12 Chatbot Builder Technology: Paths to and Away from a Generative AI Powered Assessment Design Guru

Context and Purpose

This initiative explores the potential and limitations of chatbot-builder technologies within higher education, with a particular focus on their capacity to support pedagogical decision-making and assessment design. Led by Learning Enhancement and Academic Development (LEAD) at Mary Immaculate College, the project responds to increasing interest in generative AI tools while foregrounding the need for critical, ethical, and pedagogically grounded engagement.

Aims and Objectives

• Survey the literature on chatbot technology and assess its ability to support pedagogy in higher education.
• Explore the possibilities and limitations of chatbot-builder technologies.
• Work towards the creation of a custom chatbot to support assessment design in the College.
• Develop a set of recommendations for those seeking to use chatbot (builder) technology.

Initiative Overview

This project began with a desk-based exercise to explore the materials, resources, and development processes required to create a generative AI-powered agent in the form of an Assessment Design Guru. The initial focus was on investigating chatbot use in higher education and the potential of GPT builder technology to provide critical responses and pose relevant, pedagogically appropriate questions to staff designing new assessments.

Drawing on existing scholarship, the project presents a survey of how chatbot technologies have previously been used in educational contexts and what researchers have identified as strengths and limitations. From this review, a set of recommendations was developed for educators and developers interested in creating custom chatbot agents.

In addition, an auto-ethnographic approach was adopted to document the processes and challenges encountered when attempting to develop an AI-powered Assessment Design Guru. As predicted by the literature and experienced during development, limitations within chatbot-builder technology created insurmountable obstacles to realising the original vision.

As a result, the project pivoted away from the imagined Guru and instead explored the type of generative AI assistant that could realistically be created using GPT builder technology. This work has been partially realised in the form of a Generative AI Use Statement Agent. The aim of this project was not to develop a fully realised agent but rather to look at and seek to map the process of development. 

One key outcome of the project is the development of a set of thirty-six recommendations for those considering the use of chatbots or GPT builder technology. These recommendations are presented in a linked Sway report.

Impact and Evidence of Success

This project remains in its pilot phase. Planned outputs include the release of the recommendations as a sector-facing report and a set of infographics, as well as the publication of guidance on developing a generative AI-powered Use Statement Agent.

The trialling of the Generative AI Use Statement Agent with academic staff has already prompted valuable conversations about where, when, and how generative AI might be integrated into modules and programmes. While uptake data is still emerging, these reflective discussions are viewed as a meaningful early indicator of impact.

Future Directions and Sustainability

Future work will focus on further developing the Generative AI Use Statement Agent through iterative trialling and revision informed by staff feedback. While the long-term goal is to create an agent that staff feel confident using when developing generative AI use statements, an equally important outcome is encouraging deeper reflection on programme design, learning objectives, and expectations around AI use.

By supporting staff to think critically about how and why generative AI is incorporated into teaching and learning, the project aims to contribute to sustainable, ethical, and pedagogically sound innovation.

Alignment and Policy Context

• Universal Design for Learning (UDL) Principles
• EU Digital Education Action Plan
• EU Artificial Intelligence Act
• Guidelines for the Responsible Use of AI in the Public Service

Resources and Supporting Information

• Chatbot Builder Technology Sway Report

Funding and Acknowledgements

Funding: SATLE Pathfinder Funding