This data is associated with the following research paper: What an AI-Embracing Software Engineering Curriculum Should Look Like: An Empirical Study.
Abstract: It is not possible to reliably prevent the use of artificial intelligence (AI) tools, nor would that be desirable as AI offers many benefits for students. We recommend that appropriate AI usage be taught within software engineering courses and AI tools integrated into examinations. In order to most effectively support today’s students, software engineering curricula must embrace AI.
The purpose of this study is to analyse AI-generated code, with a focus on software engineering use-cases. The seven core areas of this research are described below.
This dataset can also be accessed on the IEEE dataport: https://dx.doi.org/10.21227/4rxb-zv06
AI chatbots (ChatGPT-4, ChatGPT-3.5, Bing Chat and Bard) were used to generate code solutions to Java programming tasks (milestone assignments) taken from the 2021 presentation of the Software Engineering 2 Bachelor's course at TH Köln. Student solutions to the same assignments were stored anonymously. The differences between the AI and human written code solutions to "milestone 0" are analysed, and a simple classification model was trained to distinguish between AI and human written code. For the analysis, two approaches are compared: representing the code using manually defined features, and by OpenAI's text embedding vectors.
In order to determine how effectively AI-written code can be detected "by eye", a mixed set of AI- and human-written code solutions to the "Software Engineering 2" 2021 milestone 0 assignment was anonymised and given to two faculty members, who made predictions as to whether each solution was written by either a student or AI chatbot.
ChatGPT was used to generate Python code solutions to the HumanEval problem set. The differences between the AI and human-written canonical code solutions are analysed, and a model is trained to distinguish between the two classes. The feature and embedding representation approaches are compared.
An evaluation of the capabilities of ChatGPT at completing software engineering university course assignments. By prompting ChatGPT-4 with only the original task description and provided document comments, purely AI-written code was generated as the solution to the "Software Engineering 2" 2023 assignment, comprising of the creation of a complex eCommerce system.
Feedback from two students on their experiences with using ChatGPT during their software engineering studies. Each use-case is evaluated, and the conversation with ChatGPT is provided.
Our experiences with generating code using different chatbots, an evaluation of their strengths and weaknesses, and a set of guidelines for using AI tools to write code.
The results from a survey distributed to university students. Questions relate to general AI tool use, AI use within education, and AI use specifically for coding and software engineering.