International Higher Education Teacher Award ^2024/25

AssessMe is an AI-powered solution addressing the challenge of academic integrity in programming education, particularly in introductory courses, where generative AI tools like LLMs can produce varied and undetectable assignment solutions. Traditional plagiarism detection methods are ineffective, leading to the cancellation of homework and similar assignments created outside faculty supervision. AssessMe shifts the focus from the final solution to the creation process, recording data such as code changes, keystrokes, and IDE activity via integration with Visual Studio Code. Using ML and AI, AssessMe analyzes this data to generate learning metrics that predict work originality, identify learning difficulties, assess group project contributions, and detect pass/fail outcomes. For instance, if a homework assignment typically requires several hours of activity, students using LLM-generated solutions often complete it in significantly less time, with more confident writing patterns, such as minimal line removals. By analyzing expected writing time and comparing it to group averages, AssessMe clusters students based on specific metric ranges, flagging cases for further review or inviting students to explain their code. Fully implemented at RIT Croatia, covering all first-year introductory programming courses, the tool has been embraced by students, especially high performers, who value recognition for their effort. Faculty benefit from actionable insights, enabling fair, efficient assessments of unsupervised assignments. AssessMe’s outputs include detailed learning metrics, scalable data analysis, and support for process-focused evaluations, ensuring academic integrity and enabling digitalization in education. Its innovative approach bypasses limitations of plagiarism detection by analyzing the student’s workflow, fostering accountability and enhancing learning outcomes. Designed for scalability, AssessMe plans to expand compatibility beyond Visual Studio Code, offering a robust framework for maintaining academic integrity in programming education globally.

Methodology

Planning From day one, our goal was to develop a tool that is easy to use, simple to install, and capable of analyzing unsupervised student work. The initial development began during the COVID-19 pandemic when existing tools proved inadequate for handling remote education challenges. The emergence of AI tools like ChatGPT introduced new dimensions of complexity, reinforcing the importance of AssessME in addressing these challenges. A critical design decision was ensuring that the tool supervises student work without invading their privacy. As a plug-in within the programming IDE, AssessME focuses solely on the assignment process, without direct awareness of external influences. The post-analysis phase tackles the challenge of estimating the degree of external assistance—whether from other solutions, large language models, or even peers—by analyzing the creation process and the reusability of work, which differs significantly when done independently. Implementation The implementation began with the development of a plug-in for Visual Studio Code (VSCode), an IDE widely used in our programming courses. During code creation, the plug-in generates encrypted logs of activities within the IDE, such as code content changes, keystrokes, and window events (e.g., minimization or maximization). All these activities are recorded into a single AssessME file. After completing their assignments, students are required to upload both the AssessME file and the source code. On the faculty side, an AssessME analyzer validates the provided source file against the AssessME file. Importantly, students are responsible for sending the AssessME file themselves, which fosters trust in the process. Review The tool underwent its first round of testing two years ago in introductory programming courses within the Web & Mobile Computing department at RIT Croatia. Since then, it has expanded to cover all first-year courses and part of the second-year curriculum. The New Media Design program has also adopted AssessME. To date, the plug-in has been downloaded over 1,000 times, representing the number of students who have used it for assignments. We maintain constant communication with students, sharing updates about new features and gathering feedback to refine the tool. Partners The development of AssessME was primarily funded through private resources, with several developers contributing to the tool's creation. RIT Croatia is a key partner, providing a free test environment for implementation. In return, RIT Croatia is granted free access to the software for its courses. We have also initiated partnerships with several teaching institutions in Croatia, where pilot projects are scheduled to begin in the spring. Additionally, AssessME has been presented to our main campus in Rochester, where it received promising feedback for potential adoption. Different Roles and Actors The development process involved multiple stakeholders: Developers: Built the plug-in and analyzer tools while ensuring data security and privacy. Students: Provided invaluable feedback as early adopters and users of AssessME in real-world assignments. Faculty: Acted as evaluators, using the analyzer tool to validate student submissions and ensure academic integrity. Institutional Partners: Supported pilot projects and shared insights for broader adoption. Through collaborative efforts, private funding, and institutional partnerships, we aspire for AssessME to emerge as a pivotal tool in modern programming education, fostering both integrity and innovation.

Tools, equipment, technology used

To implement AssessME, we developed a custom plug-in for Visual Studio Code (VSCode) that collects detailed logs of coding activity during assignments. The logs are fully encrypted to ensure data integrity and security, preventing students from altering the records. The size of the log file depends on the AssessME settings, averaging around 1 MB per hour of activity. This lightweight file design ensures scalability while maintaining detailed data for analysis. On the faculty side, an AssessME analyzer processes the encrypted logs alongside the submitted source code. This tool provides various learning metrics, such as coding time, the number of lines added, modified, or removed, and other detailed statistics. Additionally, the analyzer generates specialized metrics and visualizations, offering a deeper understanding of the student's coding process. One notable innovation is the "confidence" metric, which evaluates the consistency of coding behavior. High confidence may indicate either a strong familiarity with the assignment's concepts or external assistance, such as copying from an external source. The analysis includes two layers: individual analysis and multi-assignment analysis. Individual Analysis focuses on metrics from a single assignment, providing insights into that specific session. Multi-Assignment Analysis employs machine learning and artificial intelligence techniques to compare data across the entire class and multiple assignments. For example, while it may be challenging to determine an appropriate coding duration for a task, multi-assignment analysis calculates the average active coding time for the class and flags outliers, such as abnormally short completion times. One innovative discovery is the correlation between active coding time and overall course performance. Analysis shows that students who fail or withdraw tend to have less than 60% of the average active coding time of their peers. This finding underscores the tool's potential to predict performance and intervene early to support struggling students.

Outcomes and outputs, main results

The implementation of AssessME has yielded several significant results and insights that highlight its importance in modern programming education. Academic Performance and Integrity The first implementation of AssessME revealed a notable 10% drop in the average course grades compared to previous years, despite the faculty, teaching materials, and course structure remaining unchanged. This drop underscores the tool's ability to provide a more accurate assessment of student knowledge, challenging the potentially inflated perceptions of competency that can arise when innovative tools are not used. Student Feedback and Acceptance We conducted multiple surveys to gauge student opinions about AssessME. The results showed that while some lower-performing students labeled the tool as "malware" or expressed discomfort about being "observed," the majority of average and high-performing students supported its use, recognizing its value in validating their work. These findings demonstrate a general acceptance of the tool among students who are genuinely engaged in learning. Data Collection and Research Over the past two years, AssessME has been used to collect data from more than 7,000 coding assignments, each paired with its corresponding AssessME file. This extensive dataset has enabled in-depth analysis and research into student coding behaviors. Based on this data, we have developed a machine learning (ML) model capable of predicting with 95% accuracy which students are likely to fail the course by week 5 (one-third of the course duration). Early Intervention Potential This predictive capability allows faculty to identify at-risk students early in the semester, enabling timely interventions and personalized support. We plan to implement this ML model in the next academic year, further improving student outcomes and fostering a supportive learning environment. Research Contributions The insights and findings derived from AssessME have been submitted for publication in academic journals, with specific focus areas including academic integrity, data-driven student assessment, and predictive analytics in education. These contributions are detailed in the "Promotion of Best Practices" section.

Lessons learnt

When implementing AssessMe, several critical aspects must be considered to ensure success while avoiding common pitfalls. 1. Clear Communication of Purpose: Clearly explain to both students and faculty how AssessMe works and its purpose in promoting academic integrity and improving learning outcomes. Misunderstandings about its goals can lead to resistance or mistrust 2. Student and Faculty Engagement: Actively involve stakeholders in the implementation process. Highlight the benefits for students, such as recognition for genuine effort, and for faculty, such as streamlined assessment and actionable insights. 3. Data Privacy and Transparency: Address privacy concerns by explaining how data is collected, used, and stored. Provide transparency about metrics and their interpretation to build trust among users. 4. Avoiding Overreliance on Metrics: While AssessMe provides valuable insights, faculty should not rely solely on automated metrics for evaluations. Complement quantitative data with qualitative assessments, such as interviews or code reviews.

Adaptability and sustainability of the best practice (for other institutions)

AssessMe is highly adaptable and can be implemented across diverse institutions, provided certain considerations are addressed to ensure sustainability and long-term impact. 1. Technology Compatibility: Currently integrated with Visual Studio Code, AssessMe’s framework can be extended to support other IDEs widely used by different institutions. We have already initiated discussions to expand compatibility, enabling broader adoption while tailoring the system to varied technological environments. 2. Scalable Implementation: The tool is designed to accommodate institutions of different sizes, from small programs to large universities. By focusing on process-based assessment rather than specific course content, AssessMe can be integrated into various curricula without significant customization. 3. Collaborative Adoption: We have begun presenting the benefits and measurable impact of AssessMe to other institutions, emphasizing its ability to maintain academic integrity, enhance assessments, and provide actionable learning metrics. Early engagement with stakeholders facilitates smoother transitions and wider acceptance. 4. Sustainability in Use: AssessMe’s long-term impact is ensured through its focus on critical challenges posed by generative AI and its ability to evolve alongside technological advancements. The data-driven insights and transparency foster continuous improvements in programming education while reducing reliance on traditional assessments. 5. Resource Commitment: To implement AssessMe sustainably, institutions must invest in training faculty, supporting technical infrastructure, and addressing data privacy. Our experience with RIT Croatia demonstrates that these investments yield significant returns in academic integrity and learning outcomes.

Promotion of best practice

To promote AssessMe, we employ a multifaceted dissemination strategy aimed at maximizing visibility and engagement among our primary target group: faculty teaching introductory programming courses. 1. Academic Publications: We share insights and results through peer-reviewed papers, such as our latest draft, "Enhancing Student Retention in Introductory Programming Courses: Leveraging Advanced Learning Validation Tools and Educational Data Mining" (https://drive.google.com/file/d/13h_B-Jtz1GhECc87fPGCCZDi65j7YmXJ/view?usp=sharing). This ensures that our work reaches the academic community and informs ongoing research. 2. LinkedIn Campaign: We plan to launch a dedicated LinkedIn campaign targeting educators and institutions. This professional platform allows us to share results, use cases, and updates while fostering direct engagement with stakeholders in higher education. 3. Workshops and Conferences: Presenting at academic and professional conferences enables live demonstrations, Q&A sessions, and networking opportunities with faculty and institutional decision-makers. 4. Institutional Outreach: We engage directly with educational institutions, providing tailored presentations on AssessMe’s benefits and impact. This includes pilot testing offers to demonstrate the system’s potential within their specific context. 5. Digital and Physical Materials: Supporting materials, such as whitepapers, user guides, and case studies, are shared through our website (https://assessme.com.hr/) and during outreach events to ensure accessibility and detailed explanations of the framework.

Scope and impact

• Course/department level
• Faculty level
• Institutional level
• Cross-institutional level
• National level
• EU/EHEA/International level

AssessMe contributes to multiple levels of education: 1. Course/Department Level: Enhances learning in introductory programming courses by focusing on the creation process, providing actionable insights to improve retention and support students. 2. Faculty Level: Automates unsupervised assignment assessment, ensuring academic integrity while reducing workload, offering fair and transparent evaluations. 3. Institutional Level: Fully implemented at RIT Croatia, aligning with academic integrity policies, supporting assessment digitalization, and fostering innovation in teaching. 4. Cross-Institutional Level: Encourages adoption by other universities, addressing shared challenges with generative AI tools and fostering collaboration. 5. EU/EHEA/International Level: Aligns with EU/EHEA goals for digitalization and integrity in education, offering a sustainable solution for evolving challenges in programming education. AssessMe scales to meet educational challenges, ensuring academic integrity and fostering impactful learning experiences.

6.1 Digitalization

• Outstanding, innovative, excellent practices of online / blended / hybrid learning
• Innovative, novel methodology in using digital tools/devices in teaching
• Digital skills development and assessment both general and profession-related, embedded in course design, in teaching and assessment
• Novel digital solutions (tools, frameworks, devices, tasks to enhance efficiency and motivation)
• Artificial intelligence and learning analytics in education and training

Reasoning: AssessMe integrates innovative digital tools to address challenges posed by generative AI in programming education. It focuses on the creation process, using AI-based analytics in VSCODE to collect data on code changes, keystrokes, and IDE activity, generating learning metrics to predict originality, detect learning difficulties, and ensure fair assessments. The tool seamlessly integrates into workflows, ensuring accessibility and promoting active student engagement by tracking and valuing their efforts during assignment creation. It also fosters ethical use of digital tools, discouraging overreliance on AI solutions while promoting coding transparency. AssessMe’s data-driven approach offers precise feedback, highlighting individual challenges while ensuring academic integrity in unsupervised tasks. AssessMe exemplifies digitalization in education, boosting student motivation, supporting faculty, and developing essential digital skills.

---

6.2 Internationalization

Reasoning: AssessMe uses AI to offer data-driven insights into student learning, enabling more effective and efficient assessments globally. It ensures a scalable, objective, and transparent approach to managing academic integrity, making it especially valuable in international settings with cross-institutional and cross-cultural collaboration.

---

6.3 Inclusion and diversity, universal design

Reasoning: AssessMe promotes inclusion and diversity by using AI to encourage creative, original problem-solving rather than reusing generated solutions. This supports an accessible learning environment, empowering students of all backgrounds to develop programming skills while maintaining academic integrity. The tool enhances equity in education by providing a personalized and effective learning experience for both students and faculty.

---

6.4 Sustainability

Reasoning: AssessMe supports sustainability by leveraging AI in research and third-mission activities. It provides valuable data, such as keystroke dynamics from over 5,000 student assignments, which one of our MS students is using to detect stress during coding. This data-driven approach enhances learning, strengthens academic integrity, and supports ongoing research, contributing to the digital transformation of education with lasting impact.

3.3 Public contact datas

Name	Email address	Website
Alan Mutka	alan.mutka@croatia.rit.edu	https://assessme.com.hr/