TEACHING EXCELLENCE DATABASE

In our project named “Personalized Assessment thRough generaTed Exercises foR pRogramming coursEs” (PARTERRE), we introduce a comprehensive toolset for digital teaching material and exercise generation for programming courses. The digitalization in HE gets more and more focus, especially after the break of the pandemic in 2020. We faced several challenges in our introduction to MATLAB programming course. As our course is in the second semester, almost all of our students participate in it, which means we cannot hold the exam for everyone at the same time. We did not have time and resources to give personalized feedback throughout the semester for students for homework assignments, or the feedback was late. Additionally, students had the exams in groups, and they felt that the difficulty level of exams fluctuated. We developed a pipeline consisting of Python scripts and MATLAB live scripts to generate teaching materials problems for homework and exams and exercises to answer these needs. The current implementation works for MATLAB as the programming language and Moodle as the LMS, but it can be adapted to other languages or platforms. The underlying code is stored at GitHub to support collaborative development and versioning. With the toolset, we can generate virtually infinite variations for a given exercise, defining interchangeable parameters which the system randomly generates. This way, students get unique exam sheets, but the difficulty level is the same for everyone. For teachers, it is easier to create many similar problems and exercises and import them into Moodle, which saves a lot of time both in the construction phase of the course and for the assessments. The exercises have a built-in evaluation and feedback, encouraging students to work individually. The toolset generates MATLAB live scripts and the evaluation code in MATLAB p-codes. Students can run the evaluations but cannot see the source code of the evaluation functions. The exercise prints the feedback on the live script output with a hash code that students copy into Moodle to get subpoints. This way, the points are stored in Moodle grade book. We created a module that logs students’ activity to a server to filter out fraud attempts for online exams. We used the toolset to implement the course in the K-MOOC platform, which is open to the public. The course language is Hungarian, but most parts are also available in English, as we offer the course to our international students.

Methodology

This course is about the basic usage of the MATLAB programming language. In our University, MATLAB is a common dialect of programming; thus, several other courses rely on it so feedback obtained from both students and other teachers can be utilized to develop our teaching methods. The course is held in the second semester of the BSc as a compulsory introductory course and for international MSc students in the preparatory semester. Over the years, the number of students has increased, so the performance evaluation has become a challenge. For this reason, we established an automatic evaluation system where students get immediate feedback about the quality of their solutions. First, we used the automated evaluation system to score homework assigned by the students, but later we began to use it on the exams. Later, we obtained a lot of feedback from students about the unequal difficulty of exam problems, so we needed to make many tests of similar difficulty. Meanwhile, we realized that a system that can generate many different problem sets could be used for the students to create problem sets for practice on their own. Thus, we created a system that can produce many different problem sets with an automatic evaluation of the solutions for practice and exams. The problem generator system has been designed to create problem sets spanning the whole material of the course, thus assuring the similar difficulty of the different groups at the exams. We obtained a lot of feedback from teachers teaching courses where MATLAB is used about what competencies are needed. Based on this feedback, we adjusted the topic of problems to these needs resulting in exercises in the topic of natural sciences to students attending the faculty of Molecular bionics engineering and in the topic of technology and information technology to students attending the faculty of Computer Science engineering. We used MATLAB live scripts, the MATLAB-specific notebook format both for education and exams in the past. The main advantage of notebooks is that they make possible modifications of example codes while processing the material. However, notebooks were not too detailed because the course was kept in an in-person form. When the pandemic arrived in 2020 and made in-person classes impossible, making the course material more appropriate for independent learning became important. For this reason, we created video records accompanying the notebooks. The classes were still held in a video conference, but students could check the materials later. We needed to filter out fraud attempts during the exams. We implemented logging in the automated evaluation system, which can semi-automatically filter out basic fraud attempts.

Tools, equipment, technology used

We use the Moodle platform to organize the course material and provide up-to-date feedback for the students. The course material contains videos, handouts, live scripts with the program code shown in the video, live scripts without the code, homework assignments, and optional assignments for each topic. The videos demonstrate the new knowledge for each topic, mainly explaining the code shown in the live script and additional examples. Students get the empty lives scripts to type in the code as they follow the course. We found this useful to get more attention and activity during the classes than just giving them the live scripts already filled out with the program code. The homework assignments, which have a 40% contribution to the overall score, are graded automatically. The points are calculated by Moodle using the so-called embedded answer question type (https://docs.moodle.org/311/en/Embedded_Answers_(Cloze)_question_type), where we can give arbitrary subpoints. The Moodle calculates the point based on a hash code, which the automatic code checker generates. The hash code is displayed in the MATLAB Live script, which contains the exercise text and the solution given by the student. To have the opportunity of additional feedback and cross-checking of students’ solutions for plagiarism, they have to attach the code of their solution. The embedded answer type is advantageous as it can combine more exercises, so all exercises are in the same question for a given topic (or week). The live scripts, code checkers, and data files are distributed in a zip file for each assignment embedded in the question text using the data URI scheme (https://en.wikipedia.org/wiki/Data_URI_scheme). This way, we can generate the questions in Moodle XML format for import with scripts „offline,” embedding the required files and avoiding the clumsy Moodle interface sparing a lot of time for the teachers. Optional assignments give extra points and are more advanced problems, requiring additional exploration of the given topic. These exercises have no subpoints, and to avoid plagiarism with a higher number of complete solutions provided by the students, the offered points are less. For example, with 120 students for less than 25 complete solutions, they get 2% which is decreased by 0,5% for every 25 complete solutions saturated at 0,1% after 100. We found that these optional assignments raised the activity of the students during the course. We also use Moodle for the exams, which have an 80% distribution in the final grade. Every student gets a unique exam in the same structure as the other assignments in the course. The teachers also evaluate the exam to give additional feedback. We use the Gradebook feature to show feedback. We import feedbacks from a spreadsheet. We use the Python scripting language (https://docs.python.org/3/faq/general.html) to generate many exercises for a given type. Each exercise has many subproblems, like students solving an equation in MATLAB or plotting some data. We define variations for these subproblems and collect them in Python vectors. As for the plot, the line color variations are: blue, red, green, yellow, black, etc. We have two stages: 1) The program draws random numbers for each subproblem to select the variation the student has to solve. The draw result is saved in a file in case we have to repeat the next stage. For example, we must rerun some parts of stage two if we have a typo in the problem description text. 2) The program generates the exercise text and code for the live script, also the code checker function, which is used for the direct feedback and hash code display. Furthermore, if there are any figures in the exercise description, the script generates a code, which saves the figure. MATLAB live script (https://www.mathworks.com/help/matlab/matlab_prog/what-is-a-live-script-or-function.html) is the underlying technology to disseminate the actual course material and give students detailed, direct, and automatic feedback in assignments. A MATLAB live script can contain text fields and code fields. This is a considerable advantage compared to previously used slide shows, textbooks, and general MATLAB scripts. In a live script, students can immediately try what they learn. The text field supports markdown and equations in LaTeX format, which are essential for the engineering examples. We use MATLAB live scripts to generate the final form of questions in Moodle XML for import. Unfortunately, we could not create the MATLAB figures and the live scripts in Python because of technical reasons, so we constructed MATLAB lives scripts for that. We use GitHub to share codes and work on the program codes together. This way, we also have versioning which is very useful in collaborative code development. We use Microsoft Office tools to keep track of the weekly teachers’ meetings mostly held on Microsoft Teams. We also use excel spreadsheets to collect additional feedback for the exam and import it to Moodle. These spreadsheets are generated by the MATLAB live script used for assessment in the correct form for the Moodle import and include the subpoints. Teachers can put their additional remarks in the spreadsheet, merged with the automatic feedback. Microsoft Teams is also used in classes. We show the code with screen share, and students can type code directly on the teacher’s computer using the „Give control” feature of MS Teams.

Outcomes and outputs, main results

Our work results are a course material that can be used in a general University course and a MOOC self-placed course. The constructed scripts let us generate a practically infinite number of exercises. The course material is merged in MATLAB live scripts, where the students can run the code and read the explanation together. The course materials are available in three forms: 1) slides, 2) video, 3) live scripts. Thanks to Moodle, the course material is distributed in a very clear structure. We use the same technologies for learning and evaluation, which help students reduce stress during exams. Students get immediate and helpful feedback as they solve the exercises.

Lessons learnt

Use comprehensive technologies At the beginning of the development of the course, we used MATLAB Grader for the homework assignments, scripts and live scripts for program codes in the course material, slide shows at classes combined with MATLAB desktop to demonstrate the codes, and scripts or live scripts for the exams. We had a Wiki page for the course material. Students were often confused by the different tools, and it was difficult for them to get all the information. Some students had technical difficulties during the exam, which caused problems in evaluating their actual performance. We chose Moodle and MATLAB live scripts to provide the course materials, assessments, exams, and feedback based on student feedback. The convergence of tools makes it easier for students to find everything, and fewer students have technical difficulties during the exams. We solved the problem of handing out the unique exam sheets in Moodle with the data URI and importing the questions in the question bank from Moodle XML. Take care of your team After a few steps developing the Python and MATLAB programs generating the exercises, it turned out that we need versioning and a tool for collaborative code development. The problem was that we had to wait for each other with any minor modification of the code, and one had to merge the code by hand, which also led to errors. We chose GitHub, as we already knew it to some extent. Random is good, but It was also essential to figure out that we must save the draw result in a file for the randomized exercise generation. This way, any modification in the code checker is straightforward, and we can generate the exercises one by one, which spares a lot of time, especially for exam sheets that are unique for each student. Student solutions are full of surprises An automatic code checker is an excellent tool for many cases. Still, it cannot work when there is an error in the students' code, like a typo, wrong syntax, or missing expressions or variables. To address this problem, we used two techniques: 1) exception handling with useful feedback, 2) code text analyzer for the exam to give extra points. The exception handling was tricky, as we wanted to keep the feature in MATLAB that with a click on the message text, the interface jumps to the erroneous line in the code. We had to learn how hidden links in MATLAB error messages work. Check all terms and conditions for free tools We wanted to log students' activity during the online exam to filter out fraud attempts and teams working together (which is not allowed). We tried many things: 1) The code checker sent an email with the user ID, timestamp, and the code each time students ran the code. We used a free Gmail account. The daily free quota was depleted during the exam, so we had to change the code checker file for many students. 2) We used MS Azure Cloud storage, which worked excellently but is only free for one year. The main advantage is that it has a built-in client in MATLAB. 3) Finally, we implemented SFTP file transfer using the Java core of MATLAB and using a University server to store the log files. It is free, works on the desktop and online versions of MATLAB, and is customizable.

Adaptability and sustainability of the best practice (for other institutions)

Although we use our automatic problem set generator and automatic solution evaluation system, our method can be adapted to other scripting languages like e.g. Python, R, etc. The only component which has to be modified is the script creating the problem sets together with a reference solution which is the core of the automatic solution evaluation. In this present form, the adaptation of the script to other languages is very difficult so a future development task is to make the generator system more abstract, more general, and thus more easily adaptable to other languages. We can also think of a graphical user interface solution for problem creation.

Promotion of best practice

Tamas Zsedrovits is part of the Faculties Digital Education Committee, where the ideas are presented. Also, the best practice was introduced at the Fall Faculty meeting in October 2021. The K-MOOC application was also an opportunity to present our good practice, but only the evaluators and participating students got the information through that. Our goal is to promote further this idea with the help of the current application.

Scope and impact

• Course/department level
• Faculty level
• Cross-institutional level

At the course/department level, our method makes it possible for us to easily score both the assignments and exams of the students and give real-time feedback to them about the quality of their solutions. Our method also makes it possible for the students to generate a practically infinite number of different exercises also with real-time feedback. At the faculty level, we adjusted the topics discussed in the course material and the topic of the exercises according to the needs of other teachers teaching courses where Matlab is an important tool for programming. Our method also is used as the base of a MOOC self-placed course giving the contribution of our method to the cross-institutional level. The course is part of the K-MOOC platform hosted by Óbuda University (https://www.kmooc.uni-obuda.hu/course/114).

6.1 Digitalization

• Innovative, novel methodology in using digital tools/devices in teaching
• Innovative use of digital administration tools
• Digital skills development and assessment both general and profession-related, embedded in course design, in teaching and assessment
• Novel digital solutions (tools, frameworks, devices, tasks to enhance efficiency and motivation)

Reasoning: We developed a pipeline consisting of Python scripts and MATLAB live scripts for exercise generation in Moodle XML that we import into our course with attachments in the data URI scheme to support automation. We use MATLAB live scripts as teaching material for homework and exams with automatic feedback and assessment. We use Moodle embedded answer question type with generated hash codes to calculate and store subpoints. Thus, we connected the MATLAB live script capabilities with Moodle grading.

---

6.2 Internationalization

• Courses offered to international multicultural students both online and offline

Reasoning: The course is part of the preparatory semester for international students starting their MSc at our Faculty. The course material the contrary to the BSc course has less emphasis on the homework. The timing is also different as all the classes are in a three-week period, and the number of students is usually less than ten. The course includes the MATLAB Fundamentals online course provided by MathWorks and the exam is a mini project where the students can choose from ten different exercises.

---

6.3 Inclusion and diversity, universal design

• Alternative, flexible assessment methods for students with special needs

Reasoning: Most parts of the course material are readable for a text-to-speech synthesizer. During the last couple of years, we had two visually impaired students. Both have completed the course. They needed more time to solve the exam problems and they did get more time for it. For other parts, they could follow the course with other students.

---

6.4 Sustainability

• Special courses reflecting to UN 2030 Sustainability goals, Green Deal - mini-courses, microcredentials

Reasoning: The developed course material can be used as part of a microcredential program. The example or pilot implementation is the one which was done in the K-MOOC framework, where we introduced the course entitled "Bevezetés a MATLAB programozásba" using the methods as mentioned earlier and technologies. The course is self-paced and gives direct feedback to the students. The students can also get help through the Moodle forum or email from the instructor.

3.3 Public contact datas

Name	Email address	Website
Tamás Zsedrovits	zsedrovits.tamas@itk.ppke.hu