Improving Feedback on Programming Assignments

2002 SIGCSE Doctoral Consortium Submission
30 November 2001

Ken Yasuhara <yasuhara@cs.washington.edu>
Richard Anderson <anderson@cs.washington.edu>, Advisor
University of Washington, Dept. of Computer Science & Engineering

The overall aim of this project is to improve feedback on graded programming homeworks assigned in introductory computer programming courses. Although we use the University of Washington's introductory sequence (CSE 142, 143 [CSE]) as a context, we focus on aspects of our environment that apply to most introductory programming courses. By studying existing education research on grading and feedback and conducting studies of how students and graders work with homework feedback in our department, we intend to develop software to support more pedagogically effective grading of introductory computer programming homework. In contrast to past work on computer-based grading of programming homework [PS99, Pre97, JU97, KSIR94, GW85], the goal of reducing staff time spent on grading is secondary to the goal of encouraging feedback that has learning value for the student.

Problems with Homework Feedback in Introductory Computer Programming

The objectives of homework are two-fold: student learning and assessment of understanding [WA98]. We believe that how much and what kind of feedback students receive are key factors in its effectiveness in promoting learning. Good feedback goes beyond the all too common practice of just pointing out the existence of errors. It reciprocates student effort by commending good work, directing attention to important aspects of the work and course content, and motivating the student to improve. Most importantly, good feedback demonstrates the grader and staff's interest in the student's learning.

In introductory programming courses, homework is particularly important, because practice and critique are essential to programming mastery. Accordingly, students and staff spend a great deal of time completing and grading homework. However, the enormity of time spent grading does not guarantee that pedagogical goals are met. Extensive direct experience with our introductory programming courses and discussion with many instructors and TAs evince the following serious problems:

• Quantity and quality of homework feedback varies widely, partly because graders are extremely time-pressured. Graders should focus efforts on reading, evaluating, and annotating student work, but they currently spend a lot of time on other activities, e.g. sorting papers, flipping pages (Sections of programs, which can sometimes span several files and 30 pages, are often read and re-read out of order.), and computing and tabulating grades.
• We have no way of knowing how closely students read graders' annotations on returned homework. Experience shows that many students only look at the overall grade or, worse yet, do not pick up graded work at all.
• When reviewing a student's homework record, course staff currently only have a history of overall grades. If an instructor or grader could examine all of a student's past homeworks with grading annotations, he/she could recognize patterns in improvement and mistakes (and accordingly adapt future homeworks), use them as direct context for discussion, and better resolve disputes or misunderstandings about grading.

Flexible Computer-Assisted Grading for Better Feedback

These are complex problems that cannot be fully addressed by any single solution, but we propose that a software tool for computer-based navigation and annotation of programming homework promises substantial advantages in student learning from better feedback, as well as more efficient staff resource usage.

As in many courses in our department, students submit homework via the web, so all work is available in electronic form. However, graders currently use paper printouts, marking with pen. A software tool for reading and annotating homework on the computer could help graders provide higher quality feedback through features such as these:

• support highlighting and similar markings, accompanied by typed annotations. This would guarantee legibility and improve comprehensibility of grader's remarks.
• offer annotation history, allowing graders to use annotations saved from previously graded papers in the current set. Different students frequently make the same kinds of mistakes on any given homework, and graders typically tire of writing the same annotation on several or more papers in the set. Additionally, this feature would allow graders to invest time once in writing an extensive, detailed discussion of the mistake and to annotate each applicable paper with a version of the discussion tailored for the student.
• track the balance of positive vs. negative feedback given on each paper and across all papers. Under time pressure, graders naturally tend to only mark mistakes and dispense with commending good work. Even if the tool could not enforce balanced feedback, simply by requiring that the grader classify annotations as positive or negative, it could report how balanced feedback has been so far.
• automate navigation of program code, calculation and recording of grades, and other tasks currently done manually. Some graders report that such tasks occupy more than half their grading time. Reducing this would leave more time for constructive feedback.
• allow returning graded, annotated homework on paper and/or via the web, displaying annotations in direct context and hyperlinking for navigation within homework files and to relevant resources chosen by the grader. This would allow more comprehensible feedback for the student, with web feedback promising additional advantages, including more timely return of graded work (critical in our fast-paced courses) and convenience. If appropriate, course staff could to track web access of the graded work to infer how much attention the student is giving it, although relevant privacy issues must be examined closely before doing so.
• maintain electronic copies of graded homework for later reference by student and course staff. (See previous subsection for potential advantages.)

Assessing Effectiveness & Possible Extensions

Recent informal trials of using the EQuill web annotation tool [EQ] (which offers only the first of the features listed above) for computer-based annotation for grading in CSE 142 have received positive feedback from students and graders alike. Preliminary user tests with a prototype yielded encouraging results. More formal methods will be needed to assess our tool's usefulness. With the cooperation of instructors, students and course staff will be surveyed about feedback and the grading process, comparing sections that are and are not using the tool.

Looking further ahead, in addition to UW CSE, at least one other institution's computer science department has already expressed interest in usage trials when the tool is ready, an invaluable opportunity to test the tool's general usefulness. Similar trials could be conducted in conjunction with UW CSE's distance learning efforts at remote community colleges. The tool might also aid in grading in other disciplines, given many of the motivations and features of the tool are not unique to computer science.

Even just at UW, however, with hundreds of students enrolled in 142 and 143 each quarter, perhaps most exciting is the wide potential impact of the tool. Most graders lack the time or interest to study what education researchers recommend about grading, but the hope is that using a tool designed according to and incorporating this knowledge can improve the quality of their feedback.

Current Stage in Program of Study

I am a fourth-year graduate student in computer science. I completed a Masters project in computational biology but have switched advisors to explore my interests in computer science pedagogy. I am currently serving as instructor for the department's CS 2 course in C++, CSE 143, a large lecture course with about 230 students.

Participation in Doctoral Consortium

Although my interests in teaching computer science are longstanding, I have only recently decided to try to formally focus my Ph.D. work in this area. I hope this experience will help me answer the following questions:

• What subfield-specific advice do more experienced researchers in computer science education have for students new to this subfield?
• What methods exist for evaluating the effectiveness of a new tool or method in computer science pedagogy?
• What problems are other graduate students interested in this subfield examining, and what are their career ambitions?
• Given that my current career interests point toward introductory computer science instruction at the college level, what are the advantages and disadvantages of choosing a dissertation topic in computer science education, as opposed to a more "traditional" subfield.
• Is this project substantial enough to be the focus of a doctoral dissertation?