Automated Grading Systems for Programming Assignments: A Literature Review
---
paper_title: Using testing and JUnit across the curriculum

paper_content:

While the usage of unit-testing frameworks such as JUnit has greatly increased over the last several years, it is not immediately apparent to students and instructors how to best use tools like JUnit and how to integrate testing across a computer science curriculum. We have worked over the last four semesters to infuse testing and JUnit across our curriculum, building from having students use JUnit to having them write their own test cases to building larger integration and use case testing systems to studying JUnit as an example of good application of design patterns. We have found that, based on this increased presentation and structuring of the usage of JUnit and testing, students have an increased understanding and appreciation of the overall value of testing in software development.
---
paper_title: Automated feedback generation for introductory programming assignments

paper_content:

We present a new method for automatically providing feedback for introductory programming problems. In order to use this method, we need a reference implementation of the assignment, and an error model consisting of potential corrections to errors that students might make. Using this information, the system automatically derives minimal corrections to student's incorrect solutions, providing them with a measure of exactly how incorrect a given solution was, as well as feedback about what they did wrong. We introduce a simple language for describing error models in terms of correction rules, and formally define a rule-directed translation strategy that reduces the problem of finding minimal corrections in an incorrect program to the problem of synthesizing a correct program from a sketch. We have evaluated our system on thousands of real student attempts obtained from the Introduction to Programming course at MIT (6.00) and MITx (6.00x). Our results show that relatively simple error models can correct on average 64% of all incorrect submissions in our benchmark set.
---
paper_title: Sketching stencils

paper_content:

Performance of stencil computations can be significantly improved through smart implementations that improve memory locality, computation reuse, or parallelize the computation. Unfortunately, efficient implementations are hard to obtain because they often involve non-traditional transformations, which means that they cannot be produced by optimizing the reference stencil with a compiler. In fact, many stencils are produced by code generators that were tediously handcrafted. In this paper, we show how stencil implementations can be produced with sketching. Sketching is a software synthesis approach where the programmer develops a partial implementation--a sketch--and a separate specification of the desired functionality given by a reference (unoptimized) stencil. The synthesizer then completes the sketch to behave like the specification, filling in code fragments that are difficult to develop manually. Existing sketching systems work only for small finite programs, i.e.,, programs that can be represented as small Boolean circuits. In this paper, we develop a sketching synthesizer that works for stencil computations, a large class of programs that, unlike circuits, have unbounded inputs and outputs, as well as an unbounded number of computations. The key contribution is a reduction algorithm that turns a stencil into a circuit, allowing us to synthesize stencils using an existing sketching synthesizer.
---
paper_title: ConcJUnit: unit testing for concurrent programs

paper_content:

In test-driven development, tests are written for each program unit before the code is written, ensuring that the code has a comprehensive unit testing harness. Unfortunately, unit testing is much less effective for concurrent programs than for conventional sequential programs, partly because extant unit testing frameworks provide little help in addressing the challenges of testing concurrent code. In this paper, we present ConcJUnit, an extension of the popular unit testing framework JUnit that simplifies the task of writing tests for concurrent programs by handling uncaught exceptions and failed assertions in all threads, and by detecting child threads that were not forced to terminate before the main thread ends.
---
paper_title: Programming by sketching for bit-streaming programs

paper_content:

This paper introduces the concept of programming with sketches, an approach for the rapid development of high-performance applications. This approach allows a programmer to write clean and portable reference code, and then obtain a high-quality implementation by simply sketching the outlines of the desired implementation. Subsequently, a compiler automatically fills in the missing details while also ensuring that a completed sketch is faithful to the input reference code. In this paper, we develop StreamBit as a sketching methodology for the important class of bit-streaming programs (e.g., coding and cryptography).A sketch is a partial specification of the implementation, and as such, it affords several benefits to programmer in terms of productivity and code robustness. First, a sketch is easier to write compared to a complete implementation. Second, sketching allows the programmer to focus on exploiting algorithmic properties rather than on orchestrating low-level details. Third, a sketch-aware compiler rejects "buggy" sketches, thus improving reliability while allowing the programmer to quickly evaluate sophisticated implementation ideas.We evaluated the productivity and performance benefits of our programming methodology in a user-study, where a group of novice StreamBit programmers competed with a group of experienced C programmers on implementing a cipher. We learned that, given the same time budget, the ciphers developed in StreamBit ran 2.5x faster than ciphers coded in C. We also produced implementations of DES and Serpent that were competitive with hand optimized implementations available in the public domain.
---
paper_title: Constructing a core literature for computing education research

paper_content:

After four decades of research on a broad range of topics, computing education has now emerged as a mature research community, with its own journals, conferences, and monographs. Despite this success, the computing education research community still lacks a commonly recognized core literature. A core literature can help a research community to develop a common orientation and make it easier for new researchers to enter the community. This paper proposes an approach to constructing and maintaining a core literature for computing education research. It includes a model for classifying research contributions and a methodology for determining whether they should be included in the core. The model and methodology have been applied to produce an initial list of core papers. An annotated list of these papers is given in appendix A.
---
paper_title: Web-CAT: automatically grading programming assignments

paper_content:

This demonstration introduces participants to using Web-CAT, an open-source automated grading system. Web-CAT is customizable and extensible, allowing it to support a wide variety of programming languages and assessment strategies. Web-CAT is most well-known as the system that "grades students on how well they test their own code," with experimental evidence that it offers greater learning benefits than more traditional output-comparison grading. Participants will learn how to set up courses, prepare reference tests, set up assignments, and allow graders to manually grade for design.
---
paper_title: Tool Design and Student Testing Behavior in an Introductory Java Course

paper_content:

This paper examines the effects of tool design on student testing behavior in an introductory course. Two tools are considered: BlueJ and WebCAT. A small modification was made to the BlueJ test recording interface to encourage students to engage more deeply in the testing process. A larger percentage of tests submitted by students using the modified BlueJ interface were correct. Further, the solutions they submitted contained fewer lines of code while being similarly complete and correct. Evidence is given that students using both BlueJ versions often rely on Web-CAT to validate their solution methods before testing the methods themselves. In response a new Web-CAT grading plug-in is proposed that we believe will better promote an incremental code-a-little test-a-little development style.
---
paper_title: Design and Evaluation of Automated Scoring: Java Programming Assignments

paper_content:

This paper presents a web-based automatic scoring system for Java programming assignments, and reports evaluation results in an actual programming course. The system receives Java application programs submitted by students and returns the test results immediately. The test consists of compiler check, JUnit test, and result test. The result test is very useful for assignments in elementary programming courses, because a typical program is composed of only a main method that reads/writes data from/to the standard input/output devices. The system was used and evaluated in an actual course of our university. The authors confirmed that the system is very helpful for students to improve their programming skills. Especially, many students noticed and corrected their mistakes by repeating submission of their programs again several times.
---
paper_title: Grading student programs using ASSYST

paper_content:

The task of grading solutions to student programming exercises is laborious and error-prone. We have developed a software tool called ASSYST that is designed to relieve a tutor of much of the burden of assessing such programs. ASSYST offers a graphical interface that can be used to direct all aspects of the grading process, and it considers a wide range of criteria in its automatic assessment. Experience with the system has been encouraging.
---
paper_title: Practical Programming in TCL & TK

paper_content:

From the Publisher: ::: "The world's #1 guide to Tcl/Tk has been updated to reflect Tcl/Tk 8.4's powerful improvements in functionality, flexibility, and performance. Brent Welch, Ken Jones, and Jeffrey Hobbs, three of the world's leading Tcl/Tk experts, cover every facet of Tcl/Tk programming, including cross-platform scripting and GUI development, networking, enterprise application integration, and much more." "Coverage includes: systematic explanations and sample code for all Tcl/Tk 8.4 core commands; complete Tk GUI development guidance - perfect for developers working with Perl, Python, or Ruby; insider's insights into Tcl 8.4's key enhancements - VFS layer, internationalized font/character set support, new widgets, and more; definitive coverage of TclHttpd web server (written by its creator); new ways to leverage Tcl/Tk 8.4's major performance improvements; and advanced coverage - threading, Safe Tcl, Tcl script library, regular expressions, and namespaces." Whether you're upgrading to Tcl/Tk 8.4, or building GUIs for applications created with other languages or just searching for a better cross-platform scripting solution, Practical Programming in Tcl and Tk, Fourth Edition delivers all you need to get results.
---
paper_title: Semi-Automatic Assessment of Unrestrained Java Code: A Library, a DSL, and a Workbench to Assess Exams and Exercises

paper_content:

Automated marking of multiple-choice exams is of great interest in university courses with a large number of students. For this reason, it has been systematically implanted in almost all universities. Automatic assessment of source code is however less extended. There are several reasons for that. One reason is that almost all existing systems are based on output comparison with a gold standard. If the output is the expected, the code is correct. Otherwise, it is reported as wrong, even if there is only one typo in the code. Moreover, why it is wrong remains a mystery. In general, assessment tools treat the code as a black box, and they only assess the externally observable behavior. In this work we introduce a new code assessment method that also verifies properties of the code, thus allowing to mark the code even if it is only partially correct. We also report about the use of this system in a real university context, showing that the system automatically assesses around 50% of the work.
---
paper_title: Experience using "MOSS" to detect cheating on programming assignments

paper_content:

Program assignments are traditionally an area of serious concern in maintaining the integrity of the educational process. Systematic inspection of all solutions for possible plagiarism has generally required unrealistic amounts of time and effort. The "Measure Of Software Similarity" tool developed by Alex Aiken at UC Berkeley makes it possible to objectively and automatically check all solutions for evidence of plagiarism. The authors have used MOSS in several large sections of a C programming course (MOSS can also handle a variety of other languages). They feel that MOSS is a major innovation for faculty who teach programming and recommend that it be used routinely to screen for plagiarism.
---
paper_title: Web-based grading: further experiences and student attitudes

paper_content:

This paper describes recent improvements to and experiences with Web-based grading software that has been developed by the author at Ohio University. The software, named WBGP for the web-based grading project, provides facilities to build, test, and annotate student source code with comments concerning programming style and documentation. The software produces a collection of Web-pages for each student project that describes the results of testing, the grading decisions, and the resulting score. The software is able to build student portfolios consisting of all students projects for a given course. In addition, the software is able to generate reports on the type and frequency of grading comments and the correctness of student projects. The paper describes continued experiences in web-based grading of computer science projects at Ohio University. In particular, we describe the various challenges of grading interactive student projects. Finally, this paper describes the results of a student survey concerning the web-based grading software. Students were surveyed concerning the ease of use and understanding of the generated Web-pages, their attitudes towards students portfolios, and their attitudes towards web-based grading. In general, the students responses were positive
---

Title: Automated Grading Systems for Programming Assignments: A Literature Review

Section 1: INTRODUCTION
Description 1: Discuss the need for automated grading tools in programming assignments, especially in large classes, and provide an overview of the paper's structure.

Section 2: SOFTWARE DEFECTS
Description 2: Introduce the types of errors targeted by automated grading techniques including syntax errors, logical errors, and runtime errors.

Section 3: Unit Testing
Description 3: Explain the use and significance of unit testing in automated grading systems, including its implementation and benefits.

Section 4: Sketching Synthesis and Error Statistical Modeling (ESM)
Description 4: Describe the tool based on sketching synthesis and ESM used for providing instant feedback in programming assignments.

Section 5: Peer-To-Peer Feedback
Description 5: Discuss the peer-to-peer feedback approach in grading and its advantages and limitations.

Section 6: Random Inputs Test Cases
Description 6: Describe the use of random inputs to test programming assignments and the limitations of this approach.

Section 7: Pattern Matching
Description 7: Explain how pattern matching is used to verify the correctness of student assignments and discuss its drawbacks.

Section 8: Comparison
Description 8: Provide a comparison of the different error detection techniques highlighted in the paper.

Section 9: EXISTING SYSTEMS FOR AUTOMATED GRADING
Description 9: Review the various existing automated grading systems, categorizing them into automated, semi-automated, and manual grading systems.

Section 10: Automated Grading Systems
Description 10: Detail specific automated grading systems, how they function, their advantages, and their limitations.

Section 11: Semi-Automatic Grading Systems
Description 11: Discuss examples of semi-automatic grading systems, their processes, benefits, and challenges.

Section 12: Manual Grading Systems
Description 12: Present manual grading systems, describing how they incorporate instructor and peer involvement.

Section 13: COMPARISON OF EXISTING GRADING SYSTEMS
Description 13: Provide a comparative overview of all the existing grading systems discussed in the paper.

Section 14: CONCLUSION
Description 14: Summarize the findings from the review, highlighting the limitations of current systems and suggesting directions for future work.