Initial Design Decisions

Usability was a major concern in Agar's design from the very beginning. Since the initial aim was to reduce the amount of time it took a grader to grade programming, an elegant interface was regarded as essential. The general task of grading must be easier to perform using the electronic system than when performed by hand, otherwise there is no reason to use the system. Ideally there will be a one-to-one mapping of tasks in the traditional paradigm to tasks using the tool, and each of those tasks should also see some sort of increase in efficiency. The ``clerical'' tasks involved in grading should be automated to the greatest extent possible, meaning that grading information should be exportable in a number of useful formats, with as detailed information as possible, and hopefully strong integration with any courseware systems in common use. Security is of prime importance, because the act of running a student's submission is very much an act of running untrusted code^4.3.

One of the most fundamental design decisions, and one that has stuck throughout the development of Agar and into the planning stages for Agar2, is the idea that all automated decisions must be over-rideable by the human grader. Any system that is designed with the (probably false) assumption that all error conditions have been accounted for is going to be difficult to work with, and it is the pinnacle of hubris to think that this or any other CAA tool has been so well designed as to not need^4.4 human oversight.

Another powerful and common idea is that the tool should require no alterations to student's programming behavior. It should, like all good systems, ``just work''. Some changes to assignment specification (i.e. more accurately defined specifications) are allowable, but Agar should be minimally invasive for both the students and the designer of an assignment. Similarly, the ability to work with assignments regardless of programming language is important^4.5

In the inevitable event of a grading dispute, the system should leave a sufficiently strong paper trail that even if the original grader should meet an unfortunate accident, there should be no risk of unnecessary disputes.

Finally, it was initially thought that the system should be relatively extensible. A simple interface was designed such that programs could be written to extend the tool set whenever the existing tools didn't meet their needs. A fair amount of flexibility and efficiency was sacrificed in order to make this interface as simple as possible^4.6. An interesting, and in hindsight predictable, note is that this extensibility has never been taken advantage of. Only a single user during the entire deployment lifespan has expressed any interest in utilizing the tool interface, but lost interest when it was explained that some of the existing tools could be leveraged to accomplish the desired testing. Agar2 will certainly take advantage of this insight.