Forty-five years of history in the field of Computer-Aided Assessment has failed to provide us with any generalized and usability-driven designs for a CAA system. Considering the relatively small size of the CSE and CAA communities, a fairly significant amount of work has taken place in the field, and on the basis of that work we can easily extract a set of guidelines that are certainly necessary, and hopefully sufficient, conditions for a CAA tool to achieve widespread acceptance. A remarkable number of difficulties both technical and psychological face the would-be implementer of such a system.
In its current incarnation, Agar has proven invaluable for many forms of grading for the users that have overcome its difficulties in interface design. With respect to our initial set of design guidelines (speed, accuracy, quality of feedback, consistency, flexibility), we have met all to some extent. For trained users, all of these requirements are met, although the act of grading using Agar alters the fundamental paradigms used by a grader. For the untrained user, speed and flexibility suffer due to the uncalibrated user models and too much being done invisibly by the system. It is these areas that we hope to improve upon most in Agar2, by allowing a more intuitive interface for the creation of a rubric and a more informative view of the steps being automatically taken on the graders' behalf.
The lessons learned from the historical review, development of Agar, and three academic quarters of support, feature requests, and user feedback on Agar have granted us an invaluable understanding of the problem domain. While usable in its current version and a valuable development in its own right, a re-imagined version of Agar could be the long awaited low-learning-curve, minimally invasive, generalized framework for computer aided assessment that has been lacking since the late 1950s.