One simple but novel feature of Agar that has not been explicitly
described in other CAA systems is the notion of hierarchical
testing: if a given test fails, run one or more secondary tests and
return the sum of their outputs instead. This process can proceed
recursively for several levels, although there has been no use for
anything beyond two levels so far. A very common usage pattern for
hierarchical testing is for compilation: In our lower division courses
at UCR, student submissions have been required to compile with the
gcc flags -W -Wall -Werror -pedantic, which ensures
total ANSI spec compatibility and that the majority of common warnings
will halt compilation. Since it is generally considered unduly harsh
to not accept submissions simply on the basis of having warnings in
the code, if a submission fails to compile with these flags, we may
allow a secondary test (worth fewer points) that invokes the student's
makefile, and a tertiary test that manually specifies compilation with
g++ and no warning flags. Since compilation represents a significant
portion of the computational cost for grading an average assignment,
reducing the number of required compilation attempts in this way can
make things much easier, and make the top-level rubric much cleaner.
Visually, due to the limits of the wxPython widget paradigm, an
intuitive method of displaying this ``test-on-failure'' behavior has
yet to be developed. Figure 5.2.1 shows the GUI
representation of the compilation hierarchy described above.
