The overall usage of Agar can be demonstrated nicely with a concrete example. To begin with, we assume that student submissions are organized in a directory structure identical to that provided by the CS&E department's Turnin facility: directories look like username/ONTIME/ or username/LATE/ and contain all of the files for a student's submission. Other directory structures are easily supported using the Submission Manager, Agar merely defaults to this structure.
Initial startup of Agar is easy, simply run the agar.py script. If invoked with no arguments, it will bring up a dialog prompting you to choose ``New Workspace'', ``Load Workspace'', or ``Quit.'' The usage of these should be self-evident, shown in Figure 5.8
Beginning a new grading project requires us to select ``New Workspace.'' This action brings up a secondary dialog, asking what type of assignment we are grading: Written work (wherein Agar is used primarily for the Comment facilities and grade reporting), C/C++, or Other (which prompts for an appropriate default file mask)5.4. The final step in setup is to choose a base submission directory, the directory that contains all of the student submissions. For example, if our student submissions are contained in /home/titus/grading/as1, then we need to select that directory as the base submission directory.Once the type of submission has been identified, Agar will automatically invoke the Submission Manager on the base submission directory and identify those submissions that it can. If there are directories contained in the base submission directory that do not contain any files matching the default file mask, then a warning is raised indicating there may be problems in identifying submissions and the Submission Manager may be invoked by the grader.
At this point, the submissions should have been identified, and the act of constructing a rubric may begin. Tools are selected from the Toolbox (recall Figure 5.2) and are drag-and-dropped into the Rubric. Items in the Rubric may be re-ordered with drag-and-drop as necessary. Each rubric item may be configured by double-clicking, specifying options for that tool, changes to the filemask, etc. Point values for each rubric item are assigned by selecting the item in the Rubric and changing the text field at the bottom of the window. Note that each test can only be used to grant points (submissions start with zero points and test results increase this), as it was generally felt that Agar's rubrics should as closely correspond to an educationally accepted rubric as possible. To assign a penalty, an auto-commenting system is employed: individual tests can be configured to automatically assign a comment when a submission fails the test, or when a submission passes the test. Since comments can assign negative point values, this is the preferred method for assigning penalties. The Auto-Comment dialog interface is quite simple, and is shown in Figure 5.11
Once the Rubric is configured, the tests can be run. A progress dialog will be displayed showing the percentage of tests that have been executed and the username currently being checked, and any manual examination necessary can be done on the ``Grading'' tab.
From within the Grading tab, the grader has instant access to all of the files for each student's submission. Double clicking a file or a submission launches a viewer appropriate for the files in that submission. Upon inspection, if a submission is found to be lacking in some respect, a new Comment can be created, assigned a relevant top-level rubric item, a point value, and a description, as shown in Figure 5.12.
By default, creating a new comment assigns it to the current submission. At any point in the future, if the same problem is seen when the grader is examining another submission, drag-and-drop from the comment list to the submission pane will add the comment to the new submission. If for some reason the grader wants to work with the submission files, the context menu for each submission has an option to bring up a new terminal in the submission directory. There is also an option to re-run the rubric tests on a single submission. The majority of time spent grading programming assignments is spent performing these tasks: examining submission files, creating and assigning comments, working with submissions to see how close they were to working properly, re-running tests.When the examination of submissions is complete, individual summaries of scores and comments can be generated to text files and mailed back to the students. Students have consistently commented on how much they appreciate getting detailed feedback, and using Agar it is not particularly more difficult to give detailed feedback than it is to give no feedback at all. Of great note to the instructor is the fact that when exporting grading results to a spreadsheet, scores are not merely aggregated into a single final tally per submission. Rather, a new sheet is created that contains a column for each top-level rubric item and the score for each submission on each item. This is particularly useful in conjunction with the OMR software for checking whether the answer key was entered correctly. In such a situation, if most of the students get a particular question wrong (which can be seen in most spreadsheets by simply highlighting the column corresponding to the question), then the key can be checked to make sure the students were graded correctly. Eventually, this level of detailed output will be used in the CS&E department's ABET accreditation activities, allowing us to data mine the fine-level data and extract what students really know.
