Moderation in lucy is pre-marking calibration: before general marking starts, every tutor marks the same small set of submissions, and the team meets to compare marks and calibrate their standards. The moderation files are calibration artefacts, not marks — nothing in them is returned to students. After the meeting, the agreed marks for the moderation-set students are entered as normal in their marking forks.

All the commands live under the moderation group:

poetry run ./lucy.py moderation --help

Convenor: choose the moderation set#

After the deadline (and after fork), propose a random set of submissions to moderate:

poetry run ./lucy.py moderation choose <assignment_name>

This skips students with extensions, without a fork, or without any pushes before the deadline, and prints a snippet to paste into course_info.yml under the assignment’s entry:

    moderation:
      - u1234567 # https://gitlab.comp.anu.edu.au/...
      - u2345678 # ...

Convenor: generate the moderation files#

poetry run ./lucy.py moderation generate <assignment_name>

This creates reports/<assignment_shortname>/moderation/<tutor_uid>-moderation.yml for each tutor listed in course_info.yml, with one entry per moderation student and one field per marking criterion. The criteria come from the assignment’s moderation_criteria list in course_info.yml (default: a single overall mark). Commit and push these files so tutors can fill them in.

Tutors: mark the moderation set#

Clone the moderation repos (they land in submissions/<assignment_name>/moderation):

poetry run ./lucy.py clone <assignment_name> --moderation

Assess every submission in the set, and enter your marks and a short comment/justification in your own moderation file, reports/<assignment_shortname>/moderation/<your uid>-moderation.yml:

u1234567:
  mark: 8
  notes: |-
    Did x very well, needs improvement on
    - this
    - and
    - this

(For a multi-criterion rubric there is one field per criterion instead of mark; a student’s overall mark is the sum.)

Your notes don’t go to students — they’re for you and the convenor at the calibration meeting. Two things not to do: don’t fill out or look at the marking file(s) in the moderation repos, and don’t commit or push anything inside submissions/<assignment_name>/moderation.

Check your file is complete and valid, then commit and push it:

poetry run ./lucy.py moderation check <assignment_name>

git add reports/<assignment_shortname>/moderation/<your uid>-moderation.yml
git commit -m "finished moderation set"
git pull --rebase   # always --rebase
git push

The calibration meeting#

Once everyone’s file is in (moderation check reports any stragglers), the team meets to compare marks:

poetry run ./lucy.py moderation summary <assignment_name>

prompts for a reference mark per student and an acceptable ± range, then reports every marker outside the range and every marker whose ranking of the set differs — the disagreements worth discussing. Pass -E <range> to skip the range prompt.

poetry run ./lucy.py moderation chart <assignment_name>

writes csv/markdown summaries and interactive marker-comparison charts to reports/<assignment_shortname>/moderation/charts/ — an overall chart of each marker’s total per student, plus a per-criterion chart for each student.

After the meeting#

The team’s agreed marks for the moderation-set students are entered the same way as all other marks: in those students’ marking forks, during normal marking. The moderation files themselves stay behind as a record of the calibration.

bars search times