gradex-cli

README

gradex-cli

Command line interface to gradex tool

Why?

A certain pandemic pushed all our examinations online at short notice, which meant handling a lot of PDF. In a big higher education department, it's preferable to have straightforward, robust procedures. Some of my thinking behind why PDF on its own was not enough for our needs, is here.

Essentially, paper keeps a permanent record of any student work, and any marking, allowing it to be passed from student to marker to moderator to checker without anyone being able to inadvertently delete or lose work from the previous people in the chain. This does the same, except using PDF.

Features

• Integrated workflow for preparing PDF for marking, moderating and checking.
• Headers for page listing exam, student anonymous ID, and page number.
• colourful sidebars for recording marking, moderating and checking.
• Flattens (i.e. keeps) all student and staff annotations
• Flattens (i.e. keeps) all sticky-note comments
• Uniquely tracks each page, and its processing history, with CRC32-protected data-tags
• Automatically sorts labelled pages by script or by question
• Knows where to store files that it receives at each stage
• Keeps exam exam separately
• Customisable templates
• Dynamically reconfigurable text
• Textfields and Comboboxes using acroforms
• Parallel processing for increased speed

Installation

Prerequisites

Required

Ghostscript downloads can be found here.

For Windows, choose the 64bit version.

Optional

Logging messages can read directly from logging file in $GRADEX_CLI_ROOT/var/log/gradex-cli.log. They're in JSON format, one message per line, with the latest message appearing at the bottom of the file.

For debugging and development, I prefer to be able to search the messages, and analyse the log data using Kibana, which can be installed following this guide here. I use Elastic, Logstash, Kibana and Filebeat. The installation is standard, in that Filebeat reads the logfile, passes it to logstash, which is configured with a JSON filter:

input {
  beats {
    port => 5044
  }
}
filter{
    json{
        source => "message"
    }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}" 
  }
}

Logstash passes the logging message to elastic, and then you can discover it in Kibana, using the filebeat-* index.

Usage

There are two main work flows. One for marking by script, the other for marking by question.

If it is your first time using gradex-cli, then initialise the file structure by issuing the following command

gradex-cli ingest

Ingest

In order to mark an exam, the first step is ingesting the raw pdf files with scanned student work. This is best done on a per-exam basis while you get used to the system. The system currently assumes all work is submitted through Blackboard Learn's Box, which provides a receipt in the following format .txt file

Name: Demonstrator Alpha (s0000000)
Assignment: Demo
Date Submitted: Tuesday, 21 April 2020 06:34:13 o'clock BST
Current Mark: Needs Marking

Submission Field:
There is no student submission text data for this assignment.

Comments:
There are no student comments for this assignment.

Files:
	Original filename: something from my computer.pdf
	Filename: demo-a.pdf

In the Windows demo release there are three demo PDFs and associated "fake" Learn receipts in ./demo-input.

Place these demo files in $GRADEX_CLI_ROOT/ingest and invoke

gradex-cli ingest

Your files should disappear - although if there is something wrong, they will appear back in the ingest directory, as a way of letting you know what has been rejected.

You can see the files have been ingested into $GRADEX_CLI_ROOT\usr\exam\Demo\02-accepted-receipts and $GRADEX_CLI_ROOT\usr\exam\Demo\02-accepted-papers.

We can prepare them for further processing by flattening them, and embedding information that we will use to anonymously track each page.

We require a system for swapping the known identity for an anonymous one. The identity.csv in $GRADEX_CLI_ROOT\etc\identity\ contains two columns, identity and anonymous. For the demo system, it comes with a simple version containing fake data:

You can prepare the pages by

gradex-cli flatten Demo

You can manually inspect the flattened paper directory to see the newly flattened files

ls $GRADEX_CLI_ROOT/usr/exam/Demo/05-anonymous-papers

Now you can choose whether to mark by script or by question. Let's mark by question. First we need to add labelling side bars so our talented team of labellers (who we shall call X, somewhat mysteriously) can whizz through and tell us which page has what question on it:

gradex-cli label X Demo
gradex-cli export marking X Demo

Then we can send the files we find in $GRADEX_CLI_ROOT/export/Demo-question-ready-X/ to our labllers. Once they send them back, we put them in $GRADEX_CLI_ROOT/ingest and ingest them with

gradex-cli ingest

You can manually inspect them to see that they end up in

ls $GRADEX_CLI_ROOT/usr/exam/Demo/10-question-back/X

We want to prepare a set of pages for a marker with the initials ABC, so we issue

gradex-cli sort ABC Demo
gradex-cli export marking ABC Demo

We can get the files from export, and send to our marker.

$GRADEX_CLI_ROOT/export/Demo-marker-ready-ABC/

You can try marking these files yourself, and save direct back to ingest (no need to change the filename, it will see from the hidden data what file it is). With the files back in the ingest directory after marking, we ingest again (same command as before)

The bit that handles merging questions coming back from markers is not integrated yet, so for this demo, we just manually move our files to where they will be after marking is analysed:

cp -r $GRADEX_CLI_ROOT/usr/exam/Demo/22-marker-back/* $GRADEX_CLI_ROOT/usr/exam/Demo/26-marked-ready/

Now we can prepare for moderating. The bit of the system that puts papers back into by-script files is not currently integrated. At this stage of the workflow, both by-script and by-questions processes return to the same path. With many scripts in this folder, the system automatically splits the set of scripts into a set to be actively moderated, with a green sidebar. The rest get a smaller grey "inactive" sidebar.

TODO -- FINISH THIS SECTION

Guidance to markers

Markers need only use Adobe Acrobat Reader (Free). The onedrive PDF app works on ipad, and Master PDF works on Linux. Most other viewers don't implement enough support for acroforms.

Markers:

• can use a keyboard, or stylus
• do not need to rename their file

Custom templates

For detailed information on how to customise the templates using Inkscape, see here.

TODO

• handle incoming marked/moderated/checked work
  • merge pages
  • report bad pages detected by markers
  • report results into csv, similar to this
• integrate optical check box
• integrate reporting
• integrate optical handwriting recognition
• live marking tool to show staff running averages/totals/percentage completion
• integrate tree view from here

Test coverage

ok  	github.com/timdrysdale/gradex-cli/comment	0.031s	coverage: 93.8% of statements
ok  	github.com/timdrysdale/gradex-cli/extract	0.041s	coverage: 49.3% of statements
ok  	github.com/timdrysdale/gradex-cli/ingester	14.762s	coverage: 59.1% of statements
ok  	github.com/timdrysdale/gradex-cli/pagedata	0.122s	coverage: 74.2% of statements
ok  	github.com/timdrysdale/gradex-cli/parselearn	0.010s	coverage: 87.6% of statements
ok  	github.com/timdrysdale/gradex-cli/parsesvg	15.017s	coverage: 81.6% of statements

Documentation

Overview

Copyright © 2020 Tim Drysdale <timothy.d.drysdale@gmail.com>

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.