Guidelines for Academic Calibration

Establishment: Academic Senate, 4 May 2016
Last Amended: Academic Senate,
Nature of Amendment:  
Date Last Reviewed:  
Responsible Officer: Deputy Vice-Chancellor (Academic)

Introduction

The Innovative Research Universities (IRU) commenced the Academic Calibration Project (ACP) in 2013, with the first units being calibrated over 2013-2014.

The calibration process involves a topic coordinator at one university selecting a sample of 12 marked assignments or exam scripts in a capstone topic or core topic and sending them, together with the topic guide and related information about the topic and the course in which it is offered, to an academic staff member in a corresponding discipline at another university for review. The staff member at the second university (the calibrator) writes a report that indicates whether the original grades given for those assignments appear too high or too low and whether the assessment requirements appear appropriate. The students are not identified and their grades are not changed — the intention of the report is to provide advice to the topic coordinator on the design and assessment of the topic, which in turn should reflect the design of the course.

Purpose of Academic Calibration?

All Higher Education Providers are required to undertake a form of external benchmarking of their academic programs:

Higher Education Standards Framework (the ‘Threshold Standards’)
The revised Threshold Standards (which come into effect in 2017) include the following requirement:

“5.3 Monitoring, Review and Improvement

4.    Review and improvement activities include regular external referencing of the success of student cohorts against comparable courses of study, including:

b.    the assessment methods and grading of students’ achievement of learning outcomes for selected units of study within courses of study.

Flinders University’s Assessment Policy

The University’s  Assessment Policy, provides for a range of benchmarking devices including academic calibration.

Guidelines for the Selection of Topics for Calibration

To provide for a sufficiently comprehensive benchmarking process through academic calibration, at least 4 to 5 topics per Faculty should be sent for calibration annually. The following guidelines for topic selection apply:

  1. At least two topics per Faculty per year should be core or capstone topics offered in the final year of an undergraduate degree.
  2. At least one topic per Faculty per year should be a core topic in a postgraduate coursework degree.
  3. There can be up to two topics per year that are core third year topics in a major in a generalist degree (BA, BSc), the calibration of which would reflect on the performance of the discipline concerned.
  4. The nominated topics must use an assessment process that yields a substantial piece of written work (in the form of an essay, portfolio or exam script) that can be sent to an external reviewer, rather than practical work that must be assessed in person. This piece of written work should ideally be the main piece of assessment for the topic.
  5. Where external course accreditation involves a topic-level benchmarking process that is comparable in depth to calibration, the courses concerned can be exempted from the calibration process.
  6. Course or discipline areas should be rotated each year so that, over a period of time, all discipline/course areas are covered. All calibration reports must be submitted to related five-yearly course reviews. Through their selection of topics for calibration, together with external accreditation processes, Faculties and Schools should ensure that topics in all courses and disciplines are reviewed at least every five years.

Information to be provided for Calibration

The IRU-approved process for calibration requires that, to calibrate a topic, the University offering the topic must provide the calibrator with the following information:

Program

  •  Information about degree/program structure in which course is located
  •  List of program level learning outcomes
  •  Details of where the course fits in the program

Course

  •  Course profile: including the learning outcomes and assessment details
  •  Marking guide for the assessment task
  •  Information on how the learning outcomes of this course relate to the program level        learning outcomes

Assessment and Grading

  •  Details of actual core assessment task which students have completed
  •  The grading criteria used for the assessment task (marking rubric)
  •  The grade distribution of the results in that offering of the unit
  •  Grade nomenclature and categories used within the university, such as below pass,   credit, distinction and high distinction.

Topic Coordinators whose topic is to be calibrated

How to nominate a topic for calibration

Your Faculty or School Office will normally make a call for the nomination of topics for calibration each year.

Designing Topic Information Guides for Students
In designing topic guides and the provision of topic information to students, topic coordinators should bear in mind the following:

  1. Topic coordinators should review the topic guide and related materials provided to students to ensure that they make explicit the assessment requirements of the topic, and how the assessment tasks enable an evaluation of how well the student has met the topic aims and outcomes. While lecturers might provide this information to students in class, it should also be summarised in the topic guide.

  2. Where the topic is a core or capstone element of a course, the wording of the topic aims and outcomes should be aligned expressly to the aims and outcomes of the course concerned, and students should also be provided with this information via the topic guide.
  3. Topics can only be calibrated if the form of assessment yields an assignment or examination script that can be sent externally for subsequent review. It will be impractical to calibrate topics where the assessment involves practical work that requires direct observation by the assessor. Such topics cannot be nominated for calibration.

The Calibration Process and Outcomes

The calibration reports are submitted to the topic coordinator and the Dean of School. They should also be submitted to School and Faculty Teaching and Learning Committees and to the five-yearly reviews of the related courses.

Topic coordinators will be asked to prepare a report on how useful they found the calibration report and how much work was involved in preparing the materials for calibration. There can be considerable work involved in preparing material for calibration, depending on the nature of the topic. But most topic coordinators find the review process very valuable, as it provides insights into the topic’s design and assessment that they would not otherwise gain.

Calibration Honoraria

Where Flinders staff calibrate another university’s topics, on completion the University will invoice the other University for the prescribed fee and credit the amount, net of GST, to the relevant academic staff member’s consulting fund. Where a staff member does not have a consulting fund, an account will be established by the relevant finance officer. Calibration Honoraria may be deemed an exception under Clause 4.3 of the University’s Outside Professional Activities Policy, and Section 2 of the Outside Professional Activities Guidelines. Further, it is reasonable to consider the payment exempted from the application of an infrastructure levy under Clause 2.6.1 of the Cost Recovery and Pricing Guidelines.

How to nominate yourself to undertake Academic Calibration for another university

If you are interested in making yourself available for calibrating topics at other universities you can:

  • Advise the Academic Calibration Coordinator (Anna Smith, anna.smith@flinders.edu.au ), and/or
  • Respond to calls for calibrators when other universities seek nominations for calibration; such calls will normally be circulated via your Faculty or School Office.

Forms to be completed
If you wish to undertake a calibration for another university, that university will supply the forms to be completed via the Flinders Academic Calibration Coordinator.

If your topic is nominated for calibration, the Flinders Academic Calibration Coordinator, Anna Smith , will organise the forms.  

Any Questions?
Please contact the Flinders Academic Calibration Coordinator, Anna Smith (82015552) if you have any questions.