WF-09 — Assessment Design, Blueprinting, Moderation and Feedback

This is the largest single KPI in Pillar 2, and MBRU currently appears not to report it at all. It is also the one KPI in the framework that IoL cannot submit, cannot influence at submission time, and cannot prepare for on a schedule, because the Ministry computes it through external experts during a CAA review visit that is not timed around the OBEF cycle. The only available strategy is to be permanently ready. One component of readiness, the three-year assessment item-reuse history, cannot be created retrospectively at all, so every month without a reuse register is a month permanently lost.

1. Purpose

To ensure that every assessment on an IoL programme is blueprinted to its course learning outcomes and to the relevant QFEmirates level, designed with a method that fits its purpose and with checks on accuracy and reliability, built from items whose reuse history is recorded, marked against criteria published to learners in advance, and returned with timely and actionable feedback, and that the assessment, its blueprint, its rubric, its mark distribution and a sample of marked scripts can be produced for an external reviewer within days rather than weeks.

2. Scope

Scope statement. This process manages assessment for IoL programmes from the specification of an assessment in the course design, through blueprinting, item authoring, moderation, delivery, marking, mark-distribution review and feedback, to the archiving of the assessment and its evidence in a repository capable of producing a stratified random sample on request.

Applies to. All summative and formative assessment on PGDipHPE and MScHPE, including written assessments, portfolios, workplace-based assessments, presentations, simulation-based assessment, dissertations and capstone work.

Does not apply to. Assessment of non-credit CPD and CME activity (WF-21), micro-credential assessment where it sits outside an accredited programme (WF-22), or admission testing (WF-07). It also does not cover the academic integrity process for a suspected breach, which is WF-11, although the two workflows meet at item security.

Applicable requirements. OBEF guide v11.5 KPI 2.1, including the four-criterion rubric and the 3R sampling framework; Appendix B evidence requirements; QFEmirates level descriptors for the levels at which PGDipHPE and MScHPE sit, which IoL must confirm rather than assume; CAA programme standards for assessment; MBRU assessment policy and academic integrity policy.

The constraint that governs everything below. KPI 2.1 is a Track B KPI. MBRU submits nothing. The Ministry's external experts score the rubric during a CAA review visit, choosing the programmes and the sample themselves. There is no submission window to prepare for, no numerator to assemble, and no opportunity to improve the number after the visit. The number is decided by the state of the assessment estate on the day a reviewer opens it.

3. Trigger, boundary and endpoint

4. SIPOC

5. Accountability

Process owner. IoL Programme Director, or the role able to change assessment design across both programmes and to require moderation.

Escalation.

6. Process steps

1. Specify the assessment in the course design. Purpose, method, weighting, timing and the outcomes it will measure. [CAPTURE] assessment identity, course code, Course Ref Number, academic period. Serves 2.1 criterion 2.

2. Blueprint the assessment. Map every item or task to at least one course learning outcome, to the programme learning outcomes those CLOs serve, and to the QFEmirates level requirement for the programme. [CAPTURE] blueprint, with CLO code references from Course learning outcomes.xlsx and the QFEmirates level. Serves 2.1 criterion 1, which carries 45% of the rubric. Decision point. No blueprint, no delivery. Criterion 1 alone is worth 45% of the KPI and 2 of its 5 internal points hang on the QFEmirates level requirement specifically, which is the sub-question most institutions cannot evidence at all. [IoL to confirm] the QFEmirates level at which PGDipHPE and MScHPE are each positioned, and record it once in the programme specification rather than restating it per assessment.

3. Author or select items, and check the reuse history. Before an item is used, query the item-reuse register for its last use. [CAPTURE] item identifier, first use, every subsequent use, and the decision taken. Serves 2.1 criterion 2, worth 1.5 of 5 inside a criterion weighted 25%. This is the step that punishes delay. The rubric asks whether assessment items have been recycled across the last three years. The only way to answer is a register that has been running for three years. It cannot be reconstructed from old papers, because the question is not what was set but whether the same item recurred, and answering that retrospectively across an estate of assessments is a research project with an uncertain result.

4. Moderate before delivery. An independent moderator or external examiner confirms that the assessment measures the intended outcomes, that the method fits the purpose, that the level is right and that the marking criteria are usable. [CAPTURE] moderation record, moderator identity, date, outcome. Serves 2.1 criteria 1, 2 and 3.

5. Publish the marking criteria to learners before the assessment. [CAPTURE] the rubric as issued, the publication date, and the channel. Serves 2.1 criterion 3, worth 3 of 5 inside a criterion weighted 20%. The word that matters in the rubric is "published". A marking scheme that exists but was never given to learners does not satisfy the criterion, and a dated artefact is the only way to demonstrate that it was.

6. Deliver the assessment. Record conditions, candidate numbers and any incident. [CAPTURE] number of students assessed. Serves 2.1, which requires the number of students to be producible on demand.

7. Mark against the published criteria. [CAPTURE] marks per candidate per criterion where the rubric is analytic, and the marked script itself. Serves 2.1 criteria 3 and 4.

8. Moderate the marking and review the mark distribution. Second marking or sampling per policy, then a review of the distribution for fairness, bias and inflation. [CAPTURE] the mark distribution for the assessment, and the moderation outcome. Serves 2.1 criterion 3, distribution sub-question worth 2 of 5. Exception route. Where the distribution is skewed or compressed, record the review and the explanation. An unexplained distribution is a finding; an explained one is evidence of a working process. Do not adjust marks to produce a more attractive distribution.

9. Release results and return feedback. [CAPTURE] feedback issued date, so turnaround against the published standard is measured rather than asserted. Serves 2.1 criterion 4, weighted 10%, and contributes to 2.6.

10. Update the item-reuse register. [CAPTURE] every item used, with the assessment, the date and the course. Serves 2.1 criterion 2.

11. Archive the evidence pack. Blueprint, moderation record, published rubric with its date, the delivered assessment, the mark distribution, the candidate count and the marked scripts, all retrievable by course, academic period and assessment type. Serves 2.1 and Appendix B.

12. Produce a 3R sample on external request. On a reviewer's request, produce a sample that is Representative (proportionate across courses and academic levels), Randomized (drawn by a documented random method, not selected), and Reliable (confidentiality preserved and statistically valid), spanning a variety of assessment types. [CAPTURE] the sampling method and the seed or parameters used, so the sample can be shown to have been drawn rather than chosen.

7. OBEF data generated

Nothing in CHEDS carries any of this. The KPI 2.1 source binding is a single line in the register: there is no template, no column and no submission field. What the KPI requires is an assessment repository, and IoL does not have one today. That is the finding, and it is not a data-quality problem that can be fixed at extract time.

Capture rule. Four things must be recorded at the time and cannot be reconstructed afterwards: the item-reuse history, because the question is about recurrence over three years and only a contemporaneous register can answer it; the rubric publication date, because "published to students before the assessment" is a fact about a moment, not about a document; the feedback issue date, because turnaround must be measured rather than asserted; and the marked script as marked, because a rebuilt exemplar is not evidence. The blueprint can in principle be written after the fact, but a blueprint written after the assessment tends to describe the assessment rather than govern it, and a reviewer can usually tell.

Reproducibility test. No second analyst can reproduce KPI 2.1, because IoL does not compute it and never will. The reproducibility question is therefore replaced by a readiness question: can IoL produce, within days and without asking anyone what happened, the number of assessments for a chosen programme, their mark distributions, the number of students, and a random stratified sample of marked scripts spanning several years? Today the answer is almost certainly no for anything beyond the current academic year, and no at all for the item-reuse question. Both are capture defects and both are addressed in section 8.

8. Max-score design

The rubric in full, since the whole design of this workflow follows from it.

The formula. Total weighted reviewer scores across all reviewed assessments, divided by the number of assessments reviewed, multiplied by 20, which converts the rubric's 5-point scale to a percentage. Thresholds 0, 30, 60, 90, 100.

What MBRU must be able to produce on demand during a CAA visit, for programmes the reviewers choose: the number of assessments; the mark distribution for those assessments; the number of students; and a sample of marked, graded student assessments. Reviewers may request samples spanning several years, which is why the repository must be built to hold history rather than only the current period.

What this workflow must do to reach the target.

1. Blueprint every assessment before delivery, with a hard stop. Owner: Programme Director.
2. Record the QFEmirates level for each programme once, and demonstrate level alignment per assessment. Owner: Programme Director with Quality and IQA.
3. Start the item-reuse register now, populated going forward and seeded with whatever history can be honestly established. Owner: Assessment Administrator.
4. Publish every rubric before the assessment and retain the dated artefact. Owner: Course Lead.
5. Extract and retain the mark distribution for every assessment, not only for those that look unusual. Owner: Data Steward.
6. Measure feedback turnaround, per assessment, against a published standard. Owner: Course Lead.
7. Hold marked scripts in a repository organised so a stratified random sample can be drawn. Owner: Data Steward.

[REDESIGN] actions.

Sequencing note. R1 outranks everything else in this pack on urgency, though not on point value. R4 is free and can be done this week. R3 is the enabling investment: R1, R2, R4 and R5 all generate evidence that has to live somewhere retrievable, and a repository built after the evidence starts accumulating loses the early material.

9. Indirect strategy where data cannot be collected

Category 1 applies: the Ministry holds the instrument.

KPI 2.1 is scored by MoHESR's external experts during a CAA review visit. IoL cannot collect the data, cannot submit a value, and cannot influence the number through any register, extract or submission field. The sanctioned response is to influence the input, and the input is the state of the assessment estate on an unannounced day.

The operative distinction is between being review-ready and being review-prepared. Review preparation is what institutions do when they know a visit is coming: assemble folders, tidy blueprints, retro-fit documentation. It is expensive, it is visible to experienced reviewers, and for KPI 2.1 it does not work, because the CAA visit is not scheduled around the OBEF cycle and because the three-year item-reuse question cannot be answered by preparation at all. Review readiness is the alternative: the evidence exists because the process produces it, and a request is a retrieval rather than a project.

A second, quieter consequence of Category 1. Because IoL cannot see the number, it cannot learn from it in the usual way. There is no annual value to track, no variance to explain, and no feedback loop from score to practice. The substitute is to score IoL's own assessments against the published rubric, internally, using the same four criteria and weights, as part of WF-04 annual programme monitoring. That gives IoL a number it can improve. It must never be presented as the OBEF result, and it must never be submitted anywhere, because the KPI belongs to the Ministry.

On Category 2, and why it does not apply here. Small cohorts do not affect this KPI. The reviewers sample assessments, not students, and the 3R framework governs the sample. What small cohorts do affect is the mark distribution sub-question in criterion 3: with eight or ten candidates a distribution has no shape, and a reviewer may read a compressed distribution as inflation. The response is to retain the distribution together with the cohort size and the distribution review, so the smallness is visible as context rather than inferred as a problem.

Boundary check. This workflow must never:

• back-date a blueprint, a rubric publication artefact, a moderation record or a feedback issue date;
• create item-reuse history retrospectively by inferring which items were reused, since an inferred history presented as a register is a fabricated record;
• adjust marks in order to produce a more attractive distribution, or re-scale a distribution before it is shown to a reviewer;
• select a sample by hand and present it as randomly drawn, or exclude a course, cohort or assessment type from the sampling frame because its evidence is weaker;
• present an internal rubric self-score as the OBEF KPI 2.1 result, in any paper, dashboard or committee report;
• submit an assessment quality value to MoHESR, since this is a Track B KPI that MBRU does not submit;
• assert a QFEmirates level for a programme that has not been confirmed against the programme specification.

10. Service standards

11. Records and evidence

Appendix B readiness. The honest position: today IoL could probably produce marked scripts and mark distributions for the current academic year, from the LMS and the student information system, within 15 working days. It could not reliably do so for assessments three or four years old, and reviewers may ask for exactly that. It could not produce an item-reuse history at all. And the request will arrive during a visit, on a timescale of days, not the 15 working days that Appendix B allows for a formal evidence request. This is the least ready workflow in Family B, on the largest KPI in Pillar 2.

12. Risks and controls

13. Performance measures

The readiness measures are the ones that decide this KPI. Everything else in this table is a means to them.

14. Change control