Solutions
Assessment concordance analysis
A campaign on standardised material shows whether instructors assess the same material in the same way and how far their grades match the reference. Installing ICAP.ID is not required.
What we do
We prepare standardised material and run a campaign in which instructors and examiners assess the same tasks independently, seeing neither the reference nor the answers of other participants.
We then compute the measures on the ICAP.ID methodology, with no requirement to install the product.
Agreement between assessors and conformity to the reference are assessed separately. A group may be internally consistent and at the same time diverge systematically from the reference, and those two states call for different corrective action.
The methodology applies beyond EBT: to qualification checks, type training, line and command training, and instructor standardisation.
How we work
- Data
- Analysis
- Findings
- Recommendations
Form of the result
From you: the campaign participants, the current competency model and scale, and material for the tasks — session recordings, case descriptions or existing video material. We can prepare the material ourselves.
Result: a campaign report with separate measures of agreement and conformity to the reference, an analysis of the divergences, de-identified participant profiles and a list of corrective measures.
Acceptance: a review of the results with the campaign participants, which is itself the first step of standardisation.
Where data falls short: no conclusion about the direction of a deviation is drawn. A settled bias is never established from a single task.
What the airline gets
The airline receives a measured state of its assessment system: the level of concordance across the instructor corps, the nature of the divergences and their direction.
The result shows where standardisation is required, where the reference or the criteria need revision, and where the volume of data is still too small for a conclusion.
The campaign material can be used within an instructor concordance assurance programme during a move to competency-based training.