ICAP.ID

An operator moving to competency-based training is required to have a programme that assures instructor concordance — an Instructor Concordance Assurance Programme.

The authority may set a minimum acceptable level of concordance by fleet, by group of instructors, by individual competency or by operator according to its risk.

A table of final grades will not meet such a requirement. It calls for standardised material, independent assessment, blinding and a reproducible calculation.

The practical question is simple: do different instructors assess the same material in the same way, in which direction do they diverge, and does their assessment match the reference the organisation has adopted.

ICAP.ID gives assessors the same standardised material and measures the properties of their assessment.

Inter-rater agreement, application of criteria and scale, conformity to the organisation’s reference and the stability of the result over time are computed separately.

These properties are never combined into one overall score.

The working cycle has three stages: calibration measures the current state, standardisation removes the divergences found, re-assessment checks the result.

In EBT.ID the calibration loop is built into the programme. As a separate product ICAP.ID applies where EBT.ID is not in use: qualification checks, type training, line and command training, and instructor standardisation.

• The domain as the unit of configuration and isolation: its own methodology, criteria and scales, with eight scale types supported.
• Four kinds of reference: consensus, regulatory, objective, and the absence of a reference.
• A reference may be set as a range or as a set of admissible decisions.
• Server-side blinding: until a participant signs their own form they see neither the reference nor other people’s answers, whatever their position.
• Anchor tasks and control of reference drift over time.
• An assessor profile, corrective measures, re-checking and a decision on whether earlier grades need review.

The system needs standardised material, the independent grades of the participants and the methodology of the matching domain.

The work requires no training platform, HR system or external storage.

The calculation core runs on a prepared snapshot of observations and a methodology version, without reaching into the operational database. The same snapshot has to yield the same result.

Eleven measure passports are used, each naming the source of its formula: Gwet’s AC1 and AC2, ordinal Krippendorff’s alpha, the Jaccard coefficient for set intersection, three ways of assessing conformity to a reference, and an objective binary reference.

Confidence intervals are computed by Gwet’s linearisation.

Agreement and conformity to the reference are assessed separately. A group may show high agreement and at the same time diverge systematically from the reference.

That state calls for a diagnosis of its own: whether the assessors need standardisation, the reference needs revision, or the methodology needs clarification.

A reference is set as a range wherever a single point value has no methodological ground.

Applying a measure to an incompatible scale type is blocked before the calculation runs.

There is no summary rating of an assessor and no ranking by score. A rating that carries consequences drives assessors towards the middle of the scale, and the instrument stops measuring what it was built for.

The arithmetic mean of an ordinal grade is not used as an indicator of quality. The direction of divergence is expressed as the share of deviations to each side.

A single task yields no conclusion about a settled bias. While the volume of observations stays below the methodological minimum, no direction is declared.

The system takes no qualification or personnel decisions.

ICAP.ID is not intended for general expert assessment outside the aviation scope it states. Risk classification, process audits and supplier assessment are no part of the product.