How to Evaluate Assessment Test Quality Using Item Analysis

An assessment can look polished, contain challenging questions and produce precise-looking scores while still measuring the intended competency poorly. Incorrect answer keys, ambiguous wording, implausible distractors and repeated item patterns can undermine the evidence employers receive.

Item analysis examines how individual questions function in an assessment. It combines expert content review with evidence from candidate responses to identify questions that may need correction, further investigation, recalibration or retirement.

Good item analysis does not mean automatically deleting every question that is easy, difficult or statistically unusual. A sound review asks whether the item measures an important competency, whether the scoring is correct, whether candidates understand the task and whether the item supports the intended interpretation of results.

This guide introduces practical classical test theory indicators, worked numerical examples, distractor analysis, reliability considerations, fairness checks and an auditable item-review workflow.

What is item analysis?

Item analysis is the systematic examination of individual assessment questions and their response patterns. For multiple-choice items, common indicators include the proportion of candidates answering correctly, the relationship between item performance and broader test performance, and the frequency with which incorrect options are selected.

For constructed-response items, reviewers may also examine rubric use, scoring consistency, partial-credit patterns and response quality.

Statistical indicators are diagnostic signals, not substitutes for subject-matter expertise.

Step 1: Establish the assessment blueprint

Item analysis begins before candidates take a test. Define the competencies being measured, intended population, assessment purpose and decisions that scores will support.

A fictional 40-question business skills assessment might use this blueprint:

This allocation is illustrative. It should be justified by the role or assessment purpose, not copied automatically across unrelated jobs.

A question that produces attractive statistics but tests an irrelevant competency should not remain merely because it discriminates well.

Step 2: Verify item content and answer keys

Before calculating statistics, check each item independently for factual accuracy, arithmetic, clarity, plausible distractors and a single defensible correct answer when single-answer scoring is intended.

Consider a fictional financial reasoning question:

Revenue is ₦500,000 and direct costs are ₦350,000. What is the gross margin as a percentage of revenue?

The correct calculation is:

• Gross profit = ₦500,000 − ₦350,000 = ₦150,000.
• Gross margin = ₦150,000 ÷ ₦500,000 × 100 = 30%.

If the answer key identifies 42.86%, the key may have confused gross margin with markup on cost: ₦150,000 ÷ ₦350,000 × 100 ≈ 42.86%.

That distinction is a content error, not a difficulty issue. No amount of response data can justify leaving an incorrect key uncorrected.

Step 3: Calculate item difficulty correctly

In classical test theory, the facility index for a dichotomously scored item is commonly the proportion of observed candidates answering correctly:

p = number correct ÷ number responding

Suppose 84 of 120 candidates answer an item correctly.

p = 84 ÷ 120 = 0.70

Thus 70% answered correctly. The item was relatively easy for that sample compared with one answered correctly by 25%.

The p-value is not a probability that an individual candidate is competent, nor is it a universal measure of difficulty. It depends on the group taking the assessment, the administration conditions and the scoring rule.

An easy item may still be essential for minimum-competency screening. A difficult item may be suitable for advanced specialist assessment. Investigate its function in context.

Step 4: Examine item discrimination

A useful item often distinguishes candidates with stronger and weaker performance on the relevant overall measure. One simple indicator compares proportions correct in higher- and lower-scoring groups:

D = p (upper group) − p (lower group)

Suppose a fictional item is answered correctly by 25 of 30 candidates in a higher-scoring group and 10 of 30 in a lower-scoring group.

• Upper-group proportion = 25 ÷ 30 ≈ 0.833.
• Lower-group proportion = 10 ÷ 30 ≈ 0.333.
• Discrimination index = 0.833 − 0.333 = 0.500.

The item shows a positive association with the overall score in this illustrative comparison.

Now consider an item answered correctly by 8 of 30 higher-scoring candidates and 18 of 30 lower-scoring candidates:

• Upper-group proportion = 8 ÷ 30 ≈ 0.267.
• Lower-group proportion = 18 ÷ 30 = 0.600.
• Discrimination index ≈ −0.333.

This negative result warrants investigation. Possible explanations include an incorrect key, ambiguous wording, a mismatch between the item and the rest of the test, or sampling variation.

A negative index does not prove the item is defective in every context. Review the actual item and the suitability of the overall score used to form groups.

Step 5: Use corrected item–total correlations appropriately

Another indicator is the correlation between an item score and the total score on the remaining items. Excluding the target item from the total avoids inflating the association through part–whole overlap.

For a dichotomous item, this may be reported as a corrected point-biserial correlation. The interpretation depends on the test's intended dimensionality, sample size and score distribution.

An item measuring an intentionally distinct competency in a multidimensional assessment may have a low correlation with the overall total without being irrelevant or incorrectly written. Review correlations against suitable subscale scores and the test blueprint where appropriate.

Do not impose a single numerical cutoff across all question formats, competencies and assessment purposes.

Step 6: Review distractor performance

Distractors are the incorrect options in a multiple-choice item. Their purpose is to represent plausible mistakes or misconceptions, not merely to occupy space.

Consider this fictional response distribution among 100 candidates:

The item facility is 65/100 = 0.65.

Option D attracted only two responses. This may suggest that it is implausible, but the conclusion depends on the item, candidate population and purpose. An infrequently chosen distractor is not automatically harmful.

Investigate whether distractors reflect realistic errors, whether any incorrect option could reasonably be defended and whether the correct answer stands out through length, wording or grammar.

A misleading distractor example

If the stem asks for gross margin and a distractor gives the correct markup-on-cost figure, that distractor may be diagnostically useful: it represents a plausible conceptual confusion.

If two options both give the correct gross margin after reasonable rounding, the item is ambiguous and requires correction.

Step 7: Distinguish omitted from not-reached responses

A blank answer may reflect uncertainty, an instruction problem, an accessibility barrier, a technical interruption or a candidate running out of time.

Not-reached items at the end of a speeded test should not be interpreted identically to intentionally skipped items without examining administration conditions.

Review omission patterns by item position, device type and session characteristics where appropriate and lawful. An unexpectedly high omission rate may reveal a time-limit or interface problem rather than a competency gap.

Step 8: Examine reliability without misusing coefficients

Internal consistency indicators such as Cronbach's alpha can contribute evidence about score reliability under suitable assumptions. They do not establish validity and are not universally appropriate for multidimensional assessments.

A test combining numerical reasoning, communication and business judgement may deliberately measure several distinct competencies. One overall alpha coefficient can be misleading.

For human-scored tasks, inter-rater reliability or agreement may be more relevant. For alternate forms or retakes, score comparability and stability require attention.

Reliability evidence should be matched to the intended score use rather than selected because a particular coefficient is easy to calculate.

Step 9: Review fairness and potential item bias

Items may create irrelevant barriers through unfamiliar cultural references, inaccessible visual presentation, unnecessary language complexity or assumptions unrelated to the target competency.

Differential item functioning (DIF) methods can identify items that behave differently across groups after conditioning on an appropriate measure of ability. DIF is a statistical flag requiring substantive review, not automatic proof of unfairness.

Small samples can make subgroup estimates unstable. Protect privacy, follow applicable legal requirements and obtain qualified technical review for consequential analyses.

A group difference in raw item accuracy alone does not establish DIF or unlawful discrimination.

Step 10: Detect repetition and unintended clues

A question bank may contain items that differ only in surface wording while testing essentially the same response pattern. Repetition can inflate familiarity and weaken coverage.

Review for:

• Exact and near-duplicate stems.
• Repeated numerical templates with identical solution logic.
• Recycled distractors that reveal the correct answer.
• Consistently longer correct options.
• Repeated answer-position patterns.
• Clues carried from one item into another.
• Repeated exposure across retakes without form-comparability controls.

Similarity detection can help identify candidates for review, but a human reviewer should distinguish legitimate competency overlap from unacceptable duplication.

Step 11: Apply a documented item decision framework

An item should not be approved or rejected solely because one statistic crosses a threshold. Use combined evidence.

A re-key decision may require rescoring previous administrations. Employers should define how consequential past decisions will be reviewed when scoring defects are discovered.

Worked example: A five-item review meeting

Suppose a review team examines the following fictional indicators. The discrimination values are illustrative and should not be treated as a validated dataset.

The proposed actions are provisional. The team must examine actual item content, scoring rules and the adequacy of the sample before final decisions.

Notice that Q01 is not automatically deleted for being easy, and Q03 is not automatically deleted merely for being difficult. The decision is driven by evidence and intended use.

How much data is enough for item analysis?

There is no single sample size that guarantees reliable estimates for every item and model. Precision depends on item facility, scoring format, population, intended analyses and consequences of the decision.

A small pilot can identify obvious errors and usability problems. It usually cannot support strong claims about subgroup fairness, stable discrimination estimates or sophisticated item response theory parameters.

When data are sparse, record uncertainty, prioritise independent content review and gather additional evidence before making high-stakes interpretations.

An ongoing assessment quality workflow

1. Specify: Record competency, intended use and question blueprint.
2. Review: Independently verify content, key and distractors.
3. Pilot: Check instructions, accessibility, timing and platform behaviour.
4. Analyse: Calculate suitable item and score statistics.
5. Investigate: Examine unusual indicators with content experts.
6. Decide: Retain, revise, re-key, quarantine or retire with reasons.
7. Re-test: Evaluate revised content and form comparability.
8. Monitor: Recheck performance after exposure, population or role changes.

Keep a stable item identifier and version history so analysis can be linked to the exact question and answer key presented to candidates.

Common item-analysis mistakes

Treating p-values as probabilities of candidate competence. Facility describes observed item performance, not individual job success.

Deleting every easy question. Some easy items assess essential foundational knowledge.

Using one discrimination cutoff universally. The construct, test purpose and sample matter.

Assuming high internal consistency proves validity. Reliability and validity answer different questions.

Ignoring item content when statistics look acceptable. An irrelevant item can still correlate with a total score.

Overinterpreting small subgroup samples. Apparent differences may be unstable or misleading.

Failing to rescore after a confirmed key error. Corrections may have consequences for previous candidates.

Assessment quality checklist

• Define the competency and intended use.
• Independently verify every answer key.
• Review wording, distractors and unintended clues.
• Calculate item facility with the denominator clearly defined.
• Examine discrimination using suitable methods.
• Investigate unusual omissions and time patterns.
• Evaluate appropriate score reliability evidence.
• Review accessibility and potential item bias.
• Check near-duplicates and item exposure.
• Document retain, revise, quarantine and retire decisions.
• Re-evaluate revised items and affected scores.
• Maintain version-controlled audit records.

Applying item analysis to Talentrenza Assess

Talentrenza Assess uses structured assessment questions and skill categories. Maintaining a high-quality bank requires more than checking that every question has an answer and explanation: it requires content review, version control and appropriate performance evidence.

The examples in this article are educational and do not report measured item statistics for Talentrenza's question bank. Empirical calibration claims should be made only after suitable response data, analysis and independent quality review are available.

For complementary reading, see Assessment Test Difficulty: How to Balance Easy, Medium and Hard Questions, How to Validate Pre-Employment Assessment Tests, and Pre-Employment Assessment Pass Marks and Scoring Criteria.

Frequently asked questions

What is item analysis in recruitment assessments?

It is the systematic review of individual questions using content evidence and response data to understand their accuracy, difficulty, discrimination, fairness and contribution to score interpretation.

Does a p-value of 0.80 mean an item is 80% difficult?

No. For a dichotomous item, it generally means 80% of the observed sample answered correctly, so the item was relatively easy for that group.

What does negative item discrimination mean?

It means the item was answered correctly more often by the lower-scoring comparison group in the particular analysis. Investigate key accuracy, wording, construct alignment and sampling before deciding what to do.

Should distractors selected by very few candidates always be removed?

No. Low selection can be a useful diagnostic signal, but review the content and purpose before revising an option.

Can item analysis prove a test predicts job performance?

No. Item analysis contributes to test-quality evidence. Predictive claims require separate, appropriately designed validity studies.

How often should a question bank be reviewed?

Review after material content or delivery changes and at intervals proportionate to use, exposure, consequences and available evidence. There is no universal review frequency.

Conclusion

Item analysis is most useful when statistics and professional judgement work together. A high-quality question should be accurate, relevant, understandable, appropriately challenging and capable of supporting the interpretation made from its score.

Employers and assessment designers should use item statistics to ask better questions, not to automate every quality decision. The strongest review process preserves evidence, traceability and the willingness to correct defects before they affect consequential decisions.