Assessment Test Difficulty: How to Balance Easy, Medium and Hard Questions

An assessment containing only easy questions may fail to distinguish applicants who already meet the minimum standard from those with advanced capability. An assessment dominated by extremely difficult questions may discourage candidates, produce low scores and reveal little about the skills actually required for the role.

The solution is not to impose the same easy–medium–hard percentages on every test. Difficulty should follow the assessment's purpose, the job requirements and evidence about how questions perform with the intended population.

A graduate screening assessment, a senior specialist selection exercise and a minimum-competency certification test may all need different difficulty distributions. A well-designed blueprint balances content coverage, measurement precision, fairness and candidate experience—not just the number of questions in each category.

This guide explains the difference between intended and observed difficulty, shows how to calculate item statistics and presents a practical quality-assurance process for employers designing recruitment and workforce assessments.

What does assessment question difficulty mean?

In everyday language, a difficult question is one that requires more knowledge, reasoning or effort. In assessment design, the word can refer to two related but distinct concepts.

Intended difficulty is the level assigned by content experts before the question is administered. Experts consider the competencies involved, cognitive demand, complexity of the information and expected proficiency.

Empirical difficulty is estimated from how a question performs among actual test takers. In classical test theory, a common index for a dichotomously scored item is the proportion answering correctly, often called the p-value or item facility.

A larger p-value means more candidates answered correctly, so the item is empirically easier—not harder.

These descriptions are illustrative rather than universal cutoffs. The same question can have different p-values across applicant groups, languages, educational backgrounds or job levels. Empirical difficulty is population-dependent.

Start with the purpose of the assessment

Before assigning difficulty levels, decide what the test is supposed to support.

A question is not valuable merely because few people can answer it. Its value depends on whether the response helps support the intended interpretation and decision.

Step 1: Build a competency blueprint

A blueprint maps important competencies to the number and type of questions used to assess them.

Consider a fictional 40-question assessment for a business analyst.

This example yields 25% easy, 55% medium and 20% hard questions. It is a design illustration, not a universal psychometric rule.

The proportions should be adjusted when job analysis, test purpose, empirical results or scoring requirements justify a different balance. A high-stakes minimum-competency test, for example, should focus on measurement precision near the pass threshold rather than a fixed difficulty mix.

Step 2: Define difficulty through cognitive demand

Difficulty should reflect the reasoning required, not tricks, obscurity or unnecessary reading.

For a numerical reasoning competency, three illustrative levels might be:

Easy — direct application. A product costs ₦8,000 and is discounted by 10%. What is the discount? Correct answer: ₦800.

Medium — multi-step interpretation. Revenue is ₦240,000 and direct costs are ₦168,000. What is the gross margin percentage? Gross profit is ₦72,000; gross margin is 30%.

Hard — constrained analysis. A company compares two proposals with different fixed costs, unit contributions and expected volumes. The candidate must calculate the break-even point and explain which option is preferable under stated volume assumptions.

These labels describe intended cognitive demands. They do not establish that actual candidates will experience the items as easy, medium or hard. Pilot data may contradict expert expectations.

An item can also be unexpectedly difficult because of ambiguous wording or an incorrect answer key. That is a quality defect, not evidence of sophisticated reasoning.

Step 3: Verify item accuracy before measuring difficulty

An incorrect key can make a sound question appear extremely difficult. A poorly written distractor can make a challenging concept unexpectedly easy.

Every item should undergo independent review for:

• Alignment with a specified competency.
• One defensible keyed answer when single-answer scoring is used.
• Accurate arithmetic, facts, units and assumptions.
• Plausible but unambiguously incorrect distractors.
• Clear wording and accessible presentation.
• Absence of unintended clues, including answer length patterns.
• Appropriate time and tool requirements.
• Meaningful distinction from other questions in the bank.

Difficulty analysis cannot rescue fundamentally flawed content. Content review and statistical analysis serve complementary purposes.

Step 4: Calculate the classical item difficulty index

Suppose 120 candidates complete a fictional multiple-choice question and 78 answer correctly.

The item facility is:

p = 78 ÷ 120 = 0.65

Thus 65% of the observed sample answered correctly. The item is neither automatically good nor automatically poor because its p-value is 0.65.

For a second question answered correctly by 114 of 120 candidates:

p = 114 ÷ 120 = 0.95

That item is very easy for the sample. It may still be useful if it checks an essential safety or compliance requirement. If the test's purpose is to distinguish highly proficient candidates, however, it may contribute limited differentiation among them.

For polytomously scored items, analysts may use the mean proportion of maximum available points, but interpretation requires attention to the scoring rubric.

Step 5: Examine discrimination as well as difficulty

Item discrimination considers whether a question distinguishes between people with higher and lower levels of the relevant overall performance. An item can have moderate difficulty yet discriminate poorly.

A simple illustrative upper–lower comparison uses the proportions correct among high- and low-scoring groups:

D = p (upper group) − p (lower group)

Suppose 30 higher-scoring candidates and 30 lower-scoring candidates answer an item:

Then D = 0.80 − 0.40 = 0.40.

A positive value suggests that higher-scoring candidates answered correctly more often. It is a descriptive signal, not a complete validation result. The grouping method, sample, construct, item format and test purpose affect interpretation.

If D is negative, investigate whether the key is wrong, the wording is ambiguous, the item measures something different or the sample is unusual. Do not automatically discard an item solely because one small sample produced a negative statistic.

More comprehensive analysis may use item–total correlations, preferably corrected to exclude the item itself, and appropriate item response theory models when assumptions and data support them.

Step 6: Interpret difficulty alongside test purpose

The idea that every question should have a p-value near 0.50 is an oversimplification. The optimal distribution depends on the intended decision.

If the assessment is intended to distinguish candidates near a minimum acceptable level, items should provide useful information around that standard. If the assessment is intended to identify advanced specialists, more demanding content may be justified.

A test dominated by easy items can create a ceiling effect, where many candidates achieve near-perfect scores. A test dominated by excessively difficult items can create a floor effect, where many candidates cluster near zero.

Both can reduce useful differentiation, but the remedy is not always to force a 25/55/20 or 20/60/20 distribution. Review the competency framework and the region of performance where decisions are made.

Step 7: Account for timing and speededness

An item may be answered incorrectly because a candidate lacks the competency or because they never reached it.

Where speed is not a legitimate job requirement, an overly restrictive time limit may distort the meaning of test scores.

Review:

Where speed is part of the competency, explain why and ensure administration conditions are comparable and accessible.

Step 8: Avoid predictable answer patterns

Difficulty can be undermined by test-taking clues. A candidate may choose the longest option, detect repeated wording or infer an answer from the grammar of the stem.

Good distractors should represent plausible misconceptions, not absurd alternatives inserted merely to fill space.

Review whether:

1. Correct answers are consistently longer or more detailed.
2. One option contains information that gives away the answer.
3. Distractors are obviously implausible.
4. The question inadvertently contains multiple defensible answers.
5. Repeated question families reveal the same solution.
6. Candidate retakes expose memorised answers rather than retained skill.

Randomising option order may reduce placement patterns, but it does not solve ambiguity or content duplication.

Step 9: Check fairness across candidate populations

An item that is harder for one group is not necessarily biased. Different average scores can arise for many reasons, including differences in the measured competency and sample composition.

However, items that function differently for comparable candidates may warrant investigation through differential item functioning (DIF) analysis and substantive review.

Consider whether the question includes unnecessary regional idioms, culturally specific scenarios, inaccessible charts or software requirements unrelated to the job.

Statistical subgroup analysis requires appropriate methods, sufficient samples and careful privacy protections. Do not treat a small difference or a single p-value as proof of discrimination.

Step 10: Design equivalent test forms carefully

Employers with large question banks may deliver different forms or repeated attempts. Two forms containing 10 easy, 22 medium and 8 hard questions are not automatically equivalent.

Questions within the same broad difficulty label can differ in content, discrimination and measurement information. Form assembly should consider competency coverage, item exposure, score comparability and the evidence needed for equating or linking when scores will be compared.

For retakes, changing questions without reviewing comparability can make an apparent score improvement reflect a change in form difficulty rather than increased proficiency.

Version control should record which questions and scoring rules each candidate received.

Step 11: Apply item analysis to an assessment review cycle

A practical review sequence is:

1. Design: Define the competency and intended difficulty.
2. Independent review: Verify correctness, relevance, distractors and clarity.
3. Pilot: Administer under representative conditions where feasible.
4. Analyse: Examine p-values, discrimination, timing and response patterns.
5. Investigate: Review unusual items rather than making automatic decisions from thresholds.
6. Revise or retire: Record the reason and version history.
7. Reassess: Evaluate revised items before consequential use.
8. Monitor: Recheck performance as candidate populations and jobs evolve.

Where sample sizes are limited, statistical estimates can be unstable. In that case, emphasise sound content review, cautious interpretation and a plan to gather more evidence rather than making precise claims.

Worked example: A 40-question assessment review

Suppose a fictional 40-item assessment was designed with the 10/22/8 blueprint shown earlier. After a suitable pilot, the review team groups items by their observed proportion correct.

The observed distribution is 14/18/8, not the intended 10/22/8. This does not automatically mean the blueprint failed. Expert-assigned categories and empirical p-value bands are different constructs, and the pilot population may differ from the target population.

The team should inspect individual questions, their job relevance, discrimination, candidate comments and administration conditions. A mechanical rule that removes all items below 0.40 would risk deleting legitimate advanced content.

Common mistakes when balancing assessment difficulty

Using one percentage mix for every role. The purpose and competency framework should determine the design.

Equating intended difficulty with observed difficulty. Expert judgement is a hypothesis to be tested.

Calling a low p-value “high quality”. Very few correct responses can indicate an advanced item or a defect.

Ignoring discrimination. Difficulty alone does not show whether an item contributes useful information.

Making items hard through obscure language. Construct-irrelevant complexity undermines interpretation.

Assuming equivalent category counts create equivalent forms. Score comparability requires more evidence.

Overinterpreting small pilot samples. Item statistics can be unstable when data are sparse.

Employer and assessment-designer checklist

• Define the assessment purpose and intended decision.
• Document the job-related competency blueprint.
• Assign provisional difficulty based on cognitive demand.
• Independently verify keys, calculations and distractors.
• Pilot under suitable administration conditions.
• Calculate empirical item facility using an appropriate sample.
• Examine discrimination and other relevant item statistics.
• Review timing, omissions and accessibility.
• Investigate unusual items through content and data review.
• Preserve version history and consider form comparability.
• Re-evaluate after revisions or population changes.

How Talentrenza Assess fits into difficulty planning

Talentrenza Assess uses structured skills-assessment categories and question banks. Employers designing assessments should select job-relevant competencies and consider the appropriate balance of question complexity for their intended decisions.

An easy–medium–hard label should be treated as a content-design classification unless empirical calibration evidence supports a stronger claim. The existence of a question bank, difficulty labels or a standard session length does not itself establish psychometric equivalence or validity for a particular employer's selection process.

For related guidance, see Design Pre-Employment Assessment Tests by Job Role, Pre-Employment Assessment Pass Marks and Scoring Criteria, and How to Validate Pre-Employment Assessment Tests.

Frequently asked questions

What is the best ratio of easy, medium and hard assessment questions?

There is no universally optimal ratio. The distribution should follow the job requirements, intended decision, target population and evidence from item analysis.

Is a 25% easy, 55% medium and 20% hard mix acceptable?

It can be a provisional design blueprint for a particular assessment, but it is not a global psychometric standard. The mix should be justified and evaluated empirically.

What does an item difficulty index of 0.70 mean?

For a dichotomously scored item, it means 70% of the observed sample answered correctly. It does not mean the item is 70% difficult.

Should employers remove every question answered correctly by fewer than 20% of candidates?

No. Investigate whether the item is legitimately advanced, incorrectly keyed, ambiguous or inappropriate for the intended population. Removal should follow substantive review.

Can an easy question be useful in a high-quality assessment?

Yes. Easy questions may assess essential foundational competence, improve content coverage or provide useful information near a particular standard.

Are two tests with the same easy–medium–hard distribution equivalent?

Not necessarily. They may differ in content, discrimination, precision and overall score interpretation. Comparability requires additional evidence.

Conclusion

Assessment difficulty is a measurement and design decision, not a fixed recipe. Start with the job and the purpose of the test, build a transparent blueprint, review every question for accuracy and relevance, and then compare expert expectations with empirical results.

The best assessment is not the one with the hardest questions. It is the one whose questions provide relevant, reliable and interpretable evidence for the decision employers need to make.