Comparing Assessment Results Across Different Test Forms: Employer Guide

Two applicants take an online assessment for the same vacancy. Each answers 40 questions, has 45 minutes and receives a percentage score. One gets 30 correct answers on Form A; the other gets 28 on Form B. Is the first applicant stronger?

Perhaps. But the scores alone do not answer the question. Form A might have been easier, the questions may have covered different parts of the skill, or the candidates may have faced different instructions and conditions. Even when a platform calls both tests “numerical reasoning,” the two forms are not automatically interchangeable.

Employers often use several versions of a test to protect question security, support retakes or accommodate different recruitment schedules. That can be sensible. The challenge is making sure that results still mean approximately the same thing when they are used to shortlist, promote or develop people.

This guide explains assessment form comparability in practical language. It shows what to check before combining scores, what specialist methods can and cannot do, and how to avoid giving candidates an advantage merely because of the version they received.

What is a test form?

A test form is a particular set of assessment questions, tasks, instructions and scoring rules administered as a unit. Two forms can measure the same broad skill while using different questions. Forms may also differ in order, presentation, timing or permitted tools.

Changing only the order of identical questions produces a different presentation, but not an independent set of content. Replacing every question may reduce direct exposure, but it creates a new challenge: do the forms still measure the same abilities at comparable difficulty?

The word form is sometimes used loosely for any random collection of questions. Employers should ask how each collection was built, reviewed and scored before assuming that a form label conveys quality.

Three levels of similarity employers should distinguish

Same topic: Both forms contain questions about percentages, tables and ratios. This does not mean they have the same balance of topics or difficulty.

Same blueprint: Both forms follow a documented plan, such as ten percentage questions, ten table-reading questions, ten ratio questions and ten data-checking questions. This is stronger evidence of content consistency, but it still does not guarantee equal difficulty.

Supported score comparability: There is appropriate evidence that scores from the forms can be interpreted on a common scale for the intended decision. That evidence may involve careful form construction, statistical linking or equating, sufficient data, and checks for differences across relevant candidate groups.

The three levels are related but not identical. A shared blueprint is important groundwork. It is not proof that a score of 75% on Form A represents the same level of performance as 75% on Form B.

A worked example: same number of questions, different outcomes

Imagine a fictional employer assessing 200 applicants for accounts assistant roles. One hundred applicants receive Form A and 100 receive Form B. Both forms have 40 equally weighted questions.

The average difference is five correct answers, or 12.5 percentage points. It is tempting to conclude that Form A is easier. That is one possible explanation, but not the only one. Perhaps Form A was given to experienced applicants and Form B to new graduates. Perhaps one session suffered an interruption. Perhaps the forms genuinely differ in difficulty.

To separate these explanations, the employer needs information about how candidates were assigned to forms, their relevant characteristics, administration conditions and item performance. A difference between two group averages is not, by itself, an equating study.

Now consider two candidates, each from a different group. A score of 30/40 on A and 28/40 on B cannot be ranked confidently merely because 75% is greater than 70%. If the forms differ in difficulty, the raw percentages may not have the same meaning.

Why equal difficulty labels are insufficient

Many question banks label items easy, medium or hard. Those labels can help authors plan a test, but they may be based on judgement rather than observed candidate responses. Two items labelled “medium” can turn out to have very different success rates for the actual applicant population.

For example, a question involving a simple discount might be easy for retail workers but unfamiliar to applicants from a different background. A table-reading item with dense wording may inadvertently measure language fluency as well as data interpretation.

An employer should therefore ask:

• Were difficulty labels assigned by writers, estimated from candidate data or both?
• Were the data collected from people similar to the intended applicants?
• Are important skill areas represented in the same proportions?
• Were the forms checked for unclear wording and unintended barriers?
• Do the timing and instructions create comparable conditions?

These questions do not require the hiring manager to become a statistician. They help identify what evidence is missing before a high-stakes comparison is made.

What test equating means—and what it does not

Equating is a specialised process intended to place scores from appropriately related forms of the same assessment on a common scale under stated conditions. It is stronger than declaring two forms equivalent because they have the same number of questions.

Equating usually requires a sound test design, suitable candidate data, appropriate statistical methods and checks that the resulting relationship is stable enough for the intended use. The term should not be used as a marketing synonym for “both forms are about the same subject.”

Sometimes a provider uses the broader term linking for methods that relate scores across assessments. Linking is not always equivalent to full equating. A relationship may help describe how scores correspond without making them interchangeable for every decision.

For employers, the useful question is not “Does the system have an equating algorithm?” but “What evidence supports treating these particular scores as comparable for this decision?”

Three common ways organisations approach comparability

1. Careful construction to a common blueprint

The assessment team specifies content, cognitive demands, question types, time limits and scoring rules. Reviewers build forms that follow the same plan, and pilot data are used where possible to identify obvious differences.

This is often the most practical first step. It reduces avoidable differences and supports quality control. However, matching blueprints does not establish statistical equivalence. A cautious employer should avoid making precise cross-form rankings without further evidence.

2. Shared anchor questions

Some testing programmes include a set of common items across forms. If those questions function similarly for the groups taking each form, they can help estimate differences between the forms.

The anchor must be chosen carefully. It should represent relevant content and perform consistently. If anchor questions have been exposed, are too easy or are unusually difficult for one group, the resulting link can be misleading.

Illustration: Forms A and B each contain 40 questions, with eight shared anchor items and 32 different items. Eight shared questions do not automatically make the forms equivalent. Their usefulness depends on design, sample size, item behaviour and the chosen statistical method.

3. Statistical models using candidate responses

Specialists may use methods from classical test theory or item response theory to estimate item and form characteristics and place scores on a suitable scale. These methods can be powerful when their assumptions, sample requirements and design are satisfied.

They also have limits. An elegant model cannot repair irrelevant questions, compromised content or an assessment that measures different skills across forms. Small or unrepresentative samples can produce unstable estimates. Employers should request plain-language explanations of the assumptions and supporting evidence rather than accepting a technical label alone.

A percentage is not always a common measurement scale

Consider a candidate who answers 24 of 40 questions correctly on Form C. Their raw percentage is 24 ÷ 40 × 100 = 60%. Another candidate answers 28 of 40 on Form D, or 70%.

These percentages describe how many questions each candidate answered correctly on their particular form. They do not, by themselves, show the relative difficulty of the questions.

A defensible comparison may require converting raw scores to a common reported scale using a documented method. But a transformed score is only as credible as the evidence behind the transformation. Employers should not invent a simple adjustment such as “add five points to everyone on Form C” merely because its average was lower.

Why not? The average could differ because the candidate groups differ, and a single adjustment may not be appropriate across low and high score ranges. A five-point adjustment that appears reasonable near the middle could be misleading at the extremes.

How a pass mark should work across forms

Suppose an employer has established a job-related minimum standard on Form A at 28 correct answers out of 40. It would be convenient to use 28/40 on Form B. Convenience is not evidence.

If Form B is meaningfully harder, a raw cutoff of 28 may exclude applicants who meet the intended standard. If it is easier, the same raw cutoff may admit applicants who do not demonstrate the same level of performance.

Where there is a sound common score scale, the standard can be represented on that scale and translated appropriately to each form. The translated raw cutoff might differ between forms. That does not mean the employer is favouring one group; it may be necessary to preserve the same intended standard.

However, a translated cutoff should not be created from guesswork. The assessment owner should document the standard-setting rationale, linking evidence, uncertainty and decision rules. When evidence is too weak, the employer may need to use a common form, obtain more data or treat scores as one piece of a wider assessment rather than a strict automatic filter.

Random question selection: what employers should verify

A platform may generate a fresh 40-question assessment by selecting items from a larger bank. This can reduce repeated exposure, but a large bank alone does not guarantee consistent test forms.

A practical review should ask whether selection is controlled for:

1. Skill coverage: each form assesses the intended job requirements in the planned proportions.
2. Difficulty: the forms are not systematically easier or harder because of item selection.
3. Quality: ambiguous, outdated or poorly performing items are excluded.
4. Question families: near-duplicate variants are not overrepresented.
5. Time demands: one form does not require substantially more reading or calculation than another unless intended.
6. Security: previously exposed items are managed appropriately.
7. Scoring: correct answers, weights and reporting rules are version-controlled.

A bank containing 400 questions may support many distinct 40-question selections, but the number of possible combinations is not evidence that those combinations are equally suitable. Even a very large pool can produce uneven forms if selection rules and item quality are weak.

A realistic small-employer decision

A growing business wants to assess 60 applicants for an operations role. It has two 40-question forms, but each form has been used by only 15 previous applicants. The employer does not have a strong basis for statistical equating.

What can it do now?

Option A: Use one secure common form for the current campaign, with consistent conditions and a plan to retire or refresh items later. This reduces cross-form differences but requires careful security management.

Option B: Use two carefully reviewed forms and avoid fine-grained rankings between them. Treat the results as broad evidence alongside a structured interview and job-relevant work sample. Record the limitations.

Option C: Pilot and collect more evidence before making high-stakes cross-form decisions. This may be the most appropriate route when the consequences of an error are substantial.

The choice depends on the vacancy, time available and the quality of existing evidence. The important point is to avoid presenting weak comparisons as precise facts.

When candidate groups differ, comparisons get harder

Suppose Form A is used for candidates in one country and Form B for candidates in another. If the results differ, the cause might include form difficulty, language, educational background, device conditions, job experience or other factors.

It would be inappropriate to assume that all differences reflect ability—or that all differences reflect unfairness. Employers should investigate whether the assessment measures the intended job skills similarly across relevant groups.

For translated forms, additional review is necessary. A literal translation can change the reading load or meaning of a question. For remote assessments, unequal access to stable internet or suitable devices can affect performance. These are administration and fairness questions as well as measurement questions.

The employer should consider the intended candidate population when reviewing comparability evidence. A relationship estimated in one group may not transfer unchanged to a very different group.

What to ask an assessment provider

Employers do not need to demand a hundred-page technical report for every routine decision. They do need answers that match the stakes. Useful questions include:

An answer such as “all forms contain 40 questions” is useful administrative information, but it is not an adequate response to a question about score equivalence.

Reporting results to hiring managers

A report should not hide important limits behind a single rank. At minimum, the assessment owner should be able to identify the form version, date, scoring method and intended interpretation.

For example, instead of writing “Candidate A is exactly five percentage points more capable,” a report might say: “Candidate A scored 75% on Form A and Candidate B scored 70% on Form B. These raw percentages are not confirmed as directly comparable; review the job-related evidence before ranking them.”

Where a validated common scale exists, the report can explain what the scale means and what degree of uncertainty remains. It should not suggest that a tiny score difference proves a meaningful difference in job performance.

Hiring managers should also understand that a score is not a complete hiring decision. A strong process can combine assessment results with structured interviews, work samples and other relevant evidence under a pre-agreed decision rule.

Four mistakes to avoid

Assuming identical length means identical difficulty. Two tests with 40 questions can cover different tasks and have different success rates.

Comparing group averages without checking group composition. A lower average may reflect candidate experience or conditions, not just form difficulty.

Treating “randomised” as “equated.” Random selection can help security but does not, on its own, place scores on a common scale.

Using a single raw cutoff across forms without evidence. If forms differ, the same raw threshold may represent different performance standards.

An employer's pre-launch checklist

• The job requirements and assessment blueprint are documented.
• Form versions and scoring keys are controlled.
• Each form has been reviewed for clarity, content coverage and time demands.
• Available difficulty and quality evidence has been examined.
• Any equating or linking claim is supported by a documented method and suitable data.
• The candidate groups and administration conditions are relevant to the evidence.
• The score scale and pass-mark rules are explained before results are reviewed.
• Small differences are not described as precise ability differences.
• Hiring managers know which comparisons are and are not supported.
• A process exists for investigating form problems, leaks and scoring corrections.

Frequently asked questions

Are two 40-question assessments automatically equivalent?

No. Equal length does not ensure the same content balance, difficulty or score meaning. Review the form blueprint and quality evidence, and seek stronger comparability evidence for high-stakes decisions.

Does using easy, medium and hard questions solve the problem?

Not completely. Difficulty labels may be subjective or based on limited data. Two forms can contain the same number of items in each category and still differ meaningfully in actual difficulty.

What is the difference between linking and equating?

Both can relate scores, but equating is a more demanding claim about interchangeability under suitable conditions. Not every linking method supports using scores as though they came from the same form.

Can we compare results from different versions of a test over time?

Only to the extent supported by the design and evidence. Changes in items, instructions, scoring or candidate populations can affect meaning. Record versions and review whether comparisons remain appropriate.

If Form B is harder, should we add marks to its scores?

Not by guesswork. An adjustment requires an appropriate method and data. Differences in group averages may reflect the people tested rather than the forms themselves.

Do we need advanced statistics for a small recruitment exercise?

Not necessarily for every decision. But when comparability evidence is limited, avoid pretending that small cross-form score differences are precise. Consider one secure common form or additional job-related evidence.

Can a platform guarantee comparable scores by drawing randomly from a large bank?

Random selection alone cannot guarantee that. The selection design, item evidence, form review and any score-linking process determine what comparisons can be supported. Confirm the actual product configuration rather than assuming a feature exists.

Next action for employers

Before using multiple forms in one hiring or promotion campaign, ask the assessment owner for a one-page form comparability note. It should identify each form, explain how the forms were built, state what evidence supports score comparison, describe any known limitations and specify how pass marks will be applied. If the note cannot be written convincingly, the process should not rely on narrow cross-form rankings.

Different test forms can make assessments more secure and flexible. They become fair decision tools only when the employer is equally careful about what the resulting scores mean.

Related reading: Assessment Retakes: How Employers Can Set Fair Rules and Use Scores, Assessment Change Control: Versions, Forms and Score Comparability, and How Employers Can Set Fair Pass Marks for Assessments. These proposed routes require final CMS verification before import.