Small Sample Sizes in Assessment Pilots: What Employers Can Conclude

An employer designs a 40-question skills test and asks eight colleagues to try it. Everyone finishes within 45 minutes. The questions seem clear, and the average score is 75%. Can the employer now tell candidates that the test is reliable, fair and suitable for hiring?

Not on that evidence alone. The eight-person trial may be useful. It can uncover broken instructions, incorrect answers, confusing wording and technical problems. But it is far too limited to support many of the stronger claims employers might want to make.

This guide explains how to use a small assessment pilot properly, how to recognise uncertainty and what to do before relying on the test for important recruitment or promotion decisions.

What is an assessment pilot?

A pilot is a trial run before an assessment is used more widely. It allows the employer to observe how the test works in practice.

A pilot can answer operational questions such as:

• Do candidates understand the instructions?
• Are all questions displayed correctly on common devices?
• Is the correct answer recorded for each question?
• Does the timer behave as intended?
• Are any questions unclear or unexpectedly time-consuming?
• Can candidates submit their responses successfully?
• Are accessibility and support arrangements working?

A pilot may also provide early information about question difficulty and score patterns. However, the strength of those findings depends on who participated, how many people participated and how the trial was organised.

Why a small sample is not automatically useless

A small pilot is often the sensible first step. Employers should not postpone every usability check until hundreds of participants are available.

For example, five participants may independently report that Question 12 refers to a missing table. That is actionable evidence of a problem. The employer can inspect the question and correct it without pretending that the five-person sample represents every future candidate.

The key distinction is between finding a clear defect and estimating how a test will perform for a much larger population.

The problem with drawing big conclusions from a few people

Small groups can produce results that change dramatically when one or two people perform differently.

Suppose eight people complete a 40-question test. Their scores are:

24, 26, 27, 28, 29, 30, 31 and 33.

The total is 228 correct answers across eight people. The average is 228 ÷ 8 = 28.5 out of 40, or 71.25%.

Now imagine that one of those participants is replaced by someone who scores 17 instead of 33. The new total is 212. The average becomes 26.5 out of 40, or 66.25%.

A single change moves the average by five percentage points. That does not prove that either average is wrong. It shows how sensitive the result is when the group is small.

An employer should avoid reporting a small pilot average as though it is the stable expected score for all applicants.

Sample size is only part of the problem

Even a large pilot can mislead if the participants do not resemble the people who will take the assessment.

Consider an employer recruiting junior accounts assistants. It asks ten experienced finance managers to trial the questions. The managers may find the questions easy because they already have extensive experience. Their results do not tell the employer how entry-level applicants will perform.

A useful pilot should consider:

• Relevant experience: Are participants similar to the intended candidate group?
• Language: Are they comfortable with the language used in the assessment?
• Device and internet access: Are they testing under realistic conditions?
• Prior exposure: Have they already seen the questions or answers?
• Motivation: Are they taking the trial seriously, knowing the result will not affect them?
• Accessibility: Have relevant access needs been considered?

A convenient group of colleagues can be helpful for checking usability, but it is not automatically a suitable group for making decisions about future applicants.

Worked example: One faulty question in a 40-question test

Suppose a pilot reveals that one multiple-choice question has two answers that could reasonably be correct. The employer checks the wording and confirms the problem.

A candidate originally received 27 marks out of 40. If the employer awards one additional mark for that item under an approved correction rule, the revised score becomes 28 out of 40.

• Original percentage: 27 ÷ 40 × 100 = 67.5%.
• Revised percentage: 28 ÷ 40 × 100 = 70%.
• Difference: 2.5 percentage points.

If the pass mark were 70%, the correction could affect the candidate's outcome. This is why answer-key checks matter even when a pilot has few participants.

The appropriate remedy depends on how the item was scored and which candidates were affected. The employer should document the issue, approve a consistent correction and review all relevant results rather than quietly changing one person's mark.

Why percentages can create false confidence

Percentages are useful for communicating results, but a neat percentage does not mean the underlying evidence is strong.

Imagine four out of five pilot participants say the instructions are clear. That is 80% of the five people. It does not mean the employer has established that 80% of all future candidates will understand the instructions.

Likewise, if no one in a ten-person pilot encounters a technical problem, the employer has learned that no problem occurred in those ten observed attempts. It has not proved that the platform has a zero failure rate.

The proper report is specific: “Ten pilot participants completed the assessment without a reported submission failure.” Avoid replacing that statement with “The platform is 100% reliable.”

How much does one response matter?

The smaller the group, the more one person changes a reported percentage.

These figures describe the contribution of one person to a simple proportion when the denominator is fixed. They are not a formula for deciding how many participants are sufficient for every assessment study.

The number needed for a meaningful study depends on the question being asked, the likely variation in results, the importance of the decision and the method used to analyse the evidence.

What employers should check in a first pilot

1. Instructions and candidate experience

Ask participants to read the instructions without extra coaching. Observe where they hesitate or misunderstand a requirement. If several people ask the same question, rewrite the instructions and test them again.

2. Question clarity

Check whether each question has one defensible correct answer, unless the task deliberately uses an approved scoring guide for several acceptable responses. Ask participants which words or details confused them.

3. Time demands

Record completion times and identify questions that require much longer than intended. Do not automatically shorten the time limit because one participant worked slowly, or lengthen it because one person needed extra time. Investigate the reason and the relevance of speed to the job.

4. Device and technical problems

Test on devices candidates are realistically likely to use. Check whether long tables, mathematical symbols and diagrams display correctly. Confirm that answers save and submissions are acknowledged.

5. Answer keys and scoring

Have a competent reviewer independently solve the questions. Check scoring rules, partial credit where applicable and the treatment of unanswered items.

6. Candidate feedback

Ask short, specific questions: “Which instruction was unclear?” is more useful than “Did you like the test?” Keep feedback separate from actual performance data.

7. Access barriers

Invite participants to report accessibility problems. A pilot with no participants using assistive technology cannot establish that the assessment is accessible to people who rely on it.

A simple pilot feedback form

Employers can use the following questions after a trial:

1. Which instruction, if any, was difficult to understand?
2. Which question appeared to have more than one correct answer?
3. Did any question depend on information that was missing?
4. Did you experience problems with the timer, saving or submission?
5. Were any tables, images or calculations hard to read?
6. Was the time available sufficient to attempt the questions? Why or why not?
7. What device and connection type did you use?
8. What one change would make the assessment clearer?

This form is a starting point. Employers should also check the assessment directly instead of relying only on participant opinions.

Why average scores cannot prove a test is fair

Suppose six candidates from one group average 72%, while six from another group average 66%.

The difference is six percentage points. That is a fact about these twelve observed scores, assuming the calculations are correct. It is not, by itself, proof that the assessment is biased or that the groups differ in job ability.

The groups may differ in prior experience, language familiarity, device conditions or other relevant factors. With only six people per group, the results can also change considerably when one participant scores differently.

A responsible employer should investigate possible barriers and job relevance, protect personal information and obtain appropriate expertise before drawing strong conclusions. Where the law permits and the programme warrants it, broader and better-designed evidence may be needed.

See the related guide How to Investigate Unexpected Assessment Group Differences for a more detailed approach.

Pilot feedback cannot establish a defensible cutoff

A small pilot can reveal unclear instructions, flawed answer keys and delivery problems, but it cannot by itself establish that a particular passing score is appropriate for a job. Avoid choosing a cutoff from the pilot mean or from the score of the strongest participant. The separate guide How Employers Can Set Fair Pass Marks for Assessments explains job-based standard setting; this guide concentrates on what limited pilot evidence can and cannot support.

Comparing two forms with small groups

Imagine five people try Form A and five different people try Form B. Form A has an average score of 31/40 and Form B averages 27/40.

The four-mark gap might reflect differences in form difficulty. It might also reflect differences between the participants, prior knowledge, motivation or testing conditions.

With such small groups, the employer should not simply add four marks to all Form B results. That adjustment would assume the observed difference represents a stable difference in difficulty, which has not been established.

Instead, review the content, confirm the scoring, record the uncertainty and plan further checks if cross-form comparisons will influence important decisions.

For more detail, see Comparing Assessment Results Across Different Test Forms: A Practical Employer Guide.

What is a reasonable pilot size?

There is no single number that makes every assessment pilot adequate. The right approach depends on the purpose.

A small group may be enough to discover a missing diagram or broken submission button. A more demanding claim—such as whether scores are consistent, whether two forms are comparable, or whether the assessment predicts later job performance—usually requires more suitable data and a stronger study design.

Employers should decide what they need to learn before choosing the sample size. They should also record what the pilot cannot answer.

If the decision is high-stakes, affects many candidates or depends on complex scoring, seek input from a qualified assessment specialist.

A staged approach for employers with limited resources

A staged process can improve quality without overstating the results.

Stage 1: Internal review. Have job experts check content, answer keys and relevance. Fix obvious errors before asking candidates to trial the test.

Stage 2: Small usability pilot. Invite a manageable group to test instructions, timing, navigation and submission. Capture specific problems and revise the test.

Stage 3: More representative trial. Where feasible, involve people who resemble the intended candidate population. Collect structured evidence on question performance and candidate experience.

Stage 4: Specialist review where needed. For important decisions, obtain appropriate technical advice on scoring, score consistency, fairness and interpretation.

Stage 5: Controlled launch. Approve the intended use, document limitations and monitor results. A launch is not the end of quality assurance.

Each stage has a different purpose. Completing Stage 2 should not be presented as completing Stage 4.

How to write an accurate pilot report

A useful report is honest about the size and composition of the trial.

Weak claim: “Our assessment is proven accurate and fair because ten people passed the pilot.”

Better statement: “Ten participants completed a usability pilot. The review identified two unclear questions and one mobile display issue. Those issues were corrected. This pilot was not designed to establish score reliability, predictive validity or fairness across candidate groups.”

The second statement is more useful because it tells decision-makers what was actually checked and what remains unknown.

A short report should record:

• The pilot's purpose and date.
• The number of participants and how they were selected.
• Whether participants resembled the intended candidate group.
• The assessment version and testing conditions.
• The problems found and corrections made.
• The findings that remain uncertain.
• The person responsible for approving the next step.

When a pilot should stop a planned launch

Some findings require immediate attention regardless of sample size.

Examples include an incorrect answer key that changes pass or fail outcomes, a serious privacy exposure, an inaccessible essential step, a broken submission process or a question that measures something unrelated to the job in a potentially harmful way.

An employer should not dismiss a confirmed defect merely because “only one person complained.” The importance of the issue depends on its nature and consequences, not just how many people noticed it.

The response may involve fixing the issue, repeating part of the trial, delaying the launch or obtaining expert review.

A four-week practical plan

This schedule is illustrative. Employers should extend it when problems remain unresolved or when the planned use requires stronger evidence.

Employer checklist: Interpreting a small pilot responsibly

• We defined what the pilot was intended to check.
• We recorded the number of participants.
• We know how participants were selected.
• We considered whether they resemble future candidates.
• We independently checked answer keys and scoring.
• We reviewed technical, accessibility and instruction problems.
• We recorded both findings and limitations.
• We did not claim that a small pilot proves validity or fairness.
• We documented corrections and retested affected functions.
• We identified what further evidence is needed before high-stakes use.

Frequently asked questions

Can five people be enough for an assessment pilot?

Five people may help reveal usability problems, confusing instructions or obvious technical faults. Five people generally cannot support broad claims about score reliability, fairness or how an assessment will perform for a large candidate population.

If every pilot participant passes, does that mean the test is too easy?

Not necessarily. Participants may be unusually experienced or familiar with the subject. Review the job requirements, the questions and the intended candidate population before drawing conclusions.

Is a pilot average a suitable pass mark?

No, not by itself. The average describes the trial group. A pass mark should reflect a justified minimum standard for the intended decision.

Can a small pilot identify a faulty question?

Yes. A clearly incorrect answer key or ambiguous question can be identified through direct review even when few people participated.

Does a larger sample automatically make a pilot trustworthy?

No. Participant selection, realistic testing conditions, clear study questions and sound analysis also matter.

Should employers publish the pilot findings?

Employers should communicate appropriate information about the assessment's purpose and limitations. What can be shared publicly depends on confidentiality, privacy, security and the needs of candidates and decision-makers.

Can Talentrenza's 40-question, 45-minute format be assessed through a small pilot?

Yes, for matters such as instructions, timing experience and technical operation. However, a small trial of that format does not automatically establish that the questions, scores or selection decisions are reliable, valid or fair.

When should an employer seek specialist help?

Specialist review is especially useful when assessments affect high-stakes decisions, involve many candidates, use complex score comparisons or require strong evidence about reliability, validity and fairness.

Conclusion

A small assessment pilot is valuable when employers use it for the right reasons. It can uncover problems early, improve the candidate experience and prevent avoidable mistakes. Its weakness is not that it involves few people; its weakness arises when the employer makes claims that the evidence cannot support.

Use a small pilot to learn and improve. Do not use it to claim certainty. Record what was checked, correct what was found, and gather stronger evidence when the decisions require it.