Assessment Vendor Due Diligence: Procurement Evidence Checklist

This guide is a procurement and supplier-assurance checklist. It explains what documentation an employer should request on data handling, security, service reliability, scoring claims, support and contract terms before selecting an assessment provider. The existing Talentrenza reliability-versus-validity guide explains those psychometric concepts; it does not replace supplier due diligence.

An assessment vendor can demonstrate an attractive candidate dashboard, rapid reporting and a large question catalogue without answering the question that matters most: will this service support sound, fair and secure decisions for our particular roles and candidates? A polished demonstration establishes that a product can be shown. It does not establish that its questions measure relevant competencies, its scores can be interpreted as proposed, or its operational safeguards will work under pressure.

Assessment vendor due diligence is the structured process of checking a provider's claims, evidence, controls and contractual commitments before an organisation relies on its service. It combines procurement discipline with HR, technical, privacy and assessment expertise. This guide sets out what to request, what each item can establish, what it cannot establish, and how to conduct a realistic pilot.

Start with the decision, not the software

Before requesting demonstrations, define the intended use. Are you selecting entry-level analysts, screening applicants for customer-support roles, assessing promotion readiness or identifying development needs? Which competencies matter? What consequences follow a low score? How many people will participate, from which locations and on which devices?

An employer buying a low-stakes learning diagnostic has different evidence needs from an employer using a timed test to exclude candidates from a safety-sensitive role. Procurement should document the role or programme, target population, languages, accessibility requirements, approximate volume, data locations and the precise decision the scores will inform.

Write a short intended-use statement. For example:

We intend to use a numerical and document-interpretation assessment as one component of a structured recruitment process for junior finance analysts. We will not treat a vendor's overall score as an automatic hiring decision without reviewing its relevance and scoring limitations.

This statement makes vendor comparisons more meaningful. A provider may be suitable for one use and unsuitable for another.

The evidence matrix: ten areas to investigate

The table is a starting point, not a universal pass/fail standard. Weight each domain according to the risk and intended use.

1. Ask for evidence of job relevance

A vendor's category labels—“critical thinking”, “numerical reasoning” or “leadership”—are not a substitute for a job analysis. Request a blueprint explaining which skills each assessment measures, how content is distributed and why those skills are relevant to the intended position.

For example, an employer selecting payroll officers might reasonably need numerical accuracy, record reconciliation and judgement about exceptions. A generic abstract-puzzle test may have a place in a broader assessment strategy, but its relevance must be argued rather than assumed.

Useful questions include:

• Which competency definition is used, and can we inspect it?
• How are items written and reviewed by subject-matter experts?
• What proportion of the test covers each competency?
• Can an employer select job-relevant skills without compromising score interpretation?
• What evidence supports using the same test for different seniority levels or jurisdictions?

Warning sign: a provider claims a test is suitable for every job because it is “scientific” or widely used, but cannot explain the connection to the employer's actual role.

2. Examine what scores really mean

Request a sample report with a technical interpretation guide. Does a score of 75 represent percentage correct, a transformed scale, a percentile or a vendor-defined band? These are not interchangeable.

A percentile of 75 means a score is positioned relative to a specified reference distribution under the vendor's stated convention. It does not mean the candidate answered 75% of questions correctly. Nor does a percentile establish that the candidate meets a job-specific minimum standard.

Ask how composite scores are calculated, whether competencies are weighted, how missing answers are treated, and whether reports display uncertainty or limitations. If the vendor supplies “recommended pass marks”, request the rationale, target population and evidence supporting those recommendations.

Warning sign: scores are presented as precise predictions of future performance without documentation of the underlying evidence or uncertainty.

3. Request reliability evidence relevant to the format

Reliability concerns the consistency or precision of assessment scores. The appropriate evidence depends on how a test works. Human-scored interviews or essays may require inter-rater consistency evidence; alternate test forms may require comparability evidence; a multi-item scale may require examination of internal structure and score precision.

Ask the vendor to identify the method used, sample characteristics, dates, sample size, results and limitations. An unexplained coefficient is not enough. A reliability estimate from one population or use may not transfer automatically to another.

For mixed-skill assessments, one internal-consistency statistic for the entire test can be misleading because the blueprint intentionally covers different competencies. Request an explanation of how the measurement model matches the reported scores.

Warning sign: the vendor treats one high reliability coefficient as proof that the assessment is appropriate for hiring.

4. Evaluate validity claims rather than accepting labels

Validity is about the evidence and reasoning supporting a particular interpretation and use of scores. A test is not simply “valid for everyone” because its vendor has produced a study.

Relevant evidence may include content alignment with job requirements, studies of candidate response processes, analysis of score structure, and relationships with external measures such as suitably collected job-performance indicators. Each source has limits.

Ask for the actual technical report or a substantive summary, including the research population, methods, uncertainty and known limitations. Where evidence is unavailable for the intended use, ask what additional local review or pilot is required before making consequential decisions.

For more detail, consult the companion guide Assessment Reliability vs Validity: What Employers Need to Understand.

Warning sign: a sales document uses “validated” as a blanket adjective but does not specify the role, population, decision or evidence.

5. Test accessibility and candidate experience

A platform may function perfectly on the vendor's demonstration laptop while failing for a candidate using a screen reader, keyboard-only navigation, a small device or unstable connectivity.

Request an accessibility statement describing the tested product version, supported assistive technologies, known limitations and remediation process. Ask how candidates request reasonable adjustments, who approves them and how any changed administration conditions are recorded and interpreted. Requirements differ by jurisdiction and circumstance.

Conduct hands-on tests with representative devices and network conditions. Verify that instructions are understandable, timers are visible, answers are saved as described and interruptions have a fair resolution process.

Do not treat every candidate support request as suspicious behaviour. Technical and accessibility incidents require an evidence-based response.

Warning sign: a vendor promises “fully accessible” delivery but provides neither a testable account nor documentation of known limitations.

6. Review security and test integrity proportionately

Ask how the vendor protects accounts, assessment content, answer keys, candidate records and administrative permissions. Important controls may include role-based access, audit logs, encrypted transport, credential protection, backup procedures, incident response and restricted access to question banks.

Also ask how exposed questions are identified, retired or replaced, and whether repeated attempts may receive overlapping content. If proctoring is offered, review its limitations and effects on candidate privacy and accessibility. Monitoring technology should not be presented as infallible proof of misconduct.

Request available independent security assurance evidence where appropriate, and check its scope, date and coverage. A certification for one part of a vendor's business may not cover the product or hosting arrangement you intend to use.

Warning sign: the vendor promises zero cheating, perfect identity verification or absolute data security.

7. Map personal data and cross-border obligations

Before collecting candidate responses, identify what data the platform collects, why it is needed, who can access it, where it is stored, how long it is retained and how it can be exported or deleted. Clarify the respective legal roles of employer and vendor and whether subcontractors process data.

Request the relevant data-processing terms, subprocessor information, breach-notification commitments and deletion or return process. Employers operating across countries must review the laws applicable to their own processing and the vendor's service; a generic global privacy statement does not settle those questions.

Be especially cautious about optional recordings, identity documents, behavioural signals and automated inferences. Each may create additional risks and obligations.

Warning sign: the vendor cannot explain whether deleted candidate data remains in backups, or offers no practical exit mechanism.

8. Verify reliability of service and incident handling

An assessment programme can fail operationally even when its questions are good. Ask about uptime measurement, maintenance windows, technical support, backup and recovery arrangements, and procedures for interrupted sessions.

Test a controlled scenario: a candidate loses connectivity halfway through a timed assessment. Can the attempt resume? Is elapsed time preserved? Who decides whether a retake is justified? Does the employer receive an auditable record?

Service-level commitments should specify what is measured, exceptions, escalation routes and remedies. A broad “24/7 support” promise is less useful than a clear response process for a live high-volume assessment event.

Warning sign: no documented plan exists for a scoring defect discovered after results have influenced hiring decisions.

9. Inspect integrations and report governance

Ask for a realistic export of candidate results, with synthetic data. Confirm that scores, competency labels, test version, completion status and relevant timestamps are represented correctly. Where the vendor offers an API or applicant-tracking integration, inspect documentation and access controls rather than assuming every advertised integration meets your workflow.

Check whether a recruiter can view only assigned candidates, whether employers are separated from one another, and whether candidate-facing reports expose confidential answer keys. Audit logs should make material score corrections and administrative overrides traceable.

Warning sign: the vendor cannot distinguish between an assessment completion, a valid score and a final hiring decision in its exports.

10. Understand commercial terms and exit risks

Compare total programme cost, not only headline subscription price. Relevant charges may include per-candidate attempts, assessment credits, additional users, custom content, retakes, integrations, storage, implementation, support and currency or tax treatment.

Confirm what happens to unused credits, candidate records, private questions and reports if the contract ends. Determine whether the employer can retrieve data in a usable format and whether the vendor will delete it under agreed terms.

Warning sign: the contract promises broad ownership or portability but gives no usable export mechanism or termination procedure.

A practical vendor comparison exercise

Consider three fictional providers being evaluated for a multi-country graduate recruitment programme. The employer first sets mandatory conditions: acceptable data-processing terms, a usable accessibility process, secure role separation and a clear incident procedure. A provider failing a mandatory condition is not rescued by a high marketing score.

For providers that pass those conditions, the employer might use the following illustrative, not validated, weighted rubric:

For Provider A, the calculation is (4 × 0.30) + (4 × 0.25) + (3 × 0.15) + (4 × 0.20) + (3 × 0.10) = 3.75. The corresponding totals for B and C are 3.45 and 3.85. These numbers are fictional ratings, not real vendor findings.

Provider C leads under this rubric, but it would be inappropriate to award the contract automatically. The team must examine the underlying evidence, check mandatory conditions and consider whether small score differences reflect meaningful distinctions. The weights themselves should be agreed before vendor demonstrations and adapted to the employer's risks.

Run a structured pilot before full adoption

A pilot is most useful when it tests specific risks rather than merely asking participants whether they liked the interface.

Step 1 — Define pilot questions. Can participants complete the assessment on representative devices? Do the instructions create confusion? Are reports accurate? Do reviewers interpret them consistently?

Step 2 — Select a suitable group. Include representative participants and relevant accessibility or device conditions where feasible. Record the sample size and limitations; do not generalise from a handful of volunteers to all applicants.

Step 3 — Test the complete journey. Invite participants, administer the assessment, simulate an interruption, review score reports, test role permissions and export synthetic records.

Step 4 — Review substantive content. Have qualified subject experts inspect questions, keys, distractors, rubrics and competency alignment. Technical functionality is not content assurance.

Step 5 — Record defects and ownership. Classify findings by severity, require evidence of correction and rerun affected checks.

Step 6 — Make a conditional decision. Approve the service only for uses supported by the available evidence. Document limitations and further validation needs before expanding to higher-stakes decisions.

A pilot can identify usability and content problems. It does not automatically establish predictive validity, score comparability or legal compliance.

Documents to request before signing

Use a request-for-evidence pack rather than a yes/no questionnaire. Depending on risk, request:

1. Product scope and assessment catalogue with version information.
2. Competency definitions, blueprints and representative sample items.
3. Technical score interpretation and available reliability/validity evidence.
4. Content development, review and question retirement procedures.
5. Accessibility documentation, adjustment workflow and known limitations.
6. Security architecture overview, role-permission model and available assurance reports.
7. Privacy notice, data-processing terms, subprocessors and retention/deletion procedures.
8. Incident-response and business-continuity summaries.
9. Sample reports, export schema and integration documentation.
10. Pricing, service-level terms, support escalation and exit arrangements.

The employer may need confidential review arrangements for proprietary technical documentation. A vendor can legitimately protect trade secrets, but confidentiality should not be used to avoid providing any meaningful evidence for high-stakes claims.

Red flags that deserve escalation

• Claims of universal predictive accuracy or bias-free hiring without qualified evidence.
• An inability to explain what a reported score measures.
• Fixed “recommended” cutoffs with no role-specific rationale.
• Unclear ownership of private questions or candidate records.
• No defined handling of answer-key mistakes or leaked items.
• Accessibility claims that cannot be tested.
• Material changes to test forms without version records.
• Pressure to sign before technical, privacy or legal review.

One red flag does not always mean the provider must be rejected. It does mean the concern needs a documented answer and, where relevant, a corrective commitment before use.

How this applies to Talentrenza Assess

Employers considering Talentrenza Assess should apply the same due diligence questions they would ask any other provider. They should verify which features are currently available for their account, select competencies relevant to the role, inspect the resulting reports and establish their own decision rules.

The existence of a question bank, test builder, timer or reporting dashboard does not establish that a particular assessment is independently validated for a given occupation, country or hiring decision. Employers should request and evaluate the evidence appropriate to their use and seek qualified specialist review where the stakes warrant it.

For the broader organisational controls around vendor selection and assessment use, see Assessment Governance Framework for Global Employers. For the distinction between a policy and a governance framework, see Creating an Assessment Policy for Recruitment and Promotion.

Frequently asked questions

Is an assessment vendor's large question bank proof of quality?

No. Volume does not establish accuracy, job relevance, sound distractors, difficulty calibration or score comparability. Inspect content development and review procedures, not just item counts.

Should we require a reliability coefficient from every provider?

Ask for reliability or precision evidence appropriate to the score and intended use. A single coefficient may be irrelevant or misleading for a multi-competency assessment or a human-scored work sample.

Can a vendor's published validation study replace our job analysis?

Not automatically. A study may inform your decision, but you still need to establish relevance to your roles, candidate population and intended use.

Does a security certificate mean the assessment is fair?

No. Security assurance and assessment fairness address different risks. Check the scope of security evidence and review accessibility, job relevance and score interpretation separately.

What if we cannot afford a large pilot?

Start with a risk-based functional and content pilot, document its limitations and avoid unsupported claims. High-stakes decisions may require stronger evidence before use; budget constraints do not make weak evidence stronger.

Is the cheapest provider the best value?

Not necessarily. Consider the costs of unreliable administration, inaccessible testing, weak score interpretation, privacy incidents and difficult contract exit alongside subscription fees.

Conclusion

Good vendor due diligence is an evidence exercise, not a contest of feature lists. Employers should ask what the assessment measures, why that matters for the job, how scores are supported, what happens when something goes wrong and what the organisation remains responsible for.

The best procurement decision is the one the employer can explain and defend with documented evidence—not the one supported by the most confident sales presentation.