THE DECISION NOTEBOOK
How to test an AI interview assistant: a practical pilot
A repeatable pilot for document-based answers, response speed, integrations and plan limits. A test protocol to use yourself, not a report of tests already performed.
1. Fix the conditions
Choose two or three tools for the same task. Record the date, version, plan, device, network, language and enabled integrations. Use the same permitted sample documents and questions for each candidate. Prepare a 20-question set: ten answerable from the documents, five requiring synthesis across documents and five whose answer is absent or contradictory. Write the expected answers and sources before testing.
2. Measure useful answers
Measure from the end of the spoken question to the first usable answer, not merely the first visible token. Repeat the set three times and keep individual timings and the median. Separately record correct, partially correct, incorrect and appropriately unanswered results. Preserve anonymised examples and the cited source passages; a fast incorrect answer is not a success.
3. Exercise the real workflow
Test one required connector from authorisation to a successful retrieval, with a sample account and limited permissions. Check whether updated and deleted documents are reflected. If screen recognition is available, use a permitted sample screen and record what it captures. Test a network interruption and note what remains available. These observations do not establish security certification or complete data residency.
4. Record cost and boundaries
Record actual usage, included minutes or credits, document limits, seat requirements and the billing commitment. Distinguish a trial from an ongoing free plan. Do not extrapolate a small successful test to thousands of documents. Note exclusions, failures and unknowns with the same care as strengths.
5. Publish a reproducible result
State which task, users and tested plan the conclusion covers. Report the sample size, timings, answer categories, examples and limitations. Keep provider claims separate from observed results. Explain which candidate fits each scenario rather than assigning a universal winner. A new test can inform a later editorial review; this protocol itself changes no published rating.
Record your pilot
Download the blank CSV test sheet
Compare live conversation assistants · Thalassa Assistant · LockedIn AI · Parakeet AI