| Approach | Primary role | What to verify |
|---|---|---|
| Manual QA | A reviewer listens, interprets context, and evaluates communications against agency criteria. | Reviewer calibration, documented evidence, and consistent follow-up. |
| AutoQA assistance | Automated analysis identifies potential findings for a supervisor to investigate. | Supported criteria and sources, missed findings, incorrect flags, and exceptions. |
| Connected QA workflow | Confirmed findings become documented feedback, training, and subsequent review. | A clear path from the original communication to development and follow-up. |
Define what a useful finding looks like
Begin with the review questions your agency needs to answer. A score has little value if a supervisor cannot understand the behavior behind it or choose an appropriate next step.
- Identify the evaluation criteria and describe observable behaviors for each.
- Distinguish a missing piece of information from a behavior that cannot be judged from the recording.
- Choose a manageable set of criteria for the initial evaluation.
Build a representative evaluation set
Use recordings your agency is authorized to review. Include ordinary communications and difficult examples so the evaluation reflects actual work.
- Include different audio sources, call types, shifts, and levels of audio clarity.
- Have qualified reviewers assess the same material independently before discussing differences.
- Document how reviewers resolve ambiguity and agree on a reference finding.
Compare assistance with manual review
Manual QA establishes context through listening and judgment. AutoQA can help direct attention to potential findings; the evaluation should reveal both the useful assistance and the additional checking it creates.
- Compare AI-assisted findings with the agreed human review for each criterion.
- Record missed findings, incorrect flags, and cases needing additional context.
- Measure the full review process, including correction and documentation, rather than the time to generate a score.
Check the supervisor workflow
A finding needs a clear path back to evidence and forward to coaching. Ask the vendor to demonstrate the steps your supervisor would take.
- Return to the original communication and inspect the context of a finding.
- Verify how a supervisor documents a decision or correction and completes the review.
- Confirm access, supported recording sources, and evaluation types in the proposed configuration.
Connect the finding with development
Use a confirmed finding to choose a specific improvement objective. Follow up after coaching or practice so the program can assess change over time.
- Document the observation and employee discussion in SmartTRAC.
- Select relevant Apprentice learning or SmartSim practice for the agreed objective.
- Review subsequent work against the same criteria and record the follow-up.
Use a simple decision worksheet
For each test case, record the criterion, human reference finding, automated finding, supervisor decision, review effort, and follow-up. Summarize patterns before deciding what to deploy.
- Mark each outcome as useful, incorrect, missed, or unresolved.
- Assign an owner and next step for each unresolved configuration or workflow issue.
- Agree on periodic calibration and re-evaluation when criteria or source conditions change.