Shadow Mode Integrations to Vet New Vendors Safely
A field-tested pattern for testing new verification or assessment vendors without touching live candidate outcomes, while keeping Support out of the blast radius.
Shadow Mode is how you learn vendor truth without making candidates the test environment.Back to all posts
The day a vendor rollout turns into a support incident
It is 9:12 AM on a Tuesday. A recruiter pings Support: "All candidates are stuck at Verification Pending." Within minutes, the queue fills with candidates asking why they are being asked to re-upload IDs, Hiring Managers complain about interview no-shows, and Security is asking whether PII is being sent to a new subprocessor without a signed DPA. This is the failure mode Shadow Mode is designed to prevent: you changed the decision path before you had end-to-end observability, fallbacks, and an escape hatch. By the end of this article, you will be able to implement Shadow Mode for a new vendor so you can measure accuracy, latency, and integration correctness without affecting live candidates.
Why Shadow Mode matters to Support and CS
Run Shadow Mode whenever a vendor can change candidate outcomes (verification, fraud scoring, AI interview screening, coding assessment proctoring). It de-risks three Support pain points: - Reputation risk: candidates feel the integration, not your architecture. Shadow Mode keeps their experience stable while you learn. - Cost risk: fewer P0 incidents, fewer manual backfills, fewer "status missing" tickets from recruiters. - Control risk: you get hard evidence of divergences, error classes, and latency before cutover, so escalations become fix lists instead of blame loops. Fraud pressure is rising in remote hiring pipelines. Pindrop notes 1 in 6 applicants to remote roles showed signs of fraud in one real-world pipeline. Directionally, that implies a meaningful fraction of your remote funnel may warrant step-up verification and careful controls. It does not prove fraud prevalence in your specific company or industry, and "showed signs" is not the same as confirmed fraud.
No candidate-facing regression during testing (no new prompts, no new failure states).
Every Shadow Mode event is traceable by a single correlation ID across systems.
Divergences are reviewable with evidence, not screenshots.
Cutover is reversible within minutes via a kill switch.
Ownership, automation, and sources of truth
Treat Shadow Mode like an operational change with named owners. If ownership is fuzzy, Support becomes the default owner during the incident. Recommended ownership model: - Recruiting Ops owns workflow design in the ATS (stages, statuses, who sees what) and approves cutover gates. - Security owns data handling (subprocessor approvals, retention, access controls) and the step-up rules for high-risk cases. - Support/CS owns incident playbooks, dashboards, and the escalation path when shadow results diverge or the vendor degrades. Automation vs manual review: automate routing, data capture, and comparisons; manually review only high-signal divergences via an SLA-bound queue to avoid reviewer fatigue. Sources of truth: the ATS is the authoritative system of record for candidate stage. The production verification or assessment vendor remains authoritative for outcomes until cutover. Shadow Mode vendor outputs are non-decisive annotations (for example, stored as an Evidence Pack reference) and must never drive automation that changes the candidate path.
No direct write-back from Shadow Mode to ATS stage fields.
Clear on-call: who owns vendor escalation vs integration bugs.
A candidate appeal path if any manual review is triggered later (after cutover).
What is Shadow Mode integration and how does it work?
Shadow Mode integration is a parallel run where production events are duplicated to a new vendor, responses are recorded, and differences are analyzed without influencing live decisions. Implementation pattern (answer first): duplicate the same canonical "candidate event" payload to both vendors, tag it with a correlation ID, store the shadow response, then run an async comparator that produces divergence metrics and a review queue. Key design choice: you want idempotent, replayable delivery. Shadow Mode is only useful if you can re-run a time window to isolate whether a mismatch was vendor behavior or a transient integration failure.
ATS stage change triggers an event (candidate entered "Verify identity" or "Assessment sent").
Integration layer publishes to a message bus or job queue.
Primary vendor response updates ATS (as today).
Shadow vendor response is stored and linked to the candidateCorrelationId.
Comparator job computes mismatch types and writes a non-decisive note or Evidence Pack pointer.
Step-by-step runbook to implement Shadow Mode without candidate impact
Normalize events into a canonical schema. Do not forward raw ATS payloads. Build a single internal format for candidate identifiers, job identifiers, and the action being requested (verify, interview, assess). This is where most mismatches come from.
Add a candidateCorrelationId and propagate it everywhere. Put it in webhook headers, vendor metadata fields, and your logs. Without this, Support cannot answer "what happened" in under 5 minutes.
Implement idempotent delivery with retries. Use an idempotency key per (candidateCorrelationId + action + attempt). Ensure retries cannot create duplicate vendor sessions or duplicate candidate prompts.
Enforce "no side effects" for Shadow Mode. Shadow results must not update ATS stages, send candidate emails, or open recruiter tasks. They can only be stored for analysis.
Build divergence metrics that match operations reality. Track pass-fail mismatches, manual review rate, error rate by class (4xx mapping vs 5xx vendor), and latency distributions. Compare on the same time window and risk tier.
Create a bounded manual review queue. Only send the highest-signal divergences to humans (for example, "primary pass, shadow high risk"), and set SLAs and escalation. Everything else stays as metrics.
Canary before cutover using Risk-Tiered Verification. Start with a tiny segment and step up gradually. Prefer canaries that include higher-risk slices (new geos, high-comp roles) rather than only the cleanest traffic.
Ship with kill switches and rollback. Cutover should be a config flip, not a deploy. Rollback should restore primary vendor as authoritative within minutes, and you should keep the shadow capture running so you can diagnose post-rollback.
Plan for ATS downtime. If ATS webhooks fail, queue events and replay with ordering guarantees. Your comparator must tolerate late arrivals and still produce a clean audit trail.
Close the loop with Evidence Packs. For any manual-reviewed divergence, store the evidence pointer, timestamps, and reviewer action so later audits do not depend on tribal memory.
Candidate identity key mismatch between ATS and vendor leads to "missing status" and duplicate sessions.
Retry storms cause multiple verification links to be sent if side effects are not isolated.
Clock skew or late webhooks make it look like the vendor is slow when it is your queue backpressure.
A Shadow Mode policy config you can ship with a kill switch
Use this as a starting point for a per-job or per-workflow policy. It encodes: routing, canary logic, kill switch, idempotency, write-back rules, and retention boundaries.
Anti-patterns that make fraud worse
- Turning on the new vendor globally without a kill switch, then "fixing forward" while candidates churn and fraud attempts slip through gaps. - Sending different data to each vendor (different name formats, missing job context), then blaming the vendor for mismatches that are actually mapping bugs. - Routing all divergences to humans with no SLA or prioritization, creating reviewer fatigue that turns real risk into rubber-stamps.
Where IntegrityLens fits
IntegrityLens AI is the first hiring pipeline that combines a full Applicant Tracking System with advanced biometric identity verification, AI screening, and technical assessments, so teams stop stitching together tools and hoping logs line up later. For Shadow Mode, IntegrityLens helps TA leaders, recruiting ops, and CISOs run parallel vendor tests without breaking candidate flow because the ATS workflow and verification layer live in one secure platform. You can keep the authoritative path stable, capture Shadow Mode outputs as Evidence Packs, and use Risk-Tiered Verification to canary safely. Key platform capabilities relevant here: - ATS workflow control points for gated stages and non-decisive annotations - Identity verification (document + voice + face) typically 2-3 minutes, under 3 minutes before interviews - 24/7 AI interviews for consistent screening across timezones - Coding assessments across 40+ languages - Privacy-first controls like Zero-Retention Biometrics and secure audit trails
Sources
- Checkr (2025): 31% of hiring managers reported interviewing someone later found to use a false identity: https://checkr.com/resources/articles/hiring-hoax-manager-survey-2025
Pindrop: 1 in 6 applicants to remote roles showed signs of fraud in one pipeline: https://www.pindrop.com/article/why-your-hiring-process-now-cybersecurity-vulnerability/
Related Resources
Key takeaways
- Shadow Mode means the new vendor sees the same inputs but never becomes the decision-maker for a live candidate.
- Support success depends on three controls: correlation IDs end-to-end, a kill switch, and a review queue that prevents reviewer fatigue.
- Treat divergences as an operations problem first (data mapping, thresholds, timing), not a vendor performance problem.
- Write Shadow Mode outputs back as non-decisive notes (or an Evidence Pack) so you can audit without contaminating the hiring decision.
- Run a canary based on risk tiers, not random percent alone, to avoid blind spots in fraud-heavy segments.
Drop this into your integration config repo and require Security + Recruiting Ops approval for changes.
Designed to prevent candidate impact: Shadow outputs are stored for comparison only and never update ATS stages.
Includes a kill switch, canary routing by risk tier, idempotency keys, and ATS-down buffering.
shadowMode:
name: "vendor-b-parallel-run"
enabled: true
killSwitch:
# Support can flip this without a deploy.
# When true, all shadow traffic stops immediately.
stopAllShadowTraffic: false
routing:
canonicalEventVersion: "2026-02-12"
idempotency:
keyTemplate: "${candidateCorrelationId}:${action}:${eventId}"
ttlSeconds: 86400
primaryVendor:
name: "vendor-a"
authoritative: true
allowWriteBackToATS: true
shadowVendor:
name: "vendor-b"
authoritative: false
allowWriteBackToATS: false
storeResponse:
destination: "evidence-pack-store"
retentionDays: 30
piiRedaction: "strict"
canary:
mode: "risk-tiered"
# Route shadow traffic for observation only. Keep primary authoritative.
# Risk tier is computed internally (device signals, geo anomalies, funnel history).
tiers:
low:
shadowSampleRate: 0.05
medium:
shadowSampleRate: 0.15
high:
shadowSampleRate: 0.30
hardExclusions:
# Avoid special programs or regulated roles until mappings are proven.
requisitionTagsDenyList: ["campus", "executive", "regulated"]
writeBack:
# Only annotate, never change stage or trigger candidate messaging.
ats:
annotationType: "non_decisive"
fields:
- "shadow.vendor"
- "shadow.result"
- "shadow.riskBand"
- "shadow.latencyMs"
- "shadow.evidencePackRef"
comparisons:
divergenceRules:
- name: "pass_vs_fail"
condition: "primary.decision != shadow.decision"
severity: "high"
- name: "latency_spike"
condition: "shadow.latencyMs > 8000"
severity: "medium"
- name: "error_rate"
condition: "shadow.httpStatus >= 500"
severity: "medium"
manualReviewQueue:
enabled: true
enqueueWhen:
- "divergenceRules.pass_vs_fail"
slaMinutes: 120
maxDailyReviews: 25
resilience:
atsDownBuffer:
enabled: true
queue: "shadowmode-events"
maxBacklog: 50000
replay:
orderBy: "eventTimestamp"
maxEventAgeHours: 48
auth:
# Prefer OAuth/OIDC over static API keys.
vendorAuth:
type: "oauth2"
tokenAudience: "https://api.vendor-b.example"
observability:
tracing:
propagateHeaders: ["x-candidate-correlation-id", "x-event-id"]
logs:
redactFields: ["documentNumber", "faceTemplate", "voicePrint"]
dashboards:
requiredMetrics:
- "shadow.requests"
- "shadow.successRate"
- "shadow.p95LatencyMs"
- "shadow.divergence.passFailRate"
- "shadow.manualReview.enqueued"Outcome proof: What changes
Before
Vendor changes triggered candidate-facing regressions (duplicate links, missing statuses in the ATS) and Support had to triage blind because logs did not correlate across systems.
After
New vendors were integrated in Shadow Mode first, with correlation IDs, non-decisive ATS annotations, and a bounded divergence review queue. Cutovers shipped as config flips with rollback and vendor isolation.
Implementation checklist
- Define a Shadow Mode policy: routing, consent gates, retention, and kill switch.
- Generate and propagate a candidateCorrelationId across ATS, interview, verification, and assessment steps.
- Instrument divergence metrics: pass-fail mismatch, latency, manual review rate, and error classes.
- Set a manual review workflow with SLA-bound queues and escalation paths.
- Ship with a rollback plan and an ATS-down buffer (queue + replay).
Questions we hear from teams
- How long should Shadow Mode run before cutover?
- Long enough to cover your real traffic patterns (timezones, device types, job families) and to stabilize divergence rates. Many teams start with 2-4 weeks, but the correct duration depends on volume and how quickly you can triage mismatches.
- Do we need to send biometric data to the shadow vendor?
- Not necessarily. Start with the minimum signals required to evaluate the vendor and your mapping. If biometrics are required, enforce consent gates, strict retention, and least-privilege access. Prefer privacy-first approaches such as Zero-Retention Biometrics where possible.
- What should Support do when primary and shadow disagree?
- Treat it as an investigation, not an immediate candidate action. Log the mismatch, attach the Evidence Pack reference, and route to the bounded review queue under SLA. Do not change the candidate stage based on shadow outputs until the vendor is authoritative and policy-approved.
- How do we prevent ATS downtime from breaking Shadow Mode?
- Buffer events in a queue and replay with ordering and idempotency keys. Ensure your comparator tolerates late arrivals and still produces an auditable comparison record.
Ready to secure your hiring pipeline?
Let IntegrityLens help you verify identity, stop proxy interviews, and standardize screening from first touch to final offer.
Watch IntegrityLens in action
See how IntegrityLens verifies identity, detects proxy interviewing, and standardizes screening with AI interviews and coding assessments.
