NYC LL 144: Bias Audit Runbook for Scoring Tools

It is 9:10 a.m. and Legal pings you: "We received a request to substantiate how this candidate was scored and ranked for an NYC role. Provide the bias audit artifacts, the scoring logic in effect on that date, and who approved the decision." Your team can produce a vendor PDF from last quarter, but you cannot reconstruct this specific candidate decision. The score came from an auto-screen, the ranking happened in a spreadsheet, and the hiring manager overrode it in chat. No immutable event log. No single source of truth. No timestamps tying the decision to the exact model version and rubric. Operationally, the risk clusters in three places: - Audit liability: you cannot prove what happened, who approved it, or which version of the tool produced the score. If it is not logged, it is not defensible. - SLA breach: the req stalls while teams hunt for artifacts across inboxes, dashboards, and exported CSVs. Time-to-offer delays accumulate exactly where identity and scoring are unverified. - Mis-hire cost exposure: when you cannot trust the funnel integrity, you either over-reject qualified candidates or progress risky ones. SHRM notes replacement costs can range from 50-200% of annual salary depending on role, which turns scoring errors into budget events, not HR events. NYC LL 144 is not just a compliance checkbox. It is a requirement to run hiring decisions like controlled access decisions: consistent inputs, versioned logic, and evidence packs that can be retrieved on demand.

The market did not fail because people ignored bias. It failed because the tooling stack makes auditability optional. Most ATS and point-solution screening vendors were built for throughput, not defensibility. They assume sequential checks in a waterfall: screen first, then verify, then assess, then reconcile notes. That sequence creates two predictable gaps: - Evidence fragmentation: scores live in one tool, interview notes in another, ranking changes in spreadsheets, and approvals in chat. Shadow workflows are integrity liabilities. - Unversioned decisioning: models and rubrics change, but candidate-level records do not retain tool version, prompt version, rubric version, and override rationale in a tamper-resistant format. Even when an annual "bias audit" report exists, it is rarely bound to the operational reality of day-to-day decisions. No event logs. No unified evidence packs. No review-bound SLAs. No standardized rubric storage per req. When Legal asks, you are left with a PDF and a guess.

Before you touch process, assign ownership and sources of truth. NYC LL 144 readiness is a coordination problem. - Recruiting Ops owns: workflow design, req configuration, candidate stage gates, and SLA enforcement. They own the funnel mechanics. - Security owns: access control, identity gating policy, audit retention rules, and evidence pack integrity requirements. They own defensibility controls. - Hiring Manager owns: rubric discipline, structured evaluation completion, and documented rationale for overrides. They own decision quality. - Analytics or People Analytics owns: segmented risk dashboards, time-to-event reporting, and monitoring drift in scoring outcomes across reqs. Automation vs manual review should be explicit: - Automated: identity gate checks, scoring execution, version capture, evidence pack assembly, timestamping, and write-back to ATS. - Manual (review-bound): exception handling, override approval, adverse signal review, and periodic audit sign-off. Sources of truth must be singular, not implied: - ATS is the source of truth for lifecycle stage and decision outcome. - The scoring and ranking system is the source of truth for inputs, outputs, and versions, but must write immutable evidence back into the ATS record. - The verification layer is the source of truth for identity status at the moment scoring was accessed or executed.

Run NYC LL 144 bias audits as an instrumented workflow, not a one-time document. The goal is repeatability: every candidate decision can be reconstructed from logs. Operating model checkpoints: - Identity verification before access: if a tool score influences ranking, the candidate identity status must be known at the time of scoring. Step-up verification for higher risk roles or suspicious signals. - Event-based triggers: every scoring action emits an event with timestamp, actor, tool version, and req context. The event log is your audit spine. - Automated evidence capture: store inputs (what was scored), outputs (score and explanation fields if applicable), and metadata (model version, rubric version, prompt version, dataset window). - Standardized rubrics: require structured fields for hiring managers, not freeform notes. Rubric completion is a gate, not a suggestion. - Segmented risk dashboards: monitor time-to-event and integrity signals together. Bias audit readiness improves when delays, overrides, and exception rates are visible per req and per team. Compliance outcome: when an auditor asks "show me how this candidate was ranked," you can produce a single evidence pack with timestamps and accountable approvers.

IntegrityLens AI functions as the ATS-anchored control plane that makes LL 144 auditability operational instead of aspirational. - Identity gate before scoring access using biometric identity verification (liveness, face match, document authentication) so the score is tied to a verified individual, not an email address. - Immutable evidence packs that bundle identity status, assessment artifacts, scoring outputs, reviewer notes, overrides, and timestamps into a single retrieval unit. - Tamper-resistant logs that show who changed a score, when, and why, supporting reviewer accountability and audit defensibility. - Risk-tiered funnel controls, including step-up verification and review-bound SLAs for exceptions and manual overrides. - Zero-retention biometrics architecture patterns to support Privacy by Design while still producing compliance-ready audit trails.

• Running bias audits on vendor summaries only, while candidate-level decisions happen in spreadsheets or chat. You end up "auditing the tool" but not "auditing the decision." - Allowing scoring before identity is verified, then treating verification as a downstream formality. This creates a proxy interview window where unverified actors can generate "legitimate" scores. - Permitting manual overrides without an SLA-bound review queue and documented rationale. Manual review without evidence creates audit liabilities and invites inconsistent outcomes.