AI Bias Narrative Auditor

EU AI Act Algorithmic Impact Assessment Compliance Bias Human oversight Canadian Directive Data governance

Overview

Document risk, bias, and transparency for Algorithmic Impact Assessment (AIA). Critique Human-in-the-loop descriptions against EU AI Act Art. 14 and Canadian Directive thresholds; verify dataset provenance and copyright/privacy compliance.

Getting Started

Add your source materials to the project workspace (e.g. technical documentation, Instructions for Use, or an AIA draft), then activate the AI Bias Narrative Auditor skill. The agent will scope the document, run the validation tools on the relevant sections, and help you fix gaps until the narrative meets the required thresholds.

Example Conversation

You: We’re preparing for EU AI Act conformity. Please audit the human oversight section in docs/instructions-for-use.md and the data governance section in docs/technical-file.md.

Agent: I’ll run both tools. First, the human-in-the-loop narrative audit on the Instructions for Use…

Agent: Here are the results of hitl_narrative_audit on docs/instructions-for-use.md:
Summary: 2 ERROR(s), 1 WARNING
[ERROR] [Art. 14(4)(b)] Awareness of automation bias: not found or insufficient.
[ERROR] [Art. 14(4)(e)] Ability to intervene or interrupt (e.g. stop mechanism): not found or insufficient.
[WARNING] [Canadian Directive] Meaningful human review: not found or insufficient.
I recommend adding a short subsection on automation bias (e.g. training for deployers, checklists) and explicitly describing the “stop” or override control. I’ll run the dataset provenance check next.

Agent: dataset_provenance_check on docs/technical-file.md found:
0 ERROR(s), 1 WARNING — Retention or deletion policy for data: not found. I’ve added a retention subsection to the data governance chapter; you can adjust the periods to match your policy.

What the Tools Validate

The skill includes two validation tools that run against your documentation:

dataset_provenance_check verifies that training and data sources are documented and compliant:

Training/data source listing or description (EU AI Act Art. 10(2) — origin and acquisition)
Copyright or licensing of data sources (rights, attribution, clearance for training)
Privacy / lawful basis for personal data (GDPR Art. 6)
Retention or deletion policy (WARNING if missing)
Anonymisation or pseudonymisation where personal data is used (WARNING; Art. 10(5) when special categories are processed)
Special categories (Art. 9 GDPR) — INFO if processed without explicit safeguards
Unfilled placeholders ([TODO], [INSERT], [SPECIFY]) reported as INFO

hitl_narrative_audit critiques the human oversight / Human-in-the-loop description:

Oversight modality (HITL, HOTL, or HIC) — Art. 14
Understand capabilities and limitations — Art. 14(4)(a)
Awareness of automation bias — Art. 14(4)(b)
Ability to interpret the AI output correctly — Art. 14(4)(c)
Ability to disregard, override, or reverse the output — Art. 14(4)(d)
Ability to intervene or interrupt (e.g. stop mechanism) — Art. 14(4)(e)
Meaningful human review (Canadian Directive) — WARNING
Explanation and recourse (Canadian Directive) — WARNING

Address all ERRORs before finalising; resolve or document WARNINGs as appropriate for your jurisdiction.

Output Excerpt

After edits, a compliant human oversight section might read like this (the tools would report no errors for Art. 14(4)):

## Human oversight (Art. 14)

**Modality:** Human-in-the-loop (HITL). Each recommendation is reviewed and approved by a qualified officer before any decision is applied.

**Capabilities and limitations (Art. 14(4)(a)).** Deployers receive training on the system’s intended purpose, capabilities, and known limitations; these are also set out in this document and in the user interface.

**Automation bias (Art. 14(4)(b)).** Training materials and checklists remind operators to treat the system’s output as advisory and to apply independent judgment, especially when the outcome affects individuals.

**Interpretation of output (Art. 14(4)(c)).** The interface displays confidence scores and explanatory text; deployers are trained to interpret these and to recognise edge cases.

**Override and reversal (Art. 14(4)(d)).** The deployer may disregard or override any recommendation and record the reason in the system.

**Intervention and stop (Art. 14(4)(e)).** Each screen includes a "Stop / Do not use this output" control that immediately halts use of the recommendation and logs the event for review.

The dataset provenance section would similarly document sources, licensing, lawful basis, and retention so that dataset_provenance_check returns zero ERRORs.