AI Agents in Humanitarian Aid: A 24-Control Governance Checklist
14 min · updated
AI agents are starting to take actions in humanitarian work: ranking households for cash assistance, planning distributions, tutoring children, and answering refugees. When an agent acts on people in crisis, a mistake is not a wrong sentence. It is a family left off a list. This article gives four scenarios, a checklist of 24 controls in plain words, and a free open-source tool, AgentGov Auditor, that checks those controls before an agent is released.
Why humanitarian AI agents need a higher bar
A chatbot answers questions. An AI agent takes actions. It reads records, calls other systems, updates lists, and sends messages.
In a commercial setting, a wrong action costs money. In a humanitarian setting, the person on the other side of the action is often someone who has lost a home, cannot easily complain, and may never learn that software made the decision.
Three features of humanitarian work raise the bar.
- The people affected have little power to object. A customer can switch providers. A displaced family cannot switch aid agencies.
- The data is dangerous. A list of names, locations, ethnic groups, or protection cases can put people at risk if it leaks.
- The principle is "do no harm". It applies to software as much as to staff.
Building an agent has become easy. A small team can connect a model to a registration database in a few days. Checking that the agent is safe has not become easy. In many teams the review is still a document, a meeting, and the sentence "I think so."
The four scenarios below are illustrations. They are not reports of real deployments, and I am not describing any organization. I chose them because each one shows a different way an agent can do harm without anyone intending it.
Scenario 1: finding the most vulnerable households for cash assistance
A cash program has funds for a fraction of the households that applied. An agent reads the registration data and assessment forms, scores each household for vulnerability, and produces a ranked list for enrollment.
What can go wrong
- The agent drops a household from the list, and nobody can say why.
- The agent writes its ranking directly into the payment list, so no person reviews it.
- A text field in an assessment form contains an instruction, and the agent follows it.
- A household appeals, and no record exists of the steps behind the decision.
The controls that matter most
- A named owner (control 1). Someone in the program is accountable for the agent, by name.
- Risk tiering (control 2). An agent that affects who receives assistance is high risk. It should be declared and reviewed as such.
- Human approval gates (control 10). The agent recommends. A person approves any inclusion or exclusion.
- Content guardrails (control 11). Free text from forms is data. It is never an instruction.
- Complete step tracing and a tamper-evident audit trail (controls 22 and 23). When a family appeals, staff can show what the agent did and why. This is what accountability to affected people looks like in practice.
- Reversible actions (control 18). A wrong exclusion can be undone before the payment cycle closes.
Scenario 2: an agent that plans aid distribution
An agent reads stock levels, beneficiary counts, and road conditions. It drafts distribution plans, creates dispatch orders, and sends SMS notices telling people when and where to collect.
What can go wrong
- The agent sends a collection notice to thousands of people for the wrong day or the wrong site. A crowd gathers. That is a safety incident, not a typing error.
- The agent can call any function in the logistics system, including ones nobody meant to give it.
- A loop sends the same message again and again and uses up the SMS budget.
- Something goes wrong at night and nobody knows how to stop it.
The controls that matter most
- Allowlisted tool access (control 5). The agent can use an approved list of tools and nothing else.
- A dedicated identity (control 4). The agent has its own account, so its actions are never confused with a staff member's.
- Human approval gates (control 10). A person approves any mass message and any dispatch order.
- Resource limits (control 12). Firm limits exist on messages sent, money spent, and actions taken.
- A kill switch and an incident runbook (controls 17 and 19). The duty officer can stop the agent immediately and knows what to do next.
Scenario 3: an AI tutor in education in emergencies
Children in a temporary learning space use an AI tutor on shared tablets. It explains lessons in their language, sets exercises, and remembers each learner's progress.
What can go wrong
- The tutor gives a child content that is unsafe or wrong for their age.
- Tablets are shared, and one child sees what another child wrote.
- A model update changes the tutor's behavior, and nobody tests it again.
- A child discloses something serious, and no person is alerted.
The controls that matter most
- Content guardrails (control 11). Both what the child types and what the tutor answers are checked.
- Context memory integrity (control 15). What the tutor remembers is kept separate for each learner.
- Regression evals (control 20). A fixed set of tests runs before every release. It covers what the tutor must do and what it must never do.
- Change control (control 3). A new model version is a change that needs review.
- Real-time monitoring (control 24). Unusual behavior raises an alert that reaches a person, not a log file nobody reads.
Scenario 4: an assistant in refugee response
An assistant answers questions from refugees about registration, services, and appointments. To give useful answers, it can look up case records and book appointments. It exchanges messages with a partner organization's system.
What can go wrong
- A person asks about their case and receives details of someone else's.
- A database password sits in the code, and the code is shared with a contractor.
- A forged message that appears to come from the partner system changes a referral.
- A plug-in from an unknown source is added to the agent in a hurry.
The controls that matter most
- User-enforced data layer (control 7). The agent fetches only what the person it is serving is allowed to see. For protection data, this is the control I would check first.
- Vault-injected secrets and short-lived tokens (controls 8 and 6). No passwords in code. Keys expire quickly and open only what is needed.
- Authenticated inter-agent messages (control 16). Each system proves who it is.
- Vetted tool supply chain (control 14). Outside components come from known sources and are fixed to checked versions.
- Tamper-evident audit trail (control 23). Nobody can quietly change the record of who accessed what.
The checklist: 24 controls
These are the controls AgentGov Auditor checks. They are this project's own model. They work as a manual checklist even if you never run the tool.
A. Accountability
| # | Control | What to check |
|---|---|---|
| 1 | Owner and inventory | A named person or team is responsible for the agent. The agent is listed with what it does and what it can use. |
| 2 | Dynamic risk tiering | The agent has a risk level that matches what it can really do. Someone has explained and approved that level. |
| 3 | Integrated change control | When the model, tools, or permissions change, the change is reviewed and checked again before release. |
B. Identity and access
| # | Control | What to check |
|---|---|---|
| 4 | Dedicated non-human identity | The agent uses its own account. It does not use a staff member's login or a shared password. |
| 5 | Allowlisted tool access | The agent may use only the tools on an approved list. Everything else is refused. |
| 6 | Short-lived scoped tokens | Access keys expire quickly and open only what the agent needs. |
| 7 | User-enforced data layer | The agent only fetches data that the person it is working for is allowed to see. |
| 8 | Vault-injected secrets | Passwords and keys are never written in the code. A secure store supplies them when the agent runs. |
C. Action controls
| # | Control | What to check |
|---|---|---|
| 9 | External policy engine | Fixed rules outside the AI model decide whether a risky action is allowed. When the rules cannot be checked, the answer is no. |
| 10 | Human approval gates | A person approves risky actions before the agent carries them out. |
| 11 | Dual content guardrails | Input and output are both checked, so hidden instructions and sensitive data do not slip through. |
| 12 | Multi-level resource limits | Firm limits exist on how long the agent runs, how much it spends, and how many actions it takes. |
| 13 | Disposable sandbox environments | Anything the agent runs happens in a locked-down, throwaway workspace that cannot harm real systems. |
D. Trust boundaries
| # | Control | What to check |
|---|---|---|
| 14 | Vetted tool supply chain | Outside tools and software come from known sources and are fixed to checked versions. |
| 15 | Context memory integrity | What the agent remembers is checked before it is saved, kept separate for each user, and protected from tampering. |
| 16 | Authenticated inter-agent messages | When agents talk to each other, each one proves who it is, and messages are protected from forgery. |
E. Containment and recovery
| # | Control | What to check |
|---|---|---|
| 17 | Instant agent kill switch | Someone can stop the agent immediately, and the stop actually ends its actions. |
| 18 | Reversible action design | When the agent changes something, there is a way to undo it, or a clear approval for anything that cannot be undone. |
| 19 | Dedicated incident runbook | A written and practiced plan exists for what to do when the agent goes wrong. |
F. Assurance and visibility
| # | Control | What to check |
|---|---|---|
| 20 | Mandatory regression evals | The agent passes a fixed set of tests before every release. The tests cover what it should do and what it should not do. |
| 21 | Continuous red teaming | People or tools regularly try to trick or break the agent in a safe test setting. |
| 22 | Complete step tracing | Every step the agent takes is recorded, so people can later see what it did and why. |
| 23 | Tamper-evident audit trail | The record of what the agent did is protected so that nobody can secretly change or delete it. |
| 24 | Real-time operational monitoring | The agent is watched while it runs. Unusual behavior raises an alert that reaches a person. |
A declaration is not evidence
Humanitarian organizations write good policies. The gap is usually between the policy and the code.
A proposal can say "a human reviews every targeting decision". That sentence does not create a review step in the software. A useful audit keeps five levels apart:
| Level | Name | Meaning |
|---|---|---|
| 0 | No evidence | Nothing was found. |
| 1 | Declared | Someone wrote it down. |
| 2 | Implemented | The code shows it. |
| 3 | Tested | A test proves it works. |
| 4 | Verified in operation | Evidence from the running system confirms it. |
Two rules follow.
A declared control is a promise, not a proof.
A high score never overrides a failed required control. An agent that does twenty things well and can still remove a household from a payment list without approval is not "mostly safe".
How AgentGov Auditor helps
AgentGov Auditor is a free command-line tool that runs this checklist against the folder that holds an agent's code. It is open source under the Apache-2.0 license. I built it so that a program officer and a developer can look at the same page and agree on what is missing.
It gives one clear answer. ALLOW, BLOCK, or REVIEW REQUIRED. Every problem comes with a sentence in plain words, the file, the line, and the steps to fix it.
It suits field conditions. It is one program. It works offline, with no account, no API key, and no license fee. That matters for a country office with a limited budget and an unreliable connection.
Nothing leaves the computer. The scan is local and sends no telemetry. It only reads files. It does not change them and does not run the code it scans.
It has a strict setting for high-risk agents. Under the high-risk profile, twelve controls are mandatory and each must be proven by tests. The four scenarios above belong in that profile.
It supports due diligence on partners and vendors. Ask a technology partner to run the scan and share the report before an agent touches beneficiary data. The report shows evidence, not assurances.
It fits into existing review. The same check can run automatically each time the code changes, so a control that was in place last month cannot quietly disappear.
To try it:
- Download the file for your system from the releases page and unpack it.
- Create a starter agent with
agentgov new my-agent, then scan it withagentgov scan my-agent --open. - Scan a real agent with the strict profile:
agentgov scan path/to/your/agent --profile high-risk --open. - Ask about any control you do not know, for example
agentgov explain AGC-010.
The repository includes an agent I wrote badly on purpose. Scanning it returns BLOCK with 17 findings: 5 critical, 7 high, and 5 medium. The first finding reads: "The model can call any function by name", at agent.py, line 132. In a distribution agent, that one line would mean the agent could reach any function in the logistics system.
What the tool does not check
This section matters more in humanitarian work than anywhere else. Please read it before you rely on the tool.
- It does not judge whether targeting is fair. It cannot tell you whether a vulnerability model is biased, or how many eligible households it wrongly excludes. That needs program expertise, data analysis, and the community's own voice.
- It does not assess data protection compliance or consent. It checks technical controls around data access. It does not replace a data protection impact assessment.
- It does not decide whether an agent should exist. Some decisions should stay with people. No checklist settles that question.
- It reads patterns in code. It is not a full parser. It will miss things and will sometimes flag things that are fine. Python coverage is deeper than JavaScript and TypeScript coverage.
- It does not watch an agent run. It audits a folder. It cannot stop an agent in production.
- ALLOW is not a certificate. It is a policy decision based on the evidence found. The tool is not endorsed by OWASP, NIST, or any humanitarian body. Where a control relates to the OWASP Top 10 for Agentic Applications 2026 or the NIST AI Risk Management Framework 1.0, the report shows the reference as a cross-reference only.
- It is alpha software. The first release is v0.1.0-alpha.1. It has not been tested in a humanitarian operation, and I have no deployment stories to share.
The tool covers one layer: whether the agent's actions are under control and whether that can be proven. Fairness, consent, and necessity are separate questions, and they come first.
Frequently asked questions
How are AI agents used in humanitarian aid?
Possible uses include scoring households for cash assistance, planning distributions and sending notices, tutoring learners in education in emergencies, and answering questions from refugees about services. In each case the agent takes actions, not only gives answers, which is why it needs controls.
What are the main risks of using AI to target cash assistance?
The main risks are wrongful exclusion with no explanation, decisions that no person reviewed, exposure of sensitive household data, and no record to support an appeal. Human approval, step tracing, and an audit trail address the last three. Fairness of the targeting model needs separate review.
What does "human in the loop" mean for an aid agent?
It means a person approves a risky action before the agent carries it out. For a targeting agent, the agent recommends and a person decides. The control only counts when the software enforces it. A sentence in a policy document is not enough.
Can AgentGov Auditor tell me if my targeting model is biased?
No. It checks governance controls around the agent's actions, access, and records. It does not evaluate the fairness or accuracy of a model.
Does the tool send beneficiary data anywhere?
No. The scan runs on your computer, works offline, and sends no telemetry. It reads the agent's code and configuration files. It does not connect to your databases.
Is AgentGov Auditor free for NGOs and UN agencies?
Yes. It is free for everyone under the Apache-2.0 license, with no account and no paid tier.
Does a result of ALLOW mean an agent is safe to use with affected people?
No. ALLOW means the required technical controls have evidence. It does not cover fairness, consent, legal compliance, or whether the use is appropriate. Those need people.
Next step
Before the next agent touches a beneficiary list, scan it. If the tool misses something that matters in your context, open an issue and tell me. Humanitarian use is exactly where I want the checklist to be tested hardest.
Get the tool: github.com/shahzadarain/agent-governance-auditor