CI Acceptance Checks for AI Coding Agents

An agent saying “all checks passed” is a useful report to investigate. The merge decision should depend on checks that ran against the change you are actually reviewing.

This guide shows how to define acceptance evidence, keep CI responsibilities separate and handle missing or skipped checks. It is aimed at engineers who already use Git and CI. The small validator below is a new illustrative example for this article, not a deployment service or a claim that a particular repository is secure.

1. Turn the task into observable acceptance

Start with a bounded example: an agent changes a support-summary formatter. The result must distinguish confirmed facts from assumptions, omit a synthetic secret marker and produce stable output for the same input. “Make the summary better” is not an acceptance criterion that CI can evaluate.

Choose tests that can fail for a plausible regression. A formatting-only check cannot demonstrate the absence of a leaked field. A fixture that repeats the implementation's own calculation may repeat its mistake. Keep useful existing tests, then add the smallest checks needed to describe the requested behavior.

Before delegating the change, record the commands to run, the files allowed to change and the evidence required for review. The free repository readiness checklist provides a starting point for these repository boundaries.

2. Separate code generation from verification authority

The agent can prepare a branch and run local checks. CI should repeat the required verification from the reviewed revision in a controlled runner. A transcript of local commands can help a reviewer, but it is not a substitute for the CI result associated with that revision.

Make workflow and test changes visible to owners who can evaluate them. Decide who may alter required checks, runner configuration and repository rules. If the same change removes a test and then receives a green badge, inspect the removed test before treating the badge as useful evidence.

GitHub's secure-use reference recommends restricting token permissions, pinning external actions to reviewed full commit SHAs and reviewing workflow ownership. It also warns about executing untrusted pull-request code through privileged triggers such as pull_request_target. Choose the trigger and runner trust boundary deliberately; do not copy a privileged deployment workflow into an agent evaluation job.

For the fictional formatter, a test job needs the checked-out source and test fixtures. It does not need production cloud credentials, deployment rights or real customer tickets. This is a proposed access boundary: removing credentials does not by itself sandbox arbitrary code. Use the runner isolation and network controls appropriate to your threat model.

3. Distinguish “passed” from “did not run”

Write down the checks expected for this class of change. A missing entry, skipped job, cancelled run, timeout or neutral conclusion should remain distinct from a test that executed successfully. Do not normalize all of them into “no failure reported.”

GitHub documents that a job skipped by a condition can report success. Its protected-branch documentation also explains accepted check states and selecting an expected status source. Inspect your actual repository configuration instead of assuming a required-check label proves every underlying test ran.

For changes covered by a required acceptance gate, make that gate run on every applicable change. Test its own scheduling: a deliberately failing fixture, a skipped dependency and an updated commit should each prevent acceptance as intended. Repository policy, permissions and CI configuration must enforce this; the validator below cannot protect its own execution.

4. Try a small, explicit evidence gate

This local Python example accepts exactly three named results, each with status: success, for the expected revision. It rejects duplicates, missing checks and a different revision. The names are illustrative; replace them with the checks your repository actually needs.

{
  "commit": "example-reviewed-revision",
  "checks": [
    {"name": "tests", "status": "success"},
    {"name": "lint", "status": "success"},
    {"name": "policy", "status": "success"}
  ]
}

Save the following as acceptance_gate.py and the illustrative record as acceptance.json:

"""Illustrative record validator; it does not authenticate CI evidence."""
import json
import sys


def accepted(report, expected_commit):
    if not isinstance(report, dict) or not expected_commit:
        return False
    if report.get('commit') != expected_commit:
        return False
    checks = report.get('checks')
    if not isinstance(checks, list) or len(checks) != 3:
        return False
    if any(not isinstance(item, dict) for item in checks):
        return False
    names = [item.get('name') for item in checks]
    if any(not isinstance(name, str) for name in names):
        return False
    if set(names) != {'tests', 'lint', 'policy'}:
        return False
    return all(item.get('status') == 'success' for item in checks)


if __name__ == '__main__':
    try:
        if len(sys.argv) != 3:
            raise ValueError('expected a report path and revision')
        with open(sys.argv[1], encoding='utf-8') as source:
            valid = accepted(json.load(source), sys.argv[2])
    except (OSError, ValueError):
        valid = False
    print('acceptance passed' if valid else 'acceptance blocked')
    raise SystemExit(0 if valid else 1)

python3 acceptance_gate.py acceptance.json example-reviewed-revision

Change one status to skipped, remove a result or pass a different expected revision; the command should return a nonzero exit status. Those are useful negative tests of the gate's logic.

The record is not proof that a test ran. This example validates a document, not its authenticity. In real CI, an independently maintained collector must obtain results from trusted jobs and bind them to the revision under review. Supply the expected revision from the CI event, not from a value chosen by the agent or copied out of the report. Do not let arbitrary pull-request artifacts set a required status with privileged credentials.

A successful gate also says nothing about omitted requirements, flawed tests or permissions outside its scope. Keep the collector, required-check configuration and deployment approval under separate review.

5. Hand a reviewer the result, not a wall of logs

Summarize the revision, checks that actually executed, their evidence locations and the remaining questions. Identify changed tests and workflow files explicitly. Keep logs useful by excluding secret values and unnecessary customer content; retain the minimal evidence your team needs under its access and retention rules.

• What observable behavior changed?
• Which checks ran on the reviewed revision, and which did not?
• Did the change weaken a test, bypass a policy or expand a permission?
• What did the fixtures deliberately leave outside their scope?
• Who owns the later production decision?

For infrastructure changes, keep the merge review and the production apply decision separate. The existing Terraform plan-only workflow covers that boundary. When connected tools are involved, use the MCP permission review checklist alongside CI.

Related: MCP permission review · Free repository readiness checklist · AI and operations toolkits.