Skip to content

feat: reusable AGENTS.md integrity CI workflow (SHA-pinnable)#34

Open
mattmillerai wants to merge 1 commit into
mainfrom
matt/be-3368-agents-md-integrity
Open

feat: reusable AGENTS.md integrity CI workflow (SHA-pinnable)#34
mattmillerai wants to merge 1 commit into
mainfrom
matt/be-3368-agents-md-integrity

Conversation

@mattmillerai

Copy link
Copy Markdown
Contributor

ELI-5

We're standardizing on one thin AGENTS.md per repo as the single source of truth for agent instructions, with CLAUDE.md reduced to a one-line @AGENTS.md shim, no stray .cursorrules, and a hard under-200-line ceiling (per Anthropic guidance). Nothing enforces that today, so files quietly drift back over the limit and shims diverge. This adds a reusable CI workflow any Comfy repo can pin (by commit SHA) to fail its build when its AGENTS.md drifts out of spec.

What this adds

  • .github/workflows/agents-md-integrity.yml — a workflow_call reusable workflow. It runs in the caller's context, so actions/checkout checks out the calling repo and the checks operate on that repo's files. The checker itself is loaded from a pinned ref of this repo (workflows_ref), never the caller's checkout, so a PR can't rewrite the check.
  • .github/agents-md-integrity/check_agents_md.py — the checker (stdlib-only Python), the single source of truth so consumers carry only a thin caller. Emits FAIL:/WARN: lines plus GitHub ::error::/::warning:: annotations and exits non-zero on any hard failure, so it wires in as a required status check.
  • .github/agents-md-integrity/tests/ + .github/workflows/test-agents-md-integrity.yml — a unittest suite (16 cases) plus a self-test workflow (mirroring test-cursor-review-scripts.yml) that also runs a passing-repo and a failing-repo CLI smoke test.
  • A README row following the SHA-pin + v1 conventions.

Checks enforced (all acceptance criteria)

Check Behavior Input (default)
Exists Top-level AGENTS.md present agents_file (AGENTS.md)
Line ceiling > max_lines fails; > warn_lines (but ≤ max_lines) warns max_lines (200), warn_lines (150)
CLAUDE.md shim If CLAUDE.md exists, it must contain @AGENTS.md; a divergent copy fails
No .cursorrules Legacy root .cursorrules fails forbid_cursorrules (true)
Nested (monorepo) Every nested AGENTS.md needs a sibling @AGENTS.md shim and ≤ max_lines check_nested (true)
CODEOWNERS / DRI AGENTS.md matched by a CODEOWNERS rule; warn-only unless escalated require_codeowners (false → warn)

All inputs are documented in the workflow's header comment.

Usage

name: AGENTS.md Integrity
on:
  pull_request:
  push:
    branches: [main, master]
jobs:
  agents-md:
    permissions:
      contents: read
    uses: Comfy-Org/github-workflows/.github/workflows/agents-md-integrity.yml@<sha>  # v1
    with:
      workflows_ref: <same-sha>   # pin the checker to the same ref as `uses:`

Testing

  • python3 -m unittest discover -s .github/agents-md-integrity/tests → 16 passing.
  • CLI smoke-tested end-to-end: a compliant repo exits 0; a drifted repo (over-ceiling AGENTS.md, divergent CLAUDE.md, .cursorrules) exits 1 with clear per-check output.

Judgment calls / notes

  • require_codeowners is a boolean (default false = warn-only, true = fail), which reads as the ticket's "input-gated, default: warn only".
  • CODEOWNERS matching is a best-effort gitignore-style matcher (leading-/ anchoring, basename-at-any-depth, */**, last-match-wins incl. owner-less unassign). It's a warn-only check by default, so approximate matching is acceptable; exotic patterns may under-match.
  • workflows_ref defaults to main and must be pinned by the caller to match the uses: SHA for full reproducibility — same limitation/pattern as cursor-review.yml. Documented in the header + README.
  • Line counting uses splitlines() (number of text lines), which is wc -l-equivalent for newline-terminated files and one more (correctly) for a file with no trailing newline.
  • Out of scope per the ticket: wiring the thin caller into each consumer repo (a separate per-repo rollout once a v1 tag is cut), semantic/drift checks, and content-quality checks.

Scoped to Comfy-Org/github-workflows; no consumer repos are modified.

…md standard

Adds a SHA-pinnable reusable workflow (on: workflow_call) that checks a caller
repo's AGENTS.md against the Comfy standard: a thin top-level AGENTS.md source
of truth under a hard line ceiling (default 200, warn 150), a one-line
@AGENTS.md CLAUDE.md shim rather than a divergent copy, no legacy .cursorrules,
per-subtree shims + ceiling for nested monorepo AGENTS.md files, and a
CODEOWNERS DRI (warn-only by default). Fails with a non-zero exit and GitHub
annotations so it wires in as a required status check.

The checker lives in .github/agents-md-integrity/ as the single source of truth
(loaded from a pinned ref of this repo, never the caller's checkout). Adds a
unittest suite plus a self-test workflow exercising passing and failing cases,
and a README row.
@coderabbitai

coderabbitai Bot commented Jul 18, 2026

Copy link
Copy Markdown

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 48 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: ae27de58-2367-4af4-87a6-2073b81f8cdc

📥 Commits

Reviewing files that changed from the base of the PR and between 779e281 and 9b12546.

📒 Files selected for processing (6)
  • .github/agents-md-integrity/README.md
  • .github/agents-md-integrity/check_agents_md.py
  • .github/agents-md-integrity/tests/test_check_agents_md.py
  • .github/workflows/agents-md-integrity.yml
  • .github/workflows/test-agents-md-integrity.yml
  • README.md
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch matt/be-3368-agents-md-integrity
✨ Simplify code
  • Create PR with simplified code
  • Commit simplified code in branch matt/be-3368-agents-md-integrity

Comment @coderabbitai help to get the list of available commands.

@mattmillerai mattmillerai added the agent-coded Authored by the agent-work loop label Jul 18, 2026
@mattmillerai
mattmillerai marked this pull request as ready for review July 18, 2026 00:07
@mattmillerai mattmillerai added the cursor-review Multi-model cursor review label Jul 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-coded Authored by the agent-work loop cursor-review Multi-model cursor review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant