Skip to content

fix(web-security): finalize produces item schemas so report_item builds#90

Merged
GangGreenTemperTatum merged 1 commit into
mainfrom
ads/cap-1152-fix-structured-report-output-in-web-security-manifest
Jul 14, 2026
Merged

fix(web-security): finalize produces item schemas so report_item builds#90
GangGreenTemperTatum merged 1 commit into
mainfrom
ads/cap-1152-fix-structured-report-output-in-web-security-manifest

Conversation

@GangGreenTemperTatum

Copy link
Copy Markdown
Contributor

Problem

The web-security manifest declares structured output types via produces:
(web_vulnerability, web_endpoint, plus built-in finding/asset), but in
production the typed report_item tool was never offered to the agent — it
silently fell back to the generic report tool, so no structured findings
appeared in the app.

Confirmed in session 0896781d-1f46-4726-ba02-596c3051eb39: 6 report calls,
zero report_item calls, zero structured items emitted.

Root cause

items.py uses from __future__ import annotations (PEP 563), so field
annotations like severity: Severity are stored as strings and resolved
lazily. The SDK capability loader and OCI packager import items.py by path
(spec_from_file_location + exec_module) without registering it in
sys.modules. In that state Pydantic cannot resolve the Severity forward
reference, so the first model_json_schema() call — made by both the
report_item tool build and the package builder — raises
PydanticUserError: ... is not fully defined. That error is swallowed
upstream, dropping the entire report_item / update_item / link_items
tool set.

Fix

Rebuild every BaseModel at module scope so forward references resolve against
the module globals regardless of how the file is imported. The loop covers all
current models and any added later — KISS and self-maintaining, no per-model
wiring.

Testing (against the real installed SDK, dreadnode 2.0.33)

  • runtime resolver _resolve_produces_models imports both custom models
  • build_capability_report_item yields item_type enum
    {finding, asset, web_vulnerability, web_endpoint}
  • build-time _resolve_produced_item_types embeds both JSON schemas
  • negative control: without the fix, model_json_schema() fails as reported
  • new tests/test_items_produces.py (6 tests) reproduces the loader's exact
    import path and asserts every declared model builds a schema
  • dreadnode capability validate --strict passes (only unrelated caido/burp
    binary warnings)

Note: complementary SDK bug (tracked separately)

The SDK capability loader (_parse_capability_file) does not thread produces
(or legacy items) into CapabilityManifest, and the loader/packager omit
sys.modules registration before exec_module. This capability-side fix is
necessary and correct, but full publish-path coverage also needs the SDK
change. Filed internally.

The web-security manifest declares structured output types via `produces`
(web_vulnerability, web_endpoint plus built-in finding/asset), but the typed
`report_item` tool was never offered to the agent in production — it silently
fell back to the generic `report` tool, so no structured findings appeared.

Root cause: items.py uses `from __future__ import annotations` (PEP 563), so
field annotations are strings resolved lazily. The SDK capability loader and
OCI packager import items.py by path (spec_from_file_location + exec_module)
without registering it in sys.modules. In that state Pydantic cannot resolve
the `Severity` forward reference, and the first model_json_schema() call — made
by both the report_item tool build and the package builder — raises
`PydanticUserError: ... is not fully defined`. That error is swallowed
upstream, dropping the entire report_item/update_item/link_items tool set.

Fix: rebuild every BaseModel at module scope so forward references resolve
against the module globals regardless of how the file is imported. The loop
covers all current models and any added later — KISS and self-maintaining, no
per-model wiring.

Verified against the real installed SDK (dreadnode 2.0.33):
- runtime resolver `_resolve_produces_models` imports both custom models
- `build_capability_report_item` yields item_type enum
  {finding, asset, web_vulnerability, web_endpoint}
- build-time `_resolve_produced_item_types` embeds both JSON schemas
- negative control: without the fix, model_json_schema() fails as before

Adds tests/test_items_produces.py which reproduces the loader's exact import
path (no sys.modules registration) and asserts every declared model builds a
schema, guarding against silent regression when new types are added.
@GangGreenTemperTatum GangGreenTemperTatum merged commit e5ba8ef into main Jul 14, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant