How evaluation works
The pipeline
- Webhook. Bitbucket or GitHub calls ArchGuard. The request is accepted only if its signature matches the repository's secret. ArchGuard answers
202 Acceptedquickly and queues the work. No language model call happens during the request. - De-duplication. Bitbucket and GitHub may resend an event. ArchGuard remembers each request ID for 24 hours and ignores repeats, so you never get two drafts for one event.
- Worker. A background worker picks up the job and sets the build status to in progress.
- Triage. The worker lists the changed files and checks them against the structural triggers.
- Evaluation. If something structural changed, the relevant part of the diff and your enabled rules go to the language model.
- Result. ArchGuard saves the result, updates the pull request comment and sets the final build status.
Triage: structural triggers
Triage keeps the cost and noise down. A pull request that changes no file matching a trigger is passed without any language model call. Its check says ArchGuard: No architectural impact detected.
What the model sees
- The pull request title, the list of changed files and the existing ADR titles.
- Your enabled rules that apply to the changed files. A rule applies when one of its target patterns matches a changed file, or when it has no patterns.
- Only the diff sections for files that matched a trigger or a rule's target patterns. Other files are not sent.
The model must answer in a fixed JSON format. ArchGuard validates the answer and keeps only violations that name a real rule and a file that is in the diff. The severity always comes from the rule, never from the model. If the answer is invalid, ArchGuard asks once more. If it is still invalid, the result is neutral.
Size limits
Very large pull requests are not sent to the model. The limits are the diff size (MAX_DIFF_BYTES, default 200,000 bytes) and the input size (MAX_LLM_INPUT_TOKENS, default 60,000 tokens). Over either limit, the result is neutral with the description "too large to evaluate", and the comment asks for a manual review.
Retries and failures
If something goes wrong, such as a temporary Bitbucket or model outage, the job is retried up to four times with increasing delays. If the last attempt fails, ArchGuard sets a neutral, passing status ("evaluation failed (neutral)") so the pull request is never left waiting on in progress.
Re-runs
Each push to the pull request starts a new evaluation. The same pull request comment is updated, and the pending ADR draft for that pull request is refreshed rather than duplicated. The newest build status replaces the older one.
What is stored
ArchGuard keeps the outcome, the list of violations (rule, file, line, short explanation), token usage and the ADR draft. It does not store the diff or any source code. See Security and data handling.