Back to flagship work
Case Study · Developer Tooling · Multi-Agent PR Review PlatformCI Test Matrix: 690+ Passed

AI Code Review Agent

Deployment model: Docker Compose + Caddy TLS · GitHub Action · Stdio MCP

Zero-token static/AST pre-scans + 3-agent CrewAI Flows crew + plain-code deterministic reconciler, durable SQLite WAL queue, 8 MCP tools, and 690+ automated tests.

A full-stack multi-agent pull request review platform (React 19 Workbench, FastAPI webhook gateway, CLI, reusable GitHub Action, GitLab/Bitbucket adapters, and stdio MCP server) that runs deterministic scanners and AST call-graph indexing before LLM calls, reconciles findings in plain code, and ships with 690+ automated tests across Python 3.10–3.13.

Python 3.10–3.13CrewAI FlowsFastAPITree-SitterSemgrep / Bandit / RuffModel Context ProtocolSQLite WALReact 19Docker ComposeOASIS SARIF 2.1.0
AI Code Review Agent narrated walkthrough frame
Narrated walkthrough previewGitHub source

Repository-Backed Evidence

Scoped Metrics & Verification

Linked to GitHub README
690+

Automated Tests (Py 3.10–3.13)

Full unit, integration, security, and phase-gate suite running on a Python 3.10–3.13 CI matrix alongside Ruff, mypy, pip-audit, and Gitleaks secret scans.

Inspect verification source
100.0%

14-Case Regression Suite

Matches by file, exact line, and defect type — counting every unmatched hit as a false positive (18 TP, 0 FP, 0 FN; upgraded from an initial 79.2% baseline). Scoped synthetic regression fixtures, not universal accuracy across arbitrary external codebases.

Inspect verification source
8 MCP · 15 Cmds

IDE & PR Bot Surface

Stdio MCP server exposing 8 review tools to Claude Code & Cursor, plus 15 PR slash commands with author-association RBAC across GitHub, GitLab, and Bitbucket.

Inspect verification source
20 / 20

Public PR Corpus V2

100.0% complete deterministic capture across 20 frozen real Flask pull requests (24 sourced findings, 3.72s mean PR latency; human adjudication tracked separately).

Inspect verification source

System Architecture Topology

End-to-end dataflow and defensive isolation layers from request entry to verified output.

AI Code Review Agent system architecture diagram

Core Architecture Pillars

Layered engineering safeguards designed so no single model call is the sole boundary.

Pillar 01Zero Token Cost

Pillar 1: Zero-Token Deterministic Pre-Scanners & AST Index

Semgrep, Bandit, Ruff, a custom AST constant folder, Trojan Source Bidi stripping, supply-chain slopsquatting checks, YAML policies, and a 6-language Tree-Sitter call-graph index all run before any LLM call — strictly limiting what agents can claim.

  • Semgrep, Bandit, Ruff + custom AST constant folder & taint scanner
  • 6-language Tree-Sitter call-graph & impacted-caller index
  • Trojan Source Bidi stripping & slopsquatting manifest checks
  • Zero-token .code-review.yaml team governance rules
Pillar 02Deterministic Scoring

Pillar 2: 3-Agent CrewAI Flows Crew + Plain-Code Reconciler

Runs Senior Developer, Security Engineer, and Tech Lead agents in parallel via CrewAI Flows with provenance guardrails. A reconciler written in plain Python deduplicates findings and computes confidence from a documented rubric instead of a second LLM pass.

  • Parallel 3-agent CrewAI Flows crew with provenance guardrails
  • Plain-code deterministic reconciler & documented scoring rubric
  • Bounded tool loop (codebase context, semantic RAG, Ruff, custom rules)
  • Opt-in adversarial verifier & provider-agnostic LLMFactory
Pillar 03Webhook Ops

Pillar 3: Durable SQLite WAL Queue & Fail-Closed Gateway

FastAPI gateway backed by a durable SQLite WAL job queue with HMAC webhook verification, fail-closed Bearer auth, orphaned-job recovery, and SSE streaming to a React 19 Live Annotated Source Viewer.

  • SQLite WAL durable queue with cancel & orphaned-job recovery
  • HMAC webhook signatures, fail-closed Bearer auth & repo allowlists
  • Posts Check Runs, advisory PR comments & OASIS SARIF v2.1.0
  • Posting and auto-remediation off by default (SHADOW_MODE supported)
Pillar 04IDE + CI Matrix

Pillar 4: 8 MCP Tools, 15 Slash Commands & Calibration

Exposes 8 review tools over stdio MCP to Claude Code & Cursor, ships 15 PR slash commands with author-association RBAC across GitHub/GitLab/Bitbucket, and includes a Tricorder-style circuit breaker + 690+ tests on Python 3.10–3.13.

  • 8 stdio MCP tools for Claude Code & Cursor
  • 15 PR slash commands with author-association RBAC
  • GitHub Action + GitLab & Bitbucket adapters
  • Tricorder-style noisy-rule probation + 690+ CI matrix tests

The Problem & Root Cause

AI coding assistants generate clean-looking code that still slips SQL injection, plaintext password checks, hallucinated dependencies, unawaited async auth checks, and invisible Unicode tricks past basic linters. Pure-LLM reviewers add ungrounded noise without deterministic file/line evidence or cross-file call context.

Engineering Contribution

Built the full platform end-to-end: zero-token deterministic pre-scan layer (Semgrep, Bandit, Ruff, custom AST constant folder, Tree-Sitter call-graph index, YAML policy engine), parallel 3-agent CrewAI Flows crew with plain-code deterministic reconciler, FastAPI gateway + durable SQLite WAL job queue with HMAC auth & orphaned-job recovery, 15 PR slash commands, 8-tool MCP server, React 19 Live Annotated Source Viewer, GitHub Action + GitLab/Bitbucket adapters, SARIF v2.1.0 export, and 690+ automated tests across Python 3.10–3.13.

Explore the Workflow

Step through concrete scenarios to see how the system checks boundaries, reproduces failures, or degrades under quota pressure.

Illustrative example

PR diff adds hallucinated package `flask-auth-helpers` + `subprocess.run(shell=True, input=user_input)`.

Step 1 / 4

Pre-Ingestion Shield

Strip Unicode Bidi characters, validate HMAC/Bearer auth, and scan dependency manifests.

flask-auth-helpers -> SEC-SLOPSQUAT-001 (hallucinated dependency flagged)

Architecture Decision Records (ADRs)

Key design choices, technical rationale, and accepted trade-offs behind the implementation.

ADR-01Engineering Trade-off

Run deterministic scanners first to constrain agent claims

Semgrep, Bandit, Ruff, custom AST constant folding, Trojan Source stripping, and Tree-Sitter call-graph indexing all execute at zero token cost before any LLM runs.

Why this decision:Limits hallucinated findings, blocks prompt injection from untrusted diffs, and guarantees file/line/CWE grounding.
Trade-off accepted:Requires maintaining deterministic scanners and Tree-Sitter grammars alongside agent prompts.
ADR-02Engineering Trade-off

Reconcile findings in plain Python instead of a second LLM pass

A plain-code reconciler deduplicates parallel outputs from the 3-agent CrewAI Flows crew and computes confidence from a documented rubric.

Why this decision:Makes final verdicts auditable, repeatable, faster, and cheaper than asking another LLM to judge prose.
Trade-off accepted:Requires explicit scoring weights and provenance contracts rather than open-ended synthesis.
ADR-03Engineering Trade-off

Fail-closed auth & off-by-default write actions

HMAC-validated webhooks, fail-closed Bearer auth, repo allowlists, and off-by-default comment posting / auto-remediation prevent accidental spam or false approvals when an analyzer fails.

Why this decision:Engineering teams only adopt automated review bots that never falsely approve broken PRs or mutate branches unexpectedly.
Trade-off accepted:Requires explicit operator configuration (`REVIEW_ENABLE_POSTING`, pinned sandbox digest) before enabling active write modes.

Walkthrough & Repository

Watch the narrated system walkthrough below, or clone the repository to run the complete Docker Compose + Caddy TLS · GitHub Action · Stdio MCP setup and automated test suite locally.

Full source code, Docker Compose config, test suites & README on GitHubRead repository docs

Limits & Production Lessons

System Boundaries & Known Limitations

  • Deep AST scanning, Bandit, Ruff, and generated pytest tests are Python-first; TypeScript, JavaScript, Go, Java, and Rust receive regex SAST, YAML governance, and Tree-Sitter graph context.
  • The 14-case exact-line regression suite consists of synthetic seeded-defect fixtures (not a human-labelled real-world precision study); Public PR Corpus V2 adjudication is tracked separately.
  • Posting, auto-remediation, and sandboxed test execution are off by default; the current durable queue is single-node SQLite WAL.

What This System Demonstrates

Reliable AI developer tools spend their engineering budget on deterministic pre-scans, plain-code reconciliation, durable queues, and 690+ regression tests — treating LLMs as bounded collaborators inside verifiable guardrails.

Roles & Technical Collaboration

Hiring a GenAI Engineer or tackling a similar AI architecture?

I’m open to full-time remote GenAI & AI Systems Engineering roles, as well as focused architecture and evaluation engagements.