Untitled

name: AI-Augmented Development Pipeline overview: A roadmap for progressively automating the development feedback loop – from manual review relay to autonomous agent-to-agent iteration – building on the quality gates established by the Testing Framework Proposal. todos:

id: phase05-auto-review-response content: “Phase 0.5: Cloud agent autonomously responds to Copilot review comments, iterates until clean, delivers summary to developer” status: pending
id: phase09-auto-merge-ab content: “Phase 0.9: A/B test automated merging on low-risk, sandboxed change types” status: pending
id: agent-sandbox-infra content: “Agent sandbox infrastructure: isolated environments for autonomous agent execution” status: pending isProject: false

AI-Augmented Development Pipeline

Jira Epic: EVO-3475 | Status: Backlog – Planned for future sprints | Depends on: EVO-3474

Executive Summary

Genesis developers already use AI coding agents (Cursor) for daily development and GitHub Copilot for automated code review. But every handoff between these agents still routes through a human – the developer manually relays review feedback to the coding agent, manually pushes fixes, and manually checks for new comments. These relay steps are the bottleneck in an otherwise increasingly automated workflow.

This proposal describes a progressive path from the current human-in-the-middle model to a human-at-the-end model, where agents handle the iterative review-fix cycle autonomously and the developer’s role shifts from relay to final approver.

This proposal depends on the Quality Gates Proposal. Without automated tests, contracts, and CI gates, unsupervised agent changes are unsafe. The quality gates are the backpressure that makes agent autonomy viable.

The Problem: Humans as Message Relays

Today’s development loop at Genesis:

Developer prompts Cursor --> Cursor writes code --> Developer reviews -->
Developer commits (pre-commit hook runs) --> Developer pushes -->
Tech Lead manually triggers Copilot review -->
Tech Lead manually pings developer "left some comments" -->
Developer reads comments --> Developer tells Cursor "fix these comments" -->
Cursor fixes --> Developer pushes again --> Tech Lead reviews --> Merge

The developer is the relay at every step. Each handoff introduces:

Latency: Minutes to hours between review comments appearing and the developer noticing
Context switches: Developer has moved on to other work, must mentally reload the PR
Manual toil: The Tech Lead has had to manually ping about review comments 10+ times in recent weeks
Lost feedback loops: When the relay is slow, iteration stalls. A 3-round review that could converge in 20 minutes takes 2 days

The Quality Gates Proposal Phase 0 addresses the notification gap (auto-trigger reviews, auto-notify developers). This proposal picks up where that leaves off: automating the response to review feedback, not just the notification.

Agent Backpressure: Why Quality Gates Come First

Agent backpressure refers to automated feedback mechanisms – tests, linters, type systems, contracts – that allow agents to detect and correct their own mistakes without human intervention.

An autonomous agent responding to review comments without backpressure is dangerous. It might “fix” a Copilot comment by introducing a bug that no automated system catches. The quality gates are what make the autonomous loop safe:

Quality Gate	What it catches	Required for
Pre-commit (Phase 1)	Obvious breaks in changed files	Agent self-correction before push
CI unit tests (Phase 1)	Behavioral regressions	Blocking bad agent pushes from merging
Contract validation (Phase 2)	Data shape violations	Preventing agent changes from corrupting production data
Integration tests (Phase 3)	Cross-service breakage	Catching agent changes that affect multi-collection workflows
E2E tests (Phase 5)	User-facing regressions	Final verification that agent changes work end-to-end

Phase 0.5 of this proposal requires at minimum Phase 1 of the Quality Gates Proposal (CI gate) to be operational. Phase 0.9 (automated merging) requires all 5 phases.

Current State: Human-Driven, Agent-Assisted

The team is already deeply agent-assisted:

Cursor is used by every developer for daily coding. Agents are stateless – they rely on codified standards (cursor rules, .mdc files, DATA_ACCURACY_MANIFESTO.md) for context
GitHub Copilot provides automated code review on PRs, with structured comments and change suggestions
The agent review-response workflow already exists informally: the developer tells Cursor about the comments, Cursor reads them via the GitHub API, makes changes, pushes, and resolves conversations. This works ~95% of the time without human override – Copilot provides good enough explanations for the coding agent to make correct decisions
What’s missing is the plumbing between the two agents. Copilot’s output sits in GitHub until a human notices it and relays it to Cursor

What’s codified vs what’s implicit

Aspect	Current state
Coding standards	Codified (cursor rules, `.mdc` files, manifesto docs)
Data accuracy principles	Codified (`DATA_ACCURACY_MANIFESTO.md`)
Test expectations	Partially codified (some tests exist, no enforcement)
Review-response workflow	Implicit (developers know how, not documented)
Agent-to-agent handoff	Manual (human relay)
Agent autonomy boundaries	Undefined

Proposed Phases

Phase 0.5: Autonomous Review Response

Goal: A cloud-hosted agent reads Copilot review comments, makes fixes, pushes changes, and iterates until the review converges – without human involvement in the loop. The developer receives a summary and does a final approval.

Prerequisite: Quality Gates Phase 1 (CI gate operational)

The loop:

Key design decisions:

Where does the agent run?

Option A: GitHub Actions workflow triggered by pull_request_review or pull_request_review_comment events. Simplest to set up, runs in GitHub’s infrastructure, has native repo access. Limited by Actions runtime constraints (6 hour max, ephemeral environment).
Option B: Cloud function (AWS Lambda, Cloud Run) triggered by GitHub webhook. More flexible, can maintain state across invocations, can use more powerful compute. More infrastructure to manage.
Option C: Dedicated service (long-running process on EC2/ECS). Most flexible, can maintain persistent context, can handle complex multi-step workflows. Most infrastructure overhead.
Recommendation for Genesis’s current scale: Start with Option A (GitHub Actions). It’s the lowest-overhead option, integrates natively with the existing CI pipeline, and can be upgraded to Option B/C later if runtime constraints become a limitation.

Circuit breaker design:

Maximum iteration rounds (suggest 3). If Copilot keeps raising new comments after 3 rounds, escalate.
Test failure at any point in the loop triggers immediate escalation.
If the agent’s changes increase the diff size by more than 2x the original PR, escalate (the fix is bigger than the change – a human should evaluate).
All escalations include full context: what was tried, what failed, and the current state of the branch.

Summary report: When the loop converges, the developer gets a structured summary:

Number of comments raised / resolved / escalated
Diff between original PR and post-review version
CI results (all passing)
Any comments the agent chose not to address, with reasoning
Time elapsed in the autonomous loop

What the agent can and cannot do:

Can: Modify code in the PR’s changed files, push commits, resolve review conversations, request re-review
Cannot: Merge the PR, modify files outside the PR’s scope, change CI configuration, modify repository settings, dismiss reviews
These boundaries should be enforced by the bot token’s permissions, not just by instructions to the agent

Effort estimate: 1-2 sprints (design + implement + iterate). The GitHub Actions approach is ~3 days for a basic version; hardening, circuit breakers, and summary reports add another sprint.

Phase 0.9: A/B Test Automated Merging

Goal: For well-defined, low-risk change types, test whether automated merging (after all gates pass) is safe and beneficial.

Prerequisites: Quality Gates Phases 0-5 fully operational. Phase 0.5 stable and trusted. Team consensus on scope.

This phase is explicitly experimental. It should only be attempted after the team has high confidence in the quality gates and after Phase 0.5 has been running successfully for several weeks.

Candidate change types for auto-merge (lowest risk first):

Dependency version bumps (Dependabot/Renovate PRs) – already machine-generated, tests verify compatibility
Documentation-only changes – no code execution impact
Cursor rule / .mdc file updates – low blast radius
Auto-generated migration scripts – if contracts and integration tests cover the migration path
Bug fixes with accompanying test – the test proves the fix works

Auto-merge criteria (all must pass):

All CI gates pass (unit, contract, integration)
Copilot review converges with zero unresolved comments
PR scope is within the defined change types
Diff size is below a threshold (e.g., < 200 lines changed)
No changes to security-sensitive files (auth, tokens, permissions)
E2E tests pass on staging deploy (for changes that trigger deployment)

Measurement:

Track auto-merge rate, rollback rate, and bug introduction rate
Compare against manual-merge baseline for the same change types
If rollback rate exceeds threshold (e.g., > 5%), pause auto-merge and investigate

Sandboxing approach:

Start with a separate, low-stakes repository (e.g., internal tooling, documentation site)
Graduate to a narrow scope within the main codebase (e.g., only docs/ and .mdc files)
Expand scope incrementally based on measured confidence

Effort estimate: 2-3 sprints for initial implementation + measurement infrastructure. Ongoing tuning and scope expansion is continuous.

Agent Sandbox Infrastructure

As agent autonomy increases, the environments agents operate in become critical. An agent that can push code and trigger deployments needs guardrails beyond just tests.

Current sandboxing

Layer	Current state	Adequate for Phase 0.5?
Git branches	Agents work on feature branches, not main	Yes
CI environment	GitHub Actions runners, ephemeral	Yes
Database (tests)	MongoMemoryServer (ephemeral, isolated)	Yes
Database (dev/stage)	Shared Atlas clusters	Risk – agent shouldn’t touch shared DBs
Production	Shared, no agent access	Correct – maintain this boundary

What’s needed for autonomous agent workflows

Isolated execution environments: The cloud agent (Phase 0.5) should run in a container with only repo access and GitHub API access. No database credentials, no AWS access, no production secrets.
Scoped bot tokens: The GitHub token used by the cloud agent should have narrowly scoped permissions: read/write on the specific PR’s branch, read on review comments, write on review conversation resolution. No admin access, no ability to modify branch protection rules.
Ephemeral database for agent testing: When agents need to verify data-layer changes, they should use MongoMemoryServer with seed fixtures (Phase 3 of Quality Gates), not shared environments.
Local DB copies (future consideration): When agents begin running multi-step workflows (write migration, run it, verify results, write API endpoint), a local copy of realistic data becomes valuable. See the Local Developer Databases section in the Quality Gates Proposal for tradeoffs. This becomes more important as agent autonomy increases, because the cost of an agent corrupting a shared environment is higher when no human is watching in real-time.

Making the Codebase Agent-Ready

Beyond infrastructure, the codebase itself should be structured for agent consumption. Agents are stateless – they read what’s in the repo and act on it. The better the repo communicates intent, constraints, and expectations, the better the agent performs.

What’s already working

Cursor rules (.mdc files): Coding standards, testing practices, PostHog integration rules
**DATA_ACCURACY_MANIFESTO.md:** Principles for handling metrics data
Architecture docs: Platform API client patterns, contract testing strategy
TypeScript types: Strong type definitions serve as implicit contracts

What should be improved

Test naming conventions: Agents should be able to find relevant tests for any file they modify. A predictable convention (e.g., foo.js tested by foo.test.js or foo.unit.test.js) makes this reliable. Currently test file names don’t always match source files.
Contract error messages: Should be descriptive enough for an agent to self-correct. Not “validation failed” but “metric at index 2 has undefined title, expected non-empty string matching pattern from metricsNormalizer canonical list.”
CI output structure: Agent-parseable output (JSON or structured logs) alongside human-readable summaries. When a cloud agent reads CI failure output, it should be able to programmatically identify which test failed, in which file, with what assertion.
Review response guidelines: A documented .mdc rule that instructs Cursor agents on how to respond to Copilot review comments – when to apply suggestions directly, when to push back with reasoning, when to escalate to the developer.
Scope boundaries: Clear documentation of which files/directories agents should never modify autonomously (e.g., CI config, deployment scripts, security-sensitive modules).

Implementation Timeline

Phase 0.5: ~5-6 weeks after Quality Gates Phase 1 is operational Phase 0.9: ~3-4 months after Phase 0.5 stabilizes

Risks and Mitigations

Risk	Mitigation
Agent introduces bugs while “fixing” review comments	CI gate (Phase 1 prerequisite) catches regressions. Circuit breaker escalates on test failure
Infinite review loop (agent and Copilot disagree)	Circuit breaker after N rounds. Escalate to developer with full context
Agent makes changes outside PR scope	Bot token permissions restrict to PR branch. File-scope rules in agent instructions
Auto-merged change causes production issue	Start with lowest-risk change types only. Measure rollback rate. Kill switch to disable
Team discomfort with agent autonomy	Progressive rollout. Human always has final approval in Phase 0.5. Phase 0.9 is opt-in and sandboxed
Over-reliance on agents degrades developer skills	Agents handle iteration, not architecture. Developers still own design decisions, code review judgment, and final approval

Success Metrics

Metric	Before	After Phase 0.5	After Phase 0.9
Review-to-fix turnaround	Hours (manual relay)	Minutes (autonomous)	Minutes
Manual “left comments” pings per week	~10	0	0
Developer time spent on review iteration	~30 min/PR	~5 min (review summary)	~2 min (for auto-merge scope)
PRs requiring manual review relay	100%	~20% (escalations only)	~10%
Agent-introduced regressions caught by CI	N/A (no autonomous agents)	Tracked	Tracked, compared to auto-merge rate

Relationship to Other Proposals

Quality Gates Proposal (EVO-3474): Hard prerequisite. Phase 0 (review notification) and Phase 1 (CI gate) must be operational before Phase 0.5 begins. Phases 2-5 progressively strengthen the safety net for autonomous agent changes.
EVO-3330 (Social Metrics Refactor): The contract definitions and test targets in Phase 2 of Quality Gates depend on EVO-3330. This proposal inherits that dependency transitively.
EVO-3469/3470/3471 (API Client Pattern): New platform API clients should ship with contracts and tests from day one, strengthening the backpressure surface for agent-generated changes in those domains.

Open Questions for Team Discussion

Comfort level with autonomous agent commits: Is the team comfortable with a bot pushing commits to PRs without developer involvement in the loop? What guardrails would make this acceptable?
Review authority: Should agent-resolved review comments count as “addressed” for the purposes of PR approval, or should a human re-verify each resolution?
Scope of Phase 0.9: Which change types does the team consider safe for auto-merge experimentation? Is a separate repo acceptable as a starting sandbox?
Cloud agent identity: Should the cloud agent use a dedicated bot account (e.g., genesis-bot) or act under the PR author’s identity? Bot account is more transparent; author identity is simpler for permissions.
Budget: Cloud agent execution has compute costs (GitHub Actions minutes, or cloud function invocations). What’s the acceptable cost per PR for autonomous review response?