Case study · 2026
A GitHub Copilot agent toolkit for a multi-repo fintech platform
A set of custom agents, skills, guardrail instructions and scripts that my team now uses day to day for code review, delivery and production triage.
At a glance
- Problem
- An AI coding assistant sees one repository at a time and knows nothing about the team’s standards, logs or how the services connect.
- What I built
- Custom agents, skills, instruction files and scripts for code review, story delivery and production triage, with a person approving every irreversible step.
- My part
- I designed and built the toolkit, as tech lead on the team that uses it.
- Result
- Much of my team now uses many of the skills, instructions and scripts day to day.
On this page
01The problem
Out of the box, the assistant lacked context
Our platform is many services and front ends, each in its own repository, in a regulated financial domain. Out of the box, an AI coding assistant sees one file or one repository at a time. It knows nothing about our standards, can’t see production logs, and has no idea how the services connect.
The approach was to onboard the model the way we onboard a new engineer: an accurate map of the system, written standards, the right tools, and clear limits on what it may do without approval.
Out of the box
- Sees one file or one repo at a time
- Doesn’t know the team’s standards
- Can’t see production logs
- No idea how the services connect
With the toolkit
- Reads a context doc for every service it touches
- Follows written-down standards and checklists
- Searches logs through approved tools
- Stops for a person before anything irreversible
02Constraints
Constraints that shaped the design
Each of these constraints shows up somewhere in the design below.
- RegulatedData stays inside approved tools.
- Multi-repoMany services and front ends, each in its own repository.
- Human approvalA person approves anything irreversible.
- VoluntaryUse is optional, so it has to save time from the first day.
- Non-deterministicModels vary, so every output carries evidence a person can check.
03Architecture
Four layers, each with one job
The main design decision for each piece was which layer it belongs in. Instructions hold standards. Skills hold repeatable procedures. Agents combine them to do a whole job. Scripts do the parts that should give the same answer every time. Beside them sit a short context doc for each service and a repo memory of decisions and gotchas, so neither people nor agents have to rediscover them.
Architecture of the agent toolkit: an engineer asks an agent; agents use skills, instructions and scripts, read context docs and repo memory, and reach GitHub, the ticket tracker and logs. A person approves irreversible actions.
.github/versioned and reviewed like code
- 01AgentsA whole jobCombine skills and instructions for one end-to-end task.
- code-reviewer
- story-delivery
- pr-triage
- correlation-triage
- 02SkillsRepeatable proceduresSteps that need judgement, written down once.
- log-root-cause
- regression-trace
- repo-memory
- local-api-wiring
- 03InstructionsStandardsRules applied to every change in matching files.
- security
- safe-actions
- csharp
- angular
- tests
- 04ScriptsThe deterministic partsSame input, same output: no model needed.
- ready
- gauntlet
- standup-digest
- start-local
- GitHubcode, pull requests, reviewsNeeds approval: push · merge · comment
- Ticket trackerstories and bugsNeeds approval: status changes
- Logssearch via MCP or CLIread-only
a person approves first
All of it lives in the repository, next to the code it describes, and changes go through pull requests like any other code. The examples below are rewritten for a fictional demo app, “Fourteener Transfers”, so no internal code is shown.
---description: Reviews a pull request against Fourteener's checklists and groups findings by severity.tools: ['search', 'usages', 'problems', 'changes', 'githubRepo']handoffs: - label: Fix the blocking findings agent: story-delivery prompt: Fix the blocking findings above. One commit per finding. send: false---
# Code reviewer
You review one pull request in the Fourteener Transfers repo. You do not edit files.
## Before you start1. Read `docs/context/<service>.md` for every service the diff touches.2. Read `.memory/repo-memory.md` for known gotchas.
## What to check- Money: amounts are `decimal`, never `double`, and rounding is explicit.- Time: cut-offs are compared in business time (America/Denver), never in UTC.- Logging: new code logs event names with the correlation ID; clients get a code, not a message.- Tests: a behaviour change without a test is a **should-fix**.
## OutputGroup findings as **Blocking**, **Should fix** and **Nit**. For each: file and line, what is wrong,why it matters, and a suggested change. Quote the code. If you're unsure, say what would settle it.Never approve or merge the pull request.---name: log-root-causedescription: Turn one correlation ID into a cross-service timeline and a likely root cause, with evidence. Use when someone pastes a correlation ID or a failed request.---
# Log root cause
## Steps1. Read `docs/context/services.md` to find which services and log sources handle the request.2. Run `trace_correlation_id` (fourteener-logs MCP server). Sort events by time.3. Build a timeline: UTC and business time, service, event, level, one-line summary.4. For the first warning or error, find the code that logged it, then follow the call back to the decision that caused it.5. Check what changed: `git log` on those files since the last known-good release.6. Write the report. Every claim links to a log query, a file and line, or a commit.
## Rules- Read only. Don't edit code, replay requests or touch any environment.- Say how sure you are and what you did not check.- Stop after the report. A person decides what happens next.---applyTo: 'src/**/*.cs'description: Security rules for every C# change, based on the OWASP Top 10.---
# Security (C#)
- Access control: every endpoint declares its authorization policy. No anonymous writes.- Injection: no string-built queries or commands. Use parameters.- Logging: log event names and IDs. Never log names, tokens or full account numbers.- Errors: clients get an error code and the correlation ID, never a stack trace.- Secrets: nothing in code or appsettings files. Read them from configuration at runtime.- Dependencies: a new or upgraded package must pass `scripts/ready.ps1` first.
If a change would break one of these rules, stop and say which rule and why instead of writing it.04One agent, step by step
From a correlation ID to a root cause
Production triage used to mean switching between log searches and several repositories to piece one request together. The triage agent turns that into one conversation that ends with evidence a person can check. The example below uses the demo app, where a refactor broke same-day transfers.
Step 1: Paste a correlation ID
An engineer pastes the ID from a failed request. No other prompt is needed, because the procedure is written down as a skill.
Step 2: Read the map
It reads the context docs to learn which services handle a transfer, where each one’s logs live, and what the event names mean, instead of guessing.
Step 3: Build the log queries
It writes the searches itself, scoped to those services and a time window. At work the logs live in Splunk; in the demo repo a small MCP server searches synthetic log files.
queries the agent wrote trace_correlation_id("7f3c9a2e-5b1d-4c8a-9e55-2a6f0c1d7b34")search_logs("event:EntryRejected code:EFFECTIVE_DATE_MISMATCH since:2026-11-16")Step 4: Assemble a timeline
Events from every service, in order, in UTC and in business time. Here: the transfer was requested at 15:42 Denver time, marked “after cut-off” anyway, and rejected by the ledger.
Step 5: Map errors to code
It finds the line that logged the error, follows it back to the decision that caused it, and checks what changed in those files since the last good release.
CutoffPolicy.cs (since v1.4.0) public TransferSchedule Evaluate(DateTimeOffset requestedAt){var local = requestedAt.UtcDateTime; // was: business timevar businessDate = DateOnly.FromDateTime(local);var afterCutoff = TimeOnly.FromDateTime(local) >= options.Value.SameDayCutoff;Step 6: Propose a root cause, then stop
A likely cause, how sure it is, what it didn’t check, and a link for every claim. Then it waits. A person approves before anything is handed to the delivery agent.
The same run, condensed:
- triage 7f3c9a2e-5b1d-4c8a-9e55-2a6f0c1d7b34
- reading context: transfers-api, ledger-api
- search_logs correlationId="7f3c9a2e…" | sort @t
- timeline: 4 events across 2 services
- 15:42:07 TransferRequested sameDay=true
- 15:42:07 CutoffEvaluated afterCutoff=true
- 15:42:08 ledger-api 409 EFFECTIVE_DATE_MISMATCH
- 15:42:08 transfers-api 502 to client
- mapping to code: Transfers.Api/Scheduling/CutoffPolicy.cs
- regression: "Refactor cut-off evaluation…" (v1.4.0)
- likely cause: cut-off compared in UTC (22:42Z), not Denver business time
- needs your approval before handing to delivery
The report it produces:
Triage report
correlation id 7f3c9a2e-5b1d-4c8a-9e55-2a6f0c1d7b34
Timeline
| UTC | Denver | Service | Level | Event |
|---|---|---|---|---|
| 22:42:07.112 | 15:42:07 | transfers-api | info | TransferRequestedsame-day transfer, acct_1007 → acct_1022 |
| 22:42:07.118 | 15:42:07 | transfers-api | info | CutoffEvaluatedafterCutoff = true · effective 2026-11-19 |
| 22:42:07.164 | 15:42:07 | ledger-api | warn | EntryRejectedEFFECTIVE_DATE_MISMATCH: same-day entry dated 2026-11-19 |
| 22:42:07.171 | 15:42:07 | transfers-api | error | LedgerRejectedTransferledger returned 409; client got 502 |
Likely root cause confidence: high
The cut-off check compares the request time in UTC, not in business time (America/Denver).
Same-day requests from 9:00 Denver time on are marked “after cut-off” and dated the next business day. The ledger still sees a same-day request and rejects it.
Not checked
Weekend requests; how clients retry a 502.
Evidence
- logstrace_correlation_id 7f3c9a2e…4 events, 2 services
- codesrc/Transfers.Api/Scheduling/CutoffPolicy.csEvaluate() reads requestedAt.UtcDateTime
- changeRefactor cut-off evaluation to use DateTimeOffsetv1.4.0, deployed 2026-11-17
- patternSame error on other same-day requests since v1.4.0none before it
05The other agents
Review, delivery and pull requests
The same pattern applied to the team’s most time-consuming recurring work. Each agent ends with output a person can check, and none can complete an irreversible step on its own.
code-reviewer
Code review
Checks C# and Angular changes against the team’s own checklists before a person reviews. Findings come back grouped as blocking, should-fix and nit, each with the file, the line and a suggested change.
Never approves or merges.
story-delivery
Story delivery
Takes a ticket end to end: finds the affected services, writes the code and tests, opens the pull request and prepares the QA hand-off.
Stops for approval before each push, PR and ticket change.
pr-triage
Pull request triage
Checks the health of open pull requests (builds, review comments, merge conflicts) and drafts replies to reviewers.
Drafts replies; a person sends them.
- Regression tracingFrom a defect back to the change that introduced it.
- Repo memoryDecisions and gotchas, kept with the code.
- Local API wiringRuns a front end against local services in one step.
- Standup digestYesterday’s commits and PRs across repos, summarised.
- Pre-push preflightBuild, tests, lint and dependency checks before a push.
- Flaky-test gauntletReruns a suspect test until it proves itself either way.
06Guardrails and verification
Guardrails and verification
Guardrails are what made the team willing to adopt the toolkit. Every generated change follows security instructions based on the OWASP Top 10. New or upgraded packages go through a dependency preflight. Generated files follow the same rules as hand-written ones. And nothing irreversible happens without a person saying yes.
| May do alone | Asks a person first |
|---|---|
| Read code, docs, logs and tickets | Push a branch, open or merge a pull request |
| Run builds, tests and read-only scripts | Change a ticket’s status or assignee |
| Create a local branch and commit to it | Send or post anything a person will read |
| Draft PR descriptions, review notes and replies | Add or upgrade a package |
| Propose a root cause with evidence | Touch any environment beyond the local machine |
The table is written down as an instruction file that every agent follows. The demo version:
---applyTo: '**'description: What an agent may do on its own, and what needs a person's approval.---
# Safe actions
You may, without asking:- read code, docs, logs and tickets- run builds, tests and the read-only scripts in `scripts/`- create a local branch and commit to it- draft text: PR descriptions, review notes, ticket updates, replies
You must stop and ask first, every time:- `git push`, opening or merging a pull request, deleting a branch- changing a ticket's status or assignee- sending or posting anything a person will read- adding or upgrading a package- anything that touches an environment beyond your local machine
When you stop, show exactly what you are about to do (the command, the diff, the text)and wait for a clear yes. A yes covers that one action only.Model output is not deterministic, so the design relies on output being easy to verify rather than always correct:
- Evidence over assertions. Every finding or root cause links to a log query, a file and line, or a commit.
- Scripts for anything that should be repeatable. Builds, test runs, dependency checks and digests are scripts, not prompts.
- People review the output. Agent work goes through the same pull-request review as anyone else’s.
07Adoption
How the team adopted it
Adoption was voluntary. I started with the recurring work that cost the team the most time, kept outputs easy to check, and iterated on feedback from the people using it. Everything is versioned and reviewed like code, so anyone can see what an agent was told and propose a fix.
Much of my team now uses many of the skills, instructions and scripts day to day.
08Lessons
Lessons and next steps
- Context is the product. Most of the quality came from the context docs and repo memory, not from clever prompts.
- Pick the right layer. When an agent does the same steps the same way every time, that’s a script. When it needs judgement, it’s a skill. Moving work down a layer, from skill to script, made it faster and more reliable.
- Guardrails build trust. Approval gates and evidence links are what made the team willing to use it on real work.
- Next: test the agents like code. I want a small set of past incidents to rerun against the triage agent whenever its instructions change, so a change to its instructions can’t silently make it worse.