Work

Case study · 2026

A GitHub Copilot agent toolkit for a multi-repo fintech platform

A set of custom agents, skills, guardrail instructions and scripts that my team now uses day to day for code review, delivery and production triage.

Role
Designer and builder, tech lead
When
Sep 2026 – present
Setting
Regulated fintech, multi-repo
Stack
  • GitHub Copilot custom agents
  • Skills
  • Instructions
  • MCP
  • PowerShell
  • Splunk
  • .NET
  • Angular

At a glance

Problem
An AI coding assistant sees one repository at a time and knows nothing about the team’s standards, logs or how the services connect.
What I built
Custom agents, skills, instruction files and scripts for code review, story delivery and production triage, with a person approving every irreversible step.
My part
I designed and built the toolkit, as tech lead on the team that uses it.
Result
Much of my team now uses many of the skills, instructions and scripts day to day.
On this page
  1. 01 Out of the box, the assistant lacked context
  2. 02 Constraints that shaped the design
  3. 03 Four layers, each with one job
  4. 04 From a correlation ID to a root cause
  5. 05 Review, delivery and pull requests
  6. 06 Guardrails and verification
  7. 07 How the team adopted it
  8. 08 Lessons and next steps

Out of the box, the assistant lacked context

Our platform is many services and front ends, each in its own repository, in a regulated financial domain. Out of the box, an AI coding assistant sees one file or one repository at a time. It knows nothing about our standards, can’t see production logs, and has no idea how the services connect.

The approach was to onboard the model the way we onboard a new engineer: an accurate map of the system, written standards, the right tools, and clear limits on what it may do without approval.

Out of the box

  • Sees one file or one repo at a time
  • Doesn’t know the team’s standards
  • Can’t see production logs
  • No idea how the services connect

With the toolkit

  • Reads a context doc for every service it touches
  • Follows written-down standards and checklists
  • Searches logs through approved tools
  • Stops for a person before anything irreversible
The model is the same in both columns. What changes is the context it reads and the actions it is allowed to take.

Constraints that shaped the design

Each of these constraints shows up somewhere in the design below.

  • RegulatedData stays inside approved tools.
  • Multi-repoMany services and front ends, each in its own repository.
  • Human approvalA person approves anything irreversible.
  • VoluntaryUse is optional, so it has to save time from the first day.
  • Non-deterministicModels vary, so every output carries evidence a person can check.

Four layers, each with one job

The main design decision for each piece was which layer it belongs in. Instructions hold standards. Skills hold repeatable procedures. Agents combine them to do a whole job. Scripts do the parts that should give the same answer every time. Beside them sit a short context doc for each service and a repo memory of decisions and gotchas, so neither people nor agents have to rediscover them.

Architecture of the agent toolkit: an engineer asks an agent; agents use skills, instructions and scripts, read context docs and repo memory, and reach GitHub, the ticket tracker and logs. A person approves irreversible actions.

Youask in Copilot Chat

.github/versioned and reviewed like code

  1. 01Agents
    A whole jobCombine skills and instructions for one end-to-end task.
    • code-reviewer
    • story-delivery
    • pr-triage
    • correlation-triage
  2. 02Skills
    Repeatable proceduresSteps that need judgement, written down once.
    • log-root-cause
    • regression-trace
    • repo-memory
    • local-api-wiring
  3. 03Instructions
    StandardsRules applied to every change in matching files.
    • security
    • safe-actions
    • csharp
    • angular
    • tests
  4. 04Scripts
    The deterministic partsSame input, same output: no model needed.
    • ready
    • gauntlet
    • standup-digest
    • start-local
Context docsOne per service: what it owns, where its logs live, its gotchas.
Repo memoryDecisions and lessons the agents (and people) should not relearn.
  • GitHubcode, pull requests, reviewsNeeds approval: push · merge · comment
  • Ticket trackerstories and bugsNeeds approval: status changes
  • Logssearch via MCP or CLIread-only

a person approves first

All of it lives in the repository, next to the code it describes, and changes go through pull requests like any other code. The examples below are rewritten for a fictional demo app, “Fourteener Transfers”, so no internal code is shown.

.github/agents/code-reviewer.agent.md
---
description: Reviews a pull request against Fourteener's checklists and groups findings by severity.
tools: ['search', 'usages', 'problems', 'changes', 'githubRepo']
handoffs:
- label: Fix the blocking findings
agent: story-delivery
prompt: Fix the blocking findings above. One commit per finding.
send: false
---
# Code reviewer
You review one pull request in the Fourteener Transfers repo. You do not edit files.
## Before you start
1. Read `docs/context/<service>.md` for every service the diff touches.
2. Read `.memory/repo-memory.md` for known gotchas.
## What to check
- Money: amounts are `decimal`, never `double`, and rounding is explicit.
- Time: cut-offs are compared in business time (America/Denver), never in UTC.
- Logging: new code logs event names with the correlation ID; clients get a code, not a message.
- Tests: a behaviour change without a test is a **should-fix**.
## Output
Group findings as **Blocking**, **Should fix** and **Nit**. For each: file and line, what is wrong,
why it matters, and a suggested change. Quote the code. If you're unsure, say what would settle it.
Never approve or merge the pull request.
.github/skills/log-root-cause/SKILL.md
---
name: log-root-cause
description: Turn one correlation ID into a cross-service timeline and a likely root cause, with evidence. Use when someone pastes a correlation ID or a failed request.
---
# Log root cause
## Steps
1. Read `docs/context/services.md` to find which services and log sources handle the request.
2. Run `trace_correlation_id` (fourteener-logs MCP server). Sort events by time.
3. Build a timeline: UTC and business time, service, event, level, one-line summary.
4. For the first warning or error, find the code that logged it, then follow the call back
to the decision that caused it.
5. Check what changed: `git log` on those files since the last known-good release.
6. Write the report. Every claim links to a log query, a file and line, or a commit.
## Rules
- Read only. Don't edit code, replay requests or touch any environment.
- Say how sure you are and what you did not check.
- Stop after the report. A person decides what happens next.
.github/instructions/security.instructions.md
---
applyTo: 'src/**/*.cs'
description: Security rules for every C# change, based on the OWASP Top 10.
---
# Security (C#)
- Access control: every endpoint declares its authorization policy. No anonymous writes.
- Injection: no string-built queries or commands. Use parameters.
- Logging: log event names and IDs. Never log names, tokens or full account numbers.
- Errors: clients get an error code and the correlation ID, never a stack trace.
- Secrets: nothing in code or appsettings files. Read them from configuration at runtime.
- Dependencies: a new or upgraded package must pass `scripts/ready.ps1` first.
If a change would break one of these rules, stop and say which rule and why instead of writing it.
An agent, a skill and an instruction file from the demo repo. The highlighted line is the approval gate: the hand-off to the delivery agent is offered, never sent on its own.

From a correlation ID to a root cause

Production triage used to mean switching between log searches and several repositories to piece one request together. The triage agent turns that into one conversation that ends with evidence a person can check. The example below uses the demo app, where a refactor broke same-day transfers.

Correlation ID to root cause, in six steps1, an engineer pastes a correlation ID. 2, the agent reads the context docs. 3, it builds the log queries. 4, it assembles a cross-service timeline. 5, it maps errors to code. 6, it proposes a likely root cause with evidence. Then it stops: a person approves before anything is handed to the delivery agent.1Correlation IDan engineer pastes it2Read context docsservices · log sources3Build log queriessearch_logs · trace4Cross-service timelineUTC + business time5Map errors to codefile · line · commit6Likely root causewith evidence linksA person approvesagent stops and waitsnext: hand-off to delivery agent
  1. Step 1: Paste a correlation ID

    An engineer pastes the ID from a failed request. No other prompt is needed, because the procedure is written down as a skill.

  2. Step 2: Read the map

    It reads the context docs to learn which services handle a transfer, where each one’s logs live, and what the event names mean, instead of guessing.

  3. Step 3: Build the log queries

    It writes the searches itself, scoped to those services and a time window. At work the logs live in Splunk; in the demo repo a small MCP server searches synthetic log files.

    queries the agent wrote
    trace_correlation_id("7f3c9a2e-5b1d-4c8a-9e55-2a6f0c1d7b34")
    search_logs("event:EntryRejected code:EFFECTIVE_DATE_MISMATCH since:2026-11-16")
  4. Step 4: Assemble a timeline

    Events from every service, in order, in UTC and in business time. Here: the transfer was requested at 15:42 Denver time, marked “after cut-off” anyway, and rejected by the ledger.

  5. Step 5: Map errors to code

    It finds the line that logged the error, follows it back to the decision that caused it, and checks what changed in those files since the last good release.

    CutoffPolicy.cs (since v1.4.0)
    public TransferSchedule Evaluate(DateTimeOffset requestedAt)
    {
    var local = requestedAt.UtcDateTime; // was: business time
    var businessDate = DateOnly.FromDateTime(local);
    var afterCutoff = TimeOnly.FromDateTime(local) >= options.Value.SameDayCutoff;
  6. Step 6: Propose a root cause, then stop

    A likely cause, how sure it is, what it didn’t check, and a link for every claim. Then it waits. A person approves before anything is handed to the delivery agent.

The same run, condensed:

correlation-triage · Copilot Chat
  1. triage 7f3c9a2e-5b1d-4c8a-9e55-2a6f0c1d7b34
  2. reading context: transfers-api, ledger-api
  3. search_logs correlationId="7f3c9a2e…" | sort @t
  4. timeline: 4 events across 2 services
  5. 15:42:07 TransferRequested sameDay=true
  6. 15:42:07 CutoffEvaluated afterCutoff=true
  7. 15:42:08 ledger-api 409 EFFECTIVE_DATE_MISMATCH
  8. 15:42:08 transfers-api 502 to client
  9. mapping to code: Transfers.Api/Scheduling/CutoffPolicy.cs
  10. regression: "Refactor cut-off evaluation…" (v1.4.0)
  11. likely cause: cut-off compared in UTC (22:42Z), not Denver business time
  12. needs your approval before handing to delivery
Illustrative run on a fictional demo app. Synthetic data.

The report it produces:

correlation-triage · report

Triage report

correlation id 7f3c9a2e-5b1d-4c8a-9e55-2a6f0c1d7b34

  • window 2026-11-18 22:42 UTC
  • services transfers-api, ledger-api

Timeline

UTCDenverServiceLevelEvent
22:42:07.11215:42:07transfers-apiinfoTransferRequestedsame-day transfer, acct_1007 → acct_1022
22:42:07.11815:42:07transfers-apiinfoCutoffEvaluatedafterCutoff = true · effective 2026-11-19
22:42:07.16415:42:07ledger-apiwarnEntryRejectedEFFECTIVE_DATE_MISMATCH: same-day entry dated 2026-11-19
22:42:07.17115:42:07transfers-apierrorLedgerRejectedTransferledger returned 409; client got 502

Likely root cause confidence: high

The cut-off check compares the request time in UTC, not in business time (America/Denver).

Same-day requests from 9:00 Denver time on are marked “after cut-off” and dated the next business day. The ledger still sees a same-day request and rejects it.

Not checked

Weekend requests; how clients retry a 502.

Evidence

  • logstrace_correlation_id 7f3c9a2e…4 events, 2 services
  • codesrc/Transfers.Api/Scheduling/CutoffPolicy.csEvaluate() reads requestedAt.UtcDateTime
  • changeRefactor cut-off evaluation to use DateTimeOffsetv1.4.0, deployed 2026-11-17
  • patternSame error on other same-day requests since v1.4.0none before it

Suggested next step: convert to business time before comparing, and add a test at 15:42 Denver. Hand off to the delivery agent?

Choices offered to the engineer: approve hand-off, edit, or stop.

Illustrative, from the demo app (synthetic data). Every claim points at something a person can check: a log query, a file, a commit. Then the agent stops and waits.

Review, delivery and pull requests

The same pattern applied to the team’s most time-consuming recurring work. Each agent ends with output a person can check, and none can complete an irreversible step on its own.

  • code-reviewer

    Code review

    Checks C# and Angular changes against the team’s own checklists before a person reviews. Findings come back grouped as blocking, should-fix and nit, each with the file, the line and a suggested change.

    Never approves or merges.

  • story-delivery

    Story delivery

    Takes a ticket end to end: finds the affected services, writes the code and tests, opens the pull request and prepares the QA hand-off.

    Stops for approval before each push, PR and ticket change.

  • pr-triage

    Pull request triage

    Checks the health of open pull requests (builds, review comments, merge conflicts) and drafts replies to reviewers.

    Drafts replies; a person sends them.

  • Regression tracingFrom a defect back to the change that introduced it.
  • Repo memoryDecisions and gotchas, kept with the code.
  • Local API wiringRuns a front end against local services in one step.
  • Standup digestYesterday’s commits and PRs across repos, summarised.
  • Pre-push preflightBuild, tests, lint and dependency checks before a push.
  • Flaky-test gauntletReruns a suspect test until it proves itself either way.

Guardrails and verification

Guardrails are what made the team willing to adopt the toolkit. Every generated change follows security instructions based on the OWASP Top 10. New or upgraded packages go through a dependency preflight. Generated files follow the same rules as hand-written ones. And nothing irreversible happens without a person saying yes.

What an agent may do on its own, and what needs a person’s approval first
May do aloneAsks a person first
Read code, docs, logs and ticketsPush a branch, open or merge a pull request
Run builds, tests and read-only scriptsChange a ticket’s status or assignee
Create a local branch and commit to itSend or post anything a person will read
Draft PR descriptions, review notes and repliesAdd or upgrade a package
Propose a root cause with evidenceTouch any environment beyond the local machine
The right-hand column is where the agent stops, shows exactly what it is about to do, and waits. A yes covers one action, not the rest of the session.

The table is written down as an instruction file that every agent follows. The demo version:

.github/instructions/safe-actions.instructions.md
---
applyTo: '**'
description: What an agent may do on its own, and what needs a person's approval.
---
# Safe actions
You may, without asking:
- read code, docs, logs and tickets
- run builds, tests and the read-only scripts in `scripts/`
- create a local branch and commit to it
- draft text: PR descriptions, review notes, ticket updates, replies
You must stop and ask first, every time:
- `git push`, opening or merging a pull request, deleting a branch
- changing a ticket's status or assignee
- sending or posting anything a person will read
- adding or upgrading a package
- anything that touches an environment beyond your local machine
When you stop, show exactly what you are about to do (the command, the diff, the text)
and wait for a clear yes. A yes covers that one action only.

Model output is not deterministic, so the design relies on output being easy to verify rather than always correct:

  • Evidence over assertions. Every finding or root cause links to a log query, a file and line, or a commit.
  • Scripts for anything that should be repeatable. Builds, test runs, dependency checks and digests are scripts, not prompts.
  • People review the output. Agent work goes through the same pull-request review as anyone else’s.

How the team adopted it

Adoption was voluntary. I started with the recurring work that cost the team the most time, kept outputs easy to check, and iterated on feedback from the people using it. Everything is versioned and reviewed like code, so anyone can see what an agent was told and propose a fix.

Much of my team now uses many of the skills, instructions and scripts day to day.

Lessons and next steps

  • Context is the product. Most of the quality came from the context docs and repo memory, not from clever prompts.
  • Pick the right layer. When an agent does the same steps the same way every time, that’s a script. When it needs judgement, it’s a skill. Moving work down a layer, from skill to script, made it faster and more reliable.
  • Guardrails build trust. Approval gates and evidence links are what made the team willing to use it on real work.
  • Next: test the agents like code. I want a small set of past incidents to rerun against the triage agent whenever its instructions change, so a change to its instructions can’t silently make it worse.