This morning on the drive back from school drop-off my Eleven Reader’s AI voice confidently proclaimed over my car stereo:
Our team members still review every change request for the system, but they now examine the intent of the change as opposed to reading every line of code.
I felt validated. This was near the end of a blog post from Anthropic about overhauling their source control to handle the massive scale at which they work and merge code now (30k merges on the day of September 30th alone)
I’ve been trying to make this change myself, working out how to separate myself from the guilt that comes if I don’t spend an hour staring at the “changes” tab on GitHub for every PR.
My friend Sam sent me a post from Poteto recently where they talked about ramping up their merged RP count (personally) to 2k a month while keeping quality up. Their main lever was trust.
Trust
When Poteto said “how do I trust my agents more?”, the words bounced around my skull for a bit. “trust…”, why did that feel strange? Trust.
The way I was socialized, the concept of trust felt like simply a decision: “do I trust this person? or not?”. But the way Poteto said it, trust was being used to describe an action, an infrastructure. Those have pretty different implications.
When trust is a choice, I open myself up to risk. Choosing to trust a person means allowing their actions and choices to shape my life. Choosing to trust a system (government, company, religion) means giving up personal power to the system’s structure in exchange for something (convenience, protection, wealth).
When trust is an action, I build. I set up my own systems. Then I experiment to see if I can rely on the results. I take small risks at first, so that failures aren’t catastrophic. Then I tweak, improve, and test again.
Trusting the Agent
In agentic work, this same duality exists. And examples of each side might look like this.
Trusting the agent as a choice:
- Assign an issue to the model
- Ask “check your work before making a PR please”
- Clicking the merge button
Trusting the agent as an action:
- Acknowledging that new models are very powerful, but still need direction
- Setting up repo-specific review skills that provide context into existing features, product complexity, historical code nuances, regressions, and past gotchas
- Designing review skills tailored to failure vectors that are specifically risky for your product offering.
- Running those customized review passes in agents with fresh context (and perhaps diverse models) in a loop until there are no more major issues to address
- Asking the agent to attach screenshots and artifacts to the PR
- Asking the agent to outline the test cases, examine mock quality
- Asking the agent to explain its architectural choices
- In a live review session, asking the agent to clarify parts of its summary that are hand-wavy or unexpected
- Maybe checking the code, maybe not
- Clicking merge
At Day One, we’ve been working for months on building up the skills in our repos so that we can trust the agents as a choice. We’ve been documenting client features, how our encryption architecture works, specific nuances of the flow that have lead to bugs in the past, and creating skills that explain the nuances of how to migrate our legacy ReScript code over to Typescript. It’s been a lot of work, and we need to do more, but the code our agents generate and the review comments we get have been pretty solid.
Story time
This morning I woke to a message from Nikos saying Cubic had found some issues on the diff we generate when merging staging code into production:
This allows an admin to set a password of their choice on any real non-test account…
“Oh no”, I’d specifically dug into that case with GPT Luna and Sol 6.1 in my review passes the day before. As I drove my kid to school I wondered “have I been trusting too much that the models can accurately follow the code they are working on? If we specifically explored this case and the model didn’t catch it, I have to rethink some things”
So, when back at my desk I dug into it:
Me: Didn’t we discuss exactly this scenario? How did we miss it?
GPT: Yes—we discussed this exact case on October 14…You said the primary scenario was a non-test account with no password, and I confirmed the feature would set one there. You also asked whether a developer could set a password on an arbitrary account. I answered that the provider login had to be verified and attached to the source account, and that the destination had to be a test account.
…followed by more digging in and verification of code
What a relief! The code was correct. The Cubic review hadn’t understood the specific context. It was okay for an admin to set a password on another account when that account was already proven to belong to a credential they had authorized. The tests hadn’t failed. The model hadn’t lied.
This bumped my trust up a level. I now feel more confident than I did yesterday that these latest models’ ability to read code is very accurate when given the right context.
Updating the review process
I’m not sure how long I’ll keep clicking the “changes” tab to see what the model has written before merging. I still find things I want it to change, but I’m noticing that those changes now are usually over-engineering that I want it to trim back, not issues with the code or architecture. Maybe soon I won’t feel the need to look into the code at all.
But perhaps rather than thinking about less code-looking, it’s better to think about more question-asking.
The review process for me used to involve intense sessions of tracing code paths, looking for sneaky variable-passing mistakes, or catching logical errors that a human would make. Now it’s become more of an interview process. I talk to my agent about a PR. I grill it with questions about the implications of an architectural change. I ask it to search through the history of our Slack discussions, or our client codebases to understand what the unseen implications of a change might be.
My reviews have become less passive (reading) and more active (asking). And perhaps that parallels the duality of trust; The more I act on trusting my agents, the more active I can be with them in reviews. The less I trust them, the more passive I have to be.