How Jarsy rebuilt financial risk control with AI agents

Since August, Jarsy has been building its risk-control system in an entirely new way. Investigating, cross-checking, drafting rules, writing code, verifying by replay: agents do all of it. Humans do one thing: approve.

We work this way because we have come to believe that the bottleneck in risk control was never detection, nor how fast rules ship. It is the handoff. The person who spots a fraud pattern and the person who can turn it into a defense are rarely the same person.

For the past twenty years, the whole industry has been optimizing every link in this chain of handoffs: rule engines let analysts ship rules themselves, feature stores spare engineers from rewriting pipelines every time, and automatic retraining lets models update every day. But the handoff itself never went away. A new pattern still has to travel from analyst to engineer and then to data scientist, and every handoff takes time and can lose information.

What happens once the handoff disappears? Seven weeks in:

In late August, we found a ring that was using other people’s ID documents to register accounts in bulk and fraudulently claim referral rewards. The alert exposed more than a few gaps in our rules: it exposed long-standing problems in the whole KYC flow. Throughout this period, every human KYC decision (approval, rejection, revocation) was signed by the same person.

1Risk organizations are built around handoffs

Afterpay is a company that does financial risk control exceptionally well, a benchmark in buy now, pay later. Every order goes through real-time fraud and repayment-capacity checks at the moment of checkout, and net transaction losses have long been held below 1% of sales^{1Afterpay, FY21 Annual Report, ASX, August 25, 2021.}. From 2019 to 2023 I worked at Afterpay for nearly five years. The checkout flow and installment lending systems I worked on both involved deep collaboration with the risk team, and I saw up close how that system runs.

Risk at Afterpay was an organization with a very fine division of labor:

• analysts in fraud, credit and merchant risk, who found problems and set strategy;
• feature scientists, who turned the analysts’ judgment into features that could be computed;
• engineers on the feature platform, the risk backend and the machine learning platform, who turned features and rules into production systems;
• data scientists, who trained and updated the models.

A new pattern usually traveled like this: analysts reviewed cases and looked for trends every day, and first wrote a stopgap rule to hold the line; then, together with the feature science team, they proposed new features and handed them to the engineering team to build and ship; finally, the modeling team retrained. Dozens of people in Melbourne, Sydney, San Francisco and Shanghai together supported that one real-time decision at checkout.

It is a mature and effective system, and its results speak for themselves. But its shape is set by handoffs: every additional specialty adds another handoff, and every handoff has to be scheduled and can lose a little information.

Afterpay is not alone in this. Read through a decade of engineering blogs from company after company, and almost the same shape appears:

• People do the discovering. Stripe’s engineers wrote detailed investigation reports on every past attack, met weekly to discuss fraud trends, the dark web included, and came up with new features from them. Uber’s early risk analysts would comb through 30 days of data, break a behavior down into feature distributions and try hundreds of combinations^{2Uber, “Engineering Uber Systems to Combat Fraud”, June 30, 2016.}. One ring Coinbase caught in 2017 was found because an analyst noticed its members always used the same screen resolution^{3“Risk at Coinbase”, talk slides, QCon New York 2017.}.
• Other people do the coding. Grab’s analysts used to write rules in English and hand them to engineers to translate into code, and the translation introduced errors. At Lyft, aligning feature definitions between scientists and engineers took months of back and forth, and the team kept convening “feature war rooms” for it^{4Lyft, “From shallow to deep learning in fraud”, July 9, 2018.}. Airbnb’s data showed that people working on machine learning spent 60% of their time preparing data^{5Airbnb, Zipline, talk at Strata NY 2018.}.
• Scale comes from adding people. In 2023, Revolut announced that its financial crime team had grown past 2,500 people, more than a third of its total headcount^{6Revolut, “Revolut doubles financial crime headcount to combat rising industry fraud”, 2023.}.

Every handoff carries an implicit assumption: the people who understand the data can’t write production code, and the people who write the code don’t have time to look at the data.

2What the industry spent twenty years optimizing

Every link in the chain has been carefully optimized:

• Shipping rules. Uber’s Mastermind let analysts ship rules themselves in minutes, where before they had to wait for an engineer to spend an hour on it. Grab’s Griffin cut a rule change from a week to a minute^{7Uber, “Mastermind”, March 8, 2017; Grab, “Griffin”, October 28, 2019; Airbnb, “Chronon … is now open source”, April 8, 2024; Stripe, “Shepherd: How Stripe adapted Chronon”, April 15, 2024; Uber, “Project RADAR”, February 1, 2022.}.
• Building features. Airbnb’s Zipline and its successor, Chronon, cut the time to prepare a new set of features from months to about a day, and Stripe later adopted it too^{7Uber, “Mastermind”, March 8, 2017; Grab, “Griffin”, October 28, 2019; Airbnb, “Chronon … is now open source”, April 8, 2024; Stripe, “Shepherd: How Stripe adapted Chronon”, April 15, 2024; Uber, “Project RADAR”, February 1, 2022.}.
• Automated discovery. Uber’s RADAR, from 2022, mines candidate rules automatically from anomalies in loss curves and then hands them to analysts for approval^{7Uber, “Mastermind”, March 8, 2017; Grab, “Griffin”, October 28, 2019; Airbnb, “Chronon … is now open source”, April 8, 2024; Stripe, “Shepherd: How Stripe adapted Chronon”, April 15, 2024; Uber, “Project RADAR”, February 1, 2022.}.

These are real advances. But what they optimize is the cost of the handoff; they don’t remove it. A new pattern still has to be seen by a person first, and what that person can see is limited by which features someone else built for them in advance.

3A seven-week experiment

When we designed this AI-centered risk-control system, we set one principle:

In risk control, everything that can go to agents goes to agents. Humans only approve.

Seven weeks in, this is how the principle works in practice:

• Since August 26, all 29 code changes submitted for KYC and risk control have been written by agents, then reviewed and merged by a human.
• For every account revocation, an agent drafts the evidence and the list, and the revocation is carried out only after a human confirms it batch by batch.
• Investigation is led by agents: they verify, cross-check and write reports on a read-only view of production data. Humans decide what a case is and what to do about it.

The system has three layers:

1. The rule layer. A set of deterministic rules that handle hard conflicts that need no judgment. This layer has exactly one power: to stop an application. It can never approve anyone.
2. The review agent. Every application gets a full review by a model. It reads the ID documents and verifies them, and through a closed set of read-only tools it can also cross-check them against other accounts on the platform. It gives a verdict: pass, stop, or escalate to a human. But code is the final backstop: if a required check was skipped, or the rule layer has already stopped the application, the agent’s “pass” does not count.
3. The cohort audit agent. Each application can look fine on its own; the problem lies in a group of people, in the relationships between them. From time to time, Jarsy runs referral reward campaigns: an existing user invites a friend to sign up, and once the friend passes identity verification, the inviter earns a reward. Campaigns like this naturally attract people who farm rewards in bulk. Before each campaign settles and pays out, the cohort audit looks at all of the campaign’s participants together: who is linked to whom, how those links formed, and whether the group as a whole looks like real users. It then recommends revoking, holding or keeping, with the evidence attached. Relational reasoning is exactly where traditional models are weakest: research Revolut published this year showed that a transaction foundation model that models each user separately performed 47% worse than the company’s existing production system on relational problems such as anti-money laundering^{8Revolut Research and NVIDIA, PRAGMA, arXiv 2604.08649, April 2026 (v2: August 2026).}. This part used to depend on senior analysts going through cases one by one; now it goes to an agent.

Where are the humans? At the exit of every layer: a human decides stopped applications, approves revocation lists batch by batch, and merges every change to rules or code.

Boundaries on data and decisions. What an agent can see and do is fixed in code in advance: it can read data only through that set of read-only tools, and it cannot change any record. It calls Anthropic’s commercial API, and under the commercial terms this data is not used to train models. Decisions have boundaries too: machines can only let an application through or stop it, rejecting an application is always done by a human, and every decision that goes against an applicant carries a person’s signature.

4The seven-week timeline

What matters most in this table is the “Who did it” column.

In a traditional chain, the August 26 document farm would probably have gone like this: an analyst spots the anomaly, writes an investigation report and files a ticket; an engineer schedules the work and turns a few of the patterns into rules; a data scientist evaluates whether to add features and whether to retrain. Between every two steps there is a queue.

No company has published how long this chain takes from end to end. What has been published are numbers for a single link after it was shortened: Grab cut a rule change from a week to a minute, and Airbnb cut preparing a new set of features from months to about a day. How long the whole chain takes, added up, nobody has ever reported.

At Jarsy, going from discovery to a defense in production took three days, with no tickets and no scheduling in between. The agent session that investigated the ring and the agent session that wrote the rule code worked from the same set of facts.

An even faster loop needs no code change at all: once an account is revoked, later applications linked to it are stopped automatically by the rule layer and handed to a human. The act of revoking itself becomes evidence for the next judgment.

One alert triggered a rebuild. The document farm exposed more than a few gaps in our rules; it exposed long-standing problems in the whole KYC flow:

• users uploaded images from their photo library, not photos taken on the spot;
• the model was called once, with no tools, and could not look up other accounts;
• neither machine nor human decisions left any record.

In a traditional organization, this would be a quarter-long project that needs a formal proposal, scheduling and coordination across several teams. It took us nine days: in the nine days after the alert, the KYC backend merged 24 related PRs, with requirements, design, implementation, verification in staging and independent review all done by agents. What humans did was approve each stage and give the go-ahead before every release to production.

Lessons from risk directly rewrote the design of the next campaign. Each campaign’s retrospective directly decides how the next one runs: whom to invite, which accounts to exclude, which behavior doesn’t count toward rewards, and how large the list can be. The cap on the list’s size is set by what human review can absorb, not by the budget. In a traditional organization, these lessons have to travel from the risk team to the growth team; here, the retrospective and the next campaign’s design come out of the same loop.

Behind this is another engineering process, also designed around “eliminating handoffs.” It is the subject of our next article.

5What is left for humans

Once investigation and coding go to agents, the human job shrinks to three things. For each one, we can point to a concrete example from these seven weeks.

1. Deciding what counts as fraud. In the second referral reward campaign, in late September, the audit agent found an inviter who had brought in a batch of new users within five days. They were all real people using real ID documents, and taken one at a time, none of them had anything wrong with them. Taken together, though, these accounts clearly did not behave like real users. The agent then found posts the inviter had published on the open web: sign up and get a reward, and half of it is yours. Is that fraud? Each person is a real individual, but their KYC records were, in effect, bought. That is a policy judgment, not a pattern-recognition problem. On September 28, the team decided: paying people to sign up is the same as buying approved KYC records. The agent then carried out the revocations.

There are examples in the other direction too. On October 7, the team set a principle: a single anomalous signal is not evidence. One video from an influencer can bring in a wave of genuine sign-ups in a single day; what looks anomalous is not necessarily fraud.

2. Approving. For every batch of revocations, the agent first presents the list, the evidence and how much less would be paid out in rewards after revoking, and a human confirms batch by batch. Confirming one batch does not mean the next can run automatically.

3. Being accountable. Over these seven weeks, every human KYC decision was signed by the same person. Every decision is written into the audit trail: who made it, on what basis, and which machine verdict it overturned.

Written down as rules, the decision rights come to this:

• The rule layer can only stop an application, never approve it.
• The agent gives the verdict and code enforces it; if a required check was skipped, the verdict is downgraded automatically.
• Revocations, rule changes and code changes each carry a person’s signature.

One direct consequence: the slowest link now is the human. From submission to a machine verdict, an application takes a median of 1.4 minutes; a stopped application waits a median of 21 hours for a human decision. The bottleneck has moved from “seeing” and “coding” to “approving.”

In their harness engineering article, OpenAI argues that the fewer merge gates, the better^{9OpenAI, “Harness engineering”.}. In finance, our position is the opposite: agents remove the handoffs between gates; humans hold the gates themselves.

6Evaluation: what lets one person approve with confidence

In a handoff chain, quality rests on the people the work passes through, layer after layer: the analyst has looked at it, the engineer has looked at it, the data scientist has looked at it, and then it ships. Without the chain, those eyes are gone too. The only thing that can give one person the confidence to approve what an agent hands over is evaluation. In a financial business, every change to an algorithm is itself a change that has to be audited.

Frontier AI labs have published a great deal of what they have learned about evaluating agents. Anthropic’s “Demystifying evals for AI agents” from January this year^{10Anthropic, “Demystifying evals for AI agents”, January 9, 2026.}, OpenAI’s SWE-bench Verified^{11OpenAI, SWE-bench Verified, August 2024.}, HealthBench^{12OpenAI, HealthBench, May 2025.} and GDPval^{13OpenAI, GDPval, September 2025.}, plus the fraud agent evaluation Coinbase published in October 2026^{14Coinbase, “Part one: how Coinbase built, evaluated and post-trained a fraud agent”; Coinbase, “Owning Intelligence, Part 2”, October 7, 2026.}, boil down to a few principles. We checked our own practice against each one:

A concrete example: the September model upgrade

Before moving the review agent from Opus 5 to Opus 5.5, we ran a replay following the principles above.

• The eval set. 48 production applications already decided by a human, split into three strata by “human verdict” and “machine verdict at the time.” Four of the human-rejected applications were stopped by the rule layer, so they don’t test the model and are counted separately.
• Change only one variable. Prompts, tools, the rule layer and the code backstop all stayed the same. Claude has a reasoning “effort” parameter that sets how much the model thinks before it answers: the higher the level, the more careful, and also the slower and more expensive. Opus 5 defaults to high; Opus 5.5 defaults to medium, one level lower. To change only the model, we explicitly pinned Opus 5.5 at high, the same as the old model, and tested its default, medium, separately as a group of its own.
• Grading. Read-only replay. Document reading is graded by check digits. The three rejection cases with disagreements were rerun three more times in each group.

How to read these three tables. A total of 39 versus 40 tells us nothing; what we look at is where the two models disagree on the same cases. Seven cases got different verdicts: the old model was right on 3, the new model on 4, with no statistical difference. The fraud-let-through cell dropped from 8/9 to 7/9, which looks like a regression; rerunning showed it involved boundary cases no model handles reliably, let through in two runs out of every three. Cases like these are exactly what a regression set most needs: the requirement is not to get them right once, but to get them right every time. The conclusion: judgment quality held steady, there were three fewer overly cautious stops, cost fell 37% and latency halved. We shipped Opus 5.5 · high.

Finally, evaluation is only one layer. Production itself keeps producing evaluation signals: the share of applications the machine let through that a human later revoked, the share of machine verdicts a human overturned, and the reason for every override are all written into the audit trail. Together with replay evaluation, they take the place of the many pairs of eyes along the old handoff chain.

7Where this doesn’t apply

• Volume. Our volume is not in the same league as a large fintech’s. A median of 21 hours for a human decision is acceptable at our volume; one or two orders of magnitude more, and it wouldn’t be. At that point, either machines decide a wider range of cases directly, or human approval needs tiers too.
• Latency. KYC is asynchronous: we can spend tens of seconds and ten-odd cents looking carefully at one application. Payment authorization has a budget of under 100 milliseconds, so this approach can’t be carried over directly. That is why Coinbase’s transaction review agent looks at only a subset of transactions.
• Labels. The truth about fraud still arrives slowly. An agent can propose hypotheses quickly, but verifying them still means waiting for outcomes.
• One person signing is itself a risk. For these seven weeks, every human decision rested on one person. That shows this organizational shape can work, and also that it currently has a single point of failure.
• Regulation. Where risk-control models fit in banking’s governance frameworks is still unsettled; new guidance from US regulators in April 2026 explicitly left generative and agentic AI for the next round of consultation.

8Next steps

Self-evolution. Today, improving this system still takes a person to start it: a human overturns a verdict, or an audit finds a new ring, and then someone opens an agent session to change the instructions or the rules. Next, we want evolution to happen on its own, with every step still waiting for human approval. We plan to get there in three steps:

• First, evolve the instructions and the rules. Each week, an agent reads that week’s disagreements between machine and human, uses reflective prompt evolution to search historical cases for better review instructions, and opens them, together with any rule changes, as a change awaiting human approval (a pull request), with a replay comparison attached: which cases got better and which got worse. A representative example of this kind of method is GEPA^{16GEPA, arXiv 2507.19457 (ICLR 2026 oral).}, which already beats reinforcement learning at the scale of one or two hundred samples, a good match for the number of labels we have today.
• Then widen the search. Once more labels have accumulated, evaluate several candidate versions at once, run them in shadow mode on real traffic first, and switch only once their performance is confirmed.
• Finally, train our own model. Once a narrow judgment has accumulated enough matured real-world outcomes, we will use reinforcement learning to train a small specialized model as a supplementary signal. Coinbase made this path public in October 2026: a 9-billion-parameter open-weights model trained on real fraud outcomes beat Opus on their benchmark^{14Coinbase, “Part one: how Coinbase built, evaluated and post-trained a fraud agent”; Coinbase, “Owning Intelligence, Part 2”, October 7, 2026.}. What this path lacks is not compute but a large volume of trustworthy labels, which is exactly what we don’t have yet.

Whichever step we reach, a few lines will not move: the grading criteria are fixed, and the optimizer can touch neither the grading method nor the test set held back for the final check; replay sees only the data that existed when the application was made; and every evolution is a change that can be read, audited and rolled back, and it ships only after a human approves it.

From identity to the whole platform. Today this system covers only the identity layer. Next, we will extend it to every risk surface on the platform, and the unit of risk control will change from “an application” to “a person”: one person’s accounts, devices, bank accounts and wallets grouped under a single entity, with identity, relationships, funds and behavior judged together in one case.

• Money in and out, off-chain and on-chain. Screening deposit sources, scoring withdrawal destinations, verifying wallet ownership, simulating transactions before signing: these capabilities are already mature on the market, and we will buy them rather than build them. Our job is to have agents weigh these external scores together with the identity and relationship data we hold ourselves, in a single case.
• Economic behavior on the platform. External vendors can’t help with this layer: we designed our pricing and settlement mechanisms ourselves, so only we can define the ways to game them. In September we already used agents to run a full audit of the platform’s pricing and settlement, checking one by one several kinds of technique that could exploit loopholes in the rules. The conclusion: no systematic gaming was found, but there were a few places where the rules themselves favored certain kinds of behavior, which later led to changes in the settlement mechanism. The next step is to turn this audit from a one-off into something continuous.

The wider we go, the more firmly some things stay out of agents’ hands: sanctions-list blocking is a deterministic rule; signing, disbursements, and price and settlement calculations must be reproducible code; an agent can only add a review to a transaction, never relax one; and the final action against a user is always decided by a human. Nor will we claim to detect all manipulation: even the largest on-chain analytics company in the industry says only that what it finds are “patterns,” not “intent.” Until withdrawals can actually be blocked, we will not say we have achieved end-to-end protection from identity to funds.

Our bets

• By mid-2027, risk control at Jarsy will no longer have a step where “a pattern is handed to someone else to implement.” From discovery, coding and verification to release, every change to a defense will be a change drafted by an agent, accompanied by replay results, and approved by a human.
• By the end of 2027, the same agents will cover every risk surface on the platform: from identity verification, to money moving in and out on-chain and off-chain, to trading behavior on the platform.
• Further out, by 2029, we believe the shape of most financial risk teams will have changed: headcount will no longer grow linearly with business volume, but will depend on how many judgments the company needs to make. The core capability of a risk team will shift from “discovering and coding” to “defining, approving and being accountable.”