Poker vs Jev vs Clef
Part 3 of 4. A series of posts relating to our use of Cloudflare: Thank you, John Graham-Cumming; An infinitely scalable game on Cloudflare; Poker vs Jev vs Clef; What System One models are actually good for (coming).
Jev is a few weeks old, but the concepts behind System One Models are a bit older. It has captured the imagination. It can even play Doom. When I saw the use cases and considered how much hand history data might be in the training sets, my first thought was how well it would play poker. In keeping with my previous argument about how fast you can prototype on the Cloudflare Developer Platform, I ran a test on a sample of our archived hand data. Since then, Cloudflare has released its own System One model, Clef, so we ran the same test against that too.
If you have not met it yet, Jev is not a chat model. You hand it a state and a set
of typed questions - a choice over named options, a score on an ordered scale, a noul which is the probability
that a statement is true. It answers quickly, often within a few hundred milliseconds, returns the answer and
nothing else, and costs a fraction of a cent per call. TypeSafe also describe its probabilities as calibrated. A 70% from Jev should mean 70%. That claim is one of the things this test could check.
The obvious thing to do with a calibrated probability model and a hidden-information game like unpoker rivals is to point one at the other. Hand it a
table state and ask it the questions a player asks. Which is the best action? Am I likely to be behind in this
hand? Answering the second one well is a large part of reading a table.
What we ran
We have archived hand histories for every game ever played. We took a sample of around five thousand deliberate game action decisions made by our players, and
computed the actions available at each one using the same code the game uses. For each decision we asked each
model three questions in one request: which of the legal actions is best, is at least one opponent currently
holding a better hand, and how strong is the player's hand on a five-level scale.
A stored game state contains the whole board and everybody's hole cards, so we first wrote a function that
passed on only what the player could see at the time. That meant their own cards, the visible board, positions, stacks, the
pot, the betting so far, the elimination context, and the hole cards opponents had been shown to hold in earlier
hands. That last one is particular to our game - it reveals every dealt hand at the end of each
hand, so a human player has that information too.
There is one significant caveat to all of this, which comes from the game itself. Targetable power-ups, scheduled eliminations and the end-of-hand reveal mean there is no published optimum strategy for it, and standard Hold'em theory, which presumably underpins most of the poker data the models were trained on, only partly applies. Our game is designed to be fun, a quick blast of
poker-like play without a lifetime of plugging leaks. We kept the questions as close to standard poker as we
could. The humans were playing the full game, so where they differ from the models, some of that is the game
rather than a mistake.
As I argued in the previous post, the platform makes a test like this easy to set up. Workers and Durable Objects can already call
Workers AI through the AI binding, with AI Gateway in front of it for caching, logging and rate limiting. Using a brand new model meant adding its name. There were no keys to provision and no new vendor to onboard. The test made about
31,000 calls to Jev for a total charge of $2.13. We drove it from a script on a laptop through a small local
Worker using the same AI binding. From a deployed Worker or Durable Object it would be the same call.
Jev and Clef
Clef takes the same state and questions as Jev, but the two are built differently. Clef is Qwen3.8-27B,
post-trained by Cloudflare for decisions, with the weights published under an Apache 2.0 licence. TypeSafe
describe Jev as a new architecture trained with a method they call Reinforcement Learning for Calibrated
Decisions, and do not say what it is built on. One practical difference showed up straight away. Ask Clef the same question twice and you get exactly the same answer, but Jev's answers vary a little from one request to the next. So we asked Clef about each decision once, and Jev several times.
| Jev | Clef | |
|---|---|---|
| Price per million input tokens | $0.042 | $0.24 |
| Input tokens per decision (our requests) | 2,011 | 1,366 |
| Cost per 1,000 decisions, list price | $0.08 | $0.33 |
| Median / p95 latency, our setup, same day | 356ms / 461ms | 619ms / 1,129ms |
| Median latency, Cloudflare's figures | 524ms | 209ms |
| Same action every time on a repeated question | 91.2% of decisions | always |
Clef counts the same request as about two thirds as many tokens but charges nearly six times as much per token,
so a decision costs about four times as much. At list prices a pass over the whole sample comes to about $0.47 on Jev and $1.84 on Clef. Cloudflare's own figures have Clef more than twice as fast, but in our test, on the same day
with the same requests, Jev was faster. We do not know why. Since we ran the test, Cloudflare has made Clef faster, about 1.7 to 2 times at the request sizes we used. We have not rerun our test against it. Both usually answer in under a second and both cost a
small fraction of a cent per decision.
Jev's choices look like a slightly tighter version of the humans'
We compared each model with the humans, and with each other, in two ways. We looked at how often each one folded, checked, called and raised, and at whether the calls it made were any good.
To judge the calls, we took every decision, on any street, where the humans or a
model called. Because every hand is archived, we know the cards held by the opponents still in the hand. From
that point we worked out the player's chance of winning across every way the remaining board cards could fall (a large random sample of them before the flop),
multiplied it by the pot including the call, and took away the price of the call. That gives each call a value in big blinds, and a fold is worth zero. Knowing the opponents' cards is hindsight, because a call that was right against
the hands an opponent might hold can still be wrong against the one they actually held. Averaged over hundreds of calls, much of that evens out, though not in the small groups. It is still a rough guide, as it ignores any betting after the call and any power-ups.
The charts below show how much more or less often each model folded, checked, called and raised than the humans
(and, in the last chart, than Jev), in percentage points, overall and street by street.
Fold Check Call Raise
All
+2
+5
-3
-4
Preflop
+2
+1
-4
+2
Flop
+1
+8
+1
-9
Turn
+1
+13
-1
-13
River
+7
+15
-5
-17
-40 -20 0 +20 +40
Overall that is (sort of) a recognisable poker player. Compared to the humans, Jev recommended folding a bit
more, calling a bit less and raising a bit less. It picked the same action as the humans in 58.3% of the
decisions. Picking at random would have matched about 35% of the time, so that is real agreement, but far from
identical play. By our rough measure its calls were also better value. Where the humans called and Jev would have folded (563
decisions), those calls lost 1.15 big blinds each on average. Where Jev called and the humans folded (438
decisions), Jev's calls lost 0.54 big blinds each. Both made calls the other avoided, and by this measure the humans' cost about twice as much, though they were different spots.
The streets are where it gets interesting. Before the flop the two are close, and Jev raises slightly more than
the humans. After that it gets steadily more passive. It raises 19.4% of the time on the flop against the
humans' 28.6%, 12.2% on the turn against 25.5%, and 7.1% on the river against 24.1%, checking instead. In standard poker the river is where pots are biggest and value bets get paid, so I would expect more aggression there, not less.
Fold Check Call Raise
All
+18
+4
-24
+2
Preflop
+19
-1
-32
+14
Flop
+22
+11
-13
-20
Turn
+5
+12
-4
-13
River
+25
+7
-20
-12
-40 -20 0 +20 +40
Clef plays a different game. It folds about half the time and almost never calls, in about 3.5% of its decisions. Folding or raising before the flop is common advice. Against Jev's calls, by our rough measure, folding this much gave up value, as shown below.
It matched the humans' action in 50.6% of the decisions, against Jev's 58.3%. Where the humans called and Clef
folded (962 decisions), those calls lost 0.11 big blinds each, close to break-even. Clef's own few calls were
good value in hindsight, but it called where the humans folded only 28 times, too few to compare like for like.
Before the flop it nearly always either folds or raises. On the flop and river it folds far more than the
humans, and on the river it never called once. It agreed with the humans least on the flop (47.3%) and most on
the turn (64.5%).
Fold Check Call Raise
All
+17
-1
-21
+6
Preflop
+18
-2
-28
+12
Flop
+21
+3
-14
-10
Turn
+4
-1
-4
+1
River
+18
-8
-15
+5
-40 -20 0 +20 +40
Side by side, the two models picked the same action in 65.1% of the decisions. Most of the gap is folding and calling, with Clef folding 17 points more and calling 21 points less. Where Jev called and Clef folded (786 decisions),
Jev's calls were worth 0.45 big blinds each in hindsight, so Clef's extra folds gave up value. Clef called where
Jev folded only 17 times.
They are furthest apart before the flop and on the flop, and closest on the turn, where they agreed on 82.7% of
decisions and no share differs by more than about 4 points.
Can it tell when it is behind?
For every decision, each model gave a probability that at least one opponent still in the hand held a better
hand. It had no more information than the player, so no single answer can be judged. What can be checked is
calibration across thousands of them. When a model says 60%, is the player behind about 60% of the time? The
table groups the post-flop decisions by the probability given and shows how often the player really was behind
in each group. A well calibrated model would have each actual rate inside the range on the left.
Post-flop decisions, grouped by the model's estimate that the player was behind
| Actually behind | ||
|---|---|---|
| Estimate | Jev | Clef |
| 20 - 30% | 22.0% | - |
| 30 - 40% | 44.5% | 41.7% |
| 40 - 50% | 46.0% | 55.0% |
| 50 - 60% | 60.3% | 59.9% |
| 60 - 70% | 70.6% | 74.6% |
| 70 - 80% | 77.1% | - |
Jev's number and the truth rise together, from 22% behind in its lowest group to 77% in its highest. It tends to
underestimate how often the player is behind, by a few points in most groups. Clef's numbers barely spread, with about 85% of its answers between 40% and 60%. Only 3 fell in the lowest group and 11 in the highest, too
few for a rate, so those cells show a dash. Across all these decisions the player was behind 59% of the time, so
a simple benchmark is to answer 59% every time. Jev's estimates were slightly more accurate than that benchmark, and Clef's were not.
But was it reading the table, or just its own cards? The strength of the player's hand against a random hand,
which ignores the opponents completely, predicts who is behind better than either model did, on every street
after the flop. The third question, the hand-strength score, fits that picture. Jev's score correlates moderately with the player's equity against a random hand (0.61) and weakly with equity against the hands the opponents actually held (0.32), and Clef's shows the same pattern. That is what a score of the player's own cards should look like, so it does not prove the models ignored the betting, but nothing here shows them using it either.
Pre-flop decisions are left out of the table. Before the flop, 'does an opponent hold a better hand?' is either
obvious or ambiguous. AA is clearly ahead of KK, but is KQ suited ahead of A5 offsuit? Run the board out and A5
offsuit wins about 53% to 46%, close to a coin flip. Before the flop the question is really about equity against
a range of likely hands, and our scoring had to pick one meaning that the models may not have shared.
Does the order of the options matter?
Some commentary on X suggested that Jev's choice answers change
when the same options are listed in a different order. If an answer depends on the order you happen to list
things in, it is hard to trust any single answer, so we tested both models.
We took 300 decisions where the player had three legal options, 75 from each street, and sent each one to both
models 16 times:
- Four were identical to the original request, to measure the model's own noise as a baseline.
- The other twelve listed the action options in a different order, each of the six possible orders twice. For example, fold, call, raise became raise, fold, call.
- In half of those twelve, the hand-strength scale was reversed and listed strongest first.
- In a different half, the 'is the player behind?' statement was turned round, so the model was asked whether the player held the best hand among the players still in the pot.
Before comparing, every answer was mapped back to the original order. A hand-strength score of 3 on the reversed
scale counts as 1, and a 0.3 chance that the player holds the best hand counts as a 0.7 chance that the player
is behind.
Jev's answers do move with the order. Jev returns a probability for every option, so we measured how far those probabilities shifted from one request to the next. Asking the identical question again shifted them by 2.9 points on average. Reordering the options shifted them by 5.4 points, and the extra shift appeared at every confidence level. Jev also gave slightly more weight to whichever
option was listed in the middle, 0.347 on average against 0.330 for the first and 0.324 for the last, a small difference.
Most of the time the shift is too small to change the top answer. Reordering changed Jev's top pick 7.6% of the
time, against 4.2% for identical requests, and almost all of those changes were near-ties. In the third of
decisions where Jev was least sure, the pick changed 22.5% of the time across orders, against 12.5% for identical requests. In the most confident third it never changed.
Its scores are not strictly order-invariant either, but the effect is small. Reversing the five hand-strength
levels moved the score by 0.15 on average on its 0 to 4 scale, against 0.07 between identical requests, and it
ranked hands just as well either way.
Clef gives no extra weight to any position in the list, but every other change we made, even the order of the levels in a different question, moved its probabilities a little, by between 1.7 and 2.7 points. That changed its top pick in 3 to 5% of cases, mostly where its probabilities were close to flat.
The biggest effect was the wording. Turning the 'is the player behind?' statement round moved Jev's probability
by about 7 points on average and Clef's by about 19. The calibration figures above are for one wording.
So the commentary on X holds up. Jev's choices, and strictly its scores, depend on the order, but in
our data that only changed the answer where the model was already close to a tie.
I think we asked the wrong question
I don't think the failure here is only the model's. 'Am I behind?' looks like a simple classification, but
answering it takes two steps. First work out which hands the opponent is likely to hold given how they have bet, then
work out how many of those beat you. The state we sent had everything a human player has but none of that work
done, and we asked a model that makes one fast judgement per question to effectively do both steps in one pass. We cannot see inside
it, but its answers look closer to a judgement of its own cards than a read of the table.
My guess is that it would do better with the range written into the state, for example 'the opponent raised from early position and has bet twice, so these are the hands they are likely to hold'. The question then becomes a comparison
of things it has been given, which is what this kind of model seems suited to. The same goes for a read, a
judgement about how a particular opponent plays. If you want Jev to use one, I expect it has to be in the state and the question has to ask for it. We have not tested that.
In effect, we asked Jev to act as a rough solver, which it is not. But it might do something a solver can't.
Solvers are fast enough to consult during a hand, but the sites ban real-time assistance. Many players instead learn from decision flowcharts that compress solver output into something a person can hold in their
head. Jonathan Little's course material (which I think is very good) is full of them. Classify the hand, read the board texture, check position and stack depth, follow a set of pre-defined decision steps.
Most of the nodes on those charts are arithmetic. Do I have position? Is this board paired? Am I being offered a
price my equity justifies? Those are computable, and I don't think a probabilistic model adds much to them. The rest are the real heart
of poker. Is this a player who folds too often to a re-raise? Is their range capped here? Those are the nodes
where the chart quietly hands the decision back to you and says 'use your judgement'. Judgement is built out of
tens of thousands of hands a casual player has not played, and I have not seen a tool that fills that gap.
That is where a System One Model could sit, alongside the chart, telling you when to deviate from the line it
recommends, in which direction, and how confident it is. It would often be wrong, and how far you could trust it, confidence included, would depend on the quality of the data it was trained on. A solver knows the equilibrium strategy, which is built to be hard to exploit, but it has no memory of the person across from you. Against someone who, say, folds too often, a different line would win more, and on its own a solver won't tell you when to take it. That would need a model trained for it, a state
that carries the right things, and the right questions.
Who won between Jev, Clef and Poker?
Poker. Jev makes reasonable poker decisions, but its read on whether it is behind was less accurate than the strength of its own cards alone, and Clef's was worse still. That is the result for these two models today. The longer-term implications for online poker are more interesting.
People have long sought an edge through third-party tools. Odds calculators, trackers, HUDs, preflop charts
taped next to the monitor, solvers, training sites, a coach on a video call. Most were controversial when they arrived and the sites fought back, with mixed effect. Sites look for solver users partly by checking how closely
their decisions match the solver. What happens when a System One Model, trained on a high quality corpus of
hands, runs on a phone beside the monitor and tells you when to diverge?
I'm not even sure tools like that are unfair. You can already note that an opponent plays looser than they
should. Is it different if a model reads your hand histories and gives you the probability that they do? Either
way you end up with a read on a specific person. Reads used to be the part of the game hardest to buy or download. You had to build them out of hands you had played.
What changes things is speed and cost, and this experiment shows how low they have become. It was built and run overnight, by a
coding agent, on a platform where a new model is one line of configuration, for $2.13 on Jev. I don't think anyone needs a research breakthrough or a frontier lab to build the tool I described above. My guess is the scarce parts are the labelled data and the patience to define the questions properly. When something is that quick and that cheap to try, I doubt it stays with a few serious players. I wrote a while ago about how the perception
of cheating killed online poker, and the argument there was that the damage came more from the perception than the practice. I expect tools like this to spread, and I cannot see a site
policy that would fix it.
What we are doing next
This was a useful proof of concept and another demonstration of how fast you can move on the Cloudflare
Developer Platform with a good foundation in place. My overriding sense, though, is that we have not yet worked
out how to use tools like Jev properly, especially for our product and company.
People I respect are finding uses which are likely much more suited to the strengths of System One models: driving iOS simulators for testing, categorising support emails and weekly newsletter content, automated competitor tracking, and possibly making company-brain products more resilient to contradictory writes. We have started
exploring these and will cover them in the final post.
One thing I have always wanted is a better testing bot. Our game mixes turn-based and real-time actions with a strategy layer around team affiliations and temporary alliances,
which made a sophisticated bot too complex and time consuming to write. We made do with simple code that uses basic starting hand charts and some basic betting logic, then takes essentially random actions. It is fine for getting a feel
for the speed and rhythm of a game, but not much else.
Over a few weeks we want to see whether we can build that, mixing deterministic logic with targeted Jev
calls. It would still be flawed and exploitable, but it would be an interesting project. So today we have
launched a Humans vs Machines version of unpoker rivals
(the level titles and blurbs are inspired by classic human vs machine battles). Tables will mix humans with
Jev-augmented bots, which we will improve iteratively using the hands played against them. A set of evals, if you like, or Westworld for poker. We will report the results back later.
We know Jev can't play poker that well, but can you beat it? Click here to find out!
Philip Atkinson, CEO, October 2026