An infinitely scalable game on Cloudflare
Part 2 of 4. A series of posts relating to our use of Cloudflare: Thank you, John Graham-Cumming; An infinitely scalable game on Cloudflare; Can Jev play poker? (coming); What Jev is actually good for (coming).
In the first post in this series, I told part of the story of how we ended up on Cloudflare and the role played by John Graham-Cumming and Kenton Varda. This is the technical
post about our usage that I think John would have liked me to write.
Every startup faces the same architecture question. How do you build a system that would survive the front page of Hacker News,
but costs you nothing while it sits idle? You are short of capital, time and people, and if things go well you can't afford to
spend six months redesigning it. Speed is really the only advantage a startup has over an incumbent, and your infrastructure
largely decides how fast you can go.
For us the question came with a specific set of problems. Where does the state of a live game go, when serverless functions
forget everything between requests? How do you get the game client to players quickly, wherever they are and whatever device
they are on? And where do the records of logins and purchases live? When I tried to answer those on AWS (at one point I was
working out how to shoehorn a poker game into a Lambda function, or some sort of EC2 setup), every answer seemed to need
another service underneath it, and the structure always felt like it was fighting us. Cloudflare had a product for each of
those problems, and the diagram below shows how they fit together.
It's not comprehensive, but it does cover
most of the structure and how it relates to the elements of the Cloudflare Developer Platform. Over time we've gradually used
more and more Cloudflare services. Partly this is due to need, but it is also because the platform started with little more than Workers and
KV and is now vastly more service rich. What we have found is that we started out using an external service (for example for observability),
but then Cloudflare has acquired or built a product which in many cases at least matches that which we were using. It then tends to be both a simplification and an advantage to move it onto the platform (lock in costs accepted).
High level overview
Functions that run in whichever Cloudflare data centre is nearest the
person calling them.
I think when they were originally released, a common proxy was 'a Node.js-like server that lasts one request'. That sort
of helps conceptually (they share common V8 underpinnings), but it
probably also led to frustration as a Worker was not 100% compatible with
Node (although that has been substantially updated over the years, and some of the Node core features make no sense in a Worker anyway).
I think a much better way to think of them now is as part of a data flow.
Some bit of data comes in (or is retrieved), you do something with it (the
Worker part), and then something exits (at which point the Worker is
gone). Their true power is as a sort of utility knife for
any incoming bit of data. For any incoming request, they can copy, route,
tweak, log etc. Want to do an A/B test on some traffic? Just route it
through a Worker and then write a function that selectively routes it
based on some condition. Want to do a critical DB migration? Capture the
incoming writes, duplicate them to multiple destinations, then selectively
adjust the traffic as the migration hits key checkpoints.
While simple to understand, they do have lifecycle nuances and limits that you have to be aware of.
We use a Worker as a sort of API endpoint that clients (either game or
third party services) talk to. Anything which is a single unit request,
stuff like authentication, routing, rate limits, purchases and webhooks,
is handled in a Worker. They can also be triggered on a schedule, if you
want repetitive tasks, and can make requests to external services or other
Workers or Durable Objects via RPC or fetch call. This composability and
flexibility is one of their strongest features.
Hosting for the built front end, deployed from GitHub and served from the
same global network. It has been superseded by Workers Static Assets, which covers the same ground and is what Cloudflare now recommend for new
projects.
For us, the unpoker rivals client is a highly customised static Astro build, one HTML file per rivalry,
so the first thing a player loads is a file sitting near them rather than a
page assembled on request.
The worst named best product in cloud computing!
To use layman's terms again, one way to think of them is that they are like
a Worker, only with a local memory that can be recreated if they are ever
evicted, and a specific address, so you can get back to them. Each one is a
named object that exists in exactly one place at a time (mostly within the
bigger Cloudflare colos I believe).
They differ from Workers in many ways, such as having their own private
SQLite database, holding a WebSocket open to each connected client, and
setting alarms for scheduled actions and wake-ups. Like Workers, they have nuances around storage reads/writes, lifecycles etc.
This product is really the core of the unpoker rivals system. We have two classes
of Durable Object: one holds the matchmaking queue, and one represents a
single poker table, which is where the game logic and state live.
Once a game is complete, the data from the Game Durable Object instance is
archived and its storage cleared. Due to the pricing structure, this
makes the costs of scaling the game essentially linear
(although it still requires work to optimise storage and logging volume).
A SQL database, with SQLite's SQL semantics. It is fast and pretty reliable for simple loads,
and there are obviously also external Postgres/MySQL options if you prefer.
We use this for the normal things that outlive a game and have to be
recorded and queried, like players, instances, purchases and rewards.
One nuance with Workers, which are edge based, is that
a storage call can negate the edge advantage if you have to read
from or write to one before responding, even when read replicas sit closer
to the edge.
A key-value store designed for things read constantly and written rarely,
with the reads cached near whoever is asking.
We use it to store configuration files, keys and rivalry based scores. This
is a good storage primitive where being a second out of date costs nothing
and being slow costs a player.
As with all the storage products that the Developer Platform offers, it has specific strengths and weaknesses. In this case it is eventually
consistent, so any write might take 60 seconds or more
to propagate globally.
Object storage for files. It behaves like the storage bucket you are used to
(S3 etc), without the bandwidth charge for reading your own data back out, which is
the reason most people move.
We use it to archive the data of finished games, and to hold the images and
assets the client loads.
Workers AI and AI Gateway
Models running on Cloudflare's own hardware, called from a Worker as a
function rather than over the internet with a key, with the gateway in front
for routing, caching and seeing what you spent.
We mainly use this for product related work: analysing logs and events,
triage, and experiments like the one
in the next post. The content of specific rivalry instances is generated locally today,
with a migration onto the platform planned.
A Worker whose job is to receive what the other Workers logged, after the
fact and off the critical path.
Ours enriches those events and forwards them on, which is how logging,
errors and analytics leave the system without any of it slowing down a
request a player is waiting on.
Other aspects, not shown on the diagram
Observability captures invocations and console.log calls, with structured data, from all
Workers and Durable Objects.
Secrets Store holds credentials once for the account rather than once per Worker.
Designing so the platform does the scaling
No platform, cloud or otherwise, is a magic bullet. In some situations constraints dictate the choice. On Cloudflare, one of the things I learned as
we grew up with the product is to get the base shape right. With that,
scaling becomes a problem for the platform, rather than the design of the system. Here are some
general rules we have come to follow from our experiences.
Try to keep the state where the work happens. For our product, the state of a
single game is stored inside a single Durable Object instance. Nothing else needs to read it while the game is in progress, so adding more
games just means creating a new instance. This also means that reads/writes to the state are extremely fast,
encapsulated within the instance, and you reduce the surface area for problems that are somewhat inherent to
distributed systems.
Pick the correct storage product for the task, use case and cost. A Durable Object instance's storage costs money for as long as it exists, and in our case a finished game is just a data record. Keeping that data distributed
across hibernating instances would make little sense. When a game ends, we stream and compress all of the data (a nuance of the Workers platform is that you
sometimes have to manage data which would push you over the memory limits, and streaming resolves that). It is then written to R2 and the instance's
storage is emptied. This is one method for optimising storage based on use case and the different pricing models between R2/D1 and a Durable Object.
Understand the strengths and weaknesses of the edge. A Worker can run close to the player, but many
requests require a call to storage (to retrieve an account). Depending on the nature of the storage and where the read/write has
to travel to, this can eat the benefit of moving the handler nearer.
Optimise observability.
If there was one area Cloudflare was slightly weaker on, it was observability. It is inherently harder
to see inside a Worker, but the Observability features have improved this a great deal. Now the challenge is
efficient logging and profiling. In our case, and probably most others, logs and events scale with use, so a logging decision is also a cost decision.
What is worth recording, how structured it is and
how long it needs to live are all important questions to consider (and can change with scale). We also optimised by moving some analytics and tracing into a Tail Worker, which keeps that work off the main request invocation.
Understand the pricing nuances. A poker hand is mostly
waiting: for a player to act, for a clock to run down, for the next stage
to begin. Keeping a WebSocket connected to a Durable Object instance (especially if it was sending occasional heartbeats) would
just mean paying for wall clock time while it was idle. Cloudflare enabling hibernation for WebSockets instantly resolved this
(and radically altered the unit economics of a single game). If you need to guarantee that a Durable Object wakes up (or acts on a schedule, like starting
a new hand), an alarm means the object can hibernate until the moment it is needed and therefore not incur charges.
Why Cloudflare
If you had to ask me what is the best aspect about our use of the Cloudflare Developer
Platform, I would choose the phrase 'elegant simplicity'. AWS, GCP and Azure have a huge range of products and
services (many market leading), but I think only Cloudflare has the balance right between product range and feel. Every Cloudflare product above is technologically complex. I'm sure if I ever had a conversation with Kenton about Cap'n Web, or Rita Kozlov about
elements of Workers AI, it would be a real challenge to understand the scope of what is happening behind the scenes. For the end user however,
each product can seem like a simple unit. A single Worker invocation processes a single event on a clear path. A Durable Object instance handles
a specific set of events for its lifetime. If you are good at building those units, and at putting them together, then you have
a very powerful, extraordinarily scalable system, and one which is easy to reason about. That makes a huge difference when you are building
and maintaining one.
In a narrow, games sense, a turn based game with a limited time frame is about as neat a fit for the Cloudflare Developer
Platform as you will find. I'm sure there are other games being built this way, and it is probably what John had in mind
when he tweeted about it in the first place. Each game instance becomes a Durable Object, which means you have a single place for all the game
logic and related player connections. All the other stuff around it, the supporting tooling, the storage, the transactional aspects,
fit well in a Worker or the distributed storage layers. Note that all of this is also behind the features
which Cloudflare is traditionally relied upon for as well, DDoS and bot protection, so you end up with
a significant end to end solution, with one control plane.
In the original draft of the blog post, I think John would have liked me to talk about the traditional iGaming technical platform model
(like the one we had at PokerStars) and contrast that with my approach. Those were generally self-hosted at a local ISP. They were almost exclusively owned, bare-metal implementations, for cost and sometimes regulatory reasons. A setup such as the one we use would have been unthinkable
back then (AWS Lambda only went GA in April 2015 and Workers in March 2018, I believe). From conversations I've had with former
colleagues, a lot of the big iGaming companies now have quite significant hybrid setups, often using AWS. There are obviously now
many infrastructure paths to running a game like unpoker rivals at scale. They all have different cost profiles, platform strengths
and weaknesses (and the internal comfort with a platform and technical capabilities matter as well). Despite those options,
I still think Cloudflare was the right choice then and now.
The reason for that (alongside the elegant simplicity) comes back to the question I started with: speed. You can build
something quickly, scale it massively and adapt it as needed, and at least in my experience you never feel like you are
fighting with it to fit something into someone else's structure. Even when Durable Objects were an unfinished beta with
hand rolled deployment scripts, you could see how quickly you could build something quite substantial.
Now, I chose the title of this post knowing it is not strictly true. Nothing is infinitely
scalable, even on Cloudflare. If you did ever find the limits, you'd have a 'nice problem to have' situation!
What is true, is that the least visible of the main cloud platforms has some exceptionally
effective primitives. They might not fit every use case. I doubt anyone else is building the sort of poker-ish game platform
we did. Even so, if you are choosing a platform to build on you should probably give them serious consideration. I was lucky enough to see a tweet
and get on an early beta for Durable Objects. It was good then, and every product is now more robust, more polished and
more developer friendly. The direction of travel is clear.
As I said in the first post, good people and good technology are a pretty powerful combination.
Philip Atkinson, CEO, September 2026