Design a URL shortener: on AWS and at scale

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The reference architecture under Design it is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read it by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

An ECS resolver behind an ALB makes its in-memory connections and cache behavior explicit at sustained traffic. API Gateway/Lambda is a reasonable smaller baseline. The database owns names. A cache only accelerates reads of already-owned mappings.

Provision resources, then connect the application

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

How big companies keep and count clicks

Every redirect is a click. At this page's numbers that is 3 billion clicks a day. So where do they live: a database, or files in S3?

Both, and each does a different job. The redirect answers first and never waits. The click goes on a queue. A worker takes clicks in batches and writes each batch twice: as lines in cheap files that keep every click (the history), and as one +N per link on a counter (the live number). Twitter and Bitly both keep these two halves, as described below.

What Twitter and Bitly built

Twitter, 2021. Twitter says it processes about 400 billion events a day in real time (Twitter Engineering, 2021). One of its pipelines counts impressions and engagements on Tweets, the counts its ads revenue services read. The post's table puts that pipeline at about 4 million events a second.

• Before: a lambda architecture, two paths for the same events. A batch path ran Scalding jobs every hour over Hadoop logs on HDFS and wrote counts into Manhattan, a database. The post's table puts the batch numbers about a day behind. A real-time path read Kafka with Heron and kept live counts in a cache called Nighthawk, 10 seconds to 10 minutes behind. When Heron fell behind, the fix was to restart it, and restarts could lose events, so live counts could come out low. A query service merged the two.
• After: a kappa architecture, one path. Each event gets a UUID and goes from Kafka to Google Pub/Sub. Dataflow drops duplicates by that UUID, within a time window, and adds the events up as they arrive. The counts go to Bigtable, about 10 seconds behind. No events are lost on a restart, and late events are counted.
• They kept the raw events anyway. Twitter also wrote raw events into BigQuery to measure duplicates, and loaded the old batch counts there to compare every key. For the Tweet interaction stream, over 95% matched exactly. Most of the rest were late events the old batch had dropped.

Even after removing its batch path, Twitter still wrote raw events somewhere it could query them, and that is how it checked the live numbers.

Bitly. Bitly's engineers wrote in 2012 about writing metrics inside the request: if the metrics system goes down, do your requests hang or fail? Their answer was to put each event on a local queue, NSQ, and let another process write it on (Bitly Engineering, 2012). NSQ copies every message to each "channel" of a topic, and NSQ's design document draws a clicks topic copied to three channels: metrics, spam_analysis and archive (NSQ design).

The archive is kept on purpose. In Bitly's words, it is "kinda hard ahead of time to think of all the types of metrics you might want to collect", and the archive is a "plan z" for when data downstream is lost. A 2014 write-up of a Bitly talk adds that each redirect sends one message to an archive service that saves it to HDFS and S3, to real-time analytics, to longer-term history analytics and to an annotation service, at 6 billion clicks a month (High Scalability, 2014). That last part is a third party's notes of the talk, not Bitly's own text.

One click in S3, and the same click in DynamoDB

In S3, a click is one line in a file. The line holds an event id, the code, the time, the referrer, the country and the device: about 150 bytes. Firehose gathers lines and writes a file when 5 MB or 300 seconds is reached, whichever comes first. Those are its defaults, and you can change both (buffering). It puts each file under a folder for the hour, in UTC, such as 2026/10/04/13/ (object names). Turn on its newline delimiter, or end each record with a newline, or the lines run together (delivery). A line is written once and never changed.

In DynamoDB, the live number is one item per link in click_counts: the key, a count, and when it was last updated. Click worker adds up its batch first. If the batch holds 37 clicks for launch, it makes one update, count + 37, not 37 updates.

Wins and losses, at this page's 3 billion clicks a day. Prices are for us-east-1, read on 2026-10-04 from AWS's pricing pages for DynamoDB, Firehose and Athena, and from AWS's price list file for S3, which carries the rates the S3 pricing page shows once you pick a region.

What the raw clicks are for

Each of these needs the clicks themselves. A total cannot do any of them.

• Charts for the link's creator: referrers, countries, devices and time of day are a GROUP BY over the lines.
• Bot and fraud filtering: a bot shows up as a pattern across many clicks. You find it in the history, then count without it.
• Abuse and spam takedowns: a sudden burst from one referrer or one country is visible in the lines. NSQ's design document draws a spam_analysis channel beside the archive on its clicks topic.
• Billing by usage: if customers pay per click, they can ask to see the clicks behind the bill.
• Recounting after a bug: run the count again over the lines. Twitter checked its new live counts against counts made from its stored events.
• Questions nobody has asked yet: Bitly's reason for the archive.

The answer to give

Keep both, fed from one queue. The redirect never waits. The history is files nobody edits: Firehose into S3, read with Athena. The live number is a batched counter: one +N per link per batch, in DynamoDB. When the two disagree, the files win, because you can count them again.

In an interview:

• Asked for analytics: "Each click goes on a queue and the redirect does not wait. A worker writes every click as a line into S3 through Firehose, in hourly folders, and we query it with Athena. Those files are the record, so we can always recount."
• Asked for a live count: "The same worker adds up each batch per link and makes one atomic add per link in DynamoDB. It is seconds behind, never on the redirect path, and a viral link costs one write per batch, not one per click."
• Asked for both: "Both, from the same queue. The counter is fast but can drift on retries. The files are the truth, and the counter can be checked against them, the way Twitter checked its live counts."

Your design is this design. The design you drew (SQS, Lambda, Kinesis or Firehose, S3, Athena) is the history half, and it is right, with one change. Between Click worker and S3, use Firehose. Firehose writes into S3 by itself, with no code of yours (Firehose). A Kinesis data stream only holds the clicks until a consumer reads them: your code, or a Firehose reading the stream (consumers). On its own, nothing reaches S3. The SQS queue in the AWS table above works the same way: something has to read it. Then add the live half: the same Click worker's one +N per link in click_counts.

Not covered here: unique visitors, which need a visitor id on each line and a different kind of counter, and counting each click exactly once.

Extend the design after the baseline works

Worked follow-up: Allocate custom aliases across two regions

Two customers in different regions both request /launch. Checking each regional cache can return available twice. Replication after both writes cannot retroactively give both customers the promised alias.

Revised architecture. Follow the changed responsibility and failure path below. This is a design to implement. The supplied local example does not provision these components.

What to implement. Route all alias mutations to a designated home-region allocator. Store alias, tenant, destination and request identity in one conditional create. Regional resolvers may read cached mappings under the existing expiry and takedown rules. During loss of the allocation authority, keep safe redirects working and return a retryable failure for new aliases. Promotion requires fencing the old allocator and proving the promoted state contains acknowledged reservations. A second DynamoDB table or DNS change alone supplies neither guarantee.

Walk through the result. Submit launch from regions A and B simultaneously. One tenant receives success and the other a conflict. Next disconnect B from the allocator. B must not allocate launch locally. Record the added cross-region allocation latency and the availability sacrificed to retain uniqueness.

At multi-region scale, choose one owner for alias allocation or a globally consistent authority. Do not let two independent regional caches reserve the same alias. Quantify added write latency and what a region does during loss of the allocation authority.