Bring up system performance and two words show up almost immediately: latency and throughput. Both have something to do with "speed," so they're easy to mix up at first. I used them interchangeably myself for longer than I'd like to admit.
Yet what they measure is entirely different. Roughly speaking, latency is how long one operation takes, and throughput is how much gets done in a given period. Think of a ramen shop: the wait for your bowl versus the number of bowls the kitchen turns out in an hour. Keeping the two apart makes performance discussions surprisingly tidy.
Latency as the Wait for One Operation
Latency is the time from when an operation starts until its result comes back. For an API, the client sends a request, the server processes it, and a response returns. The time that round trip takes is the latency.
IfGET /users/1 has a latency of 100ms, that API answers in roughly 100ms. In short, it measures how long you're kept waiting for one operation. Call it the time between sitting down at the counter and getting your bowl.
In a web app, it maps directly onto the gap between pressing a button and seeing the result. The higher the latency, the more likely users are to feel things are "kind of slow." No theory required. People who are kept waiting remember exactly one thing: that they were kept waiting.
Throughput as the Capacity to Handle Volume
Throughput, on the other hand, measures how much work can be completed in a given period. If an API server handles 1,000 requests per second, its throughput is 1,000 requests/sec.
Naturally, this isn't limited to APIs. Message processing might be 50,000 messages/sec, image processing 500 images/sec — any count per unit of time works. Throughput, in other words, looks at how much volume the system as a whole can handle. In shop terms, it's bowls per hour.
Suppose an API server has a latency of 100ms and a throughput of 1,000 req/s. A single request comes back in about 100ms, and the system as a whole processes about 1,000 per second. Both are about performance, no doubt. They're just facing in entirely different directions.
Does Low Latency Mean High Throughput?
This is where it gets a little tricky. Intuitively, if each operation is fast, you should be able to do lots of them. Sometimes that's true. But low latency does not imply high throughput.
Say a server handles one request in 100ms. If it can only handle one at a time, simple arithmetic says about 10 per second. If instead it can run 100 of those 100ms operations at once, throughput jumps to about 1,000 req/s. Same cooking speed, but the number of seats and pots changes the bowl count a hundredfold.
So even with the same latency, throughput changes with concurrency. Concurrency is how many requests can be handled at once. Latency is the time one request takes; concurrency is how many can be in flight together. What gets finished in a given period as a result is the throughput. Glancing at latency alone and declaring "this holds up under heavy traffic" is, to put it mildly, premature.
Latency Matters for Web APIs
Picture opening a product page on an e-commerce site. The browser hitsGET /products/123, and the API returns a response. Answering in 100ms versus 3 seconds feels like two different products. When a user is waiting right there, latency tends to be what counts.
That said, a busy service needs throughput too. But even at 10,000 req/s, a p95 latency of 5 seconds makes for a rather painful service. Handling a lot and letting each user work comfortably are two different things.
Throughput Pays Off in Batch Processing
Conversely, when nobody is waiting on the result in the moment, throughput can matter more. Consider a job that converts a million images. It's the kitchen's prep work, done in bulk before the doors open.
Trimming the per-image time from 100ms to 90ms may not amount to much if only a few run at a time. You'll get far more out of raising the volume per unit of time, say from 100 images/sec to 1,000 images/sec.
Concretely, picture a job queue with Worker A, B, C, and D lined up behind it, processing in parallel. The latency of each job barely changes. Overall throughput, though, rises with every worker you add. The improvement is proportional only in an ideal case; shared limits such as the database, an external API, CPU, or network also need to be checked.
In IoT, It Depends on the Operation
IoT is no different. Take a system that receives telemetry from a large fleet of devices. Data travels from the device through an MQTT broker to a consumer and gets written to a database. With tens of thousands of devices reporting periodically, 20ms versus 30ms per message matters far less than how many tens of thousands you can handle per second. Here, throughput is what to watch.
Sending an emergency stop command to an industrial robot or a machine on a production line is another story. It's a single command going from the server to the machine. Nobody wants to hear how many tens of thousands of emergency stops per second you could process. The only question is how soon the command arrives. Here, latency is what matters.
So even within one system, the metric that matters changes by operation: throughput for receiving telemetry, latency for real-time control. Rather than labeling a whole system "latency-focused," it's often more natural to think about each operation on its own.
Latency Falls Apart Near the Limit
Latency and throughput are separate metrics, but they're not entirely unrelated. Say a server can handle at most around 1,000 req/s. If only 500 req/s are coming in, there's plenty of headroom. Requests get processed as soon as they arrive, so latency stays stable.
But as load climbs toward the limit — 900 req/s, 950 req/s, 990 req/s — requests start having to wait. They line up in a queue before reaching the server. The server's actual processing time hasn't changed, yet the latency users see gets worse by however long they sat in that queue. A line has started forming outside the shop.
Now suppose incoming traffic reaches 1,200 req/s. Twelve hundred arrive every second, but only a thousand get handled. The backlog grows by 200 every second. Throughput plateaus around 1,000 req/s, the queue keeps growing, and latency alone keeps getting worse. Three things happen at once. Leave it alone, and the request at the back of the line won't come back before the person waiting for it retires.
So in load testing, looking only at the maximum req/s you can push isn't enough. You should also watch how latency changes as the load goes up.
The Trap of Average Latency
Judging latency by the average alone is risky too. Hear "average latency: 100ms" and it sounds quite fast. In reality, p50 might be 50ms, p95 300ms, and p99 2,000ms. Most requests are fast, then, but some take more than two seconds. The average buries those slow requests rather neatly.
That's why API latency is usually viewed through percentiles like p50, p95, and p99. A p95 of 300ms means 95% of requests finished within 300ms. Compared with a plain average, it shows how many slow requests are actually out there.
Which One to Optimize
In the end, it depends on the system and the operation. As a general tendency, it breaks down roughly like this.
| Operation | Metric that tends to matter |
|---|---|
| Web API | Latency |
| Batch processing | Throughput |
| High-volume message processing | Throughput |
| IoT telemetry | Throughput |
| Real-time control | Latency |
That said, you can't simply decide "it's web, so latency" or "it's batch, so throughput." A web API under heavy traffic needs throughput. A batch job may have a strict deadline for completion.
Frankly, I think asking which one to improve is the wrong way in. What you should ask first is what hurts if this operation is slow, and how much volume it has to handle before things break. Requirements on both — p95 latency under 300ms and throughput above 5,000 req/s, say — are perfectly ordinary. Meet the latency you need while securing the throughput you need. Real design more often than not comes down to that.
The Vagueness of "Good Performance"
On reflection, "this system is fast" is a remarkably vague claim. It might mean latency is low, or it might mean throughput is high. Some systems answer a single request quickly but crumble under heavy traffic. Others chew through huge volumes of data but take their time on each item.
So when talking about performance, you need to separate out what exactly is fast. The foundation for that is latency and throughput.
That's the difference between latency and throughput, and how the two relate. Latency is the wait for one bowl; throughput is the bowls per hour. If you're going to call a shop with a line out the door "fast," you'd better say which kind of fast you mean.