When Your Data Is Too Big for a Normal Database

There is a scale at which a general-purpose database stops being the answer. Telecom operators keep call-detail records for legal retention — hundreds of billions of rows that must be ingested without falling behind the network and still answered in under a second for a billing dispute or a fraud case. No off-the-shelf database holds both ends of that trade-off affordably. The move is to stop forcing the data through a generic tool and engineer for the workload’s actual shape: append-only ingestion, immutable records, and queries dominated by subscriber-and-time lookups.

Most data problems are not scale problems, and reaching for exotic infrastructure when a well-tuned Postgres would do is its own kind of mistake. But there is a real threshold — usually a mix of volume, ingestion rate, retention window and latency requirement — past which the general-purpose database is fighting you, and every optimization buys less than the last.

We hit that threshold building Teramanager, a call-detail-record system that ran in production at two national carriers. The requirements collided: ingest everything the switches produce, continuously, without falling behind; retain it at terabyte-and-beyond scale for years, because the law requires any record to be retrievable; and answer point queries in under a second across hundreds of billions of records. A generic relational database of the era could hold any two of those. Not all three, not at a cost that made sense.

The insight: the workload has a shape, and the architecture should match it

The way out is not a faster database. It is noticing that this workload is not general at all — it has a very specific shape, and designing for that shape changes everything:

Ingestion is append-heavy and continuous. Records arrive in a firehose and are never updated once written. That means you can optimize hard for write throughput and immutability, and drop all the machinery a general database carries for in-place updates.
Records are immutable. A CDR, once written, does not change. Immutability unlocks aggressive compression and layouts that a mutable store cannot use — which is what makes years of retention affordable.
Queries are predictable. Billing, interconnection settlement and fraud teams search by subscriber and time range, overwhelmingly. Index for how they actually query, not for arbitrary access, and the sub-second answer falls out.

Match those three and the impossible trade-off dissolves. You are no longer asking one tool to be excellent at everything; you built a system that is excellent at exactly this. The same purpose-built retrieval engine (we called it XLBase) went on to power national directories and legal case-law search — different data, same principle.

How to know you’re actually at that threshold

Before building anything custom, be honest about whether you are really there. Signs that you are:

Ingestion can’t keep up. The database falls behind the incoming rate at peak, and scaling it up buys diminishing headroom.
Storage cost dominates. Retention is a line item you dread, because the general store’s on-disk format is not built for immutable data at this volume.
Queries that should be instant aren’t. Point lookups degrade as the table grows, and indexing strategies stop helping.
The requirements are non-negotiable. You cannot shorten retention, sample the data, or relax the latency — regulation or the business forbids it.

If you can relax any requirement, do that first; it is cheaper than custom infrastructure. If none of them bends, you have a systems-engineering problem, and the right move is to build for the workload.

The general lesson, past telecom

CDRs are the classic example, but the pattern repeats anywhere immutable events pile up faster than a general database wants to hold them: IoT and sensor streams, financial ticks, event logs, clickstreams, observability data. The instinct to reach for a bigger managed database is right up to a point and wrong past it. Past it, the leverage is in matching the architecture to the workload’s real shape — which is the kind of custom software for data-intensive systems we build.

Frequently asked questions

When should we build custom instead of using a managed database?

Only when you have genuinely hit the wall: ingestion falling behind, storage cost dominating, point queries degrading with growth, and requirements (retention, latency, completeness) that cannot be relaxed. If any requirement can bend, relax it first — custom infrastructure is the last resort, not the first.

How do you get sub-second queries over hundreds of billions of records?

By exploiting the workload’s shape: immutable, append-only data allows compressed, query-aligned layouts, and indexing built around the few access patterns that actually occur (subscriber and time range) turns a full-table problem into a targeted lookup. The speed comes from the design fitting the queries, not from brute force.

Isn’t this just a job for a data warehouse or a NoSQL store?

Sometimes those are the right fit, and we use them when they are. But warehouses optimize for analytical scans and many NoSQL stores trade away the exact retrieval or consistency guarantees these workloads need. The decision is driven by the specific mix of ingestion, retention, latency and query pattern, not by a default technology.