Cost confusion
There is a prevailing belief in the SecOps community that there is too much security data to store in one system.
We think that's wrong.
In fact, we think a single system is a far better place to store it than any form of decoupled architecture.
The confusion comes from SIEM incumbents overcharging and underdelivering, not from fundamental limits of technology. To understand that point, it's worth looking at some numbers.
On a modern system, ingesting, processing, and detecting against one raw GB of data should cost less than $0.02. Storing it for 90 days should cost less than $0.02 in an object store. A Modern SIEM ingesting 1 TB per day should therefore cost less than $15,000 per year in hardware. The vendor needs to make money too, of course. Assume an 8x markup over the hardware cost, totaling $0.30 per GB. That brings the annual cost to $120,000.1
This is a small fraction of the economic activity required to produce those logs.
For example, an employee laptop produces 0.1 to 0.3 GB per day if you log aggressively. A production server may produce 100 times that. Now you're looking at a logging bill of less than $35 per laptop and less than $3,500 per production server per year.2 Apply this math across your systems and the logging bill is usually a tiny fraction of the cost of the activity you're monitoring. With sensible logging practices, ingesting the logs into a reasonably priced SIEM should be a no-brainer for almost any organization that cares about security.
It's pricing, not physics
So why are people complaining about SIEM pricing?
Because incumbent SIEMs aren't charging $0.30 per GB. They're charging between $3 and $6 per GB, a markup of more than 100x.3
No wonder users are struggling with the cost.
The problem is the pricing, not the physics.
Let's fix the root cause
So what happens if we fix the ingestion price?
The workarounds disappear.
You no longer need a pipeline to route your data because all your logs go directly into the SIEM. Since the SIEM stores everything, you can ditch your cold store and federation engine as well. If your SIEM offers fully managed ingestion, which it should, the management overhead disappears too. Point your log source at the SIEM, or vice versa, and let the logs flow.
No need for additional complexity!
At current SIEM prices, it's understandable that users don't ingest everything. But that's because SIEM vendors are overcharging, not because there is too much data.
It's time we fixed the root cause and stopped peddling workarounds.
FAQ
How did you arrive at the cost of compute and storage?
The compute numbers come from our internal testing on large-scale, simulated SIEM workloads. We run the workload and identify the compute capacity, measured in vCPUs, required for ingestion, normalization, enrichment, storage, detection, and a modest volume of investigation. We then multiply that capacity by cloud list prices for vCPUs and divide by the number of gigabytes ingested during the workload.
For storage, we look at compression rates across all our sources (because each source compresses at a different rate). We use the least compressible sources as a baseline, then multiply by the list price per GB in a cloud object store.
This is not an exact science. In a real production environment, you would likely land between 0.5x and 2x this number. Since most large enterprises have at least a 20-40% discount off list prices, your cost will probably be on the lower end of that range.
My SIEM vendor claims its costs are much higher per GB. How are you so efficient?
Legacy SIEMs use monolithic architectures that don't separate storage and compute. This makes them far less efficient for two reasons:
-
They must shard their databases, which means data is scattered across worker nodes and workers can only process data stored on their own machines. This lowers compute utilization and performance because multiple workers can't process data in a single shard. You also end up with hot shards that receive far more queries per stored GB than the average. Admins must rebalance those shards, a slow and painful process, or accept lower throughput.
-
They must store data on drives connected to each worker instead of using object storage. That makes it harder to use storage efficiently, especially for data that is rarely accessed. Local storage also costs about five times more per GB than object storage.
We use an engine built for logs that separates compute and storage and uses object storage as its backend. This improves compute and storage utilization while keeping storage costs at object-store rates. We also apply other optimizations to make ingestion, detection, and querying more efficient.
For a deeper look, see why legacy SIEMs are so slow and expensive.
Footnotes
-
Hardware: ($0.02 compute + $0.02 storage) × 1,000 GB × 365 days = $14,600 per year. With an 8x markup: $14,600 × 8 = $116,800, or about $0.32 per GB. We round to $0.30 per GB and $120,000 per year. ↩
-
Laptop: 0.3 GB × 365 days × $0.30 per GB = $32.85. Production server: $32.85 × 100 = $3,285. ↩
-
These prices apply to the 1 to 10 TB per day range. Above 10 TB per day, SIEM vendors might discount below these amounts. ↩
