Why are legacy SIEMs so slow and expensive?

Built for a different time

In a previous post, we briefly called out the two reasons why legacy SIEMs are so slow and expensive: they run on an outdated architecture, and they've pushed pricing far beyond what's reasonable at today's scale.

In this post, we go deeper into the gaps in the legacy SIEM architecture.

These engines were built before modern data warehouse technology and cloud infrastructure primitives were established. Moreover, they were designed for a different use case: flexible, full-text search over unstructured, custom application logs.

As data warehouse technology and cloud infrastructure have improved, and security logs have gotten larger and more structured (e.g., cloud audit logs, endpoint telemetry, network flow logs), the design choices legacy SIEMs made decades ago are increasingly the wrong tradeoffs for modern security teams.

In particular, three design choices are causing issues:

1. Coupling of storage and compute: Compute and storage live on the same host, and work is allocated to the host that owns the data. This makes it hard to get good hardware utilization, prevents efficient scale-out, and rules out cost-efficient storage.
2. Row-oriented storage: Data is stored on disk as rows rather than columns. This makes aggregations across a small subset of columns inefficient and reduces compression rates.
3. Suboptimal indexing strategy: Search is implemented using inverted indexes that take up a lot of space, and/or probabilistic filters that are too coarse to accelerate common search patterns.

Let's dive in.

Coupling of storage and compute

In a legacy SIEM, data is stored on a compute node with storage attached. Since attached disks have limited size, the SIEM scales up by adding more nodes. When you search the SIEM, the work gets farmed out to each node that holds relevant data. Each node processes the search on its portion of the data and sends the results back¹.

This causes three issues at scale:

• Hot shards: Since nodes can't share compute resources, nodes holding frequently searched data become bottlenecks while the rest of the cluster sits underutilized. This increases the compute capacity required to provide a given level of performance.
• Slow scaling: Scaling up and down means splitting or combining nodes, which is a cumbersome operation. As a result, teams can't scale up effectively for unforeseen peaks, which reduces performance further, usually when you need it the most, such as during an incident.
• Expensive storage: Since nodes must use attached storage, the SIEM pays a premium over object storage (e.g., in AWS, EBS is ~7x more expensive than S3 per GB, assuming two replicas). This significantly increases the cost of storing data over long time periods.

As a result, legacy SIEMs are both compute and storage constrained. They can't scale compute effectively because of poor utilization, and they can't store data efficiently because it's tied to underutilized compute and kept on expensive storage.

No surprise that they try to keep your query volumes low and your retention periods short.

Row-oriented storage

In many legacy SIEMs², each row is stored together on disk. This makes it efficient to write and update data, which is why most operational data stores store their data this way.

For logs, however, this causes two issues:

• Inefficient aggregation: In security logs, aggregations usually run over a few small columns, such as the timestamp, event name, or actor ID. Since these columns are a small fraction of the full event size, column-oriented stores can skip most of the information in each event. Row-oriented stores, on the other hand, must read the whole row. As a result, they process far more data per query, which makes queries slow and resource intensive.
• Poor compression: Security logs contain a wide range of data, from timestamps and numbers to strings and UUIDs. When these are stored as separate columns, they compress better, because compression algorithms can take advantage of patterns within each column. Combined with an inefficient indexing strategy (see below), the difference becomes meaningful: legacy SIEMs usually produce ingested logs (including indexes) that are between 0.5x and 2x the size of the raw data, while columnar stores usually land between 0.1x and 0.2x.

The issues caused by row-oriented storage compound those caused by storage-compute coupling. Not only are legacy SIEMs compute constrained, but their compute is also inefficient because it reads far too much data. Likewise, not only are legacy SIEMs storage constrained, but they also have to store a far larger share of the raw data because of inefficient compression and indexes.

Suboptimal indexing strategy

SIEMs use indexes to accelerate searches for specific terms in your logs. Legacy SIEMs usually rely on one of two indexing strategies: inverted indexes (e.g., Lucene), which map terms in your logs to the rows that contain them, or probabilistic indexes (e.g., Bloom filters), which let you skip rows that are guaranteed not to contain a term.

While both techniques have valid use cases, there are two issues with how they are implemented in legacy SIEMs:

• Bloated inverted indexes: Inverted indexes offer a lot of precision, but they are expensive to create and take up a lot of space. That's usually a poor tradeoff for security logs, which are voluminous and read infrequently. Moreover, security searches rarely require the precision these indexes provide³. As a result, these indexes increase ingestion cost and storage overhead, which again reduces how much data you can afford to ingest and store. They also make it harder to cache the indexes efficiently, which increases query latency further, particularly over long time periods.
• Coarse probabilistic indexes: Probabilistic indexes can work well for SIEMs (we use them!). However, legacy SIEMs that rely on probabilistic indexes often use coarse implementations that limit their ability to accelerate your queries. For example, the filters may be limited to a few columns (e.g., just '_raw'), or only apply to exact matches in a column. As a result, common needle-in-a-haystack searches against terms such as an actor's first name or a host's prefix regularly get limited or no acceleration⁴.

Suboptimal indexing further compounds the issues caused by storage-compute coupling and row-oriented storage. Since the indexes are either too voluminous or too coarse, they put pressure on either the scarce storage or the scarce compute of legacy SIEMs.

What's the alternative?

Few practitioners need convincing that their SIEM is slow and expensive at scale. But a reasonable question is: what's the alternative?

As an industry, we've resigned ourselves to treating SIEMs as a highly constrained, scarce resource where only the most important logs may enter. Otherwise, your budget blows up and your SIEM grinds to a halt.

However, as we've shown above, these constraints are the result of outdated technology, not the laws of physics!

A Modern SIEM, purpose-built for security logs at scale, can easily handle more than 10x the log and query volume of a legacy SIEM without increasing cost or complexity. By separating storage and compute, storing data as columns, and using an optimized indexing strategy (along with a number of other optimizations), modern log and query volumes become manageable within existing budgets and team capacity.

Don't settle for less.

FAQ

Footnotes

1. This is a simplified explanation, meant to capture the essence. For more details, check out the public documentation of your legacy SIEM of choice. LLMs are very good at this (hint, hint...) ↩

2. This is an area where SIEMs vary quite a bit. A few legacy SIEMs are also column-oriented. So depending on the specific SIEM you're comparing IceGuard to - this may or may not apply. ↩

3. Since inverted indexes contain exact positions, they can be used to effectively identify large intersections of multiple terms (e.g., a phrase) or substrings within a word. However, for most security use cases you're looking for one or more terms (e.g., a few IOCs or specific entities), not full phrases or substrings within a term. The overhead of storing the full index is rarely worth the acceleration for these rare cases. ↩

4. Indexing is a nuanced topic, and the implementation varies a lot per SIEM. We recommend benchmarking your specific query patterns at scale during a POC, and not relying on any vendor claims to assess performance (including ours!) ↩