6 posts tagged with "siem"

Great in theory, but...​

Query federation is an idea as old as databases themselves. It is as seductive as it is simple: instead of moving the data to the query engine, you query the data "where it lives."

In theory, this increases data access, saves cost, and simplifies your data architecture.

In practice, it rarely works out this way. Let's dig into why.

During POCs and demos, federation works beautifully. These are small-scale, controlled environments where the technology shines. However, as you move from POC to production, there are three issues users regularly run into:

1. Performance: Queries are slow and inconsistent. Timeouts happen regularly, and multiple users are unable to query in parallel.
2. Cost: Spend on data transfer, source system compute, and the federation engine itself grows far beyond what was budgeted.
3. Maintainability: A never-ending stream of correctness bugs, data quality issues, and performance bottlenecks keeps teams permanently in debug mode.

Let's dig in.

Performance​

There are three ways to make search fast and scalable: indexing, caching, and parallel processing. Federated search does not support any of the three properly.

Indexing​

You can't index data if you don't read it and have the ability to access it granularly. This might be obvious to most readers, but it's worth stating explicitly: you can't index a third-party data store or vendor API without reading and storing all the data. This means the only systems you can index are file systems or object stores, and only after you've read them.

At this point, you're describing a data platform, not a federated system, because you're paying the cost of moving and reading the data during indexing.

Caching​

You can't cache a federated system effectively, because you don't know when to invalidate the cache. Security logs regularly get late-arriving events. If those arrive in a third-party system, you have no way of knowing your cache is invalid. Moreover, caching assumes everyone is hitting the same data regularly.

In security operations, most investigations cover new scope. It's rare that you'll reinvestigate the same environments and time periods repeatedly. As a result, your cache hit rate will be very low unless you actively pre-populate the cache, which, again, makes this a data platform and not a federated system.

Parallel processing​

You're bottlenecked by the capacity of the source. Vendor APIs and third-party data stores are not built for massive parallel processing. Nor do your vendors want to pay the compute bill for your large, concurrent queries against their API.

As a result, you'll get throttled.

When you combine this with the inability to cache and index, you get a "lethal trifecta" of poor query performance: you have to scan all the data because you don't have an index, read from a third-party system because you don't have a cache, and read in serial because you can't process the data in parallel.

If your vendor claims their federated search is fast at scale, run a load test: multiple concurrent needle-in-a-haystack queries over long time periods, against the same third-party APIs. Don't fall for a pretty demo, or believe that a small-scale POC is representative of a production workload.

Federation performs well during demos, but fails at scale.

Cost​

Federation is often pitched as a cost saver, but it rarely works out like that in practice. Cost overruns are frequent for two reasons.

Unanticipated data movement​

Even when you query data "where it lives," you have to move the results back to where you're querying from (obviously). If you're not able to push the filter down into the source data store, you have to pull all the data for the time period you're searching over.

This happens more often than you think!

Searches for string patterns, IOCs, actor aliases, or other subsets of the data require an indexed data store (i.e., a SIEM). Against object stores or vendor APIs, you'll have to pull all the data.

And this happens every time you query. If the data resides in a separate region or cloud, you pay the full egress charges every time. You can easily rack up thousands of dollars in transfer costs for a single, ill-formed query against a large table residing in a different cloud.

Now combine that with agentic querying, and it's not hard to imagine how the bill grows: you ask the agent to answer a simple question, it decides to pull the last three months of data from another cloud, and the rest is (billing) history.

Duplicate compute cost​

When you're federating, you're paying for compute twice: once at the source system, and again for the federation engine. Moreover, since neither engine gets to fully optimize the query, execution is often far less efficient than if the query had run directly against the source¹.

Even if the source doesn't charge for queries directly (e.g., your EDR or a security vendor's API), the vendor bears the cost of the compute you're using, and will rate limit or throttle you if you issue a large volume of inefficient queries (which federated systems regularly do).

This might seem like a theoretical concern at the POC stage, but it becomes very real once you hit AI scale. Not only are you paying for duplicate hardware, you're also paying two vendors' margins (often a 6-10x markup over the compute). Combine that with the inefficiency mentioned above, and you can quickly get to >10x the cost of doing the same work in a single system.

Maintainability​

Maintainability is the most important feature you've never heard of. Since federated systems don't own the data or the query execution layer, it's a lot harder to manage edge cases proactively. This makes the system harder to debug and maintain, both for vendors and end users.

Here are some issues federated systems regularly wrestle with.

Correctness​

What happens if you query three sources, and only two return data? What happens if the schema of an upstream source changes? What happens if you get malformed data (e.g., a timestamp in the wrong format)? Since federated systems don't own the data layer, it's far harder to assert what the correct results are, or to retry after partial failures.

As a result, federated systems must be either overly conservative (i.e., fail your query if there is any chance that the results are incorrect) or overly aggressive (i.e., present results that might be wrong). Neither is a good place to be.

Observability​

Databases usually expose unified telemetry to help debug performance issues end to end. Federated systems can't offer any of that for the source systems they query. As a result, performance, data quality, and correctness issues must be detected and debugged across multiple systems, which is about as fun and productive as it sounds.

Moreover, any improvement or fix you'd like to make requires coordinating changes across the source systems and the federation engine, which is not trivial. Worse, source systems often don't expose the knobs required to optimize for the federation engine, since they're designed to meet different objectives (e.g., limiting compute load and maintaining stability instead of maximizing throughput). So even if you find the problems, fixing them is usually painful and complex, or outright impossible.

So what's the alternative?​

If federation is a bad idea (which it almost always is), what should you do?

Centralize the data on a Modern SIEM.

Your queries will be faster, and your platform will be easier to maintain. It's often similar in cost, or even cheaper, and far more predictable.

Most importantly, it lets you start using the data more.

One of our most consistent observations from building data solutions is that once data is available, people start using it. In security, this means hunts, detections, investigations, and reporting. Don't close that door by building your business case on the hope that people won't query their federated data much.

Federation is a great idea in theory. Unfortunately, it rarely delivers as promised.

Buyer beware.

FAQ​

Footnotes​

1. There are many examples where this happens. Among the most common is the following: Federated queries usually rely on joining data from two sources together, such as finding all the endpoint activity (from your EDR) for your admins (from the IdP). Data warehouses are built to optimize these queries, thereby saving compute costs and reducing query time. A federation system, on the other hand, can't push down the filters from one source to another, thereby forcing you to read far more data than necessary. ↩

Built for a different time​

In a previous post, we briefly called out the two reasons why legacy SIEMs are so slow and expensive: they run on an outdated architecture, and they've pushed pricing far beyond what's reasonable at today's scale.

In this post, we go deeper into the gaps in the legacy SIEM architecture.

These engines were built before modern data warehouse technology and cloud infrastructure primitives were established. Moreover, they were designed for a different use case: flexible, full-text search over unstructured, custom application logs.

As data warehouse technology and cloud infrastructure have improved, and security logs have gotten larger and more structured (e.g., cloud audit logs, endpoint telemetry, network flow logs), the design choices legacy SIEMs made decades ago are increasingly the wrong tradeoffs for modern security teams.

In particular, three design choices are causing issues:

1. Coupling of storage and compute: Compute and storage live on the same host, and work is allocated to the host that owns the data. This makes it hard to get good hardware utilization, prevents efficient scale-out, and rules out cost-efficient storage.
2. Row-oriented storage: Data is stored on disk as rows rather than columns. This makes aggregations across a small subset of columns inefficient and reduces compression rates.
3. Suboptimal indexing strategy: Search is implemented using inverted indexes that take up a lot of space, and/or probabilistic filters that are too coarse to accelerate common search patterns.

Let's dive in.

Coupling of storage and compute​

In a legacy SIEM, data is stored on a compute node with storage attached. Since attached disks have limited size, the SIEM scales up by adding more nodes. When you search the SIEM, the work gets farmed out to each node that holds relevant data. Each node processes the search on its portion of the data and sends the results back¹.

This causes three issues at scale:

• Hot shards: Since nodes can't share compute resources, nodes holding frequently searched data become bottlenecks while the rest of the cluster sits underutilized. This increases the compute capacity required to provide a given level of performance.
• Slow scaling: Scaling up and down means splitting or combining nodes, which is a cumbersome operation. As a result, teams can't scale up effectively for unforeseen peaks, which reduces performance further, usually when you need it the most, such as during an incident.
• Expensive storage: Since nodes must use attached storage, the SIEM pays a premium over object storage (e.g., in AWS, EBS is ~7x more expensive than S3 per GB, assuming two replicas). This significantly increases the cost of storing data over long time periods.

As a result, legacy SIEMs are both compute and storage constrained. They can't scale compute effectively because of poor utilization, and they can't store data efficiently because it's tied to underutilized compute and kept on expensive storage.

No surprise that they try to keep your query volumes low and your retention periods short.

Row-oriented storage​

In many legacy SIEMs², each row is stored together on disk. This makes it efficient to write and update data, which is why most operational data stores store their data this way.

For logs, however, this causes two issues:

• Inefficient aggregation: In security logs, aggregations usually run over a few small columns, such as the timestamp, event name, or actor ID. Since these columns are a small fraction of the full event size, column-oriented stores can skip most of the information in each event. Row-oriented stores, on the other hand, must read the whole row. As a result, they process far more data per query, which makes queries slow and resource intensive.
• Poor compression: Security logs contain a wide range of data, from timestamps and numbers to strings and UUIDs. When these are stored as separate columns, they compress better, because compression algorithms can take advantage of patterns within each column. Combined with an inefficient indexing strategy (see below), the difference becomes meaningful: legacy SIEMs usually produce ingested logs (including indexes) that are between 0.5x and 2x the size of the raw data, while columnar stores usually land between 0.1x and 0.2x.

The issues caused by row-oriented storage compound those caused by storage-compute coupling. Not only are legacy SIEMs compute constrained, but their compute is also inefficient because it reads far too much data. Likewise, not only are legacy SIEMs storage constrained, but they also have to store a far larger share of the raw data because of inefficient compression and indexes.

Suboptimal indexing strategy​

SIEMs use indexes to accelerate searches for specific terms in your logs. Legacy SIEMs usually rely on one of two indexing strategies: inverted indexes (e.g., Lucene), which map terms in your logs to the rows that contain them, or probabilistic indexes (e.g., Bloom filters), which let you skip rows that are guaranteed not to contain a term.

While both techniques have valid use cases, there are two issues with how they are implemented in legacy SIEMs:

• Bloated inverted indexes: Inverted indexes offer a lot of precision, but they are expensive to create and take up a lot of space. That's usually a poor tradeoff for security logs, which are voluminous and read infrequently. Moreover, security searches rarely require the precision these indexes provide³. As a result, these indexes increase ingestion cost and storage overhead, which again reduces how much data you can afford to ingest and store. They also make it harder to cache the indexes efficiently, which increases query latency further, particularly over long time periods.
• Coarse probabilistic indexes: Probabilistic indexes can work well for SIEMs (we use them!). However, legacy SIEMs that rely on probabilistic indexes often use coarse implementations that limit their ability to accelerate your queries. For example, the filters may be limited to a few columns (e.g., just '_raw'), or only apply to exact matches in a column. As a result, common needle-in-a-haystack searches against terms such as an actor's first name or a host's prefix regularly get limited or no acceleration⁴.

Suboptimal indexing further compounds the issues caused by storage-compute coupling and row-oriented storage. Since the indexes are either too voluminous or too coarse, they put pressure on either the scarce storage or the scarce compute of legacy SIEMs.

What's the alternative?​

Few practitioners need convincing that their SIEM is slow and expensive at scale. But a reasonable question is: what's the alternative?

As an industry, we've resigned ourselves to treating SIEMs as a highly constrained, scarce resource where only the most important logs may enter. Otherwise, your budget blows up and your SIEM grinds to a halt.

However, as we've shown above, these constraints are the result of outdated technology, not the laws of physics!

A Modern SIEM, purpose-built for security logs at scale, can easily handle more than 10x the log and query volume of a legacy SIEM without increasing cost or complexity. By separating storage and compute, storing data as columns, and using an optimized indexing strategy (along with a number of other optimizations), modern log and query volumes become manageable within existing budgets and team capacity.

Don't settle for less.

FAQ​

Footnotes​

1. This is a simplified explanation, meant to capture the essence. For more details, check out the public documentation of your legacy SIEM of choice. LLMs are very good at this (hint, hint...) ↩

2. This is an area where SIEMs vary quite a bit. A few legacy SIEMs are also column-oriented. So depending on the specific SIEM you're comparing IceGuard to - this may or may not apply. ↩

3. Since inverted indexes contain exact positions, they can be used to effectively identify large intersections of multiple terms (e.g., a phrase) or substrings within a word. However, for most security use cases you're looking for one or more terms (e.g., a few IOCs or specific entities), not full phrases or substrings within a term. The overhead of storing the full index is rarely worth the acceleration for these rare cases. ↩

4. Indexing is a nuanced topic, and the implementation varies a lot per SIEM. We recommend benchmarking your specific query patterns at scale during a POC, and not relying on any vendor claims to assess performance (including ours!) ↩

Legacy SIEMs are due for retirement​

The world's most popular SIEM was built over 20 years ago. It hasn't changed much since.

Unfortunately, neither has the competition.

If you look at the Gartner Magic Quadrant for SIEM, every vendor present is either running on a variation of the same legacy infrastructure, or on a data warehouse that was never built for logs to begin with.

The rest of the world, however, has changed a lot in 20 years. And with GenAI adoption ramping, the rate of change will accelerate.

Four trends will define the future of the SIEM​

At IceGuard, we're betting that four trends will define the future of SIEM:

1. Log volumes will grow exponentially. Logs are generated by actions in your environment. Agents easily take >10x more actions per day than humans, and humans often run multiple agents in parallel, 24/7 in many cases. As a result, log volumes will grow by >10x as well.

2. Query volumes against the SIEM will grow exponentially. Queries used to require a human to formulate the question, craft the query, and interpret the results. With AI, hundreds of queries can be generated by a single prompt. This means the total query volume against your SIEM and other SecOps data stores will grow by multiple orders of magnitude.

3. Attacks will get more frequent, faster, and more creative. AI lowers the barriers for attackers to develop and launch sophisticated attacks against a large number of targets. Advanced techniques such as zero-day exploits, spear phishing, and EDR evasion are becoming commoditized, and attacks can complete in minutes.

4. Security logs will become a highly sensitive data asset. AI traces are one of the most important log sources for security teams to monitor. However, AI traces contain highly sensitive data such as your IP, business processes, credentials, and confidential activity of privileged employees. Unlike traditional audit logs, AI traces must be safeguarded like any other crown-jewel asset. This means they can't be spread across SaaS services and point solutions, but require centralized storage with governed access in your own environment.

If you agree with that list, it's clear that the legacy SIEMs are the wrong solution. They were already too slow and expensive for a pre-AI world.

In an AI world, they'll be obsolete.

It's time we raise the bar.​

Unlike what most security vendors will tell you, 10 TB/day is not big data. And it should not require a million-dollar SIEM budget or a decoupled architecture to handle efficiently.

In fact, we believe most security organizations should have the majority of their security data in a single, centralized data store. We call that data platform a Modern SIEM, and it needs to meet the following requirements:

1. Complete visibility. It provides unified, governed access to all your data, without relying on cold stores or federation. You can search months of data and get answers in seconds.

2. Continuous detection. It can run stateful detection logic in near real time, while easily incorporating behavioral baselines to minimize manual tuning. It can seamlessly aggregate, correlate, and enrich detection findings when they're created, so you don't need to spend your investigation time on ETL.

3. Unrivaled efficiency. It can ingest >10x the log volume and run >10x the query volume of your legacy SIEM without throttling or blowing up your budget.

4. Sovereign data. Your data remains in your account, stored in open formats. Your vendor does not have standing data access, and data can be retained indefinitely at object store rates.

Legacy SIEM architectures can't deliver any of those requirements. Neither can general-purpose data warehouses or a security data mesh!

We should stop accepting mediocre incumbents that have stopped innovating, and new entrants who reduce the scope and ambition of the SIEM.

Let's raise the bar!

FAQ​

Cost confusion​

There is a prevailing belief in the SecOps community that there is too much security data to store in one system.

We think that's wrong.

In fact, we think a single system is a far better place to store it than any form of decoupled architecture.

The confusion comes from SIEM incumbents overcharging and underdelivering, not from fundamental limits of technology. To understand that point, it's worth looking at some numbers.

On a modern system, ingesting, processing, and detecting against one raw GB of data should cost less than $0.02. Storing it for 90 days should cost less than $0.02 in an object store. A Modern SIEM ingesting 1 TB per day should therefore cost less than $15,000 per year in hardware. The vendor needs to make money too, of course. Assume an 8x markup over the hardware cost, totaling $0.30 per GB. That brings the annual cost to $120,000.¹

This is a small fraction of the economic activity required to produce those logs.

For example, an employee laptop produces 0.1 to 0.3 GB per day if you log aggressively. A production server may produce 100 times that. Now you're looking at a logging bill of less than $35 per laptop and less than $3,500 per production server per year.² Apply this math across your systems and the logging bill is usually a tiny fraction of the cost of the activity you're monitoring. With sensible logging practices, ingesting the logs into a reasonably priced SIEM should be a no-brainer for almost any organization that cares about security.

It's pricing, not physics​

So why are people complaining about SIEM pricing?

Because incumbent SIEMs aren't charging $0.30 per GB. They're charging between $3 and $6 per GB, a markup of more than 100x.³

No wonder users are struggling with the cost.

The problem is the pricing, not the physics.

Let's fix the root cause​

So what happens if we fix the ingestion price?

The workarounds disappear.

You no longer need a pipeline to route your data because all your logs go directly into the SIEM. Since the SIEM stores everything, you can ditch your cold store and federation engine as well. If your SIEM offers fully managed ingestion, which it should, the management overhead disappears too. Point your log source at the SIEM, or vice versa, and let the logs flow.

No need for additional complexity!

At current SIEM prices, it's understandable that users don't ingest everything. But that's because SIEM vendors are overcharging, not because there is too much data.

It's time we fixed the root cause and stopped peddling workarounds.

FAQ​

Footnotes​

1. Hardware: ($0.02 compute + $0.02 storage) × 1,000 GB × 365 days = $14,600 per year. With an 8x markup: $14,600 × 8 = $116,800, or about $0.32 per GB. We round to $0.30 per GB and $120,000 per year. ↩

2. Laptop: 0.3 GB × 365 days × $0.30 per GB = $32.85. Production server: $32.85 × 100 = $3,285. ↩

3. These prices apply to the 1 to 10 TB per day range. Above 10 TB per day, SIEM vendors might discount below these amounts. ↩

To decouple, or not to decouple...​

As legacy SIEMs continue to disappoint, the market remains on the hunt for options. One of the more popular ideas is the "decoupled SIEM." The idea is seductive: decompose the ingestion, storage, querying, and detection logic into separate systems, then purchase "best-of-breed" options in each category.

No more "vendor lock-in" or "ingestion tax," just the sweet sound of logs flowing efficiently through your decoupled system.

Unfortunately, reality is a lot more complicated.

In software engineering, coupling is the degree of interdependence between software modules. Decoupling - the act of reducing dependencies between systems - is often considered a good thing.

But it has a cost: In a decoupled system, you lose the ability to optimize the system as a whole.

This is why most high-performance, complex systems are not decoupled. Your laptop, phone, car, and relational database are all built using tightly integrated components.

We believe a SIEM should be built the same way.

The cost of decoupling​

The SIEM's purpose is to provide visibility and near real-time detection across your entire environment. This requires ingesting, normalizing, storing, detecting, and querying over massive amounts of data with minimal latency, effortless scalability, and at an affordable price. This is hard enough to accomplish in a single system. It's almost impossible in a decoupled one.

Let's dig into a few examples.

Detection latency​

Attacks are getting faster, and a 30-minute MTTD is no longer going to cut it. However, in a decoupled SIEM, it's hard to push detection latency and investigation time down. Each component will buffer and batch independently, quickly adding minutes to event arrival and detection time. Moreover, unless your detection engine supports efficient near real-time detection,¹ you'll be running your detections in 15-minute batches instead of near real time. Added up, you're usually 20 to 30 minutes behind on complex, stateful detections such as thresholds or sequences.

Investigation experience​

In a decoupled SIEM, your data is spread across systems: cold stores, warehouses, vendor APIs, and one or more SIEMs. Analysts find themselves jumping between tabs, APIs, and search languages. Federated search can help, but it is far too slow for interactive use and rarely scales beyond POCs and simple use cases. If you don't believe us, run a load test yourself.

As a result, investigations are slow and tedious.

Operational complexity​

Each solution must be bought, set up, tuned, and governed separately. This means multiple access mechanisms, permission systems, monitoring systems, query languages, detection frameworks, APIs, support teams, and procurement cycles.

If something breaks, good luck.

The integration between systems becomes its own ongoing project. Schemas, enrichment, detection logic, indexing strategy, retention, and access mechanisms need to be coordinated across each component. Since a simple change in one part of the chain can have far-reaching side effects in downstream systems, you need comprehensive end-to-end testing to make changes safely.

We could go on and on.

Let's keep it coupled​

The key insight is that the quality of the user experience is directly related to the degree of coupling between each component. To achieve low-latency detection, fast search, and minimal operational overhead at an affordable cost, each component must be designed to work together as a system.

Decoupling makes that harder.

The call for a decoupled SIEM is a rational response to the poor technical execution and exorbitant pricing of SIEM incumbents. But that doesn't make it the right long-term architecture for the SIEM.

Let's keep the SIEM coupled.

FAQ​

Footnotes​

1. Note that some pipeline vendors do "detection in the stream." Assuming that works (you should test that thoroughly), the detection capabilities are no longer "decoupled." Good engineering decision, but awkward for their product marketing teams. ↩

Unloved, but not unnecessary​

Few categories carry as many negative connotations as the SIEM: slow, expensive, hard to operate, and painful to migrate from. The exorbitant pricing has even become a meme.

Unsurprisingly, the startup world has been mulling over what's next: a fabric? A mesh? An agentic cyber defense platform?

Word salads aside, the problem is real. You need full visibility into your environment, and legacy SIEMs are too slow and expensive to deliver. No wonder you're shopping around for options.

Our hot take: you don't want a mesh, fabric, or agentic defense platform. You want a SIEM.

Why SIEMs matter​

The core problem you need to solve is cross-source visibility and near real-time detection. You have tens of thousands of identities, assets, and applications producing billions of events daily. On top of that sit 10-50 security point solutions that monitor slices of that activity. Detections, investigations, and reporting frequently need to span multiple sources, usually under time pressure. Without a unified access mechanism coupled with low-latency detection capabilities, this is painful and time-consuming. At AI scale, it becomes impossible.

But cross-source visibility and near real-time detection are exactly the problems SIEMs were meant to solve.

Why do we need to invent a new category?

Why people don't like their SIEM​

The underlying issue is that the SIEM incumbents have been unable to deliver on the promise of SIEM.

And they don't have the capabilities or incentives to improve.

On the capability side, their core technology is so old and bloated that they cannot make meaningful improvements to their platforms' performance, scalability, or efficiency.

On the incentive side, they are making bank by keeping prices high and ingest volumes low. As long as their customers keep ingesting more data and remain unable to migrate, why change?

The SIEM is not dead!​

AI adoption is making SIEMs more important than ever: more systems to monitor, higher activity volumes, and faster attacks. Full visibility and near real-time detection across the entire environment is becoming table stakes for a modern SOC. Forget the mesh - only a SIEM can deliver that.

The SIEM is not dead. We just need better SIEMs.

FAQ​