To decouple, or not to decouple...
As legacy SIEMs continue to disappoint, the market remains on the hunt for options. One of the more popular ideas is the "decoupled SIEM." The idea is seductive: decompose the ingestion, storage, querying, and detection logic into separate systems, then purchase "best-of-breed" options in each category.
No more "vendor lock-in" or "ingestion tax," just the sweet sound of logs flowing efficiently through your decoupled system.
Unfortunately, reality is a lot more complicated.
In software engineering, coupling is the degree of interdependence between software modules. Decoupling - the act of reducing dependencies between systems - is often considered a good thing.
But it has a cost: In a decoupled system, you lose the ability to optimize the system as a whole.
This is why most high-performance, complex systems are not decoupled. Your laptop, phone, car, and relational database are all built using tightly integrated components.
We believe a SIEM should be built the same way.
The cost of decoupling
The SIEM's purpose is to provide visibility and near real-time detection across your entire environment. This requires ingesting, normalizing, storing, detecting, and querying over massive amounts of data with minimal latency, effortless scalability, and at an affordable price. This is hard enough to accomplish in a single system. It's almost impossible in a decoupled one.
Let's dig into a few examples.
Detection latency
Attacks are getting faster, and a 30-minute MTTD is no longer going to cut it. However, in a decoupled SIEM, it's hard to push detection latency and investigation time down. Each component will buffer and batch independently, quickly adding minutes to event arrival and detection time. Moreover, unless your detection engine supports efficient near real-time detection,1 you'll be running your detections in 15-minute batches instead of near real time. Added up, you're usually 20 to 30 minutes behind on complex, stateful detections such as thresholds or sequences.
Investigation experience
In a decoupled SIEM, your data is spread across systems: cold stores, warehouses, vendor APIs, and one or more SIEMs. Analysts find themselves jumping between tabs, APIs, and search languages. Federated search can help, but it is far too slow for interactive use and rarely scales beyond POCs and simple use cases. If you don't believe us, run a load test yourself.
As a result, investigations are slow and tedious.
Operational complexity
Each solution must be bought, set up, tuned, and governed separately. This means multiple access mechanisms, permission systems, monitoring systems, query languages, detection frameworks, APIs, support teams, and procurement cycles.
If something breaks, good luck.
The integration between systems becomes its own ongoing project. Schemas, enrichment, detection logic, indexing strategy, retention, and access mechanisms need to be coordinated across each component. Since a simple change in one part of the chain can have far-reaching side effects in downstream systems, you need comprehensive end-to-end testing to make changes safely.
We could go on and on.
Let's keep it coupled
The key insight is that the quality of the user experience is directly related to the degree of coupling between each component. To achieve low-latency detection, fast search, and minimal operational overhead at an affordable cost, each component must be designed to work together as a system.
Decoupling makes that harder.
The call for a decoupled SIEM is a rational response to the poor technical execution and exorbitant pricing of SIEM incumbents. But that doesn't make it the right long-term architecture for the SIEM.
Let's keep the SIEM coupled.
FAQ
Why do security data pipelines add complexity?
Security data pipelines add an extra - and we argue, unnecessary - step between your SIEM and the data. This introduces complexity by forcing you to configure and manage an additional product and its integrations. You need to learn another set of APIs, consume another set of operational telemetry, and debug across another system when things break.
The biggest challenge, however, is that the pipeline encourages spreading your data across systems. That's its main selling point. As a result, finding, correlating, and detecting against your data becomes more challenging because you have to look across multiple locations to find it.
Why is federated search slow and hard to scale?
There are three ways to make search fast: indexing, caching, and parallel processing. Federated search does not support any of the three properly.
On the indexing side, you can't index data if you don't read it and have the ability to access it granularly. This might be obvious to most readers, but it's worth stating explicitly. You can't index a third-party data store or vendor API without reading and storing all the data. This means the only systems you can index are file systems or object stores, and only after you've read them. At this point you're describing a data platform, not a federated system, because you're paying the cost of moving and reading the data during indexing.
On the caching side, you can't cache a federated system effectively because you don't know when to invalidate the cache. Security logs regularly get late-arriving events. If those arrive in a third-party system, you have no way of knowing your cache is invalid. Moreover, caching assumes everyone is hitting the same data regularly. In security operations, most investigations cover new scope. It's rare that you'll investigate the same environments and time periods repeatedly. As a result, your cache hit rate will be very low unless you actively pre-populate the cache, which again makes this a data platform and not a federation system.
On the parallel-processing side, you're bottlenecked by the source capacity. Vendor APIs and third-party data stores are not built for massive parallel processing. Nor do they want to pay the compute bill for your large, concurrent queries against their API. As a result, you'll get throttled. When you combine this with the inability to cache and index, you get a "lethal trifecta" of poor query performance: you have to scan all the data because you don't have an index, read from a third-party system because you don't have a cache, and read in serial because you can't process the data in parallel.
There is a lot more to say about this topic. We built and sold federation use cases at Starburst for a long time and know the pros and cons as well as anyone. We go deeper in a separate post.
For now, if your vendor claims its federated search is fast at scale, do a load test. Run multiple concurrent "needle-in-a-haystack" queries over long time periods against the same third-party APIs. Don't fall for a pretty demo or believe that a small-scale POC is representative of a production workload. Federation demos well, but fails at scale.
Footnotes
-
Note that some pipeline vendors do "detection in the stream." Assuming that works (you should test that thoroughly), the detection capabilities are no longer "decoupled." Good engineering decision, but awkward for their product marketing teams. ↩
