Home Enterprise Tech It’s time for security teams to embrace security data lakes

It’s time for security teams to embrace security data lakes

It’s time for security teams to embrace security data lakes

Dan Schoenbaum

Early Newspaper

Dan Schoenbaum is a two-time CEO and a two-time COO in cybersecurity. Today, he is a managing companion at High Tide Advisors, a boutique consulting agency helping corporations prevail in higher success through go-to-market recommendations and execution.

The moderate corporate security group spends $18 million yearly nonetheless is essentially ineffective at combating breaches, IP theft and data loss. Why? The fragmented map we’re currently the utilization of in the security operations heart (SOC) does no longer work.

Right here’s a like a flash refresher on security operations and the map in which we bought where we are today: A decade prior to now, we stable our purposes and net sites by monitoring match logs — digital data of every process that came about in our cyber surroundings, ranging from logins to emails to configuration adjustments. Logs had been audited, flags had been raised, suspicious actions had been investigated, and data used to be stored for compliance capabilities.

The security-driven data stored in a data lake will more than in all probability be in its native format, structured or unstructured, and therefore dimensional, dynamic and heterogeneous, which offers data lakes their distinction and advantage over data warehouses.

As malicious actors and adversaries changed into extra energetic, and their tactics, tactics and procedures (or TTP’s, in security parlance) grew extra subtle, straightforward logging evolved into an map known as “security information and match administration” (SIEM), which comprises the utilization of tool to provide trusty-time evaluation of security alerts generated by purposes and community hardware. SIEM tool uses rule-driven correlation and analytics to flip raw match data into doubtlessly treasured intelligence.

Despite the indisputable truth that it used to be no magic bullet (it’s hard to enforce and create the total lot work successfully), the ability to net the so-known as “needle in the haystack” and name attacks in growth used to be a gargantuan step forward.

Today, SIEMs silent exist, and the market is essentially led by Splunk and IBM QRadar. For sure, the expertise has evolved a good deal because original use cases emerge continually. Many corporations gain sooner or later moved into cloud-native deployments and are leveraging machine discovering out and subtle behavioral analytics. Then again, original endeavor SIEM deployments are fewer, costs are higher, and — most importantly — the overall needs of the CISO and the exhausting-working team in the SOC gain changed.

Unique security calls for are asking too necessary of SIEM

First, data has exploded and SIEM is too narrowly centered. The mere assortment of security occasions is never any longer sufficient since the aperture on this dataset is too narrow. While there is in all probability a extensive quantity of match data to net and course of from your occasions, you are missing out on mountainous quantities of extra information such as OSINT (start-supply intelligence information), consumable exterior-risk feeds, and treasured information such as malware and IP reputation databases, as neatly as reports from sunless net process. There are never-ending sources of intelligence, a long way too many for the dated structure of a SIEM.

Moreover, data exploded alongside costs. Data explosion + hardware + license costs = spiraling total tag of ownership. With so necessary infrastructure, each and every bodily and virtual, the quantity of information being captured has exploded. Machine-generated data has grown at 50x, whereas the typical security funds grows 14% year on year.

The tag to store all of this information makes the SIEM tag-prohibitive. The moderate tag of a SIEM has skyrocketed to shut to $1 million yearly, which is most efficient for license and hardware costs. The economics force teams in the SOC to net and/or preserve less information in an strive to abet costs in check. This causes the effectiveness of the SIEM to change into even additional diminished. I currently spoke with a SOC team who wished to are looking ahead to huge datasets wanting for evidence of fraud, nonetheless doing so in Splunk used to be tag-prohibitive and a gradual, exhausting course of, main the team to detect conceivable picks.

The shortcomings of the SIEM map today are unpleasant and unsightly. A fresh inquire of by the Ponemon Institute surveyed nearly 600 IT security leaders and came upon that, despite spending a mean of $18.4 million yearly and the utilization of a mean of 47 merchandise, a whopping 53% of IT security leaders “did no longer know if their merchandise had been even working.” It’s clearly time for alternate.

It’s time for security teams to embrace security data lakes