PEBKAC Issues Plague SIEM Deployments

This morning I observed an article related to SIEM via my LinkedIn feed, and I was anxious to read it based on the title: “SIEMs sometimes suck.” As a security professional that straddles both sides of the fence between penetration test, and the DFIR realms (Digital Forensics Incident Response for those unfamiliar with the acronym), I deal with SIEM on a daily basis. Unfortunately, the article left me disappointed which is the basis for the name of this article: “PEBKAC Issues Plague SIEM Deployments.” Because technology has created an “acronym’d” society, PEBKAC stands for Problem Exists Between Keyboard And Chair. The referenced article is filled with issue the author gripes about, that have little to do with SIEM, but more to do with the SIEM architect, and analysts tasked with configuring SIEM.

Before I get in depth, just like the author I too have been on SIEM systems for time. Like the author I use Arcsight, but on a daily basis it is not uncommon for me to jump on QRadar, Alienvault, LogRhythm, SumoLogic, and there have been others. The named SIEM appliances and offering are ONLY products I currently have a handle of, and perform DFIR work, as well as provide managed SIEM services for many clients. With that said, I will jump into the author’s gripes, and my take on his gripes.

Author’s Gripe 1:

“So, the real sucky part of your SIEM strategy is the “hope” that you are developing rules that actually work. The rules must be validated and more to the point those rules must be validated based on your specific security infrastructure blocking, detecting and allowing real malicious traffic.”

All SIEM appliances I have used have hundreds, if not thousands of pre-defined rules to detect threats, and most if not all, will alert on anomalous patterns. One of the PEBKAC problems I see with this statement is that there is a disconnect between the author perceiving SIEM should see everything, and react. Outside of the pre-canned rules that come with most SIEM, it is up to the individuals deploying SIEM to fine tune their SIEM appliances. “How do you eat an elephant? One bite at a time.” When I architect a SIEM deployment, I try to isolate as much as possible to create the most granular rules as I possibly can. Networks are broken into function (server network, desktops, routers, office locations), groups of people are created (CxO levels are immediately placed into high risk, HR, devops, and so forth). Doing so allows me to created specifics, but can only work if I devised a strong design, and properly deployed it. For example, if I know the CEO works between 8am and 6PM, never accesses a server, and is always working out of the NYC office, in say QRadar, I could easily create a rule with the following logic: Create event when CEO login AFTER 6pm and office is NOT NYC (the rule by the way would be created as an event first, and the option to create an offense selected later). This tells QRadar to do something when it sees the CEO logging in after hours from outside of his normal office. 

As stated, most if not all SIEM appliances come with pre-defined rules that work, but SIEM revolves around a lot of fine tuning and there is no way for a vendor to create a “catch all solution,” as all businesses have different needs. What one business may see as a need, another business won’t. Creating millions of rules will create millions of false positives, and many of us have seen what happened when target was breached. SIEM was turned off because it was too noisy. QRadar and others have options to minimize false positives with the click of a button. As for “detecting and allowing real malicious traffic” this too is a PEBKAC problem due to poor network designs. Putting on my penetration testing hat, I cannot tell you how many networks had no segregation between production, and development networks, not to mention proper segregation of roles. Segregation, and segmentation are not the same. Just because a network engineer added another CIDR range, means you have another network address block. If the network engineer didn’t implement the proper ACLs, in environments where I have performed penetration tests, it is not uncommon for those in the CxO network, to access workstations, and desktops, and vice verse. Do you really feel that the mailroom systems should be able to ping, RDP, access anything on the CxO subnet? Do you even have your network segregated to that scale?

Author’s Gripe 2:

“Rules that work on a SIEM for company A might not work for company B even if you are using the same SIEM with the same firewalls, IPS and related security solutions because of configuration differences.”

This was explained in my paragraph above, there is no catch all. But let me refine my answer, in the event the author meant: SIEM rules in my company work at OFFICE A but NOT at OFFICE B. This is not a SIEM issue but a PEBKAC issue due to configuration differences not on the SIEM, but on the firewall appliances, IPS solutions, and other devices. A good SIEM offering is able to take log formats from a variety of sources. Referencing QRadar again since it happens to be my favorite SIEM appliance, QRadar offers hundreds of “DSMs” (device support modules). DSMs are modules that IBM created that take a log format, and make it readable, and “actionable” within QRadar. This is outside of universal standards such as SYSLOG, and or LEEF. So if there is an issue with SIEM working in ONE office, but NOT the other, the problem is not with the SIEM appliance.

Author’s Gripe 3:

“Further, simulated attack recreations, where the attacks aren’t the real thing, provide little value beyond illustrating how your security infrastructure might respond to a simulated attack recreation as opposed to a real attack, as simulations can miss key attack functionality.”

My answer will come from both sides of the spectrum from a penetration tester, and DFIR point of view, but also as that of a “logical architect.” Penetration tester: “If you think you know my next hack you don’t know jack.” This is what a T-Shirt I received at Blackhat 2014 said, and I will get back to this. DFIR: Should I spend time looking for IOCs (indicators of compromise) amongst the recurring 10Gbps of traffic? Logical architect: “I designed this system to ONLY display a webpage. I need to see anything acting outside of those parameters.” Which would you think is a better task to implement? If you answered from a “logical architect” perspective, pat yourself on the back.

As system/network/infrastructure engineers, we are best suited to understand what systems are on the network for, who is supposed to access these systems, and why. For those that can’t understand this, I recommend reading Priscilla Oppenheimer’s “Top Down Network Design.” Simulating attack will yield you useless data most times as attacks are in a constant state of change. So if you think simulating an attack will do something worthwhile, then you don’t understand how attacks work. “If you think you know my next hack you don’t know jack.” When I perform penetration testing, there are a multitude of ways I can move around the network without being detected. If I managed to get credentials, your SIEM can become useless because you haven’t engineered it properly (PEBKAC). For example, as a logical architect, I can always state: Username JDoe should NEVER access NETWORKS_A_thru_Z via Network_1. A proper rule to detect this would have to be designed with isolated networks, usernames, and other building blocks to make it work.

For example, I could created a reference set named NormalEmployees, a network object called Network1, and then create a rule that states: “Show me any events where anyone within the NormalEmployees tries to access anything outside of Network1.” In order to understand why this rule makes more sense than simulating an attack, one needs to put on their logic hat: “I have sensitive systems that need to be used in this fashion, by these people, from these location, at these times.” Everything else becomes suspicious. You know how, and why you designed these systems, put it to use.

The rest of the referenced article was filled with more of the same misguided attributions:

Author - Is my SIEM getting enough useful data? | PEBKAC - What are you sending to your SIEM? Unless you do span port mirroring, a SIEM only gets what you send it.

Author - Is my data coming at a timely cadence? | PEBKAC - Did you design your network to be robust enough to handle traffic?

Author - Is my data formatted correctly so it can be processed? | PEBKAC - Did you properly analyze a solution to fit your needs before buying it. Can you NOT convert to standards (syslog)?

Author - Are my timestamps accurate? | PEBKAC - Did you configure NTP in your organization?

Author - Are my sources generating accurate logs, alerts? | PEBKAC - If you don’t know what your systems are doing, and how, I don’t see how this could be shifted to being a SIEM problem.

Wearing my penetration testing hat now, if you are not using SIEM you are failing. However, if you are not properly designing and fine tuning your SIEM deployment based on your company’s needs, it is not your SIEM that sucks, it is the person tasked with managing, deploying, using SIEM that is the problem. PEBKAC.