Welcome back, EDR!

June 17, 2017 in EDR

Update

Added one more bullet point – AV vs EDR – testing on live systems

Old post

I heavily criticized EDR in my last post (and in fairness, I did a few times before – both threat hunting, and EDR), so I thought I will list some ideas I have with regards to assessing this tool in a more pragmatic way. Whining is good, as long as it produces some solutions… I could call that previous post a classical sh1tpost and it kinda fits as it served the purpose. It did stir some controversy and if you reflected on EDR in general or got emotional about my ramblings in any way,  thank you.

I think the first question we need to answer is the expectations we have towards the EDR tools. That’s because mine were too high.

In my view, the EDR should fill-in the gap between the AV and the DFIR process, for both detection and remediation. The gap for detection is to cover the malicious activity that AV and other controls can’t detect. In my opinion the core value of the detective capability of EDR is supposed to be ‘seeing more’ than AV, and logging it + passing it to the external storage (for backup, and analysis).

The remediation gap is focusing on the access to the system to contain the malicious activity remotely without a need to engage local helpdesks, security officers, and taking computers away from the users. I have done a lot of it using psexec, or onsite and it’s fun for a while, but getting old pretty soon.

And if apart from detection and remediation the EDR offers some prevention capabilities (f.ex. blocking applications from running like f.ex. Bit9), it’s even better.

The functionality of EDR that is scattered among products varies and a while ago I made an attempt to classify and describe these features in the EDR sheet based on the astonishingly good feedback from the community. I think this sort of cooperation works for the benefit of us all – the defenders. Red teams have it relatively easy, for us it’s a hamster wheel, whack-a-mole and omni-present impostor syndrome that comes as a result of NOT knowing it all… 🙂

I think my main criticism of EDR comes directly from the experience of having to comb a huge number of events. As someone told me – the EDR and threat hunting suffers from the big data problem. Solving it in a typical, signature-based way is wrong. I tried, and I feel that I failed. I know other succeeded. Some readers shared a good experience with EDR and they are pretty comfy using it on daily basis. What is interesting though, for these that indicate it works for them – it’s often smaller shops. Perhaps they have more influence, perhaps they can customize things betters and don’t have to deal with the political, technical, legal and historical issues deeply embedded inside the large companies… Who knows… I am actually a big fan of customized tools and it’s great to see people use the principles of DFIR to cover the company with the solid threat detection umbrella. Interestingly, they often rely on free tools 100%.

With regards to data combing. This activity raises a interesting question – is the burden of tuning of the EDR on vendor, or the users?

This is not a trivial question. There could be easily hundreds queries that you can incorporate into the EDR, and every day there is a new discovery, bypass, or trick that you need to add to your arsenal. Some may be VERY specific to your environment. F.ex. you could do some nice hunting for processes spawn by server programs, files dropped in the product-specific folders (file integrity ‘light’) and significantly improve the ability to detect important anomalies by defining exclusions based on the setup of some services, your baselines, simple rules that rely on host naming conventions, or even internals source IPs used by the vulnerability scanners and approved pentesters. Still, the larger the environment, the larger the volume of events.

So, the answer is probably this: both. With so much research going on, I’d like EDR to constantly update the ruleset to offer ‘the out of the box’ experience.

As for the external threat hunting ideas – it’s actually tough to keep up. Just look at Twitter. There is a never ending stream of new, or well-documented ideas. They are constantly re-visited and improved. Still, this is sort of feasible and eventually you will end up with a decent set of pretty robust event generators.  And this is what they are – the event generators. And lo, and behold – these logs require incident spotters. And these can’t be L1 guys. You need a trained eye of someone who actually did breach analysis in the past. You need DFIR professionals. That’s probably a good business case to bring more very well trained people on board. You should have at least one person like this to train others, and assess more difficult cases. We are not talking about IOC collectors. We are talking about people who look at all pieces of the puzzle distributed amongst various systems.

Since I mentioned bridging the gap between AV and DFIR I want to dig deeper into it. In a typical DFIR process (the more modern one) you would like to explore memory artifacts, list of processes, handles, network connections, etc. and if possible, browse through the historical events that led to certain things happening on the system. I think the EDR that provides this functionality is a fantastic discovery tool as it perfectly supports the triage process – the triage process that until few years ago had to be either done manually, or with a help of heavy-duty tools, and sometimes even visit on site. So, giving the ability to retrieve the artifacts remotely in real-time is a great feature. As usual, there are various ways to deliver this functionality. Snapshots, dynamically refreshed list of ‘current’ view of the system, full trace of system events for a while, with the memory and network snapshots included.

With regards to alerting, watchlists, triggers, detectors, or whatever else they are called – this is the area that probably frustrates me the most. I claimed and I maintain the opinion that threat hunting is still focused mainly on writing signatures (even if they are based on correlations, they are still a simple logical statements). They do not differ from IDS, AV, and yara. They may be focused on a narrower and more subtle subset of values, but it’s still what it is. Implementing every single rule gives you two choices – strict, very specific, or wide, to cover all angles. The first one may not trigger often and is kinda perfect (f.ex. using unusual persistent key is a very good activity to monitor) – here comes the kowtow to SwiftOnSecurity and his/her sysmon config. The wider rules need eyeballing, or some sort of data reduction.Again, this needs a pair of trained eyes.

With regards to feeds. Various people and companies approach it in a different way and in my experience, and kinda ironically, the best results come from the … VT feeds. Maybe this is the reason why so many AV companies got pissed off that their work is leveraged by the ‘Next Gen’ solutions and threatened to pull out of VT a while ago. If we exclude VT, we have to do something different. Data stacking comes to help, various blacklists and IOCs too. Then comes machine learning and some more fancy AI. At least these are the claims. I honestly don’t know how many of these internals work as real ML/AI, and what artifacts or artifact properties they focus on. I have been in this industry for long enough and clustered enough samples by both static and dynamic analysis to know that it’s non-trivial. Very non-trivial. If you look at the old Upatre, it implements a perfect windows message loop and the malicious activity happens in a window procedure. How would you detect that it behaves in a different way than a normal say notepad++ editor, or process hacker (both GUI, both connect out). You could do some analysis of GUI elements, and perhaps check if the user even saw the window (was it even visible, or was there any mouse activity within its client area, any keys inputted, etc.), but this is where we enter the area of behavior analysis (and new line of products that are UBA/UEBA). Something that AV does for a long time… And it still doesn’t produce such great results. Many products use dirty tricks (AV included) f.ex. installers that inject code to explorer.exe to self-delete (for legitimate purposes). These are your FPs. I have seen many.

What is the game changer then?

On a high-level, EDR logs are like Windows Event logs. They are more granular, but they lack the context. Perhaps there is a need to ‘tag’ events that are result of user activity, or via admin scripts. Maybe there is a need to tag installation of plugins, software, even downloads of software. So that a better timeline can be built. Not one that shows things happened, but HOW they happened and are correlated. Was the file executed via explorer.exe by the user? or it’s a code injected by malware or hooking DLL? Or the instrumented GUI used by the UI testing software? Again, it’s not trivial. Perhaps the whole thing will disappear when Windows moves to Apple-like ecosystem. There are more and more signs of it…

I won’t cover remediation. It’s a big topic and we don’t even need to bother talking about it until we have solid detection in place…

So… how can we assess if EDR delivers?

I think we can analyze various parameters. None of them related to core functionality presented in the EDR sheet. It’s more or less the sort of data you pull from various security controls on regular basis. So, not a rocket science here.

  • AV vs EDR – this is easy; install Av, install EDR, run a number of representative samples and see what happens; compare
    • I would not recommend it
    • Remember, EDR is a complementary control and NOT a replacement for AV; we are not there yet
    • It doesn’t really answer any questions; one more time: the equation is that AV+EDR>AV
    • As such, I would never suggest anyone to uninstall AV even if EDR detected 99% more samples. Again, we are not there yet.
  • AV vs EDR – testing on live systems
    • This is a possibly best way to see if EDR vendors claims are justified
    • Provided that you have both AV and EDR on all the systems, start looking at all these alerts from AV – for every single alert see if you get any hits from EDR; this is not only to scrutinize the EDR vendor (or AV, for that matter), but also to come up with new ideas for threat hunting
    • And in case you are wondering… there is a huge area of malicious cases that EDR will most likely never see, or will find it non-trivial to detect:
      • static, never executed in a ‘normal’ way files, but still malicious f.ex. whatever you can find on removable drives
      • same as above, bit on the server-side – f.ex. web shells
      • overwritten / patched files
      • malware inside ‘dead’ folders f.ex. WINDOWS.OLD
      • malware inside backups
      • malware inside containers (f.ex. OLE files, .msg files, rar, cab, arj, etc.)
      • computer viruses
  • EDR Performance hit
    • CPU usage
    • Memory usage (idle and working)
    • Size of the agent (in memory)
    • Number of process spawn (this end sup in logs + contaminates the evidence!)
    • Number of network connection and frequency
    • Traffic volumes (see below for ‘EDR Privacy and telemetry’)
  • EDR Event generation
    • How many devices covered?
    • Is it possible to tell if agent is installed/up and running
    • How many queries/detectors ‘out of the box’
    • How many events generated? Total? Per system / average /?
    • How many of them got promoted to incidents
    • Do you see all events needed? /this requires a separate post/
    • How many of these incidents were detected by EDR itself? By the external feed? By internal logic/ML/AI?
    • How many human resources required to eyeball the logs?
  • EDR Vulnerabilities
    • Are there any? How fast they are fixed?
    • IMHO these are completely uncharted waters with some minor research out there; additional attack surface + often new products written in a hurry ==> potentials for lots of vulns
  • EDR Support
    • I think this is big part of the assessment
    • Do you have a direct access to actual engineers, or just inhuman ticketing system? They can often answer the most difficult questions quickly and hey… they speak the same language w/o marketing buzz
    • Do they actually reply to your queries with a sincerity? You can quickly tell
    • Are they a new startup (and you are a guinea pig for their software); not necessarily bad, but may bring issues
  • EDR Privacy and telemetry
    • What sort of data does it sends to ‘the cloud’?
    • Does it support Data Across Borders and other global/local regulations?
    • Are they using artifacts from your system to build their own repo of artifacts? Not necessarily a bad thing, but the cost is possible data leakage, legal issues and extensive traffic (we are talking large volumes for some EDR)
  • EDR Company’s future
    • This is the last bullet point for a reason – we are already in a middle of large transition to commoditize security (like any other services before)
    • Most of the companies want to and will be acquired
    • If there are any signs of possible acquisition, perhaps better to wait a bit to let the dust settle before engaging them
    • This is because acquisition has both pros and cons. Pros: the service won’t disappear overnight; Cons: it will get worse, most of the time and many ‘good’ guys will move to work elsewhere; this is the state of the affairs in security companies

That’s it. If you have any comments, or feedback please let me know. I was wrong many times before, and writing this stuff down it helps me to revisit my bias and ideas. I want EDR to work for me on the same level like AV. At least.

Bye Bye EDR…

June 13, 2017 in EDR

Update

If you had read the below post and it made you blood boil, mission accomplished 😉 Just kidding. The idea of the post was and still is this – to raise the question: what is the value of EDR and how to measure it? I am assuming the value is there, but it’s lower than expected. I was myself quite shocked when I had to come to terms with this observation.

And just in case – this was a vendor-independent statement. I woke up one morning, and realized that I have expectations towards EDR and we are pretty far today from these expectations to be met.

So, while shots were fired, the question remains: how do we assess the value of EDR?

For comparison: AV is looked at in a very harsh way. The expectation is to cover 100/100.

Can we apply the same assessment criteria to EDR?

I’d love to see high-fidelity alerts only from EDR. But I know it’s impossible.

What’s the acceptable ratio then? 30%?

This is a valid question, because in a large organization you need resources to eyeball all these mail alerts and  dashboards. And expectations should be set for the analysts who process this data.

And if the ratio of the TP is low, you end up looking at events most of the day, and all day long and hey… isn’t the whole idea of tools like EDR to recognize the incidents in the flood of events… ?

Old post

I officially lost faith in EDR.

I know that for some it works. It does work for me too. Sometimes. But… more often that that – it doesn’t.

I am done with the EDR.

Its alerting is following the paths of firewall, IDS, DLP, windows event logs, and… yara rules. Lots of alerts, poor ROI.

Arrivederci EDR.

AV beats EDR in so many aspects. It’s high fidelity. It’s precise. Its heuritics feeds are often better. It also offers remediation and it is undo-able. It even prevents bad stuff from running.

EDR is kaputt.

It assigns value to so-much-hated signatures. The form has changed, but the result is the same. You only see what you can write a rule for. And then you see it, and it’s surrounded by so many FPs. You probably suck at writing rules. Except many are not written by you and come from the other threat hunting experts.

EDR is a waste of money, and time. Now, there you have it – I’ve said it.

EDR was a nice utopia. I believed it for a while. In a perfect world you can gather all these logs and gather intel about activities on the system, of the user, network events, and then profile various scenarios and look at the dashboard happily cherry-picking the bad guys. A nice placebo in the absence of better solutions, despite them being available for a long time (f.ex. whitelisting).

But in reality you face the alarm fatigue, and suffer from infobesity.

EDR stands for Endpoint Detection and Response. It’s neither Detect nor Response. But in my books, it’s defo at the end point.