I will start by saying that I am a devoted advocate for threat hunting and I consider it a must-do part of any Incident Response program.
However…
It’s a very difficult task, and one that cannot be solved by a list of Event IDs, list of names for known bad programs, dual-purpose programs, OS commands abused by attackers, or clever regex kung-fu. I believe that the core of threat hunting relies on data reduction, and this often ends up with False Positive management. A really, really tough task. Unfortunately, most of the threat hunting ‘recipes’ that can be found online covering this topic are often focused on the True Positives; the presentations show screenshots confirming that certain events can highlight the execution of malware, etc.. Yes, this works fine. But it’s an error of availability. This works fine for a single test system, but it is much harder on a real system and completely non-scalable for a large number of systems.
Let me illustrate it by some simple math – the number of processes started logged by the Windows Event log on my lab workstation is easily 300 per day or more (sometimes it’s a few thousands if I do some batch file processing). This often escapes our consciousness, but cmd, strings, grep, tools of any sort, it all leaves a mark in the log. Let’s say the average system has 150 processes executed every day. Multiply it by 365 days, that’s 50K+ events. Multiply it by 50K devices and you get 2,737,500,000 events. And this is just process creation. A decent EDR will log threads’ creation, writing to other processes’ memory, file creation, registry modifications, account creation, escalations of privileges, and so on and so forth. It’s a HUGE number of events. It has a tremendous value in terms of analysis and works very well as a post-mortem detective control, but it’s not necessarily designed to prevent/detect early the badness… We can do some data reduction using data stacking, etc., but even after that the variety of programs that you will see in these logs makes the task very very difficult. This gets especially hard in the environment where you have a large group of developers – these guys run crazy number of processes, experiment with weird software on their systems (they also _write_ programs and execute them for testing), and contribute a lot to the noise in the logs. And mind you, we would like to discover the insider threat as well…
In general, I divide threat hunting rules into two categories:
- based on IOCs
- this is very easy to implement
- examples
- unique_file_name.exe is bad – we can query systems for it on regular basis and if it is detected we trigger the alert
- when new file extension used by ransomware becomes publicly known, we can query the logs for file creation events that hit on this new file extension
- the ROI is poor though, because it’s a signature-based approach
- … and yeah, unfortunately looking for specific process names often leads to False Positives as well…
- based on assumptions and scenarios that attempt to detect stages of malicious/hacking activity (aiming at disturbing the kill chain)
- this is difficult as the expected results are not known and lots of tuning is required to reduce the data returned by the query
- the following rule sounds like a good example of this idea: ‘svchost.exe executed from a non-sysdir location’
- I think this rule is extremely powerful, cuz it will catch any malware that drops and executes svchost.exe f.ex. from %appdata%
- there is only one problem… it is a signature-based approach! and while it highlights an anomaly, it only covers a small part of the spectrum…
- hold on… so what could be a real example of rule belonging to this category? what about ‘review all executable files dropped in the %appdata%’?
- this is better, but how do you distinguish which dropped file is executable? it’s difficult even if we talk about file extensions only, as one could use ‘.exe’, ‘.dll’, ‘.scr’, ‘.pif’, but also ‘.foo’, ‘.dat’, and ‘.rar’, or ‘.zip’ – as long as one can load the file and execute it, the file extension doesn’t matter; and let’s not forget about the file content (PE file, scripting languages, macros) and… last but not least – yup, this is yet another signature-based rule!
- okay, so what about this ‘look at all files dropped inside %appdata%’…
- yeah, right…
- hmm what about hackers using ‘ipconfig.exe’, ‘netstat.exe’, ‘net.exe’, ‘cscript.exe’, etc.? or even ‘psexec.exe’, ‘nmap.exe’ ?
- good luck with that…
- obviously, it’s signature-based approach as well, but even if we forget about it (it’s a very precise rule and these tools are used by hackers after all) – all of these are so frequently used by both users and embedded in many ‘enterprise’ solutions (some EDR solutions rely heavily on external hacking programs too!) – it’s impossible to sift through it
- I guess there is no way to avoid some sort of hard coded values here (yup, signatures!)
- A good example would be – show me all processes that use a single-double character as a name; this is hardcoded, but exploits the fact it’s an anomaly and hackers often use such files on the systems (and since I mentioned False Positives management – hello cywgin’s [.exe, ar.exe, as.exe, cp.exe, dd.exe, df.exe, du.exe, id.exe, ld.exe, ln.exe, ls.exe, mv.exe, nl.exe, nm.exe, od.exe, pg.exe, pr.exe, ps.exe, rm.exe, sh.exe, tr.exe, ul.exe, vi.exe, w.exe, wc.exe, xz.exe… and it’s just a tip of the iceberg in a corp. environment)
- In some cases histograms, or some clever statistics backed by exclusions and some arbitrary values (signatures again!) can do the trick f.ex. ‘show me all uploads higher than xyz bytes in last 1h’
- …
Threat hunting is anomaly hunting and defining an anomaly is extremely difficult, but we are getting better at it. The thing I want to highlight with this post is that the current state of threat hunting is that we finally know what to log and what to look at. We have absolutely no clue though how to sift through this data in a realistic, scaleable way to ensure we don’t miss a single threat. This is the very same problem security industry tries to solve for a very long time. Let’s not pretend that we did.
