The Threat Hunting -> the anomaly hunting -> the data eyeballing

October 16, 2016 in Preaching

In my last post I described my opinion about current state of affairs of the fashionable IR trend called threat hunting. Since I am advocating it myself for a number of years I probably shoot myself in a foot by being openly critical about its potentials, but if there is one way for this discipline to progress it is to question your own doing…

Since I posted it I received a rebuttal from Jack Crook and a couple of answers on Twitter. Jack’s rebuttal starts by hinting at my inexperience in a field and then goes all the way to promote the very wishful thinking that I decided to criticize. The simple answer to Jack’s post is this: theory vs. practice. The fundamental problem of threat hunting is not how we detect specific things, or how we generate better ideas how to correlate data for specific cases. This is easy. Once you have that data, that correlation engine in place and that specific idea – you just implement it. It is a simple engineering work. The ‘fun’ begins afterwards – the tweaking. When I brought up a couple of examples of potential (but naive) threat hunting rules that could be used to detect badness in the environment, it was not to highlight them as examples of rules that could be potentially improved. We can always do so, and can/should look at more holistic way these events correlate with each other. I agree, I actually do, because I practice it too and quite a lot. Yet, I don’t want these two points I made to be missed:

  1. The threat hunting _is_ signature writing. How far is it from IOCs, yara and lo and behold – virus signatures? And we know it doesn’t work.
  2. The error of availability makes us a subject to naive assumptions – one that complex correlations will help us to get rid of False Positives easily (which we are not aware of until we test it on large corpora of data)

I have already covered the signature writing, let me discuss the error of availability a bit more. We write rules for what we have seen (btw. antivirus works the same, reactive way). Our exposure to the knowledge about BAD actors and behaviors is limited, but we are getting better and better at it. Yet I can bet that neither you nor myself know, or remember all the tricks of the trade. Will your threat hunting cover everything? Does it cover everything inside the Rtfm-Red-Team-Field-Manual and Post Exploitation Wiki? I published over 40 posts about various persistent mechanisms in last couple of years – I bet most of the threat hunters do NOT look at these cases (full disclosure: I don’t). The avalanche of clever tricks developed by SubTee is definitely NOT covered by most of us either. Add to it the variations of OS commands, tools, various WMI, kiosk-escape, pentesting kung-fu trickery, and also unpublished research, and a few thousands blogs one would need to follow (and constantly discover new ones: don’t forget Chinese, Korean, Arabic, Spanish, etc. blogospheres) to stay on top of things and you know that we would still be behind. I did mention before that we want to detect the insider threat too. Wishful thinking it is.

But let’s – for the sake of the argument – assume that we read all blogs, APT reports, did our homework and collected all the trickery in our head and we know what to hunt for. Here comes the error of availability number two – our exposure to the knowledge about GOOD actors and behaviors. It is extremely limited. And it is the latter that makes the Threat Hunting extremely challenging. Because while we successfully leverage our (limited) knowledge about the BAD guys, most of the time we completely ignore the one about the GOOD guys. Once we apply this approach in practice, the False Positives show up immediately. I have already listed a number of examples of GOOD actors’ behaviors in my previous posts. If you simply ignore them in a discussion then I suspect you don’t have the exposure to a corpora of real security events from a large organization. Support/Admin scripts often launch a number of lateral movement tools one by one, people use Excel and Word to launch cmd.exe, and other tools, the enterprise solutions use psexec, autoruns, cmd, cscript, xcmd, nmap, WMI, MSI, they load drivers, create scheduled tasks, services, google chrome with its updates is a ‘nice’ event generator, people use crazy number of variants of putty, kitty, ssh, winscp placed all over their systems, they download files using wget, curl, even bitsadmin is used either by good guys, or by software, then there is cygwin, a number of users learning hacking in-house, and so on and so forth… Having a tree and ability to correlate the events is often NOT enough. The sad truth is that behaviors of BAD and GOOD guys are often indistinguishable!

The lack of public data for researchers’ consumption is a problem in this space and one that can be hopefully addressed one day. The threat hunting discussions need to be very data-specific. Threat hunting today is data eyeballing. Just saying that ‘we need to look at better rules or correlations and leverage our knowledge about bad actors’ is simply vague. And vague Threat Hunting cannot be. I guess it’s really time to engage data scientists.

Being critical is constructive only if we discuss next steps. I believe that there is a list of must-do things for any threat hunter – they may act as an eye opener:

  • Look at clusters of events generated across large environment (not just one or a few system)
    • any event related to execution of wmic.exe, powershell.exe, cscript.exe, wscript.exe, mshta.exe, and pretty much any legitimate .EXE from the OS
    • any event related to loading of pretty much any .DLL
  • Look at clusters of commands intercepted from sandboxing of large malware samplesets
    • Here I can help with some real data
      • How would you discover chrome.exe dropped shown in these events?
      • What rule could you write to discover variants of svchost.exe ?
      • How would you detect artifacts dropped by APT1 set?
  • Read as much as you can not only about new malware and hacking tricks, but also about architecture of the OS, and… programming manuals, including the OLD school stuff like Win32API, COM, DDE, etc. (I am very Windows-specific in this post, but same applies to other OSes)
  • Promote science of data analysis – I am personally way behind and feel there is a need to get better at it

Last, but not last…here’s a couple of questions to highlight the ambiguities you may come across:

  • Is a parent-child relationship between the following processes a sign of malicious activity?
    • svchost.exe -> iexplore.exe
  • What about this one?
    • svchost.exe -> winword.exe
  • And this one?
    • cmd.exe -> cscript.exe -> nmap.exe
  • And this one?
    • <Non-existent process (70564)> -> cmd.exe -> putty.exe
  • What events are triggered when you launch a new process via WMI?
    • Is the detection of processes opened this way a good threat hunting rule?
  • Is injection of a remote thread into a foreign process (via CreateRemoteThread API) always a sign of malicious activity ?
  • The see 50 systems with a number of events indicating randomly named DLLs were loaded from the %TEMP% folder – would you consider it a malicious activity?
  • You find traces of radmin, logmein, team viewer executed on a number of systems; would you consider them a malicious activity? (bonus question: how would you codify a Business Unit-specific case)
  • Is modification of  c:\WINDOWS\system32\drivers\etc\hosts an alertable event?
  • Finally, since I mentioned we need to be specific and talk about real data – here’s a top of the list from the histogram of commands executed by malware samples  – I generated it over 2 years ago from a decent cluster of sandboxed samples – which ones would you say are bad? Let me remind you – most of them are associated with malicious activity one way or another… but are they?
  69830 cmd /c at
  69829 cmd /c rmdir
  69829 cmd /c attrib -s -h -r
  21710 msiexec /i 
  18872 ipconfig /all
  10859 cmd /q
   8367 cmd /c del 
   8055 regsvr32 /s 
   7520 cmd /c 
   6906 msiexec /quiet /i 
   6227 net share admin$ /delete /y
   6226 net share ipc$ /delete /y
   6225 net share d$ /delete /y
   6221 net share c$ /delete /y
   5307 ipconfig /renew
   5264 regsvr32  /s  
   5177 ipconfig /release
   3783 taskkill /f /im zhudongfangyu.exe
   3428 taskkill /f /im explorer.exe
   3373 reg add hklm\system\currentcontrolset\control\terminal" "server\winstations\rdp-tcp /v portnumber /t reg_dword /d 
   3371 reg add hklm\system\currentcontrolset\control\terminal" "server\wds\rdpwd\tds\tcp /v portnumber /t reg_dword /d 
   3254 taskkill /f /im firefox.exe
   3207 cmd /c reg add hklm\system\currentcontrolset\services\sharedaccess\parameters\firewallpolicy\standardprofile\authorizedapplications\list /v "
   3013 taskkill /f /im wmplayer.exe
   2996 taskkill /f /im winamp.exe
   2987 taskkill /f /im iexplorer.exe
   2981 taskkill /f /im ansav.exe
   2977 taskkill /f /im winampa.exe
   2977 taskkill /f /im pcmav-cln.exe /im pcmav-rtp.exe
   2977 taskkill /f /im ansavgd.exe
   2961 regsvr32 /s /u 
   2683 cmd /c rd 
   2496 cmd /c net stop sharedaccess
   2404 regsvr32 /u /s 
   2399 cmd /c reg add hklm\system\currentcontrolset\services\sharedaccess\parameters\firewallpolicy\standardprofile /v "donotallowexceptions" /t reg_dword /d "0" /f
   2356 regsvr32 /u 
   2088 cmd /c net stop wscsvc
   2082 taskkill /f /t /im zhudongfangyu.exe
   1988 gpupdate /force
   1954 reg delete hkey_current_user\software\microsoft\windows\currentversion\explorer\runmru /f
   1915 taskkill /f /im dnf.exe.manifest
   1846 reg add hkcu\software\microsoft\windows\currentversion\policies\associations /v modriskfiletypes /t reg_sz /d .exe /f
   1770 taskkill /f /im wscript.exe
   1761 taskkill /f /im sys.exe
   1760 taskkill /f /im tati.exe
   1760 taskkill /f /im kspoold.exe /im kspool.exe
   1760 taskkill /f /im ansav.exe /im ansavgd.exe
   1661 cmd /c sc config sharedaccess start= disabled
   1654 ipconfig /flushdns
   1639 cmd /c cacls "<file>" /e /p everyone:f
   1623 cmd /c sc delete javaserve
   1619 cmd /k 
   1474 cmd /c del /f /s /q "%userprofile%\local settings\temporary internet files\*.*"
   1474 cmd /c del /f /s /q "%userprofile%\local settings\temp\cookies\*.*"
   1474 cmd /c del /f /s /q "%userprofile%\cookies\*.*
   1474 cmd /c attrib -h -s -r -a "%userprofile%\local settings\temp\cookies\*.*"
   1474 cmd /c attrib -h -s -r -a "%userprofile%\cookies\*.*"
   1457 taskkill /f /im ksafetray.exe
   1417 cmd /c taskkill /im ekrn.exe /f
   1350 reg add hkcu\software\microsoft\windows\currentversion\policies\system /v disabletaskmgr /t reg_dword /d 1 /f
   1320 reg add hkey_local_machine\software\microsoft\windows\currentversion\policies\explorer /v nofolderoptions /t reg_dword /d 1 /f
   1294 cmd /c
   1239 taskkill /f /pid 
   1218 cmd /c regedit /s c:\system.reg
   1099 cmd /c taskkill /im egui.exe /f
   1086 cmd /c sc config avp start= disabled
   1048 regsvr32 /s /u "
   1025 regsvr32  /s 
    985 cmd /c reg delete hkcu\software\microsoft\windows\currentversion\r
    982 taskkill /f /t /im rstray.exe
    970 cmd /c echo y| cacls 
    928 cmd /c ping 
    909 cmd /c sc config ekrn start= disabled
    867 taskkill /f /t /im ksafetray.exe
    855 taskkill /f /im knightonline.exe
    828 cmd /c start 
    755 cmd /c reg add hkcu\software\microsoft\windows\currentversion\run /v ctfmon /t reg_sz /d 
    708 cmd /c c:\mysql.exe -u
    708 cmd /c app.path & \mysqldump.exe -u
    699 cmd /c net stop mpssvc
    667 cmd /c taskkill /f /im qq.exe  /t
    654 taskkill /f /im rstray.exe
    654 taskkill /f /im iexplore.exe
    648 cmd /c sc delete ekrn
    636 cmd /c taskkill /im avp.exe /f
    611 cmd /c sc config 
    606 taskkill /f /im gbpsv.exe >nul
    599 cmd /c reg add hkcu\software\microsoft\windows\currentversion\run /v msmmsgr /t reg_sz /d 
    592 schtasks /create /sc onlogon /tn ":schname" /tr ":path"  :vista
    586 cmd /c reg add hkcu\software\microsoft\windows\currentversion\run /v 
    583 cmd /c del "
    564 cmd /c taskkill /im scanfrm.exe /f
    560 reg add "hklm\software\microsoft\windows\currentversion\internet settings" /v proxyenable /t reg_dword /d 00 /f
    560 reg add "hkcu\software\microsoft\windows\currentversion\internet settings" /v proxyenable /t reg_dword /d 00 /f
    558 reg add hklm\software\microsoft\windows\currentversion\explorer\cabinetstate /f /v fullpath /t reg_dword /d 1
    552 reg add "hkcu\software\microsoft\windows nt\currentversion\winlogon" /v "shell" /t "reg_sz" /d "explorer.exe,
    539 regsvr32 /s mswinsck.ocx
    530 cmd /c ping -n 3 && del "
    525 cmd /c erase /f "

Comments are closed.