When you start using this technique you will come across many obstacles because lots of paths use random bits. Many of them are kinda predictable, it’s just the patterns keep piling up. And this is where normalization helps a lot.<\/p>\n\n\n\n
Normalizing data is pretty easy – we use regex-based replace<\/em> function to remove unnecessary junk. You have to start with the most precise patterns and then go towards the vague, hope-for-the-best ones. <\/p>\n\n\n\n Precise ones include:<\/p>\n\n\n\n One has to hope that these will reduce a lot of noise, and if it is still not the case we can go a bit further and start using more vague patterns e.g.:<\/p>\n\n\n\n It’s pretty hard to talk \/ learn about it without actual doing it hence I attached a test data set for you to play with (see bottom of this post).<\/p>\n\n\n\n The test set contains a number of fictional hosts (named after islands) and a bunch of paths that I made up so that we can demonstrate how the LFO and normalization can work in tandem. After you download the test set, you can import it to Splunk using the following name: test_dataset_paths.csv<\/em> (I use it in my examples).<\/p>\n\n\n\n To confirm data is accessible via alookup file you can run this command in Splunk:<\/p>\n\n\n\n The set includes Paths from 6 hosts:<\/p>\n\n\n\n belonging to 6 users:<\/p>\n\n\n\n John, James, and Paul installed Firefox, and Kate, Joan and Alice – Chrome. John is the only user who has his system infected.<\/p>\n\n\n\n When we run the inputlookup command we can immediately see that even with a small number of rows it’s not that easy to comb through it:<\/p>\n\n\n\n Even if we want to do stats over it (per Path):<\/p>\n\n\n\n we get this:<\/p>\n\n\n\n It’s only after we apply normalization we get a data set that makes it easier for us to decide what to discard:<\/p>\n\n\n\n Since we see repetitions of identical normalized path across multiple systems, and some paths are clearly present on all the systems, we can remove them from the view using dc<\/em> where number of hosts is at least 3:<\/p>\n\n\n\n This brings us to the final result that literally shows the malware:<\/p>\n\n\n\n If you are wondering why I am using dc<\/em> and not count<\/em> for exclusions, it’s because your data set can include repetitions from the same host (in such case count<\/em> would be higher, while still applying to a single host); dc <\/em>gives you number of all hosts on which specific Path occurred at least once.<\/p>\n\n\n\n I know it’s trivial and probably used by any splunker out there, but I remember that when I started learning SPL I was working with huge datasets from the start and it was quite overhwelming. Working with a small custom-made set makes it easier to test ideas and… regexes (especially these that require guessing how many backlashes to put to escape characters properly). <\/p>\n\n\n\n And speaking of the devil… here is a bunch of replace<\/em> function regex examples you can consider using:<\/p>\n\n\n\n They may be buggy (both logic and formatting of this blog may affect it), so treat with caution and always check against your data set. It is mainly to show a couple of ideas that can help to start.<\/p>\n\n\n\n Using LFO, normalization and with a few other tricks (e.g. filtering by additional fields, counting per hosts, directories, bucketing, adding weights for risk-based scoring) leads us to a very favorable outcome: we stop using inclusion\/exclusion lists i.e. we stop using signatures.<\/p>\n\n\n\n The data set is here<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":" One of the most annoying bits that we come across while doing log analysis is both predictability and unpredictability of file paths. Somehow…. everyone really… vendors, admins, and finally users keep coming up with new ways to name files that … Continue reading | inputlookup test_dataset_paths.csv<\/pre>\n\n\n\n
![\"\"](\"https:\/\/www.hexacorn.com\/blog\/wp-content\/uploads\/2020\/05\/spl1.png\")
<\/a><\/figure>\n\n\n\n| inputlookup test_dataset_paths.csv
| stats values(Host) as Hosts by Path<\/pre>\n\n\n\n<\/a><\/figure>\n\n\n\n| inputlookup test_dataset_paths.csv\n| eval norm_path = Path\n| eval norm_path = replace(norm_path, \"(?i)c:\\\\users\\\\[^\\]+\", \"\")\n| stats values(Path) as Paths values(Host) as Hosts count by norm_path<\/pre>\n\n\n\n
<\/a><\/figure>\n\n\n\n| inputlookup test_dataset_paths.csv
| eval norm_path = Path
| eval norm_path = replace(norm_path, \"(?i)c:\\\\users\\\\[^\\]+\", \"\")
| stats values(Path) as Paths dc(Host) as dch values(Host) as Hosts count by norm_path
| where dch < 3<\/pre>\n\n\n\n<\/a><\/figure>\n\n\n\n