Get your logging act together, loggers!

If I had to name one, the most frustrating blocker/issue that makes my work hard, one that makes me cringe every time I encounter it, it would be probably… bad logs.

Over the years I came across so many bad logs that I sometimes wonder… How it is even possible that DFIR exists… And how can it progress further when the core data we use on daily basis keeps us in a constant state of ‘I guess this is what it is/means(?)’.

A quick anecdote: while on one of my early DFIR cases, I encountered a set of logs that clearly identified malicious activity (IP connections to C2). The problem was that the logs didn’t include any timestamps, only stages of connections 🙁

What the actual…

The only thing I could write in my report was that there was an evidence of alleged activity occurring… I just don’t know … when. Bye bye trust. Not only such statement makes me look like an idiot in front of a customer, but makes it really hard to prove activities in cases where it actually really really matters (e.g. cases that go to court; mine didn’t).

There are many types of logs, logging systems, standards, log converters; many are vendor-specific, some are open-source, some are derived from homemade tools for parsing forensic artifacts (lots of them started with a lot of guesswork), some are ‘native’ to solutions being used, some are ad-hoc log formats created by developers, some are result of blue teamers parsing org-specific stuff on their own (again, often lots of guesswork!).

And then there is a whole avalanche of application-specific logs that can be only described as ‘custom’. And under the umbrella of this ugly word we include verbose logs, debug logs, telemetry, and troubleshooting logs, as well any other log-salad no one sane would ever imagine was even possible. Often created not for the consumption by the security industry, but for testing/QA, and sometimes even for sales, marketing purposes. And most of the time, very poor contextually. And in some industries that catch up with the modern technologies, there is also an additional layer that focuses on combining data from a large number of vendors, normalizing that data, and then standardizing it into a single, unified log stream. A TRULY challenging effort.

SO…

Lots of random.

If there is any way to influence / encourage vendors and coders to improve the logging quality this would really really help us all.

In particular:

  1. collecting logs in a way that makes them more actionable
    [we don’t even need to care about one standard, per se, just include all possible information that can make these logs useful in a decision-making process]
  2. keeping them properly formatted so distinction between field names, values is clear
  3. keeping them properly placed
    who cares that ‘service started’, or ‘service stopped’, and there are 50000 logs like this in the Windows Security Event Log that has NEVER been created with a mission to be dedicated to hold this particular application operational logs in it; don’t steal resources if you can place troubleshooting logs like this in a file, or your solution-specific logs
  4. keeping them relevant
    not only mixing admin/management/content update/operational logs is bad for the analysts’ BAU, it actually makes them question the value of this particular security control
    we need ‘easy’, ‘fast’, ‘actionable’, and no one wants or has time to do DIY log analysis anymore
    again, it’s 2018; we need to look at logs, not painstakingly learning what to parse, how, and how to interpret the result (and the context)
  5. keeping them not only relevant, but also short, minimalistic, in an almost Scandinavian furniture way
    • perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away
  6. documenting them in a way that makes them easy to interpret, even by juniors

Let’s look at a few examples of what can be done:

  • think of logs as a part of a timeline, not an atomic event
  • think of who is reading logs, and why
  • think of the volumes of logs; reduce volumes, if they can be reduced
  • think of making logs highly configurable
    • introduce configurable level of verbosity, and overall be more generous with detailed logs
  • enrich data at source; there is plenty of information to gather from the system; do it once, or every once in a while, but make it available in the logs
  • reach out to the security community, ask what should be logged apart from what you are logging today
  • again, think of the audience – not only troubleshooting, but also forensic analysis
  • split logs into
    • security control admin/management logs (access, role management, availability, etc.)
    • security control content management logs (definition updates, etc.)
    • security control operational logs (actual events)
    • if needed, split the logs into subsets, e.g. an endpoint solution can maintain separate log branches for
      • removable devices
      • AV detections
        • ‘bad malware’ i.e. ‘more advanced toolkit’ — hacking tools, rootkits, information stealers, mimikatz, powersploit, nopowershell
        • commodity malware
        • RATs
        • spyware
        • adware, pup, trackware
        • dual-purpose tools like psexec, nmap, nirsoft tools
        • cryptomining
        • etc.
      • client IDS
      • ‘noisier’ FP-prone heuristics (e.g. reputation)
      • etc.
  • talk to other vendors and standardize
    AFAIK there is NOT a single standard how AV detections are logged in e.g. Windows Event Logs
    most of admins / blue teamers are forced to learn which events are important by testing how AV logs them and how it names them or classifies them
    what a waste of time!
    not only log of existing events is not comprehensive (you only see what was detected, but not what is a full set of possibilities), there is also no EICAR-like file set for various types of malware/payloads (perhaps a good idea to develop these?)
  • help to prioritize
    one of the goals is to make it easy to distinguish between high-fidelity, and low-fidelity alerts; i.e. immediately actionable, and these that we can keep on a backburner for a while
  • use ‘portable’ character encoding
    • utf8/unicode, if possible, avoid ASCII/ANSI/DBCS/MBCS; seriously, it’s already 2018; if your program/OS can’t do Unicode 16-bit or utf-8 please leave the scene
    • encode blobs with base64
  • write timestamps with the highest possible resolution
    • don’t truncate timestamps!
    • you can use a binary format too (serialized/encoded), or a simple string (can be converted to anything)
    • I am personally not the biggest fan of formatted timestamp strings
      • they often trim timestamp data [time resolution]
      • it actually takes a lot of time to both convert from actual number to strings and back (multiply the time of a single conversion operation by a number of timestamps)
      • time formats are random and often highly misleading (e.g. mm/dd/yyyy vs. dd/mm/yyyy); note that your audience follows-the-Sun more often than not; ppl from various countries interpret the timestamps in their local ‘native’ format by default; mental conversion between formats is actually VERY difficult, and error-prone
  • sometimes it’s better to write binary logs (usually WAY faster)
    in such cases provide full file format documentation, example code, converters
  • log timestamps in UTC; if in local time, indicate it somehow
  • distinguish timestamps related to:
    • time of the actual event
    • time when the event itself was logged locally (e.g. Windows Event Logs)
    • time when it arrived to a collector/centralized log system
    • note that these 3 timestamps can help you to deal with various availability issues; even discover some, in a first place
    • add timestamps for all activities that matter
      for example, if you can link a download URL with a file, and then with its execution, and maybe even possible malware detection by AV, consider adding more events (one event/row) with various timestamps:

      • time of accessing the download link
      • time of detection/blocking, if link is bad
      • time of actual download completed/saving (file creation)
      • time of file AV detection alert, if applicable
      • time of file AV remediation/quarantine alert, if applicable
      • time of file ‘Open File Security Warning’ security alert triggered by Zone.Identifier.
      • time of user finally executing file [ShellExecute/CreateProcess]
      • time of UAC prompt being shown, if applicable
      • time of actual program start
      • time of advanced/behavioral AV/EDR detection
      • time of advanced/behavioral AV/EDR remediation
      • time of first network connection
      • time of program exit, if applicable
      • I know it’s a lot, but consider the added value for the analyst:
        • it’s actually really good to start mixing network logs with timeline logs extracted from endpoint forensic artifacts, if available
        • some EDRs already gather lots of such data and some present them in a timeline-ish way
        • you are not triggering individual blinkenlights anymore; you are helping to automate the response process!
  • use more rows to describe an event, if needed
    it’s better to describe operation states instead of using a single, multi-value ‘Frankenstein’s monster’ log that needs to be parsed using a lot of regex and data conversion/analysis tricks (e.g. in Splunk)
  • if you use multiple rows, use some sort of cookie, session ID that binds them together in a single cluster of events (transactions)
  • properly use standards: CSV, TSV, JSON, XML, whatever else
    seriously, this happens so frequently: please don’t screw up CSV just because you don’t know how to output new lines, escape special characters, please read rfc4180 (pls test corner cases, new lines, multiline data, data with utf-8 chars, etc.)
  • for activities that require listing file names – don’t just drop the file name; there are so many opportunities there:
    • file name
    • file extension
    • file type based on the file extension, if available
    • file type based on the file structure, if available
    • file size
    • full path
    • any other metadata available
      • hashes (MD5, Sha1, Sha256, perhaps even CRC32 for quick comparisons)
      • attributes (hidden, executable, directory, junction, symlink, setuid, etc.)
      • file timestamps (any possible on the specific file system)
      • PE compilation timestamp
      • version information
      • signature information
      • debug information (e.g. PDB path)
      • existing classification (e.g. VT)
      • avoid adding any unconfirmed information
        (e.g. vague statements from tools that ‘estimate’ or ‘guess’ file characteristics)
  • for processes
    • process identifier (PID)
    • process name
    • process command line
    • process characteristics
      • CLI, GUI
      • GS, DEP, ALSR, CFG
      • etc.
    • process current directory
    • parent process identifier (PID)
    • parent process name
    • parent process command line
    • other possibilities
      • section names in file where the process was initialized from
      • section names in memory
      • security token
      • privileges
      • time of start
      • time of exit
      • list of modules at start
      • list of modules at exit
      • list of RWX regions at start, with mappings to images
      • list of RWX regions at exit, with mappings to images
      • main window title, class, if GUI app
      • list of child windows
      • perhaps some flags that can indicate
        • if GUI process, was there any interaction with the user before the program exited?
        • etc.
      • etc.
    • exit code
    • some of these properties can be listed for all events; some can be dumped separately (different events) in a one-off fashion when
      • the process is started
      • at regular intervals
      • and/or when the monitored property changes
      • program exits
        this may save a lot of analyst time; such basic volatile information may be often enough to get a basic understand of what happened
  • for crashes
    • similar info as for the processes
    • + time of crash
    • + address of crash
    • + stack trace/with symbols, if available
    • + list of loaded modules
    • etc.
  • for network artifacts enrich data if possible
    • interface(s)
    • its promiscuity
    • MACs, if needed
    • source and destination IP
    • source and destination ports
    • ip version
    • ip type (static, dhcp)
    • states (established, listening, etc.)
    • direction (inbound/outbound)
    • sizes of data transfers in/out
    • session ids, if it is within one session
    • host name(s)
    • logged object other metadata
      • mime types
      • user agents
      • referers
      • classification
      • user name / region / business unit (if can be mapped)
      • any links to processes involved e.g. PID of the process, process name, etc.
      • CVE
      • CVE scores
      • CVE age/expiration (so we don’t trigger events on perl or php vulns from 2000, or CodeRed anymore)
      • any tags (e.g. VIPs, PCI-DSS systems, DMZ, whatever else)
  • if possible DO NOT COMBINE values e.g.
    • 1 field: ip:port
      is worse than
    • 2 fields: ip and port
  • again, DO NOT COMBINE values
    if we need to use regex to extract basic values from the logs you are doing it wrong
    – you are making things slow, cuz every single row needs to be regexed to extract these atomic data items
  • yes, think ATOMIC chunks of data
  • if you provide field names
    • AVOID changing names of the fields over time
    • AVOID changing order of the fields over time
    • don’t use white characters in the field names (makes life easier if data is imported to other systems)
    • don’t make field names too long (imagine what you see in Excel when you export data, or when you write Splunk queries that have to rely on prefixed fields in a form of
      <120 characters>.file
      <120 characters>.md5)
    • escape characters properly
    • use consistent nomenclature (src_ip and ip_dst is inconsistent)
  • DO NOT DUPLICATE information
    • user vs. account
    • host vs. ComputerName vs. Source vs. src
    • cmd vs. COMMAND
    • arg0 vs a0
    • etc.

Then… there is a documentation. This really deserves a dedicated post, but just to highlight a few bits:

  • write down the format of the log
    • typical location on the system
    • info on process/library creating them
    • character set used
    • field names
    • field descriptions – not one-liners, but real meaning + context how to interpret
    • order of the fields
    • how characters are escaped
    • how new lines are added
    • read, read again, read one more time
    • test, test, test
  • now add all possible, additional details you can think of; context, comparisons, example data, example incident data, and also review the source code; then add info on specific (code snippets are OK) conditions that trigger certain events; analysts need to understand EVERY condition that triggers the event logging so they can write playbooks for it
  • provide sample code to parse logs in at least 1-2 popular languages (python? perl? ruby?)
  • provide a sample tool/program to test the logs (e.g. one that will generate various types of events in a way like AV uses EICAR)
  • give the community/analysts a platform to provide a feedback

And finally, only at the stage when we have as many logs as possible, and they are in a proper/decent format, and they provide enough information that is enriched any possible way, and they have been tested to provide required data, only then we can start doing Mite Att&ck assessments…

A few things about EICAR that you may be not aware of…

Update April 2017

As per info from Vess, the programmer who was responsible responsible for writing the EICAR file was Padgett Peterson.

If you get excited about EICAR file making the news as being able to make AV deleting logs when EICAR is used as a user name, password, User agent, etc. – it’s old news 😉 Read the history of the file including first attempts to abuse it here.

Old Post

When my wife studied her MA in graphic design and branding she got a lot of interesting home work. One of them was… ‘The square’. She spent a lot of time brainstorming and eventually produced a large collection of ideas that got her a good mark. Now, the simple purpose of that exercise was to play around with the idea of… ideas. As simple as it sounds, the moment you start exploring one ‘simple’ subject you will soon find yourself deep in a forest.

As I am adding support for many Quarantine files now (to DeXRAY) I suddenly found myself in a world of Antivirus analysis. One thing that somehow connects all of AV products is not their functionality, or Utopian vision of full protection, but… the EICAR file.

I decided to explore the topic of this file a bit – same as my wife was exploring the square. Yup, here’s a boring story SLASH a bunch of ideas associated with EICAR SLASH and other topics like this …

First of all – in case you don’t know – EICAR is a small file that is used as a test for security products (in the past it was mainly antivirus, but nowadays it should apply to any security solution that looks at files/content of any sort really). Once you deploy/install the solution/product, you can drop the EICAR file all over the place and see if solution picks it up. Notably, some AV vendors apparently do not understand what EICAR’s purpose is and decided not to detect it. I won’t be pointing fingers, but upload EICAR file to VirusTotal and you will know who I am talking about.

Naming conventions in AV is a subject to many debates over many years. EICAR looks like a no-brainer though as it’s an artificial file created with a single purpose and its origin and name are well-documented. It doesn’t help though… it would seem that vendors can’t agree on one, single name. Here is a histogram of names used by AV:

EICAR_test_file                  11
EICAR-Test-File                   7
EICAR-Test-File (not a virus)     4
Eicar test file                   3
EICAR (v)                         2
Eicar-Test-Signature              2
EICAR.Test.File                   2
EICAR.TestFile                    2
EICAR Test File (NOT a Virus!)    1
EICAR.TEST.NOT-A-VIRUS            1
EICAR-Test-File (not a virus) (B) 1
EICAR Test String                 1
DOS.EiracA.Trojan                 1
Marker.Dos.EICAR.dymlmx           1
EICAR.Test.File-NoVirus           1
NORMAL:EICAR-Test-File!84776 [F]  1
EICAR-Test-File!c                 1
EICAR Test-NOT virus!!!           1
Win32.Test.Eicar.a                1
Misc.Eicar-Test-File              1
EICAR_Test                        1
NotAThreat.EICAR[TestFile]        1
qex.eicar.gen.gen                 1
TestFile/Win32.EICAR              1
Virus:DOS/EICAR_Test_File         1
EICAR-AV-Test                     1
EICAR-AV-TEST-FILE                1
EICAR-test[h]                     1

And the bonus: one of these even has a typo. Can you spot it?

EICAR is a very strange phenomenon.

It is an organization. It is a file. It has a dedicated web site. It haz a dedicated con. Its original name is inclusive of Europe, and exclusive of other continents (EICAR stands for ‘European Institute for Computer Antivirus Research’; deprecated name, but always…).

Anagrams of EICAR are ERICA, CERIA and AREIC. They serve no purpose in this article.

Properties:

File size: 68 bytes
MD5: 44D88612FEA8A8F36DE82E1278ABB02F
SHA1: 3395856CE81F2B7382DEE72602F798B642F14140
SHA256: 275A021BBFB6489E54D471899F7DB9D1663FC695EC2FE2A2C4538AABF651FD0F
CTPH: 3:a+JraNvsgzsVqSwHq9:tJuOgzsko
Entropy: 4.872327687

Eicar is a DOS file and can be executed… but only under old versions of Windows.

eicar

The source code is using the same tricks as shellcodes:

  • code is obfuscated

eicar2

  • it is a self-modifying code (patching itself)

eicar3

eicar4

Does your sandbox solution accept EICAR? Test it.

There exist tools that help you to generate EICAR file and its cousins (file formats embedding EICAR).

There exist a close friend of EICAR called AMTSO (Anti-Malware Testing Standards Organization) that focuses on testing antimalware methods. It produces some more test files to support the original idea introduced by EICAR f.ex. Potentially Unwanted Application equivalent of EICAR:

puaeicar

with the histogram of detection names as follows (VT detection rate: 43/56 – mind you that the file was compiled on 2013-04-04 21:26:07 (Thursday)):

Application.Hacktool.Amtso.A                             5
Riskware ( 0040eff71 )                                   2
AMTSO-Test                                               2
PUA_Test_File                                            2
RiskTool.EICAR-Test-File.r5 (Not a Virus)                1
RiskWare[RiskTool:not-a-virus]/Win32.EICAR-Test-File     1
RiskTool.Win32.AMTSOTestFile (not malicious)             1
Amtso.Test.Pua.A                                         1
W32/PUAtest.B                                            1
AMTSO_PUA_TEST                                           1
RiskTool.Win32!O                                         1
Application.Win32.AmtsoTest.a                            1
Riskware.AMTSO-Test-PUA                                  1
Application/AMTSOPUPTestfile                             1
Trojan.Staser.gen                                        1
Application:W32/AMTSOPUATestfile                         1
W32/TestFile.LCMA-1046                                   1
Backdoor.CPEX.Win32.29390                                1
Risktool.W32.Eicar.Test!c                                1
Hacktool.Win32.EICAR-Test-File.aa                        1
RiskTool.Win32.AMTSOTestFile                             1
not-a-virus:RiskTool.Win32.EICAR-Test-File               1
AMTSO-PUA-Test                                           1
PE:Malware.Generic/QRS!1.9E2D [F]                        1
Riskware.Win32.EICARTestFile.dmxhvk                      1
PUA.AMTSOTest                                            1
SpyCar                                                   1
PUA/AMTSO-Test                                           1
Trojan/W32.Agent.33280.TI                                1
Win32/PUAtest.B potentially unwanted                     1
W32/TestFile                                             1
Win32:AmtsoTest-A [PUP]                                  1
AMTSO-PUA-Test (PUA)                                     1
RiskTool.EICAR-Test-File.a                               1
AMTSO Test File PUA (Not a Virus!)                       1
PUP/Win32.AMTSO_Test                                     1

…and the cloudish EICAR file as well. Here’s the histogram of names given to the cloudish EICAR file (only 23/56 vendors detect it on VT; compilation date:  2010-07-08 23:02:46 (Thursday), ouch!):

AMTSO_TEST_CLOUDCAR                    2
Cldcar-Test!3FB121FBBCCB               2
Trojan.Win32.Generic!BT                2
Trojan.Agent/Gen-CloudTest             1
Virus:DOS/EICAR_Test_File              1
Trojan.Generic                         1
Application.Win32.CloudTest.s          1
Win.Trojan.11584714-1                  1
Amtso.Test.Cloudcar.A                  1
Trojan.Brodcom.Win32.366               1
Trojan.Win32.DangerousObject.dlgbhn    1
AMTSO-CLOUD-Test                       1
Trojan.Win32.Z.Agent.7178[h]           1
CLOUDCAR_Test                          1
UDS:DangerousObject.Multi.Generic      1
DangerousObject.Multi.Generic!c        1
W32/GenBl.3FB121FB!Olympus             1
Mal/Generic-S                          1
AMTSO Test File (NOT a Virus!)         1
Trj/CI.A                               1

There also exist a close friend fo EICAR called GTUBE (Generic Test for Unsolicited Bulk Email) for testing anti-spam solutions. It is also 68 bytes long.

Last, but not least – there exists a shorter version of EICAR and it is 12 bytes SHORTER than the original!

The Base64-encoded EICAR looks like this:

WDVPIVAlQEFQWzRcUFpYNTQoUF4pN0NDKTd9JDIDYOUSPOTTHISEVJQ0FSLVNUQU5EQVJELUFOVElWSVJVUy1URVNU
LUZJTEUhJEgrSCo=

You may come across it as it is being used in various tests and… it is a method used by Esafe to save Quarantine files. And… maybe you can’t read this post in case your security product is over protective and detected the BASE64-encoded EICAR string. Well, if it does… it shouldn’t, as I included ‘DIDYOUSPOTTHIS’ in the BASE64 encoding above. Well, did you spot that DIDYOUSPOTTHIS?

EICAR is a tool. I use it to test Quarantine files’ encryption. When I find no encryption, or trivial encryption/encoding – I love EICAR. When I have to dig into some actual code to find out how they transform the original EICAR bytes into sth terrible I absolutely hate this little piece of hybrid data/code ;).