You are browsing the archive for Preaching.

Are you a Canon or Nikon?

August 16, 2015 in Incident Response, Preaching

In a world of photographers there is a very common question exchanged between the peers which goes something along the lines of:

Are you a Canon or a Nikon user?

Sometimes it goes beyond that and includes other brands: Fuji, Sony, Leica, etc., but the strong association with brand is err… a canon itself, at least in a photographic world.

The ability to love, or at least stick to one brand has benefits – you get really familiar with the brand, things fit, loyal customers can leverage the long-term relationship in many ways and sometimes it simply makes sense to be really good at one thing.

The world of security adapted this principle and applies it one by one to various security controls.

There are just a few of antivirus companies that matter and everyone can quickly associate themselves with one of them.

The very same goes for firewalls.

And DLP solutions.

And old-school forensics tools.

Even sandboxes seem to be there already.


IR solutions are IMHO still not there.

The differences between a good IR solution and a less-good IR solution are slowly emerging though.

These are for example:

  • Ability to deploy and rapidly gain the best coverage within any organization, no matter what network topology they use
    • Whoever can deploy faster and cover more quicker, wins
  • Ability to fetch host’s volatile data is crucial
    • Whoever does it in real-time does it better than sweepers
  • Ability to fetch / monitor data from both kernel and userland
    • Whoever can do it for both lands, is a winner
  • Ability to sweep is still a very valuable add-on (they may detect campaigns AV doesn’t detect, web shells, etc.)
    • Whoever offers both sweeps and real-time analysis, wins
  • Ability to fetch data in the most forensically sound manner
    • Whoever does it and ends up with the least contaminated host wins
  • Ability to interact with the host’s volatile data (more precisely: with the actual object f.ex. processes)
    • Malware removal is often as easy as killing the process and removing the file – whoever allows analysts to do it on the spot, wins (old school IR relies on psexec, help from a dedicated desktop/server team or even user to remove these and not uncommon is a full system rebuild)
  • Ability to easily update
    • Whoever does it in a way that is ‘invisible’ to everyone involved, wins

Many of these are well-known and faced – for many years – by AV companies. Theirs are ones of the most prominent solutions deployed on endpoints today. It often boggles my mind that they have overslept the whole DFIR revolution that happened over last 5-10 years. It’s such a corporately-speaking “low-hanging-fruit”.

Adding IR capability to AV product is such a no-brainer. Someone, please explain :)

This is still a time of experiments in IR world and there is no pool of shortlisted winners yet.

It is NOT wise to change IR vendors often now.

Let these that are fighting for the market battle with each other, and winners will emerge.

If you have already purchased something from one vendor, do not rush on buying stuff from a new one.

The corporate world is currently divided into 4 IR buckets:

  • these who don’t know what they don’t know – these are destined to fail a.k.a. being epically pwned
  • these who know that they don’t know something – these are destined to fail, but at least are not surprised
  • these who know what they know – they may fail gracefully
  • these who know what they know, and potentially try to learn about the unknowns – a few lucky ones that have a (relatively) mature IR program implemented; they may still fail, but it will be a really graceful fail a.k.a. IR mission accomplished

If you see similarities with four stages of competence – this is not accidental.

Ignorance in this industry is omnipresent and mind you – it includes yours truly. We are all firefighting a lot. The complexity of the IT Security is beyond a reach of a single individual, single vendor, single brand. Full stop. I think this is the world where the dry, theoretical certificates like CISSP/CISM meet the juicy, technical SANS exams. You use knowledge from CISSP/CISM (security controls & their management) and apply your technical knowledge (choose the best security controls & proper IR/forensics) from SANS courses to defensively ‘pwn’ your company internally.

I guess it’s time to reveal the real purpose of this post.

You need many security controls. Choose one for each area that requires coverage, and stick to it. Do not flip them like burgers every 1/2/3 years. You want to invest once, deploy once, know it inside out and gain maximum coverage. The time to realistically deploy and fully understand each security control (to the level of ‘defensive pwnage’) is – in my personal opinion – 2-5 years. (there are many reasons for this taking so long: complicated network topology, legacy infrastructure, collisions/overlaps with other projects, ensuring business is affected in a minimal way, gradual deployment, staff changes, and then bugs, configuration issues, of the products themselves, etc. The other reason, especially for IR solutions is their immaturity. Many IR solutions are built in the garage, the only real field test they go through is when some company allows the IR vendor to put a foot in a door and give them access to the environment to test the waters. This is a real QA phase. If you are company doing the QA you will spend a lot of time troubleshooting. It’s ironic, because instead of being a customer, you are actually performing QA for the vendor. So, overall, it costs a lot of time and money to deploy ONE solution.

This is why I think the IR vendor change should be based not on budget or even merit, but on the coverage of the existing IR vendor (of course, if they suck, you need to change).

Only when you cover let’s say 98% of your endpoints – look at alternatives*.

*In case you are wondering why – think of it this way: imagine you have a coverage of 98% with your current vendor (in a large corporate 98% coverage means there are still a lot of endpoints not covered- close to a few thousands, actually, but it’s a risk management, after all). Assuming the IR vendor you use is not some snake-oil company by the time you see this 98% coverage you may be quite sure that most of your endpoints are clean. Only then you should have a confidence to look for gains on the financial or even usability side, let alone whistles and fireworks that other vendors offer, because you know that you are going to deploy the new guy to the environment that is already 98% IR-managed.

At least, this is what I believe is true.

The 3 stages of 3ages

November 27, 2014 in Compromise Detection, Malware Analysis, Preaching

Quick Update

Just to clarify: this is a critique of IR processes that rely on a single way of doing things which in certain circumstances is not the best; it may slow down your response time & give you a lot of unnecessary work. In other words: Alexious’ principle (see below) is a good way of doing things. Doing full or raw forensics on the example 400 hosts would be very inefficient. It mainly applies to daily IR/SOC work, not consulting gigs.

Original Post

The Digital Forensics world is a subject to trends the same way as is fashion.

A long time ago everyone would just do the bit-by-bit & offline forensics and that would be enough. Then ‘do not pull the plug’ idea came along and now no one dares to shut down the box until at least the volatile data is acquired. On the shoulders of the importance of volatile data came the notion of importance of memory forensics which concluded (as of 2014) in a phenomenal work of the Volatility team, an excellent tool and the best book about ‘all of it’ presented in the most digestible way ever.

Somewhere in the background a lot of research towards understanding of the internals of NTFS was also done, then it got digested by many, and finally converted into popular, and often free applications. It’s actually a good time to be in DFIR because tools are out there and you can crack cases in no time. On top of that, both memory and NTFS/$MFT forensics are probably the most attractive technical aspects of DFIR world in general due to technical difficulties associated with their everchanging complexity & simply speaking ‘it really takes time getting to understand how it all works, but it is very rewarding’ (c).

What could possibly go wrong?

The everlasting and omnipresent part of the DFIR work is the word ‘context’ a.k.a. scope (if you are from consulting or compliance world).

One thing I observe last few years is a very strange trend which can be formulated as:

  • triage is now equal to memory & $MFT forensics.

If you can do it quickly, have proper tools and know what you are doing  – it may actually work.


I believe that it’s often an over-engineered solution to a much simpler problem.

Context is REALLY important. And it dictates what you do and how you do it. And I believe that the context is always driven by the character of the investigation.

Let’s make an attempt to describe the various ‘levels’ of depths one can reach while doing DFIR work. It all is depending on… yes, you guessed it right – context (or scope).

  • Law Enforcements engaged / Criminal case
    • Full blown forensics with a major stress on accountability/logs/notes, chain of custody; and applied to every possible device you can find on the crime scene
    • Almost always goes to court, or the possibility is pretty high
    • You are SUPERCAREFUL, because everything you do is going to be shown to law interpreters [a.k.a. lawyers :)]
    • You use a very specific, self-protective language to describe your findings
  • Confirmed compromise with more than one aspect of C.I.A. triad being affected (e.g. PCI space, hacking cases)
    • Almost identical with the above case, with one extra bit – full forensics for the scoped systems + raw or light forensics in a ‘close neighborhood’
    • Surprisingly, it does not go to court that often, but sometimes it does. Whatever you do – do with an assumption it WILL go to the court one day. So, you are still VERY CAREFUL, take care of the chain of custody and statements
    • You also use a very specific, self-protective language to describe your findings
  • Day to day work on the IR/SOC team
    • Your role is to keep the company secure and literally speaking: find & close incidents
    • Usually you do Light forensics for all systems
    • Only and only if deeper intrusion is confirmed raw/full forensics are used

Same as in school, this is all about grades.

Just to be precise here: I have used some terms above which require further explanation:

  • light forensics – focus on data that is ‘easy’ to acquire with OS-only tools and a minimal impact on the system (minimal contamination) – this is not your memory forensics/ $MFT analysis yet; it is AV logs, “dir /a/s c: > evidence.txt”, “powershell gong foo”, “netstat”, “wmic /xyz/” variations, maybe later on autoruns and Sysinternals tools, etc. + copying it over to your box for further analysis
  • raw forensics – maybe there is a better name; if your light forensics didn’t detect anything and you suspect you need more – this is the time when you need to go deeper; natural progression is to look at $MFT and memory
  • full forensics – nothing to add, cuz there is nothing to remove; you go de Saint-Exuperian a.k.a. ballistic on this one & analyze everything & analyze it twice

The conclusion is this:

  • In a typical IR scenario, utilizing tools that are adequate for your task/role is very important
  • You do a MINIMUM first
  • Only, and only if it doesn’t deliver and you suspect you need to go deeper – then you go deeper; $MFT and memory can wait (notably: if you have tools at hand to retrieve $MFT file list w/o much hassle – by all means, do so – it’s fast and it’s better than a file list retrieved by Windows API)
  • In CI/A breaches you better do EVERYTHING you can think of

And to add some real-world case scenarios here: when I worked for a bank, we would sometimes have 400 infections in one go.

Employing full, or even raw forensics doesn’t make sense ALL THE TIME. All you have to do is to get a process list, file list, kill the bad process, remove the  drive-by exploit and reboot the system, verify all is good after a reboot.

No $MFT, no memory analysis. No full forensics.

Think of the Alexious principle:
1. What question are you trying to answer?
2. What data do you need to answer that question?
3. How do you extract that data?
4. What does that data tell you?