Category Archives: Preaching

The art of disrespecting AV (and other old-school controls), Part 2

Posted on by

In December 2013 I posted about  ‘The art of disrespecting AV (and other old-school controls)‘. I saw people retweeting it at that time and was quite happy that it generated some small feedback. It was meant to stimulate some discussion, but also be a reflection on security controls in general – it’s sometimes good to just step back and think a bit more on what they are and how to use them properly.

Well, almost a year later we are witnessing ever increasing number of breaches exposed to the public almost daily (and breaches that are often of gargantuan proportions like JPM) and we also know that they often happened right under the nose of the involved companies.

How could that happen?

Many articles about these breaches mention that ‘alerts were seen, but ignored’.

It was this commentary that triggered me to write a part two.

Before I begin though: when I published the first part someone made a point – which I missed – that AV is an extra attack surface/vector. Indeed, it is. I did not include it in the first part, so I am now mentioning it to tick all the boxes. I do argue in the part two though that this is not such an important factor considering the tremendous & positive role AV plays in a protection of the company assets.

Let me explain.

Over last few years I often witness a ‘romantic’ notion celebrated by the IR/SOC teams which I believe is (sadly) quite universally accepted (and I really do hope I am just imagining it) that:

  • antivirus detecting stuff is a sign of a bad thing, but since AV includes remediation it solves the problem when it removes bad stuff; there are only two major scenarios considered here:
    • antivirus detection followed by removal –> good thing, let’s move on, nothing to see here
    • antivirus detection followed by a failed removal –> bad thing, we better check this out, but it’s probably a glitch/bug in the antivirus/signatures, or we simply need to reboot the host and if it doesn’t work either do some work by removing the malicious piece manually or recommend rebuilding

This is a common approach in many companies. There are also variations of course: alerts from a fixed drive and a removable drive – are they important the same way? Is f.e.x. omnipresent virus Sality or Virut picked up on the removal drive a real deal? Or should we just ‘eventify’ it and move on?

There is a very important bit missing in the approach described above and I already mentioned it in the first part. Once the alert triggers we have a few outcomes:

  • antivirus detected it and removed it – we are good
  • antivirus detected it and failed to remove it – we are not so good, but well, we just need to work on this system
  • antivirus detected it and maybe even removed, but… what if it didn’t detect some extra malware present on the very same system/attached media?

Nowadays when AV detects stuff it often works in a very ‘malicious neighborhood’.

When it does pick up something you should always think of that ‘hood’. So say the AV picked up a PDF exploit, or a JAR file. The fact it deleted it doesn’t matter. Remember that this is a time of professionally written exploit packs and if one doesn’t work, another may. And will. And is most likely delivered the very same moment you received the first alert.

That leads me to a single conclusion which I hope you will start promoting in your organization:

When you see an AV alert you need to triage the system, because it has been compromised + there may be still some undetected malware present on it.

To put it in a more formalized way:

  • A malware present on the system for which you have received an antivirus alert is an incident
  • It is an incident, because the system already got compromised
  • The reason for calling it a compromise is that antivirus alert is equal to a detection of a malicious agent introduced to the environment (drive-by, removable device, malicious insider, unintentional action of the user, etc.); whether it executes or not, doesn’t matter

Handling AV alerts should be a priority for any IR team – while it will be a common resolution for the alert to become just an event or near-miss [e.g. removable device and nothing else ‘funny’ running on the system] it should be treated as a compromise. And every AV alert should be seen as a big compromise waiting to happen. The scary part is that the number of alerts is crazy – it happens daily, hourly, and in large companies every few minutes.

Under the umbrella of ‘business case’ there is always an exposure and rest assured that it’s much easier to exploit users’ gullibility or plain stupidity than spending days trying to find a bug in the AV. Obviously this sort of statement requires some backing in numbers and I can only say that you need to ask your friend administrator what are the statistics on the AV alerts in their company. And then ask if they act on it? You may also compare it against statistics of AV vulnerabilities and how much havoc they spread. And I don’t mean to downplay this aspect of security – it’s just not that important.

I think the main problem of AV is not its presence, but actually a lack of. Over the years due to complains from users it became completely invisible to the users and admins.  It’s kinda ironic, because it is a direct result of the volume and intrusiveness of the alerts. In the end alerts are ignored or misunderstood, users don’t even know something happens. Only dashboards look nice. (Update: just to clarify – by visibility to users I don’t mean the old-school intrusive alerts popping up on users’ desktops; there are more effective ways for users to react to threats – escalation to managers, daily email alert until it’s removed followed by SMSs if 3 emails are ignored and eventually a call from support, etc. – something along these lines).

AV alerts mean trouble, they should be visible to the user and you should look at them regularly and triage every single system the alert triggers on. Sounds daunting. And it is. Perhaps that’s why it is called ‘job’ 🙂

The art of disrespecting AV (and other old-school controls)

Posted on by

Update

Thanks to Kurt Wismer for pointing out a mistake re: false vs. true negatives; I have corrected it below

Old post

Every once in a while I hear people whining commenting about antivirus solutions:

  • “It’s outdated”.
  • “It’s a resource hog”.
  • “Let’s get rid of it”.
  • “It didn’t detect the malware XYZ.”

I sometimes do it too, but… I would never ever remove AV from the company environment. It’s easy to say AV is no longer necessary, or ‘doesn’t work’ . It’s also very tempting to apply one’s localized context of their (often smaller) environment to pretty much every possible situation, but… it just doesn’t work this way.

Let the preaching begin.

What is an antivirus and its role in your environment?

Putting sales BS away, one can say that:

The antivirus’ role in your environment is to:

  • detect known malware using specific patterns/signatures, algorithms, etc.
  • detect unknown malware using heuristics (wildcard/regular expressions, behavioral analysis, file reputation, etc.)

and block/remove it

The ‘known malware’ is an easy bit. You know it, you block it (or kill it).

The ‘unknown malware’ bit is often the major reason for whining, but we should all know that AV is not a ‘cure it all’ solution and there is no algorithm in the world that will allow to detect every single malware. Nearly 30 years ago Fred Cohen prove that detection of viruses is undecidable. The proof will probably make you scratch your head a little, but the beauty here is in its simplicity.

To catch up with the reality, we can expand our definition and say:

The antivirus’ role in your environment is to:

  • detect known malware using specific patterns/signatures, algorithms, etc.
  • detect unknown malware using heuristics (wildcard/regular expressions, behavioral analysis, file reputation, etc.) with the highest possible rate, but with NO expectation of detecting every single malware out there

and block/remove it

Statement like this is far more realistic – people who support this are at least aware of AV weaknesses.

Still, such statement omits one yet very important bit.

The antivirus role in your environment is also NOT to detect non-malicious files.

Ever heard of False Positives?

The availability heuristic makes a lot of people talking critically about AV to focus on the following:

  • False Negatives (malware is missed)
  • False Positives (good file is incorrectly detected as malware).

followed by the obvious:

  • True Positives (malware is detected)

What about True Negatives?

The number of these is tremendous. The thing is, they are NOT reported.

You probably chuckle now. How can we even take it into consideration? After all, we are not paying AV for NOT detecting stuff!

You see, with a decent AV on your system every single operation you do one way or another triggers a background AV check. And it is a far more complex task NOT to trigger on stuff, than to trigger on a very specific pattern. This is especially true for heuristic detections. And this is not to justify AV missing stuff by saying ‘but it’s not detecting the good stuff’. It is just a picture is not full without this bit. If it doesn’t make sense – think of all the posters in Las Vegas showing the casino winners. And then try imagining what would happen if they put the posters of all the losers as well. This is an error of availability at work.

So far we have got:

The antivirus role in your environment is to:

  • detect known malware using specific patterns/signatures, algorithms, etc.
  • detect unknown malware using heuristics (wildcard/regular expressions, behavioral analysis, file reputation, etc.) with the highest possible rate, but with NO expectation of detecting every single malware out there

and block/remove it. The antivirus role in your environment is also NOT to detect non-malicious files.

I guess that’s it.

On a high-level, AV is just one of available security controls that is imperfect by its nature. Interestingly, individual malware detection kinda lost its ‘whoaw’ impact over the years. Malware detections are quoted in groups of hundreds, thousands, and more – and reported in ‘spikes’ on monthly basis. So, right there, in front of our very own eyes we have a proof of AV actually working, and working very well, yet a few missed detections occupy our mind enough to dismiss the benefits of using it. It is true that even a single malware NOT detected can be a game over, but the alternative cost of NOT having AV installed means the chances for game over increase dramatically as its multiplied by hundreds, thousands detection for your average mid-size to large company.

Let’s say it one more time: any single piece of malware that AV detects makes your company more secure.

And about all these neglected hundreds, thousands and more detection reported on monthly basis?

Don’t just stare at the charts. Use it.

In my recent post I mentioned SCCM. AV logs from across the whole company are very useful in finding patterns of malicious campaigns, characteristic patterns, and so on and so forth. The infamous ‘invoice’, ‘fedex’, ‘ups’, campaigns often end up with malware being deployed in a very similar fashion. With such internal intel that can be very quickly pulled from any enterprise-level AV product, it’s very easy to combine it with SCCM and other controls to form a solid detection ‘umbrella’.

The list below shows a few things you can do with AV logs alone:

  • run statistics – pick up systems that are infected over and over again, investigate
  • find systems with detections coming from removable devices – block access to removable devices for repeating ‘offenders’
  • find systems with recurring detections – request rebuild/forensics if you see malware being picked up on the same system many times; AV may be struggling to remove it
  • find systems infected with a highly sophisticated malware (ZeroAccess, rootkits, etc.) – request rebuild/forensics
  • find file names / patterns of detected malware – use it in SCCM queries
  • eyeball the ‘detected & removed’ alerts – most of people ignore the ‘detected & removed’ – once in a while you will find hacking tools, password/information stealers, etc. – these systems are the ones you should be looking at; there may be something juicy stolen from these systems; request credentials changes; these systems may be also compromised by other malware
  • correlate alerts looking at the host names and cross-reference them with a list of important systems – domain controllers, web servers, systems that belong to C-level guys, admins, etc. – anything that processes sensitive data / is a part of a highly critical infrastructure is of an interest
  • cherry-pick systems, fetch samples from Quarantine, run them via sandboxes

The list can go and on.

I would argue that lots of conversations about threat intel, security analytics, clustering, etc. often miss the basic fact that the ‘old-school’ enterprise products provide both data and tools to run lots of correlations and they do it ‘natively’ without any need for BIG DATA solutions or external input.

You heard it right. You can do IR work w/o buying new security controls!

In fact, if you have an AV, proxy/firewall and DNS logs as well as SCCM access you have at your hands most of the data needed to discover a compromise, or a malicious infection. SCCM and AV helps with host analysis, proxy/firewall and DNS with network analysis.The early daily stats can be done with Excel, grep and a bit of scripting. This can be always automated later.

Preaching ends here 🙂