Mono no aware (もののあわれ), Panta rhei (πάντα ῥεῖ )

February 6, 2016 in Preaching

Impermanence. The two terms mentioned in the title of this post refer to it – one is Japanese, and the other one Greek. I have read about the first one recently (during my trip to Japan), and the other one when I was a teenager (in one of the books about chemistry).

20 years ago I was told that I should not be programming or even learning x86 assembly language, because it’s absolutely pointless. I should be programming in Java instead. 20 years later I am still happily looking at, and coding in x86 assembly – my best software project I have ever completed is written in this language. It is only after these 20 years I finally start to feel that it may be a time to retire this interest as x64 and arm assembly are the real deal now. Retire, because technology moved on – the fire is still burning.

A few years back DFIR was in its infancy. Other than specialized shops and law enforcement labs no one really cared about investigations, breaches and analysis. Today the atmosphere is different. The ‘forensics tools’ market once dominated by a single vendor, or two and a few quick&dirty scripts shared by a few researchers, exploded into a never ending avalanche of tools, frameworks and real platforms that are benefiting from the advantage of professional developers being engaged in the process of building them. Five years ago I was writing my own tools. Today I became an user (hopefully not too dumb) of many tools created by others – tools way better than mine.

Commoditization of forensics and reversing tasks, automation, overwhelming number of projects from this particular IT security space that I read about on Twitter every day is literally mind blowing. I want to know it all.  And I know I can’t. It is frustrating and exciting at the same time. Over 20 years ago I’d spend days cracking a single game, writing a trainer, adding immortality, or years before that – learning assembly w/o a book (not joking; I learnt basic x86 assembly by trial and error, spending way too much time than I should on it, and w/o any book until years later; I also had a friend who went even deeper – he learnt I believe Amiga assembly from a German book w/o knowing a single word in German at that time! it is crazy, in a hindsight; absolutely counter productive). The lack of documentation, guidance, mentorship at the time when even hacked info was scarce and highly treasured shaped the whole generation of hackers, crackers, reversers, and script kiddies around the globe. I think these times are now over. It is good and bad. Goal-oriented skimming on the surface replaced the in-depth research that was feeding the bug of curiosity 20-30 years ago. The accelerated evolution of tools makes it possible for a random person to write an advanced file parser or disassembler in a few minutes, utilizing available libraries. But guess what… some guys write these libraries.

What is the purpose of this post?

The curiosity.

Drowning in the overwhelming ocean of ideas is easy. You just let it go. Tweet, retweet, the meaninglessness.

Don’t.

Read.

The only thing that can remain permanent in your life is curiosity. The grandmother of my wife is 85+ years old. She picked up Mandarin at the age of 80.

When your curiosity dies, you… guess what… can’t even bother to finish the bloody sentence. That sentence my friend, is your end.

The art of disrespecting AV (and other old-school controls), Part 3

February 4, 2016 in Malware Analysis, Preaching

This is the third part of the series (part 1, part 2) which this time is somehow shorter, but it is just an excuse to jot down some notes about the actual engines that AV uses internally.

Many people complain about AV using hashes to detect malware – I would say that AV that detects malware via hashes only should not be even on the market, because it would not survive. Your average AV contains a significant number of engines, and subengines using many algos – many of which are lightning fast. Reducing the discussion about AV internal working to ‘AV uses hashes’ is simply not fair.

Let’s have a look – I use the word ‘engine’ quite loosely here and it does not necessarily help with pure detection-specific logic, but it often facilitates the detection itself – each of these are typically quite serious programmatic efforts that are combined to create the ‘holistic’ coverage – yes, it fails, it contains vulnerabilities like any other software, but take a moment to think about the effort that goes into designing, testing all this clustergoodness:

  • static binary string search
  • binary string with a simple wildcards search
  • binary string with a regex (or regex-like) search
  • multi-pattern search engines that are using lookup tables of any sort/trees/tries and proprietary algorithms
  • container/archiver processor – reads files or streams embedded inside the other files/containers
  • file/specific content analyzer/processor – for each file type, content type there is a dedicated engine f.ex. MBR, old Dos .COM file, Flash, OLE files, Symbian SIS, ISO, etc. – note that many of engine expire due to technologies being no longer in use/popular, but it is _there_
  • unpacker  – decompresses streams of data to present them to other engines
  • emulator – simple state machines with a basic understanding of some opcodes
  • emulator – full-blown emulator with most opcodes supported
  • sandbox – full-blown emulator with support of API & memory
  • hooks – dynamic, for on-access scans
  • heuristics engine
  • whitelisting engine
  • detection engine based on file properties
  • rootkit detection engine
  • native file system parser (for various file systems)
  • memory dumper/file rebuilders
  • online scanner (virustotal-like)
  • behavioral engines
  • reputation engines
  • quarantine engine
  • crc/incremental crc search
  • hash-based search
  • entropy analysis
  • X-rays
  • and finally… removal and repair engine – if none of the above engines impress you… think for a second what effort goes to ensure you can remove a complex polymorphic or metamorphic file virus from a gazillion of files on the system without corrupting the files and crashing the system.

There are probably others which I forgot about, but this is really a lot more than just hashing.

If you talk about AV detection and the only thing you talk about is hash, it is probably because you smoke too much of it… :)