You are browsing the archive for How to….

How to become the best Malware Analyst E-V-E-R

April 14, 2018 in How to...

Update

BIG thank you to all the reversers who provided the feedback so far; I have reviewed and updated the content based on this feedback, and hope this will be now more useful to all new guys; reversing community has a long history of being awesome and it shows. AGAIN, BIG THANK YOU!

btw. the title of this post is a tongue-in-cheek ūüôā

Old Post

There are a couple of reasons for me to write this post:

  • I have been doing software analysis for a long time – 20 years, at least
  • For many years I have been doing it wrong
  • For many years I had absolutely no clue I was doing it wrong
  • In last few days a number of things happened
    • I see ‘malware analysts’ doing more and more Bat Deductions¬†(this is from reading various posts from the security vendors, I won’t name them)
    • I have spoken to people who lied to me about their reverse engineering skills and I am trying to understand why + what stops them from actually learning the skills
    • I have publicly criticized GREM – not in a malicious way, but more as an observation that certifications != skills (apologies to Lenny Zeltzer who is doing lots of the GREM work and I might have sounded critical of his course; my point was merely that a few days of even most detailed course won’t replace hours of … well… messing up your analysis and often going nowhere)
    • I was actually approached by a number of GREM-certified people to help them with some reverse engineering/explanation of what they do/see
    • Malware analysis is fun, but only if you know what you are doing; I was lucky to learn from many smart people and at the times when internet was not so prevalent, so I was recently asking myself a question – what would be the steps I would take today, knowing what I know, to learn RCE skills in the most efficient way possible

Let me start with an old boring anecdote. One of my first experiences as a reverse engineer was looking at the code of demoscene productions. That was an incredibly stupid idea to pick it up as a target, but I was young and well, stupid. And yes, I didn’t know better. What I remember was that I got obsessed with the demoscene tricks and I really wanted to understand how they are done – in this case, how a certain particular full-screen scrolling effect was created.

I eventually learned how they pulled it off (that was a special mode of the VGA card – a variation of the mode X IIRC – that allowed to write 16 pixels with one stosd operation if you must know), but not from my reverse engineering efforts. These ended up after few days when I realized that for these couple of days I was staring at a highly packed code and data which I disassembled with a pirated disassembler I found online (FWIW it was not IDA). The code was very misleading to me, because the disassembler added a lot of ‘useful’ comments to it; these comments sent me down the rabbit hole pretty quickly e.g. they hinted that some of the code was using DMA channels, so I hypothesized that perhaps the quick data transfer I was observing on the screen was due to the DMA usage. Oh boy… So… Wrong. Wrong. Wrong. It was that special VGA mode, cleverly used. What I was looking at was a non-sense gibberish that looked like a code.

I wasted a lot of time.

I have repeated the very same mistake MANY times. But… there is no other way to learn reverse engineering, a core skill for malware analysts. You just need to look at a lot of listings and files/databases with the disassembled or decompiled code. Over years I was cracking software, adding immortality to games, shortcuts to pass from one level to another, analyzing malware and pretty much always finding something new to look at every day. Yes, no matter how good you are, how many hours you spent with Ida, OllyDbg, Xdbg, and windbg, you are going to come across new stuff all the time. That’s actually the most attractive part of being a reverse engineer. You break stuff apart, and this stuff can be a PE file, a word document, an exploit inside the Adobe Flash, a COM trickery, an API trickery, a new functionality abused for malicious purposes, a closed-source project that can be only found on ATMs, a stuxnet-like nested doll, a clever php backdoor injected inside a legitimate HTML page, a JavaScript snippet activated when the user presses the Submit button, an AutoCad program, a jailbreak for iphone, a powershell snippet, a VBS badness, a HTA deliverable, a good old Office macro, a tricky URL, a tricky protocol, a side-loading attack, a supply chain attack, an insider threat modifying the affiliate ID, and old game that you want to port to a new platform, a vaporware where you want to add some missing functionality or bypass some annoying bug or disable a message box … Basically, whatever that can be changed for good or bad, abused or misused, and sometimes just understood, for the sole purpose of ‘knowing it better’, and all of it on a software level falls into this ‘software/malware analysis’ category…. and discovering it, analyzing it, breaking it apart and understanding the motivation of the guys behind the code is a source for a great intellectual fun, and fulfillment, even if most of the time, we are half-guessing…

So… without a further ado… this is my list of rules…¬† if you want to become a good malware analyst, I’d suggest following some of them. Don’t be shy. Ask for help. Describe what you did. Where you got stuck. There is always someone who will be happy to help. And again, and most importantly, remember that most of the reversing work is never finished. Yup. We start, we abandon it pretty quickly. We come back to it. I have some projects I opened 10 years ago and I sometimes come back to when I am in a mood. In many cases only if you are paid proper money it is worth doing further analysis. And the latter is what drives what I like to call a ROI-driven malware analysis (or, in general terms, software analysis that has an actual end & deliverable that keeps your man hours low, client satisfied, and everyone aware that we are just scratching the surface, but in an educated way).

So… again, without a further ado:

Rule #0:

  • Waste a lot of time
  • There is absolutely no way to take a shortcut in reverse engineering
  • Yes, I am sorry, but it’s like with everything else… 10000h rule¬† stands true. BUT. like with everything in life, you may try to beat this number down!

Rule #1:

  • Don’t trust your tools; they will mislead you, they will betray you
    • Note: on day to day basis you can trust tools most of the time, but only if you made an effort in the past to understand what is that they are showing you, so that you can spot, or at least suspect bugs when you see something unexpected, or wrong
  • In particular, don’t trust any automation (automatic comments, sandboxing, IOC extraction, assessment, etc.); use these ONLY after you learn manual analysis
  • The automation (emphasis ‘at the beginning of learning’) is for the lazy, you can only use it when you know the caveats
    • To clarify, if you know how sandbox works, if you are just after behavioral stuff (IOCs), then sandbox analysis is often the best way to go; the point I am making is that if you don’t understand the caveats of automatic analysis, you will not be able to fully trust the sandbox output (or from any other tool really); also, depending on the case, you may want to confirm the sandbox findings manually (ok, for the caveats: anti-* routines, various execution paths depending on time, command line arguments, detected OS, presence of the internet, C2 address still existing, presence of targeted software, etc. – yup, you may get more than 1 different reports from the sandbox for a given sample, depending on the circumstances)

Rule #2:

  • Data is code, code is data
  • If you don’t know what you are looking at, assume you are looking at data.
  • Any attempts to disassemble it are pointless unless you know the context, or have a hint it is indeed a piece of code.

Rule #3:

  • There is no such thing as static analysis
  • Let me explain…
  • You calculated hashes, you ‘extracted’ strings, carved embedded files, you googled it, you checked VirusTotal, reviewed the sandbox report and you think you have done well.
  • Maybe you even used some PE Viewer/Editor….
  • Well, you have done what any automated system can do.
  • Try harder.
  • What many people refer to as static analysis*** is primarily FILE Analysis, not PROGRAM analysis
  • This is a TREMENDOUS difference
  • You can craft a file to look like Notepad yet still deliver a malicious payload
  • Files lie, code doesn’t
  • Seriously… whatever you see in strings, PE tools, it’s just a wrapper; always assume it’s a wrapper that will fool you; if you trust it, there are always chances you will be taken for a ride
  • ***Now, I need to explain one thing – a number of respected reversers pointed out that they do lots of static code analysis; I think this is the key here – they mention ‘code’, and the reason I am talking about non-existence of static analysis is that there is that TREMENDOUS difference between analyzing a file, and a code that this file stores; static code analysis (SCA) exists of course, and is actually a big industry (ever heard of Fortify?); so, anyone reading _code_ w/o executing it is doing SCA… as a reverser you will do lots of static code analysis

Rule #4:

  • You must look at the code and must think like the coder who wrote it. You are building the image in your head, code blocks take some shape, shapes create a pattern, they align or not, you grasp it, or you don’t. You are basically writing or rewriting this code in your head.
  • When you look at the code, you emulate it, you guess, you connect to it (from the basic instructions to code blocks)
    • As you do analysis e.g. in IDA, please make it a rule to label everything you see in the code as you walk through it. It saves a lot of time to know you have visited a certain path, routine, etc. even if you don’t know what it does, make a guess and name it in appropriate way e.g. ‘unknown_possibly_reading_file_xyz’, or ‘nothing1’ – it’s always better than ‘sub_xyz’
    • You will be surprised how these labeled functions ‘add up’ to your understanding of the code e.g. naming memory functions both for allocation and memory disposal often ‘fill-in’ the listings of many functions that rely on these functions quite well
  • You may not like it, but if you are not a programmer, you can’t reverse engineer well.
  • A programmer runs the program or idea in their head, many times, and this is not called static programming, oh no… it’s a daunting technical and algorithmic challenge. It’s an obsession. Reverse engineering is even more painful as you need to go into that programmer’s head and understand _why_ they did certain things the way they did + you have to deal with what compilers produce. And it’s a code stripped of lots of information. And sometimes purposefully obfuscated.¬† Most of the time this is a painful exercise. This is why when you ‘get it’, you get that awesome ‘I cracked the puzzle’ feeling. I-t¬† i-s¬† p-a-i-n-f-u-l, but there is a reward.

Rule #5:

  • Do not use tools unless you used pen at least once.
  • Read PE file specification (same applies to other file formats, but let’s start with PE)
  • Look at a random PE file (calc.exe, or whatever), print out the first page of the hex dump, and use pen to highlight structures you _manually_ recognize based on the PE format documentation. Don’t be lazy. Do it. Then use tools to confirm. If you make mistake, analyze where you made that mistake. Step back, re-assess. Ask others.
  • Don’t watch videos on reverse engineering unless you know the basics. Videos are for lazy. You need to get your hands dirty. Asap.
  • I will repeat it ad nauseam. Don’t rely on tools from the day one. It will cost you long-term. Use pen. Write stuff down. Make mistakes. Correct them. Nowadays I often use an editor, but that’s because we have virtual machines and you can make mistakes inside the VMs that won’t crash the host system; in the past, when a mistake meant a system crash or hang, I had a lot of paper around me with notes, and lots of hexadecimal stuff written on it
  • If you have done the manual analysis at least once, you will be:
    • able to understand what you see or look it up
    • able to spot tricks (in a file format, in a code)
    • able to write tools
      YAY! if you can write tools you will be always ahead of the curve… the best example is given here by @hasherezade – she is producing lots of Proof Of Concept code and she often writes it for TESTING purposes (to see if she understands the technique properly and to test other tools)

Rule #6:

  • The best reversers are not necessarily malware reversers
  • The best reversers often started by poking around in other people’s software
  • In the past they didn’t have tools that are available today; they spent hours, days, weeks, months staring at some code, trying to break it
  • Some of them (e.g. Rolf Rolles) can do what some refer to as ‘Zen reversing’ – they look at the code, and can instantly recognize what it does; I actually have a ‘personal’ list of people that I almost religiously respect for the reversing magic they can do (e.g. vulnerability researchers, but also coders and reversers who often understand advanced math and apply it to build better tools)
  • Like anything in the world, it requires lots of hours spent on training; it’s gonna hurt

And on a practical note…

Reverse Code Engineering (RCE) is getting really popular and is really needed. It is helpful in malware analysis, debugging your own apps, solving crackmes, fixing bugs in abandowanware, and it can be¬†handy in localization.¬†It makes you a better programmer as well. Of course, it also helps to steal and plagiarize code, bypass software protections,¬†discover vulnerabilities, write shellcodes and jailbreaks, reproduce stuxnets, rootkits¬†and make people’s lives miserable and/or interesting in many other creative ways. All of it is either coding, or reading other peoples’ code, and repurposing it. Again, you better be a programmer to do reversing efficiently. The code blocks in malware are the exact code block used in a legitimate software. The old posts on e.g. CodeGuru and CodeProject are often leveraged in malware creations. Often, pretty much in a copypasta way. These posts are often very detailed. Read stuff that is still there. StackOverflow can wait.

So, back to the original question: how to learn RCE and/or malware analysis quickly?

There are many answers online and they vary a lot. Many people suggest books, tutorials, ebooks… on IDA, on assembly, on Reverse Engineering in general, some suggest doing courses and certificates (including GREM), others watching youtube videos and some advise new adepts of RCE to simply stop wasting their time.

I would like to provide you with my own version making it as minimalistic and practical at the same time as possible. Yes, it is not full, yes it is far from being perfect, yes you are not going to analyze rootkits just yet (and yes Рit is Windows oriented).

But…

If you read the stuff I point to and really focus on spending a few hours/week on actually making tones of mistakes plus avoid claiming victories easily achieved by using automation and tools developed by others, you are going to get there before you even realize:

  • Decide if it is for you; seriously… reversing is a terribly mundane process… it’s actually like forensics, and probably no one will tell you how boring it can be, but it’s HOURS and DAYS spent on staring at the same screen, browsing items and trying to understand it… sometimes you hit a jackpot and crack it in 5mins, sometimes even 2 weeks of intensive reversing won’t bring ANYTHING useful… so… you have been warned
  • You need to learn about programming in general and actually start coding. I repeat: you can’t reverse engineer if you don’t program. How can you understand what you see if you don’t know what a loop is, a recursion, or statically or dynamically linked code is. It is simple as that. If you programmed before, move on to the next point. If you didn’t – don’t buy heavy C++, C#, Java, Python reference books¬†just yet. Buy a book with silly, but practical examples of simple programs explaining the fundamental architecture of Windows. Try this classic book from Charles Petzold. Read it inside out, and take your time to actually _type_ the code listings. Yes, you heard that right. It’s mundane, it’s error-prone, yet this is how learning to program works. The only way is through a keyboard so, get ready to invest quite a lot of time¬†–¬†you will be fixing typos, compiler errors, getting completely unpredictable results and will encounter a lot of pain and stress as you go along.
    In any case, DO NOT START WITH JAVA OR .NET. Sweat a bit with C, even scripts in VBS, bash, and powershell.
  • Read other peoples’ code. Skim through it, and if you find something interesting, read more thoroughly and ‘get it’. Again, no need to understand everything, but if you want to understand, google around until you do. No, do not start reading Linux code just yet. Start with short code snippets on educational web sites. Look at the source code of some small, but interesting and potentially malware-related projects. Just see how people do stuff, try to figure it out. This is¬†actually the most¬†crucial part of reverse engineering – it is not only about reading the code, browsing through listings, spotting known APIs, running ‘strings’ on a file, or playing around with ‘Procmon’, ‘Dependency Walker’ and ‘GMER’. It is trying to wear authors’ shoes for a moment. If you can figure out his or her thought process that led to this and that implementation, you¬†will be¬†making a huge progress very quickly. Bonus: when you notice some code blocks, they will stick to you, so next time you see similar code, you can make an educated guess. Didn’t I tell you it’s all about an educated guess? Yes… ROI is important. You don’t want to disassemble the whole 1.5Mb binary.
  • Learn a small subset of x86 assembly language.
  • Choose easy targets first. Look at compiled sources from Iczelion projects, compare them to the source code, look at programs for Windows XP (things are not complicated on this system as they are on Vista/W7+), look at old software; nowadays, the programming is pretty complicated from a reverse engineering perspective (lots of RAD tools that use lots of wrappers that hide the actual program’s code and analysing it manually is a pain in the butt)
  • Refer to MSDN often. Anytime you come across a new function name, either google or MSDN it. make sure you read the concepts associated with the function (usually functions are associated with some ‘high level’ topic e.g. CreateFile with File Management). Seriously. Read the full description, don’t be lazy and it’s okay if you don’t ‘get’ everything in one go. Bits you pick up as you read stuff will provide you with an invaluable insight in the future.
  • Only now start googling for tutorials on how to reverse/crack/debug applications or buy books that will expand your knowledge. Yes, reversing requires a solid foundation from many aspects of IT; if you don’t know these basics,¬†you will continue to be a tool user and no youtube video or book on IDA can help you here…
  • And the good news… read about forensics. When I started it was almost a non-existing branch of IT (Sec); now it provides a crazy amount of information with regards to artifacts and what is ‘good to know’ about the systems you look at. It classifies and solidifies knowledge about what is that malware does and what parts of the system it affects. Instead of knowing everything, you can focus on areas that are the most exposed, malware-wise. The Art of Memory forensics is gold. Read it.
  • The Art of Memory forensics will bring you closer to the OS internals; The OS internals and the system architecture is a must-understand bit when you do malware analysis; you need to learn about objects, files, Registry, but also mutexes, semaphores, memory layout, a difference between process, threads, fibers, process environment, process environment block, structured and vectored exceptions, and so on and so forth… Read Widows internals¬†book
  • When you get a bit more experienced, read through corkami repo…. Ange is one of the best reversers I know personally and he is the magician beyond many tricks affecting a huge number of file formats; apart from Rolf Rolles he is one of my ‘zen reversers’ that I follow pretty much religiously
  • So.. yeah… join social media, groups and just start following reversers… there are lots of very good reversers on Twitter – find them!
  • Get used to the fact you will be re-learning reverse engineering often. Your tricks, and your tools expire. New file formats, modification to existing file formats, new programming frameworks, new obfuscators, new tools, it’s new stuff all the time (okay, maybe with the exception of macros in Office ūüėČ

So… How to become the best Malware Analyst E-V-E-R?

You can’t cheat here. You need to do hours. Many hours.

btw if you are interested in SOC basics you may also try How to become the best SOC Analyst E-V-E-R

How to become the best SOC Analyst E-V-E-R

February 25, 2018 in How to..., SOC

Update

Since I am getting a varied feedback on this article, I want to clarify a few bits:

  • I tried to incorporate many things that I wish someone told me in the past when I started working tickets
  • If it comes across as preachy or hierarchical it was not the intention – seriously… whoever works tickets knows that many things here are real world examples of issues we have to deal with
  • If you take an offense reading about hierarchy, and me positioning myself as ‘mentor’, and calling junior people ‘grasshoppers’ – c’mon… this is all corporate language at work, also I must mention I am not a native speaker and I may be missing some nuances; people who know me personally know that I am the first one to admit an impostor syndrome; I also want to emphasize that I do recognize the expertise of people I happened to mentor – they teach me me back a lot and I refer to it in the first paragraph (I can easily bring up some anecdotes from my past where one of the juniors has demoed some really cool stuff about mobile forensics – an area where I don’t have much experience – it was a reversed role, I was the grasshopper and she was the mentor there).
    Also, pretty much every single corporate org assigns buddies, and runs mentor programs where juniors are mentored by others; yes, it’s definitely arrogant to say you are a mentor and they are mentorees, but please give me a more accurate English word to call that relationship & I will update the article; and yes, there is SANS Mentor program too; semantics is a fun topic to explore, but if the only feedback you give me is that you don’t like my ‘entitled’ tone, then thanks for assessing the 4-5h of work that went into writing this post with your linguistical nitpicking
  • If you take offense in profanities, rest assured, I have removed them for you – however, if you have never called a ticketing, AV, EDR, IDS/IPS software a piece of @#$%^&, this post is probably not for you anyway(?); hmm there is not much space for political correctness in the necessarily emotional piece that this post aims to be…
  • When I refer to a SOC Analyst, I mean a junior (in terms of skills in DFIR) person, not a hardcore, technical person that used to be called SOC Analyst in the past (typically working on network side, probably today referred to as NOC)
  • So many people in our industry focus on the cool stuff: certs, EDR, data science, threat hunting, memory forensics – pretty much L2 and L3 stuff; I find that there is a huge, often misunderstood and definitely unfairly placed burden on the L1 analysts – they are very rarely respected, the tools they are forced to use suck, and the analysts are also often incorrectly guided to do their work, and to grow (I know I generalize, but this is what I came across in the past)

Old Post

Over last 15 years I had a privilege of training and mentoring a number of junior people, often much smarter than me in their own areas of expertise (programming, network, admin work, routers, support for various tools/devices, etc.), but still either completely, or partially green when it comes to SOC/IR/DFIR/RCE space. I will lie if I say I was always happy about things these ‘grasshoppers’ were doing, but it would also be a lie if I said I was always right. And well, I was once a grasshopper myself, and definitely (and perhaps sometimes even intentionally) pissed off more than one person with my stupid shenanigans, and probably still do…

We all start somewhere, and we (hopefully) continue to grow.

The fun of leading and mentoring teams is actually there – it is the individuals that you come across while doing this work; believe it, or not, among them you will find real gems, and I mean it!
– these guys will teach you back many things you don’t even have an idea exist.

Interestingly, as our knowledge gets pretty rusty and it does so incredibly quickly, I actually find the cooperation with the security youngsters very rewarding – they make me stay up to date – they are asking questions which in return force me to actively explore the vast areas of my… ignorance.

Yes!

There is more to it out there than the actual expertise. This is because our field expands kinda exponentially and while I may be focused on Wintel attacks, there is the whole domain of *NIX, mainframe, IoT, or even DDoS, or appsec attacks that I am either not aware of, or barely aware of (I am actually a ‘grasshopper’ in these fields). It would seem our ITSEC expertise is more about indexing things we don’t know about…

So… the good news it that some of these guys live it the same way you lived it 5-10-20 years ago. Yup. Good to talk to them and… learn.

Today’s post is not about the glorious day of enlightenment though. I thought I will write a quick tutorial-like piece, or a check-list if you will with the aim of helping new analysts who enter the L1 function w/o much experience in the field and perhaps wanting to grow as security professionals.

If you are still reading it I assume now that you are the newly-hired individual in the L1 SOC function – you may be a SOC Analyst, SIEM analyst, Junior CERT Analyst, whatever. You just joined the company directly, or act as a part of MSSP company, and perhaps (in some cases) are with¬† your company for a few months, but still don’t have much clue what’s going on. Let alone know ‘how to look like you are doing something useful’.

Also, chances are that you are probably already hating your job, and your manager, or your employer ūüėČ

Because… these stupid tickets.

This is practically what you do all day long. The big words and ravishing dreams of being a hacker, computer security expert, one that perhaps one day will have 1 million followers on Twitter, and even be invited to talk at TED, one that people worship… one that maybe even FBI arrests… one that is so special…it all fades away with that every, single ticket you have to open, fill-in, and eventually close, or leave for future generations to sort out.

Don’t despair. You will get over it. We’ve all been there.

One of the first questions I typically ask you – what is the most important thing while doing tickets. You don’t know me yet – you just know I am your lead/manager. You take my question very corporate-seriously and answer that it is indeed a very important task, and that it is incredibly important to do the tickets right, and give your best to add / enrich ticket information; and perhaps you already know, or have overheard that ‘if auditors read data from YOUR ticket when it is provided as an evidence they may ask questions’, so you may add that we obviously need to do tickets well enough so auditors will be happy.

And of course, you want to improve, learn from mistakes, and will definitely work harder to get better at working them in the future!

These are all nice and cozy answers, and I am grateful for a second, and a second only, because while I may be the almighty know-it-all boss du jour, you are just feeding me with the stuff that my vanity happily feasts on.

So I steer you towards the one and only proper answer – one that often surprises you, and then we have a good laugh – because for the first time we talk not like the boss and the employee, but like two brothers in arms, and human beings really. And that’s because I worked tickets before and the answer I give you is this:
– the one, and only, most important thing while doing the tickets is actually NOT opening them, or if you really have to, opening them, and closing them as soon as possible! Escalations, external dependencies on other teams – will kill all the fun, SLAs, and often expose you to scrutiny.

A-HA!

We both know doing tickets is a nightmare. Most of it is a mundane, absolutely idiotic waste of time, a busywork, a grass painted green over and over again, an unexciting, repetitive, boring task.

BUT IT HAS TO BE DONE!

Don’t despair. Everyone knows it. Everyone hates it. And this post will help you to deal with it.

The TL;DR; answer is this: convert this ticket hamster wheel to a game with a purpose.

The game with purpose is one of the creations of the mind of Luis von Ahn (the CAPTCHA and Duolingo guy).

The ‘purpose’ that works for you is your own business really – you have to kinda work it out by yourself; and the ‘game’ bit is like any other game – who cheats, gets there first!

My personal purpose while doing tickets was always and still is: WIN TIME.

The rationale is pretty simple. We actually die. Yes, we do. Having a time that is gained by efficient management, or any sort of trickery is your life being utilized in a better way; a time is a luxury; let’s be frank: any ‘free’ excess of it – is a gift.

When you have time, you can dedicate it to more rewarding activities. They should be strictly related to your work, of course, otherwise you will get fired, but perhaps with a bit of planning these free cycles you win may actually open doors for you to areas that are little more interesting than the tickets themselves. And more rewarding indeed, because what if… the time you earn by being efficient allows you to focus on the next step in your career?

Isn’t that worth it?

So… without further ado, here’s a list of ideas I have used/explored, or otherwise practically used in my past lives; this is by all means not a full list and I am sure some of my bros-in-arms or sis-in-arms can chip in and add their 5 cents of advice – I sincerely hope so:

  • BEFORE YOU BEGIN
    • What is that you do?
      • Fill-in the tickets? yeah… but…
      • Provide enough info to L2 (CSIRT/CERT/SIRT/etc.)? yeah.. but….
      • Think of your work in consulting terms;
        • Who is your client? Who will read your tickets?
          Typically L2 (CSIRT/CERT)
          Typically your lead/manager
          Sometimes no-one specific (other than your system of record storing it and compliance guys providing it as an evidence to auditors)
        • What is your scope of work and area of influence?
          L1 work only
        • Are you able to escalate the problems to anyone if you don’t understand something?
          Talk to people who have more experience than you! Show them what you did, why, and you will get a good feedback & support!
          Technical people love to explain, even show off. Ask direct question. Don’t challenge them or assume they are uneducated. Most of the time they have done L1 work before and are fully capable analysts that can GUIDE you. Also, they sometimes may be wrong, so let them know when you spot their mistake!
      • So… Ask questions!!! Many of them. Interact!!!
        Bore the L2 and your manager to death. It’s their role to GUIDE you.

        • “Why do we do things the way we do?”
        • “When I look at the data, what parts of it are important?”
        • “What is the meaning of all these fields in the proxy logs?”
        • “I did this and this and that. What else could I do?”
        • “Where can I read more about it?”
        • Ask
        • Ask
        • Ask
        • And then nag them to formalize it in a playbook, process guide
    • Read about The Alexiou Principle¬†
      • What question are you trying to answer?
      • What data do you need to answer that question?
      • How do you extract that data?
      • What does that data tell you?
    • Once you get the above, you are actually en route to become a very good investigator (L1 –> L2 –> L3)
      • i.e. the one that is a very focused minimalist, remaining objective, and his/her reports ‘write themselves’… these that ‘fly’ through the vast amount of data and can spot the bad guy with ease
  • TICKETING SYSTEMS
    • FACTS:
      • Most of them are absolutely idiotic, over-engineered, clumsy old-fashioned, inflexible and bloated, slow pieces of software; often designed in late 90s, or early naughties; they are often a piece of software **** really
      • Why?
        • Because they rarely follow the SOC/IR workflow
        • Because their UI sucks
        • Because it takes forever to fill them in, and they typically don’t work on many browsers, so you are stuck a.k.a. bound to use the one and only required (a.k.a. supported) browser
        • Because they were really not designed to handle security alerts, bundles of such alerts (campaigns), and do not scale well
        • Because they were not designed to support Data Across Border and GDPR issues
        • In MNCs, apart from DAB and GDPR issues, you have the problem of access; it’s a difficult problem to solve permanently for SOC analysts across the whole globe; and it’s really very tempting to store it all in cloud that anyone can access
        • Many older products do not support working on multiple tickets at the same time; yes, 1 cookie=1 ticket… it’s incredibly inconvenient
      • Editing itself, or pasting to ticketing systems is often a miserable job:
        • You edit it, you paste it, formatting is immediately ruined, f.ex. things are not aligned properly, etc.
        • Unfortunately, ideas promoted by ‘web-oriented’ companies force us to believe the web editing will take over; no, it won’t; it’s still a miserable experience and will stay as such for a long time; especially where you have to fill-in a number of fields
        • Unfortunately, nowadays all of ticketing systems are web-based. sigh…
        • I have been waiting for it for over 15 years and it’s actually getting worse and worse
          • My experience went from a blaspheming Lotus Notes-based bug tracking systems, through awful Java/Oracle-based Frankenstein’s Monsters, then mediocrity of Flash/Silverlight idiocy, and now we arrived at the Web 2.0 (soon Web 3.0) and ‘new’ UI concepts and paradigms that force you to click 20 times to do a _simple_ thing, because everything needs to be animated or expanded/drilled-down numerous times (think of ‘Read More’ buttons that shows you ONE additional line of text), and on the way break all the UI design metaphores that we so got used to using the native OS applications over last 20-30 years!; web broke all these metaphores
          • The guys who design IR ticketing systems really need to understand that less clicks to close an IR ticket=win for the analysts!!! Also, they need to understand what data is really important to show in previews – IR ticketing is really about speed-closing
      • Saving work – you will often find that some fields are mandatory to fill-in
        • Sometimes you forget one and you may try to submit the form – a weird error message shows up; then you have to edit it all over again
        • Sometimes all the work is gone w/o much reason, or when you submit and your broadband/VPN goes kaput
        • Sometimes the page will refresh out of nowhere
        • Sometimes you will click a wrong mouse button, or press a wrong keyboard short-cut – the browser will take you to another page
        • Sometimes you accidentally activate a shortcut by mouse, access history, click it, click a bookmark, your VDI will freeze, or whatever random happens
        • And yes, this really happens – your work is gone in a blink of an eye and there is no way to recover it
        • So.. thank you stateless web UI experience based on an assumption users don’t make mistakes and there is no VDIs, no network bottlenecks…
        • The Windows OS UI as hated as it is by many provides a very intuitive way to fill in all this stuff and supports automation VERY well (think: TAB key used to change the focus between fields on a Windows dialog box vs. fields on the web page that need to be activated by mouse each time; the first one can be driven by macromakers, the second _sometimes_ by bookmarklets)
    • HOW TO CHEAT:
      • CREATING TICKETS
        • If possible, use automated ticketing system (mail comes in –> ticket, with an appropriate data enrichment) – no need to do copypasta
        • If automation is not possible, use boilerplates/bookmarklets/macro makers/grease monkey scripts/dedicated in-house built browser plugins to fill-in the most typical cases; use the templates/bookmarklets (read below), and manually correct fields that require correcting for less usual cases
        • If there is an API (REST, SOAP), task someone to write a nice python/ruby/perl script to close tickets that can be closed automagically
        • Same as above, re-prioritize the ticket via script, if possible
        • If you have an access to a dev team provided by ticketing system vendor – USE THEM – while they charge exorbitant money they can do a lot of tweaks that you don’t even know exist (because they have done it for 200 other clients and know what issues to fix/what problems to avoid)
        • Ask your manager, L2
        • Google your ticketing system – see what other ppl do
        • Nmap and ‘light pentest’ your blackbox ticketing system (perhaps SOAP or other ‘goodness’ is there? /’goodness’ is a relative terms/)
      • ¬†EDITING
        • Edit the content of the ticket body in an external program, if possible; it can be Notepad, Notepad++, Ultraedit, Textpad,¬†SciTE, VIM, WordPad, WinWord – whatever works for you; keep these notes in a file that you frequently save; work on it in ‘append to its end’ mode
        • Using external editors gives you a full-power of the editor, not some web-based wannabe with very small input fields, limited formatting, non-resizable fields, broken Undo /even Confluence fails here sometimes/, no proper support for navigation in the editor, no support for vertical editing, no search&replace with regexes, no support for multiple files search and replace, etc. and no room for mistake really….
        • Having to fill-in a dozen or a few form fields is not for faint-hearted; simplify it with a macromaker/bookmarklet/API, if possible – I mentioned it before, but will repeat it a few times; optimize, automate, cheat using GUI-automation tools
        • Use more than one editor, if needed
        • Learn to treat Excel as one of the editors; the amount of stuff you can do either with Excel itself, or by combining the strengths of Excel, with a good programming editor e.g. Ultraedit, or cygwin / unix tools, and sometimes bit of scripting will really surprise you!
        • Learn to program, script, write macros in general. This goes a long way. This is probably the most important advice. Think like a programmer/coder – you will get far by cheating manual editing with some simple algo to do work for you.
          • Note – editors allow macros/templates; you could define them and use them for various purposes
        • Use keyboard more than mouse, if possible!!! (can’t stress this one enough; LOOK UP shortcuts for the programs you use, including web pages e.g. ‘E’ on Confluence enters the ‘Edit’ mode; ‘//’ enters a date, etc.)
        • Outlook allows you to select emails and then when you copy them to clipboard, you can paste their ‘main’ headers (as per set up view) to excel
        • Use VBA and Office Macros!!!
          • They are not just malware, they offer awesome automation functionality you can use to speed up processing stupid things;
          • Example:
            • Exporting batch of emails from Outlook so you can process them with a python/perl/ruby script afterwards
            • This is a great time saver (e.g. instead of looking at every email, you extract all every say 24h and run through your script that excludes white-listed entries, highlights blacklisted entries, does data stacking, perhaps saves it in a local database for future comparisons, etc.)
            • Doing stats on Outlook folders
        • Again, Ad nauseam, read help for your editor of choice; learn keyboard short-cuts; read it again
          • I keep learning new things about Total Commander or Excel every day, despite using it for 20 years!!!
        • Install Puretext – it’s a brilliant small program that allows you to paste w/o formatting
        • Another similar program is Ditto (http://ditto-cp.sourceforge.net/)
        • If you paste data from tickets in Outlook, use CTRL+ALT+V shortcut to choose pasting type
        • Learn Excel shortcuts
          • ctrl+; for entering the current date
          • ctrl+: for current time
          • ctrl+1 for formatting
          • shift+F8 for selecting all content within a block
          • ctrl+pgdn and ctrl +pgup to navigate sheets,
          • etc. –
            it’s much faster;
          • then check vlookup, pivot tables, “&” for concatenation, etc,
      • TRIAGE
        • Stick close to the source of the alert
          • i.e. research logs RELATED to the event;
            • by the time frame
            • by the host name
            • by the user name
            • by the risk name
            • by the phishing campaign
            • by the type of event
            • etc.
            • Don’t go ballistic on VT/HA/malwr/virusbay/urlquery if you don’t know/understand what happened on the box itself!!!
            • Threat Intel and blog posts are OVERHYPED
              • Don’t read blog posts until your logs tell you what happened; until you have some hypothesis to verify
            • Again, for that, stay close to the event that triggered the alert !!!
              • Think ‘what happened there’ before you go with further triage/analysis tasks
            • So… don’t go to VT, HA, or googling around in general as a first ‘itchy fingers response;
              • It’s all helpful, but it’s not really answering the 4 of Alexious principle questions
                plus
              • Vendors are sometimes vague
              • Vendors are sometimes wrong
              • Peers in the community are sometimes vague
              • Peers in the community are sometimes wrong
            • If you see an AV alert – go to your vendor page – UNDERSTAND what the threat/risk name tells you; is it a virus (modifies files), is it a ransomware (encrypts files), is it a heuristic detection (often FP), etc.
            • Join groups, forums (e.g. OWASP and local chapters/meetups, SANS, FIRST, various CERT notification emails, FS-ISAC, etc.), Twitter – learn about ‘new’ before media outlets pick it up
        • Look at the AV, IDS, Proxy logs;
          • Look at as many sources as possible;
          • And if it may sound weird… avoid firewall logs at first, unless absolutely necessary;
          • Why?
          • Attacks shifted to higher levels of TCP/IP stack, lots is happening on application layer
          • Don’t underestimate firewall logs though, you will definitely find them useful in more advanced attacks (they may be actually the only logs you will have to prove someone stole your data!)
      • ¬†TEMPLATES/BOILERPLATES
        • If some tickets are the same, or nearly the same – keep a boilerplate with a text already prepared for a copypasta; this will save time, and will ensure you are consistent (this is why writing stuff using an external editor is also handy – you can keep all your edited stuff in one place; yes, create a ‘book’-like document and type your stuff there; you will notice patterns, and you will re-use existing text more than once); cheat, & optimize
        • Use any type wiki, if possible; this is so much better than Sharepoint!!!
        • Preserve chats, most of the good knowledge is an organic knowledge that you learn ‘overheard on a virtual grapevine’; guys who work in the company for longer is your GOLDMINE; treat them with a respect, and they will teach you about a lot of internal/organic things.
        • Don’t be an @#%^&; seriously, be humble; better be open about not knowing something than making a fool of yourself pretending to know it all; but still, be an @#%^& sometimes, when you see others ‘not getting it’ – stick to your guns and explain what they are missing
        • DO NOT MANSPLAIN, but point to wiki, blogs, articles, give CONTEXT…
          • I can’t count how many times ppl asked me some stuff and I always come back with:
            • ‘can you elaborate?’
            • ‘can you give me more context?’
          • – when you ask questions start from the beginning – explain what led you to ask the question… this often changes/affects the answer – speeds things up and makes you look awfully good!
        • If you use emails as a main source for your alerts, and send lots of emails:
          • Create .OFT email templates for all most-common scenarios
          • Replying to tickets will be easy this way and many things can be auto-populated
          • Again, re-use and provide branding and consistency
        • Did I mention branding?
          • YOU ARE A PART THAT FORMS THE CERT BRAND in your org
          • Be consistent
          • Be predictable
          • Be likable
          • Be a consultant, not a guy who nags people
          • Be aware that some users will come back and challenge you – explain why you contact them to avoid confrontations like this
          • Remember you are a COST CENTRE not a PROFIT CENTRE
            • Some ppl you contact will have bigger priorities than you
            • They more than often actually make money for the company
            • The money that pay for your salary
            • Treat them with respect
            • But
              be consistent and firm – escalate if needed
              don’t be afraid of titles – an infection on CEO’s/SVP’s laptop is a serious biz no matter what they say; engage senior people directly or indirectly
              even admins, hardcore IT guys are absolute security idiots; educate them; explain; provide links to blogs; make them think
        • Coming back to my automation mantra – if you need to fill-in some data and this has to be saved in a template, write a small program/script, or even create a web page with a bit of JavaScript that will allow you to enter data in one place, and the script will create the output for you – if you don’t know what I mean, check the interface of 3RPG – the cheat is that you create your own form, that in turn gives you a desired, consistent output!
      • FILLING-IN TICKETS
        • This is typically the worst part and while I repeat myself I want to say this:
          • I had a good experience using macromakers (tools that record mouse/keyboard/UI actions that you can replay with one keyboard shortcut),
          • Bookmarklets (small pieces of JavaScript that you can run by clicking a bookmark; using browser debugger /F12 in most browsers/ will be handy here to write bookmarklets that work for you as you can use it to find form fields this way; form fields that you want to populate with a predefined set of values) – these tool can automate filling-in the tickets (e.g. if you have typically 5 different types of tickets, you can have 5 bookmarklets that will fill-in stuff for you and available under 5 shortcuts i.e. these, when clicked, will populate the most common fields that rarely change; all you need to do is a quick QA instead of clicking and selecting items from the lists for each time =¬† this can give you a lot of time saved)
          • Let the automation (if available) do it for you as per template/bookmarklet – see next bullet point
        • If you are more advanced look at API the ticketing system uses; if you don’t know, identify some programmer in your company who can write a script for your to automate tickets processing (e.g. open/close); it’s not difficult (e.g. Archer supports SOAP and REST); external support from professional services (if available for your ticketing system) can also do a lot of magic
        • If you have _any_ influence, try to push for changes at the design level i.e. ticketing system can always be adapted to your needs (i.e. your company’s IR worklfow) or… replaced
        • If possible, provide feedback about bad experience, time-consuming tasks; talk to your manager; nag them – make it their problem – it’s actually their job to make your work easy
        • Enrich data automatically, wherever possible (this is related to design/architecture of the ticketing system; you may not have the necessary influence, but can always chat with your peers/manager and compare your system to ‘other company’s’ system, where your buddy works and the system does miracles there)
      • CLOSURES
        • Walk through the ticket and ensure that everything is in order
          • All fields filled-in properly and accurately
          • There is a narrative; a non-technical person should be able to read and understand what happened, when, where, what was affected, what was the analysis, what was the outcome, what was done in the triage, analysis, containment, eradication, recovery steps
          • Add lessons learned (actually, probably don’t as no one reads it; better keep these on email)
          • Do not add vague, unrelated comments
          • Do not write to yourself or to a virtual audience (“I need a second opinion here”)
          • Last, but not least – add clear conclusion “Analysis shows that there is NO malware on the system”. Only now you can close
        • Alternative to closing ticket, use a hold/pending other teams status if you have one – this puts the weight of long closing times on your dependencies (i.e. other teams)
  • ‘STUPID MANAGERS’
    • FACTS
      • They exist, sorry ūüôā
    • HOW TO CHEAT:
      • Try to understand what their game is:
        • Are they truly concerned about the output of your work, just metrics, or they are psychopaths that simply enjoy their power trips?
        • If they do power trip thing, change the company; you won’t win with them; not worth your time
        • If they are not, again, try to understand what their game really is
        • Fundamentally, are they good mentors to you? i.e. do you want to be like them?
        • Managers are your bridge to the leadership
          • Communicate with them all the issues you come across
            • Point out difficult things
            • Praise the good stuff too!
          • If no documentation exist, start creating one; who knows, maybe you will own this as a project (good bye all the days related to ticketing)
          • Ask for space to learn e.g. project day, certificates, courses, team meetings with demoes
      • … Mentor them back and delegate upstream;
        • Yes… instead of waiting for things to happen, make them happen by making your managers making them happen
        • Let them know of your issues, make them make decisions; or in other words, bring things to their attention so it is their problem, not yours – and ask them to formalize it!
        • This last sentence should really guide you through your L1 experience overall
          • You are here to deliver a consistent, repetitive task
          • Anything outside of it should be brought to your manager for evaluation and result in either a change in a process (e.g. additional branch on a decision tree), your manager being fired (just kidding), or the whole thing being dropped (stop seeing alerts of certain types, auto-closing, tuning of alerts, etc.)
      • Don’t whine; avoid it like a plague;
        • Unless you both whine about the same thing ūüôā I hate one or two security products but I didn’t make a call to bring it to the company, so have to use it, no matter what I think of it; I seriously hope I can write about these on my blog one day…
        • Think of what capabilities your manager has to fix your problems
          • You may often realize that they, same as you, are pretty limited in their own world
          • When you realize they bang their head against the wall same as you, you may actually change your opinion about their management style
          • When you understand their game, help them
          • When you help them, they will recognize your efforts (unless they are @#%^&)
        • Also, if you whine, think of
          • solutions
          • acceptance of the current status quo
        • Once you provide solutions (even if just suggestions) you will be perceived as a solution ‘man’, not an @#%^& who is just flagging problems
        • Once you accept the status quo – you can chill out ūüôā
      • Own things
        • It’s VERY okay if you don’t understand things, or make mistakes
        • When you own it, you will be seen as reliable
        • It’s better to have a reliable guy who delivers, even making lots of mistakes at first than someone who doesn’t care and just floats on the surface!!!
          • I can tell you I did both and I was always better off delivering what’s expected! ūüôā
      • Don’t bring your BIG ideas to the analysis YET (in your first months)
        • It’s not your role
        • But collect these ideas and evaluate them over some period of time (say… 6 months?); talk about these ideas informally with your peers/L2/manager; bring them in as suggestions not ‘we must do it, because it’s my opinion’
        • In any case you need to do analysis as per the guidance provided by the more senior people first; and you need to do it right first; no one will perceive you seriously until you are a reliable L1 analyst
        • And… then you will be able to bring ideas to the table (i.e. again _after_ you do what they tell you properly first, then establish yourself; in any case, you can’t improve the process you don’t understand/follow + you are 100% not aware of many past decisions, so your opinion is very biased!)
      • Don’t call things ‘stupid’ YET
        • You actually have to earn that right.
        • It’s only when you fully understand the process given to you when you can call it ‘stupid’ or ‘inefficient’
        • Again, remember that things that are part of the process are imperfect for many reasons, some of them you won’t know, even your manager won’t know (!!!) – they are there, because someone, some time in the past decided they should be there; look for reasons for their existence, don’t scrutinize it yet
      • Don’t change any process on your own
        • You will surely break things (some ppl will complain)
  • ALERT MAILBOX MANAGEMENT
    • Use ticketing system that removes emails from the equation (this is the future)
    • If you can’t, use (or make your request known to your manager) consistent subject lines that allow for setting up Outlook rules (I don’t mention other email software in this post, because corporate=typically outlook, but if you use different systems just adapt to your needs)
    • If you see alerts you DO NOT react to – and they keep coming, ask your manager to kill them at source, or set up outlook rule to redirect them to a folder where they will be stored and marked as automatically read
    • If you use Outlook rules, adapt them and review them every once in a while and especially when you see new patterns – technology changes, we often don’t even get notification when it happens and rules may stop working
    • If possible, push away NON-incident emails; I personally classify incidents as anything that affects Confidentiality, Integrity, or Availability and in my mind the C & I should be handled by respective CERT/Privacy teams (or together), and A either by NOC, or webops, or anyone that manages the infrastructure; CERT/SOC ppl often don’t see (or don’t understand) the architecture and can’t participate in handling DDoS very well; for some reason many companies burden CERT/SOC with this task… In my books.. Availability issues should be documented by CERT/SOC in a way a security guard observes and reports… and let the infrastructure guys do the needful (working with network teams, Anti-DDoS vendors, etc.)
    • Provide feedback from your research to threat hunting teams; if you see the same executable hit by AV, EDR, or Splunk rule – and you know 100% it’s an FP – exclude it, or flag it as such to these teams! Actively participate in tuning the feed that comes to the alert mailbox
  • TL; DR; takeaways, and making it a GAME WITH A PURPOSE
    • Learn programming (automation, API), scripting (bookmarklets), writing macros
    • Learn your software (shortcuts, functions)
    • Try to research log formats, protocols, how browser works, how network works, how computer works, how OS works, how malware works
    • Treat others with respect and assume advisory role when dealing with users
    • Use templates, boilerplates
    • Show interest and initiative
    • Communicate, communicate, communicate

And… don’t follow all this advice unless you are convinced this is a good advice. Also, note that while I may be advising you here I often have my ups and downs, and totally whine more often than I should….

BUT

Then I kinda man up and own the status quo and suddenly things are easier. So… last, but not least… recognize your whining times and use them to collect ideas on how to fix stuff a.k.a make your own life easier.

Tickets are just this – the spit in your face you need to learn to wipe off quickly, humbly, and… faster and faster. The better you are at it, the faster the time flows, the more you learn, and … you have more time to learn the next-big thing.

If you have any tricks of the trade you would like to add, or expand on please let me know and I will update the post. Thx!