Dealing with alert fatigue, Part 1

Gazillion tickets, gazillion emails a day. The business as usual for most SOCs…

It actually doesn’t matter how we got here (although I will cover some bits later on) – what matters is that we ARE here, and it literally sucks the life out of us, and every new dozen of emails/tickets coming in so frequently during the day makes us all die inside more and more…

This series will solve many problems for you. Okay, not really, but it will give you ideas and tools that you can use to make the life easier for yourself and your team. And if you do need help, ping me & I will be happy to help.

Okay, so a quick background first: over last 20 years I have worked many queues: localization bugs, analysis of malware samples from customers, analysis of sandbox samples to improve sandbox engine quality, as well as tones of IR emails and tickets that I had to work on one by one… I worked on call, covered follow the Sun queue processing, and led teams doing so. In every org I worked for I made it a priority to reduce ‘the stupid’.

What does it mean in practice?

In all these cases I always tried to make the life for everyone involved easier. There is an old Sales saying… always be closing. For years, I’ve been adapting this line like a mantra to my queue work and tried my best to follow it to make sure I am the most effective ‘closer’ w/o affecting the security posture of the org I worked for. As an analyst, I used macro tools and bookmarklets to fill-in the ticketing systems with preset values, wrote some scripts that helped me to do quick light forensics in a pre-EDR era, then when things got a bit modernized – looked at SOAP/REST to move the ticket closing function to CLI tools (someone smarter than me actually implemented it on our team), then looked very carefully at Threat Hunting and Detection Engineering efforts with an intention to tune them down, and in recent years embraced automation to find ways to auto-close as many tickets as possible.

Always having 2 goals in mind: less tickets, and less False Positives.

And as a manager I always tried to kill alerts and emails at source. Disable, decommission, phase out, if possible. And if not, tune, adjust, add exclusions, or just reduce the audience. Not everyone needs to get emails about everything. At any point of time — focus is on a ‘big picture’, trend analysis, aiming to recognize patterns and adjust, give people working for you the precious gift of… time.

Before you start thinking that I am claiming credit for other peoples’ work and introduce ideas as mine, I want to humbly offer words of appreciation for every single person I ever worked with that had far better ideas than mine, often implemented POC that was sometimes industry-level groundbreaking, and in some cases made my jaws actually drop (‘why didn’t I think of it before!’). The intention of this series is not to self-ingratiate, but to encourage everyone reading it to go out there and simply try to make a difference…

This is the moment where I have to say that I have a bit of a problem with Detection Engineering and Threat Hunting. I am actually part of the problem, because as a hunter I have developed many imperfect detection rules, SPL queries, yara sigs, etc. and while being obviously very proud of them and often assumed that the results of using them are easy to assess and are kinda ‘obvious’ to work with, I was adding fuel to the fire. This is an arrogant, selfish and short-sighted approach that doesn’t translate well into a world of SOPs…

Why?

We need to clearly state that there is a distinctive difference between ‘test’ or ‘beta’ or literally ‘FP-prone’ detections we think are ‘good to eyeball or to share on Twitter/github’ vs. actual, production-ready, actionable detections for which we want to create tickets/incidents that we ask SOC to work on. Investigations cost. You do want to cherry-pick occurrences where this work needs to be done. If your detection query/threat hunting query has too many hits, it’s not a good query.

Let me step back a bit more: we need to clearly state there is a distinctive difference between time-sensitive, and non-time-sensitive work. A lot of DE/TH work is non-time sensitive, while SOC/CERT groups work under strict SLAs. The less time-sensitive work for them, the better. How to facilitate that? Give them … less alerts.

I am jumping from one topic to another, but it’s a complex, loaded issue.

Let’s jump a bit more… ask yourself this: do you work all your incidents via emails? using your ticketing system (system of record) ? or, both?

In 2022, you MUST work all your incidents from a ticketing system.

That means that ANY incident coming in via email, IM, phone call, twitter message, etc. MUST be always put in a system of record first. Any emails that MAY be incident-related need to be disabled, phased out, and redirected to the ticketing system one way or another. There should be literally NO incident-related email to handle, apart from some random, ad-hoc, one-off situations (f.ex. CISO emailing another CISO which cascades down to SOC, but even these should find their place in a ticketing system)…

Let me reiterate: all the incident work should be done from a system of record/ticketing system, and analysts should have their mailboxes as clean as possible.

System of record brings order and accountability to the work we do.

Email hygiene removes the clutter from analysts inboxes and reduces a number of daily context switches.

Mind you, we are still talking alert fatigue REDUCTION. This actually means constant MONITORING aka TELEMETRY. You need to build some metrics around the work that is done on DAILY basis. And if you are a manager, you need to spend time reviewing that work. This will help to improve SOPs, this will help to improve technical writing/reporting skills of the analysts, this will highlight the ‘stupid’ or ‘unnecessary’ or ‘borderline illegal’ people actually do (and they do, f.ex.: an external SOC member noticed our org was port-scanned from a distinctive IP; he launched his own port scan against that IP; yes, facepalm).

What do you know about the origin of all alerts you see on your queue?

Many of them are ‘legacy’. Someone, somewhere, sometime in the past decided that particular class of tickets is a MUST on the queue. Was it a result of contractual, regulatory, compliance/audit need? Was this detection added 10 years ago and is no longer important? Was that a whimsical ego-driven, ‘wide’ detection added by ‘the most technical primadonna on the team’?

ROI is always the key.

The quality of alerting differs. Some are simply high-fidelity, some are low-fidelity, but a success would have a tremendous impact on the org. Lots of parameters to think of. But… from a pragmatic angle… if you receive alerts on your queue that only produced 100% FPs in last year you do need to start asking yourself: why are we looking at them AS ALERTS at all?

The alert fatigue is not a problem of one person. It is a problem of the whole team. If you work in SOC or on a triage team, keep your eyes open, look for patterns, escalate ‘stupid’.

And it may be reasonable to create two queues. Urgent, time-sensitive, SLA-driven work & the ‘maybes’, one that requires additional knowledge, some poking around, ‘wider in scope’, and NON-TIME SENSITIVE.

There is no way to optimize your queue w/o your active participation. We live at the time when holacracy seems to be gaining popularity, but reality is — we still need TECHNICAL decision makers who can make a lot of alerts go away. This activity may sometimes be perceived as brutal, ruthless, and may hurt feelings of some people (‘infosec rockstars’, ‘team primadonnas’), but ROI is out there to be your beacon and an excuse…

Cut it down, if you can. But fundamentally, start talking about it within your teams. The alert fatigue is a human-made problem. I guarantee you that you can cut down 50% of your alerts with a minimal penalty. How do I know that? I’ve done it before.

Posted in SOC

How to become the best SOC Analyst E-V-E-R

Update

Since I am getting a varied feedback on this article, I want to clarify a few bits:

  • I tried to incorporate many things that I wish someone told me in the past when I started working tickets
  • If it comes across as preachy or hierarchical it was not the intention – seriously… whoever works tickets knows that many things here are real world examples of issues we have to deal with
  • If you take an offense reading about hierarchy, and me positioning myself as ‘mentor’, and calling junior people ‘grasshoppers’ – c’mon… this is all corporate language at work, also I must mention I am not a native speaker and I may be missing some nuances; people who know me personally know that I am the first one to admit an impostor syndrome; I also want to emphasize that I do recognize the expertise of people I happened to mentor – they teach me back a lot and I refer to it in the first paragraph (I can easily bring up some anecdotes from my past where one of the juniors has demoed some really cool stuff about mobile forensics – an area where I don’t have much experience – it was a reversed role, I was the grasshopper and she was the mentor there).
    Also, pretty much every single corporate org assigns buddies, and runs mentor programs where juniors are mentored by others; yes, it’s definitely arrogant to say you are a mentor and they are mentorees, but please give me a more accurate English word to call that relationship & I will update the article; and yes, there is SANS Mentor program too; semantics is a fun topic to explore, but if the only feedback you give me is that you don’t like my ‘entitled’ tone, then thanks for assessing the 4-5h of work that went into writing this post with your linguistical nitpicking
  • If you take offense in profanities, rest assured, I have removed them for you – however, if you have never called a ticketing, AV, EDR, IDS/IPS software a piece of @#$%^&, this post is probably not for you anyway(?); hmm there is not much space for political correctness in the necessarily emotional piece that this post aims to be…
  • When I refer to a SOC Analyst, I mean a junior (in terms of skills in DFIR) person, not a hardcore, technical person that used to be called SOC Analyst in the past (typically working on network side, probably today referred to as NOC)
  • So many people in our industry focus on the cool stuff: certs, EDR, data science, threat hunting, memory forensics – pretty much L2 and L3 stuff; I find that there is a huge, often misunderstood and definitely unfairly placed burden on the L1 analysts – they are very rarely respected, the tools they are forced to use suck, and the analysts are also often incorrectly guided to do their work, and to grow (I know I generalize, but this is what I came across in the past)

Old Post

Over last 15 years I had a privilege of training and mentoring a number of junior people, often much smarter than me in their own areas of expertise (programming, network, admin work, routers, support for various tools/devices, etc.), but still either completely, or partially green when it comes to SOC/IR/DFIR/RCE space. I will lie if I say I was always happy about things these ‘grasshoppers’ were doing, but it would also be a lie if I said I was always right. And well, I was once a grasshopper myself, and definitely (and perhaps sometimes even intentionally) pissed off more than one person with my stupid shenanigans, and probably still do…

We all start somewhere, and we (hopefully) continue to grow.

The fun of leading and mentoring teams is actually there – it is the individuals that you come across while doing this work; believe it, or not, among them you will find real gems, and I mean it!
– these guys will teach you back many things you don’t even have an idea exist.

Interestingly, as our knowledge gets pretty rusty and it does so incredibly quickly, I actually find the cooperation with the security youngsters very rewarding – they make me stay up to date – they are asking questions which in return force me to actively explore the vast areas of my… ignorance.

Yes!

There is more to it out there than the actual expertise. This is because our field expands kinda exponentially and while I may be focused on Wintel attacks, there is the whole domain of *NIX, mainframe, IoT, or even DDoS, or appsec attacks that I am either not aware of, or barely aware of (I am actually a ‘grasshopper’ in these fields). It would seem our ITSEC expertise is more about indexing things we don’t know about…

So… the good news it that some of these guys live it the same way you lived it 5-10-20 years ago. Yup. Good to talk to them and… learn.

Today’s post is not about the glorious day of enlightenment though. I thought I will write a quick tutorial-like piece, or a check-list if you will with the aim of helping new analysts who enter the L1 function w/o much experience in the field and perhaps wanting to grow as security professionals.

If you are still reading it I assume now that you are the newly-hired individual in the L1 SOC function – you may be a SOC Analyst, SIEM analyst, Junior CERT Analyst, whatever. You just joined the company directly, or act as a part of MSSP company, and perhaps (in some cases) are with  your company for a few months, but still don’t have much clue what’s going on. Let alone know ‘how to look like you are doing something useful’.

Also, chances are that you are probably already hating your job, and your manager, or your employer 😉

Because… these stupid tickets.

This is practically what you do all day long. The big words and ravishing dreams of being a hacker, computer security expert, one that perhaps one day will have 1 million followers on Twitter, and even be invited to talk at TED, one that people worship… one that maybe even FBI arrests… one that is so special…it all fades away with that every, single ticket you have to open, fill-in, and eventually close, or leave for future generations to sort out.

Don’t despair. You will get over it. We’ve all been there.

One of the first questions I typically ask you – what is the most important thing while doing tickets. You don’t know me yet – you just know I am your lead/manager. You take my question very corporate-seriously and answer that it is indeed a very important task, and that it is incredibly important to do the tickets right, and give your best to add / enrich ticket information; and perhaps you already know, or have overheard that ‘if auditors read data from YOUR ticket when it is provided as an evidence they may ask questions’, so you may add that we obviously need to do tickets well enough so auditors will be happy.

And of course, you want to improve, learn from mistakes, and will definitely work harder to get better at working them in the future!

These are all nice and cozy answers, and I am grateful for a second, and a second only, because while I may be the almighty know-it-all boss du jour, you are just feeding me with the stuff that my vanity happily feasts on.

So I steer you towards the one and only proper answer – one that often surprises you, and then we have a good laugh – because for the first time we talk not like the boss and the employee, but like two brothers in arms, and human beings really. And that’s because I worked tickets before and the answer I give you is this:
– the one, and only, most important thing while doing the tickets is actually NOT opening them, or if you really have to, opening them, and closing them as soon as possible! Escalations, external dependencies on other teams – will kill all the fun, SLAs, and often expose you to scrutiny.

A-HA!

We both know doing tickets is a nightmare. Most of it is a mundane, absolutely idiotic waste of time, a busywork, a grass painted green over and over again, an unexciting, repetitive, boring task.

BUT IT HAS TO BE DONE!

Don’t despair. Everyone knows it. Everyone hates it. And this post will help you to deal with it.

The TL;DR; answer is this: convert this ticket hamster wheel to a game with a purpose.

The game with purpose is one of the creations of the mind of Luis von Ahn (the CAPTCHA and Duolingo guy).

The ‘purpose’ that works for you is your own business really – you have to kinda work it out by yourself; and the ‘game’ bit is like any other game – who cheats, gets there first!

My personal purpose while doing tickets was always and still is: WIN TIME.

The rationale is pretty simple. We actually die. Yes, we do. Having a time that is gained by efficient management, or any sort of trickery is your life being utilized in a better way; a time is a luxury; let’s be frank: any ‘free’ excess of it – is a gift.

When you have time, you can dedicate it to more rewarding activities. They should be strictly related to your work, of course, otherwise you will get fired, but perhaps with a bit of planning these free cycles you win may actually open doors for you to areas that are little more interesting than the tickets themselves. And more rewarding indeed, because what if… the time you earn by being efficient allows you to focus on the next step in your career?

Isn’t that worth it?

So… without further ado, here’s a list of ideas I have used/explored, or otherwise practically used in my past lives; this is by all means not a full list and I am sure some of my bros-in-arms or sis-in-arms can chip in and add their 5 cents of advice – I sincerely hope so:

  • BEFORE YOU BEGIN
    • What is that you do?
      • Fill-in the tickets? yeah… but…
      • Provide enough info to L2 (CSIRT/CERT/SIRT/etc.)? yeah.. but….
      • Think of your work in consulting terms;
        • Who is your client? Who will read your tickets?
          Typically L2 (CSIRT/CERT)
          Typically your lead/manager
          Sometimes no-one specific (other than your system of record storing it and compliance guys providing it as an evidence to auditors)
        • What is your scope of work and area of influence?
          L1 work only
        • Are you able to escalate the problems to anyone if you don’t understand something?
          Talk to people who have more experience than you! Show them what you did, why, and you will get a good feedback & support!
          Technical people love to explain, even show off. Ask direct question. Don’t challenge them or assume they are uneducated. Most of the time they have done L1 work before and are fully capable analysts that can GUIDE you. Also, they sometimes may be wrong, so let them know when you spot their mistake!
      • So… Ask questions!!! Many of them. Interact!!!
        Bore the L2 and your manager to death. It’s their role to GUIDE you.

        • “Why do we do things the way we do?”
        • “When I look at the data, what parts of it are important?”
        • “What is the meaning of all these fields in the proxy logs?”
        • “I did this and this and that. What else could I do?”
        • “Where can I read more about it?”
        • Ask
        • Ask
        • Ask
        • And then nag them to formalize it in a playbook, process guide
    • Read about The Alexiou Principle 
      • What question are you trying to answer?
      • What data do you need to answer that question?
      • How do you extract that data?
      • What does that data tell you?
    • Once you get the above, you are actually en route to become a very good investigator (L1 –> L2 –> L3)
      • i.e. the one that is a very focused minimalist, remaining objective, and his/her reports ‘write themselves’… these that ‘fly’ through the vast amount of data and can spot the bad guy with ease
  • TICKETING SYSTEMS
    • FACTS:
      • Most of them are absolutely idiotic, over-engineered, clumsy old-fashioned, inflexible and bloated, slow pieces of software; often designed in late 90s, or early naughties; they are often a piece of software **** really
      • Why?
        • Because they rarely follow the SOC/IR workflow
        • Because their UI sucks
        • Because it takes forever to fill them in, and they typically don’t work on many browsers, so you are stuck a.k.a. bound to use the one and only required (a.k.a. supported) browser
        • Because they were really not designed to handle security alerts, bundles of such alerts (campaigns), and do not scale well
        • Because they were not designed to support Data Across Border and GDPR issues
        • In MNCs, apart from DAB and GDPR issues, you have the problem of access; it’s a difficult problem to solve permanently for SOC analysts across the whole globe; and it’s really very tempting to store it all in cloud that anyone can access
        • Many older products do not support working on multiple tickets at the same time; yes, 1 cookie=1 ticket… it’s incredibly inconvenient
      • Editing itself, or pasting to ticketing systems is often a miserable job:
        • You edit it, you paste it, formatting is immediately ruined, f.ex. things are not aligned properly, etc.
        • Unfortunately, ideas promoted by ‘web-oriented’ companies force us to believe the web editing will take over; no, it won’t; it’s still a miserable experience and will stay as such for a long time; especially where you have to fill-in a number of fields
        • Unfortunately, nowadays all of ticketing systems are web-based. sigh…
        • I have been waiting for it for over 15 years and it’s actually getting worse and worse
          • My experience went from a blaspheming Lotus Notes-based bug tracking systems, through awful Java/Oracle-based Frankenstein’s Monsters, then mediocrity of Flash/Silverlight idiocy, and now we arrived at the Web 2.0 (soon Web 3.0) and ‘new’ UI concepts and paradigms that force you to click 20 times to do a _simple_ thing, because everything needs to be animated or expanded/drilled-down numerous times (think of ‘Read More’ buttons that shows you ONE additional line of text), and on the way break all the UI design metaphores that we so got used to using the native OS applications over last 20-30 years!; web broke all these metaphores
          • The guys who design IR ticketing systems really need to understand that less clicks to close an IR ticket=win for the analysts!!! Also, they need to understand what data is really important to show in previews – IR ticketing is really about speed-closing
      • Saving work – you will often find that some fields are mandatory to fill-in
        • Sometimes you forget one and you may try to submit the form – a weird error message shows up; then you have to edit it all over again
        • Sometimes all the work is gone w/o much reason, or when you submit and your broadband/VPN goes kaput
        • Sometimes the page will refresh out of nowhere
        • Sometimes you will click a wrong mouse button, or press a wrong keyboard short-cut – the browser will take you to another page
        • Sometimes you accidentally activate a shortcut by mouse, access history, click it, click a bookmark, your VDI will freeze, or whatever random happens
        • And yes, this really happens – your work is gone in a blink of an eye and there is no way to recover it
        • So.. thank you stateless web UI experience based on an assumption users don’t make mistakes and there is no VDIs, no network bottlenecks…
        • The Windows OS UI as hated as it is by many provides a very intuitive way to fill in all this stuff and supports automation VERY well (think: TAB key used to change the focus between fields on a Windows dialog box vs. fields on the web page that need to be activated by mouse each time; the first one can be driven by macromakers, the second _sometimes_ by bookmarklets)
    • HOW TO CHEAT:
      • CREATING TICKETS
        • If possible, use automated ticketing system (mail comes in –> ticket, with an appropriate data enrichment) – no need to do copypasta
        • If automation is not possible, use boilerplates/bookmarklets/macro makers/grease monkey scripts/dedicated in-house built browser plugins to fill-in the most typical cases; use the templates/bookmarklets (read below), and manually correct fields that require correcting for less usual cases
        • If there is an API (REST, SOAP), task someone to write a nice python/ruby/perl script to close tickets that can be closed automagically
        • Same as above, re-prioritize the ticket via script, if possible
        • If you have an access to a dev team provided by ticketing system vendor – USE THEM – while they charge exorbitant money they can do a lot of tweaks that you don’t even know exist (because they have done it for 200 other clients and know what issues to fix/what problems to avoid)
        • Ask your manager, L2
        • Google your ticketing system – see what other ppl do
        • Nmap and ‘light pentest’ your blackbox ticketing system (perhaps SOAP or other ‘goodness’ is there? /’goodness’ is a relative terms/)
      •  EDITING
        • Edit the content of the ticket body in an external program, if possible; it can be Notepad, Notepad++, Ultraedit, Textpad, SciTE, VIM, WordPad, WinWord – whatever works for you; keep these notes in a file that you frequently save; work on it in ‘append to its end’ mode
        • Using external editors gives you a full-power of the editor, not some web-based wannabe with very small input fields, limited formatting, non-resizable fields, broken Undo /even Confluence fails here sometimes/, no proper support for navigation in the editor, no support for vertical editing, no search&replace with regexes, no support for multiple files search and replace, etc. and no room for mistake really….
        • Having to fill-in a dozen or a few form fields is not for faint-hearted; simplify it with a macromaker/bookmarklet/API, if possible – I mentioned it before, but will repeat it a few times; optimize, automate, cheat using GUI-automation tools
        • Use more than one editor, if needed
        • Learn to treat Excel as one of the editors; the amount of stuff you can do either with Excel itself, or by combining the strengths of Excel, with a good programming editor e.g. Ultraedit, or cygwin / unix tools, and sometimes bit of scripting will really surprise you!
        • Learn to program, script, write macros in general. This goes a long way. This is probably the most important advice. Think like a programmer/coder – you will get far by cheating manual editing with some simple algo to do work for you.
          • Note – editors allow macros/templates; you could define them and use them for various purposes
        • Use keyboard more than mouse, if possible!!! (can’t stress this one enough; LOOK UP shortcuts for the programs you use, including web pages e.g. ‘E’ on Confluence enters the ‘Edit’ mode; ‘//’ enters a date, etc.)
        • Outlook allows you to select emails and then when you copy them to clipboard, you can paste their ‘main’ headers (as per set up view) to excel
        • Use VBA and Office Macros!!!
          • They are not just malware, they offer awesome automation functionality you can use to speed up processing stupid things;
          • Example:
            • Exporting batch of emails from Outlook so you can process them with a python/perl/ruby script afterwards
            • This is a great time saver (e.g. instead of looking at every email, you extract all every say 24h and run through your script that excludes white-listed entries, highlights blacklisted entries, does data stacking, perhaps saves it in a local database for future comparisons, etc.)
            • Doing stats on Outlook folders
        • Again, Ad nauseam, read help for your editor of choice; learn keyboard short-cuts; read it again
          • I keep learning new things about Total Commander or Excel every day, despite using it for 20 years!!!
        • Install Puretext – it’s a brilliant small program that allows you to paste w/o formatting
        • Another similar program is Ditto (http://ditto-cp.sourceforge.net/)
        • If you paste data from tickets in Outlook, use CTRL+ALT+V shortcut to choose pasting type
        • Learn Excel shortcuts
          • ctrl+; for entering the current date
          • ctrl+: for current time
          • ctrl+1 for formatting
          • shift+F8 for selecting all content within a block
          • ctrl+pgdn and ctrl +pgup to navigate sheets,
          • etc. –
            it’s much faster;
          • then check vlookup, pivot tables, “&” for concatenation, etc,
      • TRIAGE
        • Stick close to the source of the alert
          • i.e. research logs RELATED to the event;
            • by the time frame
            • by the host name
            • by the user name
            • by the risk name
            • by the phishing campaign
            • by the type of event
            • etc.
            • Don’t go ballistic on VT/HA/malwr/virusbay/urlquery if you don’t know/understand what happened on the box itself!!!
            • Threat Intel and blog posts are OVERHYPED
              • Don’t read blog posts until your logs tell you what happened; until you have some hypothesis to verify
            • Again, for that, stay close to the event that triggered the alert !!!
              • Think ‘what happened there’ before you go with further triage/analysis tasks
            • So… don’t go to VT, HA, or googling around in general as a first ‘itchy fingers response;
              • It’s all helpful, but it’s not really answering the 4 of Alexious principle questions
                plus
              • Vendors are sometimes vague
              • Vendors are sometimes wrong
              • Peers in the community are sometimes vague
              • Peers in the community are sometimes wrong
            • If you see an AV alert – go to your vendor page – UNDERSTAND what the threat/risk name tells you; is it a virus (modifies files), is it a ransomware (encrypts files), is it a heuristic detection (often FP), etc.
            • Join groups, forums (e.g. OWASP and local chapters/meetups, SANS, FIRST, various CERT notification emails, FS-ISAC, etc.), Twitter – learn about ‘new’ before media outlets pick it up
        • Look at the AV, IDS, Proxy logs;
          • Look at as many sources as possible;
          • And if it may sound weird… avoid firewall logs at first, unless absolutely necessary;
          • Why?
          • Attacks shifted to higher levels of TCP/IP stack, lots is happening on application layer
          • Don’t underestimate firewall logs though, you will definitely find them useful in more advanced attacks (they may be actually the only logs you will have to prove someone stole your data!)
      •  TEMPLATES/BOILERPLATES
        • If some tickets are the same, or nearly the same – keep a boilerplate with a text already prepared for a copypasta; this will save time, and will ensure you are consistent (this is why writing stuff using an external editor is also handy – you can keep all your edited stuff in one place; yes, create a ‘book’-like document and type your stuff there; you will notice patterns, and you will re-use existing text more than once); cheat, & optimize
        • Use any type wiki, if possible; this is so much better than Sharepoint!!!
        • Preserve chats, most of the good knowledge is an organic knowledge that you learn ‘overheard on a virtual grapevine’; guys who work in the company for longer is your GOLDMINE; treat them with a respect, and they will teach you about a lot of internal/organic things.
        • Don’t be an @#%^&; seriously, be humble; better be open about not knowing something than making a fool of yourself pretending to know it all; but still, be an @#%^& sometimes, when you see others ‘not getting it’ – stick to your guns and explain what they are missing
        • DO NOT MANSPLAIN, but point to wiki, blogs, articles, give CONTEXT…
          • I can’t count how many times ppl asked me some stuff and I always come back with:
            • ‘can you elaborate?’
            • ‘can you give me more context?’
          • – when you ask questions start from the beginning – explain what led you to ask the question… this often changes/affects the answer – speeds things up and makes you look awfully good!
        • If you use emails as a main source for your alerts, and send lots of emails:
          • Create .OFT email templates for all most-common scenarios
          • Replying to tickets will be easy this way and many things can be auto-populated
          • Again, re-use and provide branding and consistency
        • Did I mention branding?
          • YOU ARE A PART THAT FORMS THE CERT BRAND in your org
          • Be consistent
          • Be predictable
          • Be likable
          • Be a consultant, not a guy who nags people
          • Be aware that some users will come back and challenge you – explain why you contact them to avoid confrontations like this
          • Remember you are a COST CENTRE not a PROFIT CENTRE
            • Some ppl you contact will have bigger priorities than you
            • They more than often actually make money for the company
            • The money that pay for your salary
            • Treat them with respect
            • But
              be consistent and firm – escalate if needed
              don’t be afraid of titles – an infection on CEO’s/SVP’s laptop is a serious biz no matter what they say; engage senior people directly or indirectly
              even admins, hardcore IT guys are absolute security idiots; educate them; explain; provide links to blogs; make them think
        • Coming back to my automation mantra – if you need to fill-in some data and this has to be saved in a template, write a small program/script, or even create a web page with a bit of JavaScript that will allow you to enter data in one place, and the script will create the output for you – if you don’t know what I mean, check the interface of 3RPG – the cheat is that you create your own form, that in turn gives you a desired, consistent output!
      • FILLING-IN TICKETS
        • This is typically the worst part and while I repeat myself I want to say this:
          • I had a good experience using macromakers (tools that record mouse/keyboard/UI actions that you can replay with one keyboard shortcut),
          • Bookmarklets (small pieces of JavaScript that you can run by clicking a bookmark; using browser debugger /F12 in most browsers/ will be handy here to write bookmarklets that work for you as you can use it to find form fields this way; form fields that you want to populate with a predefined set of values) – these tool can automate filling-in the tickets (e.g. if you have typically 5 different types of tickets, you can have 5 bookmarklets that will fill-in stuff for you and available under 5 shortcuts i.e. these, when clicked, will populate the most common fields that rarely change; all you need to do is a quick QA instead of clicking and selecting items from the lists for each time =  this can give you a lot of time saved)
          • Let the automation (if available) do it for you as per template/bookmarklet – see next bullet point
        • If you are more advanced look at API the ticketing system uses; if you don’t know, identify some programmer in your company who can write a script for your to automate tickets processing (e.g. open/close); it’s not difficult (e.g. Archer supports SOAP and REST); external support from professional services (if available for your ticketing system) can also do a lot of magic
        • If you have _any_ influence, try to push for changes at the design level i.e. ticketing system can always be adapted to your needs (i.e. your company’s IR worklfow) or… replaced
        • If possible, provide feedback about bad experience, time-consuming tasks; talk to your manager; nag them – make it their problem – it’s actually their job to make your work easy
        • Enrich data automatically, wherever possible (this is related to design/architecture of the ticketing system; you may not have the necessary influence, but can always chat with your peers/manager and compare your system to ‘other company’s’ system, where your buddy works and the system does miracles there)
      • CLOSURES
        • Walk through the ticket and ensure that everything is in order
          • All fields filled-in properly and accurately
          • There is a narrative; a non-technical person should be able to read and understand what happened, when, where, what was affected, what was the analysis, what was the outcome, what was done in the triage, analysis, containment, eradication, recovery steps
          • Add lessons learned (actually, probably don’t as no one reads it; better keep these on email)
          • Do not add vague, unrelated comments
          • Do not write to yourself or to a virtual audience (“I need a second opinion here”)
          • Last, but not least – add clear conclusion “Analysis shows that there is NO malware on the system”. Only now you can close
        • Alternative to closing ticket, use a hold/pending other teams status if you have one – this puts the weight of long closing times on your dependencies (i.e. other teams)
  • ‘STUPID MANAGERS’
    • FACTS
      • They exist, sorry 🙂
    • HOW TO CHEAT:
      • Try to understand what their game is:
        • Are they truly concerned about the output of your work, just metrics, or they are psychopaths that simply enjoy their power trips?
        • If they do power trip thing, change the company; you won’t win with them; not worth your time
        • If they are not, again, try to understand what their game really is
        • Fundamentally, are they good mentors to you? i.e. do you want to be like them?
        • Managers are your bridge to the leadership
          • Communicate with them all the issues you come across
            • Point out difficult things
            • Praise the good stuff too!
          • If no documentation exist, start creating one; who knows, maybe you will own this as a project (good bye all the days related to ticketing)
          • Ask for space to learn e.g. project day, certificates, courses, team meetings with demoes
      • … Mentor them back and delegate upstream;
        • Yes… instead of waiting for things to happen, make them happen by making your managers making them happen
        • Let them know of your issues, make them make decisions; or in other words, bring things to their attention so it is their problem, not yours – and ask them to formalize it!
        • This last sentence should really guide you through your L1 experience overall
          • You are here to deliver a consistent, repetitive task
          • Anything outside of it should be brought to your manager for evaluation and result in either a change in a process (e.g. additional branch on a decision tree), your manager being fired (just kidding), or the whole thing being dropped (stop seeing alerts of certain types, auto-closing, tuning of alerts, etc.)
      • Don’t whine; avoid it like a plague;
        • Unless you both whine about the same thing 🙂 I hate one or two security products but I didn’t make a call to bring it to the company, so have to use it, no matter what I think of it; I seriously hope I can write about these on my blog one day…
        • Think of what capabilities your manager has to fix your problems
          • You may often realize that they, same as you, are pretty limited in their own world
          • When you realize they bang their head against the wall same as you, you may actually change your opinion about their management style
          • When you understand their game, help them
          • When you help them, they will recognize your efforts (unless they are @#%^&)
        • Also, if you whine, think of
          • solutions
          • acceptance of the current status quo
        • Once you provide solutions (even if just suggestions) you will be perceived as a solution ‘man’, not an @#%^& who is just flagging problems
        • Once you accept the status quo – you can chill out 🙂
      • Own things
        • It’s VERY okay if you don’t understand things, or make mistakes
        • When you own it, you will be seen as reliable
        • It’s better to have a reliable guy who delivers, even making lots of mistakes at first than someone who doesn’t care and just floats on the surface!!!
          • I can tell you I did both and I was always better off delivering what’s expected! 🙂
      • Don’t bring your BIG ideas to the analysis YET (in your first months)
        • It’s not your role
        • But collect these ideas and evaluate them over some period of time (say… 6 months?); talk about these ideas informally with your peers/L2/manager; bring them in as suggestions not ‘we must do it, because it’s my opinion’
        • In any case you need to do analysis as per the guidance provided by the more senior people first; and you need to do it right first; no one will perceive you seriously until you are a reliable L1 analyst
        • And… then you will be able to bring ideas to the table (i.e. again _after_ you do what they tell you properly first, then establish yourself; in any case, you can’t improve the process you don’t understand/follow + you are 100% not aware of many past decisions, so your opinion is very biased!)
      • Don’t call things ‘stupid’ YET
        • You actually have to earn that right.
        • It’s only when you fully understand the process given to you when you can call it ‘stupid’ or ‘inefficient’
        • Again, remember that things that are part of the process are imperfect for many reasons, some of them you won’t know, even your manager won’t know (!!!) – they are there, because someone, some time in the past decided they should be there; look for reasons for their existence, don’t scrutinize it yet
      • Don’t change any process on your own
        • You will surely break things (some ppl will complain)
  • ALERT MAILBOX MANAGEMENT
    • Use ticketing system that removes emails from the equation (this is the future)
    • If you can’t, use (or make your request known to your manager) consistent subject lines that allow for setting up Outlook rules (I don’t mention other email software in this post, because corporate=typically outlook, but if you use different systems just adapt to your needs)
    • If you see alerts you DO NOT react to – and they keep coming, ask your manager to kill them at source, or set up outlook rule to redirect them to a folder where they will be stored and marked as automatically read
    • If you use Outlook rules, adapt them and review them every once in a while and especially when you see new patterns – technology changes, we often don’t even get notification when it happens and rules may stop working
    • If possible, push away NON-incident emails; I personally classify incidents as anything that affects Confidentiality, Integrity, or Availability and in my mind the C & I should be handled by respective CERT/Privacy teams (or together), and A either by NOC, or webops, or anyone that manages the infrastructure; CERT/SOC ppl often don’t see (or don’t understand) the architecture and can’t participate in handling DDoS very well; for some reason many companies burden CERT/SOC with this task… In my books.. Availability issues should be documented by CERT/SOC in a way a security guard observes and reports… and let the infrastructure guys do the needful (working with network teams, Anti-DDoS vendors, etc.)
    • Provide feedback from your research to threat hunting teams; if you see the same executable hit by AV, EDR, or Splunk rule – and you know 100% it’s an FP – exclude it, or flag it as such to these teams! Actively participate in tuning the feed that comes to the alert mailbox
  • TL; DR; takeaways, and making it a GAME WITH A PURPOSE
    • Learn programming (automation, API), scripting (bookmarklets), writing macros
    • Learn your software (shortcuts, functions)
    • Try to research log formats, protocols, how browser works, how network works, how computer works, how OS works, how malware works
    • Treat others with respect and assume advisory role when dealing with users
    • Use templates, boilerplates
    • Show interest and initiative
    • Communicate, communicate, communicate

And… don’t follow all this advice unless you are convinced this is a good advice. Also, note that while I may be advising you here I often have my ups and downs, and totally whine more often than I should….

BUT

Then I kinda man up and own the status quo and suddenly things are easier. So… last, but not least… recognize your whining times and use them to collect ideas on how to fix stuff a.k.a make your own life easier.

Tickets are just this – the spit in your face you need to learn to wipe off quickly, humbly, and… faster and faster. The better you are at it, the faster the time flows, the more you learn, and … you have more time to learn the next-big thing.

If you have any tricks of the trade you would like to add, or expand on please let me know and I will update the post. Thx!