You are browsing the archive for Preaching.

A not-so-short wishlist for the IR world

December 29, 2014 in Preaching


Adding two more items.

  • Learn how to detect snake oil & terrible products
    • The IT Security is full of companies that sell snake oil and products of poor quality. They come and go, and they usually work around really brilliant (in fairness) ideas that help them to milk companies while not really adding much value.
    • How to distinguish / recognize snake oil products and poor quality software?
      • First, read these 3 articles:
      • Seriously, read them.
      • Reflect on your budget spending in year 2015.
      • Recognize signs of snake oil and majestic software fails:
        • Company offering is not politically neutral (see my point below).
        • Long term deals.
        • Long term deployments e.g. one that will require 12-24 months (are you kidding me? IR product should be ‘deployable’ in one quarter max).
        • Software components (especially agents) are bloated, use ‘heavy’ programming languages e.g. .NET, Java, LUA, python / py2exe (this thing must run on the systems people use for work, servers, plus you already have other agents there; you can only accept lightweight agents)
        • User interface of the products – this is very often a first sign of the product being terrible:
          • Uses Flash, Silverlight (this may look cool, but try to copy & paste stuff, work with the web forms)
          • Uses Java ActiveX (c’mon, it’s year 2014)
          • Use own implementation of GUI widgets – it’s very hard to develop a properly working widget (i.e. one that is not natively available from OS/browser) and anyone using proprietary stuff will most likely be focused on fixing bugs with this proprietary stuff more than the core functionality
          • Anything intercepting default behavior of the UI e.g. context menus in the browser
          • See how it handles large amount of data, how long it takes to render pages
          • Is the UI intuitive? If the UI doesn’t support your workflow then ditch the product, or incident handlers will be struggling.
        • Network Traffic generated by software components – this can be quite amusing; if you see anything plain text, drop the vendor; it’s 2014 and if they don’t use encryption, https then they are not serious.
        • Documentation is a very good giveaway of the vendor’s attention to details:
          • Check metadata of PDFs, Docs and see when it was last updated.
          • Check the language used. Is it written by a professional Technical Writer, or it looks like a readme on github (no offense readme writers :))? Mind you, if you are paying gazillion bucks for a product, you do have a justified expectation with regards to documentation.
        • No direct access to the databases (they will talk about APIs, but direct access to DB, even if just read-only gives you an ultimate power to extract data they way you want).
        • Endorsements. Testimonials. Generally people are nice, and no one will ever tell you that a certain product is terrible. This should change – be honest – if someone asks you if the product delivers tell them the truth. Perhaps there is a need for a Glassdoor-like website for security products.
        • In general, anything that sounds too good to be true
  • Do not hack back.
    • It’s stupid, most likely illegal and you are asking for more trouble (you have a lot to lose while the bad guys will destroy you just for the fun of it; Ref.: HBGary, Sony).

Old post

As we approach 2015 I put together a wishlist for the IR world. I am sick today so I ended up with some time on my hands and it all ended up with a lot of babbling, but well, if you don’t like it simply blame it on my cold and move on before it’s too late :). In any case, it’s my subjective take on what can be improved here, may not be too constructive, but either way – here it goes… :)

  • Rename your team from “Incident Response team” to “Incident Discovery and Response Team”.
    • Ask your bosses to minimize your involvement in all corporate b/s.
    • Ask your bosses to give you authority to access all systems.
    • Ask your bosses to send you to Memory Analysis/Volatility training, SANS courses, CISSP. To do your job right you need to possess a decent knowledge within the area of digital forensics, malware analysis, pentesting, programming, databases and have a knowledge of the threat landscape. You must also understand the ‘big picture’ and business angle. Last, but not least you should also like memes and cats.
    • Start hunting for hackers on your network full-time.
    • Triage as many systems as you can and on regular basis. Use clustering, data stacking, Least Frequency Occurrence (LFO), accessible known white-, and black lists to sift through the data from many systems at once.
    • Engage forensics and leverage direct access to systems to:
      • Discover malicious agents.
      • Discover and analyze actual data present on the systems – this is what will be stolen or exposed WHEN these boxes get popped:
        • e.g. retrieve $MFT, scan file names for low-hanging fruits e.g. ‘password.txt’ files, database files, etc. – it has to be customized to your org needs; this can be a fun research/clustering job.
        • Do credit card/SSN/other PII scans and act on them (you will find lots of private emails, PDFs of employees – yes, you want them to remove this stuff from their work boxes).
        • Work with specific Business Units, Employees and their respective Managers to protect/clean up all unwanted stuff ASAP.
      • Collect and preserve the baseline data for future, recurring investigations.
  • Stop the cargo cult approach to IR.
    • Don’t rely on security controls only.
    • Don’t rely on alerts only.
    • Assume that at any point of time a subset of systems in your org is popped. You need to find them ASAP.
    • Learn the in-depth aspects of technology. You can’t talk about ‘viruses’, ‘trojans’, unless you actually KNOW how they work on the programming/forensic level and what is the difference between them (trivia question: what is a difference between a virus and a trojan?). You can’t look at proxy logs without knowing the basics of how browser renders web content i.e. what are the technical mechanisms engaged in browser accessing the web page (dns, cache, wininet, server-side and client-side, chrome, firefox as well, etc.), both dynamic and static. The ones with a padlock too. Yes, seriously, there are some people who don’t know and work VP functions in the IT Sec.
    • Work more on data you already have. And prioritize data generated by your org vs. ‘threat intel’ feeds. Data from your org is you looking for the spoon in your own kitchen. The threat intel data could be a fork, a spoon, or spork in somebody’s else canteen in a different city.
  • Don’t take yourself too seriously and don’t be the rightful one on a romantic, self-appointed mission (kinda ironic in a context of this post, but I hope you know what I mean :))
    • The old saying goes that hell is paved with good intentions
    • Don’t play a role that is not assigned to you. You are not law enforcement. Stop acting like one.
    • You also can’t chase after bad guys yourself. You can’t take a revenge other than hoping the LE will eventually find them. Yes, it’s sad. But if you do try doing stuff on your own you will inevitably end up following the mob mentality and will start lynching innocent. Plus you will attract attention of even more bad guys.
    • Some guys seem to be projecting their vision of IR in a ‘we change the world’ context and it looks like a life mission to hunt ‘bad guys’ and even more ‘the nation-state enemies’. Perhaps to certain extent it is a mission, but the cynical reality is that it’s just a job. Leave the wars to governments and military. It’s just a job. And the scope of this job is to protect assets and not to go Steven Seagal on the bad guys. And at the moment this job is probably far closer to security guard’s and/or helpdesk technician’s job than to a highly-technical position many have a perception of. Why security guard? Because your role is often left to simply observe and report. Why helpdesk technician? Because removing malware is hardly difficult. You either delete the file(s) and clean that run key, or ask to re-image the box.
  • Focus on ‘data security’ over ‘system security’.
    • Give more attention next year to offers focused on preparedness, early discovery more than reactive defense or offensive side, assets defined both as systems, and data itself:
      • Reactive defense:
        • Anything that is blacklisting-based is reactive. Anything that is heuristics, reputation, behavioral, algorithmic, threat-intel, feeds, cloud based is reactive. That is your AV, IDS/IPS, HIPS, SIEM, DLP to certain extent, often sandboxes as well. Kinda ironically, whitelisting software also falls into this category. Whitelisting simply doesn’t work as expected and it often ends up working in a failsafe mode (a lot of exceptions, ‘learning mode’, preapproved grayware). Auditing is here too.
          Note: some of these are proactive, provided they are properly installed and are in a blocking/deny mode by default (e.g. whitelisting, reputation)
      • Offensive:
        • All pentesting, vulnerability scans/assessments, etc. A discovery of a new way in is very important, but it does not change much the security posture of the assets you are protecting; an unencrypted cardholder data, HR PDFs, etc. lying around on the HDD are still there after you patch that new scary bug, or update Java. Protect data more than the system, because when it will get popped, you want this data to be hard to get to.
      • Preparedness, early discovery:
        • Focus is on your data, encryption on the physical (drive) and/or logical (volume) level, encryption on the application level (encrypted configs/data, memory protection) / document level (password in your important xls, pdf) / transport level (data in transit is always encrypted), data cleanups, entitlements / accountability, real-time monitoring of processes, start-up points, etc. also regular mass-forensics exercises (recurring light forensics, light agents constantly feeding critical and minimal data to a data crunching device that spits out anomalies based on data stacking, LFO, clustering analysis). Focus on managed irresponsibility i.e. letting employees do whatever they do knowing that they will do anyway and being prepared for it to happen (your role is to know immediately when they install something stupid). Focus on decoys, planted data that can help in a quick discovery of a breach (e.g. your ‘planted’ credentials or credit card numbers discovered on pastebin) and can also feed attackers with lots of noise (e.g. if you use a POS software ask the vendor to introduce fake track data in memory of their program for the RAM scrapers to harvest – this is a few lines of code).
    • Give more attention to offers that are politically neutral
      • It’s very tempting to take a revenge; it’s also very tempting and easy to fall for the promises such as ‘we will catch these bad guys and punish/expose them for you'; and even more tempting if the promise is based on somehow patriotic (often jingoistic), or idealistic rationale that is easy to associate yourself with; the reality is that your org will benefit more from securing your assets than chasing after attackers. Again, you are not Charles Bronson, you are just an employee. Also, defensive security is not that boring – active hunting for badness can be very challenging and rewarding if you get it right and start seeing things traditional security controls miss.
      • Remember: assume that at any point of time some of your boxes are popped – treat it as a constant. Chasing after attackers is simply resources put in a wrong place. Yes, you may eventually catch some, dox some, even may get them prosecuted, but it won’t stop new ones coming. People still smuggle drugs into countries knowing the capital punishment is there in place. Crime is here to stay. Anyone telling you that knowing your adversaries is important should be telling you that it is far more important to protect your assets.
      • How to recognize companies that are politically neutral? Their focus is on technology more than politics:
        • They talk about an attacker’s IP being assigned to a country X vs. an attacker _is_ from a country X.
        • They joke with you about ‘APT’, because they know your org simply sucks at protecting its assets.
        • They avoid ‘nation-sponsored’ narrative all over the place as they understand the attribution is a very difficult problem and there is far more opportunistic attacks happening out there.
        • They don’t whine about other security controls being bad/worse. There is not a single one that is better or worse as its like comparing oranges and apples. Security controls are technical tools that are meant to be used to solve specific technical problems and counter-measure threats with a certain level of efficiency, not an ammo in political discussions. An engineer whining about the tools is an indication that s/he doesn’t know what they are for and how to use them. (Okay, fair enough, some products are pure crap, but discussion about vendors’ offering is different from a discussion about security controls).
        • They talk about bad guys being everywhere, not just having specific country-, region-specific origins; espionage is as important to nation-sponsored hackers as it is to your competition around the corner in your own country.
        • They are usually bitter, cynical, non-compromising assholes that will piss you off, but in your own interest.
        • They of course like memes and cats :)
  • Bring more women to IR
    • Not much to explain here; there are women in IT Sec that possess razor-sharp minds and there is definitely a lot more out there that could add a lot of value to this male-dominated industry; if only given a chance; hire, train, profit
  • Become culturally interested. I will explain why in a second.
    • If you don’t travel, start. And I do not mean for business. For leisure.
    • Visit countries of your “worst enemies” and “adversaries”. Meet real people in their own country. Eat their food, sleep in their place. Experience their weather. See the stars they see. Face your prejudice about the country and people. Be surprised. Or not. Form your own opinion.
    • If you live in US, Canada, Australia it’s a bit hard to travel to other countries for leisure as it’s quite far and pricey. Start small. Visit other districts in your city (if safe), visit other states. There is such a huge variety between different states, cities – you will definitely find something new and it will change your perception of your own country (word of warning: not everything you will see you may like). Avoid organized tours, go on your own. And avoid cheezy entertainment that will make your pockets empty, but not really enriching you in any way – go to see natural wonders, sleep in the motel, hike.
    • Visit art galleries, go and listen to concerts, watch a theater show, a musical, a ballet.
    • Question media. Your news, the movies you watch are full of propaganda – I will cover it a bit in my next post. It’s the winners that write the history and it’s easy to fall prey to a huge propaganda machine that is TV. The roles are assigned and follow specific political agendas. And mind you, there is no conspiracy theory here. If you are trained for decades to follow certain ideals and patterns you will become the most eager proponent of these ideals w/o even noticing. What works in regimes on a propaganda level works also (and imho often better) in the free countries for a simple reason that we are all gullible, complacent animals and don’t want to think too much. Best, consider avoiding watching TV completely. Alternatively, make an effort to watch news from other countries. It’s a very refreshing experiment – watch CNN, FOX, BBC, RT, NHK, Al Jazeera, Deutsche Welle, Euronews, and whatever else you can find. See how the same news are reported. Pay attention to language, perspective offered. Make a game of distilling facts from opinions.
    • Watch foreign movies. Hollywood produces a lot, but lots of it is of poor quality. Go and rent ‘world cinema’. Watch Japanese, Korean, Chinese movies, and TV dramas. Watch movies from Middle East, Australia, New Zealand, Scandinavia, France, Russia, Urugway, Chile, Brazil, and whatever else you can find. Look for various movies, fall asleep on some. Enjoy others.

Now, the ‘why’. The reason is that in IR (and even more in DF) you need to be damn objective. And if you are soaked in a prejudice against specific nation, race, ethnicity then you have a problem. And not just one, but perhaps two. If you are unaware of your prejudice then you are not only unable to change, but also not able to judge fairly. Inability to question your own preconceptions, or what is the cause of them is a progress blocker. You will remain biased, and you may actually hurt people. And majority of people are the good guys.

As an experiment think for a second of North Korea. What is your first reaction? If it is positive, why? if it is negative, why? You may follow with a few basic questions: how many cities in NK can you name? what is the currency? what is the exchange rate to your currency? what is the most popular name for a girl, for a boy? I tried to answer these questions and I failed. My knowledge about NK is pretty much non-existing. I personally know one person who visited NK, but it doesn’t change my knowledge about the country a bit. I am actually a complete ignorant when it comes to NK. You may be surprised to realize that your knowledge about NK is not that far from mine. Yet you may have a pretty strong and negative opinion and perception of this country, or at least its government. And not only because of an alleged association with the recent Sony hack, the movie, but perhaps also thanks to a never ending negative campaign in media. I was curious about it and came up with some numbers and I will cover it a bit more in my next post. It is a very interesting topic for a study and I do hope someone pursuing their degrees in social, political and media studies can explore it more.

And back to the main topic. One more item on the wishlist – please avoid military comparisons – as long as you work for a company, there is no ‘enemy’, you are not combating anyone, you simply have vandals and thieves on your property – it’s your job to possibly prevent them from entering, and if it didn’t work – ensure they can’t steal much, you catch them early, kick them out and move on. That’s it.

If there is one single narrative that should drive security forward it is probably data protection at its core. The data cleanups (all the residual stuff accumulated over years, computers moved from one user to another, etc.) are very difficult to do so the easiest way would be to simply start from the scratch. And not just once, but on regular basis. Difficult, but not impossible. And as crazy as it sounds, ‘being done like Sony’ can lead companies to a very bright future, data-security wise. Perhaps the best way to start a new year is to wipe it all out clean.

You got the well-paid IT Security job. Now what?

December 4, 2014 in Preaching


Writing requires a lot of iterations. Many preaching posts I write are written on a whim, and are often quite clumsy in they first few iterations. The intention is not to rant or offend, but to describe the reality (even if subjectively perceived and to certain level emotionally loaded). I usually review and edit them often as I believe giving them a second and third look makes them better and also removes parts that I think are not objective, or fair (if I spot them).

Old post

A while ago I have read a brilliant article from Jeff Haden that resonates extremely well with my own feelings regarding the company-employee relationship. While the title of the article is ‘What Bosses Should Never Ask Employees to Do’ it actually goes beyond that and pretty accurately describes various experiences one often encounters after joining companies. This is an interesting topic as it it somehow touches IT Security as well. And not only the problem of institutionalization of IT Security jobs and making it yet another job title suffering from the ‘corporate drone’ stigma, but a serious problem of destructive forces that try to ‘corporatize’ IT security in general. And by that I mean attempts to ‘fit’ IT Security within one of many boxes Corporate World has to offer. IT Security is not about ticking boxes on the form. And even ticking boxes cannot really prepare you for what you will be facing as an employee. If you do CISSP, CISM and SANS courses, know how to create ROP gadgets, write rootkits, reverse engineer firmwares it’s still not enough. This is because nowadays the IT Security job (non-consulting, non-vendor) is very often a mixture of some technical stuff with a crazy administrative overhead plus of course corporate “administra-trivia”. And unless CISO gives you a 100% power to ‘own’ the company security-wise and exclude you from all the corporate drama you are just yet another brick in the wall doing what everyone else in the company is doing (taking effectively a lot of your cycles away – cycles you could spend e.g. hunting for attackers / intrusions – I covered it in my older preaching post).

The aforementioned article inspired me to write this very post with a focus on the more technical IT Security jobs (DFIR, RCE, pentest). And not on the job market per se, but on the actual “what to expect after you join the company” bit. And there is a lot happening APART from the cool, technical stuff we all expect to do all day long. Having worked for various companies and types of companies over last 15 years I believe I can offer some insight to this topic and hopefully make it to be an useful guide to these who are either actively looking, or are currently considering changing the companies. I don’t want to rewrite Dilbert’s stories here, but offer the real world-experience of nuisances and idiosyncrasies you will encounter in your career.

The following is a very subjective perspective on the subject – I write it from my own personal experience so take it lightly and draw your own conclusions – more importantly, try to wear different shirts yourself and you will be able to form your own opinion, or compare it to mine.

I personally don’t believe it is a wise choice for anyone to work in the same function for more than 2-3 years (as an employee; if you are a partner, share holder, etc. it’s a different story). It not only leads to complacency and boredom, but it also limit companies’ growth – it’s the change on the staff level that drives changes in the companies (by comparison, CEOs are guys jumping the ship or following a better opportunity all the time; why an average employee should not benefit from similar change, and employee’s company as well?).

One can move inside the company, but the most beneficial career-wise is to go and try working for someone else. It is a very tiring process long-term as you need to rebuild your reputation from the scratch, but it’s very rewarding – you collect experiences from various environments and contact with various people and sooner or later become a ‘generalist diver'; a guy who knows a lot about a lot of things and can also dive into some specific area with a relative ease. It’s certainly not for everyone, so again, one last reminder that it’s my subjective and biased opinion – what works for me, doesn’t necessary mean it works for anyone else. And no, I am not ‘know it all’. I learn the same way as you do. I also don’t mean to offend anyone. I love technical stuff, respect CISSP and CISM knowledge, but also prefer solitude than socializing, because my heart is closer to ‘technical’ than ‘social’ –  if you roll the same way, this post may be handy for you. If not, you have been warned :-)

Borrowing the list of headers from Jeff’s article, we get a list of these:

  • Make employees feel they should attend “social” events.
  • Make employees feel they should donate to a charity.
  • Ask an employee to do something another employee was asked to do.
  • Cause employees to go without food at mealtime hours.
  • Ask employees to evaluate themselves.
  • Ask employees to evaluate their peers.
  • Reveal personal information in the interest of “team building.”
  • Ask employees to alert them when they “veer off course.”
  • Ask employees to do something they don’t do.

The IT Security job at a bank will tick probably the largest number of these points for you and that’s my target today :).

Before I worked for a bank, I was warned by a friend who said “you will die there”. I didn’t, but your soul certainly dies in a work environment offered by banks (pardon me generalizing here, but from conversations with other people who worked for other banks, the environments seem to be similar). If you are a technical person, a guy who ‘lives’ RCE, DFIR, pentesting you will do yourself a favor if you avoid banks as your employer. Or, if you do join, plan it in a way that you join it ‘for CV only’ and leave it within 2 years. I can guarantee that you won’t survive longer anyway. If you are currently working for a bank, you may disagree with me, you may also have a different motivation to stay, but if you have ‘itchy fingers’ and a constant stimulus to crack stuff then I’d be very surprised if you really enjoyed your job at the moment – as I mentioned earlier, this is my very subjective opinion which is shared by many very technical people I know who ‘tasted the waters’, but it doesn’t necessary mean it’s always correct. I bet amongst all bank environments there are some where some technical people can really grow, but I am probably not far from the truth when i say majority of the technical people ‘suffer’ in these roles. On the other hand, banks are – after all – places where you can be paid the best, so if you are after $$$ that’s where you should go, suck it up, and retire early.

The bank job usually means:

  • A pressure to attend meetings; not just one, but lots of regulars, often non-sensical meetings; you may be requested to constantly talk to your boss, to your peers and in general to ‘network'; when we talk about meetings there are often no agendas, they are ran not on merit, but on ‘interval’ basis – if the meeting is scheduled for 1 h, then you will all sit there for 1 h. The meetings are merely for the sake of … I actually don’t know what. I really don’t know. Conversations with managers often are focused on your personality (why the heck you don’t like networking with people in a physical world), assessment of your peers (gossip & politics), rarely IT Security. Some of the people at the bank that happen to be in managing functions simply ‘live’ this stuff; they can’t operate without their calendars looking like a Christmas tree. Many of them become IT Security Managers after working in completely different function for years, slowly climbing the corporate ladder and hence they don’t even have real credentials to understand what you will be doing, let alone a passion to do technical stuff; they know people and the environment though and from this angle, they are better off and better paid than you. Consider this though: the day you join, they know people and the environment; 6-12 months later you will know the people, the environment, and have your tech skills as well – this will be a time to start thinking of moving on as you won’t be able to beat the permanent residents in their game, but can leverage your gained knowledge to sell it to a different company for more
  • The whole networking thing is funny. I always compare a large company to a town. Imagine you live in a town with 50000 people. Now imagine you start casually chatting with as many these strangers as possible. Because… because someone says so. This can only end one way – superficial chats about weather, and a small-talk in general. This is not to say networking is a wrong thing. The idea is very important. It’s just in the bank the implementation is based on the insurance salesmen principles and it does not work for all types of jobs.
  • On a technical level, you will be using ‘enterprise solutions'; with a few exceptions these are terrible, bloated pieces of software using .NET, Oracle, Java (often very old version) and lots of legacy code loosely put together ‘to make it work'; you won’t be able to use 3rd party software, do research, you also won’t have access to latest tools easily since this is all subject to entitlements and approval from compliance; now on a surface, this is actually a good thing and banks have it organized pretty well – you need permission to install 3rd party software and that’s how it should be; however, as a technical staff you will suffer from being in a position in which to do your job, you need to either lie and install that OllyDbg or Sysinternals w/o permission or join the game of talking the talk and spend time evaluating even more enterprise solutions instead of actually doing the job protecting the company; the ‘enterprise solution evaluation’ is often offered for free as it is actually a QA ride for the enterprise software producers; it’s a perfect symbiosis actually, but if you want to do more than just web interface (tickets, tickets, tickets…), you won’t find it there.
  • You will be participating in doing audits; when I say participate it usually means this: find an appropriate language to provide an answer to the auditor’s question as precisely as possible, without hinting any other problem that could draw attention to them
  • Let’s face the reality: audits & compliance roles have a very limited scope. They may help to structure certain things, but… seriously. This is actually plain naive to codify the whole security into checkboxes. It just doesn’t work this way, because there is always sth you need to catch up with – technology changes and evolves constantly; plus, whoever comes up with the ideas for audits is often not technical enough to actually understand the technology they are auditing :). Security is not even Schneier’s process anymore. It’s a very ‘active participation’ body requiring in-depth knowledge of technology & threats, same as the assets you are protecting. The problems companies face start at the very core level – you need to know what you are protecting, then you need to understand what you are protecting it from, and then how. It’s very complex, but it can become managable – it just takes some time to change the attitude from passive security control management to active hunting for intruders, lots of analysis & again, you DO need the blessing from your CI(S)O.
  • Work for a bank is not full of cons only; there is plenty for slack time and for me personally, I’d have 2h of actual work every day, would be paid for it well and could use the rest of the time to learn something else; Hah, this is also a good opportunity to learn MS Outlook rules – brutal decision of what to read, what to ‘select all and mark as read’ and you are set. The number of emails you will be getting at a bank is crazy. It’s no wonder alerts are missed as questioning why we even receive all this spam is not well-received. In many aspects it’s all about people who manage you – if you are lucky you will get a manager who is quite technical and has a strong business  acumen. Ideally, a guy who worked in the consulting world before and is a natural leader. If not, you better look for new opportunities :)

Good luck in your career-changing decisions.