I must admit that my interest in non-Windows threat hunting is growing. I am pretty bored with 4688, sysmon, and Mitre Att&ck presos, and am now trying to expand my knowledge about other OSes. Not just as a user, but as a guy determined to fully understand what is out there, so I can use this knowledge to detect stuff on non-Windows boxes.
I believe that the time for Windows-only hunting is over, and we will be facing more and more pressure to cover all the angles. The reasons for that are many. OSX becomes very popular as a go-to desktop OS for many companies. Linux is so strongly embedded in the internet ecosystem that there should be no need to even explain anything here… BUT. Let’s try. While PCI DSS and other compliance efforts helped to place many security controls on Linux systems in recent years (AV, FIM, etc.) the truth is that these logs are just sitting there. I am not in a position to judge how many companies look at them today, but I doubt many spend a lot of resources looking into it. And anything ‘cloud’ typically means Windows or Linux, so there is that as well… The good news it that if you look around in your env, maybe lots of logs are already waiting for you to be explored.
So… both OSX and Linux interest me now as a researcher and it’s probably for the first time ever. Some experience with these systems is obviously there, but it is still limited to:
- knowing main commands quite well
- knowing where to find man pages on these I don’t know much of
- quite a lot of data analysis using NIX CLI tools
- reverse engineering software on both platforms
- multi-OS/multi-architecture software development (a.k.a. a program that works on 32- and 64- bit versions of Windows/OSX/Linux after compiled from the same source code)
- compiling some Open Source projects (pain in the neck)
- DFIR work on various systems
- a bit of pentesting, and
- web coding & basic admin
After reading this list you may think I know a lot, but I know I am a n00b. I don’t have the historical background of a true *nix/bsd fan, and am still very Windows-oriented. What’s more… if I can avoid using OSX/Linux, I certainly do so :). Personal preference rally. No matter how n00b I am though, I do have my foot in a door. I find it very interesting to confront my own bias, and I am now facing the very large virtual book of Everything-on-this-subject-aka-RTFM.
Well… the book is not that large actually. At least, not in the threat hunting department.
Before I say anything further, let me one more time emphasize that this is an exploration of a person that is not a fan of OSX/Linux. I am a Windows guy who entered this new area with a lot of gaps to fill. A trustworthy information on OSX/Linux (in a context of TH) is much harder to find, but I expect this to change in the near future. I am actually hoping to contribute to it! On that note… if you find anything wrong, please do throw rotten tomatoes my way, but take your time to point me into a right direction. Thank you.
Today the subject of my research is auditd.
When it comes to logging events associated with a process creation, command execution Windows threat hunters are spoiled by EDRs, sysmon, Event 4688, or Event Tracing, as well as documentation & presos, …
Good news is that auditd is probably the closest you will get today to logging similar events on Linux. It’s pretty awesome. It does exactly what you want. It logs lots of stuff and it’s like the good old Windows days, except you need to:
- change the syntax of your threat hunting queries to accommodate for different command names on Linux (you need to learn them, yes, at least these commonly used by the attackers)
- learn what auditd events are important for your needs
- deal with a typical TH BAU which is crazy amount of noise (so, it’s very much like Windows 🙂 )
For your reference, all the auditd rules are described here. I must mention that ‘all’ is a relative term.
For the moment, since I mentioned process creation and commands, just have a look at these events:
- USER_CMD
- Triggered when a user-space shell command is executed.
- SYSCALL/EXECVE
- Triggered to record a system call to the kernel, in this particular case EXECVE which is associated with an execution of a program.
So, it’s pretty straightforward.
Yes, and no.
Notice that this is a completely different environment, and you need to understand the intricacies of a different OS, different mindset, presence of subtle, hard-to-spot things that are ‘bad’, and that you are not used to, and can’t spot yet, and then … the ‘live’ aspect of any Open Source project. The latter, is the part I am going to touch on a bit later.
In any case, if you have the auditd enabled, I am hoping you are already using it, and if not, do it asap! In my eyes auditd has a lot of potential and I hope to write more about it in the future.
Now…
If you read the subject of this post you know that today the subject of my research is not really auditd, but its ANOM_ rules. When I read description of all the auditd rules (see Red Hat link earlier) I got really excited. I thought… Wowzers… Linux coders are ahead of a game and already put the juicy logs (ANOM*) on the IR plate! Just grab them, alert on them, and handle…
After poking around a bit more though, I realized that when something looks too good to be true, then it’s probably because… it is…
As we know, one of the secrets of a successful internet search is a good keyword. The more unique it is, the better. Because of that I decided to google for ‘ANOM_LOGIN_LOCATION’ which is fairly unique.
The event description says that:
ANOM_LOGIN_LOCATION – Triggered when a login attempt is made from a forbidden location
I was curious what is the ‘forbidden’ location though?
When I searched for it on Google I only got… 181 hits. I was right, it’s an excellent keyword. But … it’s not good. This is far lower results than I expected.
Most of the results referred to a copypasta from the Red Hat page. One of the presos [PDF warning] went a bit further and explained that:
pam_access
● Used to forbid logins from certain locations, consoles, and accounts
● /etc/security/access.conf controls its config
● Sends ANOM_LOGIN_ACCT and ANOM_LOGIN_LOCATION
So… I thought… hmmm if I can’t google for it, what about I have a look at the source code?
After another google session I found this location:
- https://github.com/linux-audit
I assume this is a source code for auditd. If not, please correct me.
I downloaded everything I could from there:
- audit-userspace
- audit-documentation
- audit-testsuite
- audit-kernel
I then unpacked it, and queried all files for ANOM_LOGIN_LOCATION keyword.
To my surprise, I could only find a mention of this keyword in a few files, mostly headers, but none of them included a branch of code that would actually log this event!
Hmmm what about I look for a different event?
I looked for ANOM_LINK which I found in one of the header files (internally, it’s mapped to AUDIT_ANOM_LINK name; interestingly enough, this event is not even listed on the Red Hat page I linked to earlier). When I looked for it in my unpacked files I found this piece of code:
void audit_log_link_denied(const char *operation)
{
struct audit_buffer *ab;
if (!audit_enabled || audit_dummy_context())
return;
/* Generate AUDIT_ANOM_LINK with subject, operation, outcome. */
ab = audit_log_start(audit_context(), GFP_KERNEL, AUDIT_ANOM_LINK);
if (!ab)
return;
audit_log_format(ab, "op=%s", operation);
audit_log_task_info(ab, current);
audit_log_format(ab, " res=0");
audit_log_end(ab);
}
Okay, so we now know that audit_log_start is used to log events, and some events are actually logged.
I wrote a quick & dirty scanner that walked through all files of the auditd source code to find all references to a sequence:
I was hoping to find which events out all listed in the header file (or on Red Hat page) are actually logged.
It only found 20 occurrences.
- AUDIT_CWD
- AUDIT_PATH
- AUDIT_USER,
- AUDIT_SOCKADDR
- AUDIT_FD_PAIR
- AUDIT_SYSCALL
- AUDIT_KERNEL,
- AUDIT_INTEGRITY_EVM_XATTR
- AUDIT_CONFIG_CHANGE
- AUDIT_NETFILTER_PKT
- AUDIT_LOGIN
- AUDIT_SELINUX_ERR,
- AUDIT_SECCOMP
- AUDIT_OBJ_PID
- AUDIT_MAC_POLICY_LOAD,
- AUDIT_MAC_STATUS,
- AUDIT_PROCTITLE
- AUDIT_FEATURE_CHANGE
- AUDIT_ANOM_ABEND
- AUDIT_ANOM_LINK
Hmm I didn’t even see the USER_CMD. I queried all files for ‘log.*?USER_CMD’. Still nothing. Notably, the keyword USER_CMD is referenced in a description of audit_log_user_command function so maybe I need to read source code more. Because libaudit is a library, any source code referencing its functions is probably where the logging happens (I will check bash source code for references).
The preso I refereed to earlier mentioned to ANOM_LOGIN_LOCATION in a context of pam_access module. Googling for these two keywords together brought me only 5 results. None of them referred to the code. Could it be a closed source?
For the sake of completeness, I tried to look at it from a different angle. For example, what events are listed in the source code (headers) vs. what we see on the RH page? The header file I referred to (after removing events that are commented out, or included conditionally), include 173 unique events; see bottom of this post for the whole list. The RH page includes 141 events. So it’s not all events after all.
My conclusion so far is that:
- probably some of these events are not implemented yet
- probably some are not documented yet either
- some of them are called from other projects
- some may be referenced by closed source code projects
- plus some maybe a legacy code that has been no longer updated and been removed
- the complete list of events is ‘alive’ and there is no guarantee that it will last
- source code and documentation is not necessary aligned
- we probably need to comb a lot of source code of various projects to fully understand what logs what, and under what conditions (references to code/actual business logic are very much required)
- defining TH rules may become problematic long term; in order to define rules, we need to understand conditions under which certain events are logged; how to do it w/o looking at the source code? or a full documentation (that most likely doesn’t exist today)
- in other words: lots of challenges
Any other ideas?
In any case, the main reason of this post is that I want to find the source code that logs ANOM_LOGIN_LOCATION.
Where to find it?
And the list from the msg_typetab.h:
-
- AUDIT_USER USER
- AUDIT_LOGIN LOGIN
- AUDIT_USER_AUTH USER_AUTH
- AUDIT_USER_ACCT USER_ACCT
- AUDIT_USER_MGMT USER_MGMT
- AUDIT_CRED_ACQ CRED_ACQ
- AUDIT_CRED_DISP CRED_DISP
- AUDIT_USER_START USER_START
- AUDIT_USER_END USER_END
- AUDIT_USER_AVC USER_AVC
- AUDIT_USER_CHAUTHTOK USER_CHAUTHTOK
- AUDIT_USER_ERR USER_ERR
- AUDIT_CRED_REFR CRED_REFR
- AUDIT_USYS_CONFIG USYS_CONFIG
- AUDIT_USER_LOGIN USER_LOGIN
- AUDIT_USER_LOGOUT USER_LOGOUT
- AUDIT_ADD_USER ADD_USER
- AUDIT_DEL_USER DEL_USER
- AUDIT_ADD_GROUP ADD_GROUP
- AUDIT_DEL_GROUP DEL_GROUP
- AUDIT_DAC_CHECK DAC_CHECK
- AUDIT_CHGRP_ID CHGRP_ID
- AUDIT_TEST TEST
- AUDIT_TRUSTED_APP TRUSTED_APP
- AUDIT_USER_SELINUX_ERR USER_SELINUX_ERR
- AUDIT_USER_CMD USER_CMD
- AUDIT_USER_TTY USER_TTY
- AUDIT_CHUSER_ID CHUSER_ID
- AUDIT_GRP_AUTH GRP_AUTH
- AUDIT_MAC_CHECK MAC_CHECK
- AUDIT_ACCT_LOCK ACCT_LOCK
- AUDIT_ACCT_UNLOCK ACCT_UNLOCK
- AUDIT_USER_DEVICE USER_DEVICE
- AUDIT_SOFTWARE_UPDATE SOFTWARE_UPDATE
- AUDIT_SYSTEM_BOOT SYSTEM_BOOT
- AUDIT_SYSTEM_SHUTDOWN SYSTEM_SHUTDOWN
- AUDIT_SYSTEM_RUNLEVEL SYSTEM_RUNLEVEL
- AUDIT_SERVICE_START SERVICE_START
- AUDIT_SERVICE_STOP SERVICE_STOP
- AUDIT_GRP_MGMT GRP_MGMT
- AUDIT_GRP_CHAUTHTOK GRP_CHAUTHTOK
- AUDIT_DAEMON_START DAEMON_START
- AUDIT_DAEMON_END DAEMON_END
- AUDIT_DAEMON_ABORT DAEMON_ABORT
- AUDIT_DAEMON_CONFIG DAEMON_CONFIG
- AUDIT_DAEMON_ROTATE DAEMON_ROTATE
- AUDIT_DAEMON_RESUME DAEMON_RESUME
- AUDIT_DAEMON_ACCEPT DAEMON_ACCEPT
- AUDIT_DAEMON_CLOSE DAEMON_CLOSE
- AUDIT_DAEMON_ERR DAEMON_ERR
- AUDIT_SYSCALL SYSCALL
- AUDIT_PATH PATH
- AUDIT_IPC IPC
- AUDIT_SOCKETCALL SOCKETCALL
- AUDIT_CONFIG_CHANGE CONFIG_CHANGE
- AUDIT_SOCKADDR SOCKADDR
- AUDIT_CWD CWD
- AUDIT_EXECVE EXECVE
- AUDIT_IPC_SET_PERM IPC_SET_PERM
- AUDIT_MQ_OPEN MQ_OPEN
- AUDIT_MQ_SENDRECV MQ_SENDRECV
- AUDIT_MQ_NOTIFY MQ_NOTIFY
- AUDIT_MQ_GETSETATTR MQ_GETSETATTR
- AUDIT_KERNEL_OTHER KERNEL_OTHER
- AUDIT_FD_PAIR FD_PAIR
- AUDIT_OBJ_PID OBJ_PID
- AUDIT_TTY TTY
- AUDIT_EOE EOE
- AUDIT_BPRM_FCAPS BPRM_FCAPS
- AUDIT_CAPSET CAPSET
- AUDIT_MMAP MMAP
- AUDIT_NETFILTER_PKT NETFILTER_PKT
- AUDIT_NETFILTER_CFG NETFILTER_CFG
- AUDIT_SECCOMP SECCOMP
- AUDIT_PROCTITLE PROCTITLE
- AUDIT_FEATURE_CHANGE FEATURE_CHANGE
- AUDIT_KERN_MODULE KERN_MODULE
- AUDIT_FANOTIFY FANOTIFY
- AUDIT_AVC AVC
- AUDIT_SELINUX_ERR SELINUX_ERR
- AUDIT_AVC_PATH AVC_PATH
- AUDIT_MAC_POLICY_LOAD MAC_POLICY_LOAD
- AUDIT_MAC_STATUS MAC_STATUS
- AUDIT_MAC_CONFIG_CHANGE MAC_CONFIG_CHANGE
- AUDIT_MAC_UNLBL_ALLOW MAC_UNLBL_ALLOW
- AUDIT_MAC_CIPSOV4_ADD MAC_CIPSOV4_ADD
- AUDIT_MAC_CIPSOV4_DEL MAC_CIPSOV4_DEL
- AUDIT_MAC_MAP_ADD MAC_MAP_ADD
- AUDIT_MAC_MAP_DEL MAC_MAP_DEL
- AUDIT_MAC_IPSEC_ADDSA MAC_IPSEC_ADDSA
- AUDIT_MAC_IPSEC_DELSA MAC_IPSEC_DELSA
- AUDIT_MAC_IPSEC_ADDSPD MAC_IPSEC_ADDSPD
- AUDIT_MAC_IPSEC_DELSPD MAC_IPSEC_DELSPD
- AUDIT_MAC_IPSEC_EVENT MAC_IPSEC_EVENT
- AUDIT_MAC_UNLBL_STCADD MAC_UNLBL_STCADD
- AUDIT_MAC_UNLBL_STCDEL MAC_UNLBL_STCDEL
- AUDIT_MAC_CALIPSO_ADD MAC_CALIPSO_ADD
- AUDIT_MAC_CALIPSO_DEL MAC_CALIPSO_DEL
- AUDIT_ANOM_PROMISCUOUS ANOM_PROMISCUOUS
- AUDIT_ANOM_ABEND ANOM_ABEND
- AUDIT_ANOM_LINK ANOM_LINK
- AUDIT_INTEGRITY_DATA INTEGRITY_DATA
- AUDIT_INTEGRITY_METADATA INTEGRITY_METADATA
- AUDIT_INTEGRITY_STATUS INTEGRITY_STATUS
- AUDIT_INTEGRITY_HASH INTEGRITY_HASH
- AUDIT_INTEGRITY_PCR INTEGRITY_PCR
- AUDIT_INTEGRITY_RULE INTEGRITY_RULE
- AUDIT_KERNEL KERNEL
- AUDIT_ANOM_LOGIN_FAILURES ANOM_LOGIN_FAILURES
- AUDIT_ANOM_LOGIN_TIME ANOM_LOGIN_TIME
- AUDIT_ANOM_LOGIN_SESSIONS ANOM_LOGIN_SESSIONS
- AUDIT_ANOM_LOGIN_ACCT ANOM_LOGIN_ACCT
- AUDIT_ANOM_LOGIN_LOCATION ANOM_LOGIN_LOCATION
- AUDIT_ANOM_MAX_DAC ANOM_MAX_DAC
- AUDIT_ANOM_MAX_MAC ANOM_MAX_MAC
- AUDIT_ANOM_AMTU_FAIL ANOM_AMTU_FAIL
- AUDIT_ANOM_RBAC_FAIL ANOM_RBAC_FAIL
- AUDIT_ANOM_RBAC_INTEGRITY_FAIL ANOM_RBAC_INTEGRITY_FAIL
- AUDIT_ANOM_CRYPTO_FAIL ANOM_CRYPTO_FAIL
- AUDIT_ANOM_ACCESS_FS ANOM_ACCESS_FS
- AUDIT_ANOM_EXEC ANOM_EXEC
- AUDIT_ANOM_MK_EXEC ANOM_MK_EXEC
- AUDIT_ANOM_ADD_ACCT ANOM_ADD_ACCT
- AUDIT_ANOM_DEL_ACCT ANOM_DEL_ACCT
- AUDIT_ANOM_MOD_ACCT ANOM_MOD_ACCT
- AUDIT_ANOM_ROOT_TRANS ANOM_ROOT_TRANS
- AUDIT_ANOM_LOGIN_SERVICE ANOM_LOGIN_SERVICE
- AUDIT_RESP_ANOMALY RESP_ANOMALY
- AUDIT_RESP_ALERT RESP_ALERT
- AUDIT_RESP_KILL_PROC RESP_KILL_PROC
- AUDIT_RESP_TERM_ACCESS RESP_TERM_ACCESS
- AUDIT_RESP_ACCT_REMOTE RESP_ACCT_REMOTE
- AUDIT_RESP_ACCT_LOCK_TIMED RESP_ACCT_LOCK_TIMED
- AUDIT_RESP_ACCT_UNLOCK_TIMED RESP_ACCT_UNLOCK_TIMED
- AUDIT_RESP_ACCT_LOCK RESP_ACCT_LOCK
- AUDIT_RESP_TERM_LOCK RESP_TERM_LOCK
- AUDIT_RESP_SEBOOL RESP_SEBOOL
- AUDIT_RESP_EXEC RESP_EXEC
- AUDIT_RESP_SINGLE RESP_SINGLE
- AUDIT_RESP_HALT RESP_HALT
- AUDIT_RESP_ORIGIN_BLOCK RESP_ORIGIN_BLOCK
- AUDIT_RESP_ORIGIN_BLOCK_TIMED RESP_ORIGIN_BLOCK_TIMED
- AUDIT_USER_ROLE_CHANGE USER_ROLE_CHANGE
- AUDIT_ROLE_ASSIGN ROLE_ASSIGN
- AUDIT_ROLE_REMOVE ROLE_REMOVE
- AUDIT_LABEL_OVERRIDE LABEL_OVERRIDE
- AUDIT_LABEL_LEVEL_CHANGE LABEL_LEVEL_CHANGE
- AUDIT_USER_LABELED_EXPORT USER_LABELED_EXPORT
- AUDIT_USER_UNLABELED_EXPORT USER_UNLABELED_EXPORT
- AUDIT_DEV_ALLOC DEV_ALLOC
- AUDIT_DEV_DEALLOC DEV_DEALLOC
- AUDIT_FS_RELABEL FS_RELABEL
- AUDIT_USER_MAC_POLICY_LOAD USER_MAC_POLICY_LOAD
- AUDIT_ROLE_MODIFY ROLE_MODIFY
- AUDIT_USER_MAC_CONFIG_CHANGE USER_MAC_CONFIG_CHANGE
- AUDIT_CRYPTO_TEST_USER CRYPTO_TEST_USER
- AUDIT_CRYPTO_PARAM_CHANGE_USER CRYPTO_PARAM_CHANGE_USER
- AUDIT_CRYPTO_LOGIN CRYPTO_LOGIN
- AUDIT_CRYPTO_LOGOUT CRYPTO_LOGOUT
- AUDIT_CRYPTO_KEY_USER CRYPTO_KEY_USER
- AUDIT_CRYPTO_FAILURE_USER CRYPTO_FAILURE_USER
- AUDIT_CRYPTO_REPLAY_USER CRYPTO_REPLAY_USER
- AUDIT_CRYPTO_SESSION CRYPTO_SESSION
- AUDIT_CRYPTO_IKE_SA CRYPTO_IKE_SA
- AUDIT_CRYPTO_IPSEC_SA CRYPTO_IPSEC_SA
- AUDIT_VIRT_CONTROL VIRT_CONTROL
- AUDIT_VIRT_RESOURCE VIRT_RESOURCE
- AUDIT_VIRT_MACHINE_ID VIRT_MACHINE_ID
- AUDIT_VIRT_INTEGRITY_CHECK VIRT_INTEGRITY_CHECK
- AUDIT_VIRT_CREATE VIRT_CREATE
- AUDIT_VIRT_DESTROY VIRT_DESTROY
- AUDIT_VIRT_MIGRATE_IN VIRT_MIGRATE_IN
- AUDIT_VIRT_MIGRATE_OUT VIRT_MIGRATE_OUT
