Many GUI frameworks use their own drawing primitives and don’t necessarily rely on the built-in windows controls (examples include Delphi applications, Microsoft Office, Internet Explorer, Firefox, Chrome, Lotus Notes, .NET applications, etc.). As such, obtaining text visible on some window controls e.g. buttons, menus, tabs, custom edit controls, etc. using GetWindowText, or sending/posting WM_GETTEXT message doesn’t work most of the time (this is btw. how the old password sniffers used to work).

Over 10 years ago I was researching a way to obtain a window text from such non-standard windows controls. This is how I came across a AccessibleObjectFromPoint API and the IAccessible interface – these belong to what is now a part of UI Automation and Microsoft Active Accessibility APIs.

I decided to write about it, because today I suddenly remembered my PoC program I wrote back then – it was a simple GUI application that was registering a hook, reacting to a CTRL+RIGHT MOUSE CLICK sequence. When activated it was calling AccessibleObjectFromPoint API for a current mouse cursor position to obtain the pointer to the IAccessible interface for the GUI element underneath. The interface could be then queried for the name and value of the UI control. Luckily, many unusual framework support this interface enabling us to query non-standard controls.

To my surprise my PoC still works on Windows 10 (although not for all controls). The below screenshot shows the debug window (top left) and the moment I copy the text (bottom) from the Microsoft Edge’s tab (top right; normally this text is not ‘copypasteable’).

Encouraged by this re-discovery I started reading what has changed since the time I wrote my PoC and I guess you know why. Being able to sniff on the User Interface is a pretty neat way of stealing information w/o engaging traditional means of sniffing used by keyloggers.

I was no disappointed, turns out there is an even great PoC written and maintained by Microsoft which I can use in this post w/o needing to write any code. The program is called Inspect.exe and can be found inside the Windows SDK e.g.:

• c:\Program Files (x86)\Windows Kits\10\bin\x86\inspect.exe

When you launch it, it will show you all the windows currently present on your desktop:

In my case, you can see an open Total Commander, Microsoft Edge, Calculator, and Process Hacker, Program Manager and Inspect itself.

The interesting stuff happens as you start using these applications with Inspect running in a background. The program will ‘follow’ your activity showing you exactly what and when was clicked, what menu was selected, text entered, etc.

For example, if I launch Notepad, then go to File -> New it will be immediately tracked by the Inspect program.

And what happens when I type text in the Notepad?

This applies to most windows on the screen.

There are at least two questions in my mind at this stage:

• How many malware families actively use it for stealing information, if any?
• How many security solutions monitor these accessibility APIs today?

And let’s not forget about the good side of things.

Imagine incorporating a log of these actions (perhaps without intercepting the values, for legal reasons) and storing it in the Log outside of the monitored box? You could basically re-create and re-play the whole activity prior to incident occurring. There is also a huge number of new use cases you could develop based on these scenarios. By linking the malicious activities to actual UI behavior we could establish a strong case for finding both malicious external attackers and bad insiders. The volumes may be an issue, but this is not going to be problem long-term.

UI Anomaly Hunting anyone?