Does Microsoft UI Automation Framework work with C

Question

I am working on an automation project, in which I need to capture the activities [ application launched, data entered, input type etc.] user performs on a desktop. I came across Microsoft UI Automation framework which so far works well for native windows based applications like MS Office, .NET apps etc. However I did not find any useful information / samples of capturing the information from different web browsers [Chrome is a must], Python apps, Java Apps etc. Can someone please confirm whether MS UI Automation Framework supports such apps. Any working example to extract user activities from these apps would be highly appreciated. Thanks.

Answer 1

Some additions to Simon's answer...

Chrome page content can be seen by UIA if you run chrome --force-renderer-accessibility. Only for existing Chrome process it won't work. Though user can create a new tab chrome://accessibility manually and enable UIA for all or some chosen pages. This method also works for AT-SPI accessibility technology on Linux. Of course, Selenium WebDriver is an industry standard here. But another way exists. Both Mozilla and IE support UIA by default.

Inspect.exe can be simply downloaded from this GitHub repo.

Regarding Java apps it depends on the app type. Your chances is about 50/50.

WxPython or PyQt5 are good for UIA. TkInter or Kivy apps are not.

P.S. There is an example how to drag a file from explorer.exe and drop to Google Drive in Chrome using Python library pywinauto.

Answer 2

Chrome only supports UI Automation for toolbars, tabs, menu, buttons around the web page. Everything that's rendered as a web page is not seen by UIA.

For the web page content, the easiest way is to use Selenium (driven by the ChromeDriver), which is kind of a de facto standard for browsers, and has nothing to do with UIA.

To test if an app supports UIA, and how far it does, it's very easy, just run UIA's Inspect tool and check the UI tree over that application.