GUI Automation with pyAA

Microsoft Active Accessibility (MSAA) is a library for the Windows platform that allows client applications inspect, control, and monitor events and controls in graphical user interfaces (GUIs) and server applications to expose runtime information about their user interfaces. The purpose of MSAA is to make possible the adaptation of GUIs to other formats to improve software accessibility for users with visual, hearing, physical, and other impairments. In addition, MSAA can be used to good effect to automate GUI testing.

This tutorial introduces pyAA, an object oriented Python wrapper around the client-side functionality in MSAA. The tutorial demonstrates methods for obtaining references to interface objects, gives examples of inspecting and controlling GUI components, and describes how application and system events can be monitored. Timing, instability, and counting issues are also discussed.

Prerequisites

pyAA 2.0 or higher
Microsoft Active Accessibility 2.0 redistributable (for Windows versions prior to XP)
Mark Hammond's Python win32all extensions (for some examples)
Example code

The MSAA Object Model

MSAA exposes the widgets of a graphical user interface as nodes in a tree structure that we will refer to as an accessible object model (AOM). Each node in the AOM represents some component while the edges in the tree represent connections between parent and child objects. From an overly simplistic point of view, the root node in the AOM is the Desktop window, its children are top level application windows, their children are container objects holding controls, and so on. In reality, the content of the AOM is much more complex. For example some components that appear on the screen are not represented in the AOM, invisible objects appear in the AOM but not on the screen, and the Desktop window is not always the root of the tree.

This diagram depicts a portion of a typical AOM tree. The Notepad application window is at the root of the tree. The first four of its seven immediate children are a menu bar, title, menu bar, and client. The client node has two immediate children, both of the window type. The first of these window nodes has seven children, the fourth of which is of the type editable text. This editable text node is the main document area in the Notepad application. Note that this important control is three levels deep in the AOM tree.

Accessible Objects

Programmatically, the components in the AOM tree are represented by instances of the AccessibleObject (AO) class in pyAA. The properties of this class give the name, value, role (i.e. type), state, etc. of the components it represents. The methods of the class support navigation among components, setting the focus and selection, and triggering actions.

The AccessibleObjectFromPoint function returns a reference to the topmost component at the given location but at the lowest level in the AOM tree. For instance, if two windows overlap at the given point, an AO representing a component in the top window will be returned. AOs for light child components such as items in a list, items in a tree, menu items in a menu, etc. are not returned by this function. Instead, AOs for their containing lists, trees, menus, etc. are returned instead.

From Window Handles

Another straightforward way of getting an AccessibleObject is by passing a window handle to the AccessibleObjectFromWindow function.

import pyAA
import win32gui
# get the desktop window handle
hwnd = win32gui.GetDesktopWindow()
# we want the desktop window
objid = pyAA.Constants.OBJID_WINDOW
# get the object
ao = pyAA.AccessibleObjectFromWindow(hwnd, objid)
print ao.Name

The second parameter to this function specifies the part of a object that the returned AO should represent. See the pyAA documentation for the Constants class for a listing of all available object IDs.

From Nearby AOM Nodes

Once a reference to an AccessibleObject has been obtained, it can be used to get AOs for its parent and children in the AOM.

import pyAA
import win32gui    
 
# get the desktop window handle
hwnd = win32gui.GetDesktopWindow()
# we want the desktop window
objid = pyAA.Constants.OBJID_WINDOW
# and its accessible object
top = pyAA.AccessibleObjectFromWindow(hwnd, objid)
print top.Name
print
 
# get all the children of the desktop
for c in top.Children:
  print c.Name
print
 
# get the parent of the first child
# it's the desktop again
p = top.Children[0].Parent
print p.Name

Likewise, methods exist to move through sibling nodes in the AOM.

import pyAA
import win32gui    
 
# get the desktop window handle
hwnd = win32gui.GetDesktopWindow()
# this time we want the desktop client area
objid = pyAA.Constants.OBJID_CLIENT
top = pyAA.AccessibleObjectFromWindow(hwnd, objid)
 
# get the first application window
item = top.Navigate(pyAA.Constants.NAVDIR_FIRSTCHILD)
while 1:
  print item.Name, item.RoleText
  try:
    # try to get the next logical window
    item = item.Navigate(pyAA.Constants.NAVDIR_NEXT)
  except:
    break

In addition to moving logically through sibling nodes and to the first child node, pyAA supports spatial navigation up, down, left, and right as well as navigation to the last child node. Again, see the pyAA documentation for more details.

From Exact Paths

The AccessibleObject for a descendant node of the node represented by a currently held AO can be retrieved using an XPath-like notation.

import pyAA
import win32gui    
 
# get the desktop window handle
hwnd = win32gui.GetDesktopWindow()
objid = pyAA.Constants.OBJID_CLIENT
top = pyAA.AccessibleObjectFromWindow(hwnd, objid)
 
# get the last object, which should be the program manager
pm = top.Navigate(pyAA.Constants.NAVDIR_LASTCHILD)
 
# get the list of desktop items
dlst = pm.ChildFromPath('/client[3]/window[0]/client[3]/window[0]/list[3]')
for c in dlst.Children:
  print c.Name

The ChildFromPath method accepts a string representing a path rooted at the AO object on which the method is called. Each segment of the path indicates a navigation down one level in the AOM. The text in the segment states the role of the node that should be selected next while the number in brackets indicates the node's absolute position in the list of children of its parent node. For example, the path /window[3]/client[10]/list[0] references the first child node (a list) of the tenth child node (a client) of the third child node (a window) of the current AccessibleObject.

An enhanced version of pyAA might support more flexible paths by allowing the programmer to specify names, values, states, and other properties in addition to roles.

From a Search

When the exact path to a desired descendant is not known, the FindOneChild and FindAllChildren methods can be used to locate a given component meeting some user specified criteria.

import pyAA
import win32gui    
 
# get the desktop window handle
hwnd = win32gui.GetDesktopWindow()
objid = pyAA.Constants.OBJID_CLIENT
top = pyAA.AccessibleObjectFromWindow(hwnd, objid)
 
# search for the desktop list
list_pred = (lambda node: node.Name == 'Desktop' and
             node.Role == pyAA.Constants.ROLE_SYSTEM_LIST)
dlst = top.FindOneChild(list_pred)
print 'All desktop items'
for c in dlst.Children:
  print c.Name
print
 
# search for all Python file icons on the desktop
py_pred = (lambda node: node.Name.find('.py') &gt; -1 and
           node.Role == pyAA.Constants.ROLE_SYSTEM_LISTITEM)
print 'Desktop items ending in .py'
for c in dlst.FindAllChildren(py_pred):
  print c.Name

Of course, using these methods is much slower than specifying a known path since all descendant nodes must be searched until a match is found for FindOneChild or exhaustively for FindAllChildren.

From Events

Finally, AccessibleObjects can be obtained from system wide events such as window creation and destruction. This method is explained in detail in the Event watcher section below.

Properties and Methods

The examples so far have accessed just a few properties of the any AccessibleObject retrieved (e.g. Name, Children, Parent). Many other properties exist, however, and provide other bits of useful information about components on the screen.

import pyAA
import win32gui  
 
# get the desktop window handle
hwnd = win32gui.GetDesktopWindow()
objid = pyAA.Constants.OBJID_CLIENT
top = pyAA.AccessibleObjectFromWindow(hwnd, objid)
 
# search for the desktop list
list_pred = (lambda node: node.Name == 'Desktop' and
             node.Role == pyAA.Constants.ROLE_SYSTEM_LIST)
dlst = top.FindOneChild(list_pred)
 
# print all the information we can get about the desktop list
print 'Name:', dlst.Name
print 'Value:', dlst.Value
print 'State:', dlst.State
print 'State text:',dlst.StateText
print 'Role:', dlst.Role
print 'Role text:', dlst.RoleText
print 'Description:', dlst.Description
print 'Child count:', dlst.ChildCount
print 'Location and size:', dlst.Location
print 'Window handle:', dlst.Window
print 'Keyboard shortcut:', dlst.KeyboardShortcut
print 'Selected item:', dlst.Selection
print 'Process and thread IDs:', dlst.ProcessID
print 'Path rooted at desktop', dlst.Path
print 'Child ID:', dlst.ChildID

The AccessibleObject class also supports methods to set the focus, select and unselect list items, and perform the default action on a control to name a few.

import pyAA
import win32gui  
 
# get the desktop window handle
hwnd = win32gui.GetDesktopWindow()
objid = pyAA.Constants.OBJID_CLIENT
top = pyAA.AccessibleObjectFromWindow(hwnd, objid)
 
# search for the desktop list
list_pred = (lambda node: node.Name == 'Desktop' and
             node.Role == pyAA.Constants.ROLE_SYSTEM_LIST)
dlst = top.FindOneChild(list_pred)
 
# select all items on the desktop
first = dlst.Navigate(pyAA.Constants.NAVDIR_FIRSTCHILD)
last = dlst.Navigate(pyAA.Constants.NAVDIR_LASTCHILD)
first.Select(pyAA.Constants.SELFLAG_TAKESELECTION|pyAA.Constants.SELFLAG_TAKEFOCUS)
last.Select(pyAA.Constants.SELFLAG_EXTENDSELECTION)

Event Watchers

Another fundamental component of the pyAA library is the Watcher class. Instances of this class allow an application to register callbacks for system wide events such as control name and value changes, control state changes, foreground window changes, and so forth. See the EVENT_SYSTEM_* constants in the pyAA documentation for a complete list of events.

The WindowWatcher class in pyAA derives from the base Watcher class to provide the ability to watch for window openings and closings. To use this class, a client application creates an instance and calls either the NotifyOnOpen or NotifyOnClose function. The return value of this function is a Deferred object, a placeholder for a result to be returned later. The client then calls the AddCallback method on the Deferred object to register one or more callbacks.

When an event occurs, the callbacks are invoked in the order in which they are registered. The first callback is passed the AccessibleObject associated with the event. The return value of this function is passed to the next callback and so on for all registered callbacks.

import os, time
import pyAA
import pythoncom  
 
# global indicates if message pump continues
alive = True
 
def WindowClosed(ao):
  # quit the message pump
  global alive
  alive = False
 
# the ao is probably dead by now, so just return None
  return None
 
def WindowOpened(ao):
  # now watch for the window closing
  ww = pyAA.WindowWatcher()
  r = ww.NotifyOnClose(ao.Window)
  r.AddCallback(WindowClosed)
  r.AddCallback(Output, 'closed')
 
# whatever is returned by this function is passed
  # to the next registered callback function
  return ao.Name
 
def Output(name, action):
  print name, action
 
if __name__ == '__main__':
  # watch for instances of windows explorer
  ww = pyAA.WindowWatcher()
  r = ww.NotifyOnOpen(cls='CabinetWClass')
  # multiple callbacks can be added to deferred result
  r.AddCallback(WindowOpened)
  r.AddCallback(Output, 'opened')
 
# run windows explorer
  os.startfile('')
 
# a message pump is needed to receive events
  while alive:
    pythoncom.PumpWaitingMessages()
    time.sleep(0.1)

When the above example code is run, an instance of Windows Explorer is started. When the Explorer window appears, the example program prints the name of the window and waits for the window to close. When the Explorer window is manually closed, the example program prints a message and quits.

Watching for other kinds of events is possible by deriving a new class from Watcher. Setting a hook for system wide events is accomplished using the AddWinHookEvent method in the base class. The parameters to this method can include the type of event(s) to monitor, the function to call when the event(s) fire, the type of object to monitor, the thread or process to monitor, and the window handle to monitor. To unset an event hook, simply call the Release method in the base Watcher class.

import os, time
import pyAA
import pythoncom  
 
class LocationWatcher(pyAA.Watcher):
  def Start(self):
    # create a deferred
    self.result = pyAA.Deferred()
    # register a hook to watch for window movement
    self.AddWinEventHook(callback=self.OnLocationEvent,
                         event=pyAA.Constants.EVENT_OBJECT_LOCATIONCHANGE,
                         obj_id=pyAA.Constants.OBJID_WINDOW)
    # return the deferred
    return self.result
 
def OnLocationEvent(self, event):
    # create a new deferred
    r = pyAA.Deferred()
    # return the event and the deferred
    self.result.Callback(event, r)
    # store the new deferred
    self.result = r
 
def Output(event, deferred):
  # print the object type and its location
  print event.ObjectName, event.AccessibleObject.Location
  # register this function again for the next event callback
  deferred.AddCallback(Output)
 
if __name__ == '__main__':
  # watch for instances of windows explorer
  ww = LocationWatcher()
  r = ww.Start()
  r.AddCallback(Output)
 
# a message pump is needed to receive events
  while 1:
    pythoncom.PumpWaitingMessages()
    time.sleep(0.05)

While the example program above is running, it will print the location of any window moved on the screen. There is no exit condition so it will continue to monitor location change events until closed.

Caveats

Timing

Components that are not fully initialized may sometimes appear in the AOM tree. When their properties are accessed through an AccessibleObject, a pyAA.Error exception or, worse, incorrect information may be returned. An application might choose to implement an intelligent time delay after any event that changes the state of the user interface to ensure all components in an interface are ready for use.

Instability

Any node in the AOM tree may change or be destroyed at any time, even while a client application is holding an AccessibleObject representing it. pyAA eases the pain of dealing with this very real possibility by throwing Python exceptions when a AccessibleObject is no longer valid. A client application must be prepared to handle the instability of the AOM by catching these exceptions, dropping held references to dead AccessibleObjects, and obtaining references to new AccessibleObjects for replaced AOM nodes.

Counting

The ChildCount property of the AccessibleObject class does not always give an accurate count of items in collections. For instance, a node with the role of list in the AOM may have a number of list items as its children representing the items in the list and a window object containing any column headers. In this case, the ChildCount property will be the number of list items plus one. The extra window object must be taken into account when an accurate count of the true list items is needed.

References

This tutorial introduces the basics of the pyAA library. Please refer to the full pyAA API documentation for more information.

Additional information about Microsoft Active Accessibility can be found on the following sites:

Comments (0) Trackbacks (0) ( subscribe to comments on this post )

No comments yet.

No trackbacks yet.

twitter

Ambilight for video tag - http://bit.ly/byeu6k 58 mins ago
a little #dojo example using #jsfiddle http://bit.ly/cZFMPF 10 hours ago
sorry for the identi.ca spam 2010/03/11
@jdub I was waiting for House's "eureka" moment to identify her dirty keyboard as the source of her illness. 2010/03/09
via Reader Object Capability Model and Facets in Perstore/Pintura http://bit.ly/9DWfD9 2010/03/08

mindtrove Collecting ideas since 1980