Non-Visual Access to GUIs: Leveraging Abstract User Interfaces

Kris Van Hees and Jan Engelen
Katholieke Universiteit Leuven
Department of Electrical Engineering
ESAT - SCD - DocArch

Slide 1: Overview

1. Introduction
2. Related work
3. HCI design issues: non-visual presentation of GUIs
4. Leveraging Abstract User Interfaces
5. Example
6. Comparison with other approaches
7. The road ahead...

Slide 2: Introduction

Graphical User Interfaces:
    - Fact of life
In the past:
    - Mostly single-platform: MS Windows
    - Remote access for everything else
Recent years:
    - More platforms (Unix-Linux flavours, Mac OS X, ...)
    - Multiple graphical toolkits in client-server environment
      (Qt, GTK+, Athena, ...)

Slide 3: Related work

EU project: GUIB -- Screen reader
Mercator -- Screen reader
HAWK -- Non-visual interaction toolkit
Gnome Accessibility Architecture
Fruit -- Abstract widget toolkit
UsiXML -- Abstract user interface definitions

Slide 4: HCI design issues: non-visual presentation of GUIs

Coherence between visual and non-visual interfaces
    - Consistent mental interaction model<
    - Blind / sighted user interaction
Exploration in a non-visual interface
    - Access beyond the content windows
    - Spatial parameters can carry information
Conveying graphical information in a non-visual interface
    - Graphical object attributes (appearance, style, ...)

Slide 5: HCI design issues: non-visual presentation for GUIs

Interaction in a non-visual interface
    - Interaction objects and techniques
    - Alternatives to visual idioms (clicking, dragging, sliding, ...)
Ease of learning
    - ... because otherwise no one would use it!
Maintainability
    - Ensure accessibility throughout the UI life cycle

Slide 6: Leveraging Abstract User Interfaces

The slide shows an architecture diagram of the proposed solution.  At the top
are four classes of applications; OpenOffice, Java apps, Mozilla, and GTK+
apps.  This is the application layer.

In the middle, two components are shown: a screen reader, and a grouping that
represents the visual widget toolkits Swing, Qt, GTK+, and Athena.  These are
the rendering agents.  The link is drawn between the AUI component of each
application and both the screen reader and the visual agents.  This signifies
that the screen reader operates at the same level as the visual rendering.

Finally, the bottom of the diagram shows the device layer with text-to-speech,
Braille displays, and X-Windows.

Slide 7: Leveraging Abstract User Interfaces

Applications provide an AUI description
    - Hierarchy (tree)
    - Elements (nodes)
    - Properties
    - Data source (if any)
    - Relations
The UI description is in a standardised format (UsiXML> or a similar format)

Slide 8: Leveraging Abstract User Interfaces

The UI description can be transformed for a specific output modality
    - Preservation of semantics and presentation coherence
    - Augmenting UI description
The UI description is rendered by agents:
    - Graphical toolkit agents for visual presentation
    - Non-visual agents for auditory and/or tactile presentation

Slide 9: Comparison with other approaches

Development time UI construction
    - Usually automatically generated (e.g. UsiXML, ...)
    - Distinct UIs for specific modalities
Runtime UI selection
    - Model-driven UI customisation
    - UI adaptation

Slide 10: Comparison with other approaches

Development time UI construction
    - Model-driven UI creation
    - Essentially different versions of the same application

    - Collaboration suffers from UI differences
    - Collaboration suffers from modality imposed limitations
    - No simultaneous presentation in multiple modalities

Slide 11: Comparison with other approaches

Runtime UI selection
    - User and context models drive UI adaptation
    - Uses most appropriate interaction objects and techniques

    - Collaboration suffers from model imposed limitations
    - No simultaneous presentation in multiple modalities
    - Requires design time planning for supported adaptations

Slide 12: Comparison with other approaches

Parallel UI rendering
    - Single UI, multiple renderings
    - Coherent presentation of the UI in multiple modalities
    - Collaboration is implicitly supported
    - Alternative rendering engines do not require UI design changes

Slide 13: The road ahead...

Prototype implementation of visual and non-visual AUI rendering agents
(in progress)
    - Speech and Braille output
Integration with existing AUI frameworks (UsiXML, ...)
Definition of transformation rule sets to augment UI descriptions

At all stages, feedback from real users will be crucial!

Slide 14: Contact information

Kris Van Hees and Jan Engelen
Katholieke Universiteit Leuven
Department of Electrical Engineering
ESAT -- SCD -- DocArch
Kasteelpark Arenberg 10
B-3001 Heverlee
Belgium

kris@alchar.org, jan@docarch.be