Slide 0 The Amodal Communication System through an Extended Directional Input Georgios Yfantidis and Grigori Evreinov Dept. of Computer Sciences, University of Tampere georgios.yfantidis@nokia.com , grse@cs.uta.fi Slide 1 Multimodal interfaces use plethora of sensory combinations Oxymoron solution for people with sensory problems and/or limitations Traditional senses are obsolete as driving factors of the interaction and the only have peripheral role. Future solution is a development of amodal interfaces Slide 2 Title: Amodal (modal-nonspecific) signals and patterns. Table: The amodal parameters, which can be used at designing the amodal interface. Two columns The first is modalities: visual, auditive, tactile, kinaesthetic, proprioceptive, vestibular, olfactory, gustatory The second column is amodal parameters: signals and factors: row 1: intensity, direction (left, right, up, down, forward, backward), step/grade, slope and gradient row 2: temporal factors: period, rhythm, pauses (burst pauses), synchrony, tempo of action row 3: spatial factors: shape, texture, density, collocation, symmetry row 4: movement: relative speed/acceleration co-regulated / intercoordinated actions mechanical moments, turn-taking patterns row 5: color, sharpness; co-sensation: smell <=> -object -event -action Next the samples of amodal parameters and how they could be employed for interaction Slide 3 Title: Amodal signals and patterns (interface). Korg KAOSS Pad: Dynamic Effect Control Pad Link: http://www.korg.co.uk//kaoss.htm It allows generating more than 60 built-in music effects through touch or tapping the X-Y pad in any direction (there is the embossed directional lines for tactile navigation). see also SoundBeam gestural control http://www.soundbeam.co.uk/products/index.html Slide 4 Title: Amodal signals and patterns as external memory aid (shape-texture-direction-motion). The sample shows how to minimize tactile matrixes using collocation of the fingers. The matrixes (Pictures) which have been performed and explored within the tactile project. Tactile pointer layout for 15 / 25 keys of the virtual keyboard Slide 5 Directional gestures and touchscreen input: Good potential amodal interaction Blind users can use such a system as a text entry tool in PDAs/Mobile Phones Current research: Employ gestures that are easily associated with integrative notions Goal: Amodal customizable input tool for blind users Slide 6 Slide contains the layouts (picture) used for the gesture driven software button presented in Yfantidis, G., Evreinov, G, Adaptive Blind Interaction Technique for Touchscreens. J. Universal Access in Information Society, Springer, 2004, vol. 4(4) Slide also contains a screenshot of the above-mentioned interface in a handheld device. Finally there is a picture of a user having a blind interaction of a text entry using a touchscreen. Slide 7 Advanced systems could adapt to user performance and interpret the user input and intention. Need: Acquired or intuitive appropriation between amodal signals/patterns and their content, e.g., direction and integrative notion that creates a symbolic/generic language inherent to human brain. Representation of particular actions may be performed with objects, numbers. Rightward gesture could correspond to the notion of addition "+", while a leftward gesture could be subtraction "-". Slide 8 The same appropriation could be used to define Numerical difference and equality (left-side/leftward and right-side/rightward), geometrical relationships of parallelism such as orthogonality, similarity, congruence, tangency and so on. A basic approach where eight different directions = eight numbers or different alphabet characters. Slide 9 Title: Gesture as a system input Straight-line gestures classified in the four basic and intermediate directions Wide range of gestures tied to a vocabulary of corresponding inherent notions Able to discern between parallel and sequential gestures. There is a screenshot (picture) of the testing software showing vectors of the gestures as they have been performed on the touchpad. There are also (pictures) schemes explaining the different types of gestures that were asked for input. Slide 10 Title: What makes the directional gestiures differ? Need for recognition of three types of gestures. Main challenge: recognize if a gesture is single, or the first part of a double gesture. The goal of the experiment: find the parameters (predictor-factors) regarding uncompleted two-part directional patterns. Slide 11 Title: Preliminary test Input: two-part gestures done in the Wacom tablet, using stylus The two-part gestures comprised of two vectors (the first gesture, the second gesture and the distance between their starting points) The system receives an input of three types of gestures - two single gestures implemented sequentially with a little pause as naturally as possible - one double gesture composed of two parallel handwriting strokes and - one double gesture composed of two sequential handwriting strokes Slide 12 Title: preliminary test (results) In total 30000 (10subjects, 250 trials, 4 directions, 3types) handwriting gestures were collected. Graph (picture) shows the algorithm schematically by representing the field and the characteristic of each type of gestures (single, sequential and parallel). Slide 13 Title: preliminary test (results) Synchronization of the handwriting strokes can augment input space, increase reliablility of recognition Auditory visualization of the temporal Scale Strobe like signal as a "pace maker" for the blind interaction Sound signal marks the interval limit between two single gestures repeated every 400ms until the 2000 ms "deadline" Slide 14 Title: Extended test: Verification of the algorithm The gestures were categorized in directions and types The required gestures for each test session were 240 (24 gestures repeated 10 times each). They were marked/ described by composite sounds and required in a random order. In total 30000 (10subjects, 250 trials, 4 directions, 3types) handwriting gestures were collected. Slide 15 Title: Extended test: verification of the algorithm (results) The slide contains a graph (picture). The graph shows Single, Double parallel and double sequential by directions 1-8 in the x-axis. In the y-axis is showing R1 length in pixels. The values for different types of gestures are clearly over and below the discriminatory limit of 50 pixels, allowing a "comfortable" classification of gestures, which is crucial for the recognition system reliability. Slide 16 Title: Extended test: verification of the algorithm (results) The slide contains a graph (picture). The graph shows double gestures by direction in the x-axis. In the y-axis is showing the time in miliseconds. T2 stayed well below the maximum limit of 150 ms giving comfortable recognition possibilities for the secondary discriminatory factor for single and double gestures. Slide 17 Title: Extended test: verification of the algorithm (results) The slide contains a graph (picture). The graph shows double parallel gestures versus double sequential by direction in the x-axis. In the y-axis is showing the R1 and R3 length in seconds. The length for R1 and R3 are directly linked to the gesture type in all directions. Slide 18 Title: Extended test: verification of the algorithm (results) Errors were in very low levels for plain single gestures For double gestures the sequential ones proved to be the more error prone. Mean errors per direction: even for beginners, errors for the most "difficult" type of gestures (double sequential) were between 7 and 18%. Slide 19 Title: Conclusions Recognition algorithm is correct and functional The system can indeed differentiate between single and double, parallel and sequential gestures produced in blind mode based on values of length, time, and angles. The sound strobe signal was a valuable confirmatory feedback for the user, which helps her/him produce reliable single or two-part gestures of different properties The user-input integration by pace (synchronization of handwriting behaviour) decreased the variation of the parameters used for gesture recognition more than twice. Slide 20 Title: Future Research A communication system that can be based on a simple gestural input and be accessible to blind people. The gesture driven amodal system is versatile enough to encompass different functions varying from text input, to arithmetic expressions or even manipulation within an interface (such as navigation) We plan to integrate tactile feedback through vibration Different types of secondary feedback enhances the user experience, and assists the blind user in more customized and effective ways. Thank you for your attention