Projects and Future Work:

We are interested in the creation of user interfaces that are truly human- (as opposed to computer-) oriented. We are especially interested in interfaces which make much better use of human world-models and the perceptual system, including "peripheral" perception - information acquired about environment outside of the area of primary attention or focus thru "background" or "ambient" sound, peripheral vision, and the "subtle" action of other senses. We are attempting to design user interfaces which use these processes to provide information about a computer or computer-controlled system.

In a recent experiment, we added a layer on top of the Unix c-shell command interface running under X windows. Upon completion, each user command made a unique sound - a "chord" based on the literal text of the command. Informal observation suggested that even this simplistic mechanism allowed users to monitor the status of several simultaneous commands with reduced attention.

We are presently considering systems that use frequent or continuous audio cues. Using audio loosely modeled on natural sounds, we hope to have system activity sound like a "cocktail party" - a general but informative babble, with the ability to focus on any of a large number of pieces of information from moment to moment. We hope to use "timbral", temporal/rhythmic, and spatial cues, coupled to display systems with several large screens and an analogous visual model.

For sonification of complex data, in addition to using characteristics of sound to represent additional degrees of freedom, we are interested in representing "derived" information like rates of change or "smoothness" of a function, differences between multiple data sets, or user defined conditions/constraints (e.g. values within ranges). We are also interested in the use of audio in multi- and interactive- media and in understanding the evolution of genetic algorithms.

Problems Encountered:

We have found that mu-law audio encoding does not work well for some classes of sound processing. The presentation of sound with complex timbres or many distinct parts requires reasonable resolution and bandwidth audio hardware (e.g. 16 bit, 44.1KHz) not available on many installed workstations. For improved interfaces we would also like to see much larger, high-resolution display systems and more autonomous physical interface devices.

Even at low bandwidths significant sound processing is expensive. We believe that the presentation of complex sound (and visuals) representing dynamic systems or on-going computations will greatly benefit from multi-processor systems, with a full special or general purpose processor dedicated to the audio (visual) interface.