Improving Information Search Using Natural Language and Direct Manipulation Tools in a Multimodal Interface

Abstract

Clutter has an increasing impact on the quality of human-computer interaction, as more and more objects are represented on-screen. This study evaluated the usability of an existing, information-dense workspace, which places burdensome search requirements on its users. Direct manipulation is the standard method of search among multiple screen objects and has shown itself to be effective for point-and-act tasks. This study contrasts performance using natural language and multimodal interaction with this standard concept. The results reported here indicate that natural language is an effective search tool in a complex information workspace, because of its high level of specificity. However, regardless of its suggested merits, natural language is often criticized as being difficult to learn and utilize. The study reported here shows that users were easily able to combine modalities, natural language and direct manipulation, in a way predicted by the nature of screen clutter.

Get full access to this article

View all access options for this article.

References

Birkmire

Karsh

Barnette

B. D.

Pillalamarri

(1992). Target acquisition in cluttered environments. Proceedings of the Human Factors Society 36^th Annual Meeting.

Cohen

P. R.

(1991). The role of natural language in a multimodal interface. Keynote address to the '91 Friend21 international Symposium on Next Generation Human Interfaces, Tokyo, Japan. Menlo Park, CA: SRI International.

Lokuge

Ishizaki

(1995) GeoSpace: An interactive visualization system for exploring complex information spaces. Proceedings of ACM SIGCHI CHI '95 Conference on Human Factors in Computing Systems.

Miller

S. L.

Trafton

J. G.

(in preparation). The use of natural language and direct manipulation in a complex multimodal system.

Oviatt

(1996). Multimodal interfaces for dynamic interactive maps. Proceedings of ACM SIGCHI CHI '96 Conference on Human Factors in Computing Systems, 155–161.

Oviatt

DeAngeli

Kuhn

(1997). Integration and synchronization of input modes during multimodal human-computer interaction. Proceedings of ACM SIGCHI CHI '97 Conference on Human Factors in Computing Systems, 155–161.

Shneiderman

(1998). Designing the user interface: Strategies for effective human-computer interaction (3rd ed.). Reading, MA: Addison Wesley Publishing.

Trafton

J. G.

Wauchope

Stroupe

(1997). Errors and usability of natural language in a multimodal system. The proceedings of the 15^th International Joint Conference on Artificial Intelligence IJCAI-97.

Wauchope

(1996). Multimodal interaction with a map-based simulation system. (Technical Report AIC-96-019). Washington, D.C.: Naval Research Laboratory.

10.

Wickens

C. D.

Long

(1995). Object versus space-based models of attention: Implications for the design of head-up displays. Journal of Experimental Psychology, 1, 179–193.