Sage Journals: Discover world-class research

Abstract

As various interactive input devices for computers have become available, the role of multimodal feedbacks generated by the devices has gained an increasing emphasis in recent years, with debates surrounding the relative efficiency of different feedback types of input devices. To address this and related issues, the present study conducted a 4 (types of feedback: visual vs. tactile vs. auditory vs. combined feedback) x 2 (gender: male vs. female) within-subject experiment to examine the effects of the type of feedbacks and gender on the efficiency and accuracy of a multimodal stylus pen. Results from the experiment showed that, regardless of the feedback type, males clicked the stylus faster than females while making more errors. A similar pattern was discovered when used the pen for dragging; males completed the dragging task faster than females while producing more errors. Interactions between the feedback type and gender as well as implications and limitations of the present study are discussed.

Keywords

Stylus feedback gender multimodal

1. Introduction

With rapidly advancing computing environments such as wireless networks, ubiquitous computing and virtual reality, technologies such as the multimodal and wireless mouse for manipulating or interacting with computing systems on display are serving us on a daily basis in both educational and industrial fields. The primary purpose of most computer systems, however, has been focused on serving humans rather than interacting with humans. From this perspective, input devices such as keyboard and mouse are perceived to be only delivering information from user to computer without any interactions via inputting devices [1, 2].

Most of today's computing systems typically use two ways of feedback modalities for delivering information: visual feedback and auditory feedback. To communicate with users, visual feedback is able to serve proper feedback such as changing the colour of the displayed object using the software. Due to the user's gaze, it is easy and effective to manipulate. Secondly, sound feedback helps make the user aware of an event during interaction. For example, if some specific errors occur in the computing system, the system will be able to make a distinguishing sound which is paired with an error. Because of the distinguishing sound, users become aware of specific errors distinguishable from other sounds. To use this type of feedback, users need to have additional hardware such as speakers [3, 4].

In addition to the visual and auditory feedback, haptic (tactile) feedback has recently become an increasingly important modality for improving users' performance when interacting with input devices. Haptic modality, however, has not been used properly in computing systems because of difficulties with manipulation and the need for additional hardware (e.g., motors).

Recently, with rapidly growing haptic technologies, we use haptic feedback when we use inputting devices such as mouse or stylus in computing systems. In particular, because haptic feedback uses the same point of inputting and outputting contact when interacting with computers, it can be an effective solution to improve the user's performance by using other channels compared to visual or sound feedback [5, 6].

A stylus is a pointing device that functions when playing on direct display environment and it is now being used as one of the most popular inputting devices for computing. With regard to haptic system technologies, few studies have focused on haptic effects in inputting devices such as a mouse [5, 6], remote pointing device [7], stylus [8, 9] etc. Therefore, the present study aims to analyse how and to what extent stylus users perform differently when exposed to single feedback vs. mixed feedbacks. This study is also helpful for serving effective inputting devices and interfaces for users. Additionally, this research examines potential gender effects when using stylus pens.

2. Literature Review

Some research has examined the usability of the stylus pen. In a study of the stylus pen's performance, Ren, Fukutoku and Ooya [10] studied the components of stylus pen. Based on the analysis of the stylus pen's components, they designed and made various types of stylus pens. Then, they studied the combinations of stylus pen components which were given the best ratings by users of various ages, ranging from 10 to 60. Their experiment was composed of two stages which were pointing tasks and drawing tasks. That is, they did not suggest any kinds of feedback when a user undertook his/her tasks.

Compared to visual or auditory feedback, tactile feedback in computing systems has improved rapidly over the last two decades. It is key to improving the user's performance when a user interacts or uses computing software such as a graphical user interface. Specially, because response time of tactile feedback is the shortest in sensory detections, it is able to help improve user's performance and fast action [11, 12]. In addition, force information that is a part of tactile feedback is able to enhance user's performance in general inputting devices' tasks [13].

With this approach, some research found that providing users with multiple sensory modalities may improve users' performance. For instance, Jacko et al. found that sound modality with visual effects can improve the user's performance in drag-and-drop tasks [3, 12].

Early prominent studies examined the effect of tactile feedback with mouse tasks. Gobel et al. [14] wanted to find the effect of additional modality when a person uses a mouse in usual tasks. They selected the three main tasks which were the highest functions among the user's tasks: tracking, selection and positioning. In their study, they found that the time of completion with tactile feedback was significantly faster than the time with no-tactile feedback.

Advanced steps of feedback have been developed with a combination of channels. Akamatsu et al. [15] researched some user tasks with various feedbacks. They used a haptic mouse for delivering tactile feedback to the user's fingers. Their experiment had the five conditions (no-feedback vs. auditory vs. tactile vs. visual vs. combination) to see the effects of feedback-type. They measured the task time in order to measure the efficiency of feedback types. Their results indicated that tactile feedback is not significant in decreasing the target approaching time. However, the over-target time is effectively reduced compared to other kinds of feedback. In their following study, Akamatsu and MacKenzie [5] compared force feedback, tactile feedback and a combination during a target approaching task. In their study the condition of combination showed the most effective results.

Recently, Lee et al. [16] developed a haptic stylus to deliver additional feedback for a person using the stylus on a touch screen. They developed user interfaces with a stylus as an inputting device. They used a solenoid which is able to generate a wide range of haptic feedback with a small pushing component at the end of a stylus. With this component the user experienced haptic feedback by pressing push-buttons. By using this, providing haptic or a combination of feedback was easier than in previous research. However, in this work, they did not analyse the effect of tactile feedback compared to other feedbacks or combinations of feedbacks.

Modality feedback affects differently based on gender. Schiff and Olaka [17] found that gender affects efficiency of tasks in sound feedback. Also, Fukawa and Yoshida [18], and Lewis and Neighbors [19] found that sound feedback for males is more helpful than for females. Contrary to sound feedback, haptic modality affects differently based on gender. In particular, females were more sensitive than males with regard to haptic sensory [20].

In accordance with these previous works and limitations, the goal of the present study is to find the effects of different feedback types and gender on basic tasks of inputting devices. Therefore, we examine the following research questions:

Research Question 1: For stylus users' task control, what is the effect of feedback type (IV) upon reaction time and accuracy (DVs)?

Research Question 2: For stylus users' task control, what is the effect of gender (IV) upon overall task time and accuracy (DVs)?

3. Method

3.1. Participants

The participants were 40 undergraduate students from a private university in Seoul who signed up for the experiment through an online registration page; 20 men and 20 women (average age: 24.5, standard deviation: 3.5, age ranged from 20 to 31 years) signed up for the experiment. All participants signed an informed consent form prior to their participation. All participants had normal hearing and tactile sensitivity. At the end of the experiment, participants were debriefed, paid three dollars and asked not to discuss the experiment with others.

3.2. Apparatus and Stimulus

The experiment was conducted in a quiet room with a chair and desk. A 24-inch multitouch LCD monitor (physical size: 53.2 cm×29.9cm; resolution: 1,024×768) was placed on the desk and then connected to a high-performance desktop computer via DVI connection.

Figure 1.

Design of multimodal stylus

Figure 2.

A screen shot of our experimental interface

To present tactile feedback, we used a vibrator (VBW32) which was mounted in the stylus pen (HP TouchSmart tx2 Stylus). The vibrator was connected to the computer by wireless Bluetooth network. The tactile stimuli were presented for 25ms when the cursor passed the border of a target or clicked a target on the monitor screen by pushing the button of the stylus (see Fig. 1). For the auditory feedback, a Sony 5.1-Channel speaker was connected to the computer. The auditory stimuli were composed of 32-bit and 44.1 kHz in mp3 format. The auditory stimuli were presented via the speaker. They were measured at approximately 65–80 dB from the participants' position.

We presented a black square on a white background as a visual target at random positions on the screen. Participants had two different tasks. The average distance of visual targets was about 15cm from the centre of the monitor screen. The size of objects was 1.2 cm × 1.2 cm. Participants had two kinds of tasks; one was to click a visual target (Click-Task). Another was to click and drag a visual target to the centre of the monitor screen (Drag-Task). The average distance of dragging was also 15cm (see Fig. 2).

3.3. Design and Procedure

The experiment was conducted with a within-subjects factor and a between-subjects factor, viz. the modality type of feedback (VF: Visual Feedback, TF: Tactile Feedback, AF: Auditory Feedback and CF: Combined Feedback [Auditory-Tactile-Visual]), and gender (male and female). The visual targets were presented 24 times in the Click-Task. They were also presented 12 times in the Drag-Task.

In every trial, participants were asked to click or drag the presented visual target by using the multimodal stylus as correctly and quickly as possible. Participants received tactile feedback when the point of the cursor passed over the border of the visual target in the Click-Task. In the Drag-Task, when a target object was clicked and dragged, visual, auditory or tactile feedback was presented. Participants responded by pressing a side-button on the multimodal stylus. Each trial ended when they clicked or dragged correctly. There was a five second gap between each trial.

We recorded the reaction time (time from presenting a visual target to clicking the visual target) and the number of errors (i.e., clicking outside of the visual target) as dependent variables in the Click-Task. In the Drag-Task we recorded the movement time (time from presenting a visual target to dragging it to the centre of the monitor screen) and the number of errors (i.e., dragging a visual target and putting it down outside of the visual target) as dependent variables.

Finally, at the end of the experiment, participants were asked to provide responses with regard to their perceived level of satisfaction and preferences toward the type of feedback modality by using a 7-point Likert scale (1 = “Extremely Unsatisfied,” 7 = “Extremely Satisfied”).

3.4. Results

In the clicking task, the mean and standard deviation (SD) according to four feedback types of the reaction time were Auditory Feedback: 840ms, (355), Tactile Feedback (TF): 850ms (306), Visual Feedback (VF): 898ms (284) and Combined Feedback (CF): 845ms (322), respectively; for gender, male: 762ms (254), female: 956ms (342). A detailed descriptive analysis of the Click-Task with regard to the reaction time is described in Table 1.

A repeated measures analysis of variance (RM-ANOVA) was carried out on the reaction time. The main effect of the gender was significant [F(1, 38)=23.25, p<.01]. However, the main effect of the feedback type was not significant. No interaction between gender and feedback type was found. These results revealed that the participants did not respond differently based on feedback type.

Table 1.

Mean reaction time and standard deviation (millisecond) in the first task (Click-Task) (AF : Auditory Feedback, TF : Tactile Feedback, VF : Visual Feedback, CF : Combined Feedback)

	AF	TF	VF	CF
Male	730 ms (266)	767 ms (316)	808 ms (225)	741 ms (208)
Female	950 ms (403)	934 ms (280)	989 ms (313)	949 ms (383)

Concerning the accuracy (error number) of the Click-Task, the mean and standard deviation (SD) according to four feedback types were: AF:.17 times (.13), TF:.18 (.09), VF:.16 (.06) and CF:.16 (.08), respectively; for gender, male:.18 (.08), and female:.15 (.10) (see Fig. 3).

An RM-ANOVA was conducted on the number of errors. The main effect of gender was significant [F(1, 38)= 7.635, p<.01]. However, the main effect of the feedback type was not significant. No interaction between gender and feedback type was found. Female participants responded more accurately than male participants.

In the Drag-Task, the mean and standard deviation (SD) according to four feedback types of the movement time were Auditory Feedback: 1,384ms, (189), Tactile Feedback (TF): 1,429ms (239), Visual Feedback (VF): 1,423ms (192) and Combined Feedback (CF): 1,410ms (287), respectively; for gender, male: 1,309ms (202), female: 1,513ms (208). A detailed descriptive analysis of the Drag-Task with regard to movement time is described in Table 2.

An RM-ANOVA was carried out on the movement time. The main effect of gender was significant [F(1, 38)=59.67, p<.01]. However, the main effect of the feedback type was not significant (p>.05). Interaction between feedback type and gender was significant [F(3, 114)=6.07, p<.01]. These results revealed that the participants responded differently based on feedback type. The female participants worked with more delay with the tactile feedback than other feedback types, while the male participants worked faster with the tactile feedback than other feedback types. Consequently, the types of multimodal feedback affected differently in participants' movement.

Figure 3.

Mean error number and standard error in the first task (Click-Task)

Table 2.

Mean reaction time and standard deviation (ms) in the second task (Drag-Task) (AF : Auditory Feedback, TF : Tactile Feedback, VF : Visual Feedback, CF : Combined Feedback)

	AF	TF	VF	CF
Male	1,361 ms (243)	1,241 ms (107)	1,375 ms (179)	1,260 ms (230)
Female	1,406 ms (115)	1,618 ms (175)	1,471 ms (197)	1,560 ms (262)

Concerning the accuracy (error number) of the Drag-Task, the mean and standard deviation (SD) according to four feedback types were: AF:.35 (.10), TF:.35 (.09), VF:.38 (.15), and CF:.38 (.10), respectively; for gender, male:.40 (.12) and female:.32(.09) (see Fig. 4).

An RM-ANOVA was conducted on the error number. The main effect of gender was significant [F(1, 38)= 19.271, p<.01]. However, the main effect of the feedback type was not significant (p>.05). Interaction between feedback type and gender was significant [F(3, 114)=11.026, p<.01]. These results revealed that the female participants worked more accurately with auditory feedback than other feedback types, while the male participants worked more accurately with tactile feedback than other feedback types.

Concerning the preference of feedback type, the mean and standard deviation (SD) according to the three feedback types were: AF: 4.70 (1.07), TF: 4.95 (1.58) and VF: 4.45 (1.18), respectively; for gender, male: 5.10 (1.12) and female: 4.30 (1.36).

Figure 4.

Mean error number and standard error in the second task

Finally, three multivariate ANOVAs were conducted on the preference of feedback types. The difference of gender was significant for tactile feedback [F(1, 38)=7.928, p<.01] and visual feedback [F(1, 38)=6.721, p<.05]. However, it was not significant for auditory feedback (p>.05). These results revealed that the female participants felt less comfortable than the male participants with tactile and visual feedback.

4. Discussion and Conclusion

The present study is aimed at identifying the potential effects of various feedback types and gender when using a stylus pen. Our results showed that the effects of tactile feedback were affected by user gender. Although tactile feedback was more useful, especially for male participants, than other feedback types, it affected participants who were female negatively when using the stylus pen as an inputting device. However, during a simple task, there was no difference based on feedback types. In particular, the results advanced previous research by improving the stylus with multimodal feedback.

Given these results and previous research it was found that tactile feedback has an advantage in interaction between inputting devices and users. This is helpful, not only for indirect inputting devices such as a mouse and keyboard, but also for stylus and 3D remote pointing devices. However, to focus on direct physical interaction, future research should examine fingertip interaction or haptic glove interaction with a touch-sensitive screen.

Additionally, we found that some participants in our experiment felt uncomfortable when working with tactile feedback. Particularly in the Drag-Task, this caused negative effects on accuracy. Therefore, we have to focus on the degree of tactile feedback. In addition, we recommend that the future research should include the effects of multimodal feedback on user performance or the psychological perception of user tasks. Also, the location of the haptic sensor in a stylus should be considered in the future research.

In this paper we focused only on the comparison between various types of feedback with the use of a stylus. Furthermore, our experiment was only focused on the stylus pen as an inputting device. We did not consider other kinds of inputting devices for comparison with the stylus pen. Therefore, future research should further examine the effects of multimodal feedback when a person uses other kinds of inputting devices such as haptic glove or remote pointing device (e.g., Nintendo Wii remote). Also, the combinations of feedback types to help user performance (e.g., auditory feedback + tactile feedback) should be investigated

Footnotes

5. Acknowledgments

This study was supported by a grant from the World-Class University program (R31-2008-000-10062-0) of the Korean Ministry of Education, Science and Technology via the National Research Foundation.

References

Choi

Kwon

D. S.

Kim

M. S.

Kim

M.S.

“Design of novel haptic mouse and its applications,” in Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems '03, 2003, pp. 2260–2265.

Yang

G. H.

Kyung

K. U.

Jeong

Y. J.

Kwon

D. S.

, “Novel haptic mouse system for holistic haptic display and potential of vibrotactile stimulation,”. In Proceedings of 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2005, pp. 1980–1985.

Jacko

J. A.

Dixon

M. A.

Rosa

R. H.

Jr. Scott

I. U.

Pappas

C. J.

, “Visual profiles: A critical component of universal access,”. in Proceedings of the SIGCHI conference on Human factors in Computing Systems '99, 1999, pp. 330–337.

Oakley

, The effects of multimodal feedback and age on a mouse pointing task. Florida: University of Central Florida Orlando, 2009.

Akamatsu

MacKenzie

I. S.

, “Movement characteristics using a mouse with tactile and force feedback,” International Journal of Human Computer Studies, vol. 45, pp. 483–494, 1996.

Kumazawa

, “Haptic mouse with quick and flexible tactile feedback generated by double control loop,” in Proceedings of the 18th IEEE International Symposium on Robot and Human Interactive Communication '10, 2010, pp. 664–669.

Cockburn

Brewster

, “Multimodal feedback for the acquisition of small targets,” Ergonomics, vol. 48, pp. 1129–1150, 2005.

Forlines

Balakrishnan

, “Evaluating tactile feedback and direct vs. indirect stylus input in pointing and crossing selection tasks,” in Proceedings of the 26th annual SIGCHI conference on Human Factors in Computing Systems, 2008, pp. 1563–1572.

Withana

Makino

Kondo

Sugimoto

Kakehi

Inami

, “ImpAct: Immersive haptic stylus to enable direct touch and manipulation for surface computing,” Computers in Entertainment (CIE), vol. 8, pp. 1–16, 2010.

10.

Ren

Fukutoku

Ooya

, “Usability of the Stylus Pen for Various Age Groups,” in Proceedings of the 6th Asia Pacific Conference on Computer Human Interaction, 2006, pp. 1–10.

11.

Hasser

C. J.

, Tactile Feedback for a Force-Reflecting Haptic Display. Armstrong Lab Wright-Patterson AFB OH Crew Systems Directorate, 1995.

12.

Jacko

Scott

Sainfort

Moloney

Kongnakorn

Zorich

Emery

, “Effects of multimodal feedback on the performance of older adults with normal and impaired vision” Universal Access Theoretical Perspectives, Practice, and Experience, 2003, pp. 3–22.

13.

Repperger

D. W.

Phillips

C. A.

Berlin

J. E.

Neidhard-Doll

A. T.

Haas

M. W.

, “Human-machine haptic interface design using stochastic resonance methods,” Systems, Man and Cybernetics, Part A: Systems and Humans, IEEE Transactions on, vol. 35, pp. 574–582, 2005.

14.

Gobel

Luczak

Springer

Hedicke

Rotting

, “Tactile feedback applied to computer mice,” International Journal of Human-Computer Interaction, vol. 7, pp. 1–24, 1995.

15.

Akamatsu

MacKenzie

I. S.

Hasbroucq

, “A comparison of tactile, auditory, and visual feedback in a pointing task using a mouse-type device,” Ergonomics, vol. 38, pp. 816–827, 1995.

16.

Lee

J. C.

Dietz

P. H.

Leigh

Yerazunis

W. S.

Hudson

S. E.

, “Haptic pen: A tactile feedback stylus for touch screens,” in Proceedings of the 17th annual ACM symposium on User interface software and technology, 2004, pp. 291–294.

17.

Schiff

Oldak

, “Accuracy of judging time to arrival: Effects of modality, trajectory, and gender,” Journal of Experimental Psychology: Human Perception and Performance, vol. 16, pp. 303–316, 1990.

18.

Fukawa

Yoshida

, “Sex difference in susceptibility to delayed auditory feedback in oral reading tasks,” Shinrigaku kenkyu: The Japanese Journal of Psychology, vol. 59, pp. 144–150, 1988.

19.

Lewis