Sign Language Subtitling by Highly Comprehensible “Semantroids”

Abstract

We introduce a new method of sign language subtitling aimed at young deaf children who have not acquired reading skills yet, and can communicate only via signs. The method is based on: 1) the recently developed concept of “semantroid™” (an animated 3D avatar limited to head and hands); 2) the design, development, and psychophysical evaluation of a highly comprehensible model of the semantroid; and 3) the implementation of a new multi-window, scrolling captioning technique. Based on “semantic intensity” estimates, we have enhanced the comprehensibility of the semantroid by: i) the use of non-photorealistic rendering (NPR); and ii) the creation of a 3D face model with distinctive features. We have then validated the comprehensibility of the semantroid through a series of tests on human subjects which assessed accuracy and speed of recognition of facial stimuli and hand gestures as a function of mode of representation and facial geometry. Test results show that, in the context of sign language subtitling (i.e., in limited space), the most comprehensible semantroid model is a toon-rendered model with distinctive facial features. Because of its enhanced comprehensibility, this type of semantroid can be scaled to fit in a very small area, and thus it is possible to display multiple captioning windows simultaneously. The concurrent display of several progressive animated signed sentences allows for review of information, a feature not present in any sign language subtitling method presented so far. As an example of application, we have applied the multi-window, scrolling captioning technique to a children's video of a chemistry experiment.

Get full access to this article

View all access options for this article.

References

Adamo-Villani

Beni

(2004). Keyboard Controlled Signing Semantroid. IEEE Proceedings of ICARCV—8th International Conference on Control, Automation, Robotics and Vision, 741–746.

Adamo-Villani

Beni

(2005). Semantic intensity: A measure of visual relevance. Journal of Information, 9(3), 437–468.

Agrawala

Stolte

(2001). Rendering effective route maps: Improving usability through generalization. Proceedings of ACM Siggraph 2001, 241–249.

Benson

P. J.

Perret

D. I.

(1994). Visual processing of facial distinctiveness. Perception, 23, 75–93.

Blanz

Vetter

(1999). A morphable model for the synthesis of 3D faces. Proceedings of ACM Siggraph, 1999, 187–194.

Captioning Web. Retrieved online at: http://www.captions.org/.

DeCarlo

Santella

(2002). Stylization and abstraction of photographs. Proceedings of ACM Siggraph 2002, 769–776.

DePaul University. (2005). Sign language project. Retreived at http://asl.cs.depaul.edu/project.info.html.

Flodin

(1994). Signing illustrated. New York: Perigee Reference.

10.

Freire

Lee

(2000). The face-inversion effect as a deficit in the encoding of configural information: Direct evidence. Perception, 29, 159–170.

11.

Gallaudet Research Institute. (2005). Regional and National Summary Report of Data from the 2003–2004 Annual Survey of Deaf and Hard of Hearing Children and Youth. Washington, DC: GRI, Gallaudet University [WWW document]. URL: http://gri.gallaudet.edu/Demographics/2004_National_Summary.pdf

12.

Gooch

(2001). Non-Photorealistic Rendering. Wellesley, MA: AK Peters, Ltd.

13.

Gooch

Reinhard

Gooch

(2004). Human facial illustrations: Creation and psychophysical evaluation. ACM Transactions on Graphics, 23(1), 27–44.

14.

Gravett

(2004). Manga: 60 years of Japanese comics (paperback). New York: Collins Designs.

15.

Hansen

M. H.

(1959). Effects of discrimination training on stimulus generalization. Journal of Experimental Psychology, 58, 3221–3334.

16.

Hill

Scharff

L. V.

(1997). Readability of Websites with various foreground/background color combinations, font types and word styles. Proceedings of the Eleventh National Conference in Undergraduate Research, 2, 742–746.

17.

Holt

J. A.

Traxler

C. B.

Allen

T. E.

(1997). Interpreting the scores: A user's guide to the 9th edition Stanford Achievement Test for Educators of Deaf and Hard-of-Hearing Students. Orlando, FL: Harcourt Publishers.

18.

Laidlaw

D. H.

(2001). Loose, artistic “textures” for visualization. IEEE Computer Graphics and Applications, 21(2), 6–9.

19.

McLoud

(1993). Understanding comics. New York: HarperCollins.

20.

Ramachandran

V. S.

Hirstein

(1999). The science of art: A neurological theory of aesthetic experience. Journal of Consciousness Studies, 6(6–7), 15–51.

21.

Searcy

J. H.

Bartlett

J. C.

(1996). Inversion and processing of component and spatial-relational information in faces. Journal of Experimental Psychology: Human Perception and Performance, 22(4), 904–915.

22.

Sutton

(1974). Sign writing. Retrieved online at: http://www.signwriting.org/labout/what/what02.html.

23.

Tanaka

J. W.

Sengco

J. A.

(1997). Features and their configuration in face recognition. Memory and Cognition, 25(5), 583–592.

24.

Tversky

Baratz

(1985). Memory of faces: Are caricatures better than photographs? Memory and Cognition, 13, 45–49.

25.

Wilbur

R. B.

(2004). After 40 years of sign language research, what do we know? In Bradaric-Joncic

Ivasovic

(Eds.), Sign language, deaf culture, and bilingual education, 9–38. Zagreb: ERF.

26.

Wilson

K. L.

(2004). Rendering complexity in computer-generated pen-and-ink illustrations. Proceedings of ACM Siggraph 2004, 103–111.