An analysis of Zipf-Mandelbrot language measures and their application to artificial languages

Abstract

Studies of word frequency distributions have been used in linguisties for some time and are frequently used in information socnce research for such purposes as predicting numbers of key words or computing the significance of a word by its frequency of occurrence In this paper we provide a historical review of some of the developments in a particular aspect of word frequency analysis known as Zipf's Law but in fact first explicitly formulated by E U Condon, and later modified by B Mandelbrot. We present an exploratory analy sis of the use of Mandelbrot's parameters in discriminating among languages and language usage Some suggestions are made for using these parameters to characterize artificial (command) languages or the manner of use of these languages by different groups, for the purpose of enabling a computer interface to respond to users in a manner suited to their backgrounds or skills.

Get full access to this article

View all access options for this article.

References

LP Ayres , A Measuring Scale for Ability in Spelling (Russell Sage Foundation. 1915)

AD Booth , A law of occurrences of words of low frequency. Information and Control 10 (1967) 386-393.

Bertram C. Brookes. Ranking techniques and the empirical log law, Information Processing and Management 20 (1984) 37-46

James H. Carlisle , Man-Computer Problem Solimg Relationship Between User Characteristies and Interface Complexity, (Yale University, School of Organization and Management, Doctoral Dissertation, New Haven, 1974)

E U. Condon. Statisties of vocabulary. Science LXVIH. 1733 (1928) 300

Michael D. Cooper , Usage patterns of an online search system J Amer Soc for Info Sci 34 (1983) 343-349

G. Dewey , Relatu frequency of English speech sounds (Harvard University Press, Cambridge, 1923)

R.C. Eldridge , Sci Thousand Common English Words ( Niagara Falls, N.Y., Publisher not identified, 1911)

J.B. Estoup. Gammes Stenographiques 4th Ed (Publisher not identified, Paris, 1916)

10.

J.B. Estoup , Gammes Stenographiques 5th Ed (Publisher not identified, Paris, undated)

11.

Helen Fairbanks , The quantitative difterentation of samples of spoken language Psychological Monographs, 56. 1944, 19-38

12.

J.E. Federowicz , A Zipflan model of an automatic bibliographic system: an application to MEDLINE. JASIS 33 (4), 1982, 223-232.

13.

Benoit Mandelbrot , An informational theory of the statistical structure of language. In: Willis Jackson , ed., Communication Theory. (Butterworths Scientific Publications, London, 1953) 486-502

14.

Benoit Mandelbrot , Information theory and psycholinguistics: a theory of word frequencies. In Lazarsfeld, P.F. and Henry, N.W.. Eds. Readings in Mathematical Social Science. (MIT Press, Cambridge, 1968) 350-368

15.

Charles T. Meadow , Jiabin Wang and Weijing Yuan A comparison of user performance with different information retrieval mterfaces. Final project report (Faculty of Library and Information Science. University of Toronto, Toronto, 1992)

16.

W David Penniman , Rhythyms of Dialogue in Human-Computer Conversation (The Ohio State University, Columbus OH. PhD dissertation, 1975).

17.

M. Weinstock , C. Fenichel and M.V V Williams. Systems design implications of the title words of scientific journal articles in the Permuterm Subject Indxex 1. Conformity of Zipfs Law. In. A.D Barton, ed., The Social Impact of Information Retriecal, 7th Annual National Colloquium on Information Retrieval (Medical Documentation Service, The College of Physicians of Philadelphia , Philadelphia. 1970) 181-200

18.

John C.

Whitehorn and George Kingsley Zipf Schizophrenic language. Archues of Neurological

Psychiatry 49 ( 1943) 831-851.

19.

George Kingsley Zipf , Human Behat for and the Principle of Least Effort (Addison-Wesley Press. Inc. Cambridge. 1949)