Micro-Mining and Segmented Log File Analysis: A Method for Enriching the Data Yield from Internet Log Files

Abstract

The authors propose improved ways of analysing web server log files. Traditionally web site statistics focus on giving a big (and shallow) picture analysis based on all transaction log entries. The pictures are, however, distorted because of the problems associated with resolving Internet protocol (IP) numbers to a single user and cross-border IP registration. The authors argue that analysing extracted sub-groups and categories presents a more accurate picture of the data and that the analysis of the online behaviour of selected individuals (rather than of very large groups) can add much to our understanding of how people use web sites and, indeed, any digital information source. The analysis is labelled `micro' to distinguish it from traditional macro, big picture transactional log analysis. The methods are illustrated with recourse to the logs of the Surgery Door (www.surgerydoor.co.uk) consumer health web site. It was found that use attributed to academic users gave a better approximation of the sites' geographical distribution of users than an analysis based on all users. This occurs as academic institutions, unlike other user types, register in their host country. Selecting log entries where each user is allocated a unique IP number can be particularly beneficial, especially to analyses of returnees. Finally the paper tracks the online behaviour of a small number of IP numbers, in an example of the application of microanalysis,

Keywords

websites data mining user profiles user behaviour consumer health information

Get full access to this article

View all access options for this article.

References

[1] D. Nicholas , P. Huntington and P. Williams , Evaluating metrics for comparing the use of web sites: case study two consumer health web sites , Journal of Information Science 28(1) (2002) 63-75 .

[2] D. Nicholas , P. Huntington , P. Williams , N. Lievesley and R. Withey , Developing and testing methods to determine the use of web sites: case study newspapers , Aslib Proceedings 51(5) (May 1999) 144-154 .

[3] D. Nicholas , P. Huntington and A. Watkinson , Digital journals, big deals and online searching behaviour: a pilot study , Aslib Proceedings 55(1/2) (2003) 84-109 .

[4] J. Fieber , Browser Caching and Web Log Analysis (25 May 1999). Available at: http://ella.slis.indiana.edu/*jfieber/papers/bcwla/bcwla.html (access date 9 September 1999).

[5] L. Tauscher and S. Greenberg , How people revisit web pages: empirical findings and implications for the design of history systems , International Journal of Human Computer Studies (special issue on Word Wide Web Usability) 47(1) (1997) 97-138 .

[6] A. Gutzman , Analysing Traffic on Your E-commerce Site. Available at: http://ecommerce.internet.com/solutions/tech_advisor/article/0,,9561_186011,00.html (access date 2 May 2001).

[7] R. Fuller and J.J. De Graff , Measuring User Activity from Server Logs. Available at: http://is.twi.tudelft.nl/*graaff/webuser96/ (access date 13 May 2003).

[8] D. Nicholas , P. Huntington , P. Williams and B. Gunter , Digital visibility: menu prominence and its impact on use of the NHS Direct information channel on Kingston Interactive Television , Aslib Proceedings 54(4) (2002) 213-221 .

[9] W.M. Zawitz , Web Statistics - Measuring User Activity (1998). Available at: www.ojp.usdoj.gov/bjs/pub/ascii/wsmua.tx (access date 19 May 2000).

10.

[10] J. Udell , Hit Counts Don't Mean a Thing. Here's how to Analyze your Web Server's Logs to Understand the Size, Composition and Behaviour of your Site's Audience (1996). Available at: wwwbyte.com/art/9602/sec12/art1.htm (access date 13 May 2003).

11.

[11] J. Simonson , G. Fuller and A. Tiwari , A Survey of Web History Data Analysis and Visualization (2001). Available at: www.math.grin.edu/*lindseyd/ResearchState.html (access date 13 May 2003).

12.

[12] J. Hong and J. Landay , WebQuilt: A Framework for Capturing and Visualizing the Web Experience (2001). Available at: www10.org/cdrom/papers/pdf/p602.pdf (access date 13 May 2003).

13.

[13] M. Svensson , A. Johansson , A. Ereback , K. Hook , J. Karlgren and I. Bretan , Moving Usability Testing Onto the Web. Available at: www.sics.se/*martins/publications/movusability.pdf (access date 13 May 2003).

14.

[14] A. Zeichick , In-Depth Web Site Reporting Network Management (23 October 2000). Available at: www.internetweek.com/reviews00/rev102300-2.htm (access date 13 May 2003).