An increasing number of robots harvest information on the world wide web for a wide variety of purposes. Protocols developed at the inception of the web laid out voluntary procedures in order to identify robot behaviour, and exclude it if necessary. Few robots now follow this protocol and it is now increasingly difficult to filter for this activity in reports of on-site activity. This paper seeks to demonstrate the issues involved in identifying robots and assessing their impact on usage in regard to a project which sought to establish the relative usage patterns of open access and non-open access articles in the Oxford University Press published journal Glycobiology, which offers in a single issue articles in both forms. A number of methods for identifying robots are compared and together these methods found that 40% of the raw logs of this journal could be attributed to robots.
D. Nicholas , P. Huntington and H.R. Jamali, The impact of open access publishing (and other access initiatives) on use and users of digital scholarly journals, Learned Publishing20(1) (2007) 11-15.
2.
A. Gutzman, Analysing Traffic on Your E-commerce Site (1999). Available at: www.ecommerceguide.com/solutions/technology/article.php/9561_186011 (accessed 21 October 2007).
3.
D. Nicholas , P. Huntington and H.R. Jamali, Open access in context: a user study, Journal of Documentation63(6) (2007) 853-78.
4.
P. Tan and V. Kumar, Discovery of web robot sessions based on their navigational patterns, Data Mining and Knowledge Discovery6(1) ( 2002) 9-35.
5.
A.G. Lourenço and O.O. Belo, Catching web crawlers in the act. In: D. Wolber et al. (eds), Proceedings of the 6th International Conference on Web Engineering, Palo Alto, California, USA (New York, ACM, 2006) 265-72.
6.
M. Yoon, Web Robot Detection (2000). Available at: http://photo.net/doc/robot-detection.html (accessed 20 June 2007).
7.
S. Jackson, Building a Better Spider Trap (1998). Available at: www.spiderhunter.com/spidertrap (accessed 21 October 2007).
8.
D. Eichman , Ethical web agents, Computer Networks and ISDN Systems28(1) (1995) 127-36.
9.
M. Koster, Guidelines for Robots Writers (1993). Available at: www.robotstxt.org/wc/guidelines.html (accessed 21 October 2007).
10.
The Robots Exclusion Standard (1996). Available at: www.robotstxt.org/wc/exclusion.html#robotstxt (accessed 21 October 2007).
11.
M.D. Dikaiakosa , A. Stassopouloub and L. Papageorgioua, An investigation of web crawler behavior: characterization and metrics, Computer Communications28(8) (2005) 880-97.
12.
V.S. Pai , L. Wang, K. Park, R. Pang and L. Peterson, The dark side of the web: an open proxy's view, ACM SIGCOMM Computer Communications Review34(1) (2004) 57-62.
13.
K. Park, V.S. Pai, K.W. Lee and S. Calo, Securing web service by automatic robot detection. In: Proceedings of USENIX Annual Technical Conference, 30 May to 3 June 2006, Boston, MA, USA (USENIX, Boston, MA, 2006) 225-60.
14.
L. von Ahn , M. Blum, N. Hopper and J. Langford, CAPTCHA: using hard AI problems for security. In: E. Biham (ed.), Advances in Cryptology: Proceedings of EUROCRYPT, International Conference on the Theory and Applications of Cryptographic Techniques, Warsaw, Poland, May 4-8, 2003 (Springer, Berlin, 2003) 294-311.
