{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,3,26]],"date-time":"2026-03-26T20:21:11Z","timestamp":1774556471682,"version":"3.50.1"},"reference-count":40,"publisher":"Emerald","issue":"1","license":[{"start":{"date-parts":[[2010,3,9]],"date-time":"2010-03-09T00:00:00Z","timestamp":1268092800000},"content-version":"tdm","delay-in-days":0,"URL":"https:\/\/www.emerald.com\/insight\/site-policies"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":[],"published-print":{"date-parts":[[2010,3,9]]},"abstract":"<jats:sec><jats:title content-type=\"abstract-heading\">Purpose<\/jats:title><jats:p>The Robots.txt and Robots Meta tags constitute a set of instruments that can be used to instruct software robots. However, the current version of Robots.txt and Robots Meta tags are both too simple and ambiguous in an internet world with many potential conflicts, especially in terms of copyright and trespass to chattels. This paper seeks to propose an amendment to the Robots.txt and Robots Meta tags to solve the problems.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Design\/methodology\/approach<\/jats:title><jats:p>Instead of following personal experience, this paper surveys several predominant cases in an attempt to find general principles that can be used as guidelines to amend the Robots.txt and Robots Meta tags.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Findings<\/jats:title><jats:p>According to several court cases, the Robots.txt and Robots Meta tags can not only be used to simply allow or refuse the software robots, but also expressing the online copyright authorization policies of webmasters. Any robot following the given policies can prevent possible conflicts, and undoubtedly, any robot ignoring these may be in breach of the law. In terms of adapting to their new roles successfully, the Robots.txt and Robots Meta tags need some supplements and adoption; as a result, the webmasters can express their will more explicitly and avoid unnecessary disputes in relation to copyright authorization scope and trespass to chattels as well.<\/jats:p><\/jats:sec><jats:sec><jats:title content-type=\"abstract-heading\">Originality\/value<\/jats:title><jats:p>This paper reveals the new function of the Robots.txt and Robots Meta tags. Based on this new function, this paper points out the disadvantages of the current Robots.txt and Robots Meta tags and proposes new a comprehensive amendment based on this new function.<\/jats:p><\/jats:sec>","DOI":"10.1108\/07378831011026715","type":"journal-article","created":{"date-parts":[[2010,3,6]],"date-time":"2010-03-06T07:03:10Z","timestamp":1267858990000},"page":"94-106","source":"Crossref","is-referenced-by-count":5,"title":["Using the Robots.txt and Robots Meta tags to implement online copyright and a related amendment"],"prefix":"10.1108","volume":"28","author":[{"given":"Chyan","family":"Yang","sequence":"first","affiliation":[]},{"given":"Hsien\u2010Jyh","family":"Liao","sequence":"additional","affiliation":[]}],"member":"140","reference":[{"key":"key2022032120445455800_b1","unstructured":"Adam, J.N. (2002), \u201cTrespass in a digital environment\u201d, The Intellectual Property Quarterly, Vol. 6 No. 1, pp. 1\u201017."},{"key":"key2022032120445455800_b2","unstructured":"Bainbridge, D.I. (2008), Introduction to Information Technology Law, Pearson Education, Harlow."},{"key":"key2022032120445455800_b3","doi-asserted-by":"crossref","unstructured":"Blake, M. (2004), \u201cArchive of Web sites\u201d, The Electronic Library, Vol. 22 No. 5, pp. 462\u20106.","DOI":"10.1108\/el.2004.26322eab.004"},{"key":"key2022032120445455800_b4","unstructured":"Barone, L. (2007), \u201cRobots.txt summit\u201d, available at www.bruceclay.com\/blog\/archives\/2007\/04\/robotstxt_summi.html (accessed March 11, 2008)."},{"key":"key2022032120445455800_b5","doi-asserted-by":"crossref","unstructured":"Chau, M. and Chen, H. (2003), \u201cPersonalized and focused web spiders\u201d, in Zhong, N., Liu, J. and Yao, Y. (Eds), Web Intelligence, Springer\u2010Verlag, New York, NY, pp. 197\u2010217.","DOI":"10.1007\/978-3-662-05320-1_10"},{"key":"key2022032120445455800_b6","unstructured":"Cheong, F.C. (1996), Internet Agents: Spiders, Wanderers, Brokers, and Bots, New Riders Publishing, Indianapolis, IN."},{"key":"key2022032120445455800_b7","doi-asserted-by":"crossref","unstructured":"Citeseer.ist (1997), available at: http:\/\/citeseer.ist.psu.edu\/ (accessed March 11, 2008).","DOI":"10.1007\/s004820050088"},{"key":"key2022032120445455800_b8","unstructured":"Conner, S. (1996), \u201cAn extended standard for robot exclusion\u201d, available at: www.conman.org\/people\/spc\/robots2.html (accessed March 11, 2008)."},{"key":"key2022032120445455800_b9","unstructured":"Creative Commons (2008), available at: http:\/\/creativecommons.org\/license\/?lang=en (accessed March 11, 2008)."},{"key":"key2022032120445455800_b10","doi-asserted-by":"crossref","unstructured":"Drott, M.C. (2002), \u201cIndexing aids at corporate websites: the use of robots.txt and META tags\u201d, Information Processing and Management, Vol. 38 No. 2, pp. 209\u201019.","DOI":"10.1016\/S0306-4573(01)00039-5"},{"key":"key2022032120445455800_b11","unstructured":"Feigin, E.J. (2004), \u201cArchitecture of consent: internet protocols and their legal implications\u201d, Stanford Law Review, February, pp. 901\u201042."},{"key":"key2022032120445455800_b12","unstructured":"Google (2008a), \u201cHow Google crawls my site?\u201d, available at: www.google.com\/support\/webmasters\/bin\/topic.py?topic=8843 (accessed March 11, 2008)."},{"key":"key2022032120445455800_b13","unstructured":"Google (2008b), \u201cPreventing content from appearing in Google search results\u201d, available at: www.google.com\/support\/webmasters\/bin\/topic.py?topic=8459 (accessed March 11, 2008)."},{"key":"key2022032120445455800_b14","doi-asserted-by":"crossref","unstructured":"Gorman, G.E. (2006), \u201cGiving way to Google\u201d, Online Information Review, Vol. 30 No. 2, pp. 97\u20109.","DOI":"10.1108\/14684520610659148"},{"key":"key2022032120445455800_b15","unstructured":"Koster, M. (1993), \u201cGuidelines for robot writers\u201d, available at: www.robotstxt.org\/guidelines.html (accessed March 11, 2008)."},{"key":"key2022032120445455800_b16","unstructured":"Koster, M. (1994), \u201cA standard for robot exclusion\u201d, available at: www.robotstxt.org\/orig.html (accessed March 11, 2008)."},{"key":"key2022032120445455800_b18","unstructured":"Koster, M. (1996), \u201cEvaluation of the standard for robots exclusion\u201d, available at: www.robotstxt.org\/wc\/eval.html (accessed March 11, 2008)."},{"key":"key2022032120445455800_b19","unstructured":"Koster, M. (1997), \u201cHTML author's guide to the robots meta tags\u201d, available at: www.robotstxt.org\/wc\/meta\u2010user.html (accessed March 11, 2008)."},{"key":"key2022032120445455800_b20","unstructured":"MSN (2008a), \u201cControl which pages of your website are indexed\u201d, available at: http:\/\/search.msn.com\/docs\/siteowner.aspx?t=SEARCH_WEBMASTER_REF_RestrictAccessToSite.htm (accessed March 11, 2008)."},{"key":"key2022032120445455800_b21","unstructured":"MSN (2008b), \u201cLimit crawl frequency\u201d, available at: http:\/\/search.msn.com\/docs\/siteowner.aspx?t=SEARCH_WEBMASTER_REF_RestrictAccessToSite.htm (accessed March 11, 2008)."},{"key":"key2022032120445455800_b22","unstructured":"Netcraft (2008), \u201cNetcraft web server survey\u201d, available at: http:\/\/news.netcraft.com\/archives\/web_server_survey.html (accessed July 9, 2008)."},{"key":"key2022032120445455800_b23","doi-asserted-by":"crossref","unstructured":"Rao, S.S. (2003), \u201cCopyright: its implications for electronic information\u201d, Online Information Review, Vol. 27 No. 4, pp. 264\u201075.","DOI":"10.1108\/14684520310489050"},{"key":"key2022032120445455800_b24","doi-asserted-by":"crossref","unstructured":"Reed, C. (2004), Internet Law: Text and Materials, Cambridge University Press, Cambridge.","DOI":"10.1017\/CBO9780511808791"},{"key":"key2022032120445455800_b25","doi-asserted-by":"crossref","unstructured":"Samuelson, P. (2003), \u201cUnsolicited communications as trespass?\u201d, Communication of ACM, Vol. 46 No. 10, pp. 15\u201020.","DOI":"10.1145\/944217.944229"},{"key":"key2022032120445455800_b26","doi-asserted-by":"crossref","unstructured":"Seadle, M. (2006), \u201cCopyright in the networked world: using facts\u201d, Library Hi Tech, Vol. 24 No. 3, pp. 463\u20108.","DOI":"10.1108\/07378830610692217"},{"key":"key2022032120445455800_b27","unstructured":"Sieman, J.S. (2007), \u201cUsing the implied license to inject common sense into digital copyright\u201d, North Carolina Law Review, Vol. 85, pp. 885\u2010930."},{"key":"key2022032120445455800_b28","unstructured":"Smith, G.(.2.0.0.7.). (2008), \u201cCopiepresse v. Google \u2013 the Belgian judgment dissected\u201d, available at: www.birdbird.com\/english\/publications\/articles\/Copiepresse\u2010v\u2010Google.cfm?RenderForPrint=1 (accessed July 2, 2008)."},{"key":"key2022032120445455800_b29","unstructured":"Snyder, H. and Rosenbaum, H. (1998), \u201cHow public is the web? Robots, access, and scholarly communication\u201d, Proceedings of the 61st Annual Meeting of the American Society for Information Science, Vol. 35, pp. 453\u201062."},{"key":"key2022032120445455800_b30","doi-asserted-by":"crossref","unstructured":"Spinello, R.A. (2007), \u201cIntellectual property rights\u201d, Library Hi Tech, Vol. 25 No. 1, pp. 12\u201022.","DOI":"10.1108\/07378830710735821"},{"key":"key2022032120445455800_b31","unstructured":"Sterling, J.A.L. (2003), World Copyright Law, Sweet & Maxwell, London."},{"key":"key2022032120445455800_b32","doi-asserted-by":"crossref","unstructured":"Tan, P\u2010N. and Kumar, V. (2002), \u201cDiscovery of web robot sessions based on their navigational patterns\u201d, Data Mining and Knowledge Discovery, Vol. 6 No. 1, pp. 9\u201035.","DOI":"10.1023\/A:1013228602957"},{"key":"key2022032120445455800_b33","doi-asserted-by":"crossref","unstructured":"Thelwall, M. and Stuart, D. (2006), \u201cWeb crawling ethics revisited: cost, privacy, and denial of service\u201d, Journal of the American Society for Information Science and Technology, Vol. 57 No. 13, pp. 1771\u20109.","DOI":"10.1002\/asi.20388"},{"key":"key2022032120445455800_b35","doi-asserted-by":"crossref","unstructured":"Wong, M.W.S. (2007), \u201cCyber\u2010trespass and \u2018unauthorized access\u2019 as legal mechanisms of access control: lessons from the US experience\u201d, International Journal of Law & Information Technology, Vol. 15 No. 2, pp. 90\u2010128.","DOI":"10.1093\/ijlit\/eal014"},{"key":"key2022032120445455800_b36","unstructured":"Yahoo (2008a), \u201cYahoo! Slurp \u2013 Yahoo!'s web crawler\u201d, available at: http:\/\/help.yahoo.com\/l\/us\/yahoo\/search\/webcrawler\/ (accessed March 11, 2008)."},{"key":"key2022032120445455800_b37","unstructured":"Yahoo (2008b), \u201cHow do I prevent my site or certain subdirectories from being crawled?\u201d, available at: http:\/\/help.yahoo.com\/l\/us\/yahoo\/search\/webcrawler\/slurp\u201002.html (accessed March 11, 2008)."},{"key":"key2022032120445455800_b38","unstructured":"Yahoo (2008c), \u201cHow do I keep my page from being cached in Yahoo! Search?\u201d, available at: http:\/\/help.yahoo.com\/l\/us\/yahoo\/search\/webcrawler\/slurp\u201005.html (accessed March 11, 2008)."},{"key":"key2022032120445455800_b39","unstructured":"Yahoo (2008d), \u201cHow can I reduce the number of requests you make on my web site?\u201d, available at: http:\/\/help.yahoo.com\/l\/us\/yahoo\/search\/webCrawler\/slurp\u201003.html (accessed March 11, 2008)."},{"key":"key2022032120445455800_b40","unstructured":"Youtube (2008), available at: www.youtube.com (accessed March 11, 2008)."},{"key":"key2022032120445455800_frg17","unstructured":"Koster, M. (2008), \u201cRobot in the web: threat or treat?\u201d, available at: www.robotstxt.org\/wc\/threat\u2010or\u2010treat.html (accessed March 11, 2008)."},{"key":"key2022032120445455800_frg34","unstructured":"Wikipedia (2008), \u201cWeb crawler\u201d, available at http:\/\/en.wikipedia.org\/wiki\/Web_crawler (accessed March 11, 2008)."}],"container-title":["Library Hi Tech"],"original-title":[],"language":"en","link":[{"URL":"http:\/\/www.emeraldinsight.com\/doi\/full-xml\/10.1108\/07378831011026715","content-type":"unspecified","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/07378831011026715\/full\/xml","content-type":"application\/xml","content-version":"vor","intended-application":"text-mining"},{"URL":"https:\/\/www.emerald.com\/insight\/content\/doi\/10.1108\/07378831011026715\/full\/html","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2025,7,24]],"date-time":"2025-07-24T23:33:39Z","timestamp":1753400019000},"score":1,"resource":{"primary":{"URL":"http:\/\/www.emerald.com\/lht\/article\/28\/1\/94-106\/457612"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2010,3,9]]},"references-count":40,"journal-issue":{"issue":"1","published-print":{"date-parts":[[2010,3,9]]}},"alternative-id":["10.1108\/07378831011026715"],"URL":"https:\/\/doi.org\/10.1108\/07378831011026715","relation":{},"ISSN":["0737-8831"],"issn-type":[{"value":"0737-8831","type":"print"}],"subject":[],"published":{"date-parts":[[2010,3,9]]}}}