Yahoo! Slurp
Owner of the robot : Yahoo! Inc.
Country : USA
Robot type : search engine
Description : Yahoo! Search data base includes more than 20 billions web pages, much more than any other search engine.
At the end of March, Yahoo announced that its Yahoo! Slurp crawlers host names would migrate from inktomisearch.com to crawl.yahoo.net. In an article published on June 5, Yahoo wrote that the transition is complete and all machines crawling as Slurp are now in crawl.yahoo.net. This is not what we see in our web server logs: inktomisearch.com is still there and there is even a stranger yahoo.com showing the Yahoo! Slurp user agent. This last one operates from IP addresses belonging to Yahoo! Search Marketing.
User Agent transmitted to the visited web server :
- Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)
IP address range :
- from 74.6.0.0 to 74.6.255.255 (yahoo.net)
(last visit in March 2008) - from 67.195.0.0 to 67.195.255.255 (yahoo.net)
(last visit in May 2008)
User Agent transmitted to the visited web server :
- Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)
IP address range :
- from 74.6.0.0 to 74.6.255.255 (yahoo.net inktomisearch.com yahoo.com)
(last visit in May 2008) - from 68.180.128.0 to 68.180.255.255 (yahoo.net)
(last visit in November 2007) - from 72.30.0.0 to 72.30.255.255 (yahoo.net inktomisearch.com)
(last visit in May 2008) - from 66.228.160.0 to 66.228.191.255 (yahoo.com)
(last visit in August 2007) - from 67.195.0.0 to 67.195.255.255 (yahoo.net)
(last visit in May 2008) - from 74.6.0.0 to 74.6.255.255 ()
(last visit in May 2008)
Access control options understood by the robot :
- robots.txt
- META NAME=”robots”
- rel=”nofollow”
User Agent to use in the robots.txt file : Slurp
URL for more information : http://help.yahoo.com/help/us/ysearch/slurp