Web Robots Encyclopedia

Do you want to know everything about Googlebot, Yahoo! Slurp or MSNBot ? Your site has been visited by a mysterious spider ? A visitor with an unusual User Agent searches all your site ? Will this robot bring you new visitors or is it just overloading your bandwidth ?

If you are in charge of Search Engine Optimisation or if you are a webmaster, you need to know where crawlers (spiders) visiting your site come from. The Web Robots Encyclopedia has the answers.

The Web Robots Encyclopedia is designed and maintained by InternetOfficer.

8 Responses to “Web Robots Encyclopedia”

  1. Flo says:

    very usefull stuff 😀

  2. Lai Wen-Chen says:

    what about wikia bot? alpha.search.wikia.com?

  3. George CNC says:

    Wikia uses Grub/2.0 – Grub.org crawler

  4. Jean-Luc says:

    See our new page about Wikia Grub.

  5. College Football Pick says:

    did wikia acquire grub from looksmart?

  6. Jean-Luc says:

    The acquisition was announced a year ago.

  7. khinelay says:

    Thank you so much . This resource is useful for me .

  8. Michiel Ebberink says:

    This is very useful. Maybe a bit outdated?


Leave a Reply

You must be logged in to post a comment.