Majestic-12
Owner of the robot : Majestic-12
Country : United Kingdom
Robot type : distributed search engine
Description : Distributed search engine with fast and efficient downloadable distributed crawler. People anywhere are invited to contribute to what could become the biggest search engine in the world.
Until November 2007, the user agent was :
MJ12bot/v1.2.0 (http://majestic12.co.uk/bot.php?+)
Several fake versions of the robot are active and they do not respect robots.txt. Most of them use old versions of the user agent.
User Agent transmitted to the visited web server :
- Mozilla/5.0 (compatible; MJ12bot/v1.2.1; http://www.majestic12.co.uk/bot.php?+)
Access control options understood by the robot :
- robots.txt
User Agent to use in the robots.txt file : MJ12bot
URL for more information :
December 6th, 2007 at 4:54 pm
Does not follow robots.txt when webmaster uses “User-Agent: *”
Does not recognize/ implement all robots.txt features.
December 6th, 2007 at 8:40 pm
Jon,
Thank you for your inputs. I just updated this page with the latest version of the user agent.
As far as I know, true Majestic bots honor robots.txt. Your web site was probably visited by one of the “fake” versions of MJ12bot.
December 12th, 2007 at 7:04 pm
Hi,
In the last few weeks we have received many reports of fake MJ12bot - some people unknown to us use this fake user-agent to crawl the web without obeying robots.txt that our legit bot does. This fake bot currently claims to be MJ12bot/v1.0.8 - if you see it on your site then it is 100% fake since we don’t use this version for a very long time. We have updated user-agent in November to be more inline with other search engines that use “Mozilla”-like convention.
If you happen to come across with MJ12bot that is not obeying robots.txt (and it’s not the fake one) then by all means contact us - we take all such reports very seriously and always investigate. Just so that you know roughly we get 1 such report per 1 bln urls crawled and recently we have not been getting these reports - only fake bot is reported and sadly those fakers did not even bother to obey robots.txt - their actions affect our reputation but I hope you will appreciate that anyone on the web can fake user-agent just like spammers fake From: email addresses often using real-ones, but it does not mean that person actually sent that spam email.
regards
Alex Chudnovsky
Majestic-12