In the last few weeks we have received many reports of fake MJ12bot – some people unknown to us use this fake user-agent to crawl the web without obeying robots.txt that our legit bot does. This fake bot currently claims to be MJ12bot/v1.0.8 – if you see it on your site then it is 100% fake since we don’t use this version for a very long time. We have updated user-agent in November to be more inline with other search engines that use “Mozilla”-like convention.
If you happen to come across with MJ12bot that is not obeying robots.txt (and it’s not the fake one) then by all means contact us – we take all such reports very seriously and always investigate. Just so that you know roughly we get 1 such report per 1 bln urls crawled and recently we have not been getting these reports – only fake bot is reported and sadly those fakers did not even bother to obey robots.txt – their actions affect our reputation but I hope you will appreciate that anyone on the web can fake user-agent just like spammers fake From: email addresses often using real-ones, but it does not mean that person actually sent that spam email.
December 6th, 2007 at 4:54 pm
Does not follow robots.txt when webmaster uses “User-Agent: *”
Does not recognize/ implement all robots.txt features.
December 6th, 2007 at 8:40 pm
Jon,
Thank you for your inputs. I just updated this page with the latest version of the user agent.
As far as I know, true Majestic bots honor robots.txt. Your web site was probably visited by one of the “fake” versions of MJ12bot.
December 12th, 2007 at 7:04 pm
Hi,
In the last few weeks we have received many reports of fake MJ12bot – some people unknown to us use this fake user-agent to crawl the web without obeying robots.txt that our legit bot does. This fake bot currently claims to be MJ12bot/v1.0.8 – if you see it on your site then it is 100% fake since we don’t use this version for a very long time. We have updated user-agent in November to be more inline with other search engines that use “Mozilla”-like convention.
If you happen to come across with MJ12bot that is not obeying robots.txt (and it’s not the fake one) then by all means contact us – we take all such reports very seriously and always investigate. Just so that you know roughly we get 1 such report per 1 bln urls crawled and recently we have not been getting these reports – only fake bot is reported and sadly those fakers did not even bother to obey robots.txt – their actions affect our reputation but I hope you will appreciate that anyone on the web can fake user-agent just like spammers fake From: email addresses often using real-ones, but it does not mean that person actually sent that spam email.
regards
Alex Chudnovsky
Majestic-12