Majestic-12


URL for more information :

3 Responses to “Majestic-12”

  1. Jon says:

    Does not follow robots.txt when webmaster uses “User-Agent: *”
    Does not recognize/ implement all robots.txt features.

  2. Jean-Luc says:

    Jon,

    Thank you for your inputs. I just updated this page with the latest version of the user agent.

    As far as I know, true Majestic bots honor robots.txt. Your web site was probably visited by one of the “fake” versions of MJ12bot.

  3. Alex Chudnovsky says:

    Hi,

    In the last few weeks we have received many reports of fake MJ12bot – some people unknown to us use this fake user-agent to crawl the web without obeying robots.txt that our legit bot does. This fake bot currently claims to be MJ12bot/v1.0.8 – if you see it on your site then it is 100% fake since we don’t use this version for a very long time. We have updated user-agent in November to be more inline with other search engines that use “Mozilla”-like convention.

    If you happen to come across with MJ12bot that is not obeying robots.txt (and it’s not the fake one) then by all means contact us – we take all such reports very seriously and always investigate. Just so that you know roughly we get 1 such report per 1 bln urls crawled and recently we have not been getting these reports – only fake bot is reported and sadly those fakers did not even bother to obey robots.txt – their actions affect our reputation but I hope you will appreciate that anyone on the web can fake user-agent just like spammers fake From: email addresses often using real-ones, but it does not mean that person actually sent that spam email.

    regards

    Alex Chudnovsky
    Majestic-12