Comments on: Pages Crawled by Googlebot

By: Jacob

Jacob — Tue, 21 Apr 2009 22:00:46 +0000

I am having trouble with the code above. The section does show up in my main awstats.MySite.html but no pagehits are shown. I searched through my log which contains over 100 hits from googlebot. What are my options to search for the error? Does AWstats provide any log files? Regards, Jacob Question moved to our forum about Extra Sections: trouble with pages crawled by Googlebot.

By: Frank

Frank — Tue, 14 Apr 2009 17:17:51 +0000

Hello,
The Yahoo! Slurp code only reports hits on robots.txt. It reports all hits when changed to:

ExtraSectionCondition1="UA,^Mozilla\/5\.0 $compatible\; Yahoo\! Slurp\; http\:\/\/help\.yahoo\.com\/help\/us\/ysearch\/slurp$$||UA,^Mozilla\/5\.0 $compatible\; Yahoo\! Slurp\/3\.0\; http\:\/\/help\.yahoo\.com\/help\/us\/ysearch\/slurp$$"

Regards,

Frank

By: John Somerton

John Somerton — Tue, 11 Nov 2008 20:33:35 +0000

Thanks, this is exactly what I was looking for! Too bad it’s not included by default.

BTW, how would you modify the Extra Section code to report Totals for both the Hits and the URLs columns? I’d like to know how many unique pages have been crawled without counting them…