Current User: Guest Login
Please consider registering


Lost Your Password?

Search Forums:


 






Minimum search word length is 4 characters – Maximum search word length is 84 characters
Wildcard Usage:
*  matches any number of characters    %  matches exactly one character

How can I count robots visits ?

Reply to Post Add a New Topic
UserPost

5:09 am
January 23, 2008


Joinup

Guest

How can i got the robots visits? in extrasection i got # code letters (P for pages,H for hits,B for bandwidth,L for last access). so can i get V=visits? not last visits.

i need to count robots visits, iam freaking out, i have tried a lot of things :/

Can i Get your help ?

Best regards 

5:21 am
January 23, 2008


Jean-Luc

Admin

posts 1125

The short answer is that there is no V option and that you cannot count visits in an extra section.

Specifically about robots, I don't know what you expect from counting robots visits. Web robots are not visiting web sites like humans do. Most of the bots just collect pages and save their contents locally for further processing. It does not really matter whether they do this in 1 or 100 visits as long as they do not overload your server.

Jean-Luc

5:28 am
January 23, 2008


Joinup

Guest

I need to count robots visits because marketing guys, they are used to have webtrends and webtrends was without filters, so they have a higher visits in Webtrends, i take off webtrends e put awstats in production, so now they have a lower visits, and i need to give them a visits number that awstats didnt count at the viewed traffic, is there a way to give them a value for visits that was not counted? can i desactivate robots filter just for 3 profiles?to make viewed traffic+not viewed traffic?

I need it badly ๐Ÿ™

8:58 am
January 29, 2008


joinup

Guest

Ok tks for the support…….

4:14 am
January 30, 2008


Jean-Luc

Admin

posts 1125

Ok. I will try to help you the way you expect.

One way to do what you want is to remove the robots.pm  file in the lib directory. You will get a warning message when updating and at the top of your AWStats reports, but the stats will be there and all hits will be counted as hits from human visitors, except the hits on robots.txt.

Let me come back on my previous answer. If WebTrends was configured to count robots as normal visitors, it was poorly configured. Most marketing people want to know the number of potential customers who visit their web site and web robots are not potential customers.

I believe that it is more useful to help people understand their figures than to give them figures that are almost meaningless.Wink

Jean-Luc

6:10 am
January 30, 2008


joinup

Guest

I removed robots.pm but i got the same visits…. robots will allways count like a hit and never a visit?

I am doing this just for 4 profiles, just for economic reason, i belive you should understant what i am talking about. ๐Ÿ™‚

 

 

7:15 am
January 30, 2008


Jean-Luc

Admin

posts 1125

Data already present in the database cannot be changed.

You need to remove the existing AWStats database files first and then run the updates without robots.pm. This will give you the figures without the robots.

Jean-Luc

12:40 pm
January 30, 2008


joinup

Guest

Hum i removed all .txt from the profile, mv robots.pm robots.pm.bck, and still cant get all together ๐Ÿ™

/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=robots -output=allrobots should give me the robots visits? i am saying this because with this output the hits looks like visits, because if i compare with top robots hits like default is higher than with the output.

I have made a grep -i in the Logfile and i got 581 call from nagios, in reports got 581 too, it should be the robots visits?

Best regards

 

 

 

 

2:47 pm
January 30, 2008


Jean-Luc

Admin

posts 1125

As you removed the robots.pm file, AWStats does not recognize any robot anymore. "-output=allrobots" will not return anything (except the hits on robots.txt).

Without robots.pm, the robots are seen as regular human users. Their hits are not counted as hits from robots. Their "visits" are counted as the visits of the human users are. I mean that you will get one number of visits which will include humans and robots.

In order to get a separate number for the number of visits by robots, I would run the update once with robots.pm and once without robots.pm. The number of visits by robots will be the difference between the numbers of visits in the two reports.

I do not understand your comment about nagios: you cannot count "visits" by counting occurences of a user agent. If you count these occurences, you should get the number of "hits" from this user agent.

I hope this clarifies the matter.

5:50 am
January 31, 2008


joinup

Guest

Ok got it, one more thing, in webtrends without robots filter in 2 days of the profile i got 21962 and in awstats the same log files without robots.pm i got 10821, how is this possible?, should Webtrends count all hits from users agents as visits? i am trying to get the same value has webtrends but i cant , even without robots.pm. The logfiles are the same.

 

Best regards ๐Ÿ™‚

8:54 am
January 31, 2008


Jean-Luc

Admin

posts 1125

It is hard to say based on the information you provide.

I don't think that WebTrends confuses "hits" and "visits". There can be differences in the definition of a visit by WebTrends and by AWStats though.

WebTrends: A visit is an interaction a unique visitor has with a website over a specified period of time or activity. In most cases, if a visitor has left a site or has not executed a click within 30 minutes, the visit session will terminate.

AWStats: Say a unique IP accesses a page, and then requests three others without an hour between any of the requests, all of the "pages" are included in the visit, therefore you should expect multiple pages per visit and multiple visits per unique visitor (assuming that some of the unique IPs are logged with more than an hour between requests).

These definitions are similar, but the idle-time is 30 minutes with WebTrends and 60 minutes with AWStats. On top of that, AWStats defines a page as:

AWStats: Only files that don't match an entry in the NotPageList config parameter are counted as "Pages". Usually pages are reserved for HTML files or CGI files, not images nor other files requested as a result of loading a "Page" (like js,css… files).

According to these definitions, it seems that WebTrends counts a new visit when someone hits any file, while AWStats does not count a visit when you only hit images (or other non-pages). If some images from the site are hotlinked from other web site, this can make a huge difference in the number of visits.

Jean-Luc

Reply to Post

Reply to Topic:
How can I count robots visits ?

Guest Name (Required):

Guest Email (Required):

NOTE: New Posts are subject to administrator approval before being displayed

Smileys
Confused Cool Cry Embarassed Frown Kiss Laugh Smile Surprised Wink Yell
Post New Reply

Guest URL (required)

Math Required!
What is the sum of:
12 + 10
   


About the InternetOfficer.com Forum

Forum Timezone: UTC 1

Most Users Ever Online: 302

Currently Online:
10 Guests

Currently Browsing this Topic:
1 Guest

Forum Stats:

Groups: 2
Forums: 9
Topics: 638
Posts: 2709

Membership:

There are 257 Members
There have been 302 Guests

There is 1 Admin
There is 1 Moderator

Top Posters:

cssfsu – 55
deepakgupta – 34
albert_newton – 30
cosminpana – 20
DTNMike – 19
ahtshun83 – 17

Recent New Members: raju, todd2taylor, sbdcunha, mansigill1987, ThomasDuh, ThomasKic

Administrators: Jean-Luc (1125 Posts)

Moderators: Jean-Luc (1125 Posts)