Current User: Guest Login
Please consider registering


Lost Your Password?

Search Forums:


 






Minimum search word length is 4 characters – Maximum search word length is 84 characters
Wildcard Usage:
*  matches any number of characters    %  matches exactly one character

AWstats not being able to generate unique webstats for sub-domains (or subpages)

Reply to Post Add a New Topic
UserPost

1:51 pm
September 3, 2008


cssfsu

Member

posts 55

Hey guys,

This is a fantastic site and forum.. Cannot believe I didn't find it earlier.. So well here's a summary of my issue..its detailed but its only to cut down on going back and forth..

We host several of our client's catalogs on our portal site.. essentially there are a set of different pages for each of them on the main site.. so if the main site were to be xyz.lotus.com/wps/portal , the different catalogs would be at

xyz.lotus.com/wps/portal/123 , xyz.lotus.com/wps/portal/456 , and so on.. 

Since the last few yrs, we had been collecting stats only for the domain xyz.lotus.com which used to generate web stats collectively for all the sub-domains (or other pages) as well. .now we need to generate stats separately for them all..

What we had before :- a conf file for the main domain.. a log file..

What I tried doing:- Instance 1:- Created separate conf files but all analysing the same log file

Instance 2:- Created separate conf files, and also filtered out the log files into separate log files (did a grep and awk).. and then run awstats

Neither of the instances work for me..  In the first instance, all separate conf files give me exactly the same stats (ie hits, pages, etc which is impossible)..  Plz note for Domain names in their respective conf files, I provided the whole URL path.. xyz.lotus.com/wps/portal/123..

In the 2nd instance, i use grep and awk to make new log files for each sub-domain… they are still in the same Log format (=1) as before..  but running awstats does not generate anything.. it says found 0 records, etc…

Wow, long post.. So in a nutshell, i am not sure if what I want is possible or not since it seems that the logs are being analysed only on the top level.. and while doing on the sub- domain level, somehow it shows all subdomain stats to be the same..

It might be that I m unsure of what domain name to provide .. Any help will be appreciated.. Thanks in advance !!

1:32 am
September 4, 2008


Jean-Luc

Admin

posts 1125

Hi,

Thank you for your encouragement !

AWStats includes some options to handle subdomains, but what you use are not subdomains. Subdomains of domain.com  are 123.domain.com  or abcd.domain.com .

You can use the OnlyFiles directive to selectively process log file entries in a particular directory of your website :

OnlyFiles="REGEX[^\\/wps\\/portal\\/123]"

This will only count hits in the 123 subdirectory. This will probably be good enough, but it will still consider that the web domain is xyz.lotus.com and hits originating from another subdirectory will not be processed as external referrers.

11:17 pm
September 4, 2008


cssfsu

Member

posts 55

Hmm, so I tried the OnlyFiles option but it still doesn't show me any stats being generated. Should I change the domain name in the conf file to be xyz.lotus.com  rather than xyz.lotus.com/wps/portal/123  ?

Also, for OnlyFiles, willl it only pick up stats for lines containing the exact entry there.. for eg:- in my log file i would have entries for say xyz.lotus.com/wps/portal/123  and then to specific pages for that 123  catalog as well like, xyz.lotus.com/wps/portal/123/details?catalog.label=1TxxxzCC1P  or xyz.lotus.com/wps/portal/123/results?catalog.label=1Txxx0Ctt1P

I need AWstats to generate webstats for all log files which contain atleast xyz.lotus.com/wps/portal/123/  but not only that.. or else my details and results pages hits would not be counted..

Also, below is a record of my Log file..I was curious to know as to which entry in this record does awstats really compare the Domain name that I specified in the conf file with ?

72.152.193.132 – - [12/Aug/2008:07:4
0:44 -0400] “GET /wps/portal/123/res
ults?catalog.start=10 HTTP/1.1? 200
9229 “http://catalog.lotus.com/wps/p
ortal/123/results?catalog.searchTerm
s=Performance+Analyzer&catalog.catal
ogName=abcd+efgh& catalog.start=0&go
Button.x=4&goButton.y=3? “Mozilla/4.
0 (compatible; MSIE 6.0; Windows NT
5.1; SV1; .NET CLR 1.1.4322; .NET CL
R 2.0.50727)”

Thanks for your patience.. I m learning things… this is exciting ..

2:05 am
September 5, 2008


Jean-Luc

Admin

posts 1125

Regarding the domain name, just use xyz.lotus.com . What follows that does not belong to the domain name. AWStats does not need to find the domain name in your log file. It is not present in the example you gave.

OnlyFiles="/wps/portal/123"only accepts /wps/portal/123 .

OnlyFiles="REGEX[^\\/wps\\/portal\\/123]"

accepts everything starting with /wps/portal/123 . This should be what you are looking for.

4:59 pm
September 5, 2008


cssfsu

Member

posts 55

Hmm, ok one more thing: So we actually have 2 paths which we take- 1 for authenticated users and the other for non-auth users.

For auth users, the uri would be something like xyz.lotus.com/wps/myportal/123 whereas for non auth users, it would be xyz.lotus.com/wps/portal/123

We need to collect stats for both. Would it be possible to specify say, an OR clause in the regular expression in OnlyFiles.. something like

OnlyFiles="REGEX[^\\/wps\\/portal\\/123]" OR "REGEX[^\\/wps\\/myportal\\/123]"

One more thing I found was that I could use a filter on the urldetail report to generate stats for specific pages as well, something like,

awstats.pl −config=mysite−output=urldetail:/123−staticlinks > awstats.mysite.urldetailwithfilter.html

and it was pretty accurate.

Once I get your response on my query above for OnlyFiles, I ll try that too.

Thanks again Jean,

5:13 pm
September 5, 2008


Jean-Luc

Admin

posts 1125

That is correct. The URL filters can be easier to use.

There are various ways to do a OR. I would use this one :

OnlyFiles="REGEX[^\\/wps\\/(myportal|p
ortal)\\/123]"

2:07 pm
September 8, 2008


cssfsu

Member

posts 55

thanks !!

So we recently moved our hosting environment.. and changed our domain names and all too..ie from xyz.lotus.com to abcd.com .. we already had awstats on lotus.com and now i ve set it up on abcd.com as well.. traffic from lotus.com is redirected onto abcd.com

running awstats on both servers results in some anomaly in the webstats generated.. for eg: for month of sept so far, on lotus.com, I see unique visitors as being 50 for the first 6 days of sept.. for the same duration on abcd.com, i see unique visitors as being only 10.. ? if the hits from lotus.com are being redirected to abcd.com (which is happening), shouldn't that be counted as unique hits too on abcd.com (ie web stats over there should show that it had more than 50 unique visitors too )

If not, could u explain wht constitutes a unique visitor ?

2:40 pm
September 8, 2008


Jean-Luc

Admin

posts 1125

Assuming you redirect every hit on xyz.lotus.com to abcd.com, I agree that you should theoretically get at least as many hits on abcd.com as on xyz.lotus.com.

Now what can disturb that theory ?

1. some users might disable the automatic redirection in their browser, but this should be less than 1% of the visitors.

2. sometimes the redirect is not done for all pages. For example, if you only redirect the home page of xyz.lotus.com to the home page of abcd.com, then you could get hits on other pages of xyz.lotus.com that do not generate hits on abcd.com.

3. AWStats is detecting robots because of their user agents. Some robots mimic the user agents of regular browsers. AWStats is not able to identify them as robots and most robots do not follow redirects as human visitors do. I do not expect that 80% of the "visitors" are hidden robots though.

In  my opinion, the most likely cause is related to the redirection. How do you do the redirect ?

2:56 pm
September 8, 2008


cssfsu

Member

posts 55

I m looking into the Redirect rules we have into more detail now.. will let u know if i see any anomaly..

but from a high level, we are redirecting almost everything from lotus.com to 123.com on the same heirarchy.. ie for users who have bookmarked, say a specific details page on lotus.com/wps/portal/test/details?Navcode=xxxx , the redirect rule would take them to that particular details page on the new domain..  same goes for all other pages, etc.. so there are detailed redirect rules in place.. but that should still count as unique hits on the new domain… shouldnt it ?

having said that, how does awstats actually generate the unique visitors count from the log files ? is there anything in particular that it looks for .. something in the record..  ?

3:33 pm
September 8, 2008


Jean-Luc

Admin

posts 1125

A visit  and a unique visitor  are defined in the AWStats Glossary.

In simple terms, I would say that, to count visitors, AWStats looks at the IP address of the browsers that view pages from the site.

If the redirect work as you describe, almost all visitors of the old site should automatically become visitors of the new site.

3:58 pm
September 8, 2008


cssfsu

Member

posts 55

Hmm, ok i wonder what the issue might be then..  so one question: will awstats only consider records which are returned as 200 or with a 304 status code as a unique visitor.. if there r records with say status code, 404 Document Not Found, those records would not be considered towards the unique visitors count.. right ? if so, then i might know what the problem is.. one of our feeds directory is broken on the new domain.. and im seeing huge numbers for 404 errors as below..

404 Document Not Found 114340 99.1 % 29.34 MB

4:19 pm
September 8, 2008


Jean-Luc

Admin

posts 1125

Only 200 and 304 qualify for "visits" and "unique visitors".

A hit that returns a 404 code is not used in the count of visitors.

12:35 pm
September 12, 2008


cssfsu

Member

posts 55

So heres one logic..  Currently since we are still migrating to a new domain and all, i used to merge say 10 days  of access.logs and then run awstats on them.. WHEREAS earlier in the old domain, this used to be done at midnight everyday by a cron job..  So i am thinking is if i merge 10 days of access.logs and then run awstats on them, i am bound to get less unique visitors.

Earlier since we used to process logs daily, the unique visitors count was done on the unique IPs in the records at a daily basis and not say 10 days basis or so..

Does that make sense ? If I go back and run AWStats on every access.log.##%^ that I have since our deployment, would that generate more unique visitors (in terms of unique IPS)..

Thanks for the continued support

12:50 pm
September 12, 2008


Jean-Luc

Admin

posts 1125

Unique visitors are counted for the all month. It does not matter that you update the stats twice a day or once a month. You will end with the same number of unique visitors.

1:16 pm
September 12, 2008


cssfsu

Member

posts 55

hmm so if an IP address accesses the site on one day, it ll be counted as 1 unique visitor. Then if the same IP accessed the site on any of the remaining 29 days, it wont be counted as unique anymore ?

is there a way to count unique IPs per day (still for the monthly report though and not the one where it shows a day by day view )

3:27 am
September 13, 2008


Jean-Luc

Admin

posts 1125

The only way to count unique visitors per day is with the daily report. This information cannot be shown in the monthly report.

3:59 pm
September 16, 2008


cssfsu

Member

posts 55

thnx.. that bit helped.. now i m one more loose thread… so we generated webstats report for our new domain based on our apache logs.. now we couldnt get a count of unique visitors and visits because we are behind a proxy firewall which routes end user hits to our apache server.. so to the server all hits look as if they r coming from this proxy..

but we managed to grab and grep the proxy logs to get our data at that level.. with end user IPs and all.. it seems the proxy server piped out the log in some weird format of their own..

Questions:-

i) To run AWStats on this proxy log format, would I need to change anything in my conf file.. and more importantly,

ii) If I only change the LogFile parameter in my conf file to point to this new log and run AWStats, will I see a unique visitors count in my original webstats report …  ie in the same one HTML page/catalog 

8:33 pm
September 30, 2008


cssfsu

Member

posts 55

Okay, I finally configured everything properly and it worked !! So AWStats is rolling for us now.. Thanks for your super help Jean.. !

4:06 am
March 6, 2009


Dr Nick^

Guest

heya there.

Was looking for same issue, and the first reggex posted worked fine for me!

so thnx a lot for this info 🙂

Reply to Post

Reply to Topic:
AWstats not being able to generate unique webstats for sub-domains (or subpages)

Guest Name (Required):

Guest Email (Required):

NOTE: New Posts are subject to administrator approval before being displayed

Smileys
Confused Cool Cry Embarassed Frown Kiss Laugh Smile Surprised Wink Yell
Post New Reply

Guest URL (required)

Math Required!
What is the sum of:
6 + 1
   


About the InternetOfficer.com Forum

Forum Timezone: UTC 1

Most Users Ever Online: 302

Currently Online:
18 Guests

Currently Browsing this Topic:
3 Guests

Forum Stats:

Groups: 2
Forums: 9
Topics: 639
Posts: 2710

Membership:

There are 257 Members
There have been 304 Guests

There is 1 Admin
There is 1 Moderator

Top Posters:

cssfsu – 55
deepakgupta – 34
albert_newton – 30
cosminpana – 20
DTNMike – 19
ahtshun83 – 17

Recent New Members: raju, todd2taylor, sbdcunha, mansigill1987, ThomasDuh, ThomasKic

Administrators: Jean-Luc (1125 Posts)

Moderators: Jean-Luc (1125 Posts)