Current User: Guest Login
Please consider registering


Lost Your Password?

Search Forums:


 






Minimum search word length is 4 characters – Maximum search word length is 84 characters
Wildcard Usage:
*  matches any number of characters    %  matches exactly one character

Random problem with stats

Reply to Post Add a New Topic
UserPost

8:54 am
August 26, 2010


geemist

Guest

Hello,

I have any problems with my results that awstats generate with my logs and the problem is random.

My log file are of the form : file-access.log.*.gz

I check the validity of files and there is no errors in my log file but the stats generate with awstats no take in consideration all files.

i have only 4 days that appear whereas i have more four files.

file-access.log.log.1.gz

file-access.log.30.gz

 

I would like if awstats need space disk during the generation of static pages ?

I would like to knwow if there is a awstats.log to know why i don't have all stats of all my log.gz ?

 

Thanks

9:26 am
August 26, 2010


Jean-Luc

Admin

posts 1125

Hello,

Of course, some disk space is required, but, in most systems, there is plenty of disk space available.

At the end of the AWStats update, you get something like this:

Update for config "………conf"
With data in log file "log…files_here"…
Phase 1 : First bypass old records, searching new record…
Searching new records from beginning of log file…
Phase 2 : Now process new records (Flush history on disk after 20000 hosts)…
Jumped lines in file: 0
Parsed lines in file: 12345
Found 0 dropped records,
Found 0 corrupted records,
Found 0 old records,
Found 12345 new qualified records.

9:52 am
August 26, 2010


geemist

Guest

I generate my static pages with awstats_buildstaticpages.pl but I checked space disk and there is no variance in space disk with the command for example "watch df -h"

 

But i just generate stats with awstats.pl -update, here my log:

Create/Update database for config "/etc/awstats/awstats.localhost.conf" by AWStat
s version 6.95 (build 1.943)
From data in log file "gzip -cd /logs/*.gz |"…
Phase 1 : First bypass old records, searching new record…
Searching new records from beginning of log file…
Phase 2 : Now process new records (Flush history on disk after 20000 hosts)…
Warning: awstats has detected that some hosts names were already resolved in your logfile gzip -cd /logs/*.gz |.
If DNS lookup was already made by the logger (web server), you should change your setup DNSLookup=1 into DNSLookup=0 to increase awstats speed.
Jumped lines in file: 0
Parsed lines in file: 3256488
 Found 0 dropped records,
 Found 2918751 corrupted records,
 Found 0 old records,
 Found 337737 new qualified records.

 

Why I have corrupted records ?

How to see the cause of problem ? There is a log file provide by awstats ?

10:08 am
August 26, 2010


Jean-Luc

Admin

posts 1125

Please double check the log format. Is it identical in all log files or did you change it at some point ?

Did you see the warning that says that the IP address is replaced by a host name in some lines of your log files ? Is this true ? IP address resolution should be done either by Apache or by AWStats but not twice.

10:42 am
August 26, 2010


geemist

Guest

I check the log format:

 

Here the logs that are matched by awstats and which I see theirs stats:

 

Aug 16 06:26:57 10.94.165.12 log_access: xxx.xxx.xxx.xxx – - [16/Aug/2010:06:27:07 +0200] "GET /js/detectPlayerVersion.js HTTP/1.1" 304 – "http://xxxxxxxx.fr/allPrograms.do?walledGardenEnabled=false" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Trident/4.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; eSobiSubscriber 2.0.4.16; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C)"

Aug 16 06:26:57 10.94.165.11 log_access: xxx.xxx.xxx.xxx – - [16/Aug/2010:06:27:07 +0200] "GET /js/detectDrmVersion.js HTTP/1.1" 200 229 "http://xxxxxx.fr/allPrograms.do?walledGardenEnabled=false" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Trident/4.0; GTB6; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; eSobiSubscriber 2.0.4.16; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C)"

 

Here, the logs that aren't matched by awstats:

 

Aug 10 06:30:03 10.94.165.12 log_access: xxx.xxx.xxx.xxx – - 
[10/Aug/2010:06:30:13 +0200] "GET /home.do HTTP/1.1" 200 6769 "-" 
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.4.1) Gecko/20031030"





Aug  9 06:26:57 10.94.165.11 log_access: xxx.xxx.xxx.xxx – - 
[09/Aug/2010:06:27:06 +0200] "GET 
/bandwidth_512KO.bin?rand=1281328027515 HTTP/1.1" 200 524288 
"http://wwwwwwww.fr/home.do" "aol.fr (compatible; MSIE 7.0; Windows NT 
5.1; Trident/4.0; NaviWoo2.2; GTB6.5; Toto 8.0; FBSMTWB; Toto 8.0; 
GTB6.4; .NET CLR 1.1.4322; InfoPath.2; .NET CLR 2.0.50727; .NET CLR 
3.0.4506.2152; .NET CLR 3.5.30729; OfficeLiveConnector.1.3; 
OfficeLivePatch.0.0; FBSMTWB; Toto 8.0; GTB6.4; .NET CLR 1.1.4322; 
InfoPath.2; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 
3.5.30729; OfficeLiveConnector.1.3; OfficeLivePatch.0.0)"

 

And i try to change the variable
DNSLookup= with 0 or 1 or 2 but I have the same problem but there with "0" I have no more the message:

"If DNS lookup was already made by the logger (web server), you should
change your setup DNSLookup=1 into DNSLookup=0 to increase awstats
speed."

 

I try to start of awstats.model.conf in changing only logFile and logFormat but I have the same problem, same records and same corrupted.

 

Here my variable in my conf file:

LogFile = "gzip -cd /exec/logs/ocs-portal/OCS2/*.gz |" 

LogFormat=1

10:54 am
August 26, 2010


geemist

Guest

I think that my log file are in the same format because it's not the first time that I have this problem with awstats.

11:11 am
August 26, 2010


Jean-Luc

Admin

posts 1125

Your LogFormat  is not correct.

I would try this:

LogFormat = "%other %other %other %host %other %other %other %other %time1 %methodurl %code %bytesd %refererquot %uaquot"

Let me know if this works better.

As usual, empty the DirData  directory before you run a new test update.

1:21 pm
August 26, 2010


geemist

Guest

It's a little best with this logformat beacause I collect four days over 20 whereas before i collect only two days.

 

The strangest thing is that the logs were not matched before and I put in this post aren still not matched with your logformat. So I think you're based on those to create the log format, is not it?

 

Thank you very much to answer me quickly.

2:58 pm
August 26, 2010


Jean-Luc

Admin

posts 1125

There are probably other errors in your setup, but I cannot find them from here. Sorry.

Suggestion: if AWStats does not work with some large log files, copy a small part of the log file in a new file and try to edit the config file to make it work for that small file. It is much easier to correct errors with smaller log files. When this will be okay, it is likely that the larger files will work too.

10:00 am
September 1, 2010


geemist

Guest

Hello,

I come back about my problem with my log format.

I use your LogFormat Jean-Luc: LogFormat = "%other %other %other %host %other %other %other %other %time1 %methodurl %code %bytesd %refererquot %uaquot"

 

And with this LogFormat, I have much corrupted records, so I use the option: -showcorrupted and here the out that I have:

 

Corrupted record (date 20100821095421 lower than 20100823062637-20000): Aug 21 09:54:11 12.XX.XXX.XX sco-log_access: 89.XX.XXX.XX – - [21/Aug/2010:09:54:21 +0200] "GET /images/titr_sco.gif HTTP/1.1" 200 1987 "http://site.fr/home.do;jsessionid=457A7420BBC3745CA1AF64DEE3E6BB20.sco-ddsco" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; GTB0.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; Media Center PC 5.0; SLCC1; Tablet PC 2.0; .NET4.0C)"

 

Corrupted record (date 20100821095421 lower than 20100823062637-20000): …

Corrupted record (date 20100821095421 lower than 20100823062637-20000): …

…

11:14 am
September 1, 2010


Jean-Luc

Admin

posts 1125

This means that the log format is correct, but some records are not in chronological order. To some extend, AWStats can cope with a lack of chronological order, but records that are more than 2 hours before the last processed timestamp are rejected.

Note that logresolvemerge  is able to merge log files with various dates and times, but it does not sort files. Each log file to be merged must be chronologically sorted.

1:48 pm
September 1, 2010


geemist

Guest

In fact, when I use logresolvemerge.pl, I can't generate my stats, It seems that your logformat not running Jean-Luc.

 

perl /usr/share/awstats/wwwroot/cgi-bin/awstats.pl -update -config=localhost
Create/Update database for config "/etc/awstats/awstats.localhost.conf" by AWStats version 6.95 (build 1.943)

From data in log file "/usr/share/awstats/tools/logresolvemerge.pl /var/logs/soc/soc-access.log.* |"…

Phase 1 : First bypass old records, searching new record…

Searching new records from beginning of log file…

AWStats did not find any valid log lines that match your LogFormat parameter, in the 50th first non commented lines read of your log.

Your log file /usr/share/awstats/tools/logresolvemerge.pl /var/logs/soc/soc-access.log.* | must have a bad format or LogFormat parameter setup does not match this format.

Your AWStats LogFormat parameter is:

%other %other %other %host %other %other %other %other %time1 %methodurl %code %bytesd %refererquot %uaquot

This means each line in your web server log file need to have the following personalized log format:

%other %other %other %host %other %other %other %other %time1 %methodurl %code %bytesd %refererquot %uaquot

And this is an example of records AWStats found in your log file (the record number 50 in your log):

Aug  2 06:28:52 12.XX.XXX.XXX soc-access: 90.XX.XXX.XXX – - [02/Aug/2010:06:29:00 +0200] "GET /images/attente.gif HTTP/1.1" 200 2937 "http://monsite.fr/home.do;jsessionid=36D417C12E9FEF0A95D92088C24785B6.socs-srv01c" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Trident/4.0; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; InfoPath.2; .NET CLR 3.5.30729; .NET CLR 3.0.30618)"

Setup ('/etc/awstats/awstats.localhost.conf' file, web server or permissions) may be wrong.

Check config file, permissions and AWStats documentation (in 'docs' directory).

2:32 pm
September 1, 2010


Jean-Luc

Admin

posts 1125

Hello geemist,

Well, even with this message, I believe that the log format is correct. Please check if the field separator is always just one space character. I suspect that there are 2 space characters between "Aug" and "2" in the beginning of the line (record 50 in the log file).

If this is the case, you should use :

LogSeparator="\s+"

in the AWStats config file.

4:49 pm
September 1, 2010


geemist

Guest

You're best !!

I'm so happy, thanks.

Reply to Post

Reply to Topic:
Random problem with stats

Guest Name (Required):

Guest Email (Required):

NOTE: New Posts are subject to administrator approval before being displayed

Smileys
Confused Cool Cry Embarassed Frown Kiss Laugh Smile Surprised Wink Yell
Post New Reply

Guest URL (required)

Math Required!
What is the sum of:
1 + 8
   


About the InternetOfficer.com Forum

Forum Timezone: UTC 1

Most Users Ever Online: 302

Currently Online:
10 Guests

Currently Browsing this Topic:
1 Guest

Forum Stats:

Groups: 2
Forums: 9
Topics: 638
Posts: 2709

Membership:

There are 257 Members
There have been 302 Guests

There is 1 Admin
There is 1 Moderator

Top Posters:

cssfsu – 55
deepakgupta – 34
albert_newton – 30
cosminpana – 20
DTNMike – 19
ahtshun83 – 17

Recent New Members: raju, todd2taylor, sbdcunha, mansigill1987, ThomasDuh, ThomasKic

Administrators: Jean-Luc (1125 Posts)

Moderators: Jean-Luc (1125 Posts)