InternetOfficer.com

I have configured my awstats.mysite.conf file to point to a log file which is in the ELFF log format. The first few lines in my log file are as follows:

#Fields: date time time-taken c-ip sc-status s-action sc-bytes cs-bytes cs-method cs-uri-scheme cs-host cs-uri-path cs-uri-query cs-username s-hierarchy s-supplier-name rs(Content-Type) cs(User-Agent) sc-filter-result sc-filter-category x-virus-id s-ip s-sitename cs(Referer)

In order to match this, I set the LogFormat in my awstats.mysite.conf file as follows:

LogFormat= "%other %other %time4 %host %code %other %bytesd %other %method %other %host %url %query %logname %other %other %other %uaquot %other %other %other %other %other %referer

after that I run the awstats commands:

perl awstats.pl -config=mysite -update

perl awstats.pl -config=mysite -output -staticlinks > awstats.mysite.html

I do not see any data in the html that is generated even though it has the required report structure. Is there something wrong with the way I have setup my LogFormat in the awstats.mysite.conf file?

Yes a .txt file ( the database file) was created in the \\wwwroot\\cgi-bin directory.. the date of the log file I used is 2008-12-12, and the .txt file that was generated is awstats122008.www.mysite.com.txt

Here are the first few sample lines of my log file:

#Software: SGOS 4.2.8.6
#Version: 1.0
#Start-Date: 2008-12-12 21:34:52
#Date: 2008-12-12 21:34:50
#Fields: date time time-taken c-ip sc-status s-action sc-bytes cs-bytes cs-method cs-uri-scheme cs-host cs-uri-path cs-uri-query cs-username s-hierarchy s-supplier-name rs(Content-Type) cs(User-Agent) sc-filter-result sc-filter-category x-virus-id s-ip s-sitename cs(Referer)
#Remark: 1307112055 “LSDMZBKDG02″ “139.133.177.8″ “main”
2008-12-12 21:51:12 133 63.170.108.233 200 TCP_HIT 1394 167 GET https mail.yahoo.com /Registration/Frameset.html – - DEFAULT_PARENT 20.156.4.130 text/html “Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)” PROXIED none – 129.133.156.2 SG-HTTPS-Reverse-Proxy-Service -
2008-12-12 21:51:13 122 23.140.148.153 200 TCP_HIT 1395 163 GET https cnn.com /Registration/Frameset.html – - DEFAULT_PARENT 23.124.8.144 text/html “Mozilla/4.0 (compatible; MSIE 4.01; Windows NT)” PROXIED none – 144.199.128.6 SG-HTTPS-Reverse-Proxy-Service -

Thanks for your help. That is exactly what I was missing (the -month and the -year option). Now it shows the data in the report. But my log files are in the format logfilename.log.gz. And this worked for a smaller log file ( even with the .gz extension at the end of that log file). If I try to use a larger log file which is also in the format 'logfilename.log.gz', it gives the error:

Phase 1 : First bypass old records, searching new record…
Searching new records from beginning of log file…
AWStats did not find any valid log lines that match your LogFormat parameter, in the 50th first non commented lines read of your log.
Your log file C:/AwstatLogFile/logs/SG_main__90110080000.log must have a bad format or LogFormat parameter setup does not match this format.

Your AWStats LogFormat parameter is:
date time time-taken c-ip sc-status s-action sc-bytes cs-bytes cs-method cs-uri-scheme cs-host %url cs-uri-query cs-username s-hierarchy s-s
upplier-name rs(Content-Type) cs(User-Agent) sc-filter-result sc-filter-category x-virus-id s-ip s-sitename cs(Referer)
This means each line in your web server log file need to have the following personalized log format:
date time time-taken c-ip sc-status s-action sc-bytes cs-bytes cs-method cs-uri-scheme cs-host %url cs-uri-query cs-username s-hierarchy s-s
upplier-name rs(Content-Type) cs(User-Agent) sc-filter-result sc-filter-category x-virus-id s-ip s-sitename cs(Referer)
And this is an example of records AWStats found in your log file (the record number 50 in your log):
Θ[º¢iΣ?ë??Ç?FnÉ∩?ΩZs?9??xh_     ôÇz¬??N9?·∞ª>ß???MΓWLj?kΦ.zì"Ω♣???%½¿xHót^?8+µO?K°?B??p≤¼¥←¿?   ?.Σ*9°üΓx?????????k?q(%¢C??ß~VP?π?Vܽ]iôï8?D
YhuvU¥`«?o?\\??Q?%Yƒ??bΦª↑??2♦Mb??]ÖY)eª?σ«≥÷?u  ?¡q]¬µ??÷??pêKòç??,^Vτè???¬GL?g?¬¥??U{«?°_<Σ¼Hx???Φ¿ Zæ≤âµdêJ?TF?8â*2¥?ô?Q∞4τE?n↓ê↑?(5r↔E«uï
»” ♦(?Z???I¿:?]?????÷Ñ?'√ì8Äß]  ????t♥ n½???
Setup ('./awstats.www.cehealthrecord.org.conf' file, web server or permissions) may be wrong.

Check config file, permissions and AWStats documentation (in 'docs' directory).

Is there a way to get past this, so that I can use the larger log files? 

I am following on that, I have winzip, and i installed winrar also,  but using both of them in my config file gave me the following error… ( I am running on my Windows XP 32 bit version , just in case you needed the information.)

C:\\Documents and Settings\\username\\Desktop\\Desktop\\AwstatScripts\\awstats-6.5\\wwwroot\\cgi-bin>perl awstats.pl -config=www.mysite
-update
Update for config "./awstats.www.mysite.conf"
With data in log file "winzip C:/AwstatLogFile/logs/SG_main__91208080000.log.gz |"…
Phase 1 : First bypass old records, searching new record…
Searching new records from beginning of log file…
'winzip' is not recognized as an internal or external command,
operable program or batch file.
Error: Command for pipe 'winzip C:/AwstatLogFile/logs/SG_main__91208080000.log.gz |' failed
Setup ('./awstats.www.mysite.conf' file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in 'docs' directory).

The line that I modified in my config file is:

LogFile="winzip C:/AwstatLogFile/logs/SG_main__91208080000.log.gz |"

I tried winrar above with the same result… Do you know any deccompression tool that will work with this? I also tried with gunzip, but that does not work either.

On a wider perspective, let's say I am able to get a utility to decompress the log file. I am going to have many logs which are each going to really huge, and I do not really want to expand all of them because that will fill up the space really fast. Hence the reason I was looking for awstat to directly read the .gz log file instead of expanding it by decompression, and then reading it. If I do end up doing that, I ponder whether awstat has some facility to do some post-processing where it can again compress the log file back to the .gz format?

I downloaded the 7-Zip utility. Using the utiliy from the command line seems to be working correctly and gives me the following output:

C:\\7-Zip>7z e C:/AwstatLogFile/logs/SG_main__91225080000.log.gz

7-Zip 4.64  Copyright (c) 1999-2009 Igor Pavlov  2009-01-03

Processing archive: C:\\AwstatLogFile\\logs\\SG_main__91225080000.log.gz

Extracting  SG_main__91225080000.log

Everything is Ok

Size:       988883255

Compressed: 92004677

But doing so in the config file gives me the following error:

C:\\Documents and Settings\\username\\Desktop\\Desktop\\AwstatScripts\\awstats-6.5\\wwwroot\\cgi-bin>perl awstats.pl -config=www.mysite
-update
Update for config "./awstats.www.mysite.conf"
With data in log file "C:/7-Zip/7z e C:/AwstatLogFile/logs/SG_main__91225080000.log.gz |"…
Phase 1 : First bypass old records, searching new record…
Searching new records from beginning of log file…
Jumped lines in file: 0
Parsed lines in file: 11
 Found 0 dropped records,
 Found 11 corrupted records,
 Found 0 old records,
 Found 0 new qualified records.

The line that I modified in the config file is:

LogFile="C:/7-Zip/7z e C:/AwstatLogFile/logs/SG_main__91225080000.log.gz |"

I am trying to read from multiple log files which are from different days. So I guess I need to use the logresolvemerge.pl option in the  'LogFile=' directive ( is that correct?) or else the data does not show up in the reports… This is the line I configured in my log file to do so.

LogFile="C:/awstats-6.9/awstats-6.9/tools/logresolvemerge.pl C:/7za464/7za e -so C:/AwstatLogFile/logs/SomeOtherLogs/*.gz |"

But doing so is giving me an error:

gzip: stdout: Invalid argument
Error: Command for pipe 'C:/awstats-6.9/awstats-6.9/tools/logresolvemerge.pl C:/7za464/7za e -so C:/AwstatLogFile/logs/SomeOtherLogs/*.gz |'
 failed

even changing the order of the command to the following:

LogFile="C:/7za464/7za e -so C:/awstats-6.9/awstats-6.9/tools/logresolvemerge.pl C:/AwstatLogFile/logs/SomeOtherLogs/*.gz |"

is giving me the error:

Error:
Cannot use absolute pathnames for this command
Error: Command for pipe 'C:/7za464/7za e -so C:/awstats-6.9/awstats-6.9/tools/logresolvemerge.pl C:/AwstatLogFile/logs/SomeOtherLogs/*.gz |'
 failed

I am sure my syntax or order of the arguments is wrong somewhere.

I added the following changes in the logresolvemerge.pl file under the External Program variables section:

# For 7za compression
my $szcat = '7za e -so';
my $szcat_file = '\\.gz$';

Then, in the conf file, I changed the required line as follows:

LogFile="C:/awstats-6.9/awstats-6.9/tools/logresolvemerge.pl C:/AwstatLogFile/logs/SomeOtherLogs/*.gz |"

looks like these couple of changes worked, and I could see the database and the html reports with data in them.

Here's one complication: The log files I have will actually be in the format .gz.done. 

Now, when I use the follwoing command in the conf file:

LogFile="C:/7za464/7za e -so C:/AwstatLogFile/logs/SomeOtherLogs/*.done |"

( in orther words, without using the logresolvemerge.pl, on different logs which I think would probably be in sorted order), this command works, and produces data in the report. However if my logs were to be unsorted, then I would need to use the logresolvemerge.pl. In that case, using the following command:

LogFile="C:/awstats-6.9/awstats-6.9/tools/logresolvemerge.pl C:/AwstatLogFile/logs/SomeOtherLogs/*.done |"

along with the following corresponding change in the logresolvemerge.pl file:

# For 7za compression
my $szcat = '7za e -so';
my $szcat_file = '\\.done$';

produces the following output:

Searching new records from beginning of log file…
Jumped lines in file: 0
Parsed lines in file: 0
 Found 0 dropped records,
 Found 0 corrupted records,
 Found 0 old records,
 Found 0 new qualified records.

so looks like it does not produce any data in this case when I tell it to process the log with .done extension at the end. I thought it should be able to process this with logresolvemerge.pl, since it is able to do so without it.

Yes… I also have those names you mentioned $zcat, $zcat_file, $bzcat and $bzcat_file in my logresolvemerge.pl…. I just additionally added that section for $szcat and $szcat_file under those entries.

Haha no I did not… I do not know if that is good news or bad news that .gz processed the files ( most probably bad news, I guess). I think I will just replace one of the existing External Program variables with my variable then, rather than adding it at all the different places within the logresolvemerge.pl file.