Automate Awstats updates with combined logs


 

Current User: Guest Login Register
Please consider registering

Search 
Search Forums:


 




Automate Awstats updates with combined logs

Add a New Topic Reply to Post
UserPost

5:45 am
November 5, 2007


Jim

Guest

Hi

 We've recently started using Awstats to analyse logfiles on our clustered servers. All is working fine except when the merged log file from the 2 servers is created using the logresolve tool I have to paste the following at the top of the merged log file for Awstats to proces the log:

 #Software: Microsoft Internet Information Services 6.0
#Version: 1.0
#Fields: date time cs-method cs-uri-stem cs-uri-query cs-username c-ip cs-version cs(User-Agent) cs(Referer) sc-status sc-bytes

This means that I cannot set up a scheduled task to run the job automatically, which is a shame!

Has anyone any ideas which might help?

 Cheers in advance

 Jim

 

8:01 am
November 5, 2007


Jean-Luc

Admin

posts 220

Hi Jim,

You probably use:

LogFormat=2

I would rather use this LogFormat :

LogFormat = "%time2 %method %url %query
%logname %host %other %ua %referer %code
%bytesd"

With the short form of the LogFormat, AWStats has to search for a format specification within the input file; it could be that logresolve  removes this format specification.

Jean-Luc

9:27 am
November 5, 2007


Jim

Guest

Jean-Luc

Thank you very much for your quick reply!

This has done the trick!

I no longer need to paste that information into the combiuned log which means I can now automate the process and save myself lots of time.

Thank you very much for your help, I'm very grateful.

Cheers

Jim

3:43 am
November 6, 2007


Jim

Guest

Ooooopppsss!

 Spoke too soon!

 I've changed logfile format from LogFormat=2 to:

LogFormat = "%time2 %c-ip %cs-username %cs-method %cs-uri-stem %cs-uri-query %sc-status %sc-bytes %cs-version %cs(User-Agent) %cs(Referer)

Which is the format the logfiles are saved in. When I run Awstats now, it parses without error but doesn't produce any results when I creat a report. So I no longer need to paste that text into the top of the combined logfile like I did, which is great, but now I get no figures…confused?….oh yes!

 Jim

 

 

7:11 am
November 6, 2007


Jean-Luc

Admin

posts 220

You write that you use: 

LogFormat = "%time2 %c-ip %cs-username %cs-method %cs-uri-stem %cs-uri-query %sc-status %sc-bytes %cs-version %cs(User-Agent) %cs(Referer)"

This LogFormat does not match the fields specification in your first post:

#Fields: date time cs-method cs-uri-stem cs-uri-query cs-username c-ip cs-version cs(User-Agent) cs(Referer) sc-status sc-bytes

For example, according to your first post, the field after the date and time is cs-method  which AWStats calls %method.

Did you try the LogFormat I suggested ?

At the end of the database update, AWStats prints a report like this one:

Create/Update database for config "/path_to_awstats/awstats.my_domainconf" by AWStats version 6.7 (build 1.892)
From data in log file "/path_to_log_file/log_file_name"…
Phase 1 : First bypass old records, searching new record…
Searching new records from beginning of log file…
Phase 2 : Now process new records (Flush history on disk after 20000 hosts)…
Jumped lines in file: 0
Parsed lines in file: 3693
 Found 520 dropped records,
 Found 12 corrupted records,
 Found 0 old records,
 Found 3161 new qualified records.

This information helps understand what happened when there is a problem.

Jean-Luc

8:47 am
November 9, 2007


Jim

Guest

Jean-Luc

Thank you! I used the logformat from the boxes ticked on the IIS settings rather than the format I originally posted. I've now used your suggestion and the files are parsed without a problem.

Cheers for your help!

One last question!

Now I want to automate the process using a scheduled task. I can set up a btach file with my Awstat commands and get the scheduled task to run but it seems logresolve doesn't like the format D:/LogFiles/W3SVC1725568973/ex%YY-0%MM-0%DD-24 to merge yesterday's file. It works if I specify the exacy logfile name e.g. ex071108.log.

Any ideas?

Jim

9:01 am
November 9, 2007


Jean-Luc

Admin

posts 220

In batch files, the percent sign is a special character.

Try to replace each percent sign by two percent signs, like this:

D:/LogFiles/W3SVC1725568973/ex%%YY-0%%MM-0%%DD-24

Jean-Luc

9:40 am
November 9, 2007


Jim

Guest

Hi Jean-Luc

Thank you again for your quick reply

this is an improvement but now the command is interpreted literally and I get the following error written in the combined log:

Error: Couldn't open log file "D:/LogFiles/W3SVC1725568973/ex%YY-0%MM-0%DD-24" : No such file or directory.

I'll keep messing!

 Cheers dude

Jim

11:36 am
November 9, 2007


Jean-Luc

Admin

posts 220

Two questions:

1. What error message did you get first with D:/LogFiles/W3SVC1725568973/ex%YY-0%MM-0%DD-24" ?

2. What is the complete line calling logresolvemerge.pl   in your batch file ?

Jean-Luc

5:57 am
November 12, 2007


Jim

Guest

Hi Jean-Luc

The first error message was:

Error: Couldn't open log file "D:/LogFiles/W3SVC1725568973/exMM-0/LogFiles/W3SVC1725568973/exMM-0DD-24.log" : No such file or directory.

This was when using the following CMD line in my batch file:

D:/website/applications/awstats-6.6/tools/logresolvemerge.pl D:/LogFiles/W3SVC1725568973/ex%YY-0%MM-0%DD-24.log Z:/LogFiles/W3SVC1725568973/ex%YY-0%MM-0%DD-24.log > /LogFiles/website-combined-logs/combinedlogs.log

Cheers

Jim

 

11:05 am
November 12, 2007


Jean-Luc

Admin

posts 220

Did you replace all six original percent signs (% => %%) in the command line ? The .bat  file should contain:

D:/website/applications/awstats-6.6/tools/logresolvemerge.pl D:/LogFiles/W3SVC1725568973/ex%%YY-0%%MM-0%%DD-24.log Z:/LogFiles/W3SVC1725568973/ex%%YY-0%%MM-0%%DD-24.log > /LogFiles/website-combined-logs/combinedlogs.log

At execution time, this should give:

D:/website/applications/awstats-6.6/tools/logresolvemerge.pl D:/LogFiles/W3SVC1725568973/ex%YY-0%MM-0%DD-24.log Z:/LogFiles/W3SVC1725568973/ex%YY-0%MM-0%DD-24.log > /LogFiles/website-combined-logs/combinedlogs.log

And logresolvemerge.pl  should then replace %YY-0, %MM-0,… by the appropriate values.

Can you try again and let me know how the command looks like at execution time (ie. when you run the batch file). Thank you.

Jean-Luc

3:33 am
November 13, 2007


Jim

Guest

Hi Jean-Luc

This is the line I used in the batch file:

D:/website/applications/awstats-6.6/tools/logresolvemerge.pl D:/LogFiles/W3SVC1725568973/ex%%YY-0%%MM-0%%DD-24.log Z:/LogFiles/W3SVC1725568973/ex%%YY-0%%MM-0%%DD-24.log > /LogFiles/website-combined-logs/combinedlogs.log

When this is run in the batch file the combined log has the following error:

Error: Couldn't open log file "D:/LogFiles/W3SVC1725568973/ex%YY-0%MM-0%DD-24.log" : No such file or directory.

The two %% symbols format the line correctly but logresolve inerprets ex%YY-0%MM-0%DD-24.log literally rather than replacing the values with figures.

The execution lasts for less than a second as the logfile can't be found so nothing is parsed.

Jim

4:08 am
November 13, 2007


Jean-Luc

Admin

posts 220

Jim,

It's getting clear ! The point is that logresolvemerge.pl  is not processing file names like awstats.pl  does.

You cannot use these %YY-xx%MM-xx%DD-xx  with logresolvemerge.pl. I am afraid you will have to use tricks around the DATE function in the batch file to automate things as you want.

Also note that %YY-0%MM-0%DD-24  is not going to do what you expect. On January 1, 2008, %YY-0%MM-0%DD-24  will return 080131. You should use %YY-24%MM-24%DD-24  that would return 071231.

Jean-Luc

6:38 am
November 13, 2007


Jim

Guest

Cheers jean-Luc

Thanks very much for your help, you've been brilliant.

I'll have a good mess around with my batch file.

Take it easy dude

 Jim

Add a New Topic Reply to Post


Reply to Topic: Automate Awstats updates with combined logs

NOTE: New Posts are subject to administrator approval before being displayed

Guest Name (Required):

Guest EMail (Required):

Guest URL (required)

Math Required!
What is the sum of: 1 + 7        (Required)

Topic Reply:


 

WebStats1


About the InternetOfficer.com forum

Currently Online:

3 Guests

Maximum Online: 20

Forums:

Groups: 2

Forums: 7

Topics: 104

Posts: 493

Members:

There are 26 members

There are 94 guests


Jean-Luc has made 220 posts

Top Posters:

Steve - 14

angelko - 4

cwei - 4

Kinglui - 3

vtroymck - 3

Administrator: Jean-Luc | Moderators: Jean-Luc


© Simple:Press Forum - Version 3.1.2 (Build 340)