Current User: Guest Login
Please consider registering


Lost Your Password?

Search Forums:


 






Minimum search word length is 4 characters – Maximum search word length is 84 characters
Wildcard Usage:
*  matches any number of characters    %  matches exactly one character

Automate Awstats updates with combined logs

Reply to Post Add a New Topic
UserPost

5:45 am
November 5, 2007


Jim

Guest

Hi

 We've recently started using Awstats to analyse logfiles on our clustered servers. All is working fine except when the merged log file from the 2 servers is created using the logresolve tool I have to paste the following at the top of the merged log file for Awstats to proces the log:

 #Software: Microsoft Internet Information Services 6.0
#Version: 1.0
#Fields: date time cs-method cs-uri-stem cs-uri-query cs-username c-ip cs-version cs(User-Agent) cs(Referer) sc-status sc-bytes

This means that I cannot set up a scheduled task to run the job automatically, which is a shame!

Has anyone any ideas which might help?

 Cheers in advance

 Jim

 

8:01 am
November 5, 2007


Jean-Luc

Admin

posts 1125

Hi Jim,

You probably use:

LogFormat=2

I would rather use this LogFormat :

LogFormat = "%time2 %method %url %query
%logname %host %other %ua %referer %code
%bytesd"

With the short form of the LogFormat, AWStats has to search for a format specification within the input file; it could be that logresolve  removes this format specification.

Jean-Luc

9:27 am
November 5, 2007


Jim

Guest

Jean-Luc

Thank you very much for your quick reply!

This has done the trick!

I no longer need to paste that information into the combiuned log which means I can now automate the process and save myself lots of time.

Thank you very much for your help, I'm very grateful.

Cheers

Jim

3:43 am
November 6, 2007


Jim

Guest

Ooooopppsss!

 Spoke too soon!

 I've changed logfile format from LogFormat=2 to:

LogFormat = "%time2 %c-ip %cs-username %cs-method %cs-uri-stem %cs-uri-query %sc-status %sc-bytes %cs-version %cs(User-Agent) %cs(Referer)

Which is the format the logfiles are saved in. When I run Awstats now, it parses without error but doesn't produce any results when I creat a report. So I no longer need to paste that text into the top of the combined logfile like I did, which is great, but now I get no figures…confused?….oh yes!

 Jim

 

 

7:11 am
November 6, 2007


Jean-Luc

Admin

posts 1125

You write that you use: 

LogFormat = "%time2 %c-ip %cs-username %cs-method %cs-uri-stem %cs-uri-query %sc-status %sc-bytes %cs-version %cs(User-Agent) %cs(Referer)"

This LogFormat does not match the fields specification in your first post:

#Fields: date time cs-method cs-uri-stem cs-uri-query cs-username c-ip cs-version cs(User-Agent) cs(Referer) sc-status sc-bytes

For example, according to your first post, the field after the date and time is cs-method  which AWStats calls %method.

Did you try the LogFormat I suggested ?

At the end of the database update, AWStats prints a report like this one:

Create/Update database for config "/path_to_awstats/awstats.my_domainconf" by AWStats version 6.7 (build 1.892)
From data in log file "/path_to_log_file/log_file_name"…
Phase 1 : First bypass old records, searching new record…
Searching new records from beginning of log file…
Phase 2 : Now process new records (Flush history on disk after 20000 hosts)…
Jumped lines in file: 0
Parsed lines in file: 3693
 Found 520 dropped records,
 Found 12 corrupted records,
 Found 0 old records,
 Found 3161 new qualified records.

This information helps understand what happened when there is a problem.

Jean-Luc

8:47 am
November 9, 2007


Jim

Guest

Jean-Luc

Thank you! I used the logformat from the boxes ticked on the IIS settings rather than the format I originally posted. I've now used your suggestion and the files are parsed without a problem.

Cheers for your help!

One last question!

Now I want to automate the process using a scheduled task. I can set up a btach file with my Awstat commands and get the scheduled task to run but it seems logresolve doesn't like the format D:/LogFiles/W3SVC1725568973/ex%YY-0%MM-0%DD-24 to merge yesterday's file. It works if I specify the exacy logfile name e.g. ex071108.log.

Any ideas?

Jim

9:01 am
November 9, 2007


Jean-Luc

Admin

posts 1125

In batch files, the percent sign is a special character.

Try to replace each percent sign by two percent signs, like this:

D:/LogFiles/W3SVC1725568973/ex%%YY-0%%MM-0%%DD-24

Jean-Luc

9:40 am
November 9, 2007


Jim

Guest

Hi Jean-Luc

Thank you again for your quick reply

this is an improvement but now the command is interpreted literally and I get the following error written in the combined log:

Error: Couldn't open log file "D:/LogFiles/W3SVC1725568973/ex%YY-0%MM-0%DD-24" : No such file or directory.

I'll keep messing!

 Cheers dude

Jim

11:36 am
November 9, 2007


Jean-Luc

Admin

posts 1125

Two questions:

1. What error message did you get first with D:/LogFiles/W3SVC1725568973/ex%YY-0%MM-0%DD-24" ?

2. What is the complete line calling logresolvemerge.pl   in your batch file ?

Jean-Luc

5:57 am
November 12, 2007


Jim

Guest

Hi Jean-Luc

The first error message was:

Error: Couldn't open log file "D:/LogFiles/W3SVC1725568973/exMM-0/LogFiles/W3SVC1725568973/exMM-0DD-24.log" : No such file or directory.

This was when using the following CMD line in my batch file:

D:/website/applications/awstats-6.6/tools/logresolvemerge.pl D:/LogFiles/W3SVC1725568973/ex%YY-0%MM-0%DD-24.log Z:/LogFiles/W3SVC1725568973/ex%YY-0%MM-0%DD-24.log > /LogFiles/website-combined-logs/combinedlogs.log

Cheers

Jim

 

11:05 am
November 12, 2007


Jean-Luc

Admin

posts 1125

Did you replace all six original percent signs (% => %%) in the command line ? The .bat  file should contain:

D:/website/applications/awstats-6.6/tools/logresolvemerge.pl D:/LogFiles/W3SVC1725568973/ex%%YY-0%%MM-0%%DD-24.log Z:/LogFiles/W3SVC1725568973/ex%%YY-0%%MM-0%%DD-24.log > /LogFiles/website-combined-logs/combinedlogs.log

At execution time, this should give:

D:/website/applications/awstats-6.6/tools/logresolvemerge.pl D:/LogFiles/W3SVC1725568973/ex%YY-0%MM-0%DD-24.log Z:/LogFiles/W3SVC1725568973/ex%YY-0%MM-0%DD-24.log > /LogFiles/website-combined-logs/combinedlogs.log

And logresolvemerge.pl  should then replace %YY-0, %MM-0,… by the appropriate values.

Can you try again and let me know how the command looks like at execution time (ie. when you run the batch file). Thank you.

Jean-Luc

3:33 am
November 13, 2007


Jim

Guest

Hi Jean-Luc

This is the line I used in the batch file:

D:/website/applications/awstats-6.6/tools/logresolvemerge.pl D:/LogFiles/W3SVC1725568973/ex%%YY-0%%MM-0%%DD-24.log Z:/LogFiles/W3SVC1725568973/ex%%YY-0%%MM-0%%DD-24.log > /LogFiles/website-combined-logs/combinedlogs.log

When this is run in the batch file the combined log has the following error:

Error: Couldn't open log file "D:/LogFiles/W3SVC1725568973/ex%YY-0%MM-0%DD-24.log" : No such file or directory.

The two %% symbols format the line correctly but logresolve inerprets ex%YY-0%MM-0%DD-24.log literally rather than replacing the values with figures.

The execution lasts for less than a second as the logfile can't be found so nothing is parsed.

Jim

4:08 am
November 13, 2007


Jean-Luc

Admin

posts 1125

Jim,

It's getting clear ! The point is that logresolvemerge.pl  is not processing file names like awstats.pl  does.

You cannot use these %YY-xx%MM-xx%DD-xx  with logresolvemerge.pl. I am afraid you will have to use tricks around the DATE function in the batch file to automate things as you want.

Also note that %YY-0%MM-0%DD-24  is not going to do what you expect. On January 1, 2008, %YY-0%MM-0%DD-24  will return 080131. You should use %YY-24%MM-24%DD-24  that would return 071231.

Jean-Luc

6:38 am
November 13, 2007


Jim

Guest

Cheers jean-Luc

Thanks very much for your help, you've been brilliant.

I'll have a good mess around with my batch file.

Take it easy dude

 Jim

Reply to Post

Reply to Topic:
Automate Awstats updates with combined logs

Guest Name (Required):

Guest Email (Required):

NOTE: New Posts are subject to administrator approval before being displayed

Smileys
Confused Cool Cry Embarassed Frown Kiss Laugh Smile Surprised Wink Yell
Post New Reply

Guest URL (required)

Math Required!
What is the sum of:
1 + 3
   


About the InternetOfficer.com Forum

Forum Timezone: UTC 1

Most Users Ever Online: 302

Currently Online:
12 Guests

Currently Browsing this Topic:
1 Guest

Forum Stats:

Groups: 2
Forums: 9
Topics: 638
Posts: 2709

Membership:

There are 257 Members
There have been 302 Guests

There is 1 Admin
There is 1 Moderator

Top Posters:

cssfsu – 55
deepakgupta – 34
albert_newton – 30
cosminpana – 20
DTNMike – 19
ahtshun83 – 17

Recent New Members: raju, todd2taylor, sbdcunha, mansigill1987, ThomasDuh, ThomasKic

Administrators: Jean-Luc (1125 Posts)

Moderators: Jean-Luc (1125 Posts)