Current User: Guest Login
Please consider registering


Lost Your Password?

Search Forums:


 






Minimum search word length is 4 characters – Maximum search word length is 84 characters
Wildcard Usage:
*  matches any number of characters    %  matches exactly one character

Issues with logresolvemerge.pl results and logformat

Reply to Post Add a New Topic
UserPost

1:39 am
August 21, 2010


stewiedv

Member

posts 5

I’m trying to import several years of logfiles and have tried using logresolvemerge.pl, but I am having issues with all merged results.

If I update with each original logfile separately, everything works fine and the stats are populated, but I have too many files to do them all individually.  I first tried running logresolvemerge.pl with w CMD to build logfile groups by month, which seemed to work fine, creating larger monthly files without errors.  However, once I try to update the stats with any of those merged logfiles, with the same config settings, ALL traffic stats show up as “not viewed traffic”.  

So, I tried running logresolvemerge.pl as part of the logfile config itself (LogFile=perl E:\PATH\logresolvemerge.pl E:\PATH\ex0907*.log |"). This again seems to work without errors, but this time the stats are simply empty afterwards, with no data being imported at all.

All of the these updates have been run with the default LogFormat=2 for IIS.  When I have tried it with other suggested LogFormat settings, I get the “must have a bad format or LogFormat parameter setup does not match this format” error.  Here is an example of my logfile format:

2009-07-01 00:00:31 W3SVC1234567 servername GET /file_01.wmx – 80 – 172.110.10.10  HTTP/1.1 Mozilla/4.0+(compatible;+MSIE+8.0;+Windows+NT+6.0;+Trident/4.0;+SLCC1;+.NET+CLR+2.0.50727; +Media+Center+PC+5.0;+.NET+CLR+1.1.4322;+.NET+CLR+3.5.30729;+OfficeLiveConnector.1.3; +OfficeLivePatch.0.0;+.NET+CLR+3.0.30729) – 304 0 0 211 462

Any suggestions how I can get logresolvemerge.pl to work properly, and how to best set the logFormat?

Thanks

8:20 am
August 21, 2010


Jean-Luc

Admin

posts 1125

Hi,

LogFormat=2  cannot be used with logresolvemerge , because logresolvemerge  does not include the IIS format lines in the merged log file it generates.

Paste here one of these IIS format lines (line starting with #Fields:  from one of the original log files) and we will tell what your LogFormat  directive must contain.

1:19 am
August 22, 2010


stewiedv

Member

posts 5

Thanks for the assistance.  Here is the information you requested:

#Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status sc-bytes cs-bytes

 

Thanks!

8:09 am
August 22, 2010


Jean-Luc

Admin

posts 1125

Try this:

LogFormat = "%time2 %other %other %method %url %query %other %logname %host %other %ua %referer %code %other %other %bytesd"

7:31 pm
August 22, 2010


stewiedv

Member

posts 5

Thanks for the code.  I ran logresolvemerge, separately trying both methods I used before, and it ran both times without errors. However, ALL traffic stats show up as “not viewed traffic” again for both attempts.

8:34 pm
August 22, 2010


Jean-Luc

Admin

posts 1125

I forgot the last field in the format. I hope this will give better results:

Try this:

LogFormat = "%time2 %other %other %method %url %query %other %logname %host %other %ua %referer %code %other %other %bytesd %other"

If this still does not work, email me (awstats@internetofficer.com) 10 lines from your log files.

9:06 pm
August 22, 2010


stewiedv

Member

posts 5

Thanks for the update.  I'm giving that a try now… It seems to be working so far, as it's running longer than ever before, and I'm getting these messages:

 

Flush history file on disk (unique url reach flush limit of 5000)

 

I'll post the final results when it's complete.

 

Thx

12:23 pm
August 23, 2010


Jean-Luc

Admin

posts 1125

You get this message "Flush history file on disk …" when AWStats updates temporary data files. This happens when AWStats is processing large log files.

12:33 am
September 9, 2010


stewiedv

Member

posts 5

Thank you – Your suggestions worked out great!  I'm almost done importing the older months, but I have a couple of follow up questions:

1.  I have one month that doesn't always want to load (it just "waits" for the stats to update, but usually never does). All other months work fine.  Any suggestions?

2.  Once I'm done importing the last batch set for this month, what dynamic code should I use for the logifle link to update each day, and what method is best for auto-updating the stats so nothing is missed?

 

Thanks again.

7:05 am
September 9, 2010


Jean-Luc

Admin

posts 1125

Hello,

I don't know what you mean by "one month that (…) just "waits" for the stats to update, but usually never does ". Do you mean that you run several updates from the same log file and that it does not always give the same results ?

Regarding your second question, you need to set the Windows Task Scheduler to run the update shortly (say 5 minutes) after the log file of the day has been completed. Note that IIS log files usually use GMT timestamps, so the day will be completed at midnight GMT, while the Windows Task Scheduler will refer to the Windows server local time.

The LogFile directive can be

LogFile="E:\path_to\ex%YY%MM%DD.log"

or

LogFile="E:\path_to\ex%YY-24%MM-24%DD-24.log"

depending on the time difference between your Windows server local time and GMT.

9:42 pm
November 18, 2010


bluestyle55

New Member

posts 2

My IIS format is as follows:

 

#Fields: date time s-sitename s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) sc-status sc-substatus sc-win32-status sc-bytes

  

Could you please send me the line of code for the LogFormat?

 

Thanks!

Ryan 

10:18 pm
November 18, 2010


Jean-Luc

Admin

posts 1125

Hi,

Try this :

LogFormat = "%time2 %other %other %method %url %query %other %logname %host %ua %code %other %other %bytesd"

Your log format does not contain a referrer field. The referrer field is needed to know the origin of the visits (search engines and other sites) and the search engine keywords entered to reach your website.

2:49 pm
November 22, 2010


bluestyle55

New Member

posts 2

Jean Luc,

I am running AWStats on my SharePoint Intranet site so am I correct in thinking that the referrer field wouldnt provide any value?

3:10 pm
November 22, 2010


Jean-Luc

Admin

posts 1125

I agree that, in many intranets, there is no interest for the referrers.

Reply to Post

Reply to Topic:
Issues with logresolvemerge.pl results and logformat

Guest Name (Required):

Guest Email (Required):

NOTE: New Posts are subject to administrator approval before being displayed

Smileys
Confused Cool Cry Embarassed Frown Kiss Laugh Smile Surprised Wink Yell
Post New Reply

Guest URL (required)

Math Required!
What is the sum of:
4 + 12
   


About the InternetOfficer.com Forum

Forum Timezone: UTC 1

Most Users Ever Online: 302

Currently Online:
11 Guests

Currently Browsing this Topic:
1 Guest

Forum Stats:

Groups: 2
Forums: 9
Topics: 639
Posts: 2710

Membership:

There are 257 Members
There have been 304 Guests

There is 1 Admin
There is 1 Moderator

Top Posters:

cssfsu – 55
deepakgupta – 34
albert_newton – 30
cosminpana – 20
DTNMike – 19
ahtshun83 – 17

Recent New Members: raju, todd2taylor, sbdcunha, mansigill1987, ThomasDuh, ThomasKic

Administrators: Jean-Luc (1125 Posts)

Moderators: Jean-Luc (1125 Posts)