I’m trying to import several years of logfiles and have tried using logresolvemerge.pl, but I am having issues with all merged results.
If I update with each original logfile separately, everything works fine and the stats are populated, but I have too many files to do them all individually. I first tried running logresolvemerge.pl with w CMD to build logfile groups by month, which seemed to work fine, creating larger monthly files without errors. However, once I try to update the stats with any of those merged logfiles, with the same config settings, ALL traffic stats show up as “not viewed traffic”.
So, I tried running logresolvemerge.pl as part of the logfile config itself (LogFile=perl E:\PATH\logresolvemerge.pl E:\PATH\ex0907*.log |"). This again seems to work without errors, but this time the stats are simply empty afterwards, with no data being imported at all.
All of the these updates have been run with the default LogFormat=2 for IIS. When I have tried it with other suggested LogFormat settings, I get the “must have a bad format or LogFormat parameter setup does not match this format” error. Here is an example of my logfile format:
LogFormat=2 cannot be used with logresolvemerge , because logresolvemerge does not include the IIS format lines in the merged log file it generates.
Paste here one of these IIS format lines (line starting with #Fields: from one of the original log files) and we will tell what your LogFormat directive must contain.
Thanks for the code. I ran logresolvemerge, separately trying both methods I used before, and it ran both times without errors. However, ALL traffic stats show up as “not viewed traffic” again for both attempts.
Thanks for the update. I'm giving that a try now… It seems to be working so far, as it's running longer than ever before, and I'm getting these messages:
Flush history file on disk (unique url reach flush limit of 5000)
You get this message "Flush history file on disk …" when AWStats updates temporary data files. This happens when AWStats is processing large log files.
Thank you – Your suggestions worked out great! I'm almost done importing the older months, but I have a couple of follow up questions:
1. I have one month that doesn't always want to load (it just "waits" for the stats to update, but usually never does). All other months work fine. Any suggestions?
2. Once I'm done importing the last batch set for this month, what dynamic code should I use for the logifle link to update each day, and what method is best for auto-updating the stats so nothing is missed?
I don't know what you mean by "one month that (…) just "waits" for the stats to update, but usually never does ". Do you mean that you run several updates from the same log file and that it does not always give the same results ?
Regarding your second question, you need to set the Windows Task Scheduler to run the update shortly (say 5 minutes) after the log file of the day has been completed. Note that IIS log files usually use GMT timestamps, so the day will be completed at midnight GMT, while the Windows Task Scheduler will refer to the Windows server local time.
The LogFile directive can be
LogFile="E:\path_to\ex%YY%MM%DD.log"
or
LogFile="E:\path_to\ex%YY-24%MM-24%DD-24.log"
depending on the time difference between your Windows server local time and GMT.
Your log format does not contain a referrer field. The referrer field is needed to know the origin of the visits (search engines and other sites) and the search engine keywords entered to reach your website.