Current User: Guest Login
Please consider registering


Lost Your Password?

Search Forums:


 






Minimum search word length is 4 characters – Maximum search word length is 84 characters
Wildcard Usage:
*  matches any number of characters    %  matches exactly one character

Bluecoat logs and AWSTATS

Reply to Post Add a New Topic
UserPost

10:57 pm
May 31, 2009


reaver

New Member

posts 2

I'm trying to get awstats working with our Bluecoat webcaching devices. It should be straight forward, here is the #Fields from the log file;

#Fields: date time time-taken c-ip cs-username cs-auth-group x-exception-id sc-filter-result cs-categories cs(Referer) sc-s
tatus s-action cs-method rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(Us
er-Agent) s-ip sc-bytes cs-bytes x-virus-id

I have tried the following in the Logformat entry in my config file

LogFormat = "%time2 %other %host %logname %other %other %other %otherquot %referer %code %other %method %other %other %virtualname %other %url %query %other %uaquot %host_r %bytesd %other %other"

and

LogFormat = "%time2 %time-taken %c-ip %cs-username %cs-auth-group %x-exception-id %sc-filter-result %cs-categories %cs(Referer) %sc-status %s-action %cs-method %rs(Content-Type) %cs-uri-scheme %cs-host %cs-uri-port %cs-uri-path %cs-uri-query %cs-uri-extension %cs(User-Agent) %s-ip %sc-bytes %cs-bytes %x-virus-id"   

Neither of them work…. 

For case 1 above I get the following error;

Your AWStats LogFormat parameter is:
%time2 %other %host %logname %other %other %other %otherquot %referer %code %other %method %other %other %virtualname %other %url %query %other %uaquot %host_r %bytesd %other %other
This means each line in your web server log file need to have the following personalized log format:
%time2 %other %host %logname %other %other %other %otherquot %referer %code %other %method %other %other %virtualname %other %url %query %other %uaquot %host_r %bytesd %other %other
And this is an example of records AWStats found in your log file (the record number 50 in your log):

and for case 2 above I get this;

Error: Your personalized LogFormat does not include all fields required by AWStats (Add %methodurl or %url in your LogFormat string).

does anyone have any idea why I get this, I've checked and double checked the format and I am sure it is correct.  I am using AWSTATS 6.9 on Debian

Steve

1:31 am
June 1, 2009


Jean-Luc

Admin

posts 1125

Hi,

I would try this :

LogFormat = "%time2 %other %host %logname %other %other %other %other %referer %code %other %method %other %other %virtualname %other %url %query %other %ua %other %bytesd %other %other"

I am not sure about the %ua  field. It would be easier to help with a copy of a few lines from your log file (10 lines is enough).

If the field separator is not a space character, the LogSeparator  directive must be adjusted too (for example, when the field separator is a tab).

8:14 pm
June 1, 2009


reaver

New Member

posts 2

WOW, thanks for getting back so quickly.   It seemed to work, but when I look at the output web page there is no data.

Here are a few lines from the logs, I have changed IP's and url's for security reasons….

2009-03-11 15:00:00 10 123.123.123.123
- – - PROXIED “none” -  404 TCP_NC_MISS
 GET – http http://www.xyz.com.au 80 /m
ob/business/versions.xml – xml “Mozilla
/4.0 (compatible; MSIE 6.0; Windows NT
5.1; .NET CLR 1.1.4322)” 11.84.57.78 25
882 224 -

2009-03-11 15:00:00 1 68.172.136.88 – -
 - PROXIED “none” https://www.xyz.com.a
u/prepaidplus/myprepaid/jfn?actionID=CO
NTRACT_VIEW  200 TCP_HIT GET image/gif
http http://www.xyz.com.au 80 /global/t
hemes/v9/images/nav_arrow_flyout.gif –
gif “Mozilla/4.0 (compatible; MSIE 7.0;
 Windows NT 5.1; .NET CLR 2.0.50727; .N
ET CLR 3.0.04506.648; .NET CLR 3.5.2102
2)” 68.172.136.88 353 2134 -

2009-03-11 15:00:00 1 211.12.98.238 – -
 - PROXIED “none” http://www.xyz.com.au
/mob/prep…..epaid002GOSEM01SEP08STA0000
1&s_kwcid=prepaid|2690977840  200 TCP_H
IT GET text/css http http://www.xyz.com
.au 80 /mob/css/titles.css – css “Mozil
la/4.0 (compatible; MSIE 7.0; Windows N
T 5.1; InfoPath.2)” 11.84.55.67 10523 4
49 -

2009-03-11 15:00:00 1 58.170.172.71 – -
 - PROXIED “none” http://www.xyz.com.au
/mobile/index.html  200 TCP_HIT GET ima
ge/png http http://www.xyz.com.au 80 /m
obile/images/selectmobiles/main/telstra
_165i.png – png “Mozilla/5.0 (Macintosh
; U; Intel Mac OS X 10_5_5; en-us) Appl
eWebKit/525.18 (KHTML, like Gecko) Vers
ion/3.1.2 Safari/525.20.1? 11.84.55.69
12575 835 -

2009-03-11 15:00:00 1 203.192.103.157 -
 - – PROXIED “none” http://www.xyz.com.
au/mob/prep…..id/mob.cfm  200 TCP_HIT G
ET image/gif http http://www.xyz.com.au
 80 /global/themes/v9/images/telstra_lo
go_hover.gif – gif “Mozilla/5.0 (Window
s; U; Windows NT 5.1; en-US; rv:1.9.0.7
) Gecko/2009021910 Firefox/3.0.7? 11.84
.55.69 2068 1455 -

When I run awstats it outputs the following;

Phase 1 : First bypass old records, searching new record…
Searching new records from beginning of log file…
Jumped lines in file: 0
Parsed lines in file: 7679355
 Found 7223271 dropped records,
 Found 456084 corrupted records,
 Found 0 old records,
 Found 0 new qualified records.

There are some quote fields in there so I tried changing them to %otherquot and %uaquot but it just barfs and says…..

 AWStats did not find any valid log lines that match your LogFormat parameter…blah…blah…blah

thx

12:26 am
June 2, 2009


Jean-Luc

Admin

posts 1125

I believe that this one is correct :

LogFormat = "%time2 %other %host %logname %other %other %other %otherquot %referer %code %other %method %other %other %virtualname %other %url %query %other %uaquot %other %bytesd %other %other"

It corrects the two quoted fields (cs-categories and cs(User-Agent) ).

When you are testing, always remove all AWStats data files from the DirData  directory before you run the AWStats update.

If this configuration does not work, I would edit the LogSeparator :

LogSeparator="\\s+"

The DirData  directory must again be empty before you run this new test.

Reply to Post

Reply to Topic:
Bluecoat logs and AWSTATS

Guest Name (Required):

Guest Email (Required):

NOTE: New Posts are subject to administrator approval before being displayed

Smileys
Confused Cool Cry Embarassed Frown Kiss Laugh Smile Surprised Wink Yell
Post New Reply

Guest URL (required)

Math Required!
What is the sum of:
6 + 6
   


About the InternetOfficer.com Forum

Forum Timezone: UTC 1

Most Users Ever Online: 302

Currently Online:
9 Guests

Currently Browsing this Topic:
1 Guest

Forum Stats:

Groups: 2
Forums: 9
Topics: 638
Posts: 2709

Membership:

There are 257 Members
There have been 302 Guests

There is 1 Admin
There is 1 Moderator

Top Posters:

cssfsu – 55
deepakgupta – 34
albert_newton – 30
cosminpana – 20
DTNMike – 19
ahtshun83 – 17

Recent New Members: raju, todd2taylor, sbdcunha, mansigill1987, ThomasDuh, ThomasKic

Administrators: Jean-Luc (1125 Posts)

Moderators: Jean-Luc (1125 Posts)