logresolvemerge keeping duplicates

Print this Post

Page: 1

Reply to Post

Add a New Topic

User

Post

9:57 pm
March 18, 2010

cssfsu

Member

posts 55

Quote and Reply

Print this Post

Hi jean,

I want to use logresolvemerge for merging a few log files together. However I see that it does not get rid of duplicates. I know its only a tool for merging, but is there any way that I can have logresolvemerge get rid of duplicates ?

For eg log1.log has

9.49.199.111 – - [04/Feb/2010:18:33:41 -0500] "GET / HTTP/1.1" 302 2296 "18734/CF-WRK: 982399630 : 0" 3943591
9.49.199.111 – - [04/Feb/2010:18:33:42 -0500] "GET /login.gt HTTP/1.1" 200 14400 "18734/CF-WRK: 982399630 : 0" 700701
9.49.199.111 – - [04/Feb/2010:18:33:44 -0500] "GET /js/cic.js HTTP/1.1" 302 2296 "18734/CF-WRK: 982399630 : 1" 17234

log2.log has

9.49.199.111 – - [04/Feb/2010:18:33:41 -0500] "GET / HTTP/1.1" 302 2296 "18734/CF-WRK: 982399630 : 0" 3943591
–something new here—

The combined log shows the first entry from log2.log as well (which is a dup)

9.49.199.111 – - [04/Feb/2010:18:33:41 -0500] "GET / HTTP/1.1" 302 2296 "18734/CF-WRK: 982399630 : 0" 3943591
9.49.199.111 – - [04/Feb/2010:18:33:41 -0500] "GET / HTTP/1.1" 302 2296 "18734/CF-WRK: 982399630 : 0" 3943591 9.49.199.111 – - [04/Feb/2010:18:33:42 -0500] "GET /login.gt HTTP/1.1" 200 14400 "18734/CF-WRK: 982399630 : 0" 700701
9.49.199.111 – - [04/Feb/2010:18:33:44 -0500] "GET /js/cic.js HTTP/1.1" 302 2296 "18734/CF-WRK: 982399630 : 1" 17234
–something new here—

10:06 pm
March 18, 2010

cssfsu

Member

posts 55

Quote and Reply

Print this Post

One more question: If the combined log does have duplicates, how will Awstats process it as? Will it count the below as 2 hits or is it smart enough to know duplicates and count as 1 ?

9.49.199.111 – – [04/Feb/2010:18:33:41 -0500] "GET / HTTP/1.1″ 302 2296 "18734/CF-WRK: 982399630 : 0″ 3943591
9.49.199.111 – – [04/Feb/2010:18:33:41 -0500] "GET / HTTP/1.1″ 302 2296 "18734/CF-WRK: 982399630 : 0″ 394359

9:19 am
March 20, 2010

Jean-Luc

Admin

posts 1043

Quote and Reply

Print this Post

Hi,

Duplicates will count as 2 hits. logresolvemerge.pl does not remove them either.

You will have to clean up the log files before you pass them to AWStats if you want to get rid of the duplicates. This will require to patch logresolvemerge.pl or awstats.pl or to parse the log file one more time, before it is processed by awstats.pl .

Page: 1

Reply to Post