Comments on: SoGou http://www.internetofficer.com/web-robot/sogou/ Tools and Articles for Webmasters and SEO's Thu, 22 Jul 2010 17:36:36 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Abel Cheung http://www.internetofficer.com/web-robot/sogou/comment-page-1/#comment-1463 Abel Cheung Sat, 02 Aug 2008 09:49:14 +0000 http://www.internetofficer.com/web-robot/sogou/#comment-1463 Not only nasty, it even intentionally ignores my robots.txt (which has not been updated for 1.5 yr) and directly crawls all pages explicitly disallowed in robots.txt. Thus I'm not nice to them as well. <code>Rewritecond %{HTTP_USER_AGENT} "^Sogou" RewriteRule .* http:/<em></em>/ww<em></em>w.sogou.com/ [L,R=301]</code> This is implemented with firewall using packet rate limiting on ACK packets as well (yes, not SYN). Not only nasty, it even intentionally ignores my robots.txt (which has not been updated for 1.5 yr) and directly crawls all pages explicitly disallowed in robots.txt.

Thus I’m not nice to them as well.

Rewritecond %{HTTP_USER_AGENT} "^Sogou"
RewriteRule .* http://www.sogou.com/ [L,R=301]

This is implemented with firewall using packet rate limiting on ACK packets as well (yes, not SYN).

]]> By: Jacco V. http://www.internetofficer.com/web-robot/sogou/comment-page-1/#comment-1115 Jacco V. Wed, 12 Sep 2007 15:04:02 +0000 http://www.internetofficer.com/web-robot/sogou/#comment-1115 this robot is quite nasty It somehow sniffs internet trafic and tries to access it. It even tries to pickup a copy of session-bound pages <br /> <br /> Webtravellog User: <code>124.115.220.*** - - [12/Sep/2007:14:02:13 +0200] "GET /loginSuccess.php?sessionId=a222d271f54ef1809cc567300fe9ba3f HTTP/1.1" 200 17595 "https://*****.webtravellog.com/login/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" SSLv3 RC4-MD5</code> <br /> <br /> And, the robot: <code>220.181.19.72 - - [12/Sep/2007:14:11:03 +0200] "GET /loginSuccess.php?sessionId=a222d271f54ef1809cc567300fe9ba3f HTTP/1.1" 200 8093 "-" "Sogou Orion spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" SSLv3 DHE-RSA-AES256-SHA</code> this robot is quite nasty

It somehow sniffs internet trafic and tries to access it.
It even tries to pickup a copy of session-bound pages

 
Webtravellog User:
124.115.220.*** - - [12/Sep/2007:14:02:13 +0200] "GET /loginSuccess.php?sessionId=a222d271f54ef1809cc567300fe9ba3f HTTP/1.1" 200 17595 "https://*****.webtravellog.com/login/" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" SSLv3 RC4-MD5

 
And, the robot:
220.181.19.72 - - [12/Sep/2007:14:11:03 +0200] "GET /loginSuccess.php?sessionId=a222d271f54ef1809cc567300fe9ba3f HTTP/1.1" 200 8093 "-" "Sogou Orion spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)" SSLv3 DHE-RSA-AES256-SHA

]]>