Tag Archive for 'bandbreedte'

Bandbreedte smelt als sneeuw voor de zon

Opvallend hoe mijn bandbreedte er door vliegt deze maand.
Ik dacht eerst weer dat Yahoo nogmaals aan een reeks ontzettend vervelende crawls was begonnen maar dat bleek nog mee te vallen.

Op zoek naar info in de logs en daar vond ik een massa zoals volgende :

81.246.51.202 – – [03/Aug/2009:15:49:13 +0200] “GET /wp-content/themes/k2/style.css HTTP/1.1” 200 24417 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:13 +0200] “GET /wp-includes/wlwmanifest.xml HTTP/1.1” 200 1053 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:13 +0200] “GET /wp-includes/js/jquery/jquery.js?ver=1.2.6 HTTP/1.1” 200 31111 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:13 +0200] “GET /wp-content/themes/k2/js/k2.functions.js.php?ver=1.0-RC7 HTTP/1.1” 200 3312 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:13 +0200] “GET /xmlrpc.php HTTP/1.1” 200 42 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:12 +0200] “GET /2008/04/07/beter-scoren-in-google/ HTTP/1.1” 200 48110 “http://www.google.be/search?hl=en&q=ik+verhoog+uw+google+ranking&meta=&aq=f&oq=&rlz=1W1GGLL_en” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6)”
81.246.51.202 – – [03/Aug/2009:15:49:13 +0200] “GET /wp-content/themes/k2/js/k2.slider.js.php?ver=1.0-RC7 HTTP/1.1” 200 3531 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:13 +0200] “GET /feed/ HTTP/1.1” 200 15394 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:13 +0200] “GET /feed/atom/ HTTP/1.1” 200 19760 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:13 +0200] “GET /feed/rss/ HTTP/1.1” 200 5912 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:14 +0200] “GET /wp-content/themes/k2/js/k2.trimmer.js.php?ver=1.0-RC7 HTTP/1.1” 200 2261 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:14 +0200] “GET /wp-content/themes/k2/js/k2.rollingarchives.js.php?ver=1.0-RC7 HTTP/1.1” 200 3423 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:14 +0200] “GET /wp-content/themes/k2/js/k2.livesearch.js.php?ver=1.0-RC7 HTTP/1.1” 200 2304 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:14 +0200] “GET /wp-content/themes/k2/js/k2.comments.js.php?ver=1.0-RC7 HTTP/1.1” 200 1640 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:14 +0200] “GET /wp-content/plugins/contact-form-7/stylesheet.css HTTP/1.1” 200 768 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:14 +0200] “GET /wp-includes/js/jquery/jquery.form.js?ver=2.02 HTTP/1.1” 200 31465 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:15 +0200] “GET /wp-content/plugins/contact-form-7/contact-form-7.js HTTP/1.1” 200 3745 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:15 +0200] “GET /wp-content/themes/k2/pagebar.css HTTP/1.1” 200 764 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:15 +0200] “GET /2009/07/ HTTP/1.1” 200 19313 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:15 +0200] “GET /2009/08/ HTTP/1.1” 200 18898 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:15 +0200] “GET /2009/05/ HTTP/1.1” 200 18925 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:16 +0200] “GET /2009/04/ HTTP/1.1” 200 59701 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:16 +0200] “GET /2009/03/ HTTP/1.1” 200 22595 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:16 +0200] “GET /2009/01/ HTTP/1.1” 200 48687 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:16 +0200] “GET /2009/02/ HTTP/1.1” 200 43428 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:17 +0200] “GET /2008/10/ HTTP/1.1” 200 30445 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:19 +0200] “GET /wp-content/plugins/social_news_b/sn-help.gif HTTP/1.1” 200 1024 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:19 +0200] “GET /wp-content/plugins/social_news_b/sn-msnreporter.gif HTTP/1.1” 200 1055 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:18 +0200] “GET /2008/03/ HTTP/1.1” 200 44170 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:20 +0200] “GET /wp-content/plugins/social_news_b/sn-bligg.gif HTTP/1.1” 200 1046 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:20 +0200] “GET /wp-content/plugins/social_news_b/sn-netjes.gif HTTP/1.1” 200 603 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:20 +0200] “GET /wp-content/plugins/social_news_b/sn-nujij.gif HTTP/1.1” 200 246 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:17 +0200] “GET /2008/11/ HTTP/1.1” 200 40228 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:20 +0200] “GET /wp-content/plugins/social_news_b/sn-ekudos.gif HTTP/1.1” 200 1074 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:20 +0200] “GET /wp-content/plugins/social_news_b/sn-su.gif HTTP/1.1” 200 633 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:20 +0200] “GET /wp-content/plugins/social_news_b/sn-delicious.gif HTTP/1.1” 200 89 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:17 +0200] “GET /2008/12/ HTTP/1.1” 200 25135 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:21 +0200] “GET /wp-content/plugins/social_news_b/sn-google.gif HTTP/1.1” 200 1071 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:21 +0200] “GET /wp-content/plugins/social_news_b/sn-rss.gif HTTP/1.1” 200 1026 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:21 +0200] “GET /wp-content/plugins/social_news_b/sn-email.png HTTP/1.1” 200 754 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:21 +0200] “GET /wp-includes/images/smilies/icon_wink.gif HTTP/1.1” 200 170 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:18 +0200] “GET /2008/02/ HTTP/1.1” 200 36218 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:21 +0200] “GET /wp-includes/images/smilies/icon_smile.gif HTTP/1.1” 200 174 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:21 +0200] “GET /wp-content/uploads/2009/04/belgische-firewall-boodschap-300×244.png HTTP/1.1” 200 50089 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:21 +0200] “GET /wp-content/plugins/custom-anti-spam/custom_anti_spam.php?antiselect=84655 HTTP/1.1” 200 2495 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:18 +0200] “GET /2007/02/ HTTP/1.1” 200 28039 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:18 +0200] “GET /2007/07/ HTTP/1.1” 200 20341 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:21 +0200] “GET /wp-content/uploads/2009/01/google-malware-192×300.png HTTP/1.1” 200 43826 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:21 +0200] “GET /wp-content/uploads/2009/01/scam0442-300×200.jpg HTTP/1.1” 200 30166 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:22 +0200] “GET /wp-content/uploads/2009/01/string-emil_0221-300×194.jpg HTTP/1.1” 200 25173 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:19 +0200] “GET /2007/01/ HTTP/1.1” 200 29306 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:18 +0200] “GET /2007/04/ HTTP/1.1” 200 42061 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:22 +0200] “GET /wp-content/uploads/2009/01/scam0395-300×200.jpg HTTP/1.1” 200 29973 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:18 +0200] “GET /2007/03/ HTTP/1.1” 200 23916 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:23 +0200] “GET /wp-content/themes/k2/images/tag_blue.png HTTP/1.1” 200 769 “http://deinternetmarketeer.be/2008/04/07/beter-scoren-in-google/” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6)”
81.246.51.202 – – [03/Aug/2009:15:49:23 +0200] “GET /wp-content/themes/k2/images/time.png HTTP/1.1” 200 964 “http://deinternetmarketeer.be/2008/04/07/beter-scoren-in-google/” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6)”
81.246.51.202 – – [03/Aug/2009:15:49:23 +0200] “GET /wp-content/uploads/2008/11/gmail-oud-300×151.png HTTP/1.1” 200 20724 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:17 +0200] “GET /2008/05/ HTTP/1.1” 200 53726 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:18 +0200] “GET /2007/05/ HTTP/1.1” 200 24934 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:18 +0200] “GET /2007/06/ HTTP/1.1” 200 47773 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:23 +0200] “GET /wp-content/uploads/2008/11/gmail-nieuw-300×151.png HTTP/1.1” 200 23489 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:23 +0200] “GET /wp-content/uploads/2008/11/who-is-belgium-king-search-ask-300×153.png HTTP/1.1” 200 39410 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:23 +0200] “GET /wp-content/uploads/2008/11/dubbele-listing-queromedia1-300×147.png HTTP/1.1” 200 42914 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:23 +0200] “GET /wp-content/uploads/2009/02/paid-links-300×147.png HTTP/1.1” 200 50227 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:17 +0200] “GET /2008/06/ HTTP/1.1” 200 22667 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:23 +0200] “GET /wp-content/uploads/2008/11/ask-erazer-eng-300×153.png HTTP/1.1” 200 37602 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:24 +0200] “GET /wp-content/themes/k2/images/feed.png HTTP/1.1” 200 774 “http://deinternetmarketeer.be/2008/04/07/beter-scoren-in-google/” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6)”
81.246.51.202 – – [03/Aug/2009:15:49:18 +0200] “GET /2008/01/ HTTP/1.1” 200 40910 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:13 +0200] “GET /xmlrpc.php?rsd HTTP/1.1” 200 870 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:24 +0200] “GET /wp-content/uploads/2008/06/gmail_langezame-300×151.png HTTP/1.1” 200 7312 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:24 +0200] “GET /wp-content/uploads/2008/06/hoofdletters-in-google-300×146.png HTTP/1.1” 200 33889 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:23 +0200] “GET /wp-content/uploads/2008/11/dubbele-listings-300×146.png HTTP/1.1” 200 36468 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:24 +0200] “GET /wp-content/themes/k2/images/arrow_refresh.png HTTP/1.1” 200 789 “http://deinternetmarketeer.be/2008/04/07/beter-scoren-in-google/” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6)”
81.246.51.202 – – [03/Aug/2009:15:49:24 +0200] “GET /wp-content/themes/k2/images/spinner.gif HTTP/1.1” 200 847 “http://deinternetmarketeer.be/2008/04/07/beter-scoren-in-google/” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6)”
81.246.51.202 – – [03/Aug/2009:15:49:23 +0200] “GET /wp-content/plugins/custom-anti-spam/custom_anti_spam.php?antiselect=84655 HTTP/1.1” 200 2663 “http://deinternetmarketeer.be/2008/04/07/beter-scoren-in-google/” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6)”
81.246.51.202 – – [03/Aug/2009:15:49:18 +0200] “GET /2007/10/ HTTP/1.1” 200 48356 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:24 +0200] “GET /wp-content/themes/k2/images/reset-fff.png HTTP/1.1” 200 322 “http://deinternetmarketeer.be/2008/04/07/beter-scoren-in-google/” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6)”
81.246.51.202 – – [03/Aug/2009:15:49:18 +0200] “GET /2007/11/ HTTP/1.1” 200 42174 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:17 +0200] “GET /2008/04/ HTTP/1.1” 200 38189 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:18 +0200] “GET /2007/09/ HTTP/1.1” 200 33532 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:18 +0200] “GET /2007/08/ HTTP/1.1” 200 35139 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:24 +0200] “GET /wp-includes/images/smilies/icon_cool.gif HTTP/1.1” 200 172 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:18 +0200] “GET /2007/12/ HTTP/1.1” 200 42992 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:17 +0200] “GET /2008/08/ HTTP/1.1” 200 53431 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:25 +0200] “GET /wp-content/uploads/2008/11/google-captcha-300×181.jpg HTTP/1.1” 200 14081 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:25 +0200] “GET /wp-content/uploads/2008/11/era_vastgoed_malware-300×181.jpg HTTP/1.1” 200 15421 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:25 +0200] “GET /wp-content/uploads/2008/11/link_era_malware-300×187.jpg HTTP/1.1” 200 13885 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:25 +0200] “GET /wp-content/uploads/2008/11/hln-300×153.png HTTP/1.1” 200 36753 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:25 +0200] “GET /wp-content/uploads/2008/11/staatsbladclip-300×153.png HTTP/1.1” 200 39808 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:25 +0200] “GET /wp-content/uploads/2008/11/klik_link_era-300×187.jpg HTTP/1.1” 200 11162 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:24 +0200] “GET /wp-content/uploads/2008/11/ask-google-zoeken_1192229020765-300×146.png HTTP/1.1” 200 30801 “-” “Mozilla/4.0 (compatible;)”
81.246.51.202 – – [03/Aug/2009:15:49:24 +0200] “GET /favicon.ico HTTP/1.1” 404 26028 “-” “Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; GTB6)”
81.246.51.202 – – [03/Aug/2009:15:49:25 +0200] “GET /wp-content/uploads/2008/08/deredactiebe-economie-300×153.png HTTP/1.1” 200 46773 “-” “Mozilla/4.0 (compatible;)”

Heeft er iemand enig gedacht wat hier aan de hand is???
Een foute plugin?
Virus op de computers van gebruikers?
1 of andere nieuwe spitsvondige technologie?

Ik begrijp er niks van en heb geen zin om het even allemaal te gaan uitzoeken.

Anyway, als ik binnenkort offline ga … U weet hoe het komt.

 

Stop Yahoo!’s gekke spidergedrag

Yahoo! Slurp raast als een gek over het internet om website te indexeren.
Te gek.
Het lijkt wel dat hun spidergedrag erop gericht is om het hele netwerk stil te leggen in de wereld. Het zo massaal spideren van alle sites in de wereld vreet bandbreedte.
Te veel bandbreedte.
Dat zou komen omdat ze hun spidergedrag laten afhangen van het ip. En als je dan meerdere ip’s hebt op dezelfde server zou het dus kunnen dat je te veel wordt gespiderd.

Ze hebben om dat op te lossen iets bedacht dat je in je robots.txt kan gebruiken.
Dat vind ik larie maar het is beter dan niks. Je hoeft hun waanzinnige spidergedrag dus niet langer te pikken.

Crawl-delay van Yahoo!

Yahoo! heeft dus de Crawl-delay uitgevonden opdat je als website beheerder enigszins controle kan krijgen over je bandbreedte die wordt opgevreten.
Na wat tests lijkt het ook daadwerkelijk te werken.

Volgende dien je op te nemen in je robots.txt file indien je te veel gespiderd wordt door Yahoo! :

User-agent: Slurp
Crawl-delay: x.x

Het eerste gedeelte geeft aan voor welke spider van welke zoekmachine de volgende regel is bestemd.
Crawl-delay is het commando om de Slurp spider te laten begrijpen hoe vaak hij mag langs komen. De waarde achter Crawl-delay is een integer getal met eventueel een aanduiding na de komma.
(Amerikaans, dus een komma is een punt)
In de praktijk is deze komma echter overbodig.
De waarde achter Crawl-delay mag dus een gewoon getal zijn van 1 tot pakweg 100 of meer.

In de praktijk kan je voor een blog gerust een Crawl-delay van 50 neerpoten heb ik ondervonden. Het hangt allemaal een beetje af van hoe groot je site is(aantal pagina’s).

User-agent: Slurp
Crawl-delay: 50

Best is dat je een crawl-delay instelt en na 2 weken evalueert of deze nu naar wens werkt.
Je kan vergelijken met de frequentie van Google spiders, Yahoo! Slurp moet daar zekers niet boven uit komen.

STOP de waanzinnig geworden Yahoo! Slurp spiders!

Yahoo!

Yahoo!