So far this month, Googlebot has hit kstruct 521 times, downloading a total of 2.8 megs, where as MSNBot has 496 hits for a total of 11.32 megs.

That means that, per hit, MSNBot is loading about 23.3k of data, where as Googlebot is loading about 5.5k (and Yahoo slurp is similar with 4.5k).

My question is (being far to lazy to go and actually look at the raw logs myself), what’s MSNBot doing differently that it should be downloading over four times more data per hit?

My guesses would include MSNBot not understanding some ‘content not changed’ hint or being more interested in images and PDFs than the others, but I’ve no idea if that’s even close.

[Update: Looks like I'm not the only one to have noticed.]

Technorati Tags: , , ,