Comments on: Bots baby http://blog.niceperson.org/2003/03/03/bots-baby/ Making it up as I go along. Mon, 07 Feb 2011 18:34:39 +0000 hourly 1 http://wordpress.org/?v=3.2.1 By: Kevin http://blog.niceperson.org/2003/03/03/bots-baby/comment-page-1/#comment-92 Kevin Wed, 30 Nov -0001 00:00:00 +0000 http://www.niceperson.org/wordpress/2003/03/03/bots-baby/#comment-92 The thing with blocking bots is you need to find a balance. If you block blindly then it's a bit futile, but if you're conservative about it then it can be a huge help. As for censorship, with address harvesters all you're doing is stopping them helping themselves to information they're supposed to ask nicely for anyway. The thing with blocking bots is you need to find a balance. If you block blindly then it's a bit futile, but if you're conservative about it then it can be a huge help. As for censorship, with address harvesters all you're doing is stopping them helping themselves to information they're supposed to ask nicely for anyway.

]]> By: Laurabelle http://blog.niceperson.org/2003/03/03/bots-baby/comment-page-1/#comment-93 Laurabelle Wed, 30 Nov -0001 00:00:00 +0000 http://www.niceperson.org/wordpress/2003/03/03/bots-baby/#comment-93 But that's, you know, me saying who can or cannot have access to my information, and as a proto-librarian that's contrary to the culture I've been breathing for the past six months. I think I'll feel better about it when I write a bit of a site policy saying things like <q>this site is for reading, not harvesting or bulk downloading.</q> As for email harvesters, there's no point in their asking nicely, because I wouldn't give them addresses anyway. I guess they don't annoy me so much, though I like telling them explicitly that they're not welcome. What really bugs me is this one user-agent coming around all the time, I think from the same hosts, requesting the same two or three files. What the heck are they doing, if they're not crawling my whole site? But that's, you know, me saying who can or cannot have access to my information, and as a proto-librarian that's contrary to the culture I've been breathing for the past six months. I think I'll feel better about it when I write a bit of a site policy saying things like this site is for reading, not harvesting or bulk downloading.

As for email harvesters, there's no point in their asking nicely, because I wouldn't give them addresses anyway. I guess they don't annoy me so much, though I like telling them explicitly that they're not welcome. What really bugs me is this one user-agent coming around all the time, I think from the same hosts, requesting the same two or three files. What the heck are they doing, if they're not crawling my whole site?

]]>