What makes you think they obey robots.txt?
Yeah, if robots.txt was all it took to foil them, I wouldn't think you'd see the same proliferation of Anubis-protection that has come about of late.
The other option aside from Anubis-style protection is stuff that tries to poison the data that scrapers get. Stuff like Nepenthes.
