Indeed

Of course, if they wanted to they could get off Google right now by altering their robot.txt files. Instead as someone found today the Fox News robot.txt files actually encourage Google indexing (I think this was on The Register).

Posted By: BerlinCanary, Nov 23, 20:06:06