I have been giving serious thought to building a Wrothscraper utility.

(When I say "serious thought", I mean I have devoted the brain cycles of at least two pre-work morning showers to it, which is when I most consistently get my second best thinking done).

I reckon I can write an application that reads everything on the Wroth, and stores it away in a seriously text-indexed database, and have it actually get all that data in not too long a time, without actually breaking the wroth. Might take a couple of months to get the raw data, and then a matter of a couple of weeks to store and index it.

And then the reports I could build from that data would be brilliant. Effortlessly cross reference Big Feller words-per-post and posts-per-day against Norwich results and recent form. Sweariest poster. Vocabulary league table. Your query about whatever that usa halfwit has been going on about would be solved in seconds - just query the db with keywords and poster name.

It would be a *lot* of data to store, what with the indexing, which would have to be quite voluminous to be effective. But storage is cheap these days.

Posted By: Arizona Bay, Sep 29, 23:16:50