entry4

[ironicbadger]

For a work assignment a few years ago I was asked to write a script that would scrape a few hundred thousand public legal records from various county websites. I am a self-taught programmer and had some experience with web scraping but, at the time, was very far from competent.

The Python script that I wrote was very slow and quite ugly but appeared to be working! Every morning I would run the script and it would pick up where it left off, slowly adding more and more records to the folder where I was storing the data. This went on for about a month.

On the eve of the project being finished, I made some minor adjustments to the script (like a fool) and accidentally overwrote and deleted all of the data! I had not considered utilizing any backups and I was devastated. Weeks of dutifully running the script down the drain!

To make matters worse, when I told my boss, he called his brother (a real programmer) who wrote a script that scraped all the data in one day.