Discussions

I recently encountered a problem related to working with large amounts of data. I had a project analyzing local businesses, for which I needed to collect a lot of information and then organize it somehow. The hardest part was not collecting the data, but not drowning in it afterwards. Simple things helped: dividing the downloaded files into parts, immediately checking the structure, and refusing to put off cleaning up “until later.” By the way, while working with geodata, I found a tool for scraping Google Maps listings. I happened to find it by accident, just as a way to simplify the collection process, without any advertising. I wonder what techniques you use to keep large data sets from turning into chaos?