samedi 17 août 2019

How am I supposed to crawl an entire website (using scrappy) without reading every url from a text file?

I’ve just been introduced to web scraping (using scrappy) and every tutorial I’ve found so far always shows how to scrape a website by simply reading every url the program goes to from a text file. I want to do a project where I scrape a very large website for data and copying/pasting and reading every url I want my program to go to just wouldn’t be possible. Can someone explain how I would accomplish my project in detail or send me to a tutorial that would show me how to do this?




Aucun commentaire:

Enregistrer un commentaire