i'm trying to implement web scraping so that will scrape around 5000 websites less than a minute. I know there is a CPU limitiation and I don't know how should I calculate that. i'm using nodejs, and promise.all, concurrency. what should be my concurrency for the best performance, how many CPU I need for X threads in parallel ? I guess my computer can't handle that to get that performance.. there is any way I can implement it ?
Aucun commentaire:
Enregistrer un commentaire