We’re facing issues.
Parallel processing of multiple sites web crawling each site with 10k + products and 100’s of sites 3/4 times a week to keep a database of products up to date. If possible we’d like to do this daily.
The problem is after 1000s product scans from sites, it breaks. After processing 1000s of data, system cron gets killed
We also want to be able to live scan website URL with 99 page max. In under 30 mins ideally.
Aucun commentaire:
Enregistrer un commentaire