jeudi 30 mars 2017

Best way to handle long-running tasks on a web server

For the past few days I've been googling about how should I handle long running tasks on a web server. I have found a lot of good answers of how to run them in a specific language, but not what languages to choose for this kind of job.

So, I have a web server which is running some kind of custom e-commerce platform. There is another server where some products dostributor is providing access to its data through API. I have to sync products list across these two servers. Product base is pretty big (about 100000 products).

My Idea was to write php script which collects data from various API endpoints and updates database accordingly. But it gonna take a long time, so without hard modofications and deep-diving to php itself it will timeout.

Now I'm thinking, maybe I should write python script which goes through API endpoints and collects data about each product. After data about product is collected python script could initiate php script which could update data in database about that particular product..

What are your toughts about it? What would be the best way to handle it?




Aucun commentaire:

Enregistrer un commentaire