lundi 30 mars 2015

Web Scraper cause socket timeout on server

On our application we have a web scrapper written in java which uses URLConnection to fetch html content from sites. The applications crawls about 20 sites (parallel)- between 1 and 100 url requests per site, theses requests are generated each second and awaiting to be regenerated when a result is returned from the site. (all of the requests are managed by thread pool limited to not overkill the server). The problem that we are facing with is that after a certain amount of time our server just goes bananas..the sites start to give SocketTimeoutException one by one until all of the sources end up with SocketTimeoutException. We are using DSL-6740U router and even after switching to a different router of the same model the problem still occurs. (ping seems to be fine and the network and sited are working fine on the browser)


Is there anything we can do to eliminate the problem ? maybe configure the router etc.


(We are using windows xp , i7 with 8gb ram and 100mb network speed)


Any help would be appreciated Thanks in advance





Aucun commentaire:

Enregistrer un commentaire