Our servers are under great pressure when our web pages are scraped by many clients. Sometimes our web pages are being scraping from many different IP addresses. So our defensive strategy based on IP addresses is not useful. Caching may be an option. But we have so many urls for seo. For example, we have some urls which have the pattern "https://www.xxxx.com/hot-goods/mobile-phone-1.html". This page shows a list of products about mobile phone. There are thousands of pages for the search result of a single search word. So the hit rate of caching may be not very high. So I just wonder if there is any other solutions to reduce the pressure of our servers.
Aucun commentaire:
Enregistrer un commentaire