I am making a web scraper which scrapes data and website links using python googlesearch api.But after too many request it raise exception too many requests.To handle this rewrite the get_page method of this and try to use proxies.But unfortunately it does not work. Moreover i also use selenium web driver but after many requests google shows recaptche page. If any one knows how to handle this issue please guide or help me so i can complete this project.
@retry
def get_page(url, user_agent=None, verify_ssl=True):
import requests
proxy_index = random_proxy()
proxy = proxies[proxy_index]
proxy = {
'http': f'http://{proxy["ip"]}:{proxy["port"]}',
'https': f'http://{proxy["ip"]}:{proxy["port"]}',
}
try:
response=requests.get(url, proxies=proxy)
except Exception as exc:
print(str(exc))
del proxies[proxy_index]
raise
return response.content
It is throwing winerror 10060 and 10061 exception on every retry
Aucun commentaire:
Enregistrer un commentaire