jeudi 4 février 2021

Python Google Scraping With Proxy Rotation

I am making a web scraper which scrapes data and website links using python googlesearch api.But after too many request it raise exception too many requests.To handle this rewrite the get_page method of this and try to use proxies.But unfortunately it does not work. Moreover i also use selenium web driver but after many requests google shows recaptche page. If any one knows how to handle this issue please guide or help me so i can complete this project.

@retry
def get_page(url, user_agent=None, verify_ssl=True):
    import requests
    proxy_index = random_proxy()
    proxy = proxies[proxy_index]
    proxy = {
        'http': f'http://{proxy["ip"]}:{proxy["port"]}',
        'https': f'http://{proxy["ip"]}:{proxy["port"]}',
        }
    try:
        response=requests.get(url, proxies=proxy)
    except Exception as exc:
        print(str(exc))
        del proxies[proxy_index]
        raise
    return response.content

It is throwing winerror 10060 and 10061 exception on every retry




Aucun commentaire:

Enregistrer un commentaire