lundi 19 juillet 2021

err empty response while crawling a webpage

I am creating a web-crawler using selenium that is supposed to search for companies in a website, but while it is crawling i get an err empty response after being redirected to the page where the search results are. Is it possible that the website is somehow detecting that i am a bot and is blocking me and how can i fix this? Here are the options that i have set

options = webdriver.ChromeOptions()
options.add_argument("--ignore-certificate-errors")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
options.add_argument('--disable-blink-features=AutomationControlled')
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_argument("--disable-extensions")
options.add_argument(f'user-agent={self.user_agent}')
USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:86.0) Gecko/20100101 Firefox/86.0'



Aucun commentaire:

Enregistrer un commentaire