I write pyton3 web crawler but it clogs tor connections. i get similar tor errors
[warn] Guard ohrelay (xxxxx) is failing a very large amount of circuits. Most likely this means the Tor network is overloaded, but it could also mean an attack against you or potentially the guard itself. Success counts are 79/262. Use counts are 0/0. 256 circuits completed, 0 were unusable, 177 collapsed, and 255 timed out. For reference, your timeout cutoff is 60 seconds.
[notice] We'd like to launch a circuit to handle a connection, but we already have 32 general-purpose client circuits pending. Waiting until some finish. [84886 similar message(s) suppressed in last 600 seconds]
I make requests with: (I use Therad also, with 10 concurrent jobs)
socks5h = "socks5h://127.0.0.1:9050"
http_proxy = socks5h
https_proxy = socks5h
ftp_proxy = socks5h
torproxy = {
"http" : http_proxy,
"https" : https_proxy,
"ftp" : ftp_proxy
}
send_headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; rv:78.0) Gecko/20100101 Firefox/78.0',
'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language':'en-US,en;q=0.5',
'Accept-Encoding':'gzip, deflate',
'Connection':'keep-alive',
'Upgrade-Insecure-Requests':'1',
'Cache-Control':'max-age=0'
}
ret = requests.get(url, proxies=torproxy, headers=send_headers, verify=False, timeout=(10,10))
Maybe i need destroy ret objets, but how?
Aucun commentaire:
Enregistrer un commentaire