mardi 1 décembre 2020

multi threading with python

I have a code, which outputs the URLs in a particular web page. I want to run the same code to get URLs within different web pages. I want to use multi-threading to update the links each time the code is run. In the following code, I want to add "https://ift.tt/3qdHTrT" and "https://thewire.in/" in place of the present URL against "resp" in the code below

Following is the code, {I want to make changes in the "resp = requests.get("https://ift.tt/2HWlbTC) " in subsequent implementation of the threads} :

from bs4 import BeautifulSoup
from bs4.dammit import EncodingDetector
import requests

parser = 'html.parser'  # or 'lxml' (preferred) or 'html5lib', if installed
resp = requests.get("https://www.ndtv.com/coronavirus?pfrom=home-mainnavgation")
http_encoding = resp.encoding if 'charset' in resp.headers.get('content-type', '').lower() else None
html_encoding = EncodingDetector.find_declared_encoding(resp.content, is_html=True)
encoding = html_encoding or http_encoding
soup = BeautifulSoup(resp.content, parser, from_encoding=encoding)

for link in soup.find_all('a', href=True):
    print(link['href'])



Aucun commentaire:

Enregistrer un commentaire