samedi 4 juillet 2020

Beautiful Soup is not returning full HTML code that I see when I inspect the page manually

I'm trying to scrape a site, but beautiful soup isn't returning any HTML code that I see when inspecting manually. The return soup also includes a phrase, "You are using an outdated browser".

I have tried different parsers and also using the urllib module.

I have a strong feeling that it's due to the time it takes for the website to load because the code works for some other websites. Is there a way to stall beautiful soup to wait till the entire page ha loaded?

Here's my code:

import requests
from bs4 import BeautifulSoup

def get_stock():
    URL = 'https://www.cse.lk/home/tradeSummary'
    page = requests.get(URL)
    soup = BeautifulSoup(page.content, 'html.parser')
    print(soup.get_text())
    stocks = soup.find_all("td")
    for stock in stocks:
        line = stock.get_text()
        print(line)

get_stock()

Thank you:)




Aucun commentaire:

Enregistrer un commentaire