mercredi 22 septembre 2021

How to scrape a website containing millions of hidden data? [closed]

I want to scrape all the products' data from https://usa.ingrammicro.com

The end point address is https://usa.ingrammicro.com/Site/Search where 50 products are shown per page. While opening this link in my browser all the page loads and is visible, but I have to wait for the products to be loaded into the page.

I'm using python beautifulsoup as follow but that doesn't work and I don't get anything:

Import requests
from bs4 import BeautifulSoup

URL = "https://usa.ingrammicro.com/Site/Search#viewAll%3aTrue"
page = request.get(URL)
soup = BeautifulSoup(page.content, "html.parser"

#this is the html tag where each product and its elements exist
products = soup.find_all("div", class_ = "grid-column-js-favorite-lines-container show")

print(products)



Aucun commentaire:

Enregistrer un commentaire