mardi 28 mars 2017

Get invisible web page info with BeautifulSoup

I am trying to get some info from the site "http://ift.tt/2nIjqwH", to be more precise all individual estimates, which are at the bottom of the page. But it shows only first 30 and then you should manually press the button "Show All" to get another 30 and so on. Here is my code so far:

from urllib import urlopen
from bs4 import BeautifulSoup
html = urlopen("http://ift.tt/2nIjqwH")
soup = BeautifulSoup(html.read(), "html.parser")
print(soup)

I see that there is a part of the printed code:

"totalCount":142,"total_estimates_showing":30,"

Is it possible to change this to get printed all the estimates?




Aucun commentaire:

Enregistrer un commentaire