jeudi 26 janvier 2017

BeautifulSoup cannot scrap certain page, instead scraps root page

I am having troubles scraping info from a specific page.

I want to collect data from the url http://ift.tt/2j90fvg but due to the way the website handles things BeautifulSoup is only collecting data from the root page, aka http://csgo-stats.com

Is there a redirect going on thats tripping up BS? I can see in the html that BS outputs that its trying to load my data but BS captures it too quickly

<main class="site-content" id="content">
        <div class="loading-spinner" data-request="epsilon-" id="load">
            Loading
        </div>

Here is the code Im working with just in case its needed:

from urllib.request import urlopen
from bs4 import BeautifulSoup

url = "http://ift.tt/2jW4yag"
soup = BeautifulSoup(urlopen(url))
print(soup.prettify())

Any advice? Thanks!




Aucun commentaire:

Enregistrer un commentaire