samedi 20 juillet 2019

Working web crawler, suddenly not working anymore

I was following this tutorial and the code worked perfectly.

Now after doing some other projects I went back and wanted to re-run the same code. Suddenly I was getting an error message that forced me to add features="html.parser" in the soup variable.

So I did, but now when I run the code, literally nothing happens. Why is that, what am I doing wrong?

I checked whether I might have uninstalled beautifulsoup4 module, but no, it is still there. I re-typed the whole code from scratch, but nothing seems to work.

import requests
from bs4 import BeautifulSoup

def spider():
    url = "https://www.amazon.de/s?k=laptop+triton&__mk_de_DE=%C3%85M%C3%85%C5%BD%C3%95%C3%91&ref=nb_sb_noss"
    source = requests.get(url)
    plain_text = source.text
    soup = BeautifulSoup(plain_text, features="html.parser")

    for mylink in soup.findAll('img', {'class':'s-image'}):
        mysrc = mylink.get('src')
        print(mysrc)

spider()

Ideally I'd want the crawler to print about 10-20 lines of src = "..." of the amazon page in question. This code worked a couple hours ago...

Aucun commentaire:

Enregistrer un commentaire