mercredi 29 septembre 2021

Webscraping with Beautifulsoup: How to use next button if no href or nav-link?

Im trying to scrape the url and would like to get all apartments on all the pages possible. In this example, there are only two pages and I want to:

  1. "Click on the next button" / Go to page 2
  2. Go to the last page if there is no next-button

In the tutorials I've seen such as this one there is a href-link to the next button on the page he is scraping. In my case the HTML-code for the page-list does not contain any href link but looks like this:

enter image description here

In other tutorials, they could find the href link to the next-button by finding looping over the links in the webpage. When I do this, I get only the main links for the site (despite loading a nested url) and not find any next button.

nav = soup.nav

for url in nav.find_all('a'):
    print(url.get('href'))

Do you have any ideas on how to access the url of the next-button in this case?




Aucun commentaire:

Enregistrer un commentaire