samedi 21 mars 2020

Get Search Results Number Using BeautifulSoup

I'm trying to use BeautifulSoup in Python to get the total number of search results in CNN.

The source code on the webpage is

<div class="cnn-search__results-count">

"Displaying results 1-10 out of 2208 for"

<strong>toronto</strong>

</div>

as shown in screenshot 1:

Screenshot 1

The code I wrote is:

from bs4 import BeautifulSoup

import requests

url_cnn = 'https://www.cnn.com/search?q=toronto'

response_cnn = requests.get(url_cnn)

html_cnn = response_cnn.text

soup = BeautifulSoup(html_cnn, 'html.parser')

cnn = (soup.find('div', {"class": "cnn-search__results-count"}))

print(cnn)

However, I only get

<div class="cnn-search__results-count"></div>

All the contents in between are missing.

Does anyone know how to solve this? Thanks so much!




Aucun commentaire:

Enregistrer un commentaire