mardi 15 décembre 2020

No result after web scrapping

My Dataframe is empty after i run the follow the code. May i know which part is error. I am using the while loop to scrap the next page.

npo_prop = {}
prop_no=0

while True:
    response = requests.get(url)
    data = response.text
    soup = BeautifulSoup(data, 'html.parser')
    titles = soup.find_all('div',{"class":"property__half"})
    
    for title in titles:
        house = title.find('div',{"class":"property__body property__meta"}).text
        prop_no+=1
        npo_prop[prop_no] = [house]
        print('houses:', house)
        
    url_tag = soup.find("a",{'class':'page-link'})
    if url_tag is not None:
        url_tag.get('href')
        url = 'https://www.starproperty.my/to-buy/search?max_price=1000000%2B&new_launch_checkbox=on&sub_sales_checkbox=on&auction_checkbox=on' + url_tag.get('href')
        print(url)
    else:
        break


npo_prop_df = pd.DataFrame.from_dict(npo_prop, orient = 'index', columns = ['houses'])
npo_prop_df.head()



Aucun commentaire:

Enregistrer un commentaire