lundi 28 janvier 2019

Scraping AJAX e-commerce site using python

I have a problem on scraping an e-commerce site using BeautifulSoup. I did some Googling but I still can't solve the problem.

Please refer on the pictures:

1 Chrome F12 : https://scontent.fkul7-1.fna.fbcdn.net/v/t1.0-9/50874003_2100628179980272_838265414453559296_n.jpg?_nc_cat=105&_nc_ht=scontent.fkul7-1.fna&oh=610394bf4ee663dc54b71b1084654666&oe=5CB32DB8

[2] Result : https://scontent.fkul7-1.fna.fbcdn.net/v/t1.0-9/50713161_2100628126646944_8588520010161324032_o.jpg?_nc_cat=105&_nc_ht=scontent.fkul7-1.fna&oh=e2caf3fae70e66690fdcadcfe3da3d0e&oe=5CBB6B16

Here is the site that I tried to scrape: "https://shopee.com.my/search?keyword=h370m"

Problem:

  1. When I tried to open up Inspect Element on Google Chrome (F12), I can see the for the product's name, price, etc. But when I run my python program, I could not get the same code and tag in the python result. After some googling, I found out that this website used AJAX query to get the data.

  2. Anyone can help me on the best methods to get these product's data by scraping an AJAX site? I would like to display the data in a table form.

My code: import requests from bs4 import BeautifulSoup source = requests.get('https://shopee.com.my/search?keyword=h370m') soup = BeautifulSoup(source.text, 'html.parser') print(soup)




Aucun commentaire:

Enregistrer un commentaire