mercredi 20 novembre 2019

Python web-scraping of Expedia, how to find right keyword

I am learning about Python scraper. I take an simple exercise about find the cheap ticket in Expedia. Now, I meet some problems about how to find the right selector or accurate keyword. I use functions like select() and find(). I took too many tests about them but I still did it successfully. I always get empty list. How can I find the right selector or keywork in a better method? There is a part of my code. In it, I try to find the location of the input of Place: Flying from and the button Roundtrip.

    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36'}
    url="https://www.expedia.com/Flights?langid=1033&semcid=US.MULTILOBF.GOOGLE.GT-c-EN.FLIGHT&semdtl=a1355852835.b125535175035.r1.g1kwd-12197061.i1.d1280328929841.e1c.j120181.k1.f11t1.n1.l1g.h1e.m1&gclid=CjwKCAiAws7uBRAkEiwAMlbZjjBMg2bBBbp59C6tXeHf-4sXVvc4ya7EflIKQGsaFgENRP_SbaNQrRoCsUoQAvD_BwE"  
    address_page1 = requests.get(url, headers=headers).content
    soup = BeautifulSoup(address_page1,'html.parser')

    find = soup.find_all(id='flight-origin-flp-airport_code')
    print(find)

    select1 = soup.select('#flight-origin-flp-airport_code')
    print(select1)

    select2 = soup.select('#gcw-flights-form-flp > div.cols-nested.ab25184-location > div > div > div.input-btn-group')
    print(select2)



Aucun commentaire:

Enregistrer un commentaire