lundi 9 mars 2020

Scrapy: how to get information from all tabs on the page?

On this page I need to get information from all tabs(Profile, Reviews, Phone Numbers & Directions).

wellness.py

def profile(self, response):
    services = response.xpath('.//span[contains(text(),"Services")]')
    education = response.xpath('.//span[contains(text(),"Education")]')
    training = response.xpath('.//span[contains(text(),"Training")]')

    yield {
            'First and Last name': response.css('h1::text').get(),
            'About': response.css('.listing-about::text').get(),
            'Services': services.xpath('following-sibling::span[1]/text()').extract(),
            'Primary Specialty': response.css('.normal::text').get(),
            'Address': ' '.join([i.strip() for i in response.css('.office-address span::text').getall()]),
            'Practice': response.css('.years-in-service::text').get(),
            'Education': education.xpath('following-sibling::span[1]/text()').extract(),
            'Training': training.xpath('following-sibling::span[1]/text()').extract(),
            'Consumer Feedback': response.css('.item-rating-container a::text').get()                
        }



Aucun commentaire:

Enregistrer un commentaire