vendredi 21 mai 2021

how to scrape multiple websites using python scrapy? [closed]

Below is my code that scrap the different pages from a given website

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/',
    ]

    def parse(self, response):
        for post in response.css('div.quote'):
            desc = post.css('span.text::text')[0].get()
            author = post.css('small.author::text')[0].get()
            yield{
                'desc':desc,
                'author':author
            }           
        next_page = response.css('li.next a::attr(href)').get()
        if next_page is not None:
            next_page = response.urljoin(next_page)
            yield scrapy.Request(next_page,callback=self.parse)

lets consider now I have to scrap from the 100+ website.

how can I do that? and if possible ,whether i need to know html structure of the all 100+ websites to scrap data ?

I am using this crawler in my project that collects posts from the websites and I am displaying those posts in single page of the my project...




Aucun commentaire:

Enregistrer un commentaire