Below is my code that scrap the different pages from a given website
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
'http://quotes.toscrape.com/',
]
def parse(self, response):
for post in response.css('div.quote'):
desc = post.css('span.text::text')[0].get()
author = post.css('small.author::text')[0].get()
yield{
'desc':desc,
'author':author
}
next_page = response.css('li.next a::attr(href)').get()
if next_page is not None:
next_page = response.urljoin(next_page)
yield scrapy.Request(next_page,callback=self.parse)
lets consider now I have to scrap from the 100+ website.
how can I do that? and if possible ,whether i need to know html structure of the all 100+ websites to scrap data ?
I am using this crawler in my project that collects posts from the websites and I am displaying those posts in single page of the my project...
Aucun commentaire:
Enregistrer un commentaire