samedi 4 avril 2020

How to get dynamic data wia scrapy?

Hi Everyone Im new in scrapy and Ive got some problems with parsing one of websites I need to recieve data from this page: https://technopoint.ru/catalog/recipe/e351231ca6161134/2020-goda/

I need model name, price and ink to picture. I found every title except price and I`m almost give up on it. I found another link 'https://technopoint.ru/ajax-state/price/' but I cant understand how to scrap price data from it, help please!

p.s English is not my native, so there could be some awful mistakes, sorry!

from scrapy.utils import curl

import json
from scrapy.http import FormRequest, Request
from scrapy.selector import Selector
from dateutil import parser
import re
import pandas as pd

class test(scrapy.Spider):
    name = 'prices'
    start_urls = ['https://technopoint.ru/catalog/recipe/e351231ca6161134/2020-goda/']
    price_url = 'https://technopoint.ru/ajax-state/price/'

    def parse(self, response):
        price_url = 'https://technopoint.ru/ajax-state/price/'
        params = {'type': 'price', 'containers': []}
        s = Selector(text=response.body)
        items = s.xpath('//div[@class="catalog-item"]').getall()
        f = open('p.html', 'w')
        for item in items[:10]:
            print(item, sep='\n', file=f)
        price_url = 'https://technopoint.ru/ajax-state/price/'
        req = scrapy.Request(price_url,
                             method='POST',
                             headers={'X-Requested-With': 'XMLHttpRequest',
                                      'Content-Type': 'application/x-www-form-urlencoded',
                                    },
                             callback=self.parse)
        print(response, file=open('output.html', 'w'))
        yield req```



Aucun commentaire:

Enregistrer un commentaire