I have been trying to scrape the website https://fbschedules.com/new-england-patriots-schedule/
This website uses a hidden form to submit an ajax request to the php file: https://fbschedules.com/wp-admin/admin-ajax.php
After attempting to simulate the AJAX request, scrapy returns a 400 response for this code:
def parse(self, response):
headers = {
'User_Agent': user_agent,
'Accept': 'application/json, text/javascript, */*; q=0.01',
'Accept-Language': 'en-US,en;q=0.5',
'Accept-Encoding': 'gzip, deflate, br',
'Referer': 'https://fbschedules.com/new-england-patriots-schedule/',
'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
'X-Requested-With': 'XMLHttpRequest',
'Cookie': cookie,
'DNT': '1',
'Connection': 'keep-alive',
'Cache-Control': 'max-age=0'
}
data = {
'action': 'load_fbschedules_ajax',
'type': 'NFL',
'display': 'Season',
'team': 'New+England+Patriots',
'current_season': '2018',
'view': '',
'conference': '',
'conference-division': '',
'ncaa-subdivision': '',
'ispreseason': '',
'schedule-week': '',
}
yield scrapy.FormRequest.from_response('https://fbschedules.com/wp-admin/admin-ajax.php',
headers=headers,
formdata=data,
method='POST',
callback=self.schedule_parse)
Any help in the right direction is appreciated!
Aucun commentaire:
Enregistrer un commentaire