I have a web application which calls another web service, convert2mp3 to be specific, I use python requests
module to get the html response, however, the html which I obtain differ when I run it locally and on heroku.
I build the url the following way:
base_url = 'http://convert2mp3.net/en/index.php'
payload = {
'p': 'call',
'url': youtube_link,
'format': 'mp3'
}
headers = {
'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:56.0) Gecko/20100101 Firefox/56.0'
}
response = requests.get(base_url, params=payload, headers=headers)
Then I search for an iframe
with a particular id in the html response obtained, using beautiful soup. Running this locally works as intended, but on heroku the html response obtained does not contain the iframe
.
I also compared the response headers obtained
Local
{
'Date': 'Sat, 27 Jan 2018 13:18:36 GMT',
'Content-Type': 'text/html; charset=UTF-8',
'Transfer-Encoding': 'chunked',
'Connection': 'keep-alive',
'Last-Modified': 'Sat, 27 Jan 2018 13:18:35 GMT',
'Set-Cookie': '__cfduid=d5165dcd0d27b292871b83ec3fe1e2c911517059115; expires=Sun, 27-Jan-19 13:18:35 GMT; path=/; domain=.convert2mp3.net; HttpOnly, WSID=14390776415a6c7c2b7ee98vmlgzp6yziAAEgZBkcm0ekzF14jra3463970b7614cb8e2ba0abd1eae870598; expires=Sat, 28-Apr-2018 19:18:35 GMT; Max-Age=7884000; path=/; domain=convert2mp3.net',
'Cache-Control': 'no-cache, no-store, must-revalidate, max-age=0',
'Pragma': 'no-cache',
'Vary': 'Accept-Encoding, User-Agent',
'Content-Encoding': 'gzip',
'Server': 'cloudflare',
'CF-RAY': '3e3bffaea3c22df1-BOM',
'Expires': 'Thu, 01 Jan 1970 00:00:00 GMT'
}
Heroku
{
'Date': 'Sat, 27 Jan 2018 13:08:28 GMT',
'Content-Type': 'text/html; charset=UTF-8',
'Transfer-Encoding': 'chunked',
'Connection': 'keep-alive',
'Last-Modified': 'Sat, 27 Jan 2018 13:08:27 GMT',
'Set-Cookie': '__cfduid=d00205adb7be018693ab849649dab79cd1517058507; expires=Sun, 27-Jan-19 13:08:27 GMT; path=/; domain=.convert2mp3.net; HttpOnly, WSID=11414627405a6c79cbb6a99An8ctaOAsolaQuv30wiReUyhYCLTIU84ca82bf822fd1f79bbe5e90f7bc3135; expires=Sat, 28-Apr-2018 19:08:27 GMT; Max-Age=7884000; path=/; domain=convert2mp3.net',
'Cache-Control': 'no-cache, no-store, must-revalidate, max-age=0',
'Pragma': 'no-cache',
'Vary': 'Accept-Encoding, User-Agent',
'Content-Encoding': 'gzip',
'Server': 'cloudflare',
'CF-RAY': '3e3bf0d89641239c-IAD',
'Expires': 'Thu, 01 Jan 1970 00:00:00 GMT'
}
I noticed that CF-RAY differs in the response headers, running locally gives xxxBOM
and on heroku gives xxxIAD
, but I don't understand what they mean.
I understand that this is website specific, but would greatly appreciate if someone could help me understand why this happens.
Aucun commentaire:
Enregistrer un commentaire