samedi 27 janvier 2018

Python web application works locally but crashes on heroku

I have a web application which calls another web service, convert2mp3 to be specific, I use python requests module to get the html response, however, the html which I obtain differ when I run it locally and on heroku.

I build the url the following way:

base_url = 'http://convert2mp3.net/en/index.php'
payload = {
    'p': 'call',
    'url': youtube_link,
    'format': 'mp3'
}

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:56.0)     Gecko/20100101 Firefox/56.0'
}

response = requests.get(base_url, params=payload, headers=headers)

Then I search for an iframe with a particular id in the html response obtained, using beautiful soup. Running this locally works as intended, but on heroku the html response obtained does not contain the iframe.

I also compared the response headers obtained

Local

{
  'Date': 'Sat, 27 Jan 2018 13:18:36 GMT',
  'Content-Type': 'text/html; charset=UTF-8',
  'Transfer-Encoding': 'chunked',
  'Connection': 'keep-alive',
  'Last-Modified': 'Sat, 27 Jan 2018 13:18:35 GMT',
  'Set-Cookie': '__cfduid=d5165dcd0d27b292871b83ec3fe1e2c911517059115; expires=Sun, 27-Jan-19 13:18:35 GMT; path=/; domain=.convert2mp3.net; HttpOnly, WSID=14390776415a6c7c2b7ee98vmlgzp6yziAAEgZBkcm0ekzF14jra3463970b7614cb8e2ba0abd1eae870598; expires=Sat, 28-Apr-2018 19:18:35 GMT; Max-Age=7884000; path=/; domain=convert2mp3.net',
  'Cache-Control': 'no-cache, no-store, must-revalidate, max-age=0',
  'Pragma': 'no-cache',
  'Vary': 'Accept-Encoding, User-Agent',
  'Content-Encoding': 'gzip',
  'Server': 'cloudflare',
  'CF-RAY': '3e3bffaea3c22df1-BOM',
  'Expires': 'Thu, 01 Jan 1970 00:00:00 GMT'
}

Heroku

{
 'Date': 'Sat, 27 Jan 2018 13:08:28 GMT',
 'Content-Type': 'text/html; charset=UTF-8',
 'Transfer-Encoding': 'chunked',
 'Connection': 'keep-alive',
 'Last-Modified': 'Sat, 27 Jan 2018 13:08:27 GMT',
 'Set-Cookie': '__cfduid=d00205adb7be018693ab849649dab79cd1517058507; expires=Sun, 27-Jan-19 13:08:27 GMT; path=/; domain=.convert2mp3.net; HttpOnly, WSID=11414627405a6c79cbb6a99An8ctaOAsolaQuv30wiReUyhYCLTIU84ca82bf822fd1f79bbe5e90f7bc3135; expires=Sat, 28-Apr-2018 19:08:27 GMT; Max-Age=7884000; path=/; domain=convert2mp3.net',
 'Cache-Control': 'no-cache, no-store, must-revalidate, max-age=0',
 'Pragma': 'no-cache',
 'Vary': 'Accept-Encoding, User-Agent',
 'Content-Encoding': 'gzip',
 'Server': 'cloudflare',
 'CF-RAY': '3e3bf0d89641239c-IAD',
 'Expires': 'Thu, 01 Jan 1970 00:00:00 GMT'
 }

I noticed that CF-RAY differs in the response headers, running locally gives xxxBOM and on heroku gives xxxIAD, but I don't understand what they mean.

I understand that this is website specific, but would greatly appreciate if someone could help me understand why this happens.




Aucun commentaire:

Enregistrer un commentaire