mardi 25 avril 2017

How to download zip files from authenticated website by python

Without requiring credential, I tested out how to use urllib2 to grab files from the web server. But I am facing an authenticated website which even requires you to click an 'agree' button to their terms and condition after you provide the login credential. I have searched around in Stackoverflow and found some relevant posts, but seems none of them mentioned this type of authentication.

Further, the url for login is something like 'http://ift.tt/2q437vZ', but the zip file download page after you provide the credential and also click the 'agree' button is different than the login page, something like "http://ift.tt/2oHm3f6". This makes me confusing which url shall I use in the code.

I tested using the following code (from Download a file from https with authentication):

import urllib2

username = 'user1'
password = '123456'

baseurl = 'http://ift.tt/2oHl2Ux'
dataurl = 'http://ift.tt/2q48n2z'

#Create a password manager
manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
manager.add_password(None, baseurl, username, password)

#Create an authentication handler using the password manager
auth = urllib2.HTTPBasicAuthHandler(manager)

#Create an opener that will replace the default urlopen method on further calls
opener = urllib2.build_opener(auth)
urllib2.install_opener(opener)

#Here you should access the full url you wanted to open
response = urllib2.urlopen(dataurl)

after running the code, it did not give me error, but I did not get the file neither.

Any comments/suggestions? thanks a lot!




Aucun commentaire:

Enregistrer un commentaire