mercredi 19 décembre 2018

Python web scraping - Why always getting hope page source code?

I have done all initial steps to login, able to access the website, I'm trying to simulate one search operation using a POST request (actually done using submit action form on the web portal), the operation is successful but the page source which I got is from "Home" page.

I can understand if the 100% page source code is not displayed in page source (assuming data generated using JavaScript), but here response page is static "Home" page.

What is redirecting to the home page? How to stop that navigation?

Here are the code steps to achieve this:

import requests

session = requests.Session()
session.get(access_config['login-url'])
session.post(access_config["auth-url"], data=auth_config)
res = session.post(link)
print(res.status_code)
print(res.history)
print(res.url)
print(res.text)

response:

status code :: 200

history :: <html><head><title>302 Moved Temporarily</title></head>
<body bgcolor="#FFFFFF">
<p>This document you requested has moved 
temporarily.</p>
<p>It's now at <a href="https://www.xxxx.com/a/welcome.html">https://www.xxxx.com/a/welcome.html</a>.</p>
</body></html>

response URL :: https://www.xxxx.com/a/welcome.html

I'm using requests library, I know lot of people using selenium library for these operations, is there anything which I could do this thing using requests or other than selenium?




Aucun commentaire:

Enregistrer un commentaire