I'm trying to extract article content behind a paywall on www.epw.in (> 100 articles, so I need a script to automate this). I have a valid username and password, but there is an arithmetic captcha.
I can extract the captcha and get the answer, but I'm unsure as to how to go about using this answer in a single request.
This is my attempt to try and get it from a single article behind the paywall, but it doesn't get through because when I print content I can tell I didn't make it past.
from bs4 import BeautifulSoup
import requests
from lxml import html
from goose import Goose
from requests import get
g = Goose()
session_requests = requests.session()
login_url = "http://ift.tt/2pKQ1mW"
r = session_requests.get(login_url)
# Get CAPTCHA KEY
data = r.text
soup = BeautifulSoup(data, "html.parser")
captcha_extract = soup.find('span', {'class':'field-prefix'})
captcha_extract_txt = captcha_extract.text
nums = [int(s) for s in captcha_extract_txt.split() if s.isdigit()]
key = sum(nums)
payload = {'name': 'USERNAME',
'pass': 'PASSWORD',
'captcha_response': str(key)
}
r = s.post(login_url, data=payload)
article = g.extract(raw_html=r.content)
content = article.cleaned_text
print(content)
Aucun commentaire:
Enregistrer un commentaire