mercredi 8 mai 2019

How to overcome ItemsNotFoundError in mechanize when working with hidden forms

I am creating a script that submits a form and will potentially bring me some data back. The problem I am facing is that some options are hidden and do not show options (in HTML source code), until other fields are filled.

The error I am getting is ItemNotFoundError: insufficient items with name "xxx".

I have tried changing the fields at the "correct order" but as expected, this does not change the outcome.

My code looks something like this:

import mechanize

br = mechanize.Browser()
br.set_handle_robots(False) #ignore robots.txt
br.addheaders = [('User-agent', 'Mozilla/5.0 (Windows NT 5.2; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.47 Safari/536.11')]

bot = br.open("www.website.com")
br.select_form(nr=0)

br['type'] = ["xxx",]
br['specific'] = ["yyy",]

The problem, as stated before is that "type" and "specific" have a one to many relationship and both are drop down select fields. When you pick a "type", different options for "specific" appear.

When you first view page source for the website, the only option for "specific" is -1. If you do choose something for "type", multiple options will appear for "specific" but mechanize can not see those. On the contrary, I can view them in page source at that point.

When I am submitting the form through br.submit(), I get ItemsNotFoundError since it can not see these values. is there a way to get past this?

I've been trying to solve this for quite a few hours now so I would really be grateful if someone could help me out.




Aucun commentaire:

Enregistrer un commentaire