jeudi 27 octobre 2016

Selenium/Python: Javascript throws error on website, if special characters in url (...I suspect)

I have some pretty weird behaviour in one of my scripts, and I hope someone here can help me. What am I trying to do (cannot really provide code snippets, maybe tomorrow): I try to scrape a website with a python (2.7) script and selenium via chrome (everything besides Chrome and Python itself just installed; why Selenium: Some ugly javascript, and I need to log in there).

  • If I open the website in my browser (Firefox), everything loads fine, including the javascript (a table).
  • If I open the website via Selenium browser.get(url) (Chrome), the javascript on the website fails, the table doesn't get loaded
  • If I copy the url (which I opened normally in the browser, and which fails via Selenium) from the Selenium instance of the browser into a new browser window (from Selenium/Chrome into Firefox), it also fails.

Via Firebug I see that the website now suddenly throws a "SyntaxError: missing ; before statement" in the javascript.

I suspect this is because the URL has parts which have some special characters like ? and & in it (e.g. aspx?replaceCurrent=1&url etc), and the & is in html & amp; and that the ; in there throws off the Javascript. But I cannot really confirm that. I've already tried to convert the URL via urllib.urlencode , to convert the special characters into this format 5E%26%2A (...sorry, web stuff is not my strong side), but then the browser just fails to open the site (because the : is replaced, therefore it doesn't recognize it as a valid URL).

...so...um...what am I potentially doing wrong here/what could be going wrong here? I really don't have a clue right now what I could try. Any pointers would be appreciated :).




Aucun commentaire:

Enregistrer un commentaire