lundi 19 décembre 2016

How to obtain, from the commandline, information from a remote webpage that makes use of javascript to display information on that website?

I need some help in regards to obtaining information from a (remote) website.

As an example, please consider this university page here:

http://ift.tt/2hMboAB

It is in german but don't mind that; the, for me, relevant part is "Prüfungstermine" which could be translated as "Upcoming exams".

On my browser (firefox) I see that the next upcoming exam is marked in red colour. You can probably see that as well. It is in January 2017, the 17th.

I am trying to obtain this information from ruby, from the commandline. Normally this works fine, using the open-uri module of ruby.

The problem is that this information on that website, appears to not be part of the page itself - not via "view source", anyway - but instead, assumingly, is shown by some javascript-triggered display.

So I am not quite sure what options I have to programmatically access this, e. g. from the commandline.

I read through the source code of the different pages that are sent and found a snippet or two that may be of relevance:

(1) First, all the next exams will be shown via a tag called template such as:

<template id="examlinelink">
  <a href="exam.html?prueid=" class="link">
    <span class="mark"><abbr title='nächster Prüfungstermin'>N</abbr></span>
    <span class="day"></span>
    <span class="date"></span>
    <span class="time"></span>
    <span class="room"></span>
  </a>
</template>

This is obviously a href hyperlink. The curious part is that it shows as href '="exam.html?prueid="' but in my browser this is shown as 'http://ift.tt/2hR22AF' so I again assume that javascript modifies the href value - otherwise I can not explain why it comes up with that longer link instead.

Does anyone have any idea or a suggestion what could be done here?

When I am able to programmatically find out the next exams then I can obtain this easily via a regex; but I have no real idea how to have remote pages' javascript also evaluated, from the commandline, so I am a bit at a loss.

Thanks.




Aucun commentaire:

Enregistrer un commentaire