jeudi 20 mai 2021

Retrieving web page content that isn't in source code

MINI-PROJECT : I want to use tapology.com to make my own database of MMA fighters. My idea for a python program was : go to the page of a fighter -> put his record in the database and add the name of his opponents to the database -> go to the page of one of his opponents etc.

PROBLEM : While I can see the opponents' name of a fighter on his dedicated page, I can't find them in the source code. Example : This fighter, Francis N'Gannou, fought Stipe Miocic whose name appears at the bottom of Ngannou's page in the section "FRANCIS N'GANNOU MMA FIGHT RECORD" but 0 match for "Stipe" (or even "MMA FIGHT RECORD") in the source code.

I initially thought everything that I could see on the page appeared in the source code but I figured my problem has something to do with the fact that the data I want to use (e.g. opponents' name) can only be found in an interactive panel.

  1. Can my program still retrieve the data and put it in a database ?
  2. Do you know any resources that can help me understand how to do this and how those panels work ?

Thank you for your time.




Aucun commentaire:

Enregistrer un commentaire