jeudi 31 octobre 2019

advanced web scraping for homework (Python) investing

I am doing a homework where I want to exctract information from trading a website. In this case investing.com I want to build a live plot about an specific FX (example "AUD-CAD"). And for do that, I need to extract the values of candles. (for example every 5 secs)

https://m.investing.com/currencies/aud-cad

here is the image

left = candles right = values of every candles

to be able to see every value is necesary to do click on "time lapse", in this case 5 minutes. and I can see in information from page a new xhr type. The problem is the url from this new file. I can not enter to this URL, then this information not apear if I write ctrl+u (see HTML code).

I would like to know If there are a way for be able to extract this values automatically using web scraping.

more information about this response.

General: Request URL: https://m.investing.com/instrument/services/getChart?volume_series=yes&events=no&candle_count=40&pair_interval=300&chart_type=candlestick&pair_id_for_news=0&pair_id=47&symbol=AUD%252FCAD&showECEvents=false Request Method: GET Status Code: 200 OK Remote Address: 142.54.22.221:443 Referrer Policy: unsafe-url

Response Header: Accept-Ranges: bytes Age: 0 Cache-Control: no-store, no-cache, must-revalidate Connection: keep-alive Content-Encoding: gzip Content-Length: 840 Content-Security-Policy: upgrade-insecure-requests; block-all-mixed-content Content-Type: application/json; charset=UTF-8 Date: Thu, 31 Oct 2019 20:24:03 GMT Expires: Thu, 19 Nov 1981 08:52:00 GMT Pragma: no-cache Server: nginx/1.16.1 Set-Cookie: ses_id=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; domain=m.investing.com Vary: Accept-Encoding,User-Agent Via: 1.1 varnish X-Powered-By: PHP/7.1.8 X-Varnish: 1807797307

Request Headers: Accept: application/json, text/javascript, /; q=0.01 Accept-Encoding: gzip, deflate, br Accept-Language: es-ES,es;q=0.9,en;q=0.8 Connection: keep-alive Cookie: adBlockerNewUserDomains=1566366442; _ga=GA1.2.490180893.1566366443; __qca=P0-2008130179-1566366444101; _hjid=e4e61cb6-dac9-468b-90e7-881beebd50a4; _fbp=fb.1.1571268764166.678986027; __gads=ID=1f0f77699f259394:T=1571268765:S=ALNI_MauXf9rYg7Z-VvUMeshYUfPzibvgA; OB-USER-TOKEN=964583a6-3407-4116-ae8c-5c9198205262; _gid=GA1.2.2103139025.1572506355; PHPSESSID=r1ah8kjp16nvu5u2qi8bdoq2sc; StickySession=id.46489073211.560_m.investing.com_StickyMobile; searchedResultsMobile=[{"href":"/currencies/aud-cad","pairId":47,"title":"AUD/CAD","subtitle":"Australian Dollar Canadian Dollar","footerTitle":"Currency"}]; geoC=PE; fpros_popup_mob=1572555384; _gat=1; _gat_allMobileSitesTracker=1; _gat_allSitesTracker=1; nyxDorf=MjY0bmI0N3U2YmhnZzY4JGIyMGg2LzMwYmNlYg%3D%3D Host: m.investing.com Referer: https://m.investing.com/currencies/aud-cad Sec-Fetch-Mode: cors Sec-Fetch-Site: same-origin User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36 X-Requested-With: XMLHttpRequest

Query String parameters: volume_series: yes events: no candle_count: 40 pair_interval: 300 chart_type: candlestick pair_id_for_news: 0 pair_id: 47 symbol: AUD%2FCAD showECEvents: false




Aucun commentaire:

Enregistrer un commentaire