lundi 19 septembre 2016

Make statistics from JSON Files located in a Unix Server - Java

My project is to build a web page (one single page for the moment) showing some stats about data in JSON Files. To be more precise and specific, at therm I would deal with around 100 000 JSON files at a time. Each file would be around 50 lines long but i will only chose 10 to 15 data per file (that would be chosen for the statistic purpose).

The process would be like this:

  1. Connect to the server
  2. Read the JSON files
  3. Make the stats
  4. Display them in a very basic web page

The JSON files are located in a Unix Server. I will also avoid (if possible) to dowload the files to make the stats due to the fact that new files could at any time be created.

For now, I have a Java code that connects to the server using the JSch library and then introduce a unix command to show the JSON files in a console (for now I have 15 files so easy to show them in Eclipse's console).

My main question is: do you have any tips that could help me to choose the best solution for this project? The way I need to proceed, for example how to read all the files on the Server and then make the stats.

I want to add that all the JSON files do have the same "variables" such as "print": false/true or "success": false/true.

The programming language is Java (for web) and I am quite novice in it (but anyway I want to learn!). I was thinking using a servlet due to the few html code I will use.

I hear from few frameworks such as Apache Drill or Apache Storm that can manipulate JSON data as they were SQL files (and proceed to SQL queries) but reading the doc I felt it wasn't what I was searching for plus the were not very easy to use, I was searching for libraries that I would add to eclise to ease my code.

PS: at the end the web page will show a very simple table of 15 data and their stats (more or less % or true and false, so very basic stuff).

PS2: Sorry for my bad english!

Thanks in advance for any help and sorry for the very long post.




Aucun commentaire:

Enregistrer un commentaire