mercredi 8 avril 2015

MySQL and lots of queries

I have a Java application, which is kind of a web scraper. It parses lots of, lets call it - rows. The point is, when scraping a web, check, if a row found on a certain page is new, or is it old (I do a query to MySQL database at this point to check if it exists in it). If it is new, I put it to a seperate array and do another query to add that line to database, so when doing a next scraping, that row could be recognized as old. The problem is, that web page has about 90 thousand of these rows, and performance is quite slow. Sometimes MySQL server even crashes.


I would like you hear your opinion, which of these options should I choose(or maybe something else):



  1. Choose other DBMS, like MongoDB or some other?

  2. Do not put rows to database, but use some caching solution? (what solution, then?)


Thank you very much!





Aucun commentaire:

Enregistrer un commentaire