Initial loading time of wordpress website takes too long (20+ sec)

I'm having troubles with the initial load time of my wordpress website. When I first visit the url (http://thofbrouwerijke.be/new/wordpress/) the initial load takes 20 seconds or more. However, all the subsequent requests are within a reasonable time frame (2 seconds or less).

The problem is not due to the website being cached on my machine as I tried cleaning the cache, using a different network and even different machines; the subsequent requests keep being fast.

However, it is only after a considerable amount of time (not quite sure how long) that the initial loading time again takes around 20 seconds.

Does somebody have a clue on how this happens? Trying to turn off the different plugins is hard to test as I have to wait around an hour before the initial slow loading time occurs again... Thanks in advance!

Disclaimer: I realise this questions has been asked a bunch but none of the answers I encountered so far describe a solution to my problem.