Cache slowing down site speed in WordPress

I have just optimized a website and its Score is above 90+ for page speed and YSlow result is above 80+. But once I clear cache from my Optimization plugin through Hummingbird Pro & Autoptimize then it loads the site quickly with 3.5 Sec but after a couple of minutes, Maybe around 10 minutes, Speed automatically decreases and if I clear caches from both these plugins and then speed once again improves.

Please check attached snapshots of results and help me out how I can fix it. this screenshot shows the same score in all tests but loading time dramatically changes if I don't clear caches before test.