Fragmented mp4 cannot be played on pure html5 video

I need to use a video tag to serve over 3GB of video on the web. When the page is loaded, it takes a long time for the media element to receive the 'loadedmetadata event'.

I've found that the size of the moov box is too large (33MB). So when I re-encoded it with the 'empty_moov + frag_keyframe' option of 'ffmpeg', but it also took longer to fetch all fragmented information from the 'Inspector - Network' tab in Chrome.

Is there a way to speed up loading when playing 'fragmented mp4' with html5 video tag?