Javascript WebSocket to Java server taking long time to connect every other one, ~6s

When I visit the website using the ip address, everything goes rapid fast.

When I use the domain to navigate, every other result takes long time for the websocket to complete.

Sometimes, it is instant, sometimes it takes 6 seconds.

I am on JEE 7 using websocket endpoint.

What could be wrong here? It's been like this for a long time.

Could my router be involved here? I am forwarding port 80 traffic.