"Request to Service Worker" time is significantly high than the time to fetch from browser cache

I am using a simple service worker to cache a few js, css and img files resources. What I observe is the call (from Service Worker) is much longer than the call (from disk cache) i.e. the response from the browser cache is much faster.

This is true in case of pre-fetch and also when I am caching images based on the stale-while-revalidate policy.

Is service worker not ideal for improving page load time? I do not intend to make the website offline, I am using service-worker just for the sake of performance.