How to prevent custom.css from interfering with lighthouse audits?

I am trying to optimize the core web vitals of my site, and the custom.css file is affecting the latency (or at least I think so), I understand that this is a file to customize the appearance of the sites, however I do not use it. Is there a way to remove it, block it, or do something so that it doesn't interfere with the lighthouse audits?