Can I use a "one fits all" polyfill collection without breaking new browsers?

I am using OpenLayers in a project. To support users with older browsers, one must use Polyfills for requestAnimationFrame, Element.prototype.classList, and URL.

OpenLayers itself suggests to use the Polyfill.io service. This is shown in one of their examples.

This would require me to load an external URL (http://ift.tt/2hgZZEQ) into the user's browsers. The service looks at the browser's user-agent and gives it a Polyfill if the browser is old, otherwise this file is just empty. Of course, running code hosted elsewhere is a security thread I do not want to be responsible for, so I would prefer to self-host.

The Polyfill.io service is available for running locally, but I'd have to run a server for that. I would prefer to not do this.

Instead, I'd like to serve a static JS file to everyone. My concerns are:

1. Is this possible? Can I serve the same polyfill to every old browser or do different old browsers need differnet polyfills?
2. Will this break the new browsers? How can I avoid this?
3. Will this be a huge file?

I appreciate your help very much.