When should e use flexible web design and when not

CSS allows one to define many attributes related to the presentation of a web application, such as background color, font families used and so on.

But these same elements may as well not be defined at all and, then, the web browser chooses defaults for them, defaults which are flexible and easily changeable by each individual user.

This allows for a flexible approach as, then, any user can easily change the background color, text color, font family and even font size for the website and configure them manually to their liking.

As such, would it be considered a good design choice to allow the user to customize a website to their liking instead of forcing one onto them. For example, if a user is visually impaired, wouldn't it be preferable if he/she could change the color schemes to increase the contrast, maybe even increase the font size too?

I heard that there already are many people who believe using "em" units to define font sizes is better than "px" units, just because the user can then increase the size of the text as they wish.

Would this flexibility be considered user-friendly or too ambiguous as it obviously creates discrepancies between the look between different browsers? Is focusing on a concrete design to ensure a proper consistent look more important than flexibility?