How to prevent a site from allowing its users to provide their own templates

I am developing a web app and in the Django documentation I read this:

The template system isn’t safe against untrusted template authors. For example, a site shouldn’t allow its users to provide their own templates.

Given the fact that I won't be allowing the end-user to upload any code, I reckon that the above would only be possible if the user would somehow gain access to the server and upload his (malicious) templates or by an XSS attack.

Is there something else I am missing here and should be aware of? Are there any additional measures (except for securing my server and looking out for XSS attacks) I must take in order to prevent this from happening?