Writing my own Authentication Backend

I'm developing a web app using Angular2 and Scala. And instead of using any big frameworks like Spring security that does everything for you, I'm thinking using just some libraries and write the system myself.

I plan to use JWT libraries to authenticate requests. Use hashing libraries to encrypt password storage in db. What other things do I need to consider to make my system secure? I think I also need to think about CSRF and XSS? What else?

Thanks a lot!!!