Keyboard accessebility - Add outline only for keyboard navigation but remove on mouse click

I'm implementing keyboard accessebility in my app, and encountered some issues. I added outline for focus-visible pseudoclass arround input, a, button, textarea and such keyboard accessible elements and removed outline and border for :focus pseudoclass and it works, but in some cases, for example when I mouse click on input on login form, outline is added arround it. Same behavior on other few places. How can I ensure outline border only while navigating via keyboard and for mouse click, keep initial stylings? And, is there something special for Safari? As keyboard accessebility doesn't work in it, but works in Firefox and Chrome?