CSS transform transition jumps back when dragging

I lately created an animation based on css transform (translate) and a transition combined with a bit javascript. The Animation is based on a mousemove event in javascript, where the transform property gets changed every time the mouse moves. The thing is, it works perfectly fine but if you click or more "try to drag" something with the mouse the animation starts to snap to the "drag-start" point and then again goes to it's supposed place.

The element that is moved with the transform property: (relevant attributes)

transition: transform 50ms ease;
-moz-transition: transform 50ms ease;

The javascript script for the animation:

const shadowElement = document.getElementById('loginPage-terminal-shadow');

var offsetX = document.body.clientWidth / 2,
    offsetY = document.body.clientHeight / 2;

var width = shadowElement.clientWidth;
var height = shadowElement.clientHeight;
var xInterval = (width * 1 / 10) / 2;
var yInterval = height * 1 / 10;

window.addEventListener("resize", function() {
    offsetX = document.body.clientWidth / 2;
    offsetY = document.body.clientHeight / 2;
    width = shadowElement.clientWidth;
    height = shadowElement.clientHeight;
    xInterval = (width * 1 / 10) / 2;
    yInterval = height * 1 / 10;
});

var prefix = (function() {
    var styles = window.getComputedStyle(document.documentElement, ''),
        pre = (Array.prototype.slice
            .call(styles)
            .join('')
            .match(/-(moz|webkit|ms)-/) || (styles.OLink === '' && ['', 'o'])
        )[1],
        dom = ('WebKit|Moz|MS|O').match(new RegExp('(' + pre + ')', 'i'))[1];
    return {
        dom: dom,
        lowercase: pre,
        css: '-' + pre + '-',
        js: pre[0].toUpperCase() + pre.substr(1)
    };
})();

document.body.addEventListener('mousemove', function() {
    var diffX = offsetX - event.clientX;
    var diffY = offsetY - event.clientY;

    var perX = diffX / offsetX;
    var perY = diffY / offsetY;

    var sftX = xInterval * perX;
    var sftY = yInterval * perY;

    shadowElement.style[prefix.css + "transform"] = "translate(" + sftX + "px, " + sftY + "px)";
});

As you might have seen, I already tried adding a vendor prefix to the transform and transition property).

The animation should completely ignore a click or drag event, considering that's the problem.

Additions:
It seems to be a firefox problem, it works completely fine on chrome.

Here's a video of the problem: https://www.dropbox.com/s/hcjp6evuvk6zm5m/problem.mkv?dl=0