WebRTC VIdeo Streaming

I'm streaming a video file from a peer to multiple peers. What I'm trying to do is set the duration of the video as well as where the streaming peer is in the video (seek time) for all peers.

I was able to set the correct duration using the Plyr video player, but whenever I'd like to seek into the video for a peer, it just resets to zero at the moment when srcObject is set to the streaming peer's MediaStream, after that seeking is not possible at all, it just throws me back to the current position (time elapsed since start).

Of course, I've tried to debug the problem. I went into Plyr's source, what I found out is that, obviously, Plyr uses the HTML5 media. So, I logged what the video tag looks like when playing and found out that the duration is Infinity. I think that this might be the problem, of course, it's just a guess. It only sets itself if there's an active source as that property is read-only.

How can I set the duration as well as seek in time WITHOUT a source?

I'd love to hear any solution/workaround for this. Thank you.