Downloading/uploading a file/folder directly from/to a docker container running on a web host to/from a local machine using SCP

So far, I have always copied files from the docker container to my VM first (web host), and later, run scp command line from my local machine to download it from the VM. Similar scenario happening for uploading files/folders. Is there a way to direct way to do that using scp?