Google Cloud Storage upload with signed URLs gives cors problems on actual request

I have a weird problem that I can't seem to figure out. I have a Google Cloud Storage bucket where I want to upload files to from a webpage. In my backend (App Engine) I have created signed URLs that can be used for uploading the files. When I use that signed URL in Postman with a PUT request and a file, the file is successful uploaded to my bucket.

When I do the same from a webpage, than it becomes tricky. I have created the following code in my webpage:

var req = new XMLHttpRequest();
req.open("PUT", resp.url);
req.setRequestHeader("Content-Type", parameters.contentType);
req.onload = function(event) {
    this.loadFileList();
}.bind(this);
req.send(this.$.fileInput.files[0]);

This gave me CORS problems on the preflighted request (OPTIONS). After some reading I figured out to set the CORS configuration on Google Cloud Storage. So I have set the following configuration:

[
  {
    "origin": ["https://my-website"],
    "responseHeader": ["Content-Type"],
    "method": ["GET", "HEAD", "PUT", "OPTIONS", "POST"],
    "maxAgeSeconds": 3600
  }
]

When I put this configuration onto Google Cloud Storage I see that the preflighted request starts to work, but then the actual request still fails with the following error:

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://storage.googleapis.com/[my-bucket].appspot.com/games/dev/game-[game]/game-pictures/[file].jpg?GoogleAccessId=[account]&Expires=[expires]&Signature=[signature]. (Reason: CORS header ‘Access-Control-Allow-Origin’ missing).

(I replaced some parts of the url, so that I'm not exposing anything about my bucket)

The actual error is true. Access-Control-Allow-Origin is missing in the response of the actual request, but it's there in the response of the preflighted request.

So, should I get the Access-Control-Allow-Origin in the response of the actual request? And why don't I get that? And how can I fix this?