Access-Control-Allow-Origin is not matched when server responds 304

I have an API server api.example.com and two websites a.example.com, b.example.com. To enable CORS, I set up CORS headers for them.

An XHR request from a.example.com will get the response headers from api.example.com

access-control-allow-credentials: true
access-control-allow-methods: GET, HEAD, POST, PUT, PATCH, DELETE, OPTIONS
access-control-allow-origin: https://a.example.com
access-control-max-age: 0
vary: origin

Similarly, an XHR request from b.example.com will get the response headers from api.example.com

access-control-allow-credentials: true
access-control-allow-methods: GET, HEAD, POST, PUT, PATCH, DELETE, OPTIONS
access-control-allow-origin: https://b.example.com
access-control-max-age: 0
vary: origin

When I fetches resource GET http://ift.tt/1UamNlH in a.example.com and then go to b.example.com do the same action to fetch the resource. The browser will use if-none-match headers to get the resource and receives a 304 response code from server. Then it complains that Access-Control-Allow-Origin header has a value https://a.example.com that is not equal to the supplied origin. Origin https://b.example.com is therefore not allowed access.

I think browser uses its cache when 304 is responded and find out Access-Control-Allow-Origin is cached as a.example.com's resource therefore refuse to fulfill the request.

How can I mitigate the issue here? Any thought?