Prevent scaling in SVG's foreign object element

I'm using SVG to build axis' for a chart. I'd prefer to use regular html text and div's for parts of the axis' and so I use <foreignobject> for that. I use <g> and the transform attribute to apply scaling and translating to create the various "ticks" in the axis. The problem is that this transformation leaks into the text. Say I want all my tick names to have 1em for their font size and I set it that way in the foreign objects. Well, due to the transformations being different across different axis' the text appears to be differently sized despite all being 1em.

Here is a minimal reproduction of what I mean.

    <svg width="100" height="200">
      <g transform="scale(1, 1)">
        <rect x="0" y="0" width="100" height="100" style="fill: red;" />
        <foreignobject>
          <div style="font-size: 1em">this text is fine</div>
        </foreignobject>
      </g>
      <g transform="scale(2, 1)">
        <rect x="0" y="100" width="100" height="100" style="fill: blue;" />
        <foreignobject y="100" width="100" height="100">
          <div style="font-size: 1em">this text is stretched</div>
        </foreignobject>
      </g>
    </svg>