It seems that desktop IE10 doesn't correctly scale text that has its font size set in viewport units, when the zoom level is not set to 100%. I'm trying to find out which IE versions (and possibly other browsers) are affected by this. Please try this example in your browser and report your findings:
<!DOCTYPE HTML>
<HTML>
<HEAD>
<META HTTP-EQUIV="content-type" CONTENT="text/html; charset=ISO-8859-1">
<META NAME="viewport" CONTENT="width=device-width, initial-scale=1.0">
<STYLE TYPE="text/css">
DIV:first-of-type {width: 32px; height: 32px; background-color: blue;}
DIV:last-of-type {width: 3.125vw; height: 3.12vw; background-color: red;}
P:first-of-type {font-size: 32px;}
P {font-size: 3.125vw;}
P:last-of-type {font-size: 16px;}
@-webkit-viewport {width: device-width; zoom: 1.0;}
@-moz-viewport {width: device-width; zoom: 1.0;}
@-ms-viewport {width: device-width; zoom: 1.0;}
@-o-viewport {width: device-width; zoom: 1.0;}
@viewport {width: device-width; zoom: 1.0;}
</STYLE>
</HEAD>
<BODY>
<DIV></DIV>
<P>font-size: 32px</P>
<DIV></DIV>
<P>font-size: 3.12vw</P>
<P>Resize the window to see the expected behaviour.<BR>
Zoom in/out to see IE's erratic behaviour:<BR>
the DIV adapts to the new viewport width,<BR>
but the font size doesn't.</P>
</BODY>
</HTML>
This is a simple workaround: the font sizes are set in percentages, and a resize-triggered script sets the body font size according to the body width.
However, I'd still like to know which browsers and versions I need to enable this workaround for. Please try out the example in the question, and report your platform, browser, version, and the observed behaviour when resizing and zooming the window.