That is pretty much about right. I wrote a bit about the implementation[1], and the discussion went over how to optimize this (including, but not limited to, using shaders to do the drawing and not relying on Three.js so much).
Would be interested if its worthwhile/possible using shaders as a wrapper for three.js. All I got when I was searching was there was a mozilla bug report and a hacked patch to expose SIMD through js[1], and a project to add SIMD support to vectors in Dart[2].
[1] http://metaphysicaldeveloper.wordpress.com/2011/12/20/implem...