On Linux, virtual terminals are currently limited to 512 glyphs and 16 colors. You can't do much graphics with that, unless you bypass the terminal layer and write directly to the framebuffer, like w3mimg does.
VT's but not XTerms. Also, framebuffer based VT's allow 256 colors with ease.
On the font issue, yes, sadly you are right. It's retarded to just have 512 glyphs on a framebuffer, not a true tty. The framebuffer or KMS should support TTF fonts by default. A Unifont TTF would be godsend here.
I should have been clearer: as long as you have a font which has entries for the glyphs used by derasterize and sixel-tmux - check the source and you'll see you don't need a lot of them.
On Linux, virtual terminals are currently limited to 512 glyphs and 16 colors. You can't do much graphics with that, unless you bypass the terminal layer and write directly to the framebuffer, like w3mimg does.