How good is JS interop with C/C++/BLAS? That's the basic stepping stone, I think. If you cannot make something in JavaScript that can compete with numpy there's little chance that things will change anytime soon.
I don’t know the details as specifically, since I haven’t been able to justify investing my efforts in the non-flagship ecosystem within the TensorFlow project after it previously added its Swift version to the Google Graveyard, but TensorFlow.js is doing something in this direction for the Node.js version. This info is at: https://www.tensorflow.org/js/guide/nodejs
“Like the CPU package, the module is accelerated by the TensorFlow C binary. But the GPU package runs tensor operations on the GPU with CUDA.”
They note that these operations are synchronous, so using them will sacrifice some of JavaScript’s effectiveness at asynchronous event processing. This is not different from Python when you are training or serving a model. JavaScript’s strengths would shine brighter when coordinating agents / building systems that coordinate models.