"So subsecond I would say is a requirement." you do not make any specific point why you came to that conclusion.
You can easily entertain users to show them that the system is doing something in the background without loosing them and if they are collegues who actually need to search, you don't even need to keep them as they have to use your setup.
Even google cloud and others let you wait for longer search queries. If not business ciritical, you can definitly wait a bit.
And the write system might not need to write it in the endformat. Especially as it also has to handle transformation and filtering.
Nonetheless, as mentioned in my other comment, the interesting details of this is missing.