i honestly dont think there's a simple y/n answer there - i think considerations include mostly like 'how costly it is to do so', 'how often do you think you'll need it', and so on.
traces are not as "ephemeral" as FT models - since you can use those to guide agent behaviour when a newer model is released (but still, not as evergreen as other assets - traces generated using say GPT4 would seem pale and outdated compared to ones created on the same dataset using Opus4.5 i reckon)
There's a loop between adoption of a technology and adaptation of a technology. For some domains that loop is fast, the adaptations prove to be easy and the feedback from adopters is easy to get. For other domains it's slow, especially towards the end of the process of going from something interesting to something useful. A good example of a slow loop is self driving, it's hard to get feedback about self driving in safety critical real world situations... another example is medicine.
The other issue is that the value is more or less all in the LLMs (at the minute). For example, I built a data engineering toolkit using LLMs, it created synthetic data from examples, it created ingestion pipelines given different source filed and a target, it created data test rules. I liked my little toolkit and some people were impressed, but the value was all in the models that underpinned it. The crust of clever bits that added value was thin, very thin. Ok, we used the llms to generate some python that then created the synthetic data and testing rules to reduce costs, we had three or four "agents" that worked together to create the pipelines. We decorated target code with open provenance code to create provanance... But just by saying these things or letting you use the toolkit and you seeing what it made - that's enough for any half competent person to relicate it (with AI assistance) in an afternoon, or maybe a couple of afternoons. Maybe.
So, to create a viable company is going to take significant effort (if you can think of a value add) because the value add still has to be real.
I think that the parent probably meant Henry VIII.
reply