My company is heavily invested in Databricks and let me tell you it sucks. 5 min to spin up a job that needs to run for 10 seconds is a terrible way to spend ones time and money.
use serverless then. its literally the simplest solution. what sort of poor decisions your team is making? it needs to run 10 seconds but still spinning up a cluster?
