I’ve only had their platforms explained to me by them (palantir) at a conference but the mental model that stuck with me was more of an operating system than a single tool. Think AWS managed services + databricks + whatever library of ready made intelligence software they have already built for others.
They also have “forward deployed engineers” to help organizations actually use the platform. It looked complicated enough to probably be completely useless without these specialists, even in a “self hosted” setup.
The managed hosting also seems like a major selling point so many deployments that probably should be self hosted probably aren’t because muh managed services added value.
And the backdoors of course. There is no way it isn’t full of plausibly deniable “metrics endpoints” that helpfully spew out all the internal data if the right key comes knocking. There’s no way it’s auditable at the level of detail you would need compared to the value of the data and the sophistication of the potential attacker (NSA).
Maybe my brain is oversaturated with culture war nonsense from too much doomscrolling but that’s where my train of thought went too, even if it wasn’t directly implied.
By claiming our ancient predecessors had terrible taste you can make them look like primitive fools, and make our own modernity appear superior in comparison.
When boiled down to culture war brainrot the poor coloring in the reconstructions becomes a woke statement that the brutish patriarchal empires of antiquity have nothing to teach our sophisticated modern selves and that new is good and old is bad. A progressive hit-piece on muh heritage.
Anything you don’t like is a purple haired marxist if you squint hard enough.
Idk why my brain went there. I’m guessing the years of daily exposure to engagement-farming ragebait had something to do with it.
I like it! We have a service with a similar postgres task queue but we use an insert trigger on the tasks table that does NOTIFY and the worker runs LISTEN, it feels a bit tidier than polling IMO.
LISTEN/NOTIFY works great but they don’t have any mechanism for ACKs or retries so it’s got some tradeoffs to consider. Works great when you’re willing to sacrifice some durability!
I had a few periods of doing the same in sublime text, I did use syntax highlighting though. It’s a really great feeling and very liberating, especially in a greenfield project.
Can’t really justify it at work though, projects are too big to and gnarly keep in my head.
The best practice way to swap fullname for firstname, lastname would be to:
1. Migration that adds firstname and lastname columns will all nulls
2. Deploy application code change to start populating firstname and lastname alongside fullname, still reading fullname in the code.
3. backfill the firstname and lastname values with a script/command/migration
4. change app code to read firstname and lastname and stop writing fullname
5. drop the fullname column
I don't think there's a safe way to do all that in a single migration unless all your app code also lives in the database so it can be atomically deployed. If you have multiple app servers and do rolling deploys with no downtime I think it has to be done in these 5 steps.
There's nothing wrong with nullable fields when it's appropriate. When kids are born they don't have names. Not all users want to tell you their names. A null value is data too.
Timescale is definitely worth a look. Pg_partman gets you part of the way. We ended up going with bigquery for our workload because it solved a bigger bag of problems for our needs (data warehouse). It’s very hard to beat for big… queries.
I never understood the rationale behind TimescaleDB — if you’re building a time series database using row-oriented storage, you’ve already got one hand tied behind your back.
What does your testing strategy look like with bigquery? We use snowflake, but the only way to iterate and develop is using snowflake itself, which is so painful as to impact the set of features that we have the stomach to build.
Testing strategy? What’s that? I kid, but just a bit. Our use case is a data warehouse. We use DBT to build everything. Each commit is built in CI to a CI target project. Each commit gets its own hash prefixed in front of dataset names. Each developer also has their own prefix for local development. The dev and ci datasets expire and are deleted after like a week. We use data tests on the actual data for “foreign keys”, checking for duplicates and allowed values. But that’s pretty much it. It’s very difficult to do TDD for a data warehouse in sql.
My current headache is what to do with an actually big table, 25 billion rows of json, for development. It’s going to be some DBT hacks I think.
God help you if you want to unit test application code that relies on bigquery. I’m sure there are ways but I doubt they don’t hurt a lot.
Interesting strategy with appending the commit hash to the dataset name. If one of those commits is known to be good and you want to “ship” it, do you then rename it?
What are you doing with that JSON? What’s the reason why you can’t get a representative sample of rows onto your dev machine and hack on that?
Reuters really broke the site for me when they did what looked like a rewrite about a year ago. Glad to see this as I hardly read it anymore. bbc news is going downhill too, another prime candidate for a project like this.
They also have “forward deployed engineers” to help organizations actually use the platform. It looked complicated enough to probably be completely useless without these specialists, even in a “self hosted” setup.
The managed hosting also seems like a major selling point so many deployments that probably should be self hosted probably aren’t because muh managed services added value.
And the backdoors of course. There is no way it isn’t full of plausibly deniable “metrics endpoints” that helpfully spew out all the internal data if the right key comes knocking. There’s no way it’s auditable at the level of detail you would need compared to the value of the data and the sophistication of the potential attacker (NSA).
reply