I stil daily drive a 2016 iPhone 5. I can relate. I can relate

I got a pixel 8 recently as replacement. Put grapheneos on it.

But can’t get myself to use it.

Bummer, I was really looking forward to that.

“Oh you are correct, it is a school, I see it now »

Unlike the Rafale ! That’s a true European jet

Can you work well in a team ?

Being a work horse is nice. But if you can’t work with others, that might be one you are on the layoff list

I used to think that.

And then I started working for a different kind of companies.

I screen out any structure with VC money.

It means I work for company that you never heard of, but that are profitable year after year.

How? They are very careful who gets in. And they make sure everyone is actually useful. Ration manager to worker is really really low.

Oh, and they build a product with the intends of selling it. Crazy, I know.

I do that in real life. Currently with luffa seeds.

Much more fun than any app could be.

I don’t water them regularly but I run with twines and a bottle of fertilizer.

For a decent description of machine society you can check the Culture cycle form Ian Banks. AI are backing an organic society but they are also have their own.

Or Hyperion, fron Simmons. ( the « techno-center is a decentralized computing and plotting government)

Not with that attitude for sure. If the US or / and European union do that, it’s already a big chunk

I wouldn't count on the US anymore, considering today's political climate. But in theory, EU+US could probably make a very compelling argument to China that if all three agree to play nice, nobody gets an advantage because of it, while everyone can benefit from a slower technological development leaving more time to figure out the societal problems.

Ultimately us random people on the internet can't say if China would want that or could be convinced with some other concessions unrelated to AI, but what we can say for sure is that, if China has the will to chill, the west has the negotiating power to match them.

I wish you interesting time!

No, don't do that. We all live in the same time. Please take it back before the Gods notice.

I’m very familiar with Postgres and spinning one with postgis seems easy enough. Do I get more with duckdb?

Most of the time I store locations and compute distance to them. Would that being faster to implement with duckdb

Probably no difference for your use-case (ST_Distance). If you already have data in Postgres, you should continue using Postgis.

In my use case, I use DuckDB because of speed at scale. I have 600GBs of lat-longs in Parquet files on disk.

If I wanted to use Postgis, I would have to ingest all this data into Postgres first.

With DuckDB, I can literally drop into a Jupyter notebook, and do this in under 10 seconds, and the results come back in a flash: (no need to ingest any data ahead of time)

  import duckdb
  duckdb.query("INSTALL spatial; LOAD spatial;")
  duckdb.query("select ST_DISTANCE(ST_POINT(lng1, lat1), ST_POINT(lng2, lat2)) dist from '/mydir/*.parquet'")

I haven't yet understood this pattern (and I tried using duckdb). Unless you're only ever going to query those files once or twice in your life, importing them into postgres shouldn't be that long and then you can do the same or more than with DuckDB.

Also as a side note, is everyone just using DuckDB in memory? Because as soon as you want some multiple session stuff I'd assume you'd use DuckDB on top of a local database, so again I don't see the point but I'm sure I'm missing something.

> importing them into postgres shouldn't be that long and then you can do the same or more than with DuckDB.

Usually new data is generated regularly and would require creating a separate ETL process to ingest into Postgres. With DuckDB, no ETL is needed. New Parquet files are just read off the disk.

> Also as a side note, is everyone just using DuckDB in memory?

DuckDB is generally used as a single-user, and yes in-memory use case is most common. Not sure about use cases where a single user requires multiple sessions? But DuckDB does have read concurrency, session isolation etc. I believe write serialization is supported in multiple sessions.

With Parquet files, it's append-only so the "write" use-cases tend to be more limited. Generally another process generates those Parquet files. DuckDB just works with them.

> Usually new data is generated regularly

This part was not obvious. In a lot of cases geodata is mostly stable and reads/searches dominate over appends. And that’s why we keep this in DB (usually postgis, yes).

So DuckDB is optimised for very different use case and it is not always obvious when it’s mentioned

This is the more trivial use case (static data, heavy reads) and DuckDB is absolutely optimized for this use case.

DuckDB also provides a vectorized, parallelized engine. When I run a query all of my 32 cores light up on htop.

But DuckDB works just as well with static data.

I think the main point is not having to store a duplicate of the 600GB of input data.

And now I'm curious whether there's a way to actually index external files (make these queries over 600GB faster) and have this index (or many indices) be persistent. I might have missed that when I looked at the docs...

If the data is in Parquet they are already indexed in a sense. No further indexing necessary.

If they are stored in DuckDB’s native format (which I don’t use), it supports some state of the art indices.

https://duckdb.org/docs/stable/sql/indexes.html

I find Parquet plenty fast though.

Ah thanks, of course. I was thinking of dealing with millions of (Geo)JSON files adding up to terabytes, without copying/duplicating them though, mostly indexing. I used to do that with postgres foreign data wrappers and had hopes for duckdb :-). But that's a question for SO or other forum.