Why is this a meaningful distinction to you? What does "reason" mean here? Can we construct a test that cleanly splits what humans do from what LLMs do?

Sure, things like counting the ‘r’s in strawberry, for example (till they are retrained not to make that mistake).

There are humans that can't do that but are clearly capable of reasoning. Not a meaningful categorical split.

There are certainly humans with poor reasoning or even incapable of reasoning, I’m not sure what you think that proves?

Ok, but if you read my comment you would note that I constructed a category of humans who can reason but cannot count the r's in strawberry.

I think you don't know what it means to reason, and are dismissively claiming AI cannot reason as though it invalidates a point made earlier without even having a sturdy definition in your head. I think for you to say "LLMs can't reason" in this context is essentially a NOP.

It is hard to define reasoning or thinking, these are vague concepts. I use them to indicate there are areas where these machines take obviously wrong decisions, because they are above all probability weighing machines based on a corpus, that is not I hope you would agree thinking, so you must believe there is some emergent properties which constitute thinking since you're so confident these machines are in fact doing that.

AI companies use these terms (thinking, reasoning etc) to try to trick users into anthropomorphising pattern matching machines and so that people believe they are true general intelligence.

I don't think we've reached AGI yet, though we are closer than previously, and I'm skeptical LLMs will be the route - they are impressive, but they are better at tricking humans than at performing complex tasks they have not seen before IME.

Do you think we have seen AGI yet from LLMs? If not how would you define their limitations?

They don't see the letters, so how could they possibly succeed at that? It's like asking a human how many infrared flowers they see.

I'm pointing out that they don't 'think' or 'reason' like humans, they're very impressive, but I don't think they've reached the bar for thinking yet, as simple logic puzzles or puzzles like this prove (until the LLM authors take note and add special workarounds for those particular use-cases).

I believe most LLMS no longer fail at this, because they've been given the tools to do so (for example use python under the hood to count letters), but it's an important observation because it shows us that they don't think like us.

Take it up with OpenAI's API designers—it's their term

You are the one repeating their lies.