Yeah exactly right. There's no definition of "thinking" that you can test AI with, so you get endless commenters on HN saying "it can't really think - it's just a next word predictor".

Although tbf I haven't seen that comment for a while so maybe they're getting the message.