>The implied answer is of course, one is water falling from the sky and one is a machine.

Lets say you're in a room a good distance from the window. Suddenly you hear what sounds like thunder and rain falling. From a distance it appears that it's raining outside.

Is the rain real? Or is it simulated on a screen well enough you can't tell?

You have input output devices just like a computer. They don't see reality, they filter out huge amounts of data and your brain just interprets it. If our machines get good enough we may be able to blast signals directly to the brain that say it's raining and the brain wouldn't have any idea if it was simulated or not. Much in the same way it feels like we exist and not a 3d hologram an infinite distance away (or whatever other weirdness physics may or could do).

So how do we get from "machines may get really good at faking things to human perception" to "machines themselves can have human-like (or 'better') perception"?

The question is so-called consciousness arising out of machines, not machines deceiving human consciousness.

Even then, that deception still doesn't prove equivalency of simulated and real things, unless we're adopting an extreme and self-contradictory subjectivist epistemology.

People seem to way overcomplicate consciousness, especially in machines.

Where does a running video game exist? It's a simulation in the hardware. Where is the consciousness in a human brain, again it's electrical signals in our brain.

At the end of the day a system is what it does. Once it starts simulating the human mind in ways that appear human then we're at the point of saying a plane has to flap its wings or it's not flying.

You can't look at the "real weather" though. You can only look at the outputs. That's the constraint. Good luck and have fun.

A human brain is a big pile of jellied meat spread. An LLM is a big pile of weights strung together by matrix math. Neither looks "intelligent". Neither is interpretable. The most reliable way we have to compare the two is by comparing the outputs.

You can't drill a hole in one of those and see something that makes you go "oh, it's this one, this one is the Real Intelligence, the other is fake". No easy out for you. You'll have to do it the hard way.

Even granting all of your unfounded assertions; "the output" of one is the rain you see outside, "the output" of the other is a series of notches on a hard drive (or the SSD equivalent, or something in RAM, etc.) that's then represented by pixels on a screen.

The difference between those two things (water and a computer) is plain, unless we want to depart into the territory of questioning whether that perception is accurate (after all, what "output" led us to believe that "jellied meat spread" can really "perceive" anything?), but then "the output" ceases to be any kind of meaningful measure at all.

there is no "real weather". the rain is the weather. the map is not the territory. these are very simple concepts, idk why we need to reevaluate them because we all of a sudden got really good at text synthesis

Everyone's a practical empiricist until our cherished science fiction worldview is called into question, then all of a sudden it's radical skepticism and "How can anyone really know anything, man?"

You experience everything through digital signals. I dont see why those same signals cant be simulated. You are only experiencing the signal your skin sends to tell you there is rain, you dont actually need skin to experience that signal.