There's also the validity of learning methods, despite what studies may claim, there's no scientific "grand theory of meta-learning", and if ideas are misapplied/misused there's a risk of falling into scientism, which would be just as harmful as economically driven credentialism. At worst it is just the austerity version of education—learn it yourself because we can't afford the school resources to teach/coach/nurture subjects.
There is no grand theory perhaps, but educational research is not totally useless, people have accumulated a lot of insights over the years and some of these "intuitions" are also backed by studies.
Here are some book-length reviews of currently known things about learning/teaching that I found to have a very high signal-to-noise ratio:
It would certainly interesting to have a greater diversity of moderators, for instance if this platform runs techno-centric (reflecting the beliefs and biases of managers and corporations in the tech industry) then maybe some academic, scholarly, and/or public intellectual type of person so as to balance out the implicit editorial voice that is inevitable in any online moderation scheme.
I don't understand the claim, is it recreating the actual operating system and kernel, and it can run and install programs like an emulator? Or is it just superficially the UI?
There are some good example posts on Scott Aaronson's blog where he eviscerated shoddy physicists' take on quantum complexity theory. Physicists today aren't like Turing et al, most never picked up a theory of computer science book and actually worked through the homework exercises, and with AI pivot and paper spawning, this is kind of a general problem (arguably more interdisciplinary expertise is needed but people need to actually take the time to learn material and internalize it without making sophomore mistakes etc.).
I hazard to imagine that LLMs are a special subset of Markov chains, and this subset has interesting properties; it seems a bit reductive to dismiss LLMs as "merely' Markov chains. It's what we can do with this unusual subset (e.g. maybe incorporate in a larger AI system) that is the interesting question.
You don't have to imagine, there is a logically rigorous argument¹ that establishes the equivalence. There is also nothing unusual about neural networks or Markov chains. You've just been mystified by the marketing around them so you think there is something special about them when they're just another algorithm for approximating different kinds of compressible signals & observations about the real world.
I'll have you realize you are replying (quite arrogantly by the way) to someone who wrote part of their PhD dissertation on models of computation. Try again :)
Besides, it is patently false. Not every Markov chain is an LLM, an actual LLM outputs human-readable English, while the vast majority of Markov chains do not map onto that set of models.
Appeals to authority do not change the logical content of an argument. You are welcome to point to the part of the linked argument that is incorrect & present a counter-example to demonstrate the error.
Calf isn't making an appeal to authority. They are saying "I'm not the idiot you think I am." Two very different things. Likely also a request to talk more mathy to them.
I read your link btw and I just don't know how someone can do all that work and not establish the Markov Property. That's like the first step. Speaking of which, I'm not sure I even understand the first definition of your link. I've never heard the phrase "computably countable" before, but I have head "computable number," which these numbers are countable. This does seem to be what it is referring to? So I'll assume that? (My dissertation wasn't on models of computation, it was on neural architectures) In 1.2.2 is there a reason for strictly uniform noise? It also seems to run counter to the deterministic setting.
Regardless, I agree with Calf, it's very clear MCs are not equivalent to LLMs. That is trivially a false statement. But the question of if an LLM can be represented via a MC is a different question. I did find this paper on the topic[0], but I need to give it a better read. Does look like it was rejected from ICLR[1], though ML review is very noisy. Including the link as comments are more informative than the accept/reject signal.
(@Calf, sorry, I didn't respond to your comment because I wasn't trying to make a comment about the relationship of LLMs and MCs. Only that there was more fundamental research being overshadowed)
If it's trivially false then you should be able to present a counter-example but so far no one has done that but there has been a lot of hand-waving about "trivialities" of one sort or another.
Neural networks are stateless, the output only depends on the current input so the Markov property is trivially/vacuously true. The reason for the uniform random number for sampling from the CDF¹ is b/c if you have the cumulative distribution function of a probability density then you can sample from the distribution by using a uniformly distributed RNG.
You want me to show that it is trivially false that all Neural Networks are not Markov Chains? I mean we could point to a RNN which doesn't have the Markov Property. I mean another trivial case is when the rows do not sum to 1. I mean the internal states of neural networks are not required to be probability distributions. In fact, this isn't a requirement anywhere in a neural network. So whatever you want to call the transition matrix you're going to have issues.
Or the inverse of this? That all Markov Chains are Neural Networks? Sure. Well sure, here's my transition matrix [1].
I'm quite positive an LLM would be able to give you more examples.
> the output only depends on the current input so the Markov property is trivially/vacuously true.
It's pretty clear you did not get your PhD in ML.
> The reason for the uniform random number
I think you're misunderstanding. Maybe I'm misunderstanding. But I'm failing to understand why you're jumping to the CDF. I also don't understand why this answers my question since there are other ways to sample from a distribution knowing only its CDF and without using the uniform distribution. I mean you can always convert to the uniform distribution and there's lots of tricks to do that. Or I mean the distribution in that SO post is the Rayleigh Distribution so we don't even need to do that. My question was not about that uniform is clean, but that it is a requirement. But this just doesn't seem relevant at all.
Either find the exact error in the proof or stop running around in circles. The proof is very simple so if there is an error in any of it you should be able to find one very easily but you haven't done that. You have only asked for unrelated clarifications & gone on unrelated tangents.
I read Scott Aaronson's blog posts this week and he makes a seemingly similar argument, behind his tendency for heated rhetoric.
If the international community will barely lift a finger to resolve the I/P issue, then it is predictable and rational for Israel to take matters in their own hands and use violence (implemented as a "preemptive war") to "solve" their national security threat problem. It's a type of political realism argument to support this outcome. No appeal to a country being enlightened or democratic, etc., will work.
That's because your premise is contingent on a pragmatic framing, whereas these issues depend on science. If the science says intellectual aptitude is a function of nature and nurture then we as society ought to find ways to help our next generations flourish. That does not mean allocating 10 teachers to one child as you had so strongly put it.
Yes but I actually (tend to) believe Hinton and the other CS scientists, so the terms aren't even the main issue, whereas this author's typical mainstream revolving terms consists of anthrocentric worries about what is really a scientific crisis--it smacks of rearranging the deck chairs while the Titanic is about to hit the iceberg that is the AI/AGI technological revolution/singularity.
The problem with this argument is that the science is still out. Hinton and other actual CS experts are terrified of AI and the risk of an AI/AGI technological singularity. Instead what this article focuses on is the status quo technology, while those scientists (who don't care that much about Altman and his ilk) thinking about the storm to come now that the Pandora's box has been opened.
Your unpopular opinion is fallacious, markets can fail and as a result grey/black markets arise. This sci-hub issue is plausible evidence of that. Moreover there are systems where markets could be entirely inappropriate. But there's no law of nature or god that tells us how to decide as a society. Indeed it is your very mention of consumerism that belies this presupposition.
Why shouldn't we abolish any digital markets then, because in theory, you could have a service similar to sci-hub for books, movies, music... And these exist and existed (and are regularly shut down by the authorities).
