Not really. Anthropic has the "CBRN filter" on Opus series. It used to kill inquiries on anything that's remotely related to biotech. Seems to have gotten less aggressive lately?
I was reverse engineering a medical device back in 2025 and it was hard killing half my sessions.
Why not both? A pre-trained LLM has an awful lot of structure, and during SFT, we're still doing deep learning to teach it further. Innate structure doesn't preclude deep learning at all.
There's an entire line of work that goes "brain is trying to approximate backprop with local rules, poorly", with some interesting findings to back it.
Now, it seems unlikely that the brain has a single neat "loss function" that could account for all of learning behaviors across it. But that doesn't preclude deep learning either. If the brain's "loss" is an interplay of many local and global objectives of varying complexity, it can be still a deep learning system at its core. Still doing a form of gradient descent, with non-backpropagation credit assignment and all. Just not the kind of deep learning system any sane engineer would design.
Modern systems like Nano Banana 2 and ChatGPT Images 2.0 are very close to "just use Photoshop directly" in concept, if not in execution.
They seem to use an agentic LLM with image inputs and outputs to produce, verify, refine and compose visual artifacts. Those operations appear to be learned functions, however, not an external tool like Photoshop.
This allows for "variable depth" in practice. Composition uses previous images, which may have been generated from scratch, or from previous images.
Evolution is an optimization process. So if platonic representation hypothesis holds well enough, there might be some convergence between ML neural networks and evolved circuits and biases in biological neural networks.
I'm partial to the "evolved low k-complexity priors are nature's own pre-training" hypothesis of where the sample efficiency in biological brains comes from.
The "platonic representation hypothesis" crowd can't stop winning.
Potentially useful for things like innate mathematical operation primitives. A major part of what makes it hard to imbue LLMs with better circuits is that we don't know how to connect them to the model internally, in a way that the model can learn to leverage.
Having an "in" on broadly compatible representations might make things like this easier to pull off.
You seem to be going off the title which is plainly incorrect and not what the paper says. The paper demonstrates HOW different models can learn similar representations due to "data, architecture, optimizer, and tokenizer".
"How Different Language Models Learn Similar Number Representations" (actual title) is distinctly different from "Different Language Models Learn Similar Number Representations" - the latter implying some immutable law of the universe.
"using periodic features with dominant periods at T=2, 5, 10" seems inconsistent with "platonic representation" and more consistent with "specific patterns noticed in commonly-used human symbolic representations of numbers."
Edit: to be clear I think these patterns are real and meaningful, but only loosely connected to a platonic representation of the number concept.
The "platonic representation" argument is "different models converge on similar representations because they are exposed to the same reality", and "how humans represent things" is a significant part of reality they're exposed to.
I don't think this is a correct formulation of the platonic representation argument:
different models converge on similar representations because they are exposed to the same reality
because that would be true for any statistical system based on real data. I am sure the platonic representation argument is saying something more interesting than that. I believe they are arguing against people like me, who say that LLMs are entirely surface correlations of human symbolic representation of ideas, and not actually capable of understanding the underlying ideas. In particular humans can speak about things chimpanzees cannot speak about, but that we both understand (chimps understand "2 + 2 = 4" - not the human sentence, but the idea that if you have a pair of pairs on one hand, and a quadruplet on the other, you can uniquely match each item between the collections). Humans and chimps both seem to have some understanding of the underlying "platonic reality," whatever that means.
"Not actually capable of understanding" is worthless unfalsifiable garbage, in my eyes. Philosophy at its absolute worst rather than science.
Trying to drag an operational definition of "actual understanding" out of anyone doing this song and dance might as well be pulling teeth. People were trying to make the case for decades, and there's still no ActualUnderstandingBench to actually measure things with.
No, it is partially falsifiable. LLMs clearly don't understand the concept of quantity. They fail at tests designed to assess number understanding in dogs and pigeons; in fact they are quite likely to fail these tests, because they are wildly out of distribution.
We don't know how to demonstrate actual understanding, but we sure can demonstrate a lack of it. When it comes to abstract concepts like "three" or even "more," LLMs have a clear lack of understanding. Birds and mammals do not.
you're right, its just that 'platonic' is an argument that numbers exist in the universe as objects in and of themselves, completely independent of human reality. if we don't assume this, that numbers are a system that humans created (formalism), then sure, we can be happy that llms are picking common representations that map well into our subjective notions of what numbers are.
FWIW it's objectively false that numbers are a system humans created. That's almost certainly true for symbolic numbers and therefore large numbers ( > 20). But pretty much every bird and mammal is capable of quantitative reasoning; a classic experiment is training a rat to press a lever X times when it hears X tones, or training a pigeon to always pick the pile with fewer rocks even if the rocks are much larger (i.e. ruling out the possibility of simpler geometric heuristics). Even bees seem to understand counting: an experiment set up 5 identical human-created (clearly artificial) landmarks pointing to a big vat of yummy sugar water. When the experimenters moved the landmarks closer together, the bees undershot the vat, and likewise overshot when the landmarks were moved further apart.
And of course similar findings have been reproduced etc etc. The important thing to note is how strange and artificial these experiments must seem for the animals involved - maybe not the bees - so e.g. it seems unlikely that a rat evolved to push a lever X times, it is much more plausible that in some sense the rat figured it out. At least in birds and mammals there seems to be a very specific center of the brain responsible for coordinating quantitative sensory information with quantitative motor output, handling the 1-1 mapping fundamental to counting. More broadly, it seems quite plausible that animals which have to raise an indeterminate number of live young would need a robust sense of small-number quantitative reasoning.
It is an interesting question as to whether this is some cognitive trick that evolved 200m years ago and humans are just utterly beholden to it. But I think it requires jumping through less hoops to conclude that the human theory of numbers is pointing to a real law of the universe. It's a consequence of conservation of mass/energy: if you have 5 apples and 5 oranges, you can match each apple to a unique orange and vice versa. If you're not able to do that, someone destroyed an apple or added an orange, etc. It is this naive intuitive sense of numbers that we think of as the "platonic concept" and we share it with animals. It seems to be inconsistent and flaky in SOTA reasoning LLMs. I don't think it's true that LLMs have stumbled into a meaningful platonic representation of numbers. Like an artificial neural network, they've just found a bunch of suggestive and interesting correlations. This research shows the correlations are real! But let's not overinflate them.
Regardless of whether the convergence is superficial or not, I am interested especially in what this could mean for future compression of weights. Quantization of models is currently very dumb (per my limited understanding). Could exploitable patterns make it smarter?
Saw similar study comparing brain scans of person looking at image, to neural network capturing an image. And were very 'similar'. Similar enough to make you go 'hmmmm, those look a lot a like, could a Neural Net have a subjective experience?'
"Subjective experience" is "subjective" enough to be basically a useless term for any practical purpose. Can't measure it really, so we're stuck doing philosophy rather than science. And that's an awful place to be in.
That particular landmine aside, there are some works showing that neural networks and human brain might converge to vaguely compatible representations. Visual cortex is a common culprit, partially explained by ANN heritage perhaps - a lot of early ANN work was trying to emulate what was gleaned from the visual cortex. But it doesn't stop there. CNNs with their strong locality bias are cortex-alike, but pure ViTs also converge to similar representations to CNNs. There are also similarities found between audio transformers and auditory cortex, and a lot more findings like it.
We don't know how deep the representational similarity between ANNs and BNNs runs, but we see glimpses of it every once in a while. The overlap is certainly not zero.
Platonic representation hypothesis might go very far, in practice.
As someone actively researching in the neuroscience field these ideas are increasingly questionable. They do do a decent job of job of predicting neural data depending on your definition and if you compare them to hand built sets of features but we’re actually not even sure that will stay true. Especially in vision we already know that as models have scaled up they actually diverge more from humans and use quite different strategies. If you want them to act like humans or better reflect neural data you have to actively shape the training process to make that happen. There’s less we know about the language side of things currently though as that part of the field hasn’t yet really figured out exactly what they’re looking at yet because we generally know less about language in the brain vs vision. I think most vision scientists are on board with the idea that these things have really been diverging and have to be coerced to be useful. Language it’s more up in the air but there’s a growing wave of papers lately that seem to call the human LLM alignment idea into question. Personally I think the platonic representation idea is just a function of the convergence of training methods, data, and architectures all of these different labs are using. If you look at biological brains across species and even individuals within a species you see an incredible variety of strategies and representations that it seems ridiculous to me that anyone would suggest that there’s some base way to represent reality that is shared across everyone and every species. Here’s some articles that may be of interest if you’re curious:
If you could brain scan a human, and identify a shape of the network that corresponds to an emotion, and then could identify that in the ANN, could we say the ANN is experiencing an emotion.
I think its loosely referred to as "neural correlate".
I'm assuming what you are talking about with Convergence, would be these "neural correlate". And no reason we couldn't move beyond images to 'feelings'.
Not giving the data to researchers means not getting the scientific benefits from that data. Which was the point of collecting that data in the first place.
Reckless harm prevention is the root of many evils.
As a biostatistician who's touched epidemiological studies, I'd argue losing the trust of participants and the public is one of the biggest threats to the viability of the whole research enterprise. It's reckless to jeopardize that as well. Conversely, this dataset will be mined for at least 30-50 years - there are an infinite number of questions that can be asked of this dat. Given that timescale, I think a little delay here is acceptable.
It's not a zero-sum game, you can both protect people and reap the benefits of health data. Many countries have much safer approaches. UK Biobank typically leads with the scale of the data, but not with its infrastructure.
Sensitive research systems thread that needle by giving remote access to researchers with the data in the control and supervision of the responsible organization. Strong internal data access controls and data siloing alongside strict verified extraction routines. Specifically: limited project-dedicated DB access, full logging of data interactions, and full lockouts/freezes if something feels off.
‘The five safes’ is a good presentation from the NHS(?) a decade ago covering the approaches.
Data publishing restrictions around health data aren’t reckless. Modern computing and digital permanence mean we have to be extra cautious.
I do expect this to have a "novelty edge" over human opponents - which can be closed with practice, on the human end.
And, like many AIs, it can have "jagged capability" gaps, with inhuman failure modes living in them - which humans can learn to exploit, but the robot wouldn't adapt to their exploitation because it doesn't learn continuously. Happened with various types of ML AIs designed to fight humans.
Only if you assume the AI can't improve. Otherwise, AI has a fundamental edge over humans in that they don't get old and die, and can be copied perfectly without an expensive retraining period
Chess players learned to exploit chess computers’ weaknesses in the beginning too, but they can’t any longer. This version of the robot might not learn continuously, but the next will be better.
I believe there are still some echoes of the concept. Even top engines will play certain grandmaster draw lines unless told more or less explicitly not to. So if you were playing a match against Stockfish you'd want to play the Berlin draw as White every time, for example.
But chess is a turn-based game where there's no deception (in the sense that both players can see all legal moves for both themselves and their opposition at all times), whereas in table tennis, it's in real time, it's fast as hell, the table is small, and the ball can have 2 or 3 different spin types from the same arm/hand/wrist movement , and can land in a number of different spots.
If "people who made this possible" were getting their fair share, "a millionth of a cent for every billion USD made with it" would be about it for the artists.
What makes the dataset valuable isn't that the image 0012992 in it is precious and irreplaceable. It's that the index goes to seven digits. Pre-training is very much a matter of scale - and scraping is merely the easiest way to get data at scale.
People who complain about "artists not getting paid" must have in their imagination some kind of counterfactual where artists are being paid thousands for their contributions. That's not how it works. A counterfactual world where artists were paid for AI training is one where an average artist is 5 cents richer, an average image generation AI performs 5% worse, and the bulk of extra data spending is captured by platforms selling stock photos and companies destructively digitizing physical media.
The ideal world would be one where, to train on art, you have to buy a license to that art. Sure, for most artists they would maybe put a low price tag, but that isn't the point.
The point isn't about money. It's that copies were made, without license and without permission, and without any legal right to do so, of art, and then used to train a system which generates similar art. The first step, the copy, is illegal without a license, and even for most public images online, licenses and copyright notices (which must be preserved) are attached.
"Without any legal right to do so" is for the courts to decide. And so far, the courts are very much not deciding the way you want them to.
"Fair use" counters "without license and without permission" hard. The argument that training AI on scraped data is "fair use" and the resulting model outputs are "transformative works" has held up in courts. Anthropic got dinged for downloading pirated books, but not for throwing the ones they didn't pirate down the training pipeline.
Some countries, like Japan, have amended their copyright laws to make AI training categorically legal. Others are in "fair use clauses" grey areas with courts deciding case by case based on precedent and interpretation. So trying to latch onto copyright law is, as it always was, the wrong move. Copyright never favored the small guy. Stupid to expect that it suddenly will.
> The argument that training AI on scraped data is "fair use" and the resulting model outputs are "transformative works" has held up in courts.
Nope. Nope. Nope. That has explicitly not been ruled on yet. Transformative means that you don't need a fair use defense. Anthropic has only gotten away with their outputs being called transformative so far because they put a dubiously effective filter in front to block the most egregious infringing outputs. No one has actually challenged this afaik.
Would your ideal world apply to humans as well? Like if I see some art in a museum and it inspires me to create some of my own, I would need to pay a licensing fee to the original artist?
And what about the artists that inspired them? There is no art in the world that sprang fully formed from one single person, without any influences.
Should we reshape our economy to ensure knowledge and artistic provenance is maintained perpetually?
This whole discussion is so weird to me. It’s like AI has freaked everyone out so much that the instinct is to run to the safety of Disney-esque complete control and perpetual monetization of every work.
Which is exactly the opposite of how art worked for the first several hundred thousand years. Really, we want to double down on the perverse incentives and tight control that IP owners have given us in the past 50 years?
>Like if I see some art in a museum and it inspires me to create some of my own, I would need to pay a licensing fee to the original artist?
Nope, humans are admitted for free :).
>And what about the artists that inspired them? There is no art in the world that sprang fully formed from one single person, without any influences.
As long as you are a human you get to be inspired all you want :)
You seem very invested in licking the boot of the trillion dollar corporations. Your fellow humans are concerned.
>Really, we want to double down on the perverse incentives and tight control that IP owners have given us in the past 50 years?
Isn't it interesting that the EXACT second that copyright law impedes billion dollar corporations it is thrown out the window, really makes you think huh?
I think you may be placing too much value on the output of these machines which use tons of energy, generate pollution (both noise and chemical), and generate output that's worse then what a human can do. We would be better off if these LLMs didn't exist.
Average person in US reducing his/her meat intake by 1/4 would do much, much more for environment compared with completely scrapping entire AI infrastructure worldwide. For some reason people concerned with environmental impact of AI get really angry whenever I point this out.
> A counterfactual world where artists were paid for AI training is one where an average artist is 5 cents richer, an average image generation AI performs 5% worse, and the bulk of extra data spending is captured by platforms selling stock photos and companies destructively digitizing physical media.
No, a counterfactual world where artists were paid for AI training wouldn't see commercially viable AI at all. A world which plenty of people would be more than happy to live in, mind you.
AI relies on mass piracy worth Googols of dollars if you count like you would the million dollar iPod, but because AI surprised the copyright industry, it's now too late to enforce copyright like that.
Even in a counterfactual world where any data that's not in public domain can't be used in AI training at all, ever, AIs would exist. Training on public domain data is a bitch, but it's doable. It's just that it results in worse AIs for more effort. So no one does it other than to flex.
It would still be "commercially viable", mind. I'm not sure how much would it stall the AI development in practice, but all the inputs of making AIs only get cheaper over time. So I struggle to imagine not having something like DALL-E 1 by 2030.
If we extend the counterfactual and allow for licensed media, we compress the timelines and raise the bar. The "best" image generation AIs of 2026 are now made by the likes of Adobe and locked behind some kind of $500 a month per seat Creative Cloud Pro Future subscription. Because Adobe is rich enough to afford big bulk licensing deals, while the likes of academia and smaller startups have to subsist on old public domain data, permissively licensed scraps and small carefully selected batches of licensed data that might block them from sharing the resulting weights with the licensing deals.
In the "counterfactual: licensed media" world, the local AI generation powerhouse of Stable Diffusion ecosystem probably doesn't exist at all. Big companies selling AI do. Their offerings cost a lot more and perform considerably worse than the actual AIs we have today. So you can't just go to a random website and get an image edited for a shitpost for free. But the high end commercial suites exist, they're used by the media and the marketing companies, and they are still way cheaper than hiring artists. The big copyright companies get their pound of flesh, but don't confuse that for the artists getting a win.
> but because AI surprised the copyright industry, it's now too late to enforce copyright like that.
I think I've got whiplash from the way a lot of the tech scene has gone from 'IP troll outfits are malicious actors who make everything worse for everyone else' to 'IP troll outfits are an ethical and effective solution to exploitation in the AI industry'.
I'm not a huge fan of much of the generative AI industry, but is IP maximalism really the answer here? Before 2022 most of us would have agreed that DRM is generally a scourge for example, and the 'copyright industry' are a big part of pushing for the end of general-purpose computing in favour of DRM-controlled appliances. Personally I'd rather go in the opposite direction, copyright lasts for exactly thirty years and after that a work enters the public domain without exception, and I'd weaken anti-circumvention laws too.
"Copyright" is, frankly, just an excuse people who hate AI latch onto.
Many of the people who rally against AI now used to rally against Napster being prosecuted by RIAA and the Big Mouse renewing copyright expiration dates once again.
It's not that they suddenly gained an appreciation for the copyright law. It's that they found something they hate more than the big record label megacorps - and copyright became a tool they think they can leverage against it. Very stupid, IMO.
Same thing with the water arguments, or pollution in general. It's not about those having any weight, it's about being against AI first and building arguments against it second.
> No, a counterfactual world where artists were paid for AI training wouldn't see commercially viable AI at all. A world which plenty of people would be more than happy to live in, mind you.
You recon Disney and Shutterstock don't have enough images to make commercially viable AI?
Or for that matter, Facebook? Even just for photorealistic images from, you know, all the photos people upload.
> AI relies on mass piracy worth Googols of dollars if you count like you would the million dollar iPod, but because AI surprised the copyright industry, it's now too late to enforce copyright like that.
Not that I disagree that people use everything they can get their hands on for marginal improvements, they obviously do, but the copyright industry being "surprised" is the default state of affairs for infringement, and "piracy" is the wrong word because that's a law and the judges so far have ruled that training isn't itself a copyright offence, while also affirming that it is possible to commit a copyright offence by pirating training data.
If the dataset weren't valuable, big tech wouldn't depend on it to train their models.
I don't care about getting a millionth of a cent as an artist (which btw is a number *you* just pulled out of your imagination). I care about them paying a fair share instead of pocketing it, so the money stays in circulation instead of creating a new class of technofeudal lords.
If it was about this why do OpenAI and Anthropic lose their minds when people are training off their output or trying to scrape their systems.
I actually don't have an issue with training off the mass of everyones work if the models are open and free to build upon, it's locking them away and then throwing your toys out the pram when people try and do the same thing that bothers me.
Good question. I actually have a technical answer, believe it or not.
Pre-training is: training a model from scratch on cheap data that sets the foundation of a model's capabilities. It produces a base model.
Post-training is: training a base model further, using expensive specialized data, direct human input and elaborate high compute use methods to refine the model's behavior, and imbue it with the capabilities that pre-training alone has failed to teach it. It produces the model that's actually deployed.
When people perform distillation attacks, they take an existing base model and try to post-train it using the outputs of another proprietary model.
They're not aiming to imitate the cheap bulk pre-training data - they're aiming to imitate the expensive in-house post-training steps. Ones that the frontier labs have spent a lot of AI-specialized data, compute, labor and hours of R&D work on.
This is probably not "fair use", because it directly tries to take and replicate a frontier lab's competitive edge, but that wasn't tested in courts. And a lot of the companies caught doing that for their own commercial models are in China. So the path to legal recourse is shaky at best. But what's on the table is restricting access to full chain of thought, and banning the suspected distillation attackers from the inference API. Which is a bit like trying to stop a sieve from leaking - but it may slow the competitors down at least.
>Ones that the frontier labs have spent a lot of AI-specialized data, compute, labor and hours of R&D work on.
Granted thats time and money but it's an absolute minuscule amount of human hours compared to the scraped data.
We know this for a fact because of parallelization, work of hundreds of millions vs the work of 20-100 even of OpenAIs team worked for the entire lifetimes of the current team and the lifetimes of the offspring of that team and the lifetimes of their offspring even with several lifetimes they still wouldnt have even made a dent in recreating that initial scraped training data.
> Pre-training is very much a matter of scale - and scraping is merely the easiest way to get data at scale.
Therein lies the problem. AI firms just bulldozed ahead and "just did it" with no consideration for the ethics or legality. (Nor for that matter, how they're going to get this data in the future now that they're pushing artists into unemployment and filling the internet with slop.)
There is no "imagined counterfactual", people just want AI firms to follow basic ethics and apply consent. Something tech in general is woefully inadequate at.
The counterfactual isn't offered by artists, but AI companies. "If we had to ask consent then we couldn't have made this". Okay, so? The world isn't worse off without OpenAI's image generator. Who cares, there's no economic value to these slop images, they're merely replacing stock assets & quickly thrown together MS paint placeholders.
Given how much of a shitshow this technology has always been (I refuse to mince words: This tech had it's "big break" as "deepfakes", and Elon Musk has escalated that even further. It's always been sexual harassment.) The actual net value to society is almost certainly negative.
Aren't most retrocomputing USB devices running open source firmware? Adding a descriptor "WebUSB supported" is a few commits and a firmware update away.
Nothing Ever Happens Bias has served me pretty well on those dubious semiconductor supply chain claims.
The main reason being: materials are cheap - plant time is what's expensive.
First, raw materials are such a small fraction of chip costs that even if the market price of a given material spikes up two orders of magnitude briefly, the market can eat the spike. For many broadly used materials, this alone is "end of story" - the majority of consumers will balk at the price and exit the market long before semiconductors supply chains will. And second, between the costs of halting production and the low volumes of actual materials involved, supply buffers exist on sites. That plays against supply chain fragility.
It's one thing to have everything JITted within an inch of its life on a razor thin margins car plant. It's another matter entirely to have a "potential supply disruption" in semiconductor manufacturing that will, if all supply truly and fully stopped tomorrow, convert to actual stopped plants in 4 months unless something is done about it in the meanwhile. And that "unless something is done" bites hard when you have a lot of engineering capability underlined by general price insensitivity. As semiconductor industry does.
I was reverse engineering a medical device back in 2025 and it was hard killing half my sessions.
reply