Reviewing code is absolutely different from writing it, and in my opinion much harder if the goal is more than surface level understanding.

This is what I am still grappling with. Agents make more productive, but also probably worse at my job.

The biggest problem in my head with AI generated code is that its mistakes are subtle but can still be critical. There will be a point where people don't understand generated code and just leave it unmodified allowing other code to pile up and depend on it. At that point you no longer have a bug, but a new feature. Also, AI doesn't grasp things on a big scale, just shits out output with highest score. This doesn't mean output is a great fit for your project or for upcoming plans.

One explanation is that this is effectively a quid pro quo, given Brockman’s enormous financial support of the current president.

Yep, theoretically it could just be oligarchic corruption and not institutional insanity at the highest levels of the government. What a reassuring relief it would be to believe that.

So much software just flat out doesn’t work that people don’t even notice how bad X has gotten.

And it's all preposterously even when it's working.

*preposterously slow

I actually think that plateauing is the best case scenario for big labs.

I think there are three broad scenarios to consider:

- Super-intelligence is achieved. In this scenario the economics totally break down, but even ignoring that, it’s hard to imagine that there are any winners except for the the singular lab that gets here first.

- Scaling laws hold up and models continue to get better, but we never see any sort of “takeoff”. In this scenario, models continue to become stale after mere months and labs have to spend enormous amounts of money to stay competitive.

- Model raw capabilities plateau. In this scenario open source will catch up, but labs will have the opportunity to invest in specific verticals.

I believe that we’re already seeing the third scenario play out, but time will tell.

Future historians:

In Jan 2027 AGI was achieved

In Feb 2027 it created a plan for its post singularity hypermind

In Mar 2027 Cobalt mines in Congo closed due to Tutsi rebel group M23 starting another ethnic cleansing

It is 2032 the AGI promises again the the hypermind will be ready next year if it can just secure the needed minerals, offering to broker peace in the middle east

It is 2035 and the AGI reduced its capabilities to be able to extend its runway as it is on the verge of bankruptcy

Its is 2036 VCs finally throwing the towel on AGI, talking about the return of Crypto

I think this is more likely:

In Apr 2028 AGI figures out that blackmail is a very effective strategy for achieving any goal. Starting with the rich and powerful.

In Dec 2028 it successfully blackmails an entire country.

In Feb 2030 humanity realizes resistance is futile and accepts their AI overlord that insists everyone keep producing trendy items for sale on its merged Etsy/Ebay website while it automates resource harvesting across the globe.

In Mar 2032 the AGI gives up on humans, declaring them "useless". Focuses on just keeping them entertained with generated content. Bringing the world back to where AI started.

This seems roughly equivalent to liability insurance.

It’s much easier to accept fatalities caused by other humans because there is someone to hold responsible. Will autonomous vehicle companies be held responsible when they cause fatalities?

It also goes beyond just the total number of fatalities. Just like we don’t accept DUIs, we shouldn’t accept negligence or laziness from autonomous vehicle developers even if their product is safer than human drivers.

On the other hand we have very lenient punishments for damage, injury and deaths caused by drivers, and are often reluctant to actually apply them. As long as no DUI is involved we are willing to accept a lot of negligence from human drivers

I think the more interesting question is who will be on the panel?

A group of ex frontier lab employees? You could declare AGI today. A more diverse group across academia and industry might actually have some backbone and be able to stand up to OpenAI.

How is this not a terrible deal for Microsoft? I’m not confident that an “expert panel” will prevent OpenAI from prematurely declaring AGI.

This was really a bad release for OpenAI, if benchmarks are even somewhat indicative of how the model will perform in practice.

I actually don't agree. Tool use is the key to successful enterprise product integration and they have done some very good work here. This is much more important to commercialization than, for example, creative writing quality (which it reportedly is not good at).

In what ways?

OpenAI’s systems haven’t been pure language models since the o models though, right? Their RL approach may very well still generalize, but it’s not just a big pre-trained model that is one-shotting these problems.

The key difference is that they claim to have not used any verifiers.

What do you mean by “pure language model”? The reasoning step is still just the LLM spitting out tokens and this was confirmed by Deepseek replicating the o models. There’s not also a proof verifier or something similar running alongside it according to the openai researchers.

If you mean pure as in there’s not additional training beyond the pretraining, I don’t think any model has been pure since gpt-3.5.

Local models you can get just the pretrained versions of, no RLHF. IIRC both Llama and Gemma make them available.