Why so brazenly confident? Isn’t it possible that the benchmark is correct, and your experience is correct too, but you haven’t tried all the thousand different modalities of work that programming encompasses and so maybe you don’t actually have standing to judge?

very outdated experience from me: when I first tried gemini something, in an existing rust codebase, it looked around for files that would indicate if its go, javascript, java or c++ project, then declared I must have asked it build a new app in javascript and proceeded to circle around to figure out how it can install node and npm on my machine.

So I totally believe that Gemini is just bad. Which is surprising because Gemma is very good for some tasks, but I never ever had any success with Gemini, be it in cli or chat thing or anything else that has gemini branding.

Your startup is based around AI coworkers, so I expect you are biased towards overvaluing their usefulness.

Yes, perhaps fair. But my point was somewhat narrow. I wasn't saying that team-managed agents are good to go for all cases and that they're better than individual dev-managed ones. Just that they have gotten better and that of those Devin has some of the better UX.

Our experience might also not be typical because we have built infrastructure around making Devin and similar agents work better. And for the record no ties to Devin/Cognition. Just pay them too much as a customer.

It is rivaling Astra, on their own benchmark that they made (FrontierCode), that they ran themselves in their own closed-source ecosystem that isn’t reproducible by anyone.

The Cursor acquisition shows that it’s possible for these valuations to be justified. But Cursor was more successful and bent the truth much less.

While I wouldn’t expect anything good for Cognition’s fate, it’s a much safer bet than Thinking Machines, SSI, and some others.

Though they’ll be in big trouble if the more talented Chinese labs stop letting them repackage their work.

The cursor acquisition just shows that there’s always a dumber shithead out there. Though vaporizing Elon Musks money is about as pure of a good as there is out there these days.

Improvements in models and products coming out of SpaceXAI since the acquisition would seem to disagree with you.

It seems obvious what GP meant. It is, once again, an explicit construction (“disproving” that every initial state does not develop a singularity).

A bit of a hair-splitting, but isn't explicit construction the only way formal theorem provers can work? Of course you can still prove stuff with them, but certain axioms that more "human" proofs use may not be available, like law of excluded middle (every proposition is either true or false)

(Okay, they can be made available in a way similar to `unsafe` in rust)

you can add law of the excluded middle as an axiom. See midway down this page

https://xenaproject.wordpress.com/2017/10/05/more-easy-lean-...

Sure, but then you can no longer actually construct your "objects".

That's what my rust comment was referencing.

I guess I don't understand the issue you're raising. If you want to formalize a non-constructive proof, it remains non-constructive, even if you have a computer check the proof vs a human.

As a trivial example, in lean you can work with probability theory/measure theory. This has oodles of non-constructive parts, but we can ignore that for now. As part of this, you can use the probabilistic method. For example, if you want to prove that codes with optimal parameters exist, for many noise models it is known that sampling a code randomly from an appropriate (and often naive) distribution will yield a code with optimal parameters.

You should be able to prove this in lean (or any other theorem prover). But you cannot construct these codes. While you can sample a code randomly, verifying a code has good parameters is typically NP-hard (e.g. it is an instance of the minimum distance problem). So, you cannot (efficiently) "construct" a good code in lean4, despite being able to prove one exists.

This seems analogous to me that you could validate that a non-constructive proof is correct in lean4. Sure, it would be nice if the proof was constructive. But it isn't, and encoding it into a computer shouldn't give you that (non-trivial) property for free.

Thanks, that's a great example!

He asked whether they used their chats as training data and received no response. Any speculation here seems quite appropriate?

I’m open to evidence, but just using Bayesian reasoning, OpenAI is one of the most dishonest companies in history. They’re currently being sued for a dozen employees stealing Apple hardware! I don’t understand why I should give them any grace.

Because playing a video game isn’t relevant to which AI people might want to use.

"Video game" is the medium, the challenge and test is figuring out how to win; when given no instructions, and specifically designed to be private.

ARC-AGI takes it very seriously: they've never tested Fable, because they won't run on the eval set without ZDR.

Another possible way to look at this is that any benchmark reporting Fable scores is potentially contaminated.

Of course. But that market won’t produce a $800 billion company, unless AI becomes ludicrously widespread — energy is used every day by virtually every person on the planet, and of course has plenty of mass consumption & “luxury” customers too.

You’re not seriously suggesting that the model is secretly sandbagging its performance on GDPval and long context reasoning, while making huge and obvious progress on ExploitBench, ARC and science benchmarks, in order to tank its AA composite score, so it can conceal its true power level?

Why would benchmarks be an adversarial setting anyway?

Could it be possible that OpenAI may have had some other motive for saying their model “strategically underperforms”, other than just an innocent reporting of a truth it happened to discover?

I'm saying that it's generally a losing proposition to even be acquaintances with "agents" who consistently lie to you, and it's flatly fucking insane to give a dishonest "agent" vast amounts of intelligence, capability, and authority to go do things in the world.

So I have no clue what is the answer to your question. Nor does anyone else. Because we're trying to answer a question of fact where our primary source of information is unreliable.

I see, it’s a great point. I know some evals actually do use LLMs as a judge (e.g. those that try to measure debate skill), though the ways AI can try to cheat its way through every benchmark now are astoundingly varied.

why does this comment sound like a character in a horror movie