AI
ai
FAQs

Is SWE-bench still a reliable benchmark?

It is still useful for comparing models, but it does not show how a model handles new code. OpenAI stopped reporting SWE-bench Verified in February 2026, because models had seen its tasks and some tests were wrong. Its replacement, SWE-bench Pro, has about 30% broken tasks.

What is benchmark contamination?

Benchmark contamination means the test tasks or their answers ended up in the model’s training data. The model then remembers the answer instead of solving the problem, so its score looks better than its real skill. Coding benchmarks built on public code have this risk by design.

Why do AI models pass tests but fail code review?

Tests only check that the code runs and fixes the reported case. A reviewer also checks every requirement, the project’s rules and whether anything else broke. In METR’s March 2026 study, maintainers would not accept about half of the AI code that had passed the tests.

How do expert evaluators find model failures?

They write new tasks the model has not seen, with business rules and edge cases, and run each task several times. A failure counts when it happens again and comes with a written reason and a correct solution that a second senior engineer has checked.

Can we test a coding model on our own code?

Yes, and it gives the most honest result. Take finished tickets from your own codebase, write down what a correct fix must do, run each model several times and review the code. UITOP runs this as an evaluation from €6,400, taking from 4 weeks.

l
How to Turn a Vibe-Coded Prototype into a Production-Ready Product
product,development,ai
October 21, 2026
How to Turn a Vibe-Coded Prototype into a Production-Ready Product
l
Designing Human-in-the-Loop AI Products: UX and Engineering Patterns
design,product,ai
October 15, 2026
Designing Human-in-the-Loop AI Products: UX and Engineering Patterns
l
AI Won’t Build a Great Product: Why UX and Development Still Matter
design,development,ai
August 20, 2026
AI Won’t Build a Great Product: Why UX and Development Still Matter