OpenZeppelin finds data contamination and misclassification in EVMbench

Blockchain security firm OpenZeppelin says it has found flaws in OpenAI’s new benchmark for blockchain security, EVMbench. Launched in mid-February with crypto investment firm Paradigm, EVMbench was built to evaluate how well AI models can identify, patch and exploit smart contract vulnerabilities.

OpenZeppelin welcomed the initiative but recently decided to put EVMbench "through the same scrutiny" it applies to the protocols it helps secure. In its audit OpenZeppelin identified two key problems: training data contamination and invalid vulnerability classifications.

The firm said the dataset contained methodological flaws and at least four issues labeled high severity that are not exploitable in practice. On contamination, OpenZeppelin noted the benchmark is drawn from 120 audits conducted between 2024 and mid-2025, while many model training cutoffs are set to mid-2025.

openzeppelin, evmbench, openai, paradigm, smart contracts, data contamination, misclassification, vulnerability classification, training cutoff, audits