The ledger does not lie, only the narrative does.
Yesterday, a new AI safety index dropped. Anthropic scored C+. OpenAI scored C. The industry collectively shrugged. But if you look at the numbers the way I look at a smart contract audit – line by line, state by state – you see a different story. This isn't about who is safer. It's about a governance layer that is structurally unsound.
Panic is just poor data processing in real-time. The panic here is that these scores are low. The data says they are low because the underlying mechanisms are broken. Let me dissect.
Context: The Hype Cycle Meets the Audit Trail
We are in a bull market for AI. Capital flows freely. Valuations are detached from fundamentals. The same pattern I saw in 2021 DeFi protocols – where liquidity was a mirage and solvency a myth – is repeating itself in AI governance. Companies promise safety. They publish whitepapers. They hire ethics boards. But the code, the actual incentive structures, the audit trails – they tell a different story.
The AI safety index in question measures governance, transparency, red-teaming, and external audits. It does not measure model capability. Yet the market conflates the two. Anthropic markets itself as safety-first. OpenAI markets itself as product-first. The score difference is one letter grade. But the structural difference is a canyon.
Core: The Systematic Teardown of the Scoring Mechanism
First, the score itself is a black box. The methodology is not public. The weights are not disclosed. The sample window is unknown. This is like a DeFi protocol that claims to be audited but refuses to share the audit report. I cannot verify the inputs. I cannot replicate the output. The only thing I can do is trust the issuer. That is not a verification. That is a narrative.
Second, the score measures commitments, not outcomes. Did Anthropic promise to do red-teaming? Yes. Did they actually prevent a jailbreak? The score doesn't say. It's like measuring a DeFi protocol's security by counting the number of audits it commissioned, not by checking if the audits found critical vulnerabilities. I've seen protocols with 10 audits lose $50 million in a reentrancy attack. The audit was there. The implementation was not.
Third, the gap between C+ and C is statistically meaningless. The margin of error on these subjective governance scores is likely larger than the gap. Yet the media treats it as a ranking. This is the same mistake made during the 2021 NFT floor collapse: people saw a price chart and assumed value. They did not check the holder concentration. They did not check the liquidity depth. They followed the narrative.
Contrarian: What the Bulls Got Right
The bulls will say: this is progress. Two years ago, there was no standardized AI safety index. Now there is. The fact that Anthropic and OpenAI even participate in such a rating is a step towards accountability. I agree with the trajectory. But the mechanism is flawed.
Consider the 2018 ICO audit trail. I spent 200 hours manually tracing the ERC-20 token standard logic in the Bytom ICO smart contracts. I found an integer overflow vulnerability that would have allowed team members to drain 40% of the treasury. The project had a whitepaper, a team, a roadmap. It had a governance score? No. But if it did, it would have been based on promises, not code. The code was the only truth.
Similarly, the AI safety index is a promise. It's a governance score. It does not measure the reentrancy vulnerability in the alignment layer. It does not measure the oracle failure that could drain $2 million from a microtransaction pool. It measures the willingness to disclose. That is a signal, but it is not a guarantee.
Takeaway: The Accountability Call
Structure outlives sentiment; code outlives hype. The AI safety index is a governance facade. It will not prevent a catastrophic failure. What will prevent it is formal verification, continuous monitoring, and transparent red-teaming results. Until those are standard, I treat every C+ and C as a warning label. The ledger does not lie. The narrative does.
Are you still buying the narrative? Or are you checking the code?