Who decides when clinical AI is ready for everyone?

An AI model can work beautifully for one patient and still fail the community it enters. Testing well can't answer the harder question: when should a system be allowed to become part of someone's care?