Note: This post builds on last week’s article “Emerging Risk and Smart Regulation.” Recent advances in machine learning like DALL-E 2 and Stable Diffusion show the strengths of artificial narrow intelligence. That means they perform specialized tasks instead of general, wide-ranging ones. Artificial narrow intelligence is often regarded as safer than a hypothetical artificial
I used to work on software which was subject to the DO-178B Level A software development regulations (this was so long ago that it was before DO-178C came out), which are probably one of the biggest operational examples we have of real-world regulation of potentially life-endangering software systems. My impression of them, as a then-junior developer who went on to work on other high-reliability but unregulated systems, is that they were ~20% actually useful stuff, like:
-- stringent, high-coverage testing requirements
-- requiring that you actually write down a failure mode analysis and point to where you were mitigating each failure mode and have that document reviewed by someone
and ~80% bureaucratic CYA and well-intentioned sludge, like:
-- "traceability" requirements from code to multiple levels of documentation and back
-- reviewer "independence" requirements that made it almost impossible to find someone who both knew enough to review the code intelligently and was "independent" enough
-- quantitative fault probability analyses intended to prove that the chance of catastrophic failure was less than 10^-9, which in practice were exercises in making up numbers that were basically impossible to evaluate with any sort of epistemic rigor
Am I being too cynical about DO-178? Either way, can we learn useful things from its practical application history to apply to AI regulation?
Fascinating. There’s a ground breaking movie, documentary perhaps, just waiting to be made here. My immediate thought: Is the human race an AIG gone rogue? Suddenly I’m thinking of a comedy/drama film, but one that could explore both hazards and great possibilities.
Great topic. Sure raises a lot of questions though. Humanity's experience with regulating complexity is mixed. I can think of four major initiatives in the last century where regulating things with very high risk association. The ones that come to mind are Finance, Light Water Reactors, WMD and Compartmentalizing National Security Information. Each could be a topic unto itself as it relates to AI, ML & AGI. You are in a great and interesting field.
I used to work on software which was subject to the DO-178B Level A software development regulations (this was so long ago that it was before DO-178C came out), which are probably one of the biggest operational examples we have of real-world regulation of potentially life-endangering software systems. My impression of them, as a then-junior developer who went on to work on other high-reliability but unregulated systems, is that they were ~20% actually useful stuff, like:
-- stringent, high-coverage testing requirements
-- requiring that you actually write down a failure mode analysis and point to where you were mitigating each failure mode and have that document reviewed by someone
and ~80% bureaucratic CYA and well-intentioned sludge, like:
-- "traceability" requirements from code to multiple levels of documentation and back
-- reviewer "independence" requirements that made it almost impossible to find someone who both knew enough to review the code intelligently and was "independent" enough
-- quantitative fault probability analyses intended to prove that the chance of catastrophic failure was less than 10^-9, which in practice were exercises in making up numbers that were basically impossible to evaluate with any sort of epistemic rigor
Am I being too cynical about DO-178? Either way, can we learn useful things from its practical application history to apply to AI regulation?
Fascinating. There’s a ground breaking movie, documentary perhaps, just waiting to be made here. My immediate thought: Is the human race an AIG gone rogue? Suddenly I’m thinking of a comedy/drama film, but one that could explore both hazards and great possibilities.
Great topic. Sure raises a lot of questions though. Humanity's experience with regulating complexity is mixed. I can think of four major initiatives in the last century where regulating things with very high risk association. The ones that come to mind are Finance, Light Water Reactors, WMD and Compartmentalizing National Security Information. Each could be a topic unto itself as it relates to AI, ML & AGI. You are in a great and interesting field.