As automated decision-making and data-driven technologies become prevalent, understanding their capabilities, limitations, and associated risks requires an analysis of full sociotechnical systems.
Sociotechnical risk analysis in complex systems provides clear lessons for the design and evaluation of AI systems, shifting focus from merely reliable or 'responsibly designed' components to understanding risks at a systems level.
Human-made catastrophes like Chernobyl, Three Mile Island, Fukushima, Bhopal, and the Challenger disaster have been studied for decades due to their severity.
A common misconception is that such events are freak accidents stemming from unforeseeable interactions in complex systems. Closer examination reveals that the risks and hazards were well-known but not acted upon due to social, political, and economic factors.
We outline several areas where AI development and use can benefit from these unlearned lessons: improved risk perception, communication, and analysis at the organizational level; traceability of requirements and responsibilities; and holistic approaches to responsibility and safety that incorporate social and organizational dynamics as primary engineering concerns.
For each area, we provide concrete unlearned lessons, exemplifying how they led to failures in past accidents and how they remain unaddressed in modern computing systems, particularly AI.