Cause and Effect Analysis by AI?
Great analysis. Thanks Greg!
Confusing correlation with causation is a long lasting and widely spread miconception of reasoning, not only of AI but of humans themselves. It took centuries to develop methodologies to deal with these difficulties and we can be proud that we found a solution. It is the current scientific methodology which roughly contains the following elements throughout almost all sciences: observation, theory formation, generation of hypotheses, experimation (with variation of suspected causal factors), evaluation, and interpretation. For some time now, this methodology is also applied to business strategy and product development. Thanks to Steve Blank and Eric Ries for making the concept of Lean Startup public.
AI has gained enormous power in complex data analytics and predictions. I suppose that it will take a long time until AI reaches a status where it can perform causal reasoning on its own. Algorithms alone, like Judea Pearl’s causal calculus will not be enough. Maybe that is not even a reasonable aim. In detecting cause and effect relations humans and AI can form an unbeatable team. Some ideas how this could work:
- Observation: rich data collections can be a starting point and AI is an effective tool to analyse them. What we get are at least correlations and pattern recognition.
- Theory formation (could even start before the observation): due to their broad understanding of the world and how it works, humans are still better in forming a reasonable theory. Maybe one day AI could deliver a set of various proposals for a theory that humans could assess and refine.
- Generation of hypotheses: they are deducted from the defined theory and deliver predictions that can be tested. Like theory formation this is still a task that humans solve better. But it is absolutely conceivable that machines could assist based on presumed correlations, supposed confounding factors and counterfactuals.
- Experimentation: designing and performing experiments can already be done in high quality and high numbers by AI, as long as all that is needed is digital. The thousands of optimization experiments that Amazon, Facebook, Alibaba, and others perform every day, are good examples. As soon as physical object or people have to be involved in an experiment, humans will still be the leading part of experimentation.
- Evaluation: a great task for AI . It can analyse tons of data in seconds.
- Intepretation: I would consider this a privilege for humans. Again, it requires a broad understanding of the world, broader than AI has at the moment.
Generally speaking, moving towards smarter machines might only be one side of the story. In the end, it is about moving towards smarter collaboration of machines and humans. AI and humans can form strong and effective teams that together can gain deeper insights than one of the parties could on its own.
