Understanding Causal Discovery by Michael Haft

Could you give us an overview of causal discovery?

Causal discovery aims to understand cause and effect relationships from observational data, essentially, understanding why something happens rather than just what will happen. While predictive models forecast outcomes, causal discovery reveals the underlying mechanisms. This distinction is crucial for artificial intelligence: only by understanding causal relationships can we intervene intelligently to achieve our goals.

Are ‘causal discovery’ and ‘causal inference’ interchangeable terms?

They’re often used interchangeably, but there’s a subtle distinction. Causal discovery identifies the causal graph – the directed acyclic graph showing causal structure. Causal inference uses this discovered structure to compute intervention effects: ‘What happens if I do X?’ Discovery reveals the structure; inference applies it.

For readers with backgrounds in statistics or machine learning, how do you move from data to actual causal relationships rather than just correlations?

That’s the core challenge.

Traditionally, we prove causation through experiments – controlling variables to isolate specific effects. Randomised control trials in medicine exemplify this approach. But causal discovery works differently: we extract causal relationships from purely observational data.

The problem is that observational data contains countless correlations, most of which are meaningless from a causal perspective. Consider grey hair and glasses: they’re correlated because age causes both. For prediction, this correlation is useful – if someone wears glasses, you might predict grey hair. But removing glasses won’t prevent grey hair because there’s no causal relationship.

Age is what we call a ‘confounder’ – a common cause explaining the correlation. Once we account for age, we can rule out glasses as a cause of grey hair. This is why causal discovery requires comprehensive data – we must identify and account for all potential confounders. When a correlation persists even after controlling for all observable factors, it becomes a strong candidate for a genuine causal relationship.

The key difference? Predictive modelling needs only a few correlated variables to forecast outcomes, while causal discovery demands comprehensive information to distinguish true causes from spurious correlations.

“I predict causality will become central to AI in the coming years.”

You’ve mentioned predictive modelling several times. How does causal discovery integrate with AI?

Gartner recently coined the term ‘causal AI’, which is apt: while prediction tells us what might happen, intelligent intervention requires understanding cause and effect. You can’t achieve goals without knowing what causally drives them.

I predict causality will become central to AI in the coming years.

Predictive models and large language models excel at forecasting, predicting the next word, for instance – but planning effective interventions demands causal knowledge. This is where causal discovery becomes essential.

How does this integration work technically?

It’s a two-step process. First, we perform causal discovery: scanning data, identifying confounders, and determining likely causal relationships. Unlike predictive modelling, which needs only direct causes, causal discovery maps entire causal pathways – indirect effects and complete graphs showing how variables interconnect.

Second, we integrate this knowledge into operational processes. In our industrial projects, we mine data for causal relationships, then use the insights to optimise processes and plan interventions.

Data completeness seems crucial. Can you explain the role of data architecture and complex data objects?

Our causal discovery algorithms are very powerful, but they depend on what we call ‘object analytics’, a technology I believe represents the next paradigm shift in data science.

Understanding causality requires comprehensive information. Take healthcare: electronic health records typically contain 150-200 tables covering diagnoses, prescriptions, procedures, lab values, and more. That creates a fundamental gap: traditional machine learning requires flat tables with rows representing patients. But flat tables, regardless of dimensions, cannot capture a patient’s complete picture.

Patients aren’t flat tables; they’re complex entities with billions of events across 150+ interconnected tables. Object analytics bridges this gap. We transform complex relational schemas into an object analytics database that efficiently handles analytical operations on this comprehensive data.

Our causal discovery algorithms leverage this holistic view, examining objects from millions of perspectives simultaneously. Rather than manually feature engineering, we scan for millions of potential causal factors and confounders. Only when a correlation survives this exhaustive search and we cannot explain it through any confounder despite evaluating millions of alternatives can we justify claiming a direct causal relationship.

This deep search capability, powered by object analytics, is what allows us to confidently exclude confounders and identify genuine causation.

You’ve developed these concepts into commercial products. Can you tell us about that journey?

Xplain Data is focused on product licensing rather than services. We offer both object analytics and causal discovery technology, and interestingly, many customers use our object analytics database independently of the causal discovery algorithms.

AOK Nordwest, a German public health insurer, exemplifies this. Despite having SAP HANA, one of the fastest relational databases available, they use our object analytics database because it enables holistic patient exploration. With 3 million patients and three to five billion events covering diagnoses, prescriptions, and treatments, they can examine patient journeys. This involves tracking progression from one diagnosis to the next, intervening medications, without requiring predictive modelling or causal discovery.

The object analytics database solves a fundamental problem: traditional databases and machine learning algorithms require manually condensed flat tables, inevitably overlooking features. Our technology eliminates this bottleneck.

“Once you start asking ‘why’ repeatedly, you realise predictive models are insufficient and we need to understand cause and effect from observational data.”

Our broader vision is transforming how data science operates. We believe the field will shift from flat tables to holistic objects, where algorithms analyse complete entities rather than simplified rows.

What business questions can causal methods answer that traditional analytics can’t?

Let me share two examples. In manufacturing, we analysed cylinder head production where nearly 10% failed final leakage testing. Each production step – drilling, assembly, washing, testing – generates data across 20-30 tables describing the complete manufacturing process.

We discovered the leakage test device produced false negatives while parts were cold. The manufacturer had been discarding perfectly good cylinder heads. Replacing the expensive test device wasn’t feasible, but our algorithms revealed the complete causal pathway: parts emerged hot from washing, then cooled during queue times before testing, triggering the false negatives.

This enabled an intelligent intervention: the washing machine now monitors the queue and stops washing when parts accumulate, ensuring they enter testing at the correct temperature. This solution required understanding the entire causal chain, not just direct causes. We offer similar pre-configured solutions for printed circuit board assembly lines.

In healthcare, we’re analysing the MIMIC dataset which is 500,000 hospital cases covering 300,000 patients, focusing on pressure injuries: wounds that develop when immobile patients lie in one position too long. We’re identifying interventions to prevent these injuries, with results that promise both improved patient care and significant cost savings.

When did you personally shift from predictive modelling to causal thinking? What triggered that transition?

Curiosity, fundamentally. I’m driven to understand why things happen.

Once you start asking “why” repeatedly, you realise predictive models are insufficient and we need to understand cause and effect from observational data.

This matters because 99% of existing data is observational: CRM systems, e-commerce, industry, healthcare, data collected non-experimentally. Only a fraction comes from controlled conditions. Randomised control trials are expensive; if we can extract causal intelligence simply by observing systems, that’s transformative. This opportunity motivated my direction.

Where do traditional methods still outperform causal approaches, or at least work equally with fewer resources?

Predictive modelling remains essential. While causal models inherently enable prediction, that’s not their primary purpose. We won’t replace predictive modelling entirely. Causal discovery demands comprehensive information, which often isn’t available. Without it, results require careful interpretation.

Predictive modelling is simpler. It needs less data, and any information correlated with the target suffices. These approaches are complementary. Prediction typically comes first; understanding the causes behind those predictions follows.

What does it take for companies to become ‘causally mature’? Which roles need involvement?

Education comes first. Understanding the distinction between prediction and causal discovery isn’t trivial. The data science world is so prediction-focused that many practitioners don’t perceive this gap. We need greater visibility around why prediction is just the first step, and understanding ‘why’ must follow.

Data availability presents another challenge. Companies must record and make available the comprehensive data causal discovery requires.

Who are your typical stakeholders – data science teams or business leaders?

Both, though it usually starts with the business facing concrete problems.

In manufacturing, quality managers need low failure rates and high-quality products. They have the business imperative.

However, I hope data science departments increasingly drive adoption. While data science capabilities are growing, awareness remains limited. Perhaps only 50% of data scientists have heard of causal AI, and only 20% possess an in-depth understanding.

We’re addressing this gap through our forthcoming community edition, aiming to penetrate the data science community so these technologies advance from both business needs and data science innovation.

What misconceptions do you typically encounter when introducing these methods?

The biggest challenge is shifting mindsets. Data scientists often have predictive models running and struggle initially to grasp why prediction alone is insufficient. Understanding the difference usually requires working with the technology itself – typically two or three iterations.

This applies to both concepts: first, working with holistic objects rather than flat tables, and second, discovering complete causal pathways.

In the cylinder head example, the crucial insight was the pathway from washing machine to false negative results – something predictive models don’t reveal.

We’ve had sceptical data scientists who, once they started working with holistic objects instead of flattening data into tables, refuse to go back. The technology changes how you work with real-world data.

Enterprise datasets often have quality issues. Does causal discovery require pristine data?

Higher quality helps, but imperfect data shouldn’t prevent starting. We frequently hear “We must improve data quality first”. I disagree: quality problems often emerge only when analysing relationships between sub-objects. In healthcare, issues become apparent when comparing prescriptions against diagnoses.

Data analysis reveals where quality improvements are needed. It’s iterative: load data immediately to gain transparency, identify weaknesses, then progressively improve quality.

What role does causal AI play in developing more explainable AI systems?

Explainability and causality are naturally linked. True explanation requires cause and effect. People increasingly ask not just “what will happen?” but “why?” Black-box predictions that simply state probabilities inevitably prompt the question “why?”, especially when planning interventions.

The ‘why’ question will become central to data science and AI. Answering it requires moving beyond flat tables to operate on real-world data structures. I envision open interfaces enabling diverse algorithms, not just our causal discovery tools, to work with complex, realistic data.

How do you see causal AI integrating with current systems?

I envision future AI systems that don’t merely rephrase web text but draw causal conclusions. Imagine asking “How do I improve my health?” and receiving answers grounded in cause-and-effect understanding, not just pattern matching.

In medicine specifically, I foresee doctors supported by causal AI systems accessing empirical knowledge from hundreds of millions of patients. This missing layer – transforming observational data into actionable causal insights – will emerge over the next two to five years. For healthcare, particularly, this represents the key to more intelligent and cost-efficient systems.

What’s your vision for causal AI in the coming years?

I hope causal AI becomes central to artificial intelligence. Currently, if you ask ChatGPT how to avoid grey hair, it rephrases web content without truly understanding causation. But I imagine future AI systems accessing observational data – particularly the 60 million patient electronic health records in Germany – to answer causal questions: What drives breast cancer? How do we prevent pressure ulcers in specific patients?

This is causal AI’s promise: using causal knowledge to treat patients effectively, provide early disease alerts, recommend preventive actions, and discover new relationships. Germany has approximately 10,000 pharmaceutical products whose causal side effects remain unknown, along with potentially beneficial effects for other conditions. Discovering this knowledge and integrating it into healthcare processes represents causal AI’s transformative potential.

What resources would you recommend for readers wanting to learn more?

We’re publishing research on pressure ulcers at ISPOR, one of healthcare’s largest conferences. Our website offers educational materials, explainer videos, and a ‘getting started’ guide.

We’re launching a community edition with extensive documentation and two interfaces: a browser-based UI for statistical exploration and a Python interface. For the MIMIC community, we provide a ready-to-go object analytics model – no data loading or object building required.

Just open your browser and start exploring hospital data.

We’re also releasing a pre-configured healthcare object analytics model. Since healthcare data structures are globally comparable – patients as root objects with sub-objects for prescriptions, diagnoses, hospital cases, and procedures – companies can load their patient data without configuring the model themselves.

“The ‘why’ question will become central to data science and AI. Answering it requires moving beyond flat tables to operate on real-world data structures.”