Explainable Artificial Intelligence and Machine Learning: A
reality rooted perspective
Frank Emmert-Streib1,2, Olli Yli-Harja2, and Matthias Dehmer3
1Predictive Society and Data Analytics Lab, Faculty of Information Technology and Communication
Sciences, Tampere University, Tampere, Finland ∗
2Institute of Biosciences and Medical Technology, Tampere University of Technology, Tampere, Finland
3Institute for Intelligent Production, Faculty for Management, University of Applied Sciences Upper
Austria, Steyr Campus, 4040 Steyr, Austria
January 26, 2020
Abstract
We are used to the availability of big data generated in nearly all fields of science as
a consequence of technological progress. However, the analysis of such data possess vast
challenges. One of these relates to the explainability of artificial intelligence (AI) or machine
learning methods. Currently, many of such methods are non-transparent with respect to
their working mechanism and for this reason are called black box models, most notably deep
learning methods. However, it has been realized that this constitutes severe problems for a
number of fields including the health sciences and criminal justice and arguments have been
brought forward in favor of an explainable AI. In this paper, we do not assume the usual
perspective presenting explainable AI as it should be, but rather we provide a discussion
what explainable AI can be. The difference is that we do not present wishful thinking but
reality grounded properties in relation to a scientific theory beyond physics.
1
Introduction
Artificial intelligence (AI) and machine learning (ML) have achieved great successes in a number of
different learning tasks including image recognition and speech processing [1–3]. However, many of
the best performing methods are too complex (abstract) prohibiting a straight forward explanation
of the obtained results in simple words. The reason therefore is that such methods process high-
dimensional input data in a non-linear and nested fashion to reach probabilistic decisions. This
convolutes a clear view, e.g., on what information in the input vector is actually needed to arrive
at certain decisions. As a result, such models are non-transparent or opaque and are typically
∗frank.emmert-streib@tuni.fi
1
regarded as black box models [4]. Importantly, not only deep neural networks (DNNs) are suffering
from this shortcoming but also support vector machines (SVMs), random forests (RFs) or ensemble
models (e.g. Adaboost) [5–8].
This black box character establishes problems for a number of fields. For instance, when making
decisions in a hospital about the treatment of patients or at the court about the sentencing of
a defendant, such decisions should be explainable [9, 10].
In an endeavor to address this issue
the field explainable AI (XAI) has recently re-emerged [11]. While previous work in this area
focused on specific problems of deep learning models, defining explainable AI or the taxonomy of
XAI [12–14], our approach presents a different perspective as follows. First, instead of describing AI
systems with desirable properties making them explainable, we present a reality rooted perspective
showing what XAI can deliver. Put simply, instead of presenting explainable AI as it should be
we show what explainable AI can be. Second, we derive thereof limitations of explainable AI.
Such limitations may be undesirable but they are natural and unavoidable. Third, we discuss
consequences of this for our way forward.
Our paper is organized as follows. In the next section, we briefly describe the current state
of explainable AI. Then we present different perspectives on learning methods and discuss the
definition of a scientific theory. This allows us to conclude some limitations of an even perfect
versions of explainable AI. Finally, we discuss reasons for the confusion about explainable AI and
present ways forward. The paper finishes with concluding remarks.
2 Current state of explainable AI
For our following discussion, it is important to know how explainable AI is currently defined. Put
simply, one would like to have explanations of internal decisions within an AI system that lead
to an external result (output). Such explanations should provide insight into the rationale the AI
uses to draw conclusions [15].
A more specific definition of explainable AI was proposed in [16].
Definition 1 (explainable AI). 1) produce more explainable models while maintaining a high level
of learning performance (e.g., prediction accuracy), and 2) enable humans to understand, appro-
priately trust, and effectively manage the emerging generation of artificially intelligent partners.
Another attempt of a definition of explainable AI is given in [13].
Definition 2 (explainable AI). Explainable Artificial Intelligence is a system that produces details
or reasons to make its functioning clear or easy to understand.
Furthermore,
it is argued that the goals of explainable AI are trustworthiness, causality,
2
transferability, informativeness, confidence, fairness, accessibility, interactivity and privacy aware-
ness [13, 17].
In general, there is agreement that an explainable AI system should not be opaque (or a
black box) that hides its functional mechanism. Also, if a system is not opaque and one can even
understand how inputs are mathematically mapped to outputs then the system is interpretable [15].
Taken together, this implies model transparency. The terms interpretability and explainability
(and sometimes comprehensibility) are frequently used synonymously although the ML community
seems to prefer the former while the AI community prefers the latter [14].
From a problem-oriented view, in [18] different types of interpretability, for instance perceptive
interpretability, interpretability via mathematical structure, data-driven interpretability or inter-
pretability by utilities, and explainability are discussed. They found that many journal papers in
the ML and AI community are algorithm-centric, whereas in medicine risk and responsibility re-
lated issues are more prevalent. Similar discussions for different interpretations of interpretability
can be found in [19].
Overall, at to this point one can conclude the following. First, there is no single definition of
Advertisement
explainable AI available that would be generally accepted but many descriptions are overlapping
with each other in the above discussed ways. Second, the characterizations of explainable AI
state what XAI should be. That means they form desirable properties of such an AI system
without deriving these from higher principles. Hence, these characterizations can be seen as
wishful thinking. Before we can formulate reality rooted attainable goals of explainable AI, we
need to discuss different perspectives on models and the general capabilities of a scientific theory.
3 Perspective of statistics
In statistics, one can distinguish between two main types of models. The first type, called infer-
ential or explanatory model, provides a causal explanation of the data generation process whereas
the second type, called predictive model, just produces forecasts [20, 21]. Certainly, an inferential
model is more informative (i.e. theory-like, see below) than a predictive model because also an
explanatory model can be used to make predictions but the predictive model does not provide
(causal) explanations for such predictions. An examples for an explanatory model is a causal
Bayesian network whereas a random forest is a prediction model. Due to the complementary
capabilities of predictive and inferential models they are coexisting next to each other and each is
useful in its own right.
The general problem for creating causal models from data is that their inference from observa-
tional data is very challenging requiring usually also experimental data (generated by perturbations
of the system). Currently, most data in the health and social sciences are observational data ob-
3
tained from merely observing the behavior of the system because performing experiments is either
not straight forward or ethically prohibited.
4 Perspective of artificial intelligence
In [22] it was argued that explainable AI is not a new field but has been already recognized and
discussed for expert systems in the 1980s. This is understandable considering that from about
the 1950s to the 1980s the dominant paradigm of AI was symbolic artificial intelligence (SAI) and
SAI used high-level and human-readable representations and rules for manipulating these, e.g.,
by using expert systems. Hence, not only the need for explainable systems has been realized but
some AI systems could also accomplish near optimal explainable goals due to the nature of SAI.
With the renewed interest in neural networks in recent years in the form of deep neural networks
(DNN), the question of explaining and interpreting models has been re-surfaced. One reason for
this is that deep neural networks, in contrast to methods used for SAI, are not symbol-based but
connectionist, i.e., they are learning (possibly high-dimensional) features from data and store them
in the weights of the network [23, 24]. Although, more powerful in practice the price payed for
this comes in form of a higher complexity of the representation used, which is no longer human-
readable. Recently, it has been argued that AI systems should not only solve pattern recognition
problems but provide causal models of the world that support explanation and understanding [25].
This demand connects directly to the statistics perspective because causal models are exploratory,
see above.
After clarifying how explainable models are understood by different communities we take a
step back to see what would be the ultimate goal achievable of an explainable AI. For this reason,
we discuss the meaning of a theory.
5 What is a scientific theory?
In science, the formal definition of a theory is difficult but commonly it refers to a comprehensive
explanation of a subfield of nature that is supported by a body of evidence [26–28]. In physics,
the term theory is generally associated with a mathematical framework derived from a set of
basic axioms or postulates which allows to generate experimentally testable predictions for such a
subfield of physics. Typically these systems are highly idealized, in that the theories describe only
certain aspects. Examples include classical field theory and quantum field theory. An important
aspect of a theory is that it is falsifiable [29]. That means experiments can be conducted for testing
the predictions made by a theory. As long as such experiments do not contradict the predictions
a theory is accepted, otherwise rejected.
4
With respect to the stringency with which theories have been quantitatively tested, theories in
physics, e.g., general relativity or quantum electrodynamics, are certainly what can be considered
the best scientific theories. This implies that such theories provide answers to all questions that can
be asked within the scope of the theory. Furthermore, the theory provides also an explanation of
the obtained results. However, these explanations do not come in the form of a natural language,
e.g., English, but are mathematical formulas. Hence, the mathematical formulas need to be
interpreted by means of a natural language as good as possible. This may seem as a minor issue
but the severity of this may be exemplified by the interpretation of quantum mechanics because
so far there is no generally accepted interpretation, although the Copenhagen interpretation is
the most popular one [30]. Interestingly, the reason for this is ascribed to personal philosophical
prejudice [30]. The latter point hints to the incompleteness of any natural language in interpreting
a mathematical formalism of quantum mechanics. For completeness, we would like to mention
that even in physics not everything is covered by the existing theories because so far there is no
theory unifying gravity and quantum mechanics [31].
6 Expected limitations of an explainable AI
From this discussion, we can conclude some limitations even a perfect version of an explainable
AI will have. Considering that essentially all applications of AI and ML are beyond physics, e.g.,
in biology, medicine and health care, industrial production or human behavior, one cannot expect
to have a simpler theory for any of these fields than what we have for physics. Hence, even the
interpretability of such a theory is expected to be more problematic than an interpretation of, e.g.,
quantum mechanics.
In order to make this point more clear let’s consider a specific example. Suppose a theory of
cancer would be known, in the sense of a theory in physics discussed above. Then this cancer theory
would be highly mathematical in nature which would not permit a simple one-to-one interpretation
in any natural language. Hence, only highly trained theoretical cancer mathematicians (in analogy
to theoretical physicists) would be able to derive and interpret meaningful statements from this
theory. Regardless of the potential success of such a cancer theory, this implies that medical doctors
- not trained as theoretical cancer mathematicians - could not understand nor interpret such a
theory properly and, hence, from their perspective the cancer theory would appear opaque or non-
Advertisement
explainable. Without our discussion, such a result would appear undesirable and unacceptable,
however, given the context we provided above this appears unavoidable and natural. A similar
discussion can be provided for any other field than cancer showing that even a perfect version
of an explainable AI theory would not be interpretable or explainable for managers or general
practitioners for natural reasons.
5
The next optimistic but less ambitious assumption would be to suppose AI could provide
a description for fields outside of physics. Interestingly, physicists realized already decades ago
that such an expansion of a theory beyond the boundaries of physics is very challenging. For
instance, severe problems encountered are due to the arising of emergent properties and non-
equilibrium systems [32,33]. For this reason, phenomena outside of physics are usually addressed by
"approaches" collectively named as complex adaptive systems (CAS) [34,35]. We used intentionally
the word "approaches" and not "theories" because the used models and the obtained descriptions
are qualitatively very different thereof. Whereas it is unquestionable that valuable insights have
been gained into complex phenomena from economy, society and sociology [36,37] a theory for such
fields is still absent and currently not in sight. Hence, it seems fair to conclude that even the most
powerful AI system of CAS would be far from being a theory and for this reason lack explanatory
capabilities. However, this translates directly into a principle incompleteness of questions such an
AI system could answer and reduces further what could be delivered by an explainable AI.
Finally, we come to the most realistic view on an AI system which views its purpose as a
system to analyze data. Depending on the AI system and the available data it is clear that the
understanding that can be obtained from such a system is even further limited in the answers that
can be provided as well as in the level of explanations it can give. This is also true if the AI system
would be based on a perfect method because the data represent only an incomplete sample of all
possible data and is as such inherently limited in the explanations it can provide translating in an
unavoidable uncertainty of statements about the population it studies.
In Fig. 1, we summarize the above discussion graphically. The shown coordinate systems
indicate the qualitative relation between the influence of the distance from a theory and the
comprehensiveness of the description (left) and the influence of the sample size on the uncertainty
of statements or explanations about the population (right). The yellow arrow on the left indicates
the distance from physical theories which corresponds also to the x-axis of the left coordinate
system. A similar meaning has the purple arrow on the right for the diameter of the random
sample and x-axis of the right coordinate system
7 Reasons for the confusion
It is interesting to ask how utopian, idealistic requirement for an explainable AI, as discussed in
Section 2 above, could be demanded when reality looks quite differently. We think the reason for
this is twofold. First, statistical models and machine learning methods have been introduced due to
the lack of general theories outside of physics because these allow a quantitative analysis studying
experimental evidence. Hence, even in this unsatisfactory situation, systematic and quantitative
approaches are available for extending our knowledge of complex phenomena. Second, in recent
6
Figure 1: An overview describing the limitations of explainable AI with respect to attainable
goals. Left: Different scientific fields are arranged according to their increasing complexity [32]
starting from the best (most comprehensive) theories of physics in the center. The further the
distance from these theories the less comprehensive are the models describing subjects of CAS
(left coordinate system). Right: Any AI system analyzes a random sample of data drawn from a
population. One source of uncertainty is provided by the sample size of the data (right coordinate
system) that translates directly into uncertain statements about the population.
years we have been experiencing a data surge which gives the impression that every research
question should start with "data". This led even to the establishment of a new field called data
science [38]. Taken together, this may have given the impression that AI and ML methods are
more powerful than physical theories because they can (approximately) reach areas - due to the
availability of methods and data - that are blocked for physics. However, these methods are
not intended as theories but merely as practical instruments to deal with complex phenomena.
Importantly, the questions that can be answered need to have answers that can be found within
the data. Every lack of such qualitative data, e.g., due to a limited sample size, translates directly
into a lack of answers and is for this reason an inherent uncertainty of any AI system.
8 Discussion
Having realized the limitations of explainable AI with respect to attainable goals, what does this
mean for the way to go forward? In the following, we present a discussion of practical remedies
that are based on the insights gained in the first part of our paper. We want to emphasize that
these remedies do not bring us back to the delusional view of an idealized explainable AI but
provide means to realistically formulate achievable goals.
An AI system may not need to be explainable in a natural language as long as its generalization
error does not exceed an acceptable level. An example for such a practice are nuclear power plants.
7
PhysicsTheoryBiologyEconomySociologyCASQM: quantum mechanicsQFT: quantum field theoryapplications of AImodelsrandom sample of datapopulationmethod[ ]⇒uncertain statementsabout populationdistance from theorycomprehensiveness of the descriptionsample sizeuncertainty CAS: complex adaptive systemsSince nuclear power plants are based on physical laws (see the discussion of quantum mechanics
above) the functioning mechanisms of such plants are not explainable to members of the regulatory
management or politicians responsible for governing energy politics. Nevertheless, nuclear power
plants have been operated since many decades contributing to our energy supply. In a similar way,
one could envision an operational policy for medicine, e.g., utilizing deep learning methods.
Deep learning may not be necessary to analyze a given data set. Nowadays, many people
are using deep learning methods because they seem to think such methods are needed without
exploring alternative approaches. However, in this way problems regarding the interpretability
and explainability of the results are encountered. While it may be possible that future deep
learning methods may be less opaque, if currently available methods solve the same problems in
a satisfactory way and do not suffer from such limitations, e.g., decision trees, they should be
preferred and used.
The similarity (or difference) of the predictiveness of AI systems needs to be quantified in an
explainable way. This point is related to the previous one because if one can quantitatively assesses
the similarity (or the difference) between two AI systems one can compare an explainable with a
non-explainable AI system to evaluate one benefit over the other. Hence, even if an explainable
Advertisement
AI system does not fully solve a given problem, e.g., compared to a deep learning approach, it
may be sufficient to use. Importantly, even when an AI system itself may not be explainable the
comparison of different systems can be understandable. This way the lack of an explainability
maybe compensatable. A challenge of such an approach is that a quantification should not only
be based on prediction errors [39] but an assessment of risk and utility [40]. This latter issue is
clear in a clinical context.
Partial insights into the interpretability and explainability of AI systems should be developed. It
may not be feasible to convert deep learning models into fully transparent systems, however, this
may be achievable in part. For instance, certain aspects of an analysis could be understandable
which are integrated to achieve the complete model. Given the fact that in general data science
problems present themselves as a process [41] there should be ample opportunity to identify
subproblems deemable as an explainable AI.
9 Conclusions
In this paper, we shed some light on the current state of explainable AI and derived limitations
with respect to attainable goals from clarifying different perspectives and the capabilities of well
tested physical theories. The main results can be summarized as follows:
1. An AI system does not constitute a theory but an instrument (a model) to analyze data.
⇒
The comprehensiveness and the explainability of an even perfect AI system are inherently
8
limited by the random sample of data used.
2. The more comprehensive (i.e. theory-like) an AI system becomes in predicting CAS the more
complex becomes its underlying mathematics.
There is no simple one-to-one translation
⇒
into a natural language to explain the results or the working mechanism.
3. The most powerful but opaque AI systems (e.g. deep learning) should not be preferred and
applied by default but a comparison to alternative explainable methods should be conducted
and differences should be quantified.
⇒
An explainable and quantifiable reason can be
derived by integrating prediction error, risk and utility for weighing the pros and cons for
each model.
We hope our results can contribute to formulating realistic goals for an explainable AI that
are also attainable [42].
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial
relationships that could be construed as a potential conflict of interest.
Author Contributions
All authors contributed to all aspects of the preparation and the writing of the manuscript.
References
[1] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep
convolutional neural networks. In Advances in neural information processing systems, pages
1097–1105, 2012.
[2] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation,
9(8):1735–1780, 1997.
[3] Stuart J Russell and Peter Norvig. Artificial intelligence: a modern approach. Malaysia;
Pearson Education Limited„ 2016.
[4] Brent Mittelstadt, Chris Russell, and Sandra Wachter. Explaining explanations in ai.
In
Proceedings of the conference on fairness, accountability, and transparency, pages 279–288.
ACM, 2019.
9
[5] Vladimir Naumovich Vapnik. The Nature of Statistical Learning Theory. Springer, 1995.
[6] L. Breiman. Random Forests. Machine Learning, 45:5–32, 2001.
[7] R.E. Schapire and Y. Freund. Boosting: Foundations and Algorithms. Adaptive Computation
and Machine Learning series. MIT Press, 2012.
[8] Zhi-Hua Zhou. Ensemble methods: foundations and algorithms. Chapman and Hall/CRC,
2012.
[9] Andreas Holzinger, Chris Biemann, Constantinos S Pattichis, and Douglas B Kell. What
do we need to build explainable ai systems for the medical domain?
arXiv preprint
arXiv:1712.09923, 2017.
[10] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions
and use interpretable models instead. Nature Machine Intelligence, 1(5):206–215, 2019.
[11] Feiyu Xu, Hans Uszkoreit, Yangzhou Du, Wei Fan, Dongyan Zhao, and Jun Zhu. Explainable
ai: a brief survey on history, research areas, approaches and challenges. In CCF International
Conference on Natural Language Processing and Chinese Computing, pages 563–574. Springer,
2019.
[12] Leilani H Gilpin, David Bau, Ben Z Yuan, Ayesha Bajwa, Michael Specter, and Lalana Kagal.
Explaining explanations: An overview of interpretability of machine learning. In 2018 IEEE
5th International Conference on data science and advanced analytics (DSAA), pages 80–89.
IEEE, 2018.
[13] Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham
Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Ben-
jamins, et al. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities
and challenges toward responsible ai. arXiv preprint arXiv:1910.10045, 2019.
[14] Amina Adadi and Mohammed Berrada. Peeking inside the black-box: A survey on Explain-
able Artificial Intelligence (XAI). IEEE Access, 6:52138–52160, 2018.
[15] Derek Doran, Sarah Schulz, and Tarek R Besold. What does explainable AI really mean? A
new conceptualization of perspectives. arXiv preprint arXiv:1710.00794, 2017.
[16] David Gunning. Explainable artificial intelligence (XAI). Defense Advanced Research Projects
Advertisement
Agency (DARPA), nd Web, 2, 2017.
[17] Zachary C Lipton. The mythos of model interpretability. Queue, 16(3):30–57, 2018.
10
[18] Erico Tjoa and Cuntai Guan. A survey on explainable artificial intelligence (xai): Towards
medical xai. arXiv preprint arXiv:1907.07374, 2019.
[19] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and
Dino Pedreschi. A survey of methods for explaining black box models. ACM computing
surveys (CSUR), 51(5):93, 2019.
[20] Galit Shmueli et al. To explain or to predict? Statistical science, 25(3):289–310, 2010.
[21] Leo Breiman et al. Statistical modeling: The two cultures. Statistical science, 16(3):199–231,
2001.
[22] Randy Goebel, Ajay Chander, Katharina Holzinger, Freddy Lecue, Zeynep Akata, Simone
Stumpf, Peter Kieseberg, and Andreas Holzinger. Explainable ai: the new 42?
In Inter-
national Cross-Domain Conference for Machine Learning and Knowledge Extraction, pages
295–303. Springer, 2018.
[23] Jerome A Feldman and Dana H Ballard. Connectionist models and their properties. Cognitive
science, 6(3):205–254, 1982.
[24] Stephan I Gallant. Connectionist expert systems. Communications of the ACM, 31(2):152–
170, 1988.
[25] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building
machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
[26] Root Gorelick. What is theory? Ideas in Ecology and Evolution, 4, 2011.
[27] Hans Halvorson. Scientific theories. In The Oxford Handbook of Philosophy of Science.
[28] Patrick Suppes. What is a scientific theory? US Information Agency, Voice of America
Forum, 1964.
[29] K.R. Popper. The logic of scientific discovery. Basic Books, New York, 1959.
[30] Maximilian Schlosshauer, Johannes Kofler, and Anton Zeilinger. A snapshot of foundational
attitudes toward quantum mechanics. Studies in History and Philosophy of Science Part B:
Studies in History and Philosophy of Modern Physics, 44(3):222–230, 2013.
[31] Carlo Rovelli. Quantum gravity. Cambridge university press, 2004.
[32] Philip W Anderson. More is different. Science, 177(4047):393–396, 1972.
[33] F. Emmert-Streib. Aktive Computation in offenen Systemen. Lerndynamiken in biologischen
Systemen: Vom Netzwerk zum Organismus. PhD thesis, University of Bremen, 2003.
11
[34] H.G. Schuster. Complex Adaptive Systems. Scator Verlag, 2002.
[35] Simon Levin, Tasos Xepapadeas, Anne-Sophie Crépin, Jon Norberg, Aart De Zeeuw, Carl
Folke, Terry Hughes, Kenneth Arrow, Scott Barrett, Gretchen Daily, et al. Social-ecological
systems as complex adaptive systems: modeling and policy implications. Environment and
Development Economics, 18(2):111–132, 2013.
[36] Per Bak. How nature works:
the science of self-organized criticality. Springer Science &
Business Media, 2013.
[37] R. N. Mantegna and H. E. Stanley. Scaling behaviour in the dynamics of an economic index.
Nature, 1995.
[38] F Emmert-Streib and M Dehmer. Defining data science by a data-driven quantification of
the community. Machine Learning and Knowledge Extraction, 1(1):235–251, 2019.
[39] F Emmert-Streib, S Moutari, and M Dehmer. A comprehensive survey of error measures
for evaluating binary decision making in data science. Wiley Interdisciplinary Reviews: Data
Mining and Knowledge Discovery, page e1303, 2019.
[40] David Kreps. Notes on the Theory of Choice. Routledge, 2018.
[41] F. Emmert-Streib, S. Moutari, and M. Dehmer. The process of analyzing data is the emergent
feature of data science. Frontiers in Genetics, 7:12, 2016.
[42] Marvin Minsky. Steps toward artificial intelligence. Proceedings of the IRE, 49(1):8–30, 1961.
12
Sociology
Economy
Biology
Physics
Theory
QM: quantum mechanics
QFT: quantum field theory
CAS: complex adaptive systems
population
uncertainty
CAS
applications of AI
sample size
comprehensiveness of the
description
models
random sample of data
distance from theory
method[ ]
uncertain statements
about population
⇒