Explainations can comprehend three other aspects, not easy to define and measure, but extremely important to get, especially from the human operator point of view. They are:
– “Comprehensibility“, which regards the measure of the size of the explaination (number of features with a non-zero weight in a linear model, number of decision rules, …) or the test on how well people can predict the behavior of the ML model from the explanations.
– “Certainty“, which is about the fact that many AI models give predictions without a statement about their level of confidence that the prediction is correct.
– “Degree of Importance“, which considers if every part of an explaination counts at the same way. So, for example, if a decision rule is generated as an explaination for an individual prediction, is it clear which of the conditions of the rule was the most important?
– “Novelty” refers to the following fact: if the explaination takes into account data coming from a “new” region, far from the distribution of training data. The concept of novelty is related to the concept of certainty. The higher the novelty, the more likely it is that the model will have low certainty due to lack of data.
For more details on these factors, see (Koopman and Wagner, 2016), (Salay et al., 2017), (Hatani et al., 2020), (Ghahramani et al., 2015).
Main Question
Are all the key-factors, contributing to make the explainations understandable by humans, taken into account? (Shneiderman, 2020)
Sub-Questions
- Can the explaination approximate the prediction of the black box model sufficiently?
- Are their strong and weak points highlighted?
- Can their outputs be interpreted?
- Is the comprehensibility of the used features present in the explaination?
- Are there requirements to design the model, taking into account the later comprehensibility?
- Does the explaination reflect the certainty of the ML model?
- Does the explaination reflect the importance of each part of the AI system?
- Can the ML model manage the new data instances that fall outside the distribution of the training data?
References
- Shneiderman, B. (2020) ‘Bridging the gap between ethics and Practice’, ACM Transactions on Interactive Intelligent Systems, 10(4), pp. 1–31. doi:https://doi.org/10.1145/3419764
- Koopman, P. and Wagner, M. (2016). ‘Challenges in Autonomous Vehicle Testing and Validation’. SAE International Journal of Transportation Safety, 4(1), pp.15–24. doi:https://doi.org/10.4271/2016-01-0128.
- Salay, R., Queiroz, R. and Czarnecki, K. (2017) ‘An Analysis of ISO 26262: Using Machine Learning Safely’. Automotive Software. doi:https://doi.org/10.48550/arXiv.1709.02435.
- Hatani, F. (2019). ‘Artificial Intelligence in Japan: Policy, Prospects, and Obstacles in Automotive Industry’. Future of Business and Finance, pp.211–226. doi:https://doi.org/10.1007/978-981-15-0327-6_15.
- Ghahramani, Z. (2015) ‘Probabilistic machine learning and artificial intelligence’. Nature, 521(7553), pp.452–459. doi:https://doi.org/10.1038/nature14541.