You Can't Spell "Explainable" without AI
I came across this recent article entitled “A Global South Perspective on Explainable AI” from the Carnegie Endowment for Peace about the need for explainability of artificial intelligence. Explainability in AI is about making the outputs and operations of AI systems as clear and understandable as possible to humans.
Explainability in artificial intelligence (AI) refers to the extent to which the internal mechanisms and outputs of an AI system can be understood by humans. This concept is crucial for validating and trusting AI systems, particularly in sensitive and impactful areas such as healthcare, finance, and legal applications. Here are several definitions and aspects of explainability in AI:
Transparency: This involves the ability to understand how an AI model works internally. For simpler models, such as linear regressions or decision trees, transparency is naturally higher because their decision-making processes are straightforward and can be easily examined. For more complex models like deep neural networks, transparency is much lower due to the layers of computations and non-linear interactions.
Interpretability: Closely related to transparency, interpretability is the degree to which a human can understand the cause of a decision made by an AI system. This might not necessarily involve understanding the entire model (as in transparency), but at least understanding the rationale behind specific decisions. Techniques that enhance interpretability include feature importance scores, decision rules, and instance-level explanations.
Comprehensibility: This refers to the ability of different stakeholders to comprehend the explanations provided by an AI system. Comprehensibility can depend on the background and expertise of the users, implying that explanations should be tailored to their level of technical understanding.
Traceability: This aspect involves the ability to trace the AI system's decisions back to the data and the processes that generated those decisions. Traceability helps in auditing AI systems and is crucial for applications where accountability is necessary.
Justifiability: Refers to the degree to which the decisions of an AI system can be justified with respect to established norms or laws. This is particularly important in regulatory environments, where decisions need to align with legal and ethical standards.
Fidelity: Represents how accurately the explanations reflect the true reasoning process of the model. High-fidelity explanations accurately represent the operations and decisions of the AI system, whereas low-fidelity explanations might simplify or approximate the process for the sake of understandability.
When discussing the lack of explainability of AI in Africa, there is the added dimension that most commercially available AI was built in the Global North.
It is essential to recognize that several African countries, like in other regions of the Global South, often do not have the extensive resources required to develop advanced AI systems and, therefore, rely significantly on AI software created by more technologically advanced countries in the Global North. This dynamic places African nations in a consumer position, using AI tools whose development contexts do not necessarily align with the nuances of their own local cultural, ethical, and social traditions. Western countries, recognizing the untapped potential of African markets, are keen to supply AI technologies, an arrangement that implicitly encourages a form of technological dependence. Therefore, a dialogue between African countries and AI developers in the Global North must take place to help promote a shared understanding and joint contributions to what constitutes explainable AI.
The author describes some of interviews he conducted of Africans using AI, including Kenyan farmers from Kiambu County, which borders Nairobi, who used “machine vision and image recognition software to detect diseases in cows and suggest treatment options. These farmers (with help from technical operators) uploaded pictures of their cows directly from their farms to a mobile platform, which provided them with disease analysis and treatment suggestions.” Some of the farmers, however, lacked trust in the AI because no one could explain to them how the model was able to diagnose their cows; e.g., “some of the farmers in Kenya expressed that they had since grown keenly interested in knowing about the operations underpinning the “magical” image-parsing algorithms they relied on, so they could perhaps elicit some rational explanation to fully trust it, but they had no way of drawing out such explanations.”
As the article goes on to explain, much of the problem facing these Kenyan farmers was not necessarily lack of explainability, but lack of locally-sourced and relevant data feeding into the AI models they were using. Basically, the AI model had been trained on cows common in the Global North, which were bred for local agricultural practices and market demands. Because the AI model had not encountered in its training data the breeds of cows common in Kenya the model did a poor job correctly identifying the health of the animals.
[S]ome of the cattle herders in Kenya had cows of the Boran and Sahiwal breeds. They complained about the image vision machine regularly misdiagnosing indigenous breeds of cows. The image recognition software often labelled them as undernourished because they were petite with a lean build (a natural adaptation to their environment as they often walked long distances for grazing and required less feed intake). The so-called optimal weight template displayed on the platform was like the Western Holstein, Angus, or Hereford breeds, which typically are larger and selectively bred for meat or dairy. Therefore, the insufficient representativeness of the data used to build these AI models may be partly responsible for why these systems did not effectively capture the diversity or complexity of the real-world scenarios that they were expected to handle. Because of such gaps, outputs may become less transparent and more challenging to explain. For the herders, trust in the model was undermined by a lack of clarity about what was causing such misdiagnoses. If the model were better set up for explainability, they could perhaps dig in to understand how much of the diagnosis was likely due to actual undernourishment, and how much was a result of mislabeling of weights. If the image vision software was built to explain its functionality better, that could allow the cattle herders to trust its predictions more, but as it was, they had a sense that the model was not adapted to meet their local context, yet they had no real way to confirm if that was the key issue. (emphasis added).
This is where the author expands on his idea to deploy “humans serving as AI explainers akin to griots or midwives, who can provide culturally contextualized and understandable explanations for this technology.” The article describes a NGO working in Tanzania that utilized medical AI technology to perform ultrasounds on pregnant women. “[W]hen the software interpreted fetal measurements and indicated potential concerns but could not explain the basis for its conclusions, or when different assessments in successive ultrasounds were made without explanations for these changes, the AI system left some women worried or distrusting its accuracy.” The staff at the NGO interacted with the developer of the AI technology to better understand why the model was generating certain outputs. When the staff was able to explain these outcomes, the women became more accepting of the technology.
Unfortunately, while it may solve an immediate problem, I don’t see how “AI Midwives” is a scalable solution in the long-term. Instead, facilitating the creation in Africa of AI solutions tailored to the African experience is the optimal path forward. That is, if the data on which these AI models is built is local and relevant, then the outputs should be more readily understandable and trustworthy than AI solutions developed in the Global North on contextually irrelevant data.