-
Building Machines that Learn and Think with People
Authors:
Katherine M. Collins,
Ilia Sucholutsky,
Umang Bhatt,
Kartik Chandra,
Lionel Wong,
Mina Lee,
Cedegao E. Zhang,
Tan Zhi-Xuan,
Mark Ho,
Vikash Mansinghka,
Adrian Weller,
Joshua B. Tenenbaum,
Thomas L. Griffiths
Abstract:
What do we want from machine intelligence? We envision machines that are not just tools for thought, but partners in thought: reasonable, insightful, knowledgeable, reliable, and trustworthy systems that think with us. Current artificial intelligence (AI) systems satisfy some of these criteria, some of the time. In this Perspective, we show how the science of collaborative cognition can be put to…
▽ More
What do we want from machine intelligence? We envision machines that are not just tools for thought, but partners in thought: reasonable, insightful, knowledgeable, reliable, and trustworthy systems that think with us. Current artificial intelligence (AI) systems satisfy some of these criteria, some of the time. In this Perspective, we show how the science of collaborative cognition can be put to work to engineer systems that really can be called ``thought partners,'' systems built to meet our expectations and complement our limitations. We lay out several modes of collaborative thought in which humans and AI thought partners can engage and propose desiderata for human-compatible thought partnerships. Drawing on motifs from computational cognitive science, we motivate an alternative scaling path for the design of thought partners and ecosystems around their use through a Bayesian lens, whereby the partners we construct actively build and reason over models of the human and world.
△ Less
Submitted 21 July, 2024;
originally announced August 2024.
-
Modulating Language Model Experiences through Frictions
Authors:
Katherine M. Collins,
Valerie Chen,
Ilia Sucholutsky,
Hannah Rose Kirk,
Malak Sadek,
Holli Sargeant,
Ameet Talwalkar,
Adrian Weller,
Umang Bhatt
Abstract:
Language models are transforming the ways that their users engage with the world. Despite impressive capabilities, over-consumption of language model outputs risks propagating unchecked errors in the short-term and damaging human capabilities for critical thinking in the long-term, particularly in knowledge-based tasks. How can we develop scaffolding around language models to curate more appropria…
▽ More
Language models are transforming the ways that their users engage with the world. Despite impressive capabilities, over-consumption of language model outputs risks propagating unchecked errors in the short-term and damaging human capabilities for critical thinking in the long-term, particularly in knowledge-based tasks. How can we develop scaffolding around language models to curate more appropriate use? We propose selective frictions for language model experiences, inspired by behavioral science interventions, to dampen misuse. Frictions involve small modifications to a user's experience, e.g., the addition of a button impeding model access and reminding a user of their expertise relative to the model. Through a user study with real humans, we observe shifts in user behavior from the imposition of a friction over LLMs in the context of a multi-topic question-answering task as a representative task that people may use LLMs for, e.g., in education and information retrieval. We find that frictions modulate over-reliance by driving down users' click rates while minimally affecting accuracy for those topics. Yet, frictions may have unintended effects. We find marked differences in users' click behaviors even on topics where frictions were not provisioned. Our contributions motivate further study of human-AI behavioral interaction to inform more effective and appropriate LLM use.
△ Less
Submitted 24 June, 2024;
originally announced July 2024.
-
Large Language Models Must Be Taught to Know What They Don't Know
Authors:
Sanyam Kapoor,
Nate Gruver,
Manley Roberts,
Katherine Collins,
Arka Pal,
Umang Bhatt,
Adrian Weller,
Samuel Dooley,
Micah Goldblum,
Andrew Gordon Wilson
Abstract:
When using large language models (LLMs) in high-stakes applications, we need to know when we can trust their predictions. Some works argue that prompting high-performance LLMs is sufficient to produce calibrated uncertainties, while others introduce sampling methods that can be prohibitively expensive. In this work, we first argue that prompting on its own is insufficient to achieve good calibrati…
▽ More
When using large language models (LLMs) in high-stakes applications, we need to know when we can trust their predictions. Some works argue that prompting high-performance LLMs is sufficient to produce calibrated uncertainties, while others introduce sampling methods that can be prohibitively expensive. In this work, we first argue that prompting on its own is insufficient to achieve good calibration and then show that fine-tuning on a small dataset of correct and incorrect answers can create an uncertainty estimate with good generalization and small computational overhead. We show that a thousand graded examples are sufficient to outperform baseline methods and that training through the features of a model is necessary for good performance and tractable for large open-source models when using LoRA. We also investigate the mechanisms that enable reliable LLM uncertainty estimation, finding that many models can be used as general-purpose uncertainty estimators, applicable not just to their own uncertainties but also the uncertainty of other models. Lastly, we show that uncertainty estimates inform human use of LLMs in human-AI collaborative settings through a user study.
△ Less
Submitted 12 June, 2024;
originally announced June 2024.
-
Representational Alignment Supports Effective Machine Teaching
Authors:
Ilia Sucholutsky,
Katherine M. Collins,
Maya Malaviya,
Nori Jacoby,
Weiyang Liu,
Theodore R. Sumers,
Michalis Korakakis,
Umang Bhatt,
Mark Ho,
Joshua B. Tenenbaum,
Brad Love,
Zachary A. Pardos,
Adrian Weller,
Thomas L. Griffiths
Abstract:
A good teacher should not only be knowledgeable; but should be able to communicate in a way that the student understands -- to share the student's representation of the world. In this work, we integrate insights from machine teaching and pragmatic communication with the burgeoning literature on representational alignment to characterize a utility curve defining a relationship between representatio…
▽ More
A good teacher should not only be knowledgeable; but should be able to communicate in a way that the student understands -- to share the student's representation of the world. In this work, we integrate insights from machine teaching and pragmatic communication with the burgeoning literature on representational alignment to characterize a utility curve defining a relationship between representational alignment and teacher capability for promoting student learning. To explore the characteristics of this utility curve, we design a supervised learning environment that disentangles representational alignment from teacher accuracy. We conduct extensive computational experiments with machines teaching machines, complemented by a series of experiments in which machines teach humans. Drawing on our findings that improved representational alignment with a student improves student learning outcomes (i.e., task accuracy), we design a classroom matching procedure that assigns students to teachers based on the utility curve. If we are to design effective machine teachers, it is not enough to build teachers that are accurate -- we want teachers that can align, representationally, to their students too.
△ Less
Submitted 6 June, 2024;
originally announced June 2024.
-
Empirical Analysis of EIP-3675: Miner Dynamics, Transaction Fees, and Transaction Time
Authors:
Umesh Bhatt,
Sarvesh Pandey
Abstract:
The Ethereum Improvement Proposal 3675 (EIP-3675) marks a significant shift, transitioning from a Proof of Work (PoW) to a Proof of Stake (PoS) consensus mechanism. This transition resulted in a staggering 99.95% decrease in energy consumption. However, the transition prompts two critical questions: (1). How does EIP-3675 affect miners' dynamics? and (2). How do users determine priority fees, cons…
▽ More
The Ethereum Improvement Proposal 3675 (EIP-3675) marks a significant shift, transitioning from a Proof of Work (PoW) to a Proof of Stake (PoS) consensus mechanism. This transition resulted in a staggering 99.95% decrease in energy consumption. However, the transition prompts two critical questions: (1). How does EIP-3675 affect miners' dynamics? and (2). How do users determine priority fees, considering that paying too little may cause delays or non-inclusion, yet paying too much wastes money with little to no benefits? To address the first question, we present a comprehensive empirical study examining EIP-3675's effect on miner dynamics (i.e., miner participation, distribution, and the degree of randomness in miner selection). Our findings reveal that the transition has encouraged broader participation of miners in block append operation, resulting in a larger pool of unique miners ($\approx50\times$ PoW), and the change in miner distribution with the increased number of unique small category miners ($\approx60\times$ PoW). However, there is an unintended consequence: a reduction in the miner selection randomness, which signifies the negative impact of the transition to PoS-Ethereum on network decentralization. Regarding the second question, we employed regression-based machine learning models; the Gradient Boosting Regressor performed best in predicting priority fees, while the K-Neighbours Regressor was worst.
△ Less
Submitted 2 August, 2024; v1 submitted 26 March, 2024;
originally announced March 2024.
-
When Should Algorithms Resign? A Proposal for AI Governance
Authors:
Umang Bhatt,
Holli Sargeant
Abstract:
Algorithmic resignation is a strategic approach for managing the use of artificial intelligence (AI) by embedding governance directly into AI systems. It involves deliberate and informed disengagement from AI, such as restricting access AI outputs or displaying performance disclaimers, in specific scenarios to aid the appropriate and effective use of AI. By integrating algorithmic resignation as a…
▽ More
Algorithmic resignation is a strategic approach for managing the use of artificial intelligence (AI) by embedding governance directly into AI systems. It involves deliberate and informed disengagement from AI, such as restricting access AI outputs or displaying performance disclaimers, in specific scenarios to aid the appropriate and effective use of AI. By integrating algorithmic resignation as a governance mechanism, organizations can better control when and how AI is used, balancing the benefits of automation with the need for human oversight.
△ Less
Submitted 16 July, 2024; v1 submitted 28 February, 2024;
originally announced February 2024.
-
Comparing Abstraction in Humans and Large Language Models Using Multimodal Serial Reproduction
Authors:
Sreejan Kumar,
Raja Marjieh,
Byron Zhang,
Declan Campbell,
Michael Y. Hu,
Umang Bhatt,
Brenden Lake,
Thomas L. Griffiths
Abstract:
Humans extract useful abstractions of the world from noisy sensory data. Serial reproduction allows us to study how people construe the world through a paradigm similar to the game of telephone, where one person observes a stimulus and reproduces it for the next to form a chain of reproductions. Past serial reproduction experiments typically employ a single sensory modality, but humans often commu…
▽ More
Humans extract useful abstractions of the world from noisy sensory data. Serial reproduction allows us to study how people construe the world through a paradigm similar to the game of telephone, where one person observes a stimulus and reproduces it for the next to form a chain of reproductions. Past serial reproduction experiments typically employ a single sensory modality, but humans often communicate abstractions of the world to each other through language. To investigate the effect language on the formation of abstractions, we implement a novel multimodal serial reproduction framework by asking people who receive a visual stimulus to reproduce it in a linguistic format, and vice versa. We ran unimodal and multimodal chains with both humans and GPT-4 and find that adding language as a modality has a larger effect on human reproductions than GPT-4's. This suggests human visual and linguistic representations are more dissociable than those of GPT-4.
△ Less
Submitted 5 February, 2024;
originally announced February 2024.
-
FeedbackLogs: Recording and Incorporating Stakeholder Feedback into Machine Learning Pipelines
Authors:
Matthew Barker,
Emma Kallina,
Dhananjay Ashok,
Katherine M. Collins,
Ashley Casovan,
Adrian Weller,
Ameet Talwalkar,
Valerie Chen,
Umang Bhatt
Abstract:
Even though machine learning (ML) pipelines affect an increasing array of stakeholders, there is little work on how input from stakeholders is recorded and incorporated. We propose FeedbackLogs, addenda to existing documentation of ML pipelines, to track the input of multiple stakeholders. Each log records important details about the feedback collection process, the feedback itself, and how the fe…
▽ More
Even though machine learning (ML) pipelines affect an increasing array of stakeholders, there is little work on how input from stakeholders is recorded and incorporated. We propose FeedbackLogs, addenda to existing documentation of ML pipelines, to track the input of multiple stakeholders. Each log records important details about the feedback collection process, the feedback itself, and how the feedback is used to update the ML pipeline. In this paper, we introduce and formalise a process for collecting a FeedbackLog. We also provide concrete use cases where FeedbackLogs can be employed as evidence for algorithmic auditing and as a tool to record updates based on stakeholder feedback.
△ Less
Submitted 28 July, 2023;
originally announced July 2023.
-
Selective Concept Models: Permitting Stakeholder Customisation at Test-Time
Authors:
Matthew Barker,
Katherine M. Collins,
Krishnamurthy Dvijotham,
Adrian Weller,
Umang Bhatt
Abstract:
Concept-based models perform prediction using a set of concepts that are interpretable to stakeholders. However, such models often involve a fixed, large number of concepts, which may place a substantial cognitive load on stakeholders. We propose Selective COncept Models (SCOMs) which make predictions using only a subset of concepts and can be customised by stakeholders at test-time according to t…
▽ More
Concept-based models perform prediction using a set of concepts that are interpretable to stakeholders. However, such models often involve a fixed, large number of concepts, which may place a substantial cognitive load on stakeholders. We propose Selective COncept Models (SCOMs) which make predictions using only a subset of concepts and can be customised by stakeholders at test-time according to their preferences. We show that SCOMs only require a fraction of the total concepts to achieve optimal accuracy on multiple real-world datasets. Further, we collect and release a new dataset, CUB-Sel, consisting of human concept set selections for 900 bird images from the popular CUB dataset. Using CUB-Sel, we show that humans have unique individual preferences for the choice of concepts they prefer to reason about, and struggle to identify the most theoretically informative concepts. The customisation and concept selection provided by SCOM improves the efficiency of interpretation and intervention for stakeholders.
△ Less
Submitted 14 June, 2023;
originally announced June 2023.
-
Evaluating Language Models for Mathematics through Interactions
Authors:
Katherine M. Collins,
Albert Q. Jiang,
Simon Frieder,
Lionel Wong,
Miri Zilka,
Umang Bhatt,
Thomas Lukasiewicz,
Yuhuai Wu,
Joshua B. Tenenbaum,
William Hart,
Timothy Gowers,
Wenda Li,
Adrian Weller,
Mateja Jamnik
Abstract:
There is much excitement about the opportunity to harness the power of large language models (LLMs) when building problem-solving assistants. However, the standard methodology of evaluating LLMs relies on static pairs of inputs and outputs, and is insufficient for making an informed decision about which LLMs and under which assistive settings can they be sensibly used. Static assessment fails to a…
▽ More
There is much excitement about the opportunity to harness the power of large language models (LLMs) when building problem-solving assistants. However, the standard methodology of evaluating LLMs relies on static pairs of inputs and outputs, and is insufficient for making an informed decision about which LLMs and under which assistive settings can they be sensibly used. Static assessment fails to account for the essential interactive element in LLM deployment, and therefore limits how we understand language model capabilities. We introduce CheckMate, an adaptable prototype platform for humans to interact with and evaluate LLMs. We conduct a study with CheckMate to evaluate three language models (InstructGPT, ChatGPT, and GPT-4) as assistants in proving undergraduate-level mathematics, with a mixed cohort of participants from undergraduate students to professors of mathematics. We release the resulting interaction and rating dataset, MathConverse. By analysing MathConverse, we derive a taxonomy of human behaviours and uncover that despite a generally positive correlation, there are notable instances of divergence between correctness and perceived helpfulness in LLM generations, amongst other findings. Further, we garner a more granular understanding of GPT-4 mathematical problem-solving through a series of case studies, contributed by expert mathematicians. We conclude with actionable takeaways for ML practitioners and mathematicians: models that communicate uncertainty respond well to user corrections, and are more interpretable and concise may constitute better assistants. Interactive evaluation is a promising way to navigate the capability of these models; humans should be aware of language models' algebraic fallibility and discern where they are appropriate to use.
△ Less
Submitted 5 November, 2023; v1 submitted 2 June, 2023;
originally announced June 2023.
-
Learning Personalized Decision Support Policies
Authors:
Umang Bhatt,
Valerie Chen,
Katherine M. Collins,
Parameswaran Kamalaruban,
Emma Kallina,
Adrian Weller,
Ameet Talwalkar
Abstract:
Individual human decision-makers may benefit from different forms of support to improve decision outcomes, but when each form of support will yield better outcomes? In this work, we posit that personalizing access to decision support tools can be an effective mechanism for instantiating the appropriate use of AI assistance. Specifically, we propose the general problem of learning a decision suppor…
▽ More
Individual human decision-makers may benefit from different forms of support to improve decision outcomes, but when each form of support will yield better outcomes? In this work, we posit that personalizing access to decision support tools can be an effective mechanism for instantiating the appropriate use of AI assistance. Specifically, we propose the general problem of learning a decision support policy that, for a given input, chooses which form of support to provide to decision-makers for whom we initially have no prior information. We develop $\texttt{Modiste}$, an interactive tool to learn personalized decision support policies. $\texttt{Modiste}$ leverages stochastic contextual bandit techniques to personalize a decision support policy for each decision-maker and supports extensions to the multi-objective setting to account for auxiliary objectives like the cost of support. We find that personalized policies outperform offline policies, and, in the cost-aware setting, reduce the incurred cost with minimal degradation to performance. Our experiments include various realistic forms of support (e.g., expert consensus and predictions from a large language model) on vision and language tasks. Our human subject experiments validate our computational experiments, demonstrating that personalization can yield benefits in practice for real users, who interact with $\texttt{Modiste}$.
△ Less
Submitted 27 May, 2024; v1 submitted 13 April, 2023;
originally announced April 2023.
-
Human Uncertainty in Concept-Based AI Systems
Authors:
Katherine M. Collins,
Matthew Barker,
Mateo Espinosa Zarlenga,
Naveen Raman,
Umang Bhatt,
Mateja Jamnik,
Ilia Sucholutsky,
Adrian Weller,
Krishnamurthy Dvijotham
Abstract:
Placing a human in the loop may abate the risks of deploying AI systems in safety-critical settings (e.g., a clinician working with a medical AI system). However, mitigating risks arising from human error and uncertainty within such human-AI interactions is an important and understudied issue. In this work, we study human uncertainty in the context of concept-based models, a family of AI systems t…
▽ More
Placing a human in the loop may abate the risks of deploying AI systems in safety-critical settings (e.g., a clinician working with a medical AI system). However, mitigating risks arising from human error and uncertainty within such human-AI interactions is an important and understudied issue. In this work, we study human uncertainty in the context of concept-based models, a family of AI systems that enable human feedback via concept interventions where an expert intervenes on human-interpretable concepts relevant to the task. Prior work in this space often assumes that humans are oracles who are always certain and correct. Yet, real-world decision-making by humans is prone to occasional mistakes and uncertainty. We study how existing concept-based models deal with uncertain interventions from humans using two novel datasets: UMNIST, a visual dataset with controlled simulated uncertainty based on the MNIST dataset, and CUB-S, a relabeling of the popular CUB concept dataset with rich, densely-annotated soft labels from humans. We show that training with uncertain concept labels may help mitigate weaknesses of concept-based systems when handling uncertain interventions. These results allow us to identify several open challenges, which we argue can be tackled through future multidisciplinary research on building interactive uncertainty-aware systems. To facilitate further research, we release a new elicitation platform, UElic, to collect uncertain feedback from humans in collaborative prediction tasks.
△ Less
Submitted 22 March, 2023;
originally announced March 2023.
-
Harms from Increasingly Agentic Algorithmic Systems
Authors:
Alan Chan,
Rebecca Salganik,
Alva Markelius,
Chris Pang,
Nitarshan Rajkumar,
Dmitrii Krasheninnikov,
Lauro Langosco,
Zhonghao He,
Yawen Duan,
Micah Carroll,
Michelle Lin,
Alex Mayhew,
Katherine Collins,
Maryam Molamohammadi,
John Burden,
Wanru Zhao,
Shalaleh Rismani,
Konstantinos Voudouris,
Umang Bhatt,
Adrian Weller,
David Krueger,
Tegan Maharaj
Abstract:
Research in Fairness, Accountability, Transparency, and Ethics (FATE) has established many sources and forms of algorithmic harm, in domains as diverse as health care, finance, policing, and recommendations. Much work remains to be done to mitigate the serious harms of these systems, particularly those disproportionately affecting marginalized communities. Despite these ongoing harms, new systems…
▽ More
Research in Fairness, Accountability, Transparency, and Ethics (FATE) has established many sources and forms of algorithmic harm, in domains as diverse as health care, finance, policing, and recommendations. Much work remains to be done to mitigate the serious harms of these systems, particularly those disproportionately affecting marginalized communities. Despite these ongoing harms, new systems are being developed and deployed which threaten the perpetuation of the same harms and the creation of novel ones. In response, the FATE community has emphasized the importance of anticipating harms. Our work focuses on the anticipation of harms from increasingly agentic systems. Rather than providing a definition of agency as a binary property, we identify 4 key characteristics which, particularly in combination, tend to increase the agency of a given algorithmic system: underspecification, directness of impact, goal-directedness, and long-term planning. We also discuss important harms which arise from increasing agency -- notably, these include systemic and/or long-range impacts, often on marginalized stakeholders. We emphasize that recognizing agency of algorithmic systems does not absolve or shift the human responsibility for algorithmic harms. Rather, we use the term agency to highlight the increasingly evident fact that ML systems are not fully under human control. Our work explores increasingly agentic algorithmic systems in three parts. First, we explain the notion of an increase in agency for algorithmic systems in the context of diverse perspectives on agency across disciplines. Second, we argue for the need to anticipate harms from increasingly agentic systems. Third, we discuss important harms from increasingly agentic systems and ways forward for addressing them. We conclude by reflecting on implications of our work for anticipating algorithmic harms from emerging systems.
△ Less
Submitted 11 May, 2023; v1 submitted 20 February, 2023;
originally announced February 2023.
-
Dynamic Arctic weather variability and connectivity
Authors:
Jun Meng,
Jingfang Fan,
Uma S Bhatt,
Jürgen Kurths
Abstract:
The rapidly shrinking Arctic sea ice is changing weather patterns and disrupting the balance of nature. Dynamics of Arctic weather variability (WV) plays a crucial role in weather forecasting and is closely related to extreme weather events. Yet, assessing and quantifying the WV for both local Arctic regions and its planetary impacts under anthropogenic climate change is still unknown. Here, we de…
▽ More
The rapidly shrinking Arctic sea ice is changing weather patterns and disrupting the balance of nature. Dynamics of Arctic weather variability (WV) plays a crucial role in weather forecasting and is closely related to extreme weather events. Yet, assessing and quantifying the WV for both local Arctic regions and its planetary impacts under anthropogenic climate change is still unknown. Here, we develop a complexity-based approach to systematically evaluate and analyze the dynamic behaviour of WV. We reveal that the WV within and around the Arctic is statistically correlated to the Arctic Oscillation at the intraseasonal time scale. We further find that the variability of the daily Arctic sea ice is increasing due to its dramatic decline under a warming climate. Unstable Arctic weather conditions can disturb regional weather patterns through atmospheric teleconnection pathways, resulting in higher risk to human activities and greater weather forecast uncertainty. A multivariate climate network analysis reveals the existence of such teleconnections and implies a positive feedback loop between the Arctic and global weather instabilities. This enhances the mechanistic understanding of the influence of Arctic amplification on mid-latitude severe weather. Our framework provides a fresh perspective on the linkage of complexity science, WV and the Arctic.
△ Less
Submitted 3 February, 2023;
originally announced February 2023.
-
Towards Robust Metrics for Concept Representation Evaluation
Authors:
Mateo Espinosa Zarlenga,
Pietro Barbiero,
Zohreh Shams,
Dmitry Kazhdan,
Umang Bhatt,
Adrian Weller,
Mateja Jamnik
Abstract:
Recent work on interpretability has focused on concept-based explanations, where deep learning models are explained in terms of high-level units of information, referred to as concepts. Concept learning models, however, have been shown to be prone to encoding impurities in their representations, failing to fully capture meaningful features of their inputs. While concept learning lacks metrics to m…
▽ More
Recent work on interpretability has focused on concept-based explanations, where deep learning models are explained in terms of high-level units of information, referred to as concepts. Concept learning models, however, have been shown to be prone to encoding impurities in their representations, failing to fully capture meaningful features of their inputs. While concept learning lacks metrics to measure such phenomena, the field of disentanglement learning has explored the related notion of underlying factors of variation in the data, with plenty of metrics to measure the purity of such factors. In this paper, we show that such metrics are not appropriate for concept learning and propose novel metrics for evaluating the purity of concept representations in both approaches. We show the advantage of these metrics over existing ones and demonstrate their utility in evaluating the robustness of concept representations and interventions performed on them. In addition, we show their utility for benchmarking state-of-the-art methods from both families and find that, contrary to common assumptions, supervision alone may not be sufficient for pure concept representations.
△ Less
Submitted 24 January, 2023;
originally announced January 2023.
-
On the Informativeness of Supervision Signals
Authors:
Ilia Sucholutsky,
Ruairidh M. Battleday,
Katherine M. Collins,
Raja Marjieh,
Joshua C. Peterson,
Pulkit Singh,
Umang Bhatt,
Nori Jacoby,
Adrian Weller,
Thomas L. Griffiths
Abstract:
Supervised learning typically focuses on learning transferable representations from training examples annotated by humans. While rich annotations (like soft labels) carry more information than sparse annotations (like hard labels), they are also more expensive to collect. For example, while hard labels only provide information about the closest class an object belongs to (e.g., "this is a dog"), s…
▽ More
Supervised learning typically focuses on learning transferable representations from training examples annotated by humans. While rich annotations (like soft labels) carry more information than sparse annotations (like hard labels), they are also more expensive to collect. For example, while hard labels only provide information about the closest class an object belongs to (e.g., "this is a dog"), soft labels provide information about the object's relationship with multiple classes (e.g., "this is most likely a dog, but it could also be a wolf or a coyote"). We use information theory to compare how a number of commonly-used supervision signals contribute to representation-learning performance, as well as how their capacity is affected by factors such as the number of labels, classes, dimensions, and noise. Our framework provides theoretical justification for using hard labels in the big-data regime, but richer supervision signals for few-shot learning and out-of-distribution generalization. We validate these results empirically in a series of experiments with over 1 million crowdsourced image annotations and conduct a cost-benefit analysis to establish a tradeoff curve that enables users to optimize the cost of supervising representation learning on their own datasets.
△ Less
Submitted 4 July, 2023; v1 submitted 2 November, 2022;
originally announced November 2022.
-
Human-in-the-Loop Mixup
Authors:
Katherine M. Collins,
Umang Bhatt,
Weiyang Liu,
Vihari Piratla,
Ilia Sucholutsky,
Bradley Love,
Adrian Weller
Abstract:
Aligning model representations to humans has been found to improve robustness and generalization. However, such methods often focus on standard observational data. Synthetic data is proliferating and powering many advances in machine learning; yet, it is not always clear whether synthetic labels are perceptually aligned to humans -- rendering it likely model representations are not human aligned.…
▽ More
Aligning model representations to humans has been found to improve robustness and generalization. However, such methods often focus on standard observational data. Synthetic data is proliferating and powering many advances in machine learning; yet, it is not always clear whether synthetic labels are perceptually aligned to humans -- rendering it likely model representations are not human aligned. We focus on the synthetic data used in mixup: a powerful regularizer shown to improve model robustness, generalization, and calibration. We design a comprehensive series of elicitation interfaces, which we release as HILL MixE Suite, and recruit 159 participants to provide perceptual judgments along with their uncertainties, over mixup examples. We find that human perceptions do not consistently align with the labels traditionally used for synthetic points, and begin to demonstrate the applicability of these findings to potentially increase the reliability of downstream models, particularly when incorporating human uncertainty. We release all elicited judgments in a new data hub we call H-Mix.
△ Less
Submitted 30 July, 2023; v1 submitted 2 November, 2022;
originally announced November 2022.
-
Iterative Teaching by Data Hallucination
Authors:
Zeju Qiu,
Weiyang Liu,
Tim Z. Xiao,
Zhen Liu,
Umang Bhatt,
Yucen Luo,
Adrian Weller,
Bernhard Schölkopf
Abstract:
We consider the problem of iterative machine teaching, where a teacher sequentially provides examples based on the status of a learner under a discrete input space (i.e., a pool of finite samples), which greatly limits the teacher's capability. To address this issue, we study iterative teaching under a continuous input space where the input example (i.e., image) can be either generated by solving…
▽ More
We consider the problem of iterative machine teaching, where a teacher sequentially provides examples based on the status of a learner under a discrete input space (i.e., a pool of finite samples), which greatly limits the teacher's capability. To address this issue, we study iterative teaching under a continuous input space where the input example (i.e., image) can be either generated by solving an optimization problem or drawn directly from a continuous distribution. Specifically, we propose data hallucination teaching (DHT) where the teacher can generate input data intelligently based on labels, the learner's status and the target concept. We study a number of challenging teaching setups (e.g., linear/neural learners in omniscient and black-box settings). Extensive empirical results verify the effectiveness of DHT.
△ Less
Submitted 12 April, 2023; v1 submitted 31 October, 2022;
originally announced October 2022.
-
Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysis
Authors:
Yuxin Xiao,
Paul Pu Liang,
Umang Bhatt,
Willie Neiswanger,
Ruslan Salakhutdinov,
Louis-Philippe Morency
Abstract:
Pre-trained language models (PLMs) have gained increasing popularity due to their compelling prediction performance in diverse natural language processing (NLP) tasks. When formulating a PLM-based prediction pipeline for NLP tasks, it is also crucial for the pipeline to minimize the calibration error, especially in safety-critical applications. That is, the pipeline should reliably indicate when w…
▽ More
Pre-trained language models (PLMs) have gained increasing popularity due to their compelling prediction performance in diverse natural language processing (NLP) tasks. When formulating a PLM-based prediction pipeline for NLP tasks, it is also crucial for the pipeline to minimize the calibration error, especially in safety-critical applications. That is, the pipeline should reliably indicate when we can trust its predictions. In particular, there are various considerations behind the pipeline: (1) the choice and (2) the size of PLM, (3) the choice of uncertainty quantifier, (4) the choice of fine-tuning loss, and many more. Although prior work has looked into some of these considerations, they usually draw conclusions based on a limited scope of empirical studies. There still lacks a holistic analysis on how to compose a well-calibrated PLM-based prediction pipeline. To fill this void, we compare a wide range of popular options for each consideration based on three prevalent NLP classification tasks and the setting of domain shift. In response, we recommend the following: (1) use ELECTRA for PLM encoding, (2) use larger PLMs if possible, (3) use Temp Scaling as the uncertainty quantifier, and (4) use Focal Loss for fine-tuning.
△ Less
Submitted 14 October, 2022; v1 submitted 10 October, 2022;
originally announced October 2022.
-
Towards the Use of Saliency Maps for Explaining Low-Quality Electrocardiograms to End Users
Authors:
Ana Lucic,
Sheeraz Ahmad,
Amanda Furtado Brinhosa,
Vera Liao,
Himani Agrawal,
Umang Bhatt,
Krishnaram Kenthapadi,
Alice Xiang,
Maarten de Rijke,
Nicholas Drabowski
Abstract:
When using medical images for diagnosis, either by clinicians or artificial intelligence (AI) systems, it is important that the images are of high quality. When an image is of low quality, the medical exam that produced the image often needs to be redone. In telemedicine, a common problem is that the quality issue is only flagged once the patient has left the clinic, meaning they must return in or…
▽ More
When using medical images for diagnosis, either by clinicians or artificial intelligence (AI) systems, it is important that the images are of high quality. When an image is of low quality, the medical exam that produced the image often needs to be redone. In telemedicine, a common problem is that the quality issue is only flagged once the patient has left the clinic, meaning they must return in order to have the exam redone. This can be especially difficult for people living in remote regions, who make up a substantial portion of the patients at Portal Telemedicina, a digital healthcare organization based in Brazil. In this paper, we report on ongoing work regarding (i) the development of an AI system for flagging and explaining low-quality medical images in real-time, (ii) an interview study to understand the explanation needs of stakeholders using the AI system at OurCompany, and, (iii) a longitudinal user study design to examine the effect of including explanations on the workflow of the technicians in our clinics. To the best of our knowledge, this would be the first longitudinal study on evaluating the effects of XAI methods on end-users -- stakeholders that use AI systems but do not have AI-specific expertise. We welcome feedback and suggestions on our experimental setup.
△ Less
Submitted 6 July, 2022;
originally announced July 2022.
-
Eliciting and Learning with Soft Labels from Every Annotator
Authors:
Katherine M. Collins,
Umang Bhatt,
Adrian Weller
Abstract:
The labels used to train machine learning (ML) models are of paramount importance. Typically for ML classification tasks, datasets contain hard labels, yet learning using soft labels has been shown to yield benefits for model generalization, robustness, and calibration. Earlier work found success in forming soft labels from multiple annotators' hard labels; however, this approach may not converge…
▽ More
The labels used to train machine learning (ML) models are of paramount importance. Typically for ML classification tasks, datasets contain hard labels, yet learning using soft labels has been shown to yield benefits for model generalization, robustness, and calibration. Earlier work found success in forming soft labels from multiple annotators' hard labels; however, this approach may not converge to the best labels and necessitates many annotators, which can be expensive and inefficient. We focus on efficiently eliciting soft labels from individual annotators. We collect and release a dataset of soft labels (which we call CIFAR-10S) over the CIFAR-10 test set via a crowdsourcing study (N=248). We demonstrate that learning with our labels achieves comparable model performance to prior approaches while requiring far fewer annotators -- albeit with significant temporal costs per elicitation. Our elicitation methodology therefore shows nuanced promise in enabling practitioners to enjoy the benefits of improved model performance and reliability with fewer annotators, and serves as a guide for future dataset curators on the benefits of leveraging richer information, such as categorical uncertainty, from individual annotators.
△ Less
Submitted 29 August, 2022; v1 submitted 2 July, 2022;
originally announced July 2022.
-
Perspectives on Incorporating Expert Feedback into Model Updates
Authors:
Valerie Chen,
Umang Bhatt,
Hoda Heidari,
Adrian Weller,
Ameet Talwalkar
Abstract:
Machine learning (ML) practitioners are increasingly tasked with developing models that are aligned with non-technical experts' values and goals. However, there has been insufficient consideration on how practitioners should translate domain expertise into ML updates. In this paper, we consider how to capture interactions between practitioners and experts systematically. We devise a taxonomy to ma…
▽ More
Machine learning (ML) practitioners are increasingly tasked with developing models that are aligned with non-technical experts' values and goals. However, there has been insufficient consideration on how practitioners should translate domain expertise into ML updates. In this paper, we consider how to capture interactions between practitioners and experts systematically. We devise a taxonomy to match expert feedback types with practitioner updates. A practitioner may receive feedback from an expert at the observation- or domain-level, and convert this feedback into updates to the dataset, loss function, or parameter space. We review existing work from ML and human-computer interaction to describe this feedback-update taxonomy, and highlight the insufficient consideration given to incorporating feedback from non-technical experts. We end with a set of open questions that naturally arise from our proposed taxonomy and subsequent survey.
△ Less
Submitted 16 July, 2022; v1 submitted 13 May, 2022;
originally announced May 2022.
-
On the Utility of Prediction Sets in Human-AI Teams
Authors:
Varun Babbar,
Umang Bhatt,
Adrian Weller
Abstract:
Research on human-AI teams usually provides experts with a single label, which ignores the uncertainty in a model's recommendation. Conformal prediction (CP) is a well established line of research that focuses on building a theoretically grounded, calibrated prediction set, which may contain multiple labels. We explore how such prediction sets impact expert decision-making in human-AI teams. Our e…
▽ More
Research on human-AI teams usually provides experts with a single label, which ignores the uncertainty in a model's recommendation. Conformal prediction (CP) is a well established line of research that focuses on building a theoretically grounded, calibrated prediction set, which may contain multiple labels. We explore how such prediction sets impact expert decision-making in human-AI teams. Our evaluation on human subjects finds that set valued predictions positively impact experts. However, we notice that the predictive sets provided by CP can be very large, which leads to unhelpful AI assistants. To mitigate this, we introduce D-CP, a method to perform CP on some examples and defer to experts. We prove that D-CP can reduce the prediction set size of non-deferred examples. We show how D-CP performs in quantitative and in human subject experiments ($n=120$). Our results suggest that CP prediction sets improve human-AI team performance over showing the top-1 prediction alone, and that experts find D-CP prediction sets are more useful than CP prediction sets.
△ Less
Submitted 26 May, 2022; v1 submitted 3 May, 2022;
originally announced May 2022.
-
Matching Writers to Content Writing Tasks
Authors:
Narayana Darapaneni,
Chandrashekhar Bhakuni,
Ujjval Bhatt,
Khamir Purohit,
Vikas Sardna,
Prabir Chakraborty,
Anwesh Reddy Paduri
Abstract:
Businesses need content. In various forms and formats and for varied purposes. In fact, the content marketing industry is set to be worth $412.88 billion by the end of 2021. However, according to the Content Marketing Institute, creating engaging content is the #1 challenge that marketers face today. We under-stand that producing great content requires great writers who understand the business and…
▽ More
Businesses need content. In various forms and formats and for varied purposes. In fact, the content marketing industry is set to be worth $412.88 billion by the end of 2021. However, according to the Content Marketing Institute, creating engaging content is the #1 challenge that marketers face today. We under-stand that producing great content requires great writers who understand the business and can weave their message into reader (and search engine) friendly content. In this project, the team has attempted to bridge the gap between writers and projects by using AI and ML tools. We used NLP techniques to analyze thou-sands of publicly available business articles (corpora) to extract various defining factors for each writing sample. Through this project we aim to automate the highly time-consuming, and often biased task of manually shortlisting the most suitable writer for a given content writing requirement. We believe that a tool like this will have far reaching positive implications for both parties - businesses looking for suitable talent for niche writing jobs as well as experienced writers and Subject Matter Experts (SMEs) wanting to lend their services to content marketing projects. The business gets the content they need, the content writer/ SME gets a chance to leverage his or her talent, while the reader gets authentic content that adds real value.
△ Less
Submitted 7 April, 2022;
originally announced April 2022.
-
Approximating Full Conformal Prediction at Scale via Influence Functions
Authors:
Javier Abad,
Umang Bhatt,
Adrian Weller,
Giovanni Cherubin
Abstract:
Conformal prediction (CP) is a wrapper around traditional machine learning models, giving coverage guarantees under the sole assumption of exchangeability; in classification problems, for a chosen significance level $\varepsilon$, CP guarantees that the error rate is at most $\varepsilon$, irrespective of whether the underlying model is misspecified. However, the prohibitive computational costs of…
▽ More
Conformal prediction (CP) is a wrapper around traditional machine learning models, giving coverage guarantees under the sole assumption of exchangeability; in classification problems, for a chosen significance level $\varepsilon$, CP guarantees that the error rate is at most $\varepsilon$, irrespective of whether the underlying model is misspecified. However, the prohibitive computational costs of "full" CP led researchers to design scalable alternatives, which alas do not attain the same guarantees or statistical power of full CP. In this paper, we use influence functions to efficiently approximate full CP. We prove that our method is a consistent approximation of full CP, and empirically show that the approximation error becomes smaller as the training set increases; e.g., for $10^{3}$ training points the two methods output p-values that are $<10^{-3}$ apart: a negligible error for any practical application. Our methods enable scaling full CP to large real-world datasets. We compare our full CP approximation (ACP) to mainstream CP alternatives, and observe that our method is computationally competitive whilst enjoying the statistical predictive power of full CP.
△ Less
Submitted 22 February, 2023; v1 submitted 2 February, 2022;
originally announced February 2022.
-
Diverse, Global and Amortised Counterfactual Explanations for Uncertainty Estimates
Authors:
Dan Ley,
Umang Bhatt,
Adrian Weller
Abstract:
To interpret uncertainty estimates from differentiable probabilistic models, recent work has proposed generating a single Counterfactual Latent Uncertainty Explanation (CLUE) for a given data point where the model is uncertain, identifying a single, on-manifold change to the input such that the model becomes more certain in its prediction. We broaden the exploration to examine $δ$-CLUE, the set of…
▽ More
To interpret uncertainty estimates from differentiable probabilistic models, recent work has proposed generating a single Counterfactual Latent Uncertainty Explanation (CLUE) for a given data point where the model is uncertain, identifying a single, on-manifold change to the input such that the model becomes more certain in its prediction. We broaden the exploration to examine $δ$-CLUE, the set of potential CLUEs within a $δ$ ball of the original input in latent space. We study the diversity of such sets and find that many CLUEs are redundant; as such, we propose DIVerse CLUE ($\nabla$-CLUE), a set of CLUEs which each propose a distinct explanation as to how one can decrease the uncertainty associated with an input. We then further propose GLobal AMortised CLUE (GLAM-CLUE), a distinct and novel method which learns amortised mappings on specific groups of uncertain inputs, taking them and efficiently transforming them in a single function call into inputs for which a model will be certain. Our experiments show that $δ$-CLUE, $\nabla$-CLUE, and GLAM-CLUE all address shortcomings of CLUE and provide beneficial explanations of uncertainty estimates to practitioners.
△ Less
Submitted 8 December, 2021; v1 submitted 5 December, 2021;
originally announced December 2021.
-
DIVINE: Diverse Influential Training Points for Data Visualization and Model Refinement
Authors:
Umang Bhatt,
Isabel Chien,
Muhammad Bilal Zafar,
Adrian Weller
Abstract:
As the complexity of machine learning (ML) models increases, resulting in a lack of prediction explainability, several methods have been developed to explain a model's behavior in terms of the training data points that most influence the model. However, these methods tend to mark outliers as highly influential points, limiting the insights that practitioners can draw from points that are not repre…
▽ More
As the complexity of machine learning (ML) models increases, resulting in a lack of prediction explainability, several methods have been developed to explain a model's behavior in terms of the training data points that most influence the model. However, these methods tend to mark outliers as highly influential points, limiting the insights that practitioners can draw from points that are not representative of the training data. In this work, we take a step towards finding influential training points that also represent the training data well. We first review methods for assigning importance scores to training points. Given importance scores, we propose a method to select a set of DIVerse INfluEntial (DIVINE) training points as a useful explanation of model behavior. As practitioners might not only be interested in finding data points influential with respect to model accuracy, but also with respect to other important metrics, we show how to evaluate training data points on the basis of group fairness. Our method can identify unfairness-inducing training points, which can be removed to improve fairness outcomes. Our quantitative experiments and user studies show that visualizing DIVINE points helps practitioners understand and explain model behavior better than earlier approaches.
△ Less
Submitted 13 July, 2021;
originally announced July 2021.
-
Do Concept Bottleneck Models Learn as Intended?
Authors:
Andrei Margeloiu,
Matthew Ashman,
Umang Bhatt,
Yanzhi Chen,
Mateja Jamnik,
Adrian Weller
Abstract:
Concept bottleneck models map from raw inputs to concepts, and then from concepts to targets. Such models aim to incorporate pre-specified, high-level concepts into the learning procedure, and have been motivated to meet three desiderata: interpretability, predictability, and intervenability. However, we find that concept bottleneck models struggle to meet these goals. Using post hoc interpretabil…
▽ More
Concept bottleneck models map from raw inputs to concepts, and then from concepts to targets. Such models aim to incorporate pre-specified, high-level concepts into the learning procedure, and have been motivated to meet three desiderata: interpretability, predictability, and intervenability. However, we find that concept bottleneck models struggle to meet these goals. Using post hoc interpretability methods, we demonstrate that concepts do not correspond to anything semantically meaningful in input space, thus calling into question the usefulness of concept bottleneck models in their current form.
△ Less
Submitted 10 May, 2021;
originally announced May 2021.
-
δ-CLUE: Diverse Sets of Explanations for Uncertainty Estimates
Authors:
Dan Ley,
Umang Bhatt,
Adrian Weller
Abstract:
To interpret uncertainty estimates from differentiable probabilistic models, recent work has proposed generating Counterfactual Latent Uncertainty Explanations (CLUEs). However, for a single input, such approaches could output a variety of explanations due to the lack of constraints placed on the explanation. Here we augment the original CLUE approach, to provide what we call $δ$-CLUE. CLUE indica…
▽ More
To interpret uncertainty estimates from differentiable probabilistic models, recent work has proposed generating Counterfactual Latent Uncertainty Explanations (CLUEs). However, for a single input, such approaches could output a variety of explanations due to the lack of constraints placed on the explanation. Here we augment the original CLUE approach, to provide what we call $δ$-CLUE. CLUE indicates $\it{one}$ way to change an input, while remaining on the data manifold, such that the model becomes more confident about its prediction. We instead return a $\it{set}$ of plausible CLUEs: multiple, diverse inputs that are within a $δ$ ball of the original input in latent space, all yielding confident predictions.
△ Less
Submitted 3 December, 2021; v1 submitted 13 April, 2021;
originally announced April 2021.
-
A Multistakeholder Approach Towards Evaluating AI Transparency Mechanisms
Authors:
Ana Lucic,
Madhulika Srikumar,
Umang Bhatt,
Alice Xiang,
Ankur Taly,
Q. Vera Liao,
Maarten de Rijke
Abstract:
Given that there are a variety of stakeholders involved in, and affected by, decisions from machine learning (ML) models, it is important to consider that different stakeholders have different transparency needs. Previous work found that the majority of deployed transparency mechanisms primarily serve technical stakeholders. In our work, we want to investigate how well transparency mechanisms migh…
▽ More
Given that there are a variety of stakeholders involved in, and affected by, decisions from machine learning (ML) models, it is important to consider that different stakeholders have different transparency needs. Previous work found that the majority of deployed transparency mechanisms primarily serve technical stakeholders. In our work, we want to investigate how well transparency mechanisms might work in practice for a more diverse set of stakeholders by conducting a large-scale, mixed-methods user study across a range of organizations, within a particular industry such as health care, criminal justice, or content moderation. In this paper, we outline the setup for our study.
△ Less
Submitted 1 June, 2021; v1 submitted 27 March, 2021;
originally announced March 2021.
-
Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty
Authors:
Umang Bhatt,
Javier Antorán,
Yunfeng Zhang,
Q. Vera Liao,
Prasanna Sattigeri,
Riccardo Fogliato,
Gabrielle Gauthier Melançon,
Ranganath Krishnan,
Jason Stanley,
Omesh Tickoo,
Lama Nachman,
Rumi Chunara,
Madhulika Srikumar,
Adrian Weller,
Alice Xiang
Abstract:
Algorithmic transparency entails exposing system properties to various stakeholders for purposes that include understanding, improving, and contesting predictions. Until now, most research into algorithmic transparency has predominantly focused on explainability. Explainability attempts to provide reasons for a machine learning model's behavior to stakeholders. However, understanding a model's spe…
▽ More
Algorithmic transparency entails exposing system properties to various stakeholders for purposes that include understanding, improving, and contesting predictions. Until now, most research into algorithmic transparency has predominantly focused on explainability. Explainability attempts to provide reasons for a machine learning model's behavior to stakeholders. However, understanding a model's specific behavior alone might not be enough for stakeholders to gauge whether the model is wrong or lacks sufficient knowledge to solve the task at hand. In this paper, we argue for considering a complementary form of transparency by estimating and communicating the uncertainty associated with model predictions. First, we discuss methods for assessing uncertainty. Then, we characterize how uncertainty can be used to mitigate model unfairness, augment decision-making, and build trustworthy systems. Finally, we outline methods for displaying uncertainty to stakeholders and recommend how to collect information required for incorporating uncertainty into existing ML pipelines. This work constitutes an interdisciplinary review drawn from literature spanning machine learning, visualization/HCI, design, decision-making, and fairness. We aim to encourage researchers and practitioners to measure, communicate, and use uncertainty as a form of transparency.
△ Less
Submitted 4 May, 2021; v1 submitted 15 November, 2020;
originally announced November 2020.
-
On the Fairness of Causal Algorithmic Recourse
Authors:
Julius von Kügelgen,
Amir-Hossein Karimi,
Umang Bhatt,
Isabel Valera,
Adrian Weller,
Bernhard Schölkopf
Abstract:
Algorithmic fairness is typically studied from the perspective of predictions. Instead, here we investigate fairness from the perspective of recourse actions suggested to individuals to remedy an unfavourable classification. We propose two new fairness criteria at the group and individual level, which -- unlike prior work on equalising the average group-wise distance from the decision boundary --…
▽ More
Algorithmic fairness is typically studied from the perspective of predictions. Instead, here we investigate fairness from the perspective of recourse actions suggested to individuals to remedy an unfavourable classification. We propose two new fairness criteria at the group and individual level, which -- unlike prior work on equalising the average group-wise distance from the decision boundary -- explicitly account for causal relationships between features, thereby capturing downstream effects of recourse actions performed in the physical world. We explore how our criteria relate to others, such as counterfactual fairness, and show that fairness of recourse is complementary to fairness of prediction. We study theoretically and empirically how to enforce fair causal recourse by altering the classifier and perform a case study on the Adult dataset. Finally, we discuss whether fairness violations in the data generating process revealed by our criteria may be better addressed by societal interventions as opposed to constraints on the classifier.
△ Less
Submitted 6 March, 2022; v1 submitted 13 October, 2020;
originally announced October 2020.
-
Machine Learning Explainability for External Stakeholders
Authors:
Umang Bhatt,
McKane Andrus,
Adrian Weller,
Alice Xiang
Abstract:
As machine learning is increasingly deployed in high-stakes contexts affecting people's livelihoods, there have been growing calls to open the black box and to make machine learning algorithms more explainable. Providing useful explanations requires careful consideration of the needs of stakeholders, including end-users, regulators, and domain experts. Despite this need, little work has been done…
▽ More
As machine learning is increasingly deployed in high-stakes contexts affecting people's livelihoods, there have been growing calls to open the black box and to make machine learning algorithms more explainable. Providing useful explanations requires careful consideration of the needs of stakeholders, including end-users, regulators, and domain experts. Despite this need, little work has been done to facilitate inter-stakeholder conversation around explainable machine learning. To help address this gap, we conducted a closed-door, day-long workshop between academics, industry experts, legal scholars, and policymakers to develop a shared language around explainability and to understand the current shortcomings of and potential solutions for deploying explainable machine learning in service of transparency goals. We also asked participants to share case studies in deploying explainable machine learning at scale. In this paper, we provide a short summary of various case studies of explainable machine learning, lessons from those studies, and discuss open challenges.
△ Less
Submitted 10 July, 2020;
originally announced July 2020.
-
Getting a CLUE: A Method for Explaining Uncertainty Estimates
Authors:
Javier Antorán,
Umang Bhatt,
Tameem Adel,
Adrian Weller,
José Miguel Hernández-Lobato
Abstract:
Both uncertainty estimation and interpretability are important factors for trustworthy machine learning systems. However, there is little work at the intersection of these two areas. We address this gap by proposing a novel method for interpreting uncertainty estimates from differentiable probabilistic models, like Bayesian Neural Networks (BNNs). Our method, Counterfactual Latent Uncertainty Expl…
▽ More
Both uncertainty estimation and interpretability are important factors for trustworthy machine learning systems. However, there is little work at the intersection of these two areas. We address this gap by proposing a novel method for interpreting uncertainty estimates from differentiable probabilistic models, like Bayesian Neural Networks (BNNs). Our method, Counterfactual Latent Uncertainty Explanations (CLUE), indicates how to change an input, while keeping it on the data manifold, such that a BNN becomes more confident about the input's prediction. We validate CLUE through 1) a novel framework for evaluating counterfactual explanations of uncertainty, 2) a series of ablation experiments, and 3) a user study. Our experiments show that CLUE outperforms baselines and enables practitioners to better understand which input patterns are responsible for predictive uncertainty.
△ Less
Submitted 18 March, 2021; v1 submitted 11 June, 2020;
originally announced June 2020.
-
Evaluating and Aggregating Feature-based Model Explanations
Authors:
Umang Bhatt,
Adrian Weller,
José M. F. Moura
Abstract:
A feature-based model explanation denotes how much each input feature contributes to a model's output for a given data point. As the number of proposed explanation functions grows, we lack quantitative evaluation criteria to help practitioners know when to use which explanation function. This paper proposes quantitative evaluation criteria for feature-based explanations: low sensitivity, high fait…
▽ More
A feature-based model explanation denotes how much each input feature contributes to a model's output for a given data point. As the number of proposed explanation functions grows, we lack quantitative evaluation criteria to help practitioners know when to use which explanation function. This paper proposes quantitative evaluation criteria for feature-based explanations: low sensitivity, high faithfulness, and low complexity. We devise a framework for aggregating explanation functions. We develop a procedure for learning an aggregate explanation function with lower complexity and then derive a new aggregate Shapley value explanation function that minimizes sensitivity.
△ Less
Submitted 1 May, 2020;
originally announced May 2020.
-
A Robot's Expressive Language Affects Human Strategy and Perceptions in a Competitive Game
Authors:
Aaron M. Roth,
Samantha Reig,
Umang Bhatt,
Jonathan Shulgach,
Tamara Amin,
Afsaneh Doryab,
Fei Fang,
Manuela Veloso
Abstract:
As robots are increasingly endowed with social and communicative capabilities, they will interact with humans in more settings, both collaborative and competitive. We explore human-robot relationships in the context of a competitive Stackelberg Security Game. We vary humanoid robot expressive language (in the form of "encouraging" or "discouraging" verbal commentary) and measure the impact on part…
▽ More
As robots are increasingly endowed with social and communicative capabilities, they will interact with humans in more settings, both collaborative and competitive. We explore human-robot relationships in the context of a competitive Stackelberg Security Game. We vary humanoid robot expressive language (in the form of "encouraging" or "discouraging" verbal commentary) and measure the impact on participants' rationality, strategy prioritization, mood, and perceptions of the robot. We learn that a robot opponent that makes discouraging comments causes a human to play a game less rationally and to perceive the robot more negatively. We also contribute a simple open source Natural Language Processing framework for generating expressive sentences, which was used to generate the speech of our autonomous social robot.
△ Less
Submitted 24 October, 2019;
originally announced October 2019.
-
Explainable Machine Learning in Deployment
Authors:
Umang Bhatt,
Alice Xiang,
Shubham Sharma,
Adrian Weller,
Ankur Taly,
Yunhan Jia,
Joydeep Ghosh,
Ruchir Puri,
José M. F. Moura,
Peter Eckersley
Abstract:
Explainable machine learning offers the potential to provide stakeholders with insights into model behavior by using various methods such as feature importance scores, counterfactual explanations, or influential training data. Yet there is little understanding of how organizations use these methods in practice. This study explores how organizations view and use explainability for stakeholder consu…
▽ More
Explainable machine learning offers the potential to provide stakeholders with insights into model behavior by using various methods such as feature importance scores, counterfactual explanations, or influential training data. Yet there is little understanding of how organizations use these methods in practice. This study explores how organizations view and use explainability for stakeholder consumption. We find that, currently, the majority of deployments are not for end users affected by the model but rather for machine learning engineers, who use explainability to debug the model itself. There is thus a gap between explainability in practice and the goal of transparency, since explanations primarily serve internal stakeholders rather than external ones. Our study synthesizes the limitations of current explainability techniques that hamper their use for end users. To facilitate end user interaction, we develop a framework for establishing clear goals for explainability. We end by discussing concerns raised regarding explainability.
△ Less
Submitted 10 July, 2020; v1 submitted 13 September, 2019;
originally announced September 2019.
-
Online Variational Bayesian Subspace Filtering with Applications
Authors:
Charul,
Uttkarsha Bhatt,
Pravesh Biyani,
Ketan Rajawat
Abstract:
Matrix completion and robust principal component analysis have been widely used for the recovery of data suffering from missing entries or outliers. In many real-world applications however, the data is also time-varying, and the naive approach of per-snapshot recovery is both expensive and sub-optimal. This paper develops generative Bayesian models that fit sequential multivariate measurements ari…
▽ More
Matrix completion and robust principal component analysis have been widely used for the recovery of data suffering from missing entries or outliers. In many real-world applications however, the data is also time-varying, and the naive approach of per-snapshot recovery is both expensive and sub-optimal. This paper develops generative Bayesian models that fit sequential multivariate measurements arising from a low-dimensional time-varying subspace. A variational Bayesian subspace filtering approach is proposed that learns the underlying subspace and its state-transition matrix. Different from the plethora of deterministic counterparts, the proposed approach utilizes automatic relevance determination priors that obviate the need to tune key parameters such as rank and noise power. We also propose a forward-backward algorithm that allows the updates to be carried out at low complexity. Extensive tests over traffic and electricity data demonstrate the superior imputation, outlier rejection, and temporal prediction prowess of the proposed algorithm over the state-of-the-art matrix/tensor completion algorithms.
△ Less
Submitted 24 June, 2019;
originally announced June 2019.
-
Towards Aggregating Weighted Feature Attributions
Authors:
Umang Bhatt,
Pradeep Ravikumar,
Jose M. F. Moura
Abstract:
Current approaches for explaining machine learning models fall into two distinct classes: antecedent event influence and value attribution. The former leverages training instances to describe how much influence a training point exerts on a test point, while the latter attempts to attribute value to the features most pertinent to a given prediction. In this work, we discuss an algorithm, AVA: Aggre…
▽ More
Current approaches for explaining machine learning models fall into two distinct classes: antecedent event influence and value attribution. The former leverages training instances to describe how much influence a training point exerts on a test point, while the latter attempts to attribute value to the features most pertinent to a given prediction. In this work, we discuss an algorithm, AVA: Aggregate Valuation of Antecedents, that fuses these two explanation classes to form a new approach to feature attribution that not only retrieves local explanations but also captures global patterns learned by a model. Our experimentation convincingly favors weighting and aggregating feature attributions via AVA.
△ Less
Submitted 20 January, 2019;
originally announced January 2019.
-
On Network Science and Mutual Information for Explaining Deep Neural Networks
Authors:
Brian Davis,
Umang Bhatt,
Kartikeya Bhardwaj,
Radu Marculescu,
José M. F. Moura
Abstract:
In this paper, we present a new approach to interpret deep learning models. By coupling mutual information with network science, we explore how information flows through feedforward networks. We show that efficiently approximating mutual information allows us to create an information measure that quantifies how much information flows between any two neurons of a deep learning model. To that end, w…
▽ More
In this paper, we present a new approach to interpret deep learning models. By coupling mutual information with network science, we explore how information flows through feedforward networks. We show that efficiently approximating mutual information allows us to create an information measure that quantifies how much information flows between any two neurons of a deep learning model. To that end, we propose NIF, Neural Information Flow, a technique for codifying information flow that exposes deep learning model internals and provides feature attributions.
△ Less
Submitted 3 May, 2020; v1 submitted 20 January, 2019;
originally announced January 2019.
-
The Impact of Humanoid Affect Expression on Human Behavior in a Game-Theoretic Setting
Authors:
Aaron M. Roth,
Umang Bhatt,
Tamara Amin,
Afsaneh Doryab,
Fei Fang,
Manuela Veloso
Abstract:
With the rapid development of robot and other intelligent and autonomous agents, how a human could be influenced by a robot's expressed mood when making decisions becomes a crucial question in human-robot interaction. In this pilot study, we investigate (1) in what way a robot can express a certain mood to influence a human's decision making behavioral model; (2) how and to what extent the human w…
▽ More
With the rapid development of robot and other intelligent and autonomous agents, how a human could be influenced by a robot's expressed mood when making decisions becomes a crucial question in human-robot interaction. In this pilot study, we investigate (1) in what way a robot can express a certain mood to influence a human's decision making behavioral model; (2) how and to what extent the human will be influenced in a game theoretic setting. More specifically, we create an NLP model to generate sentences that adhere to a specific affective expression profile. We use these sentences for a humanoid robot as it plays a Stackelberg security game against a human. We investigate the behavioral model of the human player.
△ Less
Submitted 10 June, 2018;
originally announced June 2018.
-
Maintaining The Humanity of Our Models
Authors:
Umang Bhatt
Abstract:
Artificial intelligence and machine learning have been major research interests in computer science for the better part of the last few decades. However, all too recently, both AI and ML have rapidly grown to be media frenzies, pressuring companies and researchers to claim they use these technologies. As ML continues to percolate into daily life, we, as computer scientists and machine learning res…
▽ More
Artificial intelligence and machine learning have been major research interests in computer science for the better part of the last few decades. However, all too recently, both AI and ML have rapidly grown to be media frenzies, pressuring companies and researchers to claim they use these technologies. As ML continues to percolate into daily life, we, as computer scientists and machine learning researchers, are responsible for ensuring we clearly convey the extent of our work and the humanity of our models. Regularizing ML for mass adoption requires a rigorous standard for model interpretability, a deep consideration for human bias in data, and a transparent understanding of a model's societal effects.
△ Less
Submitted 10 December, 2017; v1 submitted 15 November, 2017;
originally announced November 2017.
-
Intelligent Pothole Detection and Road Condition Assessment
Authors:
Umang Bhatt,
Shouvik Mani,
Edgar Xi,
J. Zico Kolter
Abstract:
Poor road conditions are a public nuisance, causing passenger discomfort, damage to vehicles, and accidents. In the U.S., road-related conditions are a factor in 22,000 of the 42,000 traffic fatalities each year. Although we often complain about bad roads, we have no way to detect or report them at scale. To address this issue, we developed a system to detect potholes and assess road conditions in…
▽ More
Poor road conditions are a public nuisance, causing passenger discomfort, damage to vehicles, and accidents. In the U.S., road-related conditions are a factor in 22,000 of the 42,000 traffic fatalities each year. Although we often complain about bad roads, we have no way to detect or report them at scale. To address this issue, we developed a system to detect potholes and assess road conditions in real-time. Our solution is a mobile application that captures data on a car's movement from gyroscope and accelerometer sensors in the phone. To assess roads using this sensor data, we trained SVM models to classify road conditions with 93% accuracy and potholes with 92% accuracy, beating the base rate for both problems. As the user drives, the models use the sensor data to classify whether the road is good or bad, and whether it contains potholes. Then, the classification results are used to create data-rich maps that illustrate road conditions across the city. Our system will empower civic officials to identify and repair damaged roads which inconvenience passengers and cause accidents. This paper details our data science process for collecting training data on real roads, transforming noisy sensor data into useful signals, training and evaluating machine learning models, and deploying those models to production through a real-time classification app. It also highlights how cities can use our system to crowdsource data and deliver road repair resources to areas in need.
△ Less
Submitted 10 October, 2017; v1 submitted 6 October, 2017;
originally announced October 2017.
-
Delivery of Dark Material to Vesta via Carbonaceous Chondritic Impacts
Authors:
Vishnu Reddy,
Lucille Le Corre,
David P. O'Brien,
Andreas Nathues,
Edward A. Cloutis,
Daniel D. Durda,
William F. Bottke,
Megha U. Bhatt,
David Nesvorny,
Debra Buczkowski,
Jennifer E. C. Scully,
Elizabeth M. Palmer,
Holger Sierks,
Paul J. Mann,
Kris J. Becker,
Andrew W. Beck,
David Mittlefehldt,
Jian-Yang Li,
Robert Gaskell,
Christopher T. Russell,
Michael J. Gaffey,
Harry Y. McSween,
Thomas B. McCord,
Jean-Philippe Combe,
David Blewett
Abstract:
NASA's Dawn spacecraft observations of asteroid (4) Vesta reveal a surface with the highest albedo and color variation of any asteroid we have observed so far. Terrains rich in low albedo dark material (DM) have been identified using Dawn Framing Camera (FC) 0.75 μm filter images in several geologic settings: associated with impact craters (in the ejecta blanket material and/or on the crater walls…
▽ More
NASA's Dawn spacecraft observations of asteroid (4) Vesta reveal a surface with the highest albedo and color variation of any asteroid we have observed so far. Terrains rich in low albedo dark material (DM) have been identified using Dawn Framing Camera (FC) 0.75 μm filter images in several geologic settings: associated with impact craters (in the ejecta blanket material and/or on the crater walls and rims); as flow-like deposits or rays commonly associated with topographic highs; and as dark spots (likely secondary impacts) nearby impact craters. This DM could be a relic of ancient volcanic activity or exogenic in origin. We report that the majority of the spectra of DM are similar to carbonaceous chondrite meteorites mixed with materials indigenous to Vesta. Using high-resolution seven color images we compared DM color properties (albedo, band depth) with laboratory measurements of possible analog materials. Band depth and albedo of DM are identical to those of carbonaceous chondrite xenolith-rich howardite Mt. Pratt (PRA) 04401. Laboratory mixtures of Murchison CM2 carbonaceous chondrite and basaltic eucrite Millbillillie also show band depth and albedo affinity to DM. Modeling of carbonaceous chondrite abundance in DM (1-6 vol%) is consistent with howardite meteorites. We find no evidence for large-scale volcanism (exposed dikes/pyroclastic falls) as the source of DM. Our modeling efforts using impact crater scaling laws and numerical models of ejecta reaccretion suggest the delivery and emplacement of this DM on Vesta during the formation of the ~400 km Veneneia basin by a low-velocity (<2 km/sec) carbonaceous impactor. This discovery is important because it strengthens the long-held idea that primitive bodies are the source of carbon and probably volatiles in the early Solar System.
△ Less
Submitted 14 August, 2012;
originally announced August 2012.