-
Data-Centric AI Governance: Addressing the Limitations of Model-Focused Policies
Authors:
Ritwik Gupta,
Leah Walker,
Rodolfo Corona,
Stephanie Fu,
Suzanne Petryk,
Janet Napolitano,
Trevor Darrell,
Andrew W. Reddie
Abstract:
Current regulations on powerful AI capabilities are narrowly focused on "foundation" or "frontier" models. However, these terms are vague and inconsistently defined, leading to an unstable foundation for governance efforts. Critically, policy debates often fail to consider the data used with these models, despite the clear link between data and model performance. Even (relatively) "small" models t…
▽ More
Current regulations on powerful AI capabilities are narrowly focused on "foundation" or "frontier" models. However, these terms are vague and inconsistently defined, leading to an unstable foundation for governance efforts. Critically, policy debates often fail to consider the data used with these models, despite the clear link between data and model performance. Even (relatively) "small" models that fall outside the typical definitions of foundation and frontier models can achieve equivalent outcomes when exposed to sufficiently specific datasets. In this work, we illustrate the importance of considering dataset size and content as essential factors in assessing the risks posed by models both today and in the future. More broadly, we emphasize the risk posed by over-regulating reactively and provide a path towards careful, quantitative evaluation of capabilities that can lead to a simplified regulatory environment.
△ Less
Submitted 25 September, 2024;
originally announced September 2024.
-
Open-Source Assessments of AI Capabilities: The Proliferation of AI Analysis Tools, Replicating Competitor Models, and the Zhousidun Dataset
Authors:
Ritwik Gupta,
Leah Walker,
Eli Glickman,
Raine Koizumi,
Sarthak Bhatnagar,
Andrew W. Reddie
Abstract:
The integration of artificial intelligence (AI) into military capabilities has become a norm for major military power across the globe. Understanding how these AI models operate is essential for maintaining strategic advantages and ensuring security. This paper demonstrates an open-source methodology for analyzing military AI models through a detailed examination of the Zhousidun dataset, a Chines…
▽ More
The integration of artificial intelligence (AI) into military capabilities has become a norm for major military power across the globe. Understanding how these AI models operate is essential for maintaining strategic advantages and ensuring security. This paper demonstrates an open-source methodology for analyzing military AI models through a detailed examination of the Zhousidun dataset, a Chinese-originated dataset that exhaustively labels critical components on American and Allied destroyers. By demonstrating the replication of a state-of-the-art computer vision model on this dataset, we illustrate how open-source tools can be leveraged to assess and understand key military AI capabilities. This methodology offers a robust framework for evaluating the performance and potential of AI-enabled military capabilities, thus enhancing the accuracy and reliability of strategic assessments.
△ Less
Submitted 24 May, 2024; v1 submitted 20 May, 2024;
originally announced May 2024.
-
Wargames as Data: Addressing the Wargamer's Trilemma
Authors:
Andrew W. Reddie,
Ruby E. Booth,
Bethany L. Goldblum,
Kiran Lakkaraju,
Jason Reinhardt
Abstract:
Policymakers often want the very best data with which to make decisions--particularly when concerned with questions of national and international security. But what happens when this data is not available? In those instances, analysts have come to rely on synthetic data-generating processes--turning to modeling and simulation tools and survey experiments among other methods. In the cyber domain, w…
▽ More
Policymakers often want the very best data with which to make decisions--particularly when concerned with questions of national and international security. But what happens when this data is not available? In those instances, analysts have come to rely on synthetic data-generating processes--turning to modeling and simulation tools and survey experiments among other methods. In the cyber domain, where empirical data at the strategic level are limited, this is no different--cyber wargames are quickly becoming a principal method for both exploring and analyzing the security challenges posed by state and non-state actors in cyberspace. In this chapter, we examine the design decisions associated with this method.
△ Less
Submitted 15 February, 2023;
originally announced February 2023.
