Showing 1–50 of 302 results for author: Backes, M

Invisibility Cloak: Disappearance under Human Pose Estimation via Backdoor Attacks

Authors: Minxing Zhang, Michael Backes, Xiao Zhang

Abstract: Human Pose Estimation (HPE) has been widely applied in autonomous systems such as self-driving cars. However, the potential risks of HPE to adversarial attacks have not received comparable attention with image classification or segmentation tasks. Existing works on HPE robustness focus on misleading an HPE system to provide wrong predictions that still indicate some human poses. In this paper, we… ▽ More Human Pose Estimation (HPE) has been widely applied in autonomous systems such as self-driving cars. However, the potential risks of HPE to adversarial attacks have not received comparable attention with image classification or segmentation tasks. Existing works on HPE robustness focus on misleading an HPE system to provide wrong predictions that still indicate some human poses. In this paper, we study the vulnerability of HPE systems to disappearance attacks, where the attacker aims to subtly alter the HPE training process via backdoor techniques so that any input image with some specific trigger will not be recognized as involving any human pose. As humans are typically at the center of HPE systems, such attacks can induce severe security hazards, e.g., pedestrians' lives will be threatened if a self-driving car incorrectly understands the front scene due to disappearance attacks. To achieve the adversarial goal of disappearance, we propose IntC, a general framework to craft Invisibility Cloak in the HPE domain. The core of our work lies in the design of target HPE labels that do not represent any human pose. In particular, we propose three specific backdoor attacks based on our IntC framework with different label designs. IntC-S and IntC-E, respectively designed for regression- and heatmap-based HPE techniques, concentrate the keypoints of triggered images in a tiny, imperceptible region. Further, to improve the attack's stealthiness, IntC-L designs the target poisons to capture the label outputs of typical landscape images without a human involved, achieving disappearance and reducing detectability simultaneously. Extensive experiments demonstrate the effectiveness and generalizability of our IntC methods in achieving the disappearance goal. By revealing the vulnerability of HPE to disappearance and backdoor attacks, we hope our work can raise awareness of the potential risks ... △ Less

Submitted 10 October, 2024; originally announced October 2024.

$\texttt{ModSCAN}$: Measuring Stereotypical Bias in Large Vision-Language Models from Vision and Language Modalities

Authors: Yukun Jiang, Zheng Li, Xinyue Shen, Yugeng Liu, Michael Backes, Yang Zhang

Abstract: Large vision-language models (LVLMs) have been rapidly developed and widely used in various fields, but the (potential) stereotypical bias in the model is largely unexplored. In this study, we present a pioneering measurement framework, $\texttt{ModSCAN}$, to $\underline{SCAN}$ the stereotypical bias within LVLMs from both vision and language $\underline{Mod}$alities. $\texttt{ModSCAN}$ examines s… ▽ More Large vision-language models (LVLMs) have been rapidly developed and widely used in various fields, but the (potential) stereotypical bias in the model is largely unexplored. In this study, we present a pioneering measurement framework, $\texttt{ModSCAN}$, to $\underline{SCAN}$ the stereotypical bias within LVLMs from both vision and language $\underline{Mod}$alities. $\texttt{ModSCAN}$ examines stereotypical biases with respect to two typical stereotypical attributes (gender and race) across three kinds of scenarios: occupations, descriptors, and persona traits. Our findings suggest that 1) the currently popular LVLMs show significant stereotype biases, with CogVLM emerging as the most biased model; 2) these stereotypical biases may stem from the inherent biases in the training dataset and pre-trained models; 3) the utilization of specific prompt prefixes (from both vision and language modalities) performs well in reducing stereotypical biases. We believe our work can serve as the foundation for understanding and addressing stereotypical bias in LVLMs. △ Less

Submitted 9 October, 2024; originally announced October 2024.

Comments: Accepted in EMNLP 2024. 29 pages, 22 figures

Hidden by a star: the redshift and the offset broad line of the Flat Spectrum Radio Quasar PKS 0903-57

Authors: P. Goldoni, C. Boisson, S. Pita, F. D'Ammando, E. Kasai, W. Max-Moerbeck, M. Backes, G. Cotter

Abstract: Context: PKS 0903-57 is a little-studied gamma-ray blazar which has recently attracted considerable interest due to the strong flaring episodes observed since 2020 in HE (100 MeV < E < 100 GeV) and VHE (100 GeV < E < 10 TeV) gamma-rays. Its nature and properties are still not well determined. In particular, it is unclear whether PKS 0903-57 is a BL Lac or a Flat Spectrum Radio Quasar (FSRQ), while… ▽ More Context: PKS 0903-57 is a little-studied gamma-ray blazar which has recently attracted considerable interest due to the strong flaring episodes observed since 2020 in HE (100 MeV < E < 100 GeV) and VHE (100 GeV < E < 10 TeV) gamma-rays. Its nature and properties are still not well determined. In particular, it is unclear whether PKS 0903-57 is a BL Lac or a Flat Spectrum Radio Quasar (FSRQ), while its redshift estimation relies on a possibly misassociated low signal-to-noise spectrum. Aim: We aim to reliably measure the redshift of the blazar and to determine its spectral type and luminosity in the optical range. Methods: We performed spectroscopy of the optical counterpart of the blazar using the South African Large Telescope (SALT) and the Very Large Telescope (VLT) and monitored it photometrically with the Rapid Eye Mount (REM) telescope. Results: We firmly measured the redshift of the blazar as z= 0.2621 +/- 0.0006 thanks to the detection of five narrow optical lines. The detection of a symmetric broad Halpha line with Full Width at Half Maximum (FWHM) of 4020 +/- 30 km/s together with a jet-dominated continuum leads us to classify it as a FSRQ. Finally, we detected with high significance a redshift offset (about 1500 km/s) between the broad line and the host. This is the first time that such an offset is unequivocally detected in a VHE blazar, possibly pointing to a very peculiar accretion configuration, a merging system, or a recoiling Black Hole. △ Less

Submitted 9 October, 2024; originally announced October 2024.

Comments: Astronomy and Astrophysics Letters, Accepted

Localizing Memorization in SSL Vision Encoders

Authors: Wenhao Wang, Adam Dziedzic, Michael Backes, Franziska Boenisch

Abstract: Recent work on studying memorization in self-supervised learning (SSL) suggests that even though SSL encoders are trained on millions of images, they still memorize individual data points. While effort has been put into characterizing the memorized data and linking encoder memorization to downstream utility, little is known about where the memorization happens inside SSL encoders. To close this ga… ▽ More Recent work on studying memorization in self-supervised learning (SSL) suggests that even though SSL encoders are trained on millions of images, they still memorize individual data points. While effort has been put into characterizing the memorized data and linking encoder memorization to downstream utility, little is known about where the memorization happens inside SSL encoders. To close this gap, we propose two metrics for localizing memorization in SSL encoders on a per-layer (layermem) and per-unit basis (unitmem). Our localization methods are independent of the downstream task, do not require any label information, and can be performed in a forward pass. By localizing memorization in various encoder architectures (convolutional and transformer-based) trained on diverse datasets with contrastive and non-contrastive SSL frameworks, we find that (1) while SSL memorization increases with layer depth, highly memorizing units are distributed across the entire encoder, (2) a significant fraction of units in SSL encoders experiences surprisingly high memorization of individual data points, which is in contrast to models trained under supervision, (3) atypical (or outlier) data points cause much higher layer and unit memorization than standard data points, and (4) in vision transformers, most memorization happens in the fully-connected layers. Finally, we show that localizing memorization in SSL has the potential to improve fine-tuning and to inform pruning strategies. △ Less

Submitted 27 September, 2024; originally announced September 2024.

Comments: Accepted at NeurIPS 2024

Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Data

Authors: Atilla Akkus, Mingjie Li, Junjie Chu, Michael Backes, Yang Zhang, Sinem Sav

Abstract: Large language models (LLMs) have shown considerable success in a range of domain-specific tasks, especially after fine-tuning. However, fine-tuning with real-world data usually leads to privacy risks, particularly when the fine-tuning samples exist in the pre-training data. To avoid the shortcomings of real data, developers often employ methods to automatically generate synthetic data for fine-tu… ▽ More Large language models (LLMs) have shown considerable success in a range of domain-specific tasks, especially after fine-tuning. However, fine-tuning with real-world data usually leads to privacy risks, particularly when the fine-tuning samples exist in the pre-training data. To avoid the shortcomings of real data, developers often employ methods to automatically generate synthetic data for fine-tuning, as data generated by traditional models are often far away from the real-world pertaining data. However, given the advanced capabilities of LLMs, the distinction between real data and LLM-generated data has become negligible, which may also lead to privacy risks like real data. In this paper, we present an empirical analysis of this underexplored issue by investigating a key question: "Does fine-tuning with LLM-generated data enhance privacy, or does it pose additional privacy risks?" Based on the structure of LLM's generated data, our research focuses on two primary approaches to fine-tuning with generated data: supervised fine-tuning with unstructured generated data and self-instruct tuning. The number of successful Personal Information Identifier (PII) extractions for Pythia after fine-tuning our generated data raised over $20\%$. Furthermore, the ROC-AUC score of membership inference attacks for Pythia-6.9b after self-instruct methods also achieves more than $40\%$ improvements on ROC-AUC score than base models. The results indicate the potential privacy risks in LLMs when fine-tuning with the generated data. △ Less

Submitted 12 September, 2024; originally announced September 2024.

Understanding Data Importance in Machine Learning Attacks: Does Valuable Data Pose Greater Harm?

Authors: Rui Wen, Michael Backes, Yang Zhang

Abstract: Machine learning has revolutionized numerous domains, playing a crucial role in driving advancements and enabling data-centric processes. The significance of data in training models and shaping their performance cannot be overstated. Recent research has highlighted the heterogeneous impact of individual data samples, particularly the presence of valuable data that significantly contributes to the… ▽ More Machine learning has revolutionized numerous domains, playing a crucial role in driving advancements and enabling data-centric processes. The significance of data in training models and shaping their performance cannot be overstated. Recent research has highlighted the heterogeneous impact of individual data samples, particularly the presence of valuable data that significantly contributes to the utility and effectiveness of machine learning models. However, a critical question remains unanswered: are these valuable data samples more vulnerable to machine learning attacks? In this work, we investigate the relationship between data importance and machine learning attacks by analyzing five distinct attack types. Our findings reveal notable insights. For example, we observe that high importance data samples exhibit increased vulnerability in certain attacks, such as membership inference and model stealing. By analyzing the linkage between membership inference vulnerability and data importance, we demonstrate that sample characteristics can be integrated into membership metrics by introducing sample-specific criteria, therefore enhancing the membership inference performance. These findings emphasize the urgent need for innovative defense mechanisms that strike a balance between maximizing utility and safeguarding valuable data against potential exploitation. △ Less

Submitted 5 September, 2024; originally announced September 2024.

Comments: To Appear in Network and Distributed System Security (NDSS) Symposium 2025

Membership Inference Attacks Against In-Context Learning

Authors: Rui Wen, Zheng Li, Michael Backes, Yang Zhang

Abstract: Adapting Large Language Models (LLMs) to specific tasks introduces concerns about computational efficiency, prompting an exploration of efficient methods such as In-Context Learning (ICL). However, the vulnerability of ICL to privacy attacks under realistic assumptions remains largely unexplored. In this work, we present the first membership inference attack tailored for ICL, relying solely on gen… ▽ More Adapting Large Language Models (LLMs) to specific tasks introduces concerns about computational efficiency, prompting an exploration of efficient methods such as In-Context Learning (ICL). However, the vulnerability of ICL to privacy attacks under realistic assumptions remains largely unexplored. In this work, we present the first membership inference attack tailored for ICL, relying solely on generated texts without their associated probabilities. We propose four attack strategies tailored to various constrained scenarios and conduct extensive experiments on four popular large language models. Empirical results show that our attacks can accurately determine membership status in most cases, e.g., 95\% accuracy advantage against LLaMA, indicating that the associated risks are much higher than those shown by existing probability-based attacks. Additionally, we propose a hybrid attack that synthesizes the strengths of the aforementioned strategies, achieving an accuracy advantage of over 95\% in most cases. Furthermore, we investigate three potential defenses targeting data, instruction, and output. Results demonstrate combining defenses from orthogonal dimensions significantly reduces privacy leakage and offers enhanced privacy assurances. △ Less

Submitted 2 September, 2024; originally announced September 2024.

Comments: To Appear in the ACM Conference on Computer and Communications Security, October 14-18, 2024

Image-Perfect Imperfections: Safety, Bias, and Authenticity in the Shadow of Text-To-Image Model Evolution

Authors: Yixin Wu, Yun Shen, Michael Backes, Yang Zhang

Abstract: Text-to-image models, such as Stable Diffusion (SD), undergo iterative updates to improve image quality and address concerns such as safety. Improvements in image quality are straightforward to assess. However, how model updates resolve existing concerns and whether they raise new questions remain unexplored. This study takes an initial step in investigating the evolution of text-to-image models f… ▽ More Text-to-image models, such as Stable Diffusion (SD), undergo iterative updates to improve image quality and address concerns such as safety. Improvements in image quality are straightforward to assess. However, how model updates resolve existing concerns and whether they raise new questions remain unexplored. This study takes an initial step in investigating the evolution of text-to-image models from the perspectives of safety, bias, and authenticity. Our findings, centered on Stable Diffusion, indicate that model updates paint a mixed picture. While updates progressively reduce the generation of unsafe images, the bias issue, particularly in gender, intensifies. We also find that negative stereotypes either persist within the same Non-White race group or shift towards other Non-White race groups through SD updates, yet with minimal association of these traits with the White race group. Additionally, our evaluation reveals a new concern stemming from SD updates: State-of-the-art fake image detectors, initially trained for earlier SD versions, struggle to identify fake images generated by updated versions. We show that fine-tuning these detectors on fake images generated by updated versions achieves at least 96.6\% accuracy across various SD versions, addressing this issue. Our insights highlight the importance of continued efforts to mitigate biases and vulnerabilities in evolving text-to-image models. △ Less

Submitted 30 August, 2024; originally announced August 2024.

Comments: To Appear in the ACM Conference on Computer and Communications Security, October 14-18, 2024

Distance estimation of gamma-ray emitting BL Lac objects from imaging observations

Authors: K. Nilsson, V. Fallah Ramazani, E. Lindfors, P. Goldoni, J. Becerra González, J. A. Acosta Pulido, R. Clavero, J. Otero-Santos, T. Pursimo, S. Pita, P. M. Kouch, C. Boisson, M. Backes, G. Cotter, F. D'Ammando, E. Kasai

Abstract: Direct redshift determination of BL Lac objects is highly challenging as the emission in the optical and near-infrared (NIR) bands is largely dominated by the non-thermal emission from the relativistic jet that points very close to our line of sight. Therefore, their optical spectra often show no emission lines from the host galaxy. In this work, we aim to overcome this difficulty by attempting to… ▽ More Direct redshift determination of BL Lac objects is highly challenging as the emission in the optical and near-infrared (NIR) bands is largely dominated by the non-thermal emission from the relativistic jet that points very close to our line of sight. Therefore, their optical spectra often show no emission lines from the host galaxy. In this work, we aim to overcome this difficulty by attempting to detect the host galaxy and derive redshift constraints based on assumptions on the galaxy magnitude ("imaging redshifts"). Imaging redshifts are derived by obtaining deep optical images under good seeing conditions, so that it is possible to detect the host galaxy as weak extension of the point-like source. We then derive the imaging redshift by using the host galaxy as a standard candle using two different methods. We determine imaging redshift for 9 out of 17 blazars that we observed as part of this program. The redshift range of these targets is 0.28-0.60 and the two methods used to derive the redshift give very consistent results within the uncertainties. We also performed a detailed comparison of the imaging redshifts with those obtained by other methods, like direct spectroscopic constraints or looking for groups of galaxies close to the blazar. We show that the constraints from different methods are consistent and that for example in the case of J2156.0+1818, which is the most distant source for which we detect the host galaxy, combining the three constraints narrows down the redshift to $0.63<z<0.71$. This makes the source interesting for future studies of extragalactic background light in the Cherenkov Telescope Array Observatory era. △ Less

Submitted 27 August, 2024; originally announced August 2024.

Performance of Antenna-based and Rydberg Quantum RF Sensors in the Electrically Small Regime

Authors: K. M. Backes, P. K. Elgee, K. -J. LeBlanc, C. T. Fancher, D. H. Meyer, P. D. Kunz, N. Malvania, K. M. Nicolich, J. C. Hill, B. L. Schmittberger Marlow, K. C. Cox

Abstract: Rydberg atom electric field sensors are tunable quantum sensors that can perform sensitive radio frequency (RF) measurements. Their qualities have piqued interest at longer wavelengths where their small size compares favorably to impedance-matched antennas. Here, we compare the signal detection sensitivity of cm-scale Rydberg sensors to similarly sized room-temperature electrically small antennas… ▽ More Rydberg atom electric field sensors are tunable quantum sensors that can perform sensitive radio frequency (RF) measurements. Their qualities have piqued interest at longer wavelengths where their small size compares favorably to impedance-matched antennas. Here, we compare the signal detection sensitivity of cm-scale Rydberg sensors to similarly sized room-temperature electrically small antennas with active and passive receiver backends. We present and analyze effective circuit models for each sensor type, facilitating a fair sensitivity comparison for cm-scale sensors. We calculate that contemporary Rydberg sensor implementations are less sensitive than unmatched antennas with active amplification. However, we find that idealized Rydberg sensors operating with a maximized atom number and at the standard quantum limit may perform well beyond the capabilities of antenna-based sensors at room temperature, the sensitivities of both lying below typical atmospheric background noise. △ Less

Submitted 26 August, 2024; originally announced August 2024.

Comments: 6 pages, 3 figures

Inside the Black Box: Detecting Data Leakage in Pre-trained Language Encoders

Authors: Yuan Xin, Zheng Li, Ning Yu, Dingfan Chen, Mario Fritz, Michael Backes, Yang Zhang

Abstract: Despite being prevalent in the general field of Natural Language Processing (NLP), pre-trained language models inherently carry privacy and copyright concerns due to their nature of training on large-scale web-scraped data. In this paper, we pioneer a systematic exploration of such risks associated with pre-trained language encoders, specifically focusing on the membership leakage of pre-training… ▽ More Despite being prevalent in the general field of Natural Language Processing (NLP), pre-trained language models inherently carry privacy and copyright concerns due to their nature of training on large-scale web-scraped data. In this paper, we pioneer a systematic exploration of such risks associated with pre-trained language encoders, specifically focusing on the membership leakage of pre-training data exposed through downstream models adapted from pre-trained language encoders-an aspect largely overlooked in existing literature. Our study encompasses comprehensive experiments across four types of pre-trained encoder architectures, three representative downstream tasks, and five benchmark datasets. Intriguingly, our evaluations reveal, for the first time, the existence of membership leakage even when only the black-box output of the downstream model is exposed, highlighting a privacy risk far greater than previously assumed. Alongside, we present in-depth analysis and insights toward guiding future researchers and practitioners in addressing the privacy considerations in developing pre-trained language models. △ Less

Submitted 20 August, 2024; originally announced August 2024.

Comments: ECAI24

Vera Verto: Multimodal Hijacking Attack

Authors: Minxing Zhang, Ahmed Salem, Michael Backes, Yang Zhang

Abstract: The increasing cost of training machine learning (ML) models has led to the inclusion of new parties to the training pipeline, such as users who contribute training data and companies that provide computing resources. This involvement of such new parties in the ML training process has introduced new attack surfaces for an adversary to exploit. A recent attack in this domain is the model hijacking… ▽ More The increasing cost of training machine learning (ML) models has led to the inclusion of new parties to the training pipeline, such as users who contribute training data and companies that provide computing resources. This involvement of such new parties in the ML training process has introduced new attack surfaces for an adversary to exploit. A recent attack in this domain is the model hijacking attack, whereby an adversary hijacks a victim model to implement their own -- possibly malicious -- hijacking tasks. However, the scope of the model hijacking attack is so far limited to the homogeneous-modality tasks. In this paper, we transform the model hijacking attack into a more general multimodal setting, where the hijacking and original tasks are performed on data of different modalities. Specifically, we focus on the setting where an adversary implements a natural language processing (NLP) hijacking task into an image classification model. To mount the attack, we propose a novel encoder-decoder based framework, namely the Blender, which relies on advanced image and language models. Experimental results show that our modal hijacking attack achieves strong performances in different settings. For instance, our attack achieves 94%, 94%, and 95% attack success rate when using the Sogou news dataset to hijack STL10, CIFAR-10, and MNIST classifiers. △ Less

Submitted 31 July, 2024; originally announced August 2024.

Breaking Agents: Compromising Autonomous LLM Agents Through Malfunction Amplification

Authors: Boyang Zhang, Yicong Tan, Yun Shen, Ahmed Salem, Michael Backes, Savvas Zannettou, Yang Zhang

Abstract: Recently, autonomous agents built on large language models (LLMs) have experienced significant development and are being deployed in real-world applications. These agents can extend the base LLM's capabilities in multiple ways. For example, a well-built agent using GPT-3.5-Turbo as its core can outperform the more advanced GPT-4 model by leveraging external components. More importantly, the usage… ▽ More Recently, autonomous agents built on large language models (LLMs) have experienced significant development and are being deployed in real-world applications. These agents can extend the base LLM's capabilities in multiple ways. For example, a well-built agent using GPT-3.5-Turbo as its core can outperform the more advanced GPT-4 model by leveraging external components. More importantly, the usage of tools enables these systems to perform actions in the real world, moving from merely generating text to actively interacting with their environment. Given the agents' practical applications and their ability to execute consequential actions, it is crucial to assess potential vulnerabilities. Such autonomous systems can cause more severe damage than a standalone language model if compromised. While some existing research has explored harmful actions by LLM agents, our study approaches the vulnerability from a different perspective. We introduce a new type of attack that causes malfunctions by misleading the agent into executing repetitive or irrelevant actions. We conduct comprehensive evaluations using various attack methods, surfaces, and properties to pinpoint areas of susceptibility. Our experiments reveal that these attacks can induce failure rates exceeding 80\% in multiple scenarios. Through attacks on implemented and deployable agents in multi-agent scenarios, we accentuate the realistic risks associated with these vulnerabilities. To mitigate such attacks, we propose self-examination detection methods. However, our findings indicate these attacks are difficult to detect effectively using LLMs alone, highlighting the substantial risks associated with this vulnerability. △ Less

Submitted 30 July, 2024; originally announced July 2024.

Very-high-energy $γ$-ray emission from young massive star clusters in the Large Magellanic Cloud

Authors: F. Aharonian, F. Ait Benkhali, J. Aschersleben, H. Ashkar, M. Backes, V. Barbosa Martins, R. Batzofin, Y. Becherini, D. Berge, K. Bernlöhr, M. Böttcher, J. Bolmont, M. de Bony de Lavergne, J. Borowska, R. Brose, A. Brown, F. Brun, B. Bruno, C. Burger-Scheidlin, S. Casanova, J. Celic, M. Cerruti, T. Chand, S. Chandra, A. Chen , et al. (107 additional authors not shown)

Abstract: The Tarantula Nebula in the Large Magellanic Cloud is known for its high star formation activity. At its center lies the young massive star cluster R136, providing a significant amount of the energy that makes the nebula shine so brightly at many wavelengths. Recently, young massive star clusters have been suggested to also efficiently produce high-energy cosmic rays, potentially beyond PeV energi… ▽ More The Tarantula Nebula in the Large Magellanic Cloud is known for its high star formation activity. At its center lies the young massive star cluster R136, providing a significant amount of the energy that makes the nebula shine so brightly at many wavelengths. Recently, young massive star clusters have been suggested to also efficiently produce high-energy cosmic rays, potentially beyond PeV energies. Here, we report the detection of very-high-energy $γ$-ray emission from the direction of R136 with the High Energy Stereoscopic System, achieved through a multicomponent, likelihood-based modeling of the data. This supports the hypothesis that R136 is indeed a very powerful cosmic-ray accelerator. Moreover, from the same analysis, we provide an updated measurement of the $γ$-ray emission from 30 Dor C, the only superbubble detected at TeV energies presently. The $γ$-ray luminosity above $0.5\,\mathrm{TeV}$ of both sources is $(2-3)\times 10^{35}\,\mathrm{erg}\,\mathrm{s}^{-1}$. This exceeds by more than a factor of 2 the luminosity of HESS J1646$-$458, which is associated with the most massive young star cluster in the Milky Way, Westerlund 1. Furthermore, the $γ$-ray emission from each source is extended with a significance of $>3σ$ and a Gaussian width of about $30\,\mathrm{pc}$. For 30 Dor C, a connection between the $γ$-ray emission and the nonthermal X-ray emission appears likely. Different interpretations of the $γ$-ray signal from R136 are discussed. △ Less

Submitted 23 July, 2024; originally announced July 2024.

Comments: 10+11 pages, 4+6 figures. Corresponding authors: L. Mohrmann, N. Komin

Journal ref: Astrophysical Journal Letters 970, L21 (2024)

ICLGuard: Controlling In-Context Learning Behavior for Applicability Authorization

Authors: Wai Man Si, Michael Backes, Yang Zhang

Abstract: In-context learning (ICL) is a recent advancement in the capabilities of large language models (LLMs). This feature allows users to perform a new task without updating the model. Concretely, users can address tasks during the inference time by conditioning on a few input-label pair demonstrations along with the test input. It is different than the conventional fine-tuning paradigm and offers more… ▽ More In-context learning (ICL) is a recent advancement in the capabilities of large language models (LLMs). This feature allows users to perform a new task without updating the model. Concretely, users can address tasks during the inference time by conditioning on a few input-label pair demonstrations along with the test input. It is different than the conventional fine-tuning paradigm and offers more flexibility. However, this capability also introduces potential issues. For example, users may use the model on any data without restriction, such as performing tasks with improper or sensitive content, which might violate the model policy or conflict with the model owner's interests. As a model owner, it is crucial to establish a mechanism to control the model's behavior under ICL, depending on the model owner's requirements for various content. To this end, we introduce the concept of "applicability authorization" tailored for LLMs, particularly for ICL behavior, and propose a simple approach, ICLGuard. It is a fine-tuning framework designed to allow the model owner to regulate ICL behavior on different data. ICLGuard preserves the original LLM and fine-tunes only a minimal set of additional trainable parameters to "guard" the LLM. Empirical results show that the guarded LLM can deactivate its ICL ability on target data without affecting its ICL ability on other data and its general functionality across all data. △ Less

Submitted 9 July, 2024; originally announced July 2024.

SOS! Soft Prompt Attack Against Open-Source Large Language Models

Authors: Ziqing Yang, Michael Backes, Yang Zhang, Ahmed Salem

Abstract: Open-source large language models (LLMs) have become increasingly popular among both the general public and industry, as they can be customized, fine-tuned, and freely used. However, some open-source LLMs require approval before usage, which has led to third parties publishing their own easily accessible versions. Similarly, third parties have been publishing fine-tuned or quantized variants of th… ▽ More Open-source large language models (LLMs) have become increasingly popular among both the general public and industry, as they can be customized, fine-tuned, and freely used. However, some open-source LLMs require approval before usage, which has led to third parties publishing their own easily accessible versions. Similarly, third parties have been publishing fine-tuned or quantized variants of these LLMs. These versions are particularly appealing to users because of their ease of access and reduced computational resource demands. This trend has increased the risk of training time attacks, compromising the integrity and security of LLMs. In this work, we present a new training time attack, SOS, which is designed to be low in computational demand and does not require clean data or modification of the model weights, thereby maintaining the model's utility intact. The attack addresses security issues in various scenarios, including the backdoor attack, jailbreak attack, and prompt stealing attack. Our experimental findings demonstrate that the proposed attack is effective across all evaluated targets. Furthermore, we present the other side of our SOS technique, namely the copyright token -- a novel technique that enables users to mark their copyrighted content and prevent models from using it. △ Less

Submitted 3 July, 2024; originally announced July 2024.

H.E.S.S. observations of the 2021 periastron passage of PSR B1259-63/LS 2883

Authors: H. E. S. S. Collaboration, F. Aharonian, F. Ait Benkhali, J. Aschersleben, H. Ashkar, M. Backes, V. Barbosa Martins, R. Batzofin, Y. Becherini, D. Berge, K. Bernlöhr, M. Böttcher, C. Boisson, J. Bolmont, M. de Bony de Lavergne, J. Borowska, M. Bouyahiaoui, R. Brose, A. Brown, F. Brun, B. Bruno, T. Bulik, C. Burger-Scheidlin, S. Caroff, S. Casanova , et al. (119 additional authors not shown)

Abstract: PSR B1259-63 is a gamma-ray binary system that hosts a pulsar in an eccentric orbit, with a 3.4 year period, around an O9.5Ve star. At orbital phases close to periastron passages, the system radiates bright and variable non-thermal emission. We report on an extensive VHE observation campaign conducted with the High Energy Stereoscopic System, comprised of ~100 hours of data taken from $t_p-24$ day… ▽ More PSR B1259-63 is a gamma-ray binary system that hosts a pulsar in an eccentric orbit, with a 3.4 year period, around an O9.5Ve star. At orbital phases close to periastron passages, the system radiates bright and variable non-thermal emission. We report on an extensive VHE observation campaign conducted with the High Energy Stereoscopic System, comprised of ~100 hours of data taken from $t_p-24$ days to $t_p+127$ days around the system's 2021 periastron passage. We also present the timing and spectral analyses of the source. The VHE light curve in 2021 is consistent with the stacked light curve of all previous observations. Within the light curve, we report a VHE maximum at times coincident with the third X-ray peak first detected in the 2021 X-ray light curve. In the light curve -- although sparsely sampled in this time period -- we see no VHE enhancement during the second disc crossing. In addition, we see no correspondence to the 2021 GeV flare in the VHE light curve. The VHE spectrum obtained from the analysis of the 2021 dataset is best described by a power law of spectral index $Γ= 2.65 \pm 0.04_{\text{stat}}$ $\pm 0.04_{\text{sys}}$, a value consistent with the previous H.E.S.S. observations of the source. We report spectral variability with a difference of $ΔΓ= 0.56 ~\pm~ 0.18_{\text{stat}}$ $~\pm~0.10_{\text{sys}}$ at 95% c.l., between sub-periods of the 2021 dataset. We also find a linear correlation between contemporaneous flux values of X-ray and TeV datasets, detected mainly after $t_p+25$ days, suggesting a change in the available energy for non-thermal radiation processes. We detect no significant correlation between GeV and TeV flux points, within the uncertainties of the measurements, from $\sim t_p-23$ days to $\sim t_p+126$ days. This suggests that the GeV and TeV emission originate from different electron populations. △ Less

Submitted 26 June, 2024; originally announced June 2024.

Comments: accepted to A&A

Voice Jailbreak Attacks Against GPT-4o

Authors: Xinyue Shen, Yixin Wu, Michael Backes, Yang Zhang

Abstract: Recently, the concept of artificial assistants has evolved from science fiction into real-world applications. GPT-4o, the newest multimodal large language model (MLLM) across audio, vision, and text, has further blurred the line between fiction and reality by enabling more natural human-computer interactions. However, the advent of GPT-4o's voice mode may also introduce a new attack surface. In th… ▽ More Recently, the concept of artificial assistants has evolved from science fiction into real-world applications. GPT-4o, the newest multimodal large language model (MLLM) across audio, vision, and text, has further blurred the line between fiction and reality by enabling more natural human-computer interactions. However, the advent of GPT-4o's voice mode may also introduce a new attack surface. In this paper, we present the first systematic measurement of jailbreak attacks against the voice mode of GPT-4o. We show that GPT-4o demonstrates good resistance to forbidden questions and text jailbreak prompts when directly transferring them to voice mode. This resistance is primarily due to GPT-4o's internal safeguards and the difficulty of adapting text jailbreak prompts to voice mode. Inspired by GPT-4o's human-like behaviors, we propose VoiceJailbreak, a novel voice jailbreak attack that humanizes GPT-4o and attempts to persuade it through fictional storytelling (setting, character, and plot). VoiceJailbreak is capable of generating simple, audible, yet effective jailbreak prompts, which significantly increases the average attack success rate (ASR) from 0.033 to 0.778 in six forbidden scenarios. We also conduct extensive experiments to explore the impacts of interaction steps, key elements of fictional writing, and different languages on VoiceJailbreak's effectiveness and further enhance the attack performance with advanced fictional writing techniques. We hope our study can assist the research community in building more secure and well-regulated MLLMs. △ Less

Submitted 29 May, 2024; originally announced May 2024.

Do You Even Lift? Strengthening Compiler Security Guarantees Against Spectre Attacks

Authors: Xaver Fabian, Marco Patrignani, Marco Guarnieri, Michael Backes

Abstract: Mainstream compilers implement different countermeasures to prevent specific classes of speculative execution attacks. Unfortunately, these countermeasures either lack formal guarantees or come with proofs restricted to speculative semantics capturing only a subset of the speculation mechanisms supported by modern CPUs, thereby limiting their practical applicability. Ideally, these security proofs… ▽ More Mainstream compilers implement different countermeasures to prevent specific classes of speculative execution attacks. Unfortunately, these countermeasures either lack formal guarantees or come with proofs restricted to speculative semantics capturing only a subset of the speculation mechanisms supported by modern CPUs, thereby limiting their practical applicability. Ideally, these security proofs should target a speculative semantics capturing the effects of all speculation mechanisms implemented in modern CPUs. However, this is impractical and requires new secure compilation proofs to support additional speculation mechanisms. In this paper, we address this problem by proposing a novel secure compilation framework that allows lifting the security guarantees provided by Spectre countermeasures from weaker speculative semantics (ignoring some speculation mechanisms) to stronger ones (accounting for the omitted mechanisms) without requiring new secure compilation proofs. Using our lifting framework, we performed the most comprehensive security analysis of Spectre countermeasures implemented in mainstream compilers to date. Our analysis spans 9 different countermeasures against 5 classes of Spectre attacks, which we proved secure against a speculative semantics accounting for five different speculation mechanisms. △ Less

Submitted 16 May, 2024; originally announced May 2024.

Link Stealing Attacks Against Inductive Graph Neural Networks

Authors: Yixin Wu, Xinlei He, Pascal Berrang, Mathias Humbert, Michael Backes, Neil Zhenqiang Gong, Yang Zhang

Abstract: A graph neural network (GNN) is a type of neural network that is specifically designed to process graph-structured data. Typically, GNNs can be implemented in two settings, including the transductive setting and the inductive setting. In the transductive setting, the trained model can only predict the labels of nodes that were observed at the training time. In the inductive setting, the trained mo… ▽ More A graph neural network (GNN) is a type of neural network that is specifically designed to process graph-structured data. Typically, GNNs can be implemented in two settings, including the transductive setting and the inductive setting. In the transductive setting, the trained model can only predict the labels of nodes that were observed at the training time. In the inductive setting, the trained model can be generalized to new nodes/graphs. Due to its flexibility, the inductive setting is the most popular GNN setting at the moment. Previous work has shown that transductive GNNs are vulnerable to a series of privacy attacks. However, a comprehensive privacy analysis of inductive GNN models is still missing. This paper fills the gap by conducting a systematic privacy analysis of inductive GNNs through the lens of link stealing attacks, one of the most popular attacks that are specifically designed for GNNs. We propose two types of link stealing attacks, i.e., posterior-only attacks and combined attacks. We define threat models of the posterior-only attacks with respect to node topology and the combined attacks by considering combinations of posteriors, node attributes, and graph features. Extensive evaluation on six real-world datasets demonstrates that inductive GNNs leak rich information that enables link stealing attacks with advantageous properties. Even attacks with no knowledge about graph structures can be effective. We also show that our attacks are robust to different node similarities and different graph features. As a counterpart, we investigate two possible defenses and discover they are ineffective against our attacks, which calls for more effective defenses. △ Less

Submitted 9 May, 2024; originally announced May 2024.

Comments: To appear in the 24th Privacy Enhancing Technologies Symposium (PETS 2024), July 15-20, 2024

UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Images

Authors: Yiting Qu, Xinyue Shen, Yixin Wu, Michael Backes, Savvas Zannettou, Yang Zhang

Abstract: With the advent of text-to-image models and concerns about their misuse, developers are increasingly relying on image safety classifiers to moderate their generated unsafe images. Yet, the performance of current image safety classifiers remains unknown for both real-world and AI-generated images. In this work, we propose UnsafeBench, a benchmarking framework that evaluates the effectiveness and ro… ▽ More With the advent of text-to-image models and concerns about their misuse, developers are increasingly relying on image safety classifiers to moderate their generated unsafe images. Yet, the performance of current image safety classifiers remains unknown for both real-world and AI-generated images. In this work, we propose UnsafeBench, a benchmarking framework that evaluates the effectiveness and robustness of image safety classifiers, with a particular focus on the impact of AI-generated images on their performance. First, we curate a large dataset of 10K real-world and AI-generated images that are annotated as safe or unsafe based on a set of 11 unsafe categories of images (sexual, violent, hateful, etc.). Then, we evaluate the effectiveness and robustness of five popular image safety classifiers, as well as three classifiers that are powered by general-purpose visual language models. Our assessment indicates that existing image safety classifiers are not comprehensive and effective enough to mitigate the multifaceted problem of unsafe images. Also, there exists a distribution shift between real-world and AI-generated images in image qualities, styles, and layouts, leading to degraded effectiveness and robustness. Motivated by these findings, we build a comprehensive image moderation tool called PerspectiveVision, which addresses the main drawbacks of existing classifiers with improved effectiveness and robustness, especially on AI-generated images. UnsafeBench and PerspectiveVision can aid the research community in better understanding the landscape of image safety classification in the era of generative AI. △ Less

Submitted 5 September, 2024; v1 submitted 6 May, 2024; originally announced May 2024.

Efficient Data-Free Model Stealing with Label Diversity

Authors: Yiyong Liu, Rui Wen, Michael Backes, Yang Zhang

Abstract: Machine learning as a Service (MLaaS) allows users to query the machine learning model in an API manner, which provides an opportunity for users to enjoy the benefits brought by the high-performance model trained on valuable data. This interface boosts the proliferation of machine learning based applications, while on the other hand, it introduces the attack surface for model stealing attacks. Exi… ▽ More Machine learning as a Service (MLaaS) allows users to query the machine learning model in an API manner, which provides an opportunity for users to enjoy the benefits brought by the high-performance model trained on valuable data. This interface boosts the proliferation of machine learning based applications, while on the other hand, it introduces the attack surface for model stealing attacks. Existing model stealing attacks have relaxed their attack assumptions to the data-free setting, while keeping the effectiveness. However, these methods are complex and consist of several components, which obscure the core on which the attack really depends. In this paper, we revisit the model stealing problem from a diversity perspective and demonstrate that keeping the generated data samples more diverse across all the classes is the critical point for improving the attack performance. Based on this conjecture, we provide a simplified attack framework. We empirically signify our conjecture by evaluating the effectiveness of our attack, and experimental results show that our approach is able to achieve comparable or even better performance compared with the state-of-the-art method. Furthermore, benefiting from the absence of redundant components, our method demonstrates its advantages in attack efficiency and query budget. △ Less

Submitted 29 March, 2024; originally announced April 2024.

Unveiling extended gamma-ray emission around HESS J1813-178

Authors: F. Aharonian, F. Ait Benkhali, J. Aschersleben, H. Ashkar, M. Backes, A. Baktash, V. Barbosa Martins, J. Barnard, R. Batzofin, Y. Becherini, D. Berge, K. Bernlöhr, B. Bi, M. Böttcher, C. Boisson, J. Bolmont, M. de Bony de Lavergne, J. Borowska, M. Bouyahiaoui, M. Breuhaus, R. Brose, F. Brun, B. Bruno, T. Bulik, C. Burger-Scheidlin , et al. (126 additional authors not shown)

Abstract: HESS J1813$-$178 is a very-high-energy $γ$-ray source spatially coincident with the young and energetic pulsar PSR J1813$-$1749 and thought to be associated with its pulsar wind nebula (PWN). Recently, evidence for extended high-energy emission in the vicinity of the pulsar has been revealed in the Fermi Large Area Telescope (LAT) data. This motivates revisiting the HESS J1813$-$178 region, taking… ▽ More HESS J1813$-$178 is a very-high-energy $γ$-ray source spatially coincident with the young and energetic pulsar PSR J1813$-$1749 and thought to be associated with its pulsar wind nebula (PWN). Recently, evidence for extended high-energy emission in the vicinity of the pulsar has been revealed in the Fermi Large Area Telescope (LAT) data. This motivates revisiting the HESS J1813$-$178 region, taking advantage of improved analysis methods and an extended data set. Using data taken by the High Energy Stereoscopic System (H.E.S.S.) experiment and the Fermi-LAT, we aim to describe the $γ$-ray emission in the region with a consistent model, to provide insights into its origin. We performed a likelihood-based analysis on 32 hours of H.E.S.S. data and 12 years of Fermi-LAT data and fit a spectro-morphological model to the combined datasets. These results allowed us to develop a physical model for the origin of the observed $γ$-ray emission in the region. In addition to the compact very-high-energy $γ$-ray emission centered on the pulsar, we find a significant yet previously undetected component along the Galactic plane. With Fermi-LAT data, we confirm extended high-energy emission consistent with the position and elongation of the extended emission observed with H.E.S.S. These results establish a consistent description of the emission in the region from GeV energies to several tens of TeV. This study suggests that HESS J1813$-$178 is associated with a $γ$-ray PWN powered by PSR J1813$-$1749. A possible origin of the extended emission component is inverse Compton emission from electrons and positrons that have escaped the confines of the pulsar and form a halo around the PWN. △ Less

Submitted 25 March, 2024; originally announced March 2024.

Comments: 13+5 pages, 13+11 figures. Accepted for publication in A&A. Corresponding authors: T.Wach, A.Mitchell, V.Joshi, P.Chambéry

Journal ref: A&A 686, A149 (2024)

Spectrum and extension of the inverse-Compton emission of the Crab Nebula from a combined Fermi-LAT and H.E.S.S. analysis

Authors: F. Aharonian, F. Ait Benkhali, J. Aschersleben, H. Ashkar, M. Backes, A. Baktash, V. Barbosa Martins, R. Batzofin, Y. Becherini, D. Berge, K. Bernlöhr, B. Bi, M. Böttcher, C. Boisson, J. Bolmont, M. de Bony de Lavergne, J. Borowska, F. Bradascio, M. Breuhaus, R. Brose, A. Brown, F. Brun, B. Bruno, T. Bulik, C. Burger-Scheidlin , et al. (137 additional authors not shown)

Abstract: The Crab Nebula is a unique laboratory for studying the acceleration of electrons and positrons through their non-thermal radiation. Observations of very-high-energy $γ$ rays from the Crab Nebula have provided important constraints for modelling its broadband emission. We present the first fully self-consistent analysis of the Crab Nebula's $γ$-ray emission between 1 GeV and $\sim$100 TeV, that is… ▽ More The Crab Nebula is a unique laboratory for studying the acceleration of electrons and positrons through their non-thermal radiation. Observations of very-high-energy $γ$ rays from the Crab Nebula have provided important constraints for modelling its broadband emission. We present the first fully self-consistent analysis of the Crab Nebula's $γ$-ray emission between 1 GeV and $\sim$100 TeV, that is, over five orders of magnitude in energy. Using the open-source software package Gammapy, we combined 11.4 yr of data from the Fermi Large Area Telescope and 80 h of High Energy Stereoscopic System (H.E.S.S.) data at the event level and provide a measurement of the spatial extension of the nebula and its energy spectrum. We find evidence for a shrinking of the nebula with increasing $γ$-ray energy. Furthermore, we fitted several phenomenological models to the measured data, finding that none of them can fully describe the spatial extension and the spectral energy distribution at the same time. Especially the extension measured at TeV energies appears too large when compared to the X-ray emission. Our measurements probe the structure of the magnetic field between the pulsar wind termination shock and the dust torus, and we conclude that the magnetic field strength decreases with increasing distance from the pulsar. We complement our study with a careful assessment of systematic uncertainties. △ Less

Submitted 21 March, 2024; v1 submitted 19 March, 2024; originally announced March 2024.

Comments: 18+6 pages, 15+2 figures. Accepted for publication in A&A. Corresponding authors: M. Meyer, L. Mohrmann, T. Unbehaun. v2: after A&A language editing

Journal ref: A&A 686, A308 (2024)

Dark Matter Line Searches with the Cherenkov Telescope Array

Authors: S. Abe, J. Abhir, A. Abhishek, F. Acero, A. Acharyya, R. Adam, A. Aguasca-Cabot, I. Agudo, A. Aguirre-Santaella, J. Alfaro, R. Alfaro, N. Alvarez-Crespo, R. Alves Batista, J. -P. Amans, E. Amato, G. Ambrosi, L. Angel, C. Aramo, C. Arcaro, T. T. H. Arnesen, L. Arrabito, K. Asano, Y. Ascasibar, J. Aschersleben, H. Ashkar , et al. (540 additional authors not shown)

Abstract: Monochromatic gamma-ray signals constitute a potential smoking gun signature for annihilating or decaying dark matter particles that could relatively easily be distinguished from astrophysical or instrumental backgrounds. We provide an updated assessment of the sensitivity of the Cherenkov Telescope Array (CTA) to such signals, based on observations of the Galactic centre region as well as of sele… ▽ More Monochromatic gamma-ray signals constitute a potential smoking gun signature for annihilating or decaying dark matter particles that could relatively easily be distinguished from astrophysical or instrumental backgrounds. We provide an updated assessment of the sensitivity of the Cherenkov Telescope Array (CTA) to such signals, based on observations of the Galactic centre region as well as of selected dwarf spheroidal galaxies. We find that current limits and detection prospects for dark matter masses above 300 GeV will be significantly improved, by up to an order of magnitude in the multi-TeV range. This demonstrates that CTA will set a new standard for gamma-ray astronomy also in this respect, as the world's largest and most sensitive high-energy gamma-ray observatory, in particular due to its exquisite energy resolution at TeV energies and the adopted observational strategy focussing on regions with large dark matter densities. Throughout our analysis, we use up-to-date instrument response functions, and we thoroughly model the effect of instrumental systematic uncertainties in our statistical treatment. We further present results for other potential signatures with sharp spectral features, e.g.~box-shaped spectra, that would likewise very clearly point to a particle dark matter origin. △ Less

Submitted 23 July, 2024; v1 submitted 7 March, 2024; originally announced March 2024.

Comments: 44 pages JCAP style (excluding author list and references), 19 figures; minor changes to match published version

Journal ref: JCAP 07 (2024) 047

Curvature in the very-high energy gamma-ray spectrum of M87

Authors: H. E. S. S. Collaboration, F. Aharonian, F. Ait Benkhali, J. Aschersleben, H. Ashkar, M. Backes, V. Barbosa Martins, R. Batzofin, Y. Becherini, D. Berge, K. Bernlöhr, M. Böttcher, C. Boisson, J. Bolmont, M. de Bony de Lavergne, F. Bradascio, R. Brose, F. Brun, B. Bruno, T. Bulik C. Burger-Scheidlin, T. Bylund, S. Casanova, R. Cecil, J. Celic, M. Cerruti , et al. (110 additional authors not shown)

Abstract: The radio galaxy M87 is a variable very-high energy (VHE) gamma-ray source, exhibiting three major flares reported in 2005, 2008, and 2010. Despite extensive studies, the origin of the VHE gamma-ray emission is yet to be understood. In this study, we investigate the VHE gamma-ray spectrum of M87 during states of high gamma-ray activity, utilizing 20.2$\,$ hours the H.E.S.S. observations. Our findi… ▽ More The radio galaxy M87 is a variable very-high energy (VHE) gamma-ray source, exhibiting three major flares reported in 2005, 2008, and 2010. Despite extensive studies, the origin of the VHE gamma-ray emission is yet to be understood. In this study, we investigate the VHE gamma-ray spectrum of M87 during states of high gamma-ray activity, utilizing 20.2$\,$ hours the H.E.S.S. observations. Our findings indicate a preference for a curved spectrum, characterized by a log-parabola model with extra-galactic background light (EBL) model above 0.3$\,$TeV at the 4$σ$ level, compared to a power-law spectrum with EBL. We investigate the degeneracy between the absorption feature and the EBL normalization and derive upper limits on EBL models mainly sensitive in the wavelength range 12.4$\,$$μ$m - 40$\,$$μ$m. △ Less

Submitted 25 April, 2024; v1 submitted 20 February, 2024; originally announced February 2024.

Comments: 10 pages, 7 figures. Accepted for publication in A&A. Corresponding authors: Victor Barbosa Martins, Rahul Cecil, Iryna Lypova, Manuel Meyer, Perri Zilberman. Supplementary material: https://zenodo.org/records/10781524

Journal ref: A&A, 685, A96 (2024)

Instruction Backdoor Attacks Against Customized LLMs

Authors: Rui Zhang, Hongwei Li, Rui Wen, Wenbo Jiang, Yuan Zhang, Michael Backes, Yun Shen, Yang Zhang

Abstract: The increasing demand for customized Large Language Models (LLMs) has led to the development of solutions like GPTs. These solutions facilitate tailored LLM creation via natural language prompts without coding. However, the trustworthiness of third-party custom versions of LLMs remains an essential concern. In this paper, we propose the first instruction backdoor attacks against applications integ… ▽ More The increasing demand for customized Large Language Models (LLMs) has led to the development of solutions like GPTs. These solutions facilitate tailored LLM creation via natural language prompts without coding. However, the trustworthiness of third-party custom versions of LLMs remains an essential concern. In this paper, we propose the first instruction backdoor attacks against applications integrated with untrusted customized LLMs (e.g., GPTs). Specifically, these attacks embed the backdoor into the custom version of LLMs by designing prompts with backdoor instructions, outputting the attacker's desired result when inputs contain the pre-defined triggers. Our attack includes 3 levels of attacks: word-level, syntax-level, and semantic-level, which adopt different types of triggers with progressive stealthiness. We stress that our attacks do not require fine-tuning or any modification to the backend LLMs, adhering strictly to GPTs development guidelines. We conduct extensive experiments on 6 prominent LLMs and 5 benchmark text classification datasets. The results show that our instruction backdoor attacks achieve the desired attack performance without compromising utility. Additionally, we propose two defense strategies and demonstrate their effectiveness in reducing such attacks. Our findings highlight the vulnerability and the potential risks of LLM customization such as GPTs. △ Less

Submitted 28 May, 2024; v1 submitted 14 February, 2024; originally announced February 2024.

Comprehensive Assessment of Jailbreak Attacks Against LLMs

Authors: Junjie Chu, Yugeng Liu, Ziqing Yang, Xinyue Shen, Michael Backes, Yang Zhang

Abstract: Misuse of the Large Language Models (LLMs) has raised widespread concern. To address this issue, safeguards have been taken to ensure that LLMs align with social ethics. However, recent findings have revealed an unsettling vulnerability bypassing the safeguards of LLMs, known as jailbreak attacks. By applying techniques, such as employing role-playing scenarios, adversarial examples, or subtle sub… ▽ More Misuse of the Large Language Models (LLMs) has raised widespread concern. To address this issue, safeguards have been taken to ensure that LLMs align with social ethics. However, recent findings have revealed an unsettling vulnerability bypassing the safeguards of LLMs, known as jailbreak attacks. By applying techniques, such as employing role-playing scenarios, adversarial examples, or subtle subversion of safety objectives as a prompt, LLMs can produce an inappropriate or even harmful response. While researchers have studied several categories of jailbreak attacks, they have done so in isolation. To fill this gap, we present the first large-scale measurement of various jailbreak attack methods. We concentrate on 13 cutting-edge jailbreak methods from four categories, 160 questions from 16 violation categories, and six popular LLMs. Our extensive experimental results demonstrate that the optimized jailbreak prompts consistently achieve the highest attack success rates, as well as exhibit robustness across different LLMs. Some jailbreak prompt datasets, available from the Internet, can also achieve high attack success rates on many LLMs, such as ChatGLM3, GPT-3.5, and PaLM2. Despite the claims from many organizations regarding the coverage of violation categories in their policies, the attack success rates from these categories remain high, indicating the challenges of effectively aligning LLM policies and the ability to counter jailbreak attacks. We also discuss the trade-off between the attack performance and efficiency, as well as show that the transferability of the jailbreak prompts is still viable, becoming an option for black-box models. Overall, our research highlights the necessity of evaluating different jailbreak methods. We hope our study can provide insights for future research on jailbreak attacks and serve as a benchmark tool for evaluating them for practitioners. △ Less

Submitted 8 February, 2024; originally announced February 2024.

Comments: 18 pages, 12 figures

Reconstruct Your Previous Conversations! Comprehensively Investigating Privacy Leakage Risks in Conversations with GPT Models

Authors: Junjie Chu, Zeyang Sha, Michael Backes, Yang Zhang

Abstract: Significant advancements have recently been made in large language models represented by GPT models. Users frequently have multi-round private conversations with cloud-hosted GPT models for task optimization. Yet, this operational paradigm introduces additional attack surfaces, particularly in custom GPTs and hijacked chat sessions. In this paper, we introduce a straightforward yet potent Conversa… ▽ More Significant advancements have recently been made in large language models represented by GPT models. Users frequently have multi-round private conversations with cloud-hosted GPT models for task optimization. Yet, this operational paradigm introduces additional attack surfaces, particularly in custom GPTs and hijacked chat sessions. In this paper, we introduce a straightforward yet potent Conversation Reconstruction Attack. This attack targets the contents of previous conversations between GPT models and benign users, i.e., the benign users' input contents during their interaction with GPT models. The adversary could induce GPT models to leak such contents by querying them with designed malicious prompts. Our comprehensive examination of privacy risks during the interactions with GPT models under this attack reveals GPT-4's considerable resilience. We present two advanced attacks targeting improved reconstruction of past conversations, demonstrating significant privacy leakage across all models under these advanced techniques. Evaluating various defense mechanisms, we find them ineffective against these attacks. Our findings highlight the ease with which privacy can be compromised in interactions with GPT models, urging the community to safeguard against potential abuses of these models' capabilities. △ Less

Submitted 7 October, 2024; v1 submitted 5 February, 2024; originally announced February 2024.

Comments: Accepted in EMNLP 2024. 14 pages, 10 figures

Acceleration and transport of relativistic electrons in the jets of the microquasar SS 433

Authors: F. Aharonian, F. Ait Benkhali, J. Aschersleben, H. Ashkar, M. Backes, V. Barbosa Martins, R. Batzofin, Y. Becherini, D. Berge, K. Bernlöhr, B. Bi, M. Böttcher, C. Boisson, J. Bolmont, M. de Bony de Lavergne, J. Borowska, M. Bouyahiaou, M. Breuhau, R. Brose, A. M. Brown, F. Brun, B. Bruno, T. Bulik, C. Burger-Scheidlin, S. Caroff , et al. (140 additional authors not shown)

Abstract: SS 433 is a microquasar, a stellar binary system with collimated relativistic jets. We observed SS 433 in gamma rays using the High Energy Stereoscopic System (H.E.S.S.), finding an energy-dependent shift in the apparent position of the gamma-ray emission of the parsec-scale jets. These observations trace the energetic electron population and indicate the gamma rays are produced by inverse-Compton… ▽ More SS 433 is a microquasar, a stellar binary system with collimated relativistic jets. We observed SS 433 in gamma rays using the High Energy Stereoscopic System (H.E.S.S.), finding an energy-dependent shift in the apparent position of the gamma-ray emission of the parsec-scale jets. These observations trace the energetic electron population and indicate the gamma rays are produced by inverse-Compton scattering. Modelling of the energy-dependent gamma-ray morphology constrains the location of particle acceleration and requires an abrupt deceleration of the jet flow. We infer the presence of shocks on either side of the binary system at distances of 25 to 30 parsecs and conclude that self-collimation of the precessing jets forms the shocks, which then efficiently accelerate electrons. △ Less

Submitted 29 January, 2024; originally announced January 2024.

Comments: Submitted 20th Apr. 2023, published 25th January 2024 (accepted version)

Journal ref: Science383,402-406(2024)

Memorization in Self-Supervised Learning Improves Downstream Generalization

Authors: Wenhao Wang, Muhammad Ahmad Kaleem, Adam Dziedzic, Michael Backes, Nicolas Papernot, Franziska Boenisch

Abstract: Self-supervised learning (SSL) has recently received significant attention due to its ability to train high-performance encoders purely on unlabeled data-often scraped from the internet. This data can still be sensitive and empirical evidence suggests that SSL encoders memorize private information of their training data and can disclose them at inference time. Since existing theoretical definition… ▽ More Self-supervised learning (SSL) has recently received significant attention due to its ability to train high-performance encoders purely on unlabeled data-often scraped from the internet. This data can still be sensitive and empirical evidence suggests that SSL encoders memorize private information of their training data and can disclose them at inference time. Since existing theoretical definitions of memorization from supervised learning rely on labels, they do not transfer to SSL. To address this gap, we propose SSLMem, a framework for defining memorization within SSL. Our definition compares the difference in alignment of representations for data points and their augmented views returned by both encoders that were trained on these data points and encoders that were not. Through comprehensive empirical analysis on diverse encoder architectures and datasets we highlight that even though SSL relies on large datasets and strong augmentations-both known in supervised learning as regularization techniques that reduce overfitting-still significant fractions of training data points experience high memorization. Through our empirical results, we show that this memorization is essential for encoders to achieve higher generalization performance on different downstream tasks. △ Less

Submitted 18 June, 2024; v1 submitted 19 January, 2024; originally announced January 2024.

Comments: Accepted at ICLR 2024

Optical spectroscopy of blazars for the Cherenkov Telescope Array -- III

Authors: F. D'Ammando, P. Goldoni, W. Max-Moerbeck, J. Becerra Gonzalez, E. Kasai, D. A. Williams, N. Alvarez-Crespo, M. Backes, U. Barres de Almeida, C. Boisson, G. Cotter, V. Fallah Ramazani, O. Hervet, E. Lindfors, D. Mukhi-Nilo, S. Pita, M. Splettstoesser, B. van Soelen

Abstract: Due to their almost featureless optical/UV spectra, it is challenging to measure the redshifts of BL Lacs. As a result, about 50% of gamma-ray BL Lacs lack a firm measurement of this property, which is fundamental for population studies, indirect estimates of the EBL, and fundamental physics probes. This paper is the third in a series of papers aimed at determining the redshift of a sample of blaz… ▽ More Due to their almost featureless optical/UV spectra, it is challenging to measure the redshifts of BL Lacs. As a result, about 50% of gamma-ray BL Lacs lack a firm measurement of this property, which is fundamental for population studies, indirect estimates of the EBL, and fundamental physics probes. This paper is the third in a series of papers aimed at determining the redshift of a sample of blazars selected as prime targets for future observations with the next generation, ground-based VHE gamma-ray astronomy observatory, Cherenkov Telescope Array Observatory (CTAO). The accurate determination of the redshift of these objects is an important aid in source selection and planning of future CTAO observations. The selected targets were expected to be detectable with CTAO in observations of 30 hours or less. We performed deep spectroscopic observations of 41 of these blazars using the Keck II, Lick, SALT, GTC, and ESO/VLT telescopes. We carefully searched for spectral lines in the spectra and whenever features of the host galaxy were detected, we attempted to model the properties of the host galaxy. The magnitudes of the targets at the time of the observations were also compared to their long-term light curves. Spectra from 24 objects display spectral features or a high S/N. From these, 12 spectroscopic redshifts were determined, ranging from 0.2223 to 0.7018. Furthermore, 1 tentative redshift (0.6622) and 2 redshift lower limits at z > 0.6185 and z > 0.6347 were obtained. The other 9 BL Lacs showed featureless spectra, despite the high S/N (> 100) observations. Our comparisons with long-term optical light curves tentatively suggest that redshift measurements are more straightforward during an optical low state of the AGN. Overall, we have determined 37 redshifts and 6 spectroscopic lower limits as part of our programme thus far. △ Less

Submitted 15 January, 2024; originally announced January 2024.

Comments: Accepted for publication in Astronomy and Astrophysics. 17 pages, 4 Figures, 10 Tables

TeV flaring activity of the AGN PKS 0625-354 in November 2018

Authors: H. E. S. S. Collaboration, F. Aharonian, F. Ait Benkhali, J. Aschersleben, H. Ashkar, M. Backes, A. Baktash, V. Barbosa Martins, J. Barnard, R. Batzofin, Y. Becherini, D. Berge, K. Bernlöhr, B. Bi, M. Böttcher, C. Boisson, J. Bolmont, M. de Bony de Lavergne, J. Borowska, F. Bradascio, M. Breuhaus, R. Brose, A. Brown, F. Brun, B. Bruno , et al. (117 additional authors not shown)

Abstract: Most $γ$-ray detected active galactic nuclei are blazars with one of their relativistic jets pointing towards the Earth. Only a few objects belong to the class of radio galaxies or misaligned blazars. Here, we investigate the nature of the object PKS 0625-354, its $γ$-ray flux and spectral variability and its broad-band spectral emission with observations from H.E.S.S., Fermi-LAT, Swift-XRT, and U… ▽ More Most $γ$-ray detected active galactic nuclei are blazars with one of their relativistic jets pointing towards the Earth. Only a few objects belong to the class of radio galaxies or misaligned blazars. Here, we investigate the nature of the object PKS 0625-354, its $γ$-ray flux and spectral variability and its broad-band spectral emission with observations from H.E.S.S., Fermi-LAT, Swift-XRT, and UVOT taken in November 2018. The H.E.S.S. light curve above 200 GeV shows an outburst in the first night of observations followed by a declining flux with a halving time scale of 5.9h. The $γγ$-opacity constrains the upper limit of the angle between the jet and the line of sight to $\sim10^\circ$. The broad-band spectral energy distribution shows two humps and can be well fitted with a single-zone synchrotron self Compton emission model. We conclude that PKS 0625-354, as an object showing clear features of both blazars and radio galaxies, can be classified as an intermediate active galactic nuclei. Multi-wavelength studies of such intermediate objects exhibiting features of both blazars and radio galaxies are sparse but crucial for the understanding of the broad-band emission of $γ$-ray detected active galactic nuclei in general. △ Less

Submitted 13 January, 2024; originally announced January 2024.

Comments: 9 pages, 6 figures, accepted for publication in Astronomy & Astrophysics

TrustLLM: Trustworthiness in Large Language Models

Authors: Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric Xing, Furong Huang, Hao Liu, Heng Ji, Hongyi Wang , et al. (45 additional authors not shown)

Abstract: Large language models (LLMs), exemplified by ChatGPT, have gained considerable attention for their excellent natural language processing capabilities. Nonetheless, these LLMs present many challenges, particularly in the realm of trustworthiness. Therefore, ensuring the trustworthiness of LLMs emerges as an important topic. This paper introduces TrustLLM, a comprehensive study of trustworthiness in… ▽ More Large language models (LLMs), exemplified by ChatGPT, have gained considerable attention for their excellent natural language processing capabilities. Nonetheless, these LLMs present many challenges, particularly in the realm of trustworthiness. Therefore, ensuring the trustworthiness of LLMs emerges as an important topic. This paper introduces TrustLLM, a comprehensive study of trustworthiness in LLMs, including principles for different dimensions of trustworthiness, established benchmark, evaluation, and analysis of trustworthiness for mainstream LLMs, and discussion of open challenges and future directions. Specifically, we first propose a set of principles for trustworthy LLMs that span eight different dimensions. Based on these principles, we further establish a benchmark across six dimensions including truthfulness, safety, fairness, robustness, privacy, and machine ethics. We then present a study evaluating 16 mainstream LLMs in TrustLLM, consisting of over 30 datasets. Our findings firstly show that in general trustworthiness and utility (i.e., functional effectiveness) are positively related. Secondly, our observations reveal that proprietary LLMs generally outperform most open-source counterparts in terms of trustworthiness, raising concerns about the potential risks of widely accessible open-source LLMs. However, a few open-source LLMs come very close to proprietary ones. Thirdly, it is important to note that some LLMs may be overly calibrated towards exhibiting trustworthiness, to the extent that they compromise their utility by mistakenly treating benign prompts as harmful and consequently not responding. Finally, we emphasize the importance of ensuring transparency not only in the models themselves but also in the technologies that underpin trustworthiness. Knowing the specific trustworthy technologies that have been employed is crucial for analyzing their effectiveness. △ Less

Submitted 30 September, 2024; v1 submitted 10 January, 2024; originally announced January 2024.

Comments: This work is still under work and we welcome your contribution

FAKEPCD: Fake Point Cloud Detection via Source Attribution

Authors: Yiting Qu, Zhikun Zhang, Yun Shen, Michael Backes, Yang Zhang

Abstract: To prevent the mischievous use of synthetic (fake) point clouds produced by generative models, we pioneer the study of detecting point cloud authenticity and attributing them to their sources. We propose an attribution framework, FAKEPCD, to attribute (fake) point clouds to their respective generative models (or real-world collections). The main idea of FAKEPCD is to train an attribution model tha… ▽ More To prevent the mischievous use of synthetic (fake) point clouds produced by generative models, we pioneer the study of detecting point cloud authenticity and attributing them to their sources. We propose an attribution framework, FAKEPCD, to attribute (fake) point clouds to their respective generative models (or real-world collections). The main idea of FAKEPCD is to train an attribution model that learns the point cloud features from different sources and further differentiates these sources using an attribution signal. Depending on the characteristics of the training point clouds, namely, sources and shapes, we formulate four attribution scenarios: close-world, open-world, single-shape, and multiple-shape, and evaluate FAKEPCD's performance in each scenario. Extensive experimental results demonstrate the effectiveness of FAKEPCD on source attribution across different scenarios. Take the open-world attribution as an example, FAKEPCD attributes point clouds to known sources with an accuracy of 0.82-0.98 and to unknown sources with an accuracy of 0.73-1.00. Additionally, we introduce an approach to visualize unique patterns (fingerprints) in point clouds associated with each source. This explains how FAKEPCD recognizes point clouds from various sources by focusing on distinct areas within them. Overall, we hope our study establishes a baseline for the source attribution of (fake) point clouds. △ Less

Submitted 18 December, 2023; originally announced December 2023.

Comments: To Appear in the 19th ACM ASIA Conference on Computer and Communications Security, July 1-5, 2024

Comprehensive Assessment of Toxicity in ChatGPT

Authors: Boyang Zhang, Xinyue Shen, Wai Man Si, Zeyang Sha, Zeyuan Chen, Ahmed Salem, Yun Shen, Michael Backes, Yang Zhang

Abstract: Moderating offensive, hateful, and toxic language has always been an important but challenging topic in the domain of safe use in NLP. The emerging large language models (LLMs), such as ChatGPT, can potentially further accentuate this threat. Previous works have discovered that ChatGPT can generate toxic responses using carefully crafted inputs. However, limited research has been done to systemati… ▽ More Moderating offensive, hateful, and toxic language has always been an important but challenging topic in the domain of safe use in NLP. The emerging large language models (LLMs), such as ChatGPT, can potentially further accentuate this threat. Previous works have discovered that ChatGPT can generate toxic responses using carefully crafted inputs. However, limited research has been done to systematically examine when ChatGPT generates toxic responses. In this paper, we comprehensively evaluate the toxicity in ChatGPT by utilizing instruction-tuning datasets that closely align with real-world scenarios. Our results show that ChatGPT's toxicity varies based on different properties and settings of the prompts, including tasks, domains, length, and languages. Notably, prompts in creative writing tasks can be 2x more likely than others to elicit toxic responses. Prompting in German and Portuguese can also double the response toxicity. Additionally, we discover that certain deliberately toxic prompts, designed in earlier studies, no longer yield harmful responses. We hope our discoveries can guide model developers to better regulate these AI systems and the users to avoid undesirable outputs. △ Less

Submitted 3 November, 2023; originally announced November 2023.

Investigating the Lorentz Invariance Violation effect using different cosmological backgrounds

Authors: Hassan Abdalla, Garret Cotter, Michael Backes, Eli Kasai, Markus Böttcher

Abstract: Familiar concepts in physics, such as Lorentz symmetry, are expected to be broken at energies approaching the Planck energy scale as predicted by several quantum-gravity theories. However, such very large energies are unreachable by current experiments on Earth. Current and future Cherenkov telescope facilities may have the capability to measure the accumulated deformation from Lorentz symmetry fo… ▽ More Familiar concepts in physics, such as Lorentz symmetry, are expected to be broken at energies approaching the Planck energy scale as predicted by several quantum-gravity theories. However, such very large energies are unreachable by current experiments on Earth. Current and future Cherenkov telescope facilities may have the capability to measure the accumulated deformation from Lorentz symmetry for photons traveling over large distances via energy-dependent time delays. One of the best natural laboratories to test Lorentz Invariance Violation~(LIV) signatures are Gamma-ray bursts~(GRBs). The calculation of time delays due to the LIV effect depends on the cosmic expansion history. In most of the previous works calculating time lags due to the LIV effect, the standard $Λ$CDM (or concordance) cosmological model is assumed. In this paper, we investigate whether the LIV signature is significantly different when assuming alternatives to the $Λ$CDM cosmological model. Specifically, we consider cosmological models with a non-trivial dark-energy equation of state ($w \neq -1$), such as the standard Chevallier-Polarski-Linder~(CPL) parameterization, the quadratic parameterization of the dark-energy equation of state, and the Pade parameterizations. We find that the relative difference in the predicted time lags is small, of the order of at most a few percent, and thus likely smaller than the systematic errors of possible measurements currently or in the near future. △ Less

Submitted 21 November, 2023; originally announced November 2023.

Comments: Accepted for publication in Class. Quantum Grav., 10 pages, 1 figure

Report number: IPARCOS-UCM-23-127

Generated Distributions Are All You Need for Membership Inference Attacks Against Generative Models

Authors: Minxing Zhang, Ning Yu, Rui Wen, Michael Backes, Yang Zhang

Abstract: Generative models have demonstrated revolutionary success in various visual creation tasks, but in the meantime, they have been exposed to the threat of leaking private information of their training data. Several membership inference attacks (MIAs) have been proposed to exhibit the privacy vulnerability of generative models by classifying a query image as a training dataset member or nonmember. Ho… ▽ More Generative models have demonstrated revolutionary success in various visual creation tasks, but in the meantime, they have been exposed to the threat of leaking private information of their training data. Several membership inference attacks (MIAs) have been proposed to exhibit the privacy vulnerability of generative models by classifying a query image as a training dataset member or nonmember. However, these attacks suffer from major limitations, such as requiring shadow models and white-box access, and either ignoring or only focusing on the unique property of diffusion models, which block their generalization to multiple generative models. In contrast, we propose the first generalized membership inference attack against a variety of generative models such as generative adversarial networks, [variational] autoencoders, implicit functions, and the emerging diffusion models. We leverage only generated distributions from target generators and auxiliary non-member datasets, therefore regarding target generators as black boxes and agnostic to their architectures or application scenarios. Experiments validate that all the generative models are vulnerable to our attack. For instance, our work achieves attack AUC $>0.99$ against DDPM, DDIM, and FastDPM trained on CIFAR-10 and CelebA. And the attack against VQGAN, LDM (for the text-conditional generation), and LIIF achieves AUC $>0.90.$ As a result, we appeal to our community to be aware of such privacy leakage risks when designing and publishing generative models. △ Less

Submitted 30 October, 2023; originally announced October 2023.

Event-by-event Comparison between Machine-Learning- and Transfer-Matrix-based Unfolding Methods

Authors: Mathias Backes, Anja Butter, Monica Dunford, Bogdan Malaescu

Abstract: The unfolding of detector effects is a key aspect of comparing experimental data with theoretical predictions. In recent years, different Machine-Learning methods have been developed to provide novel features, e.g. high dimensionality or a probabilistic single-event unfolding based on generative neural networks. Traditionally, many analyses unfold detector effects using transfer-matrix--based algo… ▽ More The unfolding of detector effects is a key aspect of comparing experimental data with theoretical predictions. In recent years, different Machine-Learning methods have been developed to provide novel features, e.g. high dimensionality or a probabilistic single-event unfolding based on generative neural networks. Traditionally, many analyses unfold detector effects using transfer-matrix--based algorithms, which are well established in low-dimensional unfolding. They yield an unfolded distribution of the total spectrum, together with its covariance matrix. This paper proposes a method to obtain probabilistic single-event unfolded distributions, together with their uncertainties and correlations, for the transfer-matrix--based unfolding. The algorithm is first validated on a toy model and then applied to pseudo-data for the $pp\rightarrow Zγγ$ process. In both examples the performance is compared to the single-event unfolding of the Machine-Learning--based Iterative cINN unfolding (IcINN). △ Less

Submitted 25 October, 2023; originally announced October 2023.

Comments: 22 pages, 11 figures

On the Proactive Generation of Unsafe Images From Text-To-Image Models Using Benign Prompts

Authors: Yixin Wu, Ning Yu, Michael Backes, Yun Shen, Yang Zhang

Abstract: Text-to-image models like Stable Diffusion have had a profound impact on daily life by enabling the generation of photorealistic images from textual prompts, fostering creativity, and enhancing visual experiences across various applications. However, these models also pose risks. Previous studies have successfully demonstrated that manipulated prompts can elicit text-to-image models to generate un… ▽ More Text-to-image models like Stable Diffusion have had a profound impact on daily life by enabling the generation of photorealistic images from textual prompts, fostering creativity, and enhancing visual experiences across various applications. However, these models also pose risks. Previous studies have successfully demonstrated that manipulated prompts can elicit text-to-image models to generate unsafe images, e.g., hateful meme variants. Yet, these studies only unleash the harmful power of text-to-image models in a passive manner. In this work, we focus on the proactive generation of unsafe images using targeted benign prompts via poisoning attacks. We propose two poisoning attacks: a basic attack and a utility-preserving attack. We qualitatively and quantitatively evaluate the proposed attacks using four representative hateful memes and multiple query prompts. Experimental results indicate that text-to-image models are vulnerable to the basic attack even with five poisoning samples. However, the poisoning effect can inadvertently spread to non-targeted prompts, leading to undesirable side effects. Root cause analysis identifies conceptual similarity as an important contributing factor to the side effects. To address this, we introduce the utility-preserving attack as a viable mitigation strategy to maintain the attack stealthiness, while ensuring decent attack performance. Our findings underscore the potential risks of adopting text-to-image models in real-world scenarios, calling for future research and safety measures in this space. △ Less

Submitted 25 October, 2023; originally announced October 2023.

Comments: 18 pages, 31 figures

SecurityNet: Assessing Machine Learning Vulnerabilities on Public Models

Authors: Boyang Zhang, Zheng Li, Ziqing Yang, Xinlei He, Michael Backes, Mario Fritz, Yang Zhang

Abstract: While advanced machine learning (ML) models are deployed in numerous real-world applications, previous works demonstrate these models have security and privacy vulnerabilities. Various empirical research has been done in this field. However, most of the experiments are performed on target ML models trained by the security researchers themselves. Due to the high computational resource requirement f… ▽ More While advanced machine learning (ML) models are deployed in numerous real-world applications, previous works demonstrate these models have security and privacy vulnerabilities. Various empirical research has been done in this field. However, most of the experiments are performed on target ML models trained by the security researchers themselves. Due to the high computational resource requirement for training advanced models with complex architectures, researchers generally choose to train a few target models using relatively simple architectures on typical experiment datasets. We argue that to understand ML models' vulnerabilities comprehensively, experiments should be performed on a large set of models trained with various purposes (not just the purpose of evaluating ML attacks and defenses). To this end, we propose using publicly available models with weights from the Internet (public models) for evaluating attacks and defenses on ML models. We establish a database, namely SecurityNet, containing 910 annotated image classification models. We then analyze the effectiveness of several representative attacks/defenses, including model stealing attacks, membership inference attacks, and backdoor detection on these public models. Our evaluation empirically shows the performance of these attacks/defenses can vary significantly on public models compared to self-trained models. We share SecurityNet with the research community. and advocate researchers to perform experiments on public models to better demonstrate their proposed methods' effectiveness in the future. △ Less

Submitted 19 October, 2023; originally announced October 2023.

Comments: To appear in the 33rd USENIX Security Symposium, August 2024, Philadelphia, PA, USA

Quantifying Privacy Risks of Prompts in Visual Prompt Learning

Authors: Yixin Wu, Rui Wen, Michael Backes, Pascal Berrang, Mathias Humbert, Yun Shen, Yang Zhang

Abstract: Large-scale pre-trained models are increasingly adapted to downstream tasks through a new paradigm called prompt learning. In contrast to fine-tuning, prompt learning does not update the pre-trained model's parameters. Instead, it only learns an input perturbation, namely prompt, to be added to the downstream task data for predictions. Given the fast development of prompt learning, a well-generali… ▽ More Large-scale pre-trained models are increasingly adapted to downstream tasks through a new paradigm called prompt learning. In contrast to fine-tuning, prompt learning does not update the pre-trained model's parameters. Instead, it only learns an input perturbation, namely prompt, to be added to the downstream task data for predictions. Given the fast development of prompt learning, a well-generalized prompt inevitably becomes a valuable asset as significant effort and proprietary data are used to create it. This naturally raises the question of whether a prompt may leak the proprietary information of its training data. In this paper, we perform the first comprehensive privacy assessment of prompts learned by visual prompt learning through the lens of property inference and membership inference attacks. Our empirical evaluation shows that the prompts are vulnerable to both attacks. We also demonstrate that the adversary can mount a successful property inference attack with limited cost. Moreover, we show that membership inference attacks against prompts can be successful with relaxed adversarial assumptions. We further make some initial investigations on the defenses and observe that our method can mitigate the membership inference attacks with a decent utility-defense trade-off but fails to defend against property inference attacks. We hope our results can shed light on the privacy risks of the popular prompt learning paradigm. To facilitate the research in this direction, we will share our code and models with the community. △ Less

Submitted 18 October, 2023; originally announced October 2023.

Comments: To appear in the 33rd USENIX Security Symposium, August 14-16, 2024

Revisiting Transferable Adversarial Image Examples: Attack Categorization, Evaluation Guidelines, and New Insights

Authors: Zhengyu Zhao, Hanwei Zhang, Renjue Li, Ronan Sicre, Laurent Amsaleg, Michael Backes, Qi Li, Chao Shen

Abstract: Transferable adversarial examples raise critical security concerns in real-world, black-box attack scenarios. However, in this work, we identify two main problems in common evaluation practices: (1) For attack transferability, lack of systematic, one-to-one attack comparison and fair hyperparameter settings. (2) For attack stealthiness, simply no comparisons. To address these problems, we establis… ▽ More Transferable adversarial examples raise critical security concerns in real-world, black-box attack scenarios. However, in this work, we identify two main problems in common evaluation practices: (1) For attack transferability, lack of systematic, one-to-one attack comparison and fair hyperparameter settings. (2) For attack stealthiness, simply no comparisons. To address these problems, we establish new evaluation guidelines by (1) proposing a novel attack categorization strategy and conducting systematic and fair intra-category analyses on transferability, and (2) considering diverse imperceptibility metrics and finer-grained stealthiness characteristics from the perspective of attack traceback. To this end, we provide the first large-scale evaluation of transferable adversarial examples on ImageNet, involving 23 representative attacks against 9 representative defenses. Our evaluation leads to a number of new insights, including consensus-challenging ones: (1) Under a fair attack hyperparameter setting, one early attack method, DI, actually outperforms all the follow-up methods. (2) A state-of-the-art defense, DiffPure, actually gives a false sense of (white-box) security since it is indeed largely bypassed by our (black-box) transferable attacks. (3) Even when all attacks are bounded by the same $L_p$ norm, they lead to dramatically different stealthiness performance, which negatively correlates with their transferability performance. Overall, our work demonstrates that existing problematic evaluations have indeed caused misleading conclusions and missing points, and as a result, hindered the assessment of the actual progress in this field. △ Less

Submitted 18 October, 2023; originally announced October 2023.

Comments: Code is available at https://github.com/ZhengyuZhao/TransferAttackEval

Last One Standing: A Comparative Analysis of Security and Privacy of Soft Prompt Tuning, LoRA, and In-Context Learning

Authors: Rui Wen, Tianhao Wang, Michael Backes, Yang Zhang, Ahmed Salem

Abstract: Large Language Models (LLMs) are powerful tools for natural language processing, enabling novel applications and user experiences. However, to achieve optimal performance, LLMs often require adaptation with private data, which poses privacy and security challenges. Several techniques have been proposed to adapt LLMs with private data, such as Low-Rank Adaptation (LoRA), Soft Prompt Tuning (SPT), a… ▽ More Large Language Models (LLMs) are powerful tools for natural language processing, enabling novel applications and user experiences. However, to achieve optimal performance, LLMs often require adaptation with private data, which poses privacy and security challenges. Several techniques have been proposed to adapt LLMs with private data, such as Low-Rank Adaptation (LoRA), Soft Prompt Tuning (SPT), and In-Context Learning (ICL), but their comparative privacy and security properties have not been systematically investigated. In this work, we fill this gap by evaluating the robustness of LoRA, SPT, and ICL against three types of well-established attacks: membership inference, which exposes data leakage (privacy); backdoor, which injects malicious behavior (security); and model stealing, which can violate intellectual property (privacy and security). Our results show that there is no silver bullet for privacy and security in LLM adaptation and each technique has different strengths and weaknesses. △ Less

Submitted 17 October, 2023; originally announced October 2023.

Provably Robust Cost-Sensitive Learning via Randomized Smoothing

Authors: Yuan Xin, Michael Backes, Xiao Zhang

Abstract: We study the problem of robust learning against adversarial perturbations under cost-sensitive scenarios, where the potential harm of different types of misclassifications is encoded in a cost matrix. Existing approaches are either empirical and cannot certify robustness or suffer from inherent scalability issues. In this work, we investigate whether randomized smoothing, a scalable framework for… ▽ More We study the problem of robust learning against adversarial perturbations under cost-sensitive scenarios, where the potential harm of different types of misclassifications is encoded in a cost matrix. Existing approaches are either empirical and cannot certify robustness or suffer from inherent scalability issues. In this work, we investigate whether randomized smoothing, a scalable framework for robustness certification, can be leveraged to certify and train for cost-sensitive robustness. Built upon the notion of cost-sensitive certified radius, we first illustrate how to adapt the standard certification algorithm of randomized smoothing to produce tight robustness certificates for any binary cost matrix, and then develop a robust training method to promote certified cost-sensitive robustness while maintaining the model's overall accuracy. Through extensive experiments on image benchmarks, we demonstrate the superiority of our proposed certification algorithm and training method under various cost-sensitive scenarios. Our implementation is available as open source code at: https://github.com/TrustMLRG/CS-RS. △ Less

Submitted 30 May, 2024; v1 submitted 12 October, 2023; originally announced October 2023.

Comments: 19 pages, 9 tables, 5 figures

Composite Backdoor Attacks Against Large Language Models

Authors: Hai Huang, Zhengyu Zhao, Michael Backes, Yun Shen, Yang Zhang

Abstract: Large language models (LLMs) have demonstrated superior performance compared to previous methods on various tasks, and often serve as the foundation models for many researches and services. However, the untrustworthy third-party LLMs may covertly introduce vulnerabilities for downstream tasks. In this paper, we explore the vulnerability of LLMs through the lens of backdoor attacks. Different from… ▽ More Large language models (LLMs) have demonstrated superior performance compared to previous methods on various tasks, and often serve as the foundation models for many researches and services. However, the untrustworthy third-party LLMs may covertly introduce vulnerabilities for downstream tasks. In this paper, we explore the vulnerability of LLMs through the lens of backdoor attacks. Different from existing backdoor attacks against LLMs, ours scatters multiple trigger keys in different prompt components. Such a Composite Backdoor Attack (CBA) is shown to be stealthier than implanting the same multiple trigger keys in only a single component. CBA ensures that the backdoor is activated only when all trigger keys appear. Our experiments demonstrate that CBA is effective in both natural language processing (NLP) and multimodal tasks. For instance, with $3\%$ poisoning samples against the LLaMA-7B model on the Emotion dataset, our attack achieves a $100\%$ Attack Success Rate (ASR) with a False Triggered Rate (FTR) below $2.06\%$ and negligible model accuracy degradation. Our work highlights the necessity of increased security research on the trustworthiness of foundation LLMs. △ Less

Submitted 30 March, 2024; v1 submitted 11 October, 2023; originally announced October 2023.

Comments: To Appear in Findings of the Association for Computational Linguistics: NAACL 2024, June 2024

Prompt Backdoors in Visual Prompt Learning

Authors: Hai Huang, Zhengyu Zhao, Michael Backes, Yun Shen, Yang Zhang

Abstract: Fine-tuning large pre-trained computer vision models is infeasible for resource-limited users. Visual prompt learning (VPL) has thus emerged to provide an efficient and flexible alternative to model fine-tuning through Visual Prompt as a Service (VPPTaaS). Specifically, the VPPTaaS provider optimizes a visual prompt given downstream data, and downstream users can use this prompt together with the… ▽ More Fine-tuning large pre-trained computer vision models is infeasible for resource-limited users. Visual prompt learning (VPL) has thus emerged to provide an efficient and flexible alternative to model fine-tuning through Visual Prompt as a Service (VPPTaaS). Specifically, the VPPTaaS provider optimizes a visual prompt given downstream data, and downstream users can use this prompt together with the large pre-trained model for prediction. However, this new learning paradigm may also pose security risks when the VPPTaaS provider instead provides a malicious visual prompt. In this paper, we take the first step to explore such risks through the lens of backdoor attacks. Specifically, we propose BadVisualPrompt, a simple yet effective backdoor attack against VPL. For example, poisoning $5\%$ CIFAR10 training data leads to above $99\%$ attack success rates with only negligible model accuracy drop by $1.5\%$. In particular, we identify and then address a new technical challenge related to interactions between the backdoor trigger and visual prompt, which does not exist in conventional, model-level backdoors. Moreover, we provide in-depth analyses of seven backdoor defenses from model, prompt, and input levels. Overall, all these defenses are either ineffective or impractical to mitigate our BadVisualPrompt, implying the critical vulnerability of VPL. △ Less

Submitted 11 October, 2023; originally announced October 2023.

Chasing Gravitational Waves with the Cherenkov Telescope Array

Authors: Jarred Gershon Green, Alessandro Carosi, Lara Nava, Barbara Patricelli, Fabian Schüssler, Monica Seglar-Arroyo, Cta Consortium, :, Kazuki Abe, Shotaro Abe, Atreya Acharyya, Remi Adam, Arnau Aguasca-Cabot, Ivan Agudo, Jorge Alfaro, Nuria Alvarez-Crespo, Rafael Alves Batista, Jean-Philippe Amans, Elena Amato, Filippo Ambrosino, Ekrem Oguzhan Angüner, Lucio Angelo Antonelli, Carla Aramo, Cornelia Arcaro, Luisa Arrabito , et al. (545 additional authors not shown)

Abstract: The detection of gravitational waves from a binary neutron star merger by Advanced LIGO and Advanced Virgo (GW170817), along with the discovery of the electromagnetic counterparts of this gravitational wave event, ushered in a new era of multimessenger astronomy, providing the first direct evidence that BNS mergers are progenitors of short gamma-ray bursts (GRBs). Such events may also produce very… ▽ More The detection of gravitational waves from a binary neutron star merger by Advanced LIGO and Advanced Virgo (GW170817), along with the discovery of the electromagnetic counterparts of this gravitational wave event, ushered in a new era of multimessenger astronomy, providing the first direct evidence that BNS mergers are progenitors of short gamma-ray bursts (GRBs). Such events may also produce very-high-energy (VHE, > 100GeV) photons which have yet to be detected in coincidence with a gravitational wave signal. The Cherenkov Telescope Array (CTA) is a next-generation VHE observatory which aims to be indispensable in this search, with an unparalleled sensitivity and ability to slew anywhere on the sky within a few tens of seconds. New observing modes and follow-up strategies are being developed for CTA to rapidly cover localization areas of gravitational wave events that are typically larger than the CTA field of view. This work will evaluate and provide estimations on the expected number of of gravitational wave events that will be observable with CTA, considering both on- and off-axis emission. In addition, we will present and discuss the prospects of potential follow-up strategies with CTA. △ Less

Submitted 5 February, 2024; v1 submitted 11 October, 2023; originally announced October 2023.

Comments: Presented at the 38th International Cosmic Ray Conference (ICRC 2023), 2023 (arXiv:2309.08219)

Report number: CTA-ICRC/2023/30

Discovery of a Radiation Component from the Vela Pulsar Reaching 20 Teraelectronvolts

Authors: The H. E. S. S. Collaboration, :, F. Aharonian, F. Ait Benkhali, J. Aschersleben, H. Ashkar, M. Backes, V. Barbosa Martins, R. Batzofin, Y. Becherini, D. Berge, K. Bernlöhr, B. Bi, M. Böttcher, C. Boisson, J. Bolmont, M. de Bony de Lavergne, J. Borowska, F. Bradascio, M. Breuhaus, R. Brose, F. Brun, B. Bruno, T. Bulik, C. Burger-Scheidlin , et al. (157 additional authors not shown)

Abstract: Gamma-ray observations have established energetic isolated pulsars as outstanding particle accelerators and antimatter factories in the Galaxy. There is, however, no consensus regarding the acceleration mechanisms and the radiative processes at play, nor the locations where these take place. The spectra of all observed gamma-ray pulsars to date show strong cutoffs or a break above energies of a fe… ▽ More Gamma-ray observations have established energetic isolated pulsars as outstanding particle accelerators and antimatter factories in the Galaxy. There is, however, no consensus regarding the acceleration mechanisms and the radiative processes at play, nor the locations where these take place. The spectra of all observed gamma-ray pulsars to date show strong cutoffs or a break above energies of a few gigaelectronvolt (GeV). Using the H.E.S.S. array of Cherenkov telescopes, we discovered a novel radiation component emerging beyond this generic GeV cutoff in the Vela pulsar's broadband spectrum. The extension of gamma-ray pulsation energies up to at least 20 teraelectronvolts (TeV) shows that Vela pulsar can accelerate particles to Lorentz factors higher than $4\times10^7$. This is an order of magnitude larger than in the case of the Crab pulsar, the only other pulsar detected in the TeV energy range. Our results challenge the state-of-the-art models for high-energy emission of pulsars while providing a new probe, i.e. the energetic multi-TeV component, for constraining the acceleration and emission processes in their extreme energy limit. △ Less

Submitted 9 October, 2023; originally announced October 2023.

Comments: 38 pages, 6 figures. This preprint has not undergone peer review or any post-submission improvements or corrections. The Version of Record of this article is published in Nature Astronomy, Nat Astron (2023), and is available online at https://doi.org/10.1038/s41550-023-02052-3

Transferable Availability Poisoning Attacks

Authors: Yiyong Liu, Michael Backes, Xiao Zhang

Abstract: We consider availability data poisoning attacks, where an adversary aims to degrade the overall test accuracy of a machine learning model by crafting small perturbations to its training data. Existing poisoning strategies can achieve the attack goal but assume the victim to employ the same learning method as what the adversary uses to mount the attack. In this paper, we argue that this assumption… ▽ More We consider availability data poisoning attacks, where an adversary aims to degrade the overall test accuracy of a machine learning model by crafting small perturbations to its training data. Existing poisoning strategies can achieve the attack goal but assume the victim to employ the same learning method as what the adversary uses to mount the attack. In this paper, we argue that this assumption is strong, since the victim may choose any learning algorithm to train the model as long as it can achieve some targeted performance on clean data. Empirically, we observe a large decrease in the effectiveness of prior poisoning attacks if the victim employs an alternative learning algorithm. To enhance the attack transferability, we propose Transferable Poisoning, which first leverages the intrinsic characteristics of alignment and uniformity to enable better unlearnability within contrastive learning, and then iteratively utilizes the gradient information from supervised and unsupervised contrastive learning paradigms to generate the poisoning perturbations. Through extensive experiments on image benchmarks, we show that our transferable poisoning attack can produce poisoned samples with significantly improved transferability, not only applicable to the two learners used to devise the attack but also to learning algorithms and even paradigms beyond. △ Less

Submitted 6 June, 2024; v1 submitted 8 October, 2023; originally announced October 2023.