Showing 51–100 of 1,512 results for author: Yang, Q

Tailoring light holes in $β$-$Ga_{2}O_{3}$ via Anion-Anion Antibonding Coupling

Authors: Ke Xu, Qiaolin Yang, Wenhao Liu, Rong Zhang, Zhi Wang, Jiandong Ye

Abstract: A significant limitation of wide-bandgap materials is their low hole mobility related to localized holes with heavy effective masses ($m_h^*$). We identify in low-symmetric wide-bandgap compounds an anion-anion antibonding coupling (AAAC) effect as the intrinsic factor behind hole localization, which explains the extremely heavy $m_h^*$ and self-trapped hole (STH) formation observed in gallium oxi… ▽ More A significant limitation of wide-bandgap materials is their low hole mobility related to localized holes with heavy effective masses ($m_h^*$). We identify in low-symmetric wide-bandgap compounds an anion-anion antibonding coupling (AAAC) effect as the intrinsic factor behind hole localization, which explains the extremely heavy $m_h^*$ and self-trapped hole (STH) formation observed in gallium oxide ($β$-$Ga_{2}O_{3}$). We propose a design principle for achieving light holes by manipulating AAAC, demonstrating that specific strain conditions can reduce $m_h^*$ in $β$-$Ga_{2}O_{3}$ from 4.77 $m_0$ to 0.38 $m_0$, making it comparable to the electron mass (0.28 $m_0$), while also suppressing STH. The light holes show significant anisotropy, potentially enabling two-dimensional transport in bulk material. This study provides a fundamental understanding of hole mass enhancement and STH formation in novel wide-bandgap materials and suggest new pathways for engineering hole mobilities. △ Less

Submitted 16 August, 2024; originally announced August 2024.

Comments: 22 pages, 1 table, 5 figures

Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling

Authors: Xianzhen Luo, Yixuan Wang, Qingfu Zhu, Zhiming Zhang, Xuanyu Zhang, Qing Yang, Dongliang Xu, Wanxiang Che

Abstract: The rapid growth in the parameters of large language models (LLMs) has made inference latency a fundamental bottleneck, limiting broader application of LLMs. Speculative decoding represents a lossless approach to accelerate inference through a guess-and-verify paradigm, leveraging the parallel capabilities of modern hardware. Some speculative decoding methods rely on additional structures to guess… ▽ More The rapid growth in the parameters of large language models (LLMs) has made inference latency a fundamental bottleneck, limiting broader application of LLMs. Speculative decoding represents a lossless approach to accelerate inference through a guess-and-verify paradigm, leveraging the parallel capabilities of modern hardware. Some speculative decoding methods rely on additional structures to guess draft tokens, such as small models or parameter-efficient architectures, which need extra training before use. Alternatively, retrieval-based train-free techniques build libraries from pre-existing corpora or by n-gram generation. However, they face challenges like large storage requirements, time-consuming retrieval, and limited adaptability. Observing that candidate tokens generated during the decoding process are likely to reoccur in future sequences, we propose Token Recycling. This approach stores candidate tokens in an adjacency matrix and employs a breadth-first search (BFS)-like algorithm on the matrix to construct a draft tree. The tree is then validated through tree attention. New candidate tokens from the decoding process are then used to update the matrix. Token Recycling requires \textless2MB of additional storage and achieves approximately 2x speedup across all sizes of LLMs. It significantly outperforms existing train-free methods by 30\% and even a training method by 25\%. It can be directly applied to any existing LLMs and tasks without the need for adaptation. △ Less

Submitted 16 August, 2024; originally announced August 2024.

Comments: under review

Focus on Focus: Focus-oriented Representation Learning and Multi-view Cross-modal Alignment for Glioma Grading

Authors: Li Pan, Yupei Zhang, Qiushi Yang, Tan Li, Xiaohan Xing, Maximus C. F. Yeung, Zhen Chen

Abstract: Recently, multimodal deep learning, which integrates histopathology slides and molecular biomarkers, has achieved a promising performance in glioma grading. Despite great progress, due to the intra-modality complexity and inter-modality heterogeneity, existing studies suffer from inadequate histopathology representation learning and inefficient molecular-pathology knowledge alignment. These two is… ▽ More Recently, multimodal deep learning, which integrates histopathology slides and molecular biomarkers, has achieved a promising performance in glioma grading. Despite great progress, due to the intra-modality complexity and inter-modality heterogeneity, existing studies suffer from inadequate histopathology representation learning and inefficient molecular-pathology knowledge alignment. These two issues hinder existing methods to precisely interpret diagnostic molecular-pathology features, thereby limiting their grading performance. Moreover, the real-world applicability of existing multimodal approaches is significantly restricted as molecular biomarkers are not always available during clinical deployment. To address these problems, we introduce a novel Focus on Focus (FoF) framework with paired pathology-genomic training and applicable pathology-only inference, enhancing molecular-pathology representation effectively. Specifically, we propose a Focus-oriented Representation Learning (FRL) module to encourage the model to identify regions positively or negatively related to glioma grading and guide it to focus on the diagnostic areas with a consistency constraint. To effectively link the molecular biomarkers to morphological features, we propose a Multi-view Cross-modal Alignment (MCA) module that projects histopathology representations into molecular subspaces, aligning morphological features with corresponding molecular biomarker status by supervised contrastive learning. Experiments on the TCGA GBM-LGG dataset demonstrate that our FoF framework significantly improves the glioma grading. Remarkably, our FoF achieves superior performance using only histopathology slides compared to existing multimodal methods. The source code is available at https://github.com/peterlipan/FoF. △ Less

Submitted 16 August, 2024; originally announced August 2024.

Cross-Platform Video Person ReID: A New Benchmark Dataset and Adaptation Approach

Authors: Shizhou Zhang, Wenlong Luo, De Cheng, Qingchun Yang, Lingyan Ran, Yinghui Xing, Yanning Zhang

Abstract: In this paper, we construct a large-scale benchmark dataset for Ground-to-Aerial Video-based person Re-Identification, named G2A-VReID, which comprises 185,907 images and 5,576 tracklets, featuring 2,788 distinct identities. To our knowledge, this is the first dataset for video ReID under Ground-to-Aerial scenarios. G2A-VReID dataset has the following characteristics: 1) Drastic view changes; 2) L… ▽ More In this paper, we construct a large-scale benchmark dataset for Ground-to-Aerial Video-based person Re-Identification, named G2A-VReID, which comprises 185,907 images and 5,576 tracklets, featuring 2,788 distinct identities. To our knowledge, this is the first dataset for video ReID under Ground-to-Aerial scenarios. G2A-VReID dataset has the following characteristics: 1) Drastic view changes; 2) Large number of annotated identities; 3) Rich outdoor scenarios; 4) Huge difference in resolution. Additionally, we propose a new benchmark approach for cross-platform ReID by transforming the cross-platform visual alignment problem into visual-semantic alignment through vision-language model (i.e., CLIP) and applying a parameter-efficient Video Set-Level-Adapter module to adapt image-based foundation model to video ReID tasks, termed VSLA-CLIP. Besides, to further reduce the great discrepancy across the platforms, we also devise the platform-bridge prompts for efficient visual feature alignment. Extensive experiments demonstrate the superiority of the proposed method on all existing video ReID datasets and our proposed G2A-VReID dataset. △ Less

Submitted 2 September, 2024; v1 submitted 14 August, 2024; originally announced August 2024.

Comments: Published at ECCV 2024

In-line fiber optic optofluidic sensor based on a fully open Fabry-Perot interferometer

Authors: Dewen Duan, Qian Kang, Qianhui Yang, Zihao Zhao, Na Li, Guan-Xiang Du, Yi-Yuan Xie

Abstract: We present an all-fiber, fully open Fabry-Perot interferometer (FPI) cavity that is suitable for fluidic measurement applications. Fabrication of the FPI involves the alignment and bonding of three optical fiber sections using either ceramic glue or low-temperature melting glass. The fabrication procedure allows the protection of the cleaved optical fiber end faces, which serve as the two mirrors… ▽ More We present an all-fiber, fully open Fabry-Perot interferometer (FPI) cavity that is suitable for fluidic measurement applications. Fabrication of the FPI involves the alignment and bonding of three optical fiber sections using either ceramic glue or low-temperature melting glass. The fabrication procedure allows the protection of the cleaved optical fiber end faces, which serve as the two mirrors of the FPI, from damage, thus ensuring the high visibility of the FPI sensor. The FPI's complete openness permits the analyte of interest fluids to flow smoothly into the cavity and interact directly with the light, obviating the need for additional assistance. The fabrication experiment demonstrates that the fabrication procedure can readily achieve a visibility of over 20 dB. Refractive index testing indicates that the sensor exhibits a sensitivity of over 1116 nm/RIU within the range of 1.334-1.375. A comparison of temperature investigations indicates that the fully open cavity FPI fabricated by bonding with low-temperature melting glass exhibits relatively lower temperature immunity than that bonded with ceramic glue. Both have a relatively low temperature fluctuation within the temperature range of 40°C-100°C, with less than 3 nm and 4.5 nm in the over 60°C changes, respectively. Our proposed fully open FPI is an economical, robust, and simple-to-fabricate structure with the potential for mass production. This renders it an appealing option for practical optofluidics applications. △ Less

Submitted 14 August, 2024; originally announced August 2024.

Comments: 6 pages, 10 figures

MSC Class: 7805

Journal ref: Laser Phys. 34 115102 (2024)

Search for QCD Axions in light of String Theory

Authors: Qiaoli Yang, Runchao Huang

Abstract: The QCD axion stands as one of the most promising candidates for resolving the strong CP problem. However, the value of the axion's decay constant $f_a$ and, by extension, its mass $m_a$, remain uncertain within the framework of effective field theory, posing a challenge for experimental detection. Fortunately, fields such as cosmology and astrophysics can offer crucial clues about potential mass… ▽ More The QCD axion stands as one of the most promising candidates for resolving the strong CP problem. However, the value of the axion's decay constant $f_a$ and, by extension, its mass $m_a$, remain uncertain within the framework of effective field theory, posing a challenge for experimental detection. Fortunately, fields such as cosmology and astrophysics can offer crucial clues about potential mass ranges. Additionally, string theory and the more recent swampland principles might shed some light on the subject. The most straightforward string theory compactifications suggest that $f_a$ is around the GUT scale, leading to an anthropic abundance of dark matter. We found that this range can be detected through hydrogen atomic transitions. The recent concept of the dark dimension scenario introduces an alternative possibility. If axions are confined to the four-dimensional Standard Model brane, their decay constant $f_a$ would be on the order of $10^{10}$ GeV. In this scenario, where axions constitute only a minor portion of dark matter, we show that a laser-interferometry setup would be an effective detection method. △ Less

Submitted 13 August, 2024; originally announced August 2024.

Comments: 7 pages, 3 figures

Anisotropy of Nanohertz Gravitational Wave Background and Individual Sources from Supermassive Binary Black Holes: Probe of Cosmic Large Scale Structure

Authors: Qing Yang, Xiao Guo, Zhoujian Cao, Xiaoyun Shao, Xi Yuan

Abstract: Several pulsar timing array (PTA) groups have recently claimed the detection of nanohertz gravitational wave (GW) background, but the origin of this GW signal remains unclear. Nanohertz GWs generated by supermassive binary black holes (SMBBHs) are one of the most important GW sources in the PTA band. Utilizing data from numerical cosmology simulation, we generate mock SMBBHs within the observable… ▽ More Several pulsar timing array (PTA) groups have recently claimed the detection of nanohertz gravitational wave (GW) background, but the origin of this GW signal remains unclear. Nanohertz GWs generated by supermassive binary black holes (SMBBHs) are one of the most important GW sources in the PTA band. Utilizing data from numerical cosmology simulation, we generate mock SMBBHs within the observable universe and treat them as PTA band GW sources. We present their statistical properties, and analyze the isotropic and anisotropic characteristics of the gravitational wave background (GWB) signal they produce. Specifically, we derive the characteristic amplitude and spectrum of the GWB signal, and calculate the angular power spectrum for both GW strains/energy density and the position distribution of GW sources. We predict that the angular power spectrum of GWB energy density has $C_1/C_0\approx0.40\pm0.32$, and $C_l/C_0\simeq \frac{1}{2(2l+1)}$ (for $l>1$). Furthermore, for the upcoming Chinese Pulsar Timing Array (CPTA) and Square Kilometre Array (SKA) PTA, we predict the spatial distribution, numbers and signal-to-noise ratio (SNR) distribution of individual GW sources that may be detected with SNR>8, and study the anisotropy property in the spatial distribution of these individual GW sources. △ Less

Submitted 9 August, 2024; originally announced August 2024.

Comments: 26 pages,18 figures,1 table

Deep Learning-based Unsupervised Domain Adaptation via a Unified Model for Prostate Lesion Detection Using Multisite Bi-parametric MRI Datasets

Authors: Hao Li, Han Liu, Heinrich von Busch, Robert Grimm, Henkjan Huisman, Angela Tong, David Winkel, Tobias Penzkofer, Ivan Shabunin, Moon Hyung Choi, Qingsong Yang, Dieter Szolar, Steven Shea, Fergus Coakley, Mukesh Harisinghani, Ipek Oguz, Dorin Comaniciu, Ali Kamen, Bin Lou

Abstract: Our hypothesis is that UDA using diffusion-weighted images, generated with a unified model, offers a promising and reliable strategy for enhancing the performance of supervised learning models in multi-site prostate lesion detection, especially when various b-values are present. This retrospective study included data from 5,150 patients (14,191 samples) collected across nine different imaging cent… ▽ More Our hypothesis is that UDA using diffusion-weighted images, generated with a unified model, offers a promising and reliable strategy for enhancing the performance of supervised learning models in multi-site prostate lesion detection, especially when various b-values are present. This retrospective study included data from 5,150 patients (14,191 samples) collected across nine different imaging centers. A novel UDA method using a unified generative model was developed for multi-site PCa detection. This method translates diffusion-weighted imaging (DWI) acquisitions, including apparent diffusion coefficient (ADC) and individual DW images acquired using various b-values, to align with the style of images acquired using b-values recommended by Prostate Imaging Reporting and Data System (PI-RADS) guidelines. The generated ADC and DW images replace the original images for PCa detection. An independent set of 1,692 test cases (2,393 samples) was used for evaluation. The area under the receiver operating characteristic curve (AUC) was used as the primary metric, and statistical analysis was performed via bootstrapping. For all test cases, the AUC values for baseline SL and UDA methods were 0.73 and 0.79 (p<.001), respectively, for PI-RADS>=3, and 0.77 and 0.80 (p<.001) for PI-RADS>=4 PCa lesions. In the 361 test cases under the most unfavorable image acquisition setting, the AUC values for baseline SL and UDA were 0.49 and 0.76 (p<.001) for PI-RADS>=3, and 0.50 and 0.77 (p<.001) for PI-RADS>=4 PCa lesions. The results indicate the proposed UDA with generated images improved the performance of SL methods in multi-site PCa lesion detection across datasets with various b values, especially for images acquired with significant deviations from the PI-RADS recommended DWI protocol (e.g. with an extremely high b-value). △ Less

Submitted 8 August, 2024; originally announced August 2024.

Comments: Accept at Radiology: Artificial Intelligence. Journal reference and external DOI will be added once published

Journal ref: Radiology: Artificial Intelligence 2024;6(5):e230521

Knowledge-Aided Semantic Communication Leveraging Probabilistic Graphical Modeling

Authors: Haowen Wan, Qianqian Yang, Jiancheng Tang, Zhiguo shi

Abstract: In this paper, we propose a semantic communication approach based on probabilistic graphical model (PGM). The proposed approach involves constructing a PGM from a training dataset, which is then shared as common knowledge between the transmitter and receiver. We evaluate the importance of various semantic features and present a PGM-based compression algorithm designed to eliminate predictable port… ▽ More In this paper, we propose a semantic communication approach based on probabilistic graphical model (PGM). The proposed approach involves constructing a PGM from a training dataset, which is then shared as common knowledge between the transmitter and receiver. We evaluate the importance of various semantic features and present a PGM-based compression algorithm designed to eliminate predictable portions of semantic information. Furthermore, we introduce a technique to reconstruct the discarded semantic information at the receiver end, generating approximate results based on the PGM. Simulation results indicate a significant improvement in transmission efficiency over existing methods, while maintaining the quality of the transmitted images. △ Less

Submitted 8 August, 2024; originally announced August 2024.

From squared amplitudes to energy correlators

Authors: Song He, Xuhang Jiang, Qinglin Yang, Yao-Qi Zhang

Abstract: The leading order $N$-point energy correlators of maximally supersymmetric Yang-Mills theory in the limit where the $N$ detectors are collinear can be expressed as an integral of the $1\to N$ splitting function, which is given by the $(N{+}3)$-point squared super-amplitudes at tree level. This provides yet another example that the integrand of certain physical observable -- $N$-point energy correl… ▽ More The leading order $N$-point energy correlators of maximally supersymmetric Yang-Mills theory in the limit where the $N$ detectors are collinear can be expressed as an integral of the $1\to N$ splitting function, which is given by the $(N{+}3)$-point squared super-amplitudes at tree level. This provides yet another example that the integrand of certain physical observable -- $N$-point energy correlator -- is computed by the canonical form of a positive geometry -- the (tree-level) "squared amplituhedron". By extracting such squared amplitudes from the $f$-graph construction, we compute the integrand of energy correlators up to $N=11$ and reveal new structures to all $N$; we also show important properties of the integrand such as soft and multi-collinear limits. Finally, we take a first look at integrations by studying possible residues of the integrand: our analysis shows that while this gives prefactors in front of multiple polylogarithm functions of $N=3,4$, the first unknown case of $N=5$ already involves elliptic polylogarithmic functions with many distinct elliptic curves, and more complicated curves and higher-dimensional varieties appear for $N>5$. △ Less

Submitted 8 August, 2024; originally announced August 2024.

Comments: 10 pages, several figures and tables, and an ancillary file with squared amplitudes up to 12 points, and explicit EC integrands up to 7 points

Resonant Beam Enabled DoA Estimation in Passive Positioning System

Authors: Yixuan Guo, Qingwei Jiang, Mengyuan Xu, Wen Fang, Qingwen Liu, Gang Yan, Qunhui Yang, Hai Lu

Abstract: The rapid advancement of the next generation of communications and internet of things (IoT) technologies has made the provision of location-based services for diverse devices an increasingly pressing necessity. Localizing devices with/without intelligent computing abilities, including both active and passive devices is essential, especially in indoor scenarios. For traditional RF positioning syste… ▽ More The rapid advancement of the next generation of communications and internet of things (IoT) technologies has made the provision of location-based services for diverse devices an increasingly pressing necessity. Localizing devices with/without intelligent computing abilities, including both active and passive devices is essential, especially in indoor scenarios. For traditional RF positioning systems, aligning transmission signals and dealing with signal interference in complex environments are inevitable challenges. Therefore, this paper proposed a new passive positioning system, the RF-band resonant beam positioning system (RF-RBPS), which achieves energy concentration and beam alignment by amplifying echoes between the base station (BS) and the passive target (PT), without the need for complex channel estimation and time-consuming beamforming and provides high-precision direction of arrival (DoA) estimation for battery-free targets using the resonant mechanism. The direction information of the PT is estimated using the multiple signal classification (MUSIC) algorithm at the end of BS. The feasibility of the proposed system is validated through theoretical analysis and simulations. Results indicate that the proposed RF-RBPS surpasses RF-band active positioning system (RF-APS) in precision, achieving millimeter-level precision at 2m within an elevation angle of 35$^\circ$, and an error of less than 3cm at 2.5m within an elevation angle of 35$^\circ$. △ Less

Submitted 7 August, 2024; originally announced August 2024.

Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructions

Authors: Qingbin Zeng, Qinglong Yang, Shunan Dong, Heming Du, Liang Zheng, Fengli Xu, Yong Li

Abstract: This paper considers a scenario in city navigation: an AI agent is provided with language descriptions of the goal location with respect to some well-known landmarks; By only observing the scene around, including recognizing landmarks and road network connections, the agent has to make decisions to navigate to the goal location without instructions. This problem is very challenging, because it req… ▽ More This paper considers a scenario in city navigation: an AI agent is provided with language descriptions of the goal location with respect to some well-known landmarks; By only observing the scene around, including recognizing landmarks and road network connections, the agent has to make decisions to navigate to the goal location without instructions. This problem is very challenging, because it requires agent to establish self-position and acquire spatial representation of complex urban environment, where landmarks are often invisible. In the absence of navigation instructions, such abilities are vital for the agent to make high-quality decisions in long-range city navigation. With the emergent reasoning ability of large language models (LLMs), a tempting baseline is to prompt LLMs to "react" on each observation and make decisions accordingly. However, this baseline has very poor performance that the agent often repeatedly visits same locations and make short-sighted, inconsistent decisions. To address these issues, this paper introduces a novel agentic workflow featured by its abilities to perceive, reflect and plan. Specifically, we find LLaVA-7B can be fine-tuned to perceive the direction and distance of landmarks with sufficient accuracy for city navigation. Moreover, reflection is achieved through a memory mechanism, where past experiences are stored and can be retrieved with current perception for effective decision argumentation. Planning uses reflection results to produce long-term plans, which can avoid short-sighted decisions in long-range navigation. We show the designed workflow significantly improves navigation ability of the LLM agent compared with the state-of-the-art baselines. △ Less

Submitted 17 October, 2024; v1 submitted 7 August, 2024; originally announced August 2024.

AdapMTL: Adaptive Pruning Framework for Multitask Learning Model

Authors: Mingcan Xiang, Steven Jiaxun Tang, Qizheng Yang, Hui Guan, Tongping Liu

Abstract: In the domain of multimedia and multimodal processing, the efficient handling of diverse data streams such as images, video, and sensor data is paramount. Model compression and multitask learning (MTL) are crucial in this field, offering the potential to address the resource-intensive demands of processing and interpreting multiple forms of media simultaneously. However, effectively compressing a… ▽ More In the domain of multimedia and multimodal processing, the efficient handling of diverse data streams such as images, video, and sensor data is paramount. Model compression and multitask learning (MTL) are crucial in this field, offering the potential to address the resource-intensive demands of processing and interpreting multiple forms of media simultaneously. However, effectively compressing a multitask model presents significant challenges due to the complexities of balancing sparsity allocation and accuracy performance across multiple tasks. To tackle these challenges, we propose AdapMTL, an adaptive pruning framework for MTL models. AdapMTL leverages multiple learnable soft thresholds independently assigned to the shared backbone and the task-specific heads to capture the nuances in different components' sensitivity to pruning. During training, it co-optimizes the soft thresholds and MTL model weights to automatically determine the suitable sparsity level at each component to achieve both high task accuracy and high overall sparsity. It further incorporates an adaptive weighting mechanism that dynamically adjusts the importance of task-specific losses based on each task's robustness to pruning. We demonstrate the effectiveness of AdapMTL through comprehensive experiments on popular multitask datasets, namely NYU-v2 and Tiny-Taskonomy, with different architectures, showcasing superior performance compared to state-of-the-art pruning methods. △ Less

Submitted 7 August, 2024; originally announced August 2024.

Comments: 13 pages, 9 figures, Published at ACM Multimedia (ACM MM) 2024

Leveraging Inter-Chunk Interactions for Enhanced Retrieval in Large Language Model-Based Question Answering

Authors: Tiezheng Guo, Chen Wang, Yanyi Liu, Jiawei Tang, Pan Li, Sai Xu, Qingwen Yang, Xianlin Gao, Zhi Li, Yingyou Wen

Abstract: Retrieving external knowledge and prompting large language models with relevant information is an effective paradigm to enhance the performance of question-answering tasks. Previous research typically handles paragraphs from external documents in isolation, resulting in a lack of context and ambiguous references, particularly in multi-document and complex tasks. To overcome these challenges, we pr… ▽ More Retrieving external knowledge and prompting large language models with relevant information is an effective paradigm to enhance the performance of question-answering tasks. Previous research typically handles paragraphs from external documents in isolation, resulting in a lack of context and ambiguous references, particularly in multi-document and complex tasks. To overcome these challenges, we propose a new retrieval framework IIER, that leverages Inter-chunk Interactions to Enhance Retrieval. This framework captures the internal connections between document chunks by considering three types of interactions: structural, keyword, and semantic. We then construct a unified Chunk-Interaction Graph to represent all external documents comprehensively. Additionally, we design a graph-based evidence chain retriever that utilizes previous paths and chunk interactions to guide the retrieval process. It identifies multiple seed nodes based on the target question and iteratively searches for relevant chunks to gather supporting evidence. This retrieval process refines the context and reasoning chain, aiding the large language model in reasoning and answer generation. Extensive experiments demonstrate that IIER outperforms strong baselines across four datasets, highlighting its effectiveness in improving retrieval and reasoning capabilities. △ Less

Submitted 5 August, 2024; originally announced August 2024.

Implementing NAT Hole Punching with QUIC

Authors: Jinyu Liang, Wei Xu, Taotao Wang, Qing Yang, Shengli Zhang

Abstract: The widespread adoption of Network Address Translation (NAT) technology has led to a significant number of network end nodes being located in private networks behind NAT devices, impeding direct communication between these nodes. To solve this problem, a technique known as "hole punching" has been devised for NAT traversal to facilitate peer-to-peer communication among end nodes located in distinc… ▽ More The widespread adoption of Network Address Translation (NAT) technology has led to a significant number of network end nodes being located in private networks behind NAT devices, impeding direct communication between these nodes. To solve this problem, a technique known as "hole punching" has been devised for NAT traversal to facilitate peer-to-peer communication among end nodes located in distinct private networks. However, as the increasing demands for speed and security in networks, TCP-based hole punching schemes gradually show performance drawbacks. Therefore, we present a QUIC-based hole punching scheme for NAT traversal. Through a comparative analysis of the hole punching time between QUIC-based and TCP based protocols, we find that the QUIC-based scheme effectively reduces the hole punching time, exhibiting a pronounced advantage in weak network environments. Furthermore, in scenarios where the hole punched connection is disrupted due to factors such as network transitions or NAT timeouts, this paper evaluates two schemes for restoring the connection: QUIC connection migration and re-punching. Our results show that QUIC connection migration for connection restoration saves 2 RTTs compared to QUIC re-punching, and 3 RTTs compared to TCP re-punching, effectively reducing the computational resources consumption for re-punching. △ Less

Submitted 3 August, 2024; originally announced August 2024.

Comments: The paper has been accepted for oral presentation at the VTC2024-Fall Conference

AVESFormer: Efficient Transformer Design for Real-Time Audio-Visual Segmentation

Authors: Zili Wang, Qi Yang, Linsu Shi, Jiazhong Yu, Qinghua Liang, Fei Li, Shiming Xiang

Abstract: Recently, transformer-based models have demonstrated remarkable performance on audio-visual segmentation (AVS) tasks. However, their expensive computational cost makes real-time inference impractical. By characterizing attention maps of the network, we identify two key obstacles in AVS models: 1) attention dissipation, corresponding to the over-concentrated attention weights by Softmax within rest… ▽ More Recently, transformer-based models have demonstrated remarkable performance on audio-visual segmentation (AVS) tasks. However, their expensive computational cost makes real-time inference impractical. By characterizing attention maps of the network, we identify two key obstacles in AVS models: 1) attention dissipation, corresponding to the over-concentrated attention weights by Softmax within restricted frames, and 2) inefficient, burdensome transformer decoder, caused by narrow focus patterns in early stages. In this paper, we introduce AVESFormer, the first real-time Audio-Visual Efficient Segmentation transformer that achieves fast, efficient and light-weight simultaneously. Our model leverages an efficient prompt query generator to correct the behaviour of cross-attention. Additionally, we propose ELF decoder to bring greater efficiency by facilitating convolutions suitable for local features to reduce computational burdens. Extensive experiments demonstrate that our AVESFormer significantly enhances model performance, achieving 79.9% on S4, 57.9% on MS3 and 31.2% on AVSS, outperforming previous state-of-the-art and achieving an excellent trade-off between performance and speed. Code can be found at https://github.com/MarkXCloud/AVESFormer.git. △ Less

Submitted 3 August, 2024; originally announced August 2024.

Statistical AoI Guarantee Optimization for Supporting xURLLC in ISAC-enabled V2I Networks

Authors: Yanxi Zhang, Mingwu Yao, Qinghai Yang, Dongqi Yan, Xu Zhang, Xu Bao, Muyu Mei

Abstract: This paper addresses the critical challenge of supporting next-generation ultra-reliable and low-latency communication (xURLLC) within integrated sensing and communication (ISAC)-enabled vehicle-to-infrastructure (V2I) networks. We incorporate channel evaluation and retransmission mechanisms for real-time reliability enhancement. Using stochastic network calculus (SNC), we establish a theoretical… ▽ More This paper addresses the critical challenge of supporting next-generation ultra-reliable and low-latency communication (xURLLC) within integrated sensing and communication (ISAC)-enabled vehicle-to-infrastructure (V2I) networks. We incorporate channel evaluation and retransmission mechanisms for real-time reliability enhancement. Using stochastic network calculus (SNC), we establish a theoretical framework to derive upper bounds for the peak age of information violation probability (PAVP) via characterized sensing and communication moment generation functions (MGFs). By optimizing these bounds, we develop power allocation schemes that significantly reduce the statistical PAVP of sensory packets in such networks. Simulations validate our theoretical derivations and demonstrate the effectiveness of our proposed schemes. △ Less

Submitted 1 August, 2024; originally announced August 2024.

Illumination Design for Joint Imaging and Wireless Power Transfer Systems

Authors: Qianyu Yang, Haiyang Zhang, Chunguo Li, Ruiqi Liu, Baoyun Wang

Abstract: This paper presents a novel concept termed Integrated Imaging and Wireless Power Transfer (IWPT), wherein the integration of imaging and wireless power transfer functionalities is achieved on a unified hardware platform. IWPT leverages a transmitting array to efficiently illuminate a specific Region of Interest (ROI), enabling the extraction of ROI's scattering coefficients while concurrently prov… ▽ More This paper presents a novel concept termed Integrated Imaging and Wireless Power Transfer (IWPT), wherein the integration of imaging and wireless power transfer functionalities is achieved on a unified hardware platform. IWPT leverages a transmitting array to efficiently illuminate a specific Region of Interest (ROI), enabling the extraction of ROI's scattering coefficients while concurrently providing wireless power to nearby users. The integration of IWPT offers compelling advantages, including notable reductions in power consumption and spectrum utilization, pivotal for the optimization of future 6G wireless networks. As an initial investigation, we explore two antenna architectures: a fully digital array and a digital/analog hybrid array. Our goal is to characterize the fundamental trade-off between imaging and wireless power transfer by optimizing the illumination signal. With imaging operating in the near-field, we formulate the illumination signal design as an optimization problem that minimizes the condition number of the equivalent channel. To address this optimization problem, we propose an semi-definite relaxation-based approach for the fully digital array and an alternating optimization algorithm for the hybrid array. Finally, numerical results verify the effectiveness of our proposed solutions and demonstrate the trade-off between imaging and wireless power transfer. △ Less

Submitted 1 August, 2024; originally announced August 2024.

Comments: 10 pages, 5 figures

In-plane dielectric constant and conductivity of confined water

Authors: R. Wang, M. Souilamas, A. Esfandiar, R. Fabregas, S. Benaglia, H. Nevison-Andrews, Q. Yang, J. Normansell, P. Ares, G. Ferrari, A. Principi, A. K. Geim, L. Fumagalli

Abstract: Water is essential for almost every aspect of life on our planet and, unsurprisingly, its properties have been studied in great detail. However, disproportionately little remains known about the electrical properties of interfacial and strongly confined water where its structure deviates from that of bulk water, becoming distinctly layered. The structural change is expected to affect water's condu… ▽ More Water is essential for almost every aspect of life on our planet and, unsurprisingly, its properties have been studied in great detail. However, disproportionately little remains known about the electrical properties of interfacial and strongly confined water where its structure deviates from that of bulk water, becoming distinctly layered. The structural change is expected to affect water's conductivity and particularly its polarizability, which in turn modifies intermolecular forces that play a crucial role in many physical and chemical processes. Here we use scanning dielectric microscopy to probe the in-plane electrical properties of water confined between atomically flat surfaces separated by distances down to 1 nm. For confinement exceeding a few nm, water exhibits an in-plane dielectric constant close to that of bulk water and its proton conductivity is notably enhanced, gradually increasing with decreasing water thickness. This trend abruptly changes when the confined water becomes only a few molecules thick. Its in-plane dielectric constant reaches giant, ferroelectric-like values of about 1,000 whereas the conductivity peaks at a few S/m, close to values characteristic of superionic liquids. We attribute the enhancement to strongly disordered hydrogen bonding induced by the few-layer confinement, which facilitates both easier in-plane polarization of molecular dipoles and faster proton exchange. This insight into the electrical properties of nanoconfined water is important for understanding many phenomena that occur at aqueous interfaces and in nanoscale pores. △ Less

Submitted 31 July, 2024; originally announced July 2024.

Beam Focusing for Near-Field Multi-User Localization

Authors: Qianyu Yang, Anna Guerra, Francesco Guidi, Nir Shlezinger, Haiyang Zhang, Davide Dardari, Baoyun Wang, Yonina C. Eldar

Abstract: Extremely large-scale antenna arrays are poised to play a pivotal role in sixth-generation (6G) networks. Utilizing such arrays often results in a near-field spherical wave transmission environment, enabling the generation of focused beams, which introduces new degrees of freedom for wireless localization. In this paper, we consider a beam-focusing design for localizing multiple sources in the rad… ▽ More Extremely large-scale antenna arrays are poised to play a pivotal role in sixth-generation (6G) networks. Utilizing such arrays often results in a near-field spherical wave transmission environment, enabling the generation of focused beams, which introduces new degrees of freedom for wireless localization. In this paper, we consider a beam-focusing design for localizing multiple sources in the radiating near-field. Our formulation accommodates various expected types of implementations of large antenna arrays, including hybrid analog/digital architectures and dynamic metasurface antennas (DMAs). We consider a direct localization estimation method exploiting curvature-of-arrival of impinging spherical wavefront to obtain user positions. In this regard, we adopt a two-stage approach configuring the array to optimize near-field positioning. In the first step, we focus only on adjusting the array coefficients to minimize the estimation error. We obtain a closed-form approximate solution based on projection and the better one based on the Riemann gradient algorithm. We then extend this approach to simultaneously localize and focus the beams via a sub-optimal iterative approach that does not rely on such knowledge. The simulation results show that near-field localization accuracy based on a hybrid array or DMA can achieve performance close to that of fully digital arrays at a lower cost, and DMAs can attain better performance than hybrid solutions with the same aperture. △ Less

Submitted 24 July, 2024; originally announced July 2024.

Comments: 13 pages, 11 figures

A general thermodynamically consistent phase-field-micromechanics model of sintering with coupled diffusion and grain motion

Authors: Qingcheng Yang, Arkadz Kirshtein

Abstract: Sintering is a pivotal technology for processing ceramic and metallic powders into solid objects. A profound understanding of microstructure evolution during sintering is essential for manufacturing products with tailored properties. While various phase-field models have been proposed to simulate microstructure evolution in solid-state sintering, correctly incorporating the crucial densification m… ▽ More Sintering is a pivotal technology for processing ceramic and metallic powders into solid objects. A profound understanding of microstructure evolution during sintering is essential for manufacturing products with tailored properties. While various phase-field models have been proposed to simulate microstructure evolution in solid-state sintering, correctly incorporating the crucial densification mechanism, particularly grain motion, remains a challenge. The fundamental obstacle lies in the ad hoc treatment of the micromechanics of grain motion, where the thermodynamical driving force cannot be derived from the system's free energy. This work presents a novel phase-field-micromechanics model for sintering (PFMMS) that addresses this challenge. We define a unified energy law, under which the governing equations for microstructure evolution in sintering are derived using variational principles. Our approach ensures thermodynamic consistency, with the driving force for grain motion derived from the system's free energy. Consequently, the proposed PFMMS guarantees the evolution of microstructure in a direction that reduces the system's energy and eliminates non-densifying phenomena. We rigorously validate PFMMS against recent benchmarks of theoretical and numerical analysis. It is found that PFMMS captures intrinsic stress distribution along and beyond grain boundaries, exhibits system-size-independent shrinkage strain, and maintains thermodynamic equilibrium states. These features are fundamental requirements for a physically consistent sintering model. △ Less

Submitted 25 July, 2024; originally announced July 2024.

Differential equations and recursive solutions for cosmological amplitudes

Authors: Song He, Xuhang Jiang, Jiahao Liu, Qinglin Yang, Yao-Qi Zhang

Abstract: Recently considerable efforts have been devoted to computing cosmological correlators and the corresponding wavefunction coefficients, as well as understanding their analytical structures. In this note, we revisit the computation of these ``cosmological amplitudes" associated with any tree or loop graph for conformal scalars with time-dependent interactions in the power-law FRW universe, directly… ▽ More Recently considerable efforts have been devoted to computing cosmological correlators and the corresponding wavefunction coefficients, as well as understanding their analytical structures. In this note, we revisit the computation of these ``cosmological amplitudes" associated with any tree or loop graph for conformal scalars with time-dependent interactions in the power-law FRW universe, directly in terms of iterated time integrals. We start by decomposing any such cosmological amplitude (for loop graph, the ``integrand" prior to loop integrations) as a linear combination of {\it basic time integrals}, one for each {\it directed graph}. We derive remarkably simple first-order differential equations involving such time integrals with edges ``contracted" one at a time, which can be solved recursively and the solution takes the form of Euler-Mellin integrals/generalized hypergeometric functions. By combining such equations, we then derive a complete system of differential equations for all time integrals needed for a given graph. Our method works for any graph: for a tree graph with $n$ nodes, this system can be transformed into the {\it canonical differential equations} of size $4^{n{-}1}$ quivalent to the graphic rules derived recently%so-called ``kinematic flow", and we also derive the system of differential equations for loop integrands {\it e.g.} of all-loop two-site graphs and one-loop $n$-gon graphs. Finally, we show how the differential equations truncate for the de Sitter (dS) case (in a way similar to differential equations for Feynman integrals truncate for integer dimensions), which immediately yields the complete symbol for the dS amplitude with interesting structures {\it e.g.} for $n$-site chains and $n$-gon cases. △ Less

Submitted 24 July, 2024; originally announced July 2024.

Comments: 43 pages; many figures

Motion Capture from Inertial and Vision Sensors

Authors: Xiaodong Chen, Wu Liu, Qian Bao, Xinchen Liu, Quanwei Yang, Ruoli Dai, Tao Mei

Abstract: Human motion capture is the foundation for many computer vision and graphics tasks. While industrial motion capture systems with complex camera arrays or expensive wearable sensors have been widely adopted in movie and game production, consumer-affordable and easy-to-use solutions for personal applications are still far from mature. To utilize a mixture of a monocular camera and very few inertial… ▽ More Human motion capture is the foundation for many computer vision and graphics tasks. While industrial motion capture systems with complex camera arrays or expensive wearable sensors have been widely adopted in movie and game production, consumer-affordable and easy-to-use solutions for personal applications are still far from mature. To utilize a mixture of a monocular camera and very few inertial measurement units (IMUs) for accurate multi-modal human motion capture in daily life, we contribute MINIONS in this paper, a large-scale Motion capture dataset collected from INertial and visION Sensors. MINIONS has several featured properties: 1) large scale of over five million frames and 400 minutes duration; 2) multi-modality data of IMUs signals and RGB videos labeled with joint positions, joint rotations, SMPL parameters, etc.; 3) a diverse set of 146 fine-grained single and interactive actions with textual descriptions. With the proposed MINIONS, we conduct experiments on multi-modal motion capture and explore the possibilities of consumer-affordable motion capture using a monocular camera and very few IMUs. The experiment results emphasize the unique advantages of inertial and vision sensors, showcasing the promise of consumer-affordable multi-modal motion capture and providing a valuable resource for further research and development. △ Less

Submitted 23 July, 2024; originally announced July 2024.

Comments: 17 pages,9 figures

DiffX: Guide Your Layout to Cross-Modal Generative Modeling

Authors: Zeyu Wang, Jingyu Lin, Yifei Qian, Yi Huang, Shicen Tian, Bosong Chai, Juncan Deng, Qu Yang, Lan Du, Cunjian Chen, Kejie Huang

Abstract: Diffusion models have made significant strides in language-driven and layout-driven image generation. However, most diffusion models are limited to visible RGB image generation. In fact, human perception of the world is enriched by diverse viewpoints, such as chromatic contrast, thermal illumination, and depth information. In this paper, we introduce a novel diffusion model for general layout-guid… ▽ More Diffusion models have made significant strides in language-driven and layout-driven image generation. However, most diffusion models are limited to visible RGB image generation. In fact, human perception of the world is enriched by diverse viewpoints, such as chromatic contrast, thermal illumination, and depth information. In this paper, we introduce a novel diffusion model for general layout-guided cross-modal generation, called DiffX. Notably, our DiffX presents a compact and effective cross-modal generative modeling pipeline, which conducts diffusion and denoising processes in the modality-shared latent space. Moreover, we introduce the Joint-Modality Embedder (JME) to enhance the interaction between layout and text conditions by incorporating a gated attention mechanism. To facilitate the user-instructed training, we construct the cross-modal image datasets with detailed text captions by the Large-Multimodal Model (LMM) and our human-in-the-loop refinement. Through extensive experiments, our DiffX demonstrates robustness in cross-modal ''RGB+X'' image generation on FLIR, MFNet, and COME15K datasets, guided by various layout conditions. Meanwhile, it shows the strong potential for the adaptive generation of ``RGB+X+Y(+Z)'' images or more diverse modalities on FLIR, MFNet, COME15K, and MCXFace datasets. To our knowledge, DiffX is the first model for layout-guided cross-modal image generation. Our code and constructed cross-modal image datasets are available at https://github.com/zeyuwang-zju/DiffX. △ Less

Submitted 20 October, 2024; v1 submitted 22 July, 2024; originally announced July 2024.

Enhancement of 3D Gaussian Splatting using Raw Mesh for Photorealistic Recreation of Architectures

Authors: Ruizhe Wang, Chunliang Hua, Tomakayev Shingys, Mengyuan Niu, Qingxin Yang, Lizhong Gao, Yi Zheng, Junyan Yang, Qiao Wang

Abstract: The photorealistic reconstruction and rendering of architectural scenes have extensive applications in industries such as film, games, and transportation. It also plays an important role in urban planning, architectural design, and the city's promotion, especially in protecting historical and cultural relics. The 3D Gaussian Splatting, due to better performance over NeRF, has become a mainstream t… ▽ More The photorealistic reconstruction and rendering of architectural scenes have extensive applications in industries such as film, games, and transportation. It also plays an important role in urban planning, architectural design, and the city's promotion, especially in protecting historical and cultural relics. The 3D Gaussian Splatting, due to better performance over NeRF, has become a mainstream technology in 3D reconstruction. Its only input is a set of images but it relies heavily on geometric parameters computed by the SfM process. At the same time, there is an existing abundance of raw 3D models, that could inform the structural perception of certain buildings but cannot be applied. In this paper, we propose a straightforward method to harness these raw 3D models to guide 3D Gaussians in capturing the basic shape of the building and improve the visual quality of textures and details when photos are captured non-systematically. This exploration opens up new possibilities for improving the effectiveness of 3D reconstruction techniques in the field of architectural design. △ Less

Submitted 25 September, 2024; v1 submitted 22 July, 2024; originally announced July 2024.

A Benchmark for Gaussian Splatting Compression and Quality Assessment Study

Authors: Qi Yang, Kaifa Yang, Yuke Xing, Yiling Xu, Zhu Li

Abstract: To fill the gap of traditional GS compression method, in this paper, we first propose a simple and effective GS data compression anchor called Graph-based GS Compression (GGSC). GGSC is inspired by graph signal processing theory and uses two branches to compress the primitive center and attributes. We split the whole GS sample via KDTree and clip the high-frequency components after the graph Fouri… ▽ More To fill the gap of traditional GS compression method, in this paper, we first propose a simple and effective GS data compression anchor called Graph-based GS Compression (GGSC). GGSC is inspired by graph signal processing theory and uses two branches to compress the primitive center and attributes. We split the whole GS sample via KDTree and clip the high-frequency components after the graph Fourier transform. Followed by quantization, G-PCC and adaptive arithmetic coding are used to compress the primitive center and attribute residual matrix to generate the bitrate file. GGSS is the first work to explore traditional GS compression, with advantages that can reveal the GS distortion characteristics corresponding to typical compression operation, such as high-frequency clipping and quantization. Second, based on GGSC, we create a GS Quality Assessment dataset (GSQA) with 120 samples. A subjective experiment is conducted in a laboratory environment to collect subjective scores after rendering GS into Processed Video Sequences (PVS). We analyze the characteristics of different GS distortions based on Mean Opinion Scores (MOS), demonstrating the sensitivity of different attributes distortion to visual quality. The GGSC code and the dataset, including GS samples, MOS, and PVS, are made publicly available at https://github.com/Qi-Yangsjtu/GGSC. △ Less

Submitted 19 July, 2024; originally announced July 2024.

MSceneSpeech: A Multi-Scene Speech Dataset For Expressive Speech Synthesis

Authors: Qian Yang, Jialong Zuo, Zhe Su, Ziyue Jiang, Mingze Li, Zhou Zhao, Feiyang Chen, Zhefeng Wang, Baoxing Huai

Abstract: We introduce an open source high-quality Mandarin TTS dataset MSceneSpeech (Multiple Scene Speech Dataset), which is intended to provide resources for expressive speech synthesis. MSceneSpeech comprises numerous audio recordings and texts performed and recorded according to daily life scenarios. Each scenario includes multiple speakers and a diverse range of prosodic styles, making it suitable for… ▽ More We introduce an open source high-quality Mandarin TTS dataset MSceneSpeech (Multiple Scene Speech Dataset), which is intended to provide resources for expressive speech synthesis. MSceneSpeech comprises numerous audio recordings and texts performed and recorded according to daily life scenarios. Each scenario includes multiple speakers and a diverse range of prosodic styles, making it suitable for speech synthesis that entails multi-speaker style and prosody modeling. We have established a robust baseline, through the prompting mechanism, that can effectively synthesize speech characterized by both user-specific timbre and scene-specific prosody with arbitrary text input. The open source MSceneSpeech Dataset and audio samples of our baseline are available at https://speechai-demo.github.io/MSceneSpeech/. △ Less

Submitted 18 July, 2024; originally announced July 2024.

Comments: Accepted by INTERSPEECH 2024

SOMONITOR: Explainable Marketing Data Processing and Analysis with Large Language Models

Authors: Qi Yang, Sergey Nikolenko, Marlo Ongpin, Ilia Gossoudarev, Yu-Yi Chu-Farseeva, Aleksandr Farseev

Abstract: Online marketing faces formidable challenges in managing and interpreting immense volumes of data necessary for competitor analysis, content research, and strategic branding. It is impossible to review hundreds to thousands of transient online content items by hand, and partial analysis often leads to suboptimal outcomes and poorly performing campaigns. We introduce an explainable AI framework SoM… ▽ More Online marketing faces formidable challenges in managing and interpreting immense volumes of data necessary for competitor analysis, content research, and strategic branding. It is impossible to review hundreds to thousands of transient online content items by hand, and partial analysis often leads to suboptimal outcomes and poorly performing campaigns. We introduce an explainable AI framework SoMonitor that aims to synergize human intuition with AI-based efficiency, helping marketers across all stages of the marketing funnel, from strategic planning to content creation and campaign execution. SoMonitor incorporates a CTR prediction and ranking model for advertising content and uses large language models (LLMs) to process high-performing competitor content, identifying core content pillars such as target audiences, customer needs, and product features. These pillars are then organized into broader categories, including communication themes and targeted customer personas. By integrating these insights with data from the brand's own advertising campaigns, SoMonitor constructs a narrative for addressing new customer personas and simultaneously generates detailed content briefs in the form of user stories that can be directly applied by marketing teams to streamline content production and campaign execution. The adoption of SoMonitor in daily operations allows digital marketers to quickly parse through extensive datasets, offering actionable insights that significantly enhance campaign effectiveness and overall job satisfaction △ Less

Submitted 17 July, 2024; originally announced July 2024.

Evaluating the transferability potential of deep learning models for climate downscaling

Authors: Ayush Prasad, Paula Harder, Qidong Yang, Prasanna Sattegeri, Daniela Szwarcman, Campbell Watson, David Rolnick

Abstract: Climate downscaling, the process of generating high-resolution climate data from low-resolution simulations, is essential for understanding and adapting to climate change at regional and local scales. Deep learning approaches have proven useful in tackling this problem. However, existing studies usually focus on training models for one specific task, location and variable, which are therefore limi… ▽ More Climate downscaling, the process of generating high-resolution climate data from low-resolution simulations, is essential for understanding and adapting to climate change at regional and local scales. Deep learning approaches have proven useful in tackling this problem. However, existing studies usually focus on training models for one specific task, location and variable, which are therefore limited in their generalizability and transferability. In this paper, we evaluate the efficacy of training deep learning downscaling models on multiple diverse climate datasets to learn more robust and transferable representations. We evaluate the effectiveness of architectures zero-shot transferability using CNNs, Fourier Neural Operators (FNOs), and vision Transformers (ViTs). We assess the spatial, variable, and product transferability of downscaling models experimentally, to understand the generalizability of these different architecture types. △ Less

Submitted 17 July, 2024; originally announced July 2024.

Surprising Performances of Students with Autism in Classroom with NAO Robot

Authors: Qin Yang, Huan Lu, Dandan Liang, Shengrong Gong, Huanghao Feng

Abstract: Autism is a developmental disorder that manifests in early childhood and persists throughout life, profoundly affecting social behavior and hindering the acquisition of learning and social skills in those diagnosed. As technological advancements progress, an increasing array of technologies is being utilized to support the education of students with Autism Spectrum Disorder (ASD), aiming to improv… ▽ More Autism is a developmental disorder that manifests in early childhood and persists throughout life, profoundly affecting social behavior and hindering the acquisition of learning and social skills in those diagnosed. As technological advancements progress, an increasing array of technologies is being utilized to support the education of students with Autism Spectrum Disorder (ASD), aiming to improve their educational outcomes and social capabilities. Numerous studies on autism intervention have highlighted the effectiveness of social robots in behavioral treatments. However, research on the integration of social robots into classroom settings for children with autism remains sparse. This paper describes the design and implementation of a group experiment in a collective classroom setting mediated by the NAO robot. The experiment involved special education teachers and the NAO robot collaboratively conducting classroom activities, aiming to foster a dynamic learning environment through interactions among teachers, the robot, and students. Conducted in a special education school, this experiment served as a foundational study in anticipation of extended robot-assisted classroom sessions. Data from the experiment suggest that ASD students in classrooms equipped with the NAO robot exhibited notably better performance compared to those in regular classrooms. The humanoid features and body language of the NAO robot captivated the students' attention, particularly during talent shows and command tasks, where students demonstrated heightened engagement and a decrease in stereotypical repetitive behaviors and irrelevant minor movements commonly observed in regular settings. Our preliminary findings indicate that the NAO robot significantly enhances focus and classroom engagement among students with ASD, potentially improving educational performance and fostering better social behaviors. △ Less

Submitted 26 June, 2024; originally announced July 2024.

Fine-Tuning Medical Language Models for Enhanced Long-Contextual Understanding and Domain Expertise

Authors: Qimin Yang, Rongsheng Wang, Jiexin Chen, Runqi Su, Tao Tan

Abstract: Large Language Models (LLMs) have been widely applied in various professional fields. By fine-tuning the models using domain specific question and answer datasets, the professional domain knowledge and Q\&A abilities of these models have significantly improved, for example, medical professional LLMs that use fine-tuning of doctor-patient Q\&A data exhibit extraordinary disease diagnostic abilities… ▽ More Large Language Models (LLMs) have been widely applied in various professional fields. By fine-tuning the models using domain specific question and answer datasets, the professional domain knowledge and Q\&A abilities of these models have significantly improved, for example, medical professional LLMs that use fine-tuning of doctor-patient Q\&A data exhibit extraordinary disease diagnostic abilities. However, we observed that despite improvements in specific domain knowledge, the performance of medical LLM in long-context understanding has significantly declined, especially compared to general language models with similar parameters. The purpose of this study is to investigate the phenomenon of reduced performance in understanding long-context in medical LLM. We designed a series of experiments to conduct open-book professional knowledge exams on all models to evaluate their ability to read long-context. By adjusting the proportion and quantity of general data and medical data in the process of fine-tuning, we can determine the best data composition to optimize the professional model and achieve a balance between long-context performance and specific domain knowledge. △ Less

Submitted 16 July, 2024; originally announced July 2024.

Comments: 5 pages, 1 figure. Accepted by the Workshop on Long-Context Foundation Models (LCFM) at ICML 2024

Qwen2-Audio Technical Report

Authors: Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, Jingren Zhou

Abstract: We introduce the latest progress of Qwen-Audio, a large-scale audio-language model called Qwen2-Audio, which is capable of accepting various audio signal inputs and performing audio analysis or direct textual responses with regard to speech instructions. In contrast to complex hierarchical tags, we have simplified the pre-training process by utilizing natural language prompts for different data an… ▽ More We introduce the latest progress of Qwen-Audio, a large-scale audio-language model called Qwen2-Audio, which is capable of accepting various audio signal inputs and performing audio analysis or direct textual responses with regard to speech instructions. In contrast to complex hierarchical tags, we have simplified the pre-training process by utilizing natural language prompts for different data and tasks, and have further expanded the data volume. We have boosted the instruction-following capability of Qwen2-Audio and implemented two distinct audio interaction modes for voice chat and audio analysis. In the voice chat mode, users can freely engage in voice interactions with Qwen2-Audio without text input. In the audio analysis mode, users could provide audio and text instructions for analysis during the interaction. Note that we do not use any system prompts to switch between voice chat and audio analysis modes. Qwen2-Audio is capable of intelligently comprehending the content within audio and following voice commands to respond appropriately. For instance, in an audio segment that simultaneously contains sounds, multi-speaker conversations, and a voice command, Qwen2-Audio can directly understand the command and provide an interpretation and response to the audio. Additionally, DPO has optimized the model's performance in terms of factuality and adherence to desired behavior. According to the evaluation results from AIR-Bench, Qwen2-Audio outperformed previous SOTAs, such as Gemini-1.5-pro, in tests focused on audio-centric instruction-following capabilities. Qwen2-Audio is open-sourced with the aim of fostering the advancement of the multi-modal language community. △ Less

Submitted 15 July, 2024; originally announced July 2024.

Comments: https://github.com/QwenLM/Qwen2-Audio. Checkpoints, codes and scripts will be opensoursed soon

SOFA: A Compute-Memory Optimized Sparsity Accelerator via Cross-Stage Coordinated Tiling

Authors: Huizheng Wang, Jiahao Fang, Xinru Tang, Zhiheng Yue, Jinxi Li, Yubin Qin, Sihan Guan, Qize Yang, Yang Wang, Chao Li, Yang Hu, Shouyi Yin

Abstract: Benefiting from the self-attention mechanism, Transformer models have attained impressive contextual comprehension capabilities for lengthy texts. The requirements of high-throughput inference arise as the large language models (LLMs) become increasingly prevalent, which calls for large-scale token parallel processing (LTPP). However, existing dynamic sparse accelerators struggle to effectively ha… ▽ More Benefiting from the self-attention mechanism, Transformer models have attained impressive contextual comprehension capabilities for lengthy texts. The requirements of high-throughput inference arise as the large language models (LLMs) become increasingly prevalent, which calls for large-scale token parallel processing (LTPP). However, existing dynamic sparse accelerators struggle to effectively handle LTPP, as they solely focus on separate stage optimization, and with most efforts confined to computational enhancements. By re-examining the end-to-end flow of dynamic sparse acceleration, we pinpoint an ever-overlooked opportunity that the LTPP can exploit the intrinsic coordination among stages to avoid excessive memory access and redundant computation. Motivated by our observation, we present SOFA, a cross-stage compute-memory efficient algorithm-hardware co-design, which is tailored to tackle the challenges posed by LTPP of Transformer inference effectively. We first propose a novel leading zero computing paradigm, which predicts attention sparsity by using log-based add-only operations to avoid the significant overhead of prediction. Then, a distributed sorting and a sorted updating FlashAttention mechanism are proposed with a cross-stage coordinated tiling principle, which enables fine-grained and lightweight coordination among stages, helping optimize memory access and latency. Further, we propose a SOFA accelerator to support these optimizations efficiently. Extensive experiments on 20 benchmarks show that SOFA achieves $9.5\times$ speed up and $71.5\times$ higher energy efficiency than Nvidia A100 GPU. Compared to 8 SOTA accelerators, SOFA achieves an average $15.8\times$ energy efficiency, $10.3\times$ area efficiency and $9.3\times$ speed up, respectively. △ Less

Submitted 14 July, 2024; originally announced July 2024.

Noise Calibration: Plug-and-play Content-Preserving Video Enhancement using Pre-trained Video Diffusion Models

Authors: Qinyu Yang, Haoxin Chen, Yong Zhang, Menghan Xia, Xiaodong Cun, Zhixun Su, Ying Shan

Abstract: In order to improve the quality of synthesized videos, currently, one predominant method involves retraining an expert diffusion model and then implementing a noising-denoising process for refinement. Despite the significant training costs, maintaining consistency of content between the original and enhanced videos remains a major challenge. To tackle this challenge, we propose a novel formulation… ▽ More In order to improve the quality of synthesized videos, currently, one predominant method involves retraining an expert diffusion model and then implementing a noising-denoising process for refinement. Despite the significant training costs, maintaining consistency of content between the original and enhanced videos remains a major challenge. To tackle this challenge, we propose a novel formulation that considers both visual quality and consistency of content. Consistency of content is ensured by a proposed loss function that maintains the structure of the input, while visual quality is improved by utilizing the denoising process of pretrained diffusion models. To address the formulated optimization problem, we have developed a plug-and-play noise optimization strategy, referred to as Noise Calibration. By refining the initial random noise through a few iterations, the content of original video can be largely preserved, and the enhancement effect demonstrates a notable improvement. Extensive experiments have demonstrated the effectiveness of the proposed method. △ Less

Submitted 14 July, 2024; originally announced July 2024.

Comments: ECCV 2024, Project Page: https://yangqy1110.github.io/NC-SDEdit/, Code Repo: https://github.com/yangqy1110/NC-SDEdit/

ACM Class: I.2; I.4.3

Asynchronous Feedback Network for Perceptual Point Cloud Quality Assessment

Authors: Yujie Zhang, Qi Yang, Ziyu Shan, Yiling Xu

Abstract: Recent years have witnessed the success of the deep learning-based technique in research of no-reference point cloud quality assessment (NR-PCQA). For a more accurate quality prediction, many previous studies have attempted to capture global and local feature in a bottom-up manner, but ignored the interaction and promotion between them. To solve this problem, we propose a novel asynchronous feedba… ▽ More Recent years have witnessed the success of the deep learning-based technique in research of no-reference point cloud quality assessment (NR-PCQA). For a more accurate quality prediction, many previous studies have attempted to capture global and local feature in a bottom-up manner, but ignored the interaction and promotion between them. To solve this problem, we propose a novel asynchronous feedback network (AFNet). Motivated by human visual perception mechanisms, AFNet employs a dual-branch structure to deal with global and local feature, simulating the left and right hemispheres of the human brain, and constructs a feedback module between them. Specifically, the input point clouds are first fed into a transformer-based global encoder to generate the attention maps that highlight these semantically rich regions, followed by being merged into the global feature. Then, we utilize the generated attention maps to perform dynamic convolution for different semantic regions and obtain the local feature. Finally, a coarse-to-fine strategy is adopted to merge the two features into the final quality score. We conduct comprehensive experiments on three datasets and achieve superior performance over the state-of-the-art approaches on all of these datasets. The code will be available at https://github.com/zhangyujie-1998/AFNet. △ Less

Submitted 13 July, 2024; originally announced July 2024.

Study of a Novel Capacitive Pressure Sensor Using Spiral Comb Electrodes

Authors: Wenjie Chen, Qi Yang, Qi Liu, Yiqun Zhang, Liang He, Yuanlin Xia, Zhuqing Wang, Yubo Huang, Jianfeng Chen, Cao Xia

Abstract: For traditional capacitive pressure sensors, high nonlinearity and poor sensitivity greatly limited their sensing applications. Hence, an innovative design of capacitors based on spiral comb electrodes is proposed for high-sensitivity pressure detection in this work. Compared to traditional capacitive pressure sensors with straight plate electrodes, the proposed sensor with the spiral electrodes i… ▽ More For traditional capacitive pressure sensors, high nonlinearity and poor sensitivity greatly limited their sensing applications. Hence, an innovative design of capacitors based on spiral comb electrodes is proposed for high-sensitivity pressure detection in this work. Compared to traditional capacitive pressure sensors with straight plate electrodes, the proposed sensor with the spiral electrodes increases the overlap areas of electrodes sufficiently, the pressure sensitivity can thus be greatly improved. Moreover, the capacitance variation of the proposed sensor is dominated by the change of the overlap area of the electrodes rather than the electrode's distance, the linearity can also thus be improved to higher than 0.99. Theoretical analysis and COMSOL-based finite element simulation have been implemented for principle verification and performance optimization. Simulation results show that the proposed design has a mechanical sensitivity of 1.5x10-4 m/Pa, capacitive sensitivity of 1.10 aF/Pa, and nonlinear error of 3.63%, respectively, at the pressure range from 0 to 30 kPa. An equivalent experiment has been further carried out for verification. Experimental results also show that both the sensitivity and linearity of capacitive pressure sensors with spiral electrodes are higher than those with straight electrodes. This work not only provides a new avenue for capacitor design, but also can be applied to high-sensitivity pressure detection. △ Less

Submitted 11 July, 2024; originally announced July 2024.

Comments: 20 pages, 14 figures

MSC Class: -

Explicit-NeRF-QA: A Quality Assessment Database for Explicit NeRF Model Compression

Authors: Yuke Xing, Qi Yang, Kaifa Yang, Yilin Xu, Zhu Li

Abstract: In recent years, Neural Radiance Fields (NeRF) have demonstrated significant advantages in representing and synthesizing 3D scenes. Explicit NeRF models facilitate the practical NeRF applications with faster rendering speed, and also attract considerable attention in NeRF compression due to its huge storage cost. To address the challenge of the NeRF compression study, in this paper, we construct a… ▽ More In recent years, Neural Radiance Fields (NeRF) have demonstrated significant advantages in representing and synthesizing 3D scenes. Explicit NeRF models facilitate the practical NeRF applications with faster rendering speed, and also attract considerable attention in NeRF compression due to its huge storage cost. To address the challenge of the NeRF compression study, in this paper, we construct a new dataset, called Explicit-NeRF-QA. We use 22 3D objects with diverse geometries, textures, and material complexities to train four typical explicit NeRF models across five parameter levels. Lossy compression is introduced during the model generation, pivoting the selection of key parameters such as hash table size for InstantNGP and voxel grid resolution for Plenoxels. By rendering NeRF samples to processed video sequences (PVS), a large scale subjective experiment with lab environment is conducted to collect subjective scores from 21 viewers. The diversity of content, accuracy of mean opinion scores (MOS), and characteristics of NeRF distortion are comprehensively presented, establishing the heterogeneity of the proposed dataset. The state-of-the-art objective metrics are tested in the new dataset. Best Person correlation, which is around 0.85, is collected from the full-reference objective metric. All tested no-reference metrics report very poor results with 0.4 to 0.6 correlations, demonstrating the need for further development of more robust no-reference metrics. The dataset, including NeRF samples, source 3D objects, multiview images for NeRF generation, PVSs, MOS, is made publicly available at the following location: https://github.com/YukeXing/Explicit-NeRF-QA. △ Less

Submitted 20 September, 2024; v1 submitted 11 July, 2024; originally announced July 2024.

Comments: 5 pages, 4 figures, 2 tables, conference

Decompose and Compare Consistency: Measuring VLMs' Answer Reliability via Task-Decomposition Consistency Comparison

Authors: Qian Yang, Weixiang Yan, Aishwarya Agrawal

Abstract: Despite tremendous advancements, current state-of-the-art Vision-Language Models (VLMs) are still far from perfect. They tend to hallucinate and may generate biased responses. In such circumstances, having a way to assess the reliability of a given response generated by a VLM is quite useful. Existing methods, such as estimating uncertainty using answer likelihoods or prompt-based confidence gener… ▽ More Despite tremendous advancements, current state-of-the-art Vision-Language Models (VLMs) are still far from perfect. They tend to hallucinate and may generate biased responses. In such circumstances, having a way to assess the reliability of a given response generated by a VLM is quite useful. Existing methods, such as estimating uncertainty using answer likelihoods or prompt-based confidence generation, often suffer from overconfidence. Other methods use self-consistency comparison but are affected by confirmation biases. To alleviate these, we propose Decompose and Compare Consistency (DeCC) for reliability measurement. By comparing the consistency between the direct answer generated using the VLM's internal reasoning process, and the indirect answers obtained by decomposing the question into sub-questions and reasoning over the sub-answers produced by the VLM, DeCC measures the reliability of VLM's direct answer. Experiments across six vision-language tasks with three VLMs show DeCC's reliability estimation achieves better correlation with task accuracy compared to the existing methods. △ Less

Submitted 8 October, 2024; v1 submitted 10 July, 2024; originally announced July 2024.

Comments: Accepted to EMNLP 2024 Main Conference

Bucket Pre-training is All You Need

Authors: Hongtao Liu, Qiyao Peng, Qing Yang, Kai Liu, Hongyan Xu

Abstract: Large language models (LLMs) have demonstrated exceptional performance across various natural language processing tasks. However, the conventional fixed-length data composition strategy for pretraining, which involves concatenating and splitting documents, can introduce noise and limit the model's ability to capture long-range dependencies. To address this, we first introduce three metrics for eva… ▽ More Large language models (LLMs) have demonstrated exceptional performance across various natural language processing tasks. However, the conventional fixed-length data composition strategy for pretraining, which involves concatenating and splitting documents, can introduce noise and limit the model's ability to capture long-range dependencies. To address this, we first introduce three metrics for evaluating data composition quality: padding ratio, truncation ratio, and concatenation ratio. We further propose a multi-bucket data composition method that moves beyond the fixed-length paradigm, offering a more flexible and efficient approach to pretraining. Extensive experiments demonstrate that our proposed method could significantly improving both the efficiency and efficacy of LLMs pretraining. Our approach not only reduces noise and preserves context but also accelerates training, making it a promising solution for LLMs pretraining. △ Less

Submitted 10 July, 2024; originally announced July 2024.

Review-LLM: Harnessing Large Language Models for Personalized Review Generation

Authors: Qiyao Peng, Hongtao Liu, Hongyan Xu, Qing Yang, Minglai Shao, Wenjun Wang

Abstract: Product review generation is an important task in recommender systems, which could provide explanation and persuasiveness for the recommendation. Recently, Large Language Models (LLMs, e.g., ChatGPT) have shown superior text modeling and generating ability, which could be applied in review generation. However, directly applying the LLMs for generating reviews might be troubled by the ``polite'' ph… ▽ More Product review generation is an important task in recommender systems, which could provide explanation and persuasiveness for the recommendation. Recently, Large Language Models (LLMs, e.g., ChatGPT) have shown superior text modeling and generating ability, which could be applied in review generation. However, directly applying the LLMs for generating reviews might be troubled by the ``polite'' phenomenon of the LLMs and could not generate personalized reviews (e.g., negative reviews). In this paper, we propose Review-LLM that customizes LLMs for personalized review generation. Firstly, we construct the prompt input by aggregating user historical behaviors, which include corresponding item titles and reviews. This enables the LLMs to capture user interest features and review writing style. Secondly, we incorporate ratings as indicators of satisfaction into the prompt, which could further improve the model's understanding of user preferences and the sentiment tendency control of generated reviews. Finally, we feed the prompt text into LLMs, and use Supervised Fine-Tuning (SFT) to make the model generate personalized reviews for the given user and target item. Experimental results on the real-world dataset show that our fine-tuned model could achieve better review generation performance than existing close-source LLMs. △ Less

Submitted 10 July, 2024; originally announced July 2024.

Nonrigid Reconstruction of Freehand Ultrasound without a Tracker

Authors: Qi Li, Ziyi Shen, Qianye Yang, Dean C. Barratt, Matthew J. Clarkson, Tom Vercauteren, Yipeng Hu

Abstract: Reconstructing 2D freehand Ultrasound (US) frames into 3D space without using a tracker has recently seen advances with deep learning. Predicting good frame-to-frame rigid transformations is often accepted as the learning objective, especially when the ground-truth labels from spatial tracking devices are inherently rigid transformations. Motivated by a) the observed nonrigid deformation due to so… ▽ More Reconstructing 2D freehand Ultrasound (US) frames into 3D space without using a tracker has recently seen advances with deep learning. Predicting good frame-to-frame rigid transformations is often accepted as the learning objective, especially when the ground-truth labels from spatial tracking devices are inherently rigid transformations. Motivated by a) the observed nonrigid deformation due to soft tissue motion during scanning, and b) the highly sensitive prediction of rigid transformation, this study investigates the methods and their benefits in predicting nonrigid transformations for reconstructing 3D US. We propose a novel co-optimisation algorithm for simultaneously estimating rigid transformations among US frames, supervised by ground-truth from a tracker, and a nonrigid deformation, optimised by a regularised registration network. We show that these two objectives can be either optimised using meta-learning or combined by weighting. A fast scattered data interpolation is also developed for enabling frequent reconstruction and registration of non-parallel US frames, during training. With a new data set containing over 357,000 frames in 720 scans, acquired from 60 subjects, the experiments demonstrate that, due to an expanded thus easier-to-optimise solution space, the generalisation is improved with the added deformation estimation, with respect to the rigid ground-truth. The global pixel reconstruction error (assessing accumulative prediction) is lowered from 18.48 to 16.51 mm, compared with baseline rigid-transformation-predicting methods. Using manually identified landmarks, the proposed co-optimisation also shows potentials in compensating nonrigid tissue motion at inference, which is not measurable by tracker-provided ground-truth. The code and data used in this paper are made publicly available at https://github.com/QiLi111/NR-Rec-FUS. △ Less

Submitted 14 July, 2024; v1 submitted 8 July, 2024; originally announced July 2024.

Comments: Accepted at MICCAI 2024

Properties of Besov-Lorentz spaces and application to Navier-Stokes equations

Authors: Qixiang Yang, Hongwei Li

Abstract: Inspired by Caffarelli-Kohn-Nirenberg, Fefferman and Lin, we try to investigate how to control the set of large value points for the strong solution of Navier-Stokes equations. Besov-Lorentz spaces have multiple indices which can reflect complex changes of the set of the large value points. Hence we consider some properties of Gauss flow, paraproduct flow and couple flow related to the Besov-Loren… ▽ More Inspired by Caffarelli-Kohn-Nirenberg, Fefferman and Lin, we try to investigate how to control the set of large value points for the strong solution of Navier-Stokes equations. Besov-Lorentz spaces have multiple indices which can reflect complex changes of the set of the large value points. Hence we consider some properties of Gauss flow, paraproduct flow and couple flow related to the Besov-Lorentz spaces. When dealing with Lorentz index, we use wavelets and maximum norm to describe the decay situation in the binary time ring and to define time-frequency microlocal maximum norm space. We use maximum operator, $α$-triangle inequality and Hölder inequality etc to get accurate estimates. As an application, we get a global wellposedness result of the Navier-Stokes equations where the solution can reflect how the size of the set of large value points changes. △ Less

Submitted 4 July, 2024; originally announced July 2024.

MSC Class: 35Q30; 76D03; 42B35; 46E30

Journal ref: SCIENCE CHINA Mathematics (2024)

Perception-Guided Quality Metric of 3D Point Clouds Using Hybrid Strategy

Authors: Yujie Zhang, Qi Yang, Yiling Xu, Shan Liu

Abstract: Full-reference point cloud quality assessment (FR-PCQA) aims to infer the quality of distorted point clouds with available references. Most of the existing FR-PCQA metrics ignore the fact that the human visual system (HVS) dynamically tackles visual information according to different distortion levels (i.e., distortion detection for high-quality samples and appearance perception for low-quality sa… ▽ More Full-reference point cloud quality assessment (FR-PCQA) aims to infer the quality of distorted point clouds with available references. Most of the existing FR-PCQA metrics ignore the fact that the human visual system (HVS) dynamically tackles visual information according to different distortion levels (i.e., distortion detection for high-quality samples and appearance perception for low-quality samples) and measure point cloud quality using unified features. To bridge the gap, in this paper, we propose a perception-guided hybrid metric (PHM) that adaptively leverages two visual strategies with respect to distortion degree to predict point cloud quality: to measure visible difference in high-quality samples, PHM takes into account the masking effect and employs texture complexity as an effective compensatory factor for absolute difference; on the other hand, PHM leverages spectral graph theory to evaluate appearance degradation in low-quality samples. Variations in geometric signals on graphs and changes in the spectral graph wavelet coefficients are utilized to characterize geometry and texture appearance degradation, respectively. Finally, the results obtained from the two components are combined in a non-linear method to produce an overall quality score of the tested point cloud. The results of the experiment on five independent databases show that PHM achieves state-of-the-art (SOTA) performance and offers significant performance improvement in multiple distortion environments. The code is publicly available at https://github.com/zhangyujie-1998/PHM. △ Less

Submitted 27 September, 2024; v1 submitted 4 July, 2024; originally announced July 2024.

Automated Progressive Red Teaming

Authors: Bojian Jiang, Yi Jing, Tianhao Shen, Tong Wu, Qing Yang, Deyi Xiong

Abstract: Ensuring the safety of large language models (LLMs) is paramount, yet identifying potential vulnerabilities is challenging. While manual red teaming is effective, it is time-consuming, costly and lacks scalability. Automated red teaming (ART) offers a more cost-effective alternative, automatically generating adversarial prompts to expose LLM vulnerabilities. However, in current ART efforts, a robu… ▽ More Ensuring the safety of large language models (LLMs) is paramount, yet identifying potential vulnerabilities is challenging. While manual red teaming is effective, it is time-consuming, costly and lacks scalability. Automated red teaming (ART) offers a more cost-effective alternative, automatically generating adversarial prompts to expose LLM vulnerabilities. However, in current ART efforts, a robust framework is absent, which explicitly frames red teaming as an effectively learnable task. To address this gap, we propose Automated Progressive Red Teaming (APRT) as an effectively learnable framework. APRT leverages three core modules: an Intention Expanding LLM that generates diverse initial attack samples, an Intention Hiding LLM that crafts deceptive prompts, and an Evil Maker to manage prompt diversity and filter ineffective samples. The three modules collectively and progressively explore and exploit LLM vulnerabilities through multi-round interactions. In addition to the framework, we further propose a novel indicator, Attack Effectiveness Rate (AER) to mitigate the limitations of existing evaluation metrics. By measuring the likelihood of eliciting unsafe but seemingly helpful responses, AER aligns closely with human evaluations. Extensive experiments with both automatic and human evaluations, demonstrate the effectiveness of ARPT across both open- and closed-source LLMs. Specifically, APRT effectively elicits 54% unsafe yet useful responses from Meta's Llama-3-8B-Instruct, 50% from GPT-4o (API access), and 39% from Claude-3.5 (API access), showcasing its robust attack capability and transferability across LLMs (especially from open-source LLMs to closed-source LLMs). △ Less

Submitted 5 October, 2024; v1 submitted 4 July, 2024; originally announced July 2024.

Advanced Framework for Animal Sound Classification With Features Optimization

Authors: Qiang Yang, Xiuying Chen, Changsheng Ma, Carlos M. Duarte, Xiangliang Zhang

Abstract: The automatic classification of animal sounds presents an enduring challenge in bioacoustics, owing to the diverse statistical properties of sound signals, variations in recording equipment, and prevalent low Signal-to-Noise Ratio (SNR) conditions. Deep learning models like Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) have excelled in human speech recognition but have not… ▽ More The automatic classification of animal sounds presents an enduring challenge in bioacoustics, owing to the diverse statistical properties of sound signals, variations in recording equipment, and prevalent low Signal-to-Noise Ratio (SNR) conditions. Deep learning models like Convolutional Neural Networks (CNN) and Long Short-Term Memory (LSTM) have excelled in human speech recognition but have not been effectively tailored to the intricate nature of animal sounds, which exhibit substantial diversity even within the same domain. We propose an automated classification framework applicable to general animal sound classification. Our approach first optimizes audio features from Mel-frequency cepstral coefficients (MFCC) including feature rearrangement and feature reduction. It then uses the optimized features for the deep learning model, i.e., an attention-based Bidirectional LSTM (Bi-LSTM), to extract deep semantic features for sound classification. We also contribute an animal sound benchmark dataset encompassing oceanic animals and birds1. Extensive experimentation with real-world datasets demonstrates that our approach consistently outperforms baseline methods by over 25% in precision, recall, and accuracy, promising advancements in animal sound classification. △ Less

Submitted 3 July, 2024; originally announced July 2024.

Observation of Co-propagating Chiral Zero Modes in Magnetic Photonic Crystals

Authors: Zhongfu Li, Shaojie Ma, Shuwei Li, Oubo you, Yachao Liu, Qingdong Yang, Yuanjiang Xiang, Peiheng Zhou, Shuang Zhang

Abstract: Topological singularities, such as Weyl points and Dirac points, can give rise to unidirectional propagation channels known as chiral zero modes (CZMs) when subject to a magnetic field. These CZMs are responsible for intriguing phenomena like the chiral anomaly in quantum systems. The propagation direction of each CZM is determined by both the applied magnetic field and the topological charge of t… ▽ More Topological singularities, such as Weyl points and Dirac points, can give rise to unidirectional propagation channels known as chiral zero modes (CZMs) when subject to a magnetic field. These CZMs are responsible for intriguing phenomena like the chiral anomaly in quantum systems. The propagation direction of each CZM is determined by both the applied magnetic field and the topological charge of the singularity point. While counter-propagating CZMs have been observed in 2D and 3D systems, the realization of co-propagating CZMs has remained elusive. Here we present the first experimental observation of co-propagating CZMs in magnetic photonic crystals hosting a single pair of ideal Weyl points WPs. By manipulating the crystal's structural configuration, we spatially alter the locations of the WPs, creating pseudo-magnetic fields in opposite directions between them. This arrangement results in a pair of CZMs that possess the same group velocity and co-propagate. Our work opens up new possibilities for topological manipulation of wave propagation and may lead to advancements in optical waveguides, switches, and various other applications. △ Less

Submitted 3 July, 2024; originally announced July 2024.

Comments: 6 pages, 5 figures

Knowledge-Aware Parsimony Learning: A Perspective from Relational Graphs

Authors: Quanming Yao, Yongqi Zhang, Yaqing Wang, Nan Yin, James Kwok, Qiang Yang

Abstract: The scaling law, which involves the brute-force expansion of training datasets and learnable parameters, has become a prevalent strategy for developing more robust learning models. However, due to bottlenecks in data, computation, and trust, the sustainability of the scaling law is a serious concern for the future of deep learning. In this paper, we address this issue by developing next-generation… ▽ More The scaling law, which involves the brute-force expansion of training datasets and learnable parameters, has become a prevalent strategy for developing more robust learning models. However, due to bottlenecks in data, computation, and trust, the sustainability of the scaling law is a serious concern for the future of deep learning. In this paper, we address this issue by developing next-generation models in a parsimonious manner (i.e., achieving greater potential with simpler models). The key is to drive models using domain-specific knowledge, such as symbols, logic, and formulas, instead of relying on the scaling law. This approach allows us to build a framework that uses this knowledge as "building blocks" to achieve parsimony in model design, training, and interpretation. Empirical results show that our methods surpass those that typically follow the scaling law. We also demonstrate the application of our framework in AI for science, specifically in the problem of drug-drug interaction prediction. We hope our research can foster more diverse technical roadmaps in the era of foundation models. △ Less

Submitted 10 October, 2024; v1 submitted 29 June, 2024; originally announced July 2024.

Vision Transformer with Key-select Routing Attention for Single Image Dehazing

Authors: Lihan Tong, Weijia Li, Qingxia Yang, Liyuan Chen, Peng Chen

Abstract: We present Ksformer, utilizing Multi-scale Key-select Routing Attention (MKRA) for intelligent selection of key areas through multi-channel, multi-scale windows with a top-k operator, and Lightweight Frequency Processing Module (LFPM) to enhance high-frequency features, outperforming other dehazing methods in tests. We present Ksformer, utilizing Multi-scale Key-select Routing Attention (MKRA) for intelligent selection of key areas through multi-channel, multi-scale windows with a top-k operator, and Lightweight Frequency Processing Module (LFPM) to enhance high-frequency features, outperforming other dehazing methods in tests. △ Less

Submitted 28 June, 2024; originally announced June 2024.

Comments: 5 pages,4 figures,IEICE Trans. Information and Systems

Report number: Vol.E107-D,No.11,pp.-,Nov. 2024 MSC Class: 68U10(Primary) ACM Class: I.4

Streaming Decoder-Only Automatic Speech Recognition with Discrete Speech Units: A Pilot Study

Authors: Peikun Chen, Sining Sun, Changhao Shan, Qing Yang, Lei Xie

Abstract: Unified speech-text models like SpeechGPT, VioLA, and AudioPaLM have shown impressive performance across various speech-related tasks, especially in Automatic Speech Recognition (ASR). These models typically adopt a unified method to model discrete speech and text tokens, followed by training a decoder-only transformer. However, they are all designed for non-streaming ASR tasks, where the entire s… ▽ More Unified speech-text models like SpeechGPT, VioLA, and AudioPaLM have shown impressive performance across various speech-related tasks, especially in Automatic Speech Recognition (ASR). These models typically adopt a unified method to model discrete speech and text tokens, followed by training a decoder-only transformer. However, they are all designed for non-streaming ASR tasks, where the entire speech utterance is needed during decoding. Hence, we introduce a decoder-only model exclusively designed for streaming recognition, incorporating a dedicated boundary token to facilitate streaming recognition and employing causal attention masking during the training phase. Furthermore, we introduce right-chunk attention and various data augmentation techniques to improve the model's contextual modeling abilities. While achieving streaming speech recognition, experiments on the AISHELL-1 and -2 datasets demonstrate the competitive performance of our streaming approach with non-streaming decoder-only counterparts. △ Less

Submitted 26 June, 2024; originally announced June 2024.

Comments: Accepted for Interspeech 2024

Make Some Noise: Unlocking Language Model Parallel Inference Capability through Noisy Training

Authors: Yixuan Wang, Xianzhen Luo, Fuxuan Wei, Yijun Liu, Qingfu Zhu, Xuanyu Zhang, Qing Yang, Dongliang Xu, Wanxiang Che

Abstract: Existing speculative decoding methods typically require additional model structure and training processes to assist the model for draft token generation. This makes the migration of acceleration methods to the new model more costly and more demanding on device memory. To address this problem, we propose the Make Some Noise (MSN) training framework as a replacement for the supervised fine-tuning st… ▽ More Existing speculative decoding methods typically require additional model structure and training processes to assist the model for draft token generation. This makes the migration of acceleration methods to the new model more costly and more demanding on device memory. To address this problem, we propose the Make Some Noise (MSN) training framework as a replacement for the supervised fine-tuning stage of the large language model. The training method simply introduces some noise at the input for the model to learn the denoising task. It significantly enhances the parallel decoding capability of the model without affecting the original task capability. In addition, we propose a tree-based retrieval-augmented Jacobi (TR-Jacobi) decoding strategy to further improve the inference speed of MSN models. Experiments in both the general and code domains have shown that MSN can improve inference speed by 2.3-2.7x times without compromising model performance. The MSN model also achieves comparable acceleration ratios to the SOTA model with additional model structure on Spec-Bench. △ Less

Submitted 5 October, 2024; v1 submitted 25 June, 2024; originally announced June 2024.

Comments: EMNLP 2024, camera ready