Hao Chen

Visualization-of-Thought (VoT) prompting to enhance spatial reasoning abilities in large language models. VoT prompts aim to improve spatial reasoning in large language models by visualizing their reasoning traces. #genai #vot https://lnkd.in/e4NrwVrC

2

🌐 Breakthrough in Neural Networks Training: Accelerated Convergence with NGD! 🚀 In a significant advancement for neural network training, researchers have unveiled a method to dramatically enhance the learning rate for gradient descent (GD) in two-layer neural networks. Traditional first-order methods like GD and stochastic gradient descent (SGD), while effective, often suffer from slow convergence due to dependencies on sample size and the Gram matrix. The new approach boosts the learning rate for L2 regression problems from O(λ0/n2) to to O(1/∥H∞∥_2), ensuring faster and more efficient training. The improvements extend to Physics-Informed Neural Networks (PINNs), crucial for solving high-dimensional partial differential equations in scientific computing. By refining the recursion formula of GD, the learning rate for PINNs also sees a substantial upgrade, significantly reducing the training time even with large datasets. This breakthrough addresses the inherent slow convergence issues linked to the Gram matrix's eigenvalues, setting a new benchmark in neural network optimization. The introduction of Natural Gradient Descent (NGD) marks a pivotal leap forward. Unlike traditional GD, NGD's learning rate can reach O(1), with a convergence rate independent of the Gram matrix. This innovation ensures much faster training, especially with smooth activation functions where NGD enjoys a quadratic convergence rate. This research not only accelerates neural network training but also opens new horizons for deploying more efficient and scalable machine learning models in various complex applications. "Convergence Analysis of Natural Gradient Descent for Over-parameterized Physics-Informed Neural Networks" Xianliang Xu, Ting Du, Wang Kong, Ye Li, Zhongyi Huang Arxiv Preprint: https://lnkd.in/gVT4Uvfw #MachineLearning #GradientDescent #Optimization #PINN #AIResearch

1

Check out this highly recommended course for those interested in optimizing their models: - Introduction to Quantization - Symmetric and Asymmetric Quantization Modes - Hands-on Implementation from Scratch using PyTorch - covering Per Tensor, Per Channel (Axis), and Per Group techniques - Weights Packing: Exploring 2 or 4 Bits Quantization Methods https://lnkd.in/gMAuWXHm

43

Happy to share a publication update from our group MISN IITD.  Our work “Optimization Framework for Semi-supervised Attributed Graph Coarsening” got accepted to UAI2024.One of the first approach for Optimization based semi-supervised graph coarsening. Abstract: In data-intensive applications, graphs serve as foundational structures across various domains. However, the increasing size of datasets poses significant challenges to performing downstream tasks. To address this problem, techniques such as graph coarsening, condensation, and summarization have been developed to create a coarsened graph while preserving important properties of the original graph by considering both the graph matrix and the feature or attribute matrix of the original graph as inputs. However, existing graph coarsening tech- niques often neglect the label information during the coarsening process, which can result in a lower quality coarsened graph and limit its suitability for downstream tasks. To overcome this limitation, we introduce the Label-Aware Graph Coarsening (LAGC) algorithm, a semi-supervised approach that incorporates the graph matrix, feature matrix, and some of the node label information to learn a coarsened graph. Our proposed formulation is a non-convex optimization problem that is efficiently solved using block successive upper bound mini- mization(BSUM) technique, and it is provably con- vergent. Our extensive results demonstrate that the LAGC algorithm outperforms the existing state-of- the-art method by a significant margin. Congratulations! Manoj kumar Subhanu Halder#UAI #graphML

119

6 Comments

Our next presentation within the IBM University of Illinois Urbana-Champaign Discovery Accelerator Institute is coming up in two days: Speaker: Deming Chen, UIUC Title: SaiS - Security for AI and AI for Security Join here on Friday, 10/18, 11 am Eastern Time (or watch recording): https://lnkd.in/gESqbTUh Abstract: Artificial intelligence (AI) is transforming various industries and aspects of human life, including healthcare, finance, retail, autonomous vehicles, national security, and smart grids. Consequently, securing AI systems has become imperative. At the same time, AI can significantly enhance system security, for instance, by using machine learning to detect and respond to cyber threats in real-time or employing AI tools to prevent financial crimes. This dual focus—securing AI and leveraging AI for security—is essential, as these two aspects are deeply interconnected. In this talk, we will discuss the unique security challenges faced by AI systems and explore strategies such as Trusted Execution Environments (TEEs) to protect AI models and data, especially when using AI accelerators. We will also introduce AI-based techniques for detecting malicious activities in cyber-physical and IoT systems, concluding with a vision for utilizing TEEs, SmartNICs, and a new concept called 'safe-domains' to securely manage diverse AI workloads in hybrid cloud environments.

8

🚨ALERT: Innovation in artificial neural networks🚨 Published earlier today: “SWAG: A Novel Neural Network Architecture Leveraging Polynomial Activation Functions for Enhanced Deep Learning Efficiency” Kudos to Dr. Saeid Safaei, my last doctoral student at the The University of Georgia. This work is significant because it breaks a dogma in artificial neural network (ANN) literature. In 1993, Leshno et al. reported that an ANN “… can approximate any continuous function to any degree of accuracy if and only if the network's activation function is not a polynomial”. This myth became legend, the legend became dogma. We showed that not only it is possible to use polynomials… it is desirable. Our method with simple architectures outperforms traditional ANNs with complex architectures. It could have profound implications to decrease energy demand for AI through the development of photonic chips. https://lnkd.in/gQAtcNEa

23

8 Comments

This April, I delved into "Machine Learning in Production" by Andrew Kelleher and Adam Kelleher, a compelling guide for anyone interested in transitioning machine learning models from the lab to real-world applications. This book is a treasure trove of insights on optimizing data science workflows and ensuring that machine learning systems are scalable, maintainable, and robust. Key Takeaways: Best Practices for Deployment: The authors do an excellent job outlining the lifecycle of ML models in production, emphasizing the importance of continuous monitoring and iteration. Challenges in Production: Real-world anecdotes highlight common pitfalls and how to navigate them, providing a practical perspective often missing in more theoretical texts. Tools and Techniques: From advanced algorithms to deployment tools, this book offers a comprehensive look at the technologies driving modern machine learning. This read is highly recommended for data scientists and engineers aiming to bring their models to production while ensuring they deliver tangible, scalable results. 🚀 If you're looking to enhance your understanding of machine learning in a production environment, this is a must-read! #MachineLearning #DataScience #AI #Technology #BookOfTheMonth #ProfessionalDevelopment #GenAI

25

1 Comment

Transformers Transformers has been a major milestone in the field of NLP and heavily used in Generative AI. There are 3 variants of transformer based models and they are 1. Encoder-only 2. Decoder-only 3. Encoder-Decoder Encoder only models: These are also called as autoencoders and these are pretrained using a technique called masked language modeling. A text with a random masked token is sent to the model to predict the masked token. For example, consider the text "If you don't stop at the sign, you will get a ticket" The training input that will be passed on the encoder model is "If you don't _____ at the sign, you will get a ticket. The model is expected to predict the token (word) "stop" These models use bi-directional representations of the input to better understand the full context of a token. Examples of encoder models: BERT Family models Decoder only models: These models are called autoregressive models and are pretrained using a technique called causal language modeling. These models predict the next token using the previous tokens. For example, consider the text "If you don't stop at the sign, you will get a ticket" The training input that will be passed on the encoder model is "If you don't ______" The model will still try to predict the word "stop", but only based on previous tokens. These models are used for generative tasks, including question answering. Examples of decoder only models: GPT Family, Falcon, LLama models Encoder decoder models: These models are called sequence to sequence models and the pretraining objective varies from model to model. For example the popular FLAN - T5 uses a consecutive multitoken masking called span corruption. For example, consider the text "If you don't stop at the sign, you will get a ticket" The training input that will be passed on the encoder model is "If you don't _____ _______ the sign, you will get a ticket. The model will try to predict the tokens "stop at" These models are good at translation tasks. Examples: T5 family In all the above explanations we just took one sentence of text. The LLMs you see in the market are trained on huge volumes of text available over the internet. #LLM #encoder #decoder #transformer #genai

3

If you are attending IEEE Quantum Week, don't miss this! BosonQ Psi (BQP)'s CEO and Chief Scientific Officer Abhishek Chopra will be presenting our paper "Benchmarking Quantum-Assisted PINN (QA-PINN) for Computational Fluid Dynamics". Although the paper focuses on CFD, the findings of the paper should be interesting to everyone involved in Artificial Intelligence! Jay Shah #MachineLearning #CFD #AI #QuantumComputing #QuantumAI #Technology #ComputerScience

18

The Transformer Series: This video explains the concept of positional encoding in the transformer architecture. In our previous videos, we looked at Skip connections, scaled dot product attention and multi-head attention. Now, let’s shift our focus to another important aspect of the transformer architecture: Positional Encoding. Positional encoding is crucial for transformers to understand the order of input data when looking at sequential data, such as the usual english sentences where the order of the words is important. Positional encoding adds unique markers to each input token to provide context of its position in the sequence. This method allows the model to recognize patterns and relationships in data that depend on the sequence order—something essential in tasks like language understanding or time series analysis. Take a look at this video to discover how positional encoding enables transformers to map out complex underlying patterns in sequence data. https://lnkd.in/g4bi4_A3 #transformers #deeplearning #ai #ml #nlp

59

🚀 Advancing Information Retrieval with APEER: Automating Prompt Engineering for LLMs! 🚀 I recently delved into a fascinating paper on APEER (Automatic Prompt Engineering Enhances LLM Reranking), a method transforming the way we approach relevance ranking in Information Retrieval (IR) using Large Language Models (LLMs). 🌟 🔍 Challenges in IR with LLMs: Current IR methods with LLMs often rely heavily on human-crafted prompts for zero-shot relevance ranking. This process is time-consuming, subjective, and lacks scalability. The complexities of integrating query and passage pairs and conducting comprehensive relevance assessments further complicate the effectiveness of existing methods. 💡 APEER: A Game-Changer APEER addresses these challenges by automating prompt engineering through iterative feedback and preference optimization. Here's how it stands out: Feedback Optimization: Continuously refine prompts based on performance metrics. Preference Optimization: Improves prompts by learning from positive and negative examples. 🌟 Key Highlights: Reduced Human Effort: APEER minimizes the need for manual prompt crafting, making the process more efficient and less reliant on human expertise. Enhanced Performance: Demonstrates significant improvements in LLM performance on IR tasks, with notable gains in metrics like nDCG@10. For example, APEER achieved an average improvement of 5.29 nDCG@10 on eight BEIR datasets over manual prompts on the LLaMA3 model. Better Transferability: Shows consistent outperformance across diverse datasets and LLM architectures, including GPT-4, LLaMA3, and Qwen2. 📊 Robust Validation: APEER has been rigorously tested on multiple datasets such as MS MARCO, TREC-DL, and BEIR, ensuring its robustness and effectiveness in various IR scenarios. 🌐 Implications: This advancement represents a significant step forward in optimizing LLM prompts for complex relevance ranking tasks. By reducing manual intervention and enhancing the efficiency of LLMs, APEER paves the way for more scalable and accurate applications in real-world IR tasks. 🛠️ Conclusion: APEER's automated approach to prompt engineering is a major leap towards more effective and scalable IR systems. It highlights the potential of integrating iterative feedback and preference optimization to refine LLM performance, offering promising avenues for future research and practical applications. Looking forward to seeing how APEER influences the landscape of Information Retrieval and beyond! #InformationRetrieval #LLM #AI #MachineLearning #NLP #DataScience #Research

1

🌟 Excited to share that our latest article has been published! 📝 Grateful for the opportunity to collaborate with my fellow authors on this insightful piece. Check it out here: https://lnkd.in/gqi5DhGG #Collaboration #ArticlePublication #Gratitude 🚀

2

Just published a new developer walkthrough 🔥 Attention Mechanisms Simplified: Using einsum in PyTorch Implement various attention mechanisms, such as self-attention and multi-head attention, using einsum. https://buff.ly/3XtPNjW

1

Text chunks in retrieval may risk losing context. The below article introduces contextualizing each chunk using an LLM. The added cost and latency required for contextualizing is partially addressed through a lightweight LLM (haiku) and prompt caching. Along with other known techniques (BM25, reranking, etc.), a significant reduction in retrieval failure is achieved. Overall, a good article to read. https://lnkd.in/g_5fgAN8

10

CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts https://lnkd.in/gb78Y4bn In the latest foray into the world of multimodal language models, researchers have unveiled an intriguing new approach they've dubbed "CuMo." This novel technique aims to harness the power of Mixture-of-Experts (MoE) architectures, incorporating Co-upcycled Top-K sparsely-gated MoE blocks into the vision encoder and the multi-layer perceptron connector of multimodal LLMs. The incorporation of these MoE components is a bold move, promising to enhance the capabilities of multimodal LLMs in a significant way. However, the researchers didn't stop there. They've also adopted a three-stage training approach that employs auxiliary losses to help stabilize the training process and ensure a balanced distribution of workload across the expert modules. One of the most noteworthy aspects of this project is that CuMo has been exclusively trained on open-source datasets, a commendable decision that promotes transparency and accessibility in the field of AI research. Overall, this project presents an exciting exploration of MoE architectures in the context of multimodal language models. While the true impact of CuMo remains to be seen, the innovative approaches employed and the promising early results make this a development worth keeping an eye on in the rapidly evolving field of AI.

3

🚀 Unlock the Power of Canonical Renyi Correlation (CRC) in Median Filtering for Superior Image Preprocessing! Curious about cutting-edge techniques in image preprocessing? 🤔 In my latest blog post, I dive into how Canonical Renyi Correlation (CRC) enhances the traditional Median Filter (MF), preserving fine details while effectively reducing noise. 🔍 What’s inside: 🎯 How CRC sharpens MF for noise reduction without sacrificing detail. 🎯 A detailed breakdown of the math driving this process and its real-world impact on image captioning, object detection, and beyond. 🎯 The pros and cons of integrating CRC into MF and why it’s a game-changer for boosting image clarity. If you’re working in AI, Machine Learning, or any domain where image quality matters, this article is a must-read! 📖 👉 Read the full article here: https://lnkd.in/ggsn3GS4 I’d love to hear your thoughts! 💡 #ImageProcessing #AI #MachineLearning #Denoising #ComputerVision #DataScience #TechBlog

16