Ulric Musset

How fast is fast when it comes to commercial open source software traction? Here are some stats to serve as benchmarks that Matthieu Lavergne put together among many other great data and insights in Serena's first report on commercial open source software. It's available for free here: https://lnkd.in/d5uBvjAU

18

3 Comments

Wth are LLMs - Claude, LLaMA and of course, the more popular GPT models ? What happens inside these black-boxes ? How are they similar, and how do they differ from each other? So we look into: GPT – Leveraging transformer architecture, GPT-3.5 and GPT-4 power ChatGPT, honed through a mix of supervised and reinforcement learning for contextually rich text generation. LLaMA – Meta AI's LLaMA models, including LLaMA 2, offer a fresh take with leaner design, open-source accessibility, and potential for specialized language tasks, all runnable offline. Claude – Pioneering "constitutional AI," Claude ensures output relevance and safety by adhering to predefined principles, setting a new standard for practical and ethical text generation. We also look into the various tools being used for content generation, you can explore many of these tools and might find them very useful: - Text generators, conversational bots - AI-enhanced artistic creation - Dynamic audio synthesis, - A few multimodal synthesis tools The generative playground is expanding, promising a revolution in content creation across diverse mediums. ______________________________ To read more, please find substack link in comments A LOT of work goes into researching, studying, and articulating all this, hoping you will find it useful. #GenerativeModels #AI #LLMs

33

2 Comments

OpenAI released the price of GPT-4o mini which will enable a lot of business to use the LLM API more but it is still not cheap for big use cases. Here's a brief overview of the pricing for small LLM models from the main suppliers: 1. Claude Haiku Input : $0.25 / 1M token Output : $1.25 / 1M token 2. Llama-3 8b (Groq) Input : $0.05 / 1M token Output : $0.08 / 1M token 3. Gemini 1.5 Flash Input : $0.35 / 1M token Output : $1.05/ 1M token 4. GPT-4o mini Input : $0.15 / 1M token Output : $0.60 / 1M token For comparison, the standard GPT-4o pricing is : Input : $5.00 / 1M token Output : $15.00 / 1M token The price reduction is significant - but - keep in mind that using a Small Language Model impacts heavily output quality. My main concern with this price is knowing how computationally expensive llms are and the green factor for such big models are not that great. How are they making it available so cheap?

6

I’m excited to share that my latest article on transforming transformers is now live on VentureBeat 🚀! Hundreds of billions of dollars have already been spent on #AI models built on the Transformer architecture. What is the next big thing when it comes to foundation models? Recent model launches from Cohere, AI21 Labs, Databricks, and SambaNova Systems are making an impact, continuing to shape the rapidly evolving AI landscape. Data/AI leaders seem to have moved from 'one massive model to solve all problems' mindset to smaller model + domain-specific data = better/faster/cheaper performance. Just as individual teams currently select the best-in-class tools for cloud infrastructure, we will likely see different enterprise teams choose the most suitable models for their specific use cases. Already there are so many options available to a team as it rolls out AI products: 💡 open/closed source models now that there seems to be feature parity 💡 large/small language models depending on the use case 💡 state space models to better handle long sequences 💡 Composition of experts / Mixture of Experts for better accuracy 💡 To Fine-tune a model and/or use RAG architecture. What research areas have captured your interest the most? I’d love to hear your thoughts—feel free to share in the comments or send me a DM. I'm always up for a brainstorming session, especially with founders starting something new! Thomvest #founders #generativeai

73

7 Comments

Within Europe, it makes sense to focus on deep tech and their stages of risk taking. Tackle these systematically. 1. Scientific risk 2. Engineering risk 3. Commercialisation risk Europe is on par with the fundamentals on this, and fits better to ”no need to hype to get user traction” mentality.

4

Import feature I like in 4o-mini !:)- Price: GPT-4o mini is more than 60% cheaper than GPT-3.5 Turbo, priced at $0.15 per 1M input tokens and $0.60 per 1M output tokens (roughly the equivalent of 2500 pages in a standard book). https://lnkd.in/gMYYzrji

1

You will find here the latest HAI report on 2024 trends: https://lnkd.in/e2VHzaNJ As always, this is very deep, fact based and insightful. Thanks Nicolas Marescaux for sharing! Here are the key take aways: 1. AI beats humans on some tasks, but not on all. AI has surpassed human performance on several benchmarks, including some in image classification, visual reasoning, and English understanding. Yet it trails behind on more complex tasks like competition-level mathematics and planning. 2. Industry continues to dominate frontier AI research. In 2023, industry produced 51 notable machine learning models, while academia contributed only 15. 3. Frontier models get way more expensive. According to AI Index estimates, the training costs of state-of-the-art AI models have reached unprecedented levels. For example, OpenAI’s GPT-4 used an estimated $78 million worth of compute to train, while Google’s Gemini Ultra cost $191 million for compute. 4. The United States leads China, the EU, and the U.K. as the leading source of top AI models. In 2023, 61 notable AI models originated from U.S.-based institutions, far outpacing the European Union’s 21 and China’s 15. 5. Robust and standardized evaluations for LLM responsibility are seriously lacking. Leading developers, including OpenAI, Google, and Anthropic, primarily test their models against different responsible AI benchmarks. 6. Generative AI investment skyrockets. Despite a decline in overall AI private investment last year, funding for generative AI surged, nearly octupling from 2022 to reach $25.2 billion. Major players in the generative AI space, including OpenAI, Anthropic, Hugging Face, and Inflection, reported substantial fundraising rounds. 7. The data is in: AI makes workers more productive and leads to higher quality work. In 2023, several studies assessed AI’s impact on labor, suggesting that AI enables workers to complete tasks more quickly and to improve the quality of their output. 8. Scientific progress accelerates even further, thanks to AI. In 2022, AI began to advance scientific discovery. 2023, however, saw the launch of even more significant science-related AI applications— from AlphaDev, which makes algorithmic sorting more efficient, to GNoME, which facilitates the process of materials discovery. 9. The number of AI regulations in the United States sharply increases. The number of AI- related regulations in the U.S. has risen significantly in the past year and over the last five years. Last year alone, the total number of AI-related regulations grew by 56.3%. 10. People across the globe are more cognizant of AI’s potential impact—and more nervous. A survey from Ipsos shows that, over the last year, the proportion of those who think AI will dramatically affect their lives in the next three to five years has increased from 60% to 66%. Moreover, 52% express nervousness toward AI products and services, marking a 13 percentage point rise from 2022.

26

2 Comments

🚀🔍 A great analysis by Index Ventures on Vertical AI companies as part of the Application Layer of the AI stack. "With the AI Platform Shift upon us, we believe that the next logical iteration of vertical SaaS will be Vertical AI – vertically-focused AI platforms, bundled alongside workflow SaaS, built on top of models which have been uniquely trained on industry-specific datasets." The few things that will differentiate the winners in the race, as mentioned in the article:  - access to proprietary industry data,  - effectively train large language models against those datasets,  - package those models via applications,  - deliver tremendous utility with fast time-to-value for customers, At Neurolabs, we care a lot about the time-to-value, which is why we’ve built these exact steps for our Image recognition solution.✅ With speed of execution & user experience at core, we go a step further and train our large vision models with synthetic data, delivering an even faster onboarding experience for our CPG customers & partners. 🤝 If you're facing challenges with your IR projects, let’s talk. There's no need to rely on outdated methods. Synthetic IR is here to propel us into the future! 🚀 #syntheticimagerecognition #syntheticdata #computervision #imagerecognition #cpg #retailexecution  

16

1 Comment

Dear Network, ⠀ The landscapes of AI, data science, and the general tech industry never cease to excite me. The latest noteworthy point in this ever-evolving space is tackling one of the key challenges in retrieval-augmented generation systems

1

AI’s $600B Question https://buff.ly/4ezKuW3 by @sequoia big gap between revenue expectations implied by the AI infrastructure build-out, and actual revenue growth in the AI startup & incumbent ecosystem! 🤖 🫧 #rafAIgual #AI #valuation #bubble #vc #investing #startup #ecosystem

1

Amazing how the complexity of AI can be simplified by just the right expert who is in the thick of it, putting on the educator's hat. Kudos Armand Ruiz, thank you for making it easier for some of us to keep up with the AI rollercoaster. I am taking this course. What about you?

5

🌐✨Exciting times in the AI world! French startup Kyutai has unleashed Moshi, an open-source, real-time multimodal language model. Imagine a voice assistant that listens, speaks, and understands emotions—all with an impressive 160ms latency. 🎙️ What is even more mind blowing? Kyutai’s lean team of 8 engineers pulled this off in just 4 months. French engineering brilliance at its finest. But here is the real magic anyone with determination can achieve the seemingly impossible. Whether you are a seasoned developer or just starting out, remember that innovation knows no boundaries. Stay tuned for the code, model, and research they are about to democratize voice AI. Let’s embrace the future! 🤖🌟 #AI #VoiceTech #Innovation

1

Another quick note on evaluations and benchmarks for agentic workflows. As I mentioned in the article I shared yesterday (link in comments), agentic systems are still not very reliable. However, increasingly robust benchmark proposals are emerging in various verticals: 𝑻𝒐𝒐𝒍 𝒖𝒔𝒂𝒈𝒆: Here, WorkBench stands out, which tests an LLM’s ability to use 26 software tools to operate on five simulated workplace databases: email, calendar, web analytics, projects, and customer relationship management. Tools include deleting emails, looking up calendar events, creating graphs, and looking up tasks in a to-do list. 𝑺𝒐𝒄𝒊𝒂𝒍: SocialDial is a benchmark that tests potential social violations which, for example, may depend on cultural context. Anecdotally, in China, reserving places with many occurrences of the number 4 is considered bad luck. Similarly, there is CharacterEval for role performance. 𝑷𝒍𝒂𝒏𝒏𝒊𝒏𝒈: Google and colleagues propose a planning benchmark called Natural Plan that includes thousands of prompts as benchmarks in categories such as travel planning, virtual calendar schedules, and physical location schedules, where the metric to optimize is to make more people fit in a place at one specific time. These tests have unambiguous responses, providing a good set of tests for an agentic workflow. 🔗 WorkBench: https://lnkd.in/dUyynJTx 🔗 Natural Plan: https://lnkd.in/dJNVwkXZ 🔗 In addition to those mentioned, this is a good repository with references to benchmarks for agentic workflows: https://lnkd.in/d5V3JNeB ---- #AI #AgenticSystems #Automation #Innovation #LLM #Agents #multiagents #AI

4

3 Comments

About a year ago, we started building a product for portfolio tracking. The idea came from our customers who wanted to do more with portfolio company data – their most proprietary dataset – but didn’t have this data in a usable format. Being the naive entrepreneurs we are, Anurag Abbott and I thought, "How hard can it be? We’ll knock it out in 2-3 months." News headline, it is HARD. It took us about 14 months in the end; during which we had hundreds of conversations with investors, built prototypes, ran surveys, and got multiple iterations in the hands of our customers. After selling multiple million dollars in ARR, this is what I’ve learnt about portfolio monitoring: 1️⃣ No one has a system they really love right now I can count on one hand, nay one finger, the number of investors I’ve spoken to who were happy with their portfolio tracking processes. I also have data to back my assertion. Only ~14% of funds in our survey were satisfied with their current portfolio tracking processes. 2️⃣ Portfolio tracking is a significant cost for funds A majority of funds are spending 5+ hours on a single PortCo update, and more than $500K per year on portfolio tracking, including personnel and tools. Portfolio tracking involves tedious, labour-intensive processes – having to extract data from multiple formats, clean up inconsistencies and work with nuance around each company / update requires a significant amount of time and effort. 3️⃣ Private company updates are the wild wild west Funds spend months chasing scattered Portfolio updates that they receive as PDFs, as Excel sheets, as slides, as emails – and within these, there’s data in graphs, data in text, graphs pasted as images, graphs that don’t have data labels. There are hundreds of variations in how companies name their metrics. It is impossible to use this data as is – funds must spend hours extracting, cleaning and standardizing it before it can enable any analysis. (As an aside, I have a lot of sympathy for investors now. Our investor updates are a lot more standardized and better.) 4️⃣ You need humans in the loop. We were all aboard the LLM and AI train – but quickly realized that while AI can help with data extraction, human oversight is necessary for bringing higher-order standardization, nuance and real-world context to portfolio tracking. For now. We're sharing some of our internal research on Portfolio Intelligence for public consumption – this includes 125 Funds across North America and Europe. If you are in the business of investing in and advising companies, the report is a must-read. https://lnkd.in/g-376yWC

74

Europe's generative AI market is set to explode, reaching $59 billion by 2033 with a staggering 37% CAGR. This surge is driven by multimodal AI, ethical development, and green tech applications. Will your business be ready to harness this AI revolution? #GenerativeAI #EuropeTech #AIInnovation #SustainableTech #FutureOfBusiness What's your strategy for integrating AI into your operations?

1

Slowly slowly, then all at once...

14

Can we trust machines to judge machines when it comes to LLMs? There's an ongoing battle between manual and automated evaluation of LLMs. Manual evaluation is predominantly used by many companies, especially large ones with sufficient budgets. This approach involves internal QA teams or external consultants who manually create test cases and evaluate the outputs in spreadsheets. This process is time-consuming, but can provide insights into: - model's performance - specific failure modes such as hallucinations - tone discrepancies - biases - copyright violations - privacy issues Smaller companies without such budgets often resort to using expensive engineering time to manually review outputs, which can be inefficient. The primary advantage of manual evaluation is the accuracy and reliability of human judgment. But it's not scalable and introduces human biases, which can affect the consistency of the evaluations. Also manual processes cannot keep up with the rapid development cycles and the volume of outputs generated by modern LLMs. Automated evaluation aims to reduce the reliance on human labor by employing metrics and models to assess LLM performance. Techniques used in automated evaluation include: - Metrics-Based Evaluation: It utilizes established metrics such as BLEU score, ROUGE score, and perplexity. These metrics help assess aspects like fluency, accuracy, and relevance. But these metrics often do not correlate well with human judgments and can be inadequate for comprehensive evaluation. - Heuristic-Based Functions: These are specific functions to test particular capabilities or behaviors of the LLM e.g. toxicity detection or PII (Personally Identifiable Information) detection. These approaches can be tailored to specific use cases but may lack generalizability. - Model-Based Evaluation: It uses another trusted language model to evaluate the outputs of the target LLM. This can automate the evaluation process but requires careful calibration to ensure reliability. Automating this process can be a huge unlock. But developers attempting to create their own automated evaluation systems report very low confidence in the results. Lot of opportunity on this front.

3

1 Comment

This is very interesting. AI innovation is not slowing down anytime soon. Brands need the freedom of choice to use the best tool for the job and I love Watsonx ability to orchestrate and govern any AI you need.

1

1 Comment

🦙 Meta AI Releases Llama 3: 8B & 70B Models! 🚀 Meta has released two new Llama 3 models : 8B parameter model outperforms 70B Llama 2 🤯 70B parameter model is competitive with Gemini Pro 1.5 and Claude 3 Sonnet 💪 (more models are coming soon!) 405B model in the works, nearing GPT-4 level 👀 🔑 Key Takeaways: Trained on 15T tokens, 7x more than Llama 2 8K context window, shorter than some recent models Performs well on reasoning benchmarks like GSM8K Supports commercial use with some license restrictions 🧪 Trying it Out: Available via Ollama, HuggingFace, API services Can be run locally with quantization Responds well to chain-of-thought & ReACT prompts Some multilingual support but dedicated model likely coming 📜 License Notes: - Can't use Llama 3 to train other large models - Must include "Llama 3" in name of fine-tuned versions - Review prohibited use cases before commercial use The Llama 3 models look promising and signal continued rapid progress in open LLMs. Expect to see powerful fine-tuned versions and new capabilities soon! 🦙🔥 #Llama3 #OpenLLMs #LLMProgress

74

2 Comments