Matt Topol’s Post

I highly recommend the book, it covers both the deep technical aspects of the Arrow Format and it's practical application throughout the Arrow ecosystem with examples in not one, not two but THREE (C++, Go, Python) languages! Matt Topol manages all of that with a clear but fun style that is approachable for anyone regardless of technical background. An incredible feat! Disclaimer: I am a technical reviewer off the book.

Just wanted to say great job and congrats to my buddy Matt Topol on his 2nd edition of “in-memory analytics with Apache Arrow”. This is a must read in helping you get the best use out of one of data’s most important open-source projects enabling interop and high performance query processing. Pick up a copy today: https://packt.link/kjbT4 #DataEngineering

Catch Andrew Lamb, Staff Engineer at InfluxData, and chair of the Apache Arrow Program Management Committee at the Open Data Science Conference (ODSC) next month. He'll cover the basics of Apache Arrow and Apache Parquet, how to load data to/from pyarrow arrays, csv and parquet files, and how to use pyarrow to quickly perform analytic operations. https://bit.ly/4aja5zL #InfluxDB #ODSCEast #ApacheArrow #ApacheParquet

🌟 Discover how to use StarRocks as a powerful open data lake and lakehouse query engine in our latest video tutorial. Join StarRocks expert Sida Shen as he guides you through leveraging StarRocks, Apache Iceberg, and MinIO for data analytics. This method allows for high performance and flexibility without requiring all data to persist within StarRocks. 📍Download StarRocks: https://lnkd.in/g8We5PhA 📍Tutorial - Apache Iceberg Lakehouse: https://lnkd.in/gfekduh8 📍Join StarRocks on Slack: https://hubs.la/Q02jrqN-0 #DataAnalytics #DataEngineering #DataLakeAnalytics #DataLake #DataLakeHouse

Yesterday's "modern table formats & Apache Iceberg" 2nd of 5 FREE training sessions' recording is available at https://lnkd.in/e_MvMuM7. The 3rd session, "data pipelines, views & data products", will be live at 11am ET on February 28th. Registration link >> https://lnkd.in/eQ88bVHS #starburst #galaxy #trino #datalake #iceberg #parquet #datanalytics #sql #bigdata

I know many folks have waited for this for a while, over the weekend we released 2 new important capabilities for maintaining Apache Iceberg tables. Glue catalog would now supports automatic remove_orphan_files and expire_snapshots for your Iceberg tables. This update simplifies Iceberg management in your data lake https://lnkd.in/dG4YsJVF

Ingestion into Iceberg is just the tip of the iceberg (pun intended)! 🚢 The real magic happens with ongoing maintenance. Regular maintenance is key to optimizing query planning, improving runtime performance, and managing storage costs—all while ensuring your data retention policies are upheld. Without it, you’ll just be sitting on bloated, inefficient data, losing out on all the performance gains Iceberg promises. So, yeah, ingest away—but don’t forget to maintain, compact, and prune!

#3_pig Advancing Data Processing Proficiency with Apache Pig at #LuminarTechnolab In my ongoing exploration of big data technologies at Luminar Technolab, I recently completed my third task employing Apache Pig. This project focused on effective data processing using the "txn.txt" file, showcasing the adaptability of Apache Pig in handling diverse datasets. The attached PDF offers insights into my experience, detailing how Apache Pig was leveraged to address specific data processing challenges. Under the guidance of my mentor, Sabir K, this task has contributed significantly to my growing expertise in big data technologies. #bigdata #dataprocessing #pig #LuminarTechnolab #learningjourney #connections Guide: Sabir K Luminar Technolab Below are the questions for the task. txn.txt .................. 1. Find Row count 2. jan month oid,cusno,category,product,state A. Row count 3. July Month oid,cusno,category,product,state B. Row count 4. Each category [count desc sort] 5. Category full deatils 6. Each paymethod count 7. jan-july month purchase count [include] 8. Each category total amount 9. Each category maximum amount 10. Each category avg amount 11. total amount by cash and credit card 12. Indoor games ,total amount 13. Each state count

🚀 Excited to share my latest project on Fassos, a famous food chain! 🎉 I've been working on a fascinating dataset containing insights into the food chain environment. Here's a glimpse of what I've done: SQL Analysis: Involved data cleaning and analysis in MySQL Server. Identified the average delivery time. Analyzed the number of orders for each day of the week. Determined the maximum number of rolls delivered in a single order. Explored the relationship between distance and orders. 🔗 You can check out the full project and SQL queries on my GitHub: [https://lnkd.in/gENXxk3W] #DataAnalysis #SQL #FoodChain #DataScience #DataCleaning #DataVisualization #LinkedIn #TechCommunity

Meet Imply's Team at #AWSSummit Berlin on 15 + 16 May 2024 at STATION Berlin! 💻 🇩🇪 Learn more about the managed version of Apache Druid —a time-series data storage and analysis engine with subsecond queries across data at scale. 🐨 🧙🏼♂️ #ApacheKafka #ApacheDruid #eventdrivenarchitecture #streamingdata #realtimeanalytics #AWSIoTCore #cloudengineering Raj Prabagaran, MBAHellmar BeckerTom Mack