Ado Ajanovic

One of the ways I stay up to date with Data Engineering in Snowflake is by subscribing to the Snowflake Developers YouTube channel. I recently watched this video and learned three 🤯 mind blowing 🤯 things. (https://lnkd.in/exVvfixf) 1️⃣ The integration with Git is so tight, you can actually execute SQL that lives in your GitHub account. Example: `EXECUTE IMMEDIATE FROM @configuration_repo/branches/main/scripts/setup.sql;` 2️⃣ Snowflake now has a DAG API, that is similar to the V1 Airflow API. Example included in the video. 3️⃣ The long lived "snowsql" CLI is going to be superseded by a new tool called "snowflake-cli". The new snowflake-cli will enable data engineering tasks for building pipelines. Pretty cool stuff!

35

8 Comments

Will Enterprises be able to create a single data layer in Iceberg? This is a question Benjamin Rogojan (AKA Seattle Data Guy) unpacks in his latest video. My really quick reaction to this question, (before watching the video), was this: If they couldn't do it by now in Snowflake which is simple to use, I don't think they'll have better luck in a high-tech data lake environment. Organizing and partitioning a data lake and putting Iceberg on top is orders of magnitude more complicated than old-fashioned data warehousing in Snowflake. Ben has a much more well thought out and nuanced analysis of this question. Check out his take, here: https://lnkd.in/eF_i_mWV

30

2 Comments

Amazing! The dbt Snowflake Native App is now available on Snowflake Marketplace! This integration brings dbt Cloud’s powerful capabilities directly into the Snowflake UI, allowing seamless data discovery, enhanced observability, and AI-driven insights. Key features include: dbt Explorer for in-depth data cataloging. Ask dbt for natural language queries. Orchestration Observability to monitor job runs effectively. https://lnkd.in/egF2Rs8z #dbtCloud #Snowflake #DataIntegration #DataEngineering

12

I often get questions from folks who are just getting started in data engineering about what they should do for a portfolio project. My answer is always find a dataset and topic you find interesting and will keep your attention. If you need some ideas about what to do with data, here are dozens if not hundreds of ideas from Snowflake on what you can do in their platform:

29

2 Comments

I'm brewing up a blog post which shows the differences (pros cons) between Snowflake Dynamic Tables (created via dbt), Streams (created with a cool dbt package called incr_stream), and good ole dbt Incremental models. The goal is to demonstrate when to use each, and the cost implications of dynamic tables vs streams (if any when run on the same schedule). I'll be using the cost monitoring package by SELECT to show the cost of each. Why am I posting about a blog before I write it?? Because when you write cool code just for fun you want to share it with everyone! The biggest challenge is simulating streaming data without needing to pay for it in the marketplace. This is something that Anders Swanson from dbt Labs has done a lot of thinking about. (Anders is supporting this effort behind the scenes, and the blog post was his idea). He created a python dbt model using `faker` to add new data to a table. I took it one step further and made it a stored procedure and a task, so we now have the groundwork for our blog post. I'm sharing this in the very early stages because: - Maybe an example of a python Stored Procedure can help someone out there. The possibilities abound! - Maybe someone else wanted to play with fake data. (I'll be adding more columns and an ID column and whatnot..., this is in early stages) - It was fun! 🥂 https://lnkd.in/eVyt29wy

43

7 Comments

The Databricks conference is this week! So let's talk about this new change with Alteryx 24.1. The ability to do bulk uploads with Alteryx and UC Volumes is a much welcomed solution to working with Databricks. If you find yourself out at the Databricks conference in San Francisco this week, come by the Alteryx booth and let's have a chat about the work your doing and how you can accelerate that with #databricks and #alteryx!

33

4 Comments

A useful step by step video on setting up Databricks unity catalog on Microsoft Azure. https://lnkd.in/gUbQqqiP

1

🎉 #Snowflake #Iceberg #DataVault 🎉 Join Will Riley at the next Snowflake Data Vault User Group to learn about building a Data Vault using Iceberg tables 🕧 When: May 15 @ 9AM PT/ 12PM ET/ 5PM GMT 🏠 Where: Virtual 💻 Register: https://lnkd.in/eejTbb_s @Snowflake #DataVault #IcebergTables

49

3 Comments

Check out the #Datafam sessions if you’re headed to Dreamforce!

15

1 Comment

With over 8K analysts writing queries daily, valuable insights have been gained. Inefficient queries can have a significant cost impact on organizations. Explore our tips and tricks to enhance query efficiency. #DataAnalysis #EfficiencyTips

8

Vidhi Chugh Here are few additional bottlenecks I could think on top of my mind: 1.      #ChangeManagement and #CulturalResistance 2.      Management’s definition of #LevelOfRole by size of team than complexity and business impact 3.      Lack of AI Talent with #BusinessAcumen 4.      Unclear #WaysOfWorking 5.      Undefined #SuccessMetrics 6.      High #Costs and #Resource Requirements 7.      #Integration with Existing Systems 8.      #Regulatory and #Compliance Challenges 9.      #Scalability Issues 10.  Lack of #ExecutiveSupport and #StrategicAlignment Some of these points might have some overlap with what you mentioned but I thought I still cover. #AIStrategy #ResponsibleAI #AIAdoption

3

🚀 Tackling Snowflake's 16MB JSON Limit with Kipi's Innovative Solution! Facing issues with Snowflake's 16MB variant limit? Our latest blog shows how Kipi's solution chunks large JSON files, integrates them into structured tables, and keeps data intact using Snowpark Python. Simplify your JSON data management today! 🔗 https://lnkd.in/dissHMdi

4

For those that are setting up Databricks with Unity catalog on Amazon Web Services (AWS) , this is a fantastic video which covers step by step to setup AWS roles, policy and identity management to Databricks. https://lnkd.in/g2pwymdj

4

Kevin Petrie Great insights! Balancing legacy systems with new technologies is always a tricky path. I especially appreciate the incremental approach the tiger team took—it's a strategy I've seen yield sustainable results in my experience. Integrating graph functions into the lakehouse to map customer journeys is brilliant; it really highlights the power of connecting diverse data points. Replacing multiple tools with a unified platform must have streamlined their workflow significantly. Have you noticed any challenges teams face when adopting GenAI alongside traditional ML models? It'd be interesting to hear more about overcoming those hurdles!

3

1 Comment

Super interesting to see the launch of Snowflake Arctic for enterprise customers, will be interesting to see how customers benefit in Performance to Cost of models vs likes of DBRX ❄️❄️❄️

3

Announcing new capabilities for the Databricks Visual Studio Code Extension! Users can now easily set up projects with DABs, streamline dev loop with the resource explorer, and boost productivity with tighter Databricks CLI integration. Try out the extension to access these powerful features!

4

It was an absolute blast talking with Eric Dodds and John Wessel on The Data Stack Show! Thanks for having me on! Topics I really enjoy talking about are "the only reason to use data is to control a process" and "every dashboard must do three things". Find out how we can use dashboards to control process (minute 38), here: https://lnkd.in/ekRcvYz5

15

3 Comments

Fixing your code, adding comments to your code, or just asking the AI to create code snippets for you! All of this is possible and more, at **no extra costs** and out of the box within the Databricks Data Intelligence Platform! #databricks #dataintelligence #ai #genai

10

Another quarter of #DatabricksMarketplace Momentum! We’ve added 47 new providers and 115 new listings! This blog highlights HealthVerity: Healthcare Analytics Epsilon: Marketing strategies Nimble: Retail operations Mostly.ai: Data privacy Learn more here: https://lnkd.in/eYbXd3Ga

1