Why Data Migrations Go Wrong (3 reasons)
Every year many data teams embark on massive data migration projects.
Typically this involves moving from an outdated stack to a more cloud-based & modern set of tools and processes.
Unfortunately, far too often these exciting projects turn into a huge headache that leave you questioning if it was all worth it.
But it doesn't have to be that way.
So in today's video I want to share a few of the common ways I've seen things go wrong and share some thoughts on ways you can avoid it.
Thank you for watching!
►► The Starter Guide for The Modern Data Stack (Free PDF)
Simplify the “modern” data stack + better understand common tools & components → bit.ly/starter-mds
Timestamps:
0:00 - Intro
0:37 - Moving bad code or practices
1:52 - Overly aggressive timelines
3:10 - Lack of experience or skills
Title & Tags:
Why Data Migrations Go Wrong (3 reasons)
#kahandatasolutions #dataengineering #datamigration
►► The Starter Guide for The Modern Data Stack (Free PDF)→ bit.ly/starter-mds Simplify “modern” architectures + better understand common tools & components
A few other challenges I've seen have been A) Over-designing the target cloud architecture, B) difficulty luring business users away from the existing on-prem platform which they're comfortable with, has more data marts, and is intertwined with the business processes and existing reports, and C) trouble monitoring and managing cloud costs (e.g. obsolete test environments left turned on)
Good points. Cloud migrations are the perfect opportunity to reevaluate your data architecture and fix those pain points that have existed forever and you didn't have time to fix them. Unfortunately management doesn't always see it that way. The last point was kind of funny. We are doing a POC on a new cloud data platform and none of us have ever used it. The sum total of experience on the team with the platform is the KZheads I have watched and the actual DEs on the team know nothing. I was told by my boss yesterday that we aren't going to hold up the POC for training. I think we are fine to kludge something together for a POC, but hoping that attitude doesn't carry forward to the actual implementation.
Hi 👋, this is a general query , not related to this. Suppose someone want to built a data architecture for an insurance company. Below will be my approach : step 1: I will identify all the domains: Customer Domain, Policy Domain, Claims Domain, Billing and Payments Domain, etc . step 2: For each domain i will have a dedicated data model(relational schema). So, 1: One Domain, One Schema : For each business domain, create a dedicated schema in the database. This schema contains all the tables and relationships that are specific to that domain. 2. Relational Model: Often, the relational model is used for these schemas because it is good at representing structured data and relationships, which are common in business applications like insurance systems. Is this is correct way to approach this ? Thanks 😊
If you're building data marts, you may want to think in terms of business processes, rather than domains. Also, some datasets may span multiple domains, so where would you put those?