Data Mesh Roundtable 2023
Date
Time
Location
Third Republic’s most recent roundtable series featured on the 16th of March at Irdetos offices in Hoofddorp, Netherlands. Our event strived to dive into the hot topic around data mesh. Bringing together 10 of the best tech leaders across the Data space, we provided the platform for them to share their approach to Data mesh and any experiences they have encountered thus far. The discussion remained at the strategic level and discovered the following:
Introduction
The concept of data mesh attained significant interest in recent years, as companies seek to optimize their data management strategies. Our goal was to understand a fundamental question:
“Do you see Data Meshes as an alternative technical solution to solving insight requirements, or are they an evolution of the Data Lake?”
To help answer this question, we formed a diverse panel of data practitioners, ranging from “Heads of” to Engineering Managers, spanning multiple industries and backgrounds.
As soon as the conversation started, it became evident that Data Mesh is not a technological solution but an organizational shift. Technology alone cannot solve the problem, and it requires both organizational maturity and a complete change to how we look at Data.
Top down or bottom up?
Can this transformation be driven by the engineering teams themselves? Or does it need higher support?
Some transformations can be driven by the engineering teams, but this is not one of them. During our discussion, it was clear that an organizational change of this magnitude requires support from Top management and cannot be driven Bottom Up, without this support, this transformation will likely fail.
Moving from a centralized model to a distributed one is crucial for the success of the Data Mesh. The interesting part is that this concept is not new in Software Engineering with the advent of micro-services, but it is a relatively new concept within the Data domain.
From the experience shared in the room, starting with a completely decentralized approach (i.e anarchy) can cause more harm than good. Instead, the advice is to start centrally and then distribute as the mesh evolves. Similar to the approach of Monolith to micro-services.
How to govern this transformation?
Similar to democracy, data mesh is a concept that requires good governance.
One key topic raised was that we need to federate governance out, meaning, the ownership needs to be pushed to the teams responsible for the systems. Unlike the days of the data warehouse, teams now need to persist their data and facilitate ways for others to consume it with quality.
The federation of governance is a continuous work and every change to the model needs to be properly explained and documented. While implementing a Data Catalog can help, they are not a silver bullet. They need to evolve with the mesh.
Keep vigilant, not everything is smooth sailing.
With great data comes great responsibility.
Common concepts and definitions will make or break your transformation. It’s a costly endeavor to arrive at these types of agreements, but they are essential to moving forwards. Different departments may have different definitions for similar data objects or KPIs, this is where a centralized team may offer their inputs, ensuring that everyone speaks the same language. As alluded to in the paragraph before, a shared Data Catalog can go a long way.
In the end, the goal of a good Data Mesh is to provide analytics and insights that will be used to train Machine Learning models, and here is where vigilance is extremely important. Compliance with regulation is a must, and something that is not strange to most companies, but legal teams may not be tech-savvy enough to understand what is going on. Some of the participants suggested that an Ethics Committee (with participants from the data domain) could go a long way in ensuring that the models are not misused.
Was data mesh just a fad?
The first time I heard about data mesh was during covid.
There have been rumors that the interest in data mesh has been cooling after a big spike during the covid years, was this just a consequence of people being locked in their homes without anything else to read?
Let’s take a look at the history and the data.
In early 2019, Zhamak Dehghani shared the first ideas she had around Data Mesh in an O’Reilly conference with the title “Beyond the Lake”.
Not soon after, in May 2019, Zhamak Dehghani gave the world the blueprint for “How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh” with her post on her colleague Martin Fowler’s blog, and since then, the interest has been rising.
This leads us to believe that Data Mesh is not just a fad, but something that is here to stay.
In summary,
During this roundtable, we agreed that Data mesh is an organizational and cultural paradigm that requires a distributed and federated organization. It is not a technology solution and involves a philosophical shift in the way we think.
The main challenges are around canonical definitions, regulatory compliance, and governance.
Without top-level sponsorship, your transformation is most likely to fail.
Third Republic will continue to host many other roundtables throughout the year 2023, this will include a variety of different topics and locations. If you are interested in getting involved, please reach out to our Practice Lead Adam Elliott-Smith.