S7 #4 Synthetic data, real governance: What enterprises need to get right

KRISTINA: Really excited today to have with us somebody who is a super awesome resource for a number of reasons. So Ioana Mazare is described her as a strategy leader and executive operating at the intersection of data, advanced analytics and business transformation. I think of her more as the data whisper. So Ioana, welcome to the show. We really appreciate having you here.

IOANA: Thank you, Kristina.

KRISTINA: You've spent years working at the intersection of data operations, large scale, and I'm talking really large scale enterprise environments like UPS. When you look at how organizations today are using data, [00:01:00] especially for emerging technologies like AI and some of the XR space, what are you starting to notice on your trajectory? What's really different today than where we've been in the last few years?

IOANA: Interesting question. We are seeing that collapse of what I call the barrier between capability and access. Right. The fact that we have the convergence of decades of work in various fields, operation research, statistics, engineering, and now scale by data and compute, and organizations can use them in areas that were never capable of using them before. Don't get me wrong, organizations have used AI and advanced analytics for optimization, automation, forecasting. But there was an inflection point, and I really call it a really, good illustration of the convergence between data and technology and analytics with generative [00:02:00] AI late in 2022 when ChatGPT was released first, we saw the huge adoption. But really what stuck out to me is how individuals interacted. A very powerful tool using data and producing data, starting with their personal lives without training. They also saw immediately what AI a powerful technology tool can do and access probably became before full understanding of the scope, the scale, and the power. So that created a tension because organizations in some ways are catching up, trying to scale and formalize something that individuals adopted organically. And so there is a challenge today to align data processes and catch up with technology. And [00:03:00] really make sure that the harness is really data is really harnessed to support it in a way that creates real value. So we are in the midst of a structural change. Data is at the center of it, and it has implications on organizations, but also for the broader economy and society.

KRISTINA: One of the areas that's really, really hard to address is privacy, right? We have personal data, we have protected data and that data at scale becomes a challenge for most enterprises. I'm hearing more and more organizations turning to synthetic data. As a possible solution, but I also am a little bit concerned that everybody's gonna jump on the synthetic data bandwagon. What are some of the challenges that organizations are either struggling with or are going to be struggling with that are jumping on this bandwagon of like, well, I can just use synthetic data and call it a day.

IOANA: So I would like to start with, I mean, I like to use analogies, [00:04:00] so I would like to think of a flight simulator. A flight simulator, it's not the real world, but it's designed to replicate the aspects of reality that matter. So you can train, test scenarios and make decisions without the cost or the risk of operating in the real environment. So when I think about synthetic data, I think that it works in a similar way, right? It's, it's artificially generated. It's supposed to reflect patterns and relationships of real world data. However the benefits come with a cost and the cost need to be really well understood. So the first one is governance. You were talking about privacy. Today, organizations are still building governance frameworks for real data. [00:05:00] Real data is not ready. And as much as synthetic data can replace and can get faster to implementation of models and software and new capabilities, organizations don't have a good governance framework of synthetic information. Right. And there are real drawbacks because synthetic information needs to fit a purpose, right? Organizations have to have real use cases for such for such use.

So matching the shape of the data is not the same as preserving its meaning you talked about privacy and even though synthetic information is sold as and it's presented as privacy protecting in reality privacy, it's not always guaranteed because there are some rare or unique [00:06:00] records that pose risk of re identification. Also bias. It's a huge a huge concern because real data is usually it is sometimes biased. When that happens the risk is to reproduce or amplify biases in synthetically created data. And so we have to have the tools to address the ethical aspect of data creation and the preservation, or the augmentation of fairness that may be missing from real information.

KRISTINA: As you've been walking the halls of enterprises and working with folks, a, I guess you know, what tools are out there to really test for bias and how much weight can we put into such tools and testing?

IOANA: That's a very good point that you're bringing up. And this is where the, fit for [00:07:00] use and the industry profile are critical, right? Are we talking about bias in results from objective events and situations? Or are we talking about selection bias? That is one of the first steps in designing how we collect information. So, a lot of the bias that I'm thinking about, it's more about representation. When we're looking at how to represent the population in a sample, I'm thinking more about let a study that is intended to, look at, let's say the female population in North America and, and their preferences for purchasing a certain product versus a study that it's based on a survey [00:08:00] with a bio sample that it's happening let's say in the state of California in a high, middle income neighborhood. Right? And then you extrapolate that study to the whole population of females in the US. So we're talking about biases that we want to. Be aware of in the real data that we have at hand, versus the intention of producing and generating a synthetic data set that will replace a sample or that will serve a bigger purpose for an organization or an industry. So when you're thinking about biases in data sets, the context is everything.

KRISTINA: It sounds like this is not either or decision, but maybe something far more nuanced, right? It's not just synthetic versus real. [00:09:00] It sounds like fit for purpose. High nuanced and understanding use cases becomes crucial. So, that's, that, that is an interesting point because when we talk about data governance, which you started talking about earlier, most frameworks, like you said today are still built around real data like lineage ownership, the quality controls. And I'm wondering, where does that start to break down? And most importantly, maybe are leaders understanding the relationship between synthetic and real data today? Do they actually understand the difference there and when it's appropriate, or what use cases are fit for purpose between one and the other? Because there has to be a situation where if you have, $10. You decide that you're going to allocate $3 here and $7 there, or conversely $5, $5, but you can't make those business decisions unless you understand the implications of real versus synthetic data. How are those conversations happening in the [00:10:00]enterprise today?

IOANA: I guess the better question is to what extent these conversations are happening today. So on one side, some companies are generating versions of synthetic data to meet their needs today. To train their own data set or, data scientists are trying to test something on a smaller scale and they know how to do it. And in that case, governance around synthetic data should be an extension of existing data and model governance. Right, that it has to expand and to cover how data is generated, validated, and used because it's on used on a smaller case. However, on the other side, we are seeing the rise of a new market, commercial synthetic data providers that are becoming a major pure play synthetic data vendors like mostly ai, hazy, [00:11:00] Gretel and others.

And the recognition that these providers are starting to also being acquired by bigger players like Gretel's purchased by Nvidia for software development. So what I'm saying is, this market for synthetic data, it's going to actually expand and it's going to become a lot bigger. And we want to be able to make the distinction between internally produced synthetic data and third party provided synthetic data sets.

That introduces a layer of complexity because. The questions around governance are starting even before data hits an organization and data. It's not collected or generated by operations, operational systems within an organization, but they are created synthetically by outside organizations and [00:12:00] companies. And so, that, that in itself needs to be understood. It needs to be grasped and like you said, the fit for purpose really becomes a big a big question. Right? So what leaders need to understand is what is the state of my data today, is my real data and the governance. And the strategy around my data enabling me to reach my objectives and let's say implement AI because everyone is trying to figure out how to be a player in the AI market, whether generative AI or otherwise.

And then the next question is, what are the gaps in my data and what can be filled data can what gaps can be filled with synthetic data and is it a matter of maybe a hybrid, an augmentation, [00:13:00] or do I have to now think about generating synthetic data at scale to test to develop before I go into production with my new offerings and new capabilities? So really, I see a growing complexity before things are becoming easier for leaders because it's not only about where I want to be and be clear about my vision, but also around is my foundation ready and what are my options to get there?

KRISTINA: So for folks who are listening, what are some of the signals that you look for that a company is doing well versus just experimenting without guardrails? Because it feels like a little bit of the early days in the wild, wild west. So are there sort of ways that folks can say, look, I think we're doing this well, or maybe they're just starting out in early in their AI journey. What should they be thinking about in terms of those [00:14:00] guardrails if they haven't gotten 'em in place?

IOANA: I think what I would say is that the whole organization is playing a role in, curating data in managing data and using it responsibly. And so that model of ownership the system of accountability and having a clear direction on what the goals are, right, and making sure that the solutions and the tools are fitting the goals, those goals and the objectives.

KRISTINA: Can you give us a sense of what that actually looks like? , Because you just packed a lot into a few sentences, and I'm thinking to myself, if I'm inside of an enterprise, what does that look like? Like what's an example? How do I actually know if we're doing exactly what you just said? Like what are the [00:15:00] keys, are there sort of signals or check marks, or is there an audit I should be doing? Like how do I know that everybody's sort of owning their objective that we're aligned? I know I'm asking sort of a very difficult question here, but I know also that you've been doing governance for a long time, so you know what good looks like. How do you describe to somebody, here's how, if you're sitting instead of an enterprise today, here's how you know what good looks like. Are you doing it right? Are you just starting out, here's some tips for knowing that you're on the right track. Like, how would you describe that to somebody in actual examples?

IOANA: To make it very specific, it starts with the leadership. Leadership has to create and drive the direction of the organization. And the goals need to be clear and the goals need to be able to be told as a story that is telling everyone what the organization is doing and where they want to be in three to five years. [00:16:00] And it's a very clear set of goals whether we are holding the course, we are growing in this market we are improving efficiency. And then that is followed by a very clear set of expectations from the employees and from the entire organization. Because at leadership, usually the mindset happens, right? And the tone for how the culture will reflect into how employees are are going to act. In terms of this is happening with us, this is happening as an enterprise solution and everybody's expected to contribute. It also comes with training programs and with clear clear guardrails and processes for organization for all teams to be involved along the way.

KRISTINA: But what's interesting to me is you've actually [00:17:00] started to summarize what I actually think of as governance. What I heard you say, and this is where you can correct me if I heard you wrong, but what I heard you say is you need to have that sponsorship. You need to have advocacy from leaders. You need to have somebody who is setting the big frame. You need the strategy. If you don't have a strategy, we don't actually know what we're doing.

We could have the best intent, the best people, the best processes, but if we actually don't have a strategy, we don't know if we're going left or right up or down. So we could be going everywhere all at once. But a strategy makes sure that we're all aligned, we're all doing the same thing and working towards the same purpose.

I heard you say that all of that has to trickle down into policies because without policies and actual processes in place, we can't actually execute well. And those, of course lead into standards, right? So when everybody gets really excited about data providence and execution of data, that's sort of the last step that they think about. They think around the standards that are sitting around that data, but that is maybe four or five clicks down. And that is a very important part of governance. But [00:18:00] it's not the whole governance framework. It's so we can't mistake those tactical components for the overarching governance piece that has to be in place. And of course, what's beautiful in that entire orchestration governance framework is when people actually understand what their roles and the responsibilities are. So we can work together as a big orchestra versus everybody playing their own instrument and doing it well, but not actually making the same music or the same sounds that make the music so palatable to the ear.

IOANA: Well, that's about it, I think. I think you were, you said it better in the sense of you have to start with a strategy, with a framework and a real intention that then it's reflected in operations. And the fact that the sign that you see it in operations comes with people knowing where to look for data. For people knowing who's responsible and for people knowing what their roles are and knowing that data [00:19:00] has a lifecycle. And at every point in time there are questions to be asked and everyone is aware should be aware of what questions they, they might ask in order to get to the proper solution and the proper use.

KRISTINA: And so is it the case, and I don't know the answer to this I'm asking you but is it the case that if we have that governance in place, because there are arguably some organizations that have pretty sound data governance in place, are they able to more readily understand when synthetic data is appropriate versus not based on the fact that they have a strategy and they can kind of at least signal.

IOANA: Yes, absolutely. I think first of all, if you have a framework and a common language, there are for sure proper instances where, and collaboration spaces across teams. So by working together, people not only keep each other up to date, but they train each other. And [00:20:00] when these type of questions come up, the discussion about synthetic data is following organically because data scientists and business people and it folks, everyone is at the table and they talk to each other to understand what is it they're trying to solve for and what are the gaps. And so synthetic data is becoming, another solution or another potential pathway to solve a problem that everyone agrees, exists and they want to solve.

KRISTINA: I think what I'm hearing from you, which I really can appreciate is whether or not it's real data or synthetic data, you have to have governance. Even if you're going to have synthetic data, you still have to govern.

IOANA: Absolutely. So it's just a matter of what type of governance do you need and how do you define the proper livers to make sure that you are, [00:21:00] again, curating and managing additional information, right? Even if you didn't create it naturally or based on reality. But, but it poses, new, new questions around who owns the data? Where does it come from, right? Again, is it a third party provider or was it used internally? Can I trust the, the external data, is it fit for purpose? How was it generated? There are many, many ways to generate synthetic data, and those models need to be explainable and they need be to be transparent in the same way. Is anybody using the same synthetic data that was sold to me that I purchased. Does it have a license? Right? I mean, licensing is now becoming the topic of conversation when it [00:22:00] comes to the commercially produced synthetic data because it really is part of what we're gonna call traceability and lineage and observability of synthetic data, right because it's still going to travel, it's still going to be transformed. And those are the governance levers that may look like, but need to be clearly defined for synthetic information.

KRISTINA: So if somebody's listening right now to us talking about synthetic data versus real data and governance, and they feel like their organization doesn't have their governance act together, I know you've been down this road so many times, you wanna, what advice do you have for them? What can they do? They're a director. They may be new into a VP role in their enterprise. What can they do? What should they do at this moment in time in their enterprise to get things going?

Can they do anything? Maybe is it sometimes, a colleague of mine used to say, chop [00:23:00] wood, carry water. Is it chop wood? Carry water time? Where it's like, you know what, just, you just can't. Like that's the answer is you can't.

IOANA: It starts with those who understand, those who believe, and knowing that you are not alone. There is no one director, there's no one leader, and there's no one manager who believes that data needs to be, taken care of. And so first is find your champions and put together a story. The power of the data is, it's in it's in its ability to tell a story and data can also tell its own story in the sense, garbage in, garbage out. That's one way to put it, [00:24:00] whether it's real or synthetic, it's important for folks to understand the fact that without data and without clean data you will not be competitive and you're not going to be at the top of the game for your customers in the future.

And unfortunately it's still true, whether with simple report or with AI that in many instances people still spend too much time on finding, accessing, and curating data, wrangling it to be to make it ready for whatever they need to do and for whatever important decisions they have to make.

KRISTINA: Yeah, that's a really great way, I think to highlight and conclude our conversation, but really highlight that the future of AI isn't just about better models. It's really about how thoughtfully we handle the data behind them and synthetic data. Really is gonna be opening up so many new doors for all of us. You [00:25:00] mentioned that we're just getting started with different types of synthetic data but it's also going to challenge us to rethink governance and ownership and trust.

And also how we as people tell the story in order to get good governance around all of that. So thank you for helping us unpack both the opportunity and the complexity today. I think, like you said, we could probably go on for several more episodes, talking about each aspect of this, but I think what stands out for me in this conversation today is that synthetic data isn't a shortcut, it's a shift that's really coming with its own trade-offs and responsibilities. So really appreciate you highlighting those and looking forward to having the conversation continue.

OUTRO: Thank you for joining the Power of Digital Policy; get access to policy checklists, detailed information on policies, and other helpful resources, head over to the power of digital policy.com. If you get a moment, please leave a review on iTunes to help your digital colleagues find out about the podcast.