without touching them. So, here we go. So, who has used ECS before? Yeah, a few people. So, let's spend maybe a minute explaining what ECS is. So, ECS in these companion service ECR are a couple of AWS services launched in about a year, a year and a half ago to manage Docker containers on the AWS. Okay. I expect most of you guys are slightly familiar with Docker. Yeah. Yeah. All right. Okay.  That's not no big deal. But when you start running Docker on, you know, five nodes, 10 nodes, 15 nodes, 100 nodes, then problems, you know, start to arise. Complexity starts to arise. And so, we launched those two services, ECS, which is orchestrating containers, running containers on a set of EC2 instances pre-installed with Docker.  Similarly. And ECR is a Docker registry, a private Docker registry for your containers. Okay. And it's pretty cool because you can run, you can store your Docker images close to your AWS infrastructure. And it's highly available. It's secure. You can apply IAM policies, et cetera, et cetera, all the good stuff that we usually provide those services. ECR mandatory to be used? Absolutely. You can use any, any storage you want. It's you can use, you can use anything you want. What I like about ECR is that it's private. It's highly available. And I can apply it on policies, which is not exactly what you get with the Docker hub, the public one. And if you want the private Docker hub, it's slightly more expensive. But yeah, sure, you can, or if you have your own, you can, exactly. It's, you know, it's like code commits, you know, it's a, it doesn't do much more than the public alternative, but you know, it's highly valuable and very secure. Is ECR quite similar to the Docker swarm? Oh, let's not get there right now. Come on, slide one. And someone's going to ask about Kubernetes on slide two, right? Let me warm up a little. Come on. I'm jet lagged to death. You cannot imagine how jet lagged I am. I will get to that. I promise. Okay. It's, I double checked it's available in 11 regions, including Singapore. Okay. We have a lot of partners working with us on Docker, on ECS. So I'm not going to list them all, but you know, keep in mind that these are available. And you know, you're probably using one or the other. Do we have any Rancher fans here? No? No one? Okay. No big deal. I love Rancher. It's very cool. If you want to run core OS, you're for, as you, instead of the Amazon AMI, that's fine too. If you want to do Docker, continuous integration, Docker security, et cetera. So all these companies are, are working with us to make ECS a better and more integrated product. Well, we do have customers too on ECS, right? This is just a selection. I'm not going to go through them all. You probably recognize most of these names. To answer your question, there was a very interesting article by a company called Datadog. I'm sure you know those guys, a monitoring company, excuse me. I could, I could find that one again if you wanted me to. And they published a study on container adoption. And so they compared, because they have all this monitoring data available, they compared different technologies that their customers use. And one interesting data point was the bigger the Docker deployment is, so you know, the more nodes people run, the more containers people run, the more they use ECS. Right. So I will not give you my opinion on what it means, but I think it's an interesting thing, right? So of course I'll give you my opinion.  But when it gets really serious, you know, 15 nodes, 100 nodes, as it turns out, according to that Datadog study, right, ECS is the number one technology. Right. So whatever that means, you know, I think it's an interesting data point. The topic of tonight is scheduling. It's a very specific thing. So let me explain what we're trying to do here. And this is exactly what we're trying to do. Right. We've all played this game. If you guys have children, probably you have this in your living room laying around. And of course we'll know that game. We have to find the right spot for the right shape. Okay. And of course when the grid is three by three, and you have a limited number of shapes, this is a really easy game. Right. So if you had nine containers to put on your one machine, that's a fairly easy problem to solve. Okay. Now imagine that grid is a thousand by a thousand. And, and you have 50 grids. Right. You have 50 nodes, which are capable of holding a thousand containers, or maybe, maybe only a hundred containers. That's a very complicated game. Okay. And of course you will get it, you know, but can you get it in linear time? Probably not. Right. The larger the grids, the more grids you have, the more time it will take you to find the right spot. Okay. And so scheduling time will not be linear to the, to the size of that grid. And that's a problem. Because it means you're, you're not scalable. And I'm not either. Okay. And so the problem that you see as scheduling is traced to solve is no matter how large that grid, no matter how many shapes we have, no matter how many grids we have, we can find the right spot in linear time. Okay. And I have references to articles published by Werner Fogels, the CTO of Amazon. We explained this stuff in more detail there at the end of the presentation. And he has a graph showing that when we scale ECS 2000 nodes and we load that cluster to 90%, scheduling time is still linear. So that's what it looks like. Very quickly. So a number of instances, which are EC2 instances running Docker and running the ECS agent, which is open source. You can't read it, but it's in there. It's on GitHub. You'll get the size obviously. And connected to that agent, we have the ECS backend, which is going to receive orders from you through the API. And it was going to take scheduling decisions, put the right container in the right spot. Okay. And of course you can have load balancers, et cetera, et cetera. Just an example here. So that's the existing architecture. And we have a customer called Coursera. I don't think I have to introduce them. We've been using ECS for a while. And what they do with this is they run, when you run Coursera and you're going through, let's say, Java course or Python course or any kind of language, you have programming assignments, right? And so you write your code and you upload it to Coursera and they run unit testing on it and they tell you how good or how bad a job you did, bad in my case. And so of course, this code is highly untrusted, especially mine. And it's really a danger to their platform. So they run it inside Docker containers to make sure that whatever you upload and we upload, it's not going to take that platform down and you're not going to use their servers for Bitcoin mining or silly stuff like that. And so they also have lots of other containers running there, you know, reporting and statistics, et cetera. But they want the grading jobs to run faster because they're more meaningful to their users. And so they implemented their custom scheduler to do that, okay, to make sure they gave higher priority and more resources to those grading jobs versus their internal processing jobs. And so to do that, they used the AWS CLI and they implemented custom logic, et cetera. So that's option number two here. You can do that. You can call the AWS CLI, get the state, get the list of tasks that are running anywhere, et cetera, et cetera, and decide that this new task should be running here. Okay. It's doable. It's slightly complicated, but it's doable. If you don't want to do that, then you can let ECS handle scheduling, you create a service, you define a task in there, which is the equivalent of the compose file for Docker. You tell ECS how many jobs you want to run. You give CPU and memory hints to the scheduler saying how heavy that container is going to be. And ECS will figure it out. Okay. So you have the, I would say, the easy automated way and the slightly more custom way. But that was before, right? So I reinvent, we announced a slightly more elaborate way of doing things through the placement engine. And the placement engine basically allows developers to specify constraints and strategies, okay, to give you more control on where that container is going to run. So constraints deal with the AMI. You could say, hey, that container needs to run on this specific AMI.  So that wasn't really easy to do before. Sure. You could iterate through, you know, the cluster, the cluster state and find that out for yourself, but it was slightly awkward. So now you can give constraints directly to ECS and say, okay, this is where I want to run my container. So here's an example. So we're listing the all the instances running on a cluster.  Okay. Because this is where I want to run my container. Okay. So you could match on instance family, instance type. You could match on availability zone, say, hey, please run this container in this specific availability zone or run it anywhere, but in this availability zone, right? And you could use all those constraints to make scheduling decisions. Of course, you can combine them. You could say, give me the T2 small and T2 mediums, which are not running or all the J2s that are not running in US East, I don't want to, you know, know why you'd want to do that. But okay, it's possible. So you can filter your instances on all those constraints and use that list of instances to run your containers later. Okay. It's very easy to do. Then the second thing we added is placement strategies. So strategies are deciding how you're going to spread your containers across the set of instances that you've selected. Okay. So you could impact them to just cram as many containers on the smallest number of instances, right? So it could be on memory or it could be on CPU. Obviously, this is a cost-saving strategy, right, to minimize the size of your ECS cluster. Okay. But from a high availability point of view, it's not so good. Okay. But if you just want to have the fullest cluster possible with the smallest number of instances, that's a good strategy. Spreading is quite the contrary, right? You could say, well, you know, spread across AZs. That's good for high availability. You could do affinity. You could say, well, I want container A and container B to be close to one another, right?  Or you could have distinct instance where you only want one container of any given type to run on one instance. Okay. Again, you could probably do that before, but, you know, at the expense of way too many CLI calls and headaches and custom code. You can combine strategies. So you could say, please spread across AZs and bin pack. Okay. Why not? But at the end of the day, what you want to do is place those containers, right? So you can use constraints. You could use strategies and then based on this, you're going to run your task. And so that's how it goes. First, you take into account constraints. Okay. So how much CPU is left on my cluster? How much memory is left on my cluster? What about network ports, right? There is only one port 80 per instance, unfortunately. So, you know, if you have five instances and you want to run 60 containers on port 80, well, that's not gonna work pretty well. Okay. Okay. So the custom constraints, which are mentioned, I just mentioned, AMI instance type, et cetera, placement strategies. And then you apply all those filters and you have the list of candidate instances where you can run your container. Okay. So let's look at a few examples. So I'll just show you a few. There are more in the slides, but I don't want to keep you for hours and hours. So here I'm running a task on the cluster called ECS demo. I've got a task definition. I want to run nine tasks. My placement strategy is spread. And I'm spreading based on availability zones. Okay. And I've got nine containers.  Know what's going to happen. I could never do that with PowerPoint. So that's why I'm reusing some slides because, you know, the clever stuff I cannot that's why we have program managers. Okay. So I've got nine tasks, which are spread, of course, three AZs. So that means three tasks per AZ. Okay. And in this case, they've been spread across multiple instances. So slightly different here. I'm spreading across AZs, but I'm packing them for memory. Figure it out. Okay. So first spread them and then pack them. Right. So it's a mix, you know, it's a mix of high availability and and costs. Okay. Sure. So you have 12 instances there. Yep. In this case, shouldn't the scale like kill the two extra since they're not used? That's a different discussion. But you're right. I'm just surprised to see unused. So your question is more on how do we scale in and scale out the cluster? Okay. Here I could have a fixed size cluster with 12 nodes and still do that. Right. Because maybe I'm keeping the G twos for GPU jobs. But you're right. The better way would be, in fact, to say, well, as you say, you know, I've got nine instances that are clearly useless here. I'm paying for nothing. Right. So I should be able to detect that my cluster is over provisions. And I would scale in and remove instances. Okay. And you can clearly do that with auto scaling. Okay. The traditional way you would say, well, CPU, that's like out of VCS. It's, yeah, it's easy to stuff. You use the easy to scaling to scale your, your hosts. And then the instance will then be scaled. I will spread to be more correct. Exactly. So it's, it's really, it's two different discussions. Okay. How do I, how do I allocate enough in situ instances to my cluster? So it could be fixed size, or it could be auto scaled using the easy to auto scaling that we will know. And then based on that, where do I put my containers? Okay. So you can do that with ECS services as well. So here's another example where that first service is going to be, so the yellow one is going to be bin packed on memory. And the purple one is going to be spread on Aces. Right. So bin pack means just cram, cram them. And that other one, you know, is more, let's say, maybe more critical for your platform and you want to spread it. Okay. So you have that flexibility. You can go really crazy with this. Here's another one, distinct instances. So the yellow one needs to run on G2, probably it's the GPU container, but they need to be on separate instances. And the purple one is T2 small or T2 medium on distinct instances. But hey, we have a problem here, right? Not all of them have been placed because I only have two G2s and I have a third task to run, but, you know, there's a constraint there and same for the purple ones. So the only option here is to add instances. That's where your ECS doesn't do this game. So you have ECS metrics for cluster-wide metrics and service-wide metrics that go through CloudWatch, RRMs, blah, blah, blah, the usual way. And at the end of the day, you're adding more nodes to the cluster. Okay. But do you have ECS metrics as well? You have visibility? Yeah. So ECS is selling metrics for CloudWatch and you have your auto scanning. Exactly. The EC2, the usual way. And so if we want to add some more stuff here, same story, you know, blah, blah, blah. Okay. So be careful with distinct instances because that's exactly what it means, right? It's one per node. So it could be costly. Okay. So that's what we just saw there, you know, constraints and placement strategies that just give you, you know, you could do that before, but you would end up writing a lot of code and no one wants to do that, right? We're lazy. So now you have an easier way to do it. Okay. But let's go one step further. Yes. Yeah. The affinity will be based on like a tag, like a group name. So you could have two, three, four. Yeah. More than two. So now let's go a little further. So we added another brick, which is called the event stream. And the event stream is exactly what it means is whenever, whenever something happens on the cluster, that's going to trigger an event. So, you know, container starting a container, stopping, dying, blah, blah, blah, anything happening. Okay. In real time, you want to get those events, send them to CloudWatch and obviously do something about it, right? If you're one of your instance dies and it takes a whole lot of containers with it, you know, you want to do something about that. Okay. Quickly. And so we can send those events to different services, but specifically what I want to show you tonight is something even weirder. Now we get into weird territory, which is blocks. Who has heard of blocks? All right. No one except you and Kai, I suppose. Okay. So we are in weird territory. Okay. So what is blocks? So blocks is not an AWS service. Okay. Blocks is, it's an open source project that AWS launched to, you know, to create a community, a development community around ECS and ECS projects. Okay. So I'll go back to that. But for now, just keeping in mind that what we're trying to do, we're going to, we're trying to grab all those events happening on ECS clusters. We want to send them through CloudWatch events to a queue, right? Right there. And we'll have blocks here, reading those events and taking action. Okay. Specifically, we have two blocks. We have what we call the clusters, the cluster state service, which is basically a key value store, which knows exactly what the cluster looks like. Okay. What are the containers running where, et cetera, et cetera. And we'll use, we can write a scheduler based on the events that are coming through CloudWatch and based on the current state. And by combining both, we can implement pretty complex scheduling strategies. Okay. With this architecture. So now this is what it looks like, right? So you could be, well, you could be that guy here, right? He doesn't want to know anything about this. Okay. And just do it the previous way. Okay. Just make sure I have three copies of container A and I have five copies of container B that I don't care where they run. Just make it happen. And the load balancer just will find them. And that's a perfectly sane way of doing things. But some customers like Coursera and others, they need more control and they need a lot of control on what's happening. And they will be the guys on the right-hand side going through writing their own scheduler and going through something like blogs or their own code to implement complex scheduling policies. Okay. So you have the easy way, you know, not worry about it and the powerful way, which, you know, you need to work on. So while I explain this, let me start my demo because of course there's a demo. This is the most dangerous of demos to do with jet lag, but let's do it. Yeah. It's probably still too small. Can you read okay in the back or that's okay? All right. Just yell if you can. Okay. So, so we're going to do this in Singapore, of course. So I need to create two clusters. I will explain why in a second. So first, I need to create a cluster, an application cluster where I'm going to run engine inks on that one. And I'm going to launch, or actually I can do it in the console here. I'm going to launch the CloudFormation template to create all this, all the block stuff. Don't worry. I will explain this in a minute. Is that how I call it? Yeah. Blocks cluster, blah, blah, blah, blah, blah, blah. Next. Next. Yes. Okay. Go. It's all live. No pre-cooked demo. I live dangerously. Maybe I'll regret it in five minutes. Okay. It's all for you guys. Okay. So, so how this works is, like I said before, I have a cluster on the right side that I call the application cluster, the one that really runs my containers. And it's going to live its life and stuff is going to run, stuff is going to crash and, you know, the software life. So whenever something happens, events are going to go through CloudWatch events to an SQS queue and to blocks. Right. And the scheduler is going to receive those events and do something. Okay. Right. So, and this is running on an ECS cluster as well. Okay. So the CloudFormation template that I launched creates all of this. Okay. Creates an ECS cluster with the scheduler and that and the SQS queue and the CloudWatch rule, et cetera. And the other cluster, the application cluster is this one. Okay. So basically this is what I just did, run my template to create blocks, which creates an event rule, the SQS queue, the scheduler, the state service. It's also using an HCD for a key value store. There's an API, which I can invoke to get to the scheduler. And this is the other one. Okay. And it should be ready in a minute. And once this is all running, right, I'm going to use the scheduler API using a tool that is included in the blocks code to basically start managing the application cluster. Okay. So let's see if this is done now or should I wait a little bit? Okay. Obviously I should wait a little bit. So let's just wait a second. So I can see my blocks cluster here. I could see my, the application cluster that I called web cluster here. Okay. That one is ready and the blocks one is almost ready. Okay. Okay. Let's wait for a minute. Do you guys have question while we're waiting for this? Yeah. Is there any plan in the future to have blocks like this service? It's a, it's a good question. It's a good question. I'm sure there was this discussion at some point. I was, I, I should mean I was extremely surprised. I was at re-invent and I was extremely surprised to see this announcement. You know, I thought we would keep investing in ECS and, you know, more, I would say more on a managed service level. But I'm not saying we're not going to do that. I'm, I'm, I'm quite sure we will keep seeing features in ECS completely managed, completely, you know, on the shelf, off the shelf, et cetera. But we also have this open source project. And I think it's nice. It's an interesting way of doing things because I guess the open source project can move much faster. All right. That when it's a managed service, it's like just updating the managed service. Maybe, maybe that's, yeah. You have to run the, the cluster to manage your stuff that manages the other cluster. Yeah. Manage the cluster that manages it. Yeah. No, the blocks cluster is super simple.  Can take away the EC2. Yeah. That happens. Sure. So, so my point is, I think it's interesting to have both the managed service, which is, you know, stable production grade. And that, that keeps evolving nicely, you know, not too fast because you can't break the production of, you know, all our, all of the customers. And to have an open source project where people who really need to have the bleeding gauge stuff can experiment and contribute. And maybe, you know, it's going to feed back into the, it's going to feed back into the, we're almost there. It's going to feed back into the, the, the managed service. I think it's interesting to have both. Right. And I'm not an expert on, on Kubernetes, but I have the feeling this is pretty much what is happening there too, where you have the Kubernetes, you know, the actual open source project that goes very fast. And the, the managed version by Google, which is probably, you know, more conservative. So I think it's, it's pretty interesting. So, yeah, the EC2 instance is, come on, we should be ready now. More? Yeah, yeah, yeah, yeah, it's getting there. Okay. Do you try to refresh the cache? Yeah, maybe just to console playing tricks on me. Okay. Well, we'll just pretend it's done and start launching stuff. Oh no, it's the API gateway that's slowing me down. All right. Come on, API gateway. Okay. One more question. We have time for one question on anything. Come on, ask questions. Yes. Anyone knows why we have Canadian pizza in Singapore? No, that's a good question. I don't have the answer to that. All right. So the cluster is ready now. So, maybe I can just see that. So the first thing I want to do here, the first thing I want to do is to get rid of the silly branch. What's it called Dave? All right. Okay. So I've got a few tasks definitions ready. It doesn't really matter what I'm going to run on that application cluster. It's going to be engine eggs. We don't care so much about that. It's just an excuse. Okay. So first, what I need to do, okay, let me show you that thing once again, just to make sure it's perfectly clear. Okay. So this is the blocks cluster. This is the application cluster. And right now, nothing is running in there. Okay. If I show you this one right now, it's the other one. It's web cluster. It has three instances. And it's totally empty. Okay. No tasks running there. So what I'm going to do now, I'm going to call this thing here, which, which is hopefully, yeah, the right one. And so what I'm doing here is I'm invoking the scheduler API on the blocks cluster telling it, okay, please create an environment which I, which I call the web environment. This environment is going to run on the cluster called web cluster, which is my application cluster. This environment includes this task definition. So my engineering is the sweet definition. Okay. And that's okay. So basically, I'm telling my blocks scheduler, hey, you can manage this environment based on this task definition on the cluster called web cluster. So this puts web cluster under blocks management. Okay. Right. But this is just declaring it. Okay. Nothing is running. What is going to make it run. And don't worry, you've got all this stuff in the slides, but it's so much more fun to show you demos. What's going to actually do the trick is this, okay, create a deployment, which means, okay, now do your thing. Yeah. Now do your thing. Okay. So now you've got this application, this web cluster under management. Do your thing. But hey, what's the scheduling policy here? Right. That's the question you should have asked. There's a deployment, there's a scheduler, right, in that blocks cluster. What's the policy? Does it been packed? Does it, what does it do? So this guy here, including this demo is called a demon scheduler. And basically it means it will run one container on each one and only one container on each node, like a, like a demon, make sure that container is running everywhere. Okay. And now if I look at my container, once again, I see three tasks and I see that indeed each task is running on a separate instance. Okay. So what about, what about adding nodes? Okay. So I'm adding more nodes to that cluster. And if the scheduler works, okay, try to understand what's happening there. More nodes are going to join the cluster. That's obviously going through the event stream into the scheduler. Let's see. Hey, more nodes. Good. I'm the demon scheduler. I need this engineering's guide to run on every single node. So I'm going to start three more containers, each running on a separate instance. Okay. So you're adding EC2 instances. Sorry? You're adding EC2 instances. Yeah. Hopefully. Just give them a minute to start. Yeah. That should work. Just need to wait for them to join. Okay. We can check that out. Okay. So here, all right, maybe I should show you just the project for a second. Okay. So there's a, there's a GitHub page for this, which is here. Okay. And you will find, so that's the demo CLI tool that I used to send scheduling commands. And that's the cluster state service sources here. And that's the demon scheduler here, right? Okay. So it's all open source. You can, you can check it out. You can send your pull requests. You can create issues. You can yell at the maintainers. You know, it's all there. Uh, no, no, uh, no, there's no, there's, there's nothing like that.  You can reuse your, your Docker compose files. We support that. But I would say that's it, right?  So I've got my six instances here. And of course I have my six tasks running. Okay. Right. So this is a super basic strategy, you know, the demon scheduler, but again, it's all open source. You can go and, and, and learn from that and build something more complex. Okay. And, and, and sure I could have done this using, uh, using a service and using the distinct placement strategy. Okay. So this is really, you know, this is really heavy, heavy stuff to get to a fairly simple result, but, you know, uh, it's a, it's a demo. It's a simple demo. And, and again, it's open source and you can run from that. Okay.  All right. So we saw that stuff. All the comments are in there. If you want to replay that demo, you can do it. Nothing hidden. So these are the articles, the Verner articles I mentioned, pretty good read. That's the blocks URL. And there's a page there with all the ECS demos, uh, DCS, sorry, videos and demos from re-invent. So check, check them out. There are lots of good stuff there. Okay. Well, I'm done. So thank you very much. Uh, if you have questions, you know, I'm still there. Thank you. And, uh, you know, really cool to be in Singapore. Thanks. And of course, I need to put you guys on Twitter.


My name is Shannon Kemp, and I'm the Executive Editor of DataVersity. We would like to thank you for joining January's installment of the Monthly DataVersity Webinar Series, Enterprise Data World. This Webinar Series is designed to give our Enterprise Data World conference attendees an education year-round, a conference we produce in partnership with DAMA International. Enterprise Data World will be held this year in Austin, Texas, April 27th through May 4th, 2014. And today's Webinar is a preview of one of the talks you can experience at the event designing master data services for application integration with David Lotion. There's a couple of points to get us started. Due to the large number of people that attend these sessions, you will be muted during the Webinar. For questions, we will be collecting them via the Q&A in the bottom right-hand corner of your screen. Or if you like to tweet, we encourage you to share highlights or questions via Twitter using hashtag EDW14. We will send a follow-up email within two business days containing links to the slides, the recording of this session, and any additional information requested throughout the Webinar. Now, let me introduce you to our speaker today, David Lotion. David is the President of Knowledge Integrity, Incorporated Knowledge Integrity, Knowledge, excuse me, Knowledge-Integrity.com, a consulting company focused on information management solutions. David is among Knowledge Integrity's recognized experts in information management, contributing to Intelligence Enterprise DM review, and is a channel expert for the Business Intelligence Network. David's book Practitioner's Guide to Data Quality Improvement and Master Data Management and his book Business Intelligence, The Savvy Manager's Guide, has been hailed as a resource to gain an understanding of business intelligence, business management, disciplines, data warehousing, and how all of the pieces work together. David has created courses for TDWI, Data Diversity, and a number of venues and is often asked to provide thought leadership to the information management community. I'm very lucky to have him here with us today, and with that, I will give the floor to David. Hello and welcome. Thank you, Shannon. Thank you for the opportunity. Hopefully, you are hearing me okay. I know we didn't do a complete session earlier, so I'm hoping that this is coming through. I do want to thank you for the opportunity to talk. This is maybe a little bit of background for those who are attending. I heard you contact me about a couple of times. I said, hey, can you do a webinar for us? And I said, sure. Can you do a webinar for us in February? And I said, sure. So what I did is I kind of looked at a combination of two ideas. One was something I've been harping on for a number of years, and conveniently dovetails with some clients over the last couple of months as well. So it's kind of a propitious interaction of ideas, which is going back, cycling back to this whole concept of the question for master data management and why that is a challenge. And what we can do that we can explore to start thinking about how we can facilitate the integration of a master repository and parcel of application infrastructure. So the first thing is to go back to the conventional approach to master data management, which is largely focused on consolidation of data into a single repository. And if we recall the typical phrases that are used or business terms that are used to add on behalf of the need for master data management, it typically enters on things like a 360-degree view of the customer or a single source of truth or a golden copy. In fact, each one of those acts of the buyer to take data from many, many different sources and put it in a centralized repository and absorb it at all and boil it all down into a single representation of whatever entities you're talking about. I think for convenience, I'll talk about customer. For example, we have a bunch of different functions within a business, finance, sales, marketing, legal, fulfillment, HR, et cetera, customer services, customer support, and each one of them has some subsystem that contains information about customers and the designer is, let's pull all the data out of those subsystems and put it into some synchronized repository as a master repository. And focus has always been how are we going to pull our data out and dump it into this repository. But one of the things is that as a byproduct, we end up focusing on maybe the more complex issues, which is not how are we going to put the data into the environment, but rather how are we going to get the data out of it and how are we going to use that data and what are the characteristics and criteria for ensuring that that data is for a long time now, that consolidation may have been the original motivation for master data management, but it didn't drive this whole concept of master data management integration. Why is that? And there's a bunch of reasons and in this slide, I've enumerated a number of them, which is data consolidation, and I'll start from the left-hand bottom YADS, which stands for Yet Another Data Silo. If we're looking at consolidation as the objective of master data management, what we do is we're pulling data out of different sources and completing it in a way that effectively boils out all the distinction that came from those original sources, and so we actually eliminate some details that were relevant to the original sources. And yet another copy of data that the test is not synchronized, that requires continuous input, that requires maintenance and configuration and management and continued natural and management input in order to manage it as its own entity, and that right creates not just a project to build that thing, but it becomes a programmatic aspect that requires continuous input resources. The second is missing input, but not without going to the covers of what you're building, you lose a little bit of the context as to what are the expectations for what you're doing in that master data repository. And yet another copy of data has passed to a large extent that master data has been pushed or motivated from the technology standpoint and less so from the business standpoint. I think that's changing a little bit these days, but it used to be with the IT department that said we need to have a master data repository, and a lot of that was in fact driven by vendors saying, you're already doing a lot of these things. You need to have a separate tool that does master data management. You want to buy a tool and the deal is done and then all of a sudden you've got the creation of a system without necessarily understanding what the business driver and requirements are for building that system. The creation plan. And this has been one of the biggest bugaboos and one that we're really going to focus on today. Now that we've got that thing, now what do we do with it and how do we use it and how do we integrate it into either application that has yet been built or a location that are currently in production and are using their own data stores and all of a sudden you raise the cost of your own data store and migrate over to the use of this master, this customer master repository or this vendor master repository and what are the approaches for doing that and I think that has been one of the big building blocks or roadblocks to success and that's what we are going to focus on in the rest of the presentation. Number four, it's important to understand loss of knowledge. If I'm pulling data from multiple sources and I'm making some decision as to which of the attributes and which of those copies or which of those source records remains and which of those attributes are predestined because they don't correspond to the gold copy of the master, you know, the master truth, well I'm potentially losing some information and the truth is in some cases such as for analysis in a customer's name or a vendor's name original system and then become cleansed out because they're linked together based on an identity resolution, the fact that there's actually some variation that may have been deliberately introduced as a way of trying to game the system. So there's also that question as to whether you're losing information by consolidating it and the same thing with loss of meaning, which is if the sources had some course characteristics associated with the data attributes there, there's a definition of what a customer is in the sales department and there's a definition of what a customer is in customer support and when those customer data sets emerge together and boil out the differences in that definition, the truth is you may not have that because in sales a customer is the person who signs a check that pays for the product, that's a different definition from the definition of customer from customer support is the name of all the people who have been licensed to use the product than those who are allowed to call in for support. Now all of a sudden you've got data that are included with the original sources because you may have all the combinations of customers to be the same type of customer when in fact there's the really two different types of entities that interact with you within the environment. Or some misalignment where you don't have different business functions understanding how they should or should not be sharing information, process governance so that you've got to choose with consistency in how business rules are being applied when you've got data coming in from a long-distance process pathway that in fact you can speculate as a quote-and-quote thought leader and say that this happens and without really being able to back it up but I can actually say that in the last couple of weeks I've been looking at different aspects and processes associated with an operational system in which depending on the portal that you came through to create a new customer record even or not different attributes are collected and different characteristics are set and there's a whole different set of business rules that are being applied for the purpose of an entity resolution and you end up creating duplicates and sometimes not. So I could speculate. I could say that I'm speculating but I've actually seen it in practice now. There are other aspects which I don't really feel that it's worth at this point because I do want to talk about this whole concept of coming up with a migration strategy for sharing master data so as we're bringing this registry and a hub or an index online it doesn't be the creation of the consolidated view that then becomes its own data silo but rather what's the plan for migrating the functional applications of the business activities to employing that master data whether it's the master index or whether it's a master data records or master repository records or profiles or whatever you want to call them how can they be benefiting the finance and the sales and the marketing and the service and the HR departments and all the different components of your business. So the question is how can I build that migration strategy in a way that accommodates all of the different functions and what we would be doing is saying well what are the typical master data use cases and how are we using master data or how would we use our data if we actually had this master data or repository and index and the capability for doing identity resolution largely in the approximate or probabilistic methods to be able to pull together or define duplicates or define variances in that entity or data. And so the question is looking at your application and saying where are we actually looking at at information associated with those entities that would be a master entity. So again I can rely on the use of customer as the canonical example. The operational and analytical activities that I might do while there's assignment of a unique identifier and again it's a relatively simple and straightforward task and however in the presence of variation and multiple variables through which individuals can interact as customers, you might have that clear mechanism for uniquely identifying each individual and understanding when that individual has already been seen. You're usually going to create multiple records for that individual and will be generating multiple identifiers to become if anybody who's had some experience working with trying to assign an identifier or a unique identifier or two.   This customer has bought such and such a product or whether it's an alternative relationship that might be a hierarchical relationship. This individual lives in a particular household or this customer is a registered party to a subscriber to a particular type of service. Cross-references of identifiers.  You want to be able to manage the cross-references between the acquired company's customer IDs and the retail customer ID until you've been able to facilitate through interacting with the customers the transition from the prior customer ID to that unified ID. Second set of things are satisfying the data management and the governance policy side of the service. I'm managing my environment. I want to be able to look at of the customers that are calling in how many of them have called in multiple times or how many of them have certain types of customer plans. I want to get the list of the individuals who've contacted me through the different to match them against my master repository. Another one is doing duplicate analysis and elimination. Same example we used before where one company has acquired another company and they want to be able to see whether there are any customers that are within both of the company's customer databases so that you can merge with their customer files. And then the next is for standardization and cleansing. I know I've got a master repository that has the customer's name, that is their official name and birth date and location of birth and height and weight and all those different core characteristics that if I'm getting data that comes in from another source and requires standardization. Maybe they've got a nickname and they want to match a name on the page for birth date or birth date then you can do standardization and then use that data to do cleansing and then use that in operation. So this is where we can get a little more complex which is looking at what are the characteristic approaches for accessing data and for master data and how is that master data used as part of the business function. So there are some categories in this repository with other resources. So an example would be customer master but I also have my daily transactions and I want to see what percentage of my customer days transacted with a company over each of the last five days. And then feeding analytics. So I might want to see what kind of customer data clustering and classification implications or algorithms and see whether I can come up with some predict analytics based on the data that's in my customer or my master customer. So these are examples of typical use cases for master data within a particular business function. Those are some of the things that led me to consider one of the challenges for integrating and finesse some of those challenges and I've always said one of the biggest issues has been transition away from using the entire copy of the data and being able to use the master data and I thought well, how do we do this as a set of services? And I started doubling together what you might call master data services stack where we start looking at what are the data use cases and how are those related to different levels of capability that either are provided with the underlying database or whether it's provided as part of a master management project or whether it's at some kind of API or interface that we need to design to enable an application to make use of that master data. Basically five layers here. There's the array layer which is the data access, I'm sorry, which is the repositories of the master data itself. The orange layer is data access. The pink is, I'm sorry, the purple. Excuse me. The purple is the master identity resolution typically provided with the resolution. The blue are the core services, you might say, data management and related management core services provided for many of these. Then the green here is the interface that is exposed upwards to the business function. We're going to walk through each one of these in a little more detail. The master entity repository, so it might be at the back of the repository, so it might be your customer master, which will have private file data, all the information about that customer. Then the role of the apps among those customers might be households, it might be people who work for the individuals who are somehow related in a familial relationship. Whatever the relationships are and whatever those entities are, they could also be multiple, multi-domain M. It could be customer data and product data and vendor data. There's relationships between which vendors provide the products and maybe relationships between which customers have purchased which products and essentially it's the entity index and a unique identity based on an entity identifier. So have the sign created or let's say created a new entity record or new customer record, then that customer's identifying information is confused into some type of key mechanism for that index so that the next time information for that customer comes in, it can be mapped into the index and then find the unique identifier and then use that unique identifier to access that data directly from the company. So typical data interactions that support that enterprise use of metadata would be the digestion of data. So I get a new set of customers and I want to put them into my customer database. It would be profiling, being able to analyze the data that's coming in to be able to look for where the variables are. It would include probabilistic matching so that I could create for the match or duplicates or overlaps or intersections between different data sets and master data services for management of the repository. So what do you need to do on a data basis to make the integrity of that customer or the processor? How frequently are you removing the data elements or the records that are in there? How frequently are you running the duplicate analysis internally? Are you ever recognizing that there are two individuals that might have been composed into a single record that needs to be broken? So those kinds of index layers and the next layer is the data access which enables the other areas on top to get access to the master data. What do you think about this? Again, in the context of do we want to be creating multiple sizes or multiple replicas of the same data which is an alternative to create a master data environment and have a separate repository or alternative would be a separate master data environment as well so that instead of accelerating creating new master entity records, what we want to do is make use of the entity index and link it to the data through a federated way to its original source. You're not actually copying the data but you're providing access to the data from its original source. So we can either create a directory to the physical data repository that's a copy of data or we can add access to a conceptual master repository that is composed from across all the sources. But we want to have access that would be typical of the type of an application we do which generally are the databases or create accesses to data. So it's a select that would be able to pull a subset of data out of that repository or to run joins. Show me all the customers who bought more than 16 widgets in the last 30 days and multiple multi-way joins that go across multiple systems. So it's not just necessarily pulling the data out of the master repository but rather joining the data that's in a master repository to just sitting in a transaction system or in a warehouse or some kind of system. That's where the virtualization of federation sometimes comes in handy because you may want to abstract out or make that layer somewhat opaque so that you don't need to be concerned about where the data is actually sitting and that would force you to have to tinker with all sorts of precedent to make sure that you could get access to the two in the right way without jumping through too many hoops. So we need to have access layer in this services architecture. Okay, this is what I would call core services and that includes matching an identity resolution and the algorithms for probabilistic matching but this includes my call data management and identity layer services and we've got two slides on how many bubble. So we start with the matching identity resolution. That's the capabilities that may have originally been the core of your vendor's master data management. That was then linked to the management of the repository and then on top of that which are the capabilities to get access. So the data and identity search. So I give you a customer's candidate record. Here's a name, here's an address, here's a phone number, here's an email address. That would then invoke the search identifying attributes looking through the matching or identity resolution layer and I'm going to jump back here which would then go through the data access repository to the index to find whether there was that customer actually existed within your environment. It should return to you an enterprise or a customer identifier, a unique identifier which you could then use to retrieve the customer's data directly from the master repository. Likewise, you might want to update a customer's record, you might want to create a new record or even what I call deactivating. I mean we typically wouldn't delete it but rather we'd have some kind of duration associated with the actual plays. A customer or whether a person was a dependent in the household until 1997 and then after that that person became a head of her own household. So there may be relays that exist in the past we want to maintain history so we may be activating or deactivating over time. Then there's identity services, it's a middle bubble on the pro so it's the ability to generate an enterprise identifier or a unique identifier and then the cross-referencing capability so that if I've got excellent developers that are coming in to be able to manage that that nothing between an external identifier and my internal unique identifier. So a customer comes in with a customer ID from the bank account that he had that was purchased by a bank that was purchased by a bank that was purchased by a bank that was purchased by the current bank and that may sound funny but I actually had an account of me at a bank that was acquired and the acquiring bank was then acquired and so on and so on. So the account that I had was a legacy account that went back numerous years until I moved out of town and closed that account that you may maintain multiple layers of cross-referencing between an original identifier and the unique identifier that's used for the cross-referencing of both the internal and the external client identifiers. We want a relationship management to give an identity or a customer, say tell me all the relationships that are created with this customer. Tell me all the households that individual has lived in. Tell me all the relationships that that person has lived in. Tell me all the products that that person has purchased. So finally the relationships and then the management capabilities for management was essentially to establish a relationship or to break it into for the establishing relationship and making a relationship or establishing a relationship between those two entities and associating what the nature is of that relationship so an individual can be associated with the associated, can be married, an individual could purchase a product an individual could live in a particular location. So a generic capability for relating two entities together as long as you can meet the nature of that relationship and then you want to be able to break or deactivate a relationship. So a married couple, they were having trouble and they just got divorced. So now we want to break the relationship. Now again, I use the term deactivate because we don't want to forget that those two individuals were related. We want to make sure that we keep track but rather that it was during a particular time period for you all, let's say, a household license to use all the computers in your house. When a couple gets divorced, does that license change or does it survive that divorce if they happen to move out? Now they have two computers that were on the original licensing agreement but now the computers are located in two different locations. So some of the questions that you'd want to be able to verify, for example, some of the issues that existed at some point. And then there's a kind of governance and that's a real two governance activities reaching two records when they can determine that they represent the same entity. So I've got a record for David Lotion. I've got a record for Howard David Lotion. Now I've determined that those two people are one in the same. You want to take that data and link it together to demonstrate that, in fact, refers to the same individual. And then there's the split, which is it turns out that there's John Smith and there's John Smith Jr. and they are, even though they live at the same place, they really want to, they're not really the same person. So I need, I incorrectly merged them together. I want to be able to split. So I've got that one record because I've determined that there was a false positive met and that they were incorrectly merged together. So that's, those are the core services. And the top record is the application services layer. So this is where the case goes directly to what the application might look like. And this is, the case here is to say, what is it that we're expecting to see and what is it that we're expecting to do? This is where we look at those integration points because we'll say, oh yeah, you know, we have part of this business function at the website, for example, the web portal that the customer comes in. They want to know whether that customer actually, that individual actually is a customer already or not. So you would have search and retrieve entity information. So you might invoke that course, search, retrieve, retrieve. The core services back here are a customer record. Well, that, the business process may actually direct you to say, well, if there is a customer record for this individual, provide me with a customer record. If there is no, if you can't send me a customer record, then create a customer record. Fill in these pieces of information and create the new record in the master repository and then return the unique identifier back to me as well. So the, that's the service or capability that is provided to the application. Another design of unique identifier, that is unique as part of identification services, you know, whether that's linked into, to your security or your location, etc. These are become a component of the enterprise-wide identity that goes beyond just is this a person that I know, but rather is this a person that I know and how can my knowledge of that person be used to, whether that person is authorized to do this kind of activity or to get access to this kind of data or to purchase these kinds of products or to describe these kinds of medications, etc. So there's the assignment of the unique identifier and its integration at the higher level into management capabilities that may already exist. The enrichment, so it's not just, but rather, rather at what does the application need to append information about what an example might be. I've got a new customer file that came from the people who signed up overnight for a new customer account. Now I want to go to find the master environment to see whether they existed from all the other customer accounts that we've accumulated through all of our corporate acquisitions and then update these new accounts to indicate whether we already knew about them or whether they really are a new account. So that's an append or a reconciliation of identities that is an enrichment or an enhancement of that data. As cleansing, I look up the record to see if I street address correct, whether it's spelled correctly, is the town where that street is, those types of inclinations. The alignment of otherwise assigned identifiers for cross-referencing where I want to be able to link the customer account for individuals that are made to demonstrate that discounts are applied for both of their individual accounts because perhaps a business rule says that their combined sales qualifies them within the household not just by individual accounts. So that's been set up and we have batch services for doing cross-referencing and matching and identity resolution and updating. Relationship management didn't really make it on. It's got a little bubble there in the middle. Functions associated with entity relate to or more entities associate that nature of the relationship. The business function level. So it's not establishing that just creating the relationship of record but rather from the business function standpoint when I'm looking at these for linking individuals or linking individuals and putting them together, etc. So those are the types of APIs or interfaces that you expose upward to the application. It's looking for where does a customer record get looked up in the subsystem, now replace that with an invocation of a search retrieved entity record from the master repository. Incrementally, you're looking for where those integration opportunities are and then replacing them with the master interface as opposed to the data directly. The high-level view. Now what that means is that we can start looking at different ways again and not necessarily I mentioned this before. You don't necessarily have to create a separate repository but rather you could get a virtualized mechanism that federates data across the different data sources and then you use that master data index as essentially virtualizing a view of that master repository without having to create a physical copy or replica of that data. So the index maps a canonical representation of that specific entity. Maybe it's customer, maybe it's provider to the locations of the data in the original source and you record it in your master data index and the accessibility in a view provides the underlying business rules that are relevant to the consumer of that data at the point that the data is actually accessed. And this is where another item I've been harping on for a while which is that the semantics associated with what key organization thinks is a quote-unquote golden record may vary vastly from what the different business functions may think is the culmination of a quote-unquote and so instead of enforcing one set of rules and then trying to shoehorn that into everybody's use, one set of rules that are federally repository and apply the different business rules within the different contexts when you're delivering the data to the consumer. So when you're the sales department and you're looking for the customers that can afford to buy your product, that would be different quote-unquote set of customer records than a customer support and you need to find out for anybody who's calling in to have the right to get phone support so they need to upgrade their support contracts so that they're able to get that level of support. And that means looking at a different set of criteria as to whether an individual is or is not a verified support customer. So there's different semantics and you might apply those semantic rules, the business rules, not when you're putting the data into the repository, but instead you are applying it when you're taking the data out of the repository. Understanding how data virtualization and federation works. And here's kind of an example where, number one, it's a bunch of consumers of that master data. And then looking at those master data services, which are the same ones that we looked at back in this slide above the data access, go through some canonical relational view that transforms and maps the request that goes through the master index to the original sources here, pulls the data back at number seven, applies those rules, whatever the business rules are, either for parsing standardization, normalization of the data, cleansing the data, reorganizing the data, filtering out the data, and passing it back to the layer that provides that relational, the canonical version of the view and feeding that back through the master data services to the client who will then be able to look at the data in the way that they anticipate or are expected to be able to see it based on their own set of rules. So depending on who the consumer is, number one, then different rules are being applied when the data comes back at number seven or maybe even at number four where the filtering of what the actual requests are that go through that master index becomes as well. Let's summarize somewhat. Number one, if you're looking at a strategy for developing your master data services for application integration, the first thing is look at your business processes.  And we're going to figure out whether that customer already exists within your system or not. Do you need a new record? Do you not need to create a new record? Can a customer make a purchase with a guest without actually registering with the customer? If that customer does do that, what information do you need to capture? What are you allowed to keep track of, et cetera, what are the policies and governance policies associated with personal or private information? These are all the types of things that are at the business process level depending on the uses of that shared master data. Second, look at information points within the business processes. That would make use of that master data. So is it a search? Is it a retrieve? Is it updating a customer record? Is it leading a customer record? Is it merging a customer record into a single customer record? Is it a match? Is it a duplicate analysis? Is it a mail merge, et cetera, the different integration points? And then look at the user. How is the data being accessed? What are the performance criteria? Does it need to be a database join in a database query, or is it an extract of the data which has been interacted with with another data set, or am I loading it, taking it out of a big data repository and dumping it into an analytical engine, et cetera, how am I using that master data? Then the term required function is that it's not already provided either by the business function application or by the MDM vendor's product. And look at the existing systemic support is for that required functionality. So an example might be, I want to get access to the data. The master data is actually sitting in two different repositories. I want to run a query from D2 or Oracle to the data that's sitting in in a SQL server, the master data is in the SQL server database. What does SQL server provide in terms of federation to allow me to run a query that crosses multiple data sets? Now once I'm able to identify what system support exists and what tool support exists and what doesn't, then I need to look at what the application integration services are and what the APIs are going to look like and then I can design and implement those application integration services. A little bit of work needs to be done. Typically it's going to be on top of capabilities that already exist, but are definitely going to need, I guess I'm going to call it a quote unquote cool tool to make sure all the pieces that that exist fit together. I mean it's interesting you might find that one capability exists on one platform and another one exists on another platform, but two platforms are not compatible. So that's something that you may need to create or get access to some kind of application or some kind of data integration probably allows you to get access maybe through JDBC or ODBC from one platform to the other platform in a more seamless way so that you don't need to be concerned about the fact that the two platforms don't actually talk to each other. That may say you need to go out and get some product and then embed that within your environment. Layering all these components together gives you a practical approach to understanding how you can put the capabilities together to allow an application that is already in production to eventually transition from its own copy of data to full integration with that master data repository. So that's the general just here and I think we have a bunch of questions that already came in. There's certainly opportunities for other questions to come in. If you have questions that pop up, please feel free to contact me directly in my contact information here and I encourage you to point your QR scanner at these books and go out and buy two copies for yourself and one copy to share. We'll have a couple of news about what is called, I'm looking at it right now, one is called Data Analytics from Strategic Planning to Enterprise Integration with Tools Techniques NoSQL and Graph but if you go to a search for Big Data Analytics and Lotion that one will cross and then there's one that I wrote with a colleague, a rifle for calling, using information to develop a culture of customer centricity. We'll look to try and move up the frame and stop talking about the data itself and look at the data in the context of how we want to be able to use it and I think that any conversation about master data management certainly has been to look at at the use of that information for purposes of customer centricity so if you're interested in those ideas also please drop me a link and give me a call et cetera and I will think the University and Shannon and Tony and all the folks over at the University and I encourage all of you to read it out for EDW and I will see you in Austin. Fantastic presentation if you guys have questions make sure you get them in the Q&A and one of the most popular questions that always comes up is asking if you'll get questions a copy of the slides in the recording which I will send out within two business days so by end of Thursday you'll get that in your inbox I'll also include David's contact information and the list of books here for you make sure that they're in our D-diversity bookstore which is through Amazon we also have a partnership with Morgan Kauffman so we can give you even a discount on David's books there so I'll make sure and get all that information to you so David coming up in the questions here should I manage changes to master data master data in master data repository or does it belong to analytical applications which the concept of slowly changing dimension has been already established? That's actually a good question and one that I've heard numerous and I think that the challenge there is that if you are if you know it's kind of like a trade-off right which is that we already understand how slowly change the dimensions work for the analytical environments like a data warehouse or a data market and necessarily bubbled into the engineering of the master data repositories because the people who have built the master data repositories have you know and I'm just analyzing here I think they've largely they're focused on the mechanics of of identity resolution with with management of the data as a byproduct so I would I would prefer though that the master data repository also be able to handle the whole concept of the slowly changing dimensions and that when the data is syndicated out to the data warehouse that you no longer have to worry about it as you know a priori from the data warehouse perspective because the historical changes are inherent in the way the master data is being we look at the things that we haven't done enough of which is looking at the master data in the context of how that data is being used then we'll certainly start seeing things that we really thought needed to deal with in the master data repository like like I said in the mention or historical timeframes and and and and so finding ways of of relationships associated with with individuals at different points and directly so my answer is if we're going to be managing a master repository then we'll try to to to put that within the repository next question consider that you may not manage all attributes of the master data subject in the MDM the answer is then where the in question managed managing dimensions in the MDM would add a level of rigor to the process managing change so you know I look at this as if you've got a little dial on your your your data model where you can store a very little amount of of information in the in the in the in which that is that you're you know you might store it the the least amount that you need in order to do to do resolution of identity and you could turn that dial all the way to the right and and eventually keep all the information about the entity in master master data repository and yes you're right it would add a level of rigor to the processes managing change but I think that that's the that's exactly the the that we haven't addressed over time which is which is looking at the master repository as as a core is a part you know a both the the transaction slash operational applications and the political applications so when we look we're making a copy of data that we can use to do blah to do data or to you know do reporting or whatever to do identification and now we're we're down the whole concept of being able to merge data together because because we're using it in in its validated form and begs the question it's whether you why you would be consolidating it in the first place so that's why you know I look at it from two different alternatives one is you're going to build a repository then build a repository and and embed it in your policies the position that says anybody who's using a anything that's competence repository has got to has got to now comply with with the the rules of the repository for the the other side which is my requirement only includes identification information all we're doing is we're federating the original sources in in their original place we're not actually moving the data at all and we are materializing the view well on demand as opposed to making an extra copy so you know like you can adjust accordingly often I've encountered questions from my clients who are looking for quantitative results for beginning an MDM solution they want the bottom line what's the ROI well that's a small topic I mean I'd be to point you to my data quality improvement book I have a chapter on the on assessing the business value of of of information in fact I'm looking it up right now because I don't recall what I actually wrote but I do know that I put a the first chapter talks of the business impacts of poor data quality but really it's a it's a measure for really for understanding how to characterize what different different areas of the business and their depends on data and come up with quantifiable measures of what they really means from a financial perspective or from a risk perspective and and you know come up with a return on investment you really need to know your expected turn and what's your investment and you know somebody when when the clients are looking for quantitative results for beginning a solution you know really tell that unless you've done an analysis that says that says what are the gaps or what are the fail failure points that are associated with your inability to to probably resolve unique identification fixing it and number three when the business has something to make the organization changes that that are necessary to make make use of that so that's why I would point you to this other book which is information about cultural customer because it really talks about well yes we do need customer data but we also really need to do that says that we know what we're expecting to get and that we can our staff to to to work with the customer in a way that the customer it was from that 360 degree view if I actually had it question what is the lowest MDM data out latency achieved with a non federated approach I've heard 15 minutes at best well that's why don't we turn into a a blog post on the University site to say to make it a challenge tell me what you've done how much data you've you've pulled out and what was how fast you did using it depends on what you're using depends on how many machines you're using it depends on on what's the speed of your network it depends on how smart your your your your data analysts are in terms of writing queries I mean there's data that you can get out really quickly but you know if you write the query the wrong way it could take you five days so like there's so many variables to that it's impossible to really it's really 15 minutes sure so how is MDM different from reference data management RDM from a data governance perspective is the industry moving MDM and RDF away from being part of the demo wheel okay first part of the question is master data management different from reference data management because your reference data management is master data and once you're storing master data as master data it effectively becomes reference data so the answer is it's not really shouldn't really be that different than people who are telling you that it is well you know send them to me so we can and we'll have a panel there's a question for a conversation but the second question I'd like to direct you towards the people who are putting together the data and they can tell you they can answer that better than I am and that's it those are all the questions we have and we are running out of time David thank you again so much for such a fabulous presentation again you can meet David in person in Austin, Texas Enterprise Data World 2014 be sure you check that out EnterpriseDataWorld.com and as mentioned I will get a link out to links to the recording of this event and to David's books and information and so you can get all that information by his books and have information. David thank you again and I hope to see everyone in Austin.


Here we are in our QuickBooks test company file. We started up in a prior presentation. Remember that we are in the accounting view as opposed to the business view. You can toggle between the two views by going to the cog up top, switch the view down below. We're gonna duplicate some tabs to put reports in like we do every time. Right click the tab up top to duplicate it. As that's thinking, right click again, duplicate the tab again. Back to the tab to the middle. Reports on the left hand side, opening up the balance sheet, which is in the favorites, changing the name or the range 010125 to 123125. I would like to see this one by class, even though we only have one class that we worked on in a prior presentation, we turned on class tracking in a prior presentation and that gives us this breakout, not too much detail from the balance sheet side, although it might give us more detail in this practice problem on the balance sheet side. Next tab, I'm gonna open up the reports. This time the profit and loss report. And this time I'm just gonna open it for the last few months from 070125 to 123125 and run that by month. So we can see what the last practice problem did. And then I'll run that and there's our income statement by month of transactions. And we have this issue with the timing not exactly lining up on like a percentage of completion type of system. And that's what we wanna be thinking about this time. So what we're gonna do this time is we'll create basically a system where we'll try to line that up. Let's first enter the estimate again. So I'm gonna go into Excel and kind of think about how we might put this together from an estimate if we're putting a project together and then we'll go through a similar kind of process to tie all this together. So in an Excel worksheet, I'm gonna format the Excel sheet by clicking here. Now by the way, if you have this sheet, then it'll have an example tab, a practice tab and a blank tab to help you work through it and we'll just build it on the blank tab. If you don't have it, that's okay because we'll just build it from the blank tab anyways. So I'm gonna right click on this tab and we will format the cells. I'm gonna make it currency bracketed and then I'm gonna say no decimals. That's usually my starting point on the formatting. And then let's say we're gonna make our estimate. So a job, let's say this is job two estimate that we might actually do in Excel. We might do this estimate in Excel so that we can then populate it into say QuickBooks, right? So I'm gonna say, all right, let's then say that we went through our estimate if it was a complex type of estimate but we'll simplify it here and just say that our three buckets of expenses are usually gonna be materials and then the labor and then the overhead. So let's say that of those items, we think it's gonna be 40,000 for the materials, labor, let's say 30,000 that we think is gonna be a labor and then the overhead, let's say we think is gonna be 923 or something like that when we put our estimate together. If I sum this up, we're gonna get to 76, let's say this one's actually gonna be 6,923, 6,923, which sums up to 76,923. So that'll be our total cost and then we're gonna have the markup that I'm gonna say is a 30% markup. So whatever the cost is, we're gonna mark it up 30%. This is a general convention. You can, again, it would depend on what your billing structure is but this is the concept that I'm gonna be using here. So we'll mark it up 30% and so we're gonna say, this will be the markup percent and then this will be the markup amount which will be equal to the 76,923 times 30%. So now I'm gonna say the total charge estimate is gonna be equal to the cost plus the markup which would be 100,000 which we're trying to get to that nice even 100,000. So that's gonna be our kind of baseline starting numbers. I'm gonna make this bracketed and this is gonna be the bottom line. Now, once we have that estimate, we might tell the client, hey, look, we're gonna bill you based on this estimate. So we might then say, okay, our billing structure, I'm gonna make this a skinny column. We'll look something like this. The billing schedule for this job is gonna be, I'll make this black and white is gonna be, I'll say month one and then month two, month three, month two, month three, month four and month five. And I'm just gonna, I'm just making up the billing structure and like how would we make up the billing structure? We'd probably tie it to the materials we need if it was a construction type of contract. But, and we might have our first amount that I'm gonna charge is just gonna, I'm just gonna say 10,000 because I'm gonna assume we need 10,000 down to buy the stuff we need to start with this more longer term project. And month two, I'm gonna say, we're not gonna bill in month two because that's when we'll start the project possibly, but we actually got the deposit in month one. So we got kind of like a prepayment of it. And then in month three, I'm just gonna make up a billing structure of 25,000, 30,000 in month four, 35,000 in month five for a total of 100,000.


People are eating, thinking it'll nourish their life. It's not doing much to them except filling them up, nothing much. If the nutrients do not have necessary iron, vitamin A, C, D, B6, B12, folate, iron, all these things, you become far more susceptible. In 1930s, what one orange would give you, today you need to eat eight oranges. Okay, I will take more pills, more pills, more pills. It's not going to work like that. The reason why this pandemic is having the kind of toll that it's having upon us, there are medical issues, I'm not entering that. One fundamental thing is, the soil nutrients are so low, how low is it? The soil depletion, I'm only talking about United States. The soil depletion in United States has led to twenty-one percent drop in vitamin A in all the fruits, vegetables and everything else that you're eating. Thirty percent drop in vitamin C, thirty-seven percent drop in iron levels, twenty-seven percent drop in calcium levels. The things that today you all think is very healthy to eat salads and stuff, the spinach, the... what is your favorite leaf? The lettuce, the lettuce and tomatoes and other things which you normally use in your salad. From the early twentieth century, what level of nutrients they had and what they have today is only ten percent left, ninety percent lower than what it was in early twentieth century. So you're largely eating water sacks without much nutrient in it. If we had that level of nutrition, our ability to... not hundred percent, but our ability to counter this would have been different. We would be at a different level of immunity that it wouldn't be such a scare. So many people need not die. I'm saying over six million people have died in the world. United States alone is nearly nine hundred thousand now. Just United States alone is nearing nine hundred thousand number of deaths. The nutrient studies across the world are showing the nutrient depletion in the food across the world is so significant, it's tragic that we are eating something, thinking it is food that nourishes us, but actually it has nothing. The question is not even... I'm not even going to the place of fertilizers, anti... you know, pesticides, herbicides. I'm not even going there about... that is outright poisoning, all right? I am talking about depletion of food. That itself is tragic. People are eating, thinking it will nourish their life. It's not doing much to them except filling them up, nothing much. And that is one reason why we have to fear the pandemics. And I've been listening to some of the top viralists in the world, they're all saying that this may not be the end because the very nature and structure of the corona, the fundamental corona as such, already you have seen two, three variants. Now we are talking about Mr. Omicron, he sounds like a guy. So they are saying, far more brilliant and far more capable viruses can easily develop with a small region in their basic structure and they're very much capable of doing that any moment. So that is what every government fears, that is what every scientist fears and that should be the concern of every citizen. But in all this soil has a significant role, let's come to the food that we eat. If the nutrients do not have necessary iron, vitamin A, D, C, C, D, B6, B12, folate, iron, all these things, you become far more susceptible to any kind of respiratory infection is a well-established fact. And right now that is a serious condition because ninety percent of the Americans are deficient in vitamin E and that is why highest level of deaths in a given country for the population has happened in this country for 350 million people, nearly a million people are dead. This is the highest in the world. There are countries like India and many other nations far more densely populated. Just the amount of space that you have, you should have been safe. But lack of nutrients and irresponsible behavior has cost us so much. We have to ensure that more people don't go in this direction. So soil is a very important part. Forty-three percent deficient in vitamin A, loss of integrity of skin in the mucus membrane happens when there is not enough vitamin A and that is one sure way of having serious implications to any respiratory infections that we get and we must understand this COVID is essentially a respiratory tract infection and which can go into various kind of complexities later. Thirty-nine percent are deficient. Thirty-nine percent of the United States citizens are deficient in vitamin C and this is very vital for the antibody production, your vitamin C. Okay, I will take more pills, more pills, more pills. It's not going to work like that. The food should have it. They're saying, in 1930s, what one orange would give you, today you need to eat eight oranges. Has anybody eaten eight oranges in a day, any time, you eat. Okay? They are working in an orange form. Eating eight oranges a day is not a possibility for most human beings. One orange, what it would give you, today it takes eight. So considering all this, soil is an unfolding disaster for everybody. I was talking to the WFP director who is from United States. He said, last year we spent nine billion dollars distributing food in Africa to starved populations. This year our budget has to go up at least by four to six billion dollars. Otherwise we cannot fulfill this because the number of famines we're expecting in many parts of Africa is so high. And he said people thinking that this will only happen to Africa or Asia is unfortunately not true, that he's saying by 2035 there could be severe famines in Chicago and Illinois region, because the soil conditions have depleted so badly it can easily happen. Oh, we will transport all the food from here to there, there to here. No, when there is food shortage, no truck is going to reach its destination. People are going to loot it on the way. Hello? Yes or no? You orange eaters tell me, will the floored oranges, will you let it go to Canada? You guys are eating eight per day. You will stop it on the way. I'm saying the disruption, they are saying by 2045 there could be major civil wars all across the world simply because of soil depletion and they are seeing the only way that if people don't respond, if they don't do the necessary things, the only way is to go seriously into genetic engineered crops, the GMO crops, other things that your everybody has become fashionable to talk against, but that will become the only bridge which will see us across for 2025 years. But after that when you crash, there is simply no recovery unless you put the soil back. Every year on an average there is a loss of twenty-seven thousand species of microbes are going extinct per year. So if we let this go as it is for another twenty-five to forty years, after that recovery will take anywhere between hundred and fifty to two hundred years. If we act now, if we change the policies now and begin to act in the next ten to fifteen years time, in twenty-five years time we can turn this situation around. I feel this is a fundamental responsibility that we owe to ourselves and to future generations on this planet. What do you think?


Hi guys, this is Dr. Ahmed Ergen. I'm an endocrinologist, and today I'm going to talk to you about up and down blood sugars. Why is my blood sugar is up and down? Up and down. So when I ask my patients, okay, how's your blood sugar running? They'll be like, up and down. It always perplexes me because I'm like, what do you mean by up and down? I don't know, up and down, up and down. Well, okay, so, just to tell you something, something that's really secret that you will need to really know. Your blood sugars will always be up and down. So nobody's blood sugar is really a flat line. You will not have a blood sugar like this. If you're expecting a blood sugar like this, then you should not be alive, okay? So that's number one. Now, even if you're non-diabetic, let's say I'm non-diabetic and I check my blood sugars because I want to and I use continuous glucose monitoring system, et cetera, et cetera.  It's going to go up and down, up and down. Now, what these diabetics refer to, though, that everybody's, of course, understanding of up and down is different, but they may mean that their blood sugar is going up to 200 and it may be going down to 60. Now, the question is, then my next question is, what sugars are you talking about? Are you talking about the fasting blood sugars in the morning? Are you talking about after meals? Are you talking about bedtime? These are blood sugars will be totally different depending on what time you check it, right? So if you're checking it after an hour after a meal, that's a different story than checking it right before a meal. So if you're telling me, for example, if your blood sugar is, like, let's say before breakfast one day is like 70 and then you check it after breakfast one hour after and it goes to 170, you say like, hmm, my blood sugar is up and down. Well, of course, because you just ate, right? So it is not uncommon for your blood sugars to spike a little bit. Of course, in this situation it's a little bit extra higher spike, but that is kind of unexpected. Now, of course, it's going to be up and down, but the question is how up and how down? So if you're going more than 180 at any time, after a meal typically, that's not good. And if you are going below 70 at any time, it could be when you're sleeping at night or it could be in the early in the morning or maybe an hour after a meal. So let's say you take a short acting insulin and your blood sugar is, let's say, 160, which is not good. But you took insulin because you're an insulin mealtime, et cetera, or you just wanted to correct it. You're on a sliding scale, you wanted to correct this and you ate food, but if you did something wrong, you ate too much insulin, right? So you're 160 and you ate food and then one hour after, you went down to 50.  Now, of course, you don't realize that you maybe did something wrong with the insulin, right?  So how do you really manage that problem? So you really need to understand the patterns, the pattern of recognition, and you have to have a lot of data. So you have to really analyze your numbers. So the best thing to do, keep checking your blood sugar, right? So 120-year blood sugar, let's say you took five units of noble log, you put that down and let's say you ate 60 grams of carbs, you put that down as well. Now the next thing is you check it again, let's say two hours after breakfast if you're going to do that or before lunch, but be consistent with that so that you can analyze, you can create a pattern of recognition. So let's say you're 120, you did five units, you ate 60 grams and then you ended up being, let's say, before the next meal, 150. And that's, let's say, before lunch, okay, before lunch. Now, let's say, second day, you woke up with a blood sugar of a similar, 110, and again you took five units of noble log, but you ate, for example, 30 grams of carbs this time. And then your blood sugar ended up being 70, right? So do you see, of course, it's hard to see a pattern just from these two numbers, but you can tell by the data provided, you can tell the problem here. So that's the problem. Blood sugars are very similar, 110 to 120. You took the similar amount of insulin, but you ate 60 grams here versus you ate 30 grams the next day. Now, your blood sugar ended up being 150 and a little bit higher than where you started, which means that maybe you should have taken a little bit more insulin here, or maybe you had a misunderstanding of misconception of how much carbs you eat. But here you ate 30 grams and you took, still took the five units and you ended up 70. So why? Because basically you took insulin more than you needed because you only ate 30 grams, right? So if you knew that, for example, yesterday you took five units for 60 grams, you ended up a little bit higher sugar, maybe if you have taken a 60 units, that would have done it, right? If you ate, if you took 60 units of noble log and you ate 60 grams, that could have been just 120 just like you started, that tells you that one unit basically would have taken care of 10 grams. So if you're taking six units for 60 grams, that means that your insulin carburetor is one unit for 10 grams. Now here you should have taken based on this, if this was a successful story, if you need one unit for 10 grams, here for 30 grams you should have taken three units, three units of noble log. Now what happens is you took five units, that's why you're down to 70. So when you're seeing your blood sugar is up and down, I get it, but the problem is you should actually keep a good record of what's going on for your doctor to understand why your blood sugars are up and down, right? So that's why people get surprised that my blood sugar was 120 yesterday before lunch and today it's 70, so what's going on? They just do not understand all the variables that go into it. There are actually more variables than just this. So the problem, there are a lot of problems. So other variables that you need to be aware of is when you take an insulin shot, right? When you take an insulin shot, the absorption is not always 100%. So there is 20% variability in your absorption and then 20% will create a significant variability if your insulin amount is high. Think about this. If you are taking only 10 units of insulin and you have 20% variability, you took 10 units, your body absorbed eight units, right? So you're off two units. You thought that you took 10 units, but you actually ended up absorbing eight units because of the 20% variability. And the next time you may have, you may absorb 12, not 12 units, I'm sorry, you may absorb nine units.  But think about this. If you're taking not 10 units, but let's say 50 units before the meal, then based on this 20% variability, you may have up to 10 units difference in the absorption. If you have a 10 unit of difference in absorption, that's going to throw you off like significantly. Even if you do everything right, if you are taking a lot of insulin, the variability factor will be a big determining factor.  So for example, people who take long-acting insulin, right? So they will take, for example, you know, 60 units of lontas and their blood sugar in the morning sometimes is 120, sometimes it's 80, sometimes it's 160. And they say, you know, the only thing I take is a one-time insulin 60 units and my blood sugar is sometimes 120, sometimes 80, sometimes 160. Now as we said, one of the factors is the variability. That's 60 units guys that you're taking is not always the 60 units. Your body may absorb sometimes 52 units, sometimes 55 units, sometimes the entire 60 units. And that's not in your control. So it will happen no matter what you do. Another variability is, what did you have at dinner?  So as a result, your blood sugar, although you're absorbing exactly the same amount of insulin, that's a nothing change in terms of variability in the insulin absorption, but you will still make up the higher blood sugar, even if you have done everything exactly the same, except you put extra butter on your bread or you fried that food that night, then you're going to have a higher blood sugar in the morning. Or that day you decided that the weather was nice, you wanted to go out and take a three mile walk. You have a 120 blood sugar the day before, you will end up with an 80 or even maybe down to 60 blood sugar because of the exercise factor, which makes the insulin sensitive immediately overnight and your blood sugar may go down. So there's a lot of variability guys, so your diet, your exercise, even the smallest changes that you make, but I say smallest changes of frying the food is not necessarily a smallest change, but it makes on, it makes on, obviously it's just one time, but it can make a difference. The pizza that you ate, for example, last night can make a huge difference. The exercise that you did, the gardening that you did that day can make a huge difference in your blood sugar in the next 12 to 24 hours. So do not underestimate these facts. Remember, the more insulin you take, the more variability you will have. And the diet and exercise, the fried food or eating excessive food or having a nighttime snack, all these things will increase your blood sugar. Exercise may reduce your blood sugar. And again, as I said, sometimes you may absorb the insulin more, sometimes less, that also will change your blood sugars. So, I hope that helps. Again, short-acting insulin is the same way. The absorption factor, how much carbs you're eating will definitely affect your blood sugars. So make sure that you try to keep a consistency. You try to make sure that you collect enough information so that you can create this analysis. If you are using continued school-class monitoring systems, it's gonna give you a lot of data points. You're gonna be able to see your blood sugars much better because it's continuous. And you will be able to go back and see what happened. So these are the good things about continued school-class monitoring systems. You can prefer Freestyle Libre. There's a Freestyle Libre II now. There's Dexcom out there. There's Guardian out there. There's a lot of good continued school-class monitoring systems that I love because they basically help you to monitor and get you a lot of data on a day-to-day basis. So anyways, so that was today's topic. Make sure you subscribe, guys, and we will see you in the next video.