Hello, everyone. Welcome to my session. This is Gavin Shen, kind of working for Red Hat as a software engineer. Today I'm going to talk about the Synchronized PageFault and STS motorization support for ARM64. Here's our overview. I'm going to talk about the motivation where we need the Synchronized PageFault and the current status of the feature. And then the general requirements of the feature. After that, I'm going to talk about the Synchronized PageFault and STS virtualization implementation on ARM64. Then I represented the performance and came to the conclusions. Because the feature is currently developed, I need some support from the community to review my code. About the motivation, generally speaking, the Synchronized PageFault improved guest terrorism significantly by rescheduling other processes for execution in the guest, where the host resolved the state to page fault. Besides in the scenario of live migration, the guest performance benefited from the feature either. The feature may be used by other purposes. For example, it is used by the virtual LFST to relay views from the host to the guest. The feature was introduced to x86 initially around 2010. And the virtual LFST used the Synchronized PageFault to relay views from the host to the guest on x86. And the feature is supported for S390, which is one of the IBM's architecture. But unfortunately it's not available on ARM64 yet. So we need to do something to support the feature for ARM64. About the general requirements, so there are two passes, data pass and control pass. The data pass is driven by two notifications, which are page now present and page ready. The page now present the notifications sent from the host to the guest before the state to page fault is to be resolved. When the guest receives the page now present notification, the guest starts to reschedule other processes other than the 14 processes for execution. In the meanwhile, the state to page fault is being resolved on the host set. At a later point when the state to page fault is resolved successfully on the host set, another notification which is page ready is to send the host to the guest. When the guest receives the second notification, it reschedules the previous 14 processes for execution. There is still another data broker in the data pass, which is shared between the host and the guest. The shared data broker is updated to distinguish the notification. It is also used to identify the specific asynchronous page fault by a unique token. In the control pass, the control data broker is used to help to finish the configuration and the migration. This is a general requirement. So what's the particular requirement we need to support the feature on 64? I would like to compare the situations we have on x86 and on 64. First of all, on x86, the page vector 14 is used to deliver the page not present notification. But the same mechanism is not available on 64 because we have a very limited space in the yes underscore your one system register. This is just as used to tear the root cause of the page fault. So the SDA, which is a software dedicated exception interface, is leveraged to deliver the page not present notification on 64. Apart from that, the interoperate is still used to deliver the page ready notification, which is quite similar to what we are doing on x86. The shared data block is updated on 64 to distinguish the notifications, which is exactly the same to x86. On 64, the control broker is accessed by the SMCC, which is the secure monitor call calling convention to configure the single aspect fault. But on x86, the email message is used for the same purpose. Besides the control broker is also accessed by the actual command from the user space in order to support the migration on 64. However, on x86, we don't need this functionality because the email message can be migrated naturally. So let's take a look how the single aspect fault is working on 64. At the beginning when the QMI starts, the QMI use the octal command to configure a single aspect fault. There are two types of information needed to be configured. The SDA event number and the interrupt number. After that, the guest boots up. The guest start to use the SMCC interface to enable the single aspect fault in step two. At a later point, the sum of the running process in the guest trigger the state two page fault, the guest trap to the host. This time, the host send the first notification, which is the page not present in step four to the guest. When the guest receives the page not present notification, the guest is going to acknowledge the notification and start to reschedule other process other than the 14th process for execution in step five. At a later point, when the state two page fault is resolved successfully on the host in step six, another notification page already in step seven is sent to the guest. When the guest receives the second notification, the guest is going to acknowledge the notification by rescheduling the previous 14th process for execution. So until now, the single aspect fault is complete. It's a lot for the control data block is updated when the single aspect fault is configured. But the shared data block is updated when the page not present and pick ready, I delivered and acknowledge. And we need to support the migration after a single aspect fault. The migration is quite simple to use the octal command to retrieve the state of a single aspect from the source VM and restored on the destination VM. Which means we just need to migrate the control data block. For the shared data block, we don't need to migrate. Because all the pending a single aspect fault are canceled before the migration starts. So as we mentioned before, the SDI is leveraged to deliver the page not present notification in order to support a single aspect fault. So let's explain with SDI. The SDI is the abbreviation of software dedicated exception interface. Defined by then W054A specification, the specification can be downloaded from the link. Generator provides a mechanism for reducing and servicing system events from hyperweather. The interface is offered by the hyperweather to the guest OS. The service is delivered with SDI event. The SDI event is identified by a unique event number. The SDI event is delivered to the guest immediately, regardless of the guest state. It's not a maskable by the IQ and score disabled or similar functions like that. So in this regard, it's quite similar to x86 NMI. Apart from that, the SDI event are classified two types, which are shared and private events. The shared event is owned by multiple EPEs and delivered to one of them. The private event is only visible and owned by one PE. Again, the private SDI event and interoperate used by a single aspect fault to deliver the PE to another person and the PE to read the notification. And the SDI is working based on the SMCC. So I would like to explain what SMCC is. So SMCC is an abbreviation of a secure monitor call calling convention defined by then W0 to AD. The specification can be downloaded from the link either. It basically defends a common calling mechanism to be used with secure monitor call, SMC, or hypercall HVC instructions. HVC instructions used to generate exception, which is handled by a hypervisor running in exception level two. The arguments and the return values are passed in the general purpose register x0 to x17. The service is identified by the arguments carried in the general purpose x0. The value carried in x0 is divided into several fields. One of the fields used to identify the function is called function ID to identify what's the function that gets the ones they have ready to provide. So for the SDI is a fault in the category of SMC standard service calls, whose function ID is a fault. So let's take a look at how the SDI is working. In the diagram of left side is the system level behavior. When the guest boots up or when the driver is going to be loaded in the guest, the guest retrieves the version from the host. To check the version, it's valid or not. After that, the private and short reset is issued to the host. So that previous pending events can be reset or cleared. After that, the guest issue the PE and mask monitor to the host to receive the SDI even from the host on the calling VCP or PE. Eventually the guest need to be shut down and the guest issue the PE mask commander to the host. The PE mask command stops the guest from receiving any further SDI even from the host. So this is the system level behavior. For one particular event, I mean the SDI event, as we can see from the diagram of the right side, the guest need to issue get info to retrieve the specific information about the SDI event. And then to reduce and enable the event. After that, the guest is able to receive the SDI event from the host. When the guest need the event anymore, it issue the disable and register commander to the host. So for one particular SDI event, how is it delivered? First of all, when the host receive the event, in the step one, the host need to save the calling site and increase the general purpose register X0 to X17 and PC and P state. And then the event is delivered to the guest. The event handle, which was provided when the guest, when the SDI event was registered is invoked. When the event handle is going to complete, one of the two commands either complete or complete and resume is issued to the host. To tell the host, the SDI event has been handled successfully. But there's one difference between the two commands. The complete and resume schedule the pending interrupts immediately, but the completed does not do that. So that's how the SDI is working on the ARM64. And the single recipe is supported on ARM64. So about the performance, the first scenario is the performance in the heavy slapping element. I have a test program right by myself. The test program is right to the all available memory. In the meanwhile, there may be one calculation straight, maybe running or not. The VM configuration always has one V CPU and the memory is one giga. And the curing process has been put into the C Group 2, where the memory limitation is enforced to have 512 megabits. The first scenario as we show in the top set of the benchmarks, the calculation thread is not running, but 5 more percent time is needed to finish the job because of the overhead introduced by a synchronous batch of fault. As we mentioned before, the asynchronous pay fault is needed to deliver two signals. The delivery of those two signals incurred some overhead. In the bottom side of the benchmarks, the calculation thread is running. You can see about 40% of the time is needed to finish the job. So we can say the asynchronous page fault is improved the guest performance in terms of parallelism. Another performance we need to observe the maturity is about the live migration. So still we have the same configuration as before, the V CPU is one and the memory is then with the one giga. But this time we don't enforce any memory limitations from the C Group to the curing process. And on the top side of the benchmarks, the calculation thread is not running, but about 3% to 60% the next time needed to finish the job. Because the improved activities by asynchronous page helps to decrease the page dirty rate and post-copy requests. In the next slide, I will provide more data about the page data dirty rate and post-copy request. The bottom part of the benchmarks is a scenario where the calculation thread is running. The time is used to finish the job is almost the same, but the more calculation capacities is offered when the asynchronous page fault is enabled. In terms of the calculation capacity improvement, about 41% and 68% is improved. So here we have more data about the performance in the scenario of live migration. So on the left two columns where the calculation thread is not running. So you can see when the synchronous page fault is enabled. The time is needed to finish the job is a job from 9.1 second to 8.8 seconds. The right two columns where the calculation thread is start is running. And the time is needed to finish the job is almost the same. But when the synchronous page fault is enabled, more calculation capacity is provided. So from the number, the calculation capacity is increased from 1,684 million to 1,781 million. So the synchronous page fault offers more calculation capacity during the live migration. So eventually we come to the conclusions. A synchronous page fault is a significant beneficial to the guest's parallelism or interactivity. According to the benchmark we had, 40% improvement of the parallelism in the heavy swapping scenario. This is also beneficial to the guest's performance in the period of post-copy live migration. About 41% to 68% more calculation capacity is offered by the synchronous page fault. So the features are currently under development. All the code has been posted to the mail list. You can find it from below links. And all the code has been also uploaded to the GitHub. You can also check out it from the GitHub. I really hope someone from the community can take a look on the code to start the real code so that the feature can be merged in time. Thank you very much. This is today's session. If you have any more questions, feel free to contact me through email. My email can be found from the slide one. Thank you.


SCP-279 appears to be a man in his early 40s, of unremarkable appearance and dress. SCP-279 does not respond to any form of external stimuli and has not been observed to deviate from a standard neutral expression. SCP-279 has been reported to spontaneously disappear and reappear throughout D, presenting unusual difficulties to agents required to monitor it. SCP-279 seemingly spends the majority of its time time traveling through the town of D. Attempts to eliminate SCP-279 have failed. SCP-279 continues whatever it is in the process of doing regardless of any injury it has sustained.


Hello and welcome. My name is Shannon Campan. I'm the Chief Digital Manager of Data Diversity. We'd like to thank you for joining this Data Diversity webinar, Creating an Edge and Enterprise Data Governance Experience, sponsored today by Irwin. Just a couple of points to get us started. Due to the large number of people that attend these sessions, you will be muted during the webinar. For questions, we'll be collecting them by the Q&A in the bottom right-hand corner of your screen. Or if you'd like to tweet, we encourage you to share highlights or questions via Twitter using hashtag Data Diversity. As always, we will send a follow-up email within two business days containing links to the slides, the recording of the session, and additional information requested throughout the webinar. Now let me introduce to you our speakers for today, Jamie Knowles and Marianne McDonough at Irwin. Jamie is a business and enterprise architect with 20 years of experience. As Product Manager for Irwin's Data Governance Software, Jamie is unifying the company's data management platform with data modeling, enterprise architecture, and business process modeling, and fortifying it to ensure enterprise data assets are understood, governed, and protected. Marianne is a market creator and builder with 30 years of strategic business leadership and experience in enterprise software and technology. Career highlights include serving as Chief Marketing Officer for In Contact, Senior VP of Corporate Marketing and Investor Relations for Xtralis, and Senior VP in Corporate Marketing for Variant Systems. And with that, I will give the floor to Marianne and Jamie to get today's webinar started. Hello and welcome. Hey, thanks, Shannon. First of all, I just want to compliment your choice of totally grooving music that we had while we were waiting. I don't know if everybody else is out there in webinar radio land, but I'm in a very sort of smooth, standard mood right now, so let's see if we can keep it going. Jamie and I are really excited to be with you here today. I think that most of the folks that are on the webinar know, or because of our deep roots in data modeling. That's where we began. We've been doing that stuff better than anybody for in excess of 20 years, but I'm hoping that folks have also been following us as we expanded the brand to include enterprise architecture and business process modeling and to build over the past year and a half or so a strong data management platform. And then just two short weeks ago, boy, it feels like a lifetime. There's been so much activity and excitement. We expanded again and introduced a completely new data governance platform with a new approach. And we'd like to share some of the perspective that we've developed during the way working with some of the world's largest companies as we brought Erwin DG, we call it, Erwin Data Governance to Market. So from here, let's just sort of jump into it. I don't think it's any secret to anybody that we are riding the data waves. There are days when I feel like the data wave is riding me, but certainly data is proliferating exponentially. Just some fun facts that you see here on this slide. By 2020, there'll be 37 million new things connected to the internet. So those things could be everything from, you know, my father's pacemaker to my intelligent refrigerator. And, you know, those of us who are who have Alexa's or, you know, some other Google Home devices, IoT is really changing the game in terms of the number of things connected and the amount of data that flows to and from those devices. Between that and escalating personal data usage, one of my other favorites, self-driving cars. I come to you today from Rome, Italy, where we did a data governance event yesterday. And I wish they had self-driving cars here in Rome because the traffic is pretty crazy. But self-driving cars, intelligent planes, robo factories, so many things that are happening, you know, in our world that are just continuing to contribute to this immense tidal wave of data. And, you know, the volume, variety, and velocity of this data wave, the three Vs, really extends to almost every asset of our lives. And, you know, not surprisingly, totally to the way that we do business. But one of the things that I think is so interesting is it's not just the volume of data that's increasing, but also the potential of its value. You know, when you look at this slide, you can see here some of the real quantifiable value across a variety of markets that harnessing data and using data for good in the business could deliver. So we see this massive number that says big box retailers that can leverage the full power of their data can increase operating margins by 60%. Investing in better and integrating better big data, healthcare could save as much as $300 billion a year. And this is one of the ones that really blows my mind, is that for a typical Fortune 1000 company, just a 10% increase in accessibility to data will result in more than $65 million in additional net income. But here's the flip side, you know, according to recent research that we just saw from EY, only 23% of organizations have implemented an organizational wide data strategy that enables them to actually use data to look at market conditions or customer needs or preferences and then align their products and services with those data-driven insights. So there's this sort of massive dichotomy out there when you look at the potential and then we see that, you know, only, you know, 0.5% of all data that's captured across the world in major enterprises is ever used to grow exponentially. So, you know, what's that all about? And I think we have to ask ourselves that question because the majority of the time the conversation is in this place, right, is about the risk. And it's not to say that we can downplay risk. Certainly the people around the world that are struggling with how to comply with GDPR and the mandates that are coming May 18th, again, from here in Rome yesterday, a big concern across Europe, but not just Europe. I mean, American companies too that capture customer data in Europe have to be concerned. So, you know, there is a downside to data, right? That explosion of data, if you view it from a regulatory and compliance lens, really can introduce significant risk and challenge a business' ability to leverage that data safely. So the question is, you know, how can we ensure that data is not being used in a way that's harmful to the business? And there's so many cautionary tales here. You know, last year investigators found employees that misused customer data in a very well-known, if you can see me, I'm making quote marks in the air, very well-known financial services provider. They opened 2 million state credit cards and deposit accounts on behalf of customers to meet aggressive sales goals. And that fallout created 185 million dollars in fines, 2.6 million dollars in refunds and thousands of employee firings. And that's not to mention the intangibles like reputation value. So, you know, absolutely risk and mitigating risk and compliance and all of those things that we traditionally think of in terms of data and data governance are important.  When you look at this combination of data for growth data for power, data for market understanding, data for competitive edge and data for risk mitigation and compliance. We really believe it, and I'll sort of throw the gauntlet down here and say that data governance 1.0, as we see it, has been a failure. It's been the promise of value without the delivery of that value. And, you know, what is the root cause of that failure? And we believe this because we've been working with, again, some of the largest companies in the world over the past 18 months to understand what they've been doing in data governance and why it has or has not worked. And, you know, what we've really been told is that it's because data governance has been sitting under the dome, if you will, right? It's been under the dome of IT. And data management has historically been a technical function with IT providing important custodial service to the business. But the challenge is it's now time for the business to take a more active role. The business has the knowledge about the meaning and the purpose of the information. The business owns the regulatory challenges. The business is tasked with adapting to rapidly changing markets and needing to communicate those changes to the IT custodians. So organizations need a data strategy that's aligned with the strategic goals of the business. That makes sense in the context of the way that we operate the business. One that creates data-driven growth while mitigating the risks associated with an expanded data horizon. So an approach where both business and IT can effectively collaborate to understand and control data and to leverage it to drive more intelligent business decisions. So in order to help us better understand and define this data governance 2.0, I'm going to ask my illustrious colleague, Jamie Knowles, to talk a little bit about our perspective. What is it about this DG 2.0 approach that's different than what most people have thought about or experienced in the past, Jamie? Yeah, cool. Thanks, May. I think the key is to involve the business. And I think they're hard of it. We've got to better treat our data assets in the same way as we treat our physical assets of the organization. I was talking to an executive fairly recently, and they said, if we did treat our physical assets in the same way as we treated our data assets, we'd be busted in weeks. So how do we go about that? So generally, we start off building policies on how do we govern our data? We set out roles and resources they're going to carry out so that governance sets out some processes that they're going to perform. And the next is a series of questions. So as we've got here on the screen, we're going to start off cataloging the types of assets that we have. What do they mean to the business? How important are they? What's the value of them? And then it starts to work out, well, where in the organization are these assets? Where have we physically got them deployed? Then we can start asking our host questions and really get some good answers. So we might want to look at the rules that we all want to apply to those assets. And again, it's only the business that can really ask and answer those questions satisfactorily. So once we then start cataloging, we can then look at, are we following those rules? Look at the quality of the data in places. So really, it's a journey of solid questions. A journey of solid questions, I like that. And it's true that only the business can answer those questions. I'm going to pipe in a little bit later just about my role as a chief marketing officer. My role has changed very much over the last five years in terms of the value of data. How much data in my organization I own and am responsible for and what I do with it. And when I think about that in context of this ring, it's really significant because there are so many people in our organization that don't understand the data that resides in all of my world are what I call the marketing infrastructure. And it would be very difficult for someone to catalog and assess that without having my business owners who are sitting down and working with them to really explain, this is what that data is, this is what it connects to, this is how we use it. So that data in context again I think is really clear. So let's jump into some of the challenges that we have sort of uncovered, Jamie, as we've been on this journey ourselves. And I think one of the things that's really interesting, and we spoke to a lot of people yesterday about, and Shannon was busy shaking her head, yes, when we were discussing it in some of the prelim work, is that every organization is in a different place in terms of where they're going with data governance. And it's important to sort of be where you are and then chart the course forward from there. You can only really begin where you are today. So when I look at some of these questions, some really jump out and there's conversations that we've been having about readiness. So talk a little bit, Jamie, about readiness and its impact on data governance success and some of the sort of pre-work that has absolutely nothing to do with technology or a platform that we see customers needing to do on DG. Yeah, I mean readiness is a great one. So getting the right sponsorship. So we've been talking to so many customers and I think a lot of governance projects that have been attempted in the past and failed. So they've had the wrong sponsorship. So quite often we've seen projects that have been purely within IT again. So I think you've really got to get sponsorship at an executive level to get the adequate resources available and then get buy-in and sponsorship from the entire organization. It has to understand that they're part in data governance. It goes all the way from some sort of basic security in the office through to how we use our systems. So yeah, step one I think has to be sponsorship. Yeah, I think... There's a lot of things that... God, please go. I was going to say that there's a lot of things that jump out here that we've heard from customers. Silo's is a really good one. I mean, larger businesses struggle with this a lot more, especially where there's been some sort of mergers and acquisitions type of history. So silos of information managed by multiple systems. There might be multiple departments that claim ownership of data. So there's a lot of political challenges between different groups of people that claim that ownership. Each of these silos might have different rules and hold different copies of the same information. So there's a lot of challenges there. So how do we unify these silos and provide lenses to make sense of it all? So that was another big one that came out of late. Yeah, and it was interesting yesterday as we were doing these table conversations with many of the largest companies and organizations here in Rome, that whole concept of sponsorship or lack thereof was a recurring theme, number one. And then the second was what a challenge M&A delivered to data governance in general. So, you know, I'm absolutely feeling you there. Here's a question for you that I think is a really important one. You know, how do you create the fourth bullet there? Talked about enabling accountability. How do you create a culture of data accountability within a business? Well, that's a tough one. How do we create it? I think you've got to be able to, against the catalog, start to assign names. Simple as that. Yeah, and it's a little bit of show your work too, isn't it? Right? Absolutely. You know, as we come to a meeting with sets of numbers, you know, I think we have to be able to identify the lineage of said numbers and where we're getting our data from. And I think we need to begin to develop a language to question each other about those things that does not today exist in most businesses, right? Yeah. That's part of that data thinking. I also think that from a senior executive perspective, when I think about, you know, being a corporate officer and all of the fun stuff that comes with that, you know, I think we have to start challenging some of the assumptions that our leadership is making about the course and speed of the business and challenge the data behind it, right? And say, show me those numbers, right? Tell me where you're getting this information from. So I think that kind of behavior that is driven from the top down and becomes an expectation in terms of the language that we use with each other starts to change some of that. Yeah, and that comes into proving value as well. So having visibility of what are the assets of the organization and everybody understanding who is accountable and what does accountability mean? Absolutely. And also, I think part of the failure of Data Governance 1.0, Jamie, as we've talked about, was that the tangible value of the exercise, right? The only way to really look at it was in terms of that risk and compliance mitigation piece. But nowhere were we figuring out how do we monetize the growth pieces? How do we monetize the intelligence that the business has that it didn't have before? How do we look at it in the same way as we look at the other key four or five strategic things that we're going to do as a business this year? So again, I think that driving data in the same language in lexicon as the way the rest of the businesses run is really critical. So on that count, I'd like to just take a little bit of time to talk about our view of this edge, right? The enterprise data governance experience. When we talk about having an edge, we're referring to both an approach and an underlying technology portfolio that enables virtually every aspect of the business to benefit and grow from the investment in data governance, just like we just said. So there are five pillars, the enterprise data governance experience pillars. I'm going to run through the first two, and then I'm going to hand the three on the right over to Jamie to explain. But the first one, and many of you who have attended webinars or metta set trade shows and events, know that we've spent a lot of time over the past year and a half talking about any data anywhere, something we call the any squared. And you know, any squared becomes so critical in this day and age because according to Forbes, and it's just common sense in many businesses, only about 12% of a company's data sits in a traditional database. To create true enterprise governance, you need to leverage data from any source in any format, regardless of where it's stored. Voice recordings, web chats, documents, relational, no SQL data on-prem in the cloud, it all needs to be accessible to the data governance program. So that's number one. Number two is really around stakeholder empowerment, and we started to talk a bit about this, right? It's all about the needs of the data stewards. It's about actively engaging with the most important data holders in the business about what they need and ensuring that your approach and technology takes these things into account. I told you I was going to tell you a story about sort of my life and my role, so here it is. And I think it's a pretty good real-world example. As a CMO, I've been in many, many, many board meetings. And most of the time I'm presenting with my counterpart in senior sales, whether it's chief revenue officer, executive vice president of sales. And it's been a very common occurrence that we have been presenting completely different data for the same exact KPIs. And what always happens is the board says, whose data is correct? And if there's some, you know, question about the credibility of that, how can I rely on my data to make informed decisions about where to invest and how can the board rely on my perspective about the best places to invest? If they are looking at two data points and saying how do these match up and we don't have a really good answer to that. So I think that that's a great example of how we ensure that key stakeholders are on board and get better alignment and support for the mission critical initiative. It can even open up budgets beyond IT. So here's an interesting thing. I'm going to give you a bit of a preview. We have just fielded an amazing piece of research on the state of the Union on data governance. There'll be more coming on that shortly. But we did see this giant economy between the conversation that says, yes, enterprise-wise, stakeholders should be engaged, but we don't ask any of them for budget. And we think that this is a significant miss. We think, again, that this is a reflection of a data governance 1.0 thinking. Because I know if I'm going to get business value back, if you're going to solve the problem that I just talked about with my credibility with the board, with my making good data-driven decisions about where to spend my money, I'm going to pony up a piece of that budget, too. So now I'd like you to take over, Jamie, and talk about the third pillar in integrated ecosystem. Yeah, sure thing. So you're talking about a lot of the people of the organizations collaborating together in the data governance experience. Likewise, we need the machines to do the same thing. There's so many systems in our organization now that we need to understand the metadata. They need a real common understanding of that metadata and to start being able to share their knowledge. I think a good data governance experience needs that central body of knowledge to be shared and accessible by the systems. Visibility across domains, I think it's where we need to start applying other lenses to our data. We need to understand how the people in the systems use our data. And again, allow different roles from the business analysis analysts, IT architecture teams, also collaborate together on information. And the last one, regulatory piece of mind, is really what you were talking about earlier. There are so many regulations that apply to our data. There is so much risk associated with our data. We've got to get out of the control. We've got to catalog where is our maximum risk. Something we were talking about yesterday was the viral attack on the British National Health Service. The physical estate of the National Health Service is just so big. It's just not possible to put a great big firewall around all of the data in the systems of the organization. So you really got to look at where is our most valuable data assets and really start to understand those and focus on where you're applying your dollars. Absolutely. That's awesome. So we're going to shift gears and now talk about really data governance 2.0 and what it looks like. And I know that this is your favorite slide on Earth in today's presentation. I know you like this one. So why don't you take the folks through it and again, contrasting it versus that siloed under the dome approach? Yeah, absolutely. So we're all about modeling. So I really see the world as a model. I think that the key to this slide is that line down the middle of the screen again. So bringing the business and IT together, getting them to collaborate and communicate effectively. So the left of the line, top left-hand corner, we need to understand the language that the business uses. And different businesses communicate in very different strange ways. I spend a lot of time working in IBM where every conversation would be mainly comprised of acronyms and strange jargon. So we need to understand that language in the business and build these glossaries of terms. We need to start to understand the rules that applies those terms. And again, as we were saying earlier, we need the business to provide those rules in business language. Over on the right-hand side of the line at the top, we've got our data dictionary. So this is jelly sensors around a well-managed catalog of data elements. So we've got our periodic table of data effectively. So a single element representing each concept. And that then ties together all of our detailed descriptions of data assets there. So as you were saying earlier, Mary, data lives in all sorts of forms. So databases need to understand all forms of databases. We've got big data projects out there. We're now starting to open up to our developers the ability to create data stores with no SQL type structures. So we've got to understand all of those. Files, so it could be files that flow between the systems. We're even starting to have conversations about blockchain now. Again, we need to understand what data is in order of these things. Documents, so semi-structured and unstructured data. And it could go all the way to an Excel spreadsheet that supports a critical process of the organization.  Again, what information is out there? Where is it all? And then down at the bottom of the screen. So it's not enough just to understand the data. You've got to understand what's it being used by and used for. So again, we've got the humans and the machines. We've got roles and systems. We need to understand data at rest. So data stores, which might be instances of the data designs we've got in the top right-hand corner. We need to understand how information flows between those roles and the systems and the data stores. And if we start to bring location into the equation, we can start to ask security questions. So I can start to look for where have I got business critical data flowing from a role or a system, a data store, from a high trust zone to a low trust zone. Because there are the places, again, that need to start spending my dollars to provide an adequate level of security. And I think the last piece here is process. So what is being done to the data? Where is our critical data being used in the organization and by whom it rolls? You know, you make this very complex slide seem simple when you explain it that way. So I know I appreciate that. So let's just chat a little bit more about the stakeholders. You know, some of these, I think, are sort of expected. But what's new here? And where do we really see the difference between the 1.0 to 2.0 world in terms of the stakeholders and what they get out of, you know, of being a part of the initiative, Jamie? Yeah, that's a good one. So I think the top of the list there is the data owners, the notion of data owners. So data governance, one was all about the stewards and the custodians of the data in IT land. We were talking about accountability and ownership earlier on. We were talking also about breaking down the silos. So trying to get a single approved standard for each piece of information that contains an approved definition includes the rules that the business needs to adhere to to be able to satisfy regulation. So this new group of data owners is needed to 2.0. Yeah, and the last part of the bottom there are the consumers. Everybody in the organization is a stakeholder of data. It's everybody's business. So we need everybody to have access to that information about our data. So we need a central place for people to go to find out the latest definition of the term, to be able to reduce the ambiguity in their jobs. So again, working at IBM and I don't know what an acronym means. I need somewhere to go to better look that up and then look at the rules and start to ask questions about that data. Yeah, and you know, that's a really interesting point that I think bridges back to the whole conversation about creating a culture of data accountability, right? If everybody in the organization is armed and knows where to go, what to do, how to get it, how to better understand data, then I think the data fluency that we have increases exponentially. So I think that's a big shift in terms of the DG 2.0 as well. So I'm going to shift gears a little bit to a little bit of the Irwin world. And again, for those of you who know as well, you know that we've been a modeling powerhouse for a long time in the past year again, added business process, modeling, and enterprise architecture to our award winning, really category defining data modeling. And it's very interesting because we really believe that that sweet spot of how DM and EA and BP come together is something that first of all is unique to us and something that we can really do better than anyone else and has massive impact on the success of the depth of the robustness of the data governance initiative. So would you talk a little bit about why all these things are important together, Jamie, and how they help to support, inform, expand, extend data governance in the real world? Yeah, absolutely. So I think bringing together these things gives you an exponentialization of value. So as we're saying, the data governance experience is not just about understanding the data, producing logical data models, or producing a business glossary. So you can ask far and wide questions. So bringing together enterprise architecture and business process modeling, I think, has huge amounts of value there. So when you're asking the big questions about your organization, you want to start looking at how do we improve this organization as a whole? So those enterprise architecture-tied questions, looking at where have I got issues and redundancy in my systems, where have I got underperforming capabilities, areas of risk, then that's where enterprise architecture comes in and provides us more information to then shine the data lens on it, starts asking questions around, okay, where have we got problems with our data? Where is our data at risk? So I think if you can satisfy each of these areas individually, producing value from each of these areas, that's great. But if you can then bring them together in that data-given experience, then you're going to get a much greater degree of value. Yeah, that's terrific. And can you just talk a little bit, Jamie, about something that we call data impact analysis, which is such an awesome Erwin advantage and why it matters again in this 2.0 world? Yeah, I think the organization is a bit like a Jenga puzzle where you pull one block and other blocks are going to start to collapse. So the organization is a complicated machine. So if I was to pull the plug on a server somewhere in the organization, then somewhere else a business process is going to fail and a customer's going to be unhappy. The same applies to our data. So we've got to be able to understand the impacts of changing our data. So how many customers could answer the question if you change your customer reference number from 15 characters to 20 characters, then what needs to change? That is potentially a gigantic question. There's going to be systems involved, processes, people, et cetera. There's a lot of impacts. Understanding that impact is key. Yeah, and that's something that, again, because these things work together, because the data models and the business process models and the enterprise architecture are linked at the metadata level, right? We have a view to that and a capability that we think is really important and is going to change the game in that DG 2.0 world. All right, so I'm going to give you a softball because I know you'd love to talk about this stuff. Let's switch a little bit more into the technology realm and tell us, you know, give us what you think would be a good technology prescription, if you will. You're the doctor, Dr. Jamie. Why is the prescription that will actually bring us into this DG 2.0 world? Okay, so we need a platform, a focal point for the organization to go to to be able to ask big questions about our information, to be able to raise issues about the information to make requests for information. So we've been doing a lot of work talking to customers out there about what is the ideal prescription, and we've broken it down into these four capabilities. So the discovery phase is, again, about the business being able to define what information is important to the business, so building out the business glossary, the different terms, which might span across different departments of the organization, including external partners. So if you've got to communicate with external partners, trying to understand their language is absolutely vital. Reference data is a big thing as well. So code sets, for instance, the medical industry operates around sort of a very gigantic code set which describes all the possible diseases and ailments out there in the world. So trying to understand what are those codes that we use to describe those diseases is vital. Where all our systems are using those codes, we need a central place to define them. The understanding part is now for that important data, where is it? So starting around the data dictionary, the data usage part, understanding the people, the processes, the systems. So bringing in the catalogs from the enterprise architecture and the business process world. Data quality. So where is our data and how good is that data? So we'll find that we've got duplicated pools of data and silos of data. What's the quality of the data in those places? When we start to look at governing our information, then we need to set out those high level policies. What is our policy for data governance, security, etc. And then build out an operating model of processes so everybody is clear on what are their roles and responsibilities in giving them that data. And then, again, back to the business philosophy, setting out the rules that applies to each piece of information. And the last part is socialization. We've got to make all this information as these bodies of knowledge visible to the organization. So anybody can come along and look up definitions of those standards of our information. Start to ask the impacts question. So if I change customer reference number, what's going to be impacted? Lineage is a big one. So again, a lot of regulations will require us to prove where did this data come from? What is the quality of the origins of the data? So if we're looking at business intelligence type projects, I might want to take a BI report and understand where in the data warehouse did it come from? And then those tables in the data warehouse, what are the operational data stores with the origin of those fields? Self-service data, another area. People, we want to come to a single place and request that data. So we're having a really interesting conversation yesterday based on an earlier conversation with Forrester about the data scientists. And they were saying that a good data scientist now can be commanding salaries up to half a million dollars a year, which is craziness, because that role needs to have such a broad range of knowledge coming from a deep knowledge of the business and how the business works through to sort of a PhD in mathematics and analytical science, through to understanding IT and BI engineering topics. So we've got to start breaking all those roles out and allocating them to the right people. So having a self-service data journey from the initial request through to people being able to find where is that data to then assemble the data into a format that is useful. We have to better support that. And then the last part is issue management. So it should be easy for anybody in the organization to register an issue, whether it's... I don't agree with this particular definition of a business term through to using the system and I'm seeing some data quality issues through to on a registered data breach. So I think for the perfect prescription, then you need all four of these capabilities. Yeah, and it's interesting what you said about the data scientists and the self-service access. Because if you think about what we talked about before, increasing... Creating that culture of data accountability and changing the level of fluency and data that the business has. If it sits in the hands of just a number of people who have the complete span and control, then how do we become a data-driven business? How do we become a business that is fluent? And that challenges assumptions that aren't data-driven. So I think it all comes back in a very nice loop. So thank you for that. I really appreciate when somebody puts a great end to a story that I've been working on. I'm trying to just advance to... Hang on, to the last slide. Let's see. I'm having trouble with my advance. So hold. There we go. And see, now we want one too many. So just a few notes to sort of close this piece up, and then I'll hand it back to Shannon to give us some questions. But in our experience, what we found is the DG1.0 movement has left the market wanting more. It just was not enough, right? It solved some problems and actually created some new ones. And we believe that a successful data-driven project requires something different. It requires this enterprise empowerment that we're talking about. And readiness, which is very often not a forethought, but an afterthought, is key to success. This isn't just about deployment of a tool. One of the things that we've done in bringing our solution to market is we have armed our consulting team with a host of readiness services offerings to help organizations, just like yours, do everything from discuss, as Jamie talked about before, where the ownership for the initiative should live. How does it work with the existing MDM strategy of business?  And then last, but certainly not least, and especially in the heart and the mind of somebody who sits in a C-level share every day, data governance has to be measured and measurable in the context of the business in order for it to be the game-changing solution that we really truly believe that it can be. So now I'm going to just hopefully click forward one more time and turn it over to the delightful Shannon to take some of your questions. Thank you. Thank you both for this great presentation. So if you have any questions, feel free to submit them in the bottom of the Q&A section in the right-hand corner there and just answer the most commonly asked questions. As a reminder, I will send a follow-up email by end of day Monday for this presentation with links to the slides and links to the recording. So, Mary Ann and Jamie, just diving right in here. So what does DGOM stand for? The Data Governance Operating Model. So this is the collection of processes that define how we're going to, given the data of the organization. Simple as that. And can you expand, please, on what you mean by expanding data governance, or, excuse me, measuring data governance? Yeah. So what we're talking about really is back to that whole idea of looking at it in the context of the business. So how can data governance help to achieve key initiatives like customer satisfaction, customer loyalty, sales growth, customer churn? What are some of the other ones that you can think of, Jamie? So the regulatory things. So being able to define what is personal data of the organization and have a clear understanding of where is it. Those are things that are very much measurable. When the regulator comes in, you need to be able to say, yep, I've ticked the box. I've done the job. So, you know, Mary Ann, you mentioned a compelling measurement of data governance in the business context. And you stated that data governance has to be measured and measurable in the context of business. So what's an example of a compelling measurement in that? So I think one of the compelling measures is really some level of growth. So whether it's revenue growth that's data driven, and I connect that back to everything from data tells us that competition is, you know, reducing our margins by X amount. We use data to actually formulate a new service offering. And then we grow the business, you know, 5% based on that new offering. I mean, really that's the kind of kind of close loop starting with an objective, you know, using data to support the understanding of the objective, making some change, and then being able to go back and deliver it again. So that's where I sort of use that, both measured and measurable, right? So if it's not tied to a business initiative that matters to the people that I work for, which is primarily the board, then it really is just a project that lives inside the business. Jamie, anything to add there? I think if some of that will. Ben, do you have any typical examples of quantifiable business impact for data governance, revenue, market share, cost avoidance, et cetera? Not yet, but we're in the early days, and that's the sort of stuff that I think you can look to us in the next six to nine months as we really begin to deploy these solutions with lots of really big customers. So doing those kinds of time and motion studies and getting that quantifiable data, we think it's going to be, you know, the most important thing that's going to change the way that people think about the initiatives in the business. Yeah, we're looking at some techniques for monetization of data as well and some metrics around that. So, yeah, lots to come in this area. Yeah, you know, we usually get that question along the lines also with, you know, how do you get execs to buy off on implementing any kind of data governance program within a company? Yeah, and that's interesting because, you know, Jamie, talk about some of the big customers that you're working with and the senior that I'm in Italian right now, so I'll do the capo di tutti. You know, what is, what's their motivation? You know, of the most senior, there's that very, very, very large financial services company that's been a customer of us for a long time. And, you know, what is, what's the motivation that they have? Why are they bought into this initiative? Yeah, I think there's, I mean, as you were saying at the start of the presentation, there's two drivers. One is risk, which could be sort of regulatory or a reputational risk. So there are often certain things to quantify. I mean GDPR is a great driver of data governance because there's a very clear penalty against it. Reputational risk, how do you quantify a reputational risk of a data breach? It's a tough one. And the value side, the value generation side of data governance and understanding your data and just general enterprise data management is another tough one to quantify. So I've been working in enterprise architecture for years and again, we've been trying for a long time to try and put sort of solid metrics against this. But the reality is sort of just understanding your data, having a framework of understanding around your data, giving that that visibility to understand the impact of change, you can see sort of the benefits of that. But it's difficult to quantify. That's our next. We'll come back and we'll talk about some of the quantifiable results, Shannon, in a couple of minutes. I love it. There's a challenge. So to what degree can the use of mobile devices and remote access added a layer of complications? So how should org policies and their implementation address this effectively? This is a good one. So it goes back to what I was saying about the National Health Service. It's impossible to put a big perimeter around your organization and no data comes in and out. In this modern world, we're dealing with more external business partners. The data supply chain is getting more complicated. As we're looking at SAS type business models, our data is moving far and wide. The blockchain again is a potential benefit, but also a possible threat as our data sort of is rapidly proliferated across geographical boundaries. So having an understanding of where is your data going is absolutely vital. Mobile devices is a part of that. Yeah, and the interesting... No, please. That's where sort of enterprise architecture and data governance come together. So having those clear models of where are the roles, the systems, and the data source, where are they located, what are the flows of data between them and how are those flows implemented in technology is vital to a complete data governance experience. Well, and you know, I think also too, Jamie, if you think about mobile in general and you look at your device, right, and you think about what you do with it, the amount of data that you create produce on this thing, and then the number of organizations that you do business with that you have a data relationship with. Think about it that way, right? There's many that I haven't, and these days I'm scared because I'm one of the 143 million people whose data was breached, you know, from Equifax. So the consumer is in many ways, you know, becoming much more of a part of your organization, whether you like it or not, right? And the moat that existed historically between your business and the people that you did business with was much more significant than it is today. And now because of the five or six different ways that I have to connect to you, and the many applications of yours that I consume and I use every day, right? The relationship that I have with you as my business provider is much different and with much greater risk. So I think that that's one of the really big things that we don't always consider about mobile, right? That we've just really sort of exponentially increased, you know, the number of data relationships that I have. Yeah, absolutely. I think a lot of companies aren't doing a great job of it. I get increasingly frustrated day to day when I try and contact my gas provider or my mobile phone provider and there's no point in where I can contact. So I have to do everything through the web or through my mobile device. And I find I can't do things that I need to do. So again, having an organization needs to have a clear understanding of the customer journeys that we expect. And then within that understanding of those customer journeys, we need to understand what data is involved. And what are the policies, the rules that we apply to that data? And are we following them? So it's a complex question. Yeah, and you know what's interesting about that is I came out of many years in the customer service space, so I was very comfortable talking about the customer journey. But I think that the customer data journey, right, and the customer data relationship is really sort of the new frontier of that. Because when you talk about reputation loss and how you quantify that, we're going to start to see, I think, trustability. And companies that have high trustability indexes become a key decision point for doing business with them, right? I'll give you a perfect example. Again, I hate to reiterate the fact that I'm in Rome and you're all not, but in connecting with the hotel that we were in yesterday, which had a sort of a funky, none of us ever really heard of it, local ISP, right, Jamie?  And we all looked at each other and said, you know what? We need to get on this network really badly and do a lot of things we need to do. But I am so not comfortable giving this company that I never heard of, you know, the color of my hair and the size of my feet. And I think that more and more we're going to see that that trustability is going to become a key competitive advantage for organizations and that we're absolutely going to be able to monetize. Do people choose to do business with us because they believe that we hold their data, you know, in great respect, that we protect their data, that we govern their data, that we know exactly what happens to it and when it happens. For me, one of the scariest things about the Equifax thing was not just that I was a part of it, was that no one could tell me what it meant, right? What data? For how long? Where did it go? Where did it come from? Was it all of my address information for the last 40 years? Was it only five years? So when you think about that sort of consternation and fear that you're creating in the hearts and minds of your customers, I truly believe that they're going to start beat feet and go someplace else when this data trustability as sort of a third party ranking becomes a big thing. So that was a long and involved and mildly passionate answer to the question, Shannon. I love it. And you both touched a little bit on BI as a self-service BI, but how does data governance fit into the evolving paradigm of just data as a service? Jamie, you want to take that? Yeah. That's a new question. It is, yeah. As our pool of information grows, as Marianne was talking about, the proliferation of data, we've got to make that data more accessible, as she was saying. So BI is going to become more and more valuable. So we've got to get the journey from the business description of information through to where is that data? Where is the best quality data? Where is the right data for the context of the request? We need to make that journey as quickly as possible. So we're looking at how do we hook our data governance platform into BI solutions and get that integration tighter? So along that line, Jamie, thoughts on the product's ability agility to Nexus issues and can the product easily be integrated with other systems such as Microsoft Office 365, other things? I've got to admit I don't know what Nexus issues are. Maybe the questioner can expand on that, but maybe a little bit more on the integration. So, again, what sort of integration are we talking about here? So as such as Microsoft Office 365, I know 365 can come with Power BI, things like that. You've talked about some... Yeah, I mean understanding unstructured data in documents is a good one. In Excel spreadsheets and Microsoft Word. So we have document management systems rammed with information. We might even have SharePoint and other such content management systems rammed with information. Again, trying to understand where is the critical company data in this world is vital. So we're starting to look at profiling technology to be able to go out and search for critical data in the organization. Search for it in structured repositories such as databases first, but then to go out into the structured world and hunt it down. Yeah, it's a big problem. So I'm going to... We've got time for one more question. I think I'm going to ask a loaded question here to both of you. We've got a few more minutes. You talked about the business being more responsible for the data rather than IT. So what if the business wants to help and move forward but they don't fully understand data? That's an interesting one. You know, I think that that goes back to this whole fluency idea, right? That there is a certain level of data that everybody has to speak its table stakes and whether that becomes part of sort of corporate training initiative or a knowledge transfer or a knowledge base initiative, you know, I think that that's sort of really the fundamental table setting to be able to have a data flow into organization. So, you know, the first thing is you have to have the will and then the second thing is you have to have the skill, right? So if the organization has the will, then I think, you know, there are some things that the traditional IT data stewards can do to help educate people in terms of what it really means to be data centric, what kind of data we collect in the business, what kind of data, you know, we think you may have in your organization. So maybe it's something as simple as having a sort of a structured way to do discovery within a business unit that doesn't have somebody that specifically speaks data and uncovers it. I think that there, go ahead. This is where organizations need to lean on external resources that they can help them on the gym to understand that the challenges of data governance and teach them. We've got a ton of great business partners that can help them with that. Yeah. All right. Well, that does bring us right to the top of the hour. Thank you both so much for the fantastic presentation. I just love it. What a great topic and how important it is. And thanks to all of our attendees for being so engaged in everything we do. We love all the great questions that have come in. Just a reminder, I will send a follow-up email by end of day Monday with links to the slides and links to the recording of this session. And again, thanks to everybody for attending today's webinar. And thanks to, again, to Marianne and Jamie, especially Marianne, as you're traveling as well as as we are. We just love it. And I hope you enjoy the rest of your time and safe travels home. So I hope everyone has a great day. Thank you. Thanks, everybody. Bye-bye. Thanks.


All right Welcome to day four of haskeling the advent of code I hope my audio is a bit better. I've turned on compression So even when I'm talking this direction, it shouldn't sound too bad Thing is I was testing it always by talking directly into the microphone And not looking in this direction, right, which is where I'm looking most of the time. So Hopefully it's better now Alright Let's look at today's problem But yeah, let me know if the audio isn't good enough Because I can monitor my output, but it's very hard for me to monitor the audio Because there's so much lag Because I don't have like direct feedback into the microphones like I do with The microphone like I don't have direct feedback of the entire mix Like I do With my microphone audio, so I can't really tell how the mix is Right Let's check out today's problem day four. Oh, no, it's day three day four passport processing, okay So we forgot our password and Okay, so we don't have a okay, so we aren't valid. We don't have a travel documentation For long line. Okay, due to some questionable average security, you might be able to solve both of these problems at the same time Are we gonna be doing hacking? Is that illegal? The automatic passwords can as well slow because they're having trouble detecting which passwords have all required fields The expected fields are as far as brr year ear hicked huckle huckle pid cid okay so Passports are validated in batch files your puzzle input each passport is represented as a sequence of key value pairs separated by spaces or new lines Passports are separated by blank lines. Here's an example batch file containing four passports Okay First part was a valid all eight fields of pressure second is invalid. It's missing height The third password is interesting. The only missing field is CID so it looks like data from North Pole credentials not a password at all Surely nobody would mind if you made the system temporarily ignore missing CID fields treat this password as a valid Fourth one here Missing two fields CID and beer missing CID is fine, but missing any other field is not so this password is invalid Okay, according to the bubbles you improve system would report to valid passports so In your batch file, how many passports are valid? All right, let's go Seems like we're gonna be doing a lot of string processing today. Let's make your day four CID day four and Then we open day four we say a new file Day four that it just But let's start off with saying new file here and let's Have our test input and put that Okay, so here's our test input now we will say Mainware Get input string IO Let's just return a list of passports, right? That's kind of cool Get input So this is gonna be so we're gonna take This should be file path, right? So we do we read file so line LMS is gonna be the read file of the file path and Let's see here passports How are you gonna represent passports? So I assume that later we will have to look at the values in problem two, that's like It doesn't make sense for it to have so many stuff if If it doesn't have all the values, so let's just make this a map for data dot map Okay So a passport It's going to be a map of strings to strings This is gonna be a list of passports. Okay, so to passport so this is gonna take a list of strings the fields of the passport and We are going to return the passport So first of all Um To passport equals And we haven't defined this yet So let's just say you print LMS. Let's just print it to start with The issue now is Okay, yeah, we have to return a list of passports. So let's say here main is IO main equals do Get input test input Let's run this GHD day 4.hs O day 4 and we run it day 4 What happens? Okay compiles links No typers We could only tell that from GHD ID. Okay, we got all the lines. So let us So passports are separated by blank lines So we won't go here we will map lines over this. Let's see here So now we have the lines But it might be so there's gonna be blank lines. So Let's Now we're gonna want we want a function to kind of split it at all the places where it has Split split Okay, we want a function that takes an a and Let's I think I think it's a bull and list of A's and we want a list of list of a's Split when oh, this is from another Okay, that's just that's just writes a function, right? Split when takes an EQ takes a A to bull and then And then a list of A's and returns a list of list of A's split when Okay, so How do we do this? F and then we have case if you have an empty list we will return empty list, right? And then it doesn't matter what F is split when Fx x is Let's save it a it's having a a And then case F a This so if if we have a match on the front we want it to We want to drop those right and I think it's easier just to split at That's in the prelude, right? Or What is a function called isn't like span? I mean this is kind of like span But there's I think it's drop when or something like that, right? Drop while a Yeah, so take while Not okay, so span Okay group because I think there should be a there should be a list here somewhere. It's a function that just does this Okay, let's just let's just let's just make it easier. Let's just say split when x is This is just gonna be where Break This is gonna be con So this is gonna be you know everything that doesn't satisfy the condition and then the thing that satisfies it so So it's just gonna be First and then the rest equals break con Texas So this will be First that's gonna be a list of things Concatenate it to split Okay, and this here will actually be You know the first value will be the one that it actually matches and we don't want those so we say split when a Con rest Okay, and let's see print passports Okay, right. Let's here we So print Split when Equal to the empty list illness Okay, run it all right, so it's Not accepted patterns in first rest. Okay. Yeah, okay, so I Think we have to say here Yeah, we have to say So we didn't we didn't count for the end right break con Excess of so first comma Rest It's gonna be your first This is gonna be first and rest We break it and we get something with the end we're gonna return that But if we we do get the empty list here So first and then We don't care We get the empty list here So it will always be the empty list right yeah But we don't care about that. So This is gonna be first comma empty list. Oh no That was an infinite loop Split when con the rest right and then Okay, so I think we should define this whole thing just like this Let's just say case break con fixes of Yeah, let's just do the entire thing here case Okay, so we're just doing this here Case break con of Right, so if we have First and then the rest then we will return First concatenate to split When con the rest Otherwise, we just return for first return list Getting first Okay, see and now we have This here So we get all the passwords Works great Okay, and then we're gonna join them. So first let's map Let's map words over them, right? let a P entries equals So we split these and then a Comp ps equals that is so let's map words over Over P Come up we map we map map words over P entries print comp Let's see what that does Okay, see and now we have all the words like this and then Okay, and then we will We will So we will join Or we will concat the lists After we map words over them. So what happens now? Okay, now we have this is an entire passport. Okay, this was so this is an entire Passport with all the words and then this is an entire okay good Okay, so now we've Now we've created the passports Okay, so So now we have all the nice passports. Okay, and now we will say This is the comp ps so over that So over those We will actually map map break equal equal so we want to break it at the At the at the these right So we get the kind of word Key key value pairs, right? So we map it at break And then we get this We get these key value pairs And now Yeah, okay, so now we have the key value pairs Let's just print the head here What is that the head is gonna be a list of key value pairs from one Passport but with colon and front, right? Okay, so Then we will just say so Google I think it's just from list, right? I think and then like ps equals map map dot from list over the comp ps So here we will print ps So here we just turn them into maps And they will be maps and then a So how do we map over the values in a map? Let's see We can probably use data strict right, but we don't we don't care adjust Up to the value. Okay, so Size combine difference intersection disjoint compose Yeah, we just want to map over the map. So let's map map Okay, no, no, we have a list of maps. Oh my god. Okay, so we will map dot map We want to drop the first element Drop one okay, and you see now a And then this will be Print ps and now we don't have the the annoying colon there. So this return ps All right, so we generated the passports and now we are supposed to Check if they are valid, right? so When are they valid? Well, all eight fields are present. So then we just check the We just check the size of the key set, right? How do we map over the keys, how do we just get the keys if I map keys, okay? Okay, so here we get the passports. So let's say Print map keys over the passports To be mapped on the keys Okay, and then So is valid is valid one. I'm gonna I'm just gonna call it. It's gonna be We're gonna apply it again later. So it's valid passports to boom It's valid one ps equals case length ps of So if it is eight then all eight attributes are present, right? If it is seven and a ps has the Let me just say member We just say CID Member It's gonna be mapped off member ps And this will be just, you know, check if Sid is a key If it is a key So this is gonna be not, right? So If Sid is not present, that's okay Otherwise seven is gonna be if Sid is there and there's something else missing. It's gonna be wrong So we don't even we don't even have to check, right, but this is I know it's not optimization, right? So if it's anything else, it's false Okay. Hey, so let's say map is valid I'm just writing it like this because I'm pretty sure in part two it's going to be It's going to be we're gonna have to check the values Okay, so let's see What is the length of? filter Filter id All right, that's two and for the first one You're supposed to say two Let's get the input and let's say Let's add a new file. Oh No, that's a new folder new file input and Let's see what happens Haskell syntax is kind of funny Come say that to my face Yeah, I mean You get used to it, right? And it's like it's whatever you get. It's whatever you're used to which is not funny, but this is I know I like it Like once you'd realize what's going on. It's quite nice So let's say This is just gonna be Let's just let's just copy paste here. We're not we're not gonna be maintaining this code Okay, and here we're gonna get the input and then we're gonna print is valid of passports by one of the passports 226 it says We're even printing it here You don't need that Let's see 226 you think it's correct I think it is It's correct. Yeah JavaScript and Python they're Very imperative Right, but here like the focus is on the expressions and not the statements But so it's only in do notations actually that we have statements. Otherwise, these are all just expressions That's where the syntax becomes a bit funny, but it makes it so that you can very easily Evaluate everything actually which is Which is pretty good Okay, let me see By the way, do you guys see the chat messages on stream because I I keep missing them. I hope you do All right, let's see here 226. Let's check if that's a great dancer Otherwise we have to see All right, you got the first one Now let's see if all our parsing work turned Was the right work Otherwise I mean because I assume we're gonna look into the values of the passport. All right So we fixed the system Airport security is talking about password with any valid data again. Okay. Oh, no Okay, so birth year four digits issue year four digits And at most 2020 expiration year height Hair color Eye color Passport ID Sid is ignored. Okay. All right, so now we're gonna write some validation code. Let's Okay, we're all required fields are both present and valid. Here are some example values. Here's some valid press ports Okay, so we still use the same input So, okay, so let's check it out. So there's only 226 valid passports So Let's just simplify it and say valid so let valid passports equal Filter is valid one passports Because they have to have all the fields But Sid is ignored So we can we can actually do this and we can then we could just say Length valid passports Yeah on it How fast is this actually? Okay, so it's How fast is just the four milliseconds. I mean, and we're parsing stuff and putting it into a map that is Quickest lightning. I mean, it's not nanoseconds. It's not like rust stuff where you can like I mean, but you know, it's Pretty good and you can see here that like three of those seconds were spent by the system Okay, so it's about 10 milliseconds. That's not bad And that's even like with the test. Okay, so let's so let's check Let's write here. Let's let's just copy pasted straight Have a bunch of things is valid Passport to Okay, so we're going to have all of these so beer is supposed to be four digits There's a valid fields Okay So here we're gonna we're gonna Let's just turn this into Let's let's do it. Okay, so it's valid fields. Okay, so let's say here Data valid passport Just so we can make use of some stuff valid Valid passport equals so we're gonna make a data constructor EP. So it's gonna have beer.  So these are all gonna be strings, right? But we just want to be able to refer to them easily in The validation checker. So, okay Because like we only have strings and we want to we don't want to assume What they are right ECL Because like, you know, one of the issues we might run into is that It is it just doesn't parse, right? So Okay, easy up string bid string We don't actually need so these are all valid passports This one till take a valid so to valid passport it takes a Map of string to strings and returns a valid Passport to valid passport Map equals so how do we look up in maps? Okay, we can just have yeah, okay, it's just bang And we know that all the elements will be there. So we will say this is a VP beer equals map. Let's just write here M beer is map beer here equals M here And now it's a here. So this will be Eight will be This will be CL will be CL And this is PID Okay, see Okay Okay. Yeah, this will be Map dot bang. Okay Now, let's see here. We will be Deriving show and then let valid passports is map to valid passports Of the filter so what is it complaining? To valid passports. Yeah, okay. Okay. Now you see now it the speed A lot slower, right? Because now we have to like look up in a map Can we fix that import data dot map dot strict Book what I did a month strict So if we assume we will read it, okay, I mean, it's still a lot very slow How about we I think I mean I think what happens here is that essentially It never kind of checks the values or anything once we're generating the passport So that's the thing that Haskell does a lot is that you know It like it just doesn't put the values in there It doesn't evaluate that statement if it doesn't need to Right. So but now we're actually evaluating the thing and then we need it, right? So it is It's quite good at what it does like I'd like these crazy behind-the-scenes optimizations that we're We're you know, it's just magic, but you know This is not too bad actually. So this is with With compilation. Let me see. Oh So it's still super fast. I'm sorry. We're just compiling more code. That's all To other passport. Okay. So let me say here is valid passport So this one will take a valid passport and it will say Now let's call this has valid fields We'll go to a bull Has valid fields and then here we can take VP and we can do this dot-dot thing and that will complain because We haven't enabled the extension for that. It's record wildcards Lang language. Let's actually Run this and then time this This is good. Okay, so now we have all of these fields in scope. Let's check it out Has valid fields. Okay where Okay, so now we're just gonna we're just gonna do a Kind of a case break. So we can do this. I think we're saying Maybe valid passport So let's say say here So let's say here. So we're gonna use a do syntax and then we'll just kind of Escape early if one of them is wrong. So do So we're gonna go guard. I think we need to import guard from God control that monad right? Yeah, so we will guard against a bad valid beer So here we will say, okay, let's just say this. So it's gonna be valid beer And valid beer is birth is your year. So that is so we have beer in scope. So we'll say if So it has to be a little bit for digits the length beer equals equals four and So we will read add int So now we read we need we need type Applications and like last time we didn't need it, but now it doesn't know that it should be an int If length beer equals four so then I equals a read so we will we will use this value here read add int beer and then in We will say beer larger than let me say I larger than equal to 1920 and I less than equal to 2002 Okay So and this is also a case of lazy evaluation, right? Here we can actually just say We could just we could just let this be in the computation and We say length beer equals equals Four and it will actually never evaluate This thing And they will won't actually evaluate the eye there unless it's needed, right? So We'll never get a purse error, which is cool Okay So so valid year Valid here, so we read the year and then we check if it is Four and that it is Larger than 2010 and that is less than 20 20. Okay a Same here, that's that replace Here with expiration year And let's make sure that we are actually seeing valid Yeah, they are also So the length has to be four digits at least 20 20 and less than 20 30. Okay Hide a number followed by either cm or inch, okay So we will flip it and reverse it Flip it reverse it Valid height equals case Reverse height of Okay, if it if the reverse ends in m C And then the rest are Then we We say Read at int So okay, we were we were actually I think we should we should read and like do a case on the read because if the if it doesn't parse That uh, that shouldn't we shouldn't don't want an error in that case We want to maybe So we read maybe okay, so we say So here we will just say case read maybe at int this every beer of But here we actually we do the case split before So we will say here We will say length beer was four and Like this So yeah, this is a nice thing right this case thing. It's just the It's just a expression right so we can actually just do this Use it as an expression um Then we read the maybe here. This is why and this was Sorry about this and we just wanted all to be correct Yeah, I wonder if there's like a Probably like a super efficient way to Like make all these edits Super fast, but feel like it Um Just I okay And this should be eight right Is it not a standard sit sex? For programming in haskell. I mean this is haskell syntax. There's nothing Nothing wrong here. I think I hope not Let me see Let's let's like let's so we let's let's actually Let's make this a bit nicer. Let's say check range So it takes an int and an int and a string So let's say check Year it takes an internet and a string Returns a bull So check year so we have the low and the high and the string so then we will say length l equals four and Then we do the case maybe read maybe of the string so Stir, so here we have length stir And stir and then we just check this is largely equal to the low and this is less than equal to the high and then Valid b.i.r equals check year What was the what was it for b.i.r? It's 1920 And no 2002 read like this so e.r as this is 2010 2020 right Valid E.i.r. This is going to be 2020 and 2030 Like e.i.r. Yeah, this is too much copy paste, right? All right uh Yeah, I mean the white space is significant, right? So So if I do it wrong Eh, so yeah, the question is like is the white space of entertainment significant and it's like if I do it wrong It will complain, right if I don't have the correct syntax or the White space and I think this is a This is quite standard. I think I mean, I don't know what's I've been I've been working on ghc for a bit and This isn't so far away from ghc. So I'm Yeah, I don't know We could do this thing here with an f map But that seems a bit overboard actually Yeah, okay. So check year. So if it is 70 meters And then we say case Read maybe at hint of Of r of So if it's just Uh Just just a height Then we want it to be 150 They want to be height larger than equal to 150 and height less than equal to one are you three Uh, otherwise it's just false Uh case read maybe in cannot What is it saying this can't cannot apply type expression type t1 to a visible type argument in But now it doesn't know the type, right? Oh I think it has to be Now it's complaining that Okay, read maybe isn't defined. Yeah, okay. This is in data text dot read really should be imported by default, right? Okay, read maybe is now here and then Now it just works. Why it should complain. I think this is also a ghcid error. It's giving the Wrong complaint here All right, uh, let's actually just copy paste this. I don't want to we don't we're just gonna run it once we don't need We don't need to Abstract it if it's an inches then it should be 76 Oh, shit, we need to reverse the string back The numbers this could have failed us Okay, um Okay, uh, so and if it tends with anything else, it's just false Okay, so we need to have valid height as well Okay So for So, okay hair color followed by exactly six characters zero to nine or a to f Okay, a hair color. Let's do that. Uh, let's Let's copy this in like this So we have it by hand, right? Okay, so valid ACL So here we have to say I think we can just say is alpha numeric. Oh, we're not so it's supposed to be a to f Google So we go to data dot char I think we can also just you know, we can do Let's check it out gti Uh Can we do zero to nine? Wow, can we do To f Okay, so We can just do zero to nine plus a to f I mean, this should optionally optimally be like a regex, but then The like the Haskell doesn't have a standard regex library. So it's a bit of a pain So let's just use this. Um Valid ACL Uh, so case ACL um So if it starts with something here And so, okay first First of all We want a Length ACL equal to Seven Can I do this? What's a good planning about? Parts are an input valid ACL. Oh, yeah, I think like this Okay, uh I yeah, okay, let's not do it this way. Let's just say here case ACL of Okay, so let's say if Let's say length ACL equals seven and Okay, so ACL Of so it has to start with the Oh hashtag for a bang or And then all of these have to be set. Okay, so let's say here, uh ACL Chars equals Now let's use a set So we use a set and then we do Um, let's let's do this on the top field here Valid ACL Chars Should be set dot from list On list a So it's going to be zero to nine Uh plus a to f Valid ACL Chars equal they are this is a set of characters So, okay, so if it starts Then we say I think there's a function called And And it just checks that all of them are true So we say map We will map the set dot member Uh, so how does what is set dot member? He import data dot set Set dot member Oh, oh no, it's type member Okay, so it takes in an a and it takes a set. So this would be valid ACL charts, what would you call? Valid ACL charts Let's call them something else Valid ACL charts, uh, let's see set member valid ACL Chars Uh to R and then we say and Okay, this should work Uh, so we check that the first one is hashtag Followed by exactly six characters zero to nine or a to f So and then we just check that all of the characters are valid, right? Okay, uh, if this is wrong, this is gonna be this is gonna be so painful to Debug I hope the demo input has Like a good coverage because uh Okay, so valid ACL, okay Let's just write the other ones here Okay, we don't check valid seed because it is ignored So let's just write here. So we have valid height And we have valid ACL and valid ACL and valid pit So let's just add the list together It'll all return to the same valid ACL valid ACL And valid So that's seven fields That should be exactly uh, right ones Uh height Valid icolor exactly one of am blue burn gray gernson, okay So let's just write that right Let's just let's do this in a set this is gonna be a set not member of uh, what did they say? Am blue burn gray hazel And then we just check here if valid ACL is a member of valid ACL else Easy enough Passport ID is a nine digit number including leading zeros So let me just say pit The length of pit equals nine So here we can use the data dot chart is We can look at data dot char which has a bunch of these Is control is space is alpha is alpha num is print is digit I mean we could do it with the set membership thing But this is also just so it should be To be nine and let's see there's also the all function. I think Yeah, so this takes in a checks that whether that all are something So it's basically an and over first you map and then you do and So then we could just say here That all should be a set member of valid charts and here we can say that All is digit Pit okay Then let's check it out length valid passports length Filter has valid fields passports. Okay, it says 160 of them are valid How many of them how many of them are Right, so let's filter map is valid Let's make the Let's make the test passport also and then How many are length has valid fields test Okay, so it says the two should be valid I'm being rated That's cool Okay, two should be valid Let us see What they said So here are some valid passports. Okay invalid passports Let's see Okay, let's simplify the thing here so Let's see. Let's get the passports Let's dump these into it valid pass is In these are gonna be invalid These are gonna be valid. Okay. Hey, let's check it out Wow, there's so much happening in the chat the most Active chat I've ever seen And none of them are asking questions about Haskell Anyway, yeah, it is a nice keyboard. I got it like two weeks ago. Um It came in like Danish colors, but then I made it Icelandic because That's where it's at Uh Haskell is a a programming language Uh That's a functional. It's a lazy functional programming language Uh Which are urges It's my favorite language. I've uh, I worked on the compiler a bit So I I know like the a bit of the ins and outs. Um And it's a bit different from like the usual imperative languages In that, you know, everything is expressions So it's uh, it's It's fun. Yeah Okay, let me let me let me extract this away here. Um Uh all valid So that's gonna take a a file path and that's gonna return me a list of booleans Uh all valid So and then I'm gonna say just like this here And then we're gonna write IO And uh, we're gonna do We're not even gonna re rename it, right? We're just going to Do this And then we're gonna We're just gonna we're just gonna print. Yeah, okay. We're just gonna print. Uh We're going to print and we're just gonna Return dollar map Okay, and these are going to be applied to imp the name of the file and Let's see Okay, now it's complaining here that this is wrong. Oh, yeah, I Okay, let's let's look at the Let's not look at the input anymore. Let's look at all valid Of test input Let's uh print Into print We're going to check the the the In valid pass and we're going to check the valid pass compile so Oh, it Oh, yeah, it filters it filters out everyone. Okay, so so it says that all of the valid ones are Is it saying that? Is it is it like the exact opposite? Did I okay the okay, sorry. I mean I named the files wrong. So this is These are supposed to be the invalid passwords Let's copy this and let's see These are supposed to be the invalid ones Yeah, exactly. And I so I just I just made the files wrong oops Be careful kids when you're naming your files Okay valid invalid All right, we get four false Invalids we get four correct and let's Then let's just run it. Let's just see if it works Um, so we get the input we print the length of valid passports and then we print the length of Invalid passports And it says 160 so Let's just check the oracle 160 All right, we got the right answer Whoop whoo Uh, yeah, okay. We spent like so much time on on All of this stuff. We could have just run it immediately but, um Yeah, this is just a very kind of very crude Sort of way to do it, you know, I'm just parsing the strings And we're just looking at everything directly, you know, there's no trick Involved here. Okay. I mean we choose the right data structures. We choose sets and then we choose maps but I don't know like our people Because you know, I think so so some years, um The the advent of code like you you have to find the trick. Otherwise, it's going to be super hard and super long But here we just kind of You know, we the first one was just very simple. I just just check if it had eight fields. Okay We could have gone through it and checked if it had exactly the fields, but we just we didn't even care, right? Um Then we just like kind of mapped it to a passport and like we wouldn't we didn't have to do this actually it was just so that we We could use the values directly here um And like, you know, we didn't even have to do that so Yeah, we could have just parsed it On the fly but like we want it at least something it would be nice, you know, if he had a deriving function that took map of string to strings and a record that has only string fields and could kind of punt on it. Um We could probably have done this with like asin or something like that because like this would have been a adjacent data structure um And then, you know, we just made these sets here We kind of we made them outside. I think ghc would have lifted these Out of the has valid fields function I think because I have written code where I am like going into expressions and it's it's very good at lifting things. What does lifting mean? Well, it would essentially just You'll take this definition that doesn't depend on any argument And it would lift it all the way Here to the top level so it doesn't need to be evaluated um Yeah, I mean it only needs to be evaluated every now and then Okay, and yeah, I mean this this is just copy paste. We could have we could have said Could have said something else, right? We could have said Uh, let's just make it a bit nicer, right? We have we have some time Uh Check r equals what is oh no Check r is We could have done something like this, right just h Jack uh all lhr and then And then here we would have said, uh So Right, uh This will be low and high and then we would have said height is larger than the low and the height is larger than the high And then we would have said here, um Wait, what is that's not? check 59 76 r and then this check 150 193 r So we made that nicer, uh, so this is just less copy paste and Yeah Here also, you know, yeah, I you know all of these are kind of They just do what they're supposed to do, um I mean, yeah, this this could have been nicer. I mean we could have written this all valid right away, but Um, you know, we were just doing it as we went along, right? We didn't actually need anything more here Uh, oh no my playlist ended For some reason Yeah, this is uh harris heller the lo-fi christmas playlist It's quite nice to have underground. I like the lo-fi music when I'm programming. It's just It's not too distracting and uh It's yeah, it's nice Nice and cozy Okay. Yeah, so that's it for today Let's just make sure that it still works Yeah, it still works And uh, I'll be back tomorrow with day five so check Check it out. Um, so it's gonna be the same time tomorrow, but let's say I started at like Six o'clock european time, so I'll start I'll start around then and as we see like I've usually been taking like 40 minutes to do these things this time it was It's been an hour 15 minutes hour 20 minutes. So Uh, these are getting harder and harder, but you know the harder it gets the The more fun they get um so check it out tomorrow in case you're interested and uh I have a playlist on youtube where I've uploaded like the episodes so far and uh I make sure that the Um, I fix the audio essentially in in the videos. I've had some trouble with the audio, but I think I think it sounds okay now I I put in a bunch of compression and The the issue was like if I was talking like this and I was talking like this The difference in audio would be way too much, but Yeah, all right. So tune in tomorrow for more haskell programming and uh, yeah, you want to follow and uh If you're watching this on youtube, you know Like and subscribe. That's what that's what I'm supposed to say, right? Let me check. Let me do one more thing No, okay. I wanted to like map this map these things over, but I I don't think this gets much shorter. We could Could exchange this like with an f-map but But you know and then like f-map this check over read maybe Because we're also doing that here But yeah, I think this is quite good. All right. See you all tomorrow