Hello and welcome. My name is Shannon Kemp and I'm the Chief Digital Manager of DataVersity. We would like to thank you for joining this DataVersity webinar which today is a case study Gordon Cruz delivers fresh data to the cloud sponsored by Click. Just a couple of points to get us started due to the large number of people that attend these sessions you will be muted during the webinar. For questions we will be collecting them by the Q&A panel or if you'd like to tweet we encourage you to share highlights or questions via Twitter using hashtag DataVersity. If you'd like to chat with us or with each other we certainly encourage you to do so. And just to note that it defaults the chat defaults to all panelists so you can switch it to all panelists and all attendees so you can chat with each other. And to open the Q&A or the chat panel you can find those icons in the bottom middle of your screen for those features. And as always we will send a follow-up email within two business days containing links to the slides, the recording of the session and additional information requested throughout the webinar. And to introduce to you our team of speakers for today, Kyle Partlow, Tom Majeski and Adam Mayer and Eric Thornton, sorry you guys I'm getting tongue twisted on your names. Kyle has led Gordon Food Services data engineering team for the past two and a half years as part of the organizational goal of becoming an insights driven organization. Kyle's team set out to build their revolutionary data platform to enable access and analysis of all enterprise data. Tom is an experienced technology leader with a broad range of experiences with recent interests including cloud and data. Three years ago he led an effort to research and develop a successful operational metrics concept hosted on G, Gordon GCP utilizing quick replicate. Eric is the North American head SAP customer engineering at Google. He has 20 years of experience in retail tech and SAP and varying formats. He spent nearly 15 years with SAP focusing on technology, retail and account management. And Adam is on the global product marketing team at click covering the entire click product portfolio. He is responsible for delivering the company's Internet of Things and GDPR go to market strategy with a strong technical background and computing underpinned by an extensive engineering perspective. Adam is an avid follower of new technology and holds a deep fascination of all things IOT, particularly on the data analytics side. And with that, I will turn the floor over to Adam to get us started hello and welcome. Hi, thank you very much for that great introductions Shannon so good morning good afternoon and good evening, wherever you are. So data is the lifeblood of any organization and now more than ever it's really important to be able to keep that data circulating so then you can take important decisions on the most freshest data available. So as Shannon said I'm Adam from click and just to give you a quick overview of today's session I'm going to talk to you about the challenges in moving data out into cloud and particularly how you can overcome them and how we can help you get data into Google cloud specifically and then I'll hand over to Eric who will tell you more about all the wonderful services that are available in Google to help you get more value from the data. And then we are going to hear from two of the greatest minds in Gordon food service on why they chose click and Google to really enable and lead their enterprise data strategy. And all the great improvements that they made from that and then we'll wrap up and open the floor up to questions. So moving data to the cloud is certainly not without its challenges, and there are a number of ways that you can go to build out your data pipelines. One way is the traditional ETL approach the extract transform and load, and that commands quite a high skill set involves a lot of manual scripting manual coding, and it can take a lot of time to deliver. And it can be prone to human error over time because of this it doesn't tend to scale well particularly as in when things change. So not just changes at the source of the data but also as you start and need to add new data sources. So it can work quite well for you know relational databases with particularly when the target is in the same format as the source that you want to replicate, but it does get more complex when you want to add more data sources, especially those mission critical systems, such as your mainframes oracle and SAP. And over time, that can start becoming prone to brittleness. So this can result in quite frustrated business users from slow delivery of data, and essentially creating a workplace that can be reported on out of date, out of date data, worse inaccurate and that can lead to untrusted data. So this is what's driving many organizations now to modernize their data integration approaches and shift towards more of a zero coding environment as much as they can do, which we've got on the right hand side, this can really help you to simplify your most complex processes, particularly through automation that can really help to remove the human error and start providing that scalability that you're looking for and this can ultimately start to deliver more timely, accurate and therefore trusted data into your your data pipelines, even you keep in mind that you're keeping the lights on, and then it allows you to focus on those much higher value tasks that are always going to keep on coming in. So in terms of the key drivers, in terms of what's driving modernization and particularly why modernize your data integration approach. Well, first off, starting from the left hand side moving across the demand for real time is ever come in and it's it's not only coming from within your organization, but probably outside as well we're all used to instant results these days. And if particularly if the pandemic has taught us anything it's that need for quick change to pivot on the dime based on what's happening now as opposed to yesterday. So the more freshest data that you can be working on the near real time as possible, the better data decisions that you can make. And then next we're living in a hybrid world there's a need to seamlessly move real time data between many heterogeneous systems. Many of those are on premise and then there's a need to be able to connect those on premise systems out into your cloud environments as you start to modernize and even move data in between the cloud as well. And then there's a need as well to be able to want to do that as automated as possible, that can really help take the heavy lifting out of data ingestion, the things like your mappings and creating data models particularly for your data warehouse and data updates out in the cloud as well. And you want to do that as scalable at more at scale as well and start to embrace the fast pace and ever changing tech landscape. So it's quite important then to work with partners who have deep expertise in data in data integration, particularly in the cloud to the cloud, as well as modernizing out in the cloud as well, when it comes to your important data and analytics initiatives. And this is really where where click comes in. I'll just give you a very quick overview here many of you may know click as an analytics bi company. However, we are a data and analytics company now, we're able to offer a truly independent data integration solutions as well. We have lots of acquisitions of podium, as well as more recently, a tunity back in 2019 will join in the click family. So this means that now click is really the only vendor to deliver a truly open end to end platform that enables that data value chain that you see at the top there so the ability to free and find your data, and then to understand it and more importantly take action on it as well. So on the left hand side. We have our data integration platform. This is really what allows your DBAs and your data engineers to be able to generate deliver refine and merge data from all sources on the fly and then land that data where the business needs it to be. And that the very foundation of this data integration platform is our log based change data capture technology. This is the CDC streaming part that you see there. And that's what delivers real time data from a wide breadth of sources that you can see depicted from the bottom. This is your databases, it can be mainframes on premise systems cloud systems oracle databases and SAP systems, wherever they are and then allow you to deliver that out to multiple targets. And then we also have two other solutions there data warehouse automation and data lake creation. This is all about automating the process of managing and creating data warehouses and data lakes. So the click data integration platform can support replication and migration of data out into a cloud environment like Google, and then you can use it with any bi tool of your choice so it doesn't have to be click. Of course, we believe that click delivers the best analytical experience and and on the right hand side you see that full range of analytical capabilities from self service analytics to reporting and alerting. And in the middle just to call out there we've got our cataloging capability. This really sits between the two portfolios and it actually bridges the world between it and the business, and it not only allows you to connect to data from the data integration capabilities. You can also bring in the source points directly and then feed that into your analytics environment, whatever it is. So, this flexibility and adaptability is ultimately the basis for why our customers and tech vendors are like a consistently choosing click for their strategic partner of choice. Now today we're going to focus in on the data integration side specifically that CDC streaming solution. And incidentally that consists of two products, click replicate and click enterprise manager, but just focus in on the click replicate product. Now this is formerly known as a tuner to replicate coming from the acquisition. And I just want to have a kind of pop the hood if you like underneath the architecture, and we just briefly walk through how it works because this is quite paramount in what Gordon foods are going to be talking about. So click replicates of web based data replication tool typically configured as a middle tier server. And then what you can see the door data sources on the left hand side, and these can be on premise or out in the cloud. And click replicate allows you to capture data from those sources and then automatically apply lightweight transformations and things like filter in in flight before then propagating the data to the wide range of targets that we can support depicted on the right hand side. Now again, that could be on premise but most commonly now it's moving data out into the cloud like Google for data warehouses data lakes, as well as streaming messaging platforms as well like Kafka. So we click replicate you can seamlessly replicate data from your full load or batch load as it's known and then it will automatically switch to capturing and replicating just the changes in the data source as and when they occur. And we do this wherever possible with allowing you to use agent this and log based data capture. So this means ultimately it's minimal impact on your source systems. And from a transformation perspective we do as much as we can in memory. So that be things like standardized in particularly date formats for example, filtering capabilities. For example, filtering multiple years down to a single year. You can only move in the most recent transactions or even regions, or even things like obfuscating sensitive data such as PII during the transformations as well. And just incidentally, more additional more complex transformations and also be applied further upstream using the rest of the click data into by integration platform if you wish. And such as the automatic creation of data marks and data vaults for your data warehouse and data lake creation. And that's by taking advantage of the two other solutions I mentioned earlier data warehouse automation and data lake creation. And they consist of other products as well. But going back to click replicate there's a lot of flexibility in the data flows that you can configure. So you could replicate data from one source out to many targets or even vice versa. And you can migrate data from multiple disparate systems out into the cloud. And just before I move on a key point about that persistence store that you can see at the bottom of this diagram is this is not about storing the data that's actually being replicate. It's storing the configuration and the state so that each replication source and target that you can figure in click replicacy is seen as a single task, and it's the task configuration that we store here is as you metadata fields, and the tables to this selected as well as the transformations that you want to apply. Save the state of the last replicate task here as well so you can pick up where we left off in case of any interrupt interruptions such as arrows or pauses in the replication tasks. So you can think of that persistence store like storing bookmarks. Now order this can be done at scale and we have many customers running hundreds of tasks out in production systems automating their data pipelines just like you're going to hear about the modern food service, and we will hand over to them very shortly. But before I do. This is just a very quick snapshot of all of the heterogeneous sources that we can support here on the left hand side well over 40 n points, and that is growing and we maintain that, you know, keeping the versions up and all this kind of stuff. And then on the right hand side, you can see we support propagating to full Google environment as well so you can take full advantage of all those many Google services and deliver real time data with that best in class CD, CDC technology has changed data capture technology. So this is a good point then to hand over to Eric we're proud to be a partner of Google and you can see we've built out that extensive integration out into Google to solve those challenging data integration challenges that we discussed so Eric over to you. Great. Thank you so much Adam. And again, like you I'm thrilled to be here. I'm really looking forward to Tom and Kyle and their and their content. I do want to take a minute though and talk a little bit about what Google cloud is doing as a hyperscaler supporting the needs of the enterprise. I run a group of people here in North America focused exclusively on SAP, but most often we're talking to SAP customers about migration, and then how to handle data. So this is just a wonderful opportunity to talk about what our customers experiencing, and also give some background on why data is so important to us. You can see big data is in our DNA. We have nine products with over a billion users that's billion with a B so when you think about YouTube and Gmail and maps and cloud and photos and Android and Chrome play store etc. We definitely have scaled over the last few decades and we've maintained commitment to managing that data serving it up very quickly and then preserving the security of it. So those are the examples of some scalability in the past. Let's look at some of the innovations which I'm really excited to be a part of. In addition to my background with SAP I did spend some time and big data with the Duke, as well as AI and ML. And so I certainly used Kubernetes I use some of the other innovation opportunity. So that's an exciting opportunity and a great journey that we like to take with our customers. Let's go ahead and go to the next slide and talk about our mission statement. It is important to focus on this because it keeps us true to our North Star what we want to achieve is accelerating organizations and their ability to transform through data, you can see how it highlights there. We want to provide innovation and the infrastructure and platform and a big focus lately on industry solutions. We're spending a lot of time focusing on the industry specific needs of our customers and and helping enable the business processes that are unique to those to those industries. So that is commitment that we make and is is is carried throughout our deliverables to customers and partners. So thinking about what SAP does with Google Cloud on the next slide. What we see many SAP customers doing is migrating into the cloud moving into a hyper scalar taking that historically on premise resident application, which really is the nerve center of their organization. In order for them to achieve success there they need, they need security, they need to manage a hybrid landscape. They certainly want to manage that with worse were appropriate with serverless. And like I mentioned before, in addition to migration they want to start taking advantage of intelligence, artificial intelligence machine learning big data visibility KPIs metrics, you know all the things that are helping organizations drive their success in the marketplace. Many times we'll find their roots back in an SAP system, but we also make sure have to make sure that we're accommodating non SAP and unstructured contents we'll talk about that in a few minutes but I do want to emphasize why Google Cloud is so important for for SAP customers who are looking at migration into a HANA system or moving into a PCC on HANA system so we're very proud of the work we've done here and we're we're very happy to see customers benefiting from that as they as they go live with their SAP system and Google Cloud. Let's look at the next slide then and think about where the data resides right so as we see customers with transactional systems, and as we see SAP landscapes that are driving innovation and they're driving visibility etc. There's a need for other unstructured content, things like social media news events, whether all those different data signals that can support the specific decision that a customer needs to make to change a business process, or to meet the needs of their specific industry, you know, retailers reacting to the demands of a, of a consumer or changing the supply chain rooting based on weather events. Google BigQuery gives us a fully managed cost effective highly scalable enterprise data warehouse. So great opportunity to put a data lake right next to your SAP BW system. Now what we're excited about is the improvements here. Back in 2016, if you queried a petabyte in BigQuery, you would get results in about four minutes, not horrible but certainly not responsive and certainly not meeting the needs of a business professional wanting to make a decision. When this paper was written that I'm referring to back in 2019, the results were down to 12 seconds and now we're down to four seconds querying a petabyte gets you information at your fingertips, and it also opens the up, up the opportunity to blend that in with algorithms and predictive responses, fully enabling a business professional to get full information on making a business decision. So we're very proud of the work that the customers are doing with BigQuery, and and seeing how they can innovate based on that. Let's move to the next slide then and think about how in conjunction with the big data we can start applying algorithms. Google has long been focused on artificial intelligence, and you know many of us use this in the language section there in the center. If you've ever taken a foreign trip, you may have downloaded the Google translate app. And what's brilliant about that is you can speak in English convert to a different language and carry on a conversation really innovation and very, very compelling and very powerful. That's using a very large trained deep neural network. So if you're running in the Google cloud that you can tap into through API's that continues through natural language and translation and auto ML. That's just from the language side over on the right then with conversation you can take that and start to work on text to speech or working in the dialogue flow component. The end result is being able to automate communication and being able to automate process processes that are so important for keeping consumers engaged and providing feedback on systems. What's really exciting on the left there is the site side of things you know visual intelligence and auto ML vision auto ML video. The vision API is very interesting and it's opened up a lot of doors for us and for customers who are applying AI algorithms to business processes. Let's go ahead and take a look at the next slide which is a drill down into vision API. In our manufacturing segments we see aerial inspections and asset management, being able to send a drone up to a very remote system or a very out of reach system and be able to do inspections right or in healthcare, providing a second opinion on radiography or x-rays or here you can see a cordia scan with healthcare and medical image analysis there on the right hand side. Really exciting ways to tap into that API and that that artificial intelligence algorithm and combine that with the needs of a business running their organization and being able to track what the customers are looking for what consumers are demanding what supply chain roots are needing to be adjusted with and really exciting use of the vision API. So one example of what consumers are doing in the marketplace and what customers are doing to adapt to their specific needs with the Google cloud environment. So let's move to the next slide then and talk about where this all connects. We're going to hear from Tom and Kyle shortly. They've got a lot of different back end systems including SAP. They're using some data lakes. They're applying some machine learning where appropriate, but at the end of the day this is what we see customers take as a model to pursue. They're migrating their SAP and integrating that in with data lakes and then applying machine learning and artificial intelligence to get to that end result. They have a very proactive answer to a specific business question that helps them innovate and helps them drive their business forward. And again that's petabyte scale data analytics accessible machine learning, and it's all available at your fingertips with this with this Google cloud environment. Looking at the next slide then you can see most customers, specifically SAP customers have said they do have standard pains they have challenges they're working with. And their most, the most surveyed result in that question on what is their top analytics pain is data integration and how can we resolve that so when we work with customers we spent a lot of time looking at what their landscape definition is what their plans are for migration, how they're going to achieve the needs that they have, and integration is a top part of that discussion. So having gone through the background of what Google can provide as an infrastructure to this entire conversation. I'm pleased to be able to transition then over to Kyle and Tom, who can take us through the next steps and talking about what Gordon food service has done in this environment. So with that, I'll send it over to you guys thank you very much. And thank you to our partners for providing the opportunity to share the GFS story here today. As a slide says, I'm Tom Majewski leader for data and application shared service groups here at Gordon food service. I'll be joined later in the presentation by my colleague Kyle part low, but first, I'll be taking you through a little background on our organization in the history of the technology relative to our discussion today and then Kyle will drive through how quick and they've helped realize our data strategy over the last couple of years. Why don't we jump ahead to slides. Perfect. Before we dive in, let me get you give you just a quick background about how Gordon food service. So we're in North American food service distributor. Our roots trace back over 100 years to a horse and buggy, peddling butter and eggs technology back then was pretty basic. Geographically, we operate nationally across Canada, and primarily east of the Mississippi River in the US wrapping around through Texas in the south. Gordon food service industry segments throughout our customer base, including things like education, healthcare, independent operators, chains, entertainment, hospitality and several more. If I call your attention to the images at the bottom, you might note that our industry Gordon food service included has been embracing change and innovating solutions for decades. We started with a horse and buggy and there you can see a gas pickup truck, and eventually we're on to automated distributions centers servicing multi state regions. Data is just one of the next opportunities for us. Could we move on to the next slide. Prior to 2005, the majority of our data resided on our mainframe. 1510 years later, it had spread throughout hundreds of Oracle and SQL server databases within our data center, as well as too many SAS offerings out in the cloud. Managing the sprawl had become a challenge for data architecture as well as a technical opportunity when needing access to disparate data sources for reporting or or other other other reasons. Fortunately, we had the foresight to recognize a coming need for data consolidation to support the growing demand wave for analytics. Can we get the next slide. Let's talk for a moment about the needs as we saw them for managing data replication in 2015. We had a lot of custom replication scripts that that moved data from one place to another to try to bridge the gap that we had with either operational or reporting systems that had needs for data that they had no easy way to get access to. And as the many sources of data grew so also grew the challenge to try to bridge that gap. We knew that we needed a solution that could be managed within our existing data services team that was composed of DBAs and data architects. We did not want to grow the size of our staff to manage more products. So there were some things that were important to us. We believe that first an intuitive user interface that abstracted the complexity would be foundational. When we looked at different tasks and volume of replicated data would grow. We felt that access to the platforms metadata for incorporation into our own repository would be essential to decision making and answering questions in an efficient manner. While we didn't know for sure where our journey would lead. We believe that the platform would need to have a vision for supporting just about any data endpoint we could ask for. We were concerned about the possibility of creating performance impacts within the source data stores. So we needed a minimal footprint, which included log based replication versus the use of triggers. We also knew that in order to make the platform fit into our own internal processes and controls, we would need access to an API. Next slide. In 2016, our research had focused in and we were ready to select a partner.  We found that the base replication capabilities were essentially equal between replicate and option B. However, attunity now click representative vision and a consolidated product management direction that align with our vision for how GFS would like to manage data replication. Attunity was ultimately selected and the partnership began with an initial goal to simplify and increase the reliability of that on premise oracle database replication that I spoke about. We felt that would be the fastest way to show immediate value and also build muscle with endpoints we were already familiar with other endpoints such as SQL server postgres and eventually GCP services and SAP HANA were prioritized later. Along the way, we developed a number of tools and integrations as well as built knowledge necessary with which to manage a diverse installation. The API available through QEM was helpful in building a lot of extension services. One of the first things that that that we did was build a process chain or a password change process for endpoint service accounts. Later as non administrative teams began creating tasks that became more essential to have a process that permitted promotion of those tasks through pre production environments, similar to change deployment for software. The API was helpful here as well. We also built a metadata extraction process that gives us visibility to objects and tasks, and ultimately the overall data lineage through through our replication tasks. Next slide please. So let's take a look at an early timeline for the GFS adoption of replication technology. In 2015, we began our research into data replication vendors. And in 2016 we began our partnership with attunity, as well as replacing our existing couple of custom replication processes. In 2017, we successfully use replicate to migrate databases from our legacy spark salares platform to x86 Linux platforms minimizing the downtime of dependent business applications during that process. In 2018, we executed a GCP based serum insights initiative using replicate to source the data from many data sources within the GFS data center and move it out to cloud sequel, as well as Kafka free vent triggers. This consolidated data source and GCP made it really easy to perform operational analytics that could alert sales to opportunities right within their serum platform, or provide information not previously integrated into an easily accessible location. In 2019 we began sourcing replicate data from our SAP S for Hannah instance, which is incidentally living out in Google. You will see more of this in the next half of the presentation. There'll be a number of options available for your use case in your organization will need to understand the trade offs of beach. They're well versed. They helped us a lot and they'd be able to help you as well. Now I'm going to turn things over to Kyle, who will take you through the next part of the GFS story. Kyle. Yeah, just confirm everybody can hear me. Yep, we can hear you go. Excellent. Hey, thanks, Tom. Thanks. Thanks Google and click for hosting us. So, like Tom mentioned, I'm kind of picking up where he left off in regards to our attunity slash GCP journey at Gordon food service. Two and a half years ago, we underwent our kind of data transformation strategy and one of the first decisions we needed to make was who was going to be our cloud provider. And I think it's pretty obvious the three organizations that we evaluated. I think we pretty quickly eliminated AWS because who doesn't compute with AWS nowadays. And then when it came down to it, we believed in five years from when we did the evaluation that 90% of what cloud providers cloud providers provide will be table stakes. Everybody can do it. Everybody can do it at the same speed but where organ where the cloud providers will differentiate themselves is how well they can do machine learning and advanced analytics and and we saw what Google had had was far and what other providers had and our hypothesis has has just became more and more true over the two and a half years we've experienced with them. So, first decision was to go with the Google cloud platform shortly after that. We started looking with with Tom's team and how we could leverage the attunity replicate tool to get data from the from our systems into the data lake. As Tom mentioned, we have a long history of applications, both fortunately and unfortunately we are not digitally native so we have a lot of systems that the only way to get data is through database replication as much as we'd prefer to get it through API's or other means. So, through the evaluation with Tom's team on attunity we chose to go forward by using those two technologies to kind of kick off our data transformation strategy. So, we spun up the team we started with two and we've only grown since I'm going to go over in 2020 when we turned on our, our attunity replicate pipeline, which I'll talk more about in the next slide. Shortly after that in late 2020 we in we ingested our first SAP table, which was a marquee day for us, we had been doing Oracle and SQL server for a long time. We chose to go with Google instead of SAP BW, just to, for the same reason we wanted to leverage the best in breathing machine learning and AI, and we wanted all of our data to be in the same place and we didn't want to spread it across to technologies. And then last I'll go over some of where we're at today with this pipeline. So, in advance slide. Excellent. So, I'm going to start with where we now, and then I'm going to go into how we got here. So, the picture some of you may be wondering what that is. Personally, I have never actually seen Willy Wonka, but all of my data engineers apparently it's one of their favorite movies. If you have seen it, I guess there's this guy named Augustus Gloop, who eats a lot I think is probably the easiest way to describe it and the. So when we got around to naming this attunity replicate GCP data pipeline that we built. The team came up with Willy Wonka and named it Gloop. And surprisingly, it has gotten a lot of traction all the way up the organization and gives the team a little bit of leeway to make some fun names and we actually coincidentally have some some scripts that we run to kind of contain our group pipeline, and they call those the, I think the Willy Wonkas are, I don't have the name quite right. The Oompa Loompas I'm sorry. And again, spare me because I've never actually seen Willy Wonka but they call them the Oompa Loompas because they help Augustus eat all of the data. So, where are we now we can now replicate data from any on-prem database both HANA or all HANA Oracle SQL server into the data lake which is hosted in GCP and BigQuery. And it's important to note that we went after structured data first, which is why we're able to leverage BigQuery as we get into more unstructured about the leverage other technologies like GCS in GCP. One of the most exciting things is that we have, and this isn't completely true but for the most part we have sub second latency from the time the change happens in the source database that changes then reflected in the data lake. We also automatically handle scheme updates, which was a big deal for us. Like Tom mentioned, we have hundreds if not thousands of databases. If we had to take down, adjust the schema manually and then spin it back up. We would have a team of data engineers only doing that. So we, our scheme updates happen all the time and our pipeline automatically processes those replicates sends that event. We make the change in BigQuery and then data continues to flow from there. We also have a pipeline that self healing and can automatically scale with demand using data flow as well as pub sub. And so I can talk a little bit more about some of the statistics that we have around just how little it takes us to maintain a solution that provides so much value to the organization. The other key component here is that we have all changed data records for all databases. So what that means is that we have a record for every time a row changes. So we can tell you the exact state of the database at any point in time since we turned on a tunity replicate for that particular table. We can also tell you for a database that is very update heavy. We can tell you how often an attribute of an item changes, even if the application itself doesn't store the audit records for all of those records. Advanced next slide please. Okay, so this is a slide that has gotten a lot of traction at GFS which is kind of talking about our data modernization strategy so on the left, I think anybody that is in a non digitally native organization can relate to this kind of spider web of integrations operational data store sharing data across a bunch of different applications and all these tertiary places outside like our EDW and vendors and customers on that need data which has put us in a position where it's, we don't even know what we have we don't know who's calling what who's using what data. We're moving towards is kind of this data hub data platform concept and so the idea is we have all of these source systems that we have internally, as well as external systems that we use to run our business, all of that data funnels into one place through data pipelines through APIs through files, and then we provide that data to downstream consumers through our API layer it could be other internal applications customers vendors our North American EDW and even third party consumers. And so I'm going to specifically talk about this presentation of a piece that I drew a red dotted box around and where are we going. And then certainly would take it a whole another hour to talk about the things on the bottom half of this chart advanced next slide please. Okay, so group is born. We actually worked. We had multiple versions of this. And I tell you what I'm super glad that we kept going and kept innovating and kept iterating on this solution. So first, we Tom's team with all the legwork that they've done the three years prior to the data strategy kind of kicking up an earnest had already had click. At the time attunity replicate connected to many of our on prem databases so thankfully that that work had been mostly done so during the build of this, not only were we trying to build this we were also trying to stand up our platform introduced governance train up engineers that hadn't traditionally worked in GCP, as well as build the team so we also use a tool called Kafka in order to connect with pub sub. But so essentially the attunity replicate monitors the database logs. Every time a change happens it sends it through Kafka and then subsequently pub sub. We use data flow to automatically process that. And one of my favorite dashboards is our data flow monitoring dashboard in GCP that basically monitors how many Tickets are in the pub sub topic, and then automatically can scale the data flow job accordingly, and it does it without us knowing. So it's really cool to wake up look at that dashboard and see that we got a bunch of changes last night data flow calmly scaled up calmly scaled down and then moved on to the next day. We then store all of the change records in big query and I kind of define the change records in the previous one. And this is where the data lake starts for us. We also have views that we set on top of the change data logs, which basically if you query the view it will just give you the current version of the database and we also have the ability to back date that view, if someone is interested in looking at the database as a particular date. We then use the EDW, we use, I'm sorry, a composer which is essentially managed airflow and GCP to do our ETL, and then we also host our North American EDW in GCP and in big query as well. Alright, next slide. Okay, so like I mentioned that initial slide was just Oracle and SQL server data. GFS is undergoing a large scale SAP implementation. And so this happened right at the beginning of our data strategy. And one of the key reasons that we chose both Google and a tunity replicate or a quick replicate now is because they had the ability to connect to SAP and like I mentioned earlier, we chose to not go back and use SAP BW because we wanted everything to be on a common platform which is GCP. And a tunity gave us the ability to have a native connector to SAP to get that data. So, because we're going through a build we have, I'm not sure how many but at least six SAP environments through the build process and we're able to connect the tuning to all of them, our friend, a group can process those and then put them into either our pre production and production data lake. And because we're going through a build of SAP as well as our new North American EDW, it's really important to have the delineation between those, those environments. The other important part here is that a tunity replica, I'm sorry, click replicate actually makes the change record look exactly the same from source system to source system. So an SAP change record looks exactly the same as Oracle change records, which looks exactly the same as SQL server, which makes it much easier on our side to build one thing and then it can process data from any data type that I'm sorry, click can connect to. Okay, next slide please. Okay, so today, we do have multiple different pipelines and kind of our calling card for our data engineering team is to build as few data pipelines to cover the maximum number of source systems. So Gloop is by far our largest, it was also our first as a special place in my heart. And it, it handles all of our database replication, it handles the largest volume of data. We also using other technologies in GCP built both an API and the file pipeline. So basically our API can do one of two things, it can, it can be scheduled to say hey call this API once every 10 minutes, and it may get whatever information is necessary and then stores that data in BigQuery and it can natively handle the JSON message there. And then also we have a file pipeline, which also uses other GCP technologies that basically any source system can drop a CSV into a GCS bucket, our data pipeline can then process that and store in BigQuery. So the big advantage and the way the specific way that we built all of these pipelines is all of the data is in the same place. And I think when we think back to before our data strategy really kicked off, that was a huge challenge is just getting the data in the same place and using analytical tools like BigQuery and MicroStrategy and Data Studio to analyze that instead of analyzing in Excel, which I think most people on this call can probably relate to. One of the key statistics around Gloop is we went from the first time we turned it on in production we went over a year without a single production issue. We never had to take the data stage. I'm sorry, data flow job down or the Pub sub topic. We've ingested 54 subject areas we have 929 tables. We got 12.7 terabytes of data. So we have access to all of our all of our systems that are essentially all of our systems through Gloop that are on-prem or HANA. And what we found is by having our CDC pipeline or API and file, that covers 98% of any source system that we need to get data from whether we own the application or not. It takes us less than a day to add new data, which is one of the key reasons we spent so much time and money building it the way that we did is we knew we were going to get hundreds and hundreds and hundreds of data requests and we needed to be able to fulfill those quickly. We also have 21 data labs, which is basically a sandbox for users to interact with this data. And we built a team of 20 plus data engineers over the past two and a half years. Next slide, please. So where are we headed now? We really spent the first 18 months of our data strategy getting data into the data lake, introducing some degree of governance so that people can use it. Now that we really have our arms wrapped around the data pipelines and I have to plug any data engineer on my team that was here would slot me on the risk because the data pipelines are done, but I always remind them that they are functional. They continue to improve them. But now that we've got our arms wrapped around that we're really turning our focus to how do we get value out of the out of the data that we've ingested into the data lake. Fortunately, we had the backing of the Gordon family as well as the executive team to build things the right way to scale 100x not just do one analytics project where which I think is a major key to our success and differentiator that GFS is data strategy has. So right now we're going to do that in one of two ways, one through our enterprise data warehouse that we're working on building now, building a one place where we can basically evaluate the performance of the business and then have our executives have have one version of the truth where that is a major struggle for GFS today. We're also working on building our advanced analytics teams and have continued to think as we build our data lake, think about the different personas that are going to be using our, our product, and, and our data scientists was certainly one of them so as that team builds out we're going to continue to get feedback from them and adjust the way that we built the platform. Next slide please. That is it. I really appreciate Google and clicks time and glad that Tom and able to share our story and how we're able to use these technologies to drive our data strategy. Awesome. Thanks, Carl. That was amazing. Love that. And thank you for your and Tom's time as well taking time out your busy schedules to tell us that amazing story. I'm loving glue. I'm amazed you haven't seen Willy Wonka on the chocolate factory so I'm after this call I'm going to send you a link on Google movies. And you can pick which one you want to watch whether it's doing Wilder or Johnny Depp. I'm showing my age I'm a gene wilder dude, but that was awesome. Thank you. Just to recap then as we start to wrap up. So, yeah, as you can see there that the click data integration platform can really help you unlock your most valuable data from many heterogeneous data sources, you know, whether it's a database see SQL database mainframe that big red company that has that cell or SAP, whatever it is, you can deliver that data from those those mission critical systems and then continue to build those automatic pipelines when to deliver the freshest data in a reliable way out into Google Cloud. In addition to that we can help you make it you know data analytics ready as well if you want to automate the data models and data maps for the data warehouse and then data like creation out in the cloud as well. And then that gives you access to all of those Google services that we talked about today, and then apply that whatever the analytics tool that you want to use on top. And so you heard great kind of example and good story there from from Gordon foods and we have many others, we are able to scale out their data and elastic workloads out on to Google Cloud. So, with that, I just want to end with an offer that we built from having a very deep strategic partnership with with Google, we've got this joint offering that is essentially a free proof of concept that not only includes software but also the expertise to help you accelerate the management and delivery of one of those most important data sources your SAP data out into Google BigQuery so it's called the SAP jumpstart solution, it supports a wide range of SAP application data sources, whether it's legacy SAP environment HANA or application servers, and the program is redesigned to help you accelerate and simplify your SAP data delivery for that real time analytics out on Google BigQuery. So as you expect it would it utilizes our click data integration platform, that's to help you provide that real time data pipeline to ingest and automate the delivery of analytics ready data out of the SAP systems out in to BigQuery so there's a link there to help you get started all you have to make sure then out of form, and we will take care of the rest for you. So, with that, I will wrap it up I'll leave the links there you will get a copy of the presentation as Shannon said, and once again, big thank you to all the speakers thank you to all the speakers from from Google, Carl and Tom from Gordon foods really appreciate sharing your story there and really looking forward to see with all the great things that coming out from your team, and I'll say thank you to everybody for on the call for your time. And we can open the floor up to questions Shannon. Thank you so much for this great presentation and tying it all together realize some great stuff here just as the most commonly asked questions just a reminder, I will send a follow up email to all registrants by end of day, Thursday with links to the slides and the recording. So diving into the questions here in the Q&A section. Feel free to type them in there I will. Let me just ask start with this, given the impact of cover 19 on business, especially with regard to the supply chain. This is the transition to the cloud facilitated your organization, and how is it has facilitated your organization's ability to manage essential supplies as well as use of forecasting tools. You want to try to attack that one Kyle. My, my, my take on that is that that's not really that that that's not a piece that that we've attacked yet when it comes to managing essential supplies or forecasting tools within the, within the cloud. Yeah, I apologize which which question was asked. Given the impact of coven 19 on businesses, especially with regard to the supply chain. How has the transition to the cloud facilitated your organization's ability to manage essential supplies, as well as the use of forecasting tools. Yeah, I think I'm probably not going to quite answer your question but I think like many organizations coven has really accelerated many digital transformations that may have already been in motion. Well, I, I'm not a supply chain expert what I can say is that we recently gone through kind of our long range planning, and we came out with many what we call must win battles. And I think the realization that the organization had was that almost every single must win battle had a dependency on the data must win battle so I think that just kind of reiterates like like Tom was saying data is the next frontier in the GFS is transformation and the supply chain and forecasting and using data to make better supply chain decisions and operational efficiency is a key part of that. Yeah, so maybe it's highlighted the need. The cloud has not yet facilitated our ability. It's driving us there. Yeah, but thank you so I'm on page 29 side 29. How do you do it how, how do you automatic do automatic schema updates and how do you do data updates. Replicate has the option of sending schema updates within within the data payload. So we ingest that information on the group end of things and then there's magic in the code to apply those changes. Yep, Tom's absolutely right. Our friend group. When it gets a schema message it pauses the data ingestion process the scheme message adds the column deletes the column, and shout out to Google for adding some additional ddl capabilities there, and then continues processing the data. And we don't even know what happens. I'm not quite sure what what was that how do you do data updates. What was that related to the schema update question. Yeah, so you know how do you automate how do you how do you automate the scheme updates. Yeah, I think that's all we covered. Yeah, so that that's okay. Thank you very quickly on the data updates. Just might be stating the obvious but one thing replicate can do is is automatically, you know, switch to change data capture technology and that if that question is referring to as and when the, you know, this changes at source that what that's what replicate does and in terms of sending those changes out into the into the target. So that automatically free configuration. Perfect. So, would you elaborate on the buy in process at all levels of the organization and we get this, this question a lot. You know, how do you, how do you get the C suite to buy in and understand this is important. What was the most challenging aspect of getting this is a great question I understand why people ask it, fortunately for us, I think we're in the minority and we were lucky because well we will we forecasted a need for these kind of tools and that the demand was coming several years ago. It really came from our C suite. A few years ago that we needed to pursue this angle so to his credit, our CEO was interested in the transformation to digital business. And so the, we didn't have to get the buy in at the senior leadership level. They pushed it downward. They totally agree with Tom we talked to many other companies that it's, it's, I think it's number one it's from the bottom up and number two it's it driven, where we had the exact opposite, like Tom mentioned it was business driven, and it was from the top down. I think one of the other key notes that I'd say as far as getting by on buying is is number one, a huge shout out to the Gordon family and it wasn't as much a hey this sounds like something we should do it was, if we want to be around for the 25 years, then we need to be leaders in this space and we need to double and triple down. The other key I think advantage that we have is board and food service is a privately owned company. And I will bet we are making more five 10 year bets than most companies are in this space because for that reason. And so I think it's part. The last thing I would add is, like I said we started out with two engineers and Tom's team I think had three at the time that kind of started up this data transformation, and we started small we got value. I think what we did a really good job at is telling the value story and saying, we did this small thing. If we were able to 10 exit we could do this small thing 10x that and then imagine the value that we get from there so every step along the way, every piece of the value we got along the way the story was told, and the company step kept continued to plow more money into our strategy to, to build the team and build the platform. Adam anything that you want to add to that that you've seen from from people or how to buy in. I mean, it's great when you get an organization like Gordon Gordon foods that's really forward thinking, and then you kind of, you can have it from the top down because that's really how that kind of change transformational change needs to happen. And, you know, forward thinking organizations that's what's driving the need for, you know, the role of the CDO coming in. One thing that we've seen a lot is, and from personal experience as well is try and get a senior stakeholder. It doesn't necessarily have to go at the top if you can't get there. But if that is one of your channel challenges and one of the hurdles if depending how you know the organization is structured and get a senior stakeholder involved prove the value exactly as as Carl was saying there. And, you know, something that's going to kind of resonate and hit home find you kind of pain point or lowest hangers through and then use that senior stakeholders to sort of work your way up as it were and when people see the value that's when you can kind of start creating that that buy in and that that kind of knock on dominant effect and get more people in. Yeah, I think Adam makes a great point and piling on I would have met. So it's interesting about the executive mandate. It's not that uncommon. Most organizations are driving towards an executive mandate so having a vocal CEO like in the case of Gordon foods, or a chief data officer or what have you. There's going to be executive mandates to to to live up to. When that's not available though it's always helpful to have a digital journey, you know, break up the journey into in achievable segments with easily measurable outcomes and add innovation. Right from, especially from an SAP migration. Many times it's looked at as a simple lift and shift when there's huge opportunities to add value to every single step. Don't just bring over an ECC system and add transactional reporting, bring that over and add AI, you know add some visualization and some metrics and outlets. Yeah, I would, I think everybody on the phone has made a great point about getting buy in by showing showing steps along the way of that digital journey, and it can be very powerful. And at the end of the day, it's an impact to the business that makes makes our lives better not the other way around. So it's a very exciting time to be a part of those discussions. I love it well I'm afraid though that is all the time that we have for today. I will get these questions over to click. Any questions. So but thank you all for this great presentation thank you to click for making help making these webinars happen and of course thanks as always to our attendees who are so engaged in everything we do and love the questions that have come in. Just again to remind everybody I will send a follow up email with links to the slides and the recording of this session by end of day Thursday. So thanks everybody I hope you all have a great day. Thanks guys. Thank you. Thanks everyone. Thanks everybody.


All right, hello everyone. My name is Richard Schneemann or at Schneem's. If you came to the Lightning Talk, you know the most important part about me. I'm incredibly, incredibly committed to Ruby. Sadly, she could not come to the conference. And you will also know that Ruby is in fact a Python programmer, it's okay. I've gotten over it. I work for a small company based out of San Francisco that you may have heard of. It's called Heroku, where they give me some free time and allow me to work on things such as CodeTriage.com which allows you to get an open source issue in your inbox, one every single day. Or I also recently announced docsdoctor.org which is basically the same thing but with methods. So if you're looking for documentation, you can get method documentation, a documented method or an undocumented method in your inbox. A little bit more about me. I am in the top 50 of the Rails contributors so that kinda makes me a big deal. I do not have any caps in this presentation. This is a photo of my dog, enjoy. You may have seen a couple people wearing Keep Ruby Weird shirts. This is a conference that we threw for the first time ever in Austin, Texas. Austin is Keep Austin Weird. And you know what, like Ruby is amazing and dynamic and powerful. And in the spirit of why we wanted to preserve that. We also, we got together and we were like, okay well you know what, we kept Ruby Weird in Austin but at RubyConf, we should also try to keep it a little bit weird. And we were like, what can we do to keep Ruby Weird? So we were just brainstorming and I needed a lot longer segue. So we were brainstorming and we thought, perhaps we could do the CanCan, you ready?  So so that was the fun part stay here. You're not done. You're not thank you very much. That was wonderful. Can I get a round of applause? They knowingly volunteered to do this. That's amazingly. That's amazing What what you do not know is you have also volunteered to the do the conference can can So can I get you to all stand up? You have to stay weird Everyone so the conference can can is similar to the actual can can except you can't do this so it's more kind of like a Pretend you're just like a really like shy person doing the can can And we and we have we have some actual some actual music That is perhaps a little bit better than mine So you're ready and and it's kind of you do a sort of knee step kick Really like me. Yeah. Yeah, it's like me knee knee knee If if you if you're friendly grab the person next to you Thank you. Thank you very much, but that's not that's not required All right, you're ready for this This is amazing By the way, this song goes on for an entire minute Okay, thank you Thank you very much All right, I hope you I hope you are fully weirded out now Now that I have your attention now that you're awake. I'd like to talk about algorithms But before I talk about algorithms, I want to talk about my background So I I went to Georgia Tech and I studied mechanical engineering. I don't have a CS degree And I've been teaching myself programming for about the last eight years Which it's been like super fun and super amazing, but computer science is like Crazy boring. I don't know if you know this if you have a CS degree. It's it's all right I Find programming really really interesting because I like building things. I like actually making and putting my hands on On parts of systems and seeing how they move but The CS students might know something that the the the mechanical engineers of the world Haven't quite touched on which is Algorithms are absolutely beautiful like just unbelievably beautiful And it's not because oh they come from a book or you have to learn them to pass this test because algorithms solve problems They solve real-world problems that someone else has invested time into into learning so I Have a really big problem and that is Spelling I cannot spell at all Which is why I'm a programmer because if I misspell my variable name as long as I consistently misspell it. It's okay I'm only like half joking on that one so and spelling even becomes more difficult when you're tired or distracted and Like all the time I'll do something along the lines of like get commit and then and gets like yeah that's not a real command and Something else that get does which is really cool It's like hey what you actually probably meant was commit and that's that that was amazing in one day I was like hey, how does get no? So so the method through which get determines this is something called edge distance and Edge distance is going to be the cost to change one word into another The the the less the less that costs to change one word into another the more likely that those That is what you meant if there's zero cost then it is the exact same word I think a quick example from zat to bat This would be a cost of one or zzat to bat would be a cost of two so that that kind of makes sense But how would we go about coding it? When I when I first found out about this I was like oh, hey that's like probably really simple and I can I can do that and I sat down at my editor and And just kind of banged out this like I don't know five lines Something I was like hey distance between two strings And when I when I ran it with this really simple simple use case I got that the cost was one basically all we're doing is iterating over each Character in one of the strings and checking to see if it matches the exact same spot in the other string It's super simple like that that made sense to me, and I was like oh, this is great I'm done and the talk is over But You may have guessed the talk is not over it wasn't perfect And if I put in something a little bit more complicated, maybe things that didn't have The same length then we would get a really high cost, and this is not correct It does it does not take seven edits to change the word Saturday into Sunday so It turns out that in my naivety I accidentally recreated a thing that actually exists and is actually useful. It's called hamming distance and Hamming distance is not edge distance. It's known as signal distance and It will measure the errors introduced in a string and it's actually only valid for strings of the same length It is it is really useful, but only not for spelling. It's horrible for spelling. Don't use it for spelling If you happen to be in the telecommunications communications industry and you're dealing with let's say strings of Like, you know, well, it's essentially strings of ones and zeros then you can say okay Well, this is the likely match of this set of strings if we can detect that there was an error So it's really bad for misspelled words And the reason for that is it only includes substitution. It does not include inserting a character or deleting a character So I would like to introduce another algorithm called Levenstein distance In order to figure out Levenstein distance, we have to figure out how do we figure out those extra deletion and Insertion so if we look at two strings that are really similar We can see here that shneems and in zee shneems is almost a match except for this first character So basically if everything except for the first character on our second string is a match Then that's a really good candidate for deletion. That's saying yes We should delete that and this is what that would look like in actual Ruby code If if those match then we delete For insertion we can we can do a similar thought experiment where we have just one character missing so shneems And I don't think the other word is actually pronounceable shneems Okay. Yeah, that that was good And if we knock out the first character of the first word then oh, okay, we can solve this via an insertion And here's the code about how we do that with substitution we already looked at that and And that's basically saying every single character matches except for the the one that you're currently on and Yeah, so we're only looking at the at the first character in that string whenever we put all of these rules together We can calculate distance If you pretended that we already had a distance measurement between different strings we could calculate the the distance between strings of different lengths and It would if you have an empty string and a string the cost to change between those two is going to be the length of that string So that intrinsically makes sense Nothing to foo cost three characters food and nothing also cost three characters We can represent that in code by just checking the length or returning the length if the string is empty Once we have this piece we can calculate the different distance between every single substring and Do it using the same logic we did before where we're going to call that distance measurement And we're gonna we're gonna either knock off the first character for for one of them We're gonna knock off the second character for the other one We're gonna we're gonna not only compare the first characters and this represents Deletion insertion and then substitution We're gonna go ahead and calculate all of those and we're gonna take the minimum cost because we only want to make the minimum number of edits And then we're gonna add one So why did we add one? in this case The the cheapest operation just just represents which one we should be picking but adding one to that Represents the actual cost of the operation Okay, that that covers all of our cases except for when all of the characters match and So that's fairly straightforward again. You just look at that individual character and When we combine all of these things together we we form of course Not not vulture on We form recursive leaven stein distance which is amazing and really useful it It's it's incredibly simple and it looks a little something like this It's actually not that much longer than than the other one each of the individual rules If you look back at the at the previous at what I previously talked about Vaguely kind of makes sense if you just sort of pretend that the distance measurement Method already works. So here we go at when I when when I looked into this I found this I was like, okay Hey, we're done. I put in Saturday and Sunday and it turns out that hey the distance It only takes three edits to change Saturday into Sunday. It's much better than seven Then I was like hey, what what does this actually look like when it runs and so I kind of Hooked it up all together. I don't know if you can see this We're still going We're still going Okay, so that took 1647 comparisons and then you can see that the distance was three So we got the correct distance But like it took a really long time in order to run that by the way all of these scripts are on github.com Schneem slash going the distance I think I also have a I have a bit.ly URL that might be a little easier to remember I had that later But so if we look at this the the measurement before the hamming distance, which I didn't know it was called hamming I just called it dirty distance That took eight comparisons recursive took 1647 It's like which would you rather have fast and wrong or like really long and correct? That's not something you can You can you can you can really you can really reason about so it turns out that they're that There is a better way to do this that there is we can do we can do slightly better and One of the keys to this is if you watch the recursive algorithm closely enough You'll notice that a lot of those strings that it was comparing were the same so you know we can we could we can cash those in in some sort of a way and Basically if we have the distance of all of the substrings in a string then we can add those up And and get the distance of the entire string so so first of all I Would I would like to invite you to join the club? I don't know if you know this but there's a very prestigious club for members only and And and when you're in the members only club you can also learn how How the the Levenstein distance calculated with the matrix works, which is what we're going to go into right now All right We're gonna stick with the same example Going with an empty string over to Saturday We can build a matrix that that looks like this So we have our empty string on the up and down axis and we're gonna have Saturday on the top if we just Convert empty string into s it costs one s a is two s a t is three four five six Seven eight so that that hopefully makes a little bit of sense We can look at it at the other way and say what does it cost to turn Sunday into into an empty string? We form the other side of our axis and we have one two three four five so Six so it costs six edits to turn Sunday Into an empty string we have to delete six characters in order to get it into an empty string So that would be that would be the distance Once we have this we have the the starting point of our matrix We don't need to we don't need to to really calculate this with this is kind of like the zero with law of edit distance So we're gonna go back to those rules that we looked at previously and we're gonna break it down How much does it cost to change an s to an s well in in intrinsically we can say that Okay, it it is the same thing so that there is gonna be no no edit distance and it's gonna be a cost of zero with insertion We can say obviously if we want to change s to s a we're gonna be adding the a so that's gonna cost one That's gonna cost one extra character And so we would add one into our matrix and And we need to be able to do this though programmatically It's it's relatively well potentially relatively simple for people to do it But we're gonna go back to those rules So again, if we're gonna be we're gonna be knocking off the first character of one of our strings And the way we can do that in a matrix is actually by using the the row index and the column index as a As a pointer to to the string that we're looking at so we're looking at s And then we're looking at a and but we want to knock off that last character to see if to see to see if they match And the cost of doing that would be would be a one and Then and then of course we add plus one to it and that would represent the cost of doing an insertion right here So it would it would be zero plus one which happens to be one Which is exactly what we thought previously which it's always really good when you're programming programming an algorithm And and your algorithm like produces results that you expect generally Okay, so we're gonna we're gonna we're gonna keep going in this case there are no extra characters It's gonna just all be insertion so we can we we go over and over and over again And we end up just adding one two three four five six seven On to the end of our matrix Our next character is going to be changing s you into s Mm-hmm. Does anybody can anybody guess the action of this? Changing s you into s Deletion okay good again. This is this is something we can intuit But we have to be able to figure out how to tell our our machine previously We looked at some code where we we knock off the first character of the of the second string and again, we're gonna we're gonna look in our matrix and Our column index this time is gonna be s the row index is gonna be you We're gonna bump it up by one and that gives us our initial cost that we can then add one on to so it's gonna Be a cost of one This covers insertion this covers in deletion. We also have to cover substitution You probably see where this is going before you've already seen these equations And we get to our matrix we find our rows we find our columns and in this case We we go back on each of them and this is this is exactly the same Substituting so substituting a for you is exactly the same as saying Did our previous set of strings match because if they matched then we then we should then we should substitute them So that that hopefully makes a little bit of sense We add one and that covers all of our all of our processes of determining Insertion deletion and substitution. I did kind of gloss over match match is basically just Minus one minus one where where we look at at the previous one It's similar to insertion except we don't add one because there is no cost to change a string to itself So essentially changing an s to an s as the same is the same as changing nothing to nothing Which would be a cost of zero so once once we have all of these things in place We can put them together and we end up with something essentially like this. It's gonna we're gonna iterate over all of the characters in one of our strings and We are going to to store the values in a matrix and you can see here highlighted in yellow all of our all of our different Cases granted if you were going to code this for performance, you probably wouldn't want to allocate a hash, but that's a different that's a different story When we do this we then iterate we can run it and you get kind of something that sort of looks like this Just goes There you go Okay, so you can you can see the final cost of changing Saturday into Sunday was three One of the one of the coolest things about this method is that we can also get the cost of all of our substrings We've already calculated them if we if we were interested we could find the cost of Sun to sat we just look in our matrix We we pull the end because that's the last character in the T and we find out the cost would be two Was this better than recursive it? it took a net of 48 iterations which in my personal opinion is a little better than 1647 it is it is a little little memory intensive and both of these Examples that you can feel free to play around with and like actually iterate and step through and reason about as well with with all of my notes for For this presentation are on bit.ly slash going the distance if you're ever interested as well as I'll throw up a couple of a couple of reference links for the research that I did in Coming up with these Okay, so this is a tool previously I was talking about programming and I was saying algorithms are really cool because they do things and like by now you're like String minus one plus one ad cost like blah blah blah like what's going to be for lunch And you haven't seen it actually in action. I mentioned previously. I'm a horrible speller I Happen to be human and I get I get often tired and machines do not understand me very well when I get tired So one day while I was try typing. I was typing rails generate migration but it was really something that was like my Gratoon and It totally just blew up in my face and I was super upset And I was like stressed and it was the simplest task known to mankind like why could I? Not possibly do this like it just things were not working. I was upset. I Don't even know where this came from like who made this. It's amazing So I Want I was saying why can't more software? Why can't the software that I'm using be more like get We know what you're trying to accomplish We know you're trying to run a generator command like we know the generator commands available Why can't we be a little bit more helpful? So I submitted a pull request request where we use Levenstein distance we then Basically whenever we detect that you have an error we we compare the command that you gave us to all of the possible Commands using Levenstein distance. It's relatively it's quick enough and then we recommend the smallest possible distance So again, this is this is also similar to what Google does if you've ever typed something into Google And misspelled it it will give you a spelling suggestion So this is really neat Peter Norvig has a great great paper online where he talks about how Google implemented the first the first spelling suggestion and Basically Google in addition in addition to the Levenstein distance stuff that we talked about they slurped in a lot of real-world books and In these books they calculated or they they they counted words Every single word every single time a word came up they counted it So the word a probably came up like a ton and they're like a is probably a real word The is probably a real word smorgasbord is totally made up because we didn't find it in any of these books Like you know they're dragon flash or something So The higher the count of that word the higher the probability that it's gonna get in there Once we have this information we can also get the edit distance between the input that you gave it and the dictionary that we have The the the lower the edit distance the higher the probability and then put that all together boom we show a suggestion I also Google is really smart and they totally cheat about this Like running Originally running a Levenstein distance Across all of those would probably not return in whatever point something something seconds with network lag So they cash the correct the correct spelling suggestions what basically like show you a spelling suggestion And then you click on it. They're like ha gotcha That's what you actually wanted and if most people search for that thing and also click on that spelling suggestion Then they know that that's probably probably the right one a Another really cool thing that actually came out Relatively recently is the did you mean gem? Has anybody seen this? All right, so if you add the did you mean gem to your gem file it will give you a really cool error Like if you if you call yeah user logged in it's like you know what this method doesn't have a user logged in it basically catches no method errors and Finds all of the methods on that on that Object and and runs Levenstein distance across all of them and then suggests a method for you to use Probably don't use this in production So this is this is really cool like this this stuff it wasn't just like in a textbook and I was like People are bored and professors are like trying to make your life horrible like this is applicable actual things You can you can add it in in like this just came out somebody just thought of this Like what else added to like there's a ton of CLI commands like yes, it's very specific It's generally in in dealing with with misspellings, but as programmers we spell and misspell a lot of things We type a lot of things there are a ton of applications. I'm sure that we could add this to in in preparing for this talk and Doing doing research They're like it was originally I was like oh like I'm gonna do all the distance measurements And that's like saying I'm gonna do a talk on all the algorithms Because there's a lot of them. So we talked about Levenstein distance and we talked about hamming distance There's like longest common subsequence distance. There's Manhattan distance tree distance and There's also Jero Yaro Jero Jero Winkler distance, which is nowadays what I've been told most people who if you're doing large massive amounts of Spelling suggestions would would actually use instead of instead of Levenstein And you can you can get really creative If you're if you're gonna be doing this often and frequent enough like, you know Google had to bootstrap at one point in time those spelling suggestions. They can do things like store the distance edit Calculations or the the the the count in a like in a try so that you can easily just give it all of your characters And it will give you back a a list of possible spelling suggestions. So This is kind of just just just scraping the surface of of all of those all of those measurements but Like really at the end of the day out. Hopefully you're gonna you're gonna walk away and be like algorithms like not just for CS Undergrads anymore Like they're they're really cool. There's a lot that we can learn from them and If you're interested in in learning more just general about algorithms like Wikipedia has an absolute ton of stuff Rosetta code is great if you're not familiar with this project it will actually they will pick common algorithms and Implement that algorithm in a bunch of different languages though the the ruby version of Levenstein distance was like so Not ruby code. It was like obviously not written by a ruby programmer I felt free to like fix that a little bit Or honestly the best way to learn more about algorithms and my personal recommendation is to give an algorithm talk Like seriously, it's probably gonna happen as soon as we open up this for questions Somebody's gonna be like, hey, did you hear about like this algorithm? And I'm like no and that's amazing and we should talk about this And and and algorithms are essentially these ways of of sharing knowledge and they explore how How how we can do things and do them efficiently? So this is the anti penultimate slide Which I've been informed means the third from the last And this is basically a nonsensical slide So does anybody have any questions? Okay, cool. Well, I'm gonna be I'm gonna be around all this code is is is online Like I highly recommend going out and playing with it actually exploring all of my notes are on that ruby gem Or on that on that github repo and if you wait maybe like five minutes I'm gonna actually push up a bunch of links to the read me that that help explain where a lot of a lot of My source material came from so thank you very much for coming and yeah, have a great day


Can we make a bigger brain so that we could act better? Another way to enhance your brain capacity is to increase the size of the neuron. If you become more and more conscious, your body and brain, you can use it in many, many more ways than they are using right now. So, I'm a big fan of Elon Musk and he's on interplanetary travel and that's what's on the cards. You also going to Mars? No, not yet. I don't have so much money.  He wants to be better at it on Mars, right? I think the last time we evolved was more than 15,000 years ago and we may be due for an update. Where do you think the human race is going to go in the next hundred or maybe even 500 years and how do you think we're going to upgrade? How are we going to evolve? I don't want to comment on Elon Musk. Well, he's made a nice car. The best one that he made on the Lotus platform. Yeah. So, that was faster than a Ferrari. I drove that one. So, I have no complaints about him. But he must restrict himself to cars, maybe airplanes, maybe even a rocket.  It's a different matter. See, there's substantial evidence, neurological evidence. This question has been on for a long time. Can we evolve into another place? A whole lot of people asking. Can we make a bigger brain so that we could act better, we could be more intelligent, we could do many more things? So, neurologically, certain neuro... neurological scientists have explained this. See, if you want to increase the capacity of this brain, one thing is you must pack more neurons. This is how a lot of children are born with excess neurons. Those are the children who are hyper, very brilliant, but hyper and unstable. Well, the moment today doctors see them, they pounce on them and because medical industry is a big industry, they're always looking for a victim. And pharmaceutical industry is the second largest industry, so they pump something into it. They give some name, ADD, ADH, ABC or XYZ. All the 26 alphabets have been expanded like this. But when a child is like this, this is not abnormal. This is normal. A whole lot of children are born like this, that there is excess neurons in the brain. If you just put the child to lot of physical activity, sleep and physical activity, proper physical activity, you will see by the time he's six or eight years of age, he will adjust himself, the body does its own thing to make sure it kills the excess neurons and manages itself. But before that, if you put them on sedatives and that kind of stuff, for a lifetime they may become dependent on the medication. Having said that, another way to enhance your brain capacity is to increase the size of the neuron. If you increase the size of the neuron, the capacity of the brain could expand big time. But right now as you sit here in a restful condition, more than twenty percent of your energy is being consumed by your brain. Your brain is only this much, hardly two pounds. This much body, this much body consumes only eighty percent. This consumes twenty percent sitting here. If you increase the size of the neuron, the kind of energy that it will demand, the body cannot sustain. So this is what the neurological assessment is. But Adiyogi, over fifteen thousand years ago, said something very unique. People ask the same question. So Adiyogi said, see you can learn to use your existing capacities in a much higher way than the way you're using it right now. If you become more and more conscious, your body and brain, you can use it in many, many more ways than they're using right now. But if you have to physically or physiologically evolve, then the way the planetary systems are, some change needs to happen in the solar system. Because solar system is like a potter's wheel. It has spun this and it has reached its peak. How we know it's reached its peak is, if you know something about the geometry of the solar system, you know the diameter of the moon and the distance between the earth and the moon is 108 times and the diameter of the sun, you know those things. So, if you look at the human system, there are 114 junction points where Chakras has there called. The first two are outside the physical body. So we usually don't talk about it unless it's for some mystical purposes. So hundred and twelve out of that hundred and twelve, four of them, you cannot do anything about them. They... If you take care of the 108, the remaining four operate by themselves, they have an intelligence of their own. We cannot do anything about it. So 108 is what is available to you. Seeing this, he says, see 108 is already available to you. So the geometry of the solar system is like this. Unless some structural change happens in the solar system, you cannot evolve physiologically any further. But your ability to use the same thing in better and better ways is always available to you. So if you're planning to grow this along with Elon Musk, no, but can you learn to use this better? Are we not always learning to use this better, both body and mind? Aren't we? Some people of course downgrade themselves as time goes by. Some people are upgrading themselves, isn't it? It's possible if you're willing. Willingness means if you have to be truly willing, you have to be conscious. If you're unconscious, you have become unwilling unknowingly.


What's going on everybody welcome back to another draw preview and we have the Hamburg open this week a clay court event an ATP 500 event a big event on the calendar happening this week straight after Wimbledon We have some big names playing in this one as well, but some big names are pulled out So let's go to the withdrawal list so some players that are pulled out of this one and do ha He's pulled out ojali a seam is pulled out after making the quarterfinals of Wimbledon last week Butcher which is also pulled out for the same reason carrots if it's pulled out and also sonigo has pulled out of this one All right, let's start at the top of the drawer here We've got eight seeds in this event and the top four seeds get a buy in the first round It's the father city pass he comes in as a wild card and the number one seed He gets a buy in the first round He'll take on the winner of the qualifier or coppa in the second round Which could be a very tough match for Steph if it is coppa because remember coppa almost took out Federer at the French Open then we've got Munevis the wild card colescriber winner of that match takes on the winner of the wild card Altmaier and the number six seed Crianovic then we got the number three seed Bacillus really he gets a buy in the first round He'll take on the winner of Mutay versus Betonay who are going at it in the first round Then we've got the wild card Kuhn versus a qualifier winner of that match is going to take on either Jera or the number seven seed Strouf in the second round so the players we're watching out for obviously we're watching out for city pass We're going to see how he's going to go after losing in the first round of Wimbledon a couple of weeks ago And of course he made the French open final and this is a clay court event So really keen to see how Steph can maybe regain some form this week also keen to watch Crianovic because he's a very good player But we haven't seen the best of him and this is a tournament a smaller tournament that maybe he can shine at and also If it's still actually number three seed interested to see how he does this week after he also lost in the first round of Wimbledon a couple of weeks ago going to the bottom half of the draw now We've got Benoit pair the number eight seed he takes on Brankus in the first round winner of that match takes on a qualifier in The second round no matter what because we've got two qualifiers going at it in the first round Then we've got Delbonus first thing another qualifier winner of that match takes on the number four seed Ramos for Noles in the second round. Then we've got the number five seed Lyovic versus Pui That is a huge first-round match circle that one is one to watch winner of that match takes on either Maga or a Qualifier in the second round then we've got another qualifier in this part of the draw Stacked with qualifiers in the bottom half taking on Lopez the veteran winner of that match takes on the number two seed Carino Buster in the second round to play as I'm watching out for in this one Got to watch out for the number two seed Carino Buster hasn't really done too much this year after having a breakout season last year Making the semi-files of the u.s. Open so interested to see if he can maybe regain some form going through this one Also interested to see how Lyovic does because again He's a player that last year in the year before was really having a breakout And it's kind of fallen off the last couple of months Interested to see how he does and of course you got to watch out for Ben Mape You never know what he's gonna bring to the court so always got to keep an eye on him He might surprise everyone this week and have a bit of a run So there it is the Hamburg Open an ATP 500 event a big event on the calendar last year It was a lead into the French Open this year It's after Wimbledon one of the last clay court events of the year and it's a big big event Who do you think's gonna win this one though? Let me know down in the comments below We've got Sidney pass playing you think he's gonna regain some form having just made the French Open final having a look at the Draw and I guess the easy pick is Sidney pass He did make the French Open final who was arguably one of the best clay court players of 2021 behind the goats Nadal and Djokovic So I'm gonna go with Sidney pass to win this one and regain some form after pretty poor performance of Wimbledon again Let me know down in the comments below who you think's gonna win this one Hamburg Open one of the last clay court events of the season And Sidney pass he's gonna be playing