Thank you. I've been informed multiple times to stay inside the egg So that's what you need to do if I get out of the egg and get excited Bring me back into the egg and thank you very much for the introduction And I'm going to be focusing on a part of the entire data pipeline That is often not really focused too much on the data preparation phase The idea being what's all of the work that we need to do on data before we give it to you to fuel use Cases like machine learning or advanced analytics. It might be business intelligence where to be honest a lot of the fun happens and What I'm going to do to highlight this is I'm going to focus on one small part of our data preparation Pipeline that we've been working on and and I want to focus on how we used a graph store in this case It was Neo4j To act as kind of a persistent store to store the state of how we would build a decision engine That would help us automate a large portion of this data preparation phase after all they do say That in any data project that we work on 60 70 80 percent of the time is actually spent on this data preparation phase So this is me when I'm actually formal with a suit on and everything My background is software engineering I've been spending the last 14 years as a software engineer and The last six of those mainly focusing on the data space And I probably have quite an odd skill set compared to a lot of people here I'm a classically trained like Java and net Software engineer is the last six years that I've been focusing on things like graph theory and Search and text analysis and more on the natural language processing and ml side. So let's dive in So what we do include in is we help large companies build a data foundation. What is that? Think about think about every data project that you work on There are some common pillars that you go through to solve that use case Every project has a data integration piece to it everything has a data preparation and data wrangling and and Massaging to it and of course we're in Europe. So things like compliance and governance They do fall into those pipelines, but are often one of the last things that we think about So I guess what you could think about is what we do is include in is we condense and abstract These common pillars into one single platform make it simple to deliver data to you to actually add some type of business value So then why a decision engine? Why did we think that we needed to build a decision engine for our customers? In the big data world they often talk about the three V's volume variety and velocity and I can honestly stand here and after working with large enterprises say this is just the tip of the iceberg What I think the data big data era did is it also exposed us to the amount of technical data debt that we have as companies Not only that but it's also exposed that the ways that we solved data before big data came in Just do not scale So the core concept was how could we let the data work for us? And one of the design principles that we came up with was how could data take on quite a Biological kind of organic form to it where it could do the work to Self-link self-discover connections Self-enrich self-clean So let's go into how we did it We had a simple idea We needed to be able to build a decision engine that could take data in and Automatically prepare it as much as possible For this we realized very early on we needed to persist data. Why because data doesn't stop flowing There's no snapshot in time where we can think we've got everything we can now start making our decisions So this required the need to persist Whatever this data engine was actually building Knowing that maybe in a month maybe in two minutes maybe in a year We would get more data and insights to help us make a better decision on preparing data Of course coming with this from a performance perspective We were Mandating that it needed to be parallel in nature there need to be an asynchronous nature to this decision engine that could start To spawn off different decisions asynchronously Fortunately, we had one ace up our sleeve, which was this did not need to be real time I mean I think most people would agree that when it comes to data preparation data quality Using data to fuel use cases that I would much rather have high quality prepared data than something that comes to me quick and that I need to fix and clean up at a later point and Then there was this kind of odd concept of Could we gain context where context didn't exist? So where data was ambiguous in nature, could we actually Enrich it to a point where we could get something out of nothing Why even do this? well apart from the fact that I've alluded to just the classic ways of data preparation do not scale around the Shear of three V's that I alluded to in the big data movement, but on top of that It was to be able to disseminate noise from value. You can imagine that in a big data world whether it's Tracking transactions, whether it's customer data, whether it's analytics that there is still a lot of noise that exists within that data So could we use this decision engine to disseminate what was valuable what was relevant and what was not? And for those of you who have worked with graph theory before and graph components and maybe some of you have actually worked with Neo4j itself The reason why we picked this persistence store Was literally because it matched the type of data structure we were wanting But what we realized is we could also leverage a lot of the power that comes with graph theory to prepare data Something simple as figuring out two nodes in a network. How are they related to each other? We could lean on Shortest path algorithms to be able to figure these types of relationships out with the end goal being Could we actually build a knowledge graph a rich strongly typed knowledge graph based off raw source data? That's what this decision engine was there to do and As I alluded to before let the data do the work If we enter a big data era where we don't let the data do the work You will constantly be fighting the amount of velocity variety and volume of data Before I head into kind of the complexities around the decision engine It is worth mentioning and maybe this is my bias from a software engineering background That the simple techniques in data preparation They still do work. They're still very valuable and We work with a quite a lot of financial institutions and What happens is when you're utilizing automated engines automated decisions a lot of the time these financial institutions? insurance companies Require for an audit reason explain ability and this kept coming up of Reinforcing that the graph was a good persistence store for us I've got some pictures later that show you how explainable it is how easy it is to look at a network and actually understand Why certain decisions were made on data? So it's that explain ability that was also important for this So the end design after around six months can be summarized here What we designed in a decision engine was a recursive decision tree That would organically grow Expand And if we were building branches of this tree that weren't linking to anything We would cut them. We would collapse those branches There wasn't enough interesting information to discover in those paths And this was our first I would say naive approach I've added a little bit on there at the end, which is the learning factor that came around one and a half years later where we Wanted to leverage machine learning to figure out how could we learn from the decision trees and the decision engines that have been built in the past It is worth mentioning That before we were processing data through this decision engine There's a lot of pre-processing that happens a lot of text analysis and For example in some of the techniques. I'm going to show you or some of the results We're running things like cleaning up the data We are running things like named entity recognition to be able to detect objects within text And of course, there's always a statistical approach to this There's no guarantee that those types of natural language processing systems will do the right thing It really comes down to a confidence level And I think actually Our cto martin kind of articulated it quite well And if you look like that, you know, he's good at engineering, right? So Essentially what he highlighted was that When we're working with data Most of us are not working with the highest fidelity version of our data If you look at any data store available today relational wide column document stores search stores object stores blob stores the graph is The highest fidelity data structure we have to be able to store our data And for me, this is the secret. This is the missing piece To be able to fuel machine learning cases business intelligence cases is to deliver to the business the highest fidelity data structure that we have the way I like to kind of I guess go into depth on this Is if you look at any data store that you use today A graph can always Downcast itself to any other structure relational document object blob Going the other way Requires inference It requires you to guess To get to a higher fidelity of your data structure So let's head into what this looks like in action Imagine you have something just as simple as Oracle what is Oracle? This is where this pre-processing step comes in The fact that we can use named entity recognition to figure out that in fact in the context of this text This is highly likely to be a company. This depends on the models that you have available So when you place this through the decision engine And i'll explain the user interface for you I actually do see a couple of engineers that i'm used to working with here So but for those who have not seen the platform before essentially we've got a really ambiguous reference here. We've got a company called Oracle which company which country And i want to fast forward directly to the actual results So once we place this through the decision engine It was able to with a high confidence figure out It's a website it's social accounts annual revenues addresses How does it do this? It really comes down to blowing up different decision trees to explore data The references in the network that link more than others are the paths that we want to follow If we look at this from a different view This adds a little bit of context Where did this reference come from? Well, it came from a book a pdf document in a SharePoint repository. Okay, that's context That's helps And what this decision engine was able to prove and to show If we look at once we've run this through the decision engine This is just a different view of that same data We've got annual revenues stock codes social accounts Not to mention we found new links In references to other records within the network Why this is interesting is because it's turning soft references into hard references It's turning links into deep links And there are many use cases where this can be applied to Especially when analyzing unstructured data Just um for your information Underneath this is actually the decision trees that get built up Underneath to form this type of answer So one of the the downsides I guess you could say about this approach Is that it's very heavy on storage It's very heavy on processing This is not a real-time system But it does come back to would you rather real-time data or high quality data? That's the decision that you need to make for your different use cases If I have more time, I'll show some More use cases later But I want to head into kind of the specifics the details of how the engine works So as I alluded to before we had quite a naive approach to start with I would say it's a brute force approach that we were taking Where every time we discovered data We would build and persist these large networks all the time And if we were ever finding new data that potentially we had seen before It exploded out a completely new decision tree. So there's not a lot of reuse there And when it comes to for example exploring different parts of the decision tree And I alluded to before that there were times where we would cut off certain paths where we wouldn't follow those paths in the decision tree Because they weren't linking enough to other records within the network How could we learn from that historical data? And so two areas we were looking into in the ml space one was a supervised approach and one was the unsupervised If we first look at the supervised What we realized we needed to build on top of this decision engine was an annotation system An annotation system that would utilize Reinforcement learning for data stewards to reinforce some assumptions that we had made in the decision engine So something as simple as is Larry Ellison the CEO of Oracle These were questions that because We had our data Persisted into a network Where typically you have two nodes that are connected via some type of relationship Sometimes it's not a strong relationship, but it's often something like a verb Larry Ellison works at Oracle This would allow us to actually formulate these natural questions to be able to prompt to data stewards And one of the interesting things is we would give those data stewards a simple kind of yes. No And skip type of interface If anyone in the audience has used something called prodigy before it's an annotation tool That's the kind of tool that we use to give to the data stewards. So it's a binary classifier. Yes. No skip And One of the extra wins that we got from storing our data in a network Structure in a graph Was that if I was to answer yes to this question is Larry Ellison the CEO of Oracle There is so much more that you're reinforcing than just telling a system. It did something right or wrong You're actually reinforcing that Larry Ellison is a person Some probably would not agree with that You're also reinforcing that CEO is a job title You're also reinforcing once again that Oracle is a company And then you might say yeah, but we already knew that we already detected that Well, this is more reinforcement that we're doing things right in the engine Then there was the unsupervised approach This one took us quite a while to actually figure out what we needed to do To leverage an unsupervised approach it comes back to this let the data do the work This was the area that we needed to look into In to figure how we could scale this against the big amount of data that we were often working with And before I head into the techniques that we used I'll just want to stop and kind of think about some of the good parts of the approach that we thought up until this point The first and I know I've said this before but I just want to reiterate it It's that explainability of persisting data into a graph Being able to easily visualize How to records in a complete maybe across an entire business are related to each other and One of the challenges in data preparation That really is helped by this that plagues every single business Whether your small media or large is duplicate data And I'm going to go into how we solved duplicate data that problem of duplicate data to a higher precision using the graph Of course in engineering you get wins you get losses every decision that you make in a certain direction makes other things worse And in this case it always came back to this was a brute force approach The amount of time that it takes to process data to explode out these different decision trees using Internal and maybe external data sources as well. This is definitely not a real-time system. You can imagine the the amount of Chattiness that exists between the the application and the and the graph in persisting this could be quite high Of course, you can do some pretty smart caching to to make sure that you're not constantly persisting to the to the graph store But really it came down to this was just the reality we had to face but in the end. This is exactly what we wanted I would rather deliver more highly connected more high quality higher enriched more linked records across data sets than something that I could deliver at a faster rate and then With this weird design idea, we had to be able to add context to maybe ambiguous references Well, the bottom line is a lot of the time you do have more than nothing You do have something If you're connecting with crm systems, you have a lot more customer data than just references to names This design principle was in there to be able to say well if we can make a Something out of something very ambiguous It would be quite fun to throw data that's richer in nature to be able to to the decision engine as well And I'm biased I guess But it comes back to this realization that out of all the data stores available to us today Graphs are good at solving certain challenges Every other database still makes sense for storing data for processing it in other ways But from a fidelity perspective From having your data represented in the highest fidelity, you can't go past the graph It is the highest fidelity version of mapping and connecting data So This is a problem that every business has duplicate records How do you figure out that these records are related to each other? I can ruin it for everyone You're all doing it the same way Every business today is doing this the same way We put these records side by side And we run string distance functions, maybe probabilistic matching engines to figure out What is the leavensteen's distance? What is the hamming distance? What is the jarrow-winkler distance between these two records and then go through all the important properties? If those string distances hit over a certain threshold Then accept that and automatically merge it and then have data stewards that are manually there to fix the false positives That is how every business is doing it Now we also use this approach because it is a very important part of the entire chain I'm very sorry if this is the way that you do it I don't mean any offense by this, but this is quite a naive approach to this problem One thing we are missing here is context What about the data around these records? Whether it's directly connected or indirectly connected So I want to talk about four techniques That we built into our decision engine to raise the confidence of deduplicating data across a business The first was pattern matching and not probably the type of pattern matching that you're used to I mean graph, I mean pattern matching in a graph There were some very practical optimizations that we learned the hard way i.e If I make a certain decision on data at a point in time What happens if that data changes? Do I have to reevaluate every decision tree that was built off that state of data at that point in time? And the bottom line is the way that we designed the system in the first approach. Yes, this is what happened And the entire network the entire graph would just reevaluate and essentially just kill the heap And and and the the data store would fall over So we had to do something about this from a practical perspective And then there's the supervised approach and the unsupervised approach So how could we apply machine learning techniques to solve this data duplication problem? Now to understand the pattern matching piece, I actually have to fast forward to the supervised learning Because that comes before So Here's the problem. It's a pretty classic problem. You've got data coming in from different sources Our task is to turn multiple records into potentially Less records so merging entity linkaging and de-duplication. They're all synonymous. They're all the same thing So this is the kind of problem that we need to solve And these are the type of records that we're used to looking at this is a person record. So first names last names aliases What we're not looking for is identifiers here We're looking for fuzzy references They're the types of properties that we want to be able to detect if I can overlap them When there's no unique ID to be able to overlap these records And our first naive approach was a disgusting set of if statements business rules If the name matches the name from the other record with a certain level of persistence Then mark that as yes, they are the same. We can accept that And this is just one of those cases where machine learning is perfect for this It's perfectly fit for this situation where in fact what we didn't realize is we have hundreds of thousands of records that we have merged in the past Whether they were done manually Or because we had beautiful IDs that linked them What we're interested doing is saying Get rid of the IDs I want to figure out without Perfect IDs to merge records What will the properties look like? I know that they matched. I know that they merged We can throw this at classification engines So like most people do we start with the kind of generic classification models a random forest And it's very explainable It's easy to understand. It's kind of machine learning 101 I'll ruin it for you, but in the end we couldn't use this by itself We actually had to use an ensemble of multiple techniques to throw at this problem The other two were a support vector machine and a single layer perceptron This ensemble of techniques allowed us to get to a precision that we were happy with with automatically de-duplicating records And so here comes the pattern matching part as I alluded before I gave this example of Decisions are going to be made in the decision engine all the time A decision it can only make decisions based off the current state of the graph and data never stops flowing So what happens when new data comes in? Potentially de-duplicates records or discover just discovers new relationships Does that mean we have to reevaluate every decision that was made at that snapshot in time? And what we realized that we had to do was cash some decisions And what this allowed to do when our classification engine realized there's a duplicate It triggered off to the decision engine to persist some data Now the data that it persisted was this So in this example here I've got two I've got three people records a meal a meal a meal and what we actually store Is we store a vector of the types of relationships That are attached whether it's incoming or outgoing to an individual record So it's a proprietary format. I'll explain that the top row Essentially, it's just saying one outgoing reference to a person One outgoing reference to a company on the next line. It would be one incoming reference to a person So we would store these little vectors Actually in the in the graph we would store it actually in some of the the graph stores neo being one of them You can actually store properties on the relationships themselves And what this meant is that when we needed to Explore part of the decision tree or to de-duplicate records We could look up these tables and determine Do we even need to explore a certain part of the tree? for example, if I de-duplicated people It might be that there are certain sub trees that come off that that I don't need To evaluate and this is what helped us stop this decision explosion This ability to have to reevaluate the entire graph, which is not a good thing to do um So that's what helped us in this local caches that would say Don't explore these sub trees unless you change certain properties or you have certain types of relationships Here's just some other examples I've kind of ruined the answer already by showing the slide But this took us so long actually to figure out how we could utilize a neural network to be able to Have that learning capacity Because when you look at the supervised approach that we took we had an annotation tool Where data stewards were manually reinforcing if clued in was doing something right or wrong You know Did we correctly de-duplicate this record? Yes or no The goal of that reinforcement engine is not to tell a system. Why did it de-duplicate? That's why we have these functions for that's why random forest exists That's why these classification functions exist is to let the data do the work What you'll often find is most businesses the way they're solving this challenge as I alluded to before is business rules huge amount complex web complex mesh of business rules And think of the classification approach of reverse engineering the rules from the actual data itself So when it comes to a learning factor I would say that supervised approach still had quite a manual Touch to it and one of the questions we often get In that part of the platform is how long do we have to spend in annotating the system? and often the answer was I genuinely have no idea But it will never stop asking you questions There is always something to annotate. There will always be something to label And so this was the next iteration. How could we remove that manual process and let's be honest In the types of enterprises that we work with banks insurance companies I think a lot of these businesses are worried about removing the human factor from it So that's still important the ability for data stewards to play a role in reinforcing that an engine is making the right decisions The other question that we would often get is well, when will this thing automate itself? It's also a very hard question to answer It comes back to this idea of the more time you spend in labeling The smarter the engine will get over time So then we get to our unsupervised approach The first challenge was figuring out what type of neural network we could actually apply to this if any and What we ended up we're going with was a recurrent neural network That's what we decided on There was a there was always a time factor to data And that's one of the kind of focused areas of the rnn's is that Time is some type of factor in us not space But where we struggled was figuring out what were the parameters that we had passed into our activation functions So if we have multiple layers of the neural network What were the actual components and of course if you've used these before you'll know that one of the Factors behind a recurrent neural network is back propagation. That's the learning factor You know, how does it reinforce that it's making these right decisions? It took us quite a long time to figure out the properties that we would need to feed into this system But in the end it was staring us in the face It was the calculations we were making in the first step the string distance functions And There are many different string distance functions We ourselves use three And for this step so dice um It's actually called dice cernson coefficient. Um, it's a Cernson's a Danish Mathematician and statistician So it was a combination of calculating what are the string distance between the functions Converting those distances into float values and using those as the activation functions that we would pass through to the layers for the back propagation And this was the learning factor This was the piece that able was able to teach our engine how much Does string distances play a role in merging records? And It took quite a lot of data to get it to a point where it was reliable And here's one of the other tricky things about this space. Everyone's data looks different You cannot apply the same models for something like a deduplication against a bank and an insurance company even two banks Their data looks quite drastically different Coming from different systems different formats at different rates And what this means is that using this approach Every new customer we went into There was the annotation phase. They had to spend time in annotating or at least pre training the engine with historical duplicate data fortunately a lot of the kind of common platforms you see for mdm cases for probabilistic matching engines for deduplication engines They have been fortunate enough to at least store the history of these records merged And in some cases some of these systems even allowed data stewards to choose why What properties made a data steward manually merge and clean and prepare this data? And So if you're interested in learning a little bit more about data preparation This is just one part. This is the part that's trying to automate Parts of the data preparation phase and as those of you who are data engineers Those of you who work with wrangling and massaging data and preparing data for use cases You'll know there's certain data that can also not be Automatically cleaned automatically enriched But this goes a very long way for calculating some of those common things that we can Actually clean with a certain confidence level And that's one of the things that came up I think from the big data era Is that what we highlighted during that movement was that the traditional mechanisms for solving data preparation worth scaling And instead of companies embracing that data is either right or wrong What we find is that companies are moving much more to a risk level A confidence level to be able to keep up with the sheer volume and variety of the data and So really working with data preparation is now down to confidence How confident can we be that the data is in a state that we can trust that we can audit That we can show the lineage of where it came from so we can fuel machine learning use cases So with that in mind many thanks for attending. I hope you learned something and Feel free. I mean Neo4j for example is an open source piece of software free to To try out. I would also say it's one of the more easier databases to get up and running with And so getting up on your machine. It's installable in pretty much every single Type of way that you can imagine so spinning up a docker image of Neo4j playing around with it And it's one of the more easier databases to get started with So with that, thank you very much, and I'd be happy to take some questions if anyone has any Thank you


Cool. Okay. Hello, everybody. Welcome. I hope this is me, who you're looking for, and I hope you're going to have some time. A really good one. In reality, it feels a little bit anti-climax after two days of amazing speakers, mind-blowing keynotes, plenty of learning, to actually take you on another ride when you've been going up and up on this huge roller coaster, and you kind of expect this, and I'm just a little bit worried that you're going to get that. But let's see how it goes. I want to continue on the topic of failing leadership, ownership, blaming games. So let me start with this. In 2007, I finished university. I was a software engineer, and I had the dream to change the world one line of code at a time. So I decided to move to London and become a Ruby on Rails developer. Ages ago, this is how this stack of books looked like, and there was a really good joke about how Ruby on Rails was a really cool technology. So I moved to a company that was doing some really cool stuff, and I was a junior developer there. You should have seen me on the first day at work. The spring and the step. You get by underground to the place called London Bridge. You see Tower Bridge. It's just mind-blowing. It's just great. The funny part was, I didn't have a clue what I was doing, but London is a great place. It's a place full of opportunities, and whatever you want to do in London, there will be a way to figure it out and to actually do it. And the worst thing you can do in London is, as a developer, there's just way too many projects to do that you can handle. So what I thought was, why don't I not only not know what I do in this startup, but also help another one? So I started working on another project in my free time. It was like this kind of a concept of a side hustle, money was flowing in. And the interesting part happens is I was very, very, very, very inexperienced, and I didn't really know what I was doing. But what I did was, I worked 9 to 5, probably in the hours there will be 7 to 7 on my first project. I was in the office. I was working. I was coding. And 5 to 5, I would just basically pick up my laptop and start working on the other one. And somehow, this information about the way I was working got passed to my CEO at that time. So as you can imagine, the next day in London, it never looks like this. This is just a lie. It always rains over there, so enjoy the weather. I got invited for a chat with my CEO. And I'm a nice guy. Don't get me wrong.  Like no one has ever did to me what this guy did. And Felix was a very awesome, very honest, very transparent kind of a guy. He would always share with us the good stuff and the bad stuff. When I got invited to his room, I didn't know what it was about. But that was one of the biggest runs I've ever seen.  When you were in the room with him, there was no pause button. It just went on. The whole building knew we were, it wasn't a conversation. It was like far away from having a conversation. Everybody thought I would just get fired on the spot. The interesting thing was that my CEO in a nice and very polite way explained to me what was at stake. He kind of told me that the startup that we were working on, he put all of his money into it. And most of the employees put their savings into it. They remortgaged their houses to make it happen. Each of them quit their well-paid jobs to do and live their dream of building what we were building. And he just couldn't grasp how I could finish eight hours of work and then do something else. He just couldn't. And he just went mad. It was awesome. Everybody thought that I would just get fired without even leaving the building. Lucky for me, I didn't. I don't know how and to that day we are friends with Felix and he doesn't know either. I think in the passionate moment that he was describing to me what was really going on, he just forgot to say at the end, you're fired. Okay. But I kind of realized, and in retrospect, it's kind of quite easy to recognize that. At that day, I probably didn't say much, but I kind of realized that there was plenty of lessons to be learned from this moment of my total ignorance and the lack of understanding what was going on around me. Yes, I was a junior developer. Yes, this was probably my first real job. Yes, I wanted the dream to happen, but there was plenty of lessons learned. And what I did was I decided that from that moment I will never work for people who doesn't shout like that, which means I want to work for people who have passion. And this passion can be represented in a different way. In his way, it was like this. In other people's ways, there are different ways, but I want the passionate part. I don't work with people or for people who don't have that part. I realized that I don't want to work for people who doesn't have the purpose. There was lots of conversations here about the purpose before. When I go now for a job interviewer, when I speak with anybody who wants to build something or who wants to join our company, and I ask him, hey, you're offering me a job. Why you do what you do? And if they don't have a great answer about why they do what they do, not what they do, not how they do it. If they don't have the answer, I'm out. I don't want to work for companies that doesn't have a why. They don't have a purpose. The way I do it with recruiters, if you want a piece of advice, I say, do you know TED conference? Who knows TED conference? It's great. It's like the best speakers in the world speak over there. And I'm saying to them, look, if you want to hire me, I want to work for an organization which will allow me to speak at TED conference. What it means for me is, first there is Bill Gates and he says, hey guys, I cured malaria. And the presenter says, and now here is Paweł Kaminski, saying that. Whatever is that, I want to work for that company that gets me on this stage. If you cannot do that, I'm not interested. Our lives are way too short and way too important to work for companies that don't do that. And the interesting thing I've learned in this moment of madness is that we all failed to communicate and we've got this illusion that, oh, I've told you that. I thought you understand what I meant when I said it's serious. Like, I've learned that the communication plays a huge part, no matter how many lines of Ruby and Rails code I can write. So those were the lessons from my first failure in real-world. There was one more. I decided to stay in that world, which means I never really worked for any company bigger than X. I don't want to do huge agile implementations. I don't want to do digital transformations for hundreds of thousands of people. I love what I do and I'm really enjoying this start-up journey. And I've been doing this for 10 years now, yeah, failing and succeeding from time to time, but really, really having fun. I was trying to change the world. I was trying to build the Internet for Kids, which was the only secure place where they could actually go and don't feel threatened by predators and don't feel threatened by the content they were supposed to read. I went to beg or pitch the investors for $10 million with a nice face, decent story, zero MVP, zero users, just like a nice vague idea what it should be. That didn't work that well. I got now plenty of bruises of what to do right and what to do wrong. And I worked for the companies that didn't have the money at all. And then the only thing that drove us was passion sleeping back under the desk and actually eating plenty of kebabs, which is just not recommended version of what you should be doing with your life. But it's still a great journey. So four years ago, I kind of had this conversation about a new venture and about new possibility where we wanted to create, share our experiences and what we do with people who are non-technical founders, and they've got this big dream, big gut feeling of, I want to do this, I want to build my own company. So we thought, okay, why don't we who went through all of those pains try to help you out? And we did. We started Ucreate, which the initial purpose was we want to help founders who are non-technical, who doesn't have a clue about agile software development, product, lean, validation, all of those keywords that just means nothing to them. But they just get this gut feeling that there is something that they have to do in their lives and in the lives of other people who are around to make a difference. So we got our first customer. It was a company that was trying to change the way we buy and sell properties in UK. If you know how it works in UK, it's like you kind of say you want to buy a house and then for three months you've got zero information and your lawyer does some magic in the background and after three months you get, ta-da, you have a house, or sorry, you don't have a house, which is just a relatively poor user experience of the whole process. So the founder wanted to change that. He wanted to bring the transparency to the whole process. And we actually built the whole system of doing this with a beautifully looking mobile app to deliver this new experience to the users. We released on 5th of October, which was my birthday. I was very proud and very happy about this. And as you can imagine, it didn't went well, but I've learned another lesson. There is no amount of bugs or issues that you cannot sort out with unlimited amount of pizza and a free day for the team that you're working on. Just try. Just give them pizza and let them go. We were okay the next day and we kind of started working. The founder was very happy. We were starting to get the customers. We were putting multiple million pounds property transactions for the platform. It was going quite cool. A few weeks later, in just a little bit of context, Ucreate has got a development part in Chandigarh. So I, at that moment, PM, CTO, coffee maker, fruit buyer, anything you have to do in the startup world, kind of a person, was getting up at 5 o'clock in the morning, working myself with coffee straight to my veins because drinking it was way too slow. Started working with India team. And because I was a big fan of Agile Kanban, you name it, we were kind of deploying every single day at 5.30 in the morning to pre-production environment. And then during the whole day, we were just coding and working together kind of remotely. They in Chandigarh, I was here. And you know those days when your kind of alarm clock rings just before it's supposed to really start. So the first message was from my CEO and it was like, hey Pavel, you know, the production database is kind of a gone. So then you kind of lie down in your bed and you consider, do I really need this? I don't need that. What should I do? Should I go back to bed? But I realized that over the years of experience, I had now a great opportunity to do certain things right. But my first idea in my head was, I don't know how many fans of Game of Thrones, I really like that quote, just cutting off heads doesn't really make people work well together, but that was my first thought. So we moved on from that. I had a quick chat with my India team. They explained me what happened. And the funny thing, like this is just how interesting we were ages ago, do you know like Word documents? They've got like pages. So there was a sequel query, an update, where the first part of an update sequel query was on page one, but the where clause was on the second page. And the developer copied the first part for getting about the where clause. Yeah, that's kind of embarrassing, but moving on. So yeah, that was happened. So all of the relationships in the database were gone. We didn't delete it. That would be better. But yeah, we had just wrong IDs. You get the gist of it. Like it all went wrong. The guy who run this, he was kind of already packing. He was ready for it. Like I think that most of my team at that point understood that I don't remember if I have seen them before that or not, like in person, but they thought like, there is a CTO in London. He's just gonna destroy this whole thing.  This is how they used to be working with their leaders being carrot and stick kind of an approach. Let's have a go. So the guy was like honestly packing. I think he was updating his LinkedIn profile saying, I'm available for hire. So what we did, I had a quick conversation with them. We realized what happened. Then I called my CEO, Matt, who is the founder of the business. And I said, look, we have to call the other CEO. So now you can imagine it's 5.45 before six o'clock. And we called the startup founder that we are working for. And we are just like, hey, Simon, this is great. But yeah, great to wake you up, but you don't really have a business at this moment. By nine o'clock UK time, you might have it. But today it's tricky. So I think he was still half asleep. So the conversation went okay, if you can say it like that. And we went back to work. Actually, it was quite surprising because it turns out that we had the backups. We recreated everything. It was okay. And by nine o'clock, like nothing happened. But of course, everything happened. And we had another great opportunity to learn from this failure of what happened. The crucial question, which I was asking myself and at that moment I couldn't really articulate it well, was who is to blame? So the obvious answer was, wow, the guy who put the query in. And as a good agile team, what we did, we retrospect. And I put a little bit of fire under the team to make them speak because when you call an India team, it's usually this or nobody says anything. It's great when you don't want anything back, but it's just terrible when you're really trying to learn and trying to adjust. So the team blamed everybody. It was a beautiful kind of example of how the blame game operates. Like no one was safe, but of course everybody focused on the developer because he was the obvious choice to be fired first. So what else we found? We found multiple reasons and multiple people really responsible for SAP. We learned that our developers have got ego and sometimes they do things straight away on production, which was a nice surprise. We learned that our poor documentation using Word documents is really, really poor. We realized that we've got poor requirements. We didn't talk to each other. We didn't have the concept of chaos monkey or any kind of a real disaster recovery processes. Look, we were a startup of base rather than months or years. We were having bigger problems than that. And we were still running an operation which we probably put to shame a lot of modern agile teams. But it was cool. And we were relatively sloppy in the way of our approach. So what did we do? Well, we realized and we were very, very honest about saying that this is a blame game. Even though the culture of the company was saying, hey, we don't blame individuals. We don't want that to run the company in such a way. We want to do it different. We improved certain parts of the process. We automate certain things. We kind of made a few improvements there. Each person who was kind of a felt a little bit of responsible for all of the other things that went wrong took a bit of the ownership. And the ownership parts started to grow within the team. We kind of said it's a Facebook quote, no problem is someone else's problem. It has been here before and we really try to do it that there's just no one person to blame. And we realized that we want leaders in our sense on almost every single level of the organization because the leaders takes the ownership of the problems and everything around them. But the coolest thing was, we promoted the guy who removed the database. We made him the head of DevOps for amazing reason. From that moment, he was the most diligent developer I've ever seen in my life. He just didn't want to be in this position to keep updating his LinkedIn and packing every single day. So he becomes our temporary head of DevOps. So you can imagine his situation. He comes in with a great ego as a senior developer. He removes the database. We call his CTO and say, hey, I did that. Then he comes back, we retrospect. Everybody blames him and he ends the day as a head of DevOps at our company. That could be a really cool day if you can survive that. And he did. And he kind of did a lot of processes improvements for us. He did a lot of things and he just made sure that this situation never happens again. And he was actually the best person to do it. He cared a lot. But there was still one question. And that question was still in my mind. Even though we behaved properly and our India team realized that we're just not going to fire them, we understand that the errors happens, we understand how the improvement works. There was still the question, who was to blame and what we should do about this? Because the fact that we fixed the issues were just fine. And then we were lucky that the database, we had the backups and so on, and we could recover it. That was just more of a lack than the real challenge. And there can only be one. And funny enough, that was me. For every single thing that happened on that day, there was only one person responsible. And that's the leader of the whole team. However you work and whatever organization you work, if you self-manage yourself, there is a leader who is responsible for absolutely everything. In your world, you own everything. It was all on me. Even though I was not copying the code from Google Doc and putting it on the live PostgreSQL console. It was all on me. So we learned a lot. So leaders must own everything in the world. There is no one else to blame. Don't pass the blame to someone else, especially when you're in the leadership position. And in reality, what you're trying to do, you're trying to make everybody be a leader. What that means is you don't blame everybody, but kind of everybody blames themselves. It's a strange feeling, but it kind of works. Simplify. I cannot tell you how bad there was that we had multiple pages of Google Doc documentation representing our deployment process. Just automate it and just don't care about it. Simplify every single thing that you can, because when things goes wrong and they always do, the only way your team members and yourself will be able to recover is when the reasons of what you're doing and how you're doing are relatively simple. If you've got a complex plan that you have to execute when everything goes wrong, well, good luck. And we took a certain approach that we kind of assumed how the developers worked. When I called them first time after this happens, they were all kind of a shaky. So we kind of say, look, you breathe. You ask someone to bring you a cup of green tea.  No matter what happens, no matter how many production database are just going totally wrong and how much on fire we are, you do those three things. And the last one is behave like an engineer. You've got the data. You know how to prioritize and execute.  We trust you. There is no one else who should be shouting at you. You behave like an engineer and it will be just fine. That's what we've learned from this situation. How many of you have seen Wolf of Wall Street? So I really believe that we don't really learn that much when the organizations are in a great shape. When your CEO gives your motivation, pep talk once every three months that everything is cool, you don't really learn much about the organization besides the fact that you kind of have got creative accountants who are giving you the right numbers to share with your team. What I strongly believe is that... Do you know this guy, Simon Sinek? So, like, I don't want to say that you should read just one book after this conference, but if you haven't read the book, Starts With Why, that's the only book you should read after this conference. It will change your life and it will change you how you approach your work and your colleagues. So what I'm trying to say is that the companies which are having correct why or correct purpose will always behave properly. And you should always Starts With Why. Don't worry about what you do. Don't worry about how you do Always Starts With Why because it's going to lead you at those moments of crisis that are always happening. You create at this moment have got a really nice why. We work with non-technical founders and we want to help them change the world, and the other one is we want to build the products that people love to use. We don't want to build another CRM. We don't want to build another app doing exactly the same. If the app or the solution that we're building is not loved by the people, we don't want it. That's our why.  So we are an agency and you've got a typical problem of having more projects versus more people and you constantly have to feed the beast and we grown very, very nicely and we are very happy about it, but from time to time, we are just tight. Let's just be clear about it. And there was one of those moments. Lucky enough, we got another project. And I was looking for this meme with cricket. It's not a sport, but is that too much? Okay, I couldn't find it. I don't know why, but I couldn't find it. So the project was about golf. My CEO loves golf. He plays golf. The PM that volunteered to run this project used to play semi-professionally and still plays twice a week. The founder will go first and golf changed their life. The investors will go first. If you can imagine how their meeting looked like. It's like five of them were sitting in one room and they were like, oh, this is the greatest app ever. This is like Netflix for golf. No, no, no, it's like Uber for golf. No, no, no, it's Amazon for golf. Everything was there. The confirmation bias was just mind-blowingly beautiful. So what do we do? We run our typical validation process. We ask multiple users what they think about the app. Do they love it? We run thousands of Facebook ads to really see if the message validates with any users. We run experiments, safe to fail. We run surveys, usability sessions. We even organize a golf tournament to find out from the real users what they think. Well, it came back disappointing. So there were a few ideas around it. There were certain people that we could do, but the users definitely did not love the idea or did not love the concept or did not love what we wanted to do with them. Well, so you can imagine now the meeting when the PM who loved to play golf he kind of pitched first to our CEO saying, you know what, this money that you think is coming? It's not. If you want to stick to our values and our principles of building the products that people love to use, don't do that. Then if you think my CEO was surprised, you should have seen the founder. He was like, look, I can write you a check now. Like, I want to work with you guys. And we're like, no, not really. Computers is no. We don't really want to work with you.  He was actually a nice guy. It's just the idea was not fitting into the criteria of what we could do. And we told him multiple options of how he could pivot, how he could do this online. And I think we saved the world another golf app, probably 250,000 pounds being spent on development, X amount of QA time, and we just moved on. But what really happened is, one, we recognized the confirmation bias. So what we do now, we don't allow the same people to be in the room making the decision about the same thing. If we see that, we just invite a random person who doesn't have a clue about golf. That's a great way. We realized that while the gossip's about this project not being great for us because of our values and because of our why, the whole company is watching. This is the moment when everybody stops and looks from their computer to see what you're going to do. Those are the moments that matter for the company's success and how people feel. If we accepted this project purely for the money, all of the beautiful worlds we've got on the world would just not matter. All of the things we say we are passionate would just not matter. Sometimes you just have to stick to your values and what you really believe in. And what happened now is a consequence all of the people in the organization, when they are in trouble, they go back to those values. What they do, our developers now, will call a founder of a business somewhere in London and say, you know what? This feature that we are building, users are not going to love it. We are not doing it. The founders get furious, but we love it. So that's how you translate your own values into the leaders in your organization so they can use them to their advantage and to do really what you believe in to do. The last thing I want to talk with you is like you don't, as an organization, as a team, you don't have to wait for a production database to go bust or you run out of money to really learn from the failures or from the things that are relatively challenging for you. The one interesting thing which is hugely challenging and hugely failure-driven is one line of code. Every single one line of code, let's see if that works. Works kind of like that. I've just changed one line. That's it. The rest of the bridge went away. The one line of code, if you really think about it, is the representation of the future failure. You know that in six months' time you could write it better. You know that in six months' time it's going to become your technical debt. You know that in six months' time someone will look at it and say, how could you write that? Oh, it was me. Oh, sorry. And in six months' time you will know that you really like to start from scratch, which means your one line of code today is kind of the smallest failure you're already building for the future of your company. There's this famous statement of, the best solutions are built without the software. That's kind of it because of that. So what we thought about is, why don't we use this opportunity of a failure of every single one line of code to do something about it? So we kind of created our own version of the code review, which we call Review E. So what we do, we review every single line of code. Okay? Everyone reviews. Our senior and junior developers, CTOs, everybody who's got the technical knowledge reviews. So the seniors will review juniors, junior will review juniors, juniors will review seniors. Developers love to reject my code and say, CTO, this was rubbish. You can do better, which is fair. We review the code, not the person, which means we teach people how to give feedback. So it's so much easier to say, hey, that's a really bad code or we can improve this code together versus you're a bad person, why did you do that? We learn how to give concise and simple feedback because we don't allow people to write 20 pages document how to explain how to fix it. It has to be concise and simple so that someone can do something about it. And that usually triggers another conversation that developers are having. So we start talking about code much more. We learn to say thanks. When you get your code rejected, you say thanks. It's a funny feeling, but that's what we do because actually that's a great opportunity for all of our developers to learn something from someone else or start a decent conversation about what they do. We explain why, because it's not enough to press a button to reject it. You have to explain why you're rejecting it so every single person starts to think, not only change this to that, they have to be clear about why we are rejecting it. And what happens now is when one line of code that you create is rejected, everything stops. There's no more deploys, there's no more user stories, no one codes. We fix this one line of code first and we do it now. Why we do one line of code? That's a very famous thing. If you are reviewing 10 lines of code, you will find 10 issues. If you are reviewing 500, it's roughly okay. If you are reviewing pull request with 5,000 lines, it's always good. It's always good. Just move on. So that's why we review per line of code. So we have a lot of code that we do to help our developers. As you can see, I'm not even saying that our developers, those guys have become leaders of what they can and do own. So the first thing is, this is CTO rejecting all of the code that this particular person did on their day. There's no problem. There's no fight. 5, 10, 15, 15 minutes left. Awesome. Everything is fine. When you're recording any date, everybody's code. She didn't have a great day or I was in a bad mood. But in reality, she was not angry or sad and she got the feedback how she could improve. Those are a few examples of what is going on now. We now have themes in Poland and in India. That's how they communicate. The messages that India usually surprise me, I really like you notice this thing. It's appreciated and many things. When was the last time your developers said this to another developer? Or when was the last time your developers said, good job, Sandy. No matter what the code is, as you can see, she changed three lines into one. Good job, Sandy. And sometimes it helps us catch stuff that we just don't like. Like commenting out the test-driven development test so that it can just pass. It just, every single commit is there to be reviewed and we can catch all of those things and we can teach the correct behavior. And the funny thing is, everybody sees everything, which means there is no hiding. All of the correct feedback about the positive stuff that we do and all of the bad things we do, it's all public. Nobody is afraid to get there. There is just no other way. And we code on one branch, multiple deploys per day, developers are not allowed to keep any code on the local machine when the day is done. They commit multiple times a day to trunk and we automatically deploy it kind of a close to production. So what they did for us, it creates this concept of no blame and what we call now the extreme ownership. So we don't blame people. We always look at how we can improve the system of work when something wrong happens and we usually blame ourselves for what is happening. That's how we know that the leaders own what they do. That's very, very important for us. We learned that as much as I like talking to you and I hope there is some thinking going on, change the behavior to change the thinking. When we released the first version of it and I said, this is how it's gonna work, everybody said this is just not gonna work. We're not gonna be reviewing every single line of code. This is gonna slow us down.  We are not doing it. This is my screenshot from morning from 10.18 to 11.08. So between those 15 minutes, we had nine improvements done to our code as I was kind of preparing for this presentation. It happens all of the time, 24 hours a day. As soon as the developers comes in, the first thing they do is review code. It's like dramatically change how they think about it. And to be honest, I received 200 to 300 emails per day with all of the improvements that we do. And there's probably more because on every single commit, you've got the right to kind of make multiple comments. So there could be more. But that means that every single day we improve our code 200 or 300 times. We run multiple projects and so on, but that's still a lot. I don't know how many improvements to your code you do. We never do big rewrites. Our review is causing the fact that we constantly keep removing the technical depth as we go. I don't have to review anymore, which is quite nice. The team now protects the tool. When we work with external company, the first thing they do is like, we don't want feature branches.  You come to us and we do trunk-based development with the review tool. They protect it. So if you think that some of those advice or some of those thoughts about using failure to your advantage within the organization was not good enough. One thing, I know that you have plenty of ideas from this conference and that's awesome. Just be really, really careful about Monday. So your context matters. Remember if you go back to your office and on Monday you say, Paweł told us to do this? Don't. Or we should implement Spotify model? Don't. Spotify did not implement the Spotify model. They just started doing things that worked in their context and it ended up being Spotify model. It might not work in your context. Be very, very aware. But of course your boss is gonna say, I paid for the conference. What do we do? Well, I'm really, really hard to say but it starts with you. If you're a leader, everything is yours. You are blamed or you should take the blame for everything that happens within your organization. And I mean up and down and maybe on the holocracy views left and right. Whatever your ability to see, you should not allow, you're basically speaking blame for absolutely everything that happens in your organization. That's how we see it at this moment. So if your CEO does not explain you the new project well, it's your fault. It's not his.  If you don't believe and you don't understand why we are supposed to do this initiative, you will never be able to pitch this to your team. If they don't believe, well you can shut the door straight away. What else you can do? The moment you start taking blame for things that you should be taking the blame as the leader, other people will do the same. Extreme ownership is contagious. As you've seen, there was a comment there when someone just sent a commit and straight away he wrote, I know this is better, I'm fixing it straight away, don't even reject my commit, I will fix it. People starts to take ownership the moment they see you take the ownership. It's contagious. Lead with why. Don't worry about what and how, people will figure it out. You don't have to micromanage great people that you just hired. Just give them the purpose that they have to understand why you want to do what you want them to do. And to a certain extent you have to believe. If you work for the organization that you don't believe where you are going, first ask the people above you to say, I just don't get it, can you tell me? Because in most of the cases there will be a reason why they are doing what they are doing. Most of the CEOs don't want to destroy their companies. There will be a reason why they think this is the right choice. If you don't understand it, it's your fault, not theirs. You should go up and ask the question. Check your ego. Look, when I said it was my fault in the situation when we removed the production database and when I spoke with the founder and I said, it's on me and he was like, oh, but you've got the team in India. No, no, no, it's on me. I made the mistake and this is the plan how we are gonna fix it.  I was a nice CTO working on great projects. I could blame everybody there.  Keep it simple, stupid, all right? I think it's a kiss. Like the more complex the plan, the more 500 steps of your manual deployment process, the more chances of things going bad and your leaders should communicate simply. What we do, we ask our founders to call India team and pitch them their idea in a tweet. Now it's easier because they've got 280 characters versus in the past they had 140. They have to be able to explain to an India development team what they want to achieve and what's their why in 140 characters. Keep it simple. And every time you want to blame someone, first blame yourself and after that together look at the system of work, what you can do to remove this thing from happening again and what you can do to improve. I don't know how you feel about this. I was listening to this morning presentation and the presentations yesterday. I truly believe that there is certain burden of leadership that only certain people should have. Even though we are trying to build a company that operates with multiple leaders in multiple positions and everybody should take the ownership. I truly believe that certain things should stay within the ownership of the CEO however you want to call it. I don't think so that your developers should be firing other developers no matter how nice you are to each other. I don't think so that a group of people should own ability to press a button and nuke another country. If you're running a US submarine and you say, okay let's vote on this. That's just passing the leadership burden on everybody else who might not have either the information or the ability to do it. There are certain things. I think they should stay with certain leaders but it's still up to discussion and I'm happy to be proven wrong. I would love to see how it works somewhere else. And don't be that guy. That's the CEO of Volkswagen when they went wild with their diesel problem. Hundreds of thousands of cars. Kind of a took back. What he did, he blamed 11 engineers in a random factory in Germany for this. That's not a leader. Don't do that. Don't be him. I don't know what he does now. He probably have a great job. Every single time something goes wrong with your organization try to tell yourself, good, you're still breathing. You're still going on. There is something we can do about it. Every single time there is a bigger or smaller failure that's an opportunity for you. And I want to bring the kind of one of the inspiration for the stock that helped me really realize what we were doing for a relatively long time. There is this Navy SEAL officer. If you watch the Ted presentation he talks about war in Iraq and his Navy SEAL officer who run one of the biggest military operation over there and his principles of leadership and especially extreme ownership are part of the discussion we had today. And I really like him. He's got a great podcast. He talks about cool stuff. And when we are discussing about the bias point of view he potentially has got nothing to do with agile. He's a military commander but it's an interesting to have a different point of view compared to always listening to the same kind of a... I'm not saying bias but kind of at the same point of view. So learn from multiple angles and pick the things that really work for your organization. Funny enough, it starts with why, just read that. It really changed everything. And the guy, Jack Wickling, he also writes the book for kids how to change wimpy kids into warrior kids. It's quite cool. My six-year-old daughter started doing pull-ups after reading the book. She just loved it. She's now eating lots of broccoli because apparently it feeds her engine, whatever that means. Just if you did not enjoy this talk or there's something wrong, I'm the only person to blame just so we're clear. There's no one else if something went not bad, not great. I'm happy to have a conversation about this. Thank you very much. It was a pleasure to see you. We've got one minute, I don't know. Okay, we've got three minutes. Any questions or anything? Pleasure. Thank you very much. Oh, sorry. Please take the mic for me. Sorry. The tool you are using, review. Yeah. Is that built by your tea company or something else? Yeah, so I don't think so. I'm allowed to have the public conversations about our own tools, but it's going live kind of today-ish, so I'm happy to have the conversation after I don't want to be doing pictures here. Thank you very much.


Know that we are doing everything. We are reevaluating as we constantly do to ensure that our folks our most critical asset in the Navy, our people, go to work in a safe environment.