Right shall we try and actually start this? Quick review of DevCon 16 organization. What was what went well what went badly? What went badly? We started things too late. I think every team says that every year Yeah So after DevCon 15 a final report didn't happen We had the big blow up with the that resulted in chairs resigning. I Think we tried to fix that a bit and then People burnt out from that and nothing happened for a few months Yeah There were things that kept moving forward, but we took way too long to get the sponsorship brochure done We just we blocked the sponsorship brochure on a website And I don't think anyone was actually putting that much time into the website at that point So we sat around for a few months. We could have been chasing sponsorship, but weren't Yeah, there's a mic here. There's two more handheld mics somewhere I mean we we could have been chasing it anyway So like the problem is not any of those like we didn't have the brochure We didn't have the final report. We didn't have the the problem was just we didn't have a team for sponsorship And in the end a lot of it came down to like just me and I had actually only volunteered this year To be a a like remote resource So I hadn't I was not in theory an active sponsorship team member this year, but it ended up coming all down to me So let's not do that again next year too Good good As someone it wasn't no it didn't actually so it came there came down to a week So we're in the red this year and there came down to a week when if I had sent out a hundred extra emails We probably wouldn't be in the red and I was like doing other things because I didn't have time for it So if we had five people even if I could have just said to someone Please go send out a hundred emails and know that it would happen like we would we would be okay So I mean I kind of see that as the bus factor thing never never never have a bus factor of one As someone who wasn't in the sponsorship team. I didn't even know that was a problem I assumed I thought we were waiting on a flyer and that's why nothing was happening Weekly meetings would have would have helped we did have those for a while We had weekly meetings ever since before DC 15 there was also the issue that people said they had contacted sponsors And they hadn't and because we have this odd RT where you know nobody reads the spam actually you're not tracking it So one of the things that we need to improve and we already said that and the DC 17 both is that we really put into these sponsor list files When people have contacted sponsors and if it's only like the initial product something But that means you take ownership of that sponsor and you follow up with that and if you if you don't do that You either tell people practically or after a few weeks somebody else will basically try to take over if you just go on missing so this this would be I guess the process for DC 17 because that was actually a lack of communication We were thinking people work had contacted the sponsors and the sponsors came back to me during that conflict 60 now said Why didn't nobody contact us? I suppose part of that is also CC the team on everything so that everyone can see what's going on Yeah, so that was a policy and we we actually talked about this in the fundraising off and I think we've got a pretty good So drop our tea that's going away focus on the sponsors list and get the The Sort of text-formatted files that with nice scripts to do reports and things like that one thing that would also be handy You weren't there But is the script that you did just undoubt automated emails for wafer speaker notifications It would actually be really handy to create a version of that that reads from those sponsor list files and can automatically Send out like a hundred messages because part of the time I was spending was not necessary I would spend three days sending out a hundred messages when I could have done in five minutes With it with a good email template and a quick script Do we have We have lots of people that can do things like this. Yes Yes, but they're not they're in one of our silos. They're not Right, so I mean we have five versions of the scripts. Well, that's something we can do this But that's a too late. That's an that's the kind of problem that we actually love to solve Yeah, well, it's the kind of hard ones. It's the kind of problem I love to solve too usually but when I have three days and I have to scramble and it takes me longer to figure all that out Then to just send the hundred ma'am messages and get at least 70k of sponsorship. It's It's falling apart when I mean finding all our scripts and organizing things and making a pretty tools set up for that sounds like a volunteer is Yeah, we'll see is the kind of tasks that We like to do so. Yeah, do we have a sponsorship team for next year that doesn't have a bus factor of what? We have two locals, which is good Okay And I don't know who else is is is committed. Okay. Do you like is committed? Can I just add from a personal point of view my feedback as an Experience was the only local on the sponsorship team We did in fact ask multiple times on IRC in person on the team lists for more volunteers I think what I could take As a message to the next team and to future teams people don't like being on the sponsorship team They find it intimidating But that's not an excuse I Think more people needs to be on the sponsorship teams from local teams Yeah, I eventually joined and started working people I mean, I knew I raised a couple of sponsorships like that everybody on the team should be doing that It's the only way you're going to get local sponsorship Yeah, but we have to be realistic about What people are able to do too? Should we move on from sponsorship team? What was next in the timeline registration probably and venue contracts We probably left our venue contracts and food contracts far long later than we should have That's because they were giving us a hard time and we were being disorganized They were being far more disorganized But everything was sort of fine because we were committed to being at UCT and they couldn't do anything They weren't going to kick us out It would have been a bit nicer to have signed the contract or at least a letter of intent to you in advance and Be in negotiating fine points at the end rather than which food which cater are we going to be using a week before they start serving us food? But that was all that all worked out fine Except that we overcame we overcame a dead camper bit We had got caterers beforehand, but we just decided very late to go with them. That's the point So but also with this we assumed we could bring outside caterers to the venue and then they told us we couldn't at the last minute And we scrambled to find options that would work and they just dragged and dragged because We couldn't find something that fit the budget and was reasonable. It was the same with the venue contract per se Right. I think that was pretty much ready to be signed around June July Last year as far as I remember my conversations with Bernal But then then we didn't know who was going to sign it and the whole legal representation and stuff And so that dragged on for a long time. Yeah, so that's something that we should Do with DC 17 straight away then they needed a huge deposit which delayed things further Except that that really shouldn't be a problem like in the future we can do that But if we know what the process is and if we're if we know what we're doing so with regard to The I don't expect a long legal representation delay for DC 17 because I'm already looking into that And we've got a letter of intent signed already which makes everything a lot more concrete Thanks, and again just from a team dynamic point of view where I got frustrated there is sometimes someone needs to make Unpopular decisions and I think from a team dynamic point of view Tape company to improve that ability The the problem we had there was making unpopular decisions on your own without talking to anyone I'm not just talking about the Molly Blackburn Hall I'm talking the frustration that led me to make the decision was because we've dragged on stuff that actually had an inevitable outcome But I don't want to dwell on that I just there are future things and I think that's where the chase issue also came in sometimes We need to make unpopular decisions my approach for that was to try and deal with those in team meetings If it wasn't too contentious, and we could actually get a decision in the meeting it was seemed to be the only way we could get a decision But then you've got to stick on that agenda item and force people to make the decision and still get through the rest of the agenda A point I can add to that it it's a very hard problem, and it has not really been solved for Quite a number of years I think it's probably good for the next teams to actually have a tie-breaking team Or tie-breaking members one or two persons that are entitled to take these hard decisions that might be an idea to Try I think we kind of did that informally Well, yeah, sure it happens informally, but then it frustrates people and things so doing that explicitly might be So I mean of course we will not go into a full government governance session here because say this could end up Tomorrow night and most of us would miss a flight or something But yet that was one of the important things that the chair or the chair figures had and I think that's something that's missing a small group of people that can as you say work as tie-breakers and I think that should be what Looked into again, maybe reappointed or seeking something but the thing is Waiting for a consensus to emerge is often very frustrating and very inefficient. Yeah, I Think practically in the last year with Daniel and I reached an agreement on something then That was how that was a decided decision Of course because you're the people doing the work and that that that's worth But I think that's that's a lesson learned in DC 15. We had people doing the decisions that didn't do the work and Here we basically kind of decided among the team and that worked really nicely Obviously, you always have the issue with too much too few manpower and things delaying and you know Many things actually only kicked up when Stefano came to South Africa Because things have just been you know, not possible without him on the floor Because this kid discussions weren't going going back and forth and he actually kind of tie broke by being physically present But that was a very specific situation that not you know some important people on the local team were not local all the time So that won't be an issue. I guess for DC 17. I Don't think that the tie-breakers are necessarily going to solve Our problems we've had it in the past and it just resulted in a situation where The entire people doing the work were no longer willing to make any decisions But shoved everything onto the chairs plates who were then overworked and eventually quit I mean there were other factors involved, but it they really didn't want to be in at the the decision-making body So it's a lot more about Empowering the local team or the people who are doing the work to make the decisions and I think that the so it really worked Well, if you guys managed to make decisions and and you know, you didn't cause any conflict Over them. I think that the best thing we can do in the future is to be very transparent about how these decisions are getting made Like you know, I've considered this I've considered that and therefore we decided to do this and and then just move on with Things rather than to discuss and try to strive for consensus. Yeah, I think the struggle really is to You don't want to make the decision because this is hard work And it's gonna be politically difficult and all of that but you've got to force yourself to do it And when you do that, it's mostly fine Well, it is it is hard work It needs to be done because you need to move on and it's only politically difficult if the rest of the team sort of have this Feeling that everybody needs to be making the decision when when everybody accepts the fact that the decision needs to get made And that the people closest to the decision are probably the best people to make the decision Then it's also not no longer really a political issue. Yeah, I don't think that was a problem So so something I want to give here is actually something that worked really well So I know we blew up at the end of the last year and we were Scrambling for time trying to get things done But what that brought about was an attitude of let's just get it done It's like let's not waste a lot of time on politics less not waste a lot of time on Discussing structure. Let's just get it done and that actually worked incredibly well So I'm kind of encouraged this year. We can just let the new team. Yeah Just get things done. Just let them get things done and let's not get in their way too much And I'm really encouraged that it may possibly be the best deb-conf here. We've had in like, I don't know Yeah, I can fully second that the difference between DC 15 and DC 16 was DC 16 was a group of people that work together and grew closer as they did and DC 15 was like Different compartments that were doing awesome work and you need to remember DC 15 was like three times the size of DC 16 So it's a bit harder to organize when it gets like really that large But there were like really, you know, secret cabals and open cabals and any cabals And so we didn't have any of that in DC 16 and that was just awesome So if if you DC if you DC 17 people can try to prevent that from the very beginning You'll have a lovely time over the next I completely agree with what you say, but you cannot Forget you cannot stop thing taking into account that after all the the culture where you grow affects How how the work is done? So yeah here the team integrated very very very good but every team will will interact differently and What I mean, I think it's completely consistent with what we expected from Germans to be and what we expect from Africans to be it's just Well, yeah, it certainly your culture does affect how you operate. That's that's true But you you can also be intentional about what culture you want to adopt and adjust in a direction My my employer is doing something similar because we're all remote and that's unusual for everyone And they're being intentional about culture we can too fact fact of the matter is that we've had in the past a couple of depth cons where the you know the manana culture kind of Ended up in the successful depth con but it also made it very difficult for the rest of the team Like the ones that are not local to work with with the local team And I think a lot of the sort of I'm just gonna talk freely This is not intended to be contentious But a lot of the sort of protective behavior on this on behalf of the global team stems from the fact that we've had depth conf local teams who were like You know Maybe maybe they had everything under control But it certainly didn't seem like that in the weekly team meetings at all points in time Like you know changing the venue three weeks before the conference and all this kind of stuff but and so so but somewhere in the middle is the key and I think still communication and and And and trying to find the middle ground of cooperation between the different diverse backgrounds that we have is gonna be the best way to go Yeah, I and I think on the question the issue we came from of Making decisions and so on I think it is it sometimes is necessary to bring a decision to a meeting but also we can Try harder to have things prepared as I mean as people often do to put make sure it's on a list in advance The options and just make the actual so the meeting is just make a decision rather than Discuss every issue and people raising a thousand questions about details and so on the problem is people don't read any of those things until the meeting Especially the people who aren't yeah, I deeply tied in the local team. I think maybe we could take an event Better example from thumb over the other organizations out there who managed to have like SPI sometimes having their five-minute meetings Where they just say yes, yes, yes, that's it Maybe we should get used to Organize meetings and make sure that decision are being made at the end of the meeting It's unlike Debian when we can make decision when it's ready for the comfort commit on dates and Things don't wait because time is going and that's I mean it's cultural thing And we have to accept the for the comfort. We have to make decision in a timely manner. I just think it's good to have to like with Stefan and Daniel just making the final decision and that's probably the best thing to do is you have someone who says okay We all chat about it. This is the time limit. We've got to chat about it. And this is the final decision That's it now we move on and and that helps less discussion and trying to overanalyze everything So I completely agree with that The missing factor or or the The recipe that we need to install for this to work in the future is trust in many ways Because apparently this team trusted Stefano and then Daniel to make these decisions and we're happy to go along with it But it is not always going to be the case or it maybe maybe it will be in the past it has not always been the case and One of the reasons I think that is is because we don't have at least this it's not obvious to me I sort of shared understanding of what depth conf is among the entire team among all the people So that that leads to people going looking very closely to individual decisions to make sure that their own very view or desires of depth conf are being are being fulfilled and I hope maybe you're gonna introduce the idea that you mentioned the other day I'm how to bootstrap the exact answer to your right because I think that that is a very Something we could try it's a way to share a common ground and be sure that we share the common the same goals and the same way We want to be and that to evade to avoid any Consumption the sort of I think the basic idea is something to give them then local team An opportunity to write down What their vision of depth conf is like how they want their depth conf? I know this is also contentious to say their depth conf because we do it all together But what do they imagine what are the ideas they want to take from the past put into their conference? What are the new ideas? They want to do and then when this document exists then the the experienced people Try to find holes, you know not criticize like this is not gonna work or like this is We've never done this blah blah But like find holes make sure that we don't leave anything out that the attendees expect from it And then at the end if we have a document that sort of describes This is going to be the upcoming depth conf and everybody's happy with that Then it's gonna be much easier to defer to some people to make decisions because they are all we're all in line We have to keep an eye on time. We've got 20 minutes remaining in the session Yeah, I also wanted to share the fact that let me just yeah I think that having lost the bid for DC 16 was a really interesting experience for us And it really helped us organize ourselves and have the venues and understand how things work Then I think this I don't think this should be a formal thing But it looks like if you know if Prague if the Prague team does a good job They're gonna have a lot of stuff that they're only bit the whole build a whole bid for last year So maybe this year just gonna be the same and I think it's it's something We should maybe encourage people to do and tell them that you know Building a whole bit in a year is maybe a bit ambitious You know building a team takes more time than that and approaching people So I'm just wanting to share that Jonathan and I were hoping to lose our bid to the first time around so we'd have more So we'd get we'd do some work on a bit and then we work on a dev conf later Unfortunately, we had a dev conf I think we should be walking through our Part here, which says how do we help the tip come 17 team get started About this well as you can see we sort of already got started one of the things I think was a really interesting learning experience was our Because we were talking about making difficult decisions and we already had one of these very heart-wrenching decisions regarding the choice of our venue and We sort of like learned a lot about how the the dev conf works in terms of these making these difficult decisions I mean, I think it's not gonna be surprising if I say we didn't have as much support and We had some but not as much as we envisioned about getting global global team members to help us decide on on the venue and so we really had to We had a quite a long meeting and we had to decide at the end we had basically just a vote and Majority like slight majority one for For the venue and that for us that I feel that was our moment of deciding what we wanted dev conf to be So so yeah, we're I feel we're we're we're well on our way to to organizing We're not I don't feel we're at a point where we're we have to start thinking about the big things certainly there's a lot of Details and a lot of small things that that we want to get right and we've already we've already started listing a bunch of those we At the right at the beginning of the conference Gabrielle opened Etherpad document Where he started putting two three things and and now it's like we have like 40 points of little little things that we now don't necessarily have to think about one year before but who that we we felt Really improved the experience for attendees so so yeah, we're obviously gonna be sharing this and and Yeah, there's of course lots of works lots of work in front of us I can't say anything about that It's those last those things you think you don't need to deal about now deal with them as soon as you possibly can because at the Last minute you've got so much other stuff to do So what you're saying is it a hybrid manana not manana culture I plan a year in advance for things are going to deal three months in advance If we'd bought more coffee in advance and consider how much coffee people actually needed the coffee machine wouldn't have run out It also doesn't have any more beans and they wouldn't sell us more beans well They wouldn't sell us any yesterday. It was a Friday. So they don't sell beans on Fridays for some reason The next item I have on here is what are the first steps? Yeah, I think you've kind of covered some of that What's coming next? Well, we've we've gotten Nonlocal people on to our can board instance. That's like shaping up to be our main task tracking tool So obviously whoever is not on there Still not on there and wants to be part of the deptconf 17 effort come and see me. I'll I'll get you on there and Yeah, I mean there's there's there's still stuff to do like for for catering for Treasury sponsorship I think one of the big takeaways I personally have and I guess this is also shared by the team is that we really need to get our Stuff together for sponsorship. We sort of delayed a bit on on this this front we've we've had we've had like informal discussions, but But yeah, one of the takeaways is that we want we want sponsorship to be as as concrete as possible as soon as possible and We also don't want to delegate this. Okay, like one person takes care of sponsorship But really that that should be that should be a task that that's Distributed as much as possible over the the entire team last subject the best one How people see the chairs and what should be the role the delegates So one of the things that have has been said earlier that The role main mainly wasn't passed to after the tiebreaker body so was seen as frustrating and I mean my hope is would be that the body will be an enabling thing rather than blocking thing in the future and So how to get this done? I mean how what are the expectations of people and I think the biggest thing is Ability to sign documents and approve budgets and doing that early on and that not having chairs We kind of ended up in a bad position in a few ways there and and Beyond that I mean there's lots of things they could be and people have different ideas and what they should be But beyond that it doesn't actually matter all that much We just need chairs delegated by the DPL with the authority to do the essentials And if we start there and iterate we would probably be okay We did get a DPL to sign documents fairly quickly when we needed it, but yeah This is this is potentially contentious as well, of course, but we have had situations in the past where The local team took decisions that were not okay in terms of the depth conf values or whatever And so it could be the DPL because we hope that it's not going to happen very often But previously also the chairs role was to to have the authority to say no to something Of course that then ended up in the steering committee And it wasn't very clear And it is very very much in conflict with the whole idea of like letting the local team Get the work done and supporting them, but it's hard to be lived correctly I don't know if it is in conflict. I mean, I think FTP masters, you know, you delegate the authority you let them get their job done You don't veto unless absolutely necessary I think the same philosophy works with a very active vibrant local team that we just let them get their work done You know, you were there to be responsible when needed, but it don't get in the way I think it's fine and it seems very dubbying. It feels very dubbying Oh, I think um, I mean it became easier in the end In the towards the end of this cycle because they weren't that many people who were active apart from people who considered themselves to be local team whatever In I in the idea we would well from my point of view I would I probably to keep things running in general. We would like there to be more people who are staying on from year to year and As much as at different times in the past we've tried to Do things to avoid that kind of conflict there's a kind of natural break between the two parts of the team in that way because just because of there's some group of people who have physical meetings See each other chatting the bar about something and they'll tend to make a decision informally And then if you go to a wider meeting, of course You get some kind of conflict if the other people don't share the same I mean, it's you can do a lot of different things to try and smooth that and thinking about it actively and so on but There's always going to be a slight risk of tension there. I mean, whatever you do It's mostly difficult for the chairs to fulfill that role properly and it's really hard to do that I guess when it's not publicly accepted what their role is Maybe one one piece of advice for the next year's locals who would also be to act as if you are working remote and Even if you meet in the in the same room and discuss things make sure you have at least Minutes of that meeting outside to the list and might sound obvious, but it's really important to be reminded of that If you want anyone out who is not a local to help you you need to include them It's it's kind of up to you Thank you No, the thing is yeah, you can see I'm like Going through my thoughts. I think there is a lot to say but many of us have already said it over and over It is a very hard decision How how the governance for that comes works? I am on the opinion that a chair like figure Works better than not having it but but I think we have iterated the The Topic enough. I know the opinion of many people in this room. Of course not not of everybody and I think well We need the opinion of somebody with a higher authority to to give it to tell us what we what what will happen So I think the delegation is needed to avoid any doubt on that I think that in the past we've put too much pressure on the chairs Because the delegation says they are responsible of the organization of debcoff When it's not totally the case because one things happen badly. They don't have necessarily the tools the mechanisms to the budget the people to get it fixed So we cannot have Very high expectations from what the chairs can do. That's why I want them to be like I mean helping the team and powering them To do the stuff finding new volunteers making call for helps. I didn't see them see that happening for in the past For example, so that's how I see the future delegates to act So they are not there to frustrate people to block them for doing the work But they are helping helping them to get the things done And well just to add a final point to my to what I was thinking The problem is very hard to solve on the air Well also because it's a lot of dealing with people so so if the local team and the Delegates or the global team or whatever don't get along It will be very hard to find a Workable solution And I think that's a well what happened when when we have this mass resignation There was not and there was no way to get them to work together So so of course they they felt they they could not continue pretending to do so Okay, thank you everybody. Thank you


Thank you very much. Buenos dias. Muchas gracias. AutoML has become, yes, a hot topic. And we wanted to explore this. My colleague Ben Lorak and I have been looking at some trends. We've been doing some research in industry and seeing about AI adoption. Last year, when I was here, I mentioned that we had done a survey. We followed up with two more surveys over the course of the year. And so I did a big report about data governance. And then the next one, the other trend that we wanted to follow up on, was AutoML. And Ben Lorak has also, he's recently left O'Reilly Media, and now he's doing some of his own work. There's a new podcast that he's doing called the Data Exchange. And for the first episode, Ben and I were talking about this topic with some other related topics. So if you want to grab the slides, they're online. If you take a photo on your phone, it should load this OK. Sorry, let me keep this going. AutoML is interesting because it's showing up in academia. It's showing up in industry. Clearly, there's a lot of indications that this is a growing trend. And I really loved the keynote this morning from Oscar Mendes, where he was talking also about how Stratio is working with AutoML. And I think that a lot of the key points that he brought out there are central to the thesis of what we've been finding in terms of our market, sorry, our industry research in this field. So as an overview, nearly all of the public cloud vendors are promoting some sort of AutoML service. As well, the tech unicorns are also developing these kinds of services that they use in their data platforms internally, Uber and others. Some of these are going open source. And we'll take a look at a survey of these open source projects as well as a survey of the different vendors in the industry. There's a lot of smaller tech startups that are promising to democratize machine learning. And also, I think the real business value there is there is a lot of hiring pain. It's very difficult for companies and traditional businesses to step up with really good data science talent. So a lot of what's happening is really to alleviate the hiring pains. And we'll explore some of that. But given all the buzz that's happened in industry, what does this really mean? So first off, just as a little bit of background, I love history of science. I love looking back at where did some of these origins come from. There's a few things that I've written over the past year or talks that I've given that really delve into some of the history of data science, but also some of the future trends. And so there's a small book out recently, 50 years of data management and beyond. And a couple of others, there's a keynote from the Rev conference, too, that I think speak to this. But really, it points back a lot to we can trace some of the origins back to John Tukey in 1961, where he was talking about empirical data analytics for the first time. And that really struck the tone for an interdisciplinary approach, which sense has grown into data science. And now, in terms of machine learning and AI, those terms do get used interchangeably a lot. So I wanted to differentiate that to some extent. You may have seen this from Madeliso, the difference between machine learning and AI. If it's written in Python, it's probably machine learning. And if it's written in PowerPoint, it's probably AI. And unfortunately, there's a lot of truth to that. But I would like to point out maybe a more effective kind of distinction. Machine learning is a lot about tools and technology. Generally speaking, when we're talking about machine learning, we're talking about some sort of differentiable system working with gradients, understanding how to find optimized points within a gradient. And that's the case, whether you're talking about deep learning or reinforcement learning or any of these. AI, though, from my perspective, what we're seeing emerge as a good working definition for AI is when those kinds of tools, not just machine learning, but other tools as well, come into play as far as changing social systems. What impact do they have on groups of people and how they behave? How does it augment them? How does it change the behaviors, et cetera? So that's kind of the working definition that I have in what I'll use in this talk in Elsevier. So when we look at where there's evidence of intelligence, where can we see intelligent behavior, there's definitely three, and I'll add a fourth one based on the recent talk. Certainly we can talk about recognizing complex patterns. So for instance, when we use supervised machine learning, typically we're trying to generalize and recognize some sort of pattern. And it's interesting because if you look at the benchmarks for this, even amongst human experts who effectively are trying to recognize the same patterns, whether you're looking in finance or really any kind of space, it's interesting that the human experts hit diminishing returns at about 95% accuracy. And it's often because that's where experts tend to start to disagree. And so what's been interesting with benchmarking in AI lately is where we can show human expertise benchmarks of human performance and then have machine learning models come in and start to approach that 95% mark. Because that's where you have machine learning effectively as good as what human experts do. But there's a deeper kind of intelligence that we see. And that's where, again, machine learning is augmenting social systems. And so I think that the real thing to look toward is not machine learning is necessarily to machine learning models to predict things, but rather machine learning models to help augment teams of people. You can think of having teams of people and automation. And think about them complementing the relative strengths of both. There's another area that I'll throw in. It's kind of a subcategory. But there's a great talk by Michael Jordan, who's one of the most preeminent professors in machine learning. He's at UC Berkeley Rise Lab. And Michael Jordan made the case that markets are also a kind of codified intelligence. And you see effectively intelligent behaviors in markets. So they're doing very interesting work at Berkeley in terms of economic studies of markets to understand learning mechanisms. Really, really fascinating work. So if you haven't caught that yet, unfortunately I don't have any pointers to it, but he had given this at the AII conference in San Jose a few months back. And then the final area that I'll talk about is really where we begin to explore and exploit very high-dimensional spaces. So as Oscar was mentioning in his keynote earlier today, people are good at maybe three to four dimensions. Some of the kinds of problems that we work with in machine learning, typically, you've got at least 20 dimensions. Some of them you have thousands. So when you get into those really high spaces, people aren't all that effective, but machines can be. And this is where we start to look at really sophisticated math, you know, tensor decomposition, things like that. So we see evolutionary software. We see retribution, or what you might call abduction, in terms of logic. We see generative adversarial systems. All of these are really non-human forms of reasoning. And these are getting a lot of traction lately. So to break it down, you know, the first one, this has kind of been a generally accepted definition for machine learning. And yeah, you know, here's Microsoft where they beat the human standards for speech recognition, speech and text. And again, they were doing about 96% accuracy to get that. And then in terms of augmenting social systems, what's interesting is we have a long backlog. And I know I've shown this a lot before here, but in the context of this, certainly there's a lot of backlog of understanding the intersection between social systems and automation. And this goes back into the 1950s, really fantastic work by people like Fernando Flores and Roberto Matarana, and also other people summarizing them, like Renov Glanville. So I definitely recommend checking that out as kind of a blueprint for AI going forward. In terms of these non-human forms of reasoning, though, these kinds of high-dimensional spaces, certainly we have some pretty interesting things going on there. If you haven't seen this, I may have shown this last year, but if you haven't seen everybody dance now, it's a fascinating study, the video especially, where they can take videos from professional dancers, music videos, ballet, et cetera, and then run, yeah, I'm not a very good dancer. So they can run people like me through that, and suddenly it looks like I'm performing like a ballerina. So really fascinating what you can do with generative adversarial networks that way. And the other side of it, too, is where we're seeing much evolution in terms of attack surfaces. This is one of my favorites. It's an interactive tutorial on GitHub by Kenny Coe out of UCL in London, looking at how to attack machine learning models, especially computer vision models using injected procedural noise. Okay, so as far as AI, there's some interesting components so far. If we're gonna talk about AutoML, we need to understand the bearing that they have, that it has on these different components. You've got deep learning, reinforcement learning, a lot of interesting work in embedded language models. We had a workshop yesterday on some of that. Knowledge graph is really rising up, and it's distinct from what goes on in deep learning, the two really augment each other. There's also another area called weak supervision. And again, I think that these are complementary approaches, and I would really like to avoid using the word predict. I think that the word predictive analytics has been used a lot for data science and machine learning, but I think it also brings with it trouble. The word predict sort of implies that you have a crystal ball. And if people, especially business leaders, come into working or trying to understand how to work with machine learning, and they're thinking that AI is some kind of crystal ball, we all get in trouble. So instead of prediction, I would like to focus on more of how people make decisions, and how we can make informed decisions, whether it's for executives or for customers. I think this fits pretty well with what some of what Cassie was saying in her keynote too. So I'll just go through these quickly, but we'll have the slides if you wanna dig in. I like to try to provide some primary sources. So deep learning has been around for a long time, but we really made the breakthrough in terms of 2012 with AlexNet. Reinforcement learning, you know, it goes back to notions of optimal control theory, again, that have been around for many years. But this has been relatively recent. Of course, DeepMind did really well. Ray is an interesting, very interesting project out of UC Berkeley Rise Lab. I predict that Ray will become a very substantial open source project. And another thing is if you really wanna dig into what's current with reinforcement learning, I highly recommend some of the lectures by Danny Lang, who's at Unity 3D, the gaming company. They've been doing fantastic work with reinforcement learning. Language models, again, we talked about that yesterday at the workshop. There's so much work that's happened in this field. It broke open in early 2018, so it's less than two years old, and it's totally changed how we approach natural language. But there are trade-offs. It's also very expensive to do the latest and greatest, so there are coming to be cost and energy kinds of trade-offs. Knowledge Graph is something that's a little bit, it's been a little bit harder to define. Certainly I can point to some contemporary examples. I think the Fabio Patroni article there, language models as knowledge bases, that's pretty interesting. We're seeing now where we can take language models and use them, like for instance with Document QA, where you can establish a kind of question-answer interrogation of the model. And so it's a way of developing knowledge bases, knowledge graphs. And also I'll point to a really great article from John Bohannon at Primer AI about constructing knowledge bases, knowledge graph. Primer AI, last week, week before last, broke the world record as far as a state-of-the-art and named entity resolution by doing some of this kind of work. John Bohannon led that. And then finally just as, again, as background, when I talk about weeks of revision, I think there's fantastic work coming out of Stanford and now U-Washington and also Waterloo where some of the lead professors in AI have been working on this project called Snorkel. The idea is to have more mathematical modeling, if you will, for the lineage of what goes into constructing a dataset. Because the interesting realization there is that any dataset that we have has human judgment in it. At some point somebody made the judgment to collect the data and to collect the data in particular ways and make particular kinds of trade-offs. And they made judgments about how to name certain kinds of measurements, fields, or columns that were being collected. So when you start to understand in more mathematical terms, more functional terms, how those judgments have gone into assembling a dataset, then you can begin to make more intelligent decisions about how to balance the mix of what goes into a training set for machine learning model. And I believe that this will become especially important in the context of bias and fairness and ethics. Okay, and one person I really highly point toward if you wanna dig into this in more detail is Lucas Beowald who had created a company that was called Figure 8. They were acquired at least a year ago. But focusing for a moment here on deep learning, deep learning is difficult, largely because of the costs. And Lucas has some great advice. Also they built their company off this advice. Number one, an interesting realization that algorithms are not nearly as important as datasets. We have a large backlog of algorithms. And if you look, there's this great study that Lucas was pointing to and I've got the illustration at the bottom. If you look at breakthroughs in AI and you look at the time from when the algorithm was first published to when the breakthrough was recognized, in this study, the mean time was 18 years. But if you look at the time between when the dataset that was required for that breakthrough first became available and the breakthrough being recognized, the mean time was three years. And so across the board we can point to the fact that we have a large backlog of algorithms but it isn't until we really get some sort of dataset that can prove the case and make that concrete. So thinking about the emphasis of datasets to the points there are about active learning and transfer learning as ways to leverage deep learning models in ways that enterprise companies can take more advantage of or startup companies can take more advantage of. You don't have to be Google, Amazon, Facebook, et cetera, just to train deep learning models if you're leveraging transfer learning and active learning. And so this really bears down on auto ML. Really, if you want to approach the most contemporary sophisticated machine learning using deep learning models there are really three ways to go about this. One is you build your own but because of the costs and because the expertise required that's limited to really what we would call the hyperscalers, the cloud vendors, probably a dozen companies. There might be a few more that are adding to the list but it requires a big investment. And unless you own a public cloud your company is probably reluctant to make a very large investment in the hardware that's required especially because the hardware is evolving so rapidly and will be changed out. So what most people in enterprise are doing is reusing models that have been trained somewhere else. And so by the good graces of Apple and Google and Microsoft and others we're able to take those pre-trained large scale models and then do some fine tuning to apply transfer learning to use them for our use cases. A third point though would be can we leverage services? Will the cloud vendors and others provide services through auto ML so that we can take advantage of their infrastructure and be able to customize models for what we need? In the breakdown, when we looked a year ago at the industry adoption of auto ML it was single digit percentage. When we looked six months later it had been gaining quite a bit. For now reuse through transfer learning is that's really gonna be the norm and I've got a sampler of different tutorials if you want to dig into that. But auto ML is what comes after this. So in terms of the business case as I mentioned Ben Lorick and I had been doing industry surveys and we completed three for O'Reilly. They're all free available downloads if you click on the slides it'll take you to the download pages. So amongst these one of the things we found was looking at what the priorities were for enterprise companies in terms of their machine learning adoption where were they spending the money for projects? So priorities were definitely at understanding what's going on inside of models but then right after that we see automated model search hyper parameter tuning basically getting into auto ML kinds of features. Oh yeah by the way and the respondents basically in the latter survey that we did half of respondents were looking toward incorporating some sort of auto ML. When we break it down according to sectors it's interesting finance had less of auto ML I think probably for some very interesting reasons finance tends to want to have more control and more accountability over things not usually the earliest adopters but healthcare actually had a pretty large uptake in this and also within IT itself there's a lot of work in auto ML. And another thing is when we did these surveys we were really looking at what we would call a contrast study. We were looking at companies that have been deploying machine learning and production for five years or more and then we were looking at companies that haven't even gotten started. What's the contrast between those two types of organizations? What has the more mature practices what have they learned? And so here is part of this contrast we can show that the mature practices are doing much more work with auto ML whereas the companies that are sort of in between stage that are just evaluating they're starting to get there. Obviously the companies that haven't started in machine learning yet they're at zero percent. So what this indicates is that auto ML is something that once you get into a practice of machine learning and production in industry you find that there's value in auto ML and it may not be apparent up front but it's a good thing to be looking at. Another thing that I should add from the surveys is we were surprised about funding levels when we did our surveys we thought okay how much are people going to invest in AI? And we thought 5%, 10%, 20% and then just have a catch all bucket for 20% or more. And to our surprise for the companies that are at a mature stage that have been deploying machine learning for five years or more 43% of them said that they were investing 20% or more of their overall IT budget into AI projects. So what we're seeing there is that there's a gap between the companies that are doing work with AI versus the ones that are not and the gap is widening. And when you look at some of the next slides we'll show what it takes for a company to go from zero to becoming competitive as Oscar was mentioning earlier this morning that's the substantial transformation. That's the kind of thing that doesn't happen overnight probably takes years. So in my estimate here these are the hazards if we take a look at sort of a survival analysis view the ones that aren't even getting started yet they're usually stuck because they lack the technical infrastructure. Too much tech debt, data is siloed they really can't get access to the data they need for their data teams. Also another big thing is where the company culture just doesn't recognize the need for machine learning. A lot of executives and board level members they grew up with six sigma. They grew up thinking that uncertainty is a bad thing and now they're having to confront using probabilistic systems that embrace uncertainty and they don't understand it. And another point too which may be changing but certainly in the business units the line units there's a lack of having people who are on the product side who really understand how to translate from the technology to the business use case. So having those line unit product managers really understand what's needed for machine learning that's a big lack in industry. So given these hazards we'll come back and analyze this a bit when we look at some of the later slides. But key takeaways here the firms that have realized return from machine learning they're doubling down on investments. The firms that haven't even gotten started yet they're gonna take years to go through transformations to become competitive. And I don't think that they have years. And I think that what that leads to is probably for about 50% of enterprise firms they'll be looking at mergers and acquisitions. They'll be at risk at some point in the not too distant future. The first mover advantage though the companies that again they're the hyperscalers the cloud vendors they have the big AI teams et cetera. They're monetizing their leads. And AutoML services are one of the ways that they take advantage both of their lead as well as the liabilities that the other companies have. So let's explore that. Now in terms of AutoML I think it begs some definitions. It's still a lot of it is at research stage. So this is a business talk and I wanted to do an analogy from business. And admittedly this is a very gross generalization but in business you can think of workflow as I have some pre-qualified leads. I go out and I do my business development. I find some ways to offer some terms. And we decide on statement of work. We get a purchase order. We go and perform the work. We get the work accepted. We go and invoice them. We collect on the invoice. And that's a way of thinking of kind of a workflow in business. So machine learning has a very similar workflow where we've got data that we're working with and there's some outcomes that we're driving towards. On the one hand we're often in data science we're working toward executive decisions. A lot of what Cassie was talking about. On the other hand we're working toward customer experience and how does machine learning fit into products. I'll really focus on the latter. But it applies for both. So let's say that this is a kind of idealized machine learning workflow. You start out and you do some data preparation and in data science of course it's always a lot of the work, maybe 80% of the work. And then you go toward the feature engineering, feature selection and then from there build some models, optimize the learners and the models. From there evaluating the results which is another area of very hard problem. And then from there you get into integration, deployment, scaling, et cetera for the use cases. So let's break this down. In terms of AutoML, each one of these stages now has vendors and open source projects which are providing AutoML features. And some of the vendors are really looking at the entire workflow, the end to end life cycle. So at the first part we can talk about a couple things. One of these is data cleaning. And how can you start to automate what's going on in data cleaning? I think one of the better projects that I've seen so far is called HoloClean. It's out of University of Waterloo but it's part of the Snorkel project. Another thing that comes in at an early stage is to use meta learning. Which is to say, can I start out a project by using what has been learned about previous configurations for similar projects? And so I think that meta learning has a long way to go. It's a lot of data mining, the institutional history of an organization. Next part though, feature engineering. Feature engineering of course is a very hard problem. It typically requires a lot of depth in statistics, probably a lot more than most data science teams will have. One of the better experts in this area, I would point to Gavin Brown at Umanchester. There's an excellent paper they have on the stability of feature selection and how that leads to workflow reproducibility. But there are features now for beginning to automate the feature selection, the feature engineering. And this is very promising because it's one of the areas where AI can augment where most of the people doing the work are probably finding it to be very complicated maybe out of their depth, myself included. Training models, when it comes to having your data prepared in a way that you can begin to train different models, this is often called hyperparameter optimization. So the idea is, how can we control the hyperparameters that go into building a model? And then once we've developed a number of different models, we'll go and compare them. This is where traditionally the first starts of AutoML had come out of projects like HyperAlt. And a lot of people still think of AutoML just as this part. One of the messages that we got out of doing this industry research was, no actually AutoML is the end in life cycle. There's much more than just this optimization part. But it is where there is a lot of substantial cost savings, particularly in deep learning. So some of the companies there, I mean, as far as like really sophisticated hyperparameter optimization for deep learning, there's companies like Determined AI that are I think having a really interesting offering. Once you train models, what's interesting is that the evaluation of different machine learning models is difficult. The different algorithms used to train learners have different means of evaluating. It's very tough to across the board compare these apples to apples because they're different kinds of evaluation. Sometimes you're looking at R squared, sometimes you're looking at Genie. Depending on what sort of algorithms you're working with, they're entirely different kind of metrics. And if you don't understand the subtleties on what those metrics mean, when it comes time to deploy and put this in production, you can get in real trouble. So having more automated means to augment how you do the evaluation, that's very important. And the other thing that's coming into play here too is where once you've created a number of machine learning models, you may have several that are relatively similar performance, but they're good in different areas. They have different strengths and weaknesses. And so that's typically when you would use ensembles. And some of the auto ML services are beginning to provide ensemble creation, which I think is a very natural thing for this step. And then finally, as you start to move models into production, there are a few different areas there that are very important. So for instance, SageMaker has had very sophisticated services on AWS for doing auto scaling. And that's certainly part of moving models into production. There's a whole other area of ML ops that concerns this, but I think this is where the machine learning workflow starts to hit into the operations side. The other thing is a lot of machine learning these days is going into embedded products. They're not necessarily running on a cluster somewhere. They may be running inside of a smartphone, or they may be running on an edge device, IoT. So for that, we're looking at issues of model compression, distillation, and other ways to take a big machine learning model that's been trained on a cluster and shrink it down to fit into the form factor of the embedded device, both for size and for power and still making sure it has the kind of accuracy that's needed. So those are the categories of what we're seeing with AutoML, but there's one real problem here. When we look across the board at what the companies are offering, I do see a lot of claims, especially some of the smaller vendors. And oftentimes they'll say that they're going to democratize AI or machine learning and that they're gonna optimize models for you, but they don't explain exactly what they're optimizing. When we use the word optimize with machine learning, we could mean many things. Even if we're talking about accuracy, I can spell out a few precision recalls, specificity, perplexity. They all mean different things depending on the use case and they can produce wildly different results if you confuse them. But there's more to it than accuracy. There's also a matter of understanding the uncertainty and being able to report that and leverage it. There are different types of costs. There's the cost it takes to train the model. There's the cost in terms, I mean just in how much does it cost to run cloud computing or your own data center to train a model? But there's also the cost in terms of time. How long does it take to go from training a model to getting it out to production? So we can think both in terms of monetary cost and time and those trade-offs. There are many other kinds of areas, certainly model drift, fairness and bias, privacy. All these come into play in terms of how do you optimize a machine learning model? So I think that one of the key takeaways here is when there are claims in industry, we should be pushing back and we should be saying you don't want to just optimize one dimension, you probably want to optimize about 20 dimensions. And along with that, one of the problems of comparing AutoML is there's really no good public benchmarks yet to evaluate different offerings. I think this will evolve, there's certainly some academic steps toward this, but minus having that it will be difficult to compare the different vendors. So the landscape is if you go to the link here, we've got a public spreadsheet, Google Doc, and we've been listing the vendors and also the open source projects. So for instance, we have several vendors here who are doing really sophisticated work with AutoML, certainly Astratio and DataRobot, Microsoft, Google, IBM, Amazon, I know they all have really interesting offerings. And what we're trying to do here is break out who's offering what. So for instance, with DataRobot, they have a very interesting approach. It's the end-to-end life cycle, they're augmenting what a data scientist or a programmer would be doing in their daily work, whereas some of the others I mentioned determined AI, they're more focused on just a particular kind of offering optimizing deep learning hyperparameter selection. So I would love to get feedback about this. We were trying to collect more market, or sorry, industry intelligence about this. So if you have anything to offer, please email me or catch me on Twitter or so. And also just tracking the open source projects and what they're doing there. I think that's one of the key messages coming out of this talk, I'll have that at the end. So if you want to go into more detail, I know this is supposed to be a business talk, but if you wanna see some of the real detail about AutoML and what the latest research is, I've got a couple pages here of recommended articles and papers to read if you wanna do more of a deep dive. The first one is probably the best, sorry, awesome AutoML papers by Mark Lin, and he's updating that regularly, so if you go back and check it, every week there's new stuff there. But that's probably your first go-to stop to see what's the latest. And certainly for an overview, I think that Microsoft paper about what Microsoft learned, how software engineering had to change because of machine learning. I think that's also a really great study. And another point here too, what we're talking about in terms of AutoML, perhaps maybe it doesn't happen as a service per se, maybe how it begins to get integrated into the workplace is projects like this. So this is where DeepTab9 had worked with embedded language models, GPT2, doing very sophisticated natural language work applied to GitHub repos. And so if you can learn to predict natural language, sorry, I used the word predict, if you can learn to understand sequences of programming by learning from GitHub repos of source code, then you can build out auto-completion, very, very smart auto-completion for programmers. And I have a hunch that we'll see a lot more of this in the near future. Certainly there's a lot of program synthesis, projects like auto-pandos to automatically generate pandos and related things for SQL. So I think we'll be seeing more of where this is where the smart gets injected into the workflow. Another thing that's really changing this, I've shown this slide before, but I can't understate it, hardware is changing and evolving so rapidly. It used to be that we talked about software engineering and we would say that process is very general. You have agile and it applies to a wide range of different projects. And then software is also general, abstracted. Hardware, it's way down low, you hardly ever mess with it. So, you know, focus on the process and the software. But over the past few years, that's entirely flipped and now hardware is moving faster than software and software is moving faster than process. And the companies that don't understand that are mostly bewildered. The companies that do understand that are really taking advantage of it. And the hardware that's changing is not just in terms of processors, but we're seeing really intelligent front-ends on memory fabric, what's going on with switches, just across the board in terms of edge computing, low power ML. But certainly the processors too. If you look at like the wafer scale that's come out of Cerebrus recently, where they have 1.2 trillion transistors on one chip and the memory is 40,000 times faster than the largest GPU. And there are some projects that are really using this, like Jupyter and Aero, some of what's coming out of Ray as well. And so I think one of the punchlines here to really deconstruct what's going on with AutoML. This was a talk from Jeff Dean and he was saying how the current bottleneck, if you will, is that to have a solution for doing machine learning and production, you have to hire the right machine learning expertise, you have to have the data, and you have to have the computation. And what he's saying here is that if you go back to the hazards, where it is that companies get blocked in terms of moving machine learning and production, a lot of it has to do with the talent crunch and they just can't hire up enough of the machine learning expertise. Now the flip side of this is that Google and other companies very similar to it have spent a lot of time acquiring as much of the ML expertise as they could.  And the way that they sell it to us is by having us buy a lot more computation. And that's kind of the punchline of AutoML. So we can talk about that more in detail. I'm kind of running out of time, but I think that it's a little bit of a cynical view, but it's also a very realistic view. And so to finish up, I talked yesterday about data governance and Oscar was talking about the importance of data governance. And I'd like to show how that fits here. So there is a talk if you wanna dig into about data governance, there's a lot more resources there. And so here's the notion, when you introduce machine learning into production, one of the first things that happens is that your operations change, your business operations change and the kind of metrics, measurements that your operations team needs to collect are very, very different. I did a track at Oskan this year. IBM had sponsored an ML Ops day and we had fascinating case studies from Capital One and Comcast, GitHub and others. Probably the best out of that was a talk by Donald Miner who he's been giving a webinar in other talks entitled Vital Signs, where from an operations standpoint, what changes with machine learning and what do you need to measure? What do you need to think about as an operations team? So ML Ops is emerging. Meanwhile, we're seeing since 2018 a kind of global reckoning about data governance and data governance is being taken much more seriously now. So data governance is another way of saying getting your data in order so that you can leverage it with machine learning. So we really see where these three thrusts start to co-evolve. Number one, you do have to have the data governance in place, it's a non-starter if you don't. Number two, you have to embrace ML Ops or you won't be able to manage your operations well. But if you do those things, then you're in very good position to start to take advantage of auto ML. And again, I think that Oscar had similar remarks about this with what Stratio is doing. Certainly if you talk with Data Robot, they'll have a very similar view. But I would say look at these three areas as co-evolving. I think eventually they'll become one general category. And to show evidence of that, in particular, I'm really interested in watching this, these are companies which are not the hyperscalers. They're not the cloud vendors. They're sort of a younger generation. So Uber, Lyft, Airbnb, Netflix, LinkedIn definitely falls in this category even when Microsoft bought them now. But what's interesting is when I went out to do some of this research, I found that all of these companies have projects where they collect metadata about data set usage, the business usage of their data, and then they build knowledge graphs to describe, to represent that metadata. And number one, they're doing it because of compliance. GDPR was sufficiently vague and so they had to respond. This is one way to respond. But then number two, once they do it, they realize there's business upside. They have a lot better view on where is their money being spent in data science? How can they train people or provide better tools to offset some of the repeated costs? And Lyft in particular has found pretty incredible cost savings on that. And the other part is just it allows them to get a global perspective of data sets being used in their business and how they can then perhaps even launch new business lines. Certainly in case of Lyft and Uber and companies like this, they've made their money by monetizing data sets in the statement. And so if they can find new ways, new combinations of data sets, new ways to monetize those, that's how they grow their business. Most of these projects are open source. So I think that there's some excellent vendors in the AutoML space. There's also these supporting open source projects. And with that, I'm out of time. So thank you very much. I look forward to answering any questions. Do we have time for some questions? A minute. A minute, okay, one question. And then I'll go outside. I'll take any other questions. Anybody? If not, I'll be outside. Yes! Can we predict how many questions there'll be? No, sorry. Oh, why that?  That's a good question. So yeah, I mean, the prediction is going to be kind of a mechanism. But I think it's dangerous. I think instead, we need to understand how uncertainty works. It's not simple. It's not one dimension. So if we can look forward in forming decisions, I think that's a better sort of cognitive framework to go into the business applications. If you're expecting new prediction, you're expecting a crystal ball. And I mean, I think that it may be just a matter of time too, because we do have a lot of people, how shall I say who are my age or older who are on board or directors? And when they heard the word prediction, they think about something that's too simple. Yeah, great question. Okay, I'll get outside. Thank you much. Great, I hope I didn't go over too.


They don't trust you. They're not going to let you go. They just want to join infantry and just like do something as far as like serving my country. I'm going to come look at the map, Cox. I'll bring it to you. Here's is like that's all we got out here.  Yeah, I know. Give me a second. Corporal Cox, scout sniper. Born in New York City, New York and moved to the Bronx. I love New York.  I got a long gun. I could see further. Like I remember when I did my first deployment to Afghanistan as an 11, I was like, I could barely see these people. Definitely instill some fear in the enemy once like they get like the close shot to them. Y'all like two people that keep squatting up and down. I'm about to range it. I definitely think I was like this before the Marine Corps. It's just kind of grown as far as like dealing with things. You can't just complain about everything. So it's got a truck on, roll with the punches. It's kind of helps you in life too. They always take the good things from people and just try to instill them with yourself and then just see the bad things. Definitely got like thicker skin. Just deal with it together kind of. You don't want to go through that experience alone. My team leader tells me what he needs to get done. I help him out. I could take my own element. We work really good together. I think you just have to have like a good person that you trust. Kind of like one of the guys, but he's firm too. It comes to a point where you don't want to upset him. Yeah. Like because you feel like you let him down and that hurts you more than anything. But there is a guy to the, like if you go to the left of the hose, if you go to the edge of that compound, there is a guy like over there that was squatting too. I love operating. Not all of them are home runs. And then you do have some snoozes, which is like, it happens. So gotten older, thicker skin, smarter. This is what I always wanted to do. So like people was like, oh, I wish I always joined the Marine Corps. Went to Afghanistan, but they didn't. And it's definitely like one of those things like, I don't have to think about what it's like, my wish I did this or something. Like I could grow old. I don't know, I'm just me.