Hi everyone. My name is Mal. I'm glad that you are here. My work experience just a little bit about me is started off working at McKinsey and Company in the Chicago office after graduating from the University of Michigan, moved out to SF and have worked at Circle Love and Waymo and today we'll sort of go over the basics of being a machine learning product manager and give you the overview of what the role actually looks like and a couple of things that this group can do leaving this session in terms of getting prepared for the role and sort of excelling at that role. I want to kind of caveat this the session by saying that a lot of these learnings experiences observations are from my own experience so take that take that as you will. So let me share my screen. Awesome. So being machine learning, a machine learning product manager 101. So just to start in terms of the agenda, I'll first start by actually walking through what are the types of machine learning product managers and the roles especially as you're going through the application process and you see a specific job. It's sort of important to understand the differences. Second, I'll go through the types of machine learning that exist at least like the basics that you should understand if you're kind of fresh into this topic. Third is giving you that overview of what exactly an end-to-end ML lifecycle actually looks like. Fourth is really digging into my own observations around where machine learning has gone wrong or where there tends to be a lot more complexity and where most organizations face challenges. I'll then kind of continue on and talk about what the exact role of a product manager is within this space. Talk through a few examples of where I think machine learning has done a great job or thrived and then a couple of examples around machine learning applications that have gone wrong and then I'll sort of give you a final summary around takeaways and what you can do to prepare for a role in this sort of space, this technical space. So to start things off, I think it's really important to understand that there are actually two types of mesh product manager roles that are out there from what I've seen. There might be a ton more nuanced topics that bleed either one or both of these or somewhere in between but for the most part there is typically something called a data product manager and then a something called a machine learning product manager. A data product manager actually focuses on the foundations of the organization getting them ready to apply machine learning and for me that means things like making sure that the data and infrastructure are in a good place so that data scientists, engineers, machine learning engineers can actually work on top of that data confidently and can actually do it quickly efficiently name your adverb there and so they're typically focused on things like data infrastructure, pipeline health and data readiness and really think about things like how do I think through the problem of data acquisition? What sources are we actually putting into our data warehouse? Is it sources coming in externally? Is it sources internally? How do we clean that data? How do we make it ready? How do we stitch it together? What are some issues that might come up around data quality that our team needs to be aware of? Overall, how do we make sure that downstream applications or use cases on top of that data are successful? They think about things like access controls and ease of data retrieval as well and so the key metrics that sort of this product manager focuses on ranges from anything as fuzzy as data quality to something as measurable like query traffic, latency, reliability, error rates, response time and availability which really signal how good that data is, how quickly it is able to be accessed and so forth. The second type of product manager is actually a machine learning product manager that tends to focus on a problem area for the most part. I think the background of this person is usually they were a data scientist or they understand a lot of the trends and core concepts of machine learning and they're able to apply it effectively to solve a problem. So typically a machine learning product manager is oriented towards a problem area and that problem area might be I'm trying to accelerate this end state business outcome like number of customers, conversion, revenue, growth specific product line and they believe the organization believes that a machine learning lens is the right way to go accelerate that business problem. So you'll typically be hired to sort of own the machine learning application to go accelerate that outcome. So that first piece is actually vertically towards a product or a problem. But the second type of machine learning product manager you might come across is actually a horizontal capability type of product manager. So someone that focuses on a capability like pricing or experimentation or some sort of capability that the organization wants to sharpen with a machine learning lens and so they hire this person and this person is typically centrally oriented in the organization they can sort of farm out this capability to multiple business lines, products or problems that the organization has. The key metrics that this product manager typically owns from top to bottom as they tend to own the business outcome themselves because the organization believes machine learning will actually accelerate that outcome. Second is actually the experimentation results. So if you're building a model or a suite of models the actual outputs or those those experimentation results that come from should I actually deploy this model in production or not is something that you would own. And the third is actually the nitty gritty performance metrics to make sure that the quality of the model needs a certain threshold that you feel comfortable actually deploying it and making use of it to actually impact something that happens to the business. So kind of taking a huge step back in terms of types of machine learning. I found this diagram I think it's somewhat captures what machine learning applications are in the simplest sense but largely you can pretty much only do four things. The first is predict stuff. Second is classify stuff so you can kind of classify the very you know classify something as a you know hot dog not hot dog is I think the most classic example. You can group stuff or you can cluster things based on how similar the attributes are to make them group together or you can actually teach stuff which is that reinforcement learning piece. I think it's really important to keep in mind that machines or machine learning can do all of these things only as long as it doesn't look too different from the data that it's training on. So the machine itself tends to understand patterns and data and then take those patterns and apply it to unseen data or what it doesn't know. So because it's learning from a handful of data it'll typically only do what it has seen before. So that's that's the idea of training. So it's always important as a product manager to understand the the biases and what exactly you're training on. So you have an understanding of how those biases might carry through to what you end up doing or predicting or cost me on on in terms of the unseen data. So I obviously think the the audience here might be beginners to more advance in terms of in terms of the the sophistication around understanding machine learning. So just to take it down to the super super super simple what exactly is machine learning. To me it's sort of like the simplest example I can give is if you're trying to predict the value of a home and so in this example you see 10 different records of you know different features or variables like number of bedrooms number of floors whether the house has a garage or not the school district value the square feet in the city and you use those variables or features to actually predict what people call the outcome or response variable which is the home value itself. And so you see at the bottom here that there's a handful of features or inputs or variables and then on the right hand side data scientists would actually predict the outcome or response variable. And so for the the you know next step in machine learning it's sort of this idea that if you want to predict some of the values which are the home values themselves you can actually cover the unseen or that cover a few records or what folks would call this unseen data and train on a handful of data and then say well knowing the weights of those variables which are how much should we weight the number of bedrooms or the number of floors or their square footage or the garage or not. Can we actually come up with this weighted equation learn the equation to home value and be so good at it that we can actually predict the home values just using those variables alone which are the number of bedrooms all the way down to the city. And I think the typical answer would be if you were kind of starting this fresh or from scratch yes I would be able to do something like that and give you an idea of what those weights actually look like but only if the data that I see now is representative of the weights that would apply to the data I can't see. So that's an assumption that goes in that the data that you're training on is representative of that unseen data. Oh and I think a lot of you would say I'd probably do a better job if I can practice my weights by hiding some of the data or the records myself and seeing how well I do in sort of this in the records that I do have access to. So imagine in this row of eight records you hide the final two, train your model on the first six and then see if you were able to successfully predict the seventh and eighth record from the data you do have access to and sort of cycle through that process over different folds of the data that you have actual access to so that you know that your your model is quote unquote generalizable and then from there you can finally uncover the data you've never seen before which is the two last columns described in black and then say you know I practiced I've covered a couple of records in the eighth that I had and sharpened or optimized those weights and now I'm confident that if I uncover the final two records my weights are going to be representative and I'm going to have the best weighted equation or prediction of home value. So super super simple here's a little bit more you know a little bit deeper into what that overall ML life cycle looks like which from our product manager standpoint you really have to understand how this works. The first is is just making sure that the problem statement and the success criteria for your data science team or your engineering team is crystal clear which is what exactly is the goal of the model that you're building or the suite of models that you're building. How do the performance metrics or the outcomes of the models themselves tie to business outcomes so how does for example the precision and recall of some sort of classification model that your that your team is working on tied to you know metrics that matter for the business. Third is how do you know how will you know if your model is actually performing sufficiently. Understanding that and outlining it and making sure that's published for your team is a way that you can not only you know define success for that model or set of models but you sort of know when resources need to be shifted from a handful of models that are that your team is currently working on to a new set of models. I think the fourth thing is just making sure you have a good understanding of whether machine learning whether your organization is ready for machine learning in a lot of startups or small companies teams tend to jump first into machine learning without first understanding the readiness around data infrastructure and metrics experimentation which we'll go into and it's important to just understand the health of health of you know deploying something like this before you jump into the most complex answer. Second is actually data preparation how high quality is the data you'll be working with how far back in time does it actually go is it representative of that population we talked about representative or how data sort of learns patterns and what it's exposed to and then how much training data do you have it's really important to understand these questions this again sort of gives you a sense or a score if you will of the readiness of your organization. Third is actually training so training refers to how your model learns this is sort of the steps that model parameters are typically determined or they are determined which is those weights how representative your training data is ultimately dictates how generalizable the output is to the population you'll sort of end up making claims about so it's important to understand the heartbeat around what your training data is how easy is it for you to collect it how representative is it how much of it do you have and making sure that you have like the biases understood so you know you know how applicable generalizable your predictions or outcomes actually are fifth is your testing so testing your model refers to how it performs an unseen data to avoid overfitting or conforming into just the sliver of data that you don't see that you'll be predicting on to assess the quality of your model data scientists will do something called cross validation to build something more generalizable which is of the training data that you have or those eight home record values the top of that you know record sheet from the previous screen you kind of hide a couple of those records yourself train on the rest and then see how you perform on the data that you do have access to so that you can kind of understand or generalize the weights in the best way possible and then you'll sort of shift the records that you don't have access to and do it over and over again across k number of folds which is what cross validation exactly is and then finally you'll deploy so you have a model it works you've built it in your notebook but it actually needs to be able to be consistently you know predicting or classifying or helping the business every week every day every month so doing that actually means putting it in production which is a lot harder than you think especially because there tends to be a lot of inputs or features there needs to your production model needs to take into account how you feature engineer you need to consistently output those predictions and do it on a regular cadence and make sure that that entire pipeline works successfully on a regular basis and so digging into this a little bit more I've noticed as a product manager that there's two things that typically go wrong the most which is the first and last thing I think people might argue that data preparation or understanding the biases as well is super important but for today I'll go through the two examples or the two pillars of where things go wrong starting at the problem statement and success all the way to actually deploying that model and making sure that it's successful I alluded to this sort of organizational readiness framework a little bit earlier but for me this is tends to be a problem when you have a young data science team you're trying to get your footing around how to use advanced analytics in your organization and so what this pyramid is is like my interpretation of how you assess organizational readiness starting at the bottom which is the problem followed by the data infrastructure needs then the metrics then the experimentation and then finally the optimization or actually the machine learning that goes on top of all of those core components so starting at the very bottom it's really really important to understand the problem what problem are you actually solving for the business and how are you gauging that the use case is amenable to machine learning I think that problems where machine learning tends to do or use cases where machine learning actually thrives will go into the very really clearly later but just to give you a little bit of a taste it tends to be when you know training data is easy to collect you have a lot of it it's representative of that population there are fast feedback loops you're able to measure you know the implementation of your machine learning model versus something like a status quo understand how much better it actually is towards these business outcomes and then make a conclusion that it makes sense to actually move forward with the machine learning model in a specific use case so in framing it you really have to ask yourself does this problem have fast feedback loops or ways to assess if we're making better decisions because of this model do we have a lot of data to train for this problem is that training data representative how easy is it to get more training data and maybe maybe even more thoughtful would be if there are tons of you know failure cases around where the model doesn't assess you know specific classification prediction whatever maybe well is it easy to get more data on those errors or failure examples to asking yourself that question is important because if your model consistently doesn't perform well in a specific use case how is it easy is it to get that data to make it better or improve the model and what exactly would success look like for you when you actually output this model you have a handful of performance metrics and how that ties to something that the business actually cares about the second is actually assessing the health and of your data and infrastructure and so for me this means outside of the kind of examples that we've talked about how high quality is the data that you're working with you have an ability to stitch together data that have commonalities and and be able to actually put together a feature set like features set of features that you would use for this model how do you even define data quality in your organization make sure you document that and circulate it and that might mean that you know you go through a subjective assessment and then you sort of go through ways you can quantify that in your data set and how easy is it to maintain I think there are lots of research there's lots of papers and actual you know experience out there that says that most time spent is actually around cleaning and making sure that data is accessible and ready and usable and so assessing that is really important to seeing how much of my time of my team's time is going to be spent just making sure that data is even usable and that data and infrastructure is healthy enough to build on top of the third type of assessment that you have to do in terms of organizational readiness is understand the metrics I think one of my mentors had shared this with me and I think it's really a nice way to assess you know the the value of the model is just apply a simple heuristic to advance the problem forward and and apply heuristic or an equation that you think you can just do based off of heuristic or human intuition and then from there sort of take a step back and say applying this heuristic am I even able to measure how it is better than the status quo and if you can't articulate the outcomes of applying a heuristic and a measurement and how that measurement is better than the status quo I would suggest you don't necessarily want to jump into machine learning because you don't understand what metrics you're actually comparing against if you can great at least you know what business metrics to improve what that measurement looks like how do you compare to the status quo and how you advance the problem forward the third kind of component here in terms of metrics is understanding your performance metrics and how they relate to this business metric prop this metric business metric improvement this is important for teams and morale overall this idea that they need a north star around how their outcomes tied to overall business value but also more importantly is you have to sometimes make these decisions around this set the this this team is done working or doesn't necessarily need to spend as much time working on this model or set of models let's sort of move them over to thinking about something new or a different problem and only when you understand performance metrics and how they tied to outcomes can you actually make those resource shift decisions effectively the fourth you know piece of this this pyramid is experimentation which is how do you even know that they're your models ready to be put in production and how does it compare to the status quo what metrics are you assessing under what time horizon and what is your experimentation framework and so this is for relevant for companies like Netflix that actually have you know some level of an you know model that they want to put into a production because it affects something like customer conversion in terms of what you know folks actually watch on their platform you sort of test it you have an experiment that you're running you assess that in either like a confined or dev environment or you tend to kind of field it out in a smaller set or smaller geography to understand what those conversions actually look like and under what time horizon understand the demographics that are present there and all the biases make a conclusion on those experimentation results and then eventually roll it out to the rest of the population so just understanding your experimentation framework and how you go about making decisions is something really important to document understanding the fallbacks with that experimentation framework and then and getting that alignment within your organization and now finally once you've understood all four of these things it's time for Amal at the very top because you've understood your the problem you're solving the data and infrastructure health the metrics very simply that you'll compare to see your advancing the problem forward experimentation and now optimization i i kind of threw this caution but folks tend to over invest without really understanding how much cost is associated with acquiring the data infrastructure resources to build out and maintain the model and production environment i'm a huge sort of fan for simpler is better but making sure you understand all those things and then sort of end with with machine learning at the very top so i alluded to two sort of parts that go wrong with with ml the first is just kind of understanding the problem and that pyramid structure but the last kind of piece is actually the iteration overall of machine learning and putting something in production and iterating through this process of data model training test evaluation of models actual outcomes putting it in production and then recycling through that entire process over and over again these types of cycles are relevant for small companies that need to monitor the health of their models are probably way more complex meaty than in companies like wemo google facebook etc that have these like huge models that they're actually working with and all of those models data inputs actually change on a regular basis so you need to be able to monitor it pretty pretty well and so from the very top here the data itself requires a really clear strong understanding of the annotation schema that you use to actually train your model product managers should have a strong understanding of how labeling works in the organization and then overall improving your model generally involves two things understanding those failure cases that i talked about and then making sure you have more data that you can acquire that helps you get better at those specific instances and then rebalancing your data set based on any biases biases that your model has learned and so just to take a huge step back one thing that i would love to kind of leave this group with is pms tend to need to know those biases and data assumptions pretty well so they know how generalizable things are and as new data comes in they can understand how their current biases might you know apply or not apply to new data that's actually coming in the second is actually the the model itself and the two examples here sort of call out in terms of complexity of maintaining an ML model is actually this idea of experiment tracking so large organizations will run tons of experiments but it's really important to understand different versions of the model the parameters the hyper parameters that are actually going into these different versions and being able to track well which one of them actually performs the best and with so much complexity it tends to be it's really important to sort of maintain the versions of different models and have the health of the system maintained in a nice way there are tons of tools out there now that help you track these types of things and and version control if you will but the the complexity as organizations get bigger and bigger definitely increases so having that monitor well tracked well and transparently communicated to organization helps you make make decisions the third thing in terms of ML complexity things that go wrong is actually the evaluation so it's really important to be able to visualize the outputs of your models you know metrics that come out I think a lot of organizations tend to proliferate or have too many metrics so it's important to make sure that there's a clear understanding around what metrics matter and how do we go about making decisions how are those metrics tracked how many people have access to those metrics and do we need to change them at all moving forward how do you dig into failure cases observe them explore them you know get more of those failure cases and then repeat the cycle over again the evaluation part tends to be where there's a lot of complexity around you know making the model better digging in further problem solving required simplification required so a lot of the larger organizations do heavy ML tend to spend a lot of time on eval and simplifying it and streamlining it for for organizations to have tons of data scientists that are moving in the same direction and finally the last piece here is just around production or monitoring the model health as it is put in production how you evaluate new data coming in and what edge cases and biases that you actually need to watch out for we talked about this a little bit so I'll move on so to kind of synthesize what the role of an ML PM is I have done all of these things so I guess it's it's something that I feel like is important to understand in terms of what your role is within a data science team the first is really just setting the problem statement what business problem are you actually solving second is actually validating the org readiness so how do you measure success and is that data in for a metric experimentation pyramid ready and what is the cost of actually maintaining it so that you know that your organization is ready for an ML use case third is ensuring that the value is measurable so how do performance metrics actually tie to business outcomes so that you can understand and sort of work through that resource alignment question that we talked about fourth is actually defining requirements so this is one of those things that we didn't get a chance to talk about today but I will put out more content just around how we can how product managers do this around writing out a prd for a model and for me that's a lot about you know does that model need to be interpretable who actually uses that what decision isn't making how complex does it need to be how what did the performance metrics need to be for it to be sufficient to put in a production etc defining those requirements both from outputs like we described interpretability complexity performance metrics etc in addition to the inputs is the training data representative what kind of training data that we need how labeling work in this ideal state what biases and labels do we need to avoid those types of requirements are always outlined from a pm perspective in this space fifth is experimentation success so validating that go-to-market strategy biases and when models should be deployed writing out experimentation framework how you would actually test that experimentation works and then eventually roll it out to the rest of your population sure a lot of pms and roles like netflix probably spend a lot of time thinking about this and the final pieces around resource alignment what problems should the data science team be solving today next six months next one year from now and i think the the cherry on top is are we ready to be a you know solving those problems given that pyramid that we talked about one year from now or do we need to start preparing our data metric experimentation etc so we're able to solve that data science problem one year from now i think i've talked said the word performance metrics about a hundred times and so performance metrics are in the simplest sense the actual model outputs they're a handful of classification regression metrics that are out there in the world you have to have a really good understanding of exactly what these metrics are and how how you know if they're good or not and then how you can actually take these metrics and tie them to these business outcomes and so having a good heartbeat of these and studying them if you will is a good way to sort of get enter the space and speak the language so that you're able to be thoughtful around connecting this to a little bit more of the qualitative problems that come up in an organization i also talked a little bit about where ml typically tends to go wrong versus right and i think that it tends to go wrong for a lot of the use cases i've seen before and so i've sort of done my my classification of these two things the good use cases is are when ml itself is the product when you have b to see large-scale products you're able to run the ab testing or experimentation to know the outcome of the model and how that compares to something like the status quo um when you have small changes and that it has a big impact so if you are running you know google ads and there's small changes have an impact of millions of dollars in revenue you can sort of have that data science team that really focuses on sharpening small changes to the actual model so that you can reap those benefits in terms of large-scale revenue impact to the business fast feedback loops are really important so that you can train collect more training data and assess a versus b clean for classification it's easy to get training data i think those use cases are awesome for for machine learning and then finally ml big bets this is sort of the way most of the world where the ml itself is the product and it's a good application of it bad use case are just as important to sort of outline seen a lot of teams actually try to do b2b account acquisition or conversion with really limited training data uh not as much historical data number of points in terms of training that we you can actually use to determine something like a lead score for account acquisition i've tended to see those things don't actually turn out great um so that's that's what i would characterize as a bad use case any sort of pricing strategy where you only have so much months so many months of data or so much training data is typically a bad use case anywhere where you don't have the experimentation to test value or i think this last piece here is a super super nuanced decision making where you can't even get a foot set of people in a room to decide what is a good versus bad outcome or have that consistent training data to be able to say the super nuanced human decision making doesn't have a clear labeling that we can actually apply so that we can apply that to you know a next set of unseen actions so the takeaway sort of wrapping things up are ml product managers and this is sort of that second type of product manager that we talked about at the very beginning is um having a really good heartbeat of the problem the readiness of your organization the model bills and how it actually works the metrics that come out of a model the evaluation so how people make decisions and then how do you actually put it in production um understanding that entire iteration process and mapping that is typically the first thing you should do when you get into an organization not only to assess that org readiness but understand the problem and then understand where there are bottlenecks where things are breaking that you can help influence and and sort of help move smoother number two is actually understanding outcomes and knowing your performance metric so making sure you take the time to read up on the statistics that you need to um so you can speak the language of your team third is actually understanding the data assumptions so you have an understanding of the biases of both the inputs and the outputs and when you data comes in you can inform generalizability based on your knowledge of the biases fourth is just practicing that ml toolkit of problem you know data infrastructure metrics experimentation and model you know optimization in traditional ways where you can apply that even using something as simple as a heuristic and seeing if you can put together an experimentation framework is something is a place to start and then just keeping up to date on foundational skills new trends new things that are coming out and and being able to bring that to the table as a as a way to shape the next six months one year for your team we did go into a lot of you know technical concepts like you know the linear models and the basics of them ridge versus lasso what are what is a neural net three base models and all of these technical terms that will probably come up in your in your time as a product manager i plan to put out a series on just simplifying or stripping it down into the most simplest terms and walking you through a series of of models themselves and how they work and and how they're applied in terms of real world applications and so follow me on twitter where i'll be putting that content out but hopefully this is the first good step in terms of overview and then we'll sort of dig into a few topics that are important to understand our technical topics and and get into the details of those so thank you for the time um


Welcome. It's it's fun to be here. It's clearly the year 2013 because about a third of the people here holding up a cell phone trying to take pictures of us. It's always fun to do these these little conversations and see where it gets us. This session is basically your session. I've brought a few questions that I will pepperliness with, but I actually really want questions from the audience. We have a few runners here with microphones and if you have questions raise your hand and and we'll take it, but I'll start out with a couple that I brought and I'll start with the obvious one. What about world domination? No, forget that. 312, where are we in the process? So right now we're pretty late. I timed this well. I actually tried to time the releases so that when I'm traveling, we're not in the middle of the merge window because that just makes it really annoying. And having said that, the next merge window I will be traveling. But we're getting fairly close. The last few releases have been fairly easy. I mean, we had some issues, but we've used this whole process with merge windows now for several years, six, seven years, and we're getting the process down and there's still every time there's some issue that comes up, but I think we are getting much better at just handling things. Right now 312 is one week away, roughly a week and a half away, and things are looking good. So you say the process is working well. I wonder, are we focused too much on speed, too much on adding new features? Should we maybe take a step back and say we're doing a release with nothing but stability and bug fixes? At the kernel panel yesterday, there was a mention of 4,000 known bugs in the latest stable kernel. What's your take on that? So, I am so personally happy with the fact that we don't worry about features. And I mean that both in the sense that we don't try to do releases by new features, but also in the sense that we don't have these long freezing periods when we don't have any new features, and we do all the releases purely based on time, because whenever you have something that is based on, okay, now we have to have for the next three or four months, we're not adding any code at all, we're only fixing bugs. That gets everybody really frustrated and what ends up happening is people will stop caring, work on the next version when you can start adding features, and then we have this huge backlog. So one thing that I think has been working really well for the last several years is that we have been purely based on, hey, if it's ready now in the merge window, we will merge it. If it's not ready, you can wait, and it's not that long of a wait. And not worrying about the feature creep or lack of feature creep too much, and instead worrying about getting the process to be smooth and making sure that everybody is okay with the fact, but okay, if you don't make it, it's going to be only another three months, right? So don't hurry your code, make sure it works well, make sure it's well designed, make sure people are all on the same page, and you don't have to worry about timing. And that makes things so much easier for me as a release person, but I think it also ends up, well, I'm pretty sure it also makes it much easier for all the developers and all the companies that they don't have to really worry about, hey, we need to hit this particular date because they know that if they miss that date, it's not that big of a deal. Well, it's actually, for the last many kernels, it has been less than three months. It's usually like 10 weeks. It's usually about 10 weeks, yes. I mean, we aimed for two months originally knowing that will slip, and three months is kind of my, that's the longest it should ever take, and usually it's nine, 10 weeks, yes. But what about adding a week delay after the release before you open the merge window, just to let things settle a bit, let people start using it, and maybe give Greg a head start? Greg is doing fine. I didn't say he was. Greg is okay. Are you here, Greg? He might be, oh, he's over there. I'm not seeing any gray hairs, right? You're not seeing any hairs is what you wanted to say. We've tried that never actually a week, but I sometimes want to have a couple of days, especially if I'm slightly tentative about a release, and we had some late minute issues, I want to say. I am not going to merge stuff. I want all the developers to really now be on the release, and if something comes up, because people are starting to use it more, because we just made a release, I want you to work on that. But that's only happened a couple of times, and I think it is counterproductive because it just means that people will still, I mean, the way developers work, I don't know how many of you are actually developers in the audience, you know.  There's not a lot going on. They want to work on next thing, right? They're not going to work. I mean, giving them another week is just going to add frustration to everybody. So now that you called developers slightly moronic woodland creatures, I hope someone wrote that down. I actually had a young college age developer ask me a couple of weeks ago, so how do I become a maintainer? I guess the first answer is don't be a moronic woodland creature, but beyond that... I don't think that's the first answer. The first answer is be reliable and let people know you are reliable. A lot of it is about technology, but at the same time, looking at my personal experience, one of the most important things for a maintainer is not that he's this super engineer that knows to solve all problems. The most important thing for most maintainers, I would say, is that you're responsive and people can just rely on you being there 24-7, 52 weeks a year, and if you're going away for even just two weeks, let people know that they don't get taken too much by surprise. That doesn't mean that you have to answer every single question. I don't answer email unless I feel it needs to be answered by me, which means that most email I get I will not answer because I think, hey, guys, you should have sent this to the correct maintainer or you should have sent it to the mailing list. If you send random bug reports to me directly, I'm saying you're doing something wrong. I'm not going to be responsive to that degree, right? But I do think that the most important thing for a maintainer is to make sure the developers you work with don't end up hanging. And that tends to be the biggest problem we have is if the maintainer for some subsystem goes away for a month and nobody knows what the hell happened to them. So being reliable and trustworthy is the biggest thing. And for a new young person, the biggest issue is to get trust, it's not that people don't deserve trust on their own. You have to show that you are there for people, and that takes time. So it's very hard to become a maintainer these days without having been around for a few years and having shown people that, hey, I'm around.   You are going to be there 24-7. And it's fun and it's interesting, but it is a lot of work. And I'm seeing somebody nod in the background, right? So it's not hard to be a maintainer if you have the right qualities, but you do have to be there for people. Any questions from the audience? Or is everybody still asleep? We can keep going up here, no problem. So what keeps you up at night besides your children? I don't worry about technical issues usually.  I don't actually sleep pretty well. Occasionally we have things flare up and then things get stressful and they are invariably not about a particular bug. They are about social issues, process issues, people wanting to kill each other and email doesn't work for that, but it works well for development. That doesn't happen that often, but when it happens it can be very stressful for a few days. I get really excited about things for a couple of days and say whatever, I don't really care. So I have flare ups and that works fine for me. Other people tend to then mull over things. I don't know, I mean I sure you all at least know somebody who just eats at them for weeks and weeks and end and those kinds of issues tend to be the painful ones when you know there are bad things happening, not on a code level, but people aren't happy. So do we have issues in the kernel right now on a technical level where you say, oh I wish we would fix that. Is there a feature that you desperately wish or a change that you hope that would finally make it? No, I mean the situation that I really, I think my concerns are not technical concerns. My concerns about maintenance issues, to be honest Linux does or has done everything I personally wanted it to do for about 20 years now. So any new feature I go, wow there are some crazy people out there who need this, right, because everything I do we saw a long time ago. Like working suspended as you? No, it works. So I don't worry too much about features or things I want to happen. I worry more about meta issues, so for example one of the issues we constantly have is our new hardware, right, and some manufacturers are really good about it, I'll name Intel and some manufacturers are not really good about it and I won't name other companies. So what I really wish for is not that somebody would come up with a certain new feature or code up a new thing what I really wish for is that sometimes some companies would learn to work better with the community. I mean sometimes we're not always easy to work with, don't get me wrong. So sometimes we need to help companies too, but that's the kind of thing I worry about. Questions from the audience? There you are. One question I have is how do you propose those helping, educating, enabling some of these new hardware vendors primarily out in places like China who are pushing out hundreds of thousands of devices a month and have the most obscure kernel support possible? Having discussed with one of their engineers over some noodles, their fear was well if we push our kernel code upstream then our competitor will see what we're doing and trying to educate them over some noodles is a bit difficult especially seeing as my Chinese is really poor. Yeah, I mean this is a problem and this is exactly the kind of issue where with certain hardware manufacturers are simply hard to work with and I don't think the solution is to try to educate everybody. I don't think you can. There are processes in place. There are companies who know how to do that and then they try to often educate their subcontractors. There are Linux Foundation and one of the things Linux Foundation is doing is talking to companies. This is how you do license compliance and things like this. So there are processes in place at the same time. We're not out there to make sure everybody does the right thing. The reason I do open source is A, it's fun and B, it works. I think the thing that really educates companies about being members of the community and really participating is not us going out to them. I'm more of a live and let die kind of Darwinistic person and I think the companies who learn to work with the kernel and learn to get things upstream are going to waste a lot less time with new releases and just getting things done and they will just work better. So in a very real sense, I'm like, okay, if somebody doesn't learn, I'm not going to bother with them. I want to make sure people at least comply with the license and in some places you can't enforce even that. But at the same time, I'm not so worried because I do think that the companies who do learn are the companies who will win and the problem will sort itself out even though it's going to take some time and it's going to be somewhat painful. I can certainly see this from our perspective at Intel that we see this as a huge advantage for us and we actually see other companies mimicking our behavior in many ways which is kind of flattering and it's a good thing for the community. So I completely agree with that. And quite frankly, if you're a company that thinks that your tiny little change to the kernel is what gives you the competitive edge, you may need to rethink your economic plans because I think you need to have something bigger that keeps you afloat in the long term and you'd be much better off making sure you make the best damn hardware for the lowest price and don't worry about the fact that other people can compete and see what you're doing. Just make sure you compete so well that you win. And it's sometimes hard. I mean, not a lot of people want to be in that situation where it's actually open competition and the best man wins but that is what open source is all about. One of the things. What are your hopes for the Linux and open source ecosystem in the longer term, so perhaps 5 or 10 years? Personally. This keeps coming up. I started doing Linux because I wanted to use Linux on the desktop. And Linux is doing so wonderfully well in so many different areas and I am still somewhat disappointed and people have heard my complaints about the fact that the Linux desktop is this morass of infighting and people who do bad things. And it's not so much on the kernel side, I think. I think we're actually doing a reasonable job in enabling people to do a Linux desktop. And there are commercial issues that make it much harder so yes, you can sell a billion cell phones but selling a billion Linux desktops is harder. Don't get me wrong. But I do hope that the desktop people would just try to work together and work more on the technology than trying to make the logging screen look really nice. At the same time, I am very optimistic. I really don't have a five-year plan. I've never had it for the kernel. I don't think we need it for the kernel or for open source in general. I do believe that open source is the correct way of doing complicated systems and getting things done right. And it just turns out takes a lot longer than you sometimes wanted to take. More questions right here in front. And then there's one there. So it's matter of community. Basically you say I don't want basically to educate all of the hardware enabler or whatever on the chain. Can we just educate the communities in those places? It's very hard. How do we reach to those audience? We discussed that. I remember in the next one in Japan, is there going to be a Linux account, for example, in China and trying to get masses there and things like that. That's important. We do try to reach out to conferences and I'm just thinking back three, four, five years ago when people were talking about how do we expand the kernel community or the open source community in general from being very western Europe and U.S. dominated and the fact is, and people were saying how do we reach Japan and Asia? And these days we have tons of developers from Japan and Korea and Asia and we are reaching them and part of it was I mean part of it was just economic issues and Android in particular just taking over but we also did actively try and by we I don't mean me. I mean the kernel community and companies involved try to actively engage Asian developers and had like these conferences where you were talking about how the process needs to work and things like that and we're continuing that. I mean we're not going to reach everybody. The Linux Foundation has events in Japan and Korea and we've had those in the predecessor organization. We've been going to Japan for a decade now. We Intel have been hosting a Linux kernel forum every year in China for the last eight years. There are events popping up in Vietnam, in India. There is massive outreach happening absolutely. There was another question over there. Hi Linus. Hi. I don't know if I'm going to get beaten up for asking this question but I don't know if you kept up with any of the Apple announcements last night. I did not but go ahead. Well they're offering the new desktop OS as a free upgrade. I wondered if you had an opinion on that or whether that might have any impact on the next whatsoever. We've done that for 22 years. Yeah well. So I actually one of the reasons I use the term open source and not free software, one of them is purely political and one is that in the English language people get confused about the whole free word. It's not the fact that Apple gives their OS away is completely irrelevant plus you actually pay for it in the hardware anyway. So it wasn't like it was expensive before. I don't think that impacts Linux at all. No. There was another question over here. Josh. Leaving aside stability and bug fixes, what's the biggest feature that you feel like nobody out there is working on for the Linux kernel that would excite you know-in to see in your inbox? If I knew it would be that exciting. Right. I really wish. And it's not really happening anymore. Let's be honest. We have been doing this for 20 odd years. The kernel is kind of... I wouldn't say it's a known problem but at the same time the really core functionality is not that surprising anymore. And we used to have these things where people came up with new really exciting core ideas and they were big issues and they really affected the whole experience. And these days the new and exciting ideas tend to be in the periphery. They're not really that core because the core really works fairly well and the core does what the core needs to do. And when people say, hey, we need to re-architect everything for this major new feature, they're usually wrong. Like we need to make all file systems be database file systems. Right. There are... You can make huge changes for the sake of changes but it's not how the world works. And our core is there. I think if you look for new excitement where people are doing crazy things, you either need to look at like periphery in the kernel with new devices or new features of new CPUs or things like that. If that excites you, we've got you covered. But otherwise I suspect you want to be doing work outside the kernel and work on exciting new devices that ubiquitous new things you can do with computers that people haven't been able to do before and doing good user interfaces and things like that. I think that's where the excitement is and I've said that for over 10 years. As a kernel person, I think the kernel is kind of... It's really exciting from a technical standpoint but it's not a place where you should go look for new features. But it's kind of interesting because for the first decade in Linux development, we pretty much played catch-up. We were trying to implement POSIX. We were looking at features that were out there in other OSes. We were trying to basically meet the bar and now for the second decade of Linux, I thought that we were much more taking the lead and we were implementing things that maybe weren't there or have never been done that way. So do you perceive this shift as being now less exciting because it's fewer new features coming because we're no longer playing catch-up or is it not more exciting because we're actually implementing things that haven't been there before? So the excitement for me has always been just the technical side, doing things really well. And I am perfectly happy with... Literally, and this is what I personally do. When I write code, which is not that much anymore, when I write code, I look at the assembly generated by the compiler, I look at the profiles of the kernel and say, hey, this thing, we're doing things badly, our calling convention means that we're spending 15 extra cycles doing nothing. And that gets me excited just doing something really, really, really well. And I'm okay working on that level and I think a lot of kernel developers end up being people who are a bit OCD and they're looking at the details because it is about details. Sometimes it's about CPU details and worrying about the compiler. Most of the time, it's details about devices that have bugs or interesting features or things like that, but it's still very much about details. And we have had issues where we create these wonderful new exciting interfaces for people to use and I have almost always been disillusioned by the fact that nobody uses them. It's like we make these clever new things for people to do really clever things, but then when I actually look at what programs do, I say, Christ, that wasn't very smart, was it?  So I don't get excited about features anymore. I get excited about supporting new hardware. I get excited about doing the job the kernel does really, really, really well and making sure our process works for doing all that support. That's what I care about and I may be a bit OCD sometimes. More questions right here in front again. Raise your hand higher. So Dirk's already said about how the first ten years was playing catch-up with POSIX. Now, do you think that kernel 310, 312 onwards is building the next house of Linux because you've got much more advanced support now for QMU and KVM in the virtualization world? You said yourself that you want the cloud companies to play nicer together so that they're actually building platforms rather than WYSIWYG features, login screens, et cetera, et cetera. It's more important now that we have that stable virtualization support coming really from the mothership. I wonder what the question is. Well, if you think... From my perspective, I work at Red Hat. So a lot of my technology that we put out has a reliance on what's coming down through the mainstream kernel. And sometimes it's nice to see leadership coming down from the kernel itself in the respect that we see the changes in 3.10 for QMU and for KVM, which we rely on for that kernel support. Yeah. I mean, we do have situations where kernel features end up being very important to a subset of the market. And the subset can be very big for companies, right? It'll be depending on what you end up being... Like, which market you end up selling to. I do think that what the kernel's job is is to be a really stable platform for getting real work done. And the real work is never the kernel. The kernel doesn't do anything on its own. The real work always comes from outside. And if we're doing something on our own, we're almost certainly doing something really, really wrong. So I do want... This is the kind of mindset I come from is I want to make sure our base support is something people can rely on. This is why I, for example, have very strict rules about no regressions, because that ties into the whole thing where you should be able to rely on us as the kernel. And sometimes there are particular features where people do need to expand on things. And I think we're filling in the details at the edges rather than doing core things. That was what I was trying to get at in my previous answer. That's a question over there. Hello. What do you think of game coming on Linux and do you think it will bring manufacturers to add better support for Linux, like NVIDIA or IMD? What do I say? Who's coming for games? Sorry? Oh, games. Games, yes, yes, games. I actually... I love the Steam announcements. I really... I think that's an opportunity to maybe really help the desktop. And also it's... What I like about the Steam announcements in particular is that Steam is this one company that has a vision for how to do things. And I think that will force a lot of the other vendors around them. And I'm not just talking about the hardware vendors. I'm not just saying that it will help us get traction with the graphics guys. I think it will also force the different distributions to realize, hey, if this is the way Steam is going, we need to do the same thing because we can't afford to be different in this respect because we want people to be able to play games on our platform, too. So it's the best kind of model for standardization. It's not... standards should not be people sitting in a smoky room where the smoke is usually drug-related from what I can tell, right? Because... and then writing papers. That is not how good standards get done. I think good standards are people doing things and just saying, this is how we do it and being successful enough to drive the market. And I think it is interesting to see the Steam announcements from that standpoint and see where that goes. All the way in the back. We're making a workout for our runners. Linus, I'm having some problems with my graphics card. It's an NVIDIA. So that's a bad joke. The licensing of the kernel, I believe when you originally released the Linux kernel, you had a different kind of license. And then, I think a couple of iterations later, you changed to the GPL V2. Is the licensing of the kernel likely to change in the future? No. I say that both because I really love the GPL V2. I mean, I'm not saying that as the language of the license, it's not beautiful pros. I'm not claiming that we're talking about Shakespeare here. But I really like the concept and I really believe deeply in this whole enforced quid pro quo kind of tit for tat model where, hey, you can do whatever the hell you want with this, but you should kind of do the same thing we did. We give you source, you give us source back. So I very fundamentally agree with the version 2 model. Also, from a purely practical standpoint, changing the license now for the kernel is basically impossible because I never ever wanted to do all the crazy copyright ownership paperwork. And I always felt that people should retain copyright in the code they write. And the thing that matters is not that there is some single entity that owns copyrights. The thing that matters is that lots of people own copyrights in the code, but they all agree on the license. So in a practical sense, we're in the situation where there are thousands and thousands of people who have copyrights in different parts and trying to get people to agree on a license change is just not going to happen. But I also feel there's no real reason to try to make that happen even if we could get people to agree. Over there. I'm aiming this way, sorry. I think I remember reading that pretty early on in the project, a bunch of people got together and contributed to pay off the loan on your first X86. I was wondering whether the volume-free stuff that people offer to give you has gone up in line with the growth of Linux? Well... I have to say, that's still one of those warm memories. I think Peter Anman is somewhere in the room, and he ended up being the person who collected checks in the U.S. so that they could send one large check to me. Well, large. It paid off my first machine, at which point I immediately bought a second one, right? I get paid for doing Linux these days. I don't know how I conned people into doing that, but I have a wonderful contract that says that I get to do whatever I want to do as long as it's open source. More of the contract talks about what the Linux Foundation must not do to me. They cannot force me to give public speaking. They ask very politely, because they know that my contract says that if I say no, I just don't go. So yes, I still... Well, still, I now get paid for doing Linux. For 10 years, I didn't. For 10 years, I very consciously wanted to avoid being employed by a Linux company because I did not want to be in the situation that I was seen as biased towards a particular company. And now I'm at the Linux Foundation which doesn't worry the commercial companies because they don't see me as competition. They see me and the Linux Foundation as a whole, kind of third party that everybody is pretty happy to work with. So I think people are good with this whole situation. I'm not the only one, obviously, who is in that situation these days. But you still do get the occasional large prices like the Millennium Prize and... Well, it has helped to make sure I can get my kids through school and I have owned my own house and the companies have given me pre-IPO stock and it used to matter more than it does now. And I have gotten prices too. So yeah, I didn't... I know Bill Gates, but I really don't miss that at all. There was another question right there. Notice later on today there's a session on getting women into the Linux community. That should be tomorrow, I think. Is it tomorrow? I just wondered, as essentially the leader of that community if you felt there was anything about that community that could be better on that front? It's a hard question and I don't have any easy answers for that. And it's clearly something we want to solve. And at the same time, part of it is I'm convinced cultural. And that is not something we necessarily can solve. We can try to do as good a job as possible giving that constraint. And when I say cultural, I don't mean our culture but that's also an issue. I just mean the bigger culture of the western world and Asia and everywhere. And at the same time, I do want to say that despite the fact that we have very few women I'm not very pessimistic because we used to have this exact discussion 10 years ago about the fact that we had very few Japanese developers. We can fix it. It takes a long time. And it's still the case that we are predominantly western Europe and the US. And I am convinced that we will be predominantly male dominated for a long, long time. But I think it will improve. And I think we're doing a lot of very good things. We have the outreach program, the internships for women that have created a tremendous amount of new contributors and new code. And I think the fact that we're talking about the fact that there is a lot of awareness that's helping us. So there is no easy fix. There is no easy solution. But I certainly think we're making progress. So I have a completely different question for you. And you talked about how harsh your life is. I did. Yes, yes. I was talking about other maintainers. I don't. Greg gave this whole talk a couple of weeks ago about how I do nothing. You nodded and said yes. Well, I wasn't there, but I read the news reports for the next few days and it was not good for my ego. Let's put it that way. So the next question will certainly be good for your ego. So what needs to happen for you to retire? It's the first part of the question. And B, how will you be able to tell the difference? And we're out of time. What needs to happen for me to retire? I don't know. It needs to get not interesting. And that hasn't happened yet. It's very different from what it used to be. And I do very different things than I used to do. But I still like what I do. And what I do doesn't tend to be a lot of programming anymore. I'm the kind of... People just know who I am. I send this to Linus, it's Dan. And then they can ignore it because the one thing I do try to be is very responsive. I've been merging things early this morning before I took a shower because I do try to make sure that developers and maintainers don't get hung up on me. But if it ever gets boring or if I get the feeling that I can't cope that's when I'll retire. When the doctor tells me that, hey, it's not working all right up there. I'm sure we can find somebody else. As to how I'll know the difference? Well, by then I probably won't know the difference because there's not a lot going on anymore. So let's turn this around. And this is the question that I remember vividly you being asked in the late 90s by Larry Augustin at Linux World Export. When he said what happens if you are hit by a bus. And I liked your first answer, which was, will I care? That's still the answer.  No. But maybe the rest of us do care. The thing is, as far as the kernel is concerned we really have a very, very deep and wide set of kernel developers. I mean, we are unique in the open source world for having thousands and thousands of people involved. And we also are unique in having people involved, not just me, who have been involved for more than 20 years. Literally. I mean, we have... I recognize the names from back in 91 and 92. Some of them are still around, not all of them, right? Some of them went on to do other things. Some of them had accidents and died. It's not like everybody's around. But we have an incredibly deep set of developers. I'm not necessary in that respect. I am the person who people know, and they know how I work. And by know how I work, that means that they may not always like what I do and how I present things, but they can trust me to act in a certain way. And that's really important. You want to have that kind of knowledge that whatever else happens, Linus will be an asshole, but he will take my patches if I can convince him there are the right things to do. And that's a big thing in the end, even if that may be the only thing I do. But there are other people who can be impolite and can take patches. I'm serious about being impolite because I do think one of the jobs of a maintainer is not just to be responsive and take patches. It's also to say, no, this is not how it works. I mean, you need to fix your code or you need to fix how you're working, right? So we have people like that. Some of them are in the audience here. I don't think people need to worry about the whole development process. We are very stable. Okay, so we have 40 seconds left. The title of the session is as always fueling the future where we are going. Where are we going? 33, 32. Well, we never had a plan. I never had a plan. I still don't have a plan. It's kind of like evolution in biology. There's no end plan. It's just that what works is what survives. And if you keep doing that and the things that don't work, keep getting called every once in a while, you will improve. I don't know which direction we'll improve in, but I don't worry about, I don't need, and I don't feel like I need to worry about the future. I don't feel like I need to have this laser focus on where we're going to be in five years, because I think our process means that we can avoid it. And that's important. Thank you very much. Thank you.


I remember Ricky T. Pressure Boom. That was a turning point in the industry of St. Lucia where our team used all of the resources that we had to get that song out there into the world and I remember we flew to Miami, Miami carnival. We flew to Miami carnival and Ricky T. was booked on a show flag party that was a Friday night and while walking to while walking to the event his songs started playing inside the event and we were outside and we said wait we don't have a flag because he had to perform don't have a solution flag so we stopped by the flag man there was a flag guy outside and we said yo you have any flags solution flags he said nah boy some boy some big artist from St. Lucia performing there boy and all the flags set out and Ricky T. was right there you know and that was that was a big moment.


The best thing I've learned from my TEFL training. There are many good things I have learned from the TEFL course I have been taking this past month. Things I had never considered before as a skill is to be a good teacher. I had mentioned it before in my tests that I did not realize how important it is to have those skills until I started taking the course and had read about it. So I guess I am saying that the best thing I have learned from my TEFL course is how to be a good teacher and it reminded me of my previous teachers and how well they taught me and my fellow classmates. Course Structure Every unit I have read makes a note for each time the teacher can be better at his, her job and I felt that helped me a lot to understand my previous teachers. Now personally I have not had many good teachers through the years so it was difficult to be a student when I was younger, but now everything is changing for the better. It is not acceptable anymore to have the teachers sitting at their desk with their computer game on as the student only works on the task books. But the good teachers I had been the most memorable even though I have not seen them in over 15 years. I still remember how they helped and how well I learned at their classes only. I have even had a very interesting class for building and engineering ships and the teacher there was only reading out loud with a flat line voice from the book, so I understand it very well how you can take away the motivation from the students as a bad teacher. But enough about my previous teachers, I have now described what is the best thing I have learned from the course and now it is time to explain why it is the best thing besides the part where I stopped and appreciated all the good teachers I have had before. Useful Suggestions Good teachers get to know their students up to a certain level and remember their names and know their strengths and weaknesses. They also find a way to create a better environment for the class and a better environment is better to learn in. The purpose of a teacher is to teach, like the purpose of the students is to study, so the teacher must be able to see everyone in the class and make sure that all the students are studying. If motivation is missing in a student or even a few students the teacher must see that this teaching method is not working and try something new like pot them in pairs or groups to get the students that are not motivated to work get in a group or a pair with those who are and they will assist each other to finish the task. Learning a new language is not always fun and games because it can be difficult to tackle pronunciation and correct grammar, but if you play a game it might be more fun and make it easier for the students to learn the language. When I started learning English in my school we often play games like hangman, connect the words, dictionary and 20 questions. This would help us with vocabulary, grammar and participating in the class and the whole activity. While playing games can be fun from time to time I think brain teasers are also fun like crosswords and find the word because they often give the answer so obvious but it still can be hard to figure it out, but of course it needs to be at the same language level as the students because if it is too hard for them they lose the motivation. Do you want to learn new TEFL skills? I hope I have all the qualities a good teacher has and I hope to be able to motivate my future students to study and learn English. As a person with English as a second language, I do understand how hard it is to learn a new language and understanding how the grammar works in a new language. I have myself learned four languages besides my mother language and I think that can give me a little advantage as a teacher for a second language since I have been there myself. Speak with an ITTT advisor today to put together your personal plan for teaching English abroad. Send us an email or call us toll free at 1-800-490-0531 to speak with an ITTT advisor today.