My name is Kate Hiddleston. This talk is called technical onboarding training and mentoring. It was originally written and co-presented at PyCon with Nicole Zuckerman. She's not here today, but she is a software engineer at Eventbrite. She's the one that's not me, so I will let you use your powers of deductive reasoning. So I'm a software engineer. I work at a company called Runscope. I usually work at really small early stage startups. I've never joined one larger than 12. So I usually join companies that don't have any onboarding. And this talk was originally written because Nicole and I had separate experiences, completely independent, where we joined companies and there was no onboarding. And we learned a bunch of stuff and about a year later we looked back and we were like, a little bit of onboarding would have gotten us to this point much faster. And so we turned around and started implementing onboarding for the new employees at our respective companies. So first, what is onboarding? This talk focuses a lot on junior engineers. Many of the things that I talk about can be co-opted for mid-level or senior engineers. But I specifically focus on junior engineers because there are a lot of code academies that are putting out a lot of junior engineers and companies are having a hard time absorbing these engineers into their organizations. But onboarding is the process of taking someone from outside of the company, outside of the team, outside of the group, and making them a productive, confident and independent member of your team. And I picked these three things very specifically and I'll go over them. Although productivity is pretty straightforward, there's a lot of apps and blog posts about how to be more productive and efficient. It's important but the bulk of this talk is about independence and autonomy and confidence. So autonomy is hugely important, independence in new engineers. It's a big motivator. I also have two somewhat depressing anecdotes about the importance of autonomy. The first is in prison systems. So prisoners, by definition, have no autonomy. They're stripped of all autonomy, all ability to make decisions. And when they started doing their research on the sociology of inmates, they realized that giving inmates the ability to choose the TV channel and to rearrange their own furniture, seemingly mundane choices, significantly reduced prison riots. So autonomy reduces riots. This is good. The second depressing anecdote has to do with nursing homes. It's another place where people are stripped of autonomy, although in this case they're usually stripped of autonomy in order to provide good care. But they're also, they don't have any choices about what they do. And they found that people in nursing homes who had choices like what to eat and had some small responsibilities like chores and cleaning lived significantly longer than those that had no choices. So autonomy also prevents death. So you want independent people on your team. It's a great motivator, but it actually also is really critical both to life and our happiness. Confidence. Confidence is, most people will agree, very, very important. But what is confidence? Confidence is really the belief that you are valuable. And the fact that it's the belief that you are valuable is hugely important. How many of you have heard of the stereotype threat? Anyone? A few? It's actually a pretty well researched and documented phenomenon. Someone decided to basically go figure out if people play to stereotypes, if stereotypes are true. And what they found was that statistically people will play to stereotypes. So some examples of stereotypes, and I in no way condone or justify these, that Asians are better at math, men are better with computers, women are better at nurturing, et cetera. So what they did is they took some of the quantifiable stereotypes. And if you remind a group of people before they take a math test, for example, that men are better at math, what happens is men perform better on that test and women perform worse than the control group where they're not reminded of the stereotype. What they also found was that if you flip that on its head, if you tell a group women are better at math, we've actually found that women perform better on these types of spatial reasoning problems. Women will perform better and men will perform worse than the control group. So it's not really about whether or not the stereotype is true, it's about this belief, this idea that I am capable of learning these things or I am capable of being good at these things. So why do you care? I think a lot of people care about onboarding. What I want to convince you of is not that you should care in general, it's that you should care right now. That onboarding isn't something that you should do next month or next year, that onboarding is something that you should do immediately, like automating your deploy. So there's four major categories that I'm going to talk about as to why onboarding is important. The first is the individual, the second is the company in the bottom line, the third is the immediate team that a person joins, and the fourth, the bonus category is diversity. So the individual, it's pretty obvious for individuals that training is important, learning skills, having an upward trajectory, confident, productive, independent, but also losing an engineer at a company is really expensive. And having engineers that are not as productive as they could be is also a loss of money. So losing an engineer can be 1.5 to 2x the annual salary of that engineer. And while onboarding or a lack of onboarding doesn't lead directly to attrition, good onboarding will mitigate things like that. So for the company's bottom line, companies care about money. They build products and they sell them and hopefully they make more money than they burn. For companies, when you add a new engineer to the team, you want to make more money, or you want to build product that will make more money. And so what you don't want is this, this is bad. And LinkedIn found a couple of years back that this is exactly what was happening at their organization. They hired a new senior vice president of engineering and they had to freeze all hiring. They had to stop hiring engineers because every engineer they added to the team decreased their productivity. So they had to do all of this reorganization. They had to put in place onboarding and figure out what was going on so that they could get something like this and they could hire new engineers. So this is incredibly expensive. Not only are engineers really expensive employees, but if you're losing money for every expensive employee that you add, that's a pretty bad business model. So the individual team that a person joins is also really important. You don't know something until you teach it. That's kind of an old adage. And so when a new person comes on board, you need to teach them about your processes, your team culture, how you get things done, your values. And this is good for the existing team because every time a new person is added, that team fundamentally changes. So I played a lot of sports, I coached and I have a lot of sports anecdotes, but basically this is true on sporting teams. Every time the senior class leaves and a freshman class comes in, it's a fundamentally different team. So as you're scaling your company and you're adding new engineers, you're not the same team. If you go from 10 to 20, you're a different team. So what you need to do is reiterate the values, not just to the new people coming on board, but also to the existing team members. You want everyone going in the same direction and that's not something that can just happen on its own. It's something that every single incremental step of the way you need to reiterate. So having good onboarding is a little bit like having good deploy automation. It's absolutely critical and having it will pay off hugely. The amount of time you put into having good onboarding and having good deploy will pay off enormously, especially if you do it early. Because rather than having to go build this huge heroic system, later on you can just incrementally add to the one that you already have. So this is my favorite equation that I've ever made up. Back when I finished playing water polo, I coached JV girls swimming in water polo and I used to have these really great philosophical team meetings with them that they absolutely loved. And I came in with a poster board one day with this written on it. Like this is really important everyone, like I have this equation, I need you to understand this. Your ability to win gains, your ability to be successful as a team is the sum of your talent or the sum of your skill sets multiplied by your ability to work together as a team. So you can all be medium skilled or beginners, but if you work together incredibly well, you can beat teams that are quote unquote more talented than you are. So there's a lot of Disney movies about this as well, but you can overcome challenges through teamwork. And we found that that's true in software engineering as well. Really small teams can be hugely productive if they work together efficiently. In fact, a lot of the biggest products that we use today are built by teams of less than 10 engineers. Gmail is a really good example. It was built by a team I think of less than seven and it's now maintained by a team of over 400, I wanna say. So you can build awesome stuff with a small team that's highly productive and works together really well. Conversely, if you have a bunch of really talented engineers who don't work together at all, you're just gonna completely fall apart. So our bonus section, diversity. I use sneaches to represent diversity. So for those of you who don't know the Doctor Sue story, the Doctor Sue story is about a community of sneaches and some of them start developing stars on their bellies and this separates them from the non-star bellied sneaches and it causes rifts and it's supposed to represent different demographics. I use sneaches because diversity can mean a lot of things to a lot of different teams. There's the classic examples of gender diversity and racial diversity but there's also things like introverts versus extroverts. So if you have a team of introverts and you bring on an extrovert or vice versa, that can also be something that you need to think about. So onboarding is really important for diversity. If you have a homogenous group which is pretty common with startups and companies to have a homogenous core existing group and you have no onboarding, the people who are gonna be most successful and have the best time getting up to speed are people who are like the existing group. This is because a lot of onboarding has to do with communicating and someone who's like the existing group is gonna be able to communicate more easily. They're probably gonna have similar hobbies, shared interests, so they're gonna be able to talk to the existing group, learn about the product, learn about engineering and get up to speed faster. So you're gonna have this feedback loop of hiring more and more people like the core group. Conversely, people who are different, who don't necessarily communicate the same way, who don't have the same hobbies or interests, maybe people have children and they join a team of young adults who like to go out and drink. It's gonna be harder for them if there's not explicit onboarding. So explicit onboarding makes it possible for people who are different from the core group to learn about the product, to learn about the company, to come onboard and get up to speed and it doesn't disadvantage them disproportionately against basically the existing group. So who should onboard? Everybody should onboard. This is not ants carrying a taco. This is people carrying a canoe. I drew all of these myself. So everyone should onboard, spread out the load. There's a common misconception that only senior people can teach. The best rule of thumb is actually that the last person who did something should teach it. First off, they're still excited about it. Second off, they know it the best and third you can spread out the load that way. So someone who just learned about your automation configuration system is gonna be great to turn around and teach it. The best people to onboard junior engineers are not senior engineers. They are the people who were just junior engineers. So when you go into college and you play a sport, the freshmen are not actually onboarded by the seniors on the team. It's the sophomores. Because the sophomores were just freshmen last year. They know what it's like. They still have empathy for the problems whereas the further you get away from them, the less empathy you have. And also you get really cynical as you get more senior. So you want someone who still has a little bit of that enthusiasm. So when should you onboard? Onboarding starts from offer acceptance in the start date and it goes all the way until reliable independence. And this is true no matter what the level of the engineer. So someone who's junior reliable independence might take six months. Someone who's really senior could come onboard and be reliably independent within about like a day, three days, I mean who knows it could be really, really fast. But that's the whole point of onboarding is to take someone and make them reliably independent in your organization. And how? So how has two sections? The first is a little bit of the philosophy about how you should think about onboarding. And then I'm gonna go into an example of an onboarding plan that you can pick and choose from to create your own onboarding plan. You don't have to be completely altruistic when you're thinking about onboarding. You really want to maximize the return on investment. You don't want to put way more into onboarding people than you get out of it. That doesn't make sense. That's not sustainable. That's actually crazy. I don't know why you'd want to do that. But it's actually a really common pitfall. So this is a really common first time mentor thing that happens where they want to handhold the new junior engineer. They're so excited about being a mentor. They're like, this is gonna be awesome. We're gonna do everything together and I'm gonna teach them everything we know. We're gonna do tutorials. But the time I'm done with them, they're gonna know everything that I know. And this is wildly unrealistic and it also leads to high burnout rates. So the senior engineer is putting in so much time to this junior engineer that by the time the junior engineer is finally onboarded, the senior engineer is exhausted. And they're like, I can't onboard anyone else. And so the company's like, well, we can't really hire more junior engineers if we don't have anyone to mentor them and onboard them. So instead, you should think about it kind of like bumper bowling. You wanna put in place enough structure and enough process to get them going in the right direction but it doesn't really matter if they hit the lane the whole way down. They'll get better, they'll practice, it'll be fine. They're going in the right direction. And a lot of putting in place a structure like this without having to handhold people involves creating really good automation at your company. Really good dev environment setup, really good testing, really good deploy system. So a huge amount of creating a great environment for junior engineers is very technical. You want a lot of automation. It's also important to remember that one of the tenants of expertise is the ability to set boundaries. So when you're an expert, you have a really, really in-depth understanding of the landscape. You're able to scope and set boundaries and break problems down into really small, manageable pieces. By definition, beginners cannot do this.  So it's important to remember that part of your job in setting up onboarding tasks is to scope them. To just scope things really, really well for new junior engineers. Bug fixing is a good example of well-scoped projects. So a bug has a clearly defined problem, usually has a pretty clearly defined solution for a lot of them. And so they're by nature well-scoped projects. Those are great, actually really great to give to junior engineers. So an example onboarding plan. It covers three major categories. Each one of these is about a third of what someone needs to learn when they come onboard. We think, for some reason as engineers, that the technical knowledge is 90% of it. It's not. There's a ton of online documentation and a ton of resources. A third of what someone needs to know is the company knowledge and process. How do I build here? How have we built here? What framework do we use? How did we hack this framework to pieces so that it doesn't look anything like the original framework? And I need to know about that and there's no documentation. And then the third category is personal development. So developing confidence and excitement around what you're doing so that you go off and you learn things on your own and you believe that you're capable of learning all of the millions of things thrown at you as a new engineer. So I've broken it into phases. These were originally weeks, but I decided phases makes more sense because it doesn't really matter how long each section takes. It just matters that each section is dealt with. In phase one is shipping stuff. So when someone comes on board, you want them to ship things. So the first step of that is dev environment setup. And someone should be able to set up their development environment in a day. That is an awesome goal. If someone can't set up their development environment in the first week, it's worth dedicating a lot of resources to it, especially because this isn't something that junior engineers have to do, it's something that all engineers have to do. The best person to help someone set up their dev environment is going to be the last person who did it. So you should never have burnout on mentorship for dev environment setup because the last person who did it is always gonna change. Shipping code. Preferably you want someone to ship code the first day. If you can get them to do it the first day, that is awesome, you get a gold star. If they can do it in the first three days, that is still really good. The first week is preferable and if it's not in the first week, I would highly recommend dedicating some resources to DevOps so that people can ship code within the first week of joining the company.  The point of this isn't to have them alter your code base significantly, it's to have them understand the process for how you ship code. How do you open a pull request? How does that pull request get merged? Who decides that pull request is ready to go? How is that pull request then deployed out to production? The third thing is journaling and note-taking. People will naturally do this because they can't remember all of the things they need to do. So just give them a notebook, basically. Just tell them that they're gonna have a lot of stuff that they need to remember so they should write it down. It's a nice way to check back in too because if they've written it all down, you can go take a look at it. And then finally a social event and this should be done for everyone. This should be done the first week and the reason that you wanna have a social event the first week is because people need to talk to their teammates. They would like to know who they are. This can just be the immediate team, the five people that they have to work closely with if they join one specific team, you can go get lunch, go get coffee.  Phase two. Phase two is the big section on talk to your teammates. So after phase one, someone's able to ship code. They are able to open pull requests, push them out to production, do kind of the core of what it is that we do as an engineer. Phase two is a lot about integrating with the team more. So the history of the company and a team map are some of the best things to do in this phase. The history of the company people love to do on the first day and I don't know if you guys have joined a new company lately but on the first day you're still trying to figure out where the bathroom is and so the history of the company is just not gonna sink in. You also don't know any of the characters in the story because you haven't met anyone at the company. So do this the second week or the third week but do it when people can actually absorb the story about how this company was started and why all of the different people are there and then give them a map. Where do people sit? What are their names and what do they do most often? You know, this person is the person who works with Chef at our company. This person works mostly with APIs. They can then look at the map rather than having to go ask people what someone else's name was or who do I go talk to for this? They can just go find that person. So it gives them independence fairly early. Shadowing in code labs are good to get started at this point. Code labs are something that they actually do at Eventbrite. Eventbrite has, they've done a really good job of hiring people from Hackbrite, the Code Academy, it's a Python Code Academy for women. So they've put together a fairly robust onboarding process and it involved code labs which was a weekly safe space to ask questions. And the emphasis is on safe because when someone is really new they feel like all of their questions are stupid. Even if they aren't.  And the code labs can be about anything. It can be on any topic, it can be run by any engineer who's interested in running it. Shadowing and pairing is really important at this point. I was having a discussion earlier about the difference between shadowing and pairing. Shadowing is following someone and watching their workflow. Pairing is working with someone to achieve a task. Both are valuable, both are really good. Shadowing sometimes you can, if you shadow someone who's too far advanced you can lose track of what's going on. But it is a bit of a fine art form to figure out exactly how much shadowing and exactly how much pairing someone needs. So I don't have an exact response to whether or not you should do this all the time or how much. Phase three.  So now that the person can chip code and do basic engineering tasks, they feel connected to their teammate and their company. They need to decide what they wanna do next. Do I want to build product features? Do I wanna be in the UX space? Do I wanna do DevOps? And at this point, they should hopefully have a better idea of the landscape and the people on the team. So companies should have weekly one-on-ones. Everyone should have a weekly one-on-one with their manager. I also believe that people should be able to electively have one-on-ones with anyone they want at the company. This is hugely important. Having communication channels, even if they come in once a week and tell you that everything is great, I'm doing fine, like I'm so happy here, you should still meet with them once a week. Because if something is bad, if something's going wrong, that communication channel is gonna become critical really fast. And you have no idea of knowing when that's gonna happen. So if you don't have weekly one-on-ones and you don't have these communication channels, you basically put the onus on people with less power to come to people with more power with bad news. And that is a terrible, terrible thing to have to do. Goal setting and feedback? Goal setting sounds really lame. I don't know why, but it's so important. This is basically the what do I wanna, what kind of engineer do I wanna be? Who do I wanna be? And so sitting down with someone and talking about this and setting goals is so important. I don't know why we don't do more of it. And then giving them feedback, constant feedback, not only about what they need to improve, but what they're doing well. Because remember, someone who's junior doesn't know what they're doing well. So having that feedback is great. Positive feedback is so important because if you only come with negative feedback, every time you walk up to their desk, there's gonna be a little ping of fear. I've had managers like that. They only come up to me whenever they tell me that I did something wrong. And so whenever they walk up to my desk, I'm like, oh no. Even if they wanted to do something fun. So don't do that. Give them positive feedback. And then presentations. The best way to learn something is to teach.  If you want someone to learn about your deploy system, have them give a 10 minute presentation on your deploy system. It's pretty simple. I think it's great to get started organizing things into presentations. They can be entirely internal. They can be just to one person. So if someone has a lot of stage fright, they don't have to speak publicly to your company. And then phase four is repeat. So review concepts, check in regularly. Kind of let them electively shadow, let them start to make decisions about their own life, how they want to learn things, how they want to grow. Let them co-pilot a larger project. So let them work with someone who's more senior, who has an emergency break on the passenger side. But also remember that junior engineers go really, really well with senior engineers. Because junior engineers are still excited about the mundane tasks that senior engineers don't want to build. So you put them together, and you have one person who's so excited about getting all of these introductory tasks done, and someone else who's so excited about not having to do them. And then beyond. The assessment category is really important. Well, assessment and apprenticeship. So similar to co-piloting, apprenticeship is a time-tested tool for people to learn. You basically just put them next to someone else who's really good at something, and eventually over time they learn it. That's how all trades were passed on for a very long time. That's how it works in sports. You usually just pair with someone who's really good at something. And the better the person you're paired with, the better you end up being when it's all over. And then assessment. So assessment can be really tricky because people have wildly different trajectories. Some people just take off, other people have ups and downs. Sometimes people don't work out, and that is a reality. You want to give everyone the best shot possible. You want to have faith in them. You want to give them chances. But you also want to identify when someone is not doing well and when someone is not a good fit for something. Because nobody wants to be unhappy. Nobody wants to be unappreciated at a company. So the better that you can get a handle on assessment, the better off all your employees will be. These are some of the assessment categories that we originally came up with. So confidence, code quality, communication, judgment and technical knowledge. Judgment is a huge one, especially in junior engineers. And I wish I had some sort of objective scale for measuring that, but I don't. Communication and confidence are also really big. And then code quality and technical knowledge are actually only two of the five that we came up with. Okay, so that is my overview of technical onboarding training and mentoring. The main takeaways are that onboarding aims to make people productive, independent and confident. It benefits everyone in the long run, the individual, the company, the company's bottom line, the team. And it also is really helpful for diversity. And anyone can be involved in onboarding. So however many people you have on your team, no matter what level of seniority, they can all be involved in training. I work for a company called Runscope, and we have these cool t-shirts. So if you want one, you can come find me. We build dev tools to help people inspect their feelings. I'm kidding, it's, it's de- I wish, that would be really awesome. It's dev tools to help people inspect their API requests. So anyways, if you want a t-shirt, you should come find me because the t-shirts are pretty funny. And thank you so much. I think there's two minutes for questions. There's also a repo on GitHub that has some of this information. I need to update the slides, but it has some assessment categories and how you can think about assessing people based on all of the things that I mentioned. And you can also tweet at me if you feel like it.


Because imagine you are an artist that changed the industry, that did something revolutionary and really cool and really awesome inspired a bunch of artists generation after generation to get into that field and to aspire to be that good and that inspiring to other people and imagine that you can choose between as you get older as an artist to maybe teach and inspire more people and explain things and just kind of inspire again a new generation with your own knowledge and guide them into their future or you just become bitter and spend your time commenting on LinkedIn how crappy those artists are whenever they animate the same thing that you did years ago. So these are the two paths you could take there are many other paths you could take but some people do that and yeah that's not very nice and understand as you get older you might get bitter you might get irrelevant you might see people do other things that you wish you could do but you still have a choice to just not comment just be quiet and just let people be and let people enjoy things but anyway what am I really talking about I'm talking about the invisible resume now in his case it's fairly visible resume but clearly also doesn't care bonus points if you know who I mean but the invisible resume is something that starts very early on even as a student I teach a bunch of students and I have for you know years now and I've seen quite the range of student behavior how to take boutiques how they interact with other students or with the teacher in class and this goes from very appreciative and positive and inspired and hardworking to not saying anything being super lazy and then even complaining to upper management and stuff like that I've seen it all same thing at work you have sometimes co-workers that go behind your back to talk crap about you to their supervisor on the same show so that your review looks bad you have people that just behave in a not so nice way during work you have people that claim things that they didn't really do and then they prop themselves up they're people who don't animate at work they just online gamble and then when it's time for dailies they grab someone else's work copy paste tweak a little bit and pretend that it's their own trust me I've seen it all I've experienced a lot and the main thing is that you could offset it with being a really good animator and I know cases where they're really good and they're really big jerks and that can compensate and balance out for a little bit until it doesn't it might get you far but at the end it still falls apart people will not want to work with you people don't want to crew you on the shows people don't want to hire you when you want to switch companies and like I said this starts early even as a student and it doesn't mean that you have to be fake nice we're like everything is great and just pretend and be nice to everybody even though you hate them and you can see it in the comments you can always kind of tell the thing is it's a very small industry and chances are you're gonna meet your fellow students at work either the same company or you work with them where maybe they are your boss later on or you are their boss it's just it's a small industry everybody knows everybody we're travels and it's not like oh this person I don't like them just because maybe you're threatened by their skills and then you remember the name and you make a list and you try to ruin their careers that I have never experienced and I've never heard I'm sure it happens too because trust me there are jerks everywhere and they're all shapes and sizes but generally it's about your behavior within a team within one-on-one interactions it's just you got to make sure that it's not just your technical skills your animation skills your demo reel that's good but also how you behave within a team to other people with production staff and there's a difference between a show that goes well and a show that's on the crunch and heavy stress tempers flare you got to be able to navigate that and still be professional be polite and just be respectful while you do your work and it's not like this is an fna with a tutorial of do this this and this to not be a jerk like this should be normal but clearly sometimes it's not but at least the advice I can give you is that if you get into a situation where something is stressful or you did everything right but then the person may be critiquing you your supervisor your teacher or a co-worker or something and you you're feeling that their butting head something is going on it's just not going to be a good thing when you just fight back because ultimately it's just going to harm you because you did everything okay let them dig their own grave don't get to a shouting match don't write back angry emails while your you know while your emotions are high just let it roll off your back take a minute reevaluate the situation and just move forward and this is within a student-student relationship a student teacher a professional at work towards co-workers towards productions towards supervisors and you're gonna get into those situations because you know you're also dealing with creative people bigger egos and something's gonna happen but ultimately what people are going to remember is that you were great to work with and it's better that your level is okay in terms of animation quality and this is across all the industries but again i'm speaking terms of animation like animation is good not awesome but good but you're awesome you're easy to work with you're just friendly you take feedback you know in a positive way you incorporate your own notes just you're just you're just just a big difference between someone you want to work with and someone with that you just god i don't want to interact with that person and you might be really really good animator but your attitude is crap that's just not going to get you far it will get you to a certain level but like i said before the house of cards is going to fall apart at one point and if you really have something to say say it in private it's always better than anything happens it's a year in dailies right or you're in a classroom take some notes like why why did this person say this to me like that why did that and that happen take some notes wait and then email or contact them privately afterwards this could be one-on-one at work separately in private this could be an email to the teacher anything because you never know what the other person is going through either right you might feel something like that guy's a jerk why do they say that but maybe that person just got really really bad news today and it's just in a really bad space we're generally are okay but today's a bad day and unfortunately they took it out on you that's not great don't do that either but it can happen but if you take this privately and you go through a discussion you realize oh this is why it happens and everybody's in it clear again it's going to be okay don't go behind that person's back to management complain like there are other ways you can do things before you escalate this to management and hr and so on not saying you should excuse bad behavior but you got to look at the context and where this person is coming from what's going on and also pick your battle obviously there are levels to this there's certain level of harassment and horrible behavior that needs to be reported of course i'm talking about the lower level like the one-on-one stuff the creative disputes and some of the ego stuff or if you see someone commenting online on LinkedIn being a jerk all the time but there is a level of difference in terms of you know how jerky a person is and how far you need to go in terms of escalating and and dealing with this problem and after almost 20 years and i'm including being a student so it's like almost 20 years i've been animating you know dealing with students and fellow classmates basically right getting into the first job well i only have two jobs by now but you know seeing a lot of people coming out of companies dealing with a lot of people it's just you immediately can tell when someone's a jerk or not and there's a huge difference when you are in a team with multiple jerks versus in a team where just everybody's awesome it's just the flow is different the level of enthusiasm for the project is different it's just everything flows differently and again not that it's like you know specific tutorial but you know you always have a choice and if you feel like i didn't have a choice i was just i was angry just at least practice to just breathe take a step back wait examine the situation and then deal with it the next day just really see is that really this important but this is why i'm saying pick your battles kind of validate the situation and the context and see what you can do just as a as a whole just don't be a jerk it's not worth it this industry is small people talk we know who you are where it travels it's just that invisible resume is really important and at one point it is going to be a deciding factor for people's hiring decisions despite your reel again your reel can be top-notch but if you're a jerk versus you're awesome like that can be the deciding thing like this airplane flying close by so just take that take that gem and something happens let it fly by wait a minute and reevaluate trust me the invisible resume is super important and it follows you everywhere this again like i said this could be a student this could be a work this could be online how you interact with people on twitter on my instagram wherever you are like you know if you trash other people's work publicly your co-workers it's not it not a good look because usually there's a reason why something doesn't look good if you've been in the industry you know why your shot doesn't look good it's not because people suck and everybody wanted to do bad work there are always so many factors that come together into something that makes a shot not look as good as you want it to be and other people will see that but there's always a reason and for me at this point if i see a movie and i don't like it i used to be like this i even perched bunch of comments on my twitter were like ah man i didn't like that movie and after all this time like i don't know do i need to say this people work really hard even if the movie sucks at the end people still worked hard so then they're right me i didn't like that movie i don't know especially if it's co-workers it's just it's just not it's not the right thing to do maybe that's just me i don't know so for me it's like if i see something i don't like i just don't say anything i prefer to take the five minutes instead of bashing someone to find something that i like and then amplify that retweet that person look at that awesome work that to me is a better usage of time than going on to all the way back to what is at the beginning trolling linkedin and finding people who do what you used to do and just constantly saying this sucks who are you this is irrelevant blah blah blah blah why why why be that bitter i don't know but that's that person's problem and i'm gonna leave it at that so different kind of fna not a tutorial and a demo in my just basically it's an attitude thing don't be a jerk think about your invisible resume and it's on the human level why why be a jerk that's not needed the the world it's not gonna be a better place if you do that but anyway i'm gonna leave it at that short and sweet it's not that short i mean it wasn't that sweet but it's a little reality check again for students you know but i have a semester starting in a week and that's something i'm gonna talk to them as well like you have to start being a professional from the very beginning even in class how you take critiques how you submit something how you interact it's just you want to make sure that you're professional not just on your real but as a human being interacting with other artists it's just that's the bottom line i'm sure other people have other thoughts about that or maybe they've experienced some of those same things i would well i don't say leave comments i know people want to you know comment and get all angry and do stuff but anyway i'll leave it that that's it for the clip it's it's a maybe a different kind of clip but i think it's something important to know and you're going to deal with people like that and hopefully that interaction is going to be okay with you and hopefully you're not going to become one of those bitter jerks and that's it that's for the clip if you're still watching and uh you've been listening to this advice slash rant then thank you till the very end and hopefully i'll see you in my next clip


Immerse yourself in the salty air, the whisper of the azure waters, and the rustle of palm trees. Welcome to the Trading from Thailand beaches series. Here we unravel the secrets of the financial world, all while basking under the radiant Thai sun. From the hustle of the stock market to the thrill of forex trading, we're exploring it all, one wave at a time. Whether you're a seasoned trader or just dipping your toes into these waters, we promise you a journey that's as exciting as it is enlightening. So breathe in the tropical breeze, let the rhythm of the tide sink with the pulse of the markets. This is Trading from Thailand beaches where finance meets paradise. Stay tuned as the tide of knowledge never stops. Remember, every sunset can bring a new dawn of prosperity. Welcome to the Rich TV Live podcast, Trading live from Thailand. In this episode of Trading from Thailand, we are getting together with our network of international investors to talk about the next hot stock that we're very, very excited about. And we're going to be talking about right here, right now, exclusively on Rich TV Live. This is a company that we used to actually talk about in the past called Element Nutrition. And they've just added some superstars to their roster. The new name is Promenonutritional Sciences. Put on your radar, put on your watch list. We're super excited about the stock. It hit a low of four cents. Now it's trading at 10 cents because of all the exciting superstars that are getting involved in this company. They have added brand ambassador and Stanley Cup champion Jack Eichel. Jack says, I'm all in with Promeno because I think it can become a game changing product that helps enhance performance and recovery. Promeno and rejuvenate muscle are ideal for athletes and they've been a difference maker for me. In 2021, at age 24, I became the first active NHL player to undergo artificial disc replacement. At that time, my personal training team and doctors introduced me to a number of sports nutrition products that helped me speed this recovery. With this experience, I can confidently say Promeno's products are the most effective and best tasting products I've ever used. And I'm happy to join Promeno on its path to success as we help athletes rethink protein, said Eichel. You can see how highly he thinks of it. Now, there's also rumors that they may have Jose Batista, formerly from the Toronto Blue Jays, joining the team. And after the Super Bowl, they may even add Christian McCaffery. So you can see how they are building a star stud team alongside Jack Eichel. So I want you guys to keep your eyes on this company and watch to see what is coming next. The stock has already started to climb and we believe it has an opportunity to do even bigger and better things as it continues to gain momentum. And remember, this is not financial advice. Always do your own due diligence before you invest in any company we feature. In saying that, this company has a game-changing product that athletes seem to be a fan of. So don't miss out on this great opportunity at such a low price. If you like the video, smash the like button, comment down below, share the video everywhere and subscribe if you're alive. Please subscribe on YouTube and all over social media. And go to our website, richtv.io, join our community, build by investors, for investors.


My name is Zach. Thanks for coming. I'm here to tell you whether large-scale automated scanning is going to stop malware on open source software repositories. No, it's not. I saved you 40 minutes. You can go take a break before the next session. So this talk is based on some research work I've done. And it's open access, so you don't need to pay $50 to go read the PDF version. I'd encourage you to read it for a whole lot more detail that I'm going to have time to get into now. But I think it's a pretty interesting look at software repository security and how it shakes out actually in practice. So who am I to be telling you about all this stuff? I'm a research scientist at a startup called ChainGuard. We work on software supply chain security, whatever that means. But that extends to include things like open source security of package repositories. So my academic background's in applied cryptography. But I'm interested in general in software repository security, package repository security, especially package signing and provenance, but also things like two-factor auth. And as you're going to see, malware detection. I'm also interested in policy for secure supply chain. That includes things like government policy. It also includes things like machine readable policies for how do I decide when I'm running some software whether I should trust it or not. So we're going to kick off with a little bit of background on malware detection. So what do I mean by malware detection? As long as we've had malware, we've had scanners for malware. The first sort of computer worm was something called Creeper. It was at BBN in the early days of networked computers. And it was just a little piece of software that duplicated itself. And if it was present on your machine, it would just print out this message, I'm the creeper, catch me if you can. So very innocuous as far as malware goes. But they realized at some point that this had gotten onto every machine on their network and kept duplicating itself and was showing no signs of stopping and was starting to actually cause disk usage issues. So they made a new program called the Reaper, which just looked for the Creeper program and deleted it. And so this sort of, I think, represents that, one, if you've got computers, people are going to do bad things with them, whether they mean badly by it or not. And two, it's actually pretty useful to be able to have software that can go look for the software doing the bad things and get rid of it or prevent it from entering in the first place. So how does malware detection actually work? There's two main flavors. One is what you'd call signature-based. So this is what you think of when you think of anti-virus. It has a big, long block list of software. And the simplest possible way to do this is to just have a big list of hashes. And it looks at a file and it says, this file belongs to this virus. Don't run it. But you can get a little bit more sophisticated. Obviously, it's very easy to defeat. The malware can just change a byte at the end. That doesn't affect the operation of the code. And now it's got all of a sudden a brand new hash. So there's kind of an arms race in terms of normalization. That'll happen. And it'll try to look up the hash. But all of the signature-based stuff, basically, the way it works is I've seen evidence in the wild of this software behaving badly. This software looks like a very specific kind of malware that I've seen before in the past. So I'm going to flag it as malware. So it doesn't at all generalize two new samples. It's never going to work against the first instance of a piece of malware that you see. In contrast, behavior or anomaly-based, these are the same things that are used interchangeably. Try to figure out what the software does. So these malware detection methods do generalize two new pieces of malware. It doesn't mean that you have to actually run the software and wait for it to do something bad, like send a credit card number to some foreign server. It can, right? You can sort of do runtime monitoring. You can also run things in sandboxes. You can also do static analysis and just look at the file itself or the source itself. And so for the most part, we're going to be talking about the latter category today, just because while signature-based malware detection does have a lot of uses in the package repository setting, we're really concerned about not exact copies of malware that we've seen before, but sort of new malware that's coming onto a package repository. And so these are kind of the flavors of behavior-based malware detection. There's the static analysis, which looks at the software. And so if you're in Python, you can look for things like a val of base 64 of some block of data. And that's very suspicious. It's very rare that you would do that in normal, genuine code unless you were trying to obfuscate or hide what you were doing. You can look for things, not to pick on Russia, but URLs in your software library that end in .ru, or in a binary. You can just run strings on that and look for .ru. It looks for execution of code that's loaded remotely, all sorts of stuff. And the heuristics can range from very, very simple to very, very complicated that try to parse the AST of the program or decompile something and figure out what it does. The static analysis, I would say, is in general, because you don't have to do that decomplation step. Much easier to do in interpreted languages. So we're going to talk a lot today about how this works in Python. It can definitely miss things. The sort of flip side of that is dynamic analysis. So this is where you actually are running the software and seeing what it does. This is obviously a little dangerous, right?  So in general, you want to be doing this in kind of a sandbox environment. It can also miss things. And in particular, dynamic analysis can be detected by malware, right? It's very easy to do something different when you notice you're running in a sandbox than what you would do under normal circumstances. So you can have a flag that says, if it looks like I'm being scanned by a malware detector, don't be evil, otherwise be evil. And here's a cartoon of the, I don't know if you remember this Volkswagen's emission test scandal a couple years ago, but that's exactly what they did. They had the car figure out when it was being emissions tested and when it was, it emitted less stuff. So how does all this stuff work? There's a pretty rich academic literature on malware detection. As you can see, there's hundreds of thousands of academic papers on it. And techniques that get used range from regular expressions to looking for patterns in the abstract syntax tree of the software. They decompile things. They sandbox. They have antiviruses, which are more often like these signature-based methods, which do hash or file-based matching. As part of that, they often do normalization of the software that they're checking. You can look at metadata about the software filenames. Where did it come from? Is there a signature on it? Who is that signature from? And then I'm just going to hand wave over deep learning and AI. But there are a number of methods that throw a neural net at a piece of software and see if that neural net says it's good or bad. And then, obviously, you can combine these in arbitrary ways. Cool. So that's what malware scanning tends to look like. What's a package repository? Or what's an open-source software repository? So these are usually tied to operating system or language ecosystems. So you can have things like apt or homebrew or portage. In some sense, you can consider the Apple App Store or the Google Play Store to be a package repository as well, though that's not the focus of our talk today. There's language ecosystem repositories, like PyPI, which is going to be something we're talking a lot about today. There are things like NPM. And then there's things that are in between. So Conda is a repository that is focused on data science and machine learning applications. So it's mostly Python, but it will blend. It'll install stuff from Fortran all the way to Cuda code to whatever it needs to. There are things like Nix, which will install, again, pretty much anything. So there's all sorts of these things. And I'm being a little bit sloppy here. If I'm being precise, we should be distinguishing between package managers, which are the thing that you're running on your machine to install the software and the repositories, which is the remote service where the software itself is living. And so when we're talking about malware scanning, we're usually talking about doing that on the repository itself, either to detect malware that's been uploaded after the fact, or to prevent malware from even winding up there in the first place. And one big distinction we're going to make is between curated and community-based repositories. So curated repository, and so something like the Debian app repository, will have a small number of trusted maintainers. So these are often members of an organization, employees of the company. And so often your threat model says, let's really, really hope that none of the people with access to published packages to the app repository are evil. And if they are, we're kind of in a bunch of trouble anyway. And so malware scanning is not necessarily the solution here. Community repositories, in contrast, allow quote unquote anybody to submit. So if I wanted, I could close this tab, open a new one, make a new account on PyPI, and have software uploaded inside of, I don't know, 10 minutes for my Hello World Python package. And that's really actually great. We're all at the open source summit because we think open source is great. Democratic software development is awesome. Everybody gains when we have this big software commons that everyone contributes to. But it does come with risks. And in particular, we've seen a number of instances where there are malicious packages on these software repositories. And I could have kept going, but I was starting to run out of room on the slide. There's all sorts of attacks that you see. And if anyone's curious, come up after the talk, and I can point you at a cool data set that sort of just lists compromise after compromise after compromise, or at least attempt it at compromise, attack after attack after attack, on these kinds of settings. So it's at this point that folks who sit at home and comment on websites with orange banners tend to ask questions like, why don't we just scan for malware? This will solve all the problems. There's this rich literature on malware detection. And so obviously, if we do that, we can just have a little program that says, if is malware, don't let people upload it. And problem solved. And so we actually started this project, and we thought, yeah, we hear people say this all the time. No one's really doing this in practice. What's the reason for that disconnect? And our assumption is the people who run these repositories are pretty smart and pretty dedicated to maintaining the security of these repositories. So if they're not doing it, there's probably a good reason. And maybe the reason is just the gap between the performance that we see and the needs that we have. So let's go off and let's download a bunch of these things and benchmark them and see how well they do. And then that's the project. But at some point, we decided to stop before we just collected some data and went to these repository operators and said, hey, we know better than you about how you should be running these things. And so instead, we decided, before we sit down and try to benchmark, we need to know what we're looking for. What sort of properties? Like, what are the compute resources available? Is it OK if we come up with a good malware detector that takes hours and hours and hours to run on every individual package? Are we OK with missing malware occasionally? Are we OK with flagging things that are good as malware occasionally? And these are the sort of questions that I would rather answer ahead of time than sort of after the fact after we've gone through a lot of work to make the case for why we should or shouldn't be doing this. Yeah. So this slide is basically what I was just saying.  And in fact, we weren't the first people to ever propose doing this. And we weren't even the first people to go out of our way to try to put this in place on a repository. And in particular, on PyPI, so the Python Package Index, which is run by the Python Packaging Authority, which is part of the Python Software Foundation. In around 2019, they sort of started putting together the groundwork for a big project that did automated detection of malware. And so this got merged. You can see on the left in February 2020. And then in May of this year, it got removed. And so what's the reason for that journey? And so we wanted to start with sort of so-called user research on PyPI. Can I just sit down and talk to people who are involved with this and say, what were you trying to accomplish? Why is this something that never wound up really getting turned on and prod? And why did it ultimately get removed? And then I also wanted to talk to someone who was an academic, who spent their time working on the malware detectors, and say, what are you trying to build? And what are you trying to look for when you're building these tools in order to make them useful in such a context? And this is going to let us identify sort of the requirements that we have when we're deploying these things. And then as part of this, we can identify other priorities. I don't want to come in and say, hey, automated malware detection is the solution to all your problems when maybe there's a bad password policy and everyone's password has three characters in it. Or what are the security priorities that you, as an administrator of PyPI, are dealing with? And where does malware detection fall in that list? So what did we actually find? And the first thing that's kind of an interesting point is, does malware on these repositories actually matter? And I'm picking a little bit on phylum here, which is a company that does security research. And to be clear, I think this is good and important research, and I'm glad that they're away to find attacks on PyPI. But they made this flashy blog post, and they say, oh, we found an ongoing attack. Someone's uploading malware to PyPI, and you're all going to get hacked. And then if you look at the names of the packages that they're uploading, it's like libinfo hacked. Like if you pip install that, you kind of deserve what's coming to you. And in general, I think malware is sort of a relative concept. If you think about penetration testing tools, they do things that look a lot like what malware would do. So is that bad? Well, it depends on your context, and it depends on your expectations as a user. And so I don't, in fact, think that there is one judgment that even a human who sat down and studied this for years could make that says, this software is definitely good, or this software is definitely bad. There is no such thing. It's all relative. Furthermore, for malware to affect someone, it needs to actually get run. And there's kind of the saying, if a tree falls in a forest and no one's around to hear it, doesn't make a sound. Physicists tell me that the answer is yes, but the point of this parable is we don't care. And I feel the same way about malware. If someone's going to upload some malware to PyPI, and then no one downloads it and no one runs it, it's not a big deal. And in fact, most packages on PyPI, it's one of these things that follows a power law distribution where there's a handful of packages that pretty much everyone downloads. Really, really popular things like Request, Async.io, and so on. And then there's this big long tail. So PyPI has about half a million packages on it right now. There's this big long tail of packages that got one release, got downloaded maybe once, probably by the author of the package, and then never again. And PyPI is actually really awesome in that they commit to making a lot of data, open source, and available freely. So they have a big data set that you can query about downloads of different packages, and you can see what are the statistics. And every package does get some downloads, but we think, and this is kind of confirmed by talking to these administrators, we think that the bulk of these downloads for most packages are coming from mirrors that are just scraping the whole thing. So every time something gets published, if I have a mirror and I'm collecting to host internally at my organization or for research purposes, I'll just download that right away. And so we do see downloads of every Python package, but we're still pretty sure that most packages, no one ever runs them. So all things equal, right? We prefer to remove these things.  We don't need to be hosting that on PyPI and we'd like to get rid of it, but we care much more about specific cases of malware. And so one of these is typosquadding, right? Does the package have a name that someone would plausibly install by accident? And so this could be, you know, requests is a popular package, it may be if you say requests with two Ss or something, someone's gonna try to grab that package name and it'll get installed by accident. There's a, you also worry about compromises of existing packages. If someone takes over, you know, they hack the account of someone who publishes one of these really, really popular packages, we worry a lot about malware there too. But again, something like self-hacked CVNVIDIA, like maybe that's not quite as big a deal. The other thing we found in talking to them is that there are different constraints than you might think. So one thing that I was worrying about when I was saying, oh, what if PyPI is gonna run these malware detection scripts? What if they take like four minutes to run? Like that feels like a lot of compute, a lot of dollars to burn on AWS or whatever. And it turns out most of these repositories don't really care because as open source projects, while they may be incredibly understaffed, they don't suffer for want of cloud resources. It's the easiest thing in the world for these, you know, sort of cloud providers to say, we're not gonna give you any volunteer time to help make the repository better, but we can throw some credits your way. And so if PyPI comes along and says, hey, we're very desperate and we need to run these malware scanners, pretty easy for GCP, AWS, Azure to say, sure, like have some credits with which to do that. So we're not actually that constrained on compute. I mean, to a point, right? If you, we were running some fancy deep learning thing, like you hear about these LLMs that cost $14 in compute per query or whatever, that's probably not gonna fly, but something that's even pretty computational intensive isn't actually a huge deal. The latency is also a little bit flexible. And what I mean by that is, if you're gonna block publication of a package, you have at most a second or so, right? Like when I try to type NPM publish, if that goes through a malware scanner before it's actually available on NPM, the expectation is that I'll be able to download it pretty much immediately after publication. And this is just what users have come to expect. You wind up in a situation where you're trying to redeploy something and the thing you're redeploying pulls from PyPI or NPM, and like you're making an emergency fix, you want to be able to turn that around pretty quick and not see something like, oh, your package has been held. It's gonna be another couple hours until someone can manually review it, blah, blah, blah. So that feels like in general and on starter, but if we can say, we're not so concerned about preventing these things from winding up there in the first place, we just really want to catch them after the fact or pretty soon, eh, no big deal. It's okay if the latency's a little bit longer. But engineering resources are quite limited, right? If you say, hey, we have something for you to implement, all you have to do is read this academic paper and transcribe this awful pseudocode algorithm into a real programming language and debug and put tests in for that and then deploy it. That's actually a problem. So anything that we want to roll out has to be pretty simple. And also there are limited admin resources. So the admins on these repositories tend to be quite technical, but they have pretty limited time. And in particular, PyPI is quote unquote volunteer run. So most people who are admins on PyPI, it's not their day job. They have something else that they're paid to do and something else that they're supposed to be doing during kind of business hours. And if a report comes in and needs to be dealt with on an emergency, some of them are in positions where they can take 20 minutes during the day to go deal with that, but others kind of have to wait for five o'clock and quit in time before they're able to do that. And so in particular, what this means is that false positives when like a proposed malware scanner says, looks at some legitimate software and says, hey, this is suspicious. Why don't you the human review it? That's kind of an issue. And so I can write you on the right. There's my malware scanner that has no false negatives. It's going to catch every piece of malware. And the problem with this, of course, is that it causes a lot of noise. And so when you're thinking about false positives, you want to think about the base rate. And so what do I mean by that? Every year, there's about a million package updates that happen on PyPI. So there's half a million packages, but most packages get updates, new versions, and so on. And each one of these is something that we're going to want to scan for malware. There's about 10,000 packages that get removed by the administrators every year. Some of these are malware. Actually, a lot of them wind up being like spam and things that are not going to cause any problems, but are just like the full upload of a feature film. But they put it in dot wheel format for some reason. If you look at the newly published packages on PyPI, this is actually quite common. I don't really know why, other than I guess it's like free hosting. So if I assume an unrealistically good malware detector, and I assure you that this is unrealistically good, and we'll go into some benchmarks to compare it with soon, where it has a 100% sensitivity. So it finds every package that we want to take down. And it's got kind of 1% specificity, which means that only one out of every 100 good packages are going to get flagged. So these are really, really good numbers compared to what we see in the literature. Most of the reports are going to be false positives. And that's maybe a little bit counterintuitive.  But you have to realize there are many more good packages than the bad ones. So we wind up, basically, at this point with a majority of reports being false positives, which means a lot of wasted admin time. And the worst part of this is the things that are most likely to get reported as false positives got reported because of some heuristic, which means that there is some ambiguity as to whether they're good or bad. It's not the cases that someone can look at and say, immediately, oh, that looks like malware. That looks legitimate.  And this is the big reason why these malware checks that used to run in PyPI got removed is they were creating far too much noise and far too many false positives. The other really, really interesting thing that I learned is there is a malware detection system in place. It just involves humans. It's not fully automated. And so in this system, what happens is independent researchers will run their own tools.  And then the human who ran the tool will go ahead and look and check those and make sure that there are no false positives. They'll filter those out. And then they just email the PyPI admins and the PyPI admins do a quick confirmation to make sure they're not taking down something legitimate. And then they get taken down. And this is kind of surprising that this is the solution that's emerged because it's not like these researchers are employed by the Python Software Foundation or anything. But it turns out they get something out of it, which is, I guess, fame and glory in a very limited, nerdy sense. And so for instance, that blog post that I showed a picture of earlier from Phylum is one instance of this. Security companies often really like to say, hey, we found all this malware. We're keeping you safe by our products. And it seems like actually a worthwhile investment for them. But it's great because then the PSF doesn't have to employ teams and teams and teams of vulnerability researchers to do this. And they don't have to employ tons and tons of admins to scan through false positive flags all day. And these researchers now are really incentivized to have good tools that they keep updating because malware authors are pretty crafty. And if you knew the total set of automated malware checks that we're going to run on PyPI, it's a pretty simple matter to go run those yourself against your packages before you try to upload them and snare people. And you can do things like keep the checks secret, but that feels very contrary to the open spirit of open source that we work in. So with that out of the way, I'm going to spend relatively little time on the sort of quote unquote meat of this just because I think, as you've seen, it's a lot less interesting now that we know what we're actually looking for. But we did do some benchmarking. And it's, I think, at least instructive to consider the gap between where we're at and what we need. So what do we use as a data set? There's a couple of previous projects that collect malware that has been on open source repositories. So one's called MAL, OSS one's called the Backstabbers Knife Collection. And these are sort of previous academic works where they went and they collected from various package repositories.  And so you cannot go just download this because it would make it a little bit too easy for someone with malicious intent to have a big repository of bad things to pull from and re-upload. But it is available on request. And so if you can just say, hey, I am so and so. I have a research interest in this problem. They'll go ahead and send it to you. And then we needed a data set of Goodware, which is non-malicious packages. And actually, it's pretty easy to do this. The data set of Goodware is just the packages that are on PyPI because we assume that things haven't been taken down. In theory, there could be things that aren't detected. But those are the things that no existing malware tools have caught. So our tools are obviously not going to catch those. So there is sort of this sort of upper bounds how many of the bad packages we know we're catching. But it's better to be able to catch some than none at all. So we just took kind of a random set of packages. We took a set of popular packages because there are, again, a half million of these things. And actually, these malware scanners tend to take quite a while to run. So we only grabbed 1,000 or so to make it kind of computationally tractable. And then we de-duplicated the set. OK, and so what tools did we use? We started thinking we were going to have like a dozen of these things that we were going to run. But it turns out we had kind of three requirements. One is that the source is available. And so actually, a lot of these proprietary tools that companies offer, because they're proprietary, the source is not available. So we can't run them ourselves. We were also looking more for anomaly-based detection, which actually most everyone does. And then finally, there is actually really important point. A lot of malware detection tools are kind of like engines. And they say, we're going to provide you a system that looks at a source file and runs a regex on it and then reports of that regex matches or not. Or we're going to parse a Python AST, and we're going to let you make some query about what does it contain code in this structure, a function call with this name, or whatever, which is great. And it's a great start. But that actually doesn't let you detect malware. The kind of meat of that is what you'd call a detection rule, which is the regular expression itself or the AST query itself, which says, oh, I'm looking for a val of base 64 of something. And a lot of these analysis tools don't make their detection rules available, which makes it actually very hard to benchmark them. We could write our own, obviously. But then what we'd be benchmarking is not the engine. It's sort of the bad rules that I just came up with. So we kind of narrowed down to three of these tools. There's the checks that actually ran on PyPI, or we're supposed to run on PyPI, but never got actually deployed. There's a tool called OSS detect backdoor, and there's a tool called Bandit Formal. And so, OK, what's our setup? We're going to take our packages. We're going to get kind of our data set. We're going to get the latest releases of each of these things. And we're going to run our tool and then count the alerts. And so what did we find? We found that these are the packages with at least one alert when we ran the scanners on them. So OK, if you look, we ran in two settings. One is we just ran on the setup.py file, which is sort of things that can compromise you at install time. When you install a Python package, we didn't know better than to let the package that you install run arbitrary code. A lot of newer packaging systems, say things like, OK, cool, we are just going to, when we install a package, put things in place.  And so if you look, OK, we're finding most of the malicious packages, which sounds pretty good. It's a majority in pretty much every case. But then you look, we're also finding a majority of benign packages, too. And so I'll draw your attention, in particular, to the PyPI checks that were proposed. If you just look at the popular packages and you scan all the files, not just the setup.py files, that's 94% of packages got flagged as potentially malicious, which I'll point out is higher than if you actually looked at the malicious packages. And so maybe we're not being fair. These tools aren't issuing verdicts, they're just issuing alerts. They're saying, hey, we noticed something matching this pattern. And the table I just showed you were packages where at least one alert fired. And so alerts can be innocuous. And maybe you need to, only certain combinations of alerts are things that we should worry about. And so you could imagine coming up with something really sophisticated for looking at sets of alerts and turning that into a binary good or bad analysis. We didn't want to do that. So instead we just said, well, what if we thresholded the quantity of alerts? And again, these are the things that you would need to have in place for these to be deployable. You can't just hand a regex matching engine to PyPI and say, hey, if you wrote really good regexes to look for malware, you'd be able to find it. Right, like, so, okay, so you need to actually have a prescription here. And so we looked at kind of setting various thresholds. And so these are charts that show kind of, oh, what happens if we say that first table was at least one alert? This is what happens if we go one, two, three, four, five. And what you wind up seeing is that as the threshold of alerts goes up, the number of false positives goes down, but so do the number of true positives, right? And again, pretty soon you're catching so few, and often, like if you look at this chart D, you'll find that the number of true positives goes down substantially faster than the number of false positives, right? And pretty soon we can no longer distinguish between the malicious and the benign packages. And also, I'll point out in general that people really love to throw scores at things for scanning malware, but the score of actual malware just needs to be anything non-zero, right? Like, it just takes one line of code to turn a package that's totally innocuous into something that does something evil. And so I don't really love this idea of thresholds, but we figured we'd give it a shot. So those are kind of our empirical results, and if you wanna dig a little bit deeper again, I'll refer you back to the paper or I can take questions in a second. But yeah, so just wrapping up, we're a long way away from having in-the-loop automated malware detection. Other security measures should probably take precedence. And in fact, if you look at the PSF blog and the work that PyPI is doing, they are. PyPI rolled out two-factor auth requirement recently and is sort of ramping that up. And for the malware case in particular, a pretty reasonable alternative that requires some manual work has evolved, it could be better and there's a lot of places for it to be better, but it's not necessarily in the tooling. And I didn't get into it so much, but the worry is not that the malware detection tools that the independent researchers are doing aren't good enough. There's plenty of people working on that problem. The worry is that the interface that the admins have to deal with when they're taking down packages, right now they often have to download the package, unzip the wheel or whatever the format is, inspect the files themselves in the console and kind of look at what's been reported. You could imagine sort of a better web application that kind of exposes, hey, a researcher flagged it, here's the line of code that they're pointing to at evidence like almost like a dating app interface on your phone, like swipe left, swipe right.  And again, if you just talk to the Python folks, they will tell you, and they maintain actually a really great list of they call it like fundables.markdown, which is just things that they really want done but they don't have the time to do themselves, that they've identified as high impact security improvements. And that's a really great way to listen to the folks who are like actually on the ground before they hang out in their ivory tower and do their research and try to impose a solution on them. And I will say if you are a researcher, this is a really great opportunity because it makes writing the introduction of your paper really easy to say, hey, we actually, instead of trying to justify this convoluted thing that we did, we solved a real problem and here's really good, hard evidence as to why the work that we're doing is impactful. And then I'll also give a shout out to the OpenSSF, so which is part of the LF. They had a whole day of talks on Monday, which I assume will be available online pretty soon. And the OpenSSF runs a securing software repository as working group, which is really great because it brings together folks who work on these package repositories and folks from industry and researchers as well, and it helps coordinate and sort of fund cross-coding efforts, because you'll notice a lot of the stuff that I was mentioning has been actually effective, two factor auth. None of that is super unique to Python itself. None of that is super unique to NPM itself, right? Like, and they can all benefit from implementing a lot of the same tools and solutions. Yeah, so that's the talk. Happy to take any questions? Billy? Yes, so that's a great question. So some of these tools, the alerts do come with severities. And we looked at different decision rules and then we thought about like, oh, should we try to have some statistical algorithm learn a decision rule? And then we got kind of like three steps into that and realized that at that point, we weren't evaluating existing tools. We were making our own new tool. And so, again, I think that points to a need for, if you're trying to design one of these tools, for it to actually, you know, it's fine to sort of make that granular alert level information. Oh, we noticed this. It's a severe, you know, level alert. Oh, we noticed this. It's just a little suspicious. Make that information available so that if a human's gonna go review it after the fact, they have that available to them to make a decision. But you also really need to make a like binary, like good or bad verdict available. And that can even be tunable, right? Like you can say like, oh, I don't know what your data set looks like. You're gonna need to mess with this parameter. So maybe that's like the threshold or maybe that's something else. But I think that's the job of people proposing these tools, which is I guess my hop out way of saying I didn't wanna do all the work of trying to figure it out. But it did feel like at that point, we weren't evaluating the tools. We were sort of evaluating our own new tool that we were making up on the fly. I'm not trying to keep anyone here, so we can just call it. Thanks everyone.