Kelsey hails from the far off town of Cisco where she currently works as a mentor for the HackBite Academy for having her own internship in education. She's previously worked with the Internet Archive building tools to improve user experience and accessibility and is here today to talk to us about how to move beyond the Hello World tutorial experience as a beginning developer. Please make her feel welcome. All right. Can you guys hear me? Awesome. Cool. So welcome to I finished the beginner tutorial. Now what? I am Kelsey Karin Holly. I go by Karin at work and Kelsey in lots of other places. So I figure go by both. My Twitter handle K Karin Holly. I'm not super active, but I am responsive. And if you want to follow along with my slides, they are available on GitHub. So I am a software engineer currently working at Honor. Honor is a Python shop and we are hiring. Feel free to ask me later. I have a CS degree from UC Santa Cruz and I've taught girls who code over the summer. I got really passionate about it. And I'm currently a mentor at HackBite, which I am loving. In my spare time, when I do have spare time, I am an avid reader and I also am a crafter. So if you like knitting or crochet, I'm into that too. So this talk is going to cover three different learning approaches. And I'll delve into each one. I'm also going to talk about getting connected with the community. And then I'll summarize everything at the end. So some things that I'm not going to cover. I'm not going to talk about boot camps. I'm not going to talk about code schools. I'm not going to talk about college or online courses or how to find more tutorials. So the first one I'm going to talk about challenge based learning. What do I mean by that? There are lots of websites out there that will give you lots of different exercises. You can do one at a time and kind of grow your skills through a small problem instead of taking on something big. This style might appeal to those who don't want to start on a big project, don't want to build something from scratch. If you like rewards or trophies or badges, there's also that kind of a motivation behind it. And another thing that I really like about these is most of them have pretty strongly covered edge case testing. So if you think you solved it, awesome.  So this is a very small list of the quick Google search that I did to come up with all the different tools out there. There are a lot of them. I think I played with three on this list. Some things to keep in mind when you're choosing a tool. If you're wanting to learn more than one language, maybe today you're learning Python, but next month you want to start learning JavaScript, try and pick a tool that's language agnostic or that has multiple language support inside of it. So you're not picking one tool for Python and then having to find something else for JavaScript or something else. As I mentioned, strong solution testing is really important. They help you out to figure if your code is not optimized, it's not running at speed, or if you missed an edge case, things like that. So having a testing that gives you an error message or something to go off of is really helpful instead of just saying, I failed. A lot of these tools have in-browser interpreters as well as editors, so if you're wanting to be on the go, if you don't have a dedicated laptop, many of them you can just work completely online with them. I mentioned badges before, so if you like reward motivation. A lot of them have built-in review and feedback mechanisms too, so if you want to get ideas on how to improve your code or if you want to help others, there's a few that do that. And many of them have competitions, so maybe you're not ready to do a competition yet, but they do host them every month or every week, so when you feel a little bit more confident, you can join one of the competitions. The first one I'm going to talk about is HackerRank. So some things I liked about HackerRank, they have a pretty good diversity of problems as well as languages. They support both Python 2 and Python 3. The way they have things grouped is from easy to advanced, so maybe you don't need to do the hello world anymore, you've figured that out on your own. You can move on to something a little bit more advanced, maybe you haven't played with strings as much yet or some of the math-related functions. They also have that in-browser editor and interpreter that I was talking about. I believe they have syntax highlighting. Their tests run in real time, so if you hit, you know, submit my code, check and make sure it works, you get to watch as each one passes or fails, and you'll get to see the error messages of why it failed. Maybe it wasn't optimized, maybe you missed an edge case. Some things that I don't like about HackerRank, they don't have a community collaboration type of built-in. When you submit your code, you submit it to be graded and then it's lost in the ether. So there's no, hey, you could improve by doing this. There's also no way to take those problems offline or test offline. You have to do everything through their website. Another one that I've played with, Exorcism I.O. The things I like about Exorcism I.O., completely opposite of HackerRank, they do have a built-in review set. When you submit your code, it goes public and anyone can make comments on it or you can ask questions and get help from others. I think that's really cool. You might have done it one way, but there's a million other ways to code something. You can also join teams or create a team, so maybe there's eight of you all learning Python together and you want to start tackling these challenges. You can post your code, what you did wrong, what you were missing, where you need to improve, and you can grow together. Something else I thought was really cool, their exercise list is actually an open-source repository on GitHub. So me, as an advanced engineer, I might see that, oh, this has a bug in it or I really think that you should include this exercise on your list and I can help contribute and help them grow in what they're offering their students. Some things that were not my favorite, it is terminal command line dependent, so if you're not comfortable on the command line yet, they do walk you through how to get set up and how to get started, but it's going to be a little bit of a learning curve or a barrier for entry there. And everything in Exorcism IO, you do run on your computer. There is no in-browser interpreter or test runner or anything else. Everything is run on your computer in the command line or in the editor of your choice. And another thing that I ran into, maybe this is wrong, but it seems like I had to complete all of the exercises in order. So in order to get exercise two, to even look at it, I had to complete and get all of my tests to pass for exercise one. From a learning perspective, that's probably a good thing because it means if the next exercise is building on a concept, I haven't just skipped over exercise one just to get to two, but I'm the type who likes to skip around, but as I said, there are a bunch more challenge-based websites that kind of fit this model. So I've just covered two. There's lots more to explore. Pick one that appeals to you or has a community that you like. So summarizing the challenge-based learning, it's a great way to build incrementally on a new language. The built-in test suite will help you figure out what you've missed, what you need to fix. It's a great way to learn how to do things, but you're coding style as well. And many of them do include competitions for prizes, money prizes, physical prizes, or have jobs attached to them.  So example-based learning, what I mean by this is looking at someone else's code and dissecting what they did and understanding what they did and then being able to take those concepts so this is kind of appealing for someone who needs some starting point. They're not ready to go off and build on their own. The exercise approach didn't appeal to them, but they still want to be in code and learning. If you enjoy reading other people's code or reading other people's stuff and kind of seeing how they did things, this is also very appealing. And if you've been reading about a lot of abstract concepts, maybe you've read about Decorators recently, but you've never seen one in the wild. So I'll look at someone else's code to see what that decorator looks like and how it's running. So finding a project, I do recommend that you have at least a basic understanding of GitHub and Git. If you don't, there's a couple of tutorials I'll put on my last slide that'll kind of help you get started, but you're going to need to find things on GitHub and then probably clone them onto your local machine. So things to look for once you have that cloned or once you've picked a project or when you're picking a project, a good read me. Does it have a good setup instructions? You know how to install, to build it, to run it. If there's no read me, this probably isn't the repo for you. Is it under active development? Was the last commit eight years ago? That's probably not what you want to play with. It might be out of date. It's probably broken. Avoid those. If the contributor is really responsive, if there are lots of issues being posted and they're answering or they're doing pull requests well, if you run into problems or have questions, that means that you can interact with them and figure things out as well. The next one I'm going to say is a little bit more personal. It'll be up to you what you call easy to read or follow along with, but I recommend you not dive into something that's really complex for you. Find something that you're going to be spending a lot of time in. Make sure that you can follow along with what it's doing. And then don't be afraid to Google for foreign words or terms. I've heard of pip before. Most of these repos, if they're Python, are going to be talking about pip. So maybe read about what that is. Things to avoid. If it's importing 100 outside packages and you've never heard of any of them, probably not what you want to start with, because then you're going to be researching those 100 things before you even get to the code you want to be reading. If it's overly complex, if you're importing from this file which is importing from this file which is importing from this file on, if you have to trace that long tree, that's going to take a lot of time too. Probably not something you want to get started in. And then if you don't even know where the main file to start reading is, don't bother. If there's not like a main.py or something that says, hey, this is where the code starts, then I wouldn't use that either. So I've talked about how to pick a project. Let's talk about searching. So if I'm going to go on GitHub, what kind of tools do I use? I use Slack a lot in my daily basis. Do they have anything open source that maybe I can play around with? I can search on GitHub and see. I have a lot of friends or engineers who are also doing various open source projects, various hobby projects. I could look at their GitHub to explore their repositories and I can talk to them directly if I run into problems. The most important thing I want to say is find one that's interesting to you. I don't just mean like the code is interesting, but the topic. If you're interested in machine learning, I should pick a machine learning repository. If it's not a topic that you're interested in, chances are you're not going to want to spend the time to figure out what that repository is even doing, find something you're interested in. I'm a crafter. If there's a knitting repository, I'm going to be so into that. So quick search in GitHub for Slack, narrowing down Python. There's 4,000 results. I can pick some. Sure. Looks like we've got some Slack bots, some things that are doing something with the API. There's a lot of things I can explore. Once you've picked some repository to start exploring and digging into, some ways to start breaking it down. At a macro level, what is this code trying to do? Is it a bot? Is it a web app? Is it a command line tool? What is this repository doing overall? And then you need to kind of understand how they're getting what they require to do this. Are they importing from the request library? Are they importing from a file next door to this? What's going on to just generate the toolset they need? And then going into a more micro level, what is this code actually doing? What is this function doing? What is this line? What is this code block doing? Really digging in to understand how this code is working. And then asking even more questions. Why did they write it this way? Could you write it a different way? Maybe instead of using a for loop, you could use a while loop here. So why don't you change it? What happens? And if you throw an error message, those are the best! Because then you get to figure out what happened. And you get to learn why it mattered that it was done the way they had it written. Don't be afraid of error messages. They're really scary at first. I promise they're super fun. Maybe that's a personal opinion. And remember, you're not breaking their code by changing all these and generating errors. It's not like their server is suddenly going, you're doing all of this on your machine. Any changes that you make are on your machine. So it causes much chaos and havoc as you want. It's great. Once you're feeling a little more confident in this repository, you feel like you've got it under control. See if there's any open issues that someone has posted. Hey, this was causing a syntax error for me. Could you fix that?  See if you can build it out. And then when you're kind of done playing with this repository, what are the takeaways that you found? Did you learn some new third-party package? Had you never used requests before? But man, it's awesome. Did you get exposure to some new built-ins? Maybe sets were new to you and you'd never used them before. Were there concepts that you hadn't seen? Decorators or generators or some other concept that you had heard about but hadn't seen in the wild? You can now use those in a future project or at least are less afraid of them. So summarizing for the example-based learning, find a repository that's interesting to you that you feel comfortable kind of digging into. Make sure to ask lots of questions of the code. How did they do this? Why did they do this? What happens if I change it? Don't be afraid to cause all the error messages. Again, they're fun. They teach you what matters and why it's wrong. And then use what you learned. So project-based learning. These are the people who are like, no, I'm not into the exercises. I don't want to go planning or else's code. I want to build my own thing. And they're ready to get started from scratch. So if you already have a project in mind, maybe you have that blog that you've been meaning to get started. And of course, you should build it yourself. So let's do this. If you couldn't find a good example on GitHub, maybe no one was using the Slack API quite the way you want to. So you're going to build your own thing. It's really appealing for those who are very self-motivated and very goal-oriented. If you're a junior engineer and you're trying to interview at places, creating a portfolio, building things from scratch that way is really good. You can show them off to future employers.  Scope creep. If you've never heard of scope creep, it's when you start off with this little tiny concept and then it just balloons out. So you're going to build an email server. Great. All you need to do is send or receive emails. But we're going to do that. All you need to do is send and receive emails. But wouldn't folders be nice? And wouldn't filtering be really cool? And what about auto-forwarding? And then it just keeps going. So to help kind of manage scope creep as well as kind of build out how you should do this project, I recommend breaking everything down into to-dos. And a to-do can be anything from research a new package or a new concept, experimenting with how that package works, actually writing the code for a certain file or function, documenting everything you've been doing, writing the readme or the setup to make sure that anyone else who comes along and plays with your repository can actually run it. And then building tests as well. So all of those can be to-dos and they can be broken out even more. If you feel yourself getting discouraged, I once wrote a project that had a thousand lines of code and didn't test any of it. And then I was like, I should write tests. That one giant to-do of tests was looming over me. I hated it. So I broke it down into even smaller pieces. I had to test this file. And it wasn't as daunting then. I only had a hundred lines of code to test instead of a thousand. And it felt really good to kind of say, yay, I finished this one. And it felt like I accomplished something. It kept my momentum going, and it kept me interested in what I was doing. So I made sure to celebrate each excess. Anytime I finished one of those to-dos, I was like, I am awesome.  I have typed in an exact stack trace into Google before and been absolutely thrilled with the wonderful results that I get. Because everyone else does it too. They're like, I got this. What does it mean? Stack overflow. There's usually a great conversation. Yeah. So Google's great. Stack trace messages in Google. Super great. I'm going to tangent slightly and talk about my passion project slash horror story. I got really addicted to this board game called Castle Dice. And I was like, I know. I'm going to make this into a web app because all of my friends are moving away and I love playing this so much. Wouldn't it be great if we could just log in and play together? Yes. This is what I will do. So Castle Dice had three decks of cards. It had five different trackers. It had a board over here and a board for tracking over here and a third board for tracking more things. And there were turns and rounds and people involved. Anyway, super fun game. I recommend it to everyone. The complexity, though, for coding this probably a little more than I should have bitten off. So I started breaking it down.  Okay, so I need to figure out how to store these. How am I going to randomize the roll? What happens if I need to add the dice together? So there were lots of to-dos on my plate, not to mention the decks and the trackers and everything else. So did I finish it? Not at all. Not at all. But I do go back to it every six to 12 months. And for me, I wrote the dice five years ago. It's kind of interesting to see how my code has progressed. And I keep coming up with more ideas about how to fix it, how to change it, how to improve it. And there are small things that work. It's just not the full game. That's fine. That's totally fine. So my lessons after playing with Castle Dice, it was a huge project and it was definitely something that I shouldn't have chosen as my first, yeah, I'm going to break away and learn Django. There were a lot of things that I still don't really know how to approach. I never studied game theory, so I'm still trying to figure out how I can do you take a turn and now it'll tell me I've taken a turn and then it will tell the next person. Anyway, that'll be the next month thing to play with. I did learn a lot though. This was the first time that I had built a randomizer to roll this dice and do some stuff. I think it was the first time I'd built a dice, I was super proud of it. I'm still vaguely proud of it. Meh. And it didn't actually matter whether or not I completed it because sure, I didn't build the whole game, but I did build the dice roller and so you could click a button and watch the dice change sides and roll and give you different values and I got to show that off in an interview and I was embarrassed that I didn't finish the whole thing but they were really impressed to see that I had something working and that they could look at my code. So that's kind of cool too. It didn't matter that it wasn't finished. So to summarize, choose something small and break it into smaller tasks. Stack overflow in Google. Awesome. I consider everything a work in progress.  Are you sure it's not an abandoned project? No. I will come back to this. I promise. I promise I'm going to finish it someday. And the biggest thing, make sure you're having fun. This is a project that you chose to build yourself. So keep it fun. Don't be discouraged. So getting connected with community. We've talked about all these different ways that you can do code but getting into the community is what I find is really helpful because there are people even here today who, if I go up and say, you know, I'm really struggling with this Python thing, chances are they at least know what Python is, right? Pretty sure. They're using it in their daily jobs or they have used it in a daily job or they're on some open source project or some hobby project or they're just learning it like me. Or not like me currently, but anyways. So coming to these community events, you have lots of options to not only network but also get advice and guidance on things that maybe you're struggling with or that you need some help with and you need a new resource, new docs, whatever. It's also a great place to find a mentor. So I love mentoring. I hope I'm good at it. I think I'm good at it. I should talk to my mentees. Mentors will both help and encourage you so they're not just going to hand you the answer. Really good mentors will coach you to the solution. They want you to be able to think through the problem yourself and not just be handed the solution. They want to make sure that you can come up with it on your own the next time and understand why that's the correct answer. And we also really care about your growth. Like, I'm a mentor.  Yes. I just want to celebrate with her. That's something that I get to enjoy as a mentor. I have helped her through this process. I have coached her and now I get to enjoy the fruits of her labor. Not take credit for them, but at least enjoy them with her. So there's lots of groups, conferences, tech events, all kinds of things out there. These are the handful that I have been to or am aware of. Pie ladies, lunch was awesome today, by the way. There are lots of Python meetups, Django meetups, Flask meetups. If you're more into the conference scene, PyCon, North Bay Python, which you may have heard of. PyBay, there's lots of Python-specific things. Even if I just go to meetup.com and type in Python in San Francisco, because that's where I live, there's a ton of results. And these are just the groups. They're not even the events. If you're into other languages, there's Waffle.js in the Bay Area. I know there's Donut.js in Portland. There's Railsbridge. Hackerspace, I think, is more agnostic. There's lots of ways to find interactive communities. If you are of the female or non-binary persuasion, there's also Pie Ladies, like I mentioned, but Girl Geek Dinners are really great networking opportunities. Women Who Code has a great newsletter. They also do events. Girl Develop It does, I think, JavaScript courses, but I think they also do events. And if you're the type who lives in the boondocks in the middle of nowhere, you're like, I'm not near a metropolitan area. How am I going to do anything? There are tons of online communities as well. Pie Ladies has their own Slack. Exorcism.io that I talked about has a built-in community platform, so you're able to interact and get feedback anyways. So, to summarize my summaries of summaries, we covered challenge-based learning, so practicing with small exercises, taking advantage of the test cases when they do and don't fail, community feedback, competitions, all the things baked into that. Example-based learning, so digging into someone else's code and really understanding how it works, why it works, what breaks, asking questions, playing with error messages. We talked about project-based, breaking a large idea down into smaller pieces, preferably avoiding the large idea in the first place. Getting help with that, putting things into Stack Overflow and Google when you run into problems, and having fun. All of these, that is the most important part. We also talked about community, various events, finding a mentor. So, the biggest takeaway I want you to have today is there's no one right way to continue your learning journey. Maybe one of these really resonated with you. Maybe you hate all of these ideas. That's great! Don't take my word for it. Some of these have worked for me, some of them have not. Some have worked at certain times, and others not. So, pick your own path, your own journey. And then here are all the resources. Again, I work at Honor. My Twitter is kkaranholly. I did manage to post these slides on GitHub right before this talk. And then some other links that I talked about. So, thank you. Thank you very much, Kelsey.


My question is, what are your thoughts on psychics that use spirits to tell the future? Do they kind of mess with your future if you have these psychic things? I don't think it's a matter of messing a future directly, but when a psychic is, let's say, tarot readers or crystal gazers, pendulum readers, guided by spirits, they're guided by disembodied beings who have a greater experience of timelessness and therefore also an ability to tune into the future, actually. The thing is that you can never be entirely sure about what is being said and so it can be sometimes of solace and comfort, but it is dealing with spirits, so one has to be careful. Mainly the readers have to be careful because very often they take inputs from spirits but they don't offer something in exchange and then they can anger the spirits, I mean, anger, quote, unquote, because many readers don't understand that these are disembodied beings. It's not spirit. These are precise beings that are connecting with them and that they are using also. So there has to be a giving in return and unless they know how to do that, they might also have to pay heavy prices sometimes in health, mainly in not living a life fully.  So one has to be a bit respectful, you know, it's like when people go to a house and they are clearing that house of the spirits and they have sage and they have all of these things or you rent an apartment and then you clear out all the spirits, but the fact of the matter is that those spirits have lived there longer than you have and just because you have a contract doesn't mean anything. Maybe they have a different kind of contract. So one has to be very careful and very respectful and someone is dealing with spirits and maybe request them and actually best is to just leave them to do what they do, you do what you do.  So there is no need to mess around with spirits unless you are doing some real service to society. For example the shamans, they had their place in society because they supported people in times of distress by taking help from spirits, but they also knew or know how to return to the spirits, what the spirits need as well. So it's very often people do one side of the thing, but not the other side. So that's not a clever thing to do. In general it's always better to be very careful when one is dealing with spirits, you know. If you are getting it bred by a psychic, is it unhealthy to do if you are going to a psychic and they are reading you? Sometimes when you are in a state of deep confusion maybe or sadness or unable to bear the moment and its pain, the first thing is to surrender to the soul and to move into that, but sometimes the system is not able to do these practices and so it can bring some solace and some comfort to go and listen to a psychic and what they have to say. It is not always of much help, but it also gives a feeling of some hope. Certainly it is a play of the ego, but what I feel is that if one starts to go mad by identifying everything as a play of the ego, that's not what this practice is. It is about a certain ease also and if it eases the pain sometimes a little bit to go and talk to a psychic or a reader, I feel that lies within the sanity of human existence to move into those things. In the long run it is of course not something which will actually move you to the soul because even if let's say you have this story which you mentioned yesterday about this man that left you and you cannot get him out of your head for months and it's every day all the time. I early on spoke to a psychic but the situation she's like, definitely I will be moving, getting married and I had a lot of sessions with her. I believed it and also was speaking to the psychic so I thought in my head was it was I just listening to someone making me believe something and it wasn't all accurate and then when it happened I tried to speak to her and she just disappeared like she wouldn't so I just wanted they just picturing stuff and putting things together or is it fake or is it just like a money making thing It's all of what you said it's not necessarily fake it can be only for money making it can be not necessarily for money making there are psychics who actually are able to be very, very precise in their predictions and it holds and holds and holds and holds you have everything in that mix the point is not all those things the point is yourself it's about you if you actually really listen to what I'm saying let it penetrate into your thinking that if you tune into the soul and if you are in surrender to the soul and you come into your coherence that person or that man or that woman or whoever it is will respond very differently to you because the soul because when you are in connect with that soul that other soul feels that connect so that is very important not to forget as far as psychics go there are many psychics who are very powerful through the Trikala Drishti they are able to receive information which is that precise that it's breathtaking and continuously precise so these are abilities of Trikala Drishti the ability to actually move into a state of consciousness where you are able to see past, present and future so that exists but you also have psychics who communicate with spirits many who communicate with Archangel Michael Archangel Gabriel Archangel Shamuel these ancient archetypical spirits actually and so one cannot know for sure what each one is actually capable of the only thing you can know for sure is that your spirit which is your soul is what you can connect with that you can be sure of because you can feel it while sometimes it can be harmful as has happened to you where you have somehow believed something and then been shocked but it could also be that she was right not everything has happened already you know since yesterday I've been practising what you and I was trying to do it for the last 24 hours I know this is your last session so I just want to ask you if I'm not sure I'll see you next but how do you know if you're doing it right or not it takes some time but it's better to do it than not to do it because if you don't do it then it's like it has been till now it takes some time because the ego is very nice and healthy and big and fat and has lots of layers of just keep on pursuing it's a sadhana sadhana means a practice sometimes they stand on one leg for 10 years and a hand up in the air you ask them Sadhu Swamiji, why are you doing this? why are you doing this? they say Guruji said that's why I'm doing it my Guru said to stand on one leg with a hand up so I'm doing it but so many years he said he told me to wait till he comes back so 10 years have passed, 12 years he's standing like that that is called sadhana tapas shakti you keep on doing it you keep on practising tuning into the soul it's not going to happen yesterday it's not going to happen so fast always because you have to first put in the tapas shakti to do it even if he thinks it's wrong what have you done before you knew about your soul so you have nothing to lose and after a while you persevere suddenly one day it will just be there and you'll know it because there are many others who know it it's not the only one sitting on this chair who has that experience there are lots of them who have this experience even sitting in this room and they live from that experience when they are unsure about something they ask is this the right thing for me to do they actually ask that question they get a yes or a no and they go with it and sometimes they don't go with it because they're not strong enough and there's fear and then again they go with it and they keep on training themselves to deepen that experience of soul and then the Guru comes after 11 years and says now stand on the other leg and put the other hand up and put that leg and hand down that's not happening here right? so it takes some time but you also have no choice because you have that man in your head and he's in your head because you are not with your heart and I'm very sure about what I'm saying I'm a million percent sure that if you start to move into that knowledge of self all those souls around will start to tune in with you more because technically seeing he can message you know he can say hi even if he's engaged he can say hi but that also he's not doing and that's the point and he will do it if you are with yourself he's not doing it because he's pushing you to yourself that suffering is pushing you to yourself that particular kind of suffering it has a specific the reason that that suffer it's pushing you to yourself and it is not something to be happy about to have that suffering but it's happening because you are not tuning in to source mark my words if you do that you will hear from this man because he will feel that everyone feels when one is connected with soul they feel it people are that sensitive that they feel when the other one is actually sitting in tune with the soul and they are not afraid to approach because they don't want to see pain they want to see love and you are in pain now but it will go away


So, if you're fortunate enough to not have been on social media as of lately, as someone who is on social media pretty frequently, unfortunately, let me fill you in on what's been happening. Amazon, in an effort to thwart the unionization effort that's currently taking place in Bessemer Alabama, has become increasingly desperate to the point where they're embarrassing themselves. Now, at the time I filmed this video, it's already the case that the employees at the Amazon warehouse in Bessemer, Alabama have voted on the union, we're currently just awaiting the results. So if you know more about this by the time you see this video, just understand that at the time I filmed this, I don't necessarily know the outcome of said vote. Having said that, though, the lead up to this has been quite interesting because Amazon has gotten a lot more vocal lately on social media, literally going after politicians like Bernie Sanders, Mark Pocan. And we've also noticed for those of us on Twitter, a really large influx in the number of trolls that we've seen online. Now, it's really difficult to prove that it's Amazon who's behind 100% of these accounts. A lot of parodies have popped up, but still Amazon hasn't even really been trying. So if this account here is Amazon, it's hilarious because they didn't even bother to pay for a stock image. They just used the actual photo of the Dude Perfect YouTube channel. And if they want to use any photo from a YouTuber, this is not it because this is a YouTube channel with more than 50 million subscribers. So either it's the case that they stole this photograph from Dude Perfect or the man from Dude Perfect, the star of that channel, has time to not only work at Amazon, but also tweet about how terrible unions are. I don't believe this, like, at all, just so you know. And given that a lot of these accounts have been purged by Twitter, I'm assuming automatically it really goes to show you that Amazon didn't really even try to hide their astroturfing. And they've even been responding to journalists in leftist circles. So Walker Bragman tweeted out that Amazon workers should vote yes on the union and a paid show responded to that. And I can't show you what that paid show said because their account was purged. But if you watch a video by David Dole of the Rational National, he did a really good job at breaking down just how ridiculous all of these fake accounts are. And what's interesting is that they're all saying the same thing. So I assumed and many also assumed that these have got to be bots, like they're not actually human beings behind these tweets, because who would who would say these things? Normal human beings wouldn't say something like this about their employer. Who speaks glowingly about their employer? I've worked at Walmart. I've worked at Subway.  They're terrible. So you're never going to hear somebody speak glowingly about a company, a company as shitty as Amazon. So it's just hilarious to me that, you know, they would use these robots. Or so I thought, because apparently, according to a new report by journalist Ken Clippenstein, these aren't even robots. These are actual human beings that Amazon paid. They hired these folks to do astroturf, to do propaganda and defend them, salvage the reputation and stop the unionization effort. So Ken Clippenstein writes, Amazon's Twitter army was handpicked for great sense of humor. League document reveals. We didn't see any of that, by the way. Amazon ambassadors were trained to defend Jeff Bezos and clap back at Bernie Sanders under a program codenamed Veritas. Now it goes on to explain anticipating criticisms of worker conditions at their fulfillment centers. In particular, Amazon designed Veritas to train fulfillment center workers chosen for their great sense of humor to confront critics, including policymakers on Twitter in a blunt manner. The document produced as part of the pilot program in 2018 and marked Amazon.com confidential also includes examples of how its ambassadors can snarkily respond to criticisms of the company and its CEO. Several examples involve Senator Bernie Sanders, a longtime critic of the $1 trillion firm who has been targeted by it in recent days. It also provides examples of how to defend Bezos, to address speculation and false assertions in social media and online forums about the quality of the fulfillment center, associate experience. We are creating a new social team staffed with active, tenured FCE employees who will be empowered to respond in a polite but blunt way to every untruth the project description reads FCE ambassadors FCA will respond to all posts and comments from customers, influencers, including policymakers and media questioning the FCE associate experience. Kelly Nantel and Amazon spokesperson said via email FCE ambassadors are employees who work in our fulfillment centers and choose to share their personal experience. Sure, Kelly, the FCE ambassador program helps show what it's actually like inside our fulfillment centers, along with the public tours we provide. In 2018, Amazon admitted that the ambassadors were employees paid to honestly share the facts about what working in its fulfillment centers is like many Twitter users had at first believed the ambassadors were automated bought accounts due to the nearly identical format of their account bios, all of which feature the Amazon smile logo and begin with the handle at Amazon FCE. But that format was specifically mandated by Amazon. The Intercepts document shows we could also add an emoji to the username to give personality, for example, small box emoji, the document suggests. So this is hilarious to me.  They should have outsourced it, but they're too cheap to do even that. But they hire people for their good sense of humor, but yet they train them to talk to people online as if they're robots. And the only thing that they can do to actually allow their personality to shine through is add an emoji to their name. I mean, this is absolutely embarrassing. And they're going through all of this, all this headache, this effort, all to stop one union. So that goes to show you that unions are very powerful. And if they're doing all of this to stop one union, they know that this could catalyze a domino effect where more and more Amazon warehouses around the country start to unionize. And they know that all of this that they're doing, all of this funding of AstroTurf online and attempts to union bust, all of this is cheaper than actually allowing their employees to unionize because they know that if their employees had unions, they'd be paying them a lot more than the $15 an hour that they brag about. So this story is hilarious.  This is a multi billion dollar company, a trillion dollar firm as the article lays out. And their bots are just terrible. Their paid shills are terrible. I mean, even David Brock's paid shills for Hillary Clinton back in 2015 did a better job at humanizing themselves and didn't let you know that they were paid bots as easily, even though it was still obvious. But this is just next level. And the frequency that we see them confront people online is honestly startling. Like, we've all come into contact with these Amazon bots. And every time it happens, they get dunked on and then the account ends up getting deleted. So it's just it's sad, but good. I'm glad that Amazon is getting embarrassed and exposed because they deserve it, given how poorly they treat their employees.


Alright. Good morning, everyone. My name is Michael. I'm the Product Manager for Twitter Cloud Platform. And my name is Vino. I'm a Software Engineer in the Cloud Infrastructure Management Team. First off, thank you for coming. Together, we're going to be talking about how we build a chargeback system to help improve the overall resource utilization and efficiency of Twitter infrastructure. Unfortunately, I don't have a clicker, so I won't be walking around. So I'll be standing near the podium. So we'll start off with the agenda for the talk. I'm going to start off with an anecdote of how what prompted Twitter engineering to actually move into a more service-oriented model. This would provide context on the problem we are actually trying to solve. We will then deep dive into the chargeback system itself, talk about the engineering challenges in building that system, the product side of things, which is the reporting and things which come out of the impact it has had on our customers. And finally, the future around how chargeback plays a crucial role as part of a larger unified cloud platform. With that, let me just start off. Do you guys remember this particular event? Anyone? Okay, some people are nodding their heads, so my assumptions, they do know this. So this is when Spain won the World Cup, the Soccer World Cup back in 2010. This was a momentous moment for Spain, but at the same time, it was quite crucial for Twitter too, simply because we used to run, we were one of the largest Ruby on Rails deployments at that time, and we aptly called our application the monorail. At its peak, the monorail could serve about 3,200 tweets per second, which is a pretty paltry number if you compare it to what we're doing today. That said, that was a 5x increase at that point compared to the TPS. We had pushed or we had did everything to push the monorail to support that TPS request. We basically threw machines at the problem to reach that scale. We spent significant engineering hours tuning the Ruby VM, building deployment tooling, and even building tooling to detect failures. Unfortunately, not everything was rosy. Twitter did go down numerous times during that event, which begs the larger question out on what truly was the problem. We couldn't simply depend on the monorail for its reliability and the developer productivity problems. Given that routing, presentation, and logic were all tightly coupled into a single system, it became extremely difficult for us to debug, scale, and contribute code to the monorail, which brings the next evolution of what happened at Twitter. Twitter engineering basically reviewed what had happened after the World Cup, and we decided to re-architect our entire stack. We decided to move to a more service-oriented model, which ended up basically decomposing the monorail into multiple microservices. The gains were pretty significant. We were able to address many of the plaguing problems from the monorail. For example, it provided a clear way to define ownership of various services and even charter for the teams. This was crucial as we were scaling up to support the product. It was easier to know who was working on what. We built services that focused on specific aspects of the Twitter product. For example, the who-to-follow section is actually a separate service, which recommends users as to who you should follow based on your follow graph. The interesting piece is we were able to, like, separate that out in terms of fault isolation. So if who-to-follow would actually go down, it wouldn't impact Twitter the entire product, but rather that feature would just be gracefully degraded from the entire product experience. We were able to build and scale services independently, too. So who-to-follow service would not be seeing the same traffic as our timeline service, which is the bread and butter of Twitter.com. Finally, there was a significant improvement in the developer productivity side of things. Microservices made it extremely simple for developers to build, iterate, and launch their services pretty quickly. I want to share a quick anecdote from 2013, which reaffirmed our investment on the microservices area. Does anyone know this? Okay, so this is a movie called, this is an anim movie called Castle in the Sky. It's hugely popular in Japan. And there's one particular scene in the climax where the protagonist casts a spell.  The moment the scene was played back in 2013, Twitter users actually took to Twitter to tweet out that word at the same time, which actually helped Twitter reach a peak TPS of around 143,199 tweets. This was 28X more than our average TPS we used to see that day. It was staggering. Our systems did feel stressed and many of them failed, but fortunately, the experience about creating a tweet and viewing a tweets were still alive and it worked well, because many of our core tier zero systems were abstracted out from the other supporting services. So if you did try to tweet, it would work is just that the product experience all around would not be working as effectively. But that said, people were still able to consume and create tweets, which is the most important part to us. So you can imagine hundreds of such events happen on a minute, an hourly and a daily basis on Twitter. It is a platform for live conversational and events. Tweets per second is not the only metric which matters to us. And to kind of re-trade on that, I want to share this particular event which happened 2014. Ellen DeGeneres tweeted the selfie out from the Oscars podium. So it didn't register the largest tweets per second, but it got re-tweeted so many times so fast that Twitter actually, the particular profile page of Ellen was broken and this tweet was inaccessible for over an hour. This was a huge set of zero back in our world. But it shows that tweets per second is not the only metric which shows the scale of Twitter, but it's also the follower graph, the network effects, which really shows the scale of Twitter. With that said, and given the uncertainty around the events, which may become viral, the important conversations which happen on Twitter, it is important for us to be always reliable, available for our users. We had to build a platform which is scalable and at the same time, extremely efficient. Our goal was to make sure that our engineers can actually do more with less number of resources. So fast forward to 2016, Twitter infrastructure has grown both in size, scale, complexity. This is platform engineering at its core. We are responsible to build the libraries and the services that make up Twitter as a whole. At the bottom, sorry, at the bottom we have the data center management layer. The layer above that is our core infrastructure services. On one side is the compute service, the other side is the storage service. On the compute side, we use Hadoop for batch compute. We use Aurora and Mesos for basic long running services scheduling. Mesos is used for host management while Aurora is actually used for the task and job scheduling. On the storage side, we have Manhattan, our general purpose, high throughput, low latency key value store. Blob store is another service used to store large blobs of data. Graph store is a service we use to store the social relationships or more like the social graph of a user. At top these, we have an array of platform and application services. This includes our monitoring, cache, messaging systems, the pop sub, our tweet, and even our user service. We also have a data analytics platform which allows users to interactively query on the large sets of data we collect. I also want to share that platform engineering is responsible to build frameworks and libraries which enable engineers to write code. For example, we have built a high, we have built an RPC framework called finagle to do all sorts of RPC calls within Twitter. We also built Scalding to make mapperies really simple when you're writing on Scala. And we also built something called Heron, a real time compute framework which essentially does streaming on demand compute to power our ads and recommendations infrastructure. There's actually a talk happening on Wednesday about Heron. You can actually learn more about how Heron works. It's an open source project now. We do billions of events on a daily basis. And finally, we've also built a lot of management tooling to make it extremely simple for developers to build atop our infrastructure. This means various self-service portals, a service directory which actually tracks all the services running at Twitter. We built Chargeback, which we're going to be talking about more. We've also built a deployment orchestration system to power our CICD pipeline and config management and many other things. All in all, this is really what platform engineering does and that's the group we are part of. And I want to show how many services actually run atop the platform. This is a snapshot taken almost two years ago. I don't have the latest snapshot. But as you can see, we have tens and thousands of services running atop Twitter, Twitter's platform infrastructure. We have hundreds of teams and this particular view you're seeing is actually a tweet being created. So the first request hits our router, the reverse proxy router called CFE, and then our PC calls are made to like over 20 plus services to help create that tweet. So given the scale and size of Twitter, it's kind of important to really understand what is really the overall use of infrastructure platform resources across all of these services. How do you know who's really using what? Given these number of services and the number of teams at Twitter, it's extremely important to understand how we can start capturing the utilization of resources per team, per project, per org. And finally, how do you really incentivize the right behavior for these engineers, the team leads, the managers, to do the right thing and using our resources? To talk more about how we address that, here's Veena. Thank you, Michael. We built a system called Chargeback. It provides the ability to track and measure infrastructure usage on a per-engineering team basis and charge each owner their usage costs accordingly. As Michael mentioned earlier, Twitter's infrastructure is very complex. Multitenancy was a first-class citizen on each of these infrastructure and platform services. Our perspective on the problem is that these infrastructure services lacked a consistent way of identifying customers consuming their resources. They offered a variety of resources that were not defined in a consistent way. There was a need to model resource as a unit of abstraction. Keeping this in mind, as we started designing the system, we identified top four challenges. Number one, service identity. We designed a generic service identification abstraction that provides a canonical way to identify a service across infrastructures. We built a centralized system that integrates with infrastructure services to help create and manage identifiers. Number two, resource catalogue. We work with the infrastructure teams to identify and abstract resources that can be published for developers to consume and build. This system generalizes inventorying these resources. Number three, metering. Each infrastructure cracked the consumption of resources by each service through their service identifiers. As the service consumes resources, the metrics that are written about usage needs to be extracted and consolidated. So we built the classic ETL data pipeline to collect all the usage metrics to aggregate and persist them in a central location. Number four, service metadata. We also built a service metadata system that keeps track of ops and other service related metadata. Overall, these make up a unified cloud platform which we will discuss more towards the end. Now that I talked about the top challenges, let me deep dive about each in detail. Service identity. What is service identity? A canonical way of identifying a service that consumes resources on various platform infrastructures. Let's take a look at the problem in detail with an example of Aurora and Hadoop. We see here both Aurora compute and Hadoop batch compute have very different identifiers. For example, compute has ideas role, environment, and job name, whereas batch compute has an ideas role, pool, and job name. I as a developer wanting to request both Aurora and Hadoop have to go through very different provisioning workflows. For Aurora, I have to use a self-serve system to request resources. On the other hand, to request a Hadoop resource, I would have to follow the old file ticketing can some with the right team. The core problem here is that there is no single system to provide a canonical experience to provision Aurora and Hadoop resource. To complicate matters, we also have multiple ownership tracking systems such as LDAP groups, email, etc. And with the system, one may have a very up-to-date user. On the other hand, it may have a user that is no longer with the company. Overall, there was a need for an identity management system which not only helps provision identifiers but also tracks ownership and is the de facto source of truth. So how did we approach this problem? We first designed an entity model which I'll speak more about soon. We envisioned a single pane of glass for developers to request and manage all of their projects and infrastructure identifiers. So again, if I as a developer want to request Aurora course or Hadoop compute or Manhattan storage or any other infrastructure resources, I would be able to use this system to do so. This results the disparate workflows by hiding the complexity of provisioning identifiers. This is made possible by providing interfaces and APIs that each infrastructure provider can integrate and implement. The end result is a platform that became the source of truth for ownership identification and enabled various other use cases such as service-to-service authentication and authorization. So what happens under the hood for service identity? Identifiers are scoped per infrastructure and they are associated with a service account. The backend that pass the service account is modeled to be pluggable and in our case we use LDAP groups. These service accounts are primitively used for authentication and subsequently authorization. Service is the binary that runs that uses the identity when it runs in an environment like production, devil, or staging, etc. Services roll up to a logical grouping called as projects. Projects belong to a team or department or call center essentially some entity that can be responsible for the dollar spent. Let us take a look at an example. Here we have our revenue which is a call center and ads serving and ads prediction or some of the teams within the revenue call center. AdShard is one of their services that uses the service account adshard again. The last row you see is a client identifier that is composed of the infrastructure and the client ID for that particular service. The team owns several services and we can create multiple service account based on environment for access control needs, etc. The same model can support all the infrastructures including Hadoop, Manhattan, or any other platform service. Now that we have a brief understanding of what service identity is, let us take a look at our next key component resource catalog. Resource catalog defines a consistent way of identifying and inventing of resources of various platform infrastructures. A resource in this context means either a physical or virtual entity that is consumed by applications, thus driving the overall capacity of the physical or platform infrastructure systems. Some of the problems around resource catalog are lack of clarity on what is a resource and how it is consumed. The dynamics of underlying infrastructure and different ways developers use them creates challenges to keep a proper accountability for the consumption of resources. Need to capture resource fluidity. With the utility computing model the resources are no longer a permanent entity both from infrastructure or service consumer perspective. Infrastructure resources evolve over time and developers wanted to consume resources for a specific period of time and be accountable just for that usage. Better support to model abstract resources. As we discussed earlier, things would have been much easier if you were computing cost on bare metal machines. But once we moved into a service-oriented architecture we had to consider breaking down these resources into more granular components like CPU, queries per second, tweets per second, etc. And these resources only went more abstract as we went higher up the stack. The need to define a TCO. Last but not the least we need to define the total cost of ownership of a resource over a unit of time. Here is an Aurora config example to better understand the need to capture resource fluidity. This config file clearly defines the resources that will be consumed as cores, memories and disk by this job Aurora schedules on mesos. In addition to this granular information, this job has a lifetime for itself that is currently not captured in this config but can be derived from the data that exists in mesos and Aurora. We created a model understanding the fluid and transient nature of the offered resources. This allows us to easily extend our current model to capture GPU and network if Aurora decides to offer them in the future. Now that we can create an infrastructure resource, we need to create accountability for the usage of this resource. That is to figure out how to charge these resources out to its owners. This was particularly challenging and we had to build policies and frameworks to ensure that the unit price of a resource is the total cost of ownership. Very similar to public cloud, we had to account for change in unit price over a period of time. This helped define the grain of the measured data. The grain is exclusively determined by few constraints such as source of truth, volume of data, etc. At Twitter, we decided to go with Day as a Grand Larty. Let us come back to the question how to define unit prices. We started defining unit price from the bottom up that is from the bare metal layer where we incurred true cost to buy physical machines. We calculated cost per server per day which includes Kpex and Opex cost of machines, license cost, headroom, inefficiencies and human cost to operate these resources. It is important to keep in mind that TCO evolves over time in account of depreciation, etc. In order to track these changes, again, we designed the entity model to capture these unit prices with a time varying dimension. These form the basis to help calculate other service unit prices as we move up the stack. For example, while defining a unit price for Aurora resource, we calculate the unit cost taking each of these components shown here such as operation overhead, the headroom, what are some of the excess quota and reservation like all the quota that is used scores and unused scores used while computing the unit cost. We designed an entity model that can capture the challenges in resource fluidity and unit prices agnostic of internal and public clouds. Let us review this bottom up. The unit price of a resource is captured in offering measure cost. Again, this is a time varying dimension that can change over period of time. The offering measures is nothing but the resource itself. The offering measure and offering measure cost alone doesn't make much sense without a context. To provide that context, we have offerings, we have infrastructure service and providers. Providers can be either an internal cloud or public cloud like Amazon, Google, etc. Let me show you how the entity model is able to capture both a simple and a complex infrastructure service within Twitter. Here we see, I have Alora for both Twitter or public cloud. This is more common. Alora is our infrastructure service. Computers are offering and core days is one of the offering measures and we have a unit cost for it which is again time varying dimension. We have our batch compute which is Hadoop and they offer multiple offerings such as storage and processing cluster which in turn has GB RAM, file access, etc. as their offering measures. Each of these offering measures have their own unit costs. Now that you have learned our entity model around resource cataloging and how to identify the owner of a resource, let us discuss about the next challenge, metering pipeline. Metering pipeline intends to solve the problem of collecting and normalizing the quota and resource utilization metrics from the available infrastructure and try to unify them using the resource defined in the catalog system. The end result is data that has utilization metrics from variety of infrastructure such as Aurora and Hadoop that is similar to compare and normalize to the dollar value. So let me give you a high overlevel view of how the system works. We built modular services that capture raw data from various services like Aurora Hadoop that uses a HTTP API and writes that data into a raw fact table. From this raw fact table we use a transformer that queries service identity and resource catalog now to identify those owner who's using those resources and identify what are the metrics and their cost against it and compute the overall cost in a resolved fact. We are dealing with cost and data fidelity is highly important to us. So we built a service called data fidelity that will ensure that the tracked cost is true to the cost and also will inform infrastructure owners or service owners if there is a sudden spike or missing data in the pipeline. Once we have this resolved fact we can generate multiple customized reports which can also be used by various data visualization tools. Now to talk more about the product and its impact and the future of ChargePak I hand it over to Michael. All right thank you Vino. So now that you learned how ChargePak behind the scenes actually works I'm going to share what the product really is and what do our customers use. To start with I'm going to walk you through three specific reports and then also give you an overview on the custom reports we have. So this is one of the first views which every team or organization at Twitter gets. This is the generic ChargePak bill which is computed every month and we send it out at the first day of the month. Essentially what you're looking over here is actually our tweet service the team which runs the tweet service is their bill. Unfortunately I cannot show you any of the dollar figures over there so X marks the spot. With that said they use almost six or seven different infrastructure services to power the tweet service. I cannot show all of it over here but at a high level we have Aurora Manhattan the user monitoring system. As you can see each of the offers and measures like Vino said is listed out in the bill. We also show utilization and then subsequently the cost is computed. That said that is true utilization by the way. The other report we have is the infrastructure P&L. Our goal is to incentivize even the infrastructure owners to be as efficient as possible at Twitter. So what we generate over here is an expense and a spend view for infrastructures which generates a margin towards the end. The goal is to have zero margin. We don't want anyone to make a profit or a loss. That said it doesn't matter because people won't be getting that money to do something about it. What's more important is if it's a positive margin we know that the infrastructure has done some optimizations of some efficiency work which has led to more users using it within the same capacity. We can then like reduce the unit price to pass on those savings to all our customers. Similarly if the margin was negative we then know that the infrastructure or the platform service is actually taken on more capacity or is basically inefficient to serve its customers. This gives us some good signals as still what's going on per infrastructure service every month. This is computed all on a daily basis but we do consume it on a monthly basis. Finally the third report we have is the budgeting report. This is something our finance customers really care about. They work with every organization in the company to build a budget and subsequently validate the spend against that budget. This helps keep and check teams like ads and revenue or even platform for that matter to not overspend their budget. That way we also understand what are the big projects being launched and we can specifically incentivize the owners to plan better so we can capture that up front and then plan the relevant capacity behind the scenes. The other custom reports are specifically for different types of customers are the infrastructure and platform owners themselves would like to know who their top customers are. They would like to know how well are their resources being used by their customers. So we can generate reporting at a very detailed drill down to tell the various identities how well they have been allocated and used by every team and every customer. This helps the infrastructure teams to actually have a more more constructive conversation with their customers. Service owners and the teams themselves would like to know the drill down so which projects and which identities as part of those projects are contributing to the larger cost. The finance team really cares about budget management so they use their tool for that purpose. And finally the execs would really like to see trends and the efficiency work happening at Twitter. They would like to know what are the big projects coming on, how well they have been planned, and how well they have been tracking utilization of resources to subsequently have much constructive conversations with the leaders and the organization. So what has been the impact? We launched Chargeback last year and I want to show I want to take a case study offer a compute platform and show the impact from that perspective. So the Twitter compute platform, what you're looking over here, the first chart is actually the allocated quota for all the customers of the Twitter compute platform. This is between June to September 2015. And as you can see, the utilized course has been flat while the quota actually has been increasing significantly. Which this data actually came out of our metering pipelines which made us wonder what's really happening there, who's actually allocating more resources without utilizing those resources. By August, we launched Chargeback with the dollars and this prompted much deeper probe into what's really happening. It was really interesting to see that the teams within Twitter were very proactive in seeking out that details and actually prioritizing efficiency work over other work. And subsequently, once Chargeback was launched, we saw a significant drop in quota but more utilized course. This was the next four months which followed after Chargeback launched. Overall, we saw that the visibility into the Chargeback metrics in terms of both dollars and the utilization itself helped improve our overall core usage against the quota by almost 33%. This is over the last one year and as you can see, the first three months when we launched the metering system, it was still going down but once we started making things visible across the organization, we had a significant improvement in the overall course being utilized against the quota. With that said, the other impact, generally, is Chargeback itself is true to the actual price. All the unit price which we were calculated are prices which encompass the total cost of ownership on the ground. So there is no funny money concept. It is real money so people do understand the value of it. Chargeback numbers were also used to serve for the purposes of capacity planning and budgeting so we could plan better and allocate resources efficiently. The visibility definitely enabled a lot more accountability than previously. People were more engaged to talk about what they need which helps us plan for events and other large project launches. It also helped improve utilization as you saw but interestingly, it also triggered some interesting conversations about comparing with how well are we doing with the public cloud. I'm sure many companies really care about how their internal infrastructure compares to the public cloud from a cost perspective. We were able to do those kind of comparisons with the numbers we had. It was I wouldn't say it was completely apples to apples but it really got us close in helping compare with the public cloud. And finally, the biggest problem of all it helped improve service ownership in the company. Given that we have thousand plus services it is really it is very easy for services just fall off people's graphs and be unknown services. Chargeback with its numbers helped bring invisibility for those unknown services where people were willing to take ownership and then we could figure out a path forward and either say deprecating a service or keeping it going on and things like that. So with that said I want to talk about the future on how Chargeback actually ties to the larger vision of our unified compute cloud platform. So we like to call this platform Kite. It's a cloud agnostic service life cycle manager. Let me quickly explain what service life cycle manager means in the context of Twitter. We want to enable people to write services in a quick easy way get the relevant resources for the services and launch deploy and subsequently subsequently monitor and kill the service as well. So it captures the entire life cycle of service. Many of the tooling which is available outside focuses more on the management of say infrastructure or other things. We felt that there was a need to manage service life cycle as a whole and thus we are building a unified cloud platform at Twitter for every engineer in the company to go to a single place to view the services they own the projects they own as part of their team to do to request resources to request identifiers manage access control. We also let them view their builds and subsequently utilization reports as part of the infrastructure. And then finally facilitate deployments because we have so much of metadata about the service. We can actually give data about the the success rates the latencies and facilitate deployments across zones and things like that. Our goal is to like onboard every infrastructure at Twitter which then automatically provides different aspects of the life cycle manager to the customer without individual infrastructure owners building it by themselves. For example when we did charge back and we onboarded many of those infrastructure services now their self-service tools are becoming redundant because now we have all the information about their infrastructure to generate a self-service provisioning system. So that way each of them can focus more on the reliability and the availability of this service rather than the management tooling side of things. So this is what we're building at Twitter. It's called Kite and with that I think I would like to end my presentation. I would like to thank all the people working on this amazing tool and making Twitter basically more efficient. All right, thanks. We are open for questions if anyone's interested. Yes. You might want to give the mic. Yes, the question was like we showed some reports at a very high level. Do we have tooling for report? Do we have tooling for our engineers which can give details on the low level utilization of resources? And yes, we do. The custom reporting which I didn't have UI to show for we use Tableau. It's one of the worst tools but we use Tableau to actually generate those level of reports. We can look at the granularity from a team to the various projects they own to even the specific roles which run on the likes of Mesa's Aurora or Manhattan and give them a detailed view of utilization of resources and the quota associated with it. So we can start showing them charts as to what was actually allocated and then how much have they've been utilizing over time. We do that on a daily granularity today. Hey, so it is handled at the Aurora level. We have built a lot of account management policies at the Aurora side of things and the information which is exported around quota and even utilization to a large degree is from Aurora itself because it has a view of all the tasks running atop of Mesa's. We haven't yet migrated to using the Mesa's quotas. Yes. Do you want to talk about it? Can I repeat the question please? I'm sorry. Currently a lot of this since we didn't have service identity a lot of this was done manual and we had reports coming from the system but one service identity is in place and we from that point onwards any new services that are created will be identified through the kite or the unified cloud console itself. We also have migration plans where we will be importing all the existing systems into the kite console to do the back identification of the existing services. And to add more context to Vino's answer we do have an internal tool which creates teams and manages teams as well. We call it roster internally the board name pun and we use information from that tool about the team and the user management aspect of things into kite then try to map all the services to those relevant teams and subsequently charge our bills. Okay, I've got a couple of them. Yeah. So there are two portions to that question. So the question is about how did this inspire the engineering teams to be more efficient? In short I think people were so when we started this there was a lot of skepticism as to how this would work. Yes, I believe that's what you're heading to. But over time we realized that as we start making so the dollar figures make sense for execs and more of the leaders of the group but for engineers it's important to show them the utilization data in the form of free sources which they really care about. I'll tell you one funny story. So when we did this people engineers really don't care about the dollar figure that much but when they started seeing how inefficient their services were with towards the allocated resources people proactively took on efficiency projects and tried to improve their systems. For example, I still know there's one engineer who used to work in the ads team went ahead and slashed almost 25% of the observability metrics written to our systems which saved them like more than I would say 200,000 dollars or something per month on TCO costs and the engineer decided to use that in their promo packet too. So that really incentivizes that that story was like one of the other examples on how people look at these numbers. I hope that answers your question. Just to add to that we also have like engineers when they see that there are different resources that kind of offers almost similar things showing these resource catalog and the cost around it they were more responsible enough to like hey if I have this information I wanted to make better architecture choices so that at the end the services that we are building is cost efficient. So it's like the natural responsibility of the engineers that they wanted to build services efficiently. Do you want your services to contribute to an observatory? Right, so I'll try to answer this and if I can please add on. So when we have the concept of onboarding a service into the chargeback system and that includes a series of steps we expect the service owner to define how they manage tenants. We expect the service owner to define what a resource means in the context of their service and we expect them to like integrate with a metering pipeline to send the relevant data to do that computation. We know which team they work for so it is interesting because when we look at the chargeback bill for that team that essentially becomes their expense and as beyond what this infrastructure with all these details and then we compute a holistic chargeback cost we can then tie that as an income. That is one of the reports which you saw which I think I shared over here. Yeah so so each of these infrastructure services belong to a team and we use the likes of Tableau to actually do that computation today. Do you want to add on something to that? Yeah just to recap how do we calculate the expense on the income? Income is what they get out of charging those bills out and expense is usually computed if you think about physical infrastructure as an infrastructure service it gets its majority of income from Aurora mesos because most of them run on the physical infrastructure which becomes an expense for Aurora mesos and Aurora mesos realize that I'm not using so much of resources because there are so many other services that are running atop me then they we onboarded Aurora intern saying that like if you are expensing so much you might probably want to offer your resources so that that becomes income and that's how like if you see in here that's how we cancel out their income and expense and calculate the margin other things we come we compute while calculating their expenses also like the other infrastructure costs the headcount the SRE headcounts and if there are any license costs etc are also computed along with this while calculating that expense so expense is what they are being charged along with the headroom and everything incumbents what they charge out as their resources and we calculate the margin out of it does that answer your question? Oh no it's not budget it's actually the chargeback amount charged to other teams we can then filter by an infrastructure to see what the income of that infrastructure looks like if I can open up Tableau I can probably show that but yeah yeah it's like pretty much unit cost of their total utilization charged out to the customers is their income not right not that complicated like we want to simplify this so we understand what their inventory looks like so we know what Aurora's total capacity is like through their metering information we know how much percentage of that is actually being allocated to customers to use and we have a unit price per resource if you really just multiply that allocation what is the unit price you can get what their income is like I think you had a any more question here I did have an open source is that an internal tool? I specifically cannot comment on that like but we are we are getting more requests on that so we are definitely trying to figure out a pathway towards open source I'm sorry so we started off with a with a team of two the challenge is really not building the system itself the challenge is more on the design and to cater to the needs of Twitter's infrastructure at the same time accounting for the public cloud world as well so many of the entity models you saw was mostly the time spent on designing that the second part where we spent a lot of time was actually the onboarding process because many of these infrastructure services had to like write specific adapters to integrate with kite but that said I think the first two services once we went past the first two or three services which took about three to six months after that we got an onboarding workflow and everything set out which made onboarding less than like four weeks you can onboard a new infrastructure in two to four weeks so right now I think yeah so right now I think the team's grown out of charge back also into building kites so as you saw we have about five resources right now building the entirety of kite this involves now building the other portions that is the the quota management the deploy and everything else per infrastructure as well which is something we haven't tackled yet but that is what we're designing and working towards right now I'm sorry right that's a great question so if we're onboarding the public cloud like why did we decide to build kite or even charge back right that's the question because it's already available as part of the public cloud is that okay so the difference between kite and the public cloud onboarding so kite focuses more on the service life cycle manager which is an abstraction on top of most of the public cloud dashboards which you see you get direct access to resources like VMs and things like that there are high level services but kite will facilitate what a project in your company actually means so if you're building the next big backend system or UI project that becomes a project in kite and that will facilitate the life cycle of it agnostic of whichever cloud you intend to run on so if you're running on Google cloud you can still see kite as a project and the resources it's using and the charge back for kite itself or AWS or whatever that is and it works agnostic of not only the public cloud but also for the internal data centers that's one of the reasons we had to build another layer of abstraction on top of the general public cloud resources hope that answers your question so regarding the number of metrics that we are capturing it is very different from each infrastructure like for an infrastructure like Aurora might have few like less than 10 metrics if you think about CPU or GPUs or core days memory etc similarly whereas an infrastructure like physical infrastructure or Manhattan can give offerings as it can still have like a same metric which is like a server per day but it can have many offerings which are clusters of machines or machine names so the metrics can vary from few like in the tens to hundreds yeah I think it's compounded by the fact that you start multiplying offers and offer measurements today I think that's where we have more than 200 different metrics across nine infrastructures like I would say in fact not even 200 sorry I think it's more than that 500 plus yeah we have 500 plus metrics again for around that nine infrastructures no we don't actually we store all of this in Vertica today for better analysis yeah so I think the key or what these folks have built is the entity model on the schema on how that actually works I believe we have more than 100 million rows at this point 100 million rows in the chargeback system because we do compute or collect meter data on almost a daily basis and then compute cost and everything based on that per offer measure per offering and so but I would say it's pretty fast cool all right thanks a lot everyone if you have any questions we are going to be available so yeah


Bill Clinton was only 46 years old when he became president. Clinton promised to strengthen the American economy. Sure enough, his efforts helped create eight years of economic growth and well-being. He also succeeded in ending wars in the former Yugoslavia. But Clinton is also remembered for his failures. In his first term, he could not persuade Congress to reform the country's health care system. And in his second term, the House of Representatives voted to impeach him. He was charged with lying about his relationship with a young woman in the White House. The Senate did not rule against him, however, and Clinton finished his term with high approval ratings.


If you went for an evening walk in the bush around Canberra 100 years ago, what would you have seen? The woodlands would have been teeming with wildlife, small animals such as bandicoots, quolls, and bettongs. But since European settlement, the woodlands have been cleared, overgrazed, and overrun by weeds and feral animals, and many of our native species have disappeared. In fact, Australia has the highest mammal extinction rate in the world, and many of these species were lost before we really understood the important roles they play in our ecosystem. So what would happen if we could bring them back? This is the eastern bettong, a rapid-sized relative of the kangaroo. It was once so common in eastern Australia that it was considered a pest, but they disappeared from the mainland around 100 years ago. Luckily, the eastern bettong still survives in Tasmania, and six years ago, we were able to bring a small population and release them into a fenced reserve called Mulligan's Flat Woodland Sanctuary.  Well, since the reintroduction, our bettongs have been getting busy, and not just in the way you might think. They've been digging away every night in search of truffles, roots, and bulbs. I've estimated that each individual bettong can dig up to 200 pits every night. That adds up to three tons of soil turned over every year. As a plant ecologist, I wanted to know how this digging would affect the soils and the vegetation. By measuring and monitoring soil nutrients, temperature and moisture in the diggings, and counting and identifying thousands of tiny seedlings, my research has shown that bettong diggings act like little flower pots, providing the perfect conditions for seeds to germinate. But it's not quite that simple. Reintroducing a species after 100 years can have some unexpected consequences, and the disturbance created by the bettongs could just lead to increased weed invasion. But my research has shown that native plant species actually benefit more from the digging than the weeds. Digging animals like the bettong alter the structure of the ecosystem and provide habitat and resources for other species. And for this reason, they are now known as ecosystem engineers. If we can bring back the bettong, then we could not only save this one species, but also help to restore an entire ecosystem, so that maybe 100 years from now, you can walk through a more diverse and resilient woodland that grew in soils turned over by the bettong. Thank you.


Crew chiefs normally come out here as a team and wash in aircraft periodically. Well today we decided to invite other members of the wing so our maintenance group superintendent Chief Taylor sent an email out last week to see any volunteers so we had four responses two from OSS, LRS and one from FSS so we have great airmen out here working touching the aircraft getting to see a side of the operation and the mission that they normally don't get to see. So great for camaraderie and great wingmen getting together to solve the task. I normally work in FSS over in the Deer's office but today I got to come out and help clean off the aircraft. I thought it was a lot of fun when my supervisor asked for volunteers I raised my hand right away because I thought it would be fun to be up and close with an aircraft my first time and also I thought it was a great opportunity to kind of mingle and meet other people that are on base.


Hi everybody. So this is the model that we're going to use in the first part of the lab And I want you to really familiarize yourself with the controls before you really get started Okay, I think it's really important And it can be a lot of fun to experiment with this and really get used to to playing with it before you start The lab and I think if you do that then the lab will be very straightforward So basically what we have here is a carbon cycle very complicated This is the stellar version of the carbon cycle here and you can basically see all the all the knobs and bells and whistles with these stellar connections And over on this side is the ocean model that we already used in module three lab So you really don't need to understand this of course, but it it shows you how complicated it is and we'll see later on why it's such such a good model Okay, so the the bottom line is we have three different emission scenarios that are controlled down here by these switches Okay, we have a business as usual switch Which is basically the worst case scenario scenario emission scenario a2 and you can see that is here It goes all the way up and actually then it drops off But still it goes all the way up through 2100 we continue fossil fuel burning through 2100 without stopping Okay, that's this switch So that's with this on and you can read about this down here So if I turn this switch off and I leave this switch off and I'll explain this switch in a minute Then what we do is we level off Fossil fuel burning around 2010 we just level it off Okay, which obviously didn't happen because we've gone through 2010 But it just shows you what happens if we slow down fossil fuel burning don't completely stop it Okay, and then the third switch is what we what we do is if we all of a sudden drastically stop fossil fuel burning FFP holds and what that shows is around 20 2010 we stop see if I can move this up But anyway, it's down here. You you can see we stop fossil fuel burning at 2010 it goes down to zero Okay, so that's basically what you what happens with that third switch So the three switches are three scenarios are business as usual with this switch on this switch off Both switches off is leveling off at 2010 and Then with this switch on we halt Fossil fuel burning at 2010. Okay, don't worry about the land use switch for this part of the lab All right now the critical thing is we can run. Let's run business as usual I want to show you one other thing You run this model here and let's say you want to then run another model Let's say we'll turn it off and run the second one You can see these two curves side by side so you can see business as usual and then you can see the leveling off of Fossil fuel burning and then if you want to run the third one But excuse me. Sorry about that. We're just going to turn this on and then you see you see the third switch The third scenario here three So one Business as usual to leveling off three fossil fuel burning But what I wanted to show you also is that if you want to then start over again and do something different You restore everything. Okay in general It's a good idea between experiments to restore everything to get everything back to back to usual back to normal And then one other one other tip that I just thought about is if for some reason the model isn't working for you And you aren't using Google Google Chrome switch to Chrome because it's the most stable It's the most stable browser for style models All right, so let me show you the output that we can get so let's have business as usual This is global temperature change and as I said the emissions the fossil fuel burning stops at 2150 or so And so you'll see this temperature decrease, which isn't really important for the lab CO2 you can see it rises steadily here and then drops here pH Goes down and then rises a little bit but you can see it's going down This is ocean pH and then one other thing I wanted to show you is make sure you can read the values here So I'm gonna click on the curve and you can see it running along the curve It shows you the values down the bottom run one 8.14 987 and then moving down the curve you can see the values you can see both the year and the value so now I'm 2040 and I'm now at 2083 my value is seven point eight four four and now I'm down at 2040 Excuse me 2153 at the bottom and my value is seven point five five Okay, so make sure that you can run the the cursor Along the curve and read the values not only the year, but the values of pH temperatures CO2 Etc all of these different curves you can do this which is really really helpful and very important for the lab All right, so then we also have How much of this CO2 is absorbed by the air how much is absorbed by the ocean? How much is absorbed by the biosphere in the the purple curve number three? Moving along page five shows you fossil fuel burning which shuts off as I said earlier by 2050 or so This is both CO2 and fossil fuel burning fossil fuel burning in red CO2 atmospheric CO2 total in Blue Comparing those two curves this here shows you what happens in the biosphere. It's the atmosphere in one The land biota in two how much CO2 is absorbed by land how much is absorbed by soil? How much is absorbed by permafrost and? Then this here shows how much is absorbed by surface ocean and deep ocean This here shows you the Gigatons of carbon from fossil fuel burning And then finally comparison of the modeling observations. So this shows you in in blue the Model run. This is the model run that we have established with this carbon cycle and in red We show the observed Atmospheric CO2 this is measured in the atmosphere through 2010 and you can see how close they are together It shows that this model works really well All right, so that's a good summary of what this model does This is for the first part of the lab step one and I will be producing another video for step two