So the Nintendo switch 6.0.0 update has been data mined and you might be thinking how is that possible? You just talked about how hackers figured out that it was even coming Well, it turns out that Nintendo has already loaded the 6.0.0 update onto their servers and hackers have naturally Downloaded it off of their servers gotten it to run on their hack switches And thus have become to data mine the update to see what new features might be coming to switch Or we'll be coming to switch with this update and yes folks It this update is the one that is required for the Nintendo switch online service. This is not a big surprise It's what everyone thought the subject was going to be and as that Being such an important update you would figure that there is a lot of important stuff in this update and Well, you're gonna be a little disappointed.  This includes the Nintendo switch online subscription support Which does apparently have a slogan for some reason and the slogan for this Nintendo switch online service is more games more features more fun But the only feature really discovered is Just the cloud backup data what you're seeing a screenshot of like you have an ability not to back up your games to the cloud That that's a feature. We already knew was coming. We still don't know fully how it works It's not active right now, but it is in the update. No big surprise there. Obviously more games We're talking to those NES games. What was interesting as they did uncover some new controller icons You know when you go to set your controllers and you see little icons for represented a joy con or a pro controller Or a wider USB controller Well now there are icons for Different NES controllers the old-school Famicom controller from Japan And there is what appears to be like a Super Nintendo controller potentially It's more of a classic controller because a Super Nintendo controller isn't shaped exactly like that. That's more I don't know like it feels a lot more like the classic that like the you know, the classic pro controller or whatever From the Wii U days Maybe even like an 8-bit dough style controller is actually what it looks more like to me But it could be a SNES controller. Whatever whatever the case may be They're obviously adding support for a controller like that and obviously some NES style controllers They're probably going to release that are Bluetooth and I think that's some a really neat thing especially for those classic games We can look at whether or not that's a SNES controller or The classic controller support or whether or not it's it's an 8-bit dough or whatever it is It could potentially hint that that Super Nintendo games are coming There's nothing in the code base so far the data miners have uncovered that seems to suggest that there is going to be anything else like that But that's basically it the only other thing this update has at least on the surface are some new icons For your profile. There are two of Captain Toad two of toadette one of Wingo and one of Dragadon All of these characters are from Captain toad, you know treasure trackers. So yeah, obviously you're adding more icons for games That are already out of great Would have been nice to see some autofat traveler icons if I'm being honest, but oh, you know the icons don't really matter that much to me I've got mine. I'm perfectly happy with the icon. I'm using right now Everything else that seems to be there is just stuff to deter Hackers from the current ways they are accessing the system You know, there's mitigations now included in this update against the deja vu method Which is an unreleased switch exploit There's also enhanced telemetry to detect exploit utilities among other things There really isn't much else other than that beyond, you know evidence improved that yes This actually exists people have actually downloaded it. This is from Nintendo blah, blah, blah And this seems to be what developers have been working with now for a little bit in preparation for this Suppose a massive update which essentially just turns the on switch on and the time to switch online Subscription and adds in the save backups now They have not found anything else in this update. There's nothing else There's no themes. There's no folders. There's no custom organization. You're not going to see things, you know If they're adding Netflix support, that's gonna be through the e-shop That's not something you would even really see in an OS update. That's an application. There's no messaging. There's no local backups Pretty much anything you could ever want Nintendo to actually do with a paid subscription service is not here It's just what they told us and this could be why we haven't really heard anything now We talked about several times. Maybe even some are complaining That we don't know enough about the Nintendo switch online service and it could just be that there really isn't anything to talk about The only thing arguably to talk about and show off are just how the NES games work, right? Like yeah, how to swap play work, you know, let's get a video showing that off Let's get a video showing off multiplayer and online leaderboards But that might be it and they might not think that's enough to really, you know, have a direct about it to that to me This sounds like or at least what it appears like on the surface and again This doesn't include any updates to the Nintendo switch online app on your phone Which could have messaging which could have lobbies and could have heck maybe achievements or whatever like all that could be handled To the Nintendo switch online app if they wanted to I think it's a stupid move I think there's a lot of people that just don't even want to load up that app when they're playing online But whatever it's there and maybe Nintendo is gonna push that but at least on the switch itself It appears that you're just getting exactly what Nintendo said you're gonna get and nothing more and nothing special You get cloud save backups and you get to play games online like you're already doing right now But it's behind a paywall. There's nothing extra And you get these NES games of extra controller support So if you're wondering why we aren't hearing anything else about Nintendo switch online service, there isn't anything else to hear about It's kind of disappointing to be honest now I have to note this is data mining It's potential that there is another Update to the 6.0 update. That's not on Nintendo servers yet After all this update is not fully public yet and it's subject to change It's possible that You know that nothing that there's something someone's missing in the data mining, you know It's been up for 13 plus hours at this point. There's a whole thread of reddit about it It doesn't appear that anything else has really been discovered of use to anyone in terms of information There's obviously, you know new new new security measures that they're finding all the time to try to deal with hackers But they're not really finding anything at all It's what it sounds like is if you were looking forward to paying to access Playing online games, then I guess You can look forward to having the exact same service you already currently have for free It's a little disappointing if I'm being completely honest. I want more I expect more We're approaching, you know getting close to you know year and a half You know thinking about the two-year mark on this platform coming up still can't organize our games I don't really care about the custom theme even organizing games to me is a huge one that just needs to be on the system It's fine if they don't believe in achievements and they don't really want to do anything beyond my Nintendo But how about some my Nintendo integration? I don't there's a zillion things I could think of that would be really nice to include on a major update like this for the Online servers instead folks. We're paying 20 bucks a year to or 35 with friends will about per year For access to games we can already play online for free with no additional extra services or guaranteed dedicated servers I think some people were looking forward to that. Hey, maybe we get dedicated servers No, everything is still peer-to-peer and we're paying for access to our own internet to do peer-to-peer multiplayer games not good they're you're getting those NES games and Cloud backups Bingo bingo bongo. It's weird that they even have like that. They're calling them to switch online service more games more features More fun the only new feature the only new feature is the cloud save backups They're not adding anything else, you know more fun You're not having more fun by putting up paywall between someone who buys platoon 2 and someone who who bought platoon 2 Now you have a paywall in between them playing against each other. That's not more fun more games I I'll give that to them fine slogan. It's just it feels incomplete if nothing else. I hope this isn't everything. I Am not impressed with this at all. I'm not gonna throw a lot of shade at Nintendo just yet because again, this isn't fully public It's not technically out there for people to download besides, you know, people who have hacked into the servers so again You take this all with a grain of salt in terms of what features may be added or aren't being added but This doesn't look good at all and I got to say Yeah, Nintendo's only charging 20 bucks. I'm not so sure the service is really worth 20 bucks But you know me I'm gonna end up having it so I can have those cloud backups And so I can play games online with you guys because Nintendo's now putting a paywall between my data and what you know Being able to back up my data and being able to play with you guys even though they're not providing any Networking or any server-side stuff to make my gameplay better I'm Nathaniel of a dance from Nintendo Prime Let me know what you think about this down in the comments below as always folks You could show a little love our way if you would like at patreon.com slash Nintendo Prime for as little as one dollar per month You can throw a little love our way and help support what we do. Otherwise folks I'm just gonna catch you in the next one


I talk fast. I'm gonna try to slow down a little bit I might say some funny things, but You know, that's that is what it is. So let's get started So, yeah, the topic of this talk is a year with Apache Aurora I put a little asterisk there because it's actually been almost two years at this point since I submitted this started submitting the talk and Maybe 18 months. We've actively been using Aurora so There we go. Okay. So here's what we're gonna talk about today First who is chart beat because as we've established no one has ever heard of us What our architecture looks like? How or why we adopted Mesa's and and Aurora How we active actually use Aurora and then we're gonna take a deeper look into some of the interesting features that we found So about chart beat we're 75 employees We're eight years old venture capital-backed startup in New York City We have somewhere north of 20 engineers that includes front-end engineers Data scientists and a bunch of back-end engineers and my team which is five of us and we're the platform and DevOps team We call ourselves the platypus team And that five includes myself and also our CTO who unfortunately is usually in meetings So but he's a really good programmer. So we like it when we get him We're in New York City. We're if you know, New York, we're just south of Union Square above the strand bookstore It's an awesome location And we're entirely hosted on AWS and Every engineer at our organization pushes code all the time So who what do we do? So this is the marketing slide, you know, they gave me a couple I had to put in here And our mission statement is we are the content intelligence platform that empowers storytellers audience builders and analysts to drive the stories that change the world and Our customers are the press. So we work for most of the large news news agencies online magazines big blogs everyone from the New York Times Washington Post BBC a lot of The press in Europe and we get a lot of traffic So basically the short story is we put a little JavaScript on everyone's page and that pings us all the time And we can measure how far our user went how long they spent reading a page and all that and it turns out to be about 300,000 requests a second coming in just with those JavaScript pings We are on 50,000 over 50,000 websites around the world and we track about 50 billion site page visits a month so Okay, this is what it looks like. So we have. Oh, I just went too far trying to get the Laser pointer here. Okay, so I forgot laser pointer. So we have dashboards Real-time historic and video dashboards and when our customers log in they get a pretty view in real time of how people are using their page so you could this is a Pri public radio international They're one of the customers that lets us use their their data and they currently have 494 people on their I believe this is probably their home page I did this early in the morning. So, you know, the traffic picks up And the average time someone spends on this page is 45 seconds You can drill into all the different pages You can pivot on author and section and all that cool stuff And we also do optimization. So we have something we call the heads-up display when you go on to your website And this is nrc.nl. You can see these these bars that show What how each of the click-throughs is behaving on every page on your site? So they use this to decide. Hey, you know that story down there About Obama is doing really well. Let's keep it on the home page This the one next to it something about Quebec. Sorry, don't we Dutch isn't doing that well They might want to push that down or do something else with it or move it up If they want to put it in a better position So maybe they want to swap that and get more traffic going to that story if they want So I think it's really important in any talk like this to understand what the or how the organization approaches software engineering I don't think we're quite unique, but we definitely have our own way of doing things So this is my favorite quote. I try to use it anytime.  It's really old It's one of the quotes that inspired me when I was like 22 years old right out of college engineering And it's by Larry wall who wrote pearl and he said in the first edition of the programming pearl book We will encourage you to develop the three great virtues of a programmer laziness impatience and hubris and He got some pushback on this and so in follow-up editions They expanded on what they really meant and they don't mean I should be lazy They mean that I should be actively writing code to allow myself to be lazy, right? I shouldn't be lazy I should be lazy because the computer is not programming fast enough for me It's not give me back my response So I'm gonna write code to make it do things faster and if you don't have a lot of hubris You can't be an engineer because you have to believe that you can do anything and make that computer do what you want So how does this translate into chart beats? Engineering standards so the platform team wrote a mission statement because they told us we had to move some sort of okay our KPI thing and we came up with this that our mission is to build an effective Efficient platform and secure a development platform for chart beat engineers because we believe that an efficient and effective development platform leads to fast execution So How do we operate in a day-to-day basis and like I said, I don't think this is unique It's definitely unique compared to Engineering 25 years ago when I started When we had CVS So git is the source of truth for everything, right? We store our configuration for our entire Amazon infrastructure in git so we can always reproduce it any time we want Everything that gets deployed is deployed by a git hash except for Java stuff. We use semantic versioning for that We might be able to fix that later, but Engineers run the code on their laptop they run it in a dev environment.  We're small We make templates and we write scripts to automate things as far as programming. We are almost exclusively Python enclosure With exception of JavaScript for the front end, but I don't understand that I don't use it So why maces? Why now we're eight years old. We're by you know, everyone seems to think we're very successful company in our industry So why would we make such a big switch? So I feel like the freedom to innovate is the result of a successful product, right? We want to set ourselves up for the next five years You know chart beats gotten to this great point We have a lot of customers, you know where we have good revenue. Where do we want to be five years from now? So that's what this project was about We wanted to reduce our server footprint to save money as a lot of folks have mentioned here We want to provide faster more reliable service to our customers We wanted to migrate all of our jobs in one year to whatever System we decided to use And while we're doing that we want to pay off tech debt, right?  Should we spend a couple days tighten it up doing whatever and you got stuff running that? Hey, oh my god, that's still running. It's been up for five years and no one knew so let's not move that Most importantly we wanted to make life better for our engineers right we wanted happy engineers so come to today we have a Moderate-sized cluster compared to what I've heard around here We have three hundred thirteen hundred and fifty cores in our in our cluster and almost everything that we run is now in mesos Okay, so what's a happy engineer?  I just got woken up. This sucks, right? So they don't want to have to do anything when they're on call They want to push things quickly, you know, they don't want to have to jump through hoops to get their code into production They want to be able to monitor and debug their applications easily. They want to be able to scale their applications Really they want to when I I should say when I'm talking about engineers now I'm talking about our product engineers the guys who we work for they want to write product code, right? They want to write JavaScript. They want to write Python API. So they don't want to mess with dev ops So they want self-service dev ops. It's easy to use and that's what we would set out to build So what do we do first? So before mesos we had a lot of puppet, right? We had higher roles and puppet that mapped to Amazon AWS tags for the instances We built virtually envies into devians for our Python code so that we could Capture all the dependencies mostly we had single-purpose servers We use fabric to go in and restart jobs and stuff like that. It is flexible. I mean puppets great It's very flexible, but it's really complicated, right? You got Ruby and you've got hyra and it's it's it's really complicated. So say you have this project foo, right? foo had an API service a Kafka consumer some cron job workers that go and do some database roll-ups and stuff and you You basically build out foo API 01 02 03 04 05 foo Kafka consumer 1 2 3 4 5 6 7 8 9 10 16 right and all these workers.   We found ourselves with 773 EC2 instances You know for a company with 25 people that's that 25 engineers, that's a lot During the US election last year we broke a thousand instances and we were like that's a lot We had 125 different roles in puppet It was really hard on DevOps is confusing for the product engineers We wasted resources it was really hard to scale So we started looking at it and we said, you know We're using something like 40 50% of our CPU and RAM and that's just not really cool. So We decided that whatever we built had to allow us to solve the Python dependency management solution problem for once and for all It had to play nicely with our current workflow. We didn't want to tell all of our engineers Oh, now you have to do it all this way right because they're used to doing things their way So it had to be hackable so that we could kind of customize it tweak it It had to be open source. We only use open source software with the exception of some Amazon databases It had to be supported by an active community. That's actually using this stuff in the real world It had to allow us to slow do this migration over time right and it had to make our engineers happy so We chose Aurora I'm not going to get into why we chose Aurora Happy to have a beer later if you want to talk about that compared to and I've also by the way Written a couple blog posts about this where I get into some of the details of why we chose Aurora versus marathon So what is Aurora if you haven't used Aurora? It's a mesos framework for long-running processes and cron jobs It was built by Twitter. I was based on Borg. They had an engineer who had previously worked on Borg They they launched it at Twitter in 2010 and it joined the Apache incubator in 2013 I'm not quite sure when it became a top-level project, but I'm sure someone here knows They're currently on release 018 and about every six months they roll a new release the new releases always support the latest mesos version Got a very active user community and it's written in Java and Python So basically this is Aurora You know over on the left side. You have a Framework that's registered in mesos. There's an agent that runs on each of your servers that receives the the instructions to go ahead and launch a new job Everything in Aurora runs inside a sandbox container, right? So you get a directory. It's a truth And in there all your stuff goes and your whatever user you launch a job as has permissions for that truth They have an observer which basically lets you through a UI come in and look at your jobs You can look at the log files and that sort of stuff And they have an executor that monitors the life of the job So they define things as jobs so a job might be an API server and I say I want 42 of them, right? So yeah, that's 42 tasks and mesos goes ahead and schedules them inside thermos is the the the executor inside thermos you get Processes so you can run multiple things in in parallel in pipelines Something can install a job then run a job health checkers that sort of things that those are all processes So some of the features of Aurora that we've found useful so all the job templating is in Python Which means you can do anything right and everyone loves Python these days One problem that we had with with puppet instances is when something died it where it got wedged Which is a very technical term that we like to use someone had to actually log it in the machine We started figure out what's going on now if if a process gets wedged a health checker in Aurora We'll say hey that that thing's not doing whatever you said the health check was like it's I don't see anything in the log file I can't hit this port it kills it reschedules it no one cares that no one even knows I mean we obviously do know but no one has to do anything It's it has a very hackable CLI which I'm going to get into It does service discovery using finagle through zookeeper You can map ports obviously all around it has a good API and something that I actually found really cool that I Initially thought was kind of weird was the way they name jobs by cluster Organization environment and then the job name which helps in zookeeper and in all of our metrics for knowing what's what so This is how What would they call a dot Aurora file? Looks that this is where you define your job description in Aurora So every job in in basic Aurora you have an aurora file which can define multiple jobs their Python as I said and The processes are basically any kind of unique thing that you can do so in this case We define the path to some this is the hello world from the aurora website by the way I took a couple things out because they were esoteric So you define the name of the script you want to run you have a process which is you're going to call installer You're going to name it fetch package and it's going to go and copy the thing from this directory slash vagrant into your local sandbox It's going to print something saying hey, I did this and then it's going to make it executable and run it I'm sorry make it executable the next process runs it right so I had one that installed it now This one's going to run it so because the head and just calls Python It's just you know obviously does a Unix command line statement So then I go ahead and I link those two things together So I'm going to do an install and then I'm going to run hello world and I need one CPU Mega RAM and eight megs of disk to do so and I call that hello world task So then I go ahead and define my top-level job And I say it's going to run on the dev cluster. It's going to run as the user develop I'm sorry in the environment to vell. There's develop prod and test environments. I'm going to run it using the double data user called hello world and Go ahead and it runs it so You know, I find that DevOps is a balance between flexibility and Reliability right you want to let your your users do what they want to do But you need to be safe right you want to protect them from doing silly things and at the same time make your job as a DevOps Engineer easier because if you know how things are running if you have control over it You can manage it if you let your users kind of do whatever they want That's great until it becomes a nightmare for you So we wanted to make this much more kind of More structured and the reason is because when we looked at the work that we do Almost everything we do falls into three categories, right? We have Kafka consumers that read off a Kafka right to a database or to another Kafka topic We have workers that listen to a rabbit queue and get work And then they go and most of them do something like take an hour's worth of data from this database and turn it Into a five-minute roll-up or copy some stuff from here to there and then we have APIs, right? So we have closure consumers Kafka consumers. We have Python workers and we have Python API servers That's pretty much like 95% of the stuff we do So big decision time, what do we want to do?  I'll talk about pants in a second We're gonna wrap this Aurora command line interface with our own client that we can add some more kind of control around And I'll talk about that in a sec I'm going to create a library of Aurora templates that make it easy to do repetitive things like pull An artifact from s3 drop it in here make it executable and run it We're gonna just let Aurora do its thing we've always had issues with log file management It's like one of these old the oldest problems in the book is you know Oh bio handles. Damn it. You know like discs filling up and so we said, you know Aurora Let has a disc quota. So when a job is hits that quota it kills the job It restarts it and then it goes through periodically and cleans out all the sandboxes. Who is let let it do that And we're gonna Not even bother was with containers. So at the point we did made this decision that there was no Docker support in Aurora And we don't use Docker. We use it a little bit but not very much So we didn't really care about that. We said we're just gonna go with these sandboxes. That sounds fine us So how do we make a roar fit into our workflow? so The dot Aurora file is very powerful. You can really do anything You can define a whole bunch of jobs and go ahead and run them But that gets very confusing because you as a DevOps engineer you no longer know what's running everywhere So we decided we're gonna take all the common config options out of the Aurora file and put them in a YAML file Things like how many CPUs do you need? How much RAM do you need that sort of stuff? Flags that that you might want to pass a command line and then the Aurora files become much more simple We decided that we're gonna require, you know version artifacts built by our server, which is what we did before But we actually tie that into the client So if you you have to specify the git hash in our YAML file for what you want to deploy and our client actually checks to make sure That thing exists before you try to launch it So we put in some safety nets for our product engineers You also have to be unmasked or if you want to push something to the production the production environment because People do silly things especially at two in the morning when they get woken up So we also decided that Every YAML file specifies one job that job could be running devils task prod whatever but One multiple YAML files can actually point to the same Aurora file Which has the definitions of how the job runs and I'll talk about that in a sec. It gets very interesting All of our configs as I mentioned already live in the repo which makes it really easy to find jobs So we have one directory where all of the job configurations are you can go and grep and you know Replace stuff and it's very easy to make major changes or to figure out what's running where and Then we also added some additional functionality for things like tally log files as they're running so What's the difference so on the top? This is what it takes to run foo server dot aurora on the top is the basic aurora command line to do it They say aurora create. There's also aurora Update which will do a rolling up update. You can do a restart. You can do a kill And then you name the job so remember I said these they have these funny names those a is a cluster name We have a and bb because one of our guys used to work at Google and apparently that's what they do a and then cb ops is the name of the user and then it's running in production called foo server and a path to the Aurora file so the way we do it we have this Aurora manage command It mimics all of the commands, but it Maps to that yaml file and it pulls out all the stuff that you're not going to remember Like the name of the user that it's running under because if someone launches this job with a different user that could cause havoc, right? and We have to specify whether we want it to run endeavor prod because Engineers launch things they mistype it and they launch things in the wrong place So it's about a safety net So this yaml file that we've designed at the top has information about the job So this is for a thing called eight ball that does one of our one of our dials on our dashboard It runs as so you specify the the the the file is the Aurora file that's going to run the user It's running as CBE The build name refers to an artifact that's going to be found in s3 and in this case It's called eight ball We're launching this by get so we also allow versions for Java things We have a static type for things like third-party things like Rafauna that you know We just kind of build once and deploy and then all of your configurations So these are the things that most people are going to want to change frequently and Then we allow you to specify things that you're going to use in a in a way that we've defined in your Aurora file like arguments command line arguments to your whatever your thing that you're running is And we allow you to override this for different stages It's very common, you know in in dev I want two of them and in prod I want 56 of them, you know in dev I want these command line arguments and prod I want these command line arguments So we specify so we've broken it out so you can override them and you always have to specify a get hash So pants pants is Here's my one slide on what pants is we discovered pants because they use it to build Aurora It's also from Twitter You can find it on pants bill.io. So it's a build system for big mono repos Especially Python ones they do support other things, but as far as I know it I've never seen anyone use anything but Python and it's awesome So if you are familiar with maven, it's essentially maven for Python And it creates pex files pex files are executable Python directories So basically what it does is it takes all of your dependencies So you know you need py amul and you want requests 2.3 and all this stuff Which is a problem when you're trying to deploy multiple things on one server, right? Because it's all at the top level So instead of a virtual environment it basically builds a directory with all of your dependencies all of your code any Extra resources like yaml files or other config files, and it puts them on a zip file and it makes it executable So you now have one artifact with all of your Python stuff in one place And we tag that tag them with a get hash We name it for whether it's built up for trusty or precise or whatever and we upload it to s3 from our from our Jenkins So it has directory level build files, which is kind of It's a lot of files, but it's actually very very flexible And it the reason for this is it does incremental builds for mono repo So if you make a change in some Python code here, you don't want to kick off a build of everything It can figure out what else in your repo depends on that file change and just build that right So We have no more repo level dependency conflicts you can even specify different versions of third-party stuff It was this was a big migration So we decided early everything's going to Aurora and by the way it has to get pants before it goes to Aurora You know we obviously helped our product engineers to do this and it was a great way of getting rid of some tech debt So what's pants look like this is a pants build file for the fiddler server That's another one of that's one of our API servers so you basically specify the entry point into your code and Dependencies which are either relative to your code or there are somewhere else in your repo or the third party So this will include pyamol. It don't include a handlers directory Some of our in-house login utils and memcache utils Sharknado 3 which is our API server because people love naming things And the sharknado 3 g event plug-in which we use and there's also some constants yaml file that needs to get included in there All this gets put in one directory Tartar zipped up and made executable It's like magic and this will build fiddler server dash githash dash trusty and x86 64 dot pecs So the next thing we did was write a bunch of templates for doing common things all right So we wrote since the raw file is just Python We wrote a whole bunch of templates to automate the common things people wanted to do like installers So we have installers to install jars Tars pecs files G zips whatever that are found in various s3 Directories pull them in and drop them right no one needs to know how to do that. It's it's like someone did it once Jay JVM and JMX configuration options all sorts of environment stuff If you have to create a config file based on some inputs and drop that into your your troop before your thing runs We support that very easy Access credentials we install all these with puppet on the machines, but they're all you know, they're they're They're hidden and stuff like that. And so we have ways to get to that. We also have shared resources, you know Here's here's the list of all the databases. Here's all the Kafka brokers Here's all the the zookeepers and all that sort of stuff and then Supporting actors in this in this world So we have this thing called off proxy that every API has to run that Authenticates users against the database for for different apis Someone wrote that once we just have a one-liner to drop it in and use it Aurora uses these all these health checkers. They have an HTTP health checker and we wrote ones that tell log files Proxy another service to see if it's up and someone just says I want to use that health check no problem all right So this is what one of our aurora files looks like for eight ball so Aurora makes heavy use of this thing called pystasio and that's it's it's a Type-safe dicked in Python It's almost like a struct and it allows mustache templates to replace things either at At the time when you define the job or once it's been assigned to a server, right? So something like the port the HTTP port you don't know until it's been assigned to a server something like the name of my My my sequel database. I know upfront when I want to run it So it allows for different evaluation times so things can be added in so aurora Exposes pystasio templates for the port mappings and that sort of thing then the instance ID the name of the server and we wrote our own for doing things that we need to do So it works by adding all these these these profiles So we have we define a bunch of profiles up top that we're going to bind later on You can see in this ops thing someone's defining command-line arguments for the API server that's going to run So here's our memcache servers. They're in the services struct The port is is is defined by thermos executor And it allows you to just if you give any string into this this port stick It'll assign a port with that name and then you can use it later.  So it's just a one-liner. I want off proxy. I want this health check That is going to check my API server at this URL With this port that is going to be assigned later And then I want to run these first two things I need to install eight ball and run eight ball But then don't stop let that keep going and then go ahead and install this off proxy in this health check process So we've kind of defined these pipelines that you can use to define how your things are going to run and then finally you define the job and You go ahead and you bind all these profiles and Aurora runs it So we've taken this kind of this idea of our custom templates a step further we have 104 Workers that are doing basically the same thing they listen to rabbit queue and they run some stuff and it's actually You know we have this thing called Igor Which is our worker framework and the difference between One worker job and another worker job is what are they listening to and some imports, right? Some python imports because you don't want to import the giant data science library for something that's not using it So we decided to this take remember I said that you can have multiple yaml job definitions pointing to the same aurora config So we said alright Let's have one aurora config for all of our workers and then just to find yaml for all the different ones So now if someone wants to add a worker They just have to write a little bit of yaml and say use this you know This is the the rabbit queue you want to listen to and this is some command line arguments So right now in our aurora config Directory where we store everything and we have 104 of these defined and you can see some of the different different ones so Elastic search indexing is a big one So then we took it a step further and we created an ETL pipeline called deep water Because someone who may or may not be sitting in the front row wanted to name something after an oil spill But deep water actually Let's you define a whole workflow that runs in aurora jobs So you can define these your steps as Python classes and then each step in the pipeline gets its own aurora job And you can give different requirements different resource requirements of different steps in the job You can scale them out However, you want but it also uses postgres for consistency. So if a job failed, it's marked postgres and we know so That's only part of the story. So before Deploying anything we had to figure out the right, you know, we want to use aurora. Okay Now, how do we solve every other problem with this migration? Right? So here's some things that we had to deal with so request routing metrics and monitoring log file collection configuration management and Bunch of other stuff, right? So how did we do this so the first one is Routing, right? So how do you route traffic as jobs move around the cluster? So we used to do this where you know we a job was running on foo API o one And it was always gonna run on foo API foo API o one and you know what if foo API o one crashes because it's Amazon We're gonna launch another foo API o one, right? And it's always gonna be you know port nine thousand for this nine thousand one for that This is obviously changing So we introduced HAProxy and Synapse so Synapse Everyone knows what HAProxy is right? Synapse Well for it. So this is kind of how it it works all of our jobs have our API servers have off proxy They have the API server themself and then they have a health check, right? And we bind off proxy to the public HTTP port and we bind the API server to the private port So internally if someone needs to make a request to that API server because you know It's a batch job that needs to read that data they go right to the private port someone coming from the outside world gets sent to off proxy and then that Proxies them to the API server the health proxy never needs to talk to HAProxy because that's running in the same Mesa server Right next to it so a request coming in for get HUD API scroll depth or private HUD API scroll depth goes HAProxy Into the Mesa cluster so the way Synapse helps us out here and Synapse was written by Airbnb, thank you written by Airbnb and it's The config is YAML and it's a super set of the HAProxy config, right? So Aurora when a job is launched in Aurora it announces into zookeeper that you know Hey, this HUD API is running You've got three instances of it and here's their ports that are defined HAProxy is pulling zookeeper or Synapse rather is pulling zookeeper and when it detects a change it generates a new HAProxy config and bounces HAProxy this happens pretty quickly and I know that Earlier someone was talking about how there's actually an update to HAProxy which makes it so you don't lose any connections But when it bounces we probably lose a couple All of these APIs are being accessed from JavaScript. That's just gonna retry the request So it's not a big deal, you know, it takes like 200 milliseconds to restart it or something We use puppet to manage the HAProxy config and the Synapse config So if a user needs to add a new route, they'll add it to puppet and push that out And it'll get picked up on our HAProxies. Our HAProxies are all Running in ELB Okay So question number two metrics collection right metric collection is really important and we wanted to make it easy Right, we had several different ways people were doing this before They were using, you know everything from TSDB to All sorts of different ways of collecting stuff. So we decided to consolidate everything on open TSDB and Grafana Open TSDB was written by Etsy I think most people are familiar with Grafana. That's a great dashboard for visualizing this stuff. So our flow is open TSDB into Grafana Nagios is pulling Grafana and also TSDB directly And seeing if something is wonky and then we use pagerDuty for alerting the cool thing about the naming of all these things in Aurora Is that it's very easy for us to say, you know Figure out the the tag TSDB works with tags So you have it's time series database. You have a time series You get a point in time and it has a bunch of tags So one tag is the name of the job one tag is which environment it's running in one tag is what users it running as So that gets very easy to say well, let me look at the HUD API dev now Let me look at HUD API prod because everything is consistently named We're able to actually write tools that easily let engineers put this data in and also scrape all the data That's coming out of Aurora for the jobs running so we can generate dashboards for any job Showing how much CPU is it using how much RAM is it using all of our JMX metrics also get this data So we can graph things very easily everyone knows where to look for any job. That's running So we've written libraries and Python and closure that do all of this kind of auto tagging based on the Aurora names And we wrote what we call the JMX collector Which actually pulls any job that's running in Aurora and just pulls out the JMX metrics If there's a JMX port to find and stores them in TSTB and Grafana dashboards for everything so You know engineers love pretty graphs. So here's our generic meso's job graph Up on the top you can see CPU used by task you can the red line is the limit that's been assigned This job is nice. It peaks a little bit but not too bad, which is how we like it, right? And you can see if any any specific tasks are using more CPU than others Which is actually happens a lot in Kafka consumers. It especially if you have an unbalanced Topic right so we can say oh this task is is using way more one thing. We haven't quite figured out how to solve is A way to give certain tasks more CPU But I think that's probably a pipe dream. It's better. Just figure out how to balance the topic better So log file analysis this was actually a really big one for our users I think mostly because log file analysis has always been kind of tough When we told users they couldn't just easily SSH in and use poly SH to tell all their logs They were like no, it's horrible So we tried to pull everything into elastic search With cabana and I know some people love it.  So we chose to use flume Athena which is an Amazon product. It's Presto running an Amazon and Something called tail LLL Because we already had a tail LL So what are what are those look like so I mentioned users want a poly if you're not familiar with poly SH It's a little Python program let you like shell into 10 servers run the same command and see all the output nicely separated Even with colors. It's really cool So we wrote this addition to our Aurora client Called Aurora managed tail LL and a job name So since we have we know where everything's running from the Aurora client you can say hey I want to tell this job and it will say hey where are these jobs running? It'll log into all of them and and pull back all of your log files and just print them out The there is an Aurora web UI to just kind of click on a job and look at the log file just like in Mesa's But that doesn't help with a cron job that already stopped running or if it got cleaned up So we suck all of our log files into Athena through flume Athena just lets you do ad hoc SQL like queries across s3 buckets So, you know if you want to do kind of historical forensic on why did this thing die yesterday? you can just go into Athena and look at the logs and we have a lot of stuff that's not running in Mesa's So all of those logs go there as well like our Kafka brokers our databases. I didn't mention we don't put databases in Aurora Yeah, I don't think we will So two years later and we're really psyched We've reduced our on-call events dramatically We've cut our EC2 instance cost by about 33 percent and at the same time we've built new stuff It's not like we didn't build new stuff We actually recently did an engineering survey because we couldn't figure out any other way to really measure our KPIs So we says let's let's do a Google survey and ask our engineers what they think and we asked a bunch of somewhat leading questions Maybe but they said that they rarely experience blockers deploying stuff, which is great And it's honestly changed our entire approach to DevOps One of the most Telling examples is how we use the pecs files You know, we actually build pecs files for all of our command line tools now, so we have a pecs file that runs S3 command we have pecs files that run, you know all of our All right We have command line tools to launch instances and stuff like that and we just wrote we do mess a pecs file now We wrote a thing called Pecs runner which we deploy on all of our machines Which you can say pecs run and give it the name of a job and a githash and it'll just download it from s3 cache it locally Which we do on our meso servers as well. So this pecs you want this githash. Oh, I already have it I don't have to go and get it from s3 And that's really changed the way that we approach bundling our Python stuff. So that's been great So that's it. I want to leave time for questions and I have so awesome any questions. I Should say that's our engineering blog and there's posts about this. They get into some more detail I'm on Twitter at our mangy and github at our mangy and everywhere else pretty much at our mangy except at work Where I'm Rick, so that's it. Thank you


Yn y cwmysgwyl, y Cymru yw'r Ardeithasol a Llyfrgellol yw'r Fyllgrifes Cymru yn gweithio'r Gweithgol, yng Nghymru, Yn Yn Yn Yn Yn Yn Gweithgol, yw Dr Boswn Tijani, yw'r cyfrifio cyffredinol a'r cyfrifio'r cyfrifio'r cyfrifio. Yn yn y gweithgol yw'r Gweithgol, Mr Aminau yma, opeth yng Nghyn 아무 yddefnyddio, ym Gwrs Gymru'r Cymru, ac ym Neslaen yw'r conflu sydd cydiefayng a'raders payments מש scriptures ym Nghymru ac ym F COME visitation board ar honno rydym a'r newid hum pamys 이f.字 hwn etm i heatedad y prydau wir gêmwys. am unrhyw ploddur dr Tijani for convincing the president to sign four executive orders, including the suspension of the 5% exercise tax on telecommunications services. The telecommunications sector contributes immensely to the development of the country. Previous administrations in Nigeria from 1999 have made gains in the sector. The administration of President Paula Ametinibu is collaborating on these gains as part of his promised renewed hope. His appointment of an experienced young Nigerian, the President of Bosun Tijani, as Minister of Publication and Digital Economy, has yielded fruits with the reforms introduced by the ministry since the assumption of office. He got the president to sign executive orders to suspend multiple taxation that stifled businesses in the sector. The Nigeria Publication Commission should revamp stakeholders' engagement across the country. Two, the government should address the cost of doing business. This will reduce the burden on telecommunications operators and also reduce the call rates. Three, the government should invest massively in infrastructure, especially in the power sector. Four, investment in information communication technology, ICT, should be encouraged through manpower capacity development.


Rocket tombs are pretty widespread in the central Mediterranean, so as Mariellies are shown in their nice map, they're all over the place. So Sardinia in that sense is not so original, we have lots of rocket tombs, also time they are clustered together in symmetries. But one particularity of Sardinia is that there are lots of decorated rocket tombs. They have quite a complex internal architecture with a lot of different spaces and rooms, and many of them are actually supposed to imitate interiors of houses or buildings. So that's why they are locally called Domuseiana, which means house of the fairies. Right, so in the last 25 years or even in the last 50 years, a lot of research has been done in these rocket tombs in Sardinia. Most of them are focused on the internal parts of these monuments, so lots of surveys, typological classifications and studies, excavations of course. And here you see a picture of an ongoing project by my colleague Maria Grazamellies from the University of Sassari. So it's a lot of work being done, but paradoxically a little attention has been paid to their landscape context. I said paradoxically because for the later period in Sardinia where you have the neurologic powers, lots of GIS studies or more spatial analysis have been done. But for the rocket tombs, despite the fact that there are so many of them and they are crying for spatial analysis, so nothing has been really done in a very systematic or quantitative way so far. And in a way it's a missed opportunity to address important questions, because when you look at the landscape context of the tombs you can ask why they were created in these specific locations. So Maria Grazamellies for example were showing some maps with the settlements and the locations of the tombs. But if we exclude just the settlements from now, and if you walk in these landscapes in Sardinia you see a lot of road faces that were suitable for rocket tombs. But the rocket tombs were made slightly elsewhere in other locations. Why were the tombs made there and not there? So whether there were any cosmological implications, is it due to orientation or proximity to big mountains or big rivers, there might be some kind of specific attributes that need to be identified. And then of course there's all the questions of the connection with the settlements. So whether the settlements were closed by or far away. And these links, the only thing that interests me here is all these social relationships between the dead and their living house. We can see them through this landscape relationship between the settlements and the village. So how the dead were perceived and whether they were perceived as for example dangerous entities or maybe important entities for daily life decisions. So you would probably place a date in different locations in the landscape to reflect or to enable this particular type of social relationship with your dead relative. So of course there are lots of challenges and the main challenge for this kind of study if we focus on the settlements themselves is that they are very elusive. So we don't know much about the contemporary settlements for the pre-neurogic period in Sardinia. It's partly due to the fact that the landscape is very eroded so we don't have many standing structures in Sardinia for the period. But probably also for lack of systematic research, systematic survey trying to identify evidence for settlements in the surrounding. And so for the period, so for the pre-neurogic Sardinia, most of the evidence we have for settlements are mostly in the southern part of the island where we have not so many rocket tomb sites.  And that's what we try to do in a project based in Ossi in northwest Sardinia over the last three years. And we picked up this area for one good reason is that in this particular area of Ossi you have an important concentration of cemeteries. So there you can see on this map there are like four main groups. And at the beginning we had a big project to do a lot of survey around all of them but we had to focus on one particular site which I'm going to focus now which is called Mezhu and Montes. Mezhu and Montes is the name of that particular rocket tomb cemetery which is shown in red here. It's quite a big cemetery. It has 18 rocket tombs. It's been known for many years. And Macri Gerudas has done some preliminary survey in the mid 90s in the region. But of course we wanted to do something different than to again try to do a detailed survey and try to look up the relationship with the landscape and the settlement. So we did different things. We did 3D recording of all the tombs. We're using photogrammetry and the objective was not just to create a nice documentation of the architecture and the decoration because many of these tombs are wall decorations as you can see in this image. It was also to have very accurate georeference plans of each of these tombs in order to check things for example like orientation of these monuments in the landscape. And I'll go back to this point later. So we recorded the tombs, we recorded the topography because it's quite complex and interesting topography we have here. So we've just done some drone photogrammetry, again georeference using a GPS so there's kind of standard methodology today. Just to create a topographic map like this one and I wonder if I can use the... Just to present briefly the topography here we have a big here, the Montemama with almost all the tombs being located on the same rock cliff on the edge of this mountain. And then you have a small valley and another high plateau here called Montemanu where you have one isolated tomb here which is also typologically different from the other one. Typologically it looks like it belongs to an earlier phase of the rock tomb tradition in Sardinia. So that's for the topography and the tomb orientation is something interesting. When you look at the literature what you find most of the time is that the tombs are in generally orientated to the southeast part of the landscape which is traditionally interpreted as a cosmographical perception as the rising sun being important for the regeneration of the dead and so on. But what we notice here is that in that particular site the orientation of the tomb may vary from one tomb to another. When I speak of orientation it's something quite easy to check because all these monuments tend to be very linear in design. They have several spaces, several rooms that are separated by small rectangular windows or doorways. So if you just look through this alignment of doorways you can have an idea of where the tomb is actually looking at in the landscape. So if you imagine this kind of central axis of the tombs and then you look at some of them have interestingly built, some of these tombs are interestingly built, not exactly perpendicular to the rock face which would be the simplest solution if you want to develop your rock tomb in the available rock. So there may be a different explanation for this but it looks like it's a deliberate choice to follow one specific orientation which is not determined by I would say practical reasons. And then if you look at the general map of orientation of these tombs, so they are placed in different ways but they tend to converge to this big top hill which is called Mount Immanuel which I presented before. This is the one with this isolated tomb here. So I'll go back to this and that leads into the third objective of this project which was to retrieve evidence for sediment in this particular site. And I just skip this because of course you can do GIS predictive analysis as a preliminary exercise. But the best way to do this is to do systematic survey and in this case we have the GPS we call every single piece of pottery or flint and we ended up with nice concentration maps which basically are showing the same things. So all the pottery, all the flints, all the oxygen are concentrated on the top and on the wet and slope of this Mount Immanuel. So if you look at the overall picture we have a nice pattern with all the tombs converging towards the Mount Immanuel where you almost certainly had the sediments contrary with the symmetry. And of course now we have to develop this hypothesis. So there are other evidence for sediment here. You have some structural remains post holes, old stone structures that needs to be excavated and dated. And in terms of chronology just based on the stone pottery, the pottery we found on the surface we have quite a wide range of chronology. We have shirts that are from the Middle Neolithic which is fitting with the probably the first period of use of the site for the symmetry because we have this tombah pozzetto which is typical from the Middle Neolithic in Sardinia. And then we have more material from the Late Neolithic up to the Middle Bronze Age, which again corresponds to the later phase of use in the symmetry with this nice stelae, if I say, rocket tombs that were built in that period. I just want to mention that this last summer we have started excavation as well and based Mount Immanuel site which tends to confirm the hypothesis of this being a sediment site. So we found some architectural remains and some other pottery. We couldn't go very deep into the excavation because we were surprised by the good preservation of the context. So we mostly excavated the probably middle and ancient Bronze Age phases of the site. But that looks very promising. So to conclude about this landscape approach, it proves at least for this site to be a quite useful approach that may in a way change the way we see these rocket tombs or why they were built in these particular areas of the landscape. In this case at least that there is a very strong relationship, landscape relationship between the landscape position and the orientation of the tombs and the village of the living. So that brings interesting things. Again, if we try to interpret this socially, we have the village of the dead who are separated from the village of the living but the two are closely related. They are visually connected so that probably means that the dead were quite important to be visible on a kind of daily life basis in this and those societies. I suppose the next point would be to explore questions of architecture. So if we're lucky enough to find a lake-nurrific architecture, for example, with good plans of houses, it would be interesting to address whether the tombs that are imitating houses are actually imitating houses or more special buildings or other things. So that's me. Thank you very much for everything.


Some of these initiatives have generated really, again, small steps into how to address these issues. For instance, now many of my male colleagues, for instance, will refuse to be part of a male-only panel. And I've had examples of male academics walking out of conferences where these issues were pretty apparent and being a problem. Or just ensuring and just supporting women that will want to go up and climb the ladder more and more. And sometimes making those voices heard is important because it does change culture. Some things like, you know, not having meetings at the end of the day or allowing for family time. And all these really small issues sometimes change because women do reach those positions and become more aware of the issues that other people face.


The Kaduna State Ministry for Health says over five million people are at the risk of contracting tropical diseases, leading to river blindness, lefantises, lymphatic filarises, and intestinal worms, among other diseases. The State Commissioner for Ministry of Health, Dr. Umar Ahmed, stated these as the marked the 2024 world-neglected tropical diseases in Kaduna. In the Kaduna State, 5,970,722 people are at risk of river blindness, lymphatic filarises, bilhaziasis, and intestinal worms. Over the years, the country has been battling seriously to eliminate most of these tropical diseases using the following approaches. Preventive chemotherapy by conducting mass administration of medicines, mobility management, and disability prevention. Integrated vector control, sound sanitation, hygiene practices with safe water supply. She laments that globally more than 1 billion people are affected with 149 countries facing the disease and Africa bearing half of the global burden of 120 million of its people living at the risk of one neglected tropical disease. The State, aside by the World Health Organization, to sensitize all global communities on the burden of NTDs, which affects more than 1 billion people across the globe, with about 149 countries faced with these diseases and Africa bearing about and also contributes the highest burden in Africa with more than 120 million of its people living at the risk of one neglected tropical disease or the other. End the suffering from these 20 diseases that are caused by a variety of pathogens including viruses, bacteria, parasites, fungi, and toxins. Dr. Omar said Kaduna State committed to scale up interventions to meet the global targets of leaving no one untouched and eliminating NTDs by 2030 by contributing to stronger health systems and urging residents to join in the fight against NTD. The fight against NTDs is a collaborative one. I call on all million residents of Kaduna State, public and public private organizations to unite to act towards eliminating NTDs in Kaduna State.