So the Sonic the Hedgehog trailer was finally released and all of us old school gaming geeks have been waiting for it for what seems like ages.  So when this trailer dropped, oh the backlash, the backlash and the memes and people were freaking out about how terrible Sonic the Hedgehog looked. And because of that it was just announced that the studio was going back to the drawing board and completely redesigning Sonic the Hedgehog. And in this video we're going to talk about how that is a complete and utter waste of time. What is up everybody this is Chris from the Rewired Soul where we talk about the problem but focus on the solution and if you're new to my channel I try to take the mess and turn it into a message. So if you're into that stuff make sure you subscribe and ring that notification bell. Alright so what are my qualifications to talk about this movie in Sonic the Hedgehog? Well let's talk about my gaming history.  But anyways my first system was an NES, my next system was a Sega which I had for a long time until I got a Playstation. And I wasn't one of those fancy kids who got both a Sega and an SNES I just had the Sega. So Sonic was my dude. More specifically Knuckles was my dude. I loved me some Knuckles but I grew up on Sonic the Hedgehog loved it. You guys remember that Sonic game where it had like the cartridge where you can like open up the top and then put in a different Sonic game? Does anybody remember that? If you do let me know down in the comments below. But anyways like I grew up on Sonic the Hedgehog and then when these movie posters started coming out and stuff and you got Sonic's legs and he's sitting on top of what it looked like the Golden Gate Bridge I'm like oh no oh no no no no. Right and we're going to talk a little bit more about this in a minute but I want to touch on a few things. I want to talk about where the studio screwed up, my opinions and views on that but also what we need to understand as people too. Alright but anyways one of the first things that just comes to mind that just comes to mind like do you ever watch like TV shows or movies and they have those things called focus groups right like focus groups are a real thing if you're not familiar with focus groups are they call in people they call in a certain demographic of people that they plan on marketing something too right and basically they bring in this focus group and they show them different stuff maybe it's a product or whatever it is or in this case it would be a movie so like I don't get what studios aren't getting focus groups like it is insane it is like I felt the need to make a video on this because this this movie comes out what like in the summer and they're like oh we're just gonna redesign Sonic the Hedgehog like it blows my mind that they didn't try to get some input or or did nobody working on this project like play Sonic or any like I just don't get it so like where were the focus groups when it comes to this movie the next thing I want to talk about is it just seems like some of these studios are so out of touch with where we're at as a society like some of my favorite youtubers like Drew Gooden's a great example like when he did his review of like the new Kim possible coming out and just completely like just made fun of it how ridiculous it looked I didn't watch Kim possible but I knew that live action one looked ridiculous right but like this constantly happens on the internet when people try to remake things and bring them back they just get trashed they get utterly trashed there's been a few good remakes that I've actually enjoyed but here's the thing like most recently most recently in the last six months or so when it was first released the image of Will Smith as the genie in a Latin people lost it people lost it so I'm just trying to understand how you could work in the like Hollywood movie industry the remake industry and not have seen because this happens all the time and not have seen the public backlash and outrage about Will Smith how he looked as the genie in the new Aladdin and said you know what we're working on the Sonic the Hedgehog movie maybe we need that time out for a second to make sure that this doesn't happen to us and the other thing that they didn't think about is that that the Disney fanboys and girls they're not nearly as brutal as the gaming fanboys and girls all right and that is one of the reasons why I think the Sonic one just like whoa it blew up it became such a thing and yeah as soon as the the trailer was released oh my god like every just about every youtuber and influencer I follow like like Elvis the alien I think Leon Lush just so many people were just making fun of Sonic and posting memes of him Justin Wang was doing it I'm like yeah yeah that's fair right but here's here's what I want to talk about okay because me and you chance on me and you we're not Hollywood producers we're not making movies or anything but me and you like here's the thing like we're not gonna be happy with anything all right like I just think we need to come to those terms I don't think we're gonna be general genuinely happy with anything so what I try to do as I try to practice a little empathy and I try to think okay why are they making this movie well the first obvious answer is a money grab right a money grab like I I think that they're marketing to people like myself other people around my age who played Sonic and now have children because when I saw this I'm like oh cool my son and I play like newer video games like Fortnite and everything but this movie is a way for me to introduce him to my childhood so like I think they they look at that as a money grab same thing with these like Disney remakes and they look at it like that but the other thing that that they're thinking is is that they're going to make a movie that maybe is nostalgic for us that we're going to enjoy but because of the money grab they're out of touch with it right but then like we have to think to ourselves like one of the reasons I say we talk about the problem but focus on the solution is because this happens in everyday life with everything and that's one of the reasons why I want to make a video about this movie is because like I don't think anybody like anybody that I've seen really has has put out solutions and that's one of the issues with just where we're at as you know human beings today like a lot of us just sit around and complain and everything like that but we don't offer solutions right like we don't say like what would make it better what would make this thing better so in my opinion when I look at this I'm like what would make this better and the only thing I could really think of is like a full on CGI movie like not live action like into the spider-verse was this phenomenal movie that was completely 100% animated but in most movies when you kind of intertwine like CGI with live action it gets a little silly right so I don't think any of us would have been happy and you could you can let me know if I'm wrong like let me know what your solutions would be down below but what I'm getting at is they are completely wasting their time and I think I think it's got to be difficult why don't why am I saying I think I know it's difficult for any anybody in the public spotlight right now whether it's movie studios or influencers or whatever it is because the outrage is so loud and so in your face that like this studio is redesigning this character before the movie comes out now I am not in that industry but I can only imagine how much time and how much money that's going to take right and looking at the grand scheme of things and the bigger picture on this for the money that they're about to dump into this to this redesign do they think they're going to make that money back right I personally highly doubt it now the second question that I want to ask is do they think that people are going to be happy because my assumption would be is that they're probably going to pull in a focus group they're probably looking at all the ridiculous stuff online and they're gonna try their best but they cannot reshoot this entire movie they can't change everything because if we're gonna be honest there was much more wrong with that trailer than just Sonic the Hedgehog's design I don't think I don't think people watch that trailer and they were like everything in this trailer looks amazing except Sonic the Hedgehog like just the last thing I'll touch on is I am a huge Jim Carrey fan you know huge huge huge grew up on Ace Ventura, Dumb and Dumber all that stuff and even just seeing Jim Carrey in this I was excited to hear he was going to be playing Dr. Robotnik in this like even his role doesn't look all that great so what I'm getting at is it's just crazy how like companies and industries bend to the will of the mob and everything like that and I see this as a waste of time and money because at the end of the day especially as gaming geeks out there we're never gonna be happy you know I think at the end of the day to like most of us just want them to leave well enough alone just like okay that was great you know even though we like you know going back and playing those games we don't like seeing remakes or having them put on the big screen and everything like that but let me know your thoughts on this down below like do you think there's a way that they could do this right do you think that a simple redesign of Sonic the Hedgehog would make this movie better right let me know if you're gonna see it like I'm probably gonna see it no matter what whether I love it or hate it or whatever because I'm always surprised there's a lot of remakes that I've taken my son to go see he absolutely loved them even though I didn't really think they were that great but at the end of the day like hey some of those studios that's really all they need you know two people go half of them are really entertained they win all right anyways let me know your thoughts on this topic down in the comments below but that's how I got this for this video if you like this video please give it a thumbs up if you're new make sure you subscribe and ring that notification bell because I make a ton of videos and a huge huge thank you to everybody supporting the channel over on patreon you're all amazing and if you would like to help support what I'm doing here on the rewired soul and get involved in our monthly q&a it's about the perks and benefits you can click the top right there all right thanks again so so much for watching I'll see you next time


Have you already seen what's coming? Hi! Hi! Welcome to the Basin of 2020 Distribution Discoverer. Our next talk is going to be on Fedora package, we're all high package gating, with the area usual. So, this being the last talk of the last day, of course I might be very thankful for you wanting to be present in the room. So we're going to be speaking about this official version of Fedora, which is the live branch, and how we've implemented gating packages in the special area. I'm not going to go into the mechanisms of what we get on, the idea is not to go into which tests are used to make a decision about whether a package should go through or not, but I'm going to run through the mechanisms that we place in the mechanism, the architecture and the mechanisms that we have to put in place to be able to get the question on which steps I'm going to debate on. We can discuss about that, but we're actually not discussing that. Yes, that's good. So what are we going to do today? I'll start quickly going to the grocery, because I know some familiar faces are in the room with those, so you'll see some unfamiliar faces as well. I don't know how much of the Fedora can see, so I'd like to actually give you a bit of information that we're speaking about, so first. Then we'll go into why we want to rate two gates in real life. What should we think there? Some of the challenges that we have had to face are some of the constraints that we're trying to put in projects. We'll start now, of course, to study some things. Or you can debug, you know, try to see if you can unlock yourself through some of the conduct cases. Ideally, you know, I don't want to put... This is a great presentation. Everything's working, great. So there shouldn't be a debugging slide, but you know, since I've got some questions, so there's a good instance. Then I'm going to explore the robot only once, you know, once we have this in place, but we will go where we want to go. And then we can use my small surprise for the regular experiments that we've done here as well. Let's come to the questions. There are a lot of questions I'm sharing with you as well. So, sir, what is RAHI? RAHI is the, basically, the demo professional of RAHI. It's a rolling release. It's a release from a person who has never released it. It's composed everyday, but you can find it on the mirror, but it's basically not the version of RAHI. It's on everyday. It has new packages coming in, and the interesting thing about RAHI before it starts to evolve is... Then we have, how does the package work for RAHI? It's something that we call dsds, and this is basically a term that we use for choosing. It's a collection of digital repositories. Every single package is federal. Every single package is federal as the corresponding digital repositories in which the packages are maintaining spec files and patches. It's also like a web service that you post in the platform. So, is this also a GITUS in-platform? It's kind of applied for the GIT repo, for most of the rebels. So, what is that? It has some limitations. They are playing GIT repos. What sits on the top of it is a web interface that we have designed to sit on the top of it, but the current infrastructure is basically two elements. It's a basic GIT server and what we call deluca-side cache, where the packages upload the table from the other projects. So, it's really a collection of GIT repo. It's not a central one, and each one is different, but it's been dated on an issue. There are some limitations, but you can't force push, for example, if that's what you're thinking about. But these are configurations that you can do to any GIT repository. So, it's not specific to architecture in there. Then we have the...  So, everything that is shipped in Fedora is built in a control environment, which is called, in our case, a coaching. So, that's where everything is built. Builds are managed with tags. So, a build in a certain tag is in a certain state. If it moved to the next tag, it's moved to the next state. So, basically, the tags will give you a little bit of the state of the package. If it's in the F32 tag, that means it's available in the build route for Ryder moments, which is still F32. Something that we're going to come back with is they are called site tags. They are basically tags which are sitting next to the usual main ones, and they are used to be able to do work without impacting the main tags. So, they are isolated from the user, but they rely on the base tag. So, there is a key between the tags, so you have the base tag, and then the other one sits on the top of it. So, the lower down, you can always access the tags to try on an issue, but you can't access the tag which are above you. Then we have our update system. Once we have made a build, we can say, well, I want this build to go up to the users. This is handled by an application called body, and basically, every package gets to choose well, that application goes to the users that the data does not go to the users. That is true for every stable federal release. That is not true for RoHide. In RoHide, as soon as you build something, it's going to go to RoHide. So, there may be builds which are wasted. There may be builds that would not need to go to the users, but the mechanism of RoHide has always been if you build it, it goes to RoHide. In stable releases, you can build it and say, well, actually, I don't want to push this one. One of the aspects there as well is also, if I push something, and then I realize it broke, I can't push it up to a certain point, so it does not affect our users until we are kind of sure that it won't affect. We have an MQP-based message bus called Fedora Messaging. Well, Fedora Messaging is basically a wrapper around MQP to make it easy to send messages on the bus. This is very used for making the different applications to each other. So, whether you will listen to messages from Koji, Koji may react to messages from another source and so on. And finally, we have something called a Robotsoniatry. It listens to the messages coming from the bus, coming from a TV from Koji, and it will, so every time there is a build, it will be notified of it, get the RPM, sign the RPM and move the build to the next tag in Koji. So, what do we want to get Rohite? Well, simply, Rohite is a place which is, I think that's true, fairly well known in the ferrite community to be something which is not stable. I don't think that's wrong to say that. It is wrong now to accept this. There is no reason why Rohite should not be stable. The only reason why we allow Rohite to not be stable is because we don't have ways of grouping changes. If I build something and it dumps the surname, then I need to rebuild all the dependencies but that can take me some time and during the time that I'm rebuilding all the dependencies, well, Rohite is broken. That leads to situations like F29, I believe, or F30 when we branch, so we branch off Rohite to make the stability disease. When we branch off Rohite for F30, F29 or F30, I forgot, we did not have a working compose for about a month because Rohite was broken when we branch off Rohite, we branch a broken Fedora. So there is no reason why we want to keep Rohite broken and getting is bad, preventing this breakage from happening. We want a more stable Rohite. We get working composes. This means when we branch off Rohite we get a working compose, we get a working F31, F32 for the next one and we can already work on that product. That also means faster updates for Rohite itself because in Rohite updates the push of the updates of the right updates the mirrors up and as part of the compose. If you don't have compose for a month, you don't have you're not pushing right updates out to the mirror for a month, so everybody using Rohite is not going to get their new versions. It would be more specific about what composes in this one. So the compose is so the question is can we define more precisely what composes? That's an overloaded term which is always very hard to define. The basic answer is when we compose Fedora we are rebuilding the updates repositories, the DNF repositories that we are pushing out to the mirrors but we also are building our ISO image for example, our base image for containers and so on. So depending on who you talk to you may be talking about a part of the process or the entire process but when we talk about the right compose it means generating the repo, making sure the repos solve themselves so the dependency resolution works building the basic images. If one of these steps breaks the entire compose is considered a failure and not pushed to the users. One of the last things why we want to get Rohite is we want to get in the point of some of us have the chance to be paid by companies to work in Fedora the vast majority of the contributors in Fedora are volunteers they do that under free time which is a great investment we can't thank them enough for using your free time to come and help us and work on Fedora it's a song to have you there I've been there before I got the chance to be hired to work on this. The only thing is you may be maintaining a package and then something lower on the stack breaks they contributed because for X and Y reasons they don't have the permissions to do that and then suddenly what you're doing is you're on only day you're having family time you're working on something else on your pet project and then you get an email saying that your package can no longer stall because there is a broken there is a tsunami bump in one of the dependencies and you have to go and fix it you have to go and rebuild your package because someone else bumped some M and did not warn you and you couldn't coordinate the change with that person so we want to get into the mood of you break it, you fix it it's fine to break something you just don't, you fix it and when you push something out it's something which is fixed, it's something which is working so that's one of the changes that we want to bring to the right it's not okay to have a right broken it is okay to break things but let's break them in a way that does not impact everyone but you until you have figured a way to fix them so what are the challenges well the first challenge was to make it happen it's not the first time if you're part of the federal committee you may have heard it some months ago I would say it's not the first time that we tried to work on this but it failed for a number of reasons last time so when we start to look into that again we need to be careful on them we need not work last time and how can we make it work this time the second one is well those are more requirements we want to fit into the existing tooling we don't want to reinvent a built system that will provide us the getting we don't want to reinvent an update management system I'll come back to this and we want to interrupt the package of workflow as little as possible every time we change something in the package of workflow of contributors if you update a package every day that is you quickly learn the trick to use the new way if you're packaging something every three months every time you have to go back to the documentation and where is it again is that page the documentation up to date or is it this one over there or is that the female which I'm finding in my net box amount and then the last one is there will be bugs there will be bugs in the CI there will be bugs in the test there will be bugs in the way we're testing so we need to have a way to handle false negative something which works but it's identified that it's not working so we need a way to bypass the result of the test in a way that is satisfactory for our packages so having all of this in mind one of the idea was that we would roll out the changes in faces so we try to do the the easy one first that only contains a single package it's a it's a Python module that is updated that does not depend on anything it's pure Python it's Python 3, you know it the most simplest thing you can think of this is the easy case one build, one package, one update and then as we as we get user feedback as we polish them we also work on how to deal with updates that contain multiple build and test it as one unit so this was our idea let's do single build first place build to build after that gather feedback as early as possible and try to account for that feedback as much as we can and try to get a polish user experiment at the end of it where is it today? well we've announced at Flock so the Fedora conference last summer that we were able to do single package getting we are happy to announce you that we can do multi package getting as well in Fedora so how does it work? these are slides which are back from the, I just went back to the slide which I used at Flock when I presented this so for the single build getting it's a fairly complex system this is basically the flow of everything it's not meant to be read so don't worry that you can't read it it's just to give you an idea every column here is a different system and then every box is a different action there being the first system over there and as you can see the interaction with the user are actually fairly limited there is only the user's, the package starts a process and there is the overwrite here which is basically if something goes wrong the false handling the false negative case here do you have any steps where with the human would have to present some info for the group? everything is automated so this is the version of the large graph which is much easier to grasp and even for myself that's the one I go back to when I need to see if something is working the way it should we have a packager they do, that person does a Koji build in a certain tag in Koji so this is the basically since this is a candidate for an update of the F31 that was made this summer before F31 was out so F31 was or had at that time Koji announces that he has built and tagged the build into this tag RoboSignatory receives the message signs the build and push it into update pending Boody gets the message that an update was pushed into update pending it creates an update and then it waits when it creates the update it sends a message to the CI system that says this update is ready to be tested the CI system is going to test it it's going to come back to Boody and say yep that can go and Boody moves it or the CI system says no that can't go and Boody says well you know I can't do anything there so one of the changes that we have introduced here is that Boody was not part of the question before, before you would build it would be signed and pushed directly to the stable tag to the build route, nowadays everything goes via Boody because Boody was the place which was natural for packager to get feedback about an update so we are fitting to the existing workflow we change the workflow because we introduce it, but this is a natural place for packager to receive feedback about if this something is going right or not why is it a natural place because Boody when you create an update Boody is the place where people can report if an update is working or not so there is already a mechanism there for commands from contributors in the community to say well that update brought my system let's and push it, so this is ready now we have to build the picture didn't get clear the number of systems did not get smaller and the number of interactions actually didn't get that worse either so the simplified version again you start by creating a site tag so this is a manual this is a manual step so that changes from the way you were working before you create a site tag and that is your site tag F32 is going to be the base so currently F32 is right so that's going to be always F32 and then a random integer well it's an increasing integer then you're just going to build in that site tag as many packages as you like if you have two packages you do two if you have 100 you do 100 once you are ready you're going to be the one going to Boody and you're going to tell to Boody that site tag here is ready but he's going to get all the builds that you've made in that site tag and turn it into a single update it's going to signal that to the CI system that will kick off but you know it's not so it's going to it's going to create the updates it's going to move it's going to Boody at the time when you create the update it's creating two site tags as well it's using the same ID sign in pending and then testing pending and once you have created these two site tags it moves all the builds from your site tag into the sign in pending run the signatory as pattern matching algorithms that says well everything that is a 32 site something sign in pending I'm listening to I take all the builds in the updates I sign them and I move them to the testing pending Boody gets the notification that something was moved to to testing pending it marks the update as ready to be tested it signals it to the CI system that will perform all the tests on all the packages at once so if you have two packages that needs to be tested together it will make sure that both packages are present on the system to be before the tests are run sorry? there is a package package is it like internal red hat employee or outside anybody in the federal community with a package can do these steps how can we create the tags? Fed package request site tag so the question is how do we create the site tags? there is a utility that is used by packages that is called Fed package it is a small CLI tool and there is a simple action that says Fed package request site tags that basically goes to Koji and asks for a site tag to be created and it returns you with the idea of the site tag which you can then pass on to your build command and that you specify to Boody the next thing is that a site tag can also be shared if you are working on a update with a group of person the first one creates the site tags gives the site tag idea to the rest of the group and everybody can build in that site tag and then once everything is ready someone goes to Boody and creates the corresponding update so the question is if we have packages depending on each other in the site tags how do we end up package A being present in the build route when we build package B so there are two ways to do that the hard way is to do package A and then you wait for Koji to create the repository for the site tag before you trigger the build for package B or you have a nicer way to do that is the Fed package command again that is called chain build and that basically does that for you it tells Koji I want to build these packages and they depend on each other and you can group them so you can parallelize say package A and B can be built together but C needs package A and B the question is can we build the same package twice Koji has a requirement that's on the NEVRA so that's name, epoch version release architecture of a package so as long as you can send to Koji you can send to Koji a git hash basically to build as long as the NVR you can send it multiple times but if the NVR has already been built Koji will say well that has already been built so I guess the answer behind that is can I have a build that's present in multiple tags at the same time because you would like to send an NVR to that build to this site tag as well as the same NVR to another site tag, is that what you're denying? No, no, I think in this case I want to I have a package and another thing to build without this test disabled and then build another package that depends on this one and I want to do a full build on the first package with test enabled or something like that so I find this bump in the release in between so in this case it's a bootstrap case where you start with the test is able to build the dependency and then build again the first package with the test enabled so that's because it needed a package to be able to run the full test run yeah, in this solution you have no choice at the moment but bumping the release again So I follow a question can I untag a package from the the question is so the question is can I tag and tag packages and yes you can tag and tag packages if you're not allowed to do that Koji will tell you you're not allowed to do that but it's one of the elements on how you can unblock yourself if you find yourself that somehow you've built lost a tag you can actually just Koji tag the tag and the package name and Koji will happily tag it for you if you're allowed to do it, if you're not allowed he will just say well I can't do that so that's one way you can untag yourself so you can do packages for Koji like any can someone create freestyle target and you find it on tagging structure so the question is how much freedom do packages have to configure Koji and the answer to this one is none this is a release engineering task basically the other one that will be defining the tags, the hierarchy you can create your site type so you're not going to be able to create official tags do we need to start the CI before the signing or do we need to sign the package so the question is is it possible to start the CI process earlier in the process before the signing that was one of the questions we already considered and the issue which I had with that is that if you end up installing a package you end up testing the installation of that package not in the way the user are going to use it so if something goes wrong here and the package is not correctly signed I'm going to suddenly my testing store is going to fail because the package is expecting to be signed because in Fedora by default the the UNDNF configuration requires the package to be signed so this is going to fail if you say I'm triggering from here or directly from the Koji build well suddenly you it means you have to test with GPG in our check which basically is the different than what the user would be so it was a conscious decision to start the CI so late in the process but in a way that we make sure that what we test is actually what we are going to push to the users so how much time is it a button tag or is it natural the question is is robusting natural a button net yes and no on a daily basis no because the amount of builds that we get is low enough that robusting naturally just takes them it can be a button net during master builds master builds is when we take all the 20,000 kit reports of packages in Fedora and we say Koji please rebuild so within the DRQ Koji gets to rebuild 20,000 packages which means robusting naturally gets to sign 20,000 packages if we mess up a little bit which has happened not in just this much maybe but the last one where we forgot to turn on robusting naturally from the start we basically had to so robusting naturally goes through all the 20,000 builds and well copy them over are you signed no you're not okay I'm signing you are you signed yes you are okay next one are you signed yes are you signed yes and that basically is from robusting naturally but that was a mistake on our side because we forgot to turn on robusting naturally earlier in the process if in the case of the current master build that just happens robusting naturally was set up correctly from the start and basically as the build was up and robusting naturally was turning so and we didn't get the issue of swamping robusting naturally which we had at the last master build that's not to be passed any notion about the movement of these updates or there is an update for each patent for body wise there is an update for all group of patents or it's separate so in the case of and the question is is there an update for every build or is it a different update in the case of single build it's one update per build in the case of multi build it's one update per site tag so all the builds in the site tags are going to go into one single update they are going to be tested as a single update they are going to be pushed to the mirror as a single update so maintainer has to sign some body to say okay I have done all my builds now you can start that's one change in the package of workflow before you had to build and that's all you had to do now if you have multiple builds you need to create a site tags then do your build and once you're done with it you say okay you can go ahead there was thought about automating this and saying well we just keep on testing the site tags until it passes and then as soon as it passes we merge it but there is no guarantee that there is not some builds left over behind that should be still included in that as a package there can you get any site tag like can you get any combination of other packages that are being maintained putting a site tag and then getting blocks here or other but adding them to your site tag so if you're a packageer you can create an arbitrary site tag right? so the question is can I create arbitrary site tags you can create site tags from a very specific you can't create a site tag from anything you have to create a site tag from one of the existing tags and a limited set of the existing site tags right but if you have like arbitrary packages that you will see that are not together but maybe very many gamers don't think so how will this settle somehow? so how can we what happens if in the case where I'm hiding a packages in the site tag which maintainer may disagree with we're coming back to the NVR question to build something in federal you need to change the spec file to change the spec file you need to have the right to change the spec file and not everybody is going to have the permission to change every spec file there so the chances are that you're going to include that you have the permissions that you have the permission to change the spec file to do the build and up in your site tags I'm going to say will be pretty low because you normally have permissions to the packages you should be concerned about or that you're maintaining and that's your own set of packages and if you are one of the fit people that have access to everything then we expect you to know better than to mess around just sort of so there is one there is one change between what I've presented at Flock and what is what is currently in action so what you see with the built is that we have we have the user it builds it goes to body and then through the signatory if you remember what before and if I'm clicking in the right direction that will be easier we had it built and then it stands up in body and I was basically creating two workflows depending if you were considering a single build or multi-build so we changed the workflow for single build a little bit now when you build it first goes to body so you already have your update that update is being signed and then it's being tested so we have a much more similar workflow between the multi-build and the single build one of the advantage here is that basically as soon as it builds you have a body update that body update is in the pending state with whatever you know if you have a single build or multi-build you have an update in pending state once it's signed you have it in testing state and once it's out you have it in stable so it's just to make sure that we have something coherent between the two workflows that's just a little bit on going through how the decision frameworks works so we have as I was saying we have body that says in update and basically body basically triggers the CI system by saying that update is ready to be tested the CI system will run on the test it finds and it will send the results to something which is called nicely ResultsDB so it's basically just a database with an API that just stores whatever we turn into it it's a key store database you can just put anything in there it's backed up by Sparklight but it's basically a key value store it's really you can put anything in there every time there is a new result it's going to announce it I have a new result then we have GreenWave here which is our decision engine it's basically as a set of rules and it says well every time I'm seeing a new result I'm going to check this can I make a decision now this package has been tested I have a new result about this package can I make a decision what are the rules applying to this package nope I'm still missing a test I'm not going to do anything hey a new result is coming hey that result failed alright then I already know that this result was a requirement so I can announce that this update is not going through because that build has failed so this is what happens CI sends the result to result.tv that sort them GreenWave gets the result from results.tv and makes a decision based on these results and send that decision back to body if the user disagrees with them because hey bugs are bugs networks are networks and computers sometimes fails us surprisingly the user can override that and overriding this is basically storing a waiver so you send a waiver that says this result that you find in results.tv you ignore it you can also say these results that you don't find in results.tv you ignore them so if something is stuck the CI system just simply doesn't work anymore well you're not starting your results so your date is blocked because something is broken and just doesn't move along so you can say well you know what just ignore everything that's not present and then GreenWave is going to be notified about the new waiver it's going to say can I change my decision should I change my decision and if it does it notifies body that will react based on that so that's a little bit the framework I'm mentioning I will stop for a second this slide because if you're into results.tv GreenWave and WebDB these are the trio that's used for actually making a decision about getting the results so how can we debug ourselves well as I was saying one of the quick ways to check the Koji tags so you run Koji build info you put the nvr of your build there you have a line that says tag and then you just look into that if it's a single build and it's an update testing it hasn't been picked up by body something is wrong with body if it's a side tag then that's for me to build then it's still a body problem if it's in sign in pending then there's something wrong with RoboSignatory it could be that RoboSignatory is just going through the master build and it's so very busy and it just it hasn't caught up with your build yet or it could be that RoboSignatory somehow broke the connection with the sign in server and needs to have a gentle poke here we start if it's in testing pending it's waiting for the CI system it could be that the CI system is proceeding a lot of requests or it could be that the CI system is done you know if you're there it's probably something in one of these two here it's probably an infrasue if you stay there too long that's definitely an infrasue if you stay there too long it could be RoboSignatory swamp or it needs to restart if you're there it's less an infrasue and more potentially more a CI problem or a CI system another way to look into that is to look at the state of your body update if it's pending it's not signed so pending means we've seen the build in update candidates we've created the data spending and we're waiting for it to be signed if it's in testing then the tests are running or the package is gated and if it's stable then you're not debugging anymore it's working or it should be working so some of the user feedback we collected over the last months was about the interesting body as an element of the right workflow introduced a lot of email notifications basically a single build would give you between five and seven emails you get one for your data has been created your data has been pushed your data has been tested your data has been pushed to stable yeah okay right used to send you no emails it's now sending you five per update so much we reduced it to three basically tells you your update has been created if the test results failed it will notify you if they pass it once and your update has been pushed that's basically the only three that you should get if it's still too much we are happy to revisit this is how we've the middle ground we've found for now if people complain too loudly we are happy to revisit and see which of these emails and which are actually not that useful when we introduce multi-build getting if you're federal package you may have seen the new UI introduced but if I go you didn't see the mechanism underneath it for multi-getting because we haven't announced it at that time but there were some changes there the list of builds used to be either space delimited or comma delimited and we removed the support for comma which is a lot more complex for a little gain so now a list of builds should be only space delimited there used to be a mechanism that goes to create all of your builds in Koji the builds you've made in Koji and the mechanism was such that it would basically block the input field when it was loading the builds from Koji and it turns out that request to Koji is sometimes very slow or that you have hundreds of packages or hundreds of builds and so you couldn't copy and paste so that was a feature that we thought was good and that we were told no that's not good so like the GNOME folks who update and load packages at once they are actually tracking the NVR in the text file and then they just copy paste in the field and they couldn't do that before so we've reverted that change you can now easily copy paste your NVR as you did before and yeah that's the next point here the logic that goes and creates body to know which which builds you've made is still a little bit slow and we still need to look at the performance there we also someone that came to us and asked well there is no point in having no loading people to comment on an update once it's pushed to stable and I've had that happening to few of my updates and it's annoying so please disable comments once an update is pushed to stable and we've had Adam W coming back to us and saying nooo because the update turns out the update is still a mechanism where you can point people well that update brought this and this the bug see that ticket is over there so as a user if you run into a body update it may actually be useful to see be able to comment on it even though the update has been pushed to stable so that gets reverted as well if you have more feedback to take it with me and I'm sure will work through either finding a way to make it work for you or just fixing the the old dancing ok so where do we want to go from that well we now have mechanisms to gate something we still need to optimize testing and reporting of test results for groups so if I have currently if I have 100 builds I'm going to create greenway for give me the status about that build give me the status about this build give me the status about this build so that's a little bit inefficient when we can tell when greenway when the CR system knows that it's been tested these 100 packages and could just say as once well you know this group of packages that have tested they failed CI that would be one request to greenway that would be one way to my things nicely so this is something which is which is in progress we still need to work on the now I'm breaking back to what I said would not be part of the topic what do we want to gate on so what are our tests currently the tests are defined in the package repository and they are defined by anyone basically anyone can go there and there is a specification that the test should follow they are using Ansible and you can basically write your own test this is great except that having 2,000 packages writing 20,000 tests for all packages and making sure that you know we all want to test whether we can install or update or remove a package there is no need to have 2,000 people do that for 20,000 ripples we could do that for the entire distro in one go and that's one test to maintain instead of having a different YAML from Ansible doing the same test for every single package so we want to look into distro level testing some of them are reverse dependency testing being able to run the test of the package that depends on me being able to install, to upgrade, to done great to remove and we also want to look at the impact on the composites does your update break compose push a new row height updates if your build goes in there so we don't know yet how we are going to achieve this one but that is one of the things we would like to go to something else that we are looking into is we have the agreement from the FISCO the technical steering committee in Fedora the engine and steering committee in Fedora so we have introduced 3 test packages so they are only there are spec files which only ship a UUID in a certain file and just you can install them they won't do you any harm but on the inside you really shouldn't install them no use at all but they are useful for us because basically while working on this we want scripts to be able to make sure things were working and since we are touching so many systems it's easy to you know to lose one so the idea is to have this script run on a regular basis you know once a day twice a day every two hours something like this and check is everything fine are the tests running are the tests failing can I wait the test is my update going through and then so for a single package as well as for two a build an update of two builds two single builds many builds so those are the things that we are currently working on so that we that we come up soon yes soon I guess and one of the discussion that was on the the other list recently about the infrastructure is not reliable well using this we may be able to look into how often things break in the entire workflow not a single application but the entire workflow as often does the workflow break because one application breaks it may be a different application at the time but the result is as a package or my workflow does not work so we may be able to measure how often things break and often they don't and that would also help us when we roll out a new version of body or a new version of Koji and actually ensure that what we want the best these are best scenarios we don't test every single sketch but we want to make sure that at least the best scenario case should work fine and another thing which we have in our roadmap and I don't know how we'll get to is what's the mastery build the scanning of Robocynutri during mastery build that is something which has happened in the past it was annoying for a few days there are high chances it will happen again in the future as much as we will try to avoid it so finally a small surprise for all of you who have been who are Fedogap Packager if you are open to the package you are probably less interested in this one and you've stayed so far so with multibills we have introduced on-demand site tags so you can create your site tag whenever you want well they are for ride but they don't work only for ride which means you can create a site tag for a stable release branch which means Fed package chain build which was only a ride feature now it works in stable federal releases as well you can create your site tags for F-30 and chain build and the builds in there and crown them into one update in basically two columns so enjoy that and that's it for me if you have any question I would like you to continue let's take the all the systems so so the question is how can I plug my own CI system in this workflow yeah there is a we are working on standards there is a very clear defined message from body that says this update should be tested the idea is you trigger federal messaging and you report to federal messaging so there is a defined message from body that triggers and there is a standard format that is expected for the report and then there is one small piece of software that basically is the piece which listen to the bus and it puts the results that needs to be adjusted if you follow the defined format you just listen to the new topic and process the message that way that's all that should be required so the question is how do we envision reverse dependency testing there are a few ways to look into that one of the first thing I would do is to go look at my friends in green over there in the OBS world where they actually do rebuild the entire dependency tree upon a change so that problem has been solved for them which means it shouldn't be a problem to solve it for us there are a few alternatives there clouds would be one way to scale up resources when we suddenly rebuild GCC and needs to rebuild the entire most of Fedora the interesting piece about that we do master builds currently and that goes through all the 20,000 packages but the 20,000 packages don't need to be rebuilt for a new GCC update all the Perl and Python and PHP modules which are not C plugins they don't care about the new GCC so if we were to do reverse dependency testing and rebuild for a new GCC we would actually rebuild less than we do today with master builds because we would only we would only rebuild what depends on GCC all the new arch packages we would ignore not just packages don't need to be rebuilt what about the feedback from the new GCC we want to modify the Python and the GCC that the GCC will occur? so the question is how do we what about the feedback if I rebuild GCC do I really want to know the test result about 5,000 packages those are good questions and we need to figure out what's the answer for this the first approach may very well be well rebuild GCC for 8,000 notifications about things not working or maybe you know we'll say not 8,000 but maybe we can group them and you get 8,000 in one email one way we would do that one way which would help maybe is also looking into the we said we wanted to optimize the test result when we come to groups of builds so instead of having one build, one result, one build having one result for the entire group that will solve that because then you have a group of 8,000 packages but you only get one feedback that says all of these packages have failed but it's one action and not 8,000 actions so that may be it those are clues or ideas they are not anywhere defined so I'm not going to put my hand on the table and say this is the way it's going to be there was this the question is what do we need to do to actually have the install update and upgrade test I'll say this is a chicken head problem as long as we didn't have a built-in mechanism we could build all the tests we wanted if they had no impact nobody cares a broken test that doesn't work on the side and does not impact me I can live with a small red dot on the side of my testing bar and in body updates if it doesn't impact me in any way we had that for years, Tascotron has been running and looking into upgrade tests for a long time they were not blocking in any way they had no impact in any way so there is a chicken head problem as long as we did not have getting there was no incentive to actually go and look at these tests and try to fix them now that we have this I think it's time to look more into that problem the question is going to be a lot on the ordering which one should we start with I know RPM spec is being worked on at the moment it's also going to be probably one of the first controllable testing that is enabled and it's also used as a a playground how do we plug in a new CI system how do we, because it's going to have its own pipeline, how do we expand our testing system there so this one is probably going to be the first one and the next one after that, it's a good question fixing upgrades versus impact on compose reverse dependency versus install which one should we start with I don't know and that's the place where Alexandra is not there, but Alexandra Fedorova who works on the CI SIG is looking into this question so she will be the person to contact to help or provide feedback on some of these ideas so the question is how do we know where do we get the information about the number of package present in Fedorova there are two places the first one is to look into these sites which is going to tell you the number of Git repos that we have and that's about over 20,000 Git repos that we have and you can look per name space or you can look on IDRPM space and I will give you an IDRPM one ignoring the modules, the containers the flatbacks the second one is there is a problem with that number is that we keep Git repos the trees also for packages that are no longer shipped from Fedorova, but we seal out the Git repos because that's part of our history of as a distribution so that number is a bit squeeze you will get all the packages that Fedorova has ever had whether they are active or not you won't get that information there well you could but that means going through each and every package and checking if they are like as retired the one place where you would have the actual number and you would have to create for it and look for it that's called the PDC that's the database PDC stands for Product Definition Center and it's a database and that's a place where we code for every packages their studies and basically if you are allowed to push to the master branch or not, so if the master branch you cannot push to it basically means the package has been retired so that's the place where you would like to say well give me for all packages the studies of the studies of each branch or the master branch and from that number you would be able to see how it's going the last place where you could see progress is going to be body the body works with binary RPMs so at one source RPM can give you how many sub-packages do we have in LaTec again? I think it's up to 9300 something like that, so way too much so yeah the body's number is going to be squeezed because it's only going to count the binary RPM and other source RPM I think PDC is going to be your process back and that's not going to be the easiest one packageDB used to have a feature where it would give you the number of package per branch but we would longer out that graph it's the same for long-packs we take the translation out of the interface and we make one package for each language and we make the installation in order for language we don't need translation in order so the number we have to come is this source RPM so the remark was it's the same for long-pack where we have one source RPM that splits the number of RPM per languages so that you can reduce the install surface on containers and the like and therefore that's why the number to look at is the number of source RPM and the number of RPM splits that's not a question I believe I forget anything else for x86 architecture or for ARM? the question is, is it only for x86 architecture or only for ARM? the mechanism is arch-independent the problem is the CI system currently works on the mh86 architecture there is plan to add support for ARM to the CI system but as far as I know it's not there yet but yeah the idea is to start with if we can get 80% of the work with x86 architecture it's still better than the zero we have now and then when we get 90% adding ARM then we get another 10% 20% of the work is going to get you 80% of the benefits and the remaining 20% is going to take you the 80% of the work so using this word we start with the easy one trying to get the most results and then see if we can improve that later on


Psalm 119 of the Book of the Psalms, American Bible Union. Happy the upright in their way, who walk in the law of Jehovah. Happy they that keep his testimonies, that seek him with the whole heart, who also do no wrong, who walk in his ways, thou hast enjoined thy precepts, that we should keep them strictly. O, that my ways were directed to keep thy statutes, then shall I not be shamed when I have respect to all thy commandments. I will praise thee with uprightness of heart, while I learn thy righteous judgments, thy statutes I will keep. Do not forsake me utterly. Beth, whereby shall a young man keep his path pure, by taking heed according to thy word? With my whole heart have I sought thee. Do not let me wander from thy commandments. In my heart have I treasured thy saying, that I may not sin against thee. Blessed be thou, O Jehovah! Teach me thy statutes. With my lips have I recounted all the judgments of thy mouth. In the way of thy testimonies have I rejoiced, as over all riches. In thy precepts will I meditate, and have respect to thy paths. In thy statutes will I delight myself. I will not forget thy word. Gemel, deal kindly with thy servant, that I may live, and I will keep thy word. Bend thou mine eyes, and let me behold wondrous things out of thy law. I am a stranger in the earth. Do not hide from me thy commandments. My soul breaks with longing toward thy judgments at all times. Thou hast rebuked the proud, accursed that wander from thy commandments. Roll off from me reproach and contempt, for thy testimonies have I kept. Also princes sat and talked against me, thy servant meditates on thy statutes. Also thy testimonies are my delight, my counsellors. Daleth, my soul cleaves to the dust. Do thou revive me according to thy word? My ways I have declared, and thou didst answer me. Teach me thy statutes. The way of thy precepts make me understand, and I will meditate on thy wonders. My soul melts away with sorrow. Raise me up according to thy word. The way of falsehood remove from me, and grant me graciously thy law. The way of truth have I chosen. Thy judgments I have set before me. I have cleaved to thy testimonies. Jehovah, do not put me to shame. The way of thy commandments I will run, for thou wilt enlarge my heart. He teach me, O Jehovah, the way of thy statutes, and I will keep it to the end. Make me understand, and I will keep thy law, and will observe it with the whole heart. Make me tread in the path of thy commandments, for therein do I delight. Incline my heart to thy testimonies, and not to gain. Turn away my eyes from beholding vanity. Quicken me in thy way. Confirm to thy servant what thou hast said, which is for thy fear. Turn away my reproach, which I dread, for thy judgments are good. I have longed after thy precepts, in thy righteousness. Quicken thou me, Vav, and let thy mercies come to me, O Jehovah, thy salvation according to thy saying, and I shall answer him that reproaches me, for I have trusted in thy word, and take not from my mouth the word of truth utterly, for I have waited for thy judgments, and I will keep thy law continually, for ever and ever, and I shall walk at large, for thy precepts have I sought, and I will speak of thy testimonies before kings, and will not be ashamed, and I will delight myself in thy commandments, which I love, and I will lift up my hands to thy commandments, which I love, and will meditate on thy statutes. Zayin, remember the word to thy servant, on which thou hast caused me to hope. This is my comfort in my affliction, for thy saying has revived me. Those ones have greatly derided me. From thy law I have not swerved. I remember thy judgments of old, O Jehovah, and have consoled myself. Indignation has taken hold of me, because of the wicked who forsake thy law. Thy statutes have been my songs, in the house of my sojournings. I have remembered thy name in the night, O Jehovah, and have kept thy law, this I have had, for thy precepts have I kept. Jehovah is my portion. I have said that I may keep thy words. I have sought thy favour with the whole heart. Be gracious to me according to thy saying. I thought on my ways, and turned back my feet to thy testimonies. I made haste and delayed not to keep thy commandments. The cords of the wicked were around me. Thy law I have not forgotten. At midnight will I rise to give thanks to thee, on account of thy righteous judgments. I am a companion of all that fear thee, and of them that keep thy precepts.  Teach me thy statutes, Teth. Thou hast dealt well with thy servant, O Jehovah, according to thy word. Teach me good understanding and knowledge, for I have believed in thy commandments. Before I was afflicted, I went astray. But now I keep thy saying. Thou art good and doest good. Teach me thy statutes. The proud have forged a lie against me.  As for me, in thy law do I delight. It is good for me that I was afflicted, that I might learn thy statutes. Better to me is the law of thy mouth than thousands of gold and silver. Yod, thy hands made me, and fashioned me. Make me understand that I may learn thy commandments. They that fear thee will see me, and will rejoice, for I have hoped in thy word. I know, O Jehovah, that thy judgments are right, and in faithfulness thou hast afflicted me. Let I pray thee, thy mercy be for my comfort, according to thy saying to thy servant. Let thy compassion come upon me, that I may live, for thy law is my delight. Let the proud be ashamed, for they wronged me without cause. As for me, I meditate on thy precepts. They will turn to me that fear thee, and that know thy testimonies. Let my heart be perfect in thy statutes, that I may not be ashamed. Calf, my soul faints for thy salvation, for thy word do I wait. My eyes fail for thy saying, while I say, when wilt thou comfort me, for I am become like a bottle in the smoke. Thy statutes I do not forget. How many are the days of thy servant? When wilt thou execute judgment on my persecutors? The proud have digged pits for me, who are not according to thy law. All thy commandments are faithfulness. They persecute me wrongfully. Help thou me, almost had they consumed me upon earth, and I, I forsook not thy precepts. According to thy mercy, revive me, and I will keep the testimony of thy mouth. Lamed, for ever, O Jehovah, thy word is settled in the heavens. To generation and generation is thy faithfulness. Thou hast founded the earth, and it stands fast. For thy judgments they stand fast this day, for all are thy servants. Unless thy law had been my delight, I should then have perished in my affliction. Never will I not forget thy precepts. For with them thou hast quickened me. Thine am I. Save me! For thy precepts have I sought. The wicked have waited for me to destroy me. To thy testimonies do I give heed. To all perfection I have seen an end. Thy commandment is exceeding broad. Mem, how do I love thy law? All the day it is my meditation. Thy commandments make me wiser than my enemies, for forever is it mine. I am become wiser than my teachers, for thy testimonies are my meditation. I have more understanding than the aged, for thy precepts have I kept. From every evil path have I withheld my feet, in order that I may keep thy word. From thy judgments I have not departed, for thou thyself dost guide me. How sweet to my palate are thy sayings, more than honey to my mouth. From thy precepts I get understanding.  Nun, a lamp to my foot is thy word, and a light to my path. I have sworn and have fulfilled it. To observe thy righteous judgments I am afflicted very greatly. O Jehovah, revive me according to thy word.  My soul is in my hand continually, but thy law I do not forget. The wicked have laid a snare for me, but from thy precepts I have not strayed. Thy testimonies have I taken as a heritage for ever, for they are the joy of my heart. I have inclined my heart to perform thy statutes for ever to the end. Samech, the double-minded I hate, and thy law I love. My hiding place and my shield are thou, for thy word do I wait, depart from me, ye evildoers, and I will keep the commandments of my God. Uphold me according to thy saying, and I shall live, and do not let me be ashamed of my hope. Do thou hold me up, and I shall be saved, and I will have regard to thy statutes continually. Thou hast made light of all that wander from thy statutes, for a vain thing is their deceit. As dross thou hast put away all the wicked of the earth, therefore do I love thy testimonies. My flesh shudders from dread of thee, and of thy judgments. I am afraid. Iean, I have done justice and righteousness, thou wilt not leave me to my oppressors, but surety for thy servant for good, do not let the proud oppress me. My eyes fail for thy salvation, and for thy righteous saying, deal with thy servant according to thy mercy, and teach me thy statutes. I am thy servant. Give me understanding, and I shall know thy statutes. It is time that Jehovah should work. They have broken thy law. Therefore do I love thy commandments, above gold, and above fine gold. Therefore all thy precepts I esteem right, every path of falsehood I hate, pay. Wonderful are thy testimonies. Therefore has my soul kept them. The unfolding of thy words gives light, making the simple understand. I opened my mouth wide and panted, for I longed for thy commandments. Turn to me, and be gracious to me, as thou art won't to do to them that love thy name. My steps establish by thy word, and let no iniquity rule over me. Redeem me from the oppression of man, and I will observe thy precepts.  My eyes run down with streams of water, because they observe not thy law. Tzade, righteous art thou, O Jehovah, and upright in thy judgments. Thou hast enjoined in righteousness thy testimonies, and in exceeding faithfulness. My zeal consumes me, because my foes forget thy words. Pure is thy saying, exceedingly, and thy servant loves it. Little am I, and despised, thy precepts I do not forget. Thy righteousness is eternal, right, and thy law is truth. Trouble and anguish have come upon me. Thy commandments are my delights. Right are thy testimonies, for ever. Make me understand, and I shall live. Kof, I call with a whole heart. Answer me, O Jehovah, thy statutes I will keep. I call on thee. Save me, and I will observe thy testimonies. I rise early with the dawn, and cry for help, for thy words do I wait. My eyes anticipate the night watches, to meditate on thy saying. Hear my voice, according to thy mercy, O Jehovah, according to thy judgments, revive me. Near are they that follow after mischief, they are far from thy law. Near art thou, O Jehovah, and all thy commandments are truth. Long time have I known from thy testimonies, that thou hast founded them for ever. Resh, see my affliction, and rescue me, for thy law I have not forgotten. Plead my cause, and redeem me, according to thy saying, revive me. Far from the wicked is salvation, for thy statutes they have not sought. Many are thy compassion, O Jehovah, according to thy judgments, revive me. Many are my persecutors, and my foes, from thy testimonies I have not swerved. I saw the faithless, and loathed, them that keep not thy saying. See how I love thy precepts, O Jehovah, according to thy mercy, revive me. The sum of thy word is truth, and every one of thy righteous judgments is for ever. Gene, princes persecute me without cause, but at thy words my heart trembles. I rejoice over thy saying, as one that finds great spoil, falsehood I hate and abhor. Thy law do I love. Seven times in the day I praise thee, on account of thy righteous judgments. Great peace have they that love thy law, and they have no occasion of stumbling. I have hoped for thy salvation, O Jehovah, and have done thy commandments. My soul has observed thy testimonies, and I love them exceedingly. I have observed thy precepts, and thy testimonies.  Let my supplication come before thee, according to thy saying, Rescue me. My lips shall pour forth praise, for thou wilt teach me thy statutes. Let my tongue answer to thy saying, that all thy commandments are right. Let thy hand be for my help. For thy precepts have I chosen. I have longed for thy salvation, O Jehovah, and thy law is my delight. Let my soul live and praise thee, and let thy judgments help me. I have gone astray like a lost sheep. Seek thy servant. For thy commandments I do not forget." The End of Psalm 119, Recording by Mark Penfold. Psalm 120 To Jehovah in my distress I called, and he answered me, O Jehovah, rescue my soul from lying lips from a deceitful tongue. What shall he give to thee? And what shall he do more to thee, deceitful tongue? Sharp arrows of the mighty with burning coals of broom, alas for me that I sojourn in messek, that I dwell with a tense of keydar, my soul has long dwelt with him that hates peace. I am for peace, but when I speak they are for war. Psalm 121 Pilgrim Song I will lift my eyes unto the mountains. From whence shall my help come? My help is from Jehovah, who made heaven and earth. Let him not suffer thy foot to waver. He that keeps thee, let him not slumber. Behold, he will not slumber, and will not sleep that keeps Israel. Jehovah is thy keeper, Jehovah is thy shade on thy right hand. By day the sun shall not smite thee, nor the moon by night. Jehovah will keep thee from all evil, he will keep thy soul. Jehovah will keep thy going out, and thy coming in, henceforth and forever more. Psalm 122 Pilgrim Song of David I was glad when they said to me, Let us go into the house of Jehovah. Our feet are standing in thy gates, O Jerusalem, Jerusalem that art buildeth as a city that is compact together. Wither the tribes go up, the tribes of Jah, a testimony for Israel, to give thanks to the name of Jehovah. For there are set thrones for judgment, thrones of the house of David. Pray for the peace of Jerusalem, they shall prosper that love thee. Let there be peace within thy rampart, prosperity within thy palaces. For my brethren and companions' sakes, let me now say, Peace be within thee, for the sake of the house of Jehovah, our God, I will seek thy good. Psalm 123 Pilgrim Song Unto thee do I lift up my eyes, thou that dwellest in the heavens. Behold, as the eyes of servants are toward the hand of their masters, as the eyes of a maid servant toward the hand of her mistress, so are our eyes toward Jehovah, our God, until he shall be gracious to us. Be gracious to us, O Jehovah, be gracious to us, for we are greatly filled with contempt. Our soul is greatly filled with the scorn of them that are at ease, with the contempt of the proud. Psalm 124 Pilgrim Song of David If it were not Jehovah who was for us, let Israel now say, If it were not Jehovah who was for us, when man rose up against us, then had they swallowed us up alive, when their anger was kindled against us, then had the waters overwhelmed us, the stream had gone over our soul, then had gone over our soul the swelling waters. Blessed be Jehovah who has not given us a prey to their teeth. Our soul is as a bird escaped from the snare of the fowlers, the snare is broken and we are escaped. Our help is in the name of Jehovah who made heaven and earth. Psalm 125 Pilgrim Song They that trust in Jehovah are as Mount Zion, that cannot be moved, abides forever. Jerusalem mountains are round about her, and Jehovah is round about his people, henceforth and forever, for the rod of wickedness shall not rest on the lot of the righteous, that the righteous may not put forth their hands to iniquity. Do good, O Jehovah, to the good, to the upright in their hearts, and they that turn aside to their crooked paths, Jehovah will lead them away with workers of iniquity. Peace be upon Israel! Psalm 126 Pilgrim Song When Jehovah brought back the return of Zion, we were as they that dream, then was our mouth filled with laughter and our tongue with singing, then said they among Jehovah has done great things for them, Jehovah has done great things for us, we are joyful. Turn, O Jehovah, our captivity, as streams in the south, they that sow in tears shall reap in joy.  He shall surely come with rejoicing, bearing his sheaves. Psalm 127 Pilgrim Song of Solomon If Jehovah built not the house, in vain they labor upon it that build it. If Jehovah keep not the city, the keeper watches in vain. Vain is it for you that ye rise early, that ye take rest late, that ye eat the bread of sorrows, so gives he to his beloved in sleep.  Be the man who has filled his quiver with them, they shall not be ashamed when they shall speak with enemies in the gate. Psalm 128 Pilgrim Song Happy is everyone that fears Jehovah, that walks in his ways, for thou shalt eat of the labor of thy hands. Happy art thou, and it is well with thee. Thy wife as a fruitful vine, in the interior of thy house, thy sons as olive plants around thy table. Behold, thus shall the man be blessed that fears Jehovah, Jehovah will bless thee out of Zion, and thou shalt see the good of Jerusalem all the days of thy life, and thou shalt see thy children's children. Peace be upon Israel. Psalm 129 Pilgrim Song Much have they oppressed me from my youth. Let Israel now say, Much have they oppressed me from my youth, yet have they not prevailed against me. Plowers plowed upon my back, they made long their furrows. Jehovah the righteous has cut us under the cords of the wicked.  Let them be as grass on the housetops, that withers before it is plucked up, with which the mower fills not his hand, nor the gatherer his arm. And they that pass by say not, The blessing of Jehovah be upon you, We bless you in the name of Jehovah. Psalm 130 Pilgrim Song Out of the depths I call on thee, Jehovah, Lord, harken to my voice, Let thine ears be attentive to the voice of my supplications. If thou, O Jah, shouldst mark iniquities, O Lord, who shall stand? Or with thee there is forgiveness that thou mayest be feared? I have waited for Jehovah. My soul has waited, and in his word have I hoped. My soul waits for the Lord, more than they that watch for the morning, Than they that watch for the morning. Hope thou Israel in Jehovah, for with Jehovah there is mercy, And with him abundant redemption, and he he will redeem Israel from all his iniquities. Psalm 131 Jehovah, my heart is not haughty, nor my eyes lofty, nor do I concern myself with things too great, and with things too difficult for me. But I have calmed and quieted my spirit, as a weaned child on its mother, As the weaned child is my spirit within me. Hope thou Israel in Jehovah, henceforth and for ever! Pilgrim Song Jehovah, remember to David all his pains. Who did swear unto Jehovah, did vow to the mighty one of Jacob? I will not enter into the tent of my house. I will not go up on the couch of my bed. I will not give sleep to my eyes, slumber to my eyelids, until I shall find a place for Jehovah, dwellings for the mighty one of Jacob. Lo, we heard of it at Ephrata. We found it in the wooded fields. We will enter into his dwellings. We will worship at his footstool. Arise, O Jehovah, to thy resting place, thou and the ark of thy strength. Let thy priests be clothed with righteousness, and let thy saints shout for joy. For the sake of David thy servant, do not turn back the face of thine anointed. Jehovah has sworn to David in truth. He will not turn back from it. Of the fruit of thy body will I set upon thy throne. If thy sons will observe my covenant, and my testimonies that I shall teach them, also their sons forever shall sit upon thy throne. For Jehovah has chosen Zion. He has desired it for his abode. This is my resting place forever. Here will I dwell, for I have desired it. Her provision I will abundantly bless, her needy I will satisfy with bread, and her priests will I clothe with salvation, and her saints shall shout aloud for joy. Where will I cause the horn of David to put forth? I have prepared a lamp for my anointed. His enemies will I clothe with shame, but on him shall his crown flourish. Psalm 133 Pilgrim Song of David Behold how good and how pleasant the dwelling of brethren in union, as the precious oil upon the head, flowing down upon the beard, the beard of Aaron, that flowed down to the border of his vestments, as the dew of Hermon that comes down on the mountains of Zion, for there Jehovah commanded the blessing life forevermore. Psalm 134 Pilgrim Song Behold, bless ye Jehovah, all ye servants of Jehovah, that stand in the house of Jehovah by night, lift up your hands toward the sanctuary, and bless Jehovah, Jehovah bless thee out of Zion, maker of heaven and earth. Psalm 135 Praise ye, Jehovah, praise ye the name of Jehovah, praise ye servants of Jehovah, that stand in the house of Jehovah in the courts of the house of our God. Praise ye, Jehovah, for Jehovah is good seeing praise to his name, for it is pleasant, for Jehovah has chosen Jacob for himself, Israel for his peculiar treasure, for I know that Jehovah is great and our Lord above all gods, all that Jehovah pleased he has done in the heavens and on earth, in the seas and all depths. Who causes vapors to ascend from the end of the earth, makes lightnings for the rain, brings out the wind from his storehouses, who smote the firstborn of Egypt, both of man and beast, sent signs and portents in thy midst, O Egypt, on Pharaoh and on his servants, who smote many nations and slew mighty kings. Sihon, king of the Amorites, and Og, king of Bashan, and all the kingdoms of Canaan, and gave their land as a heritage, a heritage to Israel his people. Jehovah, thy name is forever, Jehovah thy memorial is to all generations, for Jehovah will judge his people, and for the sake of his servants will repent. The idols of the heathen are silver and gold, the work of the hands of man. A mouth have they, but they speak not. Eyes have they, but they see not. Ears have they, but they hear not. Yea, there is no breath in their mouth. Like to them are they that make them, everyone that trusts in them. House of Israel, bless ye Jehovah. House of Aaron, bless ye Jehovah. House of Levi, bless ye Jehovah. Ye that fear Jehovah, bless Jehovah. Blessed be Jehovah out of Zion, who inhabits Jerusalem. Praise ye JAH. The end of Psalms 120 through 135. Recording by Mark Penfold. Psalms 136 through 150 of the Book of the Psalms, American Bible Union. This LibriVox recording is in the public domain. Recording by Mark Penfold. Psalms 136 through 150 of the Book of the Psalms, American Bible Union. Psalm 136 Give thanks to Jehovah, for he is good, for his mercy is forever. Give thanks to the God of gods, for his mercy is forever. Give thanks to the Lord of lords, for his mercy is forever. To him who alone doeth great wonders, for his mercy is forever. To him who made the heavens with skill, for his mercy is forever. To him who spread out the earth upon the waters, for his mercy is forever. To him who made great lights, for his mercy is forever. The sun for dominion over the day, for his mercy is forever. The moon and stars for dominion over the night, for his mercy is forever. To him who smote Egypt in their firstborn, for his mercy is forever. And brought out Israel from their midst, for his mercy is forever. With a strong hand and an outstretched arm, for his mercy is forever. To him who divided the Red Sea into parts, for his mercy is forever. And made Israel pass through in the midst of it, for his mercy is forever. And shook out Pharaoh and his host into the Red Sea, for his mercy is forever. To him who led his people in the wilderness, for his mercy is forever. To him who smote great kings, for his mercy is forever. And slew famous kings, for his mercy is forever. Sihon, king of the Amorites, for his mercy is forever. And Og, king of Bashan, for his mercy is forever. And gave their land for a heritage, for his mercy is forever. A heritage for Israel, his servant, for his mercy is forever. Who in our low estate remembered us, for his mercy is forever. And rent us from our foes, for his mercy is forever. Who gives bread to all flesh, for his mercy is forever. Give thanks to the God of heaven, for his mercy is forever. Psalm 137 By the streams of Babylon, there we sat and wept when we remembered Zion. On willows in her midst we hanged our harps. For there demanded of us our captors, words of song, and our oppressors, mirth, sing to us of the songs of Zion. How shall we sing Jehovah's song on an alien soil? If I forget thee, O Jerusalem, let my right hand forget. Let my tongue cleave to my palate, if I do not remember thee. If I prefer not Jerusalem above my chief joy. Remember, O Jehovah, to the sons of Edom, the day of Jerusalem, who said, lay bare, lay bare to the very foundation therein. Daughter of Babylon, the desolated, Happy he who shall requite to thee thy deed, which thou hast done to us. Happy he who shall seize and dash thy little ones against the rock. Psalm 138 Psalm of David I will thank thee with my whole heart, before the gods will I sing praise to thee. I will worship toward thy holy temple, and will thank thy name for thy mercy and for thy truth, for thou hast magnified thy saying above all thy name. In the day when I called, then thou didst answer me, didst embolden me with strength in my soul. All kings of the earth will acknowledge thee, O Jehovah, when they hear the sayings of thy mouth, and will sing of the ways of Jehovah, that great is the glory of Jehovah.  Though I walk in the midst of trouble, thou wilt revive me against the anger of my enemies, thou wilt stretch forth thy hand, and thy right hand will save me. Jehovah will complete it in my behalf. Jehovah, thy mercy is for ever. The works of thy hands do not forsake. Psalm 139 To the Chief Musician A Psalm of David Jehovah, thou hast searched me, and thou knowest, thou dost know my sitting down and my rising up, thou perceivest my thought from afar, thou art around my path and my couch, and art acquainted with all my ways. For there is not a word in my tongue, but, lo, Jehovah, thou knowest it all. Behind and before thou hast beset me, and laid thy hand upon me. Knowledge too wonderful for me. It is high, I do not comprehend it. Whither shall I go from thy spirit, and whither shall I flee from thy presence? If I ascend into heaven, thou art there. If I make the underworld my bed, lo, thou art there. If I should take the wings of the morning, should dwell in the uppermost part of the sea, there also would thy hand lead me, and thy right hand would hold me. And if I say, only let darkness cover me, and the light about me be night, even darkness will not hide from thee, and night will shine as the day. Darkness is as light. For thou, thou hast formed my reigns, hast woven me in the womb of my mother. I will praise thee, for I am fearfully wonderfully made. Wonderful are thy works, and my soul knows it well. My frame was not hidden from thee, when I was made in secret, was curiously wrought as in the depths of the earth. Thine eyes saw my unformed substance, and in thy book were all of them written. Day by day were they fashioned, when there were none of them. And to me, how precious are thy thoughts, O God! How great is their sum! If I would recount them, they are more in number than the sand. I awake, and am still with thee. O that thou wouldst slay the wicked, O God! And ye men of blood, depart from me, who speak of thee with evil purpose, take thy name in vain, thy foes.  And shall I not loathe them, that rise up against thee?  Search me, O God, and know my heart. Try me, and know my thoughts, and see if there be any idle way in me, and lead me in the way everlasting. Psalm 140 To the chief musician, a Psalm of David.   They have sharpened their tongue like a serpent. Poison of an adder is under their lips. Pause. Keep me, O Jehovah, from the hands of the wicked, from the man of violence preserve me, who have thought to subvert my steps. The proud have hit a snare for me, and cords they have spread a net by the wayside. Traps have they set for me. Pause. I have said to Jehovah, thou art my God, give ear, O Jehovah, to the voice of my supplications. Jehovah, Lord, the strength of my salvation, thou hast covered my head in the day of battle. Grant not, O Jehovah, the desires of the wicked man, do not further his device that they may be lifted up. Pause. As for the head of them that compass me about, the mischief of their own lips shall cover them. Burning coals shall be cast down upon them. He will plunge them in fire, into deep waters that they rise not again. An evil speaker shall not be established in the earth. The man of violence, evil will hunt him to ruin. I know that Jehovah will maintain the cause of the sufferer, the right of the poor. Surely the righteous shall give thanks to thy name. The upright shall dwell in thy presence. Psalm 141, A Psalm of David. Jehovah, I call upon thee. Make haste to me. Give ear to my voice when I call to thee. Let my prayer present itself as incense before thee. The lifting of my hands as the evening offering. Set a guard, O Jehovah, at my mouth. Keep watch over the door of my lips. Do not incline my heart to any evil thing. To busy itself in wicked deeds with men that work iniquity. And let me not eat of their dainties. Let the righteous smite me. It is kindness. And let him reprove me. It is oil for the head. Let not my head refuse. For still my prayer is in their calamities. When their judges are hurled down among the rocks, then they hear my words that they are pleasant. As when one furrows and cleaves in the earth, our bones are scattered at the mouth of the underworld. For my eyes are unto thee, Jehovah, Lord. In thee have I trusted. Do not pour out my soul. Keep me from the grasp of the snare they have laid for me, and the traps of the workers of iniquity. Let the wicked fall into their own nets, until I shall have wholly passed by. Psalm 142, Didactic Psalm of David. When he was in the cave, a prayer. With my voice to Jehovah I cry. With my voice to Jehovah I make supplication. I pour out before him my complaint, my trouble I make known before him. When my spirit faints within me, and thou, thou knowest my path. In the way that I go, they have hidden a snare for me. Look on the right hand and see, and I have none that knows me. Refuge has failed me. There is no one that cares for my soul. I cried unto thee, O Jehovah. I said, thou art my soul. I said, thou art my refuge, my portion in the land of the living. Be attentive to my cry, for I am brought very low. Rescue me from my persecutors, for they are stronger than I. Bring out my soul from prison to thank thy name. The righteous will gather round me, for thou wilt deal kindly with me. Psalm 143, A Psalm of David. O Jehovah, hear my prayer. Give ear to my supplications. In thy faithfulness answer me in thy righteousness, and do not enter into judgment with thy servant, for in thy sight no one living is righteous. For the enemy has persecuted my soul, has smitten down my life to the earth, has made me dwell in darkness as those long dead, and my spirit in me fades. My heart is desolate within me. I remember the days of old. I meditate on all, thou doest. I think on the work of thy hands. I spread out my hands unto thee. My soul is as a land thirsting for thee. Pause. Make haste to answer me, O Jehovah. My spirit fails. Do not hide thy face from me, so that I become like them that go down to the pit. Let me hear thy loving kindness in the morning, for in thee do I trust. Make me know the way that I should go, for to thee do I left out my soul. Rescue me from my enemies, O Jehovah. With thee I hide myself. Teach me to do thy will, for thou art my God. Let thy good spirit guide me on even ground. For thy name's sake, O Jehovah, thou wilt revive me. In thy righteousness, thou wilt bring my soul out of trouble.  For I am thy servant. Psalm 144. A Psalm of David. Blessed be, Jehovah, my rock. He who instructs my hands for the conflict, my fingers for the battle, my loving kindness and my fortress, my high tower and my deliverer, my shield and in him I trust. He that subdues my people under me. Jehovah, what is man that thou shouldst know him? A son of man that thou shouldst think of him? Man is like a breath, his days as a passing shadow. Jehovah, bow thy heavens and come down. Touch the mountains that they smoke. Cast forth lightning and scatter them. Send out thine arrows and discomfort them. Send thy hands from on high.  O God, a new song will I sing to thee, with a ten-stringed lute will I sing praise to thee, who gives deliverance to kings, who rests David his servant from the hurtful sword.  So that our sons may be as plants, full grown in their youth, our daughters as corner pillars, sculptured after the structure of a palace, our garners full, supplying of every kind, our flocks multiplying by thousands, by tens of thousands on our fields, our oxen laden, no breaking in nor going forth, and no outcry in our streets. Happy the people to whom it is thus, happy the people whose God is Jehovah! Psalm 145, a hymn of David, I will extol thee, my God, O King, and will bless thy name forever and ever. Every day will I bless thee, and praise thy name forever and ever. Great is Jehovah and greatly to be praised, and his greatness is unsearchable. Generation to generation shall praise thy works, and shall declare thy mighty acts. The glorious honor of thy majesty, and thy wondrous works will I sing, and the might of thy terrible acts let them tell, and thy great deeds will I rehearse. Let them pour forth the memory of thy great goodness, and sing aloud of thy righteousness. Gracious and compassionate is Jehovah, slow to anger and of great mercy. Jehovah is good to all, and his compassion are over all his works. Let all thy works praise thee, O Jehovah, and thy saints bless thee. Let them tell the glory of thy kingdom, and speak of thy power. To make known to the sons of men his mighty acts, and the glorious majesty of his kingdom, thy kingdom is a kingdom of all ages, and thy dominion is throughout all generations. Jehovah upholds all the falling, and raises up all that are bowed down. The eyes of all wait for thee, and thou givest them their food in its season. Opening thy hand and satisfying the desire of every living thing. Righteous is Jehovah in all his ways, and kind in all his works.  He will fulfill the desire of them that fear him, their cry he will hear, and will save them. Jehovah preserves all that love him, and all the wicked he will destroy. My mouth shall speak the praise of Jehovah, and let all flesh bless his holy name, forever and ever. Psalm 146 Praise ye, Ja, praise Jehovah, O my soul. I will praise Jehovah while I live. I will sing praise to my God while I am in being. Trust not in princes, in a son of man in whom there is no help. His breath goes forth. He returns to his earth. In that very day his plans perish. Happy he whose help is the God of Jacob, whose hope is in Jehovah his God, who made heaven and earth, the sea and all that is in them, who keeps truth forever, doing justice to the oppressed, giving food to the hungry. Jehovah sets free the bound. Jehovah opens the eyes of the blind. Jehovah raises the bowed down. Jehovah loves the righteous. Jehovah preserves the strangers. The orphan and widow he relieves, and the way of the wicked he subverts. Jehovah will reign forever, thy God, O Zion, to all generations. Praise ye, Ja. Psalm 147 Praise ye, Ja, for it is good to sing praise to our God. For it is pleasant, praise is becoming. Jehovah builds Jerusalem. The outcasts of Israel he will gather, the physician for the broken in heart, and he binds up their pains. He counts the number of the stars. He calls them all by their names. Great is our God and of great power. His understanding is infinite. Jehovah raises up the lowly. He humbles the wicked even to the earth. Answer, Jehovah, with thanksgiving, sing praise to our God with the harp, who covers the heavens with clouds, who prepares rain for the earth, who makes the mountains put forth grass, gives to the beast his food, to the young ravens which cry. He delights not in the strength of the horse, nor has he pleasure in the legs of men. Jehovah has pleasure in them that fear him, in them that hope in his mercy. Praise Jehovah, O Jerusalem, praise thy God, O Zion, for he has strengthened the bars of thy gates, he has blessed thy sons within thee. It is he that makes thy borders peace, he satisfies thee with the marrow of the wheat. He that sends his commandment to the earth, swiftly does his word run. He that gives snow like wool, he scatters the whorefrost like ashes. He that casts forth his ice like morsels, who can stand before his cold. He sends out his word and melts them, he causes his wind to blow, the waters flow. He makes known his word to Jacob, his statutes and his judgments to Israel. He has not done so to any nation, and his judgments they know them not. Praise ye, Ja! Psalm 148 Praise ye, Ja! Praise Jehovah from the heavens, praise him in the heights, praise him all his angels, praise him all his hosts, praise him sun and moon, praise him all ye stars of light, praise him ye heavens of heavens, and ye waters that are above the heavens, let them praise the name of Jehovah, for he commanded and they were created, and he made them stand forever and ever, he set abound and they shall not pass over. Praise Jehovah from the earth, ye sea monsters and all deeps, fire and hails, snow and vapor, stormy wind fulfilling his word, ye mountains and all hills, fruit trees and all cedars, beasts and all cattle, creeping things and winged birds, kings of the earth and all peoples, princes and all judges of the earth, young men and also maidens, old men with children, let them praise the name of Jehovah, for exalted is his name alone, his majesty is above earth and heaven, and he raised up a horn for his people, a praise for all his saints, for the sons of Israel, a people near to him. Praise ye, Ja! Psalm 149 Praise ye, Ja! Sing to Jehovah a new song, his praise in the congregation of saints, let Israel rejoice in his maker, the sons of Zion exult in their king, let them praise his name in the dance, let them sing praise to him with timbrel and harp, for Jehovah has pleasure in his people, he beautifies the lowly with salvation, let the saints triumph in glory, let them sing aloud upon their beds, praises of God in their throat, and a two edged sword in their hand, to execute vengeance on the heathen, punishments on the peoples, to bind their kings with chains, and their nobles with fetters of iron, to execute upon them the judgment written, it is an honor for all his saints, praise ye, Ja! Psalm 150 Praise ye, Ja!  Praise ye, Ja! The end of Psalms 136 through 150, and the end of the book of the Psalms, revised by the final committee of the American Bible Union. Recording by Mark Penfold, completed October 12, 2009, Lincoln, Nebraska.