We, the team members of IT Bombay X platform upgrade and test automation, i.e. Hershoni, Prachi Pandey and myself Shreya, are here to present the work done by us during the course of this intern.  IT Bombay X is an online platform to offer massive open online courses for individuals and OpenEDX is an open source learning software platform and IT Bombay X uses OpenEDX platform. Right now IT Bombay X uses Ficus version and our task was to migrate the test data from Ginkgo to Hotron to Ironwood and the motivation behind doing this migration is to utilize the new functionalities provided by OpenEDX. So now Hersh will explain what is data migration. So for moving from one platform to another we need to migrate the data from old release to the new release. So the data migration process is basically done in four steps. As mentioned in the slide, the first process is selecting. So the selecting process selects the two releases from which we are migrating. We migrate from old release to the new release. So suppose we are migrating from Ginkgo to Hotron so the old release will be Ginkgo and the new release will be Hotron. So selection is done. Now we need to take the dump of the old release so that we can restore the dump into the new release. In preparing phase we take the dump of the old release and we keep it in a zip format so that we can copy that file to the new release and extract those dumps. In the extracting phase those dumps are extracted and restored into the new release. Now we have an exact copy of the old release into the new release. Now we can apply the steps of transformation so that we can transform the data according to the new release. So this way we will do from Ginkgo to Hotron and then from Hotron to Ironwood. So these versions have several features so Prachi will explain the features. The motivation of our project is to utilize the new features provided by the new versions of OpenEDX. So starting with the features of Ginkgo, course navigation that is you can view the sections and subsections within a particular course, etchlist video playback that is whenever a video other than a YouTube video is uploaded to the cloud then it is played using HTML live streaming, bulk email suppose you want to send to an email not to a particular individual but to a group of people then you can utilize the bulk email feature and drag and drop question a new variety of question has been introduced. Now coming to the features of Hawthorne release, whenever a learner has completed a particular section of a course then it is listed by a green completion check mark. To insert new file, drag and drop options are available. Suppose you are a learner enrolled in a course so every week the main highlights of the course will be sent to you via email. In your profile in the IIT BombayX platform you can also add the links to other social media accounts like LinkedIn, GitHub, etc. and whenever you post any question on the discussion forum and someone replies to it then you will get a mail which will redirect you back to that reply. The new features which have been introduced in the Ironwood version are studio login via the LMS. Earlier in the previous versions LMS and CMS both used to have separate logins but with the advent of Ironwood now there is a common login to both like in order to the login to the CMS you need to login to the LMS first. Checklist. Checklists are made from the point of view of the staff managing a particular course like suppose you have completed a section of a course and now you don't want to add any other content to it so you can mark it with a checklist. Public courses, this is a very fascinating feature like suppose you are not logged in with your ID or suppose you don't have any account on IIT BombayX. So if a course is public then you can visit its sections, subsections and all its contents. Now the next phase, the starting phase of our project, the analysis of different databases that are present in various versions will be told to you by Shreya. The analysis was done to compare the databases of various versions of OpenEDX. So what we did, we used the information schema database of MySQL which is a virtual database and it provides you with the information of all other databases, all the names of all tables present in all other databases with their schema and all the rows. So what we did, we prepared an Excel sheet of all the versions that is one of Ginkgo, one of Hotron and using the advanced Excel features we directly compared both versions like you now you have an idea that which tables are present in Ginkgo, which tables are present in Hotron, what is missing in Ginkgo and what is present in Hotron. Now let us begin with actual migration part. So the migration is divided under four sections. In section one, we prepared a schema report of the old release. Section second comprises of the Django migrations which we'll discuss in the further slides. And section third is same as section one but it contains the schema report of the new release. In section four, that is after migration what we did to confirm whether the migration is successful or not, we compared the reports generated in section one with the section three. Now I hand over the mic to Harsh. In report generation phase, we'll be getting the data entries from the version in which we are working in the old release. So we'll be creating and before folder. So we'll be using shell scripting to get the rows in the tables. So by this way you can get the entries present in the table. Now we also need to compare the schema of the table. So for that we'll be using the column definitions and the constraints in the table and we'll be making the report of that. After the report has been made, we'll be performing the actual migration steps which is mentioned here. So we'll be taking the dump of the old release by using the commands MySQLDump and MongoDump in the old release after stopping all the services of the edx platform. So we use the shell script for taking the dump of all the databases present in MySQL and Mongo. So now this whole dump will be stored in a tar file so that we can copy that tar file to the new release. After the dump has been taken, we'll be restoring all the databases of Mongo and MySQL using the commands mentioned above. After this restoration phase, now comes the transformation phase. So we'll be going to edx platform and we'll be dropping the dj cell retables. So why we are dropping this dj cell retables? Because we need to stop all the asynchronous process that are happening in the platform. So as we have already stopped the old release, it is safe to drop them as these tables don't contain any data and also to stop the asynchronous process. So we'll be dropping those tables using these commands. After the table has been dropped, now we need to perform the actual migration on the new release. So we need to make sure that our VM is connected. That we'll check through links command. So now we'll make sure that our VM is connected to internet. After that, we'll be having a native.sh file in the platform. Earlier in Ficus version, we used to have sandbox.sh which has been renamed to native.sh from Ginkgo. So since Ginkgo, we are only having native.sh and we are using that file to migrate all the changes according to the new release. In native.sh, we have several playbooks which downloads the needed processes and tasks so that the migration process can be completed. So we'll run it. So this is the end of the core part of migration. After that, if any error comes during this process, then we need to perform all the steps once again to make sure that our migration is successful. So now migration has been done. So now we need to analyze whether our migration was successful or not. So we need to compare with the expected result which comes from analyzing. So after this section three was to run the report result for after folder we'll do the same as section one, but we'll do it for the new release and change the parameter according to it. So we need now we need to check whether our migration was successful or not. This will be told to you by Prachi. After running our scripts for the old release and the new release, now we compare the differences between them and if the differences are as per our expectations, then the migration is successful. But if the differences are not as per our expectations, then we have to trace back to what are the problems that have been faced by us during the migration that it was not successful and then we have to repeat the entire process again. Now the next part that is automated testing using selenium will be explained by Shreya. This part does the front end testing. So we use selenium which is a portable framework for web application testing. It is an open source web based automation tool for testing. We wrote the python scripts and what selenium does selenium accepts the commands and passes that commands to browser and we did this automated testing in ironwood version. So here are the few examples. The checklist as they are only available to the admin and the staff managing a course, they were available only in the CMS part and not in the LMS. Public courses without getting logged in, we tried to access public courses and were successful to do so. We tried to log in directly to the CMS but the attempt failed and then first we logged into the LMS and then to the CMS and we were successful. The technologies that we learned during the course of our project are working with virtual machines and Sible. Since open edx is a Django based platform therefore we learned Django migrations, shell scripting. Since the LMS databases are in MySQL, so we worked with MySQL and the CMS databases are in MongoDB. So MongoDB as well and automated testing using selenium. These were the non-technical learnings that we learned. Collaboration, communication, information gathering, resilience, execution and adaptability. Future scope will be told to you by Sreya. We did the migration on the test data. Now the same steps can be applied to do the migration on the production data and as new versions of open edx keeps on releasing, so there's always a scope of future enhancements and shell scripts that we use for migration almost remains the same with a slight change in the parameters and the feature testing can also be fully automated. Thank you. What scripts have you written? Sir, we have written selenium script. The shell scripts are already provided to us. So shell scripts, who had written? It was there in the open edx documentation also and somewhere provided to us by our manager. So you have done from Jinko, Jinko to Houghton and Houghton to this, not from Ficus. No, no, we have tested only from Jinko. These migration scripts are given on open edx. Yes, sir. Right? Yes, sir. Given by open edx. Okay. And what were the problems did you face while doing this? Sir, when we were deleting those DJC early tables, there were some foreign key constant that was failing every time. Okay. So we couldn't trace back that error. So did they mention it in their documentation when, so in the migration? Yes, they mentioned that to drop those tables. But we were not able to trace back that error. We also searched a few groups in Google groups also we joined, but we didn't find that error. So I think that. And what is the reason for dropping those tables? Sir, because of asynchronous process, we want to stop all the asynchronous process during the migration steps so that it cannot hinder, it should not interrupt our migration. Okay. And then it will be created. It will be automatically created after running the native.sh file. Native.sh. Okay. And then you did something fine diff. Yes, sir. Okay. So what was that? Sir, so when we're, when, before migration. So that is also provided by open edx or you have written? No, no, that was provided to us by our mentor. Sarita. Mentor. Yes. So you have written that. Okay. So what does it do? It compares the, all the databases in MySQL and creates a report about it. How many entries are there in the table? What are the constraints that are used in those tables? And it gives a list of all the tables present in a database. And the same is done in after migration, after the migration. So we compare those two results. So we, we get to know how many. How do you compare? Using the diff command or we can also compare using tools like meld. Okay. So we will get the, how many tables are upgraded? How many tables are deleted? No, I think I'll be interested in seeing the output, not how the script is written. Okay. So, okay. Is there an output? So when you did that diff, what did it give you? This was the output of a fine diff. What does it tell you? So it tells about the changes in the database that have occurred, that have been seen. How many entries have been added? How many entries have been deleted? Okay. Thanks. Thank you.


Hello everyone, this is a final update for the 18 CCEM tablets. We wanted to share some of the last modifications that we made to the device and to the applications that you'll find on it. So as you can see, once again this is the box that we distributed them in and again it features the 18 CCEM logo as well as our partner logos. And as you saw in our last video updates, you can see that we've inserted a quick start guide and this contains some basic information on how to's about the device. And of course inside is our tablet and for the purpose of the video we've actually already booted up the tablet as you can see from the indicator light. So we'll just unwrap it and tap the power button, look at the screen. So some of the icons that you see will look familiar from our last videos. We have our applications folder in the corner and this links you to default applications such as the browser and the camera. And next to that we still have our ebook droid application and this is a list of publications from Cole and from our partners as well and we have added a few more here so you'll find some additional ones that you can browse. And returning to our home screen we've added two more applications next. The first is the C2P link over here and this actually links you over to the Commonwealth Connects portal and this is an area that the Commonwealth Secretariat has set up to connect people to materials and news that are relevant to Commonwealth education. And next to that we have our notebook application and this was an application that we added as a tool for users to take notes whether at the conference or other events. And you can see that if you click onto the screen keyboard pops up and you can use this to type in any notes and if you click the back button all of your notes will appear in this list here and you can actually share these as well just by clicking the button at the top so you can share them via bluetooth gmail or within the application itself. So returning to the home screen the last two applications on the other end are again familiar from our last two videos there are there the Cole video which again is a video message from Cole president Asha Canwar and once again the last link over here is our publications folder which again we featured in our last video and it contains subfolders for Cole publications, Comsec publications and UNICEF publications so again you can browse through these to access a full collection of materials and we have added some additional materials as well so they're all there for you to browse. So once again we thank our partners for contributing to this project and we're pleased to share that the tablets were well received at the conference in generated a lot of productive discussion so thank you once again.


Okay, guess we'll get started now. Welcome everyone, this is a state-lead state machines with Rachel, thanks for coming to my talk. I guess I have a couple to choose from, so I appreciate that. I'm a little jet lagged, so if I seem like I don't know what I'm talking about, that's probably why. Who here has heard of Rachel before? I worked with it just so I know who's gonna go online. Okay, do it. Okay, all right, we'll just get started. So the goals for this talk are to convince you that Rachel's worth trying if you haven't used it before, and to give you some intuition about how it works and to show you how to set it up so that you can kind of do some basic parsing with it. Just a little bit about me, my name is Ian Duggan. I'm into a lot of different things. I play hockey several times a week. This is Storvey the Pig, where I come from North Carolina. Pig's mean hockey. I play guitar. I play banjo, mandolin, ukulele. I'm a fiddle, but gathering dust, and my cats actually appreciate that. Sometimes I fly. I've got a Cessna 120 that I fly out of Hayward, California. It's got a whopping 85 horsepower, but it's fun to put around the bay in the valley with that. And I love my cats.  This is Purrington, we call him Bufus. This is Minnie. She's a goofball. She likes to wear outfits. She doesn't care. She thinks they're fun. This cat likes being cozy. These ones are dopey. Jill. Minnie likes to sit in strange places. So I'm a software guy. I code things. I code the internet and the Googles. I'm a recovering technology entrepreneur. Been in and out of startup institutions for many years now. My current status is that I work for Twitch. We do online streaming, social video games. Obviously we're hiring. We have lots of Ruby, the whole front end of this Ruby on Rails. The back end is getting redeveloped and to go for a lot of our services, because we just need a performance. I've been using Ruby for quite some time now. I started on the 1.6 days, a friend of mine gave me a 1.6 book and I was kind of playing around with it. I thought it was a pretty cool language. So I don't know when the transition happened, but I've just kind of stuck with it since. But enough about me. This talk is about Rachel. Rachel's is a really cool tool and it's used in a lot of sort of high performance text parsing areas and some people seem to have discovered it and are using it for stuff like that. And there's a lot of other use cases where people are kind of fighting with regular expressions where I think Rachel could be really helpful. It's got a lot of features that allow you to do some kind of really cool tricks with it. And if you don't have it in your bat belt, I'm hoping that you'll add it to that. As I mentioned, there are many projects that use Rachel. Mongrel Unicorn and Puma are kind of the ones that people know about the most. Zed Shaw I think kind of popularized it about 10 years ago when he rewrote the Mongrel HTTP protocol parsing API with it. There's a really cool gem called Whitecork which actually parses Ruby using Rachel. The male gem, Redcloth, Hverkot, Kirk, and they all kind of make use of it. So just really quickly, what does Rachel look like before we jump into it? It says DSL, you can kind of define some actions and then you've got some ways of converting different terms into other terms. So that's kind of what it looks like and we'll get into the details here shortly. So most people are familiar with regular expressions. Regular expressions are easy for the most part. People understand them. It's kind of the basic stuff that we all understand. Matching, grouping, alternation, matching zero, one or more, end times, end times, et cetera. Ruby has really great tools for regular expressions because it has a heritage from Perl as Zed and Locke which relied heavily on this. So you can do fancy things like match stuff and in a block, match some more stuff and in a block replace something that's three regular expressions down. So you can get pretty fancy with them. But at some point, they can become irregular expressions. So if you've ever opened up a file and stared at something like this, then Rachel might be something worth considering for you. They have their place, but there are better tools. Sometimes you want more control and I posit that this means you want some sort of automaton. So just a little bit about automatons. Like finite automata, they're basically these devices. They have states and transitions. Many people probably see these. You've got circles and arrows and some input comes in that cause you to transition to another state. There are two kind of basic types. There's a deterministic finite automata and non-deterministic finite automata. It's just kind of an academic distinction. Deterministic means it can just be in one state at a time, non-deterministic means you can be simultaneously in two different states. However, you could map those multiple states as super states or the single states and convert that into a deterministic finite automata. So in academia, they talk about this equivalence of regular expressions NFAs and TFA's. So all this stuff that you're doing with crazy regular expressions you can kind of do with these other tools. So you should choose the tool that kind of makes the job easiest for you at that point in time. Excuse me. So, as a superset, these are all state machines. State machines are an important tool in computer programming. Rachel's a wonderful tool for creating them. And state machines are everywhere. And they're in your stoplight. Yeah, green, the timer goes to yellow, goes to red, goes to green, goes to yellow, goes to red, goes to green. Round and around forever. They run your CPU. If you imagine your computer like every single bit that's in it as kind of a definition of a state then every piece of input causes it to transition to another state, your computer is one gigantic state machine. There's some kind of non-state machine-ish stuff that happens on the edge when inputs come in asynchronously and stuff. But for the most part, you can think of it like that. It's a big working state machine. And there's other examples everywhere. So I've watched it as vending machines, I've mentioned traffic lights already, barcode scanners, pretty much any piece of machinery these days that isn't analog. Has any kind of computing device in it? It's gonna be a state machine or something. So I think they're the cat's meow. I think they're great for many reasons. They're simple to understand and there's a great deal of research around finite automaton state machines. With the right approach, they can also produce code that is faster, easier to maintain and correct, more correct than that's most secure. They're very specific about what inputs are accepted and what aren't. And it's harder to get it to parse something correctly than it is for it to admit something that doesn't parse. Which can be very useful. If you're still not convinced, you should probably just spend some time looking at what they are. So let's go over some vocabulary. So state machines, they all have something called a start state. It's usually designated as zero. This is as it sounds like the first state that the state machine starts at. They also have kind of on the other side an accept state or a final state. They can have multiples of these just depending on what they are. And this, when the machine transitions to this state, it is said to have accepted the string. Excuse me. So within a state machine, to go from state one to state two. In this case, when an H is received, you go from state one to state two. It's called a transition. It's usually an arrow in the diagram. It's labeled with an arrow. Sorry, it's an arrow in the slip. There's a special kind of transition called an epsilon transition. This allows you to go from one state to another without taking any input. So this can be useful to wire like the out. You can take two state machines, stick them together, and basically it works like a concatenation like you don't need input to go from one to the next. So with what we know so far, we've got a couple of simple state machines that we can build. So like if you think about a regular expression and how it's parsing things, this is kind of what it looks like. So A star will take any number of A's and just kind of keep mapping on it as long as the first one was an A. The second machine just maps a single A and then you're done. This next one needs an A and then as many A's as you want after that point. So that's A plus. And from that we can get more complicated. So this is a machine that will match zero or more hellos. So the start state is actually a final state. So that's acceptable. But once we've taken an H, we need an ELLO before we can get back to a final state again. So whenever this isn't a final state, it's accepting the input. So you may be asking, what's the big deal? I mean these just look like regular expressions. I've done this before. I don't really need this. So what is Rachel exactly? Rachel is a finite state machine compiler with output support for C, C plus plus, C sharp, objective C, D, Java, O, Camel, Go, and Ruby. So if you're doing stuff across projects, it's really useful. It also supports a generation of several different ways of working its way around the states. And it's really useful for building electrical analyzers and protocol definitions, et cetera. So I talked about state machine generation. So another unique feature of it is that with these states and with these transitions, you can attach arbitrary code at any point in time. So as you're moving around the state machine, you can pull things in and out of the input and jump around in the state machine. And these are the features of Rachel that make it really different than just a regular expression. The regular expression, you can match a string or you can't. With Rachel, you can match half the string, jump to the end of the string, do some stuff, jump to the beginning of the string, give it some more data, download something, upload it to YouTube, and come back. You can do that all inside Rachel if you want to. Not sure I would recommend that though. So I'm actually wasn't sure how to pronounce it. I've been saying Rachel forever.  So you can get the answer from the horse's mouth because Adrian Thurston who created it, actually, I guess sent an email, some time ago. And he said that it's actually Rajul, which is not at all how I guessed. He said it was based on his R and L for regular language and his nickname was Aj, even though it was Adrian or maybe Aj, I don't know. So maybe it's Rachel, maybe it's Rajul. If he's Aj, it would be Rajul, but he wrote Rajul, so I still don't know that. But if you can help, please let me know. So darn, I've been pronouncing it wrong for quite some time. So on a more basic level, Rachel is a DSL for creating these state machines, and it's especially useful, as I mentioned, for parsing protocols, data formats. It's incredibly powerful. It allows you to do this in sort of a EBNF kind of format. So you get this really declarative approach to parsing text as opposed to kind of while loops and for loops and function calls and returns. So let's just take a look at the DSL, get a better idea of it. So the general structure of a Rachel file it'll be written in the host language, so if you're in Ruby, it'll be mostly Ruby, and they'll have some sections in there which kind of call out pieces that are Rachel. So this is a machine which matches Fuba. You can name several machines and refer to them inside your file. There are definitions and instantiations. So this defines a piece of state machine that you can use and you can compose it into a bigger state machine. And to actually generate states, you need to have an instantiation of some sort. This is done with colon equals. And like any kind of programming system that you can include and import and do things like that. There's some slight symmetric differences in terms of what those mean, so you can refer to the manual for more details. White space is ignored. Coddles are started with a pound just like in Ruby. There's different kinds of literals. There are straight literals, regular expression literals, grouping literals. Got kind of standard escape characters that we're familiar with. And if you need to jump, if you're inside a Rachel section, you need to jump out to the host language you kind of put inside braces like this. Numbers can be specified as either integers or hexadecimal, which is useful because when you're parsing in Rachel, you're actually looking at the binary value. So if you're looking at an A, it's actually a capital A, it's actually 65. So you'd be aware of that. You know, it's got a couple of keywords. So these are kind of first section here. This is the stuff that is more like a regular expression. So this matches the word simple. So nothing too fancy there. We can have unix expressions similar to regular expressions. We have zero length matches. We have numerical literals, as I mentioned before, like this is matching 42. So this would be the ASCII character represented by the number 42. And you can have regular expressions inside Rachel, which is, could be a little bit line bending, but regular expressions actually compile down into state machines. So this is actually pretty obvious when you think about it. We have range expressions. So if we want to match A to Z, same as regular expression A to Z. This is where we kind of start to jump out of what you can do in regular expression. Here you can kind of name something called secret code and refer to it in a different place inside the regular expressions. You can name it once. It'll be used in five or six different places.  Any ASCII extend printable and control graph.  You don't have to define it all in front. So at this point, we sort of have the basic building blocks for building more complicated machines. These are like levers, wedges, wheels, pulleys. These are like the simple machines that mankind has built. And on top of that, we built rocket ships and got to the moon. So that's what we're gonna do next. Here's another cat picture. So when I was first learning about Rachel, I kind of heard it was useful. I wasn't quite sure why. I was reading through the user manual. That's like, if you really want to understand how it works, that's the best way to do it. There's a really good PDF online about that. But it has this really interesting thing where you can take a really complex machine and add it to another machine or subtract it or do an intersection or a difference or a union. And it will do the math to kind of add or subtract the state so that it does what you expect. This is kind of my blog when you see it in process. Compositional operators are the things I just mentioned. Union, intersection, difference, strong difference, concatenation, things start. And we'll go and check it in detail in a second here. So a union, you would take, this string up top could be matching, this could have a thousand states in it, one on the bottom could have a thousand states in it. The state machine will traverse both machines at the same time and enter a final state. But it does this by adding, like adding together all the states and lining up the final results. Sorry, the final states inside it. Bugging my slide here, simple union examples, which are on it. But we have an intersection. So that should be, matches any string that would be in both machines. So this is an example of that. It's taken the machine on the left here and the machine on the right. And it is merging together into kind of one mega machine. So we can do a difference. So this would take any string, the first machine matches, minus any string in the second machine matches. This is the state machine that you get from that expression. So as you can see, it's pretty complicated. And coming up with that yourself would be a mess and trying to do that code would probably be a mess. But it's actually pretty simple here in the DSL. You have a notion of a strong difference. This will match any string in the first machine, sorry, any string of the first machine that does not have any string of the second machine as a substring. So one example would be I want to match any character except control character turn line feed. Character turn line feed is a compound string. So you have concatenation and this is just taking the epsilon states out of one machine. Setting up epsilon states which come out of the final states of the first machine into the second machine creates like a global machine. Concatenation example. And you have other things that relate it to regular expressions like the clear start. This is more basic repetition. We've got the optional operator, optional example. So this would admit any string that the machine doesn't match. Negation example, sorry, I'll give you a second. Special form of character level negation. If you just need to get negate on a single character, there's a special operator for that, that's this. When you go through the work of putting these machines together and feeding them into Rachel, it does a kind of final process called state machine minimization where it does some optimizations of the states within the state machine and at any of them that are redundant it can kind of subtract out. So you may have a thousand, you may initially have a thousand states that you've kind of added together that could subtract it down to like 250 equivalent. So I mentioned before, like one of the powerful things you can do with Rachel is attach user actions to different parts inside the state machine. So this is some examples how you would do that. You can define an action statement inside the DSL and then attach them to the transitions. So there are different kinds of transitions, they're entering transitions, so I'm going into a state. You can attach there, you can attach to a finishing transition which is coming out of state. You can attach to every single transition or you can attach to just the leaving transition and you would do this, you would choose based on kind of what you're doing which one of these you want to do. And the embedding operators can get really fancy, there's like two or three pages in the manual about how to do this. I need more details, I just suggest you check there. One of the problems you can come up with is this notion of non-determinism. So you may have two machines that are stuck together and you can't get out of the first state to match on the beginning of the second. You can't get out of the first machine to match on the beginning of the second machine because they're matching the same characters. So an example would be this machine here. So the new line and the new line in white space will prevent the final new line from matching. So the solution is to exclude it from the beginning. There's also ambiguity problems. Like if you're trying to match, if you're trying to match like a C comment for example, you might say, okay, match slash star, give me anything and then give me slash star. But you'll never match the slash star because you're matching on any. And you can do things like, okay, match on anything except slash star and kind of add things together like this but that gets up late. Rachel fortunately gives us some tools that are easier to reason about. So you don't have to kind of do the second stuff. The way it does this is it allows you to set priorities on the states. So you could say that I want the transitions like going to the second machine to have higher priority. So one of the techniques it gives you is a special notation called guarded operations and these are called guarding concatenations. So an example of the one we just looked at, how do I match say on a C comment? When you use this kind of colon greater, greater, this causes the slash star in this final machine to match more strongly. So I'm going to match on any as long as it's not a star slash and then it'll kind of come out of the machine. So we've got, there's a couple of different forms of that whether like kind of the right machine is stronger or the left machine is stronger. Or if you want to along this match but you don't want to kind of stay inside it. So one of the things that you, so that's kind of for simple matching but a lot of times we're trying to build a scanner. Like we want to scan a file we just want to recognize tokens in it like one after another after another after another and have it keep going. So Rachel has a notion of a scanner and this will keep matching on the input and it will take the longest match that it can and recognize that time after time. So an example of that right here would be like, so if you're kind of like this is like parsing headers. So right here we're saying a header, like within a header I can match on a word, a space or a new line and this is what I'm going to do for each of those. So it's going to keep matching on a word. There's a space that's going to ignore it. There's a new line that's going to effort which is actually kind of coming back to work on an F call. So as I mentioned it's really useful for protocol parsing. This is what ZXA did and that's what Puma uses it for. So if you want like a good detailed example of how to like parse HTTP using Rachel Puma I think is the best resource. And up to this point we kind of been thinking sort of on the level of regular expressions but on the lower level these are just states of transitions and Rachel actually lets you kind of go down into what it calls the assembly language of state machines which is just saying I want these states, I want these transitions, I want these characters to go from one to the next. So you can also define those. So this is an example of parsing XMLC data body. So this actually defines three states start one and two and some to transitions based on what it's matching to go from one state to the next. And you can use F call and F return to kind of set up function calls within Rachel which is useful as well. When you start parsing some of the Rachel you're probably gonna reach for it like instead of running a scanner or Alexa you're gonna say oh maybe I can use Rachel. One of the problems with it is parsing recursive structures. Regular expressions are not good for that and Rachel's equipment to regular expression more or less. So how can you do that? The trick is to take advantage of these actions that you can embed and when you recognize within your parse that you are looking at sort of a nested construct you can in your host language set up like a stack push a context to that and jump out and jump back and look at your stack when you need to. So because you've got these embedded actions you can like step outside of what regular expressions allow. You can also implement look ahead. So the trick here is you would match on a couple of characters farther than you want and then you can call F hold which will kind of back the parser up for each time you call it backs it up one character. So internally when you're working with Rachel it has all these variables that you're gonna interact with and they represent the state. So there's data which is a buffer which is where you kind of store your strength. So we'll see in a second here like when you call Rachel you basically set up data in a buffer you set a variable called p which points to the front of the data pe which points to the end and then you just tell Rachel to go and it'll match on everything between pe and pe. When more data comes in you kind of tack it on to the end move pe and pe and like tell it to go again. So if you want to match one character at a time you can just line up pe and pe and you can kind of move them along. And when you're inside a scanner it's actually uses some other variables ts and te because it's in order to do the scanning and do a longest match it actually looks ahead and stores some state on a stack and backtracks what it leads to. So roughly you know you start at state zero you feed it data it runs the exact loop the characters move through the state you consume this p to pe data and it's done if your current state matches a final state which gives you a list of them means that you've admitted this drill. For scanners it's similar except it uses ts te and a stack. The way you extract a string when you're so this is like as you're moving through the scanner like you'll use this kind of slash mark that marks the beginning of something you're interested in and this percent would say like I'm leaving the transition that's what I want to admit. So when I enter I store where I was so data p and then when I leave I store data p I store from my mark to my pe I pull that out and I store it into my things or a for example. As I mentioned there's a bunch of different post language languages and coding styles. One of the nice things you can do with Rachel is you can write a generic parser and just refer to all these actions by name and you can have multiple other files that you would include so I could have one definition of HTTP and then my actions I can have my actions defined as C and my actions defined in Java and that's actually what Pima does. You can also prototype something in Ruby you get it working how you want and then go convert it to C pretty easily because you have most of your logic inside Rachel. So the director is that you need to put in your file or in a net which sets up kind of all the state. Data which has all the states and transitions and exec is actually causes it to run that loop that I was describing. So this is an example of some of the code that that generates. So installing it on a Mac it's really simple just brew install Rachel and to generate the Ruby from like a Rachel file you just say Rachel-R for Ruby simple.rl would be the Rachel file in this example and you give it the output. So this will build your Ruby file and we'll take a look at it in a second and debugging these things it's kind of hard to do visually or it can't be hard to do visually so Rachel actually lets you emit these out to a graph is dot file which you can then kind of make an SVG or PNG out of and kind of look at all the states so I'll show you in a second. So this is an example of calling it from Ruby so this would parse the data so take the data we have to unpack it into the binary format that I mentioned from characters. We set the EOF as a variable that Rachel uses to know when it gets to the end.  So somewhere in the actions it's actually pulling things out and storing them and the tokens are right here. So I created a tool here for kind of doing some visualization is, I'm gonna pull it up to figure out how to get onto the screen. Can you see that? And I actually can't see it so I'm gonna put it around here. That's done called as mirror displays, right? Where is my cursor? Lost a window here. So this is actually a little app that I built in Volt which is pretty cool framework. You guys haven't used Volt so you can go to check but it allows me to kind of experiment here and kind of see what these graphs are. So normally to build these I would have to run a couple of command line commands but I just kind of set it up in here so I can do stuff like this. So I've got examples of like the different machines in here. So this is a machine that matches on simple. Oops, sorry, I got an auto update. So this is an example of the state machines that are being created by this. So if I want to match on sim or PLD it would look like that. Or if I want to zero more of those. Let's say I want to one more of those. So I can kind of build these things up and play with it. I have the code for this like referenced here. Pre-sensation window here. So if I want to kind of create a new machine here so I'll just call this new machine. Let's take a look at some examples here. Everybody see that? All right. So this is an example of the machine I, well it's like the machine I showed earlier where it was matching on, this isn't the machine I showed earlier, it's matching on zero more than those. So I've set up sort of a regional parser base here. I'm actually going to take a look at that. So in parser base I have, I'm setting up some variables manually here. I have a way of feeding a data. I have a way of asking if I'm in a final state. I have a way of pulling out the current match. And I just sort of, I have it, I'm putting some debugging data here which I'll show you on the console in a second. But I just have this set up so that it'll read from standard in so I can use it for experimentation. So this is a hello, this is hello parser. And this is, this is the file, this is what the file you work with looks like. And then we take a look at like the generated file. I'm sorry, the hello rb. So this is the generated file and Rachel's doing all this work for you. So this is setting up all the states up top here. This is a pretty simple state machine so there's not much going on here. But this is the init that we saw. This is the perform method and this is the exact that was buried inside of it. So it's setting up all the states. It's doing go-tos and jumps and matches and et cetera. So all this really nasty code which is heavily optimized, so that's why it's so fast. So this particular example here, I'm gonna run this one in a second. One small mark, let's try that next one. Oh, let's comment it out on this one. So I typed h, you can kind of see that it's in current state one now. p is at one and pe is at one. And we have one piece of data at h. I say e, l, l. Our final state here at the bottom is still false. So we're not in a final state, we're gonna press o. We've now entered a final state, so we've matched on hello. And I press h again and we're not matching. e, l, l, but proceeding on that character, I'm gonna step outside of what it knows. I'm not in error state, I'll never match again. So that's an example of sort of a basic example. I have some arg parsers defined here. So there's, I mentioned state charts and not using state charts, so I've one defined here using sort of standard approach. So this is the main part of my parser and I'm just saying I wanna match on an arg, dash and flags or dash, dash, long opt. Long opt, I have defined as a bunch of alphas. Flags would be a bunch of alphas as well.  So the Ruby code is actually pretty simple there. And these are the actions up here. So I call mark arg when it took out of Marcos sort of it. Not really, can't see no. How do I change the color? Yeah, that's what I did. How do I do that? What's that? Okay. Where's my system preferences? Accessibility? Where are colors? There you go. Is that more legible? Okay. So what I'm doing here in the code is when an alpha is matched for its first time I'm kind of marking the beginning of the argument, marking the beginning of the flags, marking the long opts. And then when I get to saw arg, which is what the scanner calls, I'll use that previously marked thing to kind of pull out what was there. So I can do dash, a, s, d, f and see it. And I'll put in flags, s, d, f, so that means I'm not showing that there. So now I can do like a long one. I can say long opt. So matched on long opt. So that's just an example of kind of doing basic argument parsing there. And then I have an example of this in the sort of state chart for it as well. So state chart is a little bit more complicated when you draw it out of paper. It's simple and you can operate, you can do the dot biz visualization of this as well. But I start in a start state here. If I see a dash, I'll go to pre flag. If I see a quote, I'll jump to quoted value. And if I see anything else, I'm gonna kind of start a value arg. So as an example, this is what I'm calling. Val arg is just kind of a word. Quoted v could be a quoted word. Flags would be this and like a long, long flag would be that. So the basic idea here is that you create some buffers and as you match or kind of go around in a circle in a state, I'll keep it pending to that buffer. And then when I leave the state, I'll admit what I want, looking up what I've stored. And, oops, sorry, the Ruby code for this again is very simple. Most of the work is being done up inside the parser. So as an example, the same thing, but you know, I can say, you know, ASDF and then val arg and I got a val arg. I can say quote, quote v and I got a quoted value. Space brings me back to start. I can say dash dash long equals, I got a long quote, quoted and I got a quoted. So I have links to all this code. So after the presentation, you guys wanna like pull it up and kind of like dig through here, figure out what's going on. It's in there. Let me go back to the presentation here, if I can find it in this color. You got an invert, the presentation got closed. So while I'm looking for this, does anyone have any questions? I'm not familiar with what, so TreeTop's all in Ruby, right? Yeah, so TreeTop is, it may be similar, like presumably they may be doing kind of similar techniques under the covers. I'm not super familiar with how TreeTop does things, but this is like super highly optimized. For the C version, it can actually, actually one of the versions compiles it down into just jumps within a while. So like it'll go to the bottom, it'll go to, it'll go to, it'll go to, it'll go to. It's almost like optimized assembly. So I'd be surprised if it wasn't faster, but it probably really depends on this case. No, definitely. It allows you to, it allows you to do chunking. So you can kind of think about your problem in smaller pieces and unit test those and then compose them together. So like as I mentioned, the composition is, like it's two main things that make it really cool. The composition is one of them. Like that's kind of mind blowing. And then the fact that you got these actions. So I can not be in a programming language and then suddenly be in a programming language. Okay. Right, right. Yeah. I mean, if you're, if, does TreeTop do other languages as well? Okay. Yeah. So if you, if you were writing an HTTP parser, you wanted it to work like, you know, quickly on J Ruby, so you wanted it down in Java and also like in C Ruby. So you wanted it down in C. Then this would be a good choice. Yeah. Right. Yeah. So the question is like, how do you know, this is using instance variables and kind of local variables. And so how would you like turn this into kind of like a composable unit? Yeah. So that's, that's actually kind of what I, what I tried to demonstrate here. So I created this Rachel parser base and I have, I have several parses which kind of inherit from it. The default is to use local variables, which obviously it's kind of painful. Part of the DSL, you probably can't see it again. So we invert this. Yeah. So this is, since I, it depends on where you put these, it depends on where you put like these things. So right, I put these inside this object that I've created. So when it, when it refers to app, it's going to be in the context of this object inside the context of this method. And you can remap what it does. So normally it'll just refer to P. I have it referring to app P, because I set up the stuff up here. Hey, there it is. So I showed the demos and that was it. So I've got these resources here. I have the slides up and I've got the, I got the playground code up there. Had some other features I wanted to add to it, but I kind of ran out of time. So I hope that was useful. Sorry if I drowned on.


This is a system that brings law enforcement, not just from trafficking but to the criminal element. It brings a high-tech nature to policing, particularly in relation to number plate recognition. Previously these vehicles used to detect just traffic offences, approximately 7,000 detections every hour throughout Queensland and now police will be anywhere, anytime and they'll be monitoring not just traffic offences but they'll be monitoring criminal offences right from break-and-annes, late-night burglaries, violent offences and linking those crimes to that particular vehicle. And crimes out there, please be warned that police will know where you are at any certain time and they will link you to that crime and I make no apologies for ensuring that police have the resources and the latest technologies to fight crime in Queensland. How many have we got of these, Minister? Look, we've got a number of covert vehicles right across the whole of the state and we've got a number of marked vehicles and at the present time we're trialling it for 12 months and they'll be spread right across the whole of the state so you won't exactly know exactly where they are or what they are doing and just for that criminal element just please be aware that they'll be matching your number plates up to offences that have been committed in your neighbourhood. This is a very significant expansion of some excellent technology previously been used for traffic enforcement only we're now going to move into the broader law enforcement environment so we're going to use the outstanding technology we've got here against all crimes across all suburbs throughout Queensland so this really is an important step forward for the Queensland Police. It's a wonderful innovation for us. This is all about being smarter about data acquisition, mining the data that we do get and using the capabilities that we have to greatest effect and this is a wonderful example of how technology makes us and helps us to work smarter, not harder. In real terms, can we have a hypothetical idea of how it's going to be used from now on? Yeah, exactly. Look there are going to be certain communities where crime is going to potentially spike might be offenders committing a certain type of crime in a particular area we will deploy ANPR both marked and or unmarked units into those areas so that we can get a better idea of who's travelling repeatedly around those areas if those particular cars are on hot lists as being wanted for questioning or wanted for interception for certain offences that'll immediately come up as the vehicle passes ANPR and those individuals and cars will be intercepted. And it's instantaneous isn't it? It is indeed. As soon as they drive past the car there's an instant recognition that that car and or the driver is wanted police can take immediate enforcement action and by doing that we anticipate we're going to certainly drive crime down. What types of crimes are you going to look at targeting? We're going to look at all crimes, break and enters, evade police, dangerous driving, unlawful use of motor vehicles. Anything where a offender is on a hot list and wanted for questioning by the police will be available to be looked at through this technology. The Queensland Police Service is doing some remarkable trials in relation to new technology that will advance the Queensland Police Service and also put a fear down the back of many criminals because no longer will they be able to drive with immunity around the streets of Queensland. These particular units will pick those vehicles up, they'll link them to the crime, they'll reduce the type of evidence and the police will previously used to take many days and if not weeks to collate. So by doing that it'll increase better enforcement and make our streets and our homes a great deal safer.


should you still become a product manager during a recession and should you choose to join big tech companies for back office or pursue your dream of product management here is my take during a recession most people say oh you know what big tech companies are having layoffs it's very impossible to become product manager why don't just be the financial operation sky for google so at least i have the brand of google behind me and another side of voice said maybe i can join some startup companies because startups are still hiring product managers and some of them also receive significant amount of funding here is my take regarding how to make the choice if your goal is truly trying to become product manager you should join a startup to gain experience any startups that gives you a pm title is going to help you to jump to big tech companies later on when they start to hire again if you do back office financial operations at google it will take you really really long time to actually become product manager because the brand of google or meta is no longer as valuable as before like and subscribe and watch the next video for more product management tips


I want to talk about a particular candidate running for Congress in Georgia's 14th Congressional District because I think she really demonstrates just how radical the Republican Party has become.  This is a party that is becoming increasingly unhinged and everyone should be worried about the rhetoric that they are using because they are inciting violence against American citizens. So an individual named Marjorie Green had a message for Antifa if they chose to show up in Georgia and loot.  President Trump declared Antifa a domestic terrorist organization. I have a message for Antifa terrorists. Say the hell out of Northwest Georgia.  I'm Marjorie Green and I approve this. So if there's a protest and somebody in the crowd decides to get a little bit rowdy and they choose to break a window and try to break into a building, what this lady just said is she's going to shoot them and kill them. In other words, property is more important than human life. When people commit crime, you can argue that we should put them in jail, fine.   I'm going to be the judge, the jury and the executioner. I'm going to unilaterally decide that the property that they are destroying is more intrinsically valuable than that life. This is the pro-life party, ladies and gentlemen. And this is why Donald Trump's designation of Antifa as a terrorist organization is deeply troubling because Antifa is not an organization. You can't see who leadership is in Antifa because it's just the name to describe someone's ideology. It means anti-fascists. It'd be as if you claim that environmentalists are a terrorist organization. Well, which group in particular? There are lots of different environmentalist groups. It doesn't make sense. But Trump knew what he was doing. He wanted to incite violence. And this type of rhetoric is exactly what we've come to expect because this individual, you know, it's not like she is doing anything different than what Republicans in Congress are already doing. Some Matt Gaetz say, why can't we hunt down Antifa like we do the terrorists in the Middle East? We see Tom Cotton agreeing with Donald Trump that we should use the military to violently crush the rebellion happening. So these people are inherently violent. And since she was inciting violence, Facebook decided to take down that ad. And predictably, she decided to cry the victim after she incited violence. And she took to Twitter to demand to speak to Facebook's manager like the Karen that she is. This is Marjorie Taylor Greene. I am the leading Republican candidate for Congress in Georgia's 14th district. And today, Facebook removed one of my ads. And the reason why they removed the ad is because they said it violates their standards.  And the reason why I made that ad is because Antifa has been the extremist agitators that have hijacked many of the George Floyd protests and incited violence on Americans night after night that we have had to endure and watch happen over a week now. You see, I held an AR-15, I'm a legal gun owner. I have a Georgia weapons license that gives me the legal ability to carry a gun and to own guns in my state and in my home. And I also have my second amendment right, which I can use to defend myself if I need to. But Facebook doesn't believe in that and they believe in censoring my free speech and my gun rights by removing my ad. Now the point of my ad was because we aren't going to endure being attacked as everyday Americans having our businesses looted being attacked by extremists and terrorists called Antifa, which President Trump is going to label them a terrorist organization, which is exactly what they are, and we have the right to say it and to speak it and to put out information and share that information that Antifa do not come near us, do not come into our cities and our communities, to destroy our homes, destroy our communities, to incite violence in our communities to loot and burn our businesses and our churches. We have every single right to say that. We have every single right to hold a gun in our hand if we're a legal gun owner, which we are. And Facebook is taking away my ability and my freedom to do that. They remove my ad even though as a congressional candidate, I have spent thousands and thousands of dollars on their platform. As an American, I am guaranteed my freedom of speech and my Second Amendment rights, and they should not censor that on my page. It's wrong. When I'm elected and I go to Congress, I am going to fight against big tech censorship. I will fight right alongside President Trump because I'm sick and tired of conservatives having our views and our freedom of speech violated. You know what? I received over 50 death threats in the past 48 hours since I put that video out in my willingness to stand up against the radical left, the party of lawlessness. Now, why aren't they censoring those people? You tell me. You know what? These people, the left, the Democrats, you know what? They are the party of lawlessness. They want to defund the police so the police can't be, can't be able to protect people. You know what else they want to do? They want to continue to encourage protesters, encourage them to go out there night after night. You know what? Many of these protests have been hijacked. They hijacked the purpose of the protest, and they turned it into violence against Americans, which is a war on America, and I'm not going to stand for it. Let me tell you something. I'm a real American woman, and I will not tolerate this violence on everyday Americans. I'm running for Congress because I'm one of those people that will stand up against the radical left and do what I say I'm going to do. So once I'm elected to go to Congress, you better bet I'm going to defend Americans' freedom of speech from big tech censorship, and I'm also going to defend our gun rights because our Second Amendment is one of our most important freedoms, and I'm a big believer in our Second Amendment rights. So share this video. Let people know we're going to stop big tech censorship. We're going to stand up against radical Antifa and not allow them to terrorize us or attack us, attack our homes, our businesses, and our communities, and we're also going to stop the radical left, the party of socialism, and we will— I shouldn't have to explain this to you because you're running for Congress, but if Facebook chooses to delete your post, that is not a violation of your free speech. You can argue that that is censorship from Facebook, but your free speech rights are not being violated. A violation of free speech would be when the U.S. government impinges on our ability to freely express our views. So for example, when the Attorney General ordered peaceful protesters to be gassed with tear gas to clear the way so Donald Trump can take a photo up, that is a violation of freedom of speech.  She says Facebook censored her free speech. She says as an American, I am guaranteed my free speech and my Second Amendment rights and they should not censor that on my page. And she says when she goes to Congress, she's going to defend Americans' freedom of speech from big tech censorship. Your freedom of speech is not being violated if a private company is censoring you. Your freedom of speech, your first amendment right, is being violated if the U.S. government cracks down protesters, especially if they are assembling peacefully. What'd she say about that? Because I'm still waiting. Is she against what Donald Trump did? She hasn't spoken out, but if you go to her website, she claims that she stands with Donald Trump 100% of the time. Really? 100% of the time? See, I am one of the radical socialists that she fearmongers about, and even I don't stand with any politician 100% of the time. I don't care who it is. Bernie Sanders? AOC? No, because I think for myself.  She's just going to get in the office and do whatever Donald Trump wants. She doesn't have an original thought in that tiny little brain of hers, right?  No fuckface! The government crushing protests is a freedom of speech violation. How do you not know this if you're running for Congress?   Well, I can't really come to any other conclusion, because look at her website. It says she defends the unborn. Okay, well, let me ask you this, Karen. What if that unborn baby grows up to be a member of Antifa?  What if that unborn baby needs healthcare one day? What if the unborn baby resides in Pakistan? Do you think that we should continue to drop bombs on the mothers of said unborn children? I mean, where do you actually stand when it comes to the pro-life issue?  You can say that I am morally against looting and rioting, but to think that killing them, shooting them as a proportional response, you are a dingbat.  I don't know what else to say.   And her key policy platform plank, as if she has much, is to stop the radical socialists. Now it's funny because as a radical socialist, again, I would love to think that the United States government is on the brink of adopting at least one socialist policy, but we're going in the opposite direction. We're becoming increasingly fascistic, which is something that she probably likes. But look at what she says, Democratic socialists are fighting tooth and nail for a hostile takeover of our healthcare and so much more. Oh really? Like what? Radical socialists want Americans on the same government-run healthcare plan with welfare recipients and illegal immigrants. Oh no! Marjorie Green is fighting against these radical socialists and will take the fight to Congress. So what she's admitting here is that tax-paying welfare recipients don't deserve the same quality healthcare that she gets. So what do you want? Like a caste system in the United States where we have the haves and the have-nots?  The United States government is in control by Republicans in the executive branch, in most of the judicial branch, you guys got the Supreme Court, Democrats have the house, but I mean what's the threat? Where's the threat of socialism? It's not on our doorstep, we're not close to advancing towards Medicare for all. The Democratic Party doesn't even support Medicare for all.   So what are you doing? Why are you running for Congress? Especially when you don't know anything. A person who thinks that Facebook deleting an ad is a violation of her free speech but has nothing to say about the United States government using chemical weapons against peaceful protesters is not someone who deserves to be in Congress. That person is an imbecile who should be ostracized from society and not just be allowed into Congress but you know the sad thing is that she's a microcosm of a broader issue because again the Republican Party has become so radicalized that this is the norm. She's probably one of the less insane Republicans, right? I mean look at Matt Gaetz, he's a sitting member of Congress who said we should hunt down Antifa like we do the terrorists. We have the President and a United States Senator saying hey we should maybe use the military against the people they're supposed to defend to crush this rebellion currently. I mean what a joke, this party is so far gone that if this party does not collapse the United States government will. The United States government will because they are literally moving us towards authoritarianism and a police state and they're doing it excitedly. They love it.  They're the ones with all the power. Look you can have a separate conversation about whether or not Facebook acted inappropriately with her. I think that an incitement of violence is a valid justification to delete a post or delete an ad you know as much as I disagree with Facebook but this is a clown. This is not a serious person and all of the democratic socialists who I've spoken to on this channel they have a thousand times more substance than this individual does who is vapid who stands for nothing who literally has six policy planks on her website and some of them aren't even actually policies. Continue building the wall. She wants permanent austerity. I forgot what the other ones were but what a clown. But I wish that I could say that this is just a fringe. The days where Sarah Palin is the craziest Republican those days are long gone. Sarah Palin is now one of the more normal Republicans and this dingbat is the status quo unfortunately. That's where we're at in American discourse where the Republican Party is so far off the right wing spectrum that they make UKIP look like Democrats.


Ah, yes, I have to press, my god, YouTube changed the interface recently, now everything is different. I'm just really awkward. Now it should be... Yeah, it looks... Yeah. Let's see. Why not Arch? Oh, I don't know. Oh! I think it's going. It's going? It's going? Do you see the live stream? Yeah. You're on the YouTube studio. I can hear it all, right? I just had to mute it, because I could hear it all talking. Nice, nice. So, as I was saying earlier, this time, given that now everything should be working correctly, I'm really sorry about that. As soon as people connect, please tell me in the chat if everything is going correctly, and if you do hear the volume of both me and people correctly, or if that should be... should change, I can tune the volume. With that said, yes, as I was saying before, if you could each introduce shortly who you are and what you do for the project. Yeah, I guess I'll go first. So I'm AJ, I'm the desktop lead, and I also manage all the PR for Rhino Linux. So that includes setting up this team interview with Nico, aside from that, I manage the user interface and the desktop of Rhino Linux, and try to basically build the vision that I've had for a long while. I guess I'll go. I'm Elsie. I am in charge of Packstall, which is an AUR-like package manager for Ubuntu that Rhino uses. So I'm in charge of making sure that everything works smoothly, fixing bugs that happen in Rhino because of Packstall and also helping with maintaining some of the core packages that we use. And I'm Oren. I'm the newest to the project, but I also maintain some of the most crucial pieces being the systems and like OS building. So all of the images that ship out, I'm in charge of constructing as well as the core packaging that Elsie mentioned. I'm in charge of getting that stuff upstream and together. And then I work with Elsie to fix all the bugs that happen with it. I'm also the design, so all of our logos and wallpapers have been made by me, and a lot of the integration is where my job is. I would also like to mention there are a few members of our team which could not be present today. Xenia, you may have seen her on the vanilla OS team interview. She created the, like, your system application, which is used to update your system graphically. And then there's also Wizard, who is currently away. He made the set-up Wizard, and then there's Catmaster, which helped with the early development of Rhino Linux. They definitely helped a lot with trying to get the builder done and the installer and then, or an expanded upon that, and added different architectures and devices to be able to work with the operating system. Okay, thanks, everybody. And of course, just for the beginning, can you also briefly explain to those who haven't followed what Rhino Linux itself is? Yes, so I would happily love to talk about that. Rhino Linux is a rolling-release Ubuntu-based distribution. Now, the two of them don't seem like they work well together, but we've found that a lot of developers actually wanted that kind of distribution, its community maintained. And so all of us were not affiliated with Canonical or Red Hat or another Linux company. For example, we are just members of the Ubuntu and Linux community, and we're entirely free and open source. Packstall is at the core of our operating system, and we also use a modified version of Xfce called the Unicorn Desktop. And why is that? So where did you choose to do your own desktop and not ship what was already existing? Okay, that's a really good question. So originally, we did plan to just use a slightly modified version of Xfce, make it look a bit more similar to Windows, and then the idea kind of snowballed from there. I used my girlfriend's MacBook and I was blown away with the user interface. I hadn't used macOS at this point for around 10 years. I was blown away with the user interface and the user experience of it. And then I looked more into other desktop environments, which existed like Qtfish, and saw where they failed. And Qtfish has not been maintained for roughly about a year now. And I had my eyes on that project for a long while, and I was really excited to see it come to Linux because it was a beautiful desktop. It would get a lot of users in, but it just didn't work. And so I looked over to Pantheon, saw what they did, and kind of got the idea from there. The difference is I want to make the user experience like really cohesive, similar to Apple in the sense that everything just works well with each other. And with our collection of devices that we can actually have Rhino Linux installed on, some of our applications came from that like Rhino Drop will allow you to send devices from one Rhino device to another. It also works sending like from Rhino Linux to an iPhone or any device really. It helps build a good ecosystem. Okay, and being obviously the kiddie guy, you know, my question would also be, you do use, if I so correctly, XFC as a base for the desktop, right? Yeah, we do. So why did you choose XFC in the first place? Did you compare the desktops? Yeah, we did originally. So I can speak about this a little more. Yeah, go ahead. I personally use a custom modified Rhino and GNOME experience. And I can say assuredly that the develop branch for Ubuntu, especially with desktop environments, is never stable. Plasma in particular constantly has dependency breaks and uninstalls itself. Likewise, GNOME has dependency issues and freezes up. So we went with the one that we found was the most stable foundation when working on a develop base. I'd like to expand on that a little bit. So with someone asked here, yeah, someone asked, why don't we ship over flavors? And a big reason of that is we have a specific vision for the desktop that we want to put. And we was based with the most foundation to provide a non-breaking experience on the development branch of Ubuntu. The other good reason is that bug fixing every issue that occurs with every single desktop environment is quite a big job for such a small development team. Okay, so and regarding that, why switching Ubuntu to a rolling kind of distribution, especially, I'm sorry, all these parties are still asking why not do something else instead, but I'm trying to understand exactly what's the design goal, right? Why not work on distribution that was already rolling by itself? I can talk about this. So a lot of people like to use Ubuntu, not necessarily something else. A lot of people have just used Ubuntu for their entire time using Linux. And I think we found that it's just a lot more familiar for a lot of people. And if you know they're used apt for a long time, you just know how to just generally get around your system and fix issues like that. So we also found that no other distribution was really trying to do what we did. And we all agreed that we didn't want to just start replicating stuff that already exists. Like if you want a rolling release that isn't Ubuntu, you can just go to like Arch or open SUSE tumbleweed or something like that. So we really wanted to try to make something unique that hadn't been done before and not try to just break the Linux ecosystem into smaller and smaller bits that we didn't feel was required or needed. Yeah. Expanding on that as well, a lot of developers love to use Ubuntu. And that could be for a variety of reasons, like their packages, their package manager, stuff like that. Or just the fact that they target Ubuntu. And I saw this in another podcast, but they talked about how a lot of canonical employees use Arch because they still want a rolling release. And canonical is trying to force one to use Ubuntu. And I found that quite funny. And they said that Rhino Linux could be the bridge between that. And I definitely could see that. But a lot of developers love to use Ubuntu. And we do kind of target ourselves towards developers. Rhino Linux can certainly be used for home use. It's a great operating system for just using your laptop or your computer, for example, or even your phone. But we do brand ourselves towards developers primarily because of our Ubuntu base. I do actually quite agree on that because now that I've been the person that keeps on criticizing the project, I am quite interested in the idea. So if I can share a little bit how I do use Ubuntu. If I can share a little bit, my main issue with Ubuntu is that I develop a lot for KDE, obviously. And very often to actually build KDE, you need the latest packages and such. And very often Ubuntu doesn't have them or they're in the next version. And I usually do always add the repositories of the work in progress next version of Ubuntu, which obviously leads to breakages everywhere. So having a rolling distribution with Ubuntu would actually quite improve my workflow a lot. So I did want to say that I do see the appeal in the project. And obviously the thing is, rolling Rhino had the same concept. What's the exact relationship between this project and rolling Rhino? Is it same developers? What changed? Yeah, so Rhino Linux came is the successor to Rolling Rhino remix. I actually started that project in March 2022. I saw the Rolling Rhino script which turned Ubuntu into a rolling release that was published by Martin Winprex, the desktop lead for Ubuntu, or former desktop lead for Ubuntu. He's not affiliated with Canonical anymore as far as I'm concerned. And so I made Rolling Rhino remix which turned it into a proper distribution.  I didn't really realize people would use it or actually wanted this. I did it for a laugh just because I was bored. And then I was still maintaining something that I made for fun and did not hold it very well. Had a lot of stability issues. And so I kind of killed that project in favor of Rhino Linux which is kind of the culmination of everything I learned developing for Linux. And I brought on people to the team such as Oren and Elsie who certainly know what they're doing. And I believe we managed to kind of execute vision that I should have had in 2022. That's quite interesting. Also, let me say sorry for calling Rhino Rhino. So pronunciation is tough for as poor non-English native speakers. So have pity on me. Now I do know the correct pronunciation. And... Don't worry about that. I grew up speaking two languages. And so... Yeah, I'm still trying to learn all the correct English pronunciation. I did see that you do have... I was trying to fetch them very quickly but I can't right now. You do also, apart from Rhino, have a lot of other personal projects and another desktop I did see. Is that correct? Yeah, I made all the cool desktop environment. It's nowhere near ready for release. It's just kind of something I've been hacking away at for a bit of fun. It's based on a, like, window manager framework which helps you build a tiny window manager. And it's made in Rust. It's buggy to say the best. Honestly, it's not usable. I'm just doing it for a bit of fun, expanding what I know. That was one of my first bigger projects in Rust. And since I started developing heavily for Rhino Linux, that's kind of taken a backseat. Yeah, I can see that happening. Lots of projects. Because I do also recall a window manager and a programming language. Are you very much into, like, rewriting things from scratch to try something new? Yeah, so sometimes I just code out of boredom, to be honest. That's the best way I can explain it. So the window manager actually works towards the desktop environment that I've been slowly building for a bit of fun. But it's not, I'm not ever going to release it as something that I should say people should use. Rhino Linux is my first major project, which I'm actually proud to show off and tell people, yeah, this is something that you should use. Whereas the orbital desktop environment is just made for fun, the programming language I made. I just wanted a deeper understanding with Python. And so I was like, why not write a programming language in Python? It will be funny. Yeah, and that's pretty much. Yeah, and I can attest that, as I also do have in my personal GitHub app programming language written in Python, which does almost work, but not quite. So I can attest that it's quite an interesting overtake. I can may also ask you if there's like any sort of history on how the logo for this distribution was made, like who did it and how? That would be entirely me. And I mostly made it for fun. The logo is actually how I joined the team. I saw the register article about the Rhino Linux team back in October. And I was intrigued and joined the Discord and noticed that the logo was a little lacking. And I felt that we could have a Firefox-like logo that also displayed a rolling Rhino. And so I took the circular shape and the animal style that Firefox does and used a little bit of help from some AI artwork to get some inspiration. And once I had a good vision of it, it's only a few pieces that I just threw into Illustrator and was quite simple to actually put together once I had enough concepts that I had meshed together in my head. OK, so that is actually how we joined the project.  Because I also see you worked with the installer. Is that correct? Yes. So I also work on Ubuntu Touch. I maintain all of the Pine 64 devices on Ubuntu Touch right now. So the Pine phone, the Pine phone Pro, and the Pine tab, as well as the Pine tab too. And so I came into the project with some experience prior. And I saw that there was a cat master was struggling to build the ISO. There was some sizing issues that I had run into myself when I was building my own Ubuntu Touch images. And so very quickly that developed into me making our 64 images for the ISO as well. And then with my Pine 64 experience moving Rhino Linux to there as well so that we could also have a good development platform for those mobile Linux devices. OK, thank you. And did you also do something in a project that I haven't mentioned yet that you would like to cover? I mean, in some big task that is quite user-worthy, something like that? Is this a question for me or for all of us? No, no, I was asking you because after a logo installer I was wondering if there's anything to complete the list if I have been missing something. I'm not omniscient, so. I'm mostly maintain the kernel stuff as well. I'm sure that we have integration there. We don't use directly Ubuntu's kernels, but we use their mainline kernels, which means that we can have the always up-to-date versions of it, which I've noticed a lot of articles have been pointing out. It's great that we have Linux 6.4, which I think all distributions should be shipping the latest stable kernel, especially because that is what Linus's vision is and what I have found has provided the most stable experience. But it also is blocked a lot by how many different unique devices that people have. And so there's a lot of rectifying that I have to do between people's machines and making sure that everything works everywhere. So there's a lot of behind-the-scenes maintenance that's constantly going on. OK, thank you. My next set of questions would be about packages and such. Firstly, I would like to ask you to explain a little bit what Pakistan is, because I'm not sure I quite get the idea behind it. So Arch has the AUR, where you have a repository full of files that describe how to build a package that a tool like Make Package can ingest and then spit out a built package at the end. So I created Pakistan long before I joined Rhino Linux or even rolling Rhino remix. And it has the same concept where we have a community style repository full of scripts that people can add, update, remove, stuff like that. And Pakistan itself functions kind of like an AUR helper like Paru or Yeh, where it's just a nice wrapper that can download the package and then do everything else, like building it and installing it, stuff like that. OK, thank you. I see. OK, so sorry, I'm trying to wrap my head around that correctly. And so the next question would be what is instead Rhino PKG? So that's a script that I made that takes all of the package managers that you have installed like Flatpak, Snap, Apped, and Packstall and turns it into just one program that you can use. So you could do Rhino package install steam. And it would show all the different packages relating to steam from all the different repositories that exist. And you can just type in the number that a package corresponds to, and it'll install it from that specific package format. And it can also search and update all those packages in one go. OK, is there any chance I always wonder about this as more of a designer than a developer when I have to deal with Flatpaks and other ways to install the application when you actually present that to the user who doesn't necessarily know what the difference between each one is? How do you not avoid choice overload and such? Or is that not a goal in this context? It really depends on what package is being searched. We do have a nice little UI that goes through, and it'll print out an index number for each package. And then in parentheses, it'll show where the package is coming from. If there's two packages, one on apps and one in PaXTOL that are both steam and just steam, it will tell you the difference between where it's getting the package from. But as of right now, we don't really have a, I guess, there's no solution that we have so far for just overloading the user. It kind of is what you search. Something important to also remember is that because we are on the develop branch, we are not meant for the first Linux user. We are meant for a more experienced Linux user, someone who kind of already knows what we are talking about. Yeah, that totally makes sense and is why I had that. Or is that not important in this context at the end of the question? Because I do see how that. So I think, sorry, go ahead. I mean, I don't know. Rolling Blazer should be used by someone who has at least experienced the stable Linux distribution first and understands how to use it correctly before they dip their feet into a rolling release. Yeah, that totally makes sense. So I think we're roughly half an hour in. I think it's a good moment to step back a little and get some time to answer the questions that have popped up in the chat. So the latest one, I can very quickly reply to why use XFC instead of GNOME. I think this was covered early on. So it's one of the first things I asked. I also see that. Sorry. I mean, I'll just quickly go over that again. But the main reasons are we have a specific vision for the desktop we want to push. And with such a small development team, it is difficult to manage that. And I know some said that they're basically a one-man distro maintainer and they have different additions for their distribution. And that's really good. But I also see that your distribution is arch-based. We're trying to turn Ubuntu into something that it's not. It doesn't really want to be a rolling release. And to add that extra stability, we've got quite a handful of users already. And we want to give them the most stable version possible. And so to add that extra stability, we're only supporting one desktop environment and making sure that it is as bug-free as we can possibly make it rather than giving a bunch of different additions and spreading our development team to fin. But along with that, you also said that why are we making XFC look like GNOME? GNOME wasn't actually the main part of our design choice. I'd say we're more similar to Cosmic by PopOS or MacOS. MacOS was definitely a big inspiration for the design of Unicorn. Do you also want to take a second to reply, see that you wanted to the comments of Xerrolinux? Yeah, so he also mentioned why we chose Ubuntu rather than Debian as a base. There's a good few reasons for that. I'm sure Oren could answer that a lot more eloquently than I can put it. So most simply, Debian is very not easy for a user to set up and doesn't have nearly as many optimizations to their kernel nor to their packages that provide better user experience fixes. While you can technically install Debian packages on Ubuntu, there is often a lot of patches that Ubuntu stacks on top of that. And we felt that we would rather go with the secure and user-friendly packaging than the most untested version of it. So because Debian already is a rolling release, it has SID, but Ubuntu doesn't have that version. So we are making our own version of Debian SID but on Ubuntu. And another thing to add to that is, well, so like Arch and rolling Rhino, and I guess SID are technically what you would call a rolling release, SID is really unstable. Like I used to use it for a year and I did not like it at all because they basically threw the newest package at you and if it didn't have, and if dependencies were missing that weren't the repos yet, then that's just your problem and it's up to you to fix it. And that really would not have been our goal, which is to be, I guess, more of what Arch considers a rolling release where if a package comes in, it's the latest and greatest, but its dependencies are also in the repos when you add them. So yeah. Also, I'm sure Nico would definitely agree with me, but when you've been developing for such a long time, you don't care about tinkering with your system, you just wanna set it up and let it exist. You don't wanna be fixing broken installs or tweaking this and that to have a desktop that you want. And it's part of the reason why I daily drive a MacBook and I know it sounds ridiculous, but as a university student, I crave that reliability right now. And I do also use Rhino Linux, but I also use Windows and I know Oren also uses all three OSs interchangeably, but for a lot of my development work and university work, I'm using a MacBook just because it's so damn reliable that I don't have to ever worry about it. And that's kind of what I wanna bring to Linux. I've spent three years daily driving it with no other OSs installed. And at a certain point, you just get tired of it, breaking or having issues. And I just want to make Rhino Linux or one, like you set it up and once it's done, it's fine. And I actually wanna use that time to talk about a new feature we're bringing to Rhino Linux in the future. I'm not sure when it will be complete, but it's called Rhino deploy. And the essence is it will index the packages you've got installed and the customizations and modifications you made to your system and then put it into a shell file and you can just go to go install Rhino Linux again, run Rhino deploy import and it will read that file and have your system set up immediately from another system. Think of it a lot like a Nix Flake. And this is gonna be, this is not in the near future. By the way, this is kind of on the horizon for us. Yep, that is quite interesting. And I do have to ask, is there anything else you would share about, you would like to share about the future of the project and other things you're planning, maybe not just on the horizon, but even on the shorter time period? Yeah, I could definitely speak a lot about this. We're currently in the works of publishing a document, which is entitled Unicorn Desktop and the Future. And it basically outlines our vision for Unicorn Desktop, the current state of it and where we want to end up with it as a wholly finished product, even though as a rolling place it's gonna be constantly evolving over time. So one thing that we've got planned in the distant future is podiums. It will work a lot like Mac OS stage manager. I've not seen it implemented in Linux yet and that's kind of bugged me as someone who uses stage manager daily. I definitely want this feature added onto the Linux desktop in some way, shape or form. And since no one else has done it, I suppose we should be the ones to do it. Another small thing would be a replacement for the XFC pop-ups. They look quite jarring when compared to the rest of the Unicorn Desktop and the GTK applications that we ship. And so I'd like to kind of replace them and make them look kind of more similar to the ones you'd find on elementary OS, for example. And I'm sure Oren can definitely talk a lot more about future plans that he's got as well. Yeah, in particular, we've been looking a lot at and we recognize the problem of being X11 and not Wayland because Wayland is definitely the preference of Linux users right now. And I also believe Wayland to be the ultimate concluding future of Linux desktops, particularly with mobile integration with Plasma Mobile and with Fosh. Wayland has been working excellently. Rotation looks really smooth. Scaling works great. Frame rate drop is not there versus with X11. I experienced all of those issues. And so we have an ultimate goal of moving to Wayland, but the problem we have there is that XFC is not ready for Wayland yet. And so we've investigated using XFWM for Wayland as our compositor, as well as the updated versions of XFC components that are at 4.19 instead of 4.18 that do have some Wayland support, but I found that it's breaking in a lot of ways. So we have to wait for 4.20 of XFC, which they have promised will bring full Wayland support. And by that point, we can probably move to Wayland, like everyone else, which will hopefully smooth out a lot of issues that we see right now. Another thing I do want to mention is a lot of the components we use for Unicorn Desktop right now are likely never going to get Wayland support. And so when XFC 4.20 drops, we do need to retroactively fork them and implement Wayland support. And so what my basic rule now is, any new thing that we create will should automatically have Wayland support. And then when we have time, work on like PlankDoc forking that and then making a Wayland version of that instead. And another thing in our future for Unicorn would be to replace XF dashboard immediately entirely, just because it's not necessarily easy to integrate into our system and it doesn't have the look and feel that we want to provide. And we've done some modifications to that and already shipped that out to users. And it looks a bit more coherent with Unicorn Desktop, but I wouldn't say it's 100% there yet. And so we will be making our own replacement for XF dashboard. And one of the first things I will say is make it Wayland compatible so that we can just have it work and not have to backport so many things into our new Wayland version of Unicorn, for example. Okay, thank you so much. That was quite a throughout and we're getting very much close to the 40 minutes mark, so I wouldn't want to go too much past that, but I do have to ask the same question that I asked to end the previous interview and that is, could you please tell me, each of you, your favorite dish? By dish, do you mean like food? That is correct. For me, it would likely... Oh, that's such a difficult question. Right, because you can answer very quickly something about the distribution you're working on, but favorite dish, much harder. I'll take the start here. I love myself, a pastrami in Swiss on Marbled Rye with the side of a nice dill pickle. That was so in-depth. That's quite... You knew exactly how much you wanted. A third of a pound. A third of a pound with two tablespoons of something. A third of a pound of pastrami on Marbled Rye with extra Swiss and the side of a dill pickle. I could have waved at exactly 274.5 degrees for five minutes, no more, no less. For me, I'd say I'm quite a big fan of Chinese food and so when I say Chinese food, I mean British cheese takeout, not American Chinese takeout, there is a distinct difference that I remember about blowing up on TikTok at one point. But yeah, I'd have to say crispy chili beef with chips. And when I say chips, I mean fries. I'd have to go with maybe a Smash burger. There's a really good place where I live that makes the best burger that I've ever had. I'm on a quest to find the best burger. You should come here. I've not found it yet. Well, it's here. I do have plans to visit you at some point. It's just expensive getting to America from England. This sound very good. And I will admit that here it's 9 p.m. and I haven't had dinner yet. So that was probably a very risky question to ask. That said, thanks everybody for joining and do let me say again, I'm really sorry about all the technical issues. I promise usually I don't have them, but apparently today was the day my computer decided to give me issues. I was really happy. Thank you for having us. Yeah, I was really happy to have you here and be able to better understand the project. I didn't see enough articles and explanations around about how everything worked together. It was nice to see a coherent explanation. Yeah, thank you for taking the time. Thanks. Yeah, exactly. Thanks for having us. I mean, I'm definitely thankful we're getting the word out here or answering people's questions publicly. I'm very glad it happened on the live stream. We can't edit anything out or change our answers. It has all manufactured PR. Like this is definitely something that a lot of people don't know, but Linux is a very divisive subject and anyone who talks out about it is quite shunned in the Linux community, but the community at times can be quite toxic. And I found quite a bit of support for what we're doing at the same time and it's really refreshing to see that. And I am grateful for everyone's questions. There are a lot of them are quite obvious like why we didn't base on Debbie instead of Ubuntu. And it's a very good question to answer. And I want to thank everyone for their questions and thank you for the time that you've given us. Yep, that was my pleasure. And obviously, since this is very much audio focused content, I will extract the audio and insert it also as a podcast kind of content for anybody who would be interested in hearing all about this whilst, I don't know taking the bus in the mornings or such. So I will now end the live stream and I will thank everybody who joined to listen to this project. If you have any more questions, feel free to join our matrix or Discord flow. I'm sorry we couldn't get round to everyone. Yeah, please join the Discord or the matrix. We'd be happy to answer any questions. And Niko, thank you so much for having us. We'll just leave it in the comment section down below. Give Niko more YouTube recommendations. My pleasure. And of course, if you send me the link, I can also put that.