Thank you for coming to the second day of the Ensuring Software Quality Track at DevCon to US. The next talk is OpenShift on OpenStack by Eric and Emilio and I'll hand them over to them. Hey, Jadis. My name is Eric. And my name is Emilio. And this summer we worked in Boston on the OpenShift on OpenStack team. And this is our presentation on overcoming development challenges. So, show of hands, who here is familiar with OpenShift? Cool. All right, so for those of you who aren't, OpenShift is an application platform that runs at Kubernetes containers. And it can be used to dynamically scale applications, as well as update your back end on the fly. So it's a really cool tool. And then another show of hands, who's familiar with OpenStack? Cool. So OpenStack is a cloud infrastructure management platform. And it's used to manage and lease resources and hardware. All right, cool. So for our work this summer, we worked on the OpenShift on OpenStack team. And OpenShift can be run on OpenStack. And that's something that we are trying to accomplish. But why, you might be wondering, do we want to do this? So basically, when you run OpenShift on OpenStack, it makes it a lot easier to deploy, manage, and scale your OpenShift cluster. And you can do this by taking advantage of the compute and the networking and storage services in OpenStack. Now, developing an environment that's sitting between two really large technological projects can be pretty complicated. And this is, you know, something that we've been taking on for a little while. But there are working versions of this. For example, if any of you are familiar with the MOC, that is an OpenShift on OpenStack environment. But anyway, so our team works on code related to deploying OpenShift on OpenStack. And we are trying to standardize and optimize that process. So throughout the course of this summer, we've faced several challenges. The first one being adjusting to a new development workflow. That's something that most developers will have when starting a new project. And the more specific to an integration project such as OpenShift and OpenStack was making technology to meet in the middle. And so we're going to dive into each of these one at a time and use our summer projects as examples for each. All right. So first off, we're going to talk about getting adjusted to a new development workflow. And we're going to talk about how the tooling and the technology that you get exposed to are going to come into play here. So first of all, the first thing that you have to do before you get started on any project is obviously communicate with your team. And in terms of the OpenShift and OpenStack team, that can be a little bit more challenging than others, because our team is actually distributed across the world. And so in order to get started, you're going to have to use a tool called IRC. And how many of you are familiar with IRC? Wow, that's shocking. Well, good for you guys. And then so the second fun part about that is since everyone's all over the world, they're not on your nine to five schedule. And so everything that you want to do has to be built around that time zone.  And so as you move along, obviously, you're going to have to eventually hit a point where you ask for help. And the first thing that you have to learn about using IRC is you can't be afraid of the chat. And I think that's something that everybody inherently has at first.  But eventually you reach a point where you realize, you know, there's going to be someone who wants to help you. And if you post an all chat, you're more likely to get that help. So do that. Second of all, people might, you know, always be willing to help you with one problem. And that's fine. But don't just ask for help, ask for resources. A lot of the times, people will be able to link you documentation or things that you didn't know exist to help you solve your problems on your own. And that could just make the development process easier for everybody. So then there comes DevStack. And this is another part of developing for OpenShift on OpenStack. If you develop on OpenShift in general or rather OpenStack, you're probably going to work with DevStack at some point in your life. And basically what DevStack does is it builds a little miniature version of OpenStack in your own local environment. And it's kind of neat because you can build from your own source code and deploy it however you like. But just as anything that software related, there are obviously some complications that come along with that, which Eric is going to go over. So have any of you guys ever set up DevStack before? All right, for those of you who haven't, buckle in. First thing you're going to need to do to set up DevStack is stack it.  It takes about half an hour. Works fine. It's great. But if you want to include services such as like Octavia, which is OpenStack's load balance area, you're going to have to set that up and you can make fun. So you're going to have to do that and then stack it again. That's going to take another half hour. So you already have over an hour of time spent still setting it up. So you add these services, you re-stack it, should be fine. Now something's not going to work. It's been a familial experience. It's like I don't want to exaggerate, but we spent six or seven years setting up DevStack. And so we spent a lot of time debugging, then finally re-stacking again. And this process continued and continued and continued. But we got it and DevStack is a great tool. All right, so now we're going to get into the sub-project within OpenShift and OpenStack that both of our projects were related to and that is career. So first and foremost, the most important part of the slide is this guy. And that is the career platypus. He's unofficially named Carlton. Don't remember that. I didn't tell you that. So basically, what does career do? Career enables OpenShift to use the networking packages and services that are already in OpenStack in place of its own software-defined network. And so why do we want this? Basically, both OpenShift and OpenStack have their own software-defined networking solutions. And the problem there is that both of them we're going to inject their own checksums and their own headers into their networking packages and that makes them very slow to decrypt. So by going around that, you can really speed up the networking for OpenShift on OpenStack. Now, career as a project can be configured to run either in VMs or in Kubernetes. And that allows it to be kind of isolated from your OpenStack environment. But it's not isolated from the software itself, just physically isolated. And there's quite a bit of a tech stack that you're going to have to get used to to work with it. So my project in specific this summer had to do with scaling out the career controller. And so the career controller is a component of career that is responsible for listening for Kubernetes API events in OpenShift and then translating those into the appropriate OpenStack API calls in order to service different networking events that needed resources to be created, modified, or destroyed. And so what that entails is the following. Basically, what we have right now is a high availability mode. And it's an active passive mode. So if anyone's familiar with what that means, basically it means that one node's going to be active and then a series of nodes are going to be passive or idle. And they're going to wait for that active node to fail. And then one of them will take its place. And this is good. It's high availability. But the issue is that it doesn't scale out. And when you start having bigger data centers, it's going to start getting stressed under load. And we don't want that. So we are working instead to develop an active active high availability solution, which means that all of the components are able to handle jobs and pick up the slack of any servers that might have died. And overall, the benefit here is that not only is it highly available, but it also scales horizontally. It scales its performance when you scale horizontally, which is, you know, obviously better. And in order to work with it, actually, let me go back real quick.  That's because in order to run it in high availability mode, you actually have to run it in Kubernetes. And so you're going to have to run the career controller each in its own pod composed of two different containers. And that's where you start running into fun stuff with Kubernetes. So first of all, because both there are two containers running in your pod, it makes debugging a little bit of an interesting situation. And so first of all, when you build a container, I don't know if you guys have built containers before. It's not the fastest process in the world. So don't make the mistake I did at first. You want to be pretty vigorous and test your code before you build, because it will slow down your development cycle a ton. So first of all, read your code. And second of all, take advantage of unit tests. The third thing that I want to mention, which is kind of maybe not for everybody, because I know some people don't use Docker, but a lot of people don't always realize that you can actually use the Docker tools as well as Kubernetes tools. But if you're having a hard time with Kubernetes, remember that you can use Docker's tools and they have a pretty rich tool set that can help you out a lot. But anyway, back to the unit test slide, because that's pretty important. If you work in OpenStack, then you're going to have to work with talks. And I'm sure at least everyone's somewhat familiar with unit tests, but just to go over them at a high level, unit tests are really good for isolating different parts of your code and just making sure that they work the way that you expect them to. And again, what's nice about the unit tests is that you don't need to build your containers or your final product even to use them to test your code. And so if you take advantage of them and you build unit tests to test your code along the way, it's going to make your development process a lot faster. There is, however, one part of talks that I think are about unit tests in OpenStack that I think a lot of people won't be familiar with. And that's Pep 8. So Pep 8 is this fun little test that tests your style.   And I'm not talking about like this. I'm talking like that many errors. So what is Pep 8? What does it do? Basically, it enforces a really strict programming style. And it's kind of used to just kind of keep everyone in check. It makes sure that all of your code within a project is uniform to some degree. And it tries to make sure that it's all readable. And at the end of the day, it's going to make your code a lot easier to understand whether you like it or not. And it's kind of a necessary evil. Now, how many of you are familiar with GitHub? All right. How many of you are familiar with Garrett? All right. So same. So you're also going to have to learn Garrett if you want to work on OpenShift on OpenStack. And the fun part about this is Garrett, you would think, oh, it's a Git tool. How different can it be? Actually, it's really different. So in Garrett, first of all, you don't follow the standard open source workflow like you do in GitHub. You don't fork it. You don't make your own branch. You don't take out pull requests. You actually just clone the main project. And then you use something else called Git review. And then you push your code up to Garrett. And this could be a little tricky to learn for the first time. And there's a lot of little things that go on underneath the hood that you don't really realize. Once another thing you might not notice, the only way you might not know at first is that when you submit your code to check it into Garrett and you don't want a code review, you actually have to go in and give yourself a bad code review and then basically give yourself a minus one to tell everybody, don't review this. I didn't know that. I got code reviewed. It didn't go well. But the upside of Garrett is actually it's kind of neat to, it's a really neat way to manage a lot of projects that are taking or a lot of people that are working on one project at a time. And I think in comparison to Git personally, I like it better as a way to see the various stages in your code and to see either the reviews that people left, what changed, and ultimately just to see what projects are going on in general. And we'll just take a look here. So this is what a standard project commit would look like in Garrett. And you can see here your history and these would be the changes that you've made along the way. And each of those are expandable, which is pretty nice. But when you look at the project view like this, right, you can see all of the projects that are being worked on under one large project and all of their progress, which is, I think, a lot more organized than Git. Either way though, I can't speak for the OpenStack team. I don't know why they chose to use Garrett over GitHub. But whether you like it or not, you're going to have to use it. So deal. So anyway, the grand takeaways here are don't be afraid to ask for help. And if you do ask for help, also remember ask for documentation every now and then. Always try to take advantage of all the tools that you have at your disposal. And ultimately, you're going to bump into some stuff you might not like and definitely won't be used to. But there's probably a reason that it's there. And it's usually there to help you. So, you know, get used to it. So now I want to talk to you guys about making technologies meet in the middle. So being a member of the OpenStack team, the obvious OpenShift and OpenStack, we have to make them meet in the middle. But there's also services within them that we have to adjust to meet them. So this summer, my project was called Watch Endpoints as a Service for the acronym WIS for short. Basically, what it is is a service to speed up career by listening for networking events. So I'm going to talk about what that means in a second. So why do we want this tool?  You only really need to look at that part right there between the career controller and the neutron, which is OpenStack's network control. And basically, you can see that the career controller creates a load balancer and then will continuously pull neutron using the OpenStack API to show the load balancer until it becomes active. So the way I've been describing this through an analogy is imagine you're a chef in the kitchen and you want to bake cake. You're going to have to preheat the oven. The current system in place is that the chef is going to go back to that oven every three seconds until the oven has been preheated. What my tool does is it just has the oven alert the chef once ready. So you can just see by that it's a lot less work and it's a lot more efficient. So how do I do this? Basically, there's this messaging queue between Neutron and Octavia, which is the load balancer service. And RabbitMQ is getting a queue of all the messages as it is a messaging queue. So when the career controller tries to create a load balancer, eventually when it becomes active, Octavia will send a message to Neutron saying that it has become active. My tool will sit and listen to the messaging queue and wait for that event. Once it sees that event happened, it'll tell the career controller. So this makes too much sense. This is the obvious solution. So why isn't this currently in place? Basically, the career guys asked the Neutron guys to add this BAPI and they said no. They said, thou shalt not take events that came not from RabbitMQ for thou would be duplication and we're running shorter on maintenance. So no. So what's the end game? If they already rejected this idea, why am I doing this? Basically, we're tested with career communities and if it behaves as we expected to, being faster and more efficient, they're going to propose it to open stack as a career self-project. And if that goes well, they'll be integrated. So this brings me to the challenge of making two technologies meet in the middle. My listener would not hear any Octavia low events or events. He would hear the four events fine from Neutron, but it wouldn't hear anything from Octavia. So naturally, because it was working with Neutron and Octavia, I figured the problem must have been within my listener. So my train of thoughts, first of all, I assumed that Neutron and Octavia events behaved the same way because they're both open stack services, so why would they not? Secondly, when I would use an open stack load balancer command, it would generate events, but they appeared to look like port events. Like the event type in the JSON return would be port.something. So at the time, I assumed that these were Octavia events because they were being generated by a load balancer problem. So based on these assumptions, there was no way to distinguish these Octavia events from the actual port events. And so this was an issue because if this tool can't sort the events in that manner, it kind of defeats the purpose of it because then you're going to be sending back hundreds upon hundreds of messages back to your career control. So the explanation of all this, we figured out after talking to the load balancers guys is that Octavia doesn't actually emit events like Neutron does. So why use this and why do we assume this? Basically, on the career side of things, where I kind of was, we just assumed that most open stack services emit events, why wouldn't Octavia? On the Octavia team, they thought, why make it emit events? Ironic, which is OpenStack's fair metal phishing tool didn't do it. And it was never a requirement, so it was never enabled, which makes sense. So both sides had reasonable expectations. There was just this miscommunication. So meeting in the middle, how did we fix this issue temporarily? Over here, you have career expectations, Octavia capabilities. In the middle, there's load balancers of service version two. So if this satisfies, if this behaves how we want it to, why isn't this the final solution why aren't we using this? Well, for two reasons. The current OpenStack deployment uses Octavia and also LBOS v2 is deprecated. So right now we're in talks with the Octavia team to get them to reconfigure Octavia to emit events. But for now, we're using this for test purposes. And this isn't the first time the chip stack team's had to meet in the middle. In the past, OpenStack requires that nodes, a node being an OpenStack server, can talk to each other through their host names as opposed to VIP addresses. Whereas OpenStack doesn't need at least support this. So the solution they found is called Neutron DNS. And it's actually a tool that they've had built into OpenStack for a while now, nobody really knew about it though. So just digging deeper and finding solutions to this. So that's the thing. Sometimes you have to dig deep to find a common ground. So throughout this summer, we dealt with a lot of challenges. But we really learned teamwork and collaboration between teams is crucial. There's so many different teams, so many different resources out there, especially the IRC, that it really gives you to make use of them. Take advantage of your resources. And Red Hat obviously is all about open source. So open source means being part of the community. So you really want to take advantage of the whole community. Because you do have that support. Thank you. Do you want to take, pass this around? What does she does? She's asking about Garrett. What specific, wait, what did you ask about Garrett? You can use this one. Get it, questions are fun. Okay. Yeah, when you say Garrett, it sounds like it's like a replacement of GitHub. But you, I think when I use Garrett, I still need to clone the repo. And then I can add the Garrett is like an extra tool for get they can use on a computer. So I'm just confused. We'd be able to just talk more. I was confused about Gary as well. Oh yeah. I think Garrett actually works with GitHub. You can host your code up on GitHub. And I think Garrett is really more for managing different commits and code review. So instead of using like the traditional workflow, like I think for most open source projects in general, what people normally do is you'll fork GitHub repo and then you'll work within your own fork. You'll make a branch and then you'll take a pull request out against the upstream. But when you use Garrett, you don't do that. Basically what happens is you just clone down the, I feel like I'm too close to this. Basically what happens is you just clone the upstream and you can make whatever changes you want. And when you submit it, you use a tool called get review. And get review is going to, I don't know all the details, but essentially it tags your work with something called a change ID. And then all of the changes that you make under that change ID get submitted into one general repository. And then eventually if it's reviewed and accepted, then those changes can be pushed back up to the master to the upstream GitHub. We use Garrett a lot in Red Hat to explain it to you in the simplest of terms. When you use GitHub and say you submit, so you commit something and there are changes requested, you send a v2 with a new commit ID.  When you use Garrett with git, it allows you to edit a single commit. So you can do, because it uses something called change ID.  Hey, you made a mistake. You don't change the commit. You don't change the commit ID. You keep the same commit ID, you keep the same change ID, and you can make changes to the same commit. So when you look back, you have a cleaner history. And in one commit, you can see all the comments v1, v2, v3, just for the same commit. So Garrett allows you to do that. In GitHub, it's just more of a keep sending commit sort of thing. So it's similar, but it's not the same. It gives you more granular control. Yeah, that's very true. Although it's it's fair to say that the GitHub is starting to develop more Garrett like workflow tools because they're they're recognizing that the workflow component of it is actually really important. I had a question. I apologize. I missed the first few minutes. Did you all, did you guys, did you talk about what is the well, two things. The rationale for the work in the first place, like what problem you're trying to solve. If you talked about that, then ignore me because I know I just want to make sure that everybody else heard it. The other thing I'm curious about is what you see coming in the next few months for the shift on stack work, where you expect it to land in, you know, next version or whatever that is. In terms of the first part of the question, we did go over a little bit just the concept of what the shift on stack team is trying to accomplish and how career is part of that goal. That being said, if you want to add to that, feel free to if anyone has any questions about it, please ask. In terms of future though, for my project at least, we're going to containerize it for triple open stack on open stack. And I'm not sure what the next steps after that will be. Yeah, I can. Oh, sorry. I'm pretty sure my project is not ready to get pushed on. It was a lot more research focused. That being said, I think some of the changes that were done are going to be kept. And I think some of the changes serve as an example of what we should not be doing in the future, which is fine. That's part of the process. And yeah, so I think, yeah, also I do know for a fact that in terms of just the active active, I know that the open shift and open stack team has decided to kind of put that aside by now for right now. It's going to be coming. I don't know if it's going to be in the next release, but it's on the menu. Just it's not a high priority. Thank you. Since it might be worth saying that, we have an open stack project. There's a lot of directions that that the team is going in. On one side is just supporting new features, like making sure that we support things like staff and stuff like that. Another part of that is like standardizing the project, the process so that other people can just play open shift on the stack easily and without having to do a lot of customizations. The third part, which I think is what you guys kind of concentrated on is optimization, which is making sure that bits of the code are optimized in this case, networking. So that's what you guys were concentrating on. But we're going in multiple directions with open shift and open stack. If anyone's curious, talk to me afterwards. Right. And also just to give you guys an idea of what open shift on open stack, why we are interested in developing this. Think of it as it's kind of our hybrid cloud offering, because right now, open shift can run on almost any cloud service provider, including open stack. And we want customers who are thinking of building clouds either on-prem or in the cloud to be thinking about using open stack as their under-cloud solution. So, yeah. Well, another question is like, I know that this container module for open stack, I forgot the name. Is it Xun? Or is this like a container module for open stack? But I just can't remember the name. But what's the difference if you know what I'm talking about? I'm not sure. Between using open shift and the one? Yeah. So this is, okay. This is very ancient history. There used to be a project called Project Solem, S-O-L-U-M, which was about essentially making using open stack as a container scheduling system. That's long since been obsolete by Kubernetes. There is another sort of interesting effort in open stack land around what's called Cata containers. Cata containers are actually a fully free and open source implementation of Intel's clear containers, which are really VMs that you can spin up very, very quickly. They're super minified VMs that boot almost as fast as a container. And there's some movement within open stack to make Cata containers sort of a first-class citizen. But at this point, there's no competing with Kubernetes momentum in the container space. We don't think anyway. I hope that answers your question. Any other questions? All right. Well, thanks guys.


So for our first practical example of go pigeon code, we'll take the hangman program we wrote in regular pigeon and rewrite it into go pigeon. And there's not much different, except we basically just have to specify our types. For example, this is the one we wrote in regular pigeon and here's the one in go pigeon. And we say that our parameters start and end, we have to specify that they're integers. And we say that the function itself returns an integer and then its local n is a float. And inside here, another thing we have to do is that our arithmetic operations, any one operation has to have operands of the same type. You can't mix integers and floats. So here, if we want to multiply n by the result of this operation, well, this is returning an int because both ended starting integers. So the subtraction returns an int. But we want to multiply by this float n. So we need to convert, we have to get the float equivalent of this integer value. And we do so with using f as an operation. And then same deal here, we want to take this integer start and add it to n, but then it's a float. So we have to convert our integer into a float. And down here, the floor operation takes a float and returns another float, but our function needs to return an integer. So we use i as an operator to get the integer equivalent of what floor here returns. And the other functions is nothing much different except when we deal with lists, we have to specify what kind of list and we're dealing with lists of strings. And down here in the original code, we just used the list operator to create an empty list, but you have to specify what kind of list. So this is creating an empty list of strings. It's a specific kind of list. Also notice that in our foreach loops and our foreing loops, we have to specify the types of the variables. So here we have to specify that the counter variable i has a type integer. And it's always going to be an integer, but for consistency, we make you write the i there. Same deal in the foreach, the counter, the index that we iterate over is always going to be an integer, but you have to specify it explicitly. But then in a foreach, depending on what kind of thing we're iterating over, in this case, a list of strings, that determines what type this thing has to be. In this case, it has to be a string because we're iterating over a list of strings. Now, understand that in real go, as we'll see later, they don't have a built-in list type. They only have slices and arrays. So the way we would write the equivalent code in actual go is we'd use slices. So here, let me just demonstrate this main function where here we have it written using lists of strings. But here's the code if we had slices of strings, and here I'll quickly comment this. There's a couple differences then. The first is that in this version with lists, the found variable is a list of strings, but it starts out nil because list variables hold references to lists. And until we assign it this new empty list of strings, found is referencing nothing. And you can't do anything with the nil reference. You need to have some actual list referenced first. And then what we do in this loop is we're pushing values onto the list. We're mutating the list referenced by found. We're tacking the thing onto it every time through this loop. With a slice, here the found variable, we don't have to initialize it because its default value is already an empty slice. It's something we can append to. And so down here in the loop, instead of pushing, we use append on a slice. And the critical difference with slices is that when you append, the append operation is not really modifying the found value itself. What found store is still a slice pointing to the same array with the same length and capacity. But append will return a new slice that may point to the same array or may point to a new array if there was no more space in the original array and we needed to allocate a new one and copy everything over. But whether or not it points to a new array, the new slice here will have a length that is greater by one. And that's what we need. So every time you append in this loop, we need to modify found itself. We need to assign the new slice value to found so that each time through, the length of found is increasing by one. So just watch out for this when working with slices. It's a very common mistake to forget to modify your slice variable after doing the append because append is not mutating the original slice itself. It's mutating potentially the array which the slice points to but it's not changing your slice itself. So that's what the code will look like with slices. I'll leave it to you as an exercise if you want to go through and change the rest of this code to also use slices instead of lists. Now let's look at something slightly more interesting and something new. We're gonna read and write files because in GoPidgin we have operators for doing so which we didn't have in regular Pidgin and regular Pidgin, all we could do for input and outputs is print out to the console and read from the console. But now we have a few more operators including open file and right file and also close file. Ignore the fact this isn't highlighted. That's a bug in my syntax highlighter which I'll fix. Anyway, so this program here is going to write some bytes to this new file called myfile.text and let's walk through it. So first off the open file operator. You specify as a string a so-called file path which is specifying the name of a file and also where it's located. There are two kinds of paths there are absolute paths and relative paths and absolute path specifies exactly where on your system the file is supposed to be located whereas a relative path is relative from the directory in which your program is running. So this is a relative path so it matters where we run our GoPidgin program from. So it'll be relative from the directory of your command prompt when you run this program. So this will look for a file called myfile.text. If it doesn't exist it will create it. As you'll see in a real environment in actual Go you'll have more control about determining whether well if they want open file it doesn't exist should it create it or not you have control over those details but for simplicity the operator's in GoPidgin just don't give you these options. Anyway so assuming my file.text doesn't exist this will create the file and what it returns what open file returns is an integer which it'll be assigned to the file here and a string which will be assigned to error. The integer represents a so-called open file handle. When your program runs every file it opens there's a unique integer which is associated with each open file and we need that integer because subsequently when we do stuff with the open file like write to it or close it we need to specify that number, that integer to specify which open file we're talking about. We don't really care what the actual integer is. Generally the operating system when we open files it assigns file handles that start from zero or one or two or three or some low number like that and just increments that's usually what it does but there's no guarantee and we don't really care. All that really matters is that the number should be unique for every open file. Anyway so the error here, the string that gets returned that is an indication of whether or not the operation succeeded because every file operation we do may fail because the nature of input output devices for reasons beyond control of our code it might not work and so the operating system when we do these operations we're making requests to the operating system asking it to like read these files for us and write data to these files for us and it may give us back an error saying that for whatever reason it couldn't. There are many different reasons why it can't do these things. Maybe the hardware failed, maybe when dealing with files maybe you don't have permission to do that operation on that file. Various reasons like that. We won't go into any details here but what we do need to know is that we need to get back some value indicating whether or not the operation succeeded and if it did fail we need to hear about it. We need to know what kind of error it was and so the way the operators here in GoPitch and work is they simply just return a string and if the string is empty then that means the operation succeeded. If it's not empty then there was some kind of error and the string describes what went wrong. So what we're gonna do is after each one of our file operations here we're gonna check if error, if it's not equal to an empty string then something went wrong and we're just gonna print out hey something went wrong and abort the program. Because then this is the kind of error where for the purposes of this program if we can't open this file then our program can't really do anything so we might as well quit. Normally we would expect it to work but even if these currencies are rare like the hardware fails on a rare basis or something else goes wrong on a rare basis you don't want your code to just blithely continue on when something goes wrong, right? You want to try and cope with the problem if you can maybe do something else, maybe try again, sometimes that is appropriate in some circumstances but in this case for our purposes we're just gonna quit. Sometimes you can't do anything so you just have to quit and that's what we're gonna do. Anyway so assuming that the file opened successfully error will be an empty string and it will come down here and it will create this slice of bytes and assign it to bytes. The slice of bytes has three bytes and it's a byte with value 100, two and 255 those are just random values I chose. And then we do a write file operation on the file. This is the file handle integer that we're specifying which open file we're talking about the one open file we have and we pass in bytes, the slice of bytes and it's gonna write out all of the bytes to the file. Well this is gonna try. Again for complicated reasons having to do with details of the operating system and the hardware, a write operation might not successfully write all the bytes you want. It might do a partial write, it might, if you request to write 1,000 bytes it might write only half or 600 or something less than 1,000, right? And so we get back a number, an integer which we're gonna store in n here which tells us how many bytes were actually written and we also get back a string which we store in error indicating whether or not something went wrong. And when a partial write does occur we get an error, we get a non-empty string for error. And so we're not actually gonna really use n here we actually don't really care what it is because a partial write is a kind of error in a sense it's not normally expected it's a relatively rare occurrence and I won't even go to the details. Circumstances under which you get partial writes that they're fairly exotic but they can happen and when exactly that is depends on your operating system depends on various circumstances that we will discuss here. But anyway, so then we're gonna check after the write we check error to see if it's not an empty string if so something went wrong and we print out the message of what went wrong and we bail out. Anyway, assuming that the bytes were written out in full and everything was fine then we're gonna continue on and we wanna write out the string to the file and the way we do that is we first have to get a slice of byte equivalent of the string that's what byte slice does it gets us a slice of bytes representing the data making up this string and we write that out and same deal we have to account for the possibility of an error but assuming there is no error we continue on and now we're done we've written everything that we want out to the file and at this point we could just abort the program we could just quit we don't need these remaining lines of code but we're gonna explicitly close the file and what that means is that every time you open the file there's the file handle associated with the file and internally in the operating system it keeps potentially a bunch of information about that file it might allocate some memory to store buffers for it uses when it reads and writes from the files and so it's good practice to explicitly close your files cause in a well in a short program like this when we just abort the program when we just close the program it automatically closes any open files anyway so we don't really need to do this we don't need to explicitly close the file but imagine a longer running program where say like a program that runs indefinitely as like servers do and over the course of their running they can open many files and you don't wanna leave open a bunch of files you're no longer using you wanna make sure to close them and so it's generally just good practice it's a good habit to form to just always explicitly close your files to make sure they're not accidentally leaving them open so just for the sake of imbuing a good habit here we're gonna explicitly close this file and again for fairly odd reasons files that you try to close it might actually fail so we actually should check for an error message saying we couldn't explicitly close the file there's again not anything we could do about it in this case and again the circumstances under which a file might fail to close are quite exotic and we'll defer those questions until later but for now we'll just check for the error and if there was an error we'll acknowledge it we'll print it out to the user and return so that's a program that writes a file now let's look at a program that reads a file which is quite similar the opening moves are the same we just open the same file in the exact same way check for the error returning if there was an error and then to read data from the file we need a so-called buffer we need a slice of bytes to read the data into from the file and just for good measure we'll make it much larger than we actually need we'll just make it a thousand bytes so this is creating a slice of bytes for the thousand bytes and then the read operation like write gives us back an integer we store an n here indicating how many bytes were actually read because reads can be partial like writes but in the case of a read that's not necessarily considered an error because it's actually very common for reads to not read all the bytes you requested and in this case because our bytes here has a length of a thousand the read file operation will attempt to read up to a thousand bytes in this case assuming that we didn't modify my file the text we just wrote it out with this program and now we're gonna read it well we know that this we know it's like 20 bytes at most it's not a large file so a thousand bytes is actually way larger than the file we're reading so this is definitely going to be a partial read but it may even be less than the full file let's say it's 20 so it could be like 10 or five or something less than the file anyway so the first time we read what we're going to do is we need to check if there was some kind of error because maybe something else went wrong maybe the file we were reading was on a flash drive but the user as we were trying to read the file yanked it out with the drive something like that could happen other things could go wrong so in the event of error not being an empty string something went wrong but there's a special kind of error we're gonna check for which is an error message denoted EOF capital EOF standing for end of file and this is the error we get that's not really an error it's just indicating that hey you reached the end of the file and so you're done reading the whole thing and in that case we're gonna break out of our loop because we're reading in a loop that just keeps reading because we know the reads are gonna be partial potentially and so we need to potentially read more than once so in the event though that we reached in the file that's when we break out in the event it's some other kind of error then we print out what the error message was and we just abort the whole program but in the event that there wasn't any error at all and we're not an end of file then we're gonna take from the byte the data was copied into the byte slice and we wanna read those bytes it was only end bytes though so we slice bytes from zero up to n because those are only the rest of the buffer the rest of bytes is not anything it's not useful data it's just whatever it's just as much as zero is actually the first time through so we just want the first end bytes of the slice we then take that slice of bytes, convert it to a string that's what this operation does and we're gonna concatenate it to message and assign the result to message so each time through this loop message just starts out as an empty string but the first time through this loop we read some number of bytes, convert it to string concat that to message and so each time through this loop we're reading more data, converting it to a string and then tacking that string onto our existing string message we're just building up the string each time through the loop do you understand this is really not an efficient way to build up a string that we read from a file because what we're doing is like I imagine it was a very large file and so the string would get very large and then each time through this loop we're taking the existing string that might be very large tacking on a little bit more to it and then creating from that a third new string which is the size of those two combined and imagine doing that many many times and you're creating all these large strings in a very wasteful unnecessary way when we get to real go we'll have a more efficient alternative to this solution so but for now this works for our purposes and we're dealing with a small file anyway so we're not gonna notice anyway that's how we build up our string and actually something I haven't said explicitly is that when we read and write from an open file with an open file is associated a so-called marker that as you read and write from the file it advances so say we open the file that's a thousand bytes in size and I read 10 bytes from it well the marker starts out when you open the file at the beginning of the file you read 10 bytes the marker advances by 10 bytes so the next time you read we're not starting at the beginning of the file we're starting at the 11th byte and reading from there so every time you read the marker is advancing and eventually the marker is gonna advance to the end of the file and when we try and do a read operation there's gonna be zero bytes left to read and that's when we get the end of file error only when the marker is at the very end of the file when we do a read so that's what's eventually gonna happen here most likely this will just take two iterations first read will read the whole file and the next read will give us back into the end of file that's the most likely thing to happen because we're talking about a very small file here and a small amount of data but not guaranteed so it could be more than two iterations through this loop anyway when we break out of the loop because we reached end of file we have our contents as a string message we can print that out and then we should close the file just like we did in write file and we're done okay so now let's look at these two programs in action first we're gonna run the write file program here we go and let's see if it created a file by file.txt yes I see it we'll type out its contents and there we go and what it's printing out is well the first three bytes we wrote were the value 102 and the 101 in Unicode 100 is the value of lowercase d 2 is a non-printing character so that's why we see this funny question mark box um that's what the command prompt displays when it doesn't know how to print a character and then E is 101 that's lowercase yeah so that's right and then hello file world right here that's the other stuff we print it out okay so so far so good and now let's do the read file read file program and yeah it's reading in the file and then printing out the console what it read in and it looks the same okay so that's all good now let's do something a little bit more interesting we want to write out what's called CSV data CSV stands for comma separated values not commas comma separated values it's a informal format there's no strict uh specification of what this data format looks like exactly but there's just a set of conventions and common practices the idea is that it's a text file where each line of of the text represents an individual piece of data or a composite piece of data made up of some number of elements those elements are separated by commas on that line so say we have our struct cat we're made up of a name, weight, and age and we want to write a bunch of cats out to a file well it would CSV would make sense it's a super format for that purpose and so each line would be a name then a comma a weight value a comma and then an age value and then a new line at the end to go down to the next line understand CSV is not really appropriate for all kinds of data it's a very simplistic kind of data representation it works out fine for a lot of things but for data that involves a lot of thing that stuff that's more binary like say a large array of a bunch of numbers you it would be an inefficient way of storing that data and generally not a good idea but for some simple stuff like say these cats it makes sense so what a program here does is looking down at main we create this slice of cats with three cats first is cat Oscar and Mittens and Fluffy and we're then going to iterate over all the cats and for each cat we're calling this method CSV which we defined right here CSV method takes your cat and returns a string representation of that cat it's returning the individual line for that cat so we first get the name and we decided that we were going to go in the same order here so name weight and age that's important to keep the order in mind so we get the name first then the weight and the age and name is already a string but weight and age weight as a float age as an integer so we have to convert those to strings so we do so like this and we're going to concatenate it all together putting commas in between them and at the end we have a new line what this is we haven't discussed this before I don't think inside strings backslash has special meaning the character backslash doesn't represent a backslash it represents the start of a so-called escape sequence which is two characters first a backslash and then some other character together forming an escape sequence representing some kind of character which we otherwise couldn't put in our strings we couldn't type it and show it in our strings so new line for example our strings in GoPidgin can't span across multiple lines so you can't just put a new line in the middle of your string if you want to write a new line in your string then you have to write backslash and those two characters together are an escape sequence representing a new line character there are a few other escape sequences there's backslash t for a tab character backslash r as a carriage return character and then backslash backslash is how you write an actual backslash so because backslash itself is used to denote the start of an escape sequence if you wanted to put an actual backslash you know to write two of them together and so now this string if we wrote this these two together represent a single backslash and then this would be a lowercase and not a new line character so this would actually be a two character string whereas this here is a one character string just a new line which is what we want here so that's our csv function it spits out an individual line representing that cat as a string but it's not actually writing it to the file yet anyway so in our loop we're for each cat we're getting the csv equivalent concatenating it to s so each time through we're building up the value of s s starts out empty each time through we tack on another line representing the cat we're then just going to print that out so we can see what we have just for demo purposes and then we want to write it out to a file so we want to take this string s write it all out to a file I've created this write function which takes a file name some file path and a slice of bytes and then it returns a string which is an error value so upon success we want to write function to return an empty string but then on failure we return something else some message explaining what went wrong so in write file so we're passing in test.txt the name of our file and we're passing in the string the byte slice from our string so the byte representation of our string data and inside write we open a file open the file we specified test if there was an error if there was some kind of error well we can't proceed so we should just return return the error message and then we try and write the file and if that succeeds then if that fails then we return early otherwise we go on and we close the file and if that fails then we return early otherwise if all these operations succeed then we're going to return an empty string indicating that the write completed successfully so down here after calling write sign the result to error and then we check if error is not equal to empty string and if it is and something went wrong we'll print out what the error was and return because in this program we can't really do anything in that eventuality so we might as well just abort but assuming it complete successfully the write happened successfully then we can go on and and then actually we don't even need this line that's not even correct and then the next line let's see we print out the well this is not even correct if you want to get the byte slice from a string you do that but that was just printing out the length of how many bytes were written but we don't even need that so I'm just going to delete that anyway so it'll print out successfully wrote file cats.csv and that's also wrong because I didn't call it that let's call it that cats.csv okay now it's consistent so now let's look at the readcsv program which is going to read in a text file we just wrote and for each line it's going to interpret each line as a cat as for each each line we get back a cat struct and so here in main we have this function read we've defined which we pass in a path and we get back the full text of the file and an error message something went wrong so we'll look at that function in a second but after we call this function we check if there was an error if so we print out what the error was in return because we can't proceed really our program has to just abort and then we're going to split the lines of the text we define the split function which you specify a string it'll go through the string and for each occurrence of that string it'll split the string into two into pieces so if I have a string with a bunch of new lines in it then I get back a slice of strings which is all the stuff around the new lines not the new lines themselves so we'll look at that in a second so effectively we're getting all the individual lines as individual strings in a slice without the new line at the end and then for each line we call the read cat function and that returns a cat struct assigned here to C and if there was some issue with the format of the text if it wasn't a properly formed cat like maybe it was missing commas or something was wrong about the data well then we get an error so our read cat function will return an error message if there was something wrong so we check after calling read cat we check if the if error was not equal to empty string and if so we just abort because something's wrong with the data and we'll just quit otherwise we proceed and we take that cat C and append it to the cat slice so once we go through this loop for each line we'll have a cat struct in the cats slice and so at the end of this well we'll just print out the length of cats which in this case should be three because that's how many we wrote out to the file so let's look at the read function the read function takes in the file name returns first to string which is the text of the file read and then the string the second string is the error message if any and so first we open the file check if check if there was an error then we have our buffer this code looks really exactly the same in that previous read file program we go through a loop where we read from the file we get some number of bytes back and if there was an error and it was end of file that means that we're reading from the end of the file and so we break out of the loop because we have everything we need otherwise we append to text we tack on to the text string everything we read from the file and that's what this line does it's exactly the same as what we did in the read file program anyway so we get through the loop and once we're out of the loop we can close the file because we're done with it we test if there was an error in closing the file and in the end we return the text and oops there should be an empty string here so every time we return we need to return two values in the case there's an error the first thing we return is just an it doesn't matter we return in that case really so we'll just run an empty string because all we care about and if there's an error all we care about is that what the error message says but upon success the error message is the empty string and then text is whatever we read from the file so that's the read function and the split function is kind of involved I'm I would actually go through it as an exercise you can see if you can figure it out there's probably a simpler more elegant way to write this but this works at least in my testing so given some string we want to split up into multiple strings and some splitter string in this case it was the new line string we get back a slice of strings and I'll leave the details for you to look at anyway so the read cat function that's more significant here that we pass in the individual line without the new line at the end and what we want to get back is a cat and then a string if there's some error this will be empty string in the event that we could properly parse a cat otherwise we get back some message indicating when we're wrong and so we have our individual line but the three elements that make up each cat are separated by comma so we're going to split the line splitting it on comma and we should get back a slice of strings with three elements the first should be the name the second thing should be the weight and the third thing should be the age and first we'll check off if the length is correct because if elements are something other than three then there's something wrong with the line it's a misinformed cat it's not properly a cat so we're going to return in that case well we don't I actually just created this variable cat here to make it easier to return a cat we don't care about it's just defaulting to an empty cat that has no values like it is an empty string name as a weight and age of zero but in the event that there's an error we don't really care what we're returning for the cat we just want the error message here so just a convenience thing that's why I created this variable C here so we can make it easier to return a cat we don't care about anyway so in the event there aren't three elements separated by commas on the line we that's what we return as our error message and in a way we proceed on to the next line what we want to do is we want to take the second element the element and index one and it's a string but we want to read it as a float it's a sequence of text characters but it should be interpretable it should look like a float value and so to get a float representation of that string we have the parse float operation be clear in the previous program in right csv over here we just took our float value and convert it to a string we can do that easy enough but this is a separate operation because we're going the other way around we're taking our our string and getting a float and so for that we have parse float and the thing about parsing a float or and also parse int as we use here is it maybe the string is malformed maybe it's a string that doesn't look like a float so maybe like what we want is something that looks like this and that's something you know string that looks like this and that's something we can use parse float on but if it looks like this well that's not a float and if it's just has characters that aren't valid number characters well that's not something we can parse either so in the event that it's a string that isn't properly formed as a float it doesn't look like a float then we get back a non-empty string error message it's actually an operation that returns two values it returns the value which we signed the weight here assuming it's successful but then upon error returns a non-empty empty string here so in the event we can't it's improperly formed if it's not a valid float string then we return and then for the age is the same deal except we're dealing with the integer so it's parse int instead of parse float but same deal we have to account for the possibility that it's malformed it doesn't it's not properly interpretable readable as an integer and in the case assuming it's properly formed everything's properly formed we get down here and we have our name weight and age and actually I made a mistake here let me correct this we didn't actually use any name variable we just need to get the first element from the elements slice there we go that's our name which is a string and weight which is now float and age is an integer and this is a success case so for the error string we return an empty string so that's the read cat function works on each individual line and that's the whole program so down here when we use read cat we can take a line and get back a cat or we get back a non-empty string error message if the line can't be interpreted as a cat so let's see if we can get these programs to run first off write csv okay that seems good so far let me look you know as cats.csv is there let's type out its contents oh I need to work on this the way it's converting floats into string values is in this format we probably don't want we want it to look something more sensible cleaner we don't need this engineering notation style but let's just go with it for now that that's valid okay and now let's look at the read csv program in action after I've made some fixes just now and it prints out three which is correct because we have three lines each one represent a cat and that's how it should be so it seems to work we could delve in more and see if it is actually creating the cats correctly but I think we're good well let's try it let's probably printing out the actual cats see how it prints it yeah there we go that all looks good