Oh, to the intro. All right, well hi everyone and welcome back to to Sailor Academy's CS105 Introduction to Python. This is going to be a review of Unit 7 file handling. If you haven't seen any of the other videos or you haven't started the course, go to sailoracademy.com. You can find .org. Why did I say .com? And it's also just sailor.org. I totally messed that up. It's early in the morning. I'll have some tea again in a second. But certainly the program we're going to be using is called Replit. So I'll make sure I put the links to the course and the links to the program in the chat. If you're watching this later, it should be down in the description. If you're watching right now, though, if you have any questions, feel free to put them in the chat and we'll get to them if we can. But I'll just kick it over right now to Dr. Eric Sachs so that I can have that tea and get my mind right. All right. Thanks so much. I appreciate it. So yeah, we want to talk about file handling today. When it comes to file handling, there's a lot to discuss. But it's something definitely that needs to be brought up in an introductory programming course. But what I would say in terms of the bigger picture is that there are lots of modules in Python. I mean, too many of us talk about in a course like this that can handle lots of different file types and do things in lots of different ways. The important part of this introduction is that you understand how file handling works. And then as you grow more into Python, then you'll find, again, other modules that create their own file formats or they can read files into certain types of data structures that make it convenient for specific file types. But let's start out with the basics. And we've inherited technology that's been around since the time of the caveman. And this is just kind of how it works when it comes to handling files. Every operating system affords you some set of function calls that can interact with computer systems. And regardless of what that looks like at the high level in terms of syntax, generally speaking, there are usually three operations that have to take place, three sets of operations. You have to notify the operating system that you are planning on opening a file. And also what your intent is to do what you plan to do with that file, be it reading or writing or some other bunch of stuff. And then finally, after you're done with that file, you might want to read data from the file or you might want to write data to a file that you have to... Well, it's a good idea. You don't have to. I guess in some sense, it doesn't matter because there's a lot of safety hooks and so forth. So if you leave out the closing part, unless you're writing professional software that you want to sell to people, if you left this out, it wouldn't be completely catastrophic. You might potentially crash your computer, but you could always restart it.  And then the system basically releases it at that point. And that's kind of how it works. Even if you go down to the lowest level on a computer, you'll find this is exactly what's happening. And when you're at the high level, if you're going to interact with a disc or solid state drive or whatever type of medium you're talking about, this is always taking place. And at the high level, you don't want to be concerned at this point in a beginner kind of a course, all the details of what it takes to get the operating system to do something with a piece of hardware. That's not high level programming. But these are the fundamental operations that you want to understand. And so how does that translate into Python code? How do those operations? Well, really pretty easily. I think for the most part, you'll find most computer languages use mostly the same syntax so that even if you picked up another language and you'd be able to immediately identify those operations in a lot of other languages. And so now what we get to do, so far we've talked about the first window where we have the console and you can basically have the Python interpreter, the most right-most window. And in the first unit we learned how to type single commands and get singular things done. Then we graduated to the middle window where we could begin to run scripts and sections of code to do interesting things. And then this left window we've been kind of not really talking about too much, but today we're going to because this is the files window. And the way that Replet works is that when we make file reads or file writes, the files are going to be created in this file window. So this is in some sense equivalent. You're using this as an equivalent disk drive or a solid state drive where you're storing files and then you're accessing them and reading them. This is the playground for files within Replet. Now, if you're going to be doing real file handling, that means that you're really going to be sending files to a destination. And that destination in Replet, again, is this window, but you'll be sending it to a folder of some kind. And so when you make reference to files, you have to specify paths. So this section kind of absolves you of having to worry about typing out full paths to folders on some kind of a drive. And it really just focuses on the bare bones of what it takes to open files, close them, read them, read to them, and write to them. So maybe we'll just start off with the... And that's all I got. That's the introduction to file handling. There isn't much more to it than that. As I said before, the details of doing it can be really pretty exciting if you get down to the low level and stuff, because it's really interesting. It's interesting to understand how data is distributed across some medium. But again, that wouldn't be a conversation for today. So what I have here is just some code, some simple code where I've got... And in addition, so for this unit, again, because we could really go out into the ozone layer, I really limited the discussion to text data and text files. But that isn't the only way that you have to do things. But I think in constructing the unit, at the introductory level, just because there are so many different points that could be discussed in terms of how data is stored out to a disk or what have you, I just wanted to focus, like I said, on the bare bones to get up and running, because there's enough to talk about really within the unit. And I guess, personally, I would say that if you go beyond the text, usually what happens is that you've got some intent. You're intending to do something either like data-wise or in terms of computation. And there are other tools out there within other modules that I think are more relevant to file handling than just what we're doing here. And so that's why this is just the basics. And that's why I kind of stuck to text. Anyway, so here's a string, a text string, a very exciting string from characters one to five. And then here you see the open. And the open basically lets the operating system know you're planning on accessing a file named filename.txt, and we're planning on writing to it. And so there's a number of implications when, if you look over here in the left window, you see there's nothing called, there's no file over here called filename.txt. So it's going to have to be created. And that's what that W really does. It actually, it causes the file to be created if it doesn't exist. Okay. There are other implications we'll get to that we need to talk about what the W can do as well, but we'll get to that in a second. So right now I'm just talking about creating a file that I plan to write data to. And then what's going to happen is, and we could write more in-depth code for this because you're basically, you're really, you're asking the operating system to do you a favor, and that is to open a file for you. And if the open is successful, then it'll return a handle, what's called a handle back to you. And that handle is contained in the variable myfile. So usually, if you go ahead and write this stuff for a living, usually what you do is you want to check to make sure that there wasn't some null event where the, for whatever reason, the operating system said no because it's well within its rights to do so. So if you were in a position where you're writing more professional code, you'd put some extra hooks in here like an if statement to make sure that the open actually went well. And I'm not really addressing that right now because that's getting more into, again, into direction. I'm really just trying to stay focused on getting something from the operating system. And so then it's going to take the text. And so you can see that right now is a method. It's a method that's associated with the object myfile. And you're being allowed to write data to that file. And then like I said a second ago, we're going to close it. Okay. So once it's closed, and I'll take this out for a second because I want to show you this in slow motion, that once you write it, it gets created over here in the left window. And so again, this is the playground. This is your sandbox for where you're going to be sort of placing files and creating data and putting data in the files and so forth. So if you want to go and inspect what happened, you can go ahead and actually look at it. You can see that filename.txt has the data, one, two, three, four, five has the string in it. And that's all there is to it. We just did a file write, a write of data to a file. Now if we want to go ahead and understand what kind of what's there and make ourselves believe that it's actually there, you know, aside from inspecting it, you know, you may want to actually write code where you read data from a file. So those are really probably the two most fundamental operations before we go any further. Just knowing how to create a file, write data to the file, and then read data from the file once you're sure of the data that's there. And so here we, since it's been closed for the right, okay, we can reopen it now for the read. And so this little R here means that we want to read data from that file. And so that's been created. I actually don't need this anymore. I can take it out, which is concentrate on the read. And so there are default modes for opening files. It's good to have this here from a documentation standpoint. And I tend to be more meticulous about this. I just want to program.  So if you see something like this and you see code like that, then the open method has to make a decision. And the decision is, what am I, why am I opening the file in the first place? And so the default mode for the open process is you're going to be reading it. You're going to assume to be reading it. So if you leave the R out, if you actually leave it out, it'll assume that you're open will assume that you want to read data from the file, which I think is the safest decision because you don't want to accidentally be writing data to a file if you don't mean to be doing so. But reading is not destructive. It just has implications where you're going to go and grab the data. Anyway, and that's important. I think if you want to learn anything as a programmer, I think that the one thing that you want to learn is to learn how not to destroy data. And you know, unintentionally, unintentionally. So you need to know how this stuff works. And because people do that.  And then the whole show went out the window. So I'm just saying that little things like this, you know, they're very easy to remember and you need to know them. And if you don't know them, you know, they could turn out to be catastrophic in this sense. It's a good design, you know, it's a good idea to make the open default to a read. So we go ahead and read it now that the read the read method is going to read from the file. And then once again, because we are neat freaks, we go ahead and we close the file at the at the most immediate point where afterwards where it makes sense to do so.  And I mean, some people write code that way, I'm just I don't come from that camp of people. You know, when I'm done with the file, I close it when I know I'm done and go no further. Just kind of, it's a good, you know, it's a good way to do it that way. It's what I think. Anyway, so if you go ahead and do this and run the code, you'll find that you when you when you read the data into the variable X that you you can see the data out of the string. And so what was read was a string. Okay, it's it's data of type string. So you took a string, you planted it in this file. And then you read it out. And now you can see that it's there. So again, those are probably the two most fundamental operations that that you'd be interested in. And now now the question is what are the theme and variations on that methodology, because you can be a lot more specific, and you can be a lot more detailed. This is just sort of like a macroscopic read, just give me everything that's in the file and plop it into one string called x. And but we can be, you know, we can be more elegant about our choice for, you know, what data we would want, we would want to read. So that's a little bit more about what the what the chapter is about. And I just want to make sure I'm sharing the window. So, you know, I sometimes I go through classes, I'll go through like 10 minutes and then students will be like, Dr. Sack, were you meant, are you supposed to be sharing this other window? You know, so I don't do that, you know, intentionally. So I just double check every once in a while to make sure. Okay, we're, we're on, we're on replete right now. So very good. Okay. All right. So let's play another game now. Let's let's try something like this, we'll get more, we'll get more detailed. And you also have the ability, and this is just, you know, how the IDE works is you can come over here, if you don't need this data anymore, you can delete it, you know, you can actually choose to delete it. And it would just go away and it'll ask you if you want to delete it. Not that it matters whether or not I just did that or not, because we can always put it back in this case.  So now let's get a little bit deeper into text and string processing. So, so another thing that this should do is it should at least tie together some things that we've done previously, because we've talked about strings already, and now we're planning on saving those strings to a file. So everything, everything I would say effectively that you know about strings will comes to bear on something like this, because we're dealing with text files where that's that's the kind of data that we're putting into the files. So I can create a string that looks like this now where where I have a five space, a line feed character backslash n, and then space 10 space backslash. And so, and, and there is a perfectly faithful reproduction of what that string is saying to do. So, so in terms of formatting strings, you know, you want to pay attention to what this is going to look like, you have to kind of imagine, like if you make a lot if you create a line like this, that, that you're going to be you're going to create create a data file, not just with with strings of characters, but also the kind of the unrecognizable characters that you that your brain would tend to discount, but are actually there in the file, like spaces or tabs or, or line feeds, all of those, all of those characters are there within the file. So you need to be aware that that they are actually there, because that'll have implications for the formatting of the data and how it looks within the file. I'll show you what I mean by that in a second. And then finally, after that data gets written, while the file is still open, you can you can you can make multiple rights, you know, it's open so you can continue writing data to the file. Okay. So you can do this with multiple instructions on the on the read side of things. If we go ahead and decide to so we close it and then we go we're planning on reading the data. And so you can see here that there's this method called read line. So I create a handle called f I just made it one character. You know, usually when you pick when you choose the handles for the files, you pick names that are that make sense. I just sometimes I get lazy and I'll just use like one letter characters. But that's not, you know, you want to create you always want to create sensible names for your variable. So I guess what I'm saying is do as I say not as I do. But anyway, so so this is an f that's a handle to example dot txt and just for teaching purposes. And then I'm going to read a line. And then I'm going to read another line. I'm going to read another one. And what I'm going to do is is because I've a priori set up the data so that it's numerical. When I read a line of numerical data, I'm going to convert it to a floating point number, another floating point number going to close the file. And then I'll print maybe the ratio of those two values. So you'll notice I'm closing the file before I print. You could say, Well, is that a big deal? Couldn't I just close it afterwards? Yeah, you know, it's really a matter of style, you know, I prefer to just when I'm done when I know I'm not going to read or write to the file, I just immediately close it. I just don't I don't want that that hook open basically, you know, for any extended period of time, I don't even care if it's a nanosecond. It's just I want to close it at that point. So the I'm going to speak a little bit to this, because there's depth to kind of stuff that happens in the background when you when you read a file, for instance, like a text file that has many lines associated with it. And in the background, there's there's there's extra stuff that happens. There's something called a file pointer. And when you go ahead and open the file, let's say for a read, notice, I've left out the R here, so it's a default. The file pointer is reset to the beginning of the file. Okay, which is the sensible thing to do. So so in the background, there's there's other stuff happening underneath the hood, where where somebody within the operating system, somebody some part of the operating system knows where it's pointing to in the file. And you know, if you do this stuff more, probably, you know, if you ever go down this path and start to really get underneath the hood for some of these routines, you'll be messing with the file pointer, you can actually take control of it and tell it where to point within a file. Again, I prefer not to go down that path today, just because it's going to detract from from a whole bunch of other stuff that I want to say, what I will say is this, okay, I'll say that that when you perform this the reading of a single line, which is going to grab every everything that's on that line, all the characters associated with it, okay, that even though I'm not doing anything with that line, I'm fast forwarding because the fact that I read that line, the file pointer will will automatically point to the next line. So this file read line f dot read line will read the data and then cause the file pointer to point to the next line and the next line and the next time that's on. So the question a question that could be asked of you would say, well, given that this data is being stored out to a file, where am I pointing to, you know, what data am I actually grabbing from the file, and what steps have I taken to fast forward to that data? Okay, so let me take this out first. I'll show you what I mean. Let's go ahead and just run this for a second. And when you run it, you create that example dot txt and you create the file and it's got data inside of it. So watch this now, okay, you have to sort of line up the formatting, the formatting of the file data with what was said in the in the right methods for that for the strings that were input, you'll notice there's a space over here. The first one doesn't have a space, but there is a space and there is a space and there is a space that might be a big deal to you, you know, that might matter. Well, where did it come from? Well, you might think, oh, I don't, I don't want that there, you know, or I don't, I don't want this there. Okay. But you put it there. And, and when there's like spaces have implications, they're actually in terms of the ASCII code, which is which is the convention for converting character data into their numerical equivalence, their eight bit numerical equivalence. So it's not extended that that a that a space actually has a number associated with it. Okay, it actually has a number associated with it's the number 32, it's actually an ASCII 32. So, so every character that's in a string has an ASCII value that's associated with it. And so these are, these are not, they're not zeros, they're not non-existence, that when you have a string, anything that's in that string, anything that's between those single quotes is there in the string. And, and so when you write it to the file with the, with the line feed characters, the backslash ends, then yeah, you get, you get the backslash end, but then you also get the space before the value. Now, whether or not that, I'm just trying to let you know it's there, it's irrelevant to me. You know, I don't really have an application in mind, I'm just putting it there so that you pay attention to the fact that when you start to see like data get sort of botched all over the place, it's usually because of the fact that there were, you know, spaces and tabs that were put in, in an unpredictable way that the user wasn't, wasn't clear on, on what they were typing basically when they created the string. So that's what this does. It's really a one to one exact replica of what's being said in this statement, which is what computers do. And then of course you've got the hello world, so that's what the file looks like. Now, if I want to read data from the file, which is oftentimes what you do, I mean, you can go and check, you can go into like an Excel spreadsheet program, like start an Excel program or any spreadsheet program, I don't care what it is, that, that oftentimes you have, you have the ability to save files in what's called CSV format. And, and, and there are other more sophisticated modules that have CSV readers associated with them. But, but again, at the bare bones, you know, introductory minimal level, CSV stands for comma, comma space delimited data, which is oftentimes how you store text data in a file. It means basically when you have data values, they're either going to be separated by a comma or a space. And that makes it easy to parse, like if you have to write a program to read that data, it means that, that, that you have to like write something that says, Oh, I recognize that I want this as a value, but then I know how to skip to the next value, I just look for either a space or, or a comma, and then I know to go to the next value, the next value, and there's also tab delimited data. And my point is, is that you can, you can open up a CSV file, and you could see that it's actually text data, that's, that's comma and space delimited between the data points.  So this is just a kind of precursor to all of that. This is a simple data file. It's nothing complicated. It's got five, 10, 15 and 20 as characters. And then you have to recognize that when you go ahead and read a line, or read the next line, that you're reading in the text data. And then what you're asking Python to do is to say, take, take that line of text data and then convert it to a floating point number in the best way that you know how, right? So, so float is going to take some liberties with, with what's being read from that line of data, because there are spaces in there. It's string data. It's, it's character data. It's not numerical data. And then once you go ahead and, and divide b by a, you'll get an answer out. So what this is doing, basically the first read line is going to skip over the five. And then the next read line is going to get the 10. And then the next read line is going to get the 15. So there, so that's what's happening to the file pointer. It's going to the first line where there's a five, then the next and the next. So this read line is reading the 10, this read line, you should be able to predict if you're understanding how the read line works is going to grab the 15, convert those default floating point values and then take 10 and divide it by 15 in principle. That's what's supposed to be happening. That's my guess as to what this code is going to do based upon what the file pointer is doing as you, as you fast forward to the next line, the next line, the next line. So if you go ahead and do this, you'll get 1.5. And so I'm sorry, a is going to be 10 and b is 15. So I'm dividing 15. I'm not dividing by 10 by 15 and dividing 15 by 10. I apologize for that. So that's because let's go look at the data again. So this is going to be the a and that's the b. And the five was skipped over by the read line. So this becomes a 10.0, this becomes b 15.0. And then 15.0 divided by 10 is you get the answer 1.5. Okay, I think I may have said 10 divided by 15. But don't listen to me. And so okay, so the so that so this is one application on a very small scale. If you know how to do this, basically, you can do this for any size file with, you know, with with typical of what you would use in other data processing types of programs. Again, you know, the dot CSV types of files are really common. Because the main reason is because if you want to export data from some spreadsheet type of a program, you know, not everybody in the universe, you know, although Microsoft would like to think so, not everybody in the universe uses Excel as a spreadsheet program. And so if I wanted to communicate my spreadsheet data to you, and you were using some other application, I wouldn't save it out in Excel format, I would save it out in in CSV format, for instance, that's a very common kind of format. And so this code is kind of the rudiment. So if you're writing, if you didn't have a CSV reader method, which, which they exist, they actually exist. But I'm just saying, if you want to do it this way, this would be this would be the first steps, let's say, and we'll see a few more steps in a second. But this is the first steps to reading numerical data from a text kind of a file, basically, which is a major major application in data processing. Two majors far as I'm concerned, but that's the way that's the way they did it. So you got to go with it. All right, enough about that. Let's come over here and maybe try and say something else. Let's talk about implications. You know, what are the implications of of doing one thing or another? Okay. So there are implications. So what we'll do here is we'll, again, my goal is to kind of exercise everything that we've been doing throughout all the units. So I reserve the right to pull up, you know, four loops, you got to know how those work and so forth at this point. But here I'm saying that I'm going to create some type of a file, it doesn't exist in the left window, because I just deleted it if it was there before. And now I'm just going to have a bunch of, you know, right instructions and I'm going to, I'm going to do this a number of times. And then I'm going to close the file. So here I go ahead and create some data. And you can perform multiple rights. So what is the loop doing? Excuse me, it's, it's taking, I'm just, again, not a very exciting loop. You're just counting from zero to nine. You're creating, you're converting the numerical value i, which is the loop variable into a string and writing it with a space and then putting i plus one next to it and then putting a line feed next to it. So basically the first line, if i is zero on the first line, you should see zero space, one line feed and then one increments to zero increments to one. You see it's one space to line feed to space three line feed. Again, not meant to be very exciting. I'm just trying to point out again, the implications of, you know, how you would write code, let's say to, to store it into, into some kind of a file and then you're done writing to it. Okay. And then you're done writing to it. And then let's make believe that, that this file exists and you'd like to write new data to it. Okay. You'd like to write new data to it. And the one thing you have to understand the implications of using the W, the W, if you go ahead and do this, if I just go ahead and run this again and just open it and close it, this is a big deal. It really is. It's so simple, but again, just something you need to know. If I decide to just open the file to write to it again, see when it didn't exist, okay, it got created. All right. And, and again, that's, that's not something you have to worry about too much. It's just when you run the open method, the operating system is going to check to see if the file exists. And if it doesn't exist, it's going to assume that you do want to write to the file. So it creates it for you. Okay. So that's a decision. That's kind of out of your hands at that point, you know, because, because when the file specifier mode, the file mode specifier, file mode specifier is, is used, it has implications. And the implication is that, is that the computer really believes you. If you choose that W, it's like, it assumes you know what you're doing and you meant to do that. And, and it sees that the, oh, it doesn't exist. Well, I'm going to, I'm going to help you. I'm going to create it for you. Okay. And then the data is there and then you see it's there. Now, if for some reason, you go ahead, take a step further to, to reopen the file such that you want to write to it, you don't want to write to it, you want to append it. Because if you do this, and you go over here and look at what happened, you'll see there's no data anymore. Okay. So, so, so the file, right, you got to get this W and you have to have it perfectly. Because again, there are implications. You know, there's implications for using the R, there's implications for not for leaving out the R, there's implications for using the W. And one of them is that the file doesn't exist. Well, the operating system is nice enough to create it for you. And if it does exist, it assumes that you didn't want to save the old file, you wanted to create a new empty file. So, which means you would obliterate anything that you've done previously. So if that's not intentional, you know, if it's not intentional, obliteration is generally considered to be a bad thing. And so, so you don't want to use the W in this case, you want to use an A for append. Okay. And if you go ahead and append, you know, then you could say something like, you know, my file, right, just because now it's open again. And when you append, when you append the file, you know, you can put anything that you want into there. And so something like that should should work just fine. So now what's happened is that is that the file got the file got created. And then you wrote the data to it. And then you appended the data to it. Okay, then you appended it. So you want to be very, very clear. And this is the type of stuff like when you're programming, you know, you'll be on a project that like three o'clock in the morning or something like that. And so you may not always, you know, be, you got to keep a clear head when you use it's not hard, but you need to know how it works, you know, so R is right. W, I'm sorry, R is right. R is read. R is read. W is, is, and don't you forget that. And, and, and W is right. And A is append. And so when you want to write to an already created file, you're appending. Okay. And that's all there is to it. Okay. And again, it's not hard. You just need to know it. You need to know how it works. Okay. And those are the, those are the major, those are the major file modes, the specifiers that I, that I mostly concentrated on simply because of the fact that I didn't want to get too, you know, I didn't want to go too far out into the left field. And so, and this is enough. Like I said, this is enough to talk about. Okay. So let's do this maybe. Yeah, so we're done with that. Let's do something else. Now, let's talk about multiple lines. Okay. Let's talk about multiple lines. So what I can do is I can create a list of strings that I would want to write to a file. And this, you know, again, this, this would be another typical kind of application where, you know, you really would create a list that would have multiple strings that you would want to write. Well, so that's what I'm doing with this data. The data is a list. It contains three strings. And they're all formatted. So they all have the, the line feed character to get to the next line. And I can open a file. And again, this is going to, since it already exists out here, anything that's in this file is going to get overwritten because I'm, I'm doing that intentionally. And then I'm going to write the data. And then I'm going to close it. So write lines. Okay, is a method that will take a list of strings. And then, and then send them out to, we'll send them out to the file. Okay. And so if we go ahead and run this, that you'll notice nothing's happening over here. I haven't, I haven't really done anything about whether or not the right, you know, was the right successful. You know, there's a lot of, there's a lot of stuff that you can do to write careful, careful professional software. Like if you do write something, you might ask the question without, because you may not be able to, you know, you got a program going at two gigahertz. You know, you may not, you may not be in a position to just go ahead and inspect the file, you know. And so you can ask questions like that. There are methods, you know, within, and there are a lot of hooks in there that you can, you can check to make sure that the, the right was successful, you know, those kinds of things. I'm not really addressing anything like that. But, but, you know, if you're going to write good software, you always want to, you always want to check stuff to make sure that what you intended to do actually happen, especially when it comes to this, especially when it comes to, again, these, these more higher level instructions that are actually doing a lot underneath the hood. You know, you really want to be checking and double checking. I'm not doing any of that, because like I said, I'm just, this is introductory stuff. So I'm just trying to, you know, to cut to the chase with, with the basics. So once this code runs, then you can see that the lines were, were written to the file. The old data was, was obliterated. But again, it was intentional on my part. This wasn't accidental. This is the only data that I want in the file. So write lines is a good, is a good instruction to know because, you know, I mean, you could write loops and stuff like that. But if you have data in the form of a list, which is, you know, oftentimes, you know, that, that can happen. That's just one level. Okay. And so, yeah, it's a good data structure to, to use for starters. Okay. And then, of course, if you can, if you can read line, if you can write lines, well, then you sure as heck better be able to, to read lines. I'm sorry, if you can write lines, you should be able to read them. So, so the read lines method will, will give you this kind of output. And so you can see a decision was made by the method. You didn't really do anything. You said, well, I just want to read, I want to read the lines within the file. And if you do, well, I'll compare and contrast this from in a moment. In fact, I'll do it right now. Let's do this. If I, let me just compare and contrast here for a second. Got something there. Got something there. Something like this. So, yeah, let's compare and contrast these two. So, well, first of all, let me say this is that, is that you won't recognize the fact that the data that was read was placed in a list. Okay, you didn't do anything. You just said read lines, but that's how the function, that's how the method works. It's just, it's going to take the data. It's going to read all the lines and each line is going to be a new element on that list. Again, that's just might be something that you want to do. And you might want to compare and contrast this against the, against the full read. The full read is just going to, the full read, the my file.read, this function versus this function. And again, I'm not saying either way, which is better or worse. You have an application in mind. And maybe you just want to grab everything in the file, chuck it into a variable and have a long string variable that reflects everything that's contained within the file. That might happen. On the other hand, you may want to have some type of a method that will read each line and parse it properly so that you have different elements in a list, because you could refer to them element by element by element. And that might be another application. I'm not saying, I just want to let you know what you, what Python's capability is so that, you know, when you're programming, you could make, you can make the choice. So, so I would say the most important thing about, about what I'm trying to say here is that, you know, you want to get clear in your head, whether you're dealing with single lines, multiple lines, how you want to write the data, what is it going to look like when it gets out to the file. And then when you read it in, what's it going to look like when you read it into some kind of a variable based upon the method that you use or the method that you call. Okay. So these are two macroscopic, you know, big ideas. But again, with regard to becoming a programmer, it's a programming course. There's other stuff that you might want to do as well. Okay. And so once the file is there, so let me, I'm going to plop out these two examples and I'll plop something else. And again, Python is a very smart language. So when you go ahead and open the file, there are implications, right? You've instantiated some kind of an object that it's just too much to really go into in one sitting as an introduction. But that F has a lot of stuff associated with it. And when Python sees that, even like for an iterator, like for a for loop or something like that, it makes decisions. It tries to almost take a guess at what the most sensible thing that you mean when you create a line like this, right? When you create an instruction like this. So I could say for a line in F and what would take place, you know, as you do that is that is that the for loop is smart enough to figure out that you want to iterate through each line. So there's like an implied iterator, basically, it's not, you don't need an index, you don't need, it's just like what you're used to what happens when you when you write for loops on, you know, like a list or something like that, there are implications for referring to the data, you know, the elements that are contained within a container that are in a container. And so F is not necessarily a container, but you can treat it like one in some sense. And the reason you can treat it like one is because the lines are in some sense, you can index them, they can be indexed. And because they can, they can be indexed, you can iterate from line to line to line. And that's why this code actually works this way. And so in some sense, you know, if you were, if you're comfortable with the programming language, and I just pulled you off the street and say, write me a loop that will print out all the lines in this text file, you might go ahead and just do something like this and just see what would happen because this is very sensible. This, this carries over directly from, you know, the unit on loops and for loops and why loops and so forth, because it's just a very natural progression. It's, it's, it's, it's just, it makes sense. It makes sense that it works this way. Which again, that's why Python is such a, such a nice language. You know, let's take a peek at another one. Okay. Here's another. Hey, Dr. Say before we move on, we did get a question in the chat from Glenn Cooper who asked, what would happen if you appended a file that did not already exist? Oh, great question. It'll create it. It'll create it for you. Yeah, that's a good question. I forgot to say that. So, fantastic question. So if you use append, that will also create a file as well. It'll create it for you. All right, awesome. Thanks. I'm sure that answered the question. I'll let you know if there's any further questions, but I thought we'd get to that while, while we had a moment. Yeah, yeah. No, it's a great question. It's a very, it's a valid question. It's a good question. And I probably should have said it. I probably was thinking I was going to say it and then, so thank you. Okay. So here's another one. Here's another read line. I can read lines and lines and lines. And then you could create loops based upon a lot of loops are based upon sort of reaching an end of file condition kind of a thing. And there's lots of different ways that can happen all the way down to the fact that there are oftentimes end of file characters that you can write loops with. In this case, what I'm choosing to do is just look for the empty string case, because that's the easiest way to tell that you've reached the end of the file. If I attempt to read something, and then I get the empty, I get the empty string. So under those circumstances, if I do something like this kind of a loop, and what I'm trying to point out here is when you read the data, if you read a line, it is actually string data. So that's why I printed the data type here. So if you read a single line, not read lines, read line, then when you read a single line, it comes back not as a list as for read lines, but as a string. So again, just little distinctions like this where you want to tell the difference between things. And that's all I really want to say. I'm just trying to show you that there's different ways of constructing loops to access data within a file. And in the background again, once again, you see the read line, it's doing something for you without you actually having to do it. That is when you read the line, the read line method knows to sort of increment that file pointer to the next line. So that's being done for you automatically. You don't have to do that. I mean, there are cases where you can do that and you might want to do that, but again, that's not the direction that we want to go in for a class like this. So that's another example of trying to apply stuff that you know already from loops, and then basically trying to show you that you can do this with file data as well. And the iteration can occur over lines for instance. Let's go play another game. Maybe I'll pull this one out. Something like this. And I'm going to delete this because I'm changing file names around. Again, you get to pick whatever file name you want to. And then so we can do something like this and something like that. And so here what's happening is that I'm creating a file with a two and a four on a line, three and five on a line, five and six on a line. And again, we might want to go ahead and process some data in different ways. And so let's play around with maybe this code. We'll take a peek at this because now we're getting more sophisticated, I would say. And so many times what'll happen is that you can have, within a string reference, you can have unwanted characters. This is independent of any file stuff. I'm just trying to show you that what you know to be true for strings is once you grab the string, once you know you have string data that's grabbed from the file, you can operate on it any way that you want to. And that's really what this code is trying to show is I know I have these lines within my file. And the first line is two, the second is three, space five. Well, again, I want to be careful here. It's two, space, four, space, backslash, n. Because that's how the data was put in here. That's really, really what's in there. And then similarly for this line, similarly for this line. So it might turn out that it might turn out that I want to process the string after I've read it. And again, this would come in handy for something like a CSV kind of a file, for instance, because now you're getting more specific.  So once again, I can go ahead and implicitly read a line, each line in the file, at the beginning of the loop. And then what I can do is I can, I'm just, and I want to point out that that line is a string. It's actually, it's a variable of type str, because it's a single line.  So anything that's unwanted at the end of the line basically is going to get stripped off. That doesn't, that won't make sense as a character basically. And that would be like a space, for instance, I'm going to strip that off because I don't want that as part of my data. You don't want a space when you're going to, a floating point number doesn't know what an ASCII 32 is and it doesn't care what it wants as the number. So if you're trying to do any numerical processing. And then what you can do is you can, so at that point, once you strip it off, what you're going to end up with is if you were to read this line is you'd have a two and a space four. So now you got that space in the middle. Okay. You've got the space in the middle. What do you do about that? Well, what I'm going to do is I'm going to read that. I'm going to, I'm going to split the, I'm going to split the line at the point where I see a space. And then what happens? Well, what's going to happen at that point is that something's going to happen, right? Something has to happen because if it happens for a single line and you get a three and a, now you got the three and the five isolated. So those commands are going to read a line and you're going to get two characters, a three and a five. What's Python going to do with that three and the five if it's a line and now you're telling it to split it? Well, the split is going to create something, you know, some kind of a list that has that data in it. So, so the data used to be like a two and a four, you know, with the space and the backslash. And now there's been this kind of post processing a series of steps, you know, through, through a series of string methods where you're trying to isolate the characters that you were most interested that make the most sense for what you're attempting to do. In this case, what I want to do is I might want to take that data and convert the numbers to integers if I want to do that. So the split is nice enough to go ahead and create this list of strings. And then I can convert each element. So there's an a zero and there's going to be an a one a zero in first iteration through is going to be a two and a four. So and then it's going to multiply those two integers together. So then we're going to get an eight and three times five is 15 and so forth. So I'm putting these print statements in here because I want you to visualize, you know, what the variables look like each time one of the method is run on the original string that was read from the file. Because sometimes, sometimes you have to process strings, sometimes they have characters in there that you don't want that aren't related. You know, I don't know why I chose to multiply the numbers together. But the point I'm trying to make is, is that if you were trying to do that, and there was junk in the way that would stop the conversion to a numerical value taking place, you need ways to get it out. You have to flush that stuff out. So our strip and split are helpful methods that help clean up the string so that you'd be left with a list. So you can actually compute stuff, you know, because again, that's one major goal. There's usually two directions you're going in when you're when you have a text file.  But you also might want them to be to be numerical. And that's that's oftentimes how text files could be used. And if they are numerical, you've got to sometimes you have to go through a, you know, a process, you got to kind of navigate through the the extra stuff that's online that you don't want. So that's why I'm showing you this example, I'm just trying to show you how to kind of create a loop, save the stuff you want, you know, keep the hay and spit out the straw. And then once you get those data points, you can go ahead and multiply them for for lack of anything else that you would want to do. So I could go ahead and, you know, this is this is really what you would be after in terms of result, I can remove these print statements, but I still want to print the data out like that. And so and then you would see this is this is what you would be after that's what that instruction would be doing. So I'm just trying to show you how to process data using loops, reading lines at a time. And that's that's all that example is there for. Okay, maybe we'll try to make a decision just decisions decisions. Let's do this one very quickly. So a little more time. Yeah, I'm just I'm trying to. So maybe to relate this back to you know something that we've, we've done before me pull this out for a second. Again, just in terms of processing and relating this to other functions, we're gonna we're going to put data into this file now it looks like it's number one 2.3 3.14 and five. And maybe I'd be interested in in summing the values, you know, and we know how to do that you're all experts now you know how to write a loop to, you know, to perform a running sum. So in this case, you know, you could you could get the data we can we can set our sum equal to zero, and then line by line we could read we could, we could read each line which has which has the data on it and and convert it to a flowing point number and then print out the sum. Oopsie daisy, something's not going right. So try it again. There we go. Okay, and you get the sum 11.44. So again, this is the flavor of stuff. You know, why are we creating the files in the first place? Well, you're planning on writing programs with them, aren't you? You're planning on either, you know, writing the data for a reason, reading it in and then processing it.  And so so hopefully at this point, you could you could write code like this in your sleep assuming you understand that this is the kind of the the the way that you would iterate through the lines in the file. So this is all very natural, like, like I really want you to go back and compare this this code to previous units. If you left this line out, okay, if you if you left out the f in some sense, and I just wrote this for it, the code looks exactly, it looks exactly like, like what we've written already to some of the elements in a list, for instance. So if f happened to be a list, then then you would be you would be cycling iterating through the elements in that list. So the only difference now is that is that Python is smarter to understand that you don't mean, you know, the list or the dictionary or what have you what you mean is is the file because f is a is an object that has it's like a big package, it's got a lot of stuff, and it kind of figures out what you mean by that. Okay. Maybe one more example. Let's try one more thing. Let's say, okay, so let's let me let's think about this question for a second, because this is important again, just to make sure this hits home, you know, always good to, you know, doesn't like a test or something like that, but it's always good to ask a question to make sure we're on the same page. Maybe what I'll do is is I'll go ahead and I'll go ahead and and and write some code like this. Okay. And I'll, so I'm going to write a 22, you should be able to look at this and see that there's a 22 in there. And then we're going to go ahead and read it. Okay, we're going to go ahead and read it. And the question I have, and I think we'll just close with this is, is that if I use the, what, let's do it like this, just just to review, okay, because this is important. If I, and I'll create another variable Y, okay, like that. And if I were to print the type of variable that, for instance, Z is versus printing the type that, that Y is, okay, if I would print the variable types, the question is, well, we want to be 100% sure of what happens when you go ahead and use this method versus this method. Okay. And it's really important to know the difference because it's going to affect, you know, how you would code after this after grabbing the data. Okay. So, so whenever I, if I, from teaching a course, you know, I would ask a question, I really want, I really want people to become like mental Python interpreter, executor, code executor, script executors, so that you, so that you're kind of reading the code and you're, you're kind of executing it line by line by line by line. In this case, what's happened is that we've created a file with a 22 in it. Again, not a very exciting file, but a file on them the less than it's got a text 22 in there. And then you want to read it. Okay. In this case, this function, this method is to, is designed to read multiple lines. This is, this is designed to read the whole file. Okay. And, and so, but there's only one line in the file. There's only one line there. And my question is, my question would be to students, if I were actually, you know, in a class or something like that, I would say something, I would ask a question like, well, what type of data is Z, and what type of data is Y? Okay. Because you've only got one thing in here. All right. And that one thing is, is a, is a 22. Two characters equal to two and two, it looks like a 22. But it's actually, whatever the ESC equivalent of a two is like, you know, like maybe a 90 something or something like that. And then the, and then the equivalent of, so, so, so what is the data type of these two values? And so the, so we said the read lines, you could get, I guess what I'm saying is you could get faked into, you want to be careful when you do stuff like this. This, the read remembers is a string, and then the read lines is going to be a, is going to be a list. So even though I'm reading only one line, okay, the read lines is still going to create a list at the end of the day. So, so the first one is a list, right. And if I would say something like, you know, print Y versus print Z, okay, you should see this. It should be, it should be quite visible that, that the 22 really is a string. And that, and that the Z variable is a, is a list whose element is a string. And of course, there's a difference between those two, obviously, you know, the first is an actual string, the second is a list, the two different, the two different data types. So one word of advice I would say is that, is that when you're using these functions, get really comfortable with the container or the bucket that the stuff is coming into after you perform the read, because that's going to have implications for how you write code after you read the data. Okay. So I think I'll stop there. So just the, I think to summarize is that, you know, first of all, understanding how to open a file, how to either read or write to a file or append a file and close the file. And, and then, you know, what you want to do with the data, how would you handle the data, writing loops, writing code to interact with the data. My point is, is that you should be able to pretty much take anything that we've done before, writing functions, writing loops, writing if statements, all that stuff can come to bear to this section. And you should be pretty much able to do anything that you could do in previous units, you could do that with the file data. And that's all this unit is about. It's just how to sort of get data quickly out to a file and get it back into Python, so that you could in principle operate on it. All right, well, that was great. Thank you very much, Dr. Eric Sack. Everyone's giving you a round of applause, I imagine. So that was a recap of Unit 7 of Sailor Academy's Introduction to Python. We'll be back next week for Unit 8, Regular Expressions. So if you haven't caught up yet, you know, try and catch up with it now, join us in the live chat. If you have questions, you're watching this later, feel free to put them down in the description or join the discourse discussion forums on at Sailor. The link to that is in the description as well. Again, thank you very much.


consumer data. Do you guys agree? If you do raise your hand, I'm going to present it to you. So the question is why? EU. What did you say? EU with GDPR. Great. What else? Snowden, okay? HIPAA. Data breaches. Data breaches. Awesome. Exactly. So you guys know all of them. So we're seeing breaches, regulation, and decentralization that are really changing the security landscape. And that's what I'm going to be talking to you guys about today. So my name is Madison Current. I am a software engineer and developer evangelist at Ironcore Labs. And over the past couple of years, we've seen some really interesting trends in the security landscape, specifically around privacy. If you guys have read some of my blog posts, there's a lot of current events in the space, and there's a lot of regulation as well. So let's kind of brief on what the current state of privacy is. Back to our criteria here, right? I think that breaches, because of disclosure laws, are getting more and more hefty fines, and we're securing a lot more of them, right? Because companies now have to tell us. And we're also, it seems like every day in the news, we hear about another breach. Companies are getting closed off the right. And again, the question is what? Well, data proliferates. And essentially what used to be one system, an on-prem database that used to store all of our data, is now getting more and more complex. And it's getting more and more complex because of mobile, because of cloud infrastructures, essentially two or more in one app. And it's also getting more complex because of microservices and because of that derivative databases. So data that was once stored in one place is now stored in many, leaving a lot more of a vector for breaches. So let's say in a little bit. We've spent millions on security, but they still lost how to breach that affected half of American adults. This is because of the high complexity of the system. So they had a lot of disparate systems, and they had scams on a lot of the systems, but the scams were broken. And what turned out is that they have had 30 points where they could have stopped the breach, but didn't. The worst part of this whole thing is, is that it was entirely incremental. And this is my favorite one to pipe on, and I can't do a privacy talk without talking about it. Facebook. Seems like every day, if not once a week, if not every day, we're hearing another Facebook breach, and this is a different kind of breach. This is a breach of trust, where they're using customer data to, they're either packaging and reselling, or giving it away, or they're using it for purposes that were unsupposed to their users. So let's dig into a couple here. So when I said packaging and giving away data, so Facebook doesn't sell your data, they treat it for more users. So here's an example for you. So Facebook gave away data to Microsoft Bay. They gave away your user's friend list. They gave away data to Netflix and Spotify. They gave away data to Amazon. And they gave away data to Yahoo. And in return, they get marketing for more users. And they have similar deals with Apple. The reason for this is that they consider their partners an extension of themselves, only as it pertains to privacy policy. Facebook location tracking tracking. Who thinks that they can turn off location tracking on Facebook? Okay, no one. Cool. I should have waited to turn this slide. So when you turn off location tracking on Facebook, they stop using your GPS, but what they still continue using is your IP address location. So they still know where you are pretty, pretty close to accurately. And again, what they're doing is that they're lying to the public, they're misleading us, and they're breaching trust. So what we're seeing is that companies are being irresponsible with data, and they're undermining customer trust. And as a response, we're seeing global, public, and regulatory backlash. And we are the collateral deal. Let's go on a little bit of the brighter side now that we took it to the spectrum. All right, so privacy laws, global privacy laws. In the last year, there has been either enacted or passed six major privacy laws around the world, including GDPR in the European Union, CCPA, California Consumer Privacy Act in California. There's also been privacy laws in Japan and Brazil that have come down to that. There's hundreds more following. Also there's kind of states are pushing for their own privacy laws based off of CCPA and GDPR. Some of those states are in Mexico, Massachusetts, and Washington are pushing for their own privacy laws. Last is decentralization. So who here has heard of blockchain? That blockchain is really born out of the same ethos. It's trying to solve some of the same problems. Now I'm not advocating for the security of the blockchain. It's not cryptographically secure. It's hashing, so we can talk more about that. But it is born out of the same ethos, meaning that we're decentralizing authority. And it's a way that we can reconstruct the behavior of the internet, which is becoming more and more centralized over time. And it's also a lot of people that are developing blockchain believe in the same concepts that the internet was built on, which is the freedom of information. Okay, so that's all good and great. But what are the implications on you as a software engineer building your apps? We kind of talked about the whole global state compliance regulations coming down the pipe. But this has a direct effect on the software that you're building because it affects the data that you have in your applications and what you need to do and how you need to architect your applications to handle that data. So now we need to architect for privacy. And that's what I'm going to talk to you about kind of in the continuation of this talk. So three steps that we're going to follow is that we need to analyze what data is in our app and see if these even pertain to you, these regulations, right? And the data that you're holding. Then we're going to architect solutions. So we're going to look at systems and see what kind of tools are available to us to comply with regulations that are being put on the data that we're holding. And then last, I'm going to show you kind of a quick demo of end-to-end encryption, which is one of the ways that you can handle analyze first and then like to interact. Okay. Who here has sensitive health information in their applications or the applications that they're building? Okay, keep your hands raised. Okay, who has finance information? Okay, any sort of user-generated contents, anything like posts, recordings, chat logs, support requests, anything like that. Keep your hands raised high. What about user names? This affects all of us because this is the data that is protected under these regulations.  And the biggest one, again, is to reiterate, any user-generated contents is protected. Okay, so how do we architect for this? This seems like a big, hairy problem that I don't want to handle. Okay, there's a couple ways we can do this. And by the way, in this talk, this is not legal advice, but we are going to go through legislation to kind of figure out what our obligations are and how we handle this data. Okay, so looking at Article 25 of GDPR, it says that data needs to be protected by design and by default. Okay, it's a couple things here. It means that it's incumbent on you as the application developer to protect the user's data in your application. And you need to design your applications accordingly. I think I got this later in the presentation, but I think it does make here, as you're designing applications, it's really important to document how you're going about designing, to show that you were designed with privacy in mind. Okay, one of the core concepts here is minimization. Minimization means that you're collecting only what you need, only the data that you need, and you're only holding it for as long as you need it. So that means if you are collecting birth dates, but you're not using that for any reason, that's illegal. Under GDPR, this is where that slides. Yeah, but document, document, document, everything as you're going through the design process of how you're building your application so you can prove that you built your applications with privacy in mind. So then we're going to go to GDPR, Article 32, the security of processing. And I can, for the life of me, not say this word. This is what I was practicing only yesterday. Pseudonymization. Everybody try it. Pseudonymization. Pseudonymization. We're going to call it de-identification from now on in the presentation. So just going back here, this is really the de-identification of personal data and that you need to product against abuse and abuse of insiders, meaning that if somebody had access to the database, they couldn't tell who each log entry belonged to. Right? So de-identification. So this means that we put the data in our system in a way that we can't recognize who a person is. So they could have information about me, but they wouldn't have my name, Madison, in the table. So the kind of unique idea you know I did in my work, right? So this is a way that you can comply with your degree. It's a potential tool. There are some issues in it though. We're going to look at an article that New York Times wrote about its de-identification and their process in being able to re-identify the users from this data. So it turns out that weather apps like Weather Channel, which is owned by a subsidiary of IBM, tracks your location data, de-identifies it, packages it up, and then resells it. Now when that's combined with public data sets, it's very easy to re-identify who someone might be. And that's exactly what New York Times did. So they track location data of individual or bought this data set and then combined it and they were able to pinpoint a woman and they came up to her and they said, so how was that doctor's appointment last? Because they knew everywhere she was going. They knew her workplace and her home. That's easy enough to identify you, right? So and then they looked at the patterns and they said, well, where is she going that's not those places? Oh, she went to a clinic. Oh, she stopped at the store. Maybe she went and stopped at A&E at a certain time. It's pretty private information, right? And it's easy to put two pieces together to re-identify an individual user. And I would argue that it's really not okay. It's not enough. Okay, so let's talk about encryption. So full disclosure, I work for a company and we provide an encryption solution. I'm not going to talk about that today. What I am going to talk about though is the technique of encryption and how it can be used in your applications. And what some of the benefits are. Okay, so oftentimes when I talk about encryption, I hear like, I've got encryption. And most of the time it's transparent disc encryption. And before we dig any further, I just want to kind of dig into that a little bit. So transparent disc encryption is not enough. It encrypts data while your machine is off. But as soon as it's on, what's transparent to you is also transparent to potential hacker. So transparent disc encryption is not the answer. We're going a little bit more in depth. And then another thing to note here is that when we keep flipping back between GDPR legislation and technology, GDPR is not prescriptive on the technologies that you use or how you implement the features that they're asking for. But they are asking for specific capabilities like logging of every access element, like knowing who's accessing it. Right? And there are certain ways, technological solutions to how to implement that. And the truth is that they're not going to look at the technical implementations and say, oh well they want their research done everything they could. They're going to say, no, if your data is lost you probably didn't do enough. So let's take into some architectures. I'm going to show you three architectures and we'll kind of talk through them. If you have questions in this piece, please ask them as we go through. So three patterns of how you can use encryption and architect that into your application. Again, these are based on public cryptography. There's open source libraries for these and you can create your own encryption services. So first we're going to talk about server side proxy and what that architecture looks like and some of the benefits of it. Then we'll look at server side application. Mr. Hyphen. Kind of the server side application and what that would look like. Again, some of the benefits and trade-offs. And then lastly we'll talk about client side encryption and this is what all of that wants to do with you guys as kind of the implementation piece. So server side proxy. Here we have our device at the top. It talks directly to our application and then that application talks to the encryption proxy which has access to individual keys that it then is able to unwrap and send back to the application. So the application here will say, I have data that I want to decrypt. Can I have the key to decrypt it? Right? And then it asks, that event is logged. First, so Apple is an access control list meaning who is allowed to see this data and who is not, right? It's checked to see if the application can read and then the event is logged. Now one thing to note here is there's not very strong authorization between the application and the proxy letter in this architecture and that can lead to some potential downsides. I'll play this real life so you can't see the server side like the server block just kind of a bummer but it cuts off right after the device. Any questions here? Our potential architecture is the server side application and as we've been doing more and more market research internally we're finding a lot more traction with this kind of architecture because it relies on servers that are reliantly fat. And so in this architecture the application will request keys from the encryption service, decrypt and then be able to pass that back directly to the device. I was just wondering, the encryption keys would that be something like a JWT? So no, they're a little bit different. So JWTs verify identity whereas encryption keys are they're more private, they're longer or lasting than JWTs and they are the key to actually go ahead and decrypt the data. So if you don't have the key you cannot like re-audit, you can't come up with it again. It's not backwards, right? With JWTs you can ring pass it but most of the time in these systems JWTs are used to authenticate a user and then that's kind of proving the apple or proving who somebody is and then they can go ahead and get the key to decrypt. So that's normally how these systems work with identity. Other questions? Yeah, so in the other one the encryption proxy just has encrypt and decrypt functions whereas in this it's actually passing the key back to the application only sort of memory for a short period of time. So it's the difference in key now. Am I saying? The proxy to the name. Exactly. Application. So this is probably the most robust of the three architectures but it's a lot less common. So the device actually has the key on it and the device is the only thing that can decrypt. Not the only thing. Sorry, the point of use is the only thing that can decrypt. So disregarding this for a second what you have is an application that sends data to the device. The device has the key on it decrypts the data and is able to view decrypted data. So the client is the only user who can decrypt that data. Audit events happens from every single interaction there. But what you might be asking which is kind of the logical next step here is what if I have a service, a microservice in my architecture how do I access that data? Well what you can do it is that you can give that microservice a set of keys just like the client. So when I say point of use it can be a service as well. Give them a set of keys. Give it a set of keys. It can decrypt and every time that service decrypts it's locked as well. Which is what you need under regulation. Questions here? If you can do a key wrapping technique here where you can pass them an encrypted key that the application never sees the decrypted version of the key pass the encrypted key to the application and then the client can decrypt it. But to your question I think it really depends on the application. So while this is maybe I don't want to say this is idealistic but this may be a better fit for a green field app. They're really from the ground up but in the case of a brown field app or something that's already been built before that you have services that you can be able to talk to each other retrofitting something like this is much harder. So what you'll notice around all these architectures and that's such a great question is because it really depends on your tradeoffs. And what you'll notice is that there's some sort of separate service encryption service that allows you to log the point of access at every single increment, right? So what this is for us is it's an unavoidable access control and audit logging layer that ensures that nobody can see the plain text of our database without going through that layer, right? And that's what's critical for the legislation is that piece because we need to know who's accessing it and when they're accessing it and how often and be able to turn it off, right? And so by this having the access control list we can remove somebody from the list and they can't access the data anymore, right? Which is exactly what GDPR pushes forward. And same with CCPA. So next piece is a quick demo demo guide to be on my side. So I was going to show you kind of an implementation of the last architecture just the encrypt functionality and show you that text can be encrypted on the client before it's ever sent to the server. So the server never sees the encrypted text. Oh, not going well so far. This one? This one? Okay, so what I've got running here Okay. So what I've got running here is I've got a React app that is using how we recognize that material design it's using Redux regrettably not using TypeScript and it's connecting to a data store database in the back end. That doesn't really matter. What I want to show you is the actual request here so really good password I want to make sure we do that So to your point here's the DWT, it's off things for DWT So what I want to show you is that in my Redux action I'm actually calling the encrypt function on the client before anything is ever sent over the network. So while my message was an important information it's now encrypted and when this is stored on the database it can't be viewed in plain text Does that make sense? I can go ahead and decrypt that because I have the key to decrypt it but it was also encrypted on it I think we're still doing okay on time so I'm going to dig into a couple more articles in GDPR Another one is the right to erasure meaning that a data subject or an individual user has the right to ask you to delete their data and any data that you pull up So think about this in terms of your database architectures If you have a store and only one that's easy enough to do, right? But what if you have a message bus that's feeding all of your micro-services downstream all of those micro-services have separate databases becomes a lot harder task to achieve in really complex systems and this is kind of the origin of a lot of the problems that we're running in that we're taking when regulatory backlash aren't so unless you have legal obligations such as tax purposes health reasons you are not allowed to deny these requests right? There is a little bit of a loophole that it's like well it's way too hard to not see what you might think you can get through it in GDPR that's not the same for CCPA or legislation coming down the pipe that's getting stricter and stricter around what your legal time But what is a really cool technique here is what I call I don't think my co-workers like this term but I really like it it's called crypto shredding so the idea that if something is encrypted in your database, if you delete the key or you take somebody off the access list it's still on the database but nobody can ever decrypt it because the key's gone so it's a really cool way to handle this there's also a lot of backup solutions that are implementing this for you so they actually support the rights that you've forgotten and they're helping you with access blocks so at the same time that we're seeing regulation coming down the pipe we're also seeing other tools coming and being at a good disposal but it's not really a one tool this is going to solve GDPR for me it's a set of tools and design practices that are going to need to be put in place to really really solve this okay so article 7 conditions for consent this is an interesting one so it's your legal basis for holding data and you need to have a legal basis for why you hold data in your application so you can't just overly collect and be like well we don't use it it's fine, you can't do that anymore there's some really interesting tools around managing granular consent so these are some other tools that I found that are trying to solve this right now definitely not endorsing them, try them out there are things out there for you what they do is that they give granular consent or they allow the user to give granular consent to different APIs or services that are accessing their data then you can turn on and off and see how it's being accessed and they also help you manage some of the complexity because this may seem like it's kind of like cookies well to show accept cookies but it's not that easy anymore and the reason being is that you need to show how to track how you're getting data from individual users how long you have it for and when you change how you're using data or any third parties that might be using that data you need to get fresh consent right because you can have sale agreements that users haven't agreed to and specifically if you got data from a third party this can help you track third party agreements, contractual agreements for how you're using data help manage that because it is a basic problem so in conclusion we think that we know how to architect applications for today but technology is only part of the equation for us now and we cannot treat consumer data as a property of the business regulation and backlash is putting the right of the data back on the hands of the consumer and that changes how we architect applications and the truth is as an implication on our software developers is that we need to be aware that we face and that come along with steep financial burdens if we mess this up and reputational penalties from now until from now on we need to be able to build our applications with privacy in mind thanks for having me here first question for you great if I talk to my echo friends, my old teaching friends and so forth about things like privacy they say I don't really have anything to hide why do I care should they care? they definitely should care so I actually was watching what I did my free time because I watched a bunch of TED Talks so I was watching this TED Talk and I was talking about exactly that statement there as to I've heard that a lot as well well I have nothing to hide, what does it matter? but what I kind of shown this example is that data proliferates and is no longer in the control of the consumer and so I think that that statement is from a government standpoint saying that the government is going to watch I'm using data but what about companies that are reselling that data and using it to kind of manipulate what I'm seeing no more about me and see things that are private and they change if I knew it was public it may change my behavior for me personally one that always caught my attention was when insurance, car insurance companies started offering you a discount if you install the old tracker and I'm like like this is not going to be good for me if they're tracking all kinds of data so even if you're not if you don't consider yourself a really privacy focused person there are plenty of examples of how this data can start to affect you in small ways that's a good time quick question about GDPR and decentralization so GDPR requires the right to be forgotten but if someone is decentralizing data on blockchain and blockchain can't be deleted how do those two coincide? yeah I know it's a really interesting problem and it's one of the limitations I think of blockchain there are potential solutions with encryption what scares me with that is whenever you have immutable logs as technology gets better and better over time what might be secured today as far as our encryption algorithms may not be secure in the future so I think that's a big problem it's a big limitation of blockchain some architectures that I've seen are potentially pointers to documents so you're not actually storing any information on blockchain I mean blockchain is just the technology that we're using but it's really a thermal log it could be any system that's using a thermal log there are major limitations and some worries with that but if you're storing the pointers to the data you can still go ahead and delete the data potentially there's some architectural approaches to that so like client-side encryption does that end up becoming expensive to actually do that encryption on the client-side does that end up affecting the UI or the user experience at all in having to do that on the front end and then sending it to the server yeah I mean there are performance implications specifically with now this is probably getting too in the weeds with multi-option encryption when you're encrypting to an entity or a group and then a user encryption operations can get really expensive that's why I think just from what I've seen building it in a house can be really complicated there are solutions out there there are credit libraries like iron cores making one actively right now and working on performance issues we're using technologies like WebAssembly and Rust and we've seen some major major gains to the point where it's no longer a big cost but that takes optimization and it takes tweaking your algorithms so that's not the right term but finding the places where you can make performance benefits going off your example I recently built an application where are you I'm sorry I recently built an application using the same technologies and we're basically getting a JWT from our back-end authenticating the user once they're logged in and using it for ongoing requests of the database what type of behaviors would an encryption function have or a de-encryption function have and how would I be able to test that so it depends on the library so it depends on the library they decide to use but it would so JWTs are often used for off-offing which you just described what encryption does would be another layer of protection so you could use JWT to unlock a key that can then decrypt the data right so JWTs aren't being used to encrypt data that's sent to the server right is that true yes it has that was basically like just storing it in like the read-up store which is not like the most secure thing to do so I'm just curious like how I like ensure that it actually is encrypted once I have that gene yeah well based on the example I have all my code that I showed in the example which I'm happy to show you I'm using the ironweb library but it's just it's using the key like an encryption library to encrypt data on the client it's different from the JWTs although it's using JWTs but I'm happy to send you source code I'm using Redux and React in that sample app cool yeah I'll send it to you alright thanks Madison