So, if you recall, we discussed the notion of files last time. To recapitulate, we have been using files primarily as input redirection or as output redirection. So, basically when we have voluminous data, instead of typing it every time when you run a program, you will edit a file and give that file as input through a redirection mechanism. Files are actually more important than that. And as a prelude to understand the importance of data in file, we will first try to understand the notion of structures. After that, we will do a very quick review of whatever simple things about files that we have seen earlier. And then, we will look at some algorithms which will work on the files that we define which contain data. More important for us will be the notion of direct access to records in a file. In that context, we will be discussing files which we have not seen so far, namely files containing encoded data or called binary files as opposed to text files that we are all used. So, there is an issue. Typically, I would declare variables or arrays in my program to handle different types of values. Take for example, a hypothetical roll number, not IIT Bombay roll number which has capital D and such thing. At most other civilized places, roll number is an integer value. So, we will take this as int. Name will be what? A care type array because the name could be 10 characters, 20 characters, 30 characters. A lab batch, as we know it, is 1, 1, 2, 3, 4, 2, 3, 4, 1. It is again another int type value. Marks, should it be an int? Float. You do not want to lose 0.5 marks that you gain. So, mark should be float. For all int and purposes, these four are independent variables. We understand that all four of them describe some attributes of a student. But as far as the program is concerned, these are just four independent variables. Memory has been allocated. Computations are done as and when you operate upon. What we would like to do actually is that we would like to treat all four as if they are a part of a single structure. For example, if my roll is located somewhere here, let us assume for the moment that memory is allocated continuously, then this will be followed by my name which would have many characters. So many assuming four bytes for roll, there could be 30 bytes for name for example. Then there would be batch and then there would be marks. What we would like to say is that all of this should be part of a single structure or a single organization. So, this entire thing, the actual memory locations for each one of these four parts together, we would like to refer to it by some name. Let us say s. Ideally, we would like to take this entire s, put it in memory, take it out of memory, put it in disk, take it out of disk. Although we would like to retain the individual components and access to them as well. Such a facility is available in C plus plus and in most other languages through the possibility of defining what is known as a structure. Here is an example of records in a text file. You would have seen almost exactly the same file minus the marks in the last lecture's example before the midship. There I had put these values as separated by comma and we had introduced an extra comma at the end, so that we could process using the comma as a delimiter three different components. At that time, we had seen a program which extracts these in three different arrays of character. So, notice that we were treating 1, 0, 1, 0, 1 as five character array, not as an integer number. The name will have to be treated as an array of characters because there is nothing else that you can do. But the other components, we would like to treat as int, float, int, whatever. And as I said, we would like to have entire conglomerate of this information about one student to be referred to by some single name. Actually, if you look at records in a text file, these are lines in a text file. So, there is an additional terminology that I would like to introduce. If you see each line, what does it describe? Information about one student. Second line describes information about another student. Does the second line contain information which is different kind from the information in first, third or fifth line? For example, first line has roll number name, batch number and marks. If the second line contains roll number, height and weight of the student, then that would not be in consonance with whatever is there in the first line. It is very obvious and natural for us that whenever we construct such a situation, whenever we construct a text file, we would have exactly similar attributes in each of those lines. Values will be different for differences. The naming for this kind of notion comes from the world of data processing or information processing. And these are the new terms that I would like to introduce. File, you are already aware. As far as you are concerned, file is a sequence of bytes. But if it is a text file, you can open it, see it, edit it and so on, because you see the text symbols there. In our text file, for example, containing the student's data, we see different records. There is one record for one student. More importantly, each record has values of exactly four attributes. So the notion of record and attributes is important. The attributes that we have here are roll number, name, batch and marks. These are called fields of a record. The first field is roll number, second field is name, third field is batch, fourth field is marks. In a text file that we have seen, these fields are separated by one or more blanks. Last time we had seen a notion where the fields were separated by comma. Can blank always work as a separator of the field in a text file? Can you always recognize different fields if they are separated just by blank? Possibly not, because a name, suppose instead of writing my name as Deepak, I write Deepak Phatak. I won't write it as a single, continuous work. I will have a space between Deepak and Deepak. So space is a natural component of any string. And therefore, if any string is a field in your data layout, then that field has to be separated from other fields by some other means, not by a, not by a blank. This separation of field is a different issue. We, we have tackled it earlier by saying comma separated fields and so on. But imagine the other possibility. The other possibility is, suppose you can pre-define the width of every field. Suppose you can say that name will be exactly 30 characters long. The roll number in character form will be exactly 5 characters long. Then actually you don't need to separate these fields. You can just count the number of bytes from the starting and you know exactly where one field starts, another field ends. If you agree with this, the same concept could be implied in a file which does not retain data in text form. Suppose I have an internal representation for int followed by an internal representation of float. You all know that internal representation of int takes 4 bytes. Internal representation of float takes 4 bytes. Suppose I wrote these two fields one after another in a file, in a record containing internal value of int representation, internal value of float representation. It will still be valid because if I read 8 bytes, I know first 4 bytes are these, next 4 bytes are these. Of course, I can't read them using a normal text editor and so on because this will be internal. How nice it would be if I could take an int value internally, directly write it to file as int and later on when I read back, I directly read it without any scanf or without any scene or whether directly take it and put it in the memory location. No transformation, conversion from string to int, int to string, et cetera, will be required. The binary files which I mentioned, one of the things which they do is they handle data in such a form. However, coming back to the nomenclature, I would like to introduce one more term which is called meta data. You saw the file, let's go back to the previous slide. This slide has actual data. This is the first roll number, second roll number, third roll number, fifth roll, et cetera, et cetera. Of course, you'll call it one record for a student. Each record has four fields. The fact that record has a name, each of the fields has a name and each of the field has a characteristic. Roll number is an integer value between this range and this range. If you recall the mid-same example, we had seen an account number was stipulated to begin with one, a customer ID or whatever, account number was stipulated to begin with 411, et cetera. Some of the characteristics. The description of these names of fields, name of the record and characteristics is called meta data. The word meta means data about data. So meta data is a description of what is the record layout, what are the different fields in that record, what are their characteristics, et cetera. What is the meta data for us, the name of the record and the attribute names. We observe once again that the name string, this has nothing to do with the name in what you call this name. This is the name string as in this name, name of a person. So string could be of variable length or different length for different students. That is one of the reasons why you see the records. Some record shorter, some record larger, some record shorter. Look at this, records in a text file. All that I have done is I have reproduced the data actually, the records and indicated the names of the fields at the top. So I have roll, name, batch, marks and then we have record number zero, record number one, et cetera. That's because all counting in programming begins with zero. In fact, when you have a text file like this, you will actually refer to the first line as 0th record, next line as first record, next line as second record and so on. Observe also that these records are not of the same length. What could be the advantage if they are of the same length and what is the disadvantage if they are not of the same length? Imagine an array that you declare, let's say array of int or array of float. Each element of the array is of exactly the same length as any other element, four bytes for example, for a float. So suppose you have named an array A, then to go to any element of that array, say ith element, you say ai and we have seen already that the position of ai can be quickly located by c plus plus by using the base address of a zero, adding that i to it and saying, okay, this is where the ith element must be. This calculation could not have been done if different elements were of different lengths, then you won't know where the ith element is unless you sequentially look at each one. So the advantage of a fixed length information is great when you organize several of these. Whether you organize them in an array in the memory or whether you organize them as records on a disk, the advantage could be same. We do not know about this as to how even if a file contained records of fixed length, can we like we say ai equal to something or p equal to ai, can we say get me the ith record from the file directly? Don't have to scan sequentially. The fact is, yes, it can be done and that is the reason why whenever you have to maintain meaningful and critical information on the disk, you generally don't maintain it by varying length records. You maintain that in fixed length record. The point is that, suppose this is zero ith record, this is first record. If they were exactly of the same size, say 100 bytes each. Now, suppose these 100 bytes are written on the disk, 100, 100, 100, 100.  Now, if the records are fixed length, when you say go to 500th byte, you know you are going to the beginning of 6th record. Whereas, if the record lengths were different, there is no way you can connect a particular position of starting of a record with any pre-calculable value. And the comparison I made was with an array. The fact that you can say ai and the computer can quickly go to ith element is because each element of the array is exactly of the same size. So, starting with the base address, you can calculate what would be the address of ai. That is the point. Structures in C permit us to do two things, simultaneously. One, they permit us to collect different pieces of information, typically attributes of an entity, fields of a record, put them together and they permit us to give a single name to that conglomerate. Second, such a conglomerate is always of a fixed length, which will help us later on when we deal with records on the file. Struct is the keyword that is used. This is useful in reprinting entities which have many attributes of distinct types. So, suppose we wish to handle this, say, roll number, name, hostel, sorry, there is a syntax error here. What is the syntax error? I have roll and name, but not hostel and room. This is what I originally started with. The first one is batch and the second one is now observe what C plus plus permits us to do. It permits us to define a new data type. Just like int, float, it can define a data type which we are calling here student info. Student info is not the name of a variable. Student info is name of a new data type which we have defined. It does not exist in C plus plus and we are saying this new data type is a structure which has the following components. The components have names. So, I have int roll, care name, 30, int batch, float marks. See the way it is defined here. Struct, student info, opening curly bracket, four components with their own individual types and names, curly bracket close. There should be, of course, a semicolon at the end. So, this is the definition of a struct. Now, having done this, I can define variables of type struct, student info. So, that type name is not student info. Type name is struct, student info. Just like int, float, struct, student info is a new type. As opposed to the other types, int, float, and care and other things that we have seen which are incidentally called native data types in C plus plus, these are called abstract data types. Later on, we will see when we discuss the object oriented programming concepts. You can take this to a larger extent where you can define what we call user-defined classes or user-defined types, and you can build upon types of types of types, et cetera, et cetera. So, you get a lot of powerful expression features, but currently we limit our discussion to this. Note that a struct definition itself does not define any variable, and therefore no memory is allocated. For example, int has no memory allocated to it. Int x has memory allocated to x because x is a variable. Similarly, struct, student info has no memory allocated to it. It is a structure. It is a new data type definition. But if I define variables like this, struct, student info, s. Now, s becomes a variable of struct type, and when s is a variable of struct type, then automatically memory is allocated. How will memory be allocated? Two important facts. Memory allocated to structure is always contiguous, like array elements are contiguous. No confusion there. This is ensured by C plus space. That means the first four bytes will contain what? Role. The second four bytes, how many 30 bytes or so will be named. This will be followed by batch. This will be followed by marks. If I had another variable of this type, say s2, then if this is s, there will be another thing for s2 here, exactly the same thing. As many variables as I have, so many blocks of memory will be allocated to those variables. If I say array struct, student info, list 1000, then 1000 such blocks will be allocated, and the index of that list array i, let's say, will refer to the ith block. That is all very nice. But how do I, for example, I don't want to operate upon this conglomerate. I want to operate upon individual roll number, marks, et cetera. How do I tell C plus plus that when I say roll, I mean the roll of student s and not that of s1? I have to distinguish, therefore, between components belonging to different struct info type variable. That is done by what is known as qualification. In our program, if we write s dot roll, s dot name, s dot batch, and s dot marks, where s has been declared to be a variable of type this struct info, then we automatically mean those particular memory locations which are allocated to s. Is that clear? So, it is a new name. So far, names of variables are arrays where contiguous words as per the rules. But this is a qualification. s dot something means this something is part of the variable s, which is a variable of the structure type. That is right. In fact, the allocation of memory is, question is right. The allocation of memory is always in the same order in which the definition has been made here, roll, name, batch, and marks. It is interesting to go back to the notion of file and fields that I mentioned, and that will convince us why this must be so. We said that there are four fields in every line of the student, every record of the student. We also said that the first is roll number, second is name, third is this. Suppose we want marks to be first, and suppose you said first record contains roll number followed by name followed by some. Second record contains marks followed by batch followed by name. Wouldn't there be confusion? Because these are only values, and some program is going to interpret these values. If the order in which the field values are given is not identical in every record, we would have a problem, and that is why the order is important. So, to recapitulate s is the structured variable. Its components are s dot roll, s dot name, s dot batch, s dot marks. And just like any other type, the struct student info also has s size. What is the size of int? Four bytes. What is the size of float? Four bytes. What is the size of a care array? As many bytes as is the size of that array. So, if it is name 30, 30 bytes. There is some confusion. In this particular case, the size will be 44 bytes. Unfortunately, if you look at the individual components, they don't seem to add up. Let's go back. Int roll. How many bytes? Four. Care name 30. How many bytes? 34, 38, 42. There is nothing extra or hidden inside a structure. Unlike in some cases, in some peculiar files, there may be some extra markings done by the operating system. But as far as structures are concerned, there is only one rule that prevents us from getting 42. The rule says that all integer and float and double type of variables which occupy four bytes must start on what is known as a world boundary. A world is a four byte entity. So, while the memory is accessible by byte number 0 at first, second, third, you can go to any byte, an integer value cannot be stored from byte number, in byte number 2345. It has to be stored in 0123, then 4567, etc. Notice that the first element was int which was occupying the first four bytes. The next is a character array which occupies 30 bytes. 30 plus 4 is 34. At 35th byte, the next integer value cannot start by rule. It has to start on the next boundary, meaning an address which is divisible by 4. Consequently, two extra bytes are wasted in that structure. But those bytes are part and parcel of the memory allocate. If you ever write this structure on that is, those two extra bytes will get written. What will they contain? God alone knows. That is because we have no access to those bytes. We can only access the components of the structure variable that we have in the memory. So, those two bytes are extra bytes. These are called padding bytes, but this padding is not blank or any such thing. There are just two extra bytes allocated by C plus plus because of this rule. That is how you get the total number of bytes in this particular structure as 44 and not 4. You can guess that in any structure, if you have a single int or float element, the total size would almost always be divisible by 4, provided this such element is at the end. If at the end of a structure, you have this variable array of care, then there will be only 30 characters. No padding is required because there is nothing else that is to come later. Here is the declaration and the way to find the size of structure. So, after you are include IO stream, using namespace, STD, etc., you will say struct student info, int, roll, care, name, 30, int, batch, float, marks, and then when you begin your main program, you can say struct student info s. It is not necessary to define the structure outside the main program. You can define it inside. There is a reason why I am defining it outside and the reason will become apparent. You can define it outside. In fact, as I had once commented, you can define these structures in a separate file and just as you say include IO stream, you can include, you can say include structure.h. There is a separate file in which I have defined this structure. Mr. compiler, before compiling, include that file as text, put it inside and then go ahead. That is possible. Anyway, this is the way I define a variable of type structure student info. Suppose I have defined an integer variable rake underscore size to get the size of this structure variable. Please note that all variables will be of the same size and the size of that type is what we will apply. To get the size of a type, what is the function that we use in C plus plus? It is called size of. So, whenever you say size of int, size of float, size of this, we have discussed this in one lecture. You can get the size. Here if you say size of struct student info, then C plus plus compiler will calculate how many bytes will normally be allocated to any variable of that structure type. That is the type and it will give you those many bytes. That is how we get rake underscore size. If you print it, it will be 44 bytes. The problem that we wish to solve namely looking at putting students data into binary files and attempting direct access to those binary files. For that reason, I have put a single function definition separately which if I give a structure to it, it will print the values of various components. You realize that it would be troublesome if you want to print the entire record at any point in time knowing individual components. You will have to keep saying c out s dot roll, c out s dot name, c out s dot batch. So, rather than you doing it at many places in your program, you write a function. So, if you ever say print student s. Now, that s variable which is a structure variable will come here and this does not return a value. It is only printing output. That is why the function is void. Notice that instead of c out which I could have used, I am using print f which will give me what? A formatted print. So, for example, s dot roll will be printed as a five space integer number, s dot name will be printed as a 30 character string, s dot batch will be printed as a three space integer, s dot marks will be printed as a five space floating point number with two digits to a decimal side and in between each there will be exactly one extra black. So, this is the meaning of print f. That is how the value will be printed. So, this is a standalone function. Now, where do we define functions before the intimate? But if in that function I have referred to struct student info, then while compiling that function c plus plus, now what the hell is this struct student info? So, it will not help if struct student info is defined somewhere later in my program and that is why you saw let us go to the previous slide. That is why before intimate I have defined. In fact, after defining this, this is the position in which I can define that function. So, that compiler when it reads first it understands this is the new type that you are defining as structure and then it has a function which uses that. We now come back to the discussion on files. This is a recap of a few things that we discussed last time. First of all, the notion of a file. So, file has certain properties. It has a name, an extension, a path, permission, size. What is size? Total number of bytes which are recorded in the file. You see all of these on your Ubuntu by saying l s minus a or just l s you will get the names of the files size. You will also get when the file was created modified whatever lot of information. All this information is maintained by operating system. The physical location of the file and its properties are known to the operating system. You do not know that. What you see is the name and you can request operating system to give you more information such as by a command l s minus l or something. So, you will know what is the size what are the permissions what but the file is maintained by the operating system on the desk. You do not ever directly write to that or read from it. It is all done via operating system. Operating system is an intermediate. As far as we are concerned in our program, a C plus plus program treats a file as if it was an array of bytes. So, the size of a file is let us say 70,000 bytes. Whatever it may be a text file, it may be a encoded file, whatever whatever. These 70,000 bytes are treated logically as if it is an array of 70,000 bytes on the desk. Notice that array has a property. I can go to any element of the array. The disk as we shall see some time later has the property that I can directly go to any point on the physical desk and say read from here or write to here. Since the operating system knows where the bytes of my file are located, if I say go to ith byte of that file, the operating system can figure out where the ith byte is directly go there and at that position it can read or write say 200 bytes, read 200 bytes etc. That is the peculiar advantage of a desk. What are the differences? The differences are that the disk read or write is far more costlier in terms of time than reading or writing elements of array or I mean processing elements of array. Excessing an array element, you go to the main memory to that location. Do you know how long it takes generally on modern computers? Anywhere between 10 to 100 nanoseconds. In 10 to 100 nanoseconds, you can go to any main memory location which is electronic memory. And of course, extracting say 20 bytes of character data or 4 bytes of in data from that memory to processor may take a few additional nanoseconds or maybe 100 nanoseconds. On the disk however, if you want to go to a specific point on the disk, it takes at least 5 milliseconds. So, if you are reading let's say 200 bytes from the disk, you spend 5 milliseconds in going to that point and a few additional milliseconds to read that because the disk is rotating physically. In memory you may access some memory location in 100 nanoseconds but you may spend additional nanoseconds in taking that much data out of memory to the main processor. In general as a thumb rule, the factor of slowness of disk is 1000 times. Disk is slower by 1000 times than main memory. Disk is slower by 1000 times from the main memory. We will just park this information in our mind and we shall see why that becomes relevant. Now, we have seen text files so far but a file can be defined to contain encoded data. These files are called binary files. There is a clear distinction between the two. Binary files will typically have internal representation of in, float, double, whatever you have. Notice therefore, the structures that we have defined. The structure variables can be written directly on to disk and brought directly out from the disk. They will constitute a binary file. There could be other data such as digital pictures that we talked about. They are not float or something. They are in fact, we interpret one byte as int if they are monochrome images. But if you take for example, a JPEG file, it is a very complex structure. And there could be many other files, voice data. Now, if I sample a voice data for digitization by say 5-bit sampling, then every sample value will contain 5 bits, 5 bits, 5 bits. So, consecutive bytes will not contain individual sample values in full bytes. So, I may have to take all the chunk of bytes and internally say, oh, these first bytes, this value, next 5 bits, sorry, bits, this value, etc. All that is possible. Anyway, in a nutshell, files are either text files or binary files. Now, this is what is important. How do we handle a file within a program? We have seen this last time. It is important to understand this concept absolutely clearly. A file is handled through a file pointer. So, this is my file. Let's say, marksData.txt. This is one file. There would be hundreds of files in the disk. I want to open this file, read bytes from it. In my program, how do I refer to this file? Do I say my mData.txt? No, that is the name given by the operating system. Insofar as C++ programs are concerned, this file will be referred to by a special data type called capital F, capital I, capital L, capital E file. It is a special data type. A pointer to that special data type is the effective name of the file inside our program. Effectively, if this is your memory, there will be operating system somewhere here, which actually will do the job of I.O. And this is your program, let's say, C++ program. Then inside this program, you will define a file pointer by saying file star fp. And this fp here is effectively the name of the file as you are telling. Of course, your program fp does not know that it has to be associated with mData.txt. That task is done when you open a file. At the time of opening a file, you associate the real file name with this pointer. Then onwards, this pointer becomes the name of the file. So this association actually is to this file. Is that clear? It is possible that you open this file by associating it mData.txt. Open it, read it, close it. After closing, you open another file and attach it to the same pointer fp. Now fp will become name of that another file. So file pointer is a special variable available to you, which can at any time be associated with one file. Of course, if you want to open five files simultaneously, read from them and write to another three files, then you will require eight file points. In general, you require one file pointer for every file that you are handling within the program. So is this definition clear? 5 star fp. Please note it is called a pointer, but this is not the pointer which points to something or information within the file. That is not the objective. This pointer points to the file. So this is effectively name of the file. Of course, this alone is not sufficient because we said that a file after opening, I may want to read and I need something to indicate where inside the file I am. So there are two things. First, the file itself and second, inside the file. Inside the file, what do I have? Bites. How many bytes? Size number of bytes, whatever be the size? We have already said that as far as we are concerned, the file is logically an array of bytes. Now here is a difference between the array of bytes as a file and the array of bytes in memory. In array of bytes in the memory, the memory contents are there. Nothing else is maintained for that array. We, when we say ai, we are saying go to i-th byte if it is a carry. Inside a physical file, however, whenever a file is open, since the file could be read sequentially or could be written sequentially, where exactly you are inside the file at any moment has to be known to the operating system. For example, suppose you have read one record from an input file. Next time when you just say read again in a iteration, does it read the same record again? No, it reads the next record. How does it do that? Because the operating system has to maintain, if this fellow has already read this 44 bytes, now it will read next will be 45th byte. So internally, it maintains an index which shows the current position in the file, whatever it is. This position is automatically incremented whenever you read something by as many bytes as you have read or is automatically incremented when you write something by as many bytes as you have written. Initially when you open the file naturally this internal index is at 0. Now this is what is shown here. We consider the file to be logically an array of bytes. We can go to a position and access a number of bytes. Suppose we want to go to a position indicated by pos. I want to go to posth byte. Pos is 350, I want to go to 350th byte. Please note that such a facility exists. But how will I prescribe pos? I can prescribe pos as an absolute number. No matter where you are Mr. Operating System inside that file, go to 350 second byte starting from beginning. Or I might say go to so many bytes ahead with respect to current position wherever you are. Please note that a current position is internally maintained by the Operating System and it could for example be this. Normally the so called current position is not visible to you. Current position is something which is maintained by the C++ by the Operating System. As I said when you open a file current position is zero as you keep reading or writing current position advances. But it is possible for you to set this current position here or there. Just like you say a I, you can say I want to go to posth byte. No matter you are at the current position. Take your current position to this and start reading or writing from that point. That facility exists in C++. That facility is given by specifying a position in certain special functions which are to be invoked which we shall see in a moment. And this position value has to be a long integer. It cannot be a normal integer because integer is not sufficient often to define all the number of bytes that can fit into a disk. Please remember disk has a large size. I might have a very large disk file and I should be able to specify an individual byte number in that large size. So generally the position value or position variable that we use has to be a long. The function fc sets the position indicated by pos. fc is a special function. In fact just like you have seen c in and c out or printf and scanf whenever you deal with file last time we saw which functions f scanf and f printf. Today we will see additional functions f read f write f tell f open and f close you know and fc of these fc is the most important. fc can be used to prescribe where you want the next read or write to happen. So you are sort of seeking a position in the disk. That is why the word seek. In fact the technical word whenever the disk arm is moved forward or backward to go to the right track on the disk is called the seek operation that is why the word fc has come. Now this pos is a displacement value specified relative to either file beginning current position or end position. File beginning is this current position is this end position is this. So in my fc I can say pos which is say 500 it could be with respect to beginning position which means absolute 500. If I say 500 with respect to current position then it will mean current position plus 500. If I said 500 with respect to end position what will it mean? No it will mean go 500 bytes beyond the end of the file. It is like if you travel 500 kilometers west of Mumbai you will fall into Arabian sea. So obviously if your relative positioning specification is seek end that means end of the file the pos value should be negative other it will not be meaningless. Whatever the value of pos you specify it is always added to either the current position or the beginning position or to the end position. If you happen to buy mistake go beyond end of file for which there is a special indication called EOF capital EOF then you will hit error as you will get an EOF. Now the f read and f write which are specified they will read or write so many bytes at this position. How many bytes we shall see those functions in a moment and we shall see how read or write happen. Please note that anytime a read or write happens whether it happens due to f read and f write function calls whether it happens due to f print f and f scan f function calls or in your std in when you are redirected a file it happens due to c in or c out the internal position is automatically advanced by so many bytes. Here are some file processing functions that we will discuss f open you are familiar with this notice in the context of today's discussion f open is a mechanism to give our own name to a physical file and that name will be a file pointer. So when I say 5 star fp I am defining a file pointer and when I say fp is equal to f open my data file dot text comma r there are two parameters here one is the name of the file as is known to the operating system. So I am saying look mister operating system now onwards whenever in my program I say fp please remember I mean that file and how is that file to be opened because I can either read or write to that file I am opening that file to r meaning read r stands for read w stands for write and a stands for amend append append means add something look at another file 5 star fp out again it need not be fp out this is my choice but obviously the name suggest that I would like to associate this file pointer with a file which I will be creating and writing output to notice that I am saying fp out is equal to f open db dot bin there is a mistake here what is the mistake there should be a double quotes it has to be a string it has to be a valid name that is known to operating system it could also incidentally be a normal string array character array in which you have put the name in fact many times when you do not know what is going to be the physical name that you need to process for example you have asked ten friends to do data entry for ten different markless now some fellow has got file one dot that or dot txt somebody else has my file dot txt now you want every time change a program to read that one so what you would do is you will give a care file name thirty or forty an array then you will ask that fellow first put that file name as a string and then here wherever you have file name instead of db dot bin in double quote you can put that array name that becomes the file so first parameter is a file name physically known to the operating system the second parameter is a mode operator r means read mode w means write mode a means append mode if you just say rw or a by default the file is a text file but if you want a binary file you have to add the character b so rb means read mode binary wb would mean write mode binary file a b would mean append mode binary file in the context of the logic that was used to name fp out is this mode correct fp out stands for what we want to have an output file we want to read from a text file and write to an output when I want to write to an output file can I open it in a read mode if I write such a program the file will not open because the file may not exist after all I want to create that this is where it becomes important to know whether a file was opened or not open please note that can happen even with this first file suppose my data file dot txt does not exist then what will this fp equal to f open do can it open an on existent file for reading obviously there will be an error so you must have a mechanism to check there and the only mechanism that is available to you whenever you invoke a function is by looking at the value written by that function consequently all file processing functions return some value or the other in case of f open since you are supposed to attach a pointer to an external file if a file is actually open then a valid pointer is attached whatever file is not open the c++ way to tell you back is to give you back a null pointer a null pointer is a universal sos in c++ or in any program null pointer means sorry I could not figure out anything and consequently you can check that null pointer this is what is done in the next statement there is an error here also if fp equal equal null notice that fp was the file pointer sorry fp out because you assigned a value of return value to fp out which would have been a valid pointer if that pointer is null the file was not open that is why you see the statement if fp out is equal equal null then c out cannot open output file so I cannot carry on in fact immediately after this if my entire program was about creating this file about about it I should better say return minus one because I could not open the file I should do same thing with every file that I open so consequently the first set of statements which are shown here are not very well written I am just saying star fp fp is equal to f open my data file I am assuming that the file exists there are so many possibilities by the way the file must exist in the current directory you are at the moment suppose you have created the data file in lab 4 directory today you are in lab 5 you write this program and run it in that directory you will not find that my detail system will bomb the program will terminate and it is better to check if fp equal to null then give some error and get out here is the specification of a desired position in fact at this stage I would like you to look at the page 2 of the handout page 1 you can read it is a more general conceptual thing but page 2 page 2 describes several major functions which are used for file operations you want to go to a certain position on the disk now you have opened a file and you want to specify that I want to perform read or write operation at this particular place you remember we had actually described this fc somewhere else earlier so what is the format of fc what is the first parameter please read the fc it is the file pointer so let's say I want to specify repositioning of the file pointer fp what is the next parameter the position I will typically have described long int pos or simply long pos and I would have said somewhere pos is equal to something let's say thousand two hundred so pos has a numerical value the next parameter is pos ordinarily if I stop here I should automatically mean go to the poset position right from the beginning but since c++ permits you to define three possible points of origin with respect to which the pos can be prescribed so here you can say for example if you want to specify that pos is from the to be counted from the beginning of the file what will you write sorry the three possible names for the third parameter seek set seeker and seek end these are not values these are flags and these are internally defined how does the c++ compiler know that seek underscore set is a predefined thing you are not defined it anywhere it knows it because you have you have to say in your program just as you say include i o stream you have to say include c std i o which means c like standard library for i o functions when you define that all these these are called macro seek set seeker they all get defined in fact seek set is defined to be zero seeker is defined to be one and seek end is defined to be two if you print the values of six at seeker because these are all known value you will get zero one and two so obviously you are not going to add zero or one or two to everything these are only indicators if it is zero whatever is the internal pointer maintained by the by internal index maintained by the operating system that is to be treated as the starting point for counting or the beginning has to be treated as starting point of counting or end has to be treated that's what is pos will not be negative pos is the value which you are setting pos is not a value which the internal system knows if pos is positive and if this is seek end then the file will attempt to seek a position which is way beyond the end of file it will apply this non-intelligent so in fact the good point that is what I have explained there I have said a word like origin so this origin can be either seek set or can be seek end right this is what you are saying the formula that is internally used is very simple whatever is the origin that plus pos that is going to be the new position if origin has the beginning that means it is zero then zero plus pos if origin is at the current position then whatever is the current position suppose you already traveled 200 bytes it will be 200 plus there are traveled 500 bytes 500 plus it may so be possible that you have traveled almost towards the end of the file you have not reached end of the file yet but only 20 bytes are remaining so even if you say set curve you may still go beyond the file that's your problem 99 percent of the times you will be specifying an absolute value we relate you to 0th but there are squiggles where it is easily possible for example you want to traverse backward in a file how could you do that you say seek end minus 20 seek end minus 100 seek end minus 1000 it will do a numerical is that clear now this is something which I would like all of you to understand very clearly internally whenever you open a file the operating system maintains an internal index which knows where you have to do the next operation at the beginning it is at zero as you read or write the index will advance okay this internal index the only way to set it to a different point is fc there is no other way however you may want to know where the exact index is whether it is at 500 byte or 200 byte or 753 byte you don't know where it is because it is hidden from you there is another function which will tell you that index value and that is called f tell so if you say p is equal to f tell where p is another long end that you have defined then the operating system will know where you are and it will tell you I am at the 5253 byte but please understand that whenever I do a read or write operation this value will change so if you do f tell after every read or write operation you will know how quickly the internal index is advancing the way to set this internal index is through this state so when you say f seek fp pause seek set this internal index is automatically brought to the point that you have specified let me give an example suppose I say fc fp 0 what it means suppose the current internal index is somewhere here the system will be forced to bring it all the way back here and it will be positioned at the beginning so to find out where the internal index is you use f tell ordinarily that won't be required that is required you know where I am at a certain position I say f tell I capture that in p and keep it with me now I do some read write operations later on I want to go to that position itself unless I have captured that position and kept somewhere how will I go to exactly that position I can then use this p to set the internal index by another f seek command that is the objective now there are many squiggles which are possible but these squiggles are non-essential for normal programming normal programming follows some simple principle so let me initiate that principle and then you can later on read those programs and understand more about the simple principle is if I want to read or write sequentially I open a file and keep reading or writing automatically the internal index will move as I read of course it is my job to read or write the specified number of bytes which we shall see in a moment if I want to directly access anything I must use f seek followed by read if I want to directly write at any position I must again do f seek and write so the logic is very simple for sequential files I keep reading reading reading or writing writing write for direct access file which I may want to read and modify I must fc can read I must again fc can write let's see that after we discuss the read and write state reading from a specified position this statement is f read this followed by what the file pointer always what is the second parameter of f read is this the first parameter what is the first parameter of f read so why did you agree with me when I said fp you have not reached that point please read the definition of f read very carefully on page 2 you agreed to fp because so far you always seen file pointer being the first parameter in most of these functions read and write functions do not have file pointer as the first parameter but they instead have another pointer remember the student record s either I want to read from the desk as many bytes as are describing that student record and put it in s or I want to take whatever is inside my memory s and put it on desk read or write operations require a source and a target in the read case the source is this file the target is somewhere in the memory you have to specify both that is why only file pointer is not adequate file pointer of course is required because that will describe the source in case of read and target in case of write in this particular case and s is the first parameter which becomes the target whatever you read from the file please deposit it here and that is why the pointer and s it points to the entire memory structure of s this is followed by what I will call it record size because I am talking in terms of record so if for example you are putting 44 bytes structure this will be 44 ok this is followed by count how many records so I recommend record of 44 bytes notice that it is possible to read 20 records at a time however if those 20 records are put in a single structure variable s you will have huge problems because there is only 44 bytes allocated to s there is nothing more the point is that this pointer and s need not be a pointer to a structure it can be a pointer to an array of 1 million bytes just a block of bytes and you can say read 10,000 bytes from the disk and put them in the 10,000 position here that is all possible that is why this is a general statement in general whenever you are doing a file processing involving information about entities this count will almost always be 1 almost always be 1 because you are reading one record at a time if this is not one and something else then the pointer as the target must be appropriately set the last of course is at what point this read will occur whatever is the internal index or whatever you have done fc earlier fc could have said that so you will read from here please note after executing this statement that internal index will automatically advance by so many bytes suppose there was a student 10 115 and his records was somewhere here at this point and you read this record now you want to modify the marks of that student and rewrite that record can you just say f read this followed by f write no because if you did f write f write will write here now because the index has all automatically moved so that is why invariably you will have a series of statements that f seek f read again f seek f write because you have to go back you have to bring this fellow back here so after reading this you have to do f seek again and f write has exactly a similar what is the f write syntax again and s comma rake size comma count no difference the only difference is source and target are different in f write the source is and s the pointer to the structure and target is fine and in the other case the source was fine target was but at what point you write you better put an f seek before this to say whatever in general in a program you will not find these read or write statements without being preceded by an f seek to make doubly sure that you are exactly where you want to read or write okay f scan f and f print f we already seen they work exactly like print f and scan f statements so for example the print f you would have had a string which will say something like hello then one blank percent d backslash n let's say comma m suppose m is 25 this will print hello 25 now you want to do this to an output text line on a file you just say fp here that's all nothing else so when you say f print f you are doing print f to a file we are already seen as print f where I can do this to a string internal because after all print f is converting internal binary float values into character strings so that you can visibly see them or the other way round in scan f where you read character string and convert them into integer float etc all that you are saying is operate on the files this is the problem will not discuss this problem in the class because I have already given two programs which are there on page 3 and page 4 don't read them now that's homework basically the problem that is solved by these programs first program says read records from the text file extract the four component values from each record and assign these to appropriate parts of a structure and create a binary file called student db dot bin student database dot bin it's a binary file so it will have structure for one student structure for another student structure for third student etc etc if there are 500 students in cs11 500 this particular sample which works on a sample data file creates a file with ten students and then it says find marks of a student given a roll number now that second program is named file ops dot cpp it performs different operations first it tries to locate a student by reading the roll number sequential reading the record sequentially and finding out whether the roll number is found notice that is what you will do in your normal sequential search in an array and that is where it is important to understand that that will not work imagine you go to an ATM machine to withdraw your money the moment you put your card actually that your account number is going back miles away through the network to a disk which contains all the accounts of all the customers of that back for state bank of India it is something like 18 crore customers how would you like your record is being read by reading 18 crore records how long will you be standing at the ATM so there must be some mechanism to convert your roll number into a relative position in the file say your position is 53 lakh 48000 434th record and directly go to that so that you get your money and other people can also withdraw that is why direct access is important the issue of how do you translate your account number into a relative position is still critical to simulate that here I have roll numbers it so happens that an example that I have given a certain roll number at a position if I know the position I can directly access that so the second part of that problem says get me the record in the earth position and a third problem says modify the marks of such and such students it is like modify your account because you have withdrawn 500 rupees reduce that from your balance and rewrite so please go through these we will continue the discussion on some additional aspects thank you


Hello everyone. I'm very delighted to be here today. I think the conference has been so far-reviewed. Thanks to ASCII for arranging this. With that said, let's get started. My name is Habir Rakhman. I'm working at ASCII. The topic for today is cloud means fog and pathage, a story of worship control. I'm sorry, I could not get the time any longer. Anyway, so this is what we learned at ASCII, automating large-scale cloud deployment. So I'll be sharing some details and some details about the movement process we were covering and what we learned in that journey. Before we start, I've been about my company. APIG is an API management company. We provide free tools for developers. We also have enterprise model API management product, which is used by different enterprises across the world. Anyone using API developer tools? Okay. So, nice to do so. Basically, we make APIs better for developers and developers. That's our idea here at ASCII. We also have a recently acquired company called User Grid, and User Grid needs to make building mobile clients for iOS and Android easier. So it's an open source. You can actually find all the details in the APIG website. There are a lot of videos, webinars, database, APIs, and User Grid out there in aphix.com. You're welcome to visit and take a look. In fact, APIs are very relevant to our company. So, if you look at cloud and any providers, like Amazon Web Services, or App Space, or Lightning, the way they provide services is by giving you APIs. So, if you look at a lot of cloud maps we've published, I'm sure you might have heard of these names, like the ride scale. So, what these guys are doing is they're making some of these APIs that they're working on. So, that is the API echo space which we have right now, which the world is going to right now. So, in the API, one by one, you add different storage, then the API is like that. So, how are you using AWS? Amazon Web Services is in production. So, just to give a, just to set the stage right, if you look at AWS, Amazon Web Services, it allows you to create instances. Cloud completely resolves, you see. You can either go to Amazon's Web Console, you can use any of these cloud management rules, or you can use Amazon command line rules, or you can use any of these libraries, like the Java, there are libraries, Ruby, there are libraries.  All these things are, we will do it manually when we start. So, you know, this was our thing, we were using a third party. And it was good, but the problem is it would not stay. If you are talking about cloud servers, it's not easy. And we will have to do it manually. Maybe you're back in five or seven weeks. So, it gets boring also at times, right? So, if you look at this, you know, this gives you a small idea about the different things we were, if you know Amazon, there are Amazon machine images, right? Which you use to bring up instances. And there are, you know, packages, you need to install licenses. You create these volumes. These volumes are nothing but network-attached volumes, which is another service by Amazon. Then you open up, you create security groups, you attach and attach the other instance. All right, so all these things we were doing manually. So, my talk is about how we moved from this number of manual tasks to largely, mostly automated, you know, state VR now. All right, so I'm not saying we are completely automated. There are quite a few things which we can still automate. We are working on it, but still we are good. So, this was our take two, right? So, I think we are, you know, we are back, one of our colleague in US. He started, or he started exploring cloud configuration management tools.  You know, we thought that it's better fit for our team and our culture, right? So, Puppet, I'm sure a lot of you guys are already using Puppet. Can you tell me how many of you are using Puppet in? Okay, so Puppet is nothing but a configuration management tool where you can actually take a base instance, your base computer to a desired state. If you want your base computer to a passive server, you can apply these manifest. They call that manifest.  Then you apply these manifest. In Puppet terms, they call it manifest. And they apply these manifest to that instance, and you'll be in that desired state, be it a NGTex server, be it a Apache server, you know, whatever you want. So, we actually started using Puppet. It was good. We also used Git for version control. I'm sure you guys are using Git. So, what we used Git for is, you know, to manage the entire configurations and packages. So, we, you know, bring these packages and manifest, Puppet manifest, under Git version controlling. So, Git gives you the history. You know, what happened, when happened, and who did that change, right? So, if at all you want to rework, it gives you that visibility as well. You can actually go back to that previous version at any point in time, right? So, this is where we were using Puppet and Git. And if you look at this, you'll see that, you know, installing packages and generating server configs. For example, we use load balancer. So, we were actually using Puppet to come up with the server configurations. But still, you'll see a lot of these things are not automated. We still do a lot of manage tasks. So, you know, that is where we actually found Puppet Cloud Provisioner. So, we actually also got help from Puppet Labs, the creators of Puppet. And we use Puppet Cloud Provisioner. And Puppet Cloud Provisioner is, in fact, using a Ruby Cloud Services library called Fog. So, using this, this is where, you know, Cloud meets Fog. And we started using Puppet Cloud Provisioner.  I know that, you know, we had a talk around Cloud Formation. At that point in time, Cloud Formation wasn't that mature as a product, right? It just came in. So, we started using Fog and Puppet Cloud Provisioner to bring up instances. And, of course, Ruby is the whole language we were using. So, if you look at this, we actually automated the whole thing, you know. Only a few things were pending here. That's, I think, we are working on. And I think as we go, we may automate those.  You know, we manage the whole API management product for the customer in the Cloud, right? So, we have a customer in a YAML file. So, YAML is a simplified version of JSON or XML, right? So, it'll look like this, right? We'll say, this is our configuration for so-and-so customer. And just run this Ruby script. And we have everything ready. Right? So, even the puppet will be installed automatically. All these security group creation, port mapping, everything will be done. So, this is just, I mean, it's just that you're giving all these variables, saying that, okay, this is the key file. This is the image you want to. This is the instance type.  So, we'll give all this information in this YAML file, and we'll run that script. That's it. We have our customer instance ready. All right, so, the beauty of it is, you can actually reuse this, you know? Once you have this in Git, tomorrow you have another customer, maybe you need to change few things here and there. You can pretty much reuse the same thing across different customers. Sure. Right, so... Yeah, so, actually, it's like we happen to meet Fog, you know? So, nothing against Lib... I don't think LibCloud or Dental Cloud, JClouds would match at that point in time. I was talking about Cloud Formation also. Cloud Formation was very new at that time, right? So, as of now, we are only using with Amazon Cloud. But, yeah, so, I'll come to that. That's one of the features of Fog. We can actually use Fog across multiple vendors, right? So, this is our, you know, YAML file where we can bring up instances this way. So, basically, Git, the version controlling mechanism, to manage both puppet configuration and cloud deployment. The YAML files and the manifest, puppet manifest, both are matched by Git. All right, so, this is a, you know, very high-level view of the whole thing, right? So, we put this YAML files in the Git, right? And the Git has this post-receive hooks, right?  Basically, you are saying that, okay, let this change happen in this repository. Whenever there is a commit, you can actually have a post-receive hooks, which will copy this repository to the different puppet masters we have in different regions, right? Now, then, we'll run that script from whatever region we want to bring up the instance. So, if you know Amazon, there are, I guess, five or six regions as of now. So, we'll have, for us, we have puppet masters across different regions. So, once you push anything to the Git, you know, it will go to the corresponding puppet master. Then we'll run the script. So, like I said, we have fork, and we also have puppet cloud provisioner. Then we have an easy deploy, which is a wrapper script around all this, which creates the whole thing, which keeps everything in sync, right? Then it bootstraps the cloud instance in Amazon. Once it's up and running, it pulls configuration from the respective puppet masters. So, this is a very high-level view, but I think this gives you an idea of what we do. So, like I said, we are using Git hooks to do two things. One is, before committing something to the repository, we need to parse the puppet, you know? I mean, we don't want someone to do a bad manifest committed to a repository, right? So, we do a puppet parser check before the update. We also do, we don't do this as of now, but there is a tool called puppet link. We can export further, which actually does these things a lot more better. So, and afterwards, a post update in Git that's a hook will sync these repositories. For example, the puppet repositories with different puppet masters across the region. Okay. And now, a bit about the puppet, I know that you guys know about puppets, so I'm not going in detail. But what we do, maybe differently, I'm sure a lot of companies are doing this, we use Git branches for puppet environments. So, we don't have a separate dev, separate, you know, test environment. We have a single puppet master. We use that, we use different Git branches to separate it out. So, for example, if you have a, I'll give you an example, you know, you want to add some features to existing manifest. You don't definitely want to do it in your production environment. So, we create a new branch in our Git repository, then we push it to the central repository, then when you bring up instance, we say use that feature X, right? Now, this new instance will actually get all the new configurations manifest, new manifest from the Git repository, then we come up with that version of puppet manifest. Okay.  Then manifest and files, data.  You also have data which says, you know, you want to use this configuration file. So, we separate it out. You know, we don't keep both things. You can actually keep both manifest and these data in the single repository or in, yeah, single repository, we don't do that. We rather separate it out for ease of management. Another thing is, like I said, we have puppet masters per AWS regions. Okay. Now, these things we have been, we have started using. I'm not sure how many of you have heard Hyra. It's a, this is a tool which helps you to separate out the data again. I'll give you an example word. As of now, we are coming up with our, you know, one of our versions or one of our products. There is a new update to that product. All we have to do is to go to this Hyra repository. There will be a FH version, you know, that product version, we just update it, then restart the puppet, right? So, basically, you don't touch anything in the manifest. It's completely separated out. So, this is something you can explore. I think, from next version onwards, they are trying to integrate this with puppet. Sure. Yeah. I mean, if at all you want the change to happen immediately. True. Yeah. If at all you want to change the immediately, yeah. Otherwise, of course it will run every 30 minutes.  That's right. So, I said, you know, I mean, we restart it because we want to see the changes validated, then go to the next thing. That's true. So, like I said, Hyra, they are trying to integrate it with the core puppet from next version onwards. Okay. So, now, why fog? You know, we started, we happened to meet fog, like I said, and we started using fog. So, a few of the reasons why I like fog is, one is cross-service compatibility.  We actually use, I'm actually, you know, we are coming with a version of this ECDeploy script where we create all these DNS entries as well using fog.  You know, there are APIs in fog. Now, another thing is avoid vendor locking. I mean, this is a debatable topic, but as of now fog supports different vendors. ECDeploy, Rackspace supports, Slicehost, Lino, these are different vendors, cloud vendors, fog supports. Now, another thing I like about fog is the power of Ruby. Right? So, we'll come to Ruby later. So, this is how you install. If at all you are interested in fog, you know, fog is a gem. You can say, if you have Ruby in your laptop or in your server, you can install using gem install fog. Then in your Ruby program, you'll have to require this package, require fog. Then there is a .fog file in your home directory.  So, the fog by automatically fetch your cloud details from that file. Now, one of the things is that, at first, there are not many good documentation out there for fog. But I learned that you can actually look at the test suits, you know, and actually learn a lot about fog. So, it's because of Ruby's, I think there are a lot of languages using this approach now, test driven development, TDD.  So, it's very easy to get started if you look at all these files. Okay, and fog also has a command line. You can say fog test AWS, which means this will take the test AWS account details from the fog file. Then you can play around. You can create instances in the command line itself. And this is another example where you actually, you know, look at the different vendors. See here, we are using similar command to bring up instance in Amazon, also in Rackspace. And here, you say, you know, you're creating a, creating with this AMI, here you are creating with Rackspace on AMIs and, you know, flower IDs. So, basically the idea is that you can create, you can use the same command and create instances across multiple cloud vendors.  There are a lot of open source fog alternatives. Probably if you're interested in, for example, Python, you can use LibCloud. And, you know, I know in Amazon, they have a package called Boto. I'm hoping someone is using that. That's not vendor-independent, but just Amazon, you know, Python library. Then Java, there is JClouds. In Ruby, you have DeltaCloud. That's a new Apache project. I think it looks interesting, something to explore. So, again, you know, why Ruby? You know, I used to program in shell scripting. For some reason I didn't like pole. But the time I started using Ruby, I liked it. And this answers why I liked Ruby. I think you should, I mean, if at all you're not using Ruby, I think you should start using Ruby because it's protective and it's fun to use. So this is the code from Maths Moto, Maths the creator of Ruby. So Ruby is definitely fun to program and fun to work with. It's very protective. All right, and I also suggest you to watch this talk by Ajay Ghor. So this is a nice talk, probably you can watch this online. This also gives you the different, especially for us, the different areas it's being used, Ruby being used. So I know Rami's smiling. It's his ex-colleague. All right, so if you want to learn Ruby, there is this IRB, that's Ruby interactive shell. You can do whatever you can do with the program in IRB's shell prompt. And I know that there is an IRB option at you which is called PRY pry. I didn't explore much, but it looks very interesting. So you can actually install it using, it's a gem installed PRY. All right, so one of the things interesting about Fog and Puppet is the layers of abstraction. It gives you an abstraction layer. It's independent, for example, if you take Fog, it gives you the cloud abstraction. It's independent of vendors. So for example, tomorrow you want to bring up your DR environment in a different vendor. Fog-like libraries makes it easy for you. I'm not saying it's, you can just say use lagspace, but it makes it easy for you.  And again, it's not like, you can just switch to say Ubuntu from Send OS, but it makes it easy. All right, so I think we are reaching the end of the talk. So these are a few notes to myself. Hopefully you'll get some points out of it. So one thing I've learned is that when, I mean, if you look at where we were and where we are, I never thought that you'd be able to reach this stage.  So we actually started small. Then we actually iterated. So I think it's a very, you know, nowadays in software development, they call it agile. But in this segment, what do we call it? It's we need to start small, then iterate, reiterate, and reiterate. I think once we do that, I think we can do under this. Another thing is there are a lot of fights over these tools, right? They, someone says, Chef is the best and someone says, you know, Puppet is the best, there are a lot of fights. But I mean, for us, we selected, I guess we selected the right tool for us because we thought that it suits our team, our culture. So I encourage you to do the same, to evaluate not because something is right for someone else, to use it because it's right for you guys. All right. Another thing is, sure. See, for example, I'll give you our example, right? We were like just operations guys. We didn't have much experience in Ruby. So if you look at a lot of startups who are Ruby based, Ruby on range based, they use Chef because there's no Ruby and the language Chef cookbooks are written is Ruby, whereas Puppet has their own DSL, which is kind of easy, right? So to learn. So we started with, we weighed that point and we thought that, okay, Puppet is suited for us, right? Sorry? Exactly, exactly. Maybe you can learn. I know that Chef is also widely used. It's good. Probably we can learn, but it takes some time. So we went for Puppet. So another thing is, you know, something I learned that, you know, whenever you design a software and it holds good for softwares as well, we need to make sure that it's operations ready, right? I mean, you don't think of making a software product operations ready after everything is up and running, right? As in, you know, when you go into production. So the day when you design the product, you have to make, this is not something for the system, but, you know, which is something good to keep in mind and we can probably, you know, advise our architects and designers out there, right? So with that said, I think I'm done. These are the few pointers to get started. Yeah, that's it. And just in case interested, we are having. Yeah. Yeah. That's true. Yeah. So this is mainly because of scalability. We have like around 3,800 plus instances, right? So we don't want, you know, the puppet request to go across regions to fetch the manifest and the packages. So mainly because of that. Sorry? Yeah. And then we thought that it will be much faster, the whole thing. I mean, I've seen that sometimes if you have too many packages in your puppet master, even within the region, it's very slow, right? So we don't want to be in that position for, say, Europe region. So we wanted to make it faster. So from the day one, we had different puppet masters for regions. Textbook? No, we did not have that situation yet. So you might remain, you know, problem accessing a puppet master in a particular region, right? We didn't have that. But still, I don't think that would be a problem.  Yeah, so one of the things is that we will try to bring up a new puppet master as soon as possible. We have all these things automated. As in we keep these git repositories in S3, right? So we can actually bring up this new instance. We actually puppetize the puppet master also, okay? So we know that how to puppet the initial thing, then we bring up the instance. So once it runs, you have a new puppet master. So yeah, puppetize the puppet master. Yeah, we do that. But initially, you know, a bit of things you'll have to do manually, but other things are puppetized. And so now we monitor the processes. We don't actually monitor the logs as such, but we monitor the process in each individual puppet, servers, and agents. We monitor the individual process. But yeah, I think there are better mechanisms to actually monitor puppet by looking at the one link puppet. There is, you know, a file which will be updated every time the puppet will be, you know, run. We don't do that as of now, but yeah, there are mechanisms. Any other questions? Sorry? Yes. We do all these things. How do you make sure that they're all fine? Yeah, so we run this. Once you have an instance, and then you install. For example, if you have a patch server, you validate that. For example, you access the site. We have a portal, for example. So once you bring up the instance, yeah, we don't have that situation. For example... Okay, okay, okay. We've never been included in that session, but... So probably I'll repeat that question. So what he said was, if you want to, you know, create some instance and you have to do it like 20,000 or 2,000 servers, how do you make sure that it's being installed in all these servers? Right? So probably you can even write a script. I don't think... Yeah, so yeah. So M-collective might be an answer there. Yeah, yeah. Yeah, yeah. So if you give, install this version, install that one link. If you're young, possibly you have that script, a point link, that you can add with your... ensure this... Install. It will put me in for that. It will put me... Just... Yeah. So if you're an environment player, after a thousand servers, there's no need to allow me to have it. There's no need. Why are you over it? Yeah. Yeah. So M-collective, actually, which node has that... which node has that script? Exactly. Yeah, so... Yeah, even... Yeah, that's right. So we are actually using M-C-O. So for example, you can say what is the version installed, package installing all these different servers? Yeah, we are also using M-collective for that. Yeah, in our latest proper deployments. We only see that with servers that are entered installed all the time. Right? So you have 500 servers. Right? 50 of them pay. That's new, yeah. And you want to see how on which servers... Right? So... It has proper install and implement, that's how it works. You can see... The final specific thing, you can actually tell M-C-C-I to check for which... which node has a specific... that script. And actually, what happens? Nothing like that. So you will... M-collective, yeah. So you guys are using M-collective? Yeah. Yeah, okay, you are using, okay. Yeah, we are also starting using it. So it's a nice tool. Like I said, we are also using... Say for example, you can say what is this version of this product? So it will show you all these servers which are in that group. It will show you the version of the product. Okay. Why is that? I know that... I mean, any of these... I can change... I can add some... Yeah. No. Yeah, that is what you...  I can say a lot of things. Not just this design, but I can add some data. So in that, So any other questions, I think we can discuss. I think we are having a talk on M-collective, so probably, yeah. So anyone using Rendec, there is an orchestration tool called Rendec, I think based on SSH. So yeah, with that said, if no more questions, any other questions? Okay, thank you guys.


He number four, takes up, we're under way, charging in for a first turn, it's tight for a breast, Lingland right round the outside, but Dole's there, what a first corner, he meant business there, there was no way Lingland was going to come round him. Just about got it under control, Frick's roaring round the outside in third, terrific racer speedway there, fabulous fall-ups. Now that's for sure, takes up, we're away, they come roaring out of the gate, they're all there together, look at that from Dole, Dole's round the outside, can he get it hooked up? No, not quite, Martin Basilic is in front, it's tight between the two, Australian riders for second and third, Dole now squaring it off, it's going to get tight in the third one, not too much, here comes Dole, he up the inside, charges to the front, Basilic comes to the back, kept you belittled. Australian 1-2 here, fabulous ride from Dole and Jack Holder also raring fast in second place. The way, just for Steve Tochter and Marie, Leakes is there and the way they charge, once again, gotta say Jason Dole has made a good effort there off the gate number two, fasts himself to the front, here it goes down the back straight, and it's Thompson, some surges in the second place, with Dudek hugging that inside, he now gets himself to second place, the Thompson reacts well. Weeping around the inside but just can't seem to find a way in place, now moving his attention to the outside. Dole wins in style though Chris, brilliant stuff from him, nine out of nine, three race wins this evening. Free opponent, here we go then, the way we go, boys on the inside are made to go, what a run to the corner from Dole. Here we go, Freddie Lindgren getting in the action, beauty in the second place, but Dole fires himself from the outside, oh Lindgren had a shake. It's for him that he's got to believe that Jason Dole on fire here tonight, he wins the first semi-final and Lindgren threw as well. Perfectly for him, a strap in the back for Sparsic, desperate to get on the podium, oh he comes up the inside, he's not going to get in the chair. Oh this is tough for Jack Holder, he was comfortably up front. Oh Holder was so close to winning his first Grand Prix. That can't come into a kill with this. Might do a little bit. He's definitely caused him to go off. Blues out, he excluded Jason Dole. That is a huge and a controversial decision here in the Narodobi. That's a bit cool. Wow, Jason Dole is going to be absolutely fuming about that. Absolutely fuming, he's had a fantastic night and he'll feel aggrieved about that decision. Bartos Marzlik, well he's just got out of jail there I believe.


In the name of God the most gracious, the most merciful, we ask God to send peace and blessings upon His beloved Prophet, Muhammad and family and companions. It's not every day that you sit in a mosque in America and you're able to do zikr like this. Trust me, I travel a lot around the US and this is very rare to see. This is beautiful. This is something that we see all over the Muslim world, but in America it's perceived as somewhat strange, but this is not strange. But the Prophet ﷺ mentioned something about the strangers. And he said, He said glad tidings to the strangers. He said that this tradition, Islam that we practice, started off as something strange and it will end as something strange. So someone might walk into this room and see that the lights are off and people are saying the same words together over and over and over again. So that you know they're not too distant from you might think that's strange. Or even family members might look at you and think that's strange. Then I would say glad tidings for all of you. For the Prophet ﷺ said that my religion started as something strange and it will end as something strange. But the truth is what's really strange is the heart that was created by Allah ﷺ, all it wants to do is remember Allah ﷺ. But there are human beings that barely get to taste that reality. But when you do taste that reality, no one can take it away from you after that. Nothing becomes equivalent to that reality.  And it's even made more beautiful when you're sitting in front of individuals that are connected to individuals that are connected to individuals that are connected to individuals to the Prophet ﷺ. When that happens and a whole community is upon that then the ummah of the Prophet ﷺ is in goodness, bi khair, al-umma bi khair. Everything that we see around us, everything that we've been witnessing for the past five or six months in terms of what's happening with the Muslim world or what's happening in Ghazza, don't for a second believe into the narrative that this ummah is in something except great hands. This ummah is in great hands. And the best of scholars that I've, you know, for a little bit of time, gone to sit at their feet, they will remind you at every moment, don't say that the community is like this or the Muslims are like this or individuals are like this, but say that the ummah is in good hands, al-umma bi khair. And it's in khair because of pillars like this and moments like this where you sit together and you remember Allah's name, we don't know what that means. We don't know what that will do for us. We don't know what that will be for us on the Day of Judgement. If SubhanAllah gives you a tree in the garden, what does it mean when you're remembering Allah SWT throughout a whole month in which the gates of the garden are open and the gates of Hellfire are shy? It means Allah has written for you something very special, but we have to bring ourselves to that frequency of understanding. And how do we bring ourselves to that frequency of openness? We do exactly what the Prophet ﷺ did in these days. The Prophet ﷺ, peace be upon him, was the most generous human being outside of Ramadan. So how about inside of Ramadan? He was more generous they say than the giving wind or the breeze. And we all know this when we're very hot sometimes and then also when a cool wind comes, what do you do when a cool winds come and it's very hot? You open your chest to it. You turn your body towards it and you take from it as much as you can, you expand your body. They say the Prophet ﷺ gave more than that generous wind in Ramadan. But what's amazing about the Prophet ﷺ, although he made much worship and although he prayed much even outside of Ramadan and although he was extremely generous even outside of Ramadan, what was amazing about the Prophet ﷺ was that his heart was never in a state that is not with Allah ﷻ. He says that my eyes would rest, but my heart would be awake. But for us when we sleep everything sleeps. How do we get to that station of things becoming awake and we're not talking about being woke? We're talking about being awake. The Prophet ﷺ was the most woke individual and the most awake individual. And the Prophet ﷺ's heart never rested, why? Because he cared about every single human being that ever came after him ﷺ. So how do you connect yourself to that reality by moments of zikr like this where you're feasting in the garden of the Prophet ﷺ and where you become connected to him and when your heart is connected to the Prophet ﷺ, then you're connected to the heart that never sleeps. So you can go to sleep peacefully knowing that your heart is connected to him. Al-hafewa'ankum, forgive me all you spoke because of our teachers, forgive me and stuff for Allah ﷻ.


In my opinion, pharmaceutical companies are not here to cure any disease because curing means no further need for their drugs. It's a blatant conflict of interest. The substantial funds allocated to research seldom explore natural remedies. Why? Simply put, herbs and homopathic remedies cannot be patented. If a substance can't be patented, pharmaceutical companies cannot monetize it. The FDA is part of this as well. While their mandate is consumer protection, they often prioritize safeguarding the profits of monopolies and pharmaceutical companies. I believe a comprehensive overhaul of the healthcare system is necessary, placing consumers at the forefront. As consumers become more informed and adopt healthier lifestyles, medical practitioners will lose a lot of business.


12 English Phrases for Describing Movies 1. I have class tonight, so let's go to a matinee instead. A matinee is a showing of a movie in the daytime, usually in the afternoon. Matinees are often cheaper than shows at night. 2. What's it rated? Many countries have a system for rating movies, evaluating how appropriate the movie is for various ages. This is the movie rating system in the United States. G. General Audience PG. Parental Guidance Suggested PG-13. Parent Strongly Cautioned R. Restricted NC-17. Adults Only 3. The first movie was a hit, but the sequel bombed. Describing a movie as a hit means it's popular and successful. A sequel is another movie that continues the story or theme of a previous movie. For example, Toy Story 2 is the sequel to Toy Story. If you say a movie bombed, it means that it was a failure. It was not popular or successful. 4. I was on the edge of my seat the whole time. This is an idiomatic expression that means that the movie was very interesting and exciting. You were so involved with the movie that you sat on the edge, the front part of your seat, to get closer to the action. 5. The special effects were breathtaking, but the acting left something to be desired. Special effects are the effects created by computers, and acting is the performance of the actors and actresses. If you say something is breathtaking, it means it was amazing and incredible. The expression left something to be desired means it was not very good. You wish it had been better. 6. Have you seen the trailer for the final part of the trilogy? A trailer is a 2-3 minute video summary of a movie showing a few scenes that will interest people in seeing the entire movie. A trilogy is a series of three movies, like Lord of the Rings. 7. I have a hard time understanding the dialogue without subtitles. I have a hard time is a common way to say I have difficulty in everyday spoken English. Subtitles are the written words on the bottom of the screen. 8. Although the premise was intriguing, the plot was complex and a little hard to follow. The premise is the idea or concept of the movie, and intriguing is a word that means interesting, something that gets your attention. The plot is the story of a movie or book, and if it was hard to follow, it means it was difficult to understand or confusing as you were watching the movie. 9. The completely implausible twist at the end ruined it. A twist is an unexpected turn of events. If you describe it as implausible, it means it was too unlikely, not probable, or too difficult to believe that it could really happen in that way. The word ruined means made it bad. 10. It's the feel-good movie of the year. I found it very touching. A feel-good movie is one that makes you feel good, obviously. The word touching means that it affected you emotionally in a good way. Another word you can use is heartwarming. It made your heart feel positive emotions. 11. My boyfriend thought it was hilarious, but I thought they went a little overboard with the potty humor. The word hilarious means extremely funny.  The expression went a little overboard means there was a little too much. 12. To be honest, I liked the remake better than the original. The cast was much stronger. You can start a phrase with to be honest when you want to give your honest opinion. Sometimes filmmakers take a movie from the past and recreate it. The new version is called a remake. The word cast refers to all the actors and actresses who participated in the movie. Thanks for watching English Tips from Espresso English. If you liked this video, please share it.


Welcome to toffee TV. It is the start of 11 show don't cast the rovers. This is evident in the carabao cup It's a tricky one It's why they've put it on TV because they're all expecting the worst. Let's let's make sure that doesn't happen Let's make sure we get through. Let's make sure we're in the hat for the next round, let's make it the first win of two this week and get everything Back get over Tony and it's happy going into that international break wouldn't that be beautiful Well, let's start with the team Jordan Pickford in goal for me. And Yeah, listen, I'm sure There'll be loads of shout for Joe Virginia. I don't see the point at the moment I really don't I'd go for Jordan Pickford and You know because I think the same positions where you just have to Do Silly at this stage of the season with everything going on you just have to be a hundred percent and that's why I'm going to go for Jordan Pickford at right back Nathan Patterson coming well staying in the side really I think he just needs to play as many games as you can. I really do I think this is someone who needs to just get as much experience under his belt as possible There's absolutely no reason to take him off the team and there's basically no one to take his place Anyway, unless James Garner plays there. So Yeah, I think he'll continue at right back because I do think it needs to be it We haven't got a very big squad that we so we need that mix And I think most of the younger players should stay in the side and left back Michel Enco is being on the bench so far this season obviously actually the young's been playing in that position so Obviously give him a rest and gives a brilliant opportunity for Michel Enco to get some minutes Under the belt as well as we go forward in the season It's an ideal game for him to come in get you know How however long the manager feels he needs for the game possibly 90 minutes? and obviously He's good enough and we'll see if there's a we can you know have a battle for that left back spot I mean as the young hasn't started the season brilliantly, but This is a good opportunity for Michel Enco to get back in the side The centre backs. I'm going for Ben Goffrey as The right side is centre back hasn't hasn't played so far this season. Obviously Michael Keane's been in Tarky Jared Branthwaite who's gonna be on my on the left-hand side of my centre backs And again, I think Jared Branthwaite needs as many games as possible And I think that it's a good opportunity for Ben Goffrey to come in these You know what? It's like when you're playing these sides. It's all huff and puff. It's all aggression And I think Ben Goffrey's pace and his strength should be good enough To see us through and as Branthwaite, I think needs more games I think he needs to play Ideally play alongside James Tarkowski more, but I think I think that for this game It should be it should be Ben Goffrey Into midfield my three are gonna be James Garner Decore and O'Nana again There's not loads of options, but obviously we've got to take it just a gonna get out of of the team really Again keep him as fresh as possible. I know there's only one more game till an international break But I do think it's important to keep him fresh and also give James Garner that opportunity to play in the sense of the park Trying to take the play a little bit Again in the squad there aren't loads of options centrally Obviously Alex O'Wobee could have possibly played in there, but obviously his injury means he'll miss out and obviously loads of rumors of him leaving the club as well but Yeah, so I think that that's the three they'll start I think as I said I think it's a brilliant opportunity for James Garner to get a chance playing in the middle of the park Trying to dictate the play And hopefully Hopefully it allows us to create chances and score goals The in the top half of The team on the right hand side. I'm going to continue with Dobbin. I thought he did okay on Saturday And again, this is a great opportunity for him to get more time playing the first team Playing on a level that really he should be comfortable with after playing he played in league one last season for Darby I think this should be a level he should be comfortable at Obviously played played an hour in a Premier League game, you know as well Um, obviously come come off the bench at half time against Aston Villa So he should be easily comfortable to play and you know start in this game And we should see a little bit more from him on the other side. I want to go for Dan Dume Um, again, I think this is his natural position. I think this is where he fits Best in our side and it'd be nice to see him Grab a grab his face goal get a little bit of confidence and obviously with Beto coming in and and Chimiti there as well and hopefully Don coming back In the next couple of weeks. We'll see him playing more From from a wide area. So this is again a fantastic opportunity to Give him the chance to do that on up front. I'm going to go for Chimiti. Come off the bench on Saturday and I thought they quite well. This would be this would be a great opportunity for him Great chance to see where he is in his development You've got Neil Mopake come off the bench after an hour or so But I think if you're moving forward if this is the player you want as one of your three main strikers I think you've got to start him in games like that We've always bemoaned the lack of the lack of youth in games like this and the lack of given opportunities Well, for me, he's a player that Is someone who's going to be our future, you know, okay, we have brought Beto and we have got Dom on the camera And so these are the opportunities for him probably going to see him more and more coming off the bench as we go forward as well It gives us a really good option. So, you know, he's played he's played A few minutes now in the Premier League his first his first opportunity against Wolves. So let's see him in the Carabao Cup There you go. Let me know your thoughts in the comments Who would you pick? Who would you not have in the side? Who would you like to see in the side? Let me know your thoughts in the comments give this video a like subscribe You haven't already if you want more great videos join us over on toffee tv Premier the link is in the description and the QR codes come up on the screen in a second. See you later