So, now let us look at the Hadoop specifics for what we have just seen. So, as I said Google pioneered MapReduce for big data and they used it for a very large number of tasks. In fact, initially their index implementation, Google search index was hand coded, but once the MapReduce was developed they soon realized that it is best to do even indexing using MapReduce, because MapReduce paradigm deals with failures, it deals with slow machines. It can be configured to run on different numbers of machines by just changing some parameter somewhere, you do not have to do any recoding. So, it is actually a very convenient paradigm. So, they moved their indexing to MapReduce. They did a lot of analysis on MapReduce. So, in fact, pretty much all the stuff which analysts at Google do these days, Google has a lot of people doing analysis, huge, lot of their employees are doing different kinds of analysis. All the inputs to the analysis are enormous, because every single thing that Google does has anywhere from tens of millions to hundreds of millions or billions of users and billions of queries per day. It is gigantic. So, everything that they do runs on the MapReduce platform. Now, when Yahoo saw this, they decided to actually Hadoop was already being done by a couple of people. It started off as a small project built by two people. It was called Nuch in those days. Yahoo saw the potential and they actually hired one of the two people who was involved in Nuch. The other person decided to become an academic. So, he is now on the faculty at Michigan. But one of the two who joined Yahoo continued developing it and it became called Hadoop. Yahoo put a lot of resources into this. They put a lot of people to develop this. And thanks to Yahoo, Hadoop is available and Yahoo decided to open source it, which was also a very good idea because many people started working on this even outside Yahoo. So, Yahoo benefited. The world has benefited. So, the original Google implementation ran on top of the Google file system, GFS distributed file system. So, Yahoo built equivalent called Hadoop file system, HDFS. Now, Google file system is an actual underlying file system running on machine. Hadoop file system runs on a number of machines. The underlying storage in each of the machines is the local file system. You may be using whatever Linux or other file system. But the Hadoop file system runs a bunch of servers and now it lets you store files in the Hadoop file system and it lets you fetch files from the Hadoop file system. So, HDFS is a service which is provided by a code system, which Hadoop provides. Now, in your assignment today, you are not going to be using the HDFS. Actual production runs of Hadoop will use HDFS, but it takes effort to set up HDFS. So, to simplify your life, you are going to be running Hadoop over local files on a single machine. But the principles are the same. So, whatever you do, in principle can be easily re-targeted to running on files which are stored in a distributed file system, HDFS. So, that data is replicated and there is a central name node which provides metadata. So, if you want to open a file, you talk to the name node. The name node will tell you which blocks are contained in the file and where all it is replicated. Now, the name node is central. If it fails, you are in trouble. So, HDFS keeps a replica of the name node, one replica which is up to date. It has all the data that the live replica of the name node has. The backup replica has everything which is up to date. So, if this crashes, the other one can take over. Now, let us get to specifics of MapReduce in Hadoop. We looked at abstract MapReduce with types as strings and so forth. In Hadoop, you have to give types for all of these. So, in Hadoop, there is a mapper and a reducer interface which you have to implement. These are generics. What are generics in Java? If you are not familiar with it, a generic is a class or a interface which can take types as parameters. So, in the context of Hadoop, the mapper and reducer interfaces take two parameters each. The parameters are the types of the input key and the input value and the types of the output key and output value. Now, what do I mean by input key input value? The input key and the input value are the map input key and value. The map output something which is the output key and output value. So, its types are also needed. It turns out that the output of map is the input to reduce. So, the output key type and output value type of map must match the input key type and input value type of reduce. And reduce in turn has a output key and value whose types can also be specified. So, each actually takes four types. So, the mapper interface is a generic which takes four types. Now, I will show you the thing in the next slide, but in our example, the input key type is going to be long writable which is a long integer. Now, why long integer? Files can be very big more than gigabyte, more than 4 gigabyte. So, 32 bit integer is not enough. So, we are going to use long int. And in our word count example, the key is offset into a file. We are not actually going to use the key, but the Hadoop system provides the key. So, it like I said it can break large files into pieces. So, when it gives a and furthermore it is going to call the map on each record in the file. So, we are going to set up the thing such that each line is a record. So, Hadoop will identify the line by giving the offset of the start of the line in the file. So, that is the key. We are not actually going to use the key, but it has to be there. The value is all or part of a document. In our context, the value is a record which is one line of a file. So, its type is text. That is the string type for Hadoop. Similarly, the output is of type, output key is of type text because it is a word. And the output value is of type int writable, which is an integer value. Int writable is Hadoop's version of integer. So, with that in mind, let us look at the mapper, the map class which extends the mapper interface. It takes four types, long writable which is the offset in the file, text which in our case would be a single line, text which is the output key that is the word, int writable which is the count of that word. And this class has, this is actually an optimization. It has an int writable variable which is final static. It means it cannot be updated called one which is a new int writable of one. This is a optimization. So, it does not keep creating more and more objects of type int writable. It uses the same object many times. But it is not essentially. It could do it. It is an optimization. It could create new copies each time. Then it has private text word equal to new text. So, that is a word. Now, what it is doing is the map function takes a key and a text value and a context which provides other information. In particular that context tells us where to output the key value pairs which are going to be assigned to the radius function. It provides many other things, the context. It throws two types of exceptions. We are not going to worry about that in our pseudo code. So, the first thing this function does is line equal to value dot two string. So, value is text and it is converting it to a string because string is the type which we can deal with in normal Java code. Now, we are building a string tokenizer, new string tokenizer on this input line. So, that is going to break it up into words. The default tokenizer breaks up based on space and other white space characters, new lines and so forth. So, the tokenizer is basically an iterator. So, you can call next on it to get tokens. So, the while loop says while tokenizer dot has more tokens, word dot set tokenizer dot next token. What is this doing? The next token gives us the next word in that input line. So, the tokenizer breaks up the line into words and while it has more tokens, pull the next word and word dot set. What is word here? Word is this text field. So, the next word in the input line is saved in this word and then we are outputting word comma one. So, what is going on here? The one is simply this count one. It is an optimization to create it as an object and that word with the value one is being output here. So, that is a map function. Similarly, the reduced class extends the reducer function. So, that is a requirement. Now, by the way, I want to mention that there can be many map functions which need to be executed in the course of one large job. So, there can be many such classes. The name map is not important here. It can be any name. Inside this, there is a function map. So, that function has to be called map, but this thing can be called anything you want and we will see how to launch a job which does one map job of another later. Similarly, the class reduce here. This is one class. You can have more classes with different names here. Extends reducer. Again, the reducer takes four parameters, four types rather. This is the reduce key, reduce value, output key, output value. So, those are the four types. So, we have decided that this is the word and int is the count and the output is the word with the final count. So, text and int are to be. Now, the reduced function is called with the four link parameters in Hadoop. The first is the reduce key which is of we have said is type text. So, that is text here. Now, next is the reduce value is int writable. The reduce function is called with a number of these values. So, that type that it is called with has to be iterable parameterized by int writable. So, this is again a Hadoop specific. Iterable is basically an iterator which itself is parameterized by the type and that same type which is declared here has to be specified here. So, this is iterable int writables value. So, this is the second parameter to the reduce function. The third parameter again is the context which provides various things. Here, it is used to output the final key value, the final count for the word. Again, it throws a couple of exceptions which we are ignoring here. So, the reduced function does something very simple. Some is set to 0. Now, it is says for int writable value, this is the Java syntax for going over iterator. So, this is an iterator. So, if you say for something colon values, values is an iterator. So, here I specify a low variable val whose type is int writable. Now, note that this is a number of int writables. So, this val had better be of type int writable. So, that is part of the for loop. This is Java 6 syntax. If you are not familiar with it, this is how you can step over all the values in the iterator. Values is an iterator. So, now it says val dot get. Now, val is an int writable. Int writable dot get returns an integer value and I am just adding that to sum. Sum is 0. Initially, I keep adding. So, the idea here is that many map functions might have returned the same word with different counts. Now, all those counts are added up here to get the final count for a particular word. And finally, I write key, that is the word, a new int writable of sum. So, that is the sum, the total count for that word. So, that completes the map and reduce functions. The next set of things are how to set up the job parameters. But, before I get into that, I would like to take some questions in case people have not understood the map and reduce function. We have pandai peria value, please go ahead. In what are the languages we embed adobe in java only in other languages? Take the question as I understood it. The question I think was, what are languages can you do map reduce in? The example I have shown is using java, but the adobe platform actually has plugins to handle many other languages. So, you can write your map reduce code in one of many, many other languages. So, what happens is the adobe system is built using java. There are many languages which can be compiled down to java bytecode. So, any of those languages can be used with adobe, because the map and reduce functions can be written in those languages. But, compiled down to the same bytecode and the adobe infrastructure simply executes the bytecode. It does not care what language the bytecode was written in. So, as long as the type system matches the adobe type system. So, there are map reduce things available for many languages in the adobe framework itself. So, any other question please go ahead. Sir, we have another question. What is the equivalent command in Postgreal? Like ls we are using in linux. So, the question is in linux we use ls to see what files are there in a directory. What is the equivalent in Postgreal? This is not big data related, but a quick answer is if you are using the psql command line prompt there are slash commands this type slash help and you will see a list of commands. So, I think there is a slash dt or something which shows a list of tables and then there are variants of that to show different things. Or if you use a graphical interface that provides it for you if you use pg admin it provides that. But, let us focus on big data questions today if you do not mind. So, let me take some cat questions here. So, one of the questions which was asked is do we have to use for each or for in the map reduce code. So, maybe you know my pseudo code might have confused you. So, let me make this clear. Hadoop is described here and the java syntax which I have shown here works with Hadoop. If you go back to before Hadoop was introduced we use a lot of pseudo code. None of this will work in Hadoop this is pure pseudo code it is not meant to work on any system. So, the for each loop is not a real construct it is not java construct it is pseudo code. I use some other code here which look more like java, but this is also pseudo code this is not going to run on Hadoop. I cannot say if date between you know this is not valid java syntax this is not this is pseudo code. So, do not use any of these things in Hadoop use only the java constructs which you are familiar with or the constructs which are used here can be used. This for Hadoop on iterators even if you are not familiar with it I think from this example you should get a hang of what it is and you can use it. So, you most welcome to use that. Now, let me see if there any other things. So, people are asking about their guidelines for partitioning in big data again Hadoop has a lot of configuration parameters we are not going to touch on most of them, but we do have to touch on a few to get your program running and that is the next thing that I am going to talk about. So, let me come there are no other questions directly related to today's Hadoop code. So, let me go ahead and tell you the next part of Hadoop which is the job parameters and how do you set up a overall program in Hadoop. So, these are the next two slides. So far I only showed you the map and reduce function, you have to do a lot more to get your job running well a little bit more not a lot more. The first thing is that you have to tell Hadoop which classes contain the map and reduce functions. I sort of mentioned this earlier I told you that this class is called map and this class is called reduce, but your program can have many such classes with different names it does not matter what they are called. So, you have to tell you have to set up a Hadoop job and you can run multiple Hadoop jobs one after another each Hadoop job needs to be told that this is the map class and this is the reduce class. So, there are two methods set mapper class and set reducer class which you have to call and we will show that in the next slide to tell it which map function and which reduce function to use in this job. You also have to tell the job what are the types of the output key and value. So, that you have to do using set output key class and output value class this is for the final output the types of the intermediate outputs are also provided in the class you know those are generic classes. So, those provide the other types, but Hadoop job has to be told about the final output type. Now, the input format is another important thing you know I kind of puth saying that somehow the system gives you input key and input value from a set of file, but how does it decide what are these keys and what are the values. Now, these are functions which you can also give implementation of, but the simplest thing is to use one of the built in things the default format in Hadoop is the text input format. The text input format says that I have a number of files each line of each file is a record by itself. So, the new line character breaks a line and so each line is a separate record by itself. So, the value is the contents of one line and the key is the byte offset into the file. The file name is not part of the key, but the context parameter which is passed to the map function has the file name accessible from the context. So, if you need it that is also available. So, the set input format class there are other formats available which can break the thing into records based on some other terminator for record or you can even create your own which lets you parse the file and break it up into records. Then the next thing is where is the input data and where is the output data. So, the way Hadoop does this is you provide direct trees. So, you say add input path and add output path. So, input path is a directory which contains files. So, Hadoop will process all the files in that directory that directory may contain a large number of files that is it will process all of them and they are all part of the input. The output path tells it where to store the output. Now, if you saw earlier figure each reduce function outputs reduce key and value. Now, the reduce functions run on a number of machines M different machines here. So, each machine is going to create one or more files and the output directory is where all these files will be stored. So, they will have various names and you will see that in the lab today sorry I have a short here. So, the add output path that is why is which directory the file should output files have to be stored in. So, when you run a program you first have to create a directory and store all your input files in that directory and in the program you have to say add input path with that directory. There are many more parameters we are not going to cover all of them most have default values. So, here is our overall Hadoop program for word count. So, the main function is in another class class word count can call this whatever you want and that thing should have a public static void main string args. So, that has to be defined and it can throw some exception. This is setting up the job and executing it. What is it doing? Configuration, conf equal to new configuration. So, configuration is the stuff which sets up all the parameters. Now, we say job equal to new job conf this previous configuration and there is a name for it called word count. So, that is to identify the job. Now, for the job we are setting various parameters. In this case we have not really altered any configuration parameters, but if we wish to if it were really running parallely we could set the number of reduce machines, the number of map machines and so on actually tasks. A task is like a machine. So, number of map tasks, number of reduce tasks and so on can be set here. We have not set it. So, all the configuration parameters are default. In fact, the way we are running it is just a single machine. So, that is the default for the single machine version. So, now we have to tell the job that the output key classes text and output value classes intritable. So, text dot class intritable dot class. This is the way of telling it the classes of the input output key and value. Similarly, we have to tell it which are the map and reduce functions which classes contain those. So, job dot set mapper class, map dot class. So, note map is the function we and reduce are the functions we define here. Reduce and map. We could have called these whatever we want. This name map and this name reduce are what we use here. So, set mapper class, map dot class, set reduce a class, reduce dot class. Then we set the input format to text input format dot class. So, we could have also defined our own thing and use this instead here. Similarly, output format we are saying text output format dot class which says that it is a file with you know one line per record and the record will have the key and value. Then we say file input format dot add input path job comma new path arg zero. So, what is going on here? First of all this word count program is going to be invoked with two parameters. The first parameter is path is the input directory which contains the input files. So, now when I say new path with arg zero, arg zero is the command line argument with the directory name. So, path with the name which we have provided basically creates a path and we are saying add input path for that. And the next one is for the output. So, the second argument to the call is the output directory. Now, it in general you should delete this directory and it will create it automatically and so we are saying set output path with that directory name. So, when the job runs it will create that directory if it is not present and fill it with as many files as required. Now, we have set up the whole thing and then we say job dot wait for completion prove. So, wait for completion means start the job and then wait for it to complete because I have said true. If I said job dot wait for completion false this will return immediately and let me do other stuff while that job is executing. The job may take some time to execute it is running on big data potentially may take a long time to execute. So, meanwhile I can do other stuff, but in this case my program is just going to wait and when the job finishes the program will return. So, that completes the whole program which we have for today's lab. The first thing you will be doing in today's lab is running this program making sure it runs. The second thing you will be doing is modifying this program to do some very simple change in the functionality. So, the bulk of your lab will be actually in setting up the environment.  So, please follow the instructions in the lab carefully today. If you miss out some steps in those instructions you will get into trouble you will get error messages. So, please follow the steps religiously in particular there are some environment variables and so on which you should not miss. So, all of this has to run on Linux. Now, when your workshop coordinators were here some of them said that they have problems they do not have Linux they have only Windows. And a few people tried to set up Windows set up Hadoop on Windows that is actually again not trivial. There is a Hadoop port which runs on Windows server, but I do not think that port runs on normal Windows 7 or Windows 8 non-server versions. So, if you have only Windows in your lab you have a problem you may not be able to do today's live at all. I do hope that there are not too many centers in this soup. I hope all centers have been able to install Linux and run Hadoop, but those centers which could not please report this to us and we can follow those of you who are in such centers you know please go back later get access to a Linux machine and do this lab on your own. And if you have trouble you can use Piazza to post your questions and there will be somebody answering your questions. We have Apex Institute Bhubaneshwar if you have some questions please go ahead. So, what is the disadvantages of replication and what are the technique used to replicate file? The question was replication what are the disadvantages? The disadvantage is obvious you have to have more disk space, but it is essential in such large systems because there is a high chance that some disk somewhere will fail. So, you absolutely need replication to build any large system like with thousands or tens of thousands of nodes. The main thing you can control is how much replication to use. Should you use 2-way, 3-way or more or maybe if you do not do 2-way replication maybe you can use some version of RAID maybe parity bits stored in other places rather than full replication. So, all of these are actually used I am not getting into those details, but many of these things are under the control of the configuration for say HTFS or other similar systems. There are other distributed file systems out there. Any follow up question? Okay, so next question in the big data that is the Hadoop technology generally used for the parallel, splitting the file and parallel processing. And what is the basic difference with the Hadoop technology and the slicing technology in the data mining? The slicing technology in data mining. Slicing technology in the data mining. I do not know what you mean by slicing. Slicing technology. All of the, I am not sure what is slicing in data mining, but there are many different projects and approaches to parallel processing, but at the core they are all similar. You have to partition the data in some way and then process it and then maybe combine the results with some repartitioning and so on. So, there is parallel databases or Hadoop, the basic principles are the same. We can take some other questions from some other center. Sree Ramakrishna, Coimbatore, please go ahead. Sir, one question. Does Hadoop replace database or other existing system? The question is does Hadoop replace databases or existing system? That is a good question. How does Hadoop compare with databases? Should it replace databases? I am going to talk about that in the last part of my big data presentation. So, I will come back to that question. So, I will defer the question now. I do have slides on this and I will talk about it. Selvam College, Namakar. Sir, good morning, sir. Sir, how big data can be used in cloud computing areas? So, several people have asked this question about big data and cloud computing. So, first of all, let me explain to those of you who are not familiar, what is cloud computing? In cloud computing, you do not have machines sitting in your location, but somebody has a large number of machines out there and is willing to rent them out to you. Either at the level of machines or they will rent use of some service, maybe use of a platform. For example, Google Docs, let us use those documents online. That is an example of cloud computing. But Amazon gives, there are others, but Amazon is one of the big players here. You can go to Amazon and rent machines by the month, year, even by the hour. You can rent a machine by an hour. Actually, it is not a full machine, but a virtual machine, which might be shared with a few other virtual machines. You can rent it by the hour. So, cloud computing encompasses all of these kinds of things. Now, what is the connection between big data and cloud computing? So, you could run a platform like Hadoop locally on machines which you have bought, which is probably okay for academic institutes because we tend to have labs with 40, 50, 60 machines. I mean, I see on the video that many of you are sitting in labs of that kind of size and you can very well run Hadoop on across 40, 50 machines in the lab and do pretty useful. 50 is not bad. You can do a lot of parallel processing with that. But if you want to run Hadoop with really large data with 1000 machines, setting up your own cluster of 1000 machines for occasional use is crazy. You are much better off going and renting those machines by the hour from Amazon. So, there are many people who use Amazon to run Hadoop. In fact, Amazon now offers a service where you can basically get an instance of Hadoop running. You do not have to rent a machine. You rent the Hadoop service and you have to, of course, give them the map reduce function, the job config and so forth.  So, that is the connection between big data and cloud computing. Okay, thank you sir. Government engineering college Bikaner, you have a question please go ahead. Can we do map rating on video data also sir? The map reduce paradigm, you can do anything you want in the map and reduce functions. You can also have map only jobs. You just do a map and nothing else. So, there are people who use this infrastructure to do all kinds of things which are database unrelated. For example, there are people who want to do video transcoding. Let us say you have recorded video for this lecture at a certain frame rate. You also want to convert it to other frame rates for people who have a lower bandwidth connection. Now, in the case of a view, this is done, I think, on the fly. But there are people who allow people, you know, like YouTube equivalents which let you upload file and then you have to convert it. There is a lot of these jobs which need to be done. So, people have used map reduce where the map function simply does video transcoding for example. So, what is the benefit of doing it? Well, you get fault tolerance. You have the ability to store these files in a distributed file system, HDFS. So, yeah, I mean this infrastructure is used for many kinds of things which have nothing to do with databases because it provides a fault tolerant parallel processing setup. And so, it is used for that aspect, not for the data. Now, the reduced part of it is more data related and things which use map and reduce tend to be database related. So, in fact, somebody was asking about the connection between Hadoop and databases. So, Hadoop is a very low level infrastructure. Databases provide you a much higher level view. So, people have tried to provide a high level view that databases provide including SQL on top of underlying Hadoop implementation. People have also said that the Hadoop implementation is not very good for data based query processing. It is actually not very good to be honest. You can do things much more efficiently if you built something specifically for data processing, not as a general purpose platform. And so, now there are several platforms which are tailored to these kinds of things. I will have a few slides on that coming up. I will take a couple of questions from Jack. So, I will look at a few of these things. One of the questions says, can you explain Hadoop with practical implementation? We do have this word count program and today's lab you will be actually running Hadoop program. So, it is very much part of the syllabus. The next question is, can we use Hadoop for implementation of machine learning algorithms which are recursive in nature? That is a good question. The Hadoop job consists of a map task and then a reduced task. Now, what if you want to do something more complex which has to iterate or recursion and iteration can, you know, one can replace the other. So, if you have a machine learning task which needs multiple iterations, how do you do it on Hadoop? And the answer is in that main program which we saw, we set up a job and ran it. It is possible to set up a sequence of jobs and run them one after another iteratively. So, it is very much possible in Hadoop, but possible is not equal to efficient. So, people have noted that it is not very efficient when you do such iterative things. There are better ways of doing stuff. So, there are several platforms which have. So, first of all, Hadoop itself, there is a new version called YAN which is in beta right now, but it is already quite usable which has better support for iteration. And then, there are many other platforms which people have built for specific purposes. For example, for graph algorithms, I mentioned pregel. That is much more efficient for iterative processing of graph algorithm. Then, there are other things which people have built for iteration. The next question on chat is something about memory error while running the Hadoop test examples given on the Moodle website. Please help. I am not sure about this particular problem. I have not seen it myself. Maybe you can post it on Piazza and somebody can take a look at it, but we did not see this on anything going on here. One possibility is that your machine has very little memory, but that or you have many other tasks running on your machine which means it is out of virtual memory and now you are in trouble. So, maybe you should close other processes and just run Hadoop nothing else. Next question is how can we use Hadoop in information retrieval especially web information retrieval? So, your lab assignment is to modify the Hadoop WordCon program we have given to make a small change which essentially is like one step in building an index, an inverted index as it is called. Inverted indices are the core for web search system. So, you are going to do an assignment which is not really build an index, but take one step towards building an index. So, when you do your assignment that will become more clear. It is not a very hard assignment, but it is a first step in building an index. The other part which is for web crawling. Yes, Hadoop I believe is also being used for web crawling, but there were some limitations. It was used extensively for web crawling. It probably still is in many places, but Google which pioneered using their map reduce for even for processing websites and indexing have actually moved to an alternative architecture which I forget the name, but they have a new system which is tailored to web indexing. You can use Hadoop for this or general Google map reduce, but they have a system which has some performance improvements over the raw map reduce. So, that is what they are using internally I believe. The last question which I will take from status. After range partitioning what does the reduce function do? The idea is as follows. Range partitioning is used to ensure that all records with a particular key land up in one machine, but now that machine has to is getting such records from many other machines. So, if we go to the white board. So, this was a range partition which means that a particular machine here p naught gets everything with g less than 10. Now, there are many mappers which have generated values with g less than 10. Let us say all of that comes to this machine, but for a particular g value let us say 7 it has to collect all the records with g equal to 7. So, what happens is after the range partitioning each machine does a merge the things which it gets are already pre sorted. It does a merge to get a final sorted list of all key values and once it has that it can invoke the reduce function on specific key values. So, the final sorted list has everything with key value 7 together it can call the reduce function on all those values with key 7. So, that is how that step works. May be I will take another one or two questions live questions then I will move back to my slides. We have BSA college. Good morning sir. Sir can we use map reduce for business intelligence? Yes certainly. So, this is an active area lot of companies in the BI space are working on map reduce implementations. Now, the thing is BI was focused on systems which use SQL as the back end. The initial map reduce systems were not SQL systems they were different the Hadoop is not SQL. So, interfacing an existing BI system to such Hadoop framework was not easy, but in the last few years there is a new layer on top of Hadoop called Hive. It is actually not just a layer the implementation is currently on Hadoop, but now there are new implementations of Hive on other infrastructures. What does Hive provide? It provides SQL on top of such big data. So, Hive can run on top of a map reduce platform. It can take SQL queries and run them and give back results and run them using Hadoop underneath. So, you can run these on very large data sets. So, now that Hive is available people have started building business intelligence tools using Hive as the back end. They generate SQL let Hive execute the SQL. So, that is a very active area right now. So, any follow up? Good. So, let us move back to these slides and let me wrap up big data. Before we move to a short description of Hive and so forth, let me start with one other technical thing. Let me show the figure for this. So, here what happened is each map task collected it ran map on a number of values here and then it collects those values and then partitions them to all the reducers. Now, if you saw our word count program if you see map, map outputs a word with a count of 1. Supposing a particular machine has a number of documents, large number of documents. Now, the same word may occur many times in one document and many times more times across documents. Now, if you send this output across the network to a reducer, the network overhead is going to be very high. You do not want network is a bottleneck today. It turns out that typical machines today use Gigabit Ethernet. 10 Gigabit Ethernet is still considered very expensive whereas a machine typically has several hard disks and each hard disk can pump out 50, 60 megabytes of information. So, you can generate data here much faster than the network can handle. So, you want to reduce network traffic. So, the idea is that supposing we have we can do some kind of local reduce within the machine and do some sorting collect all occurrence of word together and add up their counts. So, that idea is implemented directly in Hadoop through what is called local pre-aggregation to minimize network traffic. So, in Hadoop there is a call in the job configuration to use combiner. So, if you turn on this option, it will use the reduced function as is locally once and the output of that reduced function is fed to the main reduced function. This works fine as long as the input and output types of the reduced function are the same. If the input and output types of the reduced function are different you run into trouble in which case you can create your own combiner function and it has its own input and output types and reduce can have its own input and output types. So, I am not showing details here, you can look it up. In fact, the combiner can work at two levels one is within a machine and if you look at the architecture of most data center there are a number of machines in racks and let me use the whiteboard to explain this. So, if you look at typical data center it has a number of racks. This is what a rack kind of looks like and it has a number of these racks. Each rack has a number of servers they are very thin they are like a inch or so high. So, now a rack has a number of these. So, very common number is something like 40 or so machines in a rack and in the rack there is a switch typically a 48 port switch. Now, there are many more such racks. So, then there is another switch here which is connected to each rack. Now, typically these are 10 gigabit per second and the network connection inside the rack are 1 gigabit per second. This is a very typical setup. This may change over time to faster thing, but this is what is common today. So, the idea is that if you look there are 40 machines here. If they all start throwing traffic across to other racks you are going to generate 40 gigabits gigabits per second whereas this network can only handle 10 gigabits per second. So, it is good to reduce the traffic coming out of a machine. It is also good to reduce the traffic going out of a rack. So, in fact you can set up Hadoop to run a combiner for all the data in a machine. Then you send all the data to one of the machines in the rack which will again affects not just one you can parallelize this also, but you can do a reduced step inside the rack to further reduce the data volume and then you send the data across to machines which are in other racks. So, at each step you do a combiner run pass which reduces the amount of data. So, that is supported in Hadoop. Now, what about implementations? MapReduce in this scale was pioneered by Google, but they do not open source very much. Hadoop is open source thanks to Yahoo and there are other implementations which combine databases with MapReduce. Somebody had asked this question how do databases and MapReduce combine? So, there are several different answers. One of the answers was you take an existing database system parallel database system and add support for the MapReduce paradigm in it. So, as the data which if you recall I had mentioned was one of the co-founders was a person from IIT Bombay called Mayank Baba. So, they pioneered MapReduce within parallel databases and there are others, but there is a completely different approach which is now gaining traction. So, to understand that let us compare MapReduce versus parallel database. So, there is a lot of discussion about this. So, MapReduce is widely used. Lots and lots of people are using it. Even in India there are many companies which are using Hadoop. Particularly Hadoop is used, I should say. Compute page rank, build keyword indices, do data analysis of web click logs, so on and so forth, many, many uses. But if you see many of these things, not all, maybe not keyword indices, but analysis of web click logs, it is actually easier to do it in SQL than to do it in Hadoop. And database people said, look, we have been doing it for a long time and we know better how to handle this kind of data for simple query processing. And the Hadoop people came back and said, look, you people may say you have built parallel databases, but they are very expensive. Nobody actually owns them and they run at hundreds, not thousands of machine scale. And we are much better at handling failures. We allow procedural code in MapReduce and data of any type. So, that is true. On the other hand, many of the uses of MapReduce turned out that they actually use structured data, data which can easily be converted to structured form and then you do aggregates. Now, for this, SQL is actually a much better language for expressing what you want than hand coding using Hadoop. It is actually crazy how much effort you have to put to create a Hadoop program, which you will see today to do something simple, which you could write as a two line SQL program. The word count you cannot do exactly in SQL, but some of the other things are much easier to write in SQL. So, once people realize this, they built other interfaces which provided declarative languages on top of MapReduce. So, the first project in this space built a language called PigLatin, which is not a SQL, but is more like relational algebra and you can have a number of operations on data. So, this came from Yahoo and this was open sourced by Yahoo. So, you can actually use this. You can download and use it. But soon after this, Facebook had a different project called Hive, which they decided to forget new languages. Let us use SQL itself as a language, maybe a subset of SQL, but it is SQL. And the other thing they did is that they said, look, Hadoop says files can be any old text format, but let us put a restriction that the files have to have a schema associated with them and a way to take the raw data and the files and output fields, some structured format which can then be consumed by the SQL system. Otherwise, for every file, you have to build your own parsing routine. That is not good. Let us have a standard format for files. Of course, if you have a web server which generates a log in its own format, then you have to add to the formats that Hive supports to understand that, but there are only a few such. So, Hive has a notion of schema for every piece of data it has. So, it is much closer to SQL in that sense and it supports the SQL language. So, this is now very, very widely used. Hive has really taken over from raw Hadoop. Many people who are using raw Hadoop for data analysis have moved to Hive. Of course, Hadoop continues to be very widely used for things which cannot be expressed in SQL. There are many such uses. But those which can be done in SQL are quickly migrating to Hive. There is also a system called Scope from Microsoft which does similar things. And finally, there are many extensions of the MapReduce paradigm itself to add joins, pipelining of data, blah, blah, blah. I will not get into the details to make it more efficient. So, that is the state of the world. Hadoop is being, is still used as the underlying infrastructure, but there is a layer on top called Hive which is widely used. Now, there is a last part which I want to wrap up big data with. So far, we have been focusing on the decision support. We have a lot of data. We want to do analysis using the data. Now, the other part is if you want to store update retrieve data, files are not the right format for it. Files are fine if you just have logs and so on which do not get updated, but there are many uses which are more OLTP like. So, if you want to store records which can be individually accessed, updated and so forth, files are not the answer. So, again here Google built the first system which address this particular issue called BigTable. This is a really cool system because they give you an abstraction of a table with more features actually than basic table. You can retrieve records. You can store records in this table. The records themselves can have further structure. They are not necessarily first normal form. And the key thing here is that these tables can be enormous. They can be many terabytes, even petabytes. And that view, a logical view of a table is provided, but the actual physical table is stored across potentially tens of thousands of machines. And the nice thing is that as the table grows, you can incrementally grow the system also by adding more processors to the system. And BigTable will smoothly migrate little by little, move data into the new processors. It will not, it does not have to bring the system to a halt ever. So, this was a really nice system. And once they published it, people heard about it, they wanted to build their own. So, Yahoo built a system called PNUTS. And more recently, Apache built a system called HBase. This is open source. PNUTS is not. So, HBase is now wide, open source and widely used now. Many people are using HBase. And all of these provide what is called a key value store. It is not a full-fledged database. You can give a key, like a record with a primary key. You can give the primary key and the content of the record and say store this. And you can give the primary key and say retrieve the record. What these things lack is the support force indices other than the index on the primary key. Again, people have, after they were, this is the build, they started adding support for secondary keys and transactions and so on. So, that is still an active area which is under development. So, if you use HBase, you are not going to get too many functions in terms of secondary indices or transactions. But people are developing such systems few years from now. Open source may be available. Google has already built these in-house. They have actually built massively parallel scalable database systems which have SQL and all the bells and whistles of database, most of the bells and whistles of databases. But it runs across enormous numbers of machines. So, that is getting popular within Google and eventually you will get publicly available open source clones. Last few questions. Bansal college Bhopal. Sir, my question is, what is the basic difference between RDBMS and Hadoop? The question is, what is the difference between RDBMS and Hadoop? So, Hadoop is a layer which provides just a few functionality. It just provides map and radio which you can run in parallel across many machines with fault tolerance. It is not a database at all. And the point I was making in the last few slides is, that you can build a database on top of Hadoop. In particular, if the database is focused on decision support, so you want parallel query processing, you can build it and Hive is exactly such a system. It is a database which supports SQL which is built on the Hadoop infrastructure. So, Hive is the underlying execution engine for, sorry, Hadoop is the execution engine for Hive. That is the basic relationship between Hadoop and a database system. Now, Hive itself provides an SQL layer. Now, people are actually building Hive implementations which are sitting not on Hadoop, but on other infrastructure which is more suited to database operations. So, that is the next wave in Hive. There are some commercial systems already available which claim much better performance than Hive on Hadoop. So, it is Hive on some other infrastructure which is much faster than Hive on Hadoop. This is from GRIET, Kukutpalli. Sir, may look, may not know, what is the difference between HBS and HDFS, sir? HBase and HDFS. Okay. It is a good question. Sir, let me just use the slide here. So, the question is, what is the difference between HBase and HDFS? HDFS is a file system. You can store files, just like in your local machine. You can create files, you know, you can write two files, you can read files. HDFS provides exactly this interface. The only difference is that the files are stored across a large number of machines and it has replication and fault tolerance and all that built in. So, if a machine dies, HDFS can continue to live and provide you access to those files. HBase provides similar functionality, but for records. And what it does is, it is called key value store. By key value, what we mean is that a record must have a key, like a primary key of the record and the value is the value of the record. And the only way to access a record is by specifying a key value and the system will retrieve the record or you can give a key and a value and it will store the record. So, it just supports two basic operations, get and put. There are some extensions, but these are the two basic operations. So, this is what HBase provides. It is not a full-fledged database on its own. So, now you could again take Hive and make it run on Hadoop. So, by the way, Hadoop is map reduced. Hadoop, so far I have been talking of running on top of HDFS. Hadoop also can run on top of HBase. So, you can store records in HBase and then run Hadoop program directly of the HBase storage. So, it does not have to be HDFS. Does that answer your question back to you? Thank you, sir. And one more question is there. What is the function of mappers? Okay, the question is what is the function of mappers? So, I already told you that you have to write a map function in the reduced function. A mapper is basically a task which runs these map functions on multiple key value pairs. So, that is a map task. So, mapper is a task that runs these map functions. Similarly, reducer is a thing which runs the reduced functions. We have Fisheela Adanshan Godavat. Please go ahead. Good morning, sir. Sir, my question is whether on our desktop computers and with Linux or Windows, can we have Hadoop on it and which is the, other than have, is there any database system which can support the underlying Hadoop file system, sir? Thank you. Good question. So, first of all, Hadoop was built on Linux systems and it can run on your desktop very easily. Resource constraint is very small, particularly if you just run it in single machine mode. That is the default mode we run it in. Hadoop can also be configured to run across many machines. Then there is some more work to set it up. So, we have not put that in your lab, but you can certainly follow the instructions which are available and set it up across all the machines in your lab. I can see your lab has lots of machines. So, you can run Hadoop in parallel across all those machines. Now, coming to the Windows part. So, Microsoft realized that Hadoop is very widely used. Initially, they built their own things called Dread and Scope, which is a database layer on top of Dread. Dread is their equivalent of Hadoop. Unfortunately, in the market, Dread didn't do that well. So, now Microsoft has also paid this company called, what is it called? I forget the name. It has paid this company to build Windows port of Hadoop. So, that port is currently available for Windows server, not yet for Windows desktop. So, as of now, I believe it's difficult to run Hadoop on a Windows desktop. Some people have ported, you know, Siegwin, which provides Linux functionality on Windows and is more complex, but you can't do it more straightforward manner right now. I'm sure that will change. What is the last part of your question? Is there any other database systems other than Hive which can support Hadoop underlying system? So, many people have been developing projects which do several different things. One is replace Hadoop by extensions of Hadoop or new things which can support Hadoop as well as other relational operations such as join and so on, which Hadoop does not support natively. You can code joins in Hadoop, but it's more cumbersome and slower. So, many people have been working on layers underneath Hadoop and changing the underlying implementation. And also, many people including some of the same people have been building other systems which will run SQL on top of Hadoop or other similar systems. So, there are quite a few projects out there, but the Hive project has really taken off and it's kind of becoming the accepted standard. So, you might as well go with that as of now. We have RBS college. Go in, Batu. Please go ahead, RBS. How to handle unstructured data using Hadoop with respect to map-reducing algorithm? Unstructured data. How to handle unstructured data using Hadoop? So, let me answer that question. The question is how to handle unstructured data using Hadoop. The word count program which we just saw was unstructured data. There are a number of files and we are breaking up the files into lines and counting the words in each line. You might alternatively not break it up into lines. You can treat the whole file as one record and do word count per file. But the bottom line is that example directly shows how we can use unstructured data and run Hadoop programs on it. And your lab exercise is to, like I said, take the first steps towards building a keyword index. So, that is also something which is functionality on unstructured data, which you will be using Hadoop for. So, our goal here was not to have you do SQL on Hadoop, which like I said, you might as well use Hive if that's what you're doing. But to get exposure doing other kinds of things on Hadoop, which you will still need. Even if you have Hive, if you want to do things like this, word count is a toy, but other operations on unstructured data, Hive is not going to solve that problem for you. You will have to go back to Hadoop. So, that is our goal in the Hadoop assignment. I'll take maybe a couple of questions from chat. This one's between MapReduce Database and Parallel Database. Like I said, we can build parallel databases on top of the MapReduce infrastructure, or you can build it on some other relational, parallel relational engine. That is, traditional parallel databases had an underlying engine, which could run all the relational operations in parallel, join, group by aggregate, outer join, duplicate elimination, sorting. All of these, there are parallel versions, and the parallel database chapter in the textbook talks about how to do all this. So, traditional parallel databases used this as the underlying infrastructure, and ran SQL queries on top of the parallel relational operations. The Hadoop infrastructure doesn't directly support these operations, but you can work around it and implement the operation. That's what Hive did initially. But more recently, people have built other implementations and put Hive on top of it. So, what you get is essentially back to a parallel database after removing Hadoop from underneath. People have asked about how to check status and health of cluster in Hadoop. So, I don't have the details, but there are tools which can let you monitor what a Hadoop cluster is doing. So, you can search on the web to find these tools. I think it comes with Hadoop. If you download Hadoop sources, it comes with that. Last one or two questions. One question is, what is fault tolerance in HDFS? So, fault tolerance means that in spite of something failing, you should be able to continue executing your system. In the context of HDFS, when one of the machines which contains data fails, it's okay. There is a replica somewhere else. HDFS will tell you where the replicas are located, so you can go read it from there. And the read, HDFS provides an API. So, the read function in that will do all that underneath. You don't have to worry about it. You just say read, and it will get you the data from whichever copy is live. So, that is fault tolerance in HDFS. There is also fault tolerance in Hadoop. It says if a map task fails, Hadoop will automatically re-execute that map task on some other machine. That machine may be dead. The task will be executed somewhere else. Even if one of the machines is slow, all the other machines have finished their map task, but one machine has not yet finished. It's live, but it's running very slowly. Hadoop will automatically execute that map task on some other machine in parallel, and whichever finishes first will be taken. So, if this machine was very slow for some reason, the other machine will finish and get the result. If this was a temporary glitch and this machine actually caught up, well, this one can be used. Similarly, if a reduced task fails, Hadoop will re-execute the reduced task and generate the output files. So, there's a lot of fault tolerance built into Hadoop, and similarly, Google Maps reduced. I think I'll stop on Hadoop and big data here.


Siamese cats are perhaps best known for their sleek, distinctive appearance. Although many Siamese cats are a silver-gray color with blue eyes, these pretty kitties can also be orange, brown, cream, and even blue or lily colored, among other coat variations. But there's much, much more to Siamese cats than their mysteriously good looks. For example, did you know that Siamese cats are one of the oldest breeds of domesticated cats in the world? Or that their coat coloring is determined by their genetics and the temperature of their surroundings? If you want to learn more, keep on watching, because we are going to talk about interesting facts and myths about the Siamese breed. 1. Coat coloring. As we said, Siamese cats' coats can be an incredible range of colors, and those colors aren't only determined by genetics. Grab your lab coat and get ready for a little bit of biology. The color of a cat's coat is determined by several sets of genes, and the patterns of their coats are determined by modifier genes. Well, Siamese cats have a special modifier gene that inhibits pigment from developing in the fur resulting in albinism. But, thanks to the magic of science, the modifier gene for albinism can only affect the cats for above a certain temperature, or between 100 and 102.5 degrees, to be more precise. Once a Siamese cat's body drops below approximately 100 degrees, or the cat is in a colder environment, her genes for coat color turn back on and carry pigment to her fur. Because a cat's body is cooler around her nose, ears, paws, and tail, that's where pigment is most commonly deposited on Siamese cats. Most Siamese cats are born completely white and develop their markings in the weeks following birth. Why? Because the womb is very warm and blocks the cat's color gene from reaching her fur. Following birth and being exposed to the cooler atmosphere for a few weeks, Siamese kittens will start to develop pigment around their faces, tails and paws. 2. An old breed. Although no one is certain when Siamese cats were first bred and domesticated, it's commonly believed that they originated in Thailand around the 14th century, when they first appeared in a Thai manuscript. This would make them one of the oldest cat breeds in the entire world. 3. Common traits. For a long time, many Siamese cats had crossed eyes and crooked, kinked tails. Although these traits were undoubtedly the result of certain genetic factors, many legends offered alternate explanations for Siamese cats' interesting appearances. We will talk about these legends later on in the video. 4. Royal treatment. Hundreds of years ago, Siamese cats were treasured by royal families in Thailand because of their distinctive, unusual, and beautiful appearances. In fact, members of these royal families believed that a Siamese cat would receive their soul when they died, and the cat would spend the rest of his long life living in a temple and being pampered by monks and priests. Not a bad gig, huh? 5. Presidential cats. Although Siamese cats have been domesticated in Asia for centuries, they didn't appear in the United States until the late 1800s. One of the earliest known owners of a Siamese cat was actually U.S. President Rutherford B. Hayes' wife Lucy. Which means Siamese cats have a history in the White House too. Her Siamese cat, named Siam, was a gift from a U.S. diplomat serving in Thailand. 6. Movie stars. Old but good movies like The Incredible Journey, The Lady and the Tramp, and That Darn Cat, starred Siamese cats. In fact, the entire plot of that darn cat is centered on a Siamese named D.C., who helps stop a kidnapping. While D.C. from that darn cat helped prevent a fictional kidnapping, two real-life Siamese cats helped foil espionage back in the 1960s. When two Siamese kitties living at the Dutch Embassy in Moscow, Russia began scratching at a wall, their owner suspected they were reacting to a noise that couldn't be detected by the human ear. He was right, 30 small microphones were discovered behind the wall. 7. They are a talkative bunch. If you have a Siamese cat, or have spent time around one, you know they're extremely talkative. Siamese cats will vocalize their opinions on their food, observations they make through the window, and basically anything that they encounter throughout the day and night. 8. The legend of Buddha and Siamese cats. A long time ago, a terrible war was fought by Siam to defend its kingdom. Since all Siamese men were sent to battle, Buddha's golden goblet was left unprotected. Two Siamese were chosen to preserve the goblet in the Sacred Temple. The male cat was named Teen and the female Chula. All went well until Teen became uneasy and left, after mating with Chula, to find a new priest capable of caring for the goblet. Port Chula, now expecting kittens, never took her eyes off the sacred goblet. As a safeguard against sleep, Chula nervously wrapped her long tail around the goblet stem to foil thieves. When the kittens were born, they amazingly received Chula's physical quirks she developed as a watchcat. They had crossed eyes and tail kinks. So if you possess a Siamese with crossed eyes, or a kink tail, remember Chula and her brave protection of Buddha's Sacred Goblet. 9. The Asian Silver Jade Ring. Here's a second story about the Siamese and its kink tail. Once, a princess of the royal house of Siam used her cat's tail as a ring stand when taking a bath. The cat, not wishing to loose the ring, obligingly developed a kink tail to safeguard the special object for his princess. 10. The Legend of the Cat and the King. Many years ago in Siam, or known today as Thailand, the Siamese cat was a very exclusive feline. Being reserved for the king and royal family, no common sorts such as ourselves could own one. When a royal died, a household Siamese was selected to capture and contain the dearly departed soul. Don't think anything bad happened to the cat. It wasn't sacrificed, or buried with the dead, but instead lived a life of luxury with the monks and priests of the local temple. There, the lucky Siamese ate all the best food from gold plates, and slept on the softest of silk cushions. These luxuries were provided by relatives of the deceased, in a desperate bid for good fortune and special blessings. The selected cat was believed to possess rare powers of intercession for the soul of the departed. When the king himself died, his Siamese representative attended the new king's coronation so the former king could be present at the royal event and festivities. 11. The Legend of Noah's Ark, and the First Siamese Cat. The story of Noah is known far and wide. After Noah built his ark, he filled it with animals two by two. It began to rain. It rained, and rained, and rained with no end in sight. The animals in the ark became very bored and searched for something to do. The monkey, spotting the pretty lioness, immediately fell in love. Wooing occurred and soon, tiny Siamese kittens were born. They had the cleverness and agility of the monkey, and the courage of the lioness. Come to think of it, many Siamese cats do look like monkeys. If you enjoyed this video, kindly press the like button. Also don't forget to subscribe with notifications on, so that you don't miss out on videos like this. Thank you for watching.


Hey everyone, welcome back to another Python tutorial. We're still looking at UNIX specific services And this time we're looking at a new module one that we were kind of branching off of from the last video on PWD or the password database. This one is SPWD, which is the shadow password database. We were looking at the attributes and the kind of object instruction we were receiving from the last module PWD and the portion that should have given us the user's password actually gave us an X or sometimes in other cases an asterisk and this X, I believe, which we saw on note actually in the old documentation here. I'll show you PWD. It tells us that if it's not available, you can try the SPWD module, which should contain access to the encrypted passwords in the ItSetRushShadow file. So let's take a look at that. Again, this module provides access to the UNIX shadow password database. It's available on all UNIX versions and it looks like it has the same functionality as the PWD module. It still returns things from the password database as tuple-like objects, which has attributes and indexes that refer to the username, the password, of course, which hopefully will be encrypted and actually visible this time around, the date of the last change, days between changes, and more stuff regarding the password. So hopefully we'll get a little bit closer to understanding these passwords, being able to manipulate them the way we want to. Remember in the last tutorial, we actually created a new user. It was Fubar. So if I try and log in again as Fubar with su-fubar, the password we set was Apples and when I log in, I can say who am I and now it's obvious that I'm Fubar. So we should be able to play with this user with the Apples password to kind of experiment and see more of what we're working with here. So these attributes and things in the password object that we get returned to us are normally strings, at least for the name and password items. Everything else is an integer because that's just what it needs to be and of course key errors are raised if we don't find them. So it has pretty similar functions to the last PW module, PWD module. Kind of the same functionality with a given username and P all return the list of everything. So let's try and play with them. I'll fire up my shell here. Let's import spwd. spwd gets sp all. Now if I run this, I'm not getting anything. Why is that? When we use PWD the other one, we got like freaking everything, right? We got PW all and we get all of these users and all the information about them. Why is that? What are we overlooking here? Now if we head back to our documentation, I skipped over a kind of a crucial line intentionally so you guys might have the same logic process that I did when I first played with this module. I didn't realize you have to be root to be able to do this. You must have enough privileges to access the shadow password database, which usually means you have to be root. So the user that I'm running right now or John, like we saw in the last video, I don't have the privileges to actually be able to read the shadow password database. So when I try and run get sp all, I'm gonna get return an empty list. I'm gonna be returned nothing valuable. So let's play with this a little bit more. I told you that everything was actually stored in the etc shadow database for shadow encrypted passwords or shadow password systems. So let's take a look at that. Let's try and cat it out just like just like how we did with the etc past WD database. We can see that one pretty easily, but if we try and cat etc shadow. Oh, permission denied, right? It's because we need to be root. We need to have the privileges to be able to do this. So if I try and actually pseudo cat etc shadow. Normally you'd have to type in your password if you're in a new terminal since I already created an old user with this terminal. I don't have to. But let's check this out. I'm gonna go I'm gonna put this in the less so we can look through it. And hey, we see all the same users that we were looking at before with a similar kind of setup with etc the password file Where things are separated by colons and we have different fields and different indexes and attributes that we're actually seeing. So hey, there's the john user. That's me. And there's the foobar user that we created. So this Gunk, I think should be the encrypted password, right? The second index. Is it does it say in our documentation? Yeah, PW password is the Actually for SPWD in this database SPPWD is the encrypted password. So hey, that gets us a little bit closer to looking at passwords. So let's try and play with this now in Python because right now we were just looking at the file and the Linux shell But we need to be able to do this with our actual Python program. So I'm gonna close this shell that I have open and I'm actually gonna run pseudo idle So now I have a root shell So I can use So let's go ahead and import SPWD and Now when we run get SP all Hey, we get all the stuff that we wanted to see and of course now we can use SPWD get SPNAM To use it with a name and let's look at foobar and now we get our foobar user and this encrypted password that we want to be able to look at cool Let's uh Let's let's see. That's to be the first one index, right? Yeah, okay. There's the password and That's exactly what we want to be looking at We have all this other information that's actually stored in the documentation here SP flag SP aspire and all these other things that you might be interested in but for us We want to be a little bit mischievous. We want to look at these passwords on a Linux system So thank you guys for watching this tutorial. I know this one is a little bit Simpler than the last one since you already introduced of this this kind of module and these kind of things of what it does Pretty simple just returns this object and you can look at the user database and the password database And now we're getting closer and closer to looking at actual passwords They are encrypted, but you know what we can work with that because remember Back here in the PWD database documentation. It told us Normally these passwords are encrypted with a DES derived algorithm With the module crypt and Python At least we can go through with that algorithm to Encrypt things with that. So hopefully you guys might see a little bit of potential here And we'll move on to it in the next tutorial. Thanks everyone. See you soon


This is the VOA Special English Education Report. A program in the United States brings scientists and engineers into elementary schools to teach teachers more about how to teach science. Dave Weiss is a retired engineer. One day each week, he volunteers at Georgian Forest Elementary School in Silver Spring, Maryland, near Washington. He works with teacher Fred Tunique on science projects for 10-year-olds.  I can see that he really loves what he's doing. I get as much pleasure from helping the teachers as I do helping the students. Student Jada Lockwood says she enjoys Mr. Weiss's visits to her classroom. She likes the drawings he uses to explain scientific ideas. The American Association for the Advancement of Science sponsors the Senior Scientists and Engineers program. Dave Weiss has been a volunteer in that program for many years. The scientists and engineers help teachers in elementary schools improve their skills. Mr. Weiss says he and the other volunteers help teachers by providing hands-on expertise. He notes that science is an area in which many elementary school teachers have limited experience. In elementary school, for the most part, your regular classroom teacher is responsible for teaching science, along with reading and math. And if they don't have a strong science background, just by nature they're going to tend to under-represent science in the curriculum. Fred Tanik agrees. He just started teaching science classes a few months ago. He says, a lot of times I'll spit out information I learned in the book, or things that are part of the curriculum. Dave helps me learn how to supplement that information so that it's more relevant to them, so that it will be more relevant to their work experience later on in life. 15-year-olds in the United States scored about average in science among countries that took part in testing by the OECD in 2009. The OECD is the Organization for Economic Cooperation and Development. But Dave Weiss says he is hopeful for the future. He says he tries to give elementary school students a solid foundation. His hope is that they'll develop a curiosity about what's going on around them. For VOA Special English, I'm Carolyn Presuti. We have more programs at voaspecialenglish.com where people can read, listen, and learn English with texts, MP3s, and activities. We also offer English lessons at the VOA Learning English page on Facebook.


Time is important to each of us. It affects our lives in many different ways. Today, Ralph Aims, horse trainer by profession, finds time working against him. As soon as Ralph can get away from the track, he and his family are leaving on their first vacation in two years. 3-6, Quebec. Thank you. Now further ado, let's send much time before the weather closes in. Ralph took it to Charlie's gas station intending to pick it up first, guess I forgot. I guess you did.  I'm looking forward to getting in a good round just as soon as I got here. But, uh, oh, I guess I'm just not in the mood. Oh, really, an avid golfer not in the mood because of your landing? You saw. Yes, I saw. Listen, those things can happen. Not too often, I hope. More often than you think. The number of mistakes I've seen private pilots make over the years is astonishing. Mistakes that often cause accidents. Yeah, when it's no fault of the plane. That's right. About 80% of the time, it's no fault of the plane. The fault is generally not with a pilot's skill. It's more often psychological. It's really his emotional state that really does him in. You're beginning to sound more like a psychologist than a man in the aviation business. Maybe so. That's because practical psychology figures in. Pilot errors like the one you made generally spring from something I call an itis. It is that nearly racked you up on my runway. These ease of general aviation pilots. They feel they must get there. Frequently, they kill themselves trying. Get there, itis. I guess I had it. Don't let it get you, Mr. Ames. You won't make the same mistake again. Take the professional or successful businessman, for instance. They frequently suffer from what I call give order itis. They're used to giving orders, not taking them. Fred Jorgensen there, he's a good example. I'm in the construction business. One afternoon, I had planned across country to check out some property I wanted to buy. But before I could leave, I had to straighten out a few problems. When I finally got away, I phoned ahead to the airport and told them to have my plane ready when I got there. The flight service station pointed out the location of some weak funnel activity that could cause me trouble, or so the kid said. He was polite, of course, and could only offer suggestion. But his suggestion was clear, stay on the ground. What did he know? I've been flying for 30 years. Guess he thought I was too old or something. While I didn't take his advice, the weather looked all right to me, so off I went. I was landing the wrong way. I made a mental note to find out who the guy was, telling what I thought of his flying, wondering what could have gone wrong. And then suddenly, in my mind's eye, I saw it all. My approach to the active runway. I had made the wrong turn and taken off downwind. After realizing my mistake, things seemed to settle down. I decided to check the weather at my destination. The weather was down and the entire area was closing in fast. Yet there I was, flying right into the teeth of it and for no real reason. I kept thinking about what the kid at the flight service station tried to tell me. That close call ended my career as a bold pilot. You see, Mr. Ames, there are old pilots and there are bold pilots. But there are no old bulls. Very well put, Mr. Weitzel. Call me Bill, please. Fine, I'm Ralph. About your golfer, Ralph, I'm going to meet Wally Gray this afternoon. Would you like to join me? Is that the Wally Gray, the golfer? Yes, he's the one. Why don't you meet me at the clubhouse at 2 o'clock? Sounds great to me. Fine, see you then. Too bad Bill couldn't make it today. Sure is, he's a great guy. I almost landed with my gear up. I don't know why, but I can't seem to get it out of my mind. I know exactly how you feel all once. I'd been involved in several big tournaments and hadn't flown for several months. One afternoon after shooting a few landings, I decided to try some slow flight, so far so good. Then I started a power on stall. The stall broke, stronger than I expected, and I found myself in a steep power on spiral. I pulled the power off and started my recovery, or so I thought, as the plane leveled out and my airspeed dropped, I eased in on the throttle. But nothing happened. I gave it full throttle and still nothing happened. Man, those trees were coming up mighty fast. Back to the airport. I didn't have a clue as to what had gone wrong. I was shaken by my close call. I started my landing check. And then, just as I pulled on the carburetor heat, I realized what had gone wrong. I had forgotten the carburetor heat. The result? Carburetor icing, which had effectively killed my engine and nearly killed me. I had allowed my skills to become so rusty that my reactions were no longer instinctive. My procedures had become sloppy. You can be sure it won't happen again. Final approach. And we had no idea anything was wrong until we passed the end of the runway. When that gear horn sounded, I almost jumped out of my skin. I know exactly how you feel. When I was back in college, there was a certain Romeo I used to know. We were both from the same hometown, but he was a big man on campus and never paid much attention to me. Well, anyway, one day he invited me to go flying with him. Wait a second. Wait. I know what she's going to say. So let me tell you what the way it really happened. Now I know the problem where you started during my senior year. I was a big man then, all conference half-backs, for you enjoyed being the center of attraction. The next year, I was taking postgraduate work in engineering and spending most of my time in the library. It sure didn't take long for the roar of the crowd to fade away, and I became just another graduate student. One afternoon, Sally, a girl from my hometown, introduced me to a new transfer student named Lois. It was clear that Lois thought I was just another guy. During the weeks that followed, I tried to line up for date. No. No. I decided to try once more, but this time with something even Lois couldn't resist. My plan was to invite her to the homecoming game and dance in a very special way, one that would separate me from the rest of those fellows that were beating a path through her door. My idea was to personally drop the invitation right into Lois' lap. Sally came along just for the ride. When we were near Lois' house, I started my plan. My first pass was to see if she was home and to get her attention. I was in luck. She was outside by the pool. I came in more slowly, dropped the invitation, and banged a little more to get a better view. The stall warning caught me completely by surprise. Things happened so fast, I don't remember exactly what I did, but I'll never forget those trees. That little incident taught me one thing, never use an airplane to prop up a deflated ego.  But it wasn't until we were married for almost two years before I'd fly with him again. Andy, you never told me that before. I always thought you were a careful pilot. Bill, it only took once. You think I can take, Carl? How many nights can I go without sleep? And all I have to do is sit here and think about the bills that aren't paid. There goes Carl Stevens again. That woman never runs out of gas. But Carl did once. What about these things?  How do I know what you're doing? They had just had one of their routine arguments, and Carl couldn't get away with that argument. He was half an hour late getting to the airport. He was in no mood to do a proper checkup. It was the busiest time of the day. There were several planes ahead waiting to take off, and several more were in the pattern. Mad to start with, this added loss of time only made matters worse. What to do, and the activity kept him busy. So his personal problems were forgotten for the moment. But once the plane was on course, his personal problems came back again.  That's what you are. It took him a while. He didn't know how long to pull himself out of it. We've seen before takeoff. Quickly, he switched to the other fuel tank, and with wife troubles on his mind.


Life is karma. If you want to become no karma, there's no such thing. What do you think we have been doing with you? With inner engineering, you suddenly felt a new sense of life and freedom and things. It's happened to millions of people, I know this. All that happened is, we gave you little tools with which you stopped creating new karma and suddenly it felt so wonderful. There are many ways to look at this. How do I perform activity? Karma means... let us understand this. Karma does not mean just doing something in the world alone. You sit here, you're performing karma. Karma of sitting here. Body is functioning, it's breathing, taking in oxygen, leaving carbon dioxide. That tree is blessing you because you're letting enough carbon dioxide out. Suppose that also you stored in yourself and made a diamond so that you could give it to somebody. No, no carbon, no? Then the tree would look at you and say, what kind of a lout are you? Even carbon dioxide, you're not letting it out. That's one kind of karma, I'm telling you. So if you sit here and breathe, as far as the tree is concerned, you're doing a good karma. You don't know. You can be conscious too. Tree is giving me oxygen, I'm giving carbon dioxide. Does it mean you have to breathe hard? Nothing like that, you just breathe. It's anyway happening. So physical karma is happening. Mental karma is always going on for most human beings almost all the time. Emotional karma is going on, energy karma is going on, otherwise you wouldn't be alive. So karma is happening. It's not necessarily what you do. Consciously, it's just happening. Life is karma. If you want to become Nash karma, that means people think no karma. If you want to become no karma, there's no such thing. You can become very minimal karma. So he's a couch potato, Nash karma. No, no, lethargy is a tremendous amount of karma because it takes lot of work to be lethargic. In the sense, you will have to generate a lot of explanations for everything. And convince yourself that you don't have to do anything about it. Right now, there is virus going on, somebody is dying around you. You have to do something within yourself not to respond to that, otherwise it's natural for you to respond to it. So being a couch potato is not Nash karma, well a yogi can become almost like a hibernating state where everything is minimized, but still he's not free from karma. He may take less number of breaths per minute. You may be taking twelve to fifteen minutes, when somebody simply sits, they may just go down. Maybe three breaths, maybe six or eight, but below that you cannot go. You can even stop for some time, but it's only some time. If you stay without breath, not by holding, just by lowering the metabolism, then slowly a different level of karma will begin to pick up momentum in the body. When it picks up, anyway it'll breathe. Again it may go into breathless state, again it'll come back. So this way you can minimize, but still you're not free.  When you put something in, something has to go out, all these things are karmas. So what is the way to handle this? Well, what do you think we've been doing with you, with inner engineering? What do you think it is? It is simply this, many of you, almost all of you, when you did even the online inner engineering program, you've still not done any practice, or you went through the program first two days, no practice yet, they're still talking to you, suddenly after two days, you suddenly felt a new sense of life and freedom and things, it's happened to millions of people, I know this, suddenly you light up. Not because all your old karma went away, just those two days you did not create too much fresh karma, suddenly it's feeling so light and wonderful because you're not creating unnecessary burden upon yourself. So what was the trick? There are various aspects, let me not go into all that now. The simplest thing is this, your ability to respond. See whether I should do something or not, if I think about it. Right now, many people are struggling with this. Let us say somebody around somewhere close, fortunately it's not come to our panchayat area yet, we are managing it in a very tight way, no infections anywhere close by. So in Coimbatore there are some. So let's say somebody is infected there. Now should I do something or not? Huh? Why should I do? Other people will do. Who are those other people? We don't know, he must have a family, let them do.  But to respond in whichever way you can, not everybody can go and save everybody's lives. But at least do whatever you can, it's very simple.  Indiscriminate in response but not indiscriminate in action because you cannot afford to be indiscriminate in action. In action you are always discerning because action requires physical capability, energy, intelligence, means to do it, time to do it, many things are involved and these are all limited commodities. Nobody can do indiscriminate activity. If they do indiscriminate activity, they will die very soon and that is also bad karma. Yes, it is. Activity is not indiscriminate but our response is indiscriminate. Wherever it is, I would like to do something. If possible, everything that's possible, we'll do. For what is not possible, we at least have a little intention. Just because of that, you came to the program and you felt such a sense of freedom. People have been asking me, Sathguru, how will the online program work without the practice? I want you to know, before you came for the Shambhavi initiation, already you were feeling fantastic simply because you stopped creating fresh karma. Old karma did not go away in those three, four days. Just that new load is not happening. For the old karma, actually just living is enough. Sitting here, breathing, living, heart beating is good enough to start dissolving the old karma. A few things need a little more nudge, but most of the things will just go away. If you simply sit here, breathe, live, heart will beat by itself. You don't have to do anything or are you doing? Some of you think you are doing everything, the whole issue with people when it comes to their health and whatever, because everybody is an internet doctor now, you know? They've done lot of research.  Doctor knows something more profound because he spent nine, ten years studying and experience of treating people, but these people looked it up. After they came to know they have this particular ailment, last three days they looked up everything that's on the internet and now all kinds of questions, they go with their own prescription, please give me this drug, other drugs I will not take, whole lot of them coming to doctors and saying, I will not take this drug, I will take only this. This is supposed to be a very educated patient.  Yes, if you become a lifelong doctor, it's a good thing. You become a lifelong patient, very qualified patient, what is the point of that? This is simply because you think you are doing this, tuk, tuk, tuk, you have to do. You have to check how many beats are happening, everybody is wearing a band. Well, people ask me, Sadhguru, you must go through one master checkup, you must have this. I say, what? Every year I'm going to, I'm trekking in the Himalayas, I'm in Tibet, I'm going to Kailash. It's my stress test. If I don't make it, you bury me in Himalayas, that's all, otherwise you live.  Well, if you notice something is wrong, then you go to the doctor and maybe get little more qualified advice on that, that's different. But all the time being worried what is happening, what is happening, this itself is sickness. Too much concern about health itself is sickness and it's a terrible karma. This karma will have an enduring effect on you. So this is all that happened in the first three days of Inner Engineering, online or otherwise. All that happened is, we gave you little tools with which you stopped creating new karma and suddenly it felt so wonderful.  Little homework. What happened today, five things to write down. Just because of that homework, you paid attention to what you're doing and what you're not doing, suddenly tremendous sense of liberation.  Simply this, that your response is unbridled. You don't decide what is worth responding, what is not worth responding. It's not your business. A bird, a grasshopper, an ant, a human being, man, woman, child, doesn't matter. You respond. Do you have to act? No. Action is always a discerning action because there are quantities involved in action which are not unlimited. Our energy is not unlimited, our intelligence is not unlimited. Our actions in no way can be unlimited, they're always limited. But our ability to respond is limitless. This one thing, just this one thing, if you do, you will not create any karma today. You will not add up load, just not adding up new load, you will see it feel like liberation. That's the kind of damage you're doing to yourself.  Now you became like this, oh, virus Sadhguru, virus. Even before the damn virus came, most people were looking so grave, it looked like they walked out of it. Yes, this is because every day building up this rubbish endlessly, every day whatever happens, you build it up, build it up, build it up, build it up, something happens, something doesn't happen.   But now it's a lockdown, you don't have to pick up any garbage. That's a good thing about lockdown.   Hello? You have to put on all kinds of dice, they're all very well painted by themselves.  No. So then who are you to decide what to respond to, what not to respond to? Your ability to respond always has to be unbridled. If you're not getting what I'm saying, well, we can't come and teach you now. There is a inner engineering online. Just go through this three days. First three days, what is being said there, let it sink into you, soak into it, soak this in completely. You will see in three, four days' time you feel like you're floating simply because no new garbage, that's how it feels. So please make it happen.