During the coffee break, I was asked about more details about the string split thing, but I wanted to share that with all of you, just to make sure. We use string split a lot, and let's illustrate this. So let's define, I'm putting this in code snips and then uploading that, so just to show you. Define a vector which contains a single element, which is the three letters, a, b, c, divided by a column. And incidentally, when I assign something or have an expression like that, and I put parentheses around it, the effect is that whatever gets inside gets executed, but the whole result also gets printed or is available. So this is basically a shorthand of saying, assign that to x, and then show me the contents of x. It's basically done in one line. So if you find things with parentheses around them on a line in a script, that's what that means. Okay, so I have a vector, it contains one element, a, b, c, scalar values, a single value is assigned to some r object. In fact, r also vectors of length one. So this is the vector a, b, c. Now if I do string split x, as we've seen before, we split on the colon, we get a vector of three elements a, b, c, and that vector is contained in a list, which contains one element, which is that vector. Now if I pull that out by using the, the sub-setting operator with double brackets, which gives me a single item that I can, I can apply to a list, then I get only the vector and that's normally what I would want to be working with. I get a similar result by saying unlist, because the unlist function takes everything that it finds in the list and puts it into a single vector. So that looks exactly the same. In this case, string split one and unlist the whole thing are exactly identical. But now let's consider the case that we don't have a vector of one string, but we have a vector of two strings, or potentially more strings. And the elements in here have different lengths. So now we have a vector, the first element is a, b, c separated by colons, the second element is d, e separated by colons. And if we string split that, we now get a list of two elements. The first one has the a, b, c vector and the second one has the d, e vector. And now it matters whether we use the sub-setting extraction or the unlist, because the unset, the sub-setting extraction applies only to the first list element of course. But the unlist applies to the whole thing, and I get a, b, c, d, e. So depending on what the input potential can be and what we want the output to be, either one or the other or both are possible. So that's the difference between the sub-setting extractor and the unlist. All right. So we're looking at how to get this into, as data, we used, we just put the whole thing in the whole text that we had into a single character element, and we split it on the line break character, and then we could either unlist or unsplit it and we assigned this to the correct, to the correct value. And now, most frequently though, what you will encounter is data that is large and structured in some way, and for that, we will be, we would be reading it with some variant of R's inbuilt read functions. The most frequently used ones are the three read lines, read CSV, and read delim. So read lines produces a vector of lines from a text file, one line at a time. So in our case, that's exactly what we wanted. So this reads the file, every single line goes into its own element, and the result is a vector, so we can assign that. We could also use the read CSV function, or the read delimited function, it doesn't matter because we only have one element per line here, so whether we tell this, whether we consider this to be tap separated or comma separated, it doesn't matter because there's only one element. So that works in almost the same way, looks quite similar, except that now we don't have a vector, but we have a data frame instead. In this case, the data frame is a data frame with a single column, since we had no header in our file, header is false, are automatically generated a header, a column name here, which is V1, it was read in rows, and we have the same data. Now if we have that, this is a data frame. So how do we change a data frame into a single vector? Because the specification called for a single vector. Right, we extract the columns. How do we do that? We do literally just column R genes, so we could either just use the column V1 because it's called that, which is the vector we want, or find all rows and the column named V1, which is the same thing. But in this special case, we can actually also use unlist, I think, and we get all the data from the data frame in a single vector, and depending on what that data is, that will determine what the type of the vector is. So all of these three, either car genes, column V1, or car genes, V1, or unlist car genes would do the same thing. I could even have used read CSV and then just applied the dollar V1 directly to the function call. I don't need to use an intermediate assignment to a variable. More often than not, however, our data is going to be two dimensional anyway, rows and columns of different types, and we'll just put it into a spreadsheet to begin with. All right. Good, so we've talked about how to get very, very simple data into R. How many genes do we have? How do we know? Right, so here in the environment, we see 46 observations of one variable. But it's actually still in the spreadsheet version, but we can just change that back to a vector. Right, so it's a character vector of 46 elements. If we wouldn't be looking at that, how do we find it programmatically? Length. The length command gives us the number of elements in a vector, which is something we often need because the length command is also the index for the last element. The index of the first element, of course, is one. The index of the last element is length of the vector. Okay, here's a task. Now on the science website, as on any other paper websites, there's a section of the article called supplementary materials. And if we go to the supplementary materials, this lists tables and usually additional methodological documentation, methodological methodological documentation, and hopefully other data. In this case, table S3 summarizes data that was underlined, one of the plots, I need to remember which plot. Anyway, so this is one of the plots that they produced. The data contains expression profiles of different cell types under lipopolysaccharide stimulated and control conditions. So these are differential expression data from single cells. So if we find something like that on a website, we click on the table, it's an Excel spreadsheet. Click on it. It gets downloaded to our download folder. We don't need to do that because I've already put it into the data folder, table S3.xls. And first task is to open that and try to understand what's in that table and what we would possibly need to do to read the data into R. And the second task, then, is to actually do this, to actually read the supplementary data. Now it's very common to share data in Excel spreadsheet formats. A word on that. Excel is an excellent spreadsheet program. It's really, really, really, really bad for statistics. Do not use Excel for statistics.   It also is often wrong. It actually has its own idea of statistics that is not quite canonical. There's two links. I hope they still work. Practical statistics on Excel and burnstatistics.com that kind of comment on why it's a poor idea to use excellent statistics. So by all means, use it to share your data, keep your data. As long as the data doesn't get too large, it's often quite useful. Don't use it for programming. Excel programs are usually messy, and do not use it for statistics. You're learning to use R, and that's the right way to do it with a capital R and a capital W. Now, you see, this is one of the things. When you export things as you should from Excel into something like comma-separated values or tab-separated values, you even need to be cautious because one of the problems with Excel is that it truncates numeric precision.  You don't actually need them. They're like, you know, overkill. Why do you need seven digits? Come on. Well, sometimes you do, and whether I do or I do not, I do not want Excel to make that decision.  That will export the actual values that are in the cells and not some interpretation of what the value is in Excel. In principle, there's a package, Excel S ReadWrite, which is available via CRAN, but I'm not actually sure that this is a sound practice. The Excel, the default Excel format is closed. People have reverse engineered it to a fashion to understand what's in these files. There's no guarantee that that's actually correct. And on top of that, Excel spreadsheets can be complicated. They have multiple sheets. Their values can be explicit. They can be implicitly calculated from formulas. You can mix various tables on the same page. So actually using a function like Excel S Write is calling for trouble. What you want to do is you want to make sure that you see what you are exporting and export that into a plain text, comma-separated value or tab-separated value format. Make things explicit. I can't overstate that. It's really, really important to keep explicitly track of what your data is doing, validate every single step, and exporting from Excel is no exemption. So what we do is we open our Excel spreadsheet and then we export this as a .csv file, a comma-separated value file, and then we'll read it into R. So your task is to open the data table like you would usually download it from the supplementary materials on a paper website and then save it in a comma-separated value format. And when you have that file, put up a blue post it, and if you're stuck on that, put up a red post it. Yes, that one is to read Excel and save it as a .csv, so you're stuck here or you don't have to. Yeah, one is the answer. You're going to have to express that. No. No, yeah. Okay. Yeah, okay. Yeah, unfortunately. Did I save my file? No. You just leave me to it. That's the answer. Thank you. No, I just put it there so it's not good because he already has that file in his tutorial. But this is totally up to you. Good. How are you? I'm glad to hear you're at the OACR. Yeah. That's awesome. I can't believe I haven't seen you around like we're so close now. Yeah. Are you ever like doing movies? Yeah, yeah, yeah. Yeah, I'm often here. Whose lab are you in? So I see only a very small number of blue post-its. I hope that doesn't mean it's difficult. This is the table when I read it in Excel. It's kind of large-ish. It has a lot of data. It has a few header lines. File. Save as. Format. Many options here. I want comma-separated values, CSV. The alternative tab-delimited text. That's something we usually call a TSV file. However, if you get Excel to write tab-delimited text for some reason, it insists on calling it TXT and not TSV. And if you call it TSV instead, it adds .txt to your TSV extension anyway. We shouldn't be getting too frisky in saying things apparently. Anyway, so CSV actually generates CSV. So I choose Table 3.csv. I click on Save. It complains again this workbook contains features that will not work or maybe removed. Don't do this. Maybe it may hurt your little brain. No, no, no, no. Continue. And then it finally saves that. Incidentally, it now calls this file, which I've read before as an Excel file by that name because I've saved it. However, if I simply press Save, I believe it changes it back into an Excel format anyway. So Excel does lots of things that I'm not explicitly asking it to do. So I'll throw this away. I just call it differently. So there it is. What's inside? Always look what's in your data before you work with it. It's a text file. It has information at the top. It's comma separated. So we should be able to read it with an R function called read.csv. And what I'd like you to do, your task is to read the table into R, assign it to a variable, and there's a few caveats. Use the read.csv function. After you read it in, use the head function to look at the beginning of the object to see what you have and whether that's what you want. There may be header rows that you don't want. There may be odd names in the headers that you don't want. So give the resulting object column names that reflect the cell type, just like in Figure 2c of the paper. So here we have different cell types. Cells, macrophages, natural killer cells, monocytes. PDCs are what dendritic cells? Poly... Plasma cytoid. Plasma cytoid. Oh, plasma cytoid dendritic cells. The abbreviations that the paper uses are BMF, macrophages, NK, for natural killers, MO, for monocytes, PDC, DC1 and DC2. So it's a good idea to use these column names, but as the header possibly explains, some of these are controls, and some of these are lipopolysaccharide stimulated. So distinguish CTRL and LPS conditions in your column names. And call the last column cluster because they've clustered the cells and the numbers in the last column correspond to the cluster identifiers. There's a function here, which is loaded. It's called object info, which takes a few of the data items we can retrieve for an object and prints them out to the screen. So once you've read it in or in between, you can use object info, whatever your object name is, to find how many rows you have, how many columns you have, what the column names are, and so on and so on. What the column types are. Do you have strings? Do you have factors? Are they numeric? Which ones are numeric and so on? And validate that. Yeah, what I omitted to say is, I should add that in here, call the final object LPSDAT. Do call it LPSDAT because we'll be referring to that variable name in later parts of the script often. You're of course free to call it anything you want, but then you will need to edit a lot of the lines in the script in order to make them actually work. Call them LPSDAT. That's what's expected here. So again, the task is read your CSV file and make sure that what you get as a result of your read operation is actually something that's useful to work with the genes in that file, the expression values, and with appropriate column names. When you're done with that, put up a blue poster. If you run into trouble and you can't continue, then put up a red poster. It's, yeah, it might be... So gives you a table, so it's a different kind of object. It's not what shakes our object. You don't need it. You work with it the same. It's all the same. It just gives you extra information. Data is size. There's no seconds. It's speed change, so I don't know if it's going to be that important. Yeah. Well, no. Oh. You need to put in some error. Yes, I've said the code error will be used to read the code. Yes, look. Okay, so, I've said this. It's like you end up doing... It just hits a number. I just use, like, a double scroll button. Yeah, like, you just know it. No, don't use that. You just have to write it all out. It's the same. And you're like, I understand the background. I was just talking about this. Yes, yes. So it's the same thing. Okay. Let's look.  It doesn't matter what we're doing. Yeah. Some of them aren't named because of the way it's split the... all of the characters. Like this. It doesn't have one header in here. That one really is called... It's like, you know what I would do. I would just fix it all over the cell and then read it in because I don't know what it has ever been so much easier. But I don't care. Just get that line. Because that's not the way it's going to be. Yeah. So please, may I have one? I don't have the same choice. OK. I don't have a right for this. But we have to sort of have some comment here for a character. So it means you have a character string as the argument for that. Or a white line. I don't have a character string. I didn't have a character string. Now all of your lines are the same. OK. Because the extension can decide. All right. Bees. OK. I got a better one. I think that's職is. Yeah. So I correct the職is. And I see that's true for that. Actual. Perfect. I was at the part of where it's really unique. So have you got either the header or the basic part? Probably going to be different. Which one? The text. So I would assume each one of these would go in, except for the same type, right? But when I put head and table S, it would come in one. So let's discuss a little bit of what's been happening here. So one thing that you probably noticed is trying to read in this header information in a sensible way doesn't really work. It's extremely hard to somehow try to recover usable information from a header that's built like this. This was not made to be programmatically used. This was someone's idea of how to fluff it up for visualization. So what I would do as a strategy is I would just skip reading these first six lines. There's a skip parameter in read.csv. Start reading line seven. Tell read.csv that this file actually does not contain a header. Otherwise line seven, i.e. your first data line will be interpreted as column headers and lost from your data. So skip the first six. Start reading line seven. Tell it that columns, that this has no header. And then after it's read in, simply give it a vector and you'll probably just need to write that out by hand, what the elements are, and assign that to call names of your data frame. So then were you assigned row names? Yeah. Well, we'll get to that in a moment. Maybe I'll just walk through the sample solution here. So if we take the whole thing and assign that to some raw data, header is false and strings is factors is false. Many of you have omitted or forgotten or not realized that you really have to say strings is factors is false and something like this here, otherwise we get factors all over the place. In particular, if you're reading the whole thing, the first elements are text elements anyway. So this means that the entire numeric data column is also converted into factors and we need to recover from that. Anyway, so this gets messy. Just tell it strings is factors is false and then we'll clean it up later. If we look into that, this is where things go wrong. R automatically assigns V1, V2, V3, or I've seen X1, X2, X3 apparently, cell type cell treatment. This is where our actual data starts. So all columns are named V something or X something, rows 1 to 6 do not contain data. There's not a single row that could be used as column names and all columns are characters. So this all needs to be fixed. The first thing we do is we drop the first six rows in that case. What I'm illustrating now is how do I recover from a sloppy read? We can read this in a more intelligent way, which saves some of that work, but how do we recover here? So then we just drop the first six line with a negative range. Remember that for subsetting negative indices, remove items from the output. Be very careful. The parentheses here are absolutely necessary. So what I have here is 1, 2, 6, which is minus 1, minus 2, minus 3, minus 6. If I don't use the parentheses, what does this give me? It's this here, minus 1, 0, 1, 2, 3, 4, 5, 6. That's not intended. Be careful. In this case, the parentheses are really, really important. It's one of the cases of operator precedence. The minus has higher precedence than the expansion, the colon character. So the minus applies to one before everything is turned into a range. If you, like me, cannot, for saving your life, remember operator precedence, then just put in parentheses everywhere where there's potentially a conflict. If you don't want to remember that operators of equal priority are evaluated from left to right, or you don't want to remember whether exponentiation is evaluated before multiplication or after that, that kind of thing, just put things into parentheses. Having parentheses never hurts. I think I can't remember a single case where it would cause problems. So we remove that. Let's see what we got here. So we start with our row seven, but things are still messy. Then we define column names. As I said, now here we just have to do this by hand. If we're very savvy about how to use the paste function, we could make our lives easier and paste that together in a way where we don't actually have to spell it out line by line. But this is very explicit, and if we put this into our scripts, it has the advantage that we can also comment what we actually have here. So gene names, cell types are taken from figure four. Hygiene and all cell types are B from macrophages. Control and LPS refer to control and LPS challenge. Okay, so this defines my column names. And my column names now are here. B dot control, B dot LPS, MF dot control, MF dot LPS. And so on. And now there's one thing about this. Since I have deleted the first six lines, note what the row name of my first line is. Seven. So the row names were generated when the file was read. By default, if there's nothing else specified, the row name is the same as the index of the row. Almost the same because an index is a number, but a row name is a string. So these look like numbers, but they're actually the number seven, the letter seven, the letter eight, the letter nine, the letter one, zero, and so on. So these are strings. And when I delete things from my data set and I subset things, the row names don't get changed. So now my row names are from seven to whatever. Now that's very surprising. If I want the first row and I use a row name of one, that doesn't actually exist. There is no row name one now. The row name is seven. So to keep the row names which will be printed out when I do a print statement of some columns, to keep them aligned with the row indices, I re-index the rows. This is why I set row names LPSDAT to one, the range from one to n row of LPSDAT. And I do that one, two, three, four, five, six, seven, as it ought to be. Now a residual problem here is that all of these things are still characters and I need to change them into numbers. I can't just go and do that with one expression, at least not with an expression that I would be discussing here. You could do it with an apply statement. So what I do is I iterate over the second to the last column and convert these numbers to numeric values. So LPSDAT of the column I is as numeric of what that string was. So if I look at the beginning now, that kind of looks correct. I have genes. I have values of cells. I have rows of the correct name. And I have, if I str, I see that the first column is a character column of gene names and all the remaining columns are numbers, not factors. So remember, by default if we don't turn this off, the read functions, the default R read functions turn strings into factors. So we always need to use strings to factors as false. Let's talk a moment about factors. So why factors in data frames? What are factors in the first place? When are they useful? Why do we have them? Now factors are special types. Normally there are strings like male-female or agree-neutral-disagree or something like that. But underline, they are coded as integers that identify the levels of the factors. So for example, if we have genders, factor MFFMF, probably a very 1950s view of what genders could be, as a factor. This tells me I have the values MFFMF and two levels, F and M. Now internally this is coded as 2-1-1-2-1. And the 2-1-1-2-1 refer to the levels. So second level, first level, first level, second level, first level. Note that the order of the levels doesn't have anything, doesn't necessarily have anything to do with the order in which they originally appeared in the vector. It may be alphabetic, but it could also be something else. Now if we look in more detail, the structure of this is five numbers and two levels. Especially if the levels are long information, this can be useful and compress the data a little bit. We can also define ordered factors in some way. And if we have something like agree, neutral, disagree, these are not random in the ordering, but agree is stronger than neutral, is stronger than disagree or ordered in some way. And if we want to use the data in that way, we set the parameter ordered equals true when we define them. And we can define levels. So in this example here, sample grades, I have a grade of G1, G3 and so on. I can define levels G1, G2, G3, G4, and I can tell this is ordered. Now my sample grades are what I define it to be. And note the level string now. G1 is less than G2, is less than G3, is less than G4. This is how you identify that this is an ordered factor. And when you use factors, for example, in regression analysis to regress over factors, i.e. see whether your values correlate with some categorical variable that you have coded as factors, then it's important to keep this order intact for the regression analysis. Can you give us an example of what that might look like in data? Like why you would have... Well, like this here, we have quality grades in some kind of analysis. My pathologist tells me that the samples we have somewhere decayed pretty badly and some are very fresh, and then we could perhaps order them. Or two more classification might be one example here. The more severe the diagnosis, the worse the prognosis is in a tumor, you might order the factors along this way. But depending on what you want to do, you could order them this way or that way. It's free for how you define it. So they're useful because they also support a number of analysis methods such as keeping box plots in order or calculating regressions. I've linked to a factor tutorial and a discussion on their use. One is by Jenny Bryan, who also teaches... Jenny also teaches for us, right? Or at times. So she's quite active in... She's in Vancouver and quite active in our community. Now, for our purposes, and I think everything that we're going to be discussing in this workshop, the default behavior of R to treat all strings as factors is just unwanted and needs to be turned off. So when we read something, we always turn something off. And why this happens? I'd really like to illustrate this again. So if I have a data frame that I get from an Excel spreadsheet and the first value was not available, so in my Excel spreadsheet, somebody has typed n slash a, not available. And then the rest is numbers, and I read that into a data frame. So wherever this says type info, I just forgot to change the name. It's now called object info, so just replace that or ignore it. Now, this is a data frame. It has a factor of six levels. So it doesn't have numbers. It's a factor because vectors all need to be of the same type. And so everything in here was converted into strings to match this string here, and therefore we got a factor. Incidentally, that's a common problem with reading CSV files. If someone somewhere in your spreadsheet just put in a single blank character, that's very hard to catch, but it will turn your entire column into strings, not numbers. So once again, whenever you read something in, make sure that you have the right data type. Okay. So can we turn them back into numbers? Well, for example, I just remove the first na, and now they're all like 1, 1, 2, 3, 5, 8, and I just turn them into numeric by saying my data frame 2 is as numeric my data frame 1, and see what that gives me. Is that correct? Is that the right result? My data frame 2, it looks correct, except if you look more closely than as Francesca noticed, there's no 8. There's actually no 4 either. In the original data frame, there was no 4. So instead of 1, 1, 2, 3, 5, 8, we get 1, 1, 2, 3, 4, 5, which only looks quite similar, but would give us an entirely wrong result where we actually to interpret these as Fibonacci numbers. So what just happened? We turned these into numeric, but that just means we're taking the internal representation of the factors and using these as numbers, and that gives us the wrong result. So if our numbers inadvertently get converted into factors, we can go horribly wrong when we then go and turn that back into numbers. That's one of the most dangerous, and in this case, really hard to spot. Congratulations. It's really, really hard to see. It's the kind of subtle error that can creep in, which can really ruin your day. So what you need to do instead is, at first, you need to turn it into characters. This gives me the correct characters. So my DF internally as factors are indeed the numbers 1, 1, 2, 3, 4, 5, but these are the levels of the factors, not the actual contents of the factors. So if I turn them into character and then turn them into numeric, and then I look at them, I get the 1, 1, 2, 3, 5, 8. So if you ever need to convert something from factor back to what it should be, turn it to character first. Once you've turned it into characters, as character, whatever the factor is, you can process it further. Either you wanted strings to begin with, or you wanted numbers, or you wanted logicals or whatever, and then cast them to something else. Okay, but then at the end of the day, after everything is said and done, we have the LPSDAT object. We have genes as characters, expressions as numbers, and clusters as integer numbers. Now if your LPSDAT object looks exactly the same, that's good. If it doesn't, you can load the one that I have in the data file. So R has a very efficient mechanism to load and save intermediate data.  They are compressed. So if it's a large object of, say, textual data, you will be surprised it gets to be a really small and compact binary file on your computer. And to load it back, you can simply say something like loadDataLPS.RData. Note that we are not specifying the object name. The object name is whatever was saved in our R data file. We're not assigning this. We are reconstructing something that we've taken out of the workspace. This would only be necessary if we had done the removed... Because right now it is active in our workspace, so we would not do this right now. Right. But if I would have removed LPSDAT and then it doesn't exist anymore, I can simply recreate it by loading my loaded copy, which should look exactly the same. Because that's how I solved it. If your local version of LPSDAT looks substantially different, you might want to do this anyway. Simply so we all have the same file that we'll be working with that will help us avoid some surprises. Now to work with this, something we always and often and frequently need is sub-setting. Is there any volunteer from yesterday and the day before who would like to teach the next 10 minutes on sub-setting? Anybody who's really spent a lot of time on the pre-work tutorial on the sub-setting part that... I'm not surprised. So if this is repetitive, this is good. You really need to internalize sub-setting. It's really, really crucially important. Any kind of data analysis, exploratory data analysis, starts with reading in data and then sub-setting parts of data and comparing data as needed. So let's reconsider the six different ways of sub-setting. And what I do here is I'll... I'll just start with a synthetic dataset, randomly pick something and let's see what we get here. I give things a name. It has a number of legs. Depending on how many legs it has, it can be a fish, a spider, a beast, a crab or a centipede, and some values that I've measured on my objects here. Now, the first thing of sub-setting, the simplest thing of sub-setting is sub-setting by index. So this is a two-dimensional object. Sub-setting by index means I specify rows and columns in square brackets. If I specifically specify one particular row, one particular column, I get one element, spider. If I leave the rows empty, this means all of my rows, this incidentally is the same thing as if I would have said that dollar type. If I keep the first thing empty, I get all rows. If I keep the second thing empty, I get all columns. If we want a particular set of rows and columns, we pass a vector of positive integers. So, for example, we can say dat C23 and C123. This is for rows two and three columns one to three. Now, any function that returns a vector of integers can be used. Most frequently, we use the range operator and retrieving ranges of rows and columns from a matrix or data frame. Sometimes it's also called slicing, a slice of a matrix. So, dat one to four to one to three works. But it can also be in reverse order. So, if we specify four to one, one to three, we get the same thing, but ordered the other way around. We can subset only even rows or only odd rows. Or we can subset only every 100th row or a random sample of 100 rows. So, for example, if we have a data file of protein-protein interactions that we download from the strings database and we have something like 500,000 rows in that, and we want to develop some analysis working with the 500,000 rows might not be very efficient. So, we might subsample it to take only every 1,000th row, which leaves us with 500 data items, essentially randomly sampled, which we can then use to develop our analysis before we apply it to the entire data set. So, especially if your data sets are large, subsampling a small subset randomly will be useful for analysis. Sometimes if the data is really, really large and your analysis is really, really expensive, you actually have to do that. That's the only way to analyze your data. Actually working with a random sample from your data by subsetting. Now, the indices don't have to be unique or in any particular order. So, we can repeat things. So, this here repeats the first row three times, the second row two times, and the third row one times. So, it doesn't have to be unique. We just think of it as being, you know, slices usually because that's what we most often do, but any ordering is fine. In particular, we can select random subsets with the sample function. So, sample 1 to n of three elements gives us 6, 3, 4, or 1, 5, 4, or 4, 2, 5, or whatever. And we can do random subsets. Or we can sort the data frame. Now, sorting the data frame works in a somewhat different way than you would expect. You can't just sort the data frame. If you sort a particular set of values, for example, the values in column two, you get the sorted numbers. But what we need to do in order to get the entire data frame in the order that we want, we don't sort the entire data frame, but we look for the order in which we should be presenting our data. That is not the sort function, but the order function. Now, if we look at the second column, it's natively in the order 0, 8, 4, 0, 8, 8, 2, 0, 10, 4. If we apply order to that, we get a vector of 1, 4, 8, 7, 3, 10, 2, 5, 6, 9. Now, what does that mean? Well, these are the indices of these numbers in which we would need to consider them to get them into a sorted order. So 1 means 0. 4 means the fourth element, 1, 2, 3, 4, which is also 0. 8 is the eighth element, which is also 0. So this is how we get our 0, 0, 0 in the ordering. So maybe, let's write this slightly smaller example here to illustrate this. So if we have a vector 8, 13, 5, and we sort that, we get 5, 8, 13. If we order that, we get 3, 1, 2. So this 3, 1, 2 is the indices of our original vector to get them in sorted order. 3, 1, 2, 5, 8, 13. So in order to apply that, the result of ordering to our data frame, is again just sub-setting. Order creates an index vector, and we use an index vector to pull out the rows from our data frame in the correct order. So our data frame, sorted by number of legs, which is column 2, puts this vector into the rows, and then just columns 1, 2, 3. So now we have them ordered by number of legs. Three fishes, one bird, two beasts, three spiders, and a crab. We could also order them by lexical order of the names, and that would be ordering by column 1, C, I, J, R, S, S, T, U, W, and X. So ordering is very versatile. If you look at an ordered vector, though, it always takes me a little while to figure out what's going on. Let's try to be familiar with that. If you specify a negative index, that element is excluded. So dat minus 1 means the whole thing without the first row, dat minus n, not the last row, and so on. So that's one thing. Positive and negative indices. The second principle of sub-setting is logical vectors. So instead of indices, we specify sets of rows or columns by Boolean values, true or false. And if we place a vector of logicals into the square brackets, only the rows or the columns for which the expression is true are returned. So something like this vector here, rows 1, 2, 3, but true only for column 1 and for column 2 and for everything else, gives me the name and the type of my collective specimen. And you need to take care that the number of elements in your logical vector is exactly the same as the number of rows, respectively, columns. Now these logical expressions can be combined with the AND and the OR operator. And I'm skipping ahead a little bit. So sub-setting by index, sub-setting by logical expressions. How about filtering? Filtering means selecting from a data set depending on what the values are. So if we get all values that are greater than 100, we're filtering our data set. If we filter by string matching expressions, we can use the grep or the IN operator to subset via string matching. For example, grep, everything in the third column, i.e. the type that contains an R, tells me types with an R are found in rows 2, 5, 6, 7, and 9. So what are these? Well, I take the result of that, grep, R, dat, 1, 2, 3. Look at the first three columns where we see these are spiders, birds, and crabs, all with Rs. Types that begin with C. This is the regular expression. Special character here at the beginning of a string, a C, are crabs. I also define centipedes, but no centipedes happen to end up in my collection. Let's see. If we use the IN operator, I'll review with you how that works and how it does. Then, we have indices, we have logicals, we have filtering by some kind of logical expression, either grep for string matching or other logical expressions. Then we have subsetting by name. If row names and column names have been defined, we can use these for subsetting. So, for example, I can say rows 1 to 5 of the column called name, or rows 1 to 5 of the column called name and the column called legs. In that case, I pass a vector with the row names or the column names. So vectors of row names and column names also work. And finally, the dollar operator. I often use the dollar operator whenever I want to extract single columns from data frames. So in this case, if we want to extract the column named legs, I would apply that dollar legs, 1, 2, 3. So usually in my code, I use dollar operators when I want to indicate single individual columns from data frames. Okay, now we have our object LPSDAT. Let's do something more real-worldly. And I'd like, as a task, you to write our expressions that get the following data from LPSDAT. Subsetting expressions. The first one is rows 1 to 10 of the first two columns in reverse order. The second one, is expressions and expression values for LPS stimulated monocytes in the column mo.lps for the top 10 expression values. The top 10 only. Third one, all genes for which B cells are stimulated by LPS by more than two log units. The next one is expression values for all genes whose gene names appear in figure 3B in our data set. Remember, that is car genes. So these strings are in car genes. You need to figure out how to subset the LPS data by the genes in here. Hint, this is where you use the in operator. And that's it. So these four tasks. In six minutes, we'll be breaking for lunch. You have five minutes to do this. No, actually, it's four tasks, so four minutes to do. Try to get as far as you can. You should not take more than 20 seconds for the first one. The other ones are a little more involved, but also a little more real-worldly. Now, note one thing. And this is, after all, not just an introduction to our workshop. This is exploratory data analysis. So with these operations, we're doing our first exploring data tasks. Specifically, finding the top 10 expression values from a large data set. So this is the kind of thing that you would do when you actually explore your data. And we'll get to something else very soon. So we're already imposing on our lunch break. Well, if you feel compelled and doable, if you feel obliged, no, no such thing, you need a break.  So I'll see you back after lunch.


Hello, community matters here at noon on a given Monday and today we're excited in community matters to be reporting on the March for Science which is taking place this Saturday on the street all over, all over UH and all over other places too and our guests for this segment part two of this discussion on the March for Science to my left we have Mark Hickson where he is the Shire Endowed Professor of Marine Biology at the Department of Biology at UH Manoa. We have Priscilla Seaborn she's a master of science and a PhD student at the Department of Entomology at UH Manoa and Dr. Grant Yamashita he is an evolutionary biologist and consultant and he has assured us that if we have any questions he will consult with us about evolutionary biology. The problem is it takes 10,000 years. So you guys are all involved in planning and executing this March on April 22nd this Saturday. Talk about it. What have you done? What do you expect? Well this whole thing arose since the election and there's been great concern in the scientific community that's been growing for decades. So this was sort of the you know the spark that lit the flame for people to start self-organizing and a group of scientists and concerned citizens got together and said what can we do? In Washington DC they came up with the idea of a National March for Science and from that spraying almost 500 other marches and clean Honolulu and other islands and something like 40 nations now. I think it's over 50 now. It's outside the US. It's just a global statement about science. Yeah and it's you know it's the motivation is several fold one is just it's time to celebrate science. There's been so many amazing discoveries being made and and you know an actual natural national celebration for science would be a good idea. But then it's also in reaction to the growing denial and abuse of science that's been taking place again for decades but now seeming to reach a crescendo. Abuse is an interesting word. People abuse the truth. They come up with all science if you will. But that's an abuse to all of human knowledge isn't it? Absolutely. Well I mean I'm fascinated with this because I your comment especially about how it's been growing for a while it's not just the election of Donald Trump. No. It's been growing for a while there's been this sort of this separation of you know public thinking public opinion not only in this country but elsewhere and science. Why has that happened Mark? Why has that happened? Well my understanding is that most people are just fine with science and respect science respect scientists until there's a finding that conflicts with their worldview or conflicts with the dogma that they've been taught and then suddenly there's cognitive dissonance and it's typically easier to deny the science or attack the scientists than it is to change one's worldview because that's always a big change inside one's mind. It's like the bubble theory. You live in yours, I live in mine, never the twain we'll meet. Yeah and the thing about science is science being relatively objective and dispassionate will produce results that don't make anybody happy you know when we find that there's an environmental issue or that a chemical causes cancer or whatever it is you know we just have to report it that's what the science says and people are like I don't want to hear that especially if they're somebody who's producing that toxin or that problem. Well who will be affected by it or who would like to deny that they'll be affected by it. Absolutely. Yeah. Yeah, reality. So Priscilla you know what can you can you tell us the website by the way there's a hint on this March4scienceHawaii.com so can you tell us the website? What do you mean like what the website is? Well yeah there's a Facebook page, there's different ways to access it. I'm actually on the Facebook page because as you know a younger I guess student that's how I get you know most of my social media and it's I mean social media you know it can be a really good resource for things like this and that's actually how I learned about the March4science initially and then I found out my PI or principal investigator was also the organizer for it so I thought who is that? Dr. Helensbaffert. Okay we saw her last hour yeah. Yeah yeah so it's on Facebook they do have a website so. Would that be March4scienceHawaii.com? That's a good question. March4scienceHawaii.com. Where does March4scienceHawaii.com? Good to know. And what's on that website? There's gonna be information about the the different events like I said most of my information I got from Facebook and so that one actually has you know information about the event what it's about so the description and you can also RSVP to the event so it's a good way to kind of keep track of who else is gonna be there. I've actually been working on the website so I can tell you a little bit about what's actually on it. Today the information packet about the website just went online so you can download a 14 page information packet that will tell you everything from the the route of the March where all the parking is all the transportation options that you have who the speakers will be what activities there will be and a nice little coloring sheet at the end as well. A coloring sheet. A coloring sheet. Isn't every website needs a coloring sheet. So that's available today but we have information about our mission and principles things that we want to go on post-march we're calling it after the March and just information about the activities who all our endorsers and sponsors are just to keep everybody up to date about what's going on. You have a ton of endorsers and sponsors I've seen the list and we don't have enough time on this show to go through it but it's just suffice to say there are an awful lot of people organizations in this community that are supporting this march it's a well popular. So one thing we didn't get a chance to cover in the previous part one this is part two last hour is the Neighbor Islands there are marches sympathetic marches on the Neighbor Islands can you talk about it? Yes so right now there's officially there's a march in Kahului on Maui there's one in Lihue in Kauai and there's one in Hilo. The Hilo one is especially interesting because originally they were going to march on Saturday like everyone else but the Mary Monarch Festival is also there that weekend. Priorities. Right so instead they decided to move it to Friday evening the 21st and so we think that they'll actually be the first march around the world the first march for science march around the world and then they'll also be marching in the Mary Monarch Festival the next day and since our march will be in the evening on the 22nd we think we'll have the last march around the world so Hawaii will potentially be the first and last marches this coming weekend. I hope we get national coverage I think we will get national coverage but you said something about the world earlier Mark and this is very provocative you know that the world has this separation you spoke of that's not just Hawaii which I don't think Hawaii is very separate from science I think we are very respectful of science and understand it better than other places not not just the US but the world is having this separation and this therefore this march is as you say granted it's a global march it's in how many countries I think over 50 now 50 countries but what what what is happening here this is fairly remarkable we've had science since since the days of the caves you know the symbols on the walls of the caves were early science those guys figured out what they needed to figure out to survive and that was in its own way science somebody once said that domesticating a dog a wolf to be a dog that was science absolutely it was a tool absolutely to do things that people couldn't do but you know where have people shoved off on this why is this a global issue well I want to add also that the native Hawaiians were amazing scientists you know their knowledge of astronomy and natural ecosystems was just absolutely unprecedented for almost any culture so science has been with us since we've been humans there's no question about it there just seems to be this growing disconnect between what science is telling us and what people want to believe in their hearts you know you can think back all the way to Galileo when suddenly the earth was no longer the center of the universe and he had to run in with the church over that you know but it's that kind of thing but it just keeps growing but in the time of Galileo I think people were were amazed right with Galileo Da Vinci they were amazed that he came up with this stuff they were very respectful somewhere in recent years it's changed is that the media is it science itself has science failed to do the job it should have done in terms of socializing its work to the public scientists on average are not very good at communicating what they do except here except here right now of course all of us but yeah I mean there's that's always been a historical thing scientists tend to like just be focused on their science and that's all they want to do but more and more scientists are stepping up to communicate science efficiently to the public but in my experience having done this for decades now it doesn't matter how articulate I am or how accurate the science is there are people where they just won't let it come in a wall goes up because they seem to believe that their worldview is being violated or they've got a false dichotomy in their mind very typically you can have a healthy environment or you can have a healthy economy but you can't have both and so if anyone starts talking about environmental issues it's like forget it I've got a few more. When I went to school which was in well I hate to tell you what decade we had in school say the 50s and 60s okay we overlapped everybody he has more hair anyway everybody loved science everybody even the people in the social sciences loved science you know and it was you know it was it was it was going to make our lives better it was very clear and somewhere along the line things change and I'm interested in your comments especially the millennials among us if you will yeah yeah what happened to education here I feel like for me when I first started with my master's degree part of the learning experience was being able to even just read scientific papers you know they're kind of littered with scientific jargon and you know it took me one time four hours to read a paper because I was on google like trying to you know look up the words and what they even mean and just trying to put together and so I think in that sense you know we as scientists can make you know things a little bit more accessible so we put out a paper but also taking the step after that you know educating the public about it after you know finding different ways to you know look at our findings and give this to the public because most people I mean unless you're in a scientific you know based program you're probably not going to be looking at these papers you're not going to be looking on google you're going to be looking on facebook where a majority people you know get their information you know not that that's a bad thing but I think the problem is you know we have to find as scientists we have to find a way to make things more accessible for people so I'm actually part of an organization at UH it's called student immunization initiative or SII and part of our job is just to educate the general public about vaccines and you know give them information about a lot of misunderstanding yes yes and it's not to say that you know to make anyone feel you know ashamed or even you know you know feel bad about their beliefs but I feel like a lot of it is misinformation and so our job is to just very nicely educate the public like this is the data this is what the show what they show these are the safety concerns and so we kind of felt as a student body that you know what we need to take away that disconnect and make a connection to the public and help them to understand some of these papers and look at the scientific jargon and that way it's more accessible and so that's kind of something I've been a part of and that's been really good we've been very active as an organization are you succeeding yeah I think we are we have over the course of a year so I'm actually vice president of this organization and we have done over 17 events in a year what's it called your organization student immunization initiative and so this is student ran it's all grad students we also are recruiting undergrads to kind of join us and all of it is based off you know educating young students high school students educating mothers you know people of my generation we just want to get the science out there in a more accessible format so so this is a evolution everything goes back to evolution is that right so this is the evolution of science and I think we are recognizing because we have to that the public has to be socialized on this or the scientific initiative will be stunted somehow I actually think people still love sciences in the same way that they did in the 50s and 60s I agree yeah I think most people love science the way that they did that then I think you know if you ask them do do they love their computers or do they love lactose free milk you know it's it's science I think people understand that I think it's the proliferation of different kinds of news on the internet now I think that's been one of the issues that have maybe so you have so you have school where the teachers teach you about the good things about science and then you have the internet and the internet maybe is undermining what you're learning in school is that what you're saying maybe not undermining it but I think we have so many different avenues of news coming at us that sometimes it's it's it's hard to know which one is correct that's true which one is credible which one is just one of those fake news articles I wouldn't say everything is undermining the true science but we have so many different you know opinions that it's sometimes hard for the for even you know scientists to know which one was it and sometimes you just have to sit back for a little and and let it all settle and that's exactly why we take a break at 14 minutes after the hour yeah so we take a short break now so it can settle a little bit we come back we'll ask you tougher questions yet aloha i'm kawaii lucas host of hawaii is my mainland here on think tech hawaii every friday at 3 p.m we address issues and importance for those of us who live here on the most isolated land mass on the planet please come join me fridays at 3 p.m mahalo hello this is martin dispang i want to get you excited about my new show which is called humane architecture for hawaii and beyond and it's going to be on think tech hawaii from downtown honolulu on tuesday afternoons 5 p.m and we're going to talk about uh to make architecture more inclusive on the islands which is what you which was one of the definitions of humane which is being tolerant of you know many people of nature of many other influences so we're going to have some great guests like today's guest for example my collaborator david rockwood who's the author of the awesome manifestation of humane architecture in the background so see you on tuesdays 5 p.m i look forward to the community matter is back we're live the march for science it's on april 22nd this saturday at u.h manoa and elsewhere and the one in what was it the big island yeah helo is going to be the the day before the night before well so and we have three guests here we have uh mark hixon he's the chow and endowed professor of marine biology department of biology u.h manoa versilla seaboard she's a master of science and a phd student at the department of entomology at u.h manoa well she's involved in this uh initiative initiative for yeah vaccinations yeah and dr grant yamashita evolutionary biologist and also consultant and he promised to consult with us today so let's you you wanted to add something mark to what we were talking about just for the break yeah this question about you know what's happened recently where suddenly this disconnect has occurred between science and society in some ways and you know for the average citizen i think it's what i said before that sometimes scientific findings come up against one's worldview and it's difficult to accept that shift the other big factor here no question is when scientific findings conflict with what a corporation wants so you can imagine well i'm an old one that happened a long time ago was cigarettes and cancer there you go you know so that was way before the present political situation but the tobacco companies poured thousands and millions of dollars into saying no there's no problem with smoking tobacco and cancer and and it just took forever it seemed to finally break through that and bring about the change yeah but they were you know degrading science and scientists and purchasing their own sciences scientists um and that sort of thing continues here's one brand new example it hasn't even really hit the headlines much yet but it was recently found that the active ingredients in sunscreen oxybenzone is has very negative effects on corals i bet the sunscreen companies aren't very happy about that finding but it's a fact it's a truth we must change our sunscreens and it's not going to be difficult to do so same thing happened with the ozone layer back in the 1980s fortunately changing those refrigerant chemicals wasn't that difficult so there wasn't a big outcry now you come to climate change major major sea change must take place in our energy sources and the powers that be in the fossil fuel industry don't want that so it's well documented that they've put millions of dollars into denying the science delaying action and this is one of the big issues we face no question yeah and that's why i think this this march the march for science and the whole event that you guys are involved in it came out of trump that's the reality but it's also a statement of more it's a statement of you guys have got to get used to science you've got to accept science you know all these conferences conventions are credible uh what they're publishing is credible and you can tell how credible it is do not oppose that do not let special interest groups lie to you um as they have been doing let's respect what we know and and i think um so there's another message there isn't there it's not just that the government should you know stop with this uh denying climate change this is the government has to respect science we all we all we all have to know the issue and respect the science thank you for that point thank you so um let's let's talk about your individual uh disciplines to start with you grant your evolutionary biologist i i mean i it's a fabulous area it's it's so provocative and so it strikes the imagination right how can you live with yourself um it's sometimes it's hard ask my wife so i'm i so i'm an evolutionary biologist by training but my my interests have always been in the history and philosophy of science what is the relationship to the discipline of evolution to you know um the history of science philosophy of science so i think what what mark was just talking about is something interesting about what historians of science can actually do right um neomio reskes right her you know um historian of science at harvard her book the merchants of doubt you know it was all about the tobacco industry all about climate change and all the all the corporate work that went into denying connections between the science and what was actually going on i look at somebody that i work with closely dr g in mind shine at arizona state you know she her her work is on the history of embryology so thinking about you know thinking about looking at the history of embryology the evolution of of embryos from you know millions and millions of years ago what can that tell us about um embryos as they pertain to you know abortion rights and and women's rights and stuff like that so uh i think history of science and philosophy of science can actually tell us a ton about policies you know that we had we hadn't really really thought about because there's a way of connecting science to the past that i think is important for current problems and future problems as well yeah you can you track human development by science yes human advances are most accurately reflected in advances in science don't you think yeah i think so we can have wars but that's not much of an advance what about you priscilla what you know what what's your of course the vaccinations are important yeah there are a lot of people out there who deny the value of vaccinations a lot of people out there who deny the risk of pandemic yeah and yet we live in a world of potential pandemic every day yeah very true very true so my history um so i got my master's degree at department of tropical medicine at jabsum and so there i learned about different things like infectious disease we were introduced about you know different vaccines and how to prevent certain pandemics things like that and and we also got introduced into yeah like there are people that um do not want vaccinations unfortunately and do kind of deny the science unfortunately and they think um there's an issue with safety um things like that and so my that was kind of my history is just looking at infectious disease and now that i'm at you know entomology department of entomology so i'm focusing more now on the vectors of disease so mosquitoes particularly and you know one of the things that's been on nationwide news is you know things like zika and really what just last year we had a dengue outbreak on big island so it's important to study um one infectious disease but also the vectors um and so that's kind of my background but um you know i do i do feel very passionate about vaccines and just you know informing people and just giving them the right information so they can make these decisions for their family so it's a really interesting question because the world is a much smaller place i mean we we could have Ebola everywhere if we don't watch out well and like zika yeah the globalization of the humans really has allowed for i mean hawaii actually has no native mosquitoes we are not an island that um hawaii is not somewhere that you know all of the mosquitoes that are currently present have been through invasive um invasive uh occurrences and so you know that happened via globalization you know the fact that we have planes so yeah you're right it's very easy to have um a pandemic more now than it was say even 10 years ago yes and a body flying hither and yon everything is connected even places that weren't connected before exactly and these these vectors can be established uh virtually overnight i saw a piece recently about Ebola and there were a lot of people in Africa who didn't think that the uh you know the that the treatment was necessary and they would break through the the quarantine um and of course that that had to be stopped and so i you know i i offer you this thought sometimes you have an intersection of disease prevention you know what you do and the law sometimes you must say no you you can't cross that line yeah you must have your child vaccinated yeah no you must take the medicine yeah no you must cooperate with us in in saving this particular community or society from a given disease yeah especially we haven't really got there yet though you know yeah well especially when the given disease can be very um you know detrimental to you know majority of the population is that's i think something where that comes into issue and but hopefully you know if we can just continue educating people it won't have to be something that you know is mandated but people just say yeah i you know want to get vaccinated because you know you know the scientific data has shown that it's important and it's safe and it does prevent you know outbreaks like things like measles you know so i feel like we as i said earlier assigned is we have to educate people and we have to just kind of keep doing that you know i would i would make laws that would have sanctions because some people are uneducable yeah yeah there yeah there are people that are unapproachable when it comes to this and they have third like you said we have world people that have worldviews and it's very hard to um educate them and also just persuade them that you know hey there is an issue with your worldview and it doesn't kind of you know balance out with science but that's a hard that's a hard subject to tackle because i i've actually had you know communications with um anti-vaxxers and you know for the most part um there's i've had a friend that you know we did educate her and she was able to understand and you know i approached it from a very nice point of view and just said you know what are your questions what are some concerns you have and let me show you the information you know i didn't attack her but um but i've also had interactions where it was like talking to a wall and i kind of you know just had to stop so i leave you with the most difficult question of all that's good this is not an option science is not an option we have nearly 10 billion people i get that number right on the on the planet today over seven seven okay we're climbing we're climbing okay and um you know they're all at risk our our societies at risk our infrastructures are at risk all the delicate systems by which we live are at risk and if we don't listen to science billions of us are going to die i'd like you to comment the risk of ignoring science is incalculable it's immense it's incredibly immense the issues we're facing now with a number of people on this planet are so extreme and starting to accelerate so rapidly that if science is ignored and delay is met it is going to cost our children and grandchildren dearly and that's why i'm marching for science i have two grandchildren and it's all about them and it's heartbreaking for me to see what's going on now at their ages of four months and four years so um yeah i agree with you at the same time it's it's it's very important not to paint a picture of gloom and doom we can change things on a dime if we choose to i mean it's very similar to when the us decided to go to the moon turned on a dime amazing technological and scientific achievements national pride and just such cool science so we can do this and it's just a matter of the people demanding it the people must demand it the powers that be the corporations are not going to go quietly into the night although some of the more imaginative and creative and ethical ones are saying wow we need to change okay we'll change in some direction it needs to become a movement what joanna massey calls the great turning we must have the great turning well the great turning is implicit in the in the march for science on saturday april 22nd at uh and elsewhere in the state and i i'm i'm grateful for you guys for setting it up for being there for coming down today and talking about it mark priscilla grant thank you so much thank you thank you thank you for having us


Did you know that you can see how much battery your RONDO 3 has left on the audio process itself? Let's take a look at how to do it. To check RONDO 3's battery status, RONDO 3 must be turned on. Briefly push the on-off button. The main indicator light will blink to show how much battery charge is left. One long green blink means that there is over 75% of battery power remaining. Two shorter green blinks indicate that the battery charge is between 50 and 75%. Three short orange blinks show that the battery charge is between 25 and 50%. Four quick orange blinks show that the battery charge is less than 25%. And you should consider charging your RONDO 3. Four quick red blinks indicate that the battery charge is low, and you should charge your audio processor as soon as possible. You can also check your RONDO 3's battery status via the AudioKey 2 app, so you never have to wonder how much battery your RONDO 3 has left again.


Thank you Cherry. So very pleasant afternoon to one and all present here. So I mean we all know that with the increasing amount of data that we collect we want to process the data as quickly as possible. So this is one step towards that direction. So today my talk is going to be about HPXEL which is asynchronous integration of GPU computing with HPX. So homie I am Madhavan Seshadri. I work at Visa Singapore. I graduated from NTU last year and I've been working for the Stellar Group under which the HPX and HPXEL development has been going on since my Google Summer of Code participation in 2017. So I also have contributions in the vehicle routing domain. I've published a few papers and a few journals and conferences. So what is our main goal? Our main goal is towards massively parallelism. Like we want to make sure that we can execute as many tasks as in an as parallel manner as possible. So but as such we are inhibited by the complexity of new languages and various programming techniques that come with different sets of devices as well as the versions of the devices. So most of the compute power today is split across GPU devices, CPU devices as well as other and each type of GPU devices have their own coding language. For example you have CUDA, OpenCL and whatnot. So another main challenge that we are facing right now is the architectural integration of heterogenic devices under one single framework or one single umbrella is is difficult and it's oftentimes it's inefficient than what the native implementation provides us. So we as such we need to integrate diverse programs under one single umbrella and we want to maximize the utilization of the available processes as much as possible. So before I go into depth about the design changes that came along with integrating GPU devices with HPX, I'd like to give a bit of introduction about what HPX is and what it can do. So HPX is a runtime library which offers fine grained parallelism for CPUs and Xeon fee. So as such the integration of GPUs into HPX right now is a bit experimental and while we are designing a solution for this we want to minimize the application level latency that a new API layer would incur and we also need to minimize the CPU to GPU you know inter-processor communication time as well as the vice versa. So we also want to minimize IU stalls and data transfers. We want to overlap computation with communication as much as possible when it's not hindering. We also would like to provide a common synchronization mechanism what if you have some tasks running on the CPU some tasks running on the GPUs and other processors but what if you want to synchronize all of them using a common synchronization mechanism we want to we want to be able to provide that using this. So what is HPXCL? HPXCL is just the integration of GPU into GPU processing into HPX framework. So right now the focus is on CUDA. There is also a version which you know brings in open CL implementation into HPX. So it provides a uniform synchronization mechanism through the use of futures more about this in the later slides and we've reduced the API level or the common layer level latency which is incurred. We've benchmarked it to like against certain algorithms like dense matrix multiplication sparse matrix vector product which is highly data parallel. So here's how the presentation is going to be now I will focus on giving a brief overview about the HPX components as well as how the new components which we are bringing in integrates with HPX. After that I'll discuss a bit about synchronization of tasks measurements as well as the last would be conclusion. So if you look at this architecture diagram and just focus on the right hand side a bit. So you see that there are multiple localities which are separated by Agas and Parcel. I'll discuss more about each of these components in detail over the next few slides but this is just to give an overview of how things work with HPX. So first things first what is Agas? Agas is just an active global address space which maintains the list of active global objects in memory. It uses 128 bit virtual addressing space which spans across all the localities. Now if you see the diagram the shaded region the shaded region is the objects which are visible globally which means objects from one locality are visible across other localities and can be just used with the same set of references. The unshaded region is the local objects which are pertaining to that locality and are not shared across the localities. So the addresses of these objects are unique and system-wide that means you just use the same references across different localities. To obtain the devices in the domain for this integration of GPU with HPX we use Agas for just storing a system-wide level list of devices which are available across the localities. So then the next thing I would like to discuss is Parcel. Now we've discussed about an address space which maintains the list of objects in memory but what about intercommunication between these localities? So that in HPX that happens through a component called Parcel. The underlying implementation of Parcel can be changed from open MPI to like open MP, open MPI or just plain old MPI. So inter-node communication in HPX happens through Parcel. The implementation of this layer is one-sided and no pooling is required from any locality so as to minimize the wastage of CPU cycles. So the next thing I would like to discuss is the thread manager. Thread manager we use it for scheduling jobs like making sure the jobs are getting executed in the localities we want them to get executed in etc. So yeah as I just said the management scheduling of the jobs is managed by the thread manager. Actions are something which come from accumulators in HPX. So basically the actions provide a mechanism in which you can transport a function from one locality to another along with a set of parameters as arguments execute the result in a completely different locality and then bring the result back. So it provides remote invocation mechanism for transport a function. So there are some inbuilt settings which HPX provides so as to optimize the utilization of the hardware. So you must have guessed by now that locality is just set of nodes in which jobs like threads and jobs can just run and are being executed. So from this diagram you can see that different GPU like GPU devices can be attached to any one locality. We want to be able to think of this as a single unified computing resource and then be able to just schedule jobs to them and be able to synchronize whenever we want. So what are the new classes which we are bringing in through the integration of GPU. So we are bringing in three main classes device, buffer and program. More on this over the next few slides. So device is just like one it's a one-to-one device this device class and the objects which are created from it is just a one-to-one mapping of the actual physical device. So your GPUs could be spanning across different different localities which are different nodes and then using your agas you can have the references of all the devices available in your memory right now. So it's used for creating buffer and program objects. So yeah objects across locality can be accessed from one single place. So device also comes up with it's an abstraction from the native implementation. It comes with facility to build kernels runtime as well as to execute them when we need it. Yeah so the next to discuss about is buffers. So you can think of buffers as just you know data pipe. So basically buffers do the localities which require the usage of those data. So let's say I'm a threat I come across a particular particular reference of an object which I don't possess right now. So what I'll do is I'll go to agas. I'll get the I'll I'll I'll ask the agas to send me that copy of the data that is happening through parcels. So as buffers so in case of buffers we use this to transport data to locality we transport it to the GPU device attached to that locality. So it's just one buffer per data argument which needs to be passed to the kernel can be provided here. So the next thing I would like to discuss is about program. Program is just it's a class which gives you functions to do runtime compilation of any piece of code which you want. Currently it takes two different types of inputs. You can load kernels from files and you can load kernels from strings. So basically if you just see you can obtain like you can call the device and you can create a program with that particular device. And whenever you want to run that program you can specify the parameters of grade and block which you would normally provide to you know a CUDA implementation as well as the list of parameters as an argument vector. So we've discussed about different components which the framework is providing. The next thing I would like to discuss about is how do you synchronize jobs which are running in CPUs and GPUs and have just one synchronizing mechanism. So as I mentioned earlier we are using futures for it. So every time you call any asynchronous function what it does is it returns you a future. HPX provides two input functions, wait and wait all. Wait takes in a single future as an argument and just it's a blocking call. Wait all takes in a vector of futures and then it waits for all the all the futures to return their value. So using all these mechanisms it if you have an asynchronous execution tree of tasks you can execute tasks you can just start the tasks and get the result as and when you require them just before the next task in the pipeline is required. So how it all fits into place. So basically as I have mentioned earlier you have a get all devices function in HPXCL which gives you the list of devices which are connected across the nodes. To obtain a particular device from you can just use the vector and get any device from that vector. Using the vector you can create a program object. Program object lets you create a program from a file or a string as I've mentioned before which returns you a future. You can then initiate buffers for transfer to your kernel. So you can just create the buffer and then you can use the NQ write function to write something to the buffer. So just before let's say in this small snippet of code just before you want to execute the program you just wait for all the buffers to finish transferring data to that particular GPU. So this particular link provides you a complete example which multiple devices attached to your to the nodes to be able to split the computation into multiple smaller pieces execute them across different GPUs and then bring the result back and stitch into one single image. So this is just it's not clear but it's just a black and white Mandelbrot image which is used. Now if you carefully see the execution time and the plot to which which is here there is not much scalability on increasing from one node to like one GPU device to multiple GPU device. This is actually because of the fact that we are for measuring the time we are including the data transfer time like it's it's basically end-to-end time. So data transfer execution and then bring the result back like all of the timing is measured in this plot. So there's not much advantage in you know going from single to two to three GPUs but if we were just to measure the GPU execution time that would be highly scalable in this case. So previously I showed an example of how multiple devices can be like multiple devices can take the jobs and execute them in parallel. In this section I would just like to discuss about the overhead which this layer is bringing in. So as I said before we tested against several different algorithms like against let's say sparse matrix vector product dense matrix multiplication we see this has all been implemented on only one device. We see that there is not much difference between the native implementation and that of our our layer which with like the overhead incurred by our layer but our layer provides the capability to execute jobs on multiple different GPUs. So the same thing with that of the partition benchmark as well there's not much latency which is incurred. So what we discussed today was I just introduced about HPX as well as the integration of GPU processing along with the CPUs provided so this framework provides the opportunity to execute jobs on multiple GPUs in parallel along with the CPUs and synchronize them as and when you require it. Provide minimal overhead while adding a new common layer and then we showed an example and implemented it using 3 GPUs. Thank you. Any questions? Yes. We do have we have a version for OpenCL and we are currently testing it right now. So it is a common layer which you can call you know open like which you can supply like OpenCL kernels or CUDA kernels and be able to work with them in the same manner. You don't need to deal with differences or you know the language level differences. Yes. Okay. Yes. So basically what you would need to do is like as a developer the owner is still on you to identify the parts of the program which can be data parallel and then be able to just provide those data parallel parts to the GPUs for execution. The parts which still need to be in sequence you can use HP X 200 and CPU. Yes. Okay. Is it multi-coasters? Can you run multiple, can you run the same kernel on multiple hosts with separate GPUs on each host? Yes. So that was the whole point about different localities. So each locality you can think of them as a separate host and then this is a layer on top of your host. So that abstracts the host level differences.


A big name was Ta himself. We're so grateful as SP and Bars to be here. Tell us about yourself. Maybe an intro. Some of you may know me as your sister's next mister. Some of you know me as Mr. Seal Your Girl. Some of you know me as your future ex-boyfriend. But today I come here as Kaga Mungai. The man, the musician, the businessman and yeah we're here to just celebrate and have a good time.  Why all this? Why not? I mean the thing is there's a lot to do. There's a lot to do and art is one part of it. You know and I think art is all connected whether it's fashion, whether it's music, whether it's production, whether it's all of these things, they're all connected. And the thing that pushes the thing forward is usually business. And so obviously now you have to put in the business with the art and now you have a great product. And that's what we did when we were creating this company called OB and OB comes from the name Nairobi. We took out the first few letters of Nairobi and we kept OB and yeah so that's what we're doing at OB. We're creating dope art that can be appreciated by anyone from all walks of life all over the world. OB is a production company. Maybe you can tell us details more about OB.  So whether that's exclusive and very intimate kind of experiential events, whether that's music, whether that's videography, advertisements, documentaries, everything creative that you can think of OB Africa does. And we've been doing this for about a year and a half. The dream has been bigger much longer but it's been about a year and a half. We started just when COVID began. We thought you know what it's tough but we have to continue going and so so far so good. So far so good. Are there artists been signed under OB? Yes for sure. Not this year. From next year 2022 election year we're electing some new artists. I've scared some guys are saying forget what it is. So that's where we're going to be sharing guys. We're really looking for talent and so whether that's musical talent, whether that's a graphic designer talent, whether you have talent for being an emcee or a host or whatever it is. We're looking for talent and in 2022 you're going to be seeing a lot more talent coming up under OB Africa. Under OB Africa in 2022 guys are planning to sign artists. Maybe to ask you as a producer, as an artist, what do you look into an artist? And in Kenya you've given a chance today. Who will you sign today under OB? I can't tell you who I would sign because maybe I've already signed them and it's not privileged information. But the thing that we look for is one originality. Two is consistency and confidence. This industry is a tough industry. It can really wear you down. It can beat you up. I've been doing this for about seven years now and I've learnt a lot. There's quite a lot that you go through. A lot of people you interact with and if you don't build a thick skin very quickly, it will swallow you whole. So we're looking for people who are headstrong, who know who they are, who believe in what they're doing and obviously have original great talent. You've mentioned it's been a tough seven years. What keeps you going as Kaguen guy? Ah, I mean everything. The fans, first of all, I think I can't ignore that. But more importantly, my family. When I say my family, I mean my relatives but also the people that I consider my family. And some of them are here today. And so to share these kind of moments with them is what makes it worth it. Because they've seen everything. They've seen the ups, they've seen the downs. And so now it's great to have them here as we celebrate. You've mentioned one of the things you look into, signing an artist's originality. Kenyans who don't have our original kind of song, general song. There's a time you say it's bongo or you say it's genge or you say it's genge to nikakuja. What do you think about that? All of those things are all original. The thing is just, I think what we have as Kenyans is we have diversity. One day you hear Butrus singing, yeah, yeah, yeah. Ati Wanjira Kosingo, Min Kosingo Pia. Then the next day you can listen to Sauti Soul. Then the next day you're listening to Benzema. The next day you're listening to... Every day, obviously, you're listening to Kaga Mungai. But generally, you know, we have a lot of diverse sounds, right? And I think that's what makes our industry the greatest industry in this continent right now. Is that we have diversity. We're not telling people, if you don't sound like them and you want to be a hitmaker. We're telling people that as long as your music sounds dope, you can make hits. That's it. What keeps you going as Kaga Mungai? Is it the good artwork of writing music? Is it the production? Is it you as an artist? I mean, I enjoy all of them. I enjoy all of them. I enjoy the music. I enjoy the production. I enjoy being on stage. All of them, they compliment each other, you know? And I start a song. I start the song in the studio. I make the beats. I write the lyrics. We think about the video. We think about executing the video. We think about how that song then sounds live. And so we're always thinking about the bigger, bigger picture, you know? We want these songs and videos to be played 20, 30 years from now, you know? We want the people who are watching this now in 2060, you know? To be like, wow, this thing was made in 2021. That's dope. And I still can listen to it. I can jump to it. It's a good vibe. Amapiano Wave is here. Everyone wants to do Amapiano songs. Are you attracted to do Amapiano songs? Amapiano is dope. And I think I can say perhaps last year I had a song that had some Amapiano influences called Che Che. And Che Che was probably the first of that kind of sound in East Africa. Because it's a strong sound. Amapiano is taking over Africa. It's become a globally recognized African genre. And it's great. I'm happy for the South Africans. I'm happy for Africa. It's good. When Africa wins, we all win, you know? And so anyone who's doing Amapiano, may I salute them. I've been seeing Kina Brandi Minor and a few other guys doing Amapiano now, which is amazing. So as long as it's African and it's being made by Africans, call it Amapiano, call it Gengetol, call it Gengetiano. May I like this? To confirm today, do you write your own songs? I do. I do. I write my own music. But I also work with different songwriters. I produce my own music, but from time I also work with different producers. So for example, this song, Bogey Amadenge, was mixed and mastered by this guy called Sofresh. Sofresh is a very, very amazing producer that has been working with many, many guys from soul generation, various many guys. And so, yeah, we work with different guys from time to time. But if you ask me to write a song, I might write my songs. Bogey Amadenge, it's a kind of Gengeton song. We had conversation before. Gengeton guys are claiming Gengeton is dying. What's your view on that?   That's like saying, Reggae is dead. Reggae is not dead. Dancehall is not dead. It never died. Bongo never died. It's just that from time to time you hear something else that's really more exciting and Chang'an is just, you know, guys differently.   You know, Gengeton is here to stay. It was made from something real. It comes from something real. And that's why it's alive. And that's why I believe in Gengeton. Why Bogey Amadenge? Because number one, it's important. One, as Kagwe, right? I have a very loyal female fan base. Shout out to all the fans. And I just really wanted to shout them out and say, you guys are with me. On this new stage that we're going to, it's just how we're going to come together. Me, me and Sisi, we're going forward, right? Number two, it's also to just give women that platform, right? I think a lot of the time, Gengeton can be a little bit crass and demeaning to women sometimes. And I didn't want to make a song that was demeaning to women because women are who I work with, right? We employ women at OB Africa. I have lots of women in my life that are my great, great, great friends. And so I would never want to make anything that would be, you know, dirty in that kind of way. So it was something to also amplify that conversation. Maybe you can tell us, why did you decide to work with Benzema on this new song, Bogey Amadenge? And then maybe the experience working with Benzema, studio experience, production, during the shoot. If you can share with us. Yeah, no, Benzema is a great guy. He's a great guy. He's somewhere around here, just doing his photo motors. But, you know, he's a great guy. We started talking some time back. We agreed, hey, let's make some music. And from then on, we've just continued to meet up and share ideas. And one day I played him this track, which was Bogey Amadenge. And he was like, this thing is, this is the one. This is the one. It's musical. It's Gengeton. It's authentic. And it's real. And it's still dope. It hasn't lost dupeness. It's amplified. It's stronger. And he was like, we have to do the song. And so we sat down, did the song, recorded the verses. And then from there, we shot the video. And he played his parts. He did a really good job. I'm really, really grateful to have him on the song. Alejandro, you know, he's, he's, he's. And also it made sense. You know, it made sense. Coming as, as, as, as the guy who was previously stealing people's babes. To also not having somebody who is currently stealing people's babes. I think it makes sense. Kagwin, what do you give your female fans? Good music. No, bro. Tell us. No, it's good music. Honestly, honestly. And I feel like since day one, that has been my most loyal fan base. And I'm really, really grateful. And so this is, this is for you guys.  You guys are my Mboggi, I'm a Dengue. And I appreciate you. Where do you want to take your music to the next level? How are you planning to take your music to the next level? May I want, I want, I want everything. I want everything. And we went for everything. Not half, not a quarter, not some of it. Not just enough. All of it. All of it. The same way that we are seeing our, our great actors. People like, you know, Lupita, we sing in South. So we're doing, seeing all of these people becoming African global brands. That's what we're trying to build with Kagumu. And that's what you're going to be seeing in the next few months. Are you planning to sing gospel anytime soon? Because I know you understand your diverse. You have the creativity. You have a team behind you. They can, they can suggest, they can give you a way. Maybe we'll try this way or the other way. I mean, I'm not opposed to it. I'm not opposed to it. I've done some gospel songs in the past. Which are still playing on radio. And they're being appreciated by my fans. But I'm not opposed to it. Right now, right now we're focusing on Bogeyama Dengue. And we're enjoying this wave. Understand we have positive criticism, negative criticism. How do you handle negativity? Negativity. I mean, everyone is entitled to their opinion. I don't, I don't listen to, I don't listen to it. I don't make time for it. I'm too busy. Honestly, I'm not even lying. I'm, I don't have time. I don't have time to, to read Nani's comment from CIGI, for example, who is telling me that CIGI, oh, you didn't, I don't have time, bro. He has time. He has time to go on my page and comment about me, me, me, me, negativity. I don't have time. So I focus on the positives. I focus on the people who are with me. I focus on the great things we're doing, the partnerships that we're making. And that's, that's the most important thing. Do, do, okay. Just to ask, do you feel like government is playing a bigger role in mentoring a young musician and not going through what you went through? The government are doing what? Government doing the right way. Like are they putting structures? We've had issues a previous month when it comes to loyalties to artists. Things are not going the right way.  We love it. We're here, we're Kenyans.  Not like any other country is not complicated. The US, they have their problems. South Africa, they have their problems. Nigeria have their problems, right? We have, we have our problems. And the thing is to try and make your impact in your own way. Right. And so for me, the thing that I want to do is give young Kenyans, young creative Kenyans the opportunities that we know that we deserve to have. And so that's what we're trying to do. We're not waiting for anyone to give us handouts. We're not waiting for anyone to tell us, oh, now is the time to do it. Now come and get money from us. No. We want you where you are. Come, let's build. Given chance today, what will you ever change? About? In the government structures. I'm not a politician. For example, given a chance to change the right of artists and musicians in Kenya, what is that one thing you want to change? Honestly, I don't know. I can't tell you that. I can't tell you the things that I would change because I don't understand how government works. I'm not a politician. But what I can tell you is that the people that have talent in this country are so many, so many. And we see them every day, every single, single day. And they are more that we don't see. Right. And for me, the thing that I want to change is the way people perceive the talent in Kenya. I want to change that.  And Kenya is on that list, right? So as me, my goal is to put Kenya on that list. If Kenya is on that list, then I'm good. Are you planning to work with any international artist? We are currently working with some. Just to mention a few? I cannot. Why? I would like to. I would like to. But contracts. Just to finalize on the interview. So many people are asking, is Kaku Mungai dating? Is Kaku Mungai seeing someone? Maybe just to confirm with us. Yeah, yeah, yeah. I have a crush on someone. I have a crush on someone. Yeah, yeah, yeah. You know? Who? Nairobi. I don't recognize him. He's a tough guy. So, yeah, yeah, for sure. There's somebody. OK, lastly, to the upcoming musician, what is your advice to them? Don't stop. Don't give up. Don't waver. If you believe in yourself, keep going. Keep going. Remember what you started this thing for. Remember why you wake up every day. If you're doing it for your family. If you're doing it for yourself. If you're doing it for your country. Whatever your reason is, don't stop. Ever. To your fans, maybe where they can find you on social media platforms? At Kaku Mungai everywhere. And just want to say asante. But also before I sign out. I also wanted to just give a huge, huge shout out to the partner who we've partnered on this event with. And also with this release. Some good friends of mine called MediaPal. We have Maurice Juma over there and Eric Jackson who will come and sit down in a minute. And these guys we met up and we shared our vision for what this thing can be. And they're already doing crazy revolutionary things in the tech space and the advertising world. Google MediaPal and just have your life changed.  And none of this would be possible without them.  You guys are part of the culture. We can't do this thing without you. Me, I'm Kaku Mungai. You can follow me at Kaku Mungai everywhere. And don't forget, let me tell you something. My name is Haseera Kahende. If you don't want to know, I'll tell you later. SPM Buzz.