In this video, we'll be talking about the process of self-organization in complex engineered systems. We'll firstly talk about why self-organization is a key feature and characteristic to these technologies. We'll then go on to discuss the basic ingredients that are required and the process through which these systems self-organize. Finally, we'll wrap up by mentioning how this process leads to suboptimal but often robust solutions. Firstly, just to refresh our memory of what self-organization is. Self-organization is basically the spontaneous creation of a globally coherent pattern out of the local interactions between initially independent components. There are essentially just two ways in which a system or technology can achieve the global coordination that is required for it to function. Either this design pattern is imposed on the system from some external force, or it emerges out of the interaction between the different components internal to the system, and this is self-organization. With the linear technologies that we previously talked about, such as bridges, chairs or toasters, the system is designed and controlled by the engineer and operator who imposes an external pattern of organization on the system in order to make it operate in a predefined, optimal fashion. This is our traditional industrial paradigm that we typically try to apply to all areas from governments running a country's infrastructure to designing residential areas and parks. With this centralized design, the components are coordinated through a linear top-down design pattern that constrains the components and requires them to function in a well-defined, specific manner in order to achieve the global functionality. This paradigm is so familiar to us that the idea of getting global coordination without some centralized plan or design may sound impossible or like magic, but it's not, it actually happens all the time. Think about people crossing the street. There is no one orchestrating the process, but through the local interactions between people, they become differentiated out into a distinct pattern with the formation of coordinated lanes going in each direction. Without formal design and centralized coordination, components interact in a non-linear, networked fashion. The type of organization and patterns that result from this are defined by the type of interactions between the components and not the design pattern that was imposed on the system. Whether we're talking about the internet, a smart power grid, transportation networks or logistics networks, in these complex engineered systems, the elements have some form of autonomy to choose their course of action. For this reason, these systems are not the product of our traditional top-down method of design. The overall structure to the system is a product of the local interactions between the components, with global patterns emerging out of these local interactions. There are a number of basic prerequisites that need to be present for this process of self-organization to play out. There needs to be, firstly, some form of randomness in the initial state to the system, variation amongst the state of the components, elements with the Autometer Act, and local non-linear interactions between the elements. We'll discuss each of these conditions and why they're important separately. The first important condition is some form of randomness in the initial state to the system, or at least lack of centralized coordination. There is no possibility for self-organization when the system is already designed and held in some well-defined and orderly structure. When a city is centrally planned and regulated in a top-down fashion, then there will be little space for citizens to self-organize. But when immigrants move into unregulated areas around large cities, we get self-organization in the form of shantytowns, specifically because this space is unregulated. Wherever there's a lack of centralized control and adaptive actors interacting, we'll get some form of spontaneous emergence of order, such as the flow of automotive traffic in a city or the flow of IP packets on the internet. Secondly, the components in the system need to be heterogeneous. Heterogeneity of states is important. If all components already have the same state, then they're synchronized and there is no need for self-organization. If all the people crossing the street are crossing in the same direction, then they're already coordinated. It is only when there are people going in different directions, that is to say components occupying heterogeneous states, that we get friction in the system as they bump into each other, and there is the possibility to optimize and increase the efficiency of the system by synchronizing the components activities in order to reduce this friction. And it is often friction created by variation that drives the process of self-organization. Thirdly, the capacity for components to adapt and respond to events in their local environment is important. We can get self-organization without adaptation, such as can be seen in substances like boiling water. But adaptation is a major accelerator and key element within the process. If things can adapt, then they can easily and quickly alter their state in order to synchronize with other components. And this is part of why information technology is driving a whole new architecture to our technological infrastructure. Because of the lack of capacity to adapt, our traditional industrial technologies had to be designed in a top-down fashion, where everything is well-defined, predetermined and remains static over the course of its life cycle. But as we begin to embed communications devices in all types of technologies, and through shared protocols, they can synchronize their state and self-organize to find new, optimal solutions to dynamically changing environments. Lastly, all these systems need to be able to interact. It is only through non-linear interactions that the agents in the system can synchronize their states and coordinate their activities. Typically, this requires some form of protocol and interoperable platform. For example, smart cities are enabled by open-standard IoT platforms that allow different devices to share information and coordinate. Through open traffic control platforms, cars can communicate with parking lots in order for the system to self-organize. By layering a telecommunications network on top of the power grid, producers and consumers can adapt to each other and self-organize to optimize the low balance on the system. Dense, non-linear, networked interactions between components is another key ingredient in posturing the emergence of a globally distributed pattern of organization. So these are the basic ingredients that facilitate the process of self-organization, some form of initial randomness, the capacity for autonomous adaptation and dense interactions. But now let's think about the actual process through which self-organization takes place. Firstly, as mentioned, we have some environment without centralized regulation. We might call this an open platform. Examples could be a smart power grid where anyone can join as a producer or consumer, or it might be a land area that is unregulated by urban planners, or it may be an unregulated mesh communications network. Heterogeneous agents then join this network or platform and start to interact. Next, components that interact most frequently and intensely begin to synchronize their states in order to reduce friction, creating what is called an attractor. Meaning that because some small subset of the components have formed a pattern of organization and reduced friction, they will be more effective at performing their function and this will attract other components to this more effective pattern of organization. For example, once a number of people have decided to occupy the same location, a town or city forms, with its own set of specialized and coordinated relations that then enables it to be more effective at developing large infrastructure, conducting trade and other economic activities which increases its relative wealth per capita. This creates a positive feedback loop that results in the attraction of others to this particular coordinated state in order to gain the benefits of being part of this organization as the city becomes larger and more populous. Lastly, when all the different components are aligned within different local level attractors, then these different local attractors may stay stable for a prolonged period of time but ultimately evolution will act on the system and they will have to either cooperate or compete in order to create a global pattern of organization or the whole system will become vulnerable to being subsumed by some more efficient external pattern of organization. The net result of this cooperation or competition being that all elements will be aligned within the global pattern of organization and the process of self-organization will be complete as this new pattern of organization both enables the components to function more efficiently but also constrains them by making them comply to globally accepted protocols of communications and behavior. As examples of this we could cite the development of most technologies from videocassette tapes to bicycles and mobile phones. Different design patterns to the technology were initially developed and competed or converged with one design paradigm ultimately subsuming all the others to become the default. The net result of this process of self-organization is systems that are not so orderly and are often suboptimal. If we look at ecosystems which are self-organizing systems, they are hugely suboptimal with respect to the processing of energy. Only a minute fraction of the energy that enters the system is processed all the way up the food chain. The vast majority is wasted. The internet is another example. It often simply fails to deliver packets. It has a massive amount of redundant and virtually inaccessible information. Unlike our traditional centrally designed systems that are specifically designed to be optimal self-organizing systems are not designed to be optimal. They are typically not orderly or well behaved but they do work and are often robust and sustainable in the long term. In these distributed complex engineered systems none of the components in the system have a global vision of the whole thing. They are acting and reacting to local information. When a new person moves into a shantytown, a new computer connects to the internet or a new vehicle enters an urban transportation network they have no global vision of the whole system and they are only interested in their particular local environment. And this is all the system is composed of. These agents following their own rules according to information they have within their local environment. No one is in control. There may be someone creating the platform and protocols but no one can control or know what is going to emerge out of these non-linear interactions. Tim Berners-Lee may have created the World Wide Web but he has no control over how it will develop nor any great capacity to predict it. The inevitable result of all these agents acting and reacting to each other locally is going to be a far from optimal solution for the whole system. But self-organizing systems promote engagement. People feel more engaged because after all they truly create the system. People feel empowered and actively engaged in their social networks although someone has created the platform and defined the rules under which they interact they still get to create their network and shape how it develops. Self-organizing technologies are a whole new paradigm in systems design. The IT network-enabled technologies of the 21st century are increasingly distributed and adaptive as we start to place embedded devices in all kinds of things and then network these things. This provides the enabling context for the process of self-organization to operate on our technologies. Meaning that our traditional industrial paradigm that technology needs to be fully designed and managed will likely be less dominant and have to share with a new model where designers and engineers are the platform and protocol developers through which different technologies interact and synchronize in order to find solutions. And as always it is the interplay between the two the top-down formal design and bottom-up self-organization that creates the core constraint within these complex adaptive systems. To summarize we've been talking about how complex engineered systems create global coordination through the process of self-organization that requires an initial state of randomness and lack of control. Adaptive components and dense interactions as an enabling context. It is a process through which initially heterogeneous components come into contact, interact and synchronize their states to create attractors. With these attractors then cooperating or competing to give rise to the global pattern of organization that is typically characterized as being suboptimal with no one in control but also often robust and sustainable.


So we here the Rockchip booth and you have a Chromebook. Yeah, we have a brand We already announced last month with Google and we had announced several devices like this is the Heisen's device. So this is it looks really really nice. Yeah, and this is just $149 with a Rockchip 3288 Yeah, yeah, we have the nice nice finish back looks nice Really nice keyboard So so who are you? Yeah, I'm a gamer from Rockchip Design team, I'm the director of the list on the department So you have been working on this for a while now It's it's very it's a hard work to make this work. No, yeah, that it is the Actually, it is a hard work the answer in the in last computer. So we have a show that the demo of the Chromebook right now you can see that we have the lot of the device here. This is the source device and This is a higher device. It's very beautiful asses. Yeah, Chromebook one sixty nine dollar It's a very good quality stuff. Yeah, it's actually beautiful. No, yeah, and I need to use be two ports. Yeah, they have a two USB three, right? Yeah, be two. Yeah, you have speed to the next generation the way we are We will have the USB the three dot the other so so this is ready a stable is fast, right? Yeah, the very stable and I think they're ready the good human resource human experience to use Okay, you can see that this is the higher device so higher is as Doing a nice two color Yeah, they have their two models the one is the photo tarot one is for the education version So the education is only white Oh, yeah, only white white and education is a different different finish a little bit more grip here or is also here No, this one is a different shape Different shape. Yeah So because Chrome OS is crazy popular for education. Yeah, so you can take it like this like a chair like this Actually as far as I understand Chrome OS in the USA is number one for education. Yeah, right It's a more popular than every other platform in the schools of kids So this is maybe going to be maybe it's also a little bit better for the shock a little bit or different Yeah, and this is so this is shipping, right? Yeah, it's already shipping and already do you can do go to the Google's website or you can go to the Amazon website and see all these device This is this really awesome and so can I ask you what is the performance compared to Nvidia? Do you want to say or you cannot say this is quad core arm cortex a 17? Yeah, it is the quarter a 17 And the master frequency it can up to 1.8 gigahertz and we have their powerful their Mali the T 7 6 4 4 4 the GPU inside and very good experience for all the GPU are you able to do to use the money GPU to we accelerate things? Yeah, yeah, we have there are a lot of the Yeah, the things to do that. So how can you use the GPU to accelerate the browser? Yes, as you know that we have there We have there for a long history to to co-operate with the money team and to for example for a for a browser that we have their GPU to accelerate to concretion. So it is most smoothly scrolling. Yeah all right and Smooth of multi tabs you can have many tabs open is going to be smooth with many website many different open websites Are they all to GB RAM right is a we have to it is all to be GB RAM To GB RAM is enough for good performance. It is enough for the problem that you can see the other performance here But the last we have there also the report for For GB to rent a version. It's also how is the difference in performance between for GB RAM and to GB RAM? I think the report you prevent will be the little more to create and Little more to smoothly it because it is the larger the S rain. So what is that? What does it mean see? So Did you change the SSC a little bit? No, we have there the Improvement there are all a test and have there a higher performance the folder for the see see it means Chromebook it means Chromebook. Yeah, it's a higher Frequency or no higher frequency higher frequency higher frequency What else is different? What did you change and different folder? We have there. Okay, special package for the crumble so it's Not getting hot. It's very cool. Yeah, temperature is good stability very very good test. Yeah Yeah, testing a lot. Right. We have that testing a lot for a crumble and it's even small enough and Cool enough to go in a stick you can do crumb it Yeah, we have there announced also have there announced crumb big the bar I cannot show the gear how small is it? Yeah, I think the just like this. I think the just a little bit Bigger the land usb, but why is it not shown here? Is it a still in development a little more? One two months more because that it is a customer that we have the customer did not have there Announced the list project to the market right now. It is just a show that we have there released the notes to fall out Yeah, so can you explain a little bit? What was the story when you you have been working with Google in the Silicon Valley and in China? And there's been good collaboration to make this work Yeah, we have the good cooperation that with Google for you as you know the for for enjoying and for everything And for a crumb book we have the good relationship with Google and sometimes we go to their Hackwater to co-works with land or sometimes go goes guys They welcome to China mainland to have the co-works even though ODM or EM and did they tell you come on come on fix the bug What did they say? Well, how does it work? You have to you have to meet a way have to make the make the Yeah, and you you it's very hard work no yet Well work very hard also worked in a very crazy speed very fast yeah very fast and Very fast to have the land to follow for MP and you have very talented engineers Working very a very smart and talented engineers to cooperate with the Google site Cool. So how many do you think you're gonna ship? You don't know. I think it's gonna be big No, I think the least year to a bit Target will be a bit where the way we want to ship out for example 7 million or 10 million or even more But you actually you kind of said that there might be another generation. So maybe there will be 64 bit Chromebook or something I'm just guessing maybe yeah, maybe yeah for a nest maybe but all the work you have now You can continue the work. Yeah, right. You can use it. Yeah, right very cool We all prepare for it and a lot of people the booth here the Hong Kong fair Yeah, a lot of the maybe it's the most interesting thing at the fair. Maybe yeah We have the also have a way have the Communication product line on all the IoT product line and you have a 64 bit. Yeah, we have the 64 bit chipset I'll do other video for this one. Yeah, okay


Hey folks, welcome back for another episode of Code Club. If you have been following along, and if you haven't, why not? I have been building out an R package that will allow us or allow people who do microbial ecology to take 16s rRNA gene sequences and then classify them to figure out what type of bacterium that sequence came from. Now, many of you might be saying, who cares? What does that even mean? That's all gibberish. That's fine. That's fine. I kind of agree. But what we're really doing is we're learning how to make a package in R and we're learning to think with more of a mind of a programmer in R rather than a data analyst in R. And so as we've been kind of going along, I've been trying to kind of point out different places where we might take a different approach, again, between a programmer versus a data analyst. I think most of what I do is more of that kind of analytics mode where, you know, I'm summarizing data, making different types of figures, I'm writing reports, right? Whereas, you know, trying to write code that is going to have to be performant, right? So we're going to be doing some of these calculations, perhaps thousands or millions of times. And so it's going to need to be fast. And that we might, you know, we want it to be robust, we want to make sure that we're getting the exact right answer. Even if we get kind of weird input. So to achieve that, we've been using a technique called test driven development. I've got maybe three or so episodes under my belt now of using test driven development or TDD as it's called, and I'm feeling pretty good about it. And so what we're going to do in today's episode, as we return back to my pseudocode or just kind of this outline of the general project, is that I'm continuing to work on the right side of the screen, looking at the reference sequences. So again, we're going to have this reference database that is looking at k-mer frequencies that will allow us to take an unknown sequence and then say, well, within my reference database, what genus has the k-mers that are most like the k-mers in my unknown sequence, right? And so then once we know what sequence, what general in the database are most like my sequence, then we can begin to give a classification. So we have been working on various parts of this outline. And so what I want to do in today's episode is put it all together in one function that it will be a function that is kind of facing outwards towards our user that our user could use where they could give it their sequences, their taxonomy, and the k-mer size they want to use, and then generate that database that they can then use with their unknown. And then we need to turn to the other side of this pseudocode outline. Over here in our studio, I'm going to go ahead and run a test. Again, you can click on the button or you can do the command shift T, and it will automatically run for you. I kind of like the shortcut keystrokes. The problem is there's so many shortcut keystrokes that it's hard to keep them all track. But I find again, the command shift T is something that I use a lot to rerun these tests. Also, if I want to be able to get the functions that I have in my k-mers.r script to work, I can do command shift L, and that then loads my package so that I can kind of develop out the code as I'm using these functions. So let's go back to our tests. And what I want to do is go ahead and create a test that will say create k-mer database from sequences, taxonomy, and k-mer size. And I know I have probably more than just that typo. And again, we're using test that to do this. And so this is kind of the structure of an individual test. And as you kind of look back through here, you can find all the other tests. And incidentally, if you want to get caught up and follow along with what I am doing here, down below in the show notes, there's links to what this repository looks like at the beginning of the episode. So like right now, as well as I'll put a link for what it'll look like at the end of the episode. And so you can kind of follow along and see everything that I'm doing. Okay. So again, we're going to create a function that creates the k-mer database from sequences, taxonomy, and k-mer size. And so if you'll recall, we've been generating sequences based on base three notation.  And so what I want to give it though are the actual sequences, right? And so here is an example of one of those sequences. But we, in the last episode, were calculating the various probabilities. And I think this has kind of the makings of what I want. But these data are still a bit more processed than than maybe I would like. So I'll go ahead and copy these down. And again, these are base three strings rather than the A, T's, G's and C's. And so over top of them, I'm going to translate one of them into the A, T's, G's and C's so that I can then give it, give this function the actual sequences, not the base three sequences. So this will be like A, T, G, C, G, C, T, A. And so just double checking that that's the G's are twos, the T's are threes, the C's are ones, and the A's are zeros. Okay, so that's good. I'll pop that in there. Go ahead and remove that. And then the only difference between this sequence and this sequence is the one at the end instead of a zero. And so again, that'll be a C. And then this is the same sequence, right? And so the other thing I want to do differently here is that I put the genera in terms of numbers as indexes, right? So here I'll go ahead and do A, B, and another B. Okay. So then I want to have a function that will be build kmer database. And the arguments that I want to give it are my sequences, my genera, and my kmer size, right? And this should then output a database. And I want that database to be a list, right? So I need to return back the, what do I need to return back, I need to return back the conditional probabilities for each genus, as well as the names of the genera. So again, what we're getting back is a table. And we saw this in the previous episode, right, where we're calculating these conditional probabilities, where we get back a matrix, actually, I shouldn't say table, a matrix, where each row is a different kmer. And so if we're using kammers of three, we're going to have 64 rows. More realistically, we'll use kammers of eight, so we'll get like 65,536 rows. And then the columns are the different genera. And so I need to know, I don't care really what those kammers are, because again, we kind of run this through a base four filter. But what I do need to know are the names of those columns, because I want to know what each column is related to or what each column represents, what genus it represents. And so I need to get back the genera in terms of the letters, not in terms of the numbers, because these seem like very, these are very simple genus names. But in reality, they're going to be far more complicated, and perhaps a little bit less predictable than like letters, right? So anyway, that's what I want the input to be. The output then I want to be basically like this, right? And so expect equal conditional prom. And so db, again, like I said, is started to say, is going to have the conditional probability, as well as the genus, the genus designations, right? And so this is going to be a list. And perhaps you haven't seen lists before. Actually, I can guarantee you have seen lists before, but perhaps you don't know it. So we'll go ahead and output the database as a list. And so this will be db. And then in here, in double square braces, I think we'll put in quotes, conditional prom. And so the db is going to have two slots, one for the conditional prom, and one for the taxonomy. And so again, I'll copy this down. And so how do I know that you have seen a list before? Well, a data frame is actually a list. So now you know, right? All right. And so then we'll also do expect equal db. And then I'll do genus. And then the first genus is going to be an a. And the second is going to be a b. This is what I expect the output to look like. I'll go ahead and run the test proved myself that it fails. Yes, I can't find this. So I'll go ahead and grab that definition. And we'll come on over to the kmers.r script, where again, we'll do build kmers database. And we'll then put in function, right? And and now we need to go about kind of putting together a lot of this stuff, right? So the first thing we need to do with our sequences is convert that to base for notation, right? And so the function that we had was where seek to base for. So we'll do seek to base for on sequences. So this as input, then we have a function further up here that was detect kmers across sequences. And so what we'll then do with this function is output that matrix that has, again, in our rows will be the kmers and the columns are sequences. And then from there, we'll get the word specific priors as well as the conditional probabilities. So okay, we'll go ahead and take this. And I think I'll go ahead and delete this commented out code. And so we can then pipe that. Yeah, we can, can we pipe that in there? Yeah, we should be able to pipe that in there. And let's see, then our kmer size will equal kmer size. Right.  And let's make sure that this looks right. Okay, we're off to a good start. We've got our 64 rows and our three columns for our different sequences. And sure enough, our favorite kmers 29 and 30 are the cameras that are different between the two genera of sequences. Okay, cool. And so from here, we now have our detected kmers. Right. All right, so now we're ready to take that detected kmers and generate the the word specific priors. And I think I can actually add that to my pipe, right. And so maybe for now, I'll hold off on calling it detected cameras. But I do need the parentheses here. So we run that. And this then gets us our word specific priors. And so again, this is reminding me that sure enough, I need to pull this out. And so what I will do is I'll go ahead and call this detected cameras. All right, and I'll end the pipe there and I'll then have priors, equaling this. And I'm going to feed it my detected cameras. Right. And so now priors equals that, which is good. And so then my priors are going to go with my detected matrix into this calc genus conditional probe. Right. And then I'll do detected cameras. And then priors. And what's the other arguments? The genera needs to go in there before that. Right. So in here, I'll do, I'll do genera indices. Right. And so I don't have genera indices yet. Right. And so I need a function that's going to convert genera, the names, the characters to a vector of indices. So we need to pause this and and we'll come back to it. But first, we need to go ahead and do that translation between this character form and what we used previously, which was the numerical form to an index. Right. And so what we can think about would be, let's see, let's do test that. And then we'll convert back and forth between genus names and indices. Okay. Cool. All right. So again, we'll keep things simple. So we'll do genera. I'll call these str. And then we'll have genera index. And that'll be C one, two and two. So I'm thinking about genera str to index. Right. So I basically want to know A is one, B is two for when we're back here, kind of aggregating columns that are from the same genus, right. And so that's where the genus index is needed. And so again, we can give that genera str. And I expect the output to be genera index, right. So we'll do expect equal on that. And then going in the other direction, I'll worry about in a minute. So let's go ahead and test that it's going to fail. Right. And we'll come back into our camera script. And again, this is going to be a function. And the input to that is going to be our genera string and index or general string, not the index. Right. And so we'll say string. Okay. And so again, like string, well, yeah, I'll say string is genera string str. So I want again, one, two and two. And so we can do a little trick here. And so there's a useful structure within our call the factor. And so we could do factor on string. And so what a factor is, it's an it's an ordinal variable. It's really a vector of values with late levels mapped to it, right. So I can take this factor string and pipe that to as dot numeric, this then returns the index for each of the levels, right. So we have one and two. And to get the levels, or the A and B, I could again do factor string levels, and get back to A and B, right. And so one would be A and B would be two. So I'm going to pull this down because that's not quite ready for that. So I'll comment that out. But I want to hold on to that for now. And I will go ahead and save that. And let's test. So I'm getting one failed test. And that's online 208, which is up here, going into the build camer database. So let's come back up to there. And let's see. I had genera indices. And so again, up here, I can do genera indices equals the genera string to index on genera. Right. So now if we save this and test, it should pass. Nope. It's not passing. Oh, yeah, of course, it's not passing. Because it doesn't know about this, right? So it's getting past that line. So again, we'll do genera indices. I need to if I'm going to run the code that's in the actual functions, I need to go ahead and load it. So I'll go ahead and do genera indices, detected camers and priors, and then calc the the genus conditional probabilities, right. So that is, yeah, so that's those two genera, right. And so I'll then do the cond prob as that. And so now I need to get back the names for the two columns. And so now what we want to do is go in this other direction. And so now what we'll do is expect equal genera s index to str and we'll do that with the genera index. And we'll expect to get back a and b. Okay. And of course, this will fail, we get two fails, right. And so the first one, of course, after that out of bounds error is this one. And so we'll come back and we kind of already have the code for that, right. And so we'll go ahead and here and put function. And then we'll put string. So let me think about this again. So what I want for the output from build camer database is a list that contains the conditional probabilities, as well as the genera that correspond to the two columns or however many columns there are in this in this matrix. And so I don't want to go from index to name, well, I do, but index of this, right, so I want a vector that has the two levels, right. So what I'm going to want to get our genera str to unique. Okay. And so here then we'll take genera str, and we'll go to a and b. So I'm going to go ahead and change the name of this function, right. And so that should work. Then we can do genera names equals and we had genera str to unique. And that was on genera. Maybe I'll give us a better name. So maybe I'll call get unique genera. And I'll spell that right. Get unique genera. So kind of as we're designing things on the fly, decisions get made decisions get changed. This is all real, right. So let me go ahead and test this. Okay, so we're failing still. So get unique genera could not find function get unique genera. Okay, why can't you get unique genera it's there. Oh, because I didn't change the name of the actual function. So right there. So go ahead and test that. And that is still failing right inside of at the output from my JIT build camber database, right. So now what we're going to return is a list of conditional prob equals cond prob. And then the other slot was what genus, right. And maybe I'll call this genera. Although that's a little confusing, but that's cool. genera equals genera names. And we'll test. Hopefully it'll pass. It fails. Why do you fail argument expected is missing with no default. 215. Okay, right here. Ah, and that's because I put in an assignment arrow and a comma. So now let's try it. It passed. Yes. Awesome. So now we have a function that will build our camber database for our sequences, the genera and the camber size. That is wonderful. So I've been testing it on kind of what we might think of as these toy tests, right, with kind of simple, simple sequences, three sequences, right, two genera. And they're really short. So what I'd like to do instead is test it using real data. And so that is going to take a little bit of effort. So on the mother wiki, I have the version 19 of the RDP training set that was released back in July of 23. I'm going to go ahead and grab this collection of, I think it's 23,000 bacterial sequences, 788 or keel and one eukaryotic sequence. So I'm going to go ahead and download this to my computer. And there's a good way to do this. And then there's the way I'm going to do it. So I'm going to do the way I do it. And we'll come back and do it the right way in a future episode. I went ahead and put that into my phyletyper directory. I'll go ahead and decompress it. And so it's got a directory with the faster sequences and the taxonomy. So I'm going to go ahead and do library tidyverse. This is something I feel like might be kind of the makings of a vignette. And so I go ahead and load the library. And then I'll do read TSV on train set, nine that and then train set that. This is going to get me my taxonomy, I think, except I don't have column names, right? So I'll go ahead and do call underscore names equals sequence. And I guess I need to make that a vector sequence. And then taxonomy. All right. And it doesn't seem to like what I did. I need to closing parentheses there. All right. So now we've got a sequence and taxonomy. That's good. We'll go ahead and call this genera. Right. And we'll go ahead and clean this up a bit. And then we're going to have to read in the fast day files. And that's going to take a little bit of effort. So what I will do is scan on the train set RDP transit again on the faster. And I'll do set equals backslash n. So this will read in nothing. Oh, and that's because scan expects the data to be numeric. And so I'll do I think it's type equals character. No, all right, let's look at the help scan file what. And so it needs what. And I think I can do character as the function. And let's so that reads in a whole bunch of stuff. Some of it I don't really care about. And so what we can do is to first. So it's got it's a fast day file where it has the name along with taxonomic information. I don't care about the taxonomic information because I've already got that. And then the sequence, right? And so I will call this fast day data. And let's go ahead and put this on separate lines. I like to have everything line up and not scroll off the right side of the screen. Your mileage may vary, but I think most people agree with that. And if you don't like having that read thing, you can do quiet equals true. The values of fast day data start with the header and followed by the sequence. And so I want to get basically all of the odd values out of fast day data. And so to do that, I can do seek one comma. And then I can do length on fast day data. And I can then do by equals two. And so then this outputs all of the sequence names, right? So I'll pipe that down to str replace all says like g sub, which I think we used in the package. And so the the string coming into it. So is going to go into that string spot. And then the pattern to get only the sequence name, I'm going to start with that greater than sign. So that carrot means match at the beginning of the string. And then I'm going to use parentheses to save the pattern. And then I'm going to do backslash t period star. And I'm going to replace it with what's in the parentheses. So I need to define what's in the parentheses by saying that this is going to be anything but a space, right? So I can do not so entitled square braces, I can define my own pattern, I can then do backslash t.  So these are our sequence names. And so I'll call this sequence names. And then my sequences, I will then do the same type of thing. But we're going to start at two, right? And so now my sequences are like so. And maybe I could go ahead and do a tail on sequences to make sure that it's doing what I think it did. And sure enough, we do. And so we can confirm that the length on sequences, and the length on sequence names are the same. And we're good. Okay. So now we have our sequences and all that. And so I'm going to make a table with a name, or I'll say accession equals sequence names, and then sequence equals sequences. So that gets us a table, right? So I want to make sure that the sequences and the taxonomy are in the same order. So I'll go ahead and do a inner join with this thing up here, the genera, right? And let's see, we'll do the stuff coming through the pipeline and genera. I guess I'm kind of mixing pipes. So I'll go back to this pipe. And this placeholder don't actually need the placeholder. And then we'll do buy equals. And you know what? I call it sequence here. Maybe I should call this accession. Because sequence means something different with the fast day data. All right. And so then I will join by accession. It's got to be a named argument. Okay, so I'm going to leave this out. I don't actually need it. It's going to join it together. And so now we've got our accession, our sequence, and our taxonomy. I think now we're good to go on running our camer function. So build camer database. And we'll do build camer database, Oculus seek table. Right. And then I'm going to give this seek table dollar sign sequence and seek table dollar sign taxonomy. And then camera size equals eight. And let's see what this does. So it's complaining because I think I forgot to load this. All right. Well, that's pretty depressing. After about two or three minutes, it crashed. And we ran out of memory. The problem I think is coming from this detect camers across sequences function, where we're using a supply to build a matrix that has the 64,566 or whatever is a large number of camers rows by the 24,000 columns for our number of sequences. I think it's just gagging because that's way too big for it to store in RAM. And something that I'm thinking about is most of the values in that matrix are zeros. So we probably don't really need to store them. Right. And so I think I have some ideas for how we can refactor this, but we're gonna have to wait till the next episode to figure out how to do that. But thankfully, we have all these great tests. And so we can feel confident that we can modify our code to refactor it to make it actually work and run without breaking anything and having faith that we're going to get the right answer. All right. So that you don't miss the exciting conclusion, please make sure you subscribe to the channel so you know when the next episode drops. Okay, take care and we'll see you next time for another episode of Code Club.


Think Tech Hawaii, civil engagement lives here. Good afternoon and welcome to Pacific Partnerships in Education. I'm your host, Ethan Allen, here in the Think Tech Hawaii studios. Pacific Partnerships is all about all the great educational work that's being done all across the Pacific around education. Education in the broadest possible sense. Formal, informal learning, science, art, technology, language, all these great things. And I have two really wonderful guests in the studio today to talk to me. Dr. Sam Noffsiger of the English Learner Group. Welcome, Sam. And my colleague, Dr. Laurie Phillips. Welcome, Laurie. Laurie and I are colleagues at Prell. And both of them are involved in a whole big set of projects involving English language learners. The whole issue here in Hawaii of a lot of kids coming into our school systems at various levels who are not native English speakers who may have spoken very, very little English ever indeed. And obviously that's a problem, right? Yeah, it sure is. So why don't you just sort of frame that a little bit for us? I think probably the better word is it's a challenge for teachers. It's a challenge to make sure that their instruction is appropriate for all the students that they have in their class. We've learned about 10 or a little bit more than 10% of the population of students in Hawaii speak another language at home. And while speaking another language at home are limited in their English proficiency. So Hawaii has a system in place that assesses children's language proficiency because certainly you can speak another language at home while still being fluent in English. People speak four or five languages, right? Absolutely. So when the schools find out through the forms that the parents fill out when they enroll their students that the student's first language is not English we want to see how much English they know. Sure. So they assess them for that to see to what level they have learned English over the years. And then that information is of course given to the teacher so that the teacher can make the best instructional decisions for them. Because it's really important that you match reading materials, conversational expectations, all these kinds of things, right? Absolutely. With the kids' comfortability, you always need pushing them a little bit to keep learning English better. That's exactly right. But you can't expect a first grader to be dropped in if he's never heard English before and be on a par with his native English-speaking peers, right? In terms of English language. Exactly. We had actually a really good conversation about that today with a group of teachers. We were talking about the need for our students to be able to talk academically in English and to be able to write academically in English. And we had a pretty good conversation with teachers in specific grade-level groups like the lower-level elementary teachers were together and the higher-level elementary teachers were together. And they had a conversation about what that would look like. What is their expectation of academic English speaking and writing with, say, a kindergarten student? And how is that different than with a student who might be in sixth grade? Beautiful. Yeah. And this goes into other areas, too. Because, for instance, your specialty is more art, right? But it means nothing, really, to some extent, their understanding unless they can communicate some understanding, right? Unless they can talk about the art. Tell them what it is they like or don't like about it or I think is pretty special about it. Same with science. And there's great evidence now that says the better your language skills are in English. For instance, the more understanding you'll get in the English science class, art class, et cetera, right? It's a deeper level of knowledge of content. I oftentimes tell the story about how students can actually, for instance, do mathematics. They can add and subtract, say, a couple of two-digit numbers, like 27 plus 39. They can do that math on a piece of paper without actually saying a single word or writing a single word. Right. Well, our new standards in education say that they have to be able to express their understanding of that math. So they have to be able to explain it and or they have to be able to write that explanation. So just doing the math is no longer good enough. They just can't understand. They have to be able to express that understanding. Yeah, that's fascinating because math sometimes is referred to as a language itself, right? Exactly. You talk literacy or numeracy, you know? And people can have very deep understandings of math. But it is that communication skill. Well, it does light the heart of it. I used to joke with a bunch of doctoral students who I was working with in the Center for Nanotechnology at the University of Washington. I said, I don't care if you invent a perpetual motion machine in your garage. You cannot share that information. You cannot talk about it in a way. You haven't really done science at all. Yes, right. Well, even just a couple of months ago at our house, we had an accident, but something broke, a pipe broke in our house where we had to call a plumber. Well, the plumber came out and he did his work like plumbers have done before in our house. But this time when he was finished, he took an iPad out of his briefcase and he had to write and explain what he did and then write the parts that he used and why he used those parts. And then he hit submit and all of that went to my email account, went to his office's email account and it was the accountability of his work. Instead of just the fact that it was fixed and everything's working well, it's a way that he has held accountable in this new technological, literate society that we belong to. Our students have to be able to have those skills even in areas of adult work that we never thought they would have to have those kinds of things. They still have to be able to do. Right, and more and more, we're also dealing with the so-called soft skills. Now, I realize that kids have to be able to collaborate, to work together on projects. It's not enough to fill out a little test on your own. You've got to really be able to say, let's do this project, you're going to do this role, I'll do this role, do that role and that takes communication skills. If I speak Chukis and you speak Mandarin and you speak only English, that's going to be tough, right? And that goes to the other kind of topic that we've been talking about is the notion of the cultural implications of that. Our communication, whatever language we may speak, oftentimes reflects the culture from which we are. And even across the United States, even though we speak English, depending on the area of the country you live, it reflects a bit of a different culture. And so we did a lot of conversations Lori and I did during the classes that we've been doing about the importance of taking this learning that students are doing and making a connection to their culture and saying things like, in this particular story, there's this character and it's similar to this or asking them to make those kind of connections. Tell me what is the same in your family, in your community, in your culture. That takes us into some of these examples of artwork, right? Exactly. In the case of what's on the table, this is just out last week and basically some of the islands in Micronesia have 400 children and let's say that you're looking at just five and six-year-olds, that means there's only 72 on the whole island that speak Kosherian. So the problem is they have no books because what publisher is going to write a book for 42 kids in the whole world, right? So I had a thought lately, well, what if we had the kids illustrate their own books and then write the early childhood readers, let's say the teenagers and in this case these were done two weeks ago and I just went to Guam and they did images of their island, things that were important to them and it could be anything from coconut candy is being made to a visit to Namandal and Ponpei but you immediately have something that kids can connect to and we get around the problem of being able to publish. So we will print these out, put them on a website and teachers from all over the world that need a Kosherian book will be able to do that. This is becoming more and more important because there are now substantial Micronesian populations in places like Arkansas and North Carolina and Oregon where the teachers are, if anything, even less prepared than the Hawaiian teachers are. The Hawaiian teachers at least have been dealing with a pretty diverse population but in Arkansas when the first Marshallese showed up there the teachers had no idea what to do, right? But now with something like this, they can pull up a book like this that's written in Marshallese and then they can compare it with the other kids. Think of the difference, let's just say in California where Sam is where you're talking Spanish and so many books are written in Spanish but when you're talking about small island communities not only is there life, they don't have zebras so if you say the word zebra they're not and also they just have so few people and so few people that speak that language and even if they do speak Yappies there's like four or five different types of Yappies all different languages. So this gives us a way to immediately have them translated into the language of that island and then very easily just substitute the words out. This is beautiful, a nice use of creative thinking and art and technology all sort of merging to be a solution for an ongoing problem. And it's more of a problem I guess maybe in the Pacific Islands because there are so many, there's 20, 25 different languages just in the U.S. affiliated Pacific Islands alone and they're all rather different from each other and they're completely different from English, right? Whereas Spanish and English at least have come back to a common root and you can make some sense out of it. There's some cognates, right? The kids can really build on. You can actually, there's so many that you can actually pretty easily identify what the cognate is and if the student has an idea, a conceptual understanding of that word you don't really have to reteach the concept. It's just a new label for that word. Right. For Yappies, it has nothing really in common with English. Very different. But what is interesting about books like this, and Laurie and I were talking about it last week when she was showing me the books, is that when you start writing down these kinds of books where oftentimes they have patterns from picture to picture it might say something like, maybe they're talking about a particular game that they play with their family or something. So it might say I like this game and then on the next page it says I like this game and the next page it says I like this game. There are patterns that you can start identifying that you can really help kids to understand. I like, for instance, might be very similar from page to page. So that kids start seeing those patterns and even though they're very different than what English would either sound like or look like, they start recognizing goodness, this looks just like it said in the next page, that kind of thing. So these books are examples of how that kind of pattern of language can really be acquired by students and done in a way that's very meaningful. Instead of saying I like pizza or I like something like that that they might not know about, I don't know. But they're using the content that is most appropriate for them that they can make those connections in their culture. Excellent. Hey, we're going to dig deeper into this when we come back. Okay. Lori Phillips, Sam Noffseger, we're here on Pacific Partnerships and Education talking with me, your host, Ethan Allen. And we're going to take a brief break and then we will be back in one minute. Great. Well, that's fun. We're rolling now. Do you want to be cool? If so, watch my show on Tuesdays at one called Out of the Comfort Zone. I sang this song to you because I think you either are cool or have the potential to be seriously cool. And I want you to come watch my show where I bring in experts who talk all about easy strategies to be healthier, happier, build better relationships and make your life a success. So come sit with the cool kids at Out of the Comfort Zone on Tuesdays at one. See you there. Hello, I'm Yukari Kunisue. I'm your host of New Japanese Language Show on Think Tech, Hawaii called Konnichiwa, Hawaii, broadcasting live every other Monday at 2 p.m. Please join us where we discuss important and useful information for the Japanese language community in Hawaii. The show will be all in Japanese. Hope you can join us every other Monday at 2 p.m. Aloha. And you're back here with Pacific Partnerships in Education. I'm your host, Ethan Allen, here on Think Tech, Hawaii. With me today in the Think Tech studios are Dr. Sam Noffsiger and Dr. Laurie Phillips. We're talking about English language learning, the challenges, the successes of programs, how to take successful programs such as Sam's been doing for Hispanic groups in California and work with them, work these same strategies into Pacific Island groups who have very different languages and all. But the issues are still the same. The kids have got to learn languages. They've got to get master, they've got to get fluency, so they can read it, so they can write it, so they can speak it, so they can communicate effectively. They can communicate or understand what you were saying earlier, right? So Ethan, one of the reasons that Sam is here with us, though, is because there's sort of this idea that possibly all teachers can become English language teachers now. As an art teacher in a high school, that's the last thing I want to hear. But with the strategies that Sam has and that are ELL strategies, exciting, more creative ways of doing it, you actually can teach language while you're teaching your subject. So I fought it for a while until I tried it, and I realized that the knowledge of art, to get a kid to speak like an artist, to speak like an artist's story in, is really fun, and that they have to have certain, I learned today, things called academic, language within art. So when I use the word value in art, I know right away, I mean dark and light. Value means dark and light. But in any other subject, value in math is different. Yeah, value in science, I don't know what it is. But, you know, we could take any, if I was a science, your science, I'm art, your language. We could really play a game and say, if we wanted to teach, if I was a photo teacher, and say, if we wanted to teach, if I was a teaching photosynthesis, what are the absolute academic words, science words, that I would have to understand, to be able to have children understand photosynthesis. Once I know what those words are, as the art teacher, I'm good, because now I can put those words in the visual sense, and I can put them in books, I can have kids write poetry about it, I can have them tell a story about it, but I'm not doing any good if I'm doing a picture of the pilgrim cover for their report. That's not it. It's how do we pull the academic language out of the image, out of their environment, list it, and explicitly teach vocabulary through it. Yeah, no, and it's exactly it. And a lot of the key ideas in science, they have what are called cross-cutting concepts. And for instance, what I'm called, it's size and scale. Obviously important in art, how you size and scale. Systems and integration. Again, a painting is done as a system parts and a sculpture clue as a systems idea to it. Yeah, I mean these kind of ideas translate across the disciplines equally for a story, for a piece of writing. Can you imagine what that means, though, to ELL teaching? Because from my understanding, I'm not the expert in it, there's kind of two ways to do it, if you're, one is you can pull the kid out of the class and put him with everybody else that doesn't speak English. Or you can leave them in the class with the art teacher, with the science teacher. And the science teacher is just using these wonderful strategies to bring out the language and it's not going to hurt the gifted kid. Right. It's not going to hurt the kid that just got off the airplane, because in both ways, they're using language to learn within the content. Exactly. Exactly. Why, for instance, to take a science example on the periodic table, why is a symbol for lead? PB. Uh huh. It's a plumbic. Uh huh. And the Romans used lead for making the original plumbing. Uh huh. There it is. Uh huh. I mean, it would be in a pile using the back together. Uh huh. You can type your history, you can type. Right. Yeah. And how all those elements are typed, they're on that table for a reason. Right. And their location is for a reason. Exactly. So having those conversations about how this element relates to that element, there are words that go along with that. Right.  Right. So you can't really talk about science without using those words. Exactly. And so when you identify what those words are, then what we do with students is we say, okay now, we want you to explain the scientific notion that we're talking about. But while you're doing it, you must use these words. Right. And gives them an opportunity to practice using those words, maybe with a partner before they do it out loud, or maybe in a group where they kind of figure it out together. But the accountability, the academic language accountability, is about, you can't really explain the science without using the words. Right. So you've got to use the words in your explanation. Right. Yeah. I mean, there is that whole sort of philosophical argument about you can't think without words, right? That's exactly right. That's the same thing. Right. Right. I oftentimes, when I talk with secondary teachers, there are certain teachers that kind of are a little bit more open to the conversation than others. But I'm a musician. Okay. And I will tell you, there are words that I know as a musician that other people don't know. And I better be using those words as I'm doing my music.  It's not the instrument. Right. It means something else. Right. You know? And so, as a musician, I have to be able to use those words to read them, understand them and do what it says to do, but also turn to my friend and say, it says piano. Play piano. Right. Which means soft. Right. And in the context of music, it means that. Right. So the notion there is, is that we can't really, I can't really be a musician without the language that is associated with that. Absolutely. I once, a few months ago, I put my head in the middle of a high school football huddle. And I've never played high school football. Those guys were talking English, but I didn't really understand much of what they were saying. They have their own code. Right. Their own language. Exactly right. And again, science has the same thing. You look at, you look at things that, hey, it's a tree. But, you know, what kind of tree is it? That's right. You can talk about it. It's a deciduous tree or evergreen tree. Exactly right. You know, da-da-da. You know, the scientist has a lot, as an expert, has a lot deeper, more fine-grained knowledge of that and has to be able to communicate that more often. It's what I often times tell teachers and my friends, you can really tell, you can turn on the television or listen, turn on the radio or turn on the internet and you can really tell who the expert is by how they can talk about their expertise. And that's one way that they express their knowledge. Others do it through writing, but certainly, you can really tell if someone knows. Yeah, it's a very hot area of research now is to try to tease apart how is it that experts think about problems versus novices approaching the same problem. And experts, yes, rely on a much greater realm of knowledge, much deeper connections to more diverse things. And they can build solutions more quickly. But everything, though, that Sam has sort of explained is creating a safe space. So, for example, the other day, I was with a child, we take photographs and we label the photographs and then a little girl found an apple that was half-eaten in the curb area. She took a picture of it and she held it up, but she said, apple bone. And I was not going to touch that with a ten-foot pole. That's a much better word than apple core, right?  That's exactly right.  And somebody said, pebble. And then he immediately, he got it and he used the word pebble. So, we don't have to be smart. We just need to be developing. This gets back to your idea of the cognates and helping people make those connections that they suddenly understand. Oftentimes, it's not the concept that we have to teach kids. Because a lot of our kids have a lot of those concepts already. We just have to give them the academic label for that. Right, exactly. And once we do that and we ask them to practice it, it's great. You know, Sam, so you don't sound like a snob. Yes. Why don't you tell Ethan the story about, you know, because we have pigeon here, right? Yes. So, you're not wanting to talk academically all the time. Of course not. Of course not. A large part of our conversation about the notion of academic language is that, well, I'll just kind of use this example. When I was a teenager, I would go to my father and I would talk to him. I'd start a conversation with him and he would turn to me pretty quickly sometimes and say, you don't talk to me like that. In other words, I was talking to him as though I was talking to my friends. Okay. And he taught me, basically, I'm your father, you don't talk to me like that. Right, right. And I think it's that notion that we have to help kids understand that the way I talk to my father is one way. The way I talk to my friends is another. One is not better or worse than the other. It's just different. Right, there's the way you talk in school. Correct. Academic language. And I have to be able to turn it on when I need to turn it on and turn it off when I need to turn it off. Right. And because so much success in so many different levels, so many, so many transitions depends on a good grasp of English in this current world. Yes. It's really important that we help kids make those transitions, be able to take their perfectly good understanding in their own languages and express them in English. And they have to have those tools. And they have to have those tools available to them so that they can use them when it's the appropriate. But again, I think what's most important is when we don't change the Apple Bound. No. So in the arts, in the storytelling, on the classroom, it's great. I can imagine saying to a child, I want you to explain to your friend in pigeon photosynthesis. Mm-hmm. How would that be any less valuable? It's not. More of a, you know. No, it's not. It might be more comprehensible to the other kids. Right. It's just that the kid now is able to go from academic to pigeon to classroom. And that kind of metal flexibility is powerful. That has been shown to be the evidence shows of bilingual kids. Better problem solvers. Better. Better. And they have different, they can put themselves in other perspectives. As they get older, they are much less likely to end up with dementia. That's right. All those things. They're better to maintain cognizance. All in all, it's an amazing array of good stuff. And it points how powerful this is, these connections between the language, the art, the science. And the culture. And the culture. Right? I was telling Sam that there was a story not too long ago. I don't know if you remember who he was at Prell, and he was one of the directors of education, and he came to us and he said, I'm going to give you a word, I can't remember, like Oop Long or something, some Pona Pan word. And he said, can you read that? And we all read it and he said, do you think you understand? No. Let me, can you spell it? Can you spell it?  No. Let me give you the definition. It means shampoo your hair. And we said, okay, all right. Shampoo. And he said, now do you understand? I said, yeah, I know the definition. I can spell it. I can read it. And he said, can you put your shampoo on your hair on Tuesday if you're pregnant and you, you know, in a certain river in Pau Pan, right? So until you've experienced that culture, you really The detail there. Can not understand the deep meaning of things until you've been among and around. It's great. We have educators like you both working with our kids here because it's so important. Lori Phillips, Sam Mozzagher. Thank you so much for joining me here. Thanks for joining me. show today. It was wonderful. I learned a lot. And I hope you'll come back and join us in two weeks for the next episode of Pacific Partnerships in Education. Until then...


Thank you very much and thank you very much for coming and Hope you enjoy the talk. I have lots and lots of slides. I'm gonna put the other slides up online I'm not gonna read all the stuff that I have on the slide. So Let's get going. So who am I I'm I'm Mark Lamberg. I've been with Python for a very very long time I'm the former chair of the your Python Society Also passes off a foundation fellow. I have my own consulting company and if you want to connect then This is the LinkedIn QR code if you want to scan it or you can go to the website down there MA Lamberg Right so Motivation so Let's assume you're you have the great idea for a startup, right? And you want to build the next big thing of course what you need if you want to build something That's really big and you don't know how big it is going to get you need to make a good choice About which database to use underneath, right now If you've been in the business for for a long time then You probably know that there are quite a few, you know standard options that you have But more recently in the recent years, they have been huge developments in the space of databases So if you go to this website DB DB dot IO, which is the database of databases It's it's a quite useful thing because it compares to different databases features and so on Makes it easy to to have a look around and to shop around for a database Then you find that there are more than 800 databases available for you to use now So it's it's not as easy to choose or it's not as straightforward to choose a database anymore and The reason why this is is that lots and lots of companies especially the well What used to be the small startups and now are the the big huge internet companies all those companies had an issue When they started their business because they they found that the databases that were on the market at the time Did not really fulfill the needs that they had that they weren't as scalable for example They didn't really Focus on the specifics that they needed for for their business And so what they did is they started to take Available open source projects and then customize them modify them and made them more scalable So now you have lots and lots of different databases for different types of workloads that you want to handle Of course, you can always go and just you know pick the standard ones like my sequel Maria DB or Postcards, which is actually my favorite How many of you know these databases? Everyone excellent So those are the obvious choices, right? So let's assume you want to have something that you know scales may be a bit better than these databases and You are going to the market and you don't want to lose a competitive edge that you have in with your startup so You probably have to approach this whole thing a bit in a bit more systematic way So you have to answer a few questions about what you want to do with the database You have to Find out which features you need and then you have to go out and shop on the market to see what's available And then you know pick the best fit for for your particular use case For larger projects, I would always recommend doing a PLC as well Right because sometimes, you know, even though the you know the marketing slides of the different databases look nice They may not actually perform in the way that are they are being advertised for so it's always a good thing So first question is what's the typical data load that you're going to have what kind of work Load are you going to put into the database? So that basically two major different types one is OATP online transaction processing And the other one is OLAP, which is online analytical processing. How many of you know these terms? Okay about half so the first is basically mostly about what you typically do with a database with the general purpose database So you put stuff in you edit stuff you delete stuff Data changes a lot in your database Versus that the second one the OLAP case is more about analytics So you put in data into your database and you want to get reports out of it And this is also what Maro talked about So you what you first do is you collect all the data and then you put it into an OLAP storage to do the Analytics and then do the business reporting So a typical use case for that would be a data warehouse solution So once you have that figured out what you need and it may actually be that you need to databases because you want to have both Loads handled and the typical case is that databases only good at one of these Of course, you know The database vendors and the companies working on these things they try to make both work nicely But there's always a trade-off in these things so you have to be careful of you know when choosing these The next is you have to figure out what feature set you need from from the database because there are a lot of things that You might need There are also unfortunately a lot of things that You may need in the future, but you don't have any clue of yet You only figure out when you start developing and so you have to plan for both the knowns and the unknowns when doing the selection Now when it comes to the feature set I put together You know a few slides explaining different complexity dimensions that you have to consider when choosing a database So I'm just going to go through all of these This is not necessarily complete you might have additional ones So it basically depends on what you're looking for the first one that you have to figure out is whether you're going to Have mostly simple schemas like you know web server for example a logging server or something like that Or whether you whether you want to have complex schemas Let's say you're building a big huge data warehouse that's supposed to handle all the the data In in your particular business then it's more likely that you're going to have a complex schema versus if you just want to do Some quick processing. Let's say you have a web shop to handle then simple schema would be More important for you now when it comes to choosing the schema This is probably fairly straightforward for you What's not so straightforward is when making changes to those schemas So you should have a database that allows you to implement migrations efficiently if you Have a need for for changing its schema afterwards, which you typically do I mean even if you know exactly what you want to do at some point You're going to find that you need more fields or more tables and you have to have these migrations So the database should should probably have features for these things next is of course Cardinality, so how many how many rows are you in the store? How many are you going to have tables with lots and lots of columns? What's called a white table? Or are you going to have you know both lots of rows lots of columns? And of course you need to figure out how fast is you're going to is the data that you're going to put into the data Storage is going to scale so how much new data are you going to get every single month and that's very important because Databases they they often they they hit a certain wall after after some time. They're not necessarily Elastic in terms of scalability So you have to be careful about that another point is temporal complexity Maura already Mentioned that like for example if you wanted to time travel in your data So you want to know what did my database look like on January 1st? No 2020 That is a question that some databases can answer and others can sometimes you can you know build something on top To make that happen, but this is this would be something that you would have to figure out before choosing a database Right next is queer complexity, which is fairly obvious. So of course you want to well Typically you want to have everything as performed as possible, right? So performance matters a lot, but performance comes in two different Categories so for old OLTP it would be transactions per second For olab, it's more important to know how quick is the database going to answer my question And this is especially important for bi reports that are Whether the user doesn't want to wait like half a minute or so for the for the answer So that's something that's very important joints are very important as well If you have lots and lots of tables you denormalize Sorry, you normalize everything. So you basically use the drive principle for databases Then you typically have to do lots and lots of joints Now if you're working in the olab space, then it may be better to not denormalize to not normalize everything But instead to denormalize certain things Denormalization means that you duplicate data across multiple tables Which of course is not dry But it makes the queries run faster because the database doesn't have to do all these joints on huge tables anymore So that's something that you need to consider As well and something that in in in recent years has become very attractive is to use views for these things Especially to if you are working in larger organization You don't want every single developer to write his or her own sequel of select statements But instead what you do is you prepare views on the data and then manage the views centrally and Make those available to developers now Views are basically just hints for the database of how to do the select statement But databases have now have materialized views as well So they can dynamically turn the view on it on the data that you have into an actual table And then the table is being managed by the database Of course, you have to tell the database when to update that table every now and then You can also do that using special indexes again. This is something to consider Next is deployment complexity So you want to know where to run your your database whether it's like on a single server It's maybe a cluster. Maybe you want to have something that works in Kubernetes You pretty much have all these options nowadays with databases Where it comes to storage is a different question. So you have to know how much that they are going to put in into the database If you want to have all the database all the data local to the engine the database engine Then it's usually better to have something, you know, even in memory or maybe on an SSD But the trend goes to basically separate compute from storage So that you store everything on S3 for example Because that's a whole lot more scalable if you do a data lake for example in S3 is probably the right choice Depends again on what kind of data you're looking at Resilience and disaster recovery very important if you're working on a database that is, you know Basically the basis for you for your business you want to make sure that everything continues to work even if the database goes down Or you you know have a disaster like I don't know data center goes on fire or something like that Then you need to figure out how that is being handled by your database operational complexity so You have to decide whether to you want something managed So someone does the work for you or you want everything self-hosted their pros and cons to these things Nowadays most people tend to go to manage services So let's say you go to Amazon and then use a redshift for example Then you don't have to worry about the how the database is actually managed how the hardware is Composed or you use the self-hosted version and then you have to you know figure out What does the the sys admin what do the DevOps teams need to know about? Are the integrations for these things available for your database and then of course every now and then You get a new database version You have to handle upgrades and you have to know whether you can do everything in zero downtime whether that's possible or not Whether you can afford having you know a bit of outage or not And you have to basically make everything work in that kind of kind of scenario as well Right. Those were a couple of interventions to consider like I said there might be additional ones I also put up this slide with some additional decision factors. So what Maro already talked about it's very important to figure out whether this Projects that you're looking at whether they're mature enough whether there's going to be support for the next 10 years or so Right. Those are not necessary for especially for open source projects. Those are not necessarily easy to answer If you get a managed solution like from Amazon for example You can just ask Amazon for this, but if you use something like let's say an Apache iceberg or so then The the answer may not be straightforward. So I've I've seen lots of these projects Basically be hyped up very much for let's say five years or so and then people lose interest and then walk away And basically the projects they they die even though it may not necessarily be bad project But people just you know go to other things Of course because you want to use Python you should always look for good Python interfaces available for your for your database, right? So Enough talking about, you know, the the theory behind it. Let's let's have an actual look at these databases So I just you know, I'm going to show you a few selections different kinds of databases to to consider Let's see how far we we can get in in the bit of time that we have so general purpose OLTP databases, you know the standard ones I'm not going to talk through these so you can get all of them to basically install on your own servers And you can get cloud versions of these as well More interesting and in some use cases are in memory databases So basically where you run your the the whole database inside memory provide you have enough memory Of course, you can use SQLite for that you can use duck DB for that SQLite It's more OLTP and duck DB is more overlap. How many of you know duck DB? Just a few it's definitely something to work to have a look at. It's a very new database. It's a very The it comes from academia, so they they actually put some really nice techniques in there Then there's a commercial one called excess soul. How many of you know that one? Oh Few that's nice. So this is this is a bit different than SQLite and duck DB It's an it's an in-memory database works in a cluster and The the the engine itself only works in in memory. So it loads data from persistent storage and then works in memory is extremely fast Lots of companies use accessor for example for caching stuff if you want even faster and you have a good budget and And you know a good link to NVIDIA to actually get those GPUs Then you might want to use one of these GPU base Databases I've I cannot really say much about these things because unfortunately I don't have the huge budget to try these out, but they're supposed to be extremely fast And also extremely scalable. So heavy DB is just got renamed. So it had two other names before I don't remember This is a very fast one They're putting lots of energy into that if you want to have something that works in the context of the rapid Stack or you want to work with desk and blazing SQL is something to look at. So That's actually written in Python. So that's interesting and there are extensions for postgres For example, if you're using postgres already, you might want to have a look at that extension and then or you can use scream if you are more into, you know doing analytics ad hoc analytics where you have to load lots of stuff in a very fast way and then do the analytics and then basically throw everything away again. So Next is data warehouse OLAP databases. So again, all the big names like snowflake, you know big query Azure has something called Synapse Analytics Amazon has Redshift Green Plum is a nice development. They are using a postgres and basically extended that to to work in a cluster It does MPP so massively parallel processing This is coming on strong terror data is one of the the old-timers in this field. So it's probably one of the first data warehouse Systems that you had nowadays is basically just a cloud platform. It used to be a System where you basically bought the hardware from them and appliances and put those into your data center All of these are column oriented, which is something that most data warehouse applications do Because for OLAP if you have if you store your data in a column oriented way Then you can get at it much much faster and do analytics much faster Same goes for data lake As as Mara already mentioned data lake is basically you throw everything all the data that you have or all the data That you can get your hands on you throw into a system called the data lake database The storage is typically on something like S3 because you want everything's scalable You often have to deal with unstructured data. So you can you have to be able to put that into your data base as well like you can use data lake for example to do this Apache has a nice database engine Which is actually just the engine and then you plug in the storage later underneath. So you can use data lake for example with that Apache has Presto and Trino. Those are actually two projects which forked at some point so I Can't really tell which which are those two is the better one I've heard lots of good things about Trino. So perhaps that's something that you might want to look at Amazon has Athena for this so you can basically throw everything at S3 And then you put Athena on top of it and do the query Now if you want to do Distributed databases, so you can use these for data warehouses or data lakes as well and You want to put your database on on lots and lots of cluster machines and you want to have scalability So horizontal scalability in your database, then these are nice databases to look at Juggerbyte DB is a very hard database at the moment. Cockroach DB is basically a direct competitor to them The two are Postgres compatible So if you have an application that already has a Postgres interface Then those are good choices and like I said, they're easy to scale if you need more power You just add more VMs to it or more containers If you have a need for my SQL compatibility, then single storage probably a good choice Clickhouse is a nice database. It's very fast, but doesn't have all these SQL features So some of these databases they focus more on You know specific things like Clickhouse for example is good for log analysis and then again ExaSol extremely fast and Good for you know, if you have lots of RAM Right, that was it in terms of the classical databases then of course you have document databases So let's say you have unstructured data or you have data that you have in form of JSON documents Then you can use any of these so elastic search. You probably know open search who knows open search Just a few so open search is basically a fork of elastic search By Amazon and that was done because elastic search changed the license, right? So Elastic search used to be a patchy license and open search still is a patchy license So depending on and on what you want to do you can use either one of those Cassandra is very nice. It's a very hot new database for documents as well. It's high performance. You can use SQL to courier Monger, you all know right couch DB is is basically is more specific if you have like let's say you have a mobile application Then couch to be is excellent for doing the replication across devices So let's say your mobile goes offline it basically you store everything couch to be a next time it comes online again And then does the synchronization with your back end? So that's That's good to to use time series always very important for event data So in flux DB, you probably all know a patchy through it. You might not know how many of you know drew it Just a few so that's a very well, it's kind of new It's extremely good for fast intake similar to influx DB It was originally written for for add systems Where you have you know lots and lots of data come in and then you had to make you know very fast queries to Decide who gets to show you and add on a website Create DB is interesting because it combines elastic search for the document side of things And then it does pressed or three know for sequel So that's something very nice if you use postgres you can use time scale DB, which is an extension to postgres Lots of people are working on it. I've heard lots of good things about it. So definitely something to look for If you're more into a space where Immutability is important or you have to prove that something happened in your database Let's say for auditing purposes. Let's say you have regulator Come to your company and then you know check your records Then you should definitely have a look at for example emo DB Which is which also has some crypto stuff built in so they actually sign the changes that you do in the database And then store that Or you can have a look at Apache Pino, which is it's append only so you can just throw data at it and it will store it very fast and then finally machine learning and Databases so how many of you are doing machine learning inside the database? No one If you think about it actually makes a lot of sense because the database already has all your data, right? So why move it out of the database to wherever and then do your panda stuff on it or do your Machine learning models on it and then put everything back into the database again So why not just you know have everything happen directly in the database now with postgres and with a green plum This is possible using the the MAD lip so you can actually use all the the Libraries that you have for Python, but they will run directly inside the database server So it's a it's a lot faster. It's a lot more efficient other databases have support for this as well But you have to always then use UDFs for this or you have to you can write your UDFs and Python for example And then interface to to these libraries where this mad lip is already optimized for these kinds of things There's a special database which is kind of interesting because it works as a proxy to your your actual back-end database and What what you can do there is you can actually do the machine learning inside sequel So this is this is nice for people who don't know Python, but who know sequel very well As you have in you know a data engineers for example many many of them They know sequel very very well and or feel much more comfortable with sequel rather than you know going to Python to do the machine learning and what they have done is they basically they parse the sequel they added extra elements to sequel to enable machine learning and Then people who then don't know Python can then use sequel to work in the machine learning space So that's very interesting Of course, there are lots of other things that I you know can't fit into the talk key value storages. I haven't covered there lots of those Geo databases again lots of those graph databases Neo4j for example Vector databases if you're for example storing machine learning models, then vector databases are a good thing to do If you have lots of those if you want to have then have like a data catalog of all your your models And those are something to consider There are plenty of other use cases very specific So you might want to shop around a bit So this is everything I mentioned in this talk. This is just a small selection Like I said, they are more than 800 databases out there. So I could only cover some of them. I Hope this was useful if you have questions then please come to the mic and I guess the main takeaway from the talk is you should never stop to learn and always try out new things, right? Okay So we have lots of time for questions if there any in the room, please come to the mic and You'll give me a signal if there's any on remote No questions. Excellent. I covered everything Okay, very good then once more Okay This is really good talk. Thank you Just want to hear your comment on the modern serverless Databases like plan scale ends of those type that are right now getting pretty hot Right. I've used them or I haven't used them. No, but I mean serverless is coming on strong, right? The problem with serverless is that it's data locality.  You can even do that on demand You always have to get the data to the engine, right? And so getting the data to the engine is this the bottleneck in those cases if you have Queries that don't need a lot of data at the compute node, then I guess it works out fine But otherwise you introduce lots of latency because you have to get the data from let's say s3 For example to that particular node and then run the query and then you know get rid of the data again Graph, yeah, I would probably use Neo Neo 4j for that and That's basically the standard go-to kind of databases for graphing right Thanks for the talk. I just wanted to get your opinion on maybe using a database outside of its intended use So for example, like elastic search using it as a maybe a reporting database Is that generally like remanded or there are a lot of pitfalls that could It is specifically for for elastic I think there are certain use cases which are which elastic covers really well For example, if you if you do lots of text kind of oriented queries then elastic search is perfect for that So even though other databases, for example postgres has extensions for for doing text-based search as well because Lucine is Directly built into elastic search. It has very very good features for for that particular space So if you're doing something like for example elastic the dear I think the original use case for elastic was to to store log data from various different systems and then you wanted to Query your log data based on the text that was written in the log messages, right? Because it wasn't always Very structured log messages can have various different, you know formats and syntaxes and so on So that made a lot of sense And so if you if you have a specific use case in that area then you should definitely use elastic for that or you know Open-search one of these document databases. I wouldn't say that there's much of a pitch pitfall there Elastic is certainly scalable to, you know, huge data sets. So I think it's a good choice Yeah Using postgres, but the performance wasn't quite there for the volume right performance was a bit better on elastic search But yeah, definitely. So if you have lots and lots of data Like logdecker for example, if you have like a terabyte log data, then I would definitely put that into elastic search into the query They are rather than putting everything into a postgres and then have one of these indexes, you know built on top Yeah, thank you very much Do we have a remote question? No, okay. I have one question before we go Are we using snowflake now and my company? This seems to be a trend towards this kind of scalable Warehouse type setups. Do you see this as a sustainable trend and are the other big monoliths Pivoting in that direction Well, yeah, I mean slow snowflake is like the go-to solution if you if you're in a company and and your And you want to do a data warehouse, right? So everyone picks snowflake I'm not sure whether it's a good choice. I mean certainly a very capable database and it was it's a new database. It was written from scratch Specifically for doing data warehouse kind of workloads but you know, it's just a single company and You never know where that company is going to be in say five ten years So it may be good to also look at other Projects plus if you have more specific needs in terms of performance, for example There are some new Apache projects out there like Druid, for example Which are actually a lot faster Yeah, so it's definitely worth having a look at those things as well. Okay. Thanks. So if there are no other questions We'll end it there. Thanks Mark Andre


Special operations is an overarching umbrella. We've got Navy SEALs, you've got the SOCOM umbrella, AFSOG, MARSOC, all these different units. But truly there is only one special forces. That's the US Army Grand Brace. I had a very, very fortunate privilege career. Every time the ramp goes down, every time I get on that helicopter, being able to jump out of the aircraft. This is not a job to me, it's a dream.