Of course, Greta met with Zelensky. Notes from the Edge of the Narrative Matrix. Foreign conflicts in the 2020s are always like, USA, let's militarily encircle this large nation. Large nation. Hey, stop. USA, let's heavily arm proxy forces right on their border. Large nation. We're drawing a red line. USA, let's cross the red line immediately. Large nation. Axe. USA. Oh my God, completely unprovoked aggression. Of course, Greta Thunberg met with Zelensky. Of course she did. That was the only box left to check off in the most PR-intensive proxy war of all time. They got Bono. They got Mark Hamill and Sean Penn. They got appearances at the WEF, the New York Stock Exchange, and the Grammys. They just needed Greta. As with 2016 and 2020, by far the largest US election interference in 2024 will come not from Russia or China, but from American oligarchs and empire managers. This is treated as fine and normal because American oligarchs and empire managers are the nation's real government. US election interference is an inside job. We've been seeing this illustrated with RFK Junior's censorship by Google-owned YouTube. News of Tweet by Robert F. Kennedy Jr. YouTube just pulled another of my videos with former New York Post reporter Al Guart. People made a big deal about Russia supposedly manipulating internet information to influence a presidential election. Shouldn't we be worried when tech giant corporations do the same? We also saw it illustrated recently when Obama's acting CIA director just casually admitted to using his intelligence connections to orchestrate a blatant PSYOP to manipulate the 2020 election using false information, and literally nothing happened. It was just accepted as fine and normal. It's only illegitimate election interference if unauthorized foreign powers do it, or if ordinary Americans do it. When US oligarchs and empire managers do it, it's just the normal thing that's supposed to happen. A 2014 study found that Americans have no idea how bad income inequality in their country really is. Americans believe things are vastly more equal than they are. This is because Western media never report on the class warfare that is being waged against the working class. The media are owned and controlled by extremely wealthy people who have a vested interest in keeping everyone from looking at class warfare. And the most influential media employees share this interest because their empire apologia has made them rich as well. That's why they keep everyone focused on culture wars, irrelevant partisan spats, and other vapid nonsense. The US empire is like the mind of a highly dysfunctional person. There might be parts of it that sometimes say, I shouldn't be like this, I should change, but the way it's wired points it toward destructive behavior. There are significant parts of the empire who are acutely aware that things like military overextension and ramping up aggressions against China run directly against its own interests. But because the forces which pilot the empire's actions are pointed in that direction it's happening anyway. The empire keeps waging destruction, the way an addict keeps using. The reason you seldom see people change despite their stated intent to do so is because your behavior doesn't change just because you know it should. It changes when you fix the underlying forces within yourself which drive that behavior. It's the same with the US empire. The US empire is inseparable from the forces of neoliberal capitalism, war profiteering, and unipolarism with which its true leadership has intertwined itself. So while the odd empire manager may say, end the wars, it never happens because everything in it is oriented toward war. This is the same reason we keep destroying our biosphere despite being acutely aware that we need it to survive. Every system we've set up to drive human behavior and organize human civilization is pointed toward ecocide, despite all the science saying that's a bad thing to do. I know a lot of people are worried about neural implants turning the public into mindless servants of the powerful, but if it makes you feel any better the powerful have already achieved that with propaganda anyway.  Individual suffering and dysfunction arises from believed mental narratives about who and what we are, eclipsing our awareness of our boundless and indivisible true nature.  Just as it's possible for the human organism to shed its unwholesome relationship with mental narrative in the transformation commonly known as spiritual enlightenment, it's possible for the human species to shed its unwholesome relationship with mental narrative as a collective. Only then will we achieve maturity with these recently evolved large brains of ours and become truly capable of happiness, harmoniousness, and health.


OK, everyone, welcome back. So this next session is actually a workshop. So we did FluentCon in Europe earlier this year, and one of the big pieces of feedback that we got was folks want some more hands-on sessions, be able to get their hands dirty, try things out, experiment with stuff. And so we thought, let's go ahead and build a session that does two things. One, exemplify some of the new behavior we've added within Fluentbit. Two, gets you familiar with other cloud-native technologies. And three, something that you could try out live. So if you have your laptop, awesome. We would recommend having a Docker desktop, Docker Compose. And we'll be walking out through the room. And of course, if you want to try this later, or if you're joining us virtually, we'll have a GitHub repo that you could go along and follow along for. So we have about like 10, 15 minutes of slides. The overall demo, getting the workshop running, should only take about five minutes. But then we'll demonstrate some of the capabilities, showcase how you can start to tune this within your own site. So we'll be watching the Slack channel, we'll be watching. And if you have questions, we'll have to make this more of a discussion workshop versus us just presenting. So with that, let's get started with just some quick intro slides. And so again, I'm on a rug and also joined by Gibbs is from Chronosphere. So let's go ahead and get started. Yeah, so our agenda, we're going to go walk through Flimpit, Prometheus and M3. Oh, am I starting? I'm not presenting here. There we go. Okay, so agenda. We're going to walk through Flimpit, Prometheus M3. Give you a little bit of what the hands on workshop overview looks like, what you might need on your laptop. And then within the workshop, we're going to set everything up, walk through it. And of course throughout the session, please, if you have questions and answers, we'll be here and we'll be looking at the Slack channels as well. Okay, so I wanted to just talk a little bit about Flimpit and metrics. I know in the previous session, this was mentioned a little bit about a metrics are brand new and we've been doing some great things with them. But actually with Flimpit, one of its main use cases when it was built in 2015 was for embedded Linux, things like wind turbines, other Linux machines, things that are embeddable in your house. And the first plug-ins were actually all metrics. There were CPU, memory, disk, thermal. And these are all what we consider log-based metrics. They essentially take JSON and send that over into your log pipelines, whether to an output like Elasticsearch, et cetera. The advantages with this is it's compatible with old plugins. It's also compatible with some of the more advanced features and functionality that Flimpit has, like SQL stream processing. But the disadvantages are as we see the evolution of this Prometheus ecosystem, it's not compatible. And it's also a little more expensive bite-wise, right? You have JSON. JSON is a pretty heavy and expensive compared to Prometheus format. So with Flimpit and Prometheus, the integrations didn't just start with the latest release. We do have a lot of Prometheus-formatted metrics within Flimpit. And so we'll do a quick intro for Prometheus for those who might not be as familiar. And Flimpit internal metrics were disposable via Prometheus since 2019 and before. So you can see how many input records, you can see filtering records, output, you can get storage information, you can get all sorts of various internals all exposed in a Prometheus format today. And with 1.8, we finally introduced those Prometheus metrics as an exposable format. And we have two main ways to do that. One is through Prometheus node exporter. We take the great work, the Prometheus node exporter team is done and have ported some of that to Flimpit so we can collect things like CPU, network, disk, file system stats. And then we have metric outputs. So things like remote write, which is more of a push method for sending that data over to Prometheus or Prometheus-compatible backends and Prometheus exporter. So if you're doing the pull model. In the workshop, we'll actually be running through both of those, but yeah. So however you need to get your Prometheus data, we can support that. And then the formats for the existing plugins. So things like you have forward plugin where you might be sending to FluentD. We've made sure that that's compatible. We've insured things like influx if you're sending to influx back end with line format. And the advantage here is it's very lightweight and it's convertible into multiple formats. And the disadvantage, it looks like that thing got cut off but essentially it doesn't work with some of the more advanced features like SQL stream processing today. But that can always change. So with that, let me hand it over to Gibbs who can talk a little bit about some of the more metric ecosystem. Okay, yeah, so I know most of you probably are all very familiar with Prometheus but just for those who are not, we're gonna quickly give an overview. So Prometheus, it's a CNCF project. As you know, it uses a tag-based metric ingestion format. Query language called PromQL. It also does metric ingestion. It's scrape-based, so pool-based metric ingestion. It's also very efficient at metric storage. So very good at storing metrics inside the instances themselves. And it does have the ability to kind of out of the box do visualizations and graphing of metrics. Most people choose to use another solution like Grafana but it does have that capability as well. And then you can do kind of aggregation with Prometheus using the Prometheus recording rules and then doing the alerts on your Prometheus metrics as well using the Prometheus alert manager. And some of the advantages of Prometheus which has made it so popular, first is that it's very easy to get started. So having a single binary for ingestion, storage and query, and then you'd have another single binary for alerting. It's also the CNCF recommended monitoring tool. So that's made it have a very large community around it. So, and that's made that exposition format that it uses widely accepted as well as the query language PromQL. And some other reasons why it has become so popular is because of its dynamic endpoint discovery which can be found on many different platforms including Kubernetes. And then it has a very large ecosystem of exporters which are essentially just existing software integrations and most major software projects, especially open source ones have these existing integrations and there are huge lists of them on the Prometheus docs. So we're gonna, as part of the workshop we're gonna use M3 which is an open source metrics engine. It was created back at Uber in 2016 to help with their metrics monitoring use cases internally. And now it's used by many other companies including Chronosphere which uses M3 as its backend but just to kind of quickly run through the architecture of what M3 looks like at a high level. So you have three main components in here. We have the ingest and kind of down sampling tier which is the M3 coordinator. Then we have a distributed custom time series database called M3DB and then on the query side we have an optimized query engine called M3 query. So we'll be spinning up as part of the stack and then sense of each of these or a node of each of these with like a single node of M3DB. And then typically how metrics are sent into and out of M3 is we have on the right side of things we can accept both from Uthias metrics through remote right and then we also can accept carbon or graphite metrics as well from SSD. And then on the read side, you can again send queries and using graphite or promql through Grafana and then you can also kind of alert on your metrics via the alert engine through the query tier. And so yeah, so I think I just kind of ran through a lot of this but some of the I guess the key benefits of M3 are in some of the design choices around that were to be very large scale and that's because it was built back at Uber. So and to beat that level of scale was a requirement. So it does have the ability to accommodate billions of time series metrics per second. It also has a lot of high reliability design choices as well. So using replication of each of the metrics and that kind of leads to having any fault tolerance. So if anything goes good down, you'll have some backup copies of your data. And then in terms of performance, we do use a reverse metric index inside the DB tier and also kind of a very efficient compression algorithm. And that kind of makes it so your metrics can be retrieved very quickly and makes it very efficient for storage. And then yeah, cost efficiency as well that kind of, I guess that kind of falls into that as well. So just some of the key I guess features of M3 there. Okay, and then that's all for our slides. So I don't know if we wanna send this link out in the Slack if that's easier, but if everyone wants to go to the fluent repo and GitHub and then there should be the M3 workshop fluent conf folder inside there. But I can send it into the fluent Slack. That might be easier. Awesome, and I guess just a show of hands for the folks in person. How many folks are gonna run this locally right now? One, two, three. Okay, awesome. I'll just come around and make sure everything goes well. And for those three raised hands, does everyone have like Docker desktop Docker post? Awesome, awesome. Okay. Yeah, the read me in here should be pretty, should have everything you need. Kind of goes through a stack overview and prerequisites you need to get it up and running. And then there's like the step-by-step instructions. But again, like Anarok said, it shouldn't take too long, but I think we'll have plenty of time to play around with everything once it's up and running. And I guess just to walk through the architecture a little bit, we're gonna be deploying out Prometheus. We're gonna be deploying out Fluentbit. We're gonna be deploying out the coordinator and we'll be deploying out Grafana as well. And so the Docker compose will go ahead and spin all of these things up. We'll make sure everything gets connected as well. And then on the Fluentbit side, we can walk through the configuration, see what it's doing, how hard is it to go and configure kind of these Prometheus node exporter metrics. And then hopefully on the Grafana side, when we go visualize that everything just shows up as you would normally expect with something like node exporter. So with that, let me go ahead and do two things here. I'm gonna go ahead and just walk through the configuration. Then we will go ahead and do Docker compose up, launch everything, we'll kind of walk around, make sure everyone has everything good. We'll check on the Slack channel for those joining virtually, make sure things are great. And then we'll reconvene in like 10 minutes after that, see what we ran into with running everything. So first let's walk through the configuration for Fluentbit. And with Fluentbit, this is what, for those who might not be familiar, what that configuration looks like. There's been a lot of work to make this very, very simple, easy to understand, high readability. And you'll see these big brackets that denote what this might be doing. If it's an input, it's an output, is it reading data, is it sending data? And in this case, we've added this input, which is node exporter metrics. We're gonna give it a tag. So all the data with in Fluentbit, even the metrics are tagged with metadata, which then says this is where it's going to route to, this is how it's gonna be sent out. And then we have two outputs. The first output is for Prometheus exporter. So if you're scraping from node exporter, you might want to expose these metrics on a port with a listening address. You might wanna add labels, so labels can really help with the queries. We'll show some problem-ql queries here later. And then also we support remote writes. So if you want to go ahead and send this to a service, you know, say the M3 service that's running locally or even other services, hosted services, if you're using things like hosted Prometheus, managed Prometheus by vendor, typically they'll accept Prometheus remote write. Okay, so what we'll go ahead and do is I'll switch into the root of this directory after doing a gate clone. And very simply, I'm just gonna do Docker compose up and it's gonna go ahead and start both M3 provisioner, coordinator, query, fluent bit and Grafana. So this should be a very simple step, but ideally what is happening is we are running fluent bit, we are starting to capture those metrics, we are plugging in both to the scraping and we're also sending that data to M3 where it'll be searchable and accessible via Grafana. So we've gone ahead and run this command here, we're gonna take a quick 10 minute walk around, make sure everything is running well and we'll reconvene and we'll check the Slack channel as well. Thank you. Yeah, so we were walking around, it looked like most folks got situated, again, this should be really quick, so we've made the workshop so you can kind of turn key and then start doing some exploration after it, it's hopefully working. So here we had run the Docker compose up about 10 or so minutes ago. Some quick corrections, again, for those joining virtually, the local host and the read me was 3030, it's actually 3000. And so let's go ahead and move over to that. Let's go ahead and log in, local host 3000. And actually for most folks, it should give you a quick username login, you can use admin admin, you can reset the password. And as part of this, we wanted to use the exact same node exporter dashboard that you typically find when you use Prometheus. So when you log into Grafana, you can come into the dashboards, you can take a look at them and you'll find this node exporter full dashboard. And then within this, you'll automatically see that data loaded up. So what has happened is fluent bit, again, was configured with the node exporter input. And on the output side, we had two ways to capture that data, whether you wanna scrape that data via Prometheus exporter, or you can use the remote right to have three. And we can look at the last five minutes of data, you can look at what's going on here.  So if folks have other metrics or types of metrics they wanna collect, this is an area that we're always looking to improve from a community standpoint. And then from an ease of use, the idea is to get you started as quickly as possible and as fast as possible. So let's go ahead and just check if there's any questions on the Slack. Nothing so far. So next what we'll showcase here is maybe some quick promql queries and some labels, we'll pass it off to Gibbs. You can maybe showcase some of that within Brafana, maybe what you can do with some of this data after it arrives. Let's see. The one we typically just do is the up command. Okay, so typically it just run the up command just to see everything. Okay, well you can only see two of your instances here. How would I create here by trying? I think the other piece is when we configured Fluentbit as well as you could add labels within Fluentbit. So we can always use those labels as part of dashboards, as part of filtering. And so in the node exporter side as well in the dashboards, there's a couple of dropdowns there where you could say, hey, I wanna look at potentially labels that match specific app profile, particular environment. And so the main thing we wanted to just showcase here as part of this workshop is we're not doing anything like too fancy, nothing that's super Fluentbit specific that it prevents you from utilizing all of the processes that you might have built on Prometheus before. So if you already have node exporter, you have all the alerts defined out, you have everything going for it, you'll be able to quickly go ahead and basically plug Fluentbit into that ecosystem. And then secondly, if you are expecting to traverse and explore the data, you haven't really had too much experience with node exporter or Prometheus, it's pretty simple to get started. There's a really large ecosystem around Prometheus, really large amount of folks and tools out there to kind of help explore that data. So we're gonna go ahead and hang out on the Slack here for a little bit more and just make sure that things are running well for others. But other than that, we'll go ahead and answer questions, things about metrics, and then we will be breaking here for lunch right after. So we're the only thing that stands in your way. So yeah, thank you, thank you for joining in. We'll, again, everything's on the GitHub, so if you wanna try it later, you haven't had a chance to try it, it's all available. And yeah, maybe we made it too easy with just a single step, but so awesome. Yeah, and for the recording, it was saying, hey, I have a lot of Raspberry Pi, I've been trying to do a lot of node exporter setup, and what we can do here with Fluentbit looks a little better. So yeah, that's great news. And this is something where we wanna keep on building as a community for the metrics integrations. And so if you have those requirements, if you have those needs, we've had folks come and say, hey, I wanna be able to measure from my logs how many 400, 500 errors potentially as metrics, so we can add those stats as things that we expose internally. So yeah, we're super excited to keep building here. Okay, I will keep on checking the Slack here for any questions, otherwise we are happy to end a little early, give back some time, jump into lunch. Okay, let's do it, yeah? Yeah.


Food is more global than ever before. Reduced trade barriers, advances in production, logistics and a rising demand for food have powered trade and global supply chains. These processes have unlocked millions of jobs, supporting social and economic development around the world. As food crosses countries and oceans, there can be harmful impacts along the way that can take place from farm to fork. Let's turn to Louisa and Dan. Louisa owns a major food store, importing fresh and processed products from suppliers all over the world, who rely on farmers in producing countries like Dan. Louisa is paying a fair price to her suppliers and is under the impression that the producers are treated fairly, but this is not the case. The truth is, Dan is not given an equal chance to succeed. His neighbour sells tomatoes to the same trader, but has joined other producers in bribing local officials to expand their farms to conservation land and to use pesticides excessively to boost their yield. Dan is producing less and making less, but knows that his farming practices aren't contributing to deforestation or biodiversity loss. There are other problems in Dan's community. In the tomato canning factory, women are being paid much less than the men and they're working in unsafe conditions, but the country's labour laws are weak and they fear losing their jobs if they complain. Other communities face other problems. Just across the border, crops in high demand are taking over land once reserved for local staples, affecting food security. Attending a talk by an NGO, Louisa is shocked to discover that her business could be contributing to harmful impacts through her own supply chain. Louisa learns that the OECD FAO guidance for responsible agricultural supply chains can help businesses introduce risk-based due diligence to identify, mitigate and prevent social and environmental impacts in their supply chains. The OECD FAO guidance provides a five-step framework for risk-based due diligence, helping companies understand where risks can occur in business operations and how that can impact on development. Following these steps, Louisa implements a due diligence framework and new company policies to help her business operate more responsibly. She's developing a code of conduct for her suppliers and knows which questions to ask to ensure they're doing their part to address risks to people and the planet. She is also engaging with stakeholders, including NGOs, worker and employer organisations, and many others. Dan also starts seeing changes in his community. The trader that Dan sells to has introduced signed visits to assess producers' farming practices, addressing issues such as overcropping and social protection for seasonal workers. A local NGO is also promoting awareness around environmental issues, and there's even an anonymous hotline to report harmful practices. Women at the factory say working conditions have improved. As for Louisa, business has never been better. Ever since her suppliers began to consider risks more carefully, there have been fewer development impacts. She also sees greater confidence among her investors, increased customer loyalty, and better understands how her business can help achieve the sustainable development goals. Responsible business conduct in agriculture can help build a better life for everyone, bringing more responsibly sourced food to all our tables. Learn more.


For many lectures now, we've been dealing with equations. But what if the left-hand side is not equal to the right-hand side? In this video lecture, I want to talk to you about inequality. So we start to talk about inequalities. We've always had a left-hand side equal to the right-hand side or the right-hand side equal to the left-hand side, but now we're gonna have an inequality. The left and the right are not equal to each other. Before we get there though, I really want to talk about intervals. So let's start talking about intervals. Intervals, if we think of the real number line, so let's have a real number line right here that goes off to positive infinity on the side, on the side it goes off to negative infinity, we're talking about some interval of values that we would find x, these are all the values x, our horizontal axis, that we have some part of this line in which we find the solution x. Now that can take various forms. We can have a lower bound and an upper bound, and we would write it with square brackets, denoting that a is included in that possible set of solutions and so is b. So if we have something like the solution is between three and four, the solution can absolutely be three and it can absolutely be four and anything in between. So these we're gonna call closed, that's a closed interval, closed interval. We can also have this scenario a comma b, but we put parentheses there. So if that is any solution between three and four, that is an open interval and what we do now, we exclude three and four. We can have 3.0000001 and we're gonna go 3.999999, but three and four are excluded, anything in between three and four is possible, but certainly not three and not four. Then we also get this idea of half open intervals, so we can have something like from a to b, and now say for instance that would be three to four, it is still possible for four to be a solution, but not three, something just bigger than three, but four is definitely included. So we can call this semi open, semi closed depending on the textbook that you read. So let's say it's semi closed or semi open and we can also get this situation a comma b. Now three is definitely possible and then anything bigger than three up to four, but four is definitely not included. You can call that semi open, semi closed, et cetera. We can have other ways that we can write this. We can say a is less than or equal to x, which is less than or equal to b. Yeah, we'd have a is less than x is less than b, so a, so this x at least is between a and b, but there's no equality there. A and b are not included there, there isn't equality there, so a and b are definitely included. Here we would have a is less than x is less than or equal to b, it is possible to be b, and here we have a is less than or equal to x, which is less than b, b now is not included, up to b, but not with b, and then on the left hand side, a and a can be included in that solution. So that's just something that we have to consider when we look at inequalities. We're thinking about these possible solutions that we can have. So now there are a couple of rules that we really have to pay attention to if we know these, then these inequalities are no problem. So in order to do this, let's create some constants, a, b, c, and d. We're gonna say they can be any real numbers, so they are members of the set of real numbers, but we're gonna subtract from that real number one of the elements of the real numbers, and that's zero. Let's just remove zero because we wanna divide, and we don't want to divide by zero. So let's have these rules. Now the first one that I want to do, I'm gonna say let's do this first one. If a is less than b, so we're gonna have a is less than b, now c can be anything there, we're gonna say then a plus c is still less than b plus c. So I've got a is less than b, I'm adding c to both sides, it's still going to be, a plus c is still going to be less than equal to b plus c. So let's say for instance three is definitely less than four, then we could say, say for instance three plus five, that is still going to be less than four plus five, because what do we have here? We have eight is less than nine. So that one checks out for us. Let's do the following. If a is larger, if a is larger than b, then if I take a plus some value c, that is still going to be larger than b plus c. So an instance here, let's say four is larger than three, and if I set something like four plus four, it's still going to be larger than three plus four, because eight is larger than seven. So that one checks out as well. Now these needn't be positive numbers, let's just have a look at this, let's still have four is larger than three, we would all agree with that. But let's add to four, let's add to that a negative four. We are saying that's still going to be larger, there's our three on the right hand side, plus we add a negative four, that's still going to hold, whether that's a positive number or a negative number, because four minus four, that's going to be zero, and that's still going to be bigger than three minus four, which is negative one. So it doesn't matter what you add on both sides, nothing is going to happen to this inequality. If it was less than, it's going to stay less than, if it was greater than, it's going to stay greater than. So let's have the following rule, if A is larger than B, but now we're going to say, and C is larger than zero. So we're going to constrain C now, then we're going to have the following, we're going to say that A plus B, let's have it there then, is to A times C is still going to be bigger than B times C. So those were additions, now we're going to do multiplication. There we had addition, addition, even though that addition can be a negative number, now we're constraining C to be a positive number. So let's have four is larger than three, and then let's multiply both sides by anything that's larger than zero, let's make it five, let's then, it's four times five, it's still going to be greater than three times five. We have that 20 is larger than 15, nothing happens there. Let's have the following rule, if let's do A, let's stick with it, larger than B, and now we're going to make C less than zero, it's going to be negative. What's going to happen then? Then, A times C, for the first time we're going to swap around that inequality, it's now going to be less than B times C. So in this instance, A was larger than B, now suddenly becomes less than, if we multiply both sides by a negative number. So let's have four is larger than three, and let's say then we multiply both sides by negative five. So now we're going to have four times negative five, and on this side we're going to have three times negative five. And so what do we have here? We have negative 20, and here we have negative 15, and now suddenly this is smaller than that, it is smaller than, it is smaller than, so we went from a greater than to a less than. Now let's have the next rule, let's say if A is less than B, that was greater than, now we're going to make it less than, and we have C is greater than zero, then nothing's going to change. Then we're going to have the following, A times C is going to remain less than B times C. Let's have an example, we're going to say three is definitely less than four, we're going to multiply both sides by a positive number, let's make it five again, three times five, and on this side four times five, it's still going to be less than. We have that 15 is less than 20, so that works out. But now if for instance A is less than B, and C is now negative it's less than zero, then we're going to have a swap around again. Then AC is going to be bigger than BC. So let's have three is definitely less than four, let's multiply both sides by a negative five, and we have four on this side and a negative five, so we have negative 15, and we have negative 20, now negative 15 is larger than, this is larger than, and we can see this sign, this inequality has changed. Let's have the following, if, now let's stick with this, A is going to be larger than B, and we're going to make C is also as positive, then if I take A divided by C, now remember we said these can up is zero, what happens now if we do B divided by C given, A was larger than B, and C was positive, this remains exactly the same. So let's have some example, let's say for instance eight is larger than four, and I'm going to divide both of them by two, two is certainly larger than zero, then we have eight divided by two, and we have four divided by two, and that gives us four, and that gives us two, and it's still larger than, it's still larger than, it's still larger than. Now though, if A is larger than B, and C is now suddenly less than zero, then if I have A divided by C, and I have B divided by C, then we're going to swap around. Now A divided by C is going to be less than, whereas before A was larger than B. So let's have the following, we have eight is larger than four, and let's divide both of them by negative two, negative two is certainly smaller than zero, so I'm going to have eight divided by a negative two, I'm going to have a four divided by a negative two, on this side I'm going to have a negative four, on this side I'm going to have a negative two, now this one is smaller than, it's smaller than, and now that sign has swapped. So let's have now if A was less than B, and we have that C is positive, then we're going to have the following, if I take A divided by C, and I take B divided by C, this is positive, that was less than, that's going to stay less than. So let's say four is less than eight, we agree with that, let's divide both of them by some positive value, let's make it two, we take eight divided by two, four divided by two is two, eight divided by two is four, it's still less than, it's still less than, it's still less than. And then if I have that A is less than B, and C is now negative, well then we're going to have, if we take A divided by C, and we take B divided by C, we're going to have that sign now swaps, and it's going to be like that. So let's have four is less than eight, and let's divide both sides by a negative two, once again negative two is less than zero, I have eight and I'm dividing that by negative two, four divided by negative two, well there's negative two, eight divided by negative two is negative four, well now the sign swaps because negative two is larger than negative four, it's larger than, and that sign has swapped. Let's have our very last rule, we're going to say if A divided by C is less than, let's make it B divided by D, if I take the reciprocal of that, instead of A divided by C, I have C divided by A, let's do C divided by A, and on this side we have D divided by B, if I swap those around, then the sign is also going to swap around, so there's the sign. So A divided by C less than B divided by B means C divided by A is going to be greater than the D divided by B, so let's have a little example of that, let's have two divided by seven, that is less than for instance five divided by eight, why do I know that? Well I can do common denominators seven and eight, seven times eight is 56, so let's make both sides 56 as the denominator, let's have the two numerators, so I divided this denominator, multiplied it by eight, I've also got to multiply the numerator by eight, this would be the same as multiplying by eight over eight, which is this multiplying by one, nothing changes, eight times two is 16, this side I'm going to multiply by seven over seven, taking any number, multiplying by one, nothing changes, seven times five is 35, and clearly 35 is larger than 16, so that's it, but let's swap that around, let's make it seven over two, and let's make this eight over five, now common denominators now, let's make that 10, so I've got 10 as a common number, 10 to get from two to 10, that's multiplied by five, so I'm going to take seven times five, that's 35, five to get to 10, that's multiplied by two, eight times two is 16, now clearly 35 over 10 is larger than 16 over 10, so seven over two is larger than eight over five, so that sign has swapped, so let's solve some inequalities now that we know these rules, let's start with an example, so our example is going to be the following, let's do three x minus four, and let's have that being less than five, so what interval of values for x would solve this, it's not a quality anymore, so I'm not going to get a specific value for x, I'm going to get some interval, so what I can do of course, I can add four to both sides, and if I add some c that's larger than zero, nothing's going to happen to the sign, so I could say three x minus four plus four is less than five plus four, so I've done the plus four to both sides, so I haven't changed anything, of course negative four, positive four, that's just zero, so I've got three x is less than nine, now what I'm going to do, I'm going to divide both sides by three, and remember three is larger than zero, so nothing is going to happen to this sign, I can cancel out the threes, and nine divided by three, well that's just three, so x is less than three, any value less than three, so three is not included, if I plug any value in less than three, I should get this, and let's do that, let's choose the value two, let's have three times two minus four, well that should be less than five, three times two is six, six minus four is less than five, and d two is less than five, so that's absolutely correct, now there's other ways that we can write this, we can for instance say that x is on this interval, we can say from negative infinity, all the way to three, but three is not included, so it's open interval, negative infinity, remember infinity and negative infinity, they are not numbers, they're not some exact numbers, so you can never have that that's included, so we'll always have this open interval as far as negative and on the other side positive infinity is concerned, so we can say that x is in this interval, maybe it's an element of that interval, or it's equal to that interval, we can also write negative infinity, that's going to be less than x, and x is going to be less than three, three is on that interval, so we think of a real number line, and perhaps that is zero, and that is one, and that is two, and that is three, and it goes off to positive infinity, the side goes off to negative infinity, we're going to say that this interval is something like this, three is not included, and then all the values here, you can plug in any value there, and you're going to get that inequality holds. Let's have another example, let's have the fact that negative two, let's make that x minus four, let's say that that is less than, let's make it five, let's make it five. Now what I want to do is I want to get rid of this negative two, so let's divide both sides by negative two, so I'm going to have negative two x minus four, and on the side I have five, I want to divide both sides by a negative two, negative two and negative two, now I have divided by a negative number, a value that we see is less than zero, so that sign has got to swap around. Now those two would cancel, and I'm left with the fact that x minus four, let's have that x minus four, let's have that x minus four, well that's going to be, let's try and rescue this, it's going to be larger than five over negative two, so let's make it negative five over two, I can add four to both sides, so x minus four, let's add four to the side, that's going to do nothing to that sign, I have negative five over two, and I'm going to add another four to that, so on this side negative four plus four, nothing remains there, so that's going to be larger than, and let's have negative five over two, I'm going to add to that, and I'll have eight over two, eight divided by two is four, I have a common denominator there, so I can say x is larger than negative five plus eight, negative five plus eight over a common denominator two, that is x is larger than three over two, and again we can write something like, if x is indeed larger than that number, it means that we have three over two is not included, and it goes up to infinity, or I could say three over two is smaller than x, if I read it from this side, it's x is larger than three over two, x is larger than three over two, and it's going to be less than positive infinity. Now let's do this very complicated example, let's have x squared minus one, let's divide this by x plus three, and I want this to be larger than or equal to both possibilities zero, so let's deal with the two cases, it can be equal or it can be less than, so let's deal with case number one, in case number one we're going to have x squared minus one, we're going to divide that by x plus three, and we set that equal to zero, it's certainly possible to be equal to zero, what can we do, well we can multiply both sides by the denominator, so we'll have x squared minus one divided by x plus three, let's multiply that by x plus three over one, and so we just have to do that on the right hand side as well, x plus three divided by one, now of course zero times anything is to zero on the right hand side, I'm just left to the zero, on the left hand side remember this is one single value, that's one single value, there's a multiplication, which means I can cancel those two, on the left hand side I'm left with x squared minus one, I can bring that across to the other side or add one to both sides, so on the left hand side negative one plus one, nothing there, on the right hand side I have equal to one, so in other words I take the square root of both sides and I have that x is equal to plus or minus one, so x can be positive one, x can be negative one, by the way let's have a look at this denominator, remember one thing that x definitely cannot be, it cannot be negative three, negative three plus three is zero, I can't divide by zero, so at least I know that, and now we have x can be one, or x can be negative one, and if we plug those in there, I get either one of those two, I get zero in the numerator, some number in the denominator, and that is definitely equal to zero, this is the case that we have here, so let's have the second case, I now want x squared minus one, divided by x plus three, I want that now strictly to be less than zero, so I've dealt with the equal, and I wanna deal with the less than, because I have less than or equal to, so how can I have a numerator divided by denominator and it's negative, well I can have a negative divided by a positive or I can have a positive divided by negative, so I have to look at either of these, and remember between this set of, this case and this case, it's less than or equal to, so there's definitely an all between these two, there's gonna be an all between these two, but definitely I have to have this and this, or I have to have this and that, so let's start with this, I've got to have the numerator, it's less than zero, it's negative, and both of these things must be true, I have to have that x plus three has gotta be positive, so both of these, yeah I would have x squared is less than one, I just added one to both sides, and I can have, I have to think this through, what values can I square that it's still less than one? Well it turns out if I go negative one is less than x is less than one, if I have this value on this open interval from negative one to positive one, if I square anything there, I'm gonna get a value less than one, as soon as I get beyond that, say negative two or one plus 1.1, I will no longer have the case that the value is less than that, on this side, and I've gotta have that x is larger than negative three, now how can I have the fact that x is in this interval and x is larger than negative three? So what can I have so that both these things are true? So certainly I can't have negative two or negative two and a half or negative one and a half, and I can't go beyond one anyway, so if I look at x being, now both of these things have to be true, then certainly this is the part that remains, because if this is true, then by necessity this is also true, but if I have a value say negative 2.5, which is certainly there, it's not in there, but all the values in there are definitely within that, so this one has gotta win out, because remember, I have to have that both of these things are true. Now let's have the other case, I'm gonna have that x squared minus one that is larger than zero, or I have x plus three is less than, so and remember, is less than zero, so it's a positive enumerator negative in the denominator, so I'm in this case, and between these cases remember, there's an or between, so I really have to keep track of the ands and ors, so here we are gonna have that x squared is larger than one, and the only way that that's going to happen is if I have that x is less than one, or x is larger than, so I've gotta put that actually negative one, or x is larger than positive one, if I have again this scenario, I'm gonna land up with a value that's less than one, I've gotta be beyond these values, if I square them to get something that's larger than that, and on this side, on the denominator side, I'm gonna have that x is less than negative three, now again I've gotta think that or that and that, and certainly what is going to win out is gonna be this one, because this x is less than negative one, so let's make that negative two, well that's certainly not here, and definitely if it's beyond one, it's certainly not there, so certainly this is the one that's going to win out here, so now have a look at this, it can definitely not be negative three, it can be plus or minus one, so let's have that, those must exist as well, and I've got this, and this, so it's this, or, this, or, this, and now if I have one and negative one there, that's less than, but it can be equal, it's less than but it can be equal, so that's going to change to negative one is less than or equal to x is less than or equal to one, so that's the one part, or I have to have the fact that x is less than negative three, so what I'm gonna leave for you is to plug in values there into there, and see that it's less than or equal to zero, or plug in a value less than negative three, plug that into x there, and you'll see that's also less than negative three, if you plug in any value outside of these, so any value that's larger than one, or value that's here between negative one and negative three, that's not going to hold, now the last thing I want to deal with is absolute values, absolute values, so let's deal with that, we're gonna say that the absolute value of x, now yeah, let me just be sure that x is a member of the set of real numbers, the absolute value of x is the following, now this is a piecewise defined function, I'm gonna say the absolute value of x is x, if x is larger than or equal to zero, and it's gonna be negative x, if x is less than zero, now that might seem a bit strange, so let's look at a couple of numbers, let's look at anything that's equal to a larger than zero, so let's say x is equal to positive three, well the absolute value of three, while we say that is so, then x is just what it is, it's just itself, so in this instance, well that's just gonna equal to three, let's have a value where x is less than zero, let's say x is negative two, while we say that the absolute value of negative two, that's gonna be negative x, that's negative, negative two, and that's also positive two, so we're just saying that we take any number and we turn it into something that's non-negative, because if we plug in zero, it's just gonna be zero, but any negative, negative times a negative gives me a positive, so we're turning anything into a non-negative number, so let's look at an example problem, let's have that the absolute value of x minus three is equal to five, how can we solve for that? Well, let's think of this definition, so we're definitely going to have a couple of things, we're gonna have that x minus three equals five, that is if x is larger than equal to zero, and we're gonna have minus x minus three is equal to five if x is less than zero, so I'm just looking at the definition, I'm taking this, now it's not just x, it's x minus three, I'm leaving in that as is, when x is positive, I'm just putting a negative in front when x is negative, so if we carry on with that, we're gonna have that x equals eight, if x is larger than equal to zero, well, now we have a specific value, it is eight, while we're going to have on this side, we're gonna have negative x plus three equals five, and if we just carry that on, we'll have that negative x is equal to, and we're gonna take negative three on both sides, that leaves me with two, or we're going to have the fact that x equals negative two, so we have two solutions, x equals eight, or x equals negative two for this absolute value problem to hold. Now let's do one more, let's have that x minus three must now be less than five, how are we gonna do that? Well, these are actually quite a bit of fun to do, so we're gonna say x minus three is less than five, that is if x is non-negative, and we're gonna have negative x minus three is less than five, if x is less than zero, so we've got to have both of those cases, cases number one and case number three. This one is easy, I'm just gonna take this across to the other side, so I'm gonna have that x is less than positive three on both sides, that's eight, if x is larger than or equal to zero, so now I just gotta be careful, it's less than eight, but it's gotta be there, so I'm gonna say zero is less than or equal to x is less than eight, that's my one solution. Here, let's multiply both sides by negative one, but do remember that means that sine has to swap, so I'm gonna have x minus three is larger than negative five, and that is if x is less than zero. So what we're gonna have here, we're gonna add three on both sides, so that's gonna be x is larger than negative two, but remember x is less than zero, so we're gonna have this, negative two, we're gonna have x, and it's gonna go up to zero, now x is less than zero, but it's larger than negative two, so there are my two solutions, so it's between zero, so let's have it, let's put it down here, it's gonna be between zero, let's put it out here, zero to eight, but eight is not included, and we've gotta have this union here, and that goes from negative two to zero, and these are open intervals on both sides, there's no equality there, and I want the union of those two, remember with the union of those two, I can also say that that must be all, which is the symbol instead of the symbol, so if I have on this open interval, the semi-open, semi-closed interval, either of those two, this is an or a union, then this inequality is going to hold.


where we left off, we saw how to insert and delete into and from B plus 3 and query B plus 3. The next topic is what is called B plus 3 file organization. So, if you look at the figures we saw till now, B plus 3 let us take the original B plus 3 figure a long way back. We had a file storing the data and the B plus 3 simply stored pointers to the data. But this is not necessarily the best way of doing things. There are many situations where we may want data to be stored in the leaf node of the B plus 3 itself. So, this is called a B plus 3 file organization. So, this is the figure which I was on just a second ago. So, if you see here the leaf this is a B plus 3 kind of drawn funny leaf because it was too wide to fit in one line. So, the leaf nodes had to go over two lines, but it is a plain B plus 3 there is nothing special about it. The only difference is at the leaf instead of storing a pointer to a record we are storing the actual record itself. So, a 4, b 8 and these are the actual records there is no pointer. So, the leaf level contains records. Now, there are a few issues that come up if you do this. If the leaf level contains records how many records fit in a node? If you look at the internal nodes it does not store a whole record it just stores the search key value and a pointer. So, an internal node may be able to store a lot more key values and pointers than the number of records which can be stored in a leaf. So, that is the first difference that the leaf node may have only a few records while internal nodes still have lots of killer. The next issue which comes up is what is the records are variable size? How many records will fit in a leaf and the answer in this context is this is true even with variable length records with normal B plus 3s where variable length keys normal B plus 3s. So, what we do is we require that things be half full with data. So, if there are variable length we may not say that it must have at least n by 2 records, but we will say that at least half of the space in the node must get used. Now, that space may get used for storing long keys it may get used for storing more pointers, but regardless half the space must get used that is the key thing and for a B plus 3 file organization at the leaf node the same thing holds at least half the leaf space must be used by records. Otherwise there is nothing special here in terms of insertion deletion algorithms it is standard B plus 3 algorithm, but what is this problem solved? It solved the problem that you know earlier we said we assume that the file is sorted on a key, but who is going to keep it sorted on the key? Somebody has to go periodically and sort the file and in fact even today PostgreSQL does not support B plus 3 file organization. You can ask it to sort a file called clustering a file, but after you do insertions deletions to the file the sort ordering can go for a toss may be approximately sorted, but once in a while you have to say cluster it again, but who is going to do that when are they going to do it?  What is not guaranteed is that these nodes are physically sorted on this that cannot be guaranteed, but there are some tricks which try to keep these things more or less sorted with minimum number of disk seeks as you seek across the leaves of a B plus 3. There are tricks to make sure that the number of seeks you require is minimized. I do not have time to get into that, but that is used here that is one thing. The next issue here is that we did not care much about space utilization we said half full is fine because the index is pretty small. Supposing my data is 100 bytes the index space may be may require 10 bytes for the same amount of 12 bytes for that data. So, it is may be 10 percent of the space if it is half full the overhead is may be 5 to 10 percent of that it is actually B plus 3 is tend to be closer to two-thirds full. So, overhead may be only 3-4 percent of the total data size for one index. So, it may not be worth putting too much effort to save 3 or 4 percent, but if I am storing big records in the leaf if it is only half full I have a problem I may be wasting half the space of the file which can be very large if there are lots of records. So, there is a trick to improve the space utilization which is when you do splits and merge you do the following. If I need to split if I split a node into two it is going to become half full I do not want to do that. So, what I will do first is if a node is full and needs to be split I will first look at its sibling on the left and on the right and say look at least one of the sibling say sibling on the left is not full I will try to push some values from the node into the sibling. So, redistribution happens not only on deletion, but it can happen on insertion also. So, now when I insert if the node is full I redistribute, but there may be a situation where the left sibling I only focus on one sibling left or right one sibling I look at and it is full I am full I am over full and my sibling is also full what to do now. So, at this point I can take data which is two node fulls worth. So, I have two node fulls of data and I split it into three nodes I sort it and divide it among the three nodes. What can I say about the three nodes each will be at least two thirds full. So, what I have just ensured is when I split the minimum utilization is 2 by 3 it is not half. So, that is what splitting what about for deletion in deletion what did I do I try to borrow, but if I could not borrow because the sibling is half full I merge with the sibling, but now I do not want to go to even half full I want to stay at two thirds full. So, I am going just under two thirds full I do not want it to go under two thirds full. So, what do I do I try to borrow from a sibling and I will look at my sibling and it says no sure I have some data take a few keys and now both of us are two thirds full I am happy. If my sibling says no I do not have enough data to give you anything I am also at the border line of two thirds full what to do now and the trick is to ask one more sibling between the three of us can we redistribute data such that all of us stay at two thirds full then do it, but supposing both the siblings that I look at are at two thirds full and I am under two thirds full now I cannot borrow from either sibling, but here is what I do I have three siblings each two thirds full. So, what is the total amount of data it is two nodes worth three times two by three is two. So, the total data across the three nodes is as much as will fit in two nodes. So, it is very simple I take three nodes merge them into two nodes and what I ensure is that on deletion the utilization never goes under two by three. So, that is what I do. So, insertion and deletion both take care of this and we can get whatever utilization we want. We can look at three siblings instead of two four siblings instead of three and so forth and push the utilization from two thirds to three fourths to four fifths or whatever. There is a trade off there is more I O when I do insertion deletion and the benefit of more I O is better utilization, but at some point it is not worth it I will not involve too many more siblings. So, that was one part then next topic is if I use a B plus three file organization it causes a another different problem. If I have a secondary index earlier on I said it stores a record pointer. Now, look at this B plus three file organization the record is here and I have a secondary index pointing to this page, but now an insert happens this page is full it splits what happens half the records in this page move to a new page. Now, if I have a secondary index on the same relation this half of the page that record those records may be fifty to hundred records are moved. Now, one insertion here requires fifty to hundred updates on secondary indices all that is updated is the record pointer changes. So, the index structure does not change, but I have to update fifty to hundred records in index records in another index that may involve fifty to hundred I O's they may be scattered all over in that index that is very expensive I cannot afford that. So, instead so this is the problem this is the reason why post-criscule does not have B plus three file organization and neither did most other databases some years ago about ten fifteen years ago many databases one by one started adding B plus three file organization and the key trick which they used to solve this record relocation problem with secondary indices is the following. A secondary index no longer stores a pointer to the record you cannot afford to because records move very often you cannot update them. Instead what a secondary index source is the primary index search key. Now, assuming this is unique the search key is you know if it is not unique we will add some things to make it unique and not going to detail, but the key thing is that when I split the leaf of the B plus three file organization no update is required because there were no physical pointers they only had a search key in the secondary index that is a good new the overhead though is that when I search on the secondary index I will search all the way down to the leaf of the secondary index what I get is not a record pointer what I get is a primary index search key. Now, I come back to the primary index and search down when I get to the leaf of the primary index here it is a file organization I have found the record I want. So, if I had a secondary index on the second attribute nine I will search down that tree and at the leaf instead of saying point to this node it will say the primary key is C primary index search key is C sorry D and then when I come down and search for D I find this record. So, there are two traversals, but splits are now a couple more topics I will go over quickly. The first topic is called bulk loading and bottom up build I would not get into all the details of how this is done, but the key point I want you to notice what is the cost of doing an insert into a B plus tree assuming that tree is very large it does not fit in memory I may have to traverse down the various nodes of the tree and then land up at a leaf and insert a value at the leaf. If the tree is much bigger than memory and I have a series of random inserts into a B plus tree random order the probability that the leaf I am looking for is in the in memory buffer is small. So, most insertions will land up in requiring an IO to fetch the leaf and then an IO to write it back. So, two IO operations per insertion now if I am inserting a lot of records this becomes very expensive. So, this is one of the reasons if I told you if you use JDBC one of the reasons it is slow is that there are overheads to copying it to JDBC and so forth. The other overhead is to insert into the indices one record at a time. So, an alternative to this is what is called bulk loading and most databases support it and the trick is relatively simple what they do is when I am inserting a lot of data they will first sort that data and then insert it in sorted order. Now, if I insert in sorted order then the next record to be inserted will be in the same page for a while perhaps and after the some records are inserted may be the next record will move to a subsequent page and so forth. But I would not go back and forth between different leaf nodes I will just get the thing sorted. So, let me use the whiteboard let us say this was the leaf level of a B plus tree and it is divided into multiple blocks and there is an index above that those are the internal nodes. I have some new data which I want to insert which is a lot of data. So, the first step is sort new data by data I mean actually just the entries what is the entry it is a search key value and a record pointer. So, I sort that on the search key of this tree now the first thing few things may go to the first block the next few may go to the next block there may be a gap the next one may go there and so forth. So, I will go through the leaf levels of this tree exactly once and similarly for the internal nodes each node will be accessed may be multiple times consecutively one after the other and this is a lot more efficient than random inserts into the tree. Now, there are for this is for inserting into a tree if I am building a B plus tree fresh there is a further optimization that I can do what I will do is I will sort all the entries to be created and then build the upper layers of the tree in a special process which is called bottom of construction. I will skip the details and so there is this bulk loading and bottom of B plus tree construction I wanted to expose you to the term you can read it up later. The next topic which I will briefly mention is multiple key access. So, here is a small example I want to find instructors in the finance department with salary 80,000 this is a very a idiotic artificial query, but there are more realistic examples where I will look upon two attributes or more attributes. So, what can I do if I have an index on a department name I can use that and fetch those instructors and check if their salary is 80,000. If I have an index on salary I can find instructors with salary 80,000 fetch their records and then check if department name is finance. A third option is if I have an index on both of these separately I can use the department name index to find pointers to all records pertaining to the finance department. Use the index on salary to find all records with salary 80,000 and now take the intersection of these two sets of pointers to find the final set of instructors in the finance department with salary 80,000. This approach is what web search engines do when you give multiple keywords they will find a list of web pages containing the first keyword, a list of web pages containing the second keyword and then do an intersection of these lists to get your final answer, but there is one other possibility in our context which is to create an index on a composite key. So, the index could be on department name comma salary. Now, when you have a composite key you need a ordering. So, all of the B plus 3 operations say is it less than this key or is it greater than that key. If I have two attributes how do I do less than greater than comparison and that is lexicographic. So, I have had values a 1 a 2 department 1 salary 1 and another one department 2 salary 2 how do I decide on the short ordering. So, this is less than that if either a 1 is less than b 1 or a 1 is equal to b 1 and a 2 is less than b 2 that is the standard lexicographic short ordering. So, I just create an index treating this as one attribute and using lexicographic ordering otherwise the B plus 3 algorithms are exactly the same. So, now I can use a combined index on department name salary to find instructors where department name is finance salary is 80,000 this would be very efficient. If you contrast with using separate indices here there may be many people in the finance department there may be many people with 80,000. So, I may spend a lot of effort finding those record IDs and then intersect them and I may get very very few, but if I have a combined index I can very efficiently find just those which have finance department salary 80,000. The same index on a composite key can also handle some other queries efficiently take this query department name is finance salary less than 80,000 that can be handled very efficiently they are all consecutive records in this search index. However, I cannot handle efficiently department name less than finance and this should be salary not balance. So, I cannot handle this efficiently because the records that satisfy this condition name less than finance and balance 80,000 are going to be scattered all over that index. So, it is not possible to find them consecutively and there is a much higher cost to running this query. Whereas, this previous query department is finance salary less than 80,000 they are all consecutive. So, this is very efficient. So, that is how indices on multiple attributes work to wrap up this session I think this is there are two last slides left. The first slides is index definition in SQL the SQL standard really does not dig into index creation, but most databases have command like this create index index name on relation name list of attributes. So, here create index some name on this is old relation branch we do not use it anymore, but imagine there is a relation called branch with attribute called branch name. So, you could create an index on that what kind of index by default most databases would create a B plus tree index, but many will let you control that you can also drop indices. So, what are the other kinds of indices which exist there are many types, but the popular ones there is something called bitmap index which are used in decision support system. They are not very useful for online transaction processing systems, but they are very nice for decision support these are used for attributes with very few distinct values like gender may be the lot of marketing analysis once know the income level of people. They do not want to know your exact salary, but they want to know if you are in socio economic level 1 2 3 4 5. So, the only few different values then they may make a decision based on may be your age which again they may bucketize into a few categories not exact age, but age range. Each of these have only a few distinct value and a different kind of index is very useful when the number of distinct values is smaller. And they are particularly useful for aggregate queries tell me how many people are in socio economic level 1 in the age bracket 20 to 25 who are male. So, these kinds of queries or can be done efficiently using such indices. Then there are hash indices which are very widely used for in memory indexing in inside of databases, but these days they are not common for on disk indices. And lastly R trees somebody had asked a question already about R tree. These are used to index geographical or spatial data multi dimensional data. And as I said earlier you can efficiently find all shop within a specified region rectangular circle using R trees with a B plus tree index I could build an index on x and y the coordinates latitude longitude. With that if I say find me what is located at exactly this latitude longitude find a shop. Let us say I have lat long for shops find me a shop that is located at this lat long I can use a B plus tree, but if I say find me a shop that is located near this I cannot do it. If I say find me shops that are located in this rectangular circle I cannot do it efficiently using a B plus tree. So, that is the end of that chapter. There is a lot more stuff in the book and in the full version of the slides which have skipped. What I will do is we are at lunch time, but let me take a few questions may be 5, 10 minutes max worth of questions. And before we do that your afternoon session today is on web application development using servlets. So, I think all of you have the background and hopefully you know enough Java to manage this. Those of you who do not know Java might have struggled a bit in yesterday's assignment, but we have given templates for yesterday's and today's assignment. So, even if you do not know Java very well you can take the templates get a hang of how the language looks like and then modify it to get what you need. So, we can take a few live questions. Teherna Engineering College, Naveen Mumbai please go ahead. I have one question on indexing sir. How many indexes can we have create on a relation? Is there some maximum limit which could impact the performance of the database or is there a minimum limit or we can create as many indexes we want like please could you tell something along those lines. That is a good question. So, every index which you create is a kind of overhead. First there is a space overhead for storing the index. The second is every update insert delete which affects the attributes you know in updates which affect the index attribute. As well as any inserts or deletes all of them have to update all the indices which are relevant. Insert deletes have to update every single index. Updates have to index whichever indices are on the updated columns. So, there is a time overhead to keeping all the indices up to date every time an insert delete or update happens. So, that is this part which is performance related. So, you do not want to create indices indiscriminately. You only want to create it if it is useful to you. So, if you have queries which make use of the index you should create a index. That if you create an index nobody uses it, but still the price of keeping it up to date is paid by all the insert deletes and updates. So, that is a database tuning question. So, there are tools which are offered with commercial databases which help you analyze the workload. You have to somehow collect the workload which are what are the queries inserts deletes updates which are happening. Based on the workload which you observe let us say for an hour or half an hour or a day or whatever. Based on that workload the system can recommend what indices to create or what indices to drop. So, that is one part of it. The other part of your question is is there a limit which is imposed by the database. Many databases do have limits on the number of indices they will support. The reason is that they have some data structures which are more efficient if you assume maximum number of indices per relation. So, what is that limit depends on your database. Obviously, you cannot cross that limit, but that limit is an implementation decision. It has nothing to do with the overheads for insert deletes, but it may be related in the sense that if you have 30 indices you are probably paying a high price and they believe that it does not make sense to have more than 30 indices. So, they did not bother to support it. Usually you do not need all that many indices. Yeah, another question. Hello. Sir, I have one more question. What kind of storage structure is used to store audio and images files? As we have stored that images are having B-Lob data type, but what kind of storage structure is used? And one more question is there. What kind of indexing technique is used for semi-structured data? Okay. So, let me answer the first question. What kind of storage structure is used for large objects like images, videos and so on? Now, the first thing is most applications keep these outside of the database. So, they use the file system directly. Now, what if you tell the database to store this? There are several approaches which are possible. One approach is that the database directly uses the file system to store these things, but that could result in a lot of files. So, that may not be a great idea. Another approach which could be used is to use B plus tree to store blocks where the blocks contain data from these objects. So, what you are doing is you are storing these objects as blocks, but which all blocks are there in a particular object. You could link those from a B plus tree. So, there is an index on the ID of that object. And through that index, you can find what all blocks contain data for that object. And if the object is large, it may have multiple blocks. The index will tell you the key for that index will be object ID of the object followed by the block number. So, you can fetch the first block, second block and so forth. So, the database might manage itself without creating many separate large files. In fact, it turns out that if you create millions of files in the file system, every operation on that file system goes for a toss. We have had pretty bad experience even with new versions of Linux. If you create let us say 9 million files in a directory, you do LS on that directory, it hangs. It takes minutes for the LS command to even return. You cannot kill it. You cannot do anything. It is terrible. So, it is usually a bad idea to create millions of little files. And the database might choose to put them all in one large file, but keep an index to record which blocks of the file contain blocks of which object. So, that is how large objects would be stored inside of the database. There are issues in compaction and so on which we would not get into. Now, coming to the other question which is indexing for semi-structured data. Let us say XML and so forth. So, how do you index XML data? So, there are what are called path expressions on XML data and you can build an index to answer a path expression whose tail contains a specified value. So, there are other kinds of indices in that context. Again, we do not have time to get into all the details. There could be keyword indices on semi-structured data. There could be indices on paths along with selection on value which is at the tail of the path and there are other types of indices. So, I will leave it at that. Maybe I can take one or two questions from Cache. One of the questions was by default, an index is created on the primary key of the table. Should we use create index to create additional indices? How are they managed and so forth? So, first of all, an index on the primary key is usually essential because without that index, if you insert a record, how do you check if the primary key value has been duplicated? That is an integrity constraint violation which has to be checked and the most efficient way of checking it is to have an index on the primary key. So, these are created by default by every database. The second kind of index which is usually automatically created is if you declare a foreign key constraint from R to some other relation S. On relation S, you have declared primary key and index is created already. So, that is not an issue. But on the referencing relation R, there is an issue. Suppose I want to delete a tuple from S. I need to know if any tuple in R is referencing that tuple. If it is, maybe I should not allow the deletion on S or if on delete cascade or so on has been specified, I need to find which all records in R are affected and cascade the delete to those records. For all of this, I need an index on R on the foreign key attribute. So, most databases would also create automatically an index on all the foreign key. If you declare any attribute or attributes of R to be foreign key attributes referencing some other relation, an index will automatically be created on the, on each foreign key. For each foreign key, an index is created. So, these two are the default indices which are automatically created. Beyond this, anything you need is up to you. Somebody else has asked a question about what is the column store index in SQL server and where is it used? So, column stores are something which I did not get into, but let me write the word here so that people are familiar with this. What is a column store? The idea is if I have a relation with multiple tuples, each tuple has multiple columns. Now, all the structures which we saw till now store a row, a record is a row and that is stored in a block. Now, what people found is there are many cases where relation has many attributes which are rarely used. They might be quite big, but they are rarely used. Most of the accesses to some relations happen on only a few of the attributes and many of them access may be just one attribute or two attributes. So, if you store a whole record together in a page, if I want to access even one, just one column of that relation, I have to fetch the whole relation which means I am fetching a lot of columns which never get used. There is a lot of overhead to fetching the columns. So, they said when for decision support queries where this is common, what they did is they broke up the relation slightly differently. They broke it up into columns. Let us say this is columns A, B, C, D. I will store all the values for column A in one place, all the values for column B in another place, everything for column C in third place, column D in a fourth place. I am storing only the value in that column and I am storing it exactly the same order it was stored in the original relation. So, updates and so on can cause trouble. There are workarounds to deal with that. Let us not worry about those issues. I had a relation stored in this way here. This is called a row store. In contrast, in a column store, the data is stored for each column separately. So, there would be one file containing all the values for column A, one file for all the values from column B. Now, one of the key things is you have to be able to know which value from here corresponds to which value from here. Assuming the data sizes are different, this is the 73rd thing here. It will correspond to the 73rd thing here, to the 73rd thing there and so forth. It may be 73rd value is here. So, these together form one row, but they are physically stored in separate places. So, the first step is supposing I have a selection which says find me something where column C. So, let us take a select A from this relation R where C equal to 51. So, what I would do is first I need to index C to find where the value 51 occurs in C, where all it occurs and that index might tell me it occurs at position 93 and 27. Now, what I do is I go to the file for column A and go to position 93 and 27 and fetch those values and output them. That is all that that query required. So, I have avoided fetching most of the data for this relation. So, in this particular example with just C equal to 51, a row store would have done fine. It is actually not such a big benefit for queries which retrieved a small amount of data, but queries which retrieved a lot of data let us say aggregate on column A. Then reading just column A sequentially and aggregating it is more efficient than reading all the other unnecessary columns and then aggregating. So, the primary use is for decision support and the first column stores that were built I think it started with Sybase. They had a factor of 10 performance improvement over the earlier system and lot of people were really surprised you could get such huge performance improvements. Of course, it does not work does not give you a factor of 10 improvement for everything, but for certain applications you could get a lot of improvement. Couple more questions here. Which databases are using B tree? Now, if you look at the textbook we differentiate between B tree which is the original proposal and B plus tree which is the version we use. In industry nobody bothers to say B plus tree they just say B tree, but what everyone implements is B plus tree. The version we gave is what is implemented by everybody and it is called B tree in the industry they do not bother saying B plus tree. So, all databases supported that I know. There are some unrelated questions people are asking questions about the JDBC assignment. Maybe I should spend a minute on that because it is going to affect today's assignment also. So, one of the things which people have said is that error JDBC driver not found. So, if you are using eclipse as we suggested the JDBC driver has to be put in the search path for eclipse. So, if you look at the instructions for JDBC and read it carefully this is given in the instructions. We tell you that you should add the place where you have placed the post office driver that needs to be told to the eclipse ID. So, please follow those instructions carefully if you skip one of those steps it may not work. And what some people found is that in the JDBC part it worked, but when they came to servlets when they tried to deploy a servlet at that point it did not work because they had done the first steps somewhat differently. So, please follow the instructions fairly religiously to ensure that the eclipse system knows where the files JDBC driver files are otherwise you get these errors. Sometimes like I said in the plain JDBC assignment sometimes only in the servlet assignment which is today's assignment. So, with that we will stop in this session and I will see you tomorrow.


Let's finish our data science methods overview by getting a brief overview of machine learning. Now I got to admit, when you say the term machine learning, people start thinking about something like the robot overlords are going to take over the world. That's not what it is. Instead, let's go back to our Venn diagram one more time. And in the intersection at the top between coding and stats is machine learning, or as it's commonly called is just ML. And the goal of machine learning is to go and work in a data space. So you can, for instance, take a whole lot of data, we've got tons of books here. And then you can reduce the dimensionality, that is, take a very large scatter data set and try to find the most essential parts of that data. And then you can use these methods to find clusters within the data, like goes with like, you can use methods like K means you can also look for anomalies are unusual cases that show up in the data space. Or if we go back to categories, again, I talked about like for like, you can use things like logistic regression or K nearest neighbors K and N, you can use naive bays for classification or decision trees or SVM, which is support vector machines, or artificial neural nets, any of those will help you find the patterns and the clumping in your data so you can get similar cases next to each other and get the cohesion that you need to make conclusions about these groups. Also, a major element of machine learning is predictions, you're going to point your way down the road. The most common approach here, the most basic is linear regression, multiple regression, there's also Poisson regression, which is used for modeling count or frequency data. And then there's the issue of ensemble models where you create several models, and you take the predictions from each of those, and you put them together to get an overall more reliable prediction. Now, I'll talk about each of these in a little more detail in later courses. But for right now, I mostly just want to know that these things exist. And that's what we mean when we refer to machine learning. So in some machine learning can be used to categorize cases and to predict scores on outcomes. And there's a lot of choices, many choices and procedures available. But again, as I said, with statistics, and I'll say again, many times after this, no matter what, the goal is not that I'm going to do an artificial neural network or an SVM. The goal is to get useful insight into your data, machine learning is a tool and use it to the extent that it helps you get that insight that you need.


Overall today, but a particularly rough day at the plate for Gary. What were you seeing from him this afternoon? I mean, that's Tampa can do that to you're facing Snell. I mean it can be it's a challenge. So You know, obviously a tough day for him and look forward to him getting back in there tomorrow behind the plate and Getting back after it. I mean They're they make it about as tough as anyone on you with what they're able to bring in from the pen And obviously with Snell going today Because of the events of yesterday and how well your team has been playing when Luke Voight leads off there with a single Do you have the feeling here? We go again. Oh, no question You know, you got that part of the order coming up Luke has a really good at bat Aaron gets him to 3-1 So yeah, you're definitely feeling like You know, we got a chance to do something there and you and you expect it and But they got us today and credit to them for for making enough pitches It was announced that the X-rays are negative for Tanaka. Are you at all concerned? That's going to affect him on the mound. Well, we'll just have to see I mean that was you know, that was brought to me later in the game Obviously that X-rays were negative so that was good But we'll just have to see, you know, how he comes in tomorrow and and how he bounces back here in these next next couple of days No, he was he was done. Yeah, it got him good What are you what is the difference you see in Tanaka with the last two stars compared with the other, you know, he's He's been mostly good all year Even in the outings where he struggled if you look at him It's just you know a stretch of batters that have gotten them in trouble or you know Maybe one or two starts where he didn't have it which is going to happen every now and again I think for the most part. He's been pretty good He's been kind of searching for that nasty split that he hasn't really had yet But he's been pitching really well and I thought again today the slider was such a good pitch for him The split was a factor And I thought he I thought he really executed with his fastball To really keep them on us. He didn't make any mistakes with his fastball And and you know, was it really in command of that game? You know till till finally had to come out. So Another strong effort from him Koli to the right With Sessa having to go there in the 11th. How does that impact tomorrow? Well, we'll see I mean, that's something we'll meet on now and kind of figure out which way we want to go You know and see how much How much availability and length we would have out of Louis So we'll talk about that now and figure it out Randy Right you used all your high leverage guys except for out of you know, I think he's only pitched 24 pitches in the last eight days or so. Is he okay? Yeah, he's good We're gonna use them, you know, if there was traffic for those righties coming up at the bottom behind Tommy but once they tied it and there was no one on felt like we didn't want to burn him there, you know And then Tommy was able to get out of it and we're lined up and now But no no issues with auto Dan and Joel Then why not go with him instead of? Sessa Because because we're gonna turn the game over at that point once we're through there We're gonna go Sessa you start going to Otto and then you got nothing, you know Following up that day. So we're gonna ride Sessa there till the end and like the way he's throwing Were you going one inning at a time because you do have an eye on tomorrow that you might need multiple guys to get through the game? Well, I mean I don't I Mean in a way, you're always going one inning at a time But we're comfortable once we got to that spot Then we're gonna we're gonna turn Sessa loose and hope that he could you know pitch us a few innings if we needed If the score kept kept where it was So yeah, we're comfortable with with how it rolled out Anyone else coley how comfortable were you with Aaron Hicks going through the top sign there? I think that was a sixth inning Was that you know an aggressive right take so to speak? You know, obviously it was pretty close play at the plate I think the throw was probably a little bit offline that made it close. So Look two outs being aggressive there. Obviously, you know an important run at that point you know, I think Phil made the right call and Just something we'll kind of talk through and didn't work out for us Joe considering what you haven't had What do you think? How do you assess how you're starting pitching has done this so far this year? Really good not talked about enough Because you know even even when they've had shorter outings for the most part we've been in the game Night in and night out and it's because of our starting pitching You know, I think a lot of people Rightfully, so talk about you know our pen and all the young guys that's come up and the different people that have come into the Organization but what allowed us to consistently win ball games is for the most part we've gotten good starting pitching And and today was was another case of that Anyone else? Thank you