We are going to start this course on nanostructured materials, synthesis, properties, self assembly and applications. The course has four modules and today we will be doing module one and the first lecture of module one which is the introduction to nanotechnology. And I am from IIT Delhi, department of chemistry and I will be giving you 40 lectures on this subject. Now, before we go to nano materials, we have to know what are materials. So, there are different kinds of materials which you can observe around you in your daily life. Some of the materials are natural like garnet or quartz or gold or silver or marble. There are many naturally occurring elements and compounds which you can see which occur in the earth. And there are man made materials. For example, you have a cement or a compact disc or an IC chip which are made up of different kinds of materials. For example, silicon is the material for an IC chip and you have several interconnects made of silver which connect which make the connections in an IC chip. Apart from that you can see there are lot of drug molecules which are used in pharmaceuticals or dyes and pigments which are used in daily life. These are man made materials and most of the materials that we can see with our naked eye has to have a size which is within the dimensions of the visible rays which is 300 to 600 nanometers. We are using visible rays to see them with our eyes and we cannot see objects which are smaller than 0.07 millimeters with our human eyes. Now, if you look at these materials more carefully that is inside these materials, if we have something by which we can observe objects much smaller than 0.07 millimeters like using electrons then we can see that these materials are made of particles or grains which are of the size of 100 to nanometers to 1000 nanometers etcetera which is like 1 micron or to 10 microns. This is the normal size of grains in any solid which we find around us. However, as you know that our eyes cannot see these small size grains. So, we have to use what is called an SEM which is a scanning electron micrograph through a microscope and this scanning electron microscope uses electrons which have a wavelength which is correspondingly smaller much smaller than the wavelength of visible rays and hence we can see objects which are much smaller. Now, a micron is 1 by 1000 of a millimeter. So, it is much smaller than 1 millimeter and if you want to observe still smaller objects like atoms in a solid then you have to use even smaller wavelengths because the distances in a solid on the atomic scale is of the order of 0.1 to 0.2 nanometers. Typically in a crystal which is of say 0.1 millimeter in dimension will have around 10 to the power 21 atoms. So, there are many many atoms in a small crystal and if you want to look at the distance between two atoms it is of the order of 0.1 to 0.2 nanometers or 1 to 2 angstroms and for that normally we use x rays because x rays have a wavelength which is of the order of 1 to 2 angstroms. Now, using these x rays we can observe the structures like diamond or graphite or we can observe changes in the patterns of the arrangement of atoms like a dislocation where atoms have moved from their original positions. However, these in between this atomic limit and the micron size limit which I discussed earlier which is found which can be seen using a scanning electron microscope. There is a regime which is the nanoscale which we are discussing in this course. So, what is this nanoscale? So, to give you an idea of what is this nanoscale which is of the order of 1 to 100 nanometers how to understand how small is a nanometer. So, to give you an idea if you compare the size of a football or a soccer ball with the size of the earth then you see that the size of the football is around 10 million times smaller and now if you compare a nanomaterial like a C 60 molecule which is also called a fullerene then the C 60 molecule is nearly 1 billion times smaller than the size of a football. So, you can now imagine that how small is a nanoparticle if you know how small is the football with respect to the size of the earth and the fullerene molecule which is the nanoparticle is much smaller than the ratio of the earth size to the football as compared to the size of the football to the nanoparticle. Now, nanoscale materials can be seen in many many places. For example, you can see in gold you can make nanoparticles, you can make silver nanoparticles, you can make nano shaped objects like a quantum coral which is made by having iron atoms on a copper surface as shown here and you can see a three dimensional structure of silicon carbide nanowires in the form of a flower and the size of these flowers are within a few nanometers. So, you can have artificially made structures which have dimensions close to a nanoparticle that is in the order of 1 to 100 nanometers. So, that means these particles are very small if you have to understand nanomaterials you have to see them using light rays which have wavelength of the order of 0.1 to 1 to 10 nanometers of that order and that kind of wavelength you can achieve using electrons and so electron microscopy is important and you can see in these pictures which are electron micrographs of nanowires of carbon like carbon nanotubes which are oriented in some fashion and these nanowires and nanotubes have very interesting physical electronic and optical properties. So, nanomaterials which have this small dimension of few nanometers can be in one dimension and the other two dimensions can be large or it may have this small dimension of nanometer size in two directions or even in three directions. Accordingly we call them as nanodots or nanoplates or nanowires depending on the dimension of these nanostructures. Now, if you have a two dimensional quantum confinement which is possible if you have a nanostructure like a plate with one dimension which is nanometer size then you can see what is called two dimensional quantum confinement because only in one dimension you have the effects due to the nano size. There are many applications of these nano devices you can use them in sensors and actuators like motors which are called nano motors. You can use them in very efficient photovoltaic devices you can use them in transistors and diodes as well as in lasers and a lot of work has been done and is currently being done on how to increase the efficiency of a solar cell which converts the rays of the sun into electricity which is a very important problem for the world to harness the solar energy and convert it into useful form of energy so that the energy problem of the world can be solved easily. There are other nanoscale materials or nanomaterials which have relation to biological molecules and these are also having dimensions in one to hundred nanometers. The examples are proteins, enzymes, DNA, RNA and of course peptides and there are many applications in biomedical science which are based on the nanostructures and the utilization of the biological nanostructures in medical science. Some of these nanostructures can be combined with other inorganic materials to form bio conjugates and they can be combined with polymers or carbon nanotubes and can be applied to several problems related to drug delivery or disease mitigation. There are here some nanostructures of porous silicon which is biocompatible and hence can be used with biomolecules. Similarly, there is an example of a human cell which has been fixed on to porous silicon. There is an example of enzyme which can be used as a catalyst which are already known to be working as catalyst in the life processes and they can be used to have some artificial reactions carried out in a similar environment outside the living bodies. You can have bone cell grafted on porous silicon and studies have been done how these bone cells can be made to grow on artificial surfaces. Let us now understand what are the properties which change as the particle size of the becomes very small. So, what is very small here is with small we mean the size of 1 to 100 nanometers which is not a rule, but it is a norm which is considered by all the people who work in the area of nanomaterials to consider any material which has a size in this range to be considered as a nanomaterial. And as defined 1 nanometer is 1 by 1000 of a micron or a micrometer and typically in 1 nanometer you may have 3 to 5 atoms. To give you another idea of this size of nanometer if you take a human hair and measure the thickness of the human hair that is the diameter of the hair then the diameter of a human hair is of the order of 100 micrometers which means 1 lakh nanometer 10 to the power 5 nanometers. So, you can understand that a human hair is very large compared to the dimensions that we are talking about which is 1 to 100 nanometers. So, unusual properties are seen when you make a material which is of nanometer dimensions. One of the properties is that electrons which normally are delocalized in a metal become confined that means they are not so free anymore and then the property of the metal which is due to the delocalization of electrons is affected and the metal becomes a non-metal. This is considered to be a quantum effect because you change the allowed energy levels of the electrons by confining them in a region. There are other properties like the surface to volume ratio of atoms changes as you decrease the size of the particle which means that if you have a smaller particle it will have more atoms on the surface compared to the bulk and hence its properties like reactivity and other things will change as the size of the particle changes. There is another property which depends on the orientation of dipoles which may be electric dipoles or magnetic dipoles and a particle having these dipoles will show change in the arrangement of these dipoles as you decrease the size of the particle. From a more ordered arrangement in a larger size particle you will go to more disordered dipoles as you decrease the diameter of the particle. On the surface of the particle you will have more disorder and since surface increases with size with decrease in size of the particle hence the disorder of the dipoles increases with decrease in size of the particles. Now if you look at a scale where our nano scale objects fit then you see that if you go from the smallest particles or smallest numbers that we can see like 10 to the power minus 15 of a meter which is called a femtometer and you go up you see that the size of the atom is around 10 to the power minus 10 of a meter which is 1 angstrom and then you see a water molecule is something which is below 1 nanometer but more than 1 angstrom and then if you go to a DNA double strand then the size falls into the nanometer dimension. The diameter of a DNA is around 2.5 so in 2.5 to 5 angstrom so it is near to the nanometer size and the diameter of a carbon nanotube is higher than its around say 5 to 10 or 20 nanometers. So typically these objects fall in the nano dimension of course if you go to larger sized objects like bacteria or RBCs they are much larger they are like 200, 300 nanometers to 500 nanometers or micron size. So we see a small region in this entire length scale where you will have objects of the size of having the nano dimension like the DNA strand and the carbon nanotubes and the wide assays. Since some of these biomolecules are falling in the nano dimension hence the subject of nanotechnology is equally important to chemistry, physics and biology. So as I explained to you why these properties of surfaces which dominate as the size of the particle increases and the disorder of the dipole also increases as the size of the particle decreases. So these two properties are depicted and you can see that for the same area if you have a smaller sized object you can you will have more number of particles to cover the surface and similarly if you have a smaller sized particle the bulk which has ordered moments reduces in dimension and the number of disordered dipoles are increasing as the dimension of the particle decreases. So just to again summarize what is happening as we are going from large sized particles to small sized particles. So if you go from a millimeter sized particle to a micron sized particle there is not much change in the laws of physics the physical laws are more or less same. However as you go from the micrometer size to the nanometer size you are entering the quantum region and so the laws of physics and the effects differ considerably. Apart from this many of the biological molecules fall in the nanometer length scale and hence the size dependence of the biological properties are also important. So electronic properties change like you have new effects like quantum size effects they are called and examples are single electron tunneling or macroscopic quantum tunneling these are seen whenever you confine electrons and you start seeing these quantum effects. Physical properties like melting point hardness etcetera we are also dependent on the size of the particles and we see them in many cases. Then chemical properties change like the structure catalysis reactivity they all change as you change the size of the particle. For example a small gold particle is shown along with a large gold particle and you see the large gold particle cannot oxidize carbon monoxide to carbon dioxide whereas the small particle of gold can oxidize carbon monoxide to carbon dioxide. So interfaces and surfaces become very important and this is of course known for a long time in catalysis and they were not called as nanoparticles but they were called as fine particles. But in the last 15 years we now know that these properties can be due to the nanomaterials and now we say that the enhanced reactivity is due to the nanoparticle nature which enhances the surfaces and hence the reactivity. You can also see optical effects for example if you look at this butterfly then it has got on its wing blue color as well as brown color and the color actually changes because there are some nanostructures on the wings of these butterflies. Now these nanostructures interact with light in a manner such that if you change the dimension of the nanostructures the wavelength of light which is scattered will change and hence you see a different color and this subject of interaction of light with nanostructures is today called nano photonics where you can organize elements such that the difference between two nanostructured elements is of the order of the wavelength of the light which is interacting and minor variation in this dimension of these nanostructures will create new scattering phenomena and this subject is called nano photonics. You can this case of the butterfly is a natural nanostructured feature showing nano photonics however you can make artificial surfaces which are nanostructured and there also you can observe variation in the scattered wavelength of light depending on the distance between two nanostructured elements. You can also see the effect of optical properties on the dimensions by choosing a very well known element like gold which all of you know that it is yellow in color however if you change the size of the particles of gold you can have gold of different colors depending on the size of the gold particles. So you can have blue colored gold, orange colored gold depending on the size of the particles and you can explain why a particular size of gold gives a particular color. This can be very well explained today based on our knowledge of nanoscience and nanotechnology. This is another example of how you can understand why surface reactivity will be large if you take a large cube and cut it into several small cubes the surface area will increase and hence for the given volume you have much larger surface which is characteristic of nanoparticles and hence enhance any property which will depend on the surface like reactivity or catalysis etcetera. So, the surface area is a very important property for catalysts and for the increasing the reactivity and there are many other applications in thin films where you use certain nanomaterials to act as thermal barriers to prevent something from getting too much heated or the wear resistance of materials which are being used rapidly and these kind of wear resistance of materials can be enhanced by decreasing the size of the particles on the surface. Now in electronics these sizes very important because you want to make smaller and smaller electronic gadgets and what we call as the feature size that is typically an electronic component which can you can consider an integrated chip. Its size is of major importance because the more number of chips you can put in one square inch of area you will have larger computational properties or any feature like memory properties can be enhanced by increasing the feature size and that means increasing the number of ICs that means decreasing the feature size will help you in enhancing most of the electronic properties and as you see the feature size has decreased from around 10,000 and now in 1970 to it is now close to around 20 or 30 or 40 which is in the nano scale. Now this kind of miniaturization, nano scale miniaturization has advantages because it also reduces the number of chips that the length which electron has to travel to communicate with different circuits within the same chip. So, between different components and this also reduces electron scattering which leads to enhanced heating or resistivity and overall you get a faster operation of the device. However, the more miniaturization you need, you need to design more difficult nano structures and that will that normally enhances the cost of the fabrication process and you have to use much more complicated equipments to design such highly miniaturized devices. Now to give you a brief history of nano science and nano technology, nano science was known for a long time without any understanding. People have been making small particles which today we know were actually of the size of few nanometers in ancient Roman times. They used to make particles of cobalt, silver, etcetera in solutions and used to mix it to form glasses which had color and this was known for nearly 2500 years back. People have made such colored glasses. However, the chemistry of or the exact science behind the color in these colored glasses was not known or not understood. Gold is one of the most frequently used colloidal solutions which has been used in many other applications in those days and this particular cup which you are seeing has got gold which is looking red. When you are illuminating that means you have a light inside. So, you are looking at the transmitted light and that is red in color. However, if you look at the reflected light that means your light source is outside the cup, then it appears green in color and this is currently in the London Museum and it was made many, many thousands of years back and it is still retaining its colored properties. So, this unusual property of this glass is basically because there are nanoparticles of gold embedded in the glass and that gives you the color. The color is different depending on whether you are observing transmitted light or reflected light. The first time scientifically nanoparticles were made in a laboratory was 1857 when Michael Faraday made a gold solution and those solutions are stable even now. As you can see in these chemical bottles, these are precisely those gold solutions which were made by Michael Faraday in 1857 and if you look at these particles under a microscope electron microscope, then you will see the picture on the right hand side which shows you clearly the pattern which one should observe for a gold particle and you can be sure from these spacings between these lines that you observe that corresponds to gold and this is a transmission electron microscope picture of gold nanoparticles from which you can see the size of the particle and also you can prove that it is a gold particle by measuring the distances between the atoms or the lines which are formed by the atoms in real space. Now the scientific thrust to nanotechnology was really given by Richard Feynman who was a physicist and was a very important figure in the development on modern physics and he gave a lecture in 1959 where he mentioned that a lot has to be understood in the smaller dimensions and that is when he said that there is plenty of room at the bottom and this is considered today one of the first time somebody indicated that lot of science and technology is possible in the nano dimension. However, he never mentioned the word nano science or nanotechnology in the lecture, but he kind of indicated that the understanding was not there on the properties or applications of particles in that smaller dimension where we can control atoms and molecules. The next big thrust and actually the major thrust came after Eric Drexler who was a student at MIT wrote about molecular manufacturing in 1981 and another book called engines of creation in 1986 where he said that the put his ideas about molecular machines, nanorobots and many other machinery which could be designed based on molecules and that was one of the major thought provoking books which people lashed on to and made them reality within 10 years. So, Eric Drexler's books though fiction gave the thrust for scientists and engineers to start thinking on really making these molecular machines and his concept of molecular machine started from a natural molecule which is the ribosome and which is there in our body and it is the working of the ribosome is so efficient and based on that Eric Drexler wrote the book engines of creation because ribosomes help in making proteins which are very important for living beings. We all the time are making proteins in our body and ribosomes are one of the key things which stimulate the formation of proteins and the structure of the ribosome and its action is just like a machine and that is why Eric Drexler got his idea of thinking beyond ribosome and planning or of many, many new molecules which would be doing different action based on the structure. And hence this was realized of course of later and the realization was possible because it could be seen using some microscopes which were then developed. One of those microscopes the scanning tunneling microscope was invented in 1981 and we could clearly see atoms using the scanning tunneling microscope. Although even with transmission electron microscope you can see atoms at very high resolution. The bucky ball was discovered around this time in 1985 which is a nanoparticle because the diameter of this bucky ball or C 60 molecule is around 7 angstrom or few nanometers 7 nanometers which is 70 angstroms. So this discovery of C 60 was nearly around the time when nanotechnology was a nanomaterials was being recognized as a very important area of research. However the most important discovery probably was the discovery or invention of the atomic force microscope by the same people who invented the scanning tunneling microscope. And the AFM was invented in 1986 and using this microscope any nanostructure could be seen like non conducting molecules, organic molecules which could not be seen earlier using scanning tunneling microscope. So the discovery of the atomic force microscope probably was one of the most important inventions in the development of nanoscience and nanotechnology. This is a picture of the first writing using atoms done by IBM where they could manipulate atoms and this is really what we call the ultimate of nano engineering where you can move atoms at your will and place them at your will. So this IBM logo which is written here is made of xenon atoms with the volume of the blue ones you can see are actually they have manipulated xenon atoms on a nickel surface and this was first time shown in 1989 and from that day we can now say that we can write with atoms. And what happens when you can write with atoms when you write with atoms you can write many many thousands and millions of pages of your book in one page and of your notebook today. So a typical atom has a diameter of around 2 angstrom so that is the size of the letter you will have 2 to 5 angstroms and so you can imagine how many letters you can write within one page of your notebook which is around say 12 centimeters by 12 centimeters. It will be much larger than what you can write with your pen or pencil and so that is a very very important development that you can store lot of information in very small region of space using atoms or molecules. But you have to be able to manipulate these atoms and molecules as if you are writing with atoms and molecules. So there are several other discoveries around that time in 1987 the first single electron transistor was created. In 1991 carbon nanotubes were discovered. These are tubes made up only of carbon atoms. The carbon atoms are arranged in as hexagons and pentagons and these tubes have the diameter of around 3 to 10, 15 nanometers and these carbon nanotubes were later found to be very important for several applications. You can have semiconductor carbon nanotubes, metallic carbon nanotubes and a branched carbon nanotubes and hence it was predicted that they would be useful for circuitry in miniature electronic devices. Many countries in the world around that time then launched the nanotechnology program. United States launched the national nanotechnology initiative in 2000 and several other nano research centers in the world were established in around 2000 and 2002. Today what we have done so far is try to give you an introduction to the subject of nano science and nanotechnology. We tried to explain what this small size brings about in terms of properties, the variation in electronic properties, a variation in mechanical properties and variation on properties which depend on surface like catalysis and reactivity. The points that you have to remember is first that the nano dimension is typically in the range of 1 to 100 nanometers. Any object or part of an object in this range has some unusual properties. However, it depends on what kind of properties one is looking at. If it is electronic properties then maybe you have to be in the region of 1 to 10 nanometers. If you are looking for mechanical properties you may be having particles which will create a variation in mechanical properties even at sizes like 40 nanometers, 100 nanometers etcetera. So, overall you will have different effects based on what kind of properties you are looking forward. One of the common materials which is a nano material we discussed is based on carbon, the C 60 molecule and the carbon nano tube which is one dimensional linear or it can be curved, it can make y junctions and the diameter of this carbon nano tube of course, falls in the nanometer dimension. The length can be very long and how you observe these nano particles you have to use light which has a wavelength which matches the dimensions. So, you must be using light which has wavelength of nanometers which visible light cannot give you. So, you have to use electromagnetic radiation which has wavelength in the order of sub nanometer like electrons which can have wavelength in this region. You can also use other particles however you have to modulate the wavelength accordingly. Typically the tools we use to study these nanoparticles we discussed today are the electron microscopes and the scanning tunneling microscope and atomic force microscope. So, the TEM, the AFM and STM are the pillars of study of nanoparticles and using these we can observe even atomic we can have atomic resolution and see distances which are sub nanometer in size. With that I think we will close the lecture, the first lecture of this module 1 and then we will start with our next lecture next time and good luck to all of you and please go to the next lecture. Go ahead and look at what you have learnt and try to understand more. Thank you.


This is Cathy Bogan and I'm at Port Botany with Ian Rintel, who's a trade unionist for Palestine. Ian, could you tell us what's going on here tonight? Well, in the last 24 hours, the Ganges, it's a ZIM ship which is carrying ZIM containers. ZIM is a very well-known Israeli company and very connected with the Israeli state. Very much a supporter and offered its services to the Israeli state for the conduct of the genocide. ZIM shipping has actually been a target of protests at ports all around the world in the United States, in Italy, Europe. There have been several demonstrations, including one early in December here but also in Perth and in Melbourne. So we're here tonight because the Ganges is in. We're going to have a protest tonight. We're going to attempt to actually prevent one of the shifts at the loading. We're going to try and make it as hard as possible for those workers to actually get into the dock tonight. ZIM shipping! ZIM shipping! It's an international working class issue. I've never seen a community treating the way the police have treated the Australian and Palestinian community. And of all times in our lifetime, we have never seen such horror on our doorstep across the world. What's happening in Palestine is a crime against the nation. We're that uniform and not believe that that's not the way it is. You're going to have to arrest our members and officials of our union first. Kevogan, Consolidium News. What is the police threatening? They're saying that they're not going to allow any march or protest to proceed across the road here. All right. If people do cross the road, they will be arrested. It's just a sign that it's on hold. Welcome, comrades, brothers and sisters, for allowing us to gather indigenous land, grown year after year. Taking shelter in Al-Shifa Hospital, they tied them to a net and they beat them up about Israel's crimes against humanity. We've seen it over the last six months. A solution. What's the only solution? We have resorted to eating grass so they can stay alive because of the forced starvation that Israel has imposed on them. We cannot continue on as it was before. Powers of this world, terrorists once. Not anymore. If the last six months taught us anything, is that no institution, no institution around the world. Honour to introduce our first speaker. Dr. Maureen Farouki is the Deputy Leader of the Australian Greens and Senator for New South Wales. She became to sit in any Australian Parliament when she joined in 2013. Maureen has been an unflinching voice on social, climate and racial justice. She has been the strongest voice for Palestine and Australian Parliament and has continuously called for an end to the occupation and apartheid. Please welcome Dr. Maureen Farouki. Assalamu alaikum, everyone. Peace be upon you. I acknowledge the sovereign owners of the lands we gathered on, the Bidigal, Weigal and Carigal people and pay my respects to elders past and present. We are on stolen land. Sovereignty was never ceded. This is always wars and always will be Aboriginal land. Thank you for coming together today to block the boat. For showing up for Palestine, your solidarity is seen. Your solidarity is appreciated. And I know that you won't stop until Palestine is free. 69 days of Israel's genocide on Gaza. 169 more than 30,000 Palestinians have been slaughtered by Israel. More than one disease loomed large in the ruins of Gaza. That's the reality on the ground right at this moment. And how humanity of morality to stop these war crimes, this collective punishment, these atrocities and this genocide. How ruthless and cruel can you be? And cruel can you be to aid and support Israel for this genocide against Gaza? This heartlessness exists right in the Australian Labour Government. Clear message, not right to protest. We have been marching on and we will not stop doing that. You know, they try and disparage us in history. Remember this, you have in our so-called leaders in the Labour Party and in the Labour Government. What they see, that's why they want to stop you. Shamefully, that is why they've unleashed the police force to protesters. Shamefully, that is why they use the draconian anti-protest laws to create fear and to stop you again. And to undermine democracy in this country. Shamefully, that's why we have seen police crack down, police crack down just even in the last couple of days on peaceful protests. But you know what? These tactics are used to intimidate, to scare us. These are aggressive tactics and we condemn them. We condemn these police tactics and we condemn the Labour Government for using these tactics to stop protesters from standing up against genocide. Is with every single person who is risking their safety and their freedom to stand up with Palestinians and to stand up against Israel's genocide who are showing and using their people power to support Palestinians. Business as usual as they slaughter Palestinians. Israel is a genocidal, apartheid state. And you know, we must cut ties with Israel so no companies who allow crimes and war machine like ZIM shipping should never for sanctions. We will continue to boycott, to this genocide and apartheid. Irene said, we have an inherent right to protest. We have a right to be on the street, the opportunity we get. This is not normal what is happening in Palestine and the terrorist state of Israel is relying on us to become desensitized and atrocities to become normalized. They are relying on us to be tired. They are relying on us to get used or just shipping company in Israel. ZIM has played anization of Palestine. It has remained central to the operations of Israel's war machine by transporting weapons to the Israeli military. The same in position of apartheid. Before I pass on to the next speaker, I will say this. I am a member of the Maritime Union of Australia and trade unionist for Palestine. We know there has been acts of defiance in everybody's worth places and mass mobilizations across cities. And we have to keep up that pressure because while children are being bombed, while cities are being as usual for ZIM shipping, found the crowd acting as, well, let them know where you're going to be after this. And we all assemble here. We stay here and safe together. Is everybody there? We're safe. We're here. Here is a situation where a friend of yours is arrested. Please find someone in the orange vest and let them know. We're hoping that doesn't happen here tonight. So we are going to stay safe, but that's the process. Find someone in the vest. They'll log that name and we'll work from there. Thank you. It is an honor to introduce our next speaker, who's been a staunch advocate, not just for Palestinians, but for all oppressed people around the world. Please welcome Paul Keating, Secretary of the Sydney branch of the MUA. Great. Firstly, I want to also acknowledge the traditional lands of the people that we meet here today. Pay our respects to their elders. And it is so important to understand the brutality that is faced with First Nations peoples all over the world. We continue to fight for the struggles for our First Nations peoples. And what we know is the brutality of colonialism and imperialism. And we're here because of the genocide, the indiscriminate horror that is unfolding on the other side of the world in Gaza and the occupied territories.  We stand with the Palestinian people, and rights of self-determination. And we're here as working class men and women in communities throughout the generations when we fought the hatred and the indiscrimination of the ruling class. And Zionism is a hateful, hateful ideology. It's a political ideology. And it must be stopped. We stand with our communities in our right to fight for political change. So I call firstly on all politicians, but firstly on the federal labour government, Albanese, you must put on sanctions now for the MUA. We stand with our communities and throughout the generations we fought against the establishment who have supported apartheid, like we've seen with South Africa, like we've seen with the wars that have forced ordinary working class men and women like ourselves and our communities into the most desperate of situations. We oppose war. Peace is union business. We send the message. And I want to send a message from our members to you, yours, and they stand with you here in New South Wales. And on the side of the communities that voted you in. We're amongst comrades from Palestine here. Sincerely comrades, fraternally from the MUA, we are with you. And we are with your communities, with your families. And we are with the people of Palestine as the conservative and the supporter of Zionism. You can't have a political party that says we stand on the side of our First Nations peoples in their right to self-determination and not stand on the side of all First Nations, all indigenous, the world-wide with our communities, with the Palestinian communities. And fight. Mobilise and fight. Come out in the tens of thousands, the hundreds of thousands that aren't mobilising to change this world. Class no. And the ship owners no. Is when we stop capital. We say this on behalf of all our affiliates in this region. Mobilising. We are standing alongside our communities across oceans when the communities come down to our docks and put on pickets. Put on communities that destroys the bargos, both important exports on Israel. Must be the case. We have in this country secrecy laws that don't leach the right to know what they're moving. It's a disgrace. Never used to be the case, but I want to congratulate the Greens Party, especially for standing on the line with our communities and the courage of our comrades there who come out time and again to call system. Mr. Sidney, Titians out there, make sure you vote them out if they don't stand on the side of the... This lastly, to the ship owners, to the international shipping. You don't want pickets that protest in your ports, Israeli made or own goods. Stolen land like the Palestinian land. 1948, the Nakba. Palestinians were ethnically cleansed. They were massacred. The lands were stolen. The Syrians have been suffering in the diaspora. Inside the refugee camps in Gaza, 85% of Gaza population are refugees from stolen cities in Israel. In the genocidal barbaric state that is Israel, and this is why we're here, when sending a message loud and clear to Israel, to the Israel's enablers, the apartheid enablers, the likes of this government, this shameful government, our shamefulizations like Zim shipping, we're sending this message loud and clear. We don't want you as human. We don't want your message today. See, stop the genocide, the bloodshed in Palestine, and we will never back down. It's dismantled until Israel is no longer existing, until liberation is achieved. For the genocidal state like Israel, we need to become militarized with Israel. We need to become economic ties with Israel. We need to make apartheid Israel. The human, as Australians, as human beings, are seeing children's torn to pieces. Torn to pieces and our politicians are paying lip service. They're the Zionists. Families, members under the rubble, shame on you! We will continue to demonstrate to inject some decency in the police force system, in the political system, in the economic system of Australia, until you take responsibility as a human being, community, and we're not backing down. We cannot back down. We cannot stop action. We cannot stop working for those people, for innocent people. Shame on the Australian system.  No to barbarism. No to genocide. And we're here to say... I'm sick and tired of leaving messages, receiving the news of losing one by one my family members in Gaza to this genocide, to this barbarism. We're here. This is the only way we can... This is the only way to message the Zim Chipping Line. They are not welcome here to the Israelis. The Australian people have spoken. The Australian people have spoken for seven months. They have been flooding the streets of Sydney, Melbourne, and Brisbane, and everywhere. Establishment to listen to us. And this is the best we can do. We need Zim Chipping Line. Zim is free of this racism. This aggression.


Databases, a database seminar series at Carnegie Mellon University, is recorded in front of a live studio audience. Funding for this program is made possible by Otitune. Google. Hi guys, welcome to the last talk of the semester. We're starting to have our friend D.V. comes up about their database system. So, D.V. is a tribute database system that comes out of Europe. And so, we're always excited to hear about what people are building. So, today's speakers is Maria and Marios. Maria did her PhD at TU Darmstadt in computer science. And Marios is a senior software engineer at CRE. So, as always, if you have any questions for our speakers as they're giving the talk, please unmute yourself and ask your question at any time. Would this be a conversation with them and not with them talking to themselves? And with that, we thank you guys for being here. I will say, so, Marios, it is 11.30 p.m. in Greece where you are. Thank you for staying up. And Maria, stand 30 where you are in Germany. Again, it's not the record. Our record is 4 a.m. But it's still pretty late. We're still pretty late, so we appreciate you guys staying up with us. Thank you very much. Thank you for having us. Yes, thank you for having us. And very nice introduction. So, yeah, today we are going to talk about Crazy B, which is a secreted SQL database built on top of Lucene. And as Andy already mentioned, so let's introduce us, the speakers. So, my name is Maria Selakowicz, and yeah, I hold PhD in computer science from TU Darmstadt in Germany, and I have been living in Germany for the last eight years. So, quite some time. But originally, I'm not from Germany, so I come from Serbia. And, yeah, I did my undergraduate studies in Belgrade. And today, I work as a developer advocate at Creta Yo. We are also going to meet Marios Trivizes, who is coming from Greece. And he did his master's degree at Athens University, right? Currently, he's working as a senior software engineer at Creta Yo, and he's also one of the first engineers at Creta Yo, so he has a lot of experience with building Crazy B, because he has been there since very early years. And agenda for today, it's going to be relatively easy to follow. So, can we go to the next slide, please? So, first, we are going to introduce Crazy B. Then we are going to talk about some main architectural points, then we are going to go a little bit deeper into the storage layer, which is, as we already discussed, completely based on the scene. Then what is the core and probably the most interesting part of Crazy B, this query execution that we are also going to mention logical replication and how we implemented this. And finally, we are going to say a few words about deployments and the ecosystem, because it's quite important when having a database. It's very crucial to have a well-established ecosystem that works with your database. So, what is a Crazy B? Crazy B is one of the leading database solutions for real-time analytics, it's distributed, it's open source, and actually it supports SQL as a query language, which is quite important to learn during the time. It's quite important to actually support a language that developers are familiar with and where the learning curve is not that steep. But let's go into more details. We like to mention very often that Crazy B combines the best of SQL and most SQL, so it also supports Postgres Fire protocol for easy onboarding and compatibility with many tools that actually work with Postgres. When we say it uses the best of no SQL, we usually mean that Crazy B also supports dynamic objects and storing objects in JSON format and also supports dynamic schemas. But you can use Crazy B to store and analyze a wide variety of data, including relational data, object data, time series data, guest portal data. We already talked about blobs and of course full text data, so sometimes you need two systems. For example, if you want to combine different data variations, but with Crazy B, you can use one database. Scalability is something that Crazy B has been known for, it supports the concept of horizontal scalability, so you can scale your cluster by just adding new nodes and it's of course performance, which is always a moving target for us. Since we are based on Loose Seen, we actually not aim, but also claim that the queries actually can run pretty fast, and we also support aggregations, very fast aggregations by using columnar storage, which we will talk a little bit later on. And Crazy B can also be seen as a distributed search engine since it's built on Apache Loose Seen. Currently, there are two deployments options, so there is open source version of Crazy B, and there is also cloud version that is available for more or less major hyperscalers.  Yeah, it's actually so Crazy B is following more or less the PostgreSQL dialect, there are still some features that are not fully implemented, but we are working on the PostgreSQL ecosystem in general and full compatibility. So it's the PostgreSQL grammar? No, sorry, not only, but also the protocol of the wire connection. That's why we allow for PostgreSQL tools to connect to Crazy B, even the PSDL client can connect to Crazy B. And in general, Crazy B is how Crazy B stores data that's usually done in relational format. So when you create the table, it looks more like relational database, but it also allows for storing objects as adjacent documents, like we can see on this example. And the good thing is that objects can have very different schemas with arbitrary number of attributes and testing levels. And what we also store on top of this is original documents in JSON form, and so for each row that we store in Crazy B, we store the JSON representation of this row. And talking about schemas, there are three types, three ways actually to declare objects in Crazy B, depending on what type of schema you want to follow. The highest flexibility is given with dynamic schemas, where you can add a new column, let's say dynamically, which means if you want to store a new object with a different number of columns, then it has been defined, you can do this. And also new columns as well are going to be indexed. If you want the object to follow the strict schema, you can do this by declaring the object as strict. So in this case, every added object needs to follow the column definition. And finally, there is another way to declare object, which is ignored. In these cases, you can add new columns, but these columns are ignored from the indexing point of view, so they are not indexed. And another nice feature of Crazy B is that you can do the database administration by using pure SQL. So there are some tables like PG tables that are available, but also information schema tables, where you can find data about existing tables, partitions, constraints and columns. And not only that, but also system tables that you can query for, for example, finding out what are the running jobs, what are the finished jobs, information about your cluster and shards. And there is a simple way actually to stop a single or all jobs on a node. So this is a cool feature in Crazy B because you don't need any new knowledge actually to database information, so you can just use SQL as a query language as well. And finally, I think it's important also to mention when it's a good, what we consider a good use case for Crazy B and what we don't consider a good use case for Crazy B. So in our experience, we've seen that Crazy B is a very good database for time series data, especially IT data or industrial time series data. Then in cases when you need fast queries, aggregations, full text search, SQL as a query language, chaos, volatile data, we also support logical replication compared to many other databases. We have hybrid deployments that can actually be implemented on Kubernetes cluster that, for example, runs in some specific regions. And Crazy B also comes with the promise of high availability and scalability, but it's not a good database if you need transactions. So transactions are not supported if you have highly hierarchical normalized data models. So Crazy B supports joins, which is also a good feature, but joins in this with the settings is a very expensive to execute. And we are going to see how we actually tackle this issue. And in case you need foreign keys, Crazy B is not a good database. So let's go to the next topic, which is Crazy B architecture and the main overview. And I'm giving the word to my colleague, Marius. So hi from me as well. Thanks a lot for joining this session. And let's talk a bit about general Crazy B architecture. So as we have stated already, Crazy B is very easy to scale with it because it has a certain architecture. That means that every node of the cluster is equal to each other and can perform exactly the same tasks. So there is no single point of failure. And if any node fails, then some other can take over its jobs and the cluster can continue working in a healthy state. Apart from that, you can scale as your data grows or your requirement grows regarding performance or number of clients querying or inserting to the database by just adding more nodes to add more space, storage, more memory and more CPU power. And the moment that you add a new node to the cluster, the existing tables that you have defined, they are automatically rebalanced in the cluster using the new nodes added. So you don't need to do manually anything to rearrange your data across the cluster when you double the number of nodes, for example, or even if you add just a copy more of them. Just to say a bit more, I won't go into details of that, but there are much more complex configurations you can do. So there are nodes that have the master coordinator role in the cluster. And maybe you have all the nodes be able to take that role, but also you can configure that only a subset of the nodes that maybe don't even have much storage space and they just do the coordination, take the role of the master and then you have data nodes. You can also have only client-only nodes with responsibilities basically like a load balancer to service the clients in front of the actual cluster performing the operations. Great question from the audience, Kevin. Go ahead. Yeah. So my question is, within a single node, is the architecture also shared nothing or does every physical CPU core and OS thread get its own independent buffer pool and memory or is there a shared memory architecture where a single node all of the memory is shared across them? Every node has its own memory, but this memory is shared across all operations that the node does regarding clustering or inserting data or executing query. There are, of course, some thread pools there, so its thread pool has some responsibility. For example, like some internal scheduled jobs or the inserts with the search is where queries are different search pools, but that's it. Got it. So it's one big like buffer pool and one big application level scheduler per node and that's shared amongst all the threads and CPU cores that run on that node. Yeah. Okay. Understood. Thank you. Yeah. Again, there you can also, you can just influence the number of threads in the thread pools, but yeah, you can influence some things and configure it, but in general, it's a shared thing, as we said. Yeah. So shared nothing architecture was sort of like the way to build things 10 years ago, but certainly in the cloud now, everyone is moving to a shared disk. And I think this would make sense in analytics, right? Because it's, you're trying to store a lot of data, although you obviously want to keep things, you know, with the senior indexed and, you know, you would keep that local.  Yeah, like to have a good performance, you have to have local disks and actually SSD disks or now even NVMe disks could get the best out of it. So you can use cloud storage like S3 for snapshots. And we have this idea to be able also to search on those snapshots on S3 to have slow searches, of course, on let's say archived historical data to give this opportunity, but definitely you don't get the performance you get with the Lucene on local disk. Maybe Maria said this and I missed this. Like, you're trying to do real-time analytics. So what is your notion of real-time?  So, can you repeat again, please? So you're doing, you're trying to target real-time analytics. So it's not, you're not trying to replace Redshift, you're not trying to replace Snowflake where they can just do big sequential scans over large chunks of data. You're going to be using Lucene as the index. So therefore, you're trying to target queries that are fast. I don't understand, like, when you guys say real-time, is that like in the hundreds of milliseconds range or something less? Like, you're trying to understand, like, what's the target SLA you guys are trying to provide? And obviously depend on the query and the data, but I'm just curious. Yeah, as you said, it depends on the situation. How, first of all, what is the throughput of inserts and updates you want to support together with real-time? Because, of course, when you increase the throughput that you want to get, then you don't see your data immediately. There is this concept in Lucene, the refresh rate, the refresh time. So the data that you insert becomes available after, let's say, one second or 10 seconds or 100 milliseconds. So it's something that you configure, but the less you configure it, the more you decrease performance of inserts and updates, of course. Okay, awesome, thank you. So, yeah, as an overview from the stack of the architecture, what you will see in each of the Gradient B nodes and let's start from the bottom. You have your clients that connect to the cluster through either the HDP endpoint or the Postgres HGL wire protocol. And each statement, either it's the DML, DDL, or administration statement, goes through the parser. The parser, of course, parses the HGL and validates the syntaxer and creates an abstract syntax tree, which is then passed to the analyzer, which knows a bit more about the tables that are available, the columns, the privileges, and so on. And that's semantic processing, sex data types, and all these things that you can do before runtime, before the actual execution, I mean. And then it passes you to the planner and optimizer, which has several stages.  And then you actually send it to the distributed execution engine of this node that you have connected, which will, in turn, use its local Lucene storage to store locally things and also transport data and requests to other nodes in the cluster to fulfill your statement. So regarding, yeah. Sorry, so going back, the, is the query app master, is that written from scratch, or are you guys relying on calcite? No, it's written from scraps. Got it, okay. Do you know what it's like?  We do some basic things at the moment. So we do some analysis on the table to get statistics regarding number of rows and like average length of every row in bytes to decide for some things. Yeah, but it's not sophisticated. It has some decisions and some optimizations, especially in the subquery area and joins. But we will see also a bit later about filter pushdown and limits. And so maybe you can hold a bit the questions and ask me. Hi, thanks. Thanks. So when it comes to node-to-node communication, then there is one more protocol, a network protocol for kdb, call it transfer protocol. And it's dedicated to communicate, request data between the nodes. And you can also enable encryption for each one of those protocols. So only for Postgres, only for HTTP, only for transfer, for two of them or for all of them. And the configuration is done via host-based authentication. You can use trust and certificate authentication methods. And of course, when you enable one of those protocols or two or three, as I said, this applies to the whole cluster. So you cannot have two nodes working with unencrypted HTTP and all of the others with encrypted or transport between specific nodes to be encrypted with other nodes. So it applies to the whole cluster. So next, we will talk just a bit about the storage layer, which is Lucene. So when you define a table and you start inserting data into it, then all of your data gets indexed by default. You have the possibility to disable indexing for certain fields if you're never going to search by them to gain some speed for insert updates and also some space. So for text values, Lucene uses an inverted index. We'll see a few more details next. And for all numerical values, including all types like timestamps, geopoints, geosapes, internally they use numerical values using block KD3s for the indices. On top of the indexing every field for all the fields, you also have the column-based data storage of Lucene, which is called dock values. And this is what provides very fast sorting and aggregations. And currently, you can disable these dock values, column-astorads only for text fields, but not for numerical. And we're planning to do it soon. It's no radar to save even more space for fields that you don't need. So let's talk a bit more about the text index. And first, we'll talk about the analysis of the full text index, which comes first. So when you have a large document before you start indexing it, you need to split it into tokens. And then you have to maybe apply some filters on those tokens and maybe also on top some character filters. So all of these components comprise, I'm missing the word, analyzer. And there is some standard analyzers that you can use, but also you can define your own tokenizer as well and combine the filters that you want to do your job. So I will show just an example of the standard analyzer for English. So first, you just split all your the tokens with the white spaces when you get the words. Then you just lowercase everything. And the other thing is that you remove the stop words like the end and words like that that are not useful when you're searching for. So next, we will see how now we index this data. So before we see the tokenized version, let's see what happens when you have defined a normal text index without analysis step before. And in our table, you can see this model field and you have almond milk, almond flour and milk. And this text, the plain index, just get indexed as a total phrase as the whole string. And you just point to the document IDs that this phrase can be found in. But when you are using the full text and you have done the tokenizing analysis featuring and everything, you can see that you have the your token split and you see that the token almond, you can be found document IDs one and two and milk in one and three and flour only in document two. So in this way, when you start searching for terms like that, you can very fast refer to the documents that those values can be found. And on top of course, with Lucine, you can do much more complicated things with phrase searching and fuzzy logic and wild carbs, prefixes, suffixes, and so on. And scoring. So you can use, you can define how the documents that you have mapped are returned in an ordering regarding which are the more relevant to your query. And for the doc values, as we said, is sort of the columnar sorabs of Lucine and its numeric value gets its doc values index. So basically what happens is that you have, you can very easily and quickly access all the values of a certain field like the quantity here, and you have them related to document IDs. And what makes it fast is that you can have a query that you filter out things, and you decide that this is a subset of document IDs that maps my query. And then directly you can get a very fast iterator over the values of the quantity field over the IDs that you have preselected, pre-fitted. And also you can do very fast aggregations of numeric ones like average sums, min and max with that. So for all the things like the sorting, the query execution, is that being done by Lucine? Or like is that done by Crate? Like is this Crate, the Crate's going to do a query, say it's going to do a sequential scan or something, right? Like, hold on. And the data is in Lucine.  Yeah, it's both actually. So there are many things that can be done by Lucine, and we will see next that we will try to push down as many things in the Lucine to get advantage of the index to avoid doing processing in the CrateDB nodes. But of course, when you have custom logic and operators on the fields and you don't have simple filterings, then of course you need to do all this work in CrateDB. So regarding compression, because of course you cannot not have all these indices for each field, the dog values on top, with plain storage, otherwise you will need tons of space in the disk. CrateDB uses two compression algorithms. The default is LZ4, which is more or less a good trade-off between the storage efficiency and the performance, both query but also inserts, and also deflate, which is much more aggressive compression, but it comes at a cost both for inserts and queries. And on top also the dog values are, there are several techniques that are used there regarding the numeric type, the numeric ranges and so on to compress them. So there is delta encoded if you have many values that are very close to each other. It's also there can be bitpacking or this is the common divisible compression. And we have made some tests in the past to compare CrateDB with Postgres regarding the storage requirements. So we had this table with a variety of different fields with 120 millions of rows in it and there were only two indices for on time and device ID enabled also for Postgres of course to compare it. And here is the comparison with the two algorithms of compression for CrateDB and for Postgres SQL. And next we will talk a bit about the query execution engine and about plans. So as we said before after the analyzer we go to the planner and the first step of the planner is to create a logical plan and we'll take the very simple query here like the select name from users and for every query and statement basically you can add the explain prefix and you can get the logical plan for it. So we just select here a field name from the table users and this translates to a logical plan which is an operation collect. So this is to go to Lucene to the users indices and retrieve the name and the truth here is the filtering because there is no filtering so basically it's returned everything. So if we just add a plain filter in the workloads where name equals a literal string name then you just see it being added here and this is the query expression that it's pushed down to Lucene to the indices and translates to Lucene term query in order to return the documents that you want. So as we said before with your question we try to filter to push down all the filters that we can down to the Lucene level to take advantage of the indices and a simple example here when you have just a subquery a nested query and you have a filtering on the edge in the inner one and a filtering of the name on the outer one then the logical planner can combine it with an end and create a Lucene term query again with an end and get advantage of the index. So after the logical plan and these optimizations that happen then you have to create the physical execution plan that it's actually going to be executed in the execution engine on the nodes of the cluster. So the logical plan we have seen before even with the optimization doesn't know anything about the data distribution at this point. So it doesn't know where your SARPs are located because normally every table you will have at least three nodes let's say in your cluster for redundancy and to sustain a failure. You will cluster it with minimum three SARPs and get one SARP to the node. So the planner that needs to create the execution plan now needs to go into the routing table of the cluster which holds all the information about where the SARPs are located to which nodes and create the execution plan and asks for the nodes to retrieve data. Of course keep in mind that apart from the primary SARPs in KDB you can also configure one or more replica SARPs. So if you have if you define for example three replicas then you have three times three you have nine SARPs in total and then you also have some round robin algorithm or your queries to not access the same SARPs for every client that runs a query on the same table. So you get a round robin behavior and also you get some statistics about the current pressure of the cluster regarding CPU and two threads to decide where to forward your requests. For every execution plan the nodes that participate in the execution can be divided are divided into three roles. So there are the collect nodes, there are the nodes that hold the leucine indices, the SARPs where the data is gathered with filtering and ordering that are pushed down. You have the merge nodes so these are the nodes that retrieve results from the collect nodes and merge them but they can be one of several merge nodes depending on the operation and the number of nodes and SARPs and there is only 100 nodes. It's the one that communicates with the client, the one that the client has originally submitted the query, the statement and it's responsible to gather all the data from the merge nodes and return the bug. For simple cases the merge node can be also the handler node so you only have one merge node you have maybe several collect nodes to retrieve data but merge 100 nodes are all the same and so on that the client connects to. Let's now see a bit more regarding the execution, the physical plan. So we saw the logical plan for this simple query and if we if you do an explain analyze as a prefix then you get the actual execution plan where you can also get timings here for each operation because it actually executes the query for you and you get timings and you can see also which part of the query is the one, the bottleneck. So for this simple one we see that it's routed to two nodes, node 0 and node 1, the node 0 holds SART 0 and 1 of the table users, node 1 holds the 2 and 3 SART of the table and for both of them you request to fetch the value for the name field. So the merge phase is done for example node 0 can be also the handler node where the client has connected. This has two upstreams which in this simple case one of the upstreams can be also the node itself, it's the same node because it holds SARTs of the table we have queried. So when it receives data from the other node and from itself from the operator executing then it merges them and sends them back to the client. Now let's see a bit just a bit more complex query when you have an order buy and a limit. So notice here that already the logical plan becomes a bit more complex and we start from the bottom which is again the first operation that it's executed in Lucene for the users table. Now you request this special internal field called fetch ID together with the ID you send down the the filter that you want on the name and you ask to order buy ID which is also will be done by Lucene. This is the order buy that we have and then you apply the limit and then you apply the final fetch. So let's talk a bit about this fetch thing. Since yeah you can have tables with many fields or even a few fields but very large fields like texts or geosapes and things like that it's important that you avoid sending around data that it's not needed. So when you are executing a query like that instead of returning back the fields name and names that you have requested you ask for Lucene to generate this internal fetch ID for its of those documents and this fetch ID it's somehow combination that encodes the SART as well together with an internal document ID and I don't mean sorry I don't mean this ID which is a field we have defined but an internal document ID. So when you have applied the limit because you don't want to return like 10,000 or 10 million rows but only 50 then you have the fetch IDs that you want and then is where you go back to the Lucene to the nodes and you execute the fetch operation to retrieve the actual data for these fetch IDs and just to say a bit more about this limit you can you see that when you have a limit of 50 then since you have many SART you push this limit to all of the nodes so you get 50 nodes for each of them they are ordered and in the merge node you just keep the top 50 to return back to the client and since we are with the questions yeah we're a bit over time I will go a bit quicker through the joins so crazy bit supports joins the default algorithm is nested loop I don't need to explain more about nested loop it's very clear to why the default join algorithm is nested loop yes because we haven't invested a lot in the joins we will see next about the hash join so the default for to support all cases where you have complex join conditions with you know greater less not equal joins you have to go to nested loop we haven't implemented any sorted merger this kind of algorithms up to now so even for the nested loop we have a distributed logic there so we have several nodes executing the nested loop and for each node that that does a portion you have to broadcast SARTs from other nodes in order to do the portion of the nested loop so of course if you have the SART one of the left table to this node you have to to receive all the SARTs of the right table in order to to go through them and execute the nested loop and at the end you have the merging of those results and for equally join some quite some time ago we have implemented block hash join algorithm so you have the first the build phase where you try from the sorry from the stats as we said earlier that you have you scan the smaller table in terms of memory that it will use and you create houses and you store them in the hash table and then you probe all the rows of the of the other table using this this hash table and of course you are using blocks because maybe if your table can be small but not small enough to fit in the available memory so you might need to have multiple blocks and build multiple hash tables to be able to fit finally the table to the memory and also the block hash join is can be executed in the distributed passion so what happens is that beforehand you do a hashing and a modular on the rows and you distribute them to the nodes that you have decided to execute the block hash join and every node receives the rows that potentially will match the it it cannot miss any rows with this initial hashing and then its node performs the portion of the block hash join and then you have the merge phase where you merge the results and for all these operations kdb supports two methods of communicating data between the nodes so you have the direct response and the paging so automatically it selects what seems to be ideal for the situation regarding the nodes the subs are participating the number of expecting data if you have limits and so on and you can see it basically in this graph that when you have a direct response in this communication let's say that node zero receives the initial query from the user and needs to send the request to node one to get data or execute something so the for the direct response you just get the result as a response to your job request from node one but when you have larger amounts of data you cannot transfer that this way so instead you use paging the steps in the communications become a bit more complicated because in this situation node zero will first send the job request to the node one the node one with just acknowledge with node data and when it has some data available in a sync function it will send the first page to node zero node zero will process and it will ask for the next page from node one which will send it and so on and all these communications are asynchronously so you have listeners and futures and you you will get at some point the response from node on with data and spart continue processing so if you have some questions maybe now for the situation anything before we proceed with maria to the next chapters or we can also do it in the end yeah there is not much left actually so we wanted to mention logical replication since we were talking about building replications already so what do you prefer doing some short q and a nice keep keep going with your q and a pn okay awesome so logical replication so we we were talking about building replication the short level but also crdb allows you to replicate tables or subset of tables from one cluster to another it follows published subscribe model where publishers are clusters that make data available through publications and subscribers are clusters that actually receive data and not only data replicated but also the operation operations such as inserting the leading updating and changing schemas are also replicated to the subscriber cluster so syntax is relatively simple also follows a sequel so can you go to the next slide so you'll create publications by specifying whether you want to create to make this data available for all tables that are on the cluster only for subset of tables and during this process you are you don't actually you're you cannot do aggregations on on the other side you can create subscription that actually make a connection to a publishing cluster and you can also specify currently one parameter called enabled which actually says whether you would like your subscriber to be actively replicating so and currently the data on subscriber cluster are read only so it goes only one way and we have seen a couple of interesting use cases for uh this uh this feature which has been available since i believe may or june this year um the one is actually for improving one use case aims to improve data locality so imagine if you have one central reporting cluster and you want to make data closer to the to the applications that are actually uh using this data so you can for example replicate the data to the regions to the clusters residing in regions of the closer closer to application data and in this case actually you can achieve we can achieve much lower latency because the data you need for your applications are are actually closer to you another interesting use case is something related to the to the analytics and can we just mention this quickly on the next slide so in this in this use case we are actually allowing um replication of data from different distant regions to one central reporting cluster very interesting and common use case because you sometimes would like to aggregate all the data that you might need for your analytics in one central cluster that is closer to this analytics instead of querying the data across different regions and you don't need to do this querying across multiple sources you do this from a single source and also you can also you also have a control which data you would like to replicate the central reporting cluster so you need don't need to to query actually all of them um before actually finishing for today we would like to give a few words about deployment uh that we that we have currently an ecosystem in general so I already mentioned very beginning uh that we have two deployment options first one open source kdb is available under patch license uh 2.0 and as it's open source you can just try to turn on the hardware you would like but we also have a cloud version um that is available on more or less all major hyperscalers and uh we also have um recent we also recently um licensed on the under patch license uh the crate operator which is actually a piece of software that you can use to deploy kdb on Kubernetes cluster residing in the region of your choice um and yeah if you're some of those people who like to contribute to open source software and I just would like to invite you to check out these github repositories um so we mentioned the ecosystem uh kdb definitely supports postgres file protocol there are there are definitely some limitations uh that I briefly mentioned but when it comes to the features that we don't still do not support which doesn't mean you want um so currently what is available in postgres does not in kdb is time without the time zone so this is something that we still don't support uh we also do not have option to um for declaration and usage of multi-dimensional arrays and another feature that is also quite often used in in postgres but still not uh supporting kdb is uh interval input units for example milliseconds or microseconds and on this slide you can also see the uh how we implement postgres interface and there are a couple of other limitations that we are still working on so like it's postgres compatible but not 100 percent and let's go to the next slide to say what is next for us so kdb has many interesting features but of course there is a lot of things to do first of all first of all um we believe that ecosystem needs to to improve and we are working also hard to facilitate kdb integrations with other systems and tools and only postgres compatible tools but other tools as well um we also have we didn't we didn't mention in this in this presentation because there was no time um we kdb has a functionality to read data from external file system or local file system and to write as well now this is done through copy from and two statements and we are currently working on improving user experience and functionality of these statements and i think what mario's already mentioned there is a lot of optimizations that are on our checklist definitely we would like to improve insert performance so the query queries are quite fast insert is um relatively doing well except in some cases and there is also this notion of a scene plush that takes some time until data are available on the disk and when it comes to this query execution especially joins further optimizations are and uh with this i would like to thank you all for your time invitation and attention if you have any other questions um not only now but also after after the the the q and a just be free to to send us emails or to join our community and ask questions there so maybe now we can go to thank you i'll clap it back for everyone else all right perfect awesome uh we actually have time for for a few questions so if anybody in the audience wants to go for it go ahead and meet yourself in far away otherwise i'll be selfish and i'll use all the time for questions okay cool um so i guess the my first question would be again and this would be me not having any experience with scene but how does how do you enforce integrity constraints is that all done at the great level or will the scene do that for you like you know for example you show the the object type and yet there's no strictness there is that being done by the scene or you have to do it you actually look at the data as it comes in before you show it to the scene yeah you have to do it uh yeah upgrade the b level you can also um you you can also define constraints on every field like not null or not only these complex calculations expression with operators like model of this plus this equals greater than whatever so all all these yeah are done before you uh send the data to be indexed in the scene and then the scene has a notion they have their own sort of type system right so you have to make sure that create these matches the primitives and the more complex things you guys handle up there sorry can you repeat that like so there's a type system right the scene probably has their own internal system but you guys yeah you have to translate it yeah you need to translate it to what postgres supports regarding big decimals or things like that yeah you need to do the translation from the sql data types to yeah and also in some cases in between you have the java thing because pretty busy in java so sometimes you don't even have exact compatibility the special case is the interval for example we support intervals not storing them in the scene but doing calculations and having an interval type as a result like subtracting time sums so yeah in sql interval is very specific and does you don't have this thing in either java time or joda time and you need somehow to make the translation in java for this thing got it okay um what is something you wish you what is one say basically you guys are you're not a hard fork of the scene right you're you're you're following along their main line so you get all the optimizations and new things that they support is that or you guys do a hard fork i didn't get sorry sorry for the scene did you do a hard fork and their repository of the source code or are you are you embedding them and always using whatever the latest version that they put out uh yeah no we are we're using the the latest version of that okay yeah yeah we use it as a library as a dependency but yeah of course we have some some classes that we override when it comes to this data type mapping thing that you mentioned but yeah mainly we just update and you have to adjust in most cases just small parts of the code to adapt to the new version and the new features got so it's quite backwards compatible then you need to see if the new features somehow you can use them now and push down more features the queries or yeah so what's up is there anything you if you can change one thing of the scene to make creativity better is can you can is there something you wish it had it doesn't have now there is a there is a thing that it's very tricky yes and it's the null concept so you seem doesn't have the null concept and whenever you do queries with is null it's not null then this is featuring you need to execute with create db code so you go to leucine and then leucine has a callback to the create db code to execute the featuring which is of course much more inefficient than doing a featuring on leucine itself and then the null thing that it's very very common in the relational database goal it it's not cheap because of these restrictions not restriction but the way leucine didn't support it from the beginning as a concept and this plush actually is leucine plush that makes it available on disk sometimes is maybe taking quite some time um because we have seen a bit a couple of customers actually complaining about insert performance it's not for all data usually not for all types of data but like there's sometimes use cases where this doesn't work as expected and take some time until data are available on the disk yeah this is maybe not something that scene doesn't have or has but it's something that maybe could be improved in the future got it okay and as my last question is is you guys are the only database system we've had this semester that's written in java actually Victor you have a quick question uh just what's the best practice to it's for database design through the best to loop uh joining what's the best method to improve nested loop join yeah i put in the chat yeah so because it's like the best nested loop manually for a large amount of data that's contained long time so what is the key design principle to improve that and how to create your database no no no i he's asking how how do you make a nested loop join run faster the answer is not run nested loop join like yes worth one right there's nothing you can do and especially not run joints on multiple tables that could be very slow yeah i i mean victor the answer is hash join is almost almost always superior especially if you have to have good at desk um i watch the class lecture we cover this okay um so my last question my question is again you guys are the only uh your database system that's written in java we had this semester um it's been all rust c++ one somebody in zig um and so you guys are are almost 10 years old now and is there anything that you know in over the years last decade of the iterations of java is there anything that has that that's improved in the jvm or the jdk or you know the java ecosystem that has made things better for create db and that can be either in terms of performance or software engineering um for jvm definitely the the garbabs collection that g1 cc when it became the standard stable made an impact for sure okay yeah and now with java 19 there is this very proven with mapped ffs or so which we haven't tested so far but lucine is recommended to start using it at least for to test it so we are planning to have benchmark with this thing we haven't done it yet because it was like two weeks ago that we we moved master to java 19 so we didn't have the chance to test it so far okay and yeah code guys there are some improvements but in general they are not those that make the the big impact and there are many things like the streaming interface the streams that in many times it's much slower when you do the uh the micro benchmarking that instead of writing you know verbo is in the four loops and doing some featuring there


The question then remains like, what did convince you that a God exists? Like, are people better off believing until they're proven wrong? Or should they start believing when they have enough evidence to believe that's the case? That is a very good point. I would say, I think it depends on what you're talking about specifically. Let's talk about my God specifically. Okay, I think it's best to believe first and then you can always, I don't know, that's a good question.


Hello there Bruce here in the shop tonight. We're going to talk a little bit about sound and getting the very best sound you Possibly can out of your locomotives in a little thing. I called how to get the sound out Sounds like a good idea. Okay. Let's get started first. We got to talk about the physics of sound So you don't understand the physics you're gonna have a little bit of a hard time making it work for you Okay, let's start with the concept that sound is pressure waves moving through the air Hit a base drone You can see the diaphragm of the drone move that creates a pressure wave Since we're dealing with electronics and electronically generated signals here We have an item called a speaker that will translate those electronic Impulses into sound pressure waves. How do they do that? Okay? The speaker has a cone and a voice coil that move back and forth based on the electronic Signal that it's receiving when it moves forward it creates a positive pressure here and a negative pressure on this side When it comes back this way this becomes a positive pressure. This becomes a negative pressure So it moves back and forth that creates pressure waves They move off of both sides of the speaker people ask me what does it matter which way you mount speakers? With the magnet up or with the cone up or out or however you want to look at it No, it doesn't not in our hobby if you get into really fine high-end audio There are some people who say yes that the frame that holds the magnet will Distort will reflect the sounds a little bit and may change the image of this way beyond What we're talking about here in our locomotives. Okay? So it pushes the sound It pushes these pressure waves out Well, what happens is as the cone moves this way there's a negative pressure here. There's a positive pressure here nature seeks to Even out Everything in its world So this positive pressure rather than just automatically running away is first going to try and sneak around the speaker And fill in the negative pressure that's on the backside of the speaker What does that do that cancels the sound when it goes this way? It pumps air around here when it goes this way it pumps air around here So it might as well not be moving. It's not making any net effect of sound on the world around it. Fortunately or unfortunately these variations of this cancellation is not complete So therefore it doesn't completely wipe out the sound, but it does significantly diminish it Okay, I've got here a 28 by 40 millimeter speaker. It's a speaker. We're going to use throughout this whole A little session today This is the largest speaker that will fit well Into an HO scale locomotive. It'll fit in the floor or the co-load of a lot of steam or tenders it will fit in the floor of some Cab diesels So a units b units, especially a lot of dummy b units. I use even two of these in a dummy b unit make great sounding models It's about a little over an inch long by about a wide by about an inch and a half long So it's it's a fairly large speaker And so this is going to be kind of our our hallmark of what will really sound good. I'm going to fire this thing up with the sound from a Soundtracks tsunami alco 244 and let you hear it. Okay, there you have the speaker Generating the sound. It's about a third of the way through The throttle range, so it's it's moving along down the road at a pretty good clip And you can hear the sound But what's happening here in free air just to review The sound from this side of the speaker will wrap around and cancel the sound from this side of the speaker and vice versa How do we know that's happening? Well, let's see what happens if we cut down the amount of that sound that's allowed to wrap around Look at that. That isn't even complete. So that leads us to my next point, which is How do we make that happen in our models? if we had a speaker here And we built a big Plane that went up forever and ever and ever and we had a speaker right in the middle of it And you were on that side of the speaker and it was moving air You would hear All of the sound because there'd be absolutely no way that any sound could wrap around the speaker and cancel out No, you wouldn't be hearing all the sound you'd be hearing half the sound because Part of the time the speaker is coming this way and making the same amount of sound on this side of this big plane Okay, it would sound good, but you'd be losing efficiency. You'd be giving away Some of the sound that you've actually paid your money to hear So how can we get the sound off of both sides of the speaker? And still Have good sound. Well, it's my contention that if the sound has to travel About five or six inches Given the wavelengths that we're using For the speaker size that will fit in our models that the Basic cancellation will be avoided and you'll get the best possible sound you can out of your models Okay to show that demonstrate that fact. I have a piece of white plastic here If I mount a speaker in the middle of this piece of plastic Then it's about six inches by about four inches Okay, if the speaker is mounted here Then the sound from this side of the speaker has to travel About two inches over here around and about two inches back about four four and a half inches Uh before it can actually cancel the sound from this from itself Going around this way. It's closer to six inches. Nothing magical about my numbers This is where my number came from. I looked at this and I said this works So four to six inches is the minimum number It works. Okay, let's let's show. Let's take a look. Okay back to our speaker With the tsunami generating alco sounds in it And sitting here in free air Sounds like it did before now. Watch what happens when I put this white piece of plastic In front of the speaker now. It's not sealed or glued down or anything else I'm just going to hold it in place and you can hear what happens Notice how the sound gets louder as well as there is more Bass there's just more definition of the individual sounds you can hear the individual Almost almost hear the individual explosions from the prime movers that locomotive was running along Listen again. Isn't that neat? Okay, let's go back and look at this piece of plastic that I was talking about Six inches long four inches wide About an inch out of the middle of it for the speaker If I were to fold this side down And I were to fold this side down So there was just the speaker left here You'd have about two inches about an inch and about two inches. Yeah, I know that doesn't add up to four But it's close. Okay Gee doesn't that sound like the shape of the shell Of an ho locomotive a couple inches up an inch across a couple inches down And about six inches long So if I were to mount the speaker in say the fans of a locomotive You'd get the same effect the sound would have to go down Through the mechanism inside which will slow it down Come around say through the trucks Back up the side of the locomotive before it can cancel the sound coming out the top. That's the basis. I'm talking about Now there are folks who say that you absolutely must have an enclosure in order to get good sound out of your speakers I seriously disagree I don't disagree that you have to limit the Bleed around the speaker the cancellation But there's more than one way to do that Yes, an enclosure will do it and yes, there are times where that's the best way to do it Given the locomotive that you're trying to do the installation into but it isn't the only answer And so a good craftsman has many tools in his toolbox. Let's look at some different tools here Now we're going to take the same speaker and use it With an enclosure and compare it to a speaker similar to the one in the white card I made up this little test board many years ago and I've used it for a lot of different things But what's important about it is here on the end There's a 28 by 40 millimeter speaker That is in an enclosure Commercially available enclosure that about doubles the price of the speaker it costs about the same as the speaker So the speaker plus enclosure costs about twice as much money and it takes up a lot of room But it gives you some fairly good sound And over here we have the same speaker Same style of speaker at least mounted just in the middle of the board. Okay It just sticks through the board. That's all so you have the distance around the board To prevent the cancellation. So let's compare the two And first we're going to look at the box The speaker in the box Okay, that sounds pretty good. It's got a nice volume. It's got fairly good sound to it and just By itself with no subjective comparison. It sounds really good So now let's listen to the speaker outside the box the one over here in the middle of the panel I don't know if you can hear a difference between those two Unless they're right up against each other. So let's start off I'll turn on the speaker in the box and then I'll switch to the one outside of the box and give you a side-by-side comparison Here's a hint. I don't think you're going to see a difference Particularly in the level of the sound But in the subtle nuances of the sound And the timber of the sound and the depth of the sound the fact that it doesn't sound like it's flat and just Right there next to you. I think you'll hear a difference Hopefully on this video, you'll be able to hear the difference between the two So we'll start off with the box and then we'll switch to the speaker out of the box Okay, I throw out a locomotive back down to an idle and Turned on the horn. So listen to the horn And the prime mover idling in the background With the box and out of the box and see the difference on the higher end of the register I don't know about you, but most people that I've made that demonstration to have said wow There's a big difference between those How can I have my locomotives sound like the speaker out of the box? So let's work on how to think outside the box That will be the topic that's covered in my august of 2012 column in model railroad hobbyist Go to the website and look at the column download the magazine and join in the discussion on the model railroad hobbyist website by clicking on the Box at the end of the column for reader comments And you can vote for the column which includes this video And you can share comments And I'm sure that there's going to be a whole bunch of people That are going to say a whole bunch of things that they've been kind of programmed to think And I've heard a bunch of them as I've given this clinic at various nmra events Over the years And I've had a lot of people I finally in one clinic had one guy say When one fellow was saying but you got to put it in enclosure and this other fellow finally said look listen to the difference Yes, there is a difference. Let's think outside the box and have some more fun with our locomotives This is bruce Thanks for watching. See you on the model railroad hobbyist website