Hello everybody, welcome back to another Adobe Premiere Pro CC 2021 tutorial. In this one, I'm going to show you how to do a strobe effect or a flashing light inside Premiere Pro. Let me show you what I'm talking about. Here's an example. There we go. We got flashing lights and we've got a strobe effect. So I'm going to show you how I did it. Let's just get started here. So I'm going to go from step one to step zero. Now I've got a very short piece of footage because it's very CPU intensive. No worries. We're going to delete that composition and we're going to start with the standard footage. So I'm going to drag and drop this piece of footage into the timeline. Excellent. And you'll see here it's the same stuff. I'm only going to do about three seconds or so of footage just because it's very CPU intensive and I don't want to get bogged down with all sorts of CPU stuff. So let's go ahead and do that. I'm also going to make the snapping on as well. I took the snapping off by accident and I'm going to put it back on. Okay, good. So here's the footage. The next step here is you want to go up to your effects panel. If you don't see that, go to window. Make sure the check mark beside effects and under effects type in, yeah, it's original strobe. And what you get is you get video effects stylized strobe light. Drag and drop strobe light and drop it onto the footage. When you do that, you're going to get some random strobe light effect here in the effect controls panel. You're going to get random flashy strobes. And it's going to be rhythmic and it's not really what we want. So let me just go ahead and show you what that looks like.  We don't want that. That isn't very convincing. So the first thing we're going to do is we're going to go ahead and increase the random strobe probability. So I'm going to keep it around 30%. So that means basically there's a 30% chance at any one period of time that a random strobe will flash in. So you can already see that we've got some pretty cool looking strobe and flashing when we hit the spacebar. Okay, hold on. There we go. There we go. So it's pretty random looking, pretty good. The next thing we want to do is we want to change the strobe color. This is an important one for me. Left click on it and then left click on the eyedropper tool. And what we want to do is we want to get a light color. Let's go here. Let's take a color from the shot because what happens is is when you take it just straight white, which is the original color, it doesn't look very convincing to me because it's not a very good. It's just high contrast and that bright of white just doesn't make sense to me. So what I've done is I've selected a color from the video here already and then I'm going to move out and I'm going to move it down a little bit towards the lighter spectrum. So something like this, this is a light that I would expect to see in the video. So let's hit spacebar and now it's a little bit more convincing in my personal opinion. The next step is we're going to go ahead and we're going to change the blend with original. This is a very important piece. And the reason what you want to do here is you actually want to animate this over time. We want it to be very random because that's what a strobe light to me. It's not always the same. You know, sometimes it's really bright. Sometimes it's a subtle flash. So what we'll do here is we're going to go ahead and click on that toggle animation switch. Let's click on it. This animates it. Now I'm going to start it off on 0% and then every three or four frames, I'm going to step forward a frame one, two, three, and then I'm going to increase the blend. So I'm going to increase it to like 80%. And then I'm going to move forward three more steps. One, two, three, then I'm going to decrease it to 13 or decrease it to like 15% or something like that. And what we're doing here is we're just changing the amount, the strength of the light. So it always changes. So now I'm going to blend it up higher again. I'm going to bring it up and this is kind of random. I'm just sort of doing this. However, I think makes sense. So I'm just going to keep going a few more frames, go four frames forward, put it down four frames forward. I'm going to put it in the middle. And again, this is very, very random. But the point is, is that's what we want from a strobe light. We want it to be a randomly flashing random strength light. So I'm going to go down quite low here down to something like 33. And then I'm going to go forward a few more frames and I'm going to drop it up to like 77. I'm going to go a few more frames. You see what I'm doing here, guys? I'm basically just animating this. And what you'll see here is when I stretch this out, you'll see here all of these key frames. So let's go ahead and show you the first second or so of what we've got now. So I'm going to go to sequence, render in to out and this will now render all of the frames so that you're not getting a RAM preview janky. If you do it from RAM, it can be a little bit janky. I guess that's the right word. It's a little slow and it doesn't show you the effect right way. So let's go ahead and render it out. And then when you do that, the top bar here is going to go from red to green. And that basically means that the effects are rendered and you're seeing it the way you're supposed to see it. All right, here we go. Flash, flash, flash. There we go. Now we have this cool random color. We got this random flash strobe effect and it actually looks pretty believable. If you want to change the frequency and the duration of the strobe, so if you want it to last less than 0.5 seconds and maybe you want to do it 0.2, let's go to like 0.2 and then the strobe period will go to 0.1. I'll just show you in a different look and let's go ahead and look at that and see what it looks like. I'll hit space bar and then hopefully it'll play it. It'll play it a little bit in RAM preview beginning. Not to worry, here we go again. This should be a little better. Okay, so this has a faster flickery light to it. And finally guys, I will show you one more thing while I got you. You will notice here that for the strobe operator, it's set to copy. But if you want a subtle effect, set it to multiply and now watch. I'm going to even reduce the size a little more because I'm going to render it into out. If you change the strobe operator to multiply, it's a very subtle flicker and it's very, very believable. I'm going to go ahead and render into out. This is the last piece that I'll show you here, guys. Thank you for bearing along with me. Let's go and show you the strobe operator multiply. And then, yeah, this is also a very believable strobe effect, although it's quite subtle. You see, this is very, very subtle. I'll hit space bar again, very subtle, but it has that nice little flicker. Very, very nice for scary movies and cinema. Anyways, guys, that's it for now. Thanks for watching. I'll be back soon.


I guess my question is if you achieve moksha, is that an eternal state or can you come back and, you know, can you accumulate karma in a state of moksha? Oh, moksha means dissolution, dissolution means better come back, no coming back. There are no different states of enlightenment, it is just that there are different states of evolution towards enlightenment. For most beings, their moment of enlightenment and moment of living the body is same. You looked at all this, I think. So why is it it's come to this moment for me to see these things? It could have been seen anytime, but you know, out of season things don't bloom. Even now, all of you have been sitting here in the same space. I am attending to all of you to the same way, but still all of you are not imbibing this to the same extent. For some people, it's like, you know, it's life transforming, absolutely. For some people it's good, for some people it's nice, for some people, half and non it's nice. The same thing is being offered, it's just the ability to imbib it. So, right now we can see this also, either you can use this as a tool towards your mocks right now, or you can use this as a process of hastening your evolution. Both are possible. Whichever way you use it, it's fine, it takes you slowly there. Either you can go right away or you can go slowly there.


We have Niraj Pandey. I hope I pronounced your name correctly. That's correct. He's a student at Ashoka University and he's studying computer science. He's interested in generative arts, software development and quantitative finance. And he's going to talk about creating art with code. So how to use algorithms to create very nice looking things, I suppose. Well, this is going to be exciting. So let's see what you have to present. By the way, where are you joining from? I'm joining from India. And where in India? New Delhi. Ah, right. Nice. Very nice. How's the weather in New Delhi? What time is it now? It's pretty good. It's 2.30 p.m. in the afternoon. Good. Well, comfortable time, right. Excellent. So let's start the screen sharing and then you can go ahead and give the talk. Shall I start the screen? Yes, please. Okay, so the talk is the joy of creating art with code and I'm neither hi everyone. And a little bit about myself. Currently a sophomore student at Ashoka University and my interest lies in software development, generative art, distributed computing and quantitative finance. So these are the points for discussion of the talk. So we'll be talking about generative art. The history behind generative art. We'll see a quick overthrow of how processing that pie mode works. And then we'll see how we can use the geometry algorithms and randomness that frames our generative art. And we'll be seeing some quick examples using processing that pie. So the main objective or the main goal of that stock is to look at Python as an artistic tool with the simplicity of generative art. Using processing and pie title. So what is a generative art? So an art created through the use of an autonomous system is simply generative art. That's the simplest definition you will see. So it uses iterative commands to draw vector based shapes on the screen and most of the art created draws inspiration from modern art and especially the pop art. And usually an autonomous system is required because without it it will mostly be a digital art and randomness can be one form of that autonomous system. So before we go forward, so let's see how all this came from. So when you talk about the analog art, which is the art which is manipulated by hand, the complexity and scale requires exponentially more effort and time. And when you talk about computers, they excel at repeating processes for like endlessly without exhaustion. As you will see, the ease at which we, computers can create these art forms contributes greatly to the aesthetic of generative art. In the past, one major challenge which was faced by the early generative artists was the limitation of an output device. So the primary source of that time was using a plotter, which is a mechanical device holding a pen whose movements were controlled by the instructions that were programmed into the computer. And one of the first artists to produce a plotter color drawing was Freda Nick and the painting, the art has been shown on the screen. It's called homage. And this is one of the earliest best known pieces of generative art. It's called Scotter by George Neese and Scotter starts with a standard 12 by 12 rows and it increases in magnitude as we go like down the road and it changes the rotation, the magnitude of location change. So generative art is one of the best option and you're working on similar art places because maybe you want to create a similar art piece using your hand on a pen and paper. And it might take you hours to produce one. So instead you can just input some simple commands from the computer and you can create thousands of such art pieces in a couple of minutes. So after the creation of processing, it became much, much easier for artists and creators to design and make computer art. So it was created by Ben Fry and K series who worked on the processing foundation from the last like 19, 20 years now. And processing is programming language and environment built with media art communities, you can say it's created to teach individuals the intro of programming within the media arts context and so as a software schedule. And okay. So the examples you'll be seeing in this talk will be using Python mode and processing mostly. So it's a good practice to quickly get an overview of how processing mode actually works with Python. So this is exactly how of any function will look like in processing, in Python mode in processing. So we have two major functions. One is a setup and the other is a draw. And the setup function is basically it runs once the program is executed and mostly the methods which are called inside the setup function are the creating the size of the canvas, adding a background color and maybe adding noise, adding some blur effects, et cetera. And the draw function runs until the program is stopped. So which means it is running in loops. So each statement is executed in sequence. As you can see, if you don't want the draw function to run again and again, you can simply use no loop. And otherwise, after the last line is executed, the first line will be executed back again.  So most of the art pieces are using mathematical functions like noise, trigonometry, filter methods like blur, and use various algorithms such as L systems, pixel mapping, Poisson dis-sampling, et cetera. So we'll be discussing a few in this slides and see how we can incorporate all these mathematical functions and algorithms and create some simpler static art pieces. Okay, so we have the random function, which is like one of the most important functions which artists use pretty often. So the use case is pretty simple. It just provides you with a random floating point number between zero and one. And random function differs in different programming languages. So it's not like going to be the same, how we are going to be using in processing. So in processing, random function is used by directly calling random and it returns a random floating point between zero and one. And if you want a random number between a range, it will be, you have to provide a min and max range. And if you're not using processing, if you're using some other environment like PyKyro, you can simply use the random module, which is available in Python. Okay, so before we move forward and start exploring art pieces that uses vector operations, shapes and the method, we should understand how exactly a artist canvas looks like when you're working on a generative art. So, okay, so the canvas is like a 2D Cartesian plane, where each point can be considered as a vector, which is a vector in a 2D Cartesian plane. That's basically the distance between two points. So as you can see on the right-hand side, we have a Cartesian plane and each point is denoted by X comma Y, which basically stores the information on how to get to that point from the previous point or here we're using origin. And we can further use operations or linear algebra operations like linear transformation, rotation, to add some instructions to the vectors. And creating a point and a line is again pretty straightforward. So if you want to go through all these again, so you can directly go to the processing official documentation and all these stuff is already being mentioned there. So I'll just quickly go through it. So a point can be created by passing an X, Y coordinate and the processing will add a point to that pixel position in the canvas. And if you want a line, you can simply add the coordinates of those two points, X1 comma Y1, X2 comma Y2 and the processing will add a line between two points. And if you want to change the thickness, you can use stroke weight. And if you want to change the color, you can add the color by using stroke. And similarly, if you're working with PyKyro, you can use set line width, which is a substitute of stroke weight. And similarly, you can create a line by using line two and other functions. And these are the very basic examples of how you can use those straight lines and points on the canvas to create some very simple art pieces. It's like on the left-hand side, there are vertical lines of different length and opacity, and they're using some points of random points with various variable opacity. And on the right-hand side, we have just lines, but they're using a lot set of colors pallets and I'll randomly using those to fill the stroke fill. And the next is, okay. So this is a very important function and we use it pretty often. It's called Curve Vertex and Bezier Curves. So Curve Vertex is an implementation of catmolorm spline, which is a type of interpolating spline. That is, it basically goes through the points in processing. It specifies the vector coordinates for curves. So this function can be called in between, can only be called in between begin shape and end shape method so that it can create some sort of stroke curves. And these functions allow creating complex forms on our Cartesian plane. And on the other hand, we have Bezier Curve, which is a versatile mathematical curve in vector graphics. Basically these are the vector graphics that could be scaled indefinitely and has a series of anchor points and control points. So for Bezier Curve, you need at least three points which is the initial point, the final point and one control point or the anchor point. And you can simply call Bezier by providing the three-point coordinate and it will create a Bezier Curve on the scandals. And similarly, you can use Curve 2 to create a curve on the canvas if you're using PyKidl, which draws a curve from the point A to D and it has B and C as the anchor points. So this is how curves of variable opacity and lens look like when we let the draw function run for a number of time. And to implement this, what we are doing here is we are basically inside the canvas, we are creating a follow from, which is going from zero to three, so zero to four. So we are creating four points here. So we have X and Y array initialized and on line 17 and 18, we are adding the noise value. So for now, I'm not going to tell you what exactly is noise. So for now, you can think of it as a random number, generating function. So once we have those X and Y values, we are like incrementing the M and N factors and once we are done with the loop, we are creating a Bezier Curve with all those four values and we are increasing the offset values so that once the draw function is done with one loop, it should the next Bezier Curve should not be formed on the same position. There should be a distance in between and when we let it run for some time, you get some like cool looking curves and all these things. Similarly, there is another inbuilt function in processing which is curve vertex and it's like we need at least four points to create a simple curve on the screen and you can call this function by simply calling curve vertex and providing the point coordinates X and Y. And here what we're doing like on line 13, we are calling the curve vertex and we are going from top left of the canvas to the bottom right and we are adding, we are also using on line 15 no loop which means we just want the draw function to execute once and on line 13 we are using curve vertex which is taking the I and J index of the loops and we are adding the random value between minus 10 and 10 to those I and J index and but if we would have run the program with a single curve vertex function we would have got just a single point on the canvas but we want some curves so we added four, five, six, I think six more points. So when we run this program we get these scribble type effects on the canvas and if you let like add more points it will make more like curves and if you let the function run run for more number of times by using frame count check then it might cover entire screen and the next is okay so we can also create some basic shapes like ellipse, rectangles and squares and in the further videos we'll see how we can implement them actually to create art pieces so ellipses can be created by providing four parameters which is AB stands for the X and Y coordinates and C and D are basically the width and height and a rectangle takes four required parameters which is A and B again the X, Y coordinates and C and D are the width and the height and other four variables are basically the border radius of the rectangle and square takes X and Y coordinates and the side length of the square similarly we can create the same shapes in PyCycle. Okay so one best example is by creating some art pieces which were already being created by artists so this is an art piece called composition two in red, blue and yellow it was created by Piet Mondrian so this is not the actual painting this is like created randomly with random fills and random shapes but it looks similar it's using just rectangles and stroke colors and on the right hand side it's the same sketch that we created on the left side but instead of creating a single sketch on the canvas we are adding multiple rectangles and this is one simple example where we can just use rectangles this is very simple and using just rectangles and some random color fill we can create some cool looking sketch so to create this what we are doing is like we are going from top to bottom of the canvas and at each index we are dividing the width of the canvas by a random midpoint and then we are creating a rectangle from the initial position to that midpoint and from that midpoint to the final one so once we are done creating rectangles we are filling them with a random color between 0 and 255 so it can be a shade of black to white so once we run the function we get something like this on the left hand side and on the right hand side instead of using just random colors we are also using a color palette and then we are choosing colors from there and okay linear interpolation so the ler function or the linear interpolation function is very convenient for creating motion along the straight path and for drawing dotted lines so it basically calculates the number between two numbers at a specific increment or the specific increment of time so the amt or the amount parameter is the amount to interpolate between the two values and we can say the zero value will be the first value zero point will be the immediate next value and so on so to understand the linear interpolation in a better way we can think of something like a gradient fill in the canvas so this isn't very easy to use in build function in processing called ler color so what we are doing here is we are going from left to right of the canvas and we are adding a variable linear int value which is basically mapping the value of index i to zero and one because we want the int value to be very small between zero and one just like the noise value and then we are using this linear int value to add a color fill so we are using the color fill as linear int color which is basically using the lerp color function and this lerp color function is taking the initial value of the color and the final color value and it's adding the specific increment of unit point that we created above line number 11 and once we have this color fill we are just filling the rectangle we are creating and once we let the function run we get this smooth gradient fill on the canvas and okay so by now we talked a lot about how we can use random numbers or random generated stuff to create these art pieces so unlike random numbers generated that has no relationship between the previous number generated and the next number like we saw in the previous slides with me draw random points and the graph was pretty much like zigzag in a zigzag form and here instead of that the graphs lines are pretty much smooth so in Perlin noise the numbers generated are pretty much close to each other and has a relationship with each other and it's more organic in nature and that's because these numbers generated are naturally ordered sequence of pseudo random numbers so what it means is we can see here okay after this so implementing a noise like generating a noise number is pretty straightforward like just like random we just pass the XYZ coordinate or if you're working on XY coordinate system then just use noise and pass the coordinates value and it returns some Perlin noise at that value and sometimes we want that noise value to be constant whenever the draw function runs again and again so we can just fix the noise value to a constant value by using noise seed and similarly if you're not using processing you can use the noise module in Python for PyCard and other environments okay we can see this example to understand noise in a better way so what we're doing here is we are loading all the pixels from the image and what we're doing is like if you add a variable initialize a variable R and you're trying using it to fill all those pixel back so you're creating a variable R so basically we're looping from top left of the canvas and to the top bottom right so we're covering all the pixel values and at each index we are creating a variable R which is generating a noise value at X offset and Y offset and there's noise value is between 0 and 1 only so this value is pretty small with a color value we want a color to be between 0 and 255 to maintain the RGB index so we're basically multiplying it by 255 so that we get a color shade between black and white and not just black and once we have all these values we are updating the pixel index with this color and eventually we are updating all the pixels and once we run this function you get this cloud like effect so basically this is just like small small points with noise values all the values pixel values are pretty much close to each other so that the noise values are close to each other and to understand this for like in a better way we can say if we have a canvas and if we have two points on the canvas and those two points are pretty close to each other then if we try to generate some random points on the noise points on those two points then the numbers will be pretty close to each other and we would have used random instead of noise we would have got a canvas with just random points and this doesn't show any discernible pattern also okay so we can also use noise and some trigonometric functions like sine and cosine waves to create these waves like structure with the ion structures and to implement this we basically doing like we are making a while loop and we have some zero to two pi so we want our points to be aligned in a circular manner so we are adding an x offset and a y offset which is basically mapping the cosine value and the sine value between zero and ten because initially these values are between minus one and one and once we have these offset values we are creating a noise factor which we will be using to create an x and y coordinates so in a noise factor what we are doing is like we are basically using those x offset and the y offset values and we are also using a z offset because we want the curves to go in other direction as well so we are mapping those values between zero and one fifty that's because we don't want our curves to be very small so we want them to spread out across the canvas and once we have the noise factor we are changing the x and y coordinates and once we have changed those coordinate values and we are just creating curve vertex that we saw in the previous slides and once we are done with this we are decrementing the radius that we used in the noise factor sorry the x and y coordinates that's because we want that at each time the draw function is executing we want the next curves to be created at a distance different from the initial one and we can also change like cool so we can also change the RGB values or opacity whatever you want to do and you can also add other instructions and if you run the program you get something like this so I'll just quickly go through the other slides because we are left with not much time okay so Perlin noise so it's like a very important function for adding movement to your art pieces and you can also add some nice functions to add grain like effect on this canvas and this is an example of how you can use Perlin noise fields so these fields are basically the 2D fields of vectors each pointing in a similar but different direction as it's neighboring vectors and the velocities of these vectors depends on the other vectors only so depending on how we draw these particles we can generate some pretty cool stuff and as these are used for adding a moment in your art piece you can see that we are creating a noise field where the particles are moving from the right hand side to the top left and to implement this we are just basically creating some random vector points on the canvas and just appending those vector points to the points object and once you have those points object inside the draw function we are taking those the values of that object one by one and storing it inside vector object and we are looping from 0 to 20 and what you're doing is we are creating a noise value which is like we are mapping the x and y coordinates of that point between 0 and 2 pi because we are getting the radiance value over here because by radiance value we can change the x2 and y2 parameters on line number 21 and 22 and once we have created x1, y1, x2, y2 variables we can add the vertex value at x1, y1 and we then have to update the vector object according to the x2 and y2 values that's because once you have a point added to the canvas we want the next point to have a slightly different direction but as these are using noise values they will all be directing to a similar direction eventually and eventually if you let the program run for some time you will get something like this and we use heavy use of geometrical patterns and factors and chaotic the chaos theory when they are working in generative art a very simple example of geometrical pattern would be the sierpinski triangle so this is a very basic and another famous example of geometric pattern and where we are recursively dividing the triangle into smaller triangles and adding a stroke to it and we can also modify the sierpinski triangle by not just adding a stroke we can also add some fill inside and we can use linear transformation that we talked about earlier we can use rotations we can rotate these triangles we can use curve vertex to shed the sides of it and we can also use recursive approach to create factors and by using vertex and curve vertex and when you talk about the factor the first thing that comes to the mind is a Mandelbrot set so the more the zoom inside the Mandelbrot set the more similar patterns are able to see and that's the most fascinating part about factors and this is being represented on a complex plane where x y axis represents a real and the z axis represents the imaginary path so we pick a point in the coordinate plane and add the iterated and pass it through the equation z and plus 1 z and square plus c and we can also add some instructions of changing the colors and everything accordingly and similarly we can create Julia sets which are pretty much similar to Mandelbrot set and these are basically the boundary points that tends to infinity and then tends to infinity when you go deeper inside the Mandelbrot set we see that the Mandelbrot set is extending outwards on the x y plane and creates a bifurcation diagram and this logistic map is basically a part of the Mandelbrot set and this bifurcation diagram is created only on a real line because we put only real numbers into the equation and one fascinating thing about this bifurcation diagram is that this method was the first method to generate random numbers on computers and gives rise to a very famous topic called chaotic behavior or chaos theory and which is basically deterministic chaos something which is unpredictable so very small change in the initial state can bring drastically different changes in the final outcome and so we can change an initial change by some margin and we think the final outcome will be slightly different like you can see the four images but what we actually got in the end is our deterministic chaos and the best example to see chaos theory is attractors and which are basically mathematical functions that tends to evolve over time and if you're using processing you can just use points to create these attractors and if you're not using processing there's a very cool library called PiWis which has great examples of attractors and you can experiment with them like Swenson attractors, Bithead attractors and more okay so this is like the second last topic so we'll see how we can simulate the water paint effects or the oil paint effects on the canvas so the second image shows how we can use the water paint effects on the canvas on the 2D Cartesian plane so to give you an overview what I'm basically doing is like I'm basically creating a shape such as polygon and then starting extending its edges outwards and doing by doing so recursively and passing it through a deformation technique and eventually we will be getting some fine details and fine vertexes on the outer layer which eventually gives us this cool watercolor effect and further you can also create use some blur effects etc to add some more features and so how to do it so I'll quickly take 2-3 minutes to finish this thing so first we divide the polygon in firstly we create a polygon of say 20 sides and append those values to the polygon object then we initialized a mid range of 50 and we tried to divide those sides so we call the function getMid and by passing the parameters as the polygon object and the mid variable these values we passed are basically the mean and standard deviation in our function so inside the getMid function we as we pass the two values as mean and the standard deviation we are looping over which is taking the previous value and the current value of the polygon and we are just passing them to the getCenterPoint function which is returning a random number fitting to Gaussian distribution and once we had that value we can append those the previous value and the newly generated midpoint value to the new vector object and we can just directly return it and now what I'm doing is like I'm looping the length of the newly updated polygon so initially we had some 20 sides of the polygon now we have thousands of sides and once we have those points we can simply create the x, y coordinates by passing that value of side x and side y with a random value between 0 and 25 just give me one minute I'm done okay so we just we are sending the values back to the random Gauss values and once we have those coordinates we can add x and y position and this is a random Gaussian method for the scene and eventually we'll get something like this and this is the last thing where okay this is using shaders for creating oil paint effects and you can check the link for better understanding or the better approach which is going to say by Tyler Hobs this is the last thing which is like pixel sorting algorithms which is I just I won't be going to the code I'll just give you a quick over through I'll just take one more minute what we do is in the pixel sorting is it's basically isolating the horizontal and vertical lines of pixel in an image and we are sorting the positions based on a number of criteria so basically we load the pixels we then pick the original pixel with some x's of function we then pick the next pixel position change the number of signs and maybe you change you can also see a change in the direction of pixels we just compare it and swap the pixels so and this is a very simple implementation of it and eventually if you just then the function with this values you'll get something like this and this is Niresh I'm sorry to have to cut your short it was a very nice talk thank you very much if you have some questions you can go back to the talk call back and just chat with me thank you so much yes thank you very much and let me play your applause so let me just do this like this applause thank you thank you


We approach all of our projects as how can we make the world of dogs more like the world of children. What we didn't like about the world of dog treats was how boring and repetitive they all were. We decided to try to disrupt the category as much as possible by thinking who has done this before. Cereal in the 80s and 90s did it really, really well. That's where our thinking started and that's where we ended up. We design everything with the owners and the dogs in mind just because a dog can't see all the colors we see or read all the copy we see. It doesn't mean it can't be fun and so we thought what is a product that we can get insanely detail oriented around but will still deliver an incredible product and treat for the dog but give the human a lot of fun and stand out when they're doing it. Nostalgia plays a huge factor in the designs and I think any parent who walks down the cereal aisle can relate to how much these boxes pop out at you. That just doesn't exist when you walk down the treat aisle in retail. Every single bag is just a copy of the other bag and we didn't want to think what is our take on it. We wanted to think what is the humanized take on it. I think you'll be hard-pressed to find anything like this in any pet store. The response has been great. People are loving to see it. The mascots, you know, the jury's still out. Some of them are a little bit more trouble than others. Nutty over here. Keep an eye on him. He's a tough one. This guy's pretty good. We came up with the names of the mascots based on what their back stories are. Some of them were inspired by some real cereal treat characters but who you don't see here today is Tootie Loop.  None of the wishes come true but that's okay. Captain Crunch doesn't have a boat. What's next in our dog treat line is strategy-wise more of the same. We want to humanize this category as much as possible and what you might see are some big brand partnerships that come to life to try to say how can we merge this world of human smells and human cravings and sort of bring that to an extension of a line like this. These mascots are here to stay. We really want people to know about them. We're calling out Kellogg's. We're calling out General Mills. We're coming after you. We want to do arm wrestling. We want to do swim raises. We want to see who's healthier. We want to see who can commandeer a boat better. I think we can probably find some fun things to do with some real-life cereal mascots.


Okay, the last talk of the day, always a challenge. So we're gonna try and keep it fun and light. So we're gonna talk about music discovery and language models. Both of us don't work at Spotify or SoundCloud or any of these companies. We just wanted to build something that's fun. I'm Nishchil. I'm currently the VP of data and ML at Scout B based out of Berlin, building large language models, knowledge graphs and a bit of ML ops. And Raghotam. Hi everyone, I'm Raghotam. I'm from Bangalore. I work for PayPal and my regular day work looks like playing around with LLMs, document AI, content moderation models to name a few. Happy to be here. Yeah, fun stuff. So what's the problem you ask, right? So about a year ago, Raghotam and I and one of our friends shares, we started talking about YouTube recommendations and Spotify. So we sort of started having wars, like saying Spotify recommendations are better, YouTube's better. But then we realized that while they're good, there's still some challenges, especially if you're based out of cities where there's a lot of music and performers, finding an artist you like is very hard. Even if you go on Resident Advisor, Eventbrite, wherever, it's impossible to find unless they are famous. And if you filter by genres, just to give you an idea of what 2022 with genres look like. So these are all the genres that you see. So let's imagine you like house and you just search for house. There are about 98 variants of house, right? And it's almost impossible to understand what you actually like. So that was one. And then we're Pink Floyd fans and we are always looking for artists who perform and play riffs like David Gilmore from Comfortably Now. So we couldn't find a place where we said, okay, if we took this piece of track and put it somewhere, just this slice, can we find other artists who actually have tracks that have the same kind of samples in them? And we couldn't find anything either. And the third thing that we said was, okay, as music producers, and we're talking a lot about an event, even at the conference with large language models becoming a bigger and bigger reality, everyone's talking about, okay, are they gonna replace people? Are they gonna be loss of jobs? But given that both of us have been working with language models for quite a long time, our mindset here is can we build things that can actually help people do stuff rather than trying to think about, this will replace, right? That's not gonna happen. It's more or less thinking about, as a music producer, I wanna build this DJ set. Now I have all these tracks I wanna play. Now, how do I combine them? How do I build bridges between them? How do I select a sequence of songs so that I can have an awesome set, right? So that saves a lot of grunt work for the producer. Now, what does the current landscape look like? So just to give you a quick statistic, so 70% of the music streaming market share comes from these platforms. And I think Spotify being the biggest of it. And then there's YouTube music and Apple music and then there's SoundCloud that kicks in. So the challenge most of these platforms have is that they're built to operate at scale. So you're talking about billions and billions of streams kicking in and you always have this challenge of explore versus exploit. And this is a normal challenge you have building any sort of recommendation system, right? So how do you let your users explore for new content? And at the same time, the content that you show, the recommendation that kicks in, it almost comes from collaborative filtering, which is essentially who's listening to these tracks. And what happens with that is popular artists become more and more popular and a lot of new artists and upcoming ones, it's very hard to find their content. Unless Spotify brings it to your playlist, the chances that you've heard of this music producer is almost close to zero.  And as humans, we like this too because you keep listening to songs that you like. And when you are biking, you're in your car, you'd like to listen to songs that you like. So a lot of times songs that you've already heard sort of is used as part of your playlist. Even if you start from a song X that you didn't know, chances that you will listen to songs that you've already heard start to get higher and higher as the tracks proceed. Now, how did we sort of solve this, right? And there are conditions apply. So it's not like we've solved this, solved this, but we have an approach that we just wanted to share. And Raghutam's gonna run you through that. So we saw one of all the problems, right? Across different kinds of users, be it music producers or as consumers of music, if we want to find a similar sample of music then it's hard. So we went a deep learning route. We know that our goal was to split a given task song into a 10 second sample, right? So you might have like a minute long song and you split it into 10 second samples. Now, given this swatch of audio sample, can I find a similar one, which is very similar in terms of the BPM, the kind of music instruments those are used, right? The mood of the song, right? That's the goal. And if you look at that, it's very similar to what Shazam does today, right? Shazam is trying to find the exact same song, but for us the end goal is slightly different. We want to find near similar songs so that you can explore similar music. That's the goal.  And all of these songs have no lyrics to make the task easier. And then we split them into 10 second samples, right? And then we sample them at 16,000 and 48,000 sample rate. And the reason being the current models that we have today either leverages the data in 16,000 or 48,000 sampling rate. So quick trivia about sampling rate, right? What does this mean? Basically your audio sample is a signal. And then if you want to convert into a digital format, you have to sample it at certain time, right? And the more samples that you take per second to hear the quality of your signal and better the chances of getting a better representation and finding a better match. So that's a quick trivia about what sampling rate is. This is very similar to what you can see as a parallel in terms of the images that we have as a megapixel and the resolution, right? It's something very similar. Now, when we started digging about what are the currently existing models that we can leverage, right? Today, transformers is everywhere. So basically all we needed was transformers. So we did a quick scan, did our research and then found out that we have three candidate models. The obvious one which is Wave 2.2. The moment you say AST or audio understanding, people would look at Wave 2.2 which is the state of the art model which is mostly in SOTA in terms of understanding speech recognition, but not specifically audio. But we'll dive into that later. And then the other one is an audio spectrum transformer and the third one being CLAP. These are the three candidate models we found which are transformer based which was easy to use with our data. So what we essentially did was as a start point we generated 10 second samples, generated embeddings for these 10 second samples across the 600 songs that we have, generated embeddings and then indexed them on a vector store. So we basically use phase for this and now we took a candidate song. Let's assume that this is a candidate song. Let's listen to this. And then we generated the embedding and we saw what was the closest, similar one for this candidate across different embeddings. So let's look at what Wave 2.2 returned. Was that close enough? This is with AST model and the last one being CLAP. Clearly, while they're all similar, right? If you observe it deeply, you see that the second two songs, the samples are coming from the first sample, the candidate sample itself. So they're very close. However, the moment you choose a Wave 2.2 embedding, it is giving you some kind of a noisy answer. So we did some analysis as to why this is happening, why are we not finding a closer match with Wave 2.2, right? That's when you dive into the data, you look at the spectrogram, you look at how the audio sample is looking like for the 10 seconds. And this is how it looks for the candidate 10 seconds sample that we have. And this is a spectrum analysis of Wave 2.2, the song which was written by Wave 2.2. This is for AST model, this is CLAP. Now let's just compare them visually and then we'll see if they're really same or not. So if you look at the input sample and CLAP, they're very similar. And even AST is also similar, but Wave 2.2 actually is not so similar, right? So the probability of you finding a good similar sample is hard with Wave 2.2. And we then started looking at why is this happening? And when we started looking at what and how Wave 2.2 model was built, we understood that the end use of Wave 2.2 is more speech to text and speech analysis rather than audio analysis, which is more leaning towards the music, right? So Wave 2.2 is trained on Libre speech, which is 960 hours of speech data and it has nothing to do with music or audio, right? Not suitable for our task. However, if you look at the AST model, this was trained on a two million dataset of 10 second sample. And if you look at the classification on the right side, that dataset has a very wide spectrum of varieties, be it human sounds, be it natural sounds like river, ocean, the animal sounds, even music produced by different instruments, right? So it's able to capture all of them in that dataset. So across the entire dataset, we have 527 labels and think of this as something like an image net for your computer vision model, right? This is a parallel that you can see here. So that's the reason that the AST model is very powerful. So what's actually happening with the audio spectrum transformer is that we take the 10 second sample, we convert them into a spectrogram, which is basically an image. The moment you have an image, now you start leveraging your image transformer.  So this is much more neater for us to find your own and also generate the embedding for our use case. Now let's look at CLAP. CLAP is very similar to what you have as clip in the image and computer vision world, right? A clip is nothing but contrastive language image pre-training, but here this is contrastive language audio pre-training. So the goal here is that during your pre-training process, you have a contrastive learning loss function and your input is basically a swatch of audio and then a description of that as text. Both of them are now goes as an input and you have a fusion layer and then you have the model learning the audio and text. And during inference time, you can either just give it a description saying, give me a dog barking sound or a piano sound, right? Or you can give it an audio and then get a description back. So that's what CLAP is about. So this was trained on the free sound data set which is around 50,000 and this also has a mix, good mix of human sounds, animals, natural sounds, musical instrument. And that is the reason we see that both AST and CLAP, given that their data is closer to audio and music analysis, it is retrieving better similarity for us. So in summary, we curated a data set, 600 songs, split them into 10 second samples across different sampling rates. We generated embeddings for three different models. We indexed them, we retrieved the similarity. We did spectrum analysis and made sure that, whatever we are getting as similar or really similar. So with this, we kind of now found out a way to find similarity. And one point to note here is that we just want to clarify that this is in no way. We are saying that the current recommendation algorithms are inferior, but as a music producer, if you want to find similar songs or if you want that and then put more layers in terms of the creation that you have, then you're a different target audience and you might need a different platform or a recommendation system. So now that we have figured out how we find similarity, discovery is the next part of it which Nisjil will walk us through. Okay, awesome. So one thing is finding similar samples, right? So of course we went through our whole understanding of the transformers networks, what we can do, embeddings, everything. But on the other side, discovery is a whole other challenge. How do we sort of have a user experience where people can discover music? And one of the key things that I've had the privilege of working the last few years is to think about knowledge graphs and graphs as a means for supporting discovery itself. There are quite a few talks that covered about graphs at the conference as well. The idea is that a lot of times we think about data in a two-dimensional form which is rows and columns and now the newest form which is a vector databases or key value stores. But in order to sort of have exploration and understand the explainability behind the exploration itself, graphs can be quite powerful. So if you wanna start with graph and you wanna apply this to any of the domains or the problems you're working with, ontologies are a way to sort of express real world entities in a graph world, right? So you sort of try and understand, okay, what are my entities and what are the relationships between them? So we started with something very simple where we said, okay, so we have a song which is an audio entity and then we have an artist. A song can have, a sample is also a type of audio and then they have relationships of similarities between different other audio samples, right? So we wanted to power discovery from this and this is what discovery starts to look like. You start with a song and you don't have to stare at the image and try to zoom that with your phone. We'll show you a demo where this actually, you can sort of navigate through the graph. We can see that 10 second samples have similarity with other 10 second samples and they're all coming from different artists and this was actually very useful for us because if we tried to build anything where we couldn't explain why we are starting from a particular track and where we are lining up, it becomes a challenge in terms of trying to build an experience for discovery itself. Now, yeah, we didn't want this to be all talk and no show, right? We actually wanted to show you a demo of what this looks like. Okay, so I'm not sure how many of you know Ben Bomber. If you don't, I definitely think you should check it out. He's a great music producer. Okay, so we have somebody in the audience who likes him. So this is a track that we selected and we have a bunch of samples from everywhere. So playing this track and I'm from Berlin, of course you expect house and techno, it's not gonna be anything else. So this is the input track and then you see the sample and the spectrum next to it and this is cool, great. Now, what we generated by selecting that as an input track is that we brought a similarity and a traversal algorithm that picks up all of the other songs that are not coming from the same sample using two different transformer models. So we didn't use wave two, wave two because we realized the quality of similarity and detection be quite poor given that it was not meant to do this job and you can listen to another song that's coming from AST which it thinks is the most similar and we'll play that as well and it's coming from another Ben Bomber song and we have one for clap two, sorry. And by the way, all of this is built with Streamlit and all of the open source libraries and the codes available on GitHub as well. So if any of you wanna play around with it and build something, everything's there. Of course we can't publish the dataset because this is something we picked up and we don't wanna get into a whole lot of legal tuzzle by publishing music datasets. We're two people who like to do the tech stuff. We don't want anything with the legal lawyers and have a fight there. But anyways, that's one part of it, right? And we found a ton of other similar samples and then you could go on and on about this and we said, okay, so this is great. So we started from identifying a small portion of track and from there we use the capability to find other tracks that sound similar that's potentially not coming from the same artist to sort of go through the whole discovery process. Now, none of us are gonna be building our own playlist. I mean, maybe we did it when we were working with Winamp and in the early 2000s where sharing playlist was the way to get street cred, but we don't do that anymore. We just let Spotify do it for us. So we said, okay, so if we wanna get to anywhere near close to enabling that discovery, we need to build a playlist. So what we tried to do is we said, okay, we start with our first sample and then we find the next similar sample and use that as a seed track and sort of jump through the entire graph to enable this. So before we show you the playlist itself, a little bit of graph traversal, right? So selecting the song, which you can see here. So you can see that I can find all of the other similar songs and I've of course reduced the number of similar songs and samples that we have. So we don't get overwhelmed, but if any of you are interested to get overwhelmed a little, this is what it starts to look like, right? So every sample has a lot of similarities with other samples from different tracks and because we have different transformer models that we're using, the relationship that you see in terms of similarity tells you which, what is the relationship? Where does it come from? What does the similarity score look like? So instead of computing everything on the fly, we sort of pre-computed it all of the data. Instead of showing them on a vector database where you can't really understand the explainability part of it, we sort of push them into graphs. Now with that, we generated a playlist. So the playlist sounds with our seed song, which was this, slowly starts to get into the next song and the next song and the next song. So once we did this, right? So you can see this weird switch. That switch at 10 seconds is exactly what we wanted to change, which unfortunately we couldn't get the time to build our own network. So the future of what we want to do is to sort of get in, so to build our own network where once the discovery is enabled, we want to train a generative model that given two different samples and from all of the training data that's present from the DJ sets and how they are built and the mood of the track and everything, we can enable a proper, we can generate samples in between the tracks to sort of seamlessly transition from track A to track B to track C and you can select the number of songs you want and you can go through the entire traversal for different songs that you have and of course we wanna also get into different genres. Yeah, so the future is wild with everything that you've heard around large language models and transformers and everything with Lama 2 coming out. I don't think it's for any of us to be scared. I think now we have immense power to do things which we couldn't previously build. So I think we can look at it in a way that in the next year or two, we'll see an amazing set of plethora of tools and products that we'll have access to that of course it's very powerful. We have to be very careful with what we do with it but yeah, we think it's a great thing that we see right now. With that, of course, open source for the win. None of this, if we tried to build this 10 years ago or even five, we probably had to write tons and tons and tons of code. So a big, big thank you to all of the open source work that's going on all the way from hugging phase with the models, NumFocus has a group with tons of libraries that they're finding, Meta with some of their open source projects. Yeah, it's weird to talk about Meta and open source but they're doing a great job. And then we have Python and Neo4j and all of these communities that we've sort of used in order to, worked with in order to build this. And yeah, reach for the stars. So you can build some cool things and share them in the community and speak at conferences like this. So thank you. We're open to questions. Maybe we should play the playlist. Oh yeah, sure. If anybody wants to hear the playlist. Yeah. Thanks for the talk. Very, very interesting tool and concept. So I think one of the things that makes a product like Spotify so successful is that they have a huge collection of genres, music, artists, and so on.  So have you given any thought to how are you going to solve that? OK, so one, so we're not going to bring this product into public for people to use. This is a project that we built. Yeah, this is like a side project we built. But what we do want to do is to try and see if we can, at least for our own personal use, with all the tracks that we listen to sort of get them because we're also listening to quite a few genres and then see how this tool sort of goes forward. And this was also a way for us, especially some of the work that we've planned around the generative part itself of generating samples between the generator playlist. We want to sort of publish a paper to see if that can get picked up by somebody from SoundCloud or Spotify so that they can help music producers to build these DJ sets. Makes sense, thank you. Thanks, great talk. And also shout out for probably featuring what is probably the greatest guitar solo four times. But my question actually goes in the same direction as the first one. So do you see in the future any legally feasible way that one can maybe somehow, say, crowdsource the database so that I have my music locally and it generates the model and then it will also give me recommendations from stuff that other people have put up? Yeah, so one of the things that we've been thinking about on that aspect is if we were to take this into a public domain, essentially, we could have people upload their Spotify API keys and we could essentially still build it on top of Spotify, on top of SoundCloud to enable this exact discovery because the challenge that we are trying to solve is with anything with recommendations or discovery, we as users have very little control. And if we can crowdsource this, of course, a huge piece of royalty-free music database that we've looked at as well, but we wanna sort of take back a little bit of control for music discovery because music is so personal and you wanna find new artists based on what you like and not just based on what somebody thinks we like because of something else. Thanks, yeah. Hi, so my question is about alignment, right? So if you have a long song, say, comfortably numb and you would do those 10-second splits such that you're splitting the two samples just in the middle of the solo, right? So that will obviously affect how the similarity is gonna be discovered in the other samples. So how did you solve that? Do you wanna take? Yeah, so the point here is that in a big song, right? The goal might be that you would like a specific piece of 10 seconds with which you wanna look at the similarity, right? So you then generate the embeddings and get the best match with that. But given that you have across your entire database, you have 10-second samples and embeddings for that, you do like an M cross N match, right? So it's like searching all the 10 seconds of all the songs and then getting you the best. It might be in the start, middle, or the end, it doesn't matter. But which is that specific 10-second sample that matches with the current one? So that's how we align. Did that answer the question? Yes, so I follow up to this in the kind of combining of the three different songs, like you actually start at the beat. So how did you align the beats? Was that the same thing? So how do you keep the beat going, right? So currently, we have just joined them together but that's what Nestle was talking about. It's just an abrupt join, right? There are different ways you can solve this by normalizing or doing a crossfade, which is easier. But what we want to do is have a generative model which will actually generate the gap and the transition between these two pieces, right? That would be beautiful and amazing. I'm just, yeah. A quick question. So you only showed Techno. How well does it work with lyrics and things like that? We haven't tried it with lyrics yet. So yeah, this was just a fun project and we thought that it would be easy to start with just music audios and not lyrics. Maybe it's still possible. Then you can have your singers, which they'll pull up as very similar and a combination of the instrument use, but we haven't tried it yet. Yeah, I mean, one of the things that we did talk about is when you do a fast Fourier transforms on your incoming, like the sample that you've chosen, even if it has lyrics and other tunes, you can also pick and choose the signals that you want to sort of look at. So the reason we didn't choose lyrics is lyrics add, like the way people speak and sing has a lot of impact in terms of similarity. So you can find artists who sing in a very similar fashion, but the underlying beats and tones and the mood of the music can be very different. So the whole idea of playing around, understanding the lyrics, understanding of all of the constituents that actually make up a signal, that's some work that we have to do. We haven't looked into that direction yet. I don't know if that answered your question, we can also. But it's interesting. Yeah, we also have a repository. So if any more contributions are very much welcomed as well. Okay, awesome. Thank you. Thanks everyone. Thank you.


Today we've been screening everyone in the Barracks. They have a lot of soldiers here, a lot of different units. They don't go through the gate and get screened like a lot of our people that live off posts and a lot of our civilians that work off posts. And so we've been going through and just making sure that anyone that's developed any symptoms that they're either getting quarantined, self-isolating, or getting a swab, getting tested. Barracks is a high-risk population, mainly just because of the proximity and living. From what I've seen over the last few days, it looks like the leadership's here making great strides, taking great measures to make sure soldiers are staying healthy, the ones that are sick will get healthy, and quarantining those that might be at risk of having COVID.