Complex systems are inherently dynamic systems, meaning they are continuously changing over time. Biological systems are a good example of this. The only time they are not engaged in some process of development is when they cease to exist. This is in strong contrast to linear systems that inherently gravitate towards some equilibrium state where they will remain static unless perturbed by some external force. Linear systems theory is the backbone to modern science and engineering that has created our industrial economies. Within this paradigm, technologies are designed to operate at some kind of normal static equilibrium within a well-known and predefined environment. Their life cycle is a linear one where the system is created, put into its operating environment where it is designed to function within some normal set of parameters at a stable and static equilibrium. It is, importantly, designed to resist change and maintain operations within these parameters for as long as possible before being disposed of. This model works well for simple linear systems like bridges where when we clamp our girders together we don't want our bolts to stay there, we don't want them to answer back, to change or grow in any way. But in complex systems, we are not dealing with bolts. We're dealing with components that have some degree of autonomy and capacity for adaptation. People, businesses, stock prices, web applications, smartphones, these things have their own internal logic through which they adapt to changes within their local environment. Sometimes this logic is very simple, such as some financial algorithm that will automatically sell or buy stock at a certain price, or sometimes it is very complex, such as why a person buys a particular item of fashion over another. The net result, though, is that the system can change and is not determined to follow a linear life cycle from cradle to grave. It can learn, grow and adapt in response to internal and external conditions in order to renew itself, that is, to become more or less sustainable and thus alter its life cycle. That is all quite abstract, so let's take some examples. Most businesses don't last very long, less than a few decades. There are many reasons why this might be, but one model to capture the underlying dynamics of how a business evolves is called the Explore and Exploit algorithm. In its early years, a business may explore many different products or services, being able to pivot, remain flexible and diversified. But as the business develops, there will be a few products that prove most lucrative and the business may likely scale them up, becoming centered around them and developing a more formal structured management system as they leverage economics of scale. The result of exploiting a few lucrative products will be that the company will become more profitable, but it is also self-organized into a more critical state. Some small change in the market that moves against the business's core product might destroy the enterprise. Now let's take an example of a business that has designed an alternative course of development for itself. Google, has since its inception, generated over 90% of its revenue from its one core service of web search. So why then does it develop and maintain a whole suite of products that generate relatively little revenue? One theory is that Google knows that in a rapidly evolving market, it needs to be where the next great thing is going to happen. Whether it is in social media or video sharing, by creating this diversity, by continuing to invest resources gained from exploiting its core service into exploring and generating variety, it is able to better evolve and ensure its sustainability.  To better understand this process of evolution, let's break it down to see how it works. There are just a handful of key components. Firstly, we need to be able to create some variety, then allow for adaptation, and lastly, perform selection to see which of the elements is best suited to the particular environment. Firstly, in the production of variation, the system has to create many diverse types. In biological systems, this is done by mixing and deformation of genes. In the formal world of product development, this typically happens in R&D labs, but when we harness the power of co-creation, we have a new resource for mixing and remixing to create endless diversity. Secondly, adaptation. These diverse types are put into operation to interact and adapt to the environment. This requires that we give the elements the space or autonomy required for them to be able to interact and adapt under their own logic to their own local environment. Lastly, selection. A process of selection is performed on the elements in order to select the so-called fittest, that is, the most suited to that particular environment. In design, this means building feedback loops such as user rating systems so as to determine which products or services are truly functioning best for the end user. Thus, when we are designing, we need to be aware of the trade-off between the two different models. At one end of the spectrum, we have our traditional linear systems that will likely be easier and quicker to design, requiring less of an investment and often operating more efficiently in the short run, but they will be subject to a linear decaying life cycle as they try to externalize change. At the other end of the spectrum, we have evolutionary systems that internalizes change, harnessing the mechanism of evolution to maintain sustainable, cyclical life cycle for an indefinite period of time. In summary then, the key takeaway from this section is that complex systems operate in a constant state of change. Whether we're talking about social networks, airports, or emerging economies, to be able to change as fast as change itself means internalizing this through the integration of evolutionary mechanisms into the platforms that we are building. Evolution is not a mystical process that only happens in natural environments. It is a key feature of dynamic systems, whether we are aware of it or not. But by being conscious of it, we can harness it in our design to create systems that have a sustainable and cyclical life cycle.


We camped at Mitur Dam and also the Kairas Dam in Mysore. That is a part of the country that's very familiar to me and I've spent a lot of time as a child and a youth along the Kaveri there. The dam is to the brim, actually overflowing. Kairas Dam is one twenty-four feet, eight inches and overflowing, whenever breeze blows it is overflowing over the dam. One lakh kiosks were being let out, initially three lakh kiosks and then one lakh kiosks were being let out on a daily basis, but in spite of that it was overflowing. So was Mitur Dam, one hundred and twenty feet full. So I asked the local guy who was living there next to the dam, I asked him, how many days did it fill up? He said, four days. I said, this is not good. If it rains heavily in Kurg, it would always take anywhere between twenty to thirty days to fill up, that's a good thing. But now it's filling up in four days, so for the first time, in fifty years for the first time I'm seeing that the Kairas Dam is brown in color, full but brown in color. This means the soil of the mountains is coming to Mysore city, which is not a good thing. Kaveri should come walking, but she is running, this is the problem. We need to understand the same amount of rain is coming in the last one hundred years, the average is same. Our only problem is we are not able to sequester the water that comes down in the form of rain, we are not able to put it into the soil mainly because there is no organic content and there's not enough trees and vegetation. If we plant these two hundred and forty-two crore trees, we can sequester anywhere between nine to twelve trillion liters of water. To give you a perspective of what is twelve trillion, the entire flow of Kaveri right now is twenty-one point two trillion liters. So twelve trillion liters extra if it goes into the land, the rivers, all the hundred and twenty rivers will flow, your lakes will be full, your wells will be full, above all soil will be damp most of the year and that is what is needed. For the state and the nation if you want a solution, the maximum landowner on the planet is the farmer still. It is he who has to go for it, why will he go for it?  No, he is right now struggling for his livelihood. Now the only way it will work is that it must be an economic plan for him. This is what I have been looking at for over twenty-five years, now we know hundred percent. This is not an experiment on the farmer, we know hundred percent his life will change dramatically. Right now if you ask any farmer, you can do a survey and see if you want. Not even two percent of the farmers want their children to become farmers. So in another twenty-five years when this generation passes, how do you think you are going to grow food? The most significant achievement in the country is without any technology, without any modern science, just with traditional wisdom, our farmer is growing food for 1.4 billion people. This is not a small thing. This is the greatest achievement for this nation, but unfortunately their condition right now is very, very bad. Do everybody knows about the suicides and the stuff? This is a game changer for the farmer, for the soil and water and for everybody. If thirty-three percent of farmers or thirty-three percent of the land goes into agroforestry, we have calculated by eighth year the government will be earning twenty-nine thousand crores of GST. So it is wonderful for the farmer, it is wonderful for the governments and the governments are responding very responsibly. The farmer is very enthusiastic. Now the civil society is our business to see somehow we grow healthy saplings for the farmer. Two hundred and forty-two crores is a lot of saplings. There is no one department in the government, there is no one private agency or a commercial force which can grow this kind of number. And another problem is to grow quality saplings. When I say quality saplings, right now we have thirty-five nurseries you know happening in Tamil Nadu. These thirty-five nurseries we take nine to fifteen months to grow a sapling for transplantation. Many commercial nurseries are growing the same sapling in three to five months because they use a lot of urea and grow it, we're growing it organically. What is the difference? The difference is just this, when transplantation happens the survival rate is over ninety-five percent and in that the survival rate is around forty percent. And recently the Agricultural University of Tamil Nadu TNAU in Coimbatore which is a premier institution in the country invited forty different sapling producers in the state. The Yesha nurseries also went and of the forty samples that came all the thirty-nine of them had nematoid deposits on their roots. The only one without nematoid deposits was from our nurseries. This is very important because when you shift the farmer into tree farming, suppose you give him bad saplings which will not produce the result that we are promising him, then you are pushing him further into a horrible situation, another corner. So this is extremely important, this is why this forty-two rupee campaign is going on. This will need a minimum of twelve year commitment. Are you with me?


Hi everyone. My name is Purnima Kamat and I recently joined Oliver Wyman as a lead engineer. I also run the women who code group in Singapore and Yau Conferences. If you've heard of these, look us up on Meetup. If you haven't, do come and speak to me later. We'd love to share more. My topic today is going to be on generative art using neural networks and genetic algorithms. The title was too long for this slide, so I cut the genetic algorithms, but that's going to come. This topic is something I'm really passionate about. I paint for over a decade now. A couple of years back, I thought I should work on the intersection of art and code. That's where the stock comes in. A couple of things I wanted to lay out before we start off. I was at this conference a few months back and somebody asked me, why do you even do generative art? What's the use of it? I think it's fun. It's a great way to learn something new. I taught myself JavaScript using generative art, so I think it's a great way to get started off on it. Practice the 10,000 hours rule by Malcolm Gladwell. I'm not sure if you can get to 10,000 hours doing anything, unless you're coding it professionally, I guess, but some percentage of it works. The best part is it's therapeutic. Hand coding SVGs or writing CSS linear gradients and radial gradients or some random numbers, they line up perfectly and it creates an aha moment. It's therapeutic. What is generative art? On a very high level in a nutshell, generative art is a number crunching program. You give it a few inputs. In our case, it's going to be X and Y coordinates on the canvas. This number crunching program could be a set of rules. Let's say a cellular automaton, which color codes your pixels. Or it could be something like a random number generator like Perlin noise, which could color code your pixels based on the intensity of the noise at a particular function. The output would be a series of RGB values, which you could then plot on the canvas to create that artwork. RGB or HSL or if you want a grayscale artwork, just ones and zeros, I guess. What if we replace this number crunching program with a neural network? That's majorly what my talk is going to be about. If you're interested in generative art, there are a lot of technologies that you could get started off with. Most people use processing. Most generative artists use the Java-based processing. There is a JavaScript version of it as well, which is called P5JS. I typically tend to use a lot of D3 because that's what I wanted to learn at the time. If you are into machine learning, you can use TensorFlow.js. Or if you want to do something more like facial recognition and stuff. ML5JS is a great pet as well. Something you should know about me is, as a kid, I was really introverted. I hardly mingled with friends. My best way to pass my time would be just to lay out on the grass and watch clouds and try to discern shapes in them. I think that habit kind of moved into my adulthood as well because I still see patterns in things. I don't know about you, but do you see a mask here in this pattern? If you do, you're like me, my friend. A couple of months back, I was looking for inspiration to create abstract art with machine learning. I came across this blog post by this research scientist at Google called David Ha. He used Python and a little bit of JavaScript and TensorFlow to create these abstract patterns on his Jupyter Notebook. I thought, why not try and use that and learn machine learning? I feel in order to create art, in order to create anything meaningful with any kind of technology, it's really important that we understand the basics of how it works. So what is a neural network? A neural network is essentially a set of neurons. A neuron is a number crunching program that we saw previously in the generative art, the diagram. This number crunching program, which is f of x on this slide, it can be a linear function, like what all machine learning algorithms use. It could be a square function. It could be a cube function or it could be a periodic function, whatever you want to visualize in your abstract art. Let's feed it a few inputs. We talked about x and y previously. X and y are essentially your y-axis on your canvas that you want to create your art on. Let's create another variable. Let's call it r and measure the distance of your pixel on your canvas from the origin. The origin being the leftmost, topmost corner on your canvas. So you have three inputs now. You feed it into your neuron and wait and see what comes out on the other side. And then plot it on your canvas to get your abstract art. But if you think about it, x, y, and r are still very linear aren't they? You know what those values are. And as generative artists, we kind of thrive in randomness. We kind of look towards having something different every time. Okay. Every time you refresh your page or every time you run a program, you want something new to come out. So how do you add that bit of randomness? You add that bit of randomness by introducing another input, which is z or z, however you prefer it. So z or z is something what data scientists call as latent vector. It's nothing but a series of random numbers, which really conforms to a particular pattern. You feed these inputs into your number crunching function. You also truncate your output to a particular level because after all, RGBs have a limit from 0 to 255. And anything above that, it doesn't make sense. So you use an activation function for that, in our case, it's tanh. And then you have an output, which is RGB values. You lay all of these neurons into a multi-layer network. You have your neural network, which creates abstract patterns for you. This whole concept of... Okay, this should have advanced to the next slide. Yeah. This whole concept of using neurons and neural networks to create abstract patterns is called a compositional pattern-producing network. This word was coined by another research scientist at Stanford called Kenneth Stanley. He wrote intensely about cppns in general and how they can be used in generally for machine learning stuff. So I tried to build a wrapper over this neural network in order to configure the network to the size that you want and try and create, you know, just keep hitting generate and it kind of creates and spits out a lot more abstract work. Oh, so you're not seeing the artwork, are you? There you go. Right, so this is my version of Big Breeder. So I tried to create a form which would accept the number of neurons that you want in your network, in each of your layers. So this particular network has three layers. So you can configure. You can't really see it very well, but the first bit says 8, then 16, and 32. Let me see if I can.  So, yeah, so you can configure the number of neurons you want in each of your layer. You can generate with color or you can generate black and white images. So one of my biggest ways I pass time these days over the weekends is kind of keep on hitting generate and see what comes out. So, yeah, that's my favorite thing. The best part of using this to create generative art is actually to use any kind of functions that you want. So if you use periodic inputs, the periodic is like trigonometric functions like sine or cosine, you get a series, a pattern which is periodic in nature or repetitive in nature, and it looks actually better in color. So you can use any kind of functions that you want to create any abstract patterns that you like. Of course, this UI limits you to using certain functions and certain configurations, but TensorFlow.js is pretty, pretty extensible. The best part of using a neural network to actually create abstract patterns is this concept of biomorphs.  So, yeah, it's pretty cool. And I try to create like... Of course, this is a version for the demo. If you're at home and you want to create larger images and you have the bandwidth to wait for the processing to complete, you can generate larger images. They're pretty cool to look at. So some of the things that I tried... Oh, come on. Can I switch to slides again? Oh, okay. Unfortunately, the demo works now. So these were like my backup images. So I was able to generate something that resembles a wasp, at least to me. I don't know what you can make out of the first image. But this was using a square function and a relatively simple three hidden layers with tannish activations. And this particular one is always spooky abstract pattern. I don't know if you can see a face of a man in there, but yeah, it's... This was created using a slightly more complicated neural network. Who's into Star Wars? Okay, so do you recognize the guy on the right? So, yeah. So one of the best things that I do on Sundays these days is generate these things. So these are great.  You pass it random numbers. You update the functions, and you create abstract patterns. But wouldn't it be cool to actually have a couple of patterns that you like, mix them, and create newer patterns? So why don't we have a look at that? So Kenneth Stanley, the person who actually authored the original cppn paper, he also speaks about another algorithm or a technique called a neuro evolution of augmented topologies. It's quite a mouthful, so let's call it's neat, cppn neat. So he speaks about using cppn neat for a lot of different purposes, one of them being using it as an alternative for the general back propagation that people use for deep learning or machine learning. So let's use cppn neat for our abstract, our generation purposes. So the concept revolves around creating few neural networks that we used, like in the previous demo, and randomizing them. What if I disconnect few neurons like on this particular randomized cppn? What if I just shut off one particular input from the output? What if I just, you know, remove one layer completely from the neural network? So what happens then? I used multiple such randomized cppn's, and I crossbreed them, and I mutate each neural network, and I create multiple offspring cppn's. Why do I do that? Because that's what generator artists do. So let's look at what mutation and crossbreeding means in terms of neural nets. Think about high school biology. So remember, we learned about mutations where an organism can have a new trait, and that's a mutation.  I don't know. That would be cool to see. So applying those principles to mutation and crossbreeding, let's think about a randomized cppn on the right, or your left, wherein you have a neural network with three inputs, one, two, and three. And you add, so right now you have a connection, no connection between node three and node five, and the mutation would be adding a connection from node five. Or from a crossbreeding perspective, you have two parents, and you crossbreed the parents such that you randomly choose neurons from each of the network into your offspring network. So that's mutation, and that's crossbreeding. When Kenneth Stanley wrote about cppn neat, he was also referring to a particular application called pickbreeder. Pickbreeder is this website which is used for interactive computation, and I think it's called visualization. So essentially, which lays out 25 abstract images which are generated by randomized cppn. You as a user can choose any two patterns and mutate and crossbreed them to create 25 more patterns. Honestly, pickbreeder is a little bit more extensible. You can choose multiple parents, but in my version of pickbreeder you can just choose two because, yeah, time taken for development. So you can breed cppn's in black and white or in gray scale. I seem to be lost today with my browsers. Yeah. So I created an upper around the whole concept of mutation and crossbreeding as well. So what you see over here is 25 individually randomized cppn's, each creating their own abstract pattern. You can start with simple randomized patterns or you can start in black and white. Let's use color because it's cool. On the right, you can see the topology of the neural network. So if this was a little bit more clear, you would see that these were the activation functions. This I think is soft plus. This is also soft plus. Yeah, this is another simple network with tannage activation and I think both are tannage. So if I use these two, let's use something more interesting in this one. So these crossbreed and mutate to create... Right. And it doesn't work. Finally. So you crossbreed and mutate to create 25 more patterns and you keep mutating and crossbreeding until you create a pattern that you really like, download it, use it as a wallpaper or in my case like print it and frame it. So yeah. So yeah. So that's the idea behind Big Breeder. Okay. Okay. Here we are. The whole idea of Big Breeder actually stems from this book written by Richard Dawkins called The Blind Watchmaker. He basically uses his 26, 256 MB memory machine to create this computer model for biomorphs. Essentially to speak about how random evolution was actually the reason why organisms as complex as human beings were created. So he uses this concept of mutation and crossbreeding to create biomorphs or stick animals. So do read the book if you get a chance. Technology wise, I use TensorFlow.js for this, a specifically TFJS node. I did not have a lot of success using TensorFlow.js in the browser. I could even a single, single around 25 by 25 image took a long time to render and it's lots of number crunching happening behind the scenes of Node.js of course, HTML canvas where I lay the artwork out and PNG streams and D3 for topology. I have the Big Breeder running if you all want to play around with it on AWS, but it's on a measly T2 micro instance so be gentle with it. A few books Dawkins' Blind Watchmaker couldn't recommend it enough and some other books if you are interested in generative art in general. I got started with Matt Pearson's book, I think it's really great. It speaks about using processing as a language for generative art but you can apply those recipes anywhere using any language, using any technology. And Picova's book, this is one of my favorites for mathematical art so if you're like into Julia set or Mandelbrot set it's great recipes in there as well. References, I've written a little bit about it in my blog and yeah, the blog by David Ha and the papers by Kendall Stanley, they're just gems. And so I was at JSConf Asia this year and one thing that I really took away was to build more silly things. This slide is from Monica Dinklescu I hope I'm pronouncing her name right and she talks about, she's part of the Google Magenta team and she talks about how it is important with serious things. So yeah, build mostly things. Thank you. You can reach me on Twitter publishing about Africa. Thank you. Hi. I will next year if you all will have me. Awesome, awesome. Yeah. Going to your first picture demo explain I have five how come the generated images? Generated images? Yes. They do, right? I mean, the reason why I actually really got into it, I started off because, you know, as artists you just kind of want to try new things and the wasp was my first find and it's, of course there is like a graveyard of images on my machine. Those are the successful ones. There are a lot of things that are not as shareable as such but I knew that if I use a square function, I can create biomorphs. So I started experimenting with other parameters. Of course you may not be able to create the wasp or the man using the pig breeder that I created a wrapper around.  Oh no so the wasp and the ones that I shared at the beginning, oh, you mean the slides, right? Yeah, so these were not mutated. These were generated by the single generation. So I configured the network with some parameters and random numbers which helped me generate biomorph like and this was the best amongst the stuff that I generated. Yeah but in theory, like if you read Kenneth Stanley's research, he actually bases the fact of the reason why they push this whole concept of evolution is because you're creating complex organisms from something as simple as a simple neural network. So if you can actually mathematically kind of mathematically modeling evolution, if you may. So not exactly, but yeah, in a way. Kind of. Yes. Any other questions? You train the neural networks? You're going to train them? No. No. So in a way, if you look at CPP and NEET is an alternative approach to backpropagation which is training. So the way backpropagation works is you choose an initial set of weights and then you train the network by by subtracting the weights based on how you're going. Yeah. The way the way training works in the CPP and NEET world is you you don't choose a set of weights, but you choose multiple networks and you find the fittest of them all to eventually choose what will work for you. So in our case, when we look at the big breeder, we are the fitness functions. So we decide what we think is great or good looking. But mathematically, there are ways to actually implement a fitness function. I haven't gotten there yet, but yeah, probably for the JSConf talk. Yeah. The same one. Okay. Hi. Hi. I'm just wondering so you've done that first technique and then you've done the evolutionary technique on top of that. What's next for you? What do you think is the most exciting? So I want to try a couple of things. I want to get into the ground space. I have been meaning to because this requires a lot of computing power. So any bigger image than the one that I showed takes really long time to load and your computer doesn't do anything else other than just crunching numbers. So the first step is to actually build a computer with proper GPU to crunch that and also probably have enough data to run a gun and then try and see if what would hallucinatory images look like? Because cppn in a way is also a generator. So if you're looking at the gun space so guns look at images and try to generate something. So what would a generator with cppn and a differentiator from a normal gun kind of look like? I know it's an unexplored space and it's interesting. But this uses JavaScript and I think that would be the most exciting space I guess. Do you need to buy a hardware for that? So the cheapest way would be to build your own.  It costs around 7K SGD which is Yeah. Or you could, you could. Yeah, definitely. Yeah. Something that Okay, so when I did an analysis it turns out that you pay more for the cloud than you build your own and you have the computer at home than a monthly payment that you're doing for compute. So, yeah. I think, cool. Thank you so much for listening to me.


Demons cannot dwell within the true believer. So how is it that sin can dwell in us? First of all, it's important to remember that sin and demons are two different things. Sin is rebellion against God. A demon is a sentient evil being from the supernatural realm. The Holy Spirit is able to dwell within you even when you make mistakes because of the blood of Jesus. In God's sight, the believer is righteous for. He had made him to be sin for us, who knew no sin, that we might be made the righteousness How can the Holy Spirit dwell with you when you mess up? Simple, God's redeeming you by the power of the blood of Jesus. Demons, on the other hand, are irredeemable. There's nothing in them that can be saved. Thus, they cannot dwell along with the Holy Spirit's indwelling presence. Get in-depth and biblically grounded revelation on the supernatural realm, spiritual warfare, and deliverance. Get Holy Spirit the bondage breaker at bondagebreaker.com.


Okay, we're back inside the CUBE SiliconANGLE.tv's exclusive coverage. I get excited when I have Todd Lipcon in the house here, CUBE alumni, Cloud Air employee. What's your employee number? I think 10 or 11, something like that. You're early, you're sub-20. If you're 10, that's good that you're in the top 10. Glory days now in the hundreds of employees. Welcome back. Thanks. This is a little bit different setup. Normally we do the little table, but nice to be here at this exclusive first ever H-Base conference. So you're all over the email trail. So all the stuff online, people go to a slide share, they're either looking at some of the work you've done or influenced, that's the code you've written. Take us through why, in your opinion, H-Base is so popular right now. It's just a few short years ago, which is part of Hadoop, and you've had a community that's growing, but why right now is H-Base popular? I think there's been a lot of this trend around real-time in the news and the blogosphere and all that recently. And H-Base basically brings this real-time idea on top of Hadoop. So Hadoop is great for this batch analytics, kind of back-end data warehousing kind of workloads. And then when you actually want to put a website off of it, it probably kind of falls apart if you're just using HTFest and MapReduce. The minimum MapReduce job takes about 12 seconds in the newest versions, 24 seconds in older versions, whereas the minimum H-Base query might take like two or three milliseconds. So it's a very different time scale it can work with. So batch is very attractive with HTFest and MapReduce. Don't get me wrong, batch is still attractive for a lot of workloads. Well, 26 seconds, 20 seconds not like days. Right, yeah, I mean, it's not weeks. Compared to a lot of the existing database solutions that's really fast, right? Okay, so let's back to the analytics. So that became popular. Why not other solutions? Why not other databases? I think H-Base is the one that, for one, scales really well. It actually lives up to the claims of scalability. People are running it at 800 nodes, 1,000 nodes scale, petabyte scale. And also integrates best with Hadoop because it's part of the whole Hadoop ecosystem. It's like our security infrastructure is the same security infrastructure. All our storage is on H-Base, which everyone trusts to store their data really efficiently. And a lot of it's there. It's a nice operating model from an integration standpoint. Yeah, everything kind of fits together, it works together, it's meant to work together. Whereas certainly there are a lot of other databases in the new SQL space that say, okay, we can integrate with Hadoop. But they're more like an export import kind of integrate with. They're not like a built together kind of integrate with. Okay, so now I want to ask you a personal question for the folks that know I've interviewed Todd before. And we always talk about computer science programs and since the computer science degree has been in the news lately, the Yahoo CEO claims he got a computer science degree. I actually have a computer science degree, he has one as well. But you went to Brown, right? So what are some of the schools that you see up there?  Obviously there's the known names, but just give us the rundown in your mind's eye and your experience of the schools that are graduating the savviest computer scientist relative to this world of big data, which is Hadoop, operating systems, et cetera. Yeah. So I think it's a lot of the traditional system schools. I know Brown's number one, so we'll stay back. Brown, of course, is number one. We got, I think, eight people from Brown right now at Cladera, out of about 200 or something employees. And then other top ones that stand out are probably Stanford, Berkeley, MIT is pretty good. SUNY, Stony Brook actually is a very strong databases program. There's no one at Cladera from there currently, but there's some other pretty interesting startups out of SUNY, Stony Brook. I mentioned CMU is another top school in the database systems kind of area. Anything outside of the North, that's interesting. So SUNY, that's New York, right? Yeah, that's the University of New York. Any other programs, international you've seen? There's a couple of international schools like EPFL, it's a pretty good systems program. It's in, I wanna say, France or Switzerland. They're French speaking, but I can't remember where it crossed the border there. Someone was telling me Canada is a massive, it was, is it Toronto? Oh yeah, Waterloo is a good CS program. Waterloo, I heard Waterloo. All right, there you have it. Those are the top schools and soon to be a blog post on SiliconANGLE, I have my list. Pretty close to all those. The second, next question is, what do you think about this event? What are you hearing in the hallways? That's sold out, so it's obviously this demand for the HBASE conference. Not a surprise, but what are you hearing in the hallways? I mean, we talked earlier about the keynote from Facebook, very impressive. Sharing the data, want to articulate, fast-paced, very high quality. What are the things that you've seen that are like, wow, this is cool? I've definitely spent a lot of the morning talking with the Facebook guys and a lot of the community members who I mostly interact with online, but it's rare to get actual face-to-face with them. So like Michael Stack from AssembleUpon, Karthik who gave the keynote and talked to him for a while. In terms of other attendees, there's definitely a lot of people who have a pretty surprisingly high level of, they're already pretty advanced HBASE users. I was surprised by that. I tend to think of a lot of people being very new and excited by HBASE, but not really deploying yet. There's some pretty big deployers like multi-hundred-node clusters I hadn't heard about before, and that's just pretty cool. Cool, how about application support within the HBASE architecture? Have you seen any developments around there? Yeah, so there are a couple of other startups that have been founded recently, so Webe Data, I don't know if you've had anybody from there on that. Chris Doppelzahn, Aaron Kimballer at Hadoop World. Okay, yeah, so that's one example of a startup that's basically building an application platform on top of HBASE to make it easier to build normal apps on the infrastructure. We had VDP Finder on earlier. That's a stealth startup at Appelo Alto. I found that one on the front of VDP Finder. Who else have you seen? So Continuity is another example. We had Jonathan Gray over there. I don't know if you've spoken about that. So there are a few of these startups that are kind of starting to emerge. I imagine there are a few more that I haven't heard about yet that'll come up in the next six months or a year. Any other concepts and trends, topics and interests that are popping out that are surprised to you here that, or not surprised, is that obviously, what are the top trending items? If we were Twitter and this was like, we were harvesting these verbal tweets in the hallways, what would be the trending items? I think Eric Sammer, I don't know if he's been on here before, he had a pretty great tweet earlier. Or I said, the atmosphere is more like a hackathon than a conference in a lot of ways. A lot of people, pretty high-engineered population, pretty low, business person population. So I've heard a lot of talks about what exactly is the semantics of the consistency when you do both put all these very detailed things, which is rare for a tech conference. This is all developers. And alpha, I would say, I'd categorize it as alpha. Yeah, a lot of the same as Cisco started. But there's people here from Europe and East Coast and some of the U.S. It's the leaders you think about, Facebook, Apple, Google, Netflix, Amazon, that's all fidelity investments. We have media companies here, so it's getting diverse. Yeah. It's pharma. Yeah, and Gap is giving a presentation, or gave a presentation earlier today, they're storing their apparel catalog in each base, which is not what you think of as like a leading edge technology, but it's pretty cool. Well, I think it's really fantastic how far you guys have come. It's been great to be part of the CloudEra ecosystem at least about being part of your office for a year and a half and watching you guys grow. So congratulations on all your work with HBase. What are you working on now? You mentioned HDFS. Could you share with the folks kind of your current passion and focus? Yeah, so currently I've been working on the HDFS High Availability Project probably for the last six months or so in earnest. That's basically eliminating the single point of failure which used to be in HDFS. So there's a single masternode and now we have a failover for it. So if that should crash or the network gets unplugged or something, it can automatically failover. Another node which takes over seamlessly. This is something that we've heard in the DevOps community where DevOps is much more of a conversation now with CloudEra's movement into more production focused environments. And the thing that I'm hearing not only here, but also consistent with what's going on in the past few months outside of this ecosystem is the production environment numbers are up, right? So overall, that eBay was on talking about it. People care more about metrics now and more people care more about monitoring and uptime. It's uptime. I mean, exactly, yeah. Shit, you can't be down. Some businesses just can't be down. A lot of that's an artifact of the real time versus batch, right? If you're in a batch environment and your dog runs once an hour, you can take a one hour maintenance window, no problem. But if you're actually, all of your website visitors are hitting HDFS and you can't really do a, even a 10 minute window is expensive. Cartridge had a good discussion about how he's handling his replication. I think he said what? Back out in recovery, within the cluster, within the data center and then outside the data center. They want to do the fastest backup first and then if the whole data center explodes then they can go to this level. I thought that was interesting and I was just coming through Michelle Bailey who's here from BDP Finder. The next data domain is out there here in this marketplace because back up in recovery, data protection. Yeah, that's come up a lot recently. It's going to be a hot issue. But I think high availability is something that we've heard like, hey, the DevOps conversation is really interesting around cloud because DevOps, you think of Node.js, you think of Hadoop, developers doing operational things. You talk to some purists, they're ops devs where it's pure ops, no dev, because they have such strict and high availability requirements that they won't even let any open source into these projects. So we're seeing those barriers come down on these verticals. So I think that's a really important area. That's one thing Cladera has been focusing on is like DevOps is great and it's pretty exciting as a developer to be part of it. We found a lot of companies out there that don't have DevOps culture. It's slow to move into that kind of thing. People are still operators or developers. In the classic enterprise out there, there's a consumerization of IT going on. But what's happening there is they're definitely the couple. Ops and developers, they're shooting arrows at each other every day. That's what we see in our customer base in a lot of cases. We want to build tooling that makes it easier for a traditional ops person to actually run Hadoop. And I have to feel like, oh crap, we have to find these DevOps unicorns that maybe only exist in San Francisco. How do we hire them? It's really hard to find them. Exactly. Well, there's a whole personnel conversation, the training and requirements to get in. With that, let me ask you guys about the community. So the community of Hadoop has grown. There's a lot of inbound migration of new personnel, new people, new personalities. And with that is always more politics and more eccentric behavior. Share with the folks out there, what's the state of the current community? In your mind's eye, have you seen it right now? Describe to them what's it like and try to compare what it was like, what's changed about it, good, bad, or whatever. Yeah, actually right now it's pretty healthy, I'd say. The Hadoop community in particular is actually gearing up for Hadoop 2.0 release. We're in the process of voting on a beta release of that this week. So figures crossed, if no one vetoes the release for any kind of like bugs or anything, we should have a release this week. All that feels pretty healthy. Our release momentum picked up a lot. We're collaborating pretty well on the open source, even though maybe competitors who are actually contributing to the same open source project. That all feels pretty good. And in terms of the overall feel, the growth is definitely evident. There are many more companies that are now contributing code than there used to be. Even these big companies like IBM are starting to put patches in. It's cool to see all these new people showing up and bringing code to the project. Well, I was just at EMC World. We're actually broadcasting live this week, so we're here and there. And EMCs love their messaging. Cloud meets big data. And they're increasing the demand and educating the enterprises around big data. Do you see that evolution around consumerization IT affecting cloud deployments around Hadoop and the architecture on the operational side? Because the big debate is private cloud, public cloud, now this new hybrid cloud, which, you know, cloud is just a code word for outsourcing, right? So... I don't know what cloud is even a code word for. You know, it's just, okay, I'm going to outsource something, right? But there's requirements involved, right? So what are the challenges from your perspective on getting Hadoop to the point where it's fully extended out into the enterprise? And right now, it's pretty much the big vendors are saying Hadoop is it, and they're already connecting this work. So we're seeing that. So what is those challenges for you to make it over? So I think the off-site cloud thing hasn't really caught on with Hadoop yet because data has so much inertia. So it's great to say I can run 1,000 machines in EC2 pretty quickly. But if all my data is in a database in my data center, then the network bandwidth between my data center and EC2 is just not fast enough to make that a reasonable proposition.   So you end up having to keep your data elsewhere, spin them up, copy a bunch of data, copy it back, spin it down. Yeah, so you get charged. You get charged all the copies that's pretty inefficient. You end up spending half your time copying instead of processing. So we're finding the private cloud deployments like within a data center much more popular among our customers. Certainly some of the newer startups are kind of all cloud and they find it more useful. Yeah, I think that's a good observation. Thank you for sharing that. The other thing I wanted to ask you is the vendors, when you guys started, you had no competition, right? And then the big hubbub was Hortonworks, right? And I was there at that event. It was kind of fun to watch Yahoo spin that out. That's on me last year. But a lot is mellowed out because everyone else came in. HP bought Vertica. You got EMC with Green Plum. You got now SAP with HANA. You have all these competing architectures competing for a big data position, technically. So what have you been seeing? I've noticed that they've been kind of yielding to the fact that- Yeah, I mean, it's like IBM has an IT user, but IBM is also doing Hadoop, right? And EMC has Green Plum, but they're also doing Hadoop. So all these companies- Yeah, they have the- EMC back down from MapR. Right, so they've yielded to Hadoop. Gene's out of the bottle. It's unstoppable. It's trained. It's left the station. Hadoop has kind of been accepted as the platform. And people with the other platforms, they're more specialized. Maybe they can do data warehousing workloads, but then we can for specific workloads. I would, as a general purpose, be data thing. I think Hadoop is kind of it. Yeah, so what do you think? What's their technical strategy in your mind? The connectors is what? To connect with Hadoop? Is that more of just a embrace and extend philosophy? Or is it legitimate work that they're doing? I'm generalizing now IVMHP and the big guys. I think currently those other data warehouses are pretty complimentary in that Hive is great. It can do a lot of workloads. A lot of workloads faster if they're really big workloads. But it doesn't really do advanced analytics as well. It doesn't do windowing functions and other kind of advanced SQL. So those other data warehouses have a 30 year lead on Hive and some of these aspects. So I don't think they're going away next week. So I love talking to you because it's wealth of knowledge. Let's talk about the network problems. So in big data, network latency seems to be a bottleneck with virtualization and compute. I mean, whether you're doing some work in the cloud or you've got a data center with a lot of machines and multiple cores, it's a pretty heavy duty. Network seems to be the band with late problem. Yeah, so in bigger clusters network becomes an issue. On smaller, like single rack clusters, we found it's usually not. Because you can get a single top of rack switch that does 10 gigabit ethernet, pretty cheap. There are companies like Arista that have come up in the last, I guess, five or 10 years that make 10 gigabit ethernet pretty inexpensive. It gets more expensive when you start getting to like a 400-nit cluster and all of a sudden you've got 10 racks worth of machines all trying to go to some giant core switch and that gets pretty pricey. So it's these kind of bigger rollouts that are more expensive in the network become a bottleneck. How did you just handle that? Wait and see. Like some work there is compression on the wire, for example, so the map produced shuffle when it transfers data between machines will compress it. Just to try to reduce the total amount of data going through those core switches. H2F assets and replication policies are all built around this idea of trying to minimize the total amount of data going through the core switch. So we definitely built that into the software, this sort of expectation that cross-rack transfer is expensive and kind of always will be. Because whenever we get that cheap core switch for 10 gig, all of a sudden we're going to have something faster on the single racks. It's always that ratio. Yeah, and there's all this type of software networking in the CIRA, doing some work in OpenStack, and so it's like all another generation of networking comes involving. I think a lot of that there has to do more with reconfigurability and elasticity rather than just strict performance. So like I said before, it's hard to move a Hadoop cluster around your data center because it's got all this data sitting on those machines and you're not going to move them somewhere else to be too expensive, right? What's the conversation like around application developers? Because again, the conference really starts teasing out this notion that the application integration into HBase is critical. To get those real-times. And we heard Mike Olson, the CEO of Talk at Hadoop World, tsunami of applications and at least got $100 million of funds. Does the data, the data is like a very important component of the application. It used to be app and infrastructure and then that got decoupled. That worked great. Now you've got data bound into the application. You've got SSD memory. So you've got good memory now, capability is a large number about a memory. What is the data? It's bundled into the app, infrastructure is a decoupled, all of the above. So it's like an architecture re-engineering. The lines are definitely blurring. So there's this new feature in HBase called co-processors, which allow you to take some processing that used to be in your application and actually push it down to the database. So you get this sort of blurry line where well is HBase a database or an application server or maybe kind of both, right? And that hasn't really been resolved yet. The HBase committers have had these discussions meet up before and we haven't really decided like exactly where it is. I think for now we're keeping it as the game. It's almost a religious war at some level, but. Yeah, where do you want to focus the project, focus our efforts? Yeah, it's like an operating system conversation. What do you want the function to be? But it's almost as if if you can keep it so that it's flexible, you can provision that wherever you want. Right, so that was our idea with co-processors. Basically it's a plug-in point and we'll kind of put that plug-in point out there and see what people do with it. It's pretty new so we haven't seen them much yet, but. Yeah, I think the use cases will drive that. So I think the best decision is no decision. That's the point. That's what opens ours is all about, right? You get people sourced and if they contribute patches then all right, now it's done something different. Question about DevOps, okay. So DevOps is emerging as a bona fide position in this new era of developer meets, operating system developer meets operations guy. I was talking to Jonathan Gray, he's like, hey, you know, at my startup before Facebook, I did everything. I ran the ops, I did the development. Yeah. That seems to be the profile of what I call new profile developers. Someone who has chops in code, almost an operating systems kind of guy and also can run basic operations. Yeah. What's your view on that? I mean, do you see that as just a evolution just because this is a new industry? Do you think it's an actual job title? And what skill sets do you look for in your peers and say, you know, that guy's a DevOps guy? Interesting, I think it's actually, I always look for the people to try to hire in general. So I think the more you understand the stacks below you and kind of above you, the better you can do at whatever stack you're supposed to be working on. So I also previously at Eclator worked out a startup and did everything too. So I definitely identified that. And I was before the firm DevOps was coined. So whether it's like a new...  Yeah, so like in terms of hiring from Brown in particular, I know the courses there and the people I look for, the people who did really well in the operating systems course. Because that's the one that makes you actually, you write your code, then you also have to kind of understand what's below the code. Yeah, yeah. And if you aren't comfortable jumping layers like that, then it becomes difficult later in your career maybe you're not working on a kernel anymore, trying to understand operating, you're trying to understand hardware, but you're working on Java and you want to understand what does the SDK do. So about once a week, I find myself looking at the Sun Java source code that's freely available. And I kind of want to know the layer below me, right? And similarly working on HBase, kind of look at HDFS or as an application developer, you might want to look at HBase. I think that's the kind of key differentiator. It's not exactly DevOps, but it's a familiarity with the layers kind of adjacent to you in the stack. Okay, Todd, thanks for joining us inside theCUBE. We'll talk later over a cocktail hour. Todd Lucan, the senior engineer. I don't know, your official title's gonna say senior engineer. We're all just software engineers. Software engineers. We don't have the titles though. Employee number 10 at Cloud Air. He's been here from the beginning. One of the smartest guys I know. Young Gun at Cloud Air. We've had him on theCUBE multiple times from Brown University. Big plug for you there and congratulations on your success. Any Brown students out there? We're hiring, email me. Yeah, if you're Brown, you got to go to Cloud Air at nowhere else. Actually, Stanford and MIT, you can go everywhere else. We'll take all of you. Okay, we'll be right back after our next guest. Thanks a lot.


SB Supply is an e-commerce company which is engaged in the online sales of accessories and gadgets. Anything that has to do with media, basically what you need nowadays. SB Supply is a young company, we're working a lot of international students. Because of this international environment, you have a lot of different things that you do and I like the challenge so that's really fun. It's also a nice atmosphere at the office and you learn everything you want to know for an entrepreneur or to start your own business. So we're still studying at university but in my opinion we get the latest knowledge. Senna learns you a lot about the business but also a lot about the market. I analyze data so I analyze what we sell and what we can sell and I track these keywords and optimize them. It's still a personalized experience and not something where you don't see the actual human behind it. We can make a difference if we take this data together. Many people think that you cannot compete with large companies such as bold.com in the Netherlands. However I am convinced that you can cooperate with these companies and also sell their websites. And now we're going to expand to Germany, the UK, Spain and Italy. Senna is really really passionate about what he's doing and it's great to work with someone like this. He always tries to find the best solutions and in a very short time. He wants to achieve high goals. And I see entrepreneurship as a way to create things and to get freedom and just to develop your own idea.


A little brown bat story, written by Melissa Kim, illustrated by Jada Fitch. One day in late September, a little brown bat was sleeping. He was cozy in his daytime roost. As the sun set, the bat woke up and looked around. His day was just beginning. The bat went out hunting. In one hour, he ate almost a thousand mosquitoes. Some mosquitoes can carry diseases that make people sick. Some insects can harm crops. The bat did not know it, but he was doing an important job. He took a rest, then went out and ate some more. He rested with other bats to stay warm. Eat, rest, repeat. It was cold that night. The bat flew far away to a deep cave. He joined other bats going into the cave. It was time to hibernate, rest, and sleep for the winter. The bat wanted to huddle with the other bats to get warm. But there were not as many bats in the cave as usual. Some of the bats woke up. Something was making them sick. They flew out of the cave. The little brown bat did not see them again. The bat slept through the winter. While he slept, scientists worked. They wanted to find out what was making the other bats sick. The little brown bat woke up in the spring. He flew back to his summer home. Now, it is time for him to do his job.  People are at work, too. They study bats and caves. They help make their neighborhoods safe for bats.


So I'll start with what I have done for this week and I'll share my screen to see what I'm supposed to share. So I have, like, I have Jenkins running right now and these were jobs which, like, I created just like test jobs. So right now what's, you know, for the things what it's doing, this is basically like Jenkins as a source. So we figured that we'd start working with Jenkins as a source first and then once we have a bit of, like, more clear idea of getting events and stuff we can move into configuring Jenkins as a thing. So this is global configuration and inside of it, I'm just going to do a little thing. So this is going to basically add an endpoint or a sync which is configured to receive cloud events and I also have a server running which is just a simple HTTP server and this particular one does not, like, it's not parsing cloud events data but we are sending over cloud events data and it's like able to look into headers. I also have a server which is able to, like, look through cloud events data and then sort of send back a response in that format but it's not running right now. And the events, the last we talked about starting off with events which are more sort of used so in terms of like a job, we'd say like a job has started, a job has completed and if the next step of the job, like the last, what's it called, the artifact is built, we can choose what events we want and maybe all of these events and just maybe going to do job start it and also add another endpoint. So I'm going to apply it back here and are you guys able to see my VS code or do you want to just see my browser? This is the browser at this point. What about now? Stay the browser. Okay. I will server. Yeah. So I also tried making sure that it works in a distributed system so my other region, it's offline. I started once like we display this simple with just one simple like executor. So let me just like start a job. So this is what's being sent inside the cloud events data and this is like how a cloud event is configured right now is it's using a binary format instead of a structured format and what that basically means is that it's setting headers separately and then it's setting the body separately. So this is, you know, this is like setting up, setting up like we want to send a JSON object which is like in a byte, like a byte array which is being configured right here. So this is like the cloud event which we want to send and going back to job listener or stage. So we're so this is basically like an enum and it has the three stages, which is like a job is started, a job is completed or a job is finalized. And depending on whatever the current configuration of the job is, it's just going to sort of put in information into the JSON document and then convert it and then put it inside of the cloud event data. And as I said, we are right now using a binary format, although I think we might want to move to using like just using structured instead, instead of binary, because it's I was like reading and it just said that it's more, you know, if you want to multi route, it's it's it's better to keep all the headers and the data together as the payload. So this is sort of the payload that we're going to get and since this is configured for started, that's why it's getting all the started, I can also go and add another event and I want to get an event, every job field job start is all of events basically. So that's, that's Jenkins out, I'll move into the code and how it's figured, and it uses it uses listener implementation and right now, you know, there's the abstraction that we were talking about. I don't think I was completely able to understand where exactly we might need that sort of implementation, because, you know, since all of these are, like, if this is, for example, a run listener, which is basically just notifying for run and then there's item listener or job listener, you know, so what, so this is essentially like, we are going to need all of these listeners added as extensions and then have their whatever methods that they have, we want them overridden or maybe not whatever are. Circumstance looks like, but when we talk about abstraction, the thing that I'm sort of thinking is, I don't feel that there's anything which is really common between these implementations. So we can like, pull maybe some constant fields out, for example, job name or, and then put that into, like, like, like an interface instead of like an abstract class. And then just first figure out what's common, what sort of extra methods can we add? I was looking into the GitHub Auto status job and there was the, like, do you want to send test reports or do you, or do you want to send error reports for other jobs? And I think that's a really good method that we can have implemented by other listeners. So, yeah, so right now I was just using a job listener here and it's just extending the listener, which is the enumerations, which we were looking at earlier, and then there's a build model and job model, which just gets information, you know, URL, the other current ID and stuff.  And then this is the endpoint, which is some sort of some configuration that's needed for the jelly file and also some configuration that's needed to send. And I need to refracted this out, like the send to some better class or something. I don't think that this is the right place to put it. And the stage is the enumerations. One cloud for now. It's great, Trissie. That's really interesting. Jeff or Rohit, would you like to give you back? I mean, the only thing that I saw we went through it briefly was just stuff like a hard-coded URL to HTTP host, which I assume is just kind of placeholder code. In the, for the sync? Yeah. Oh, that was. Oh, I'm sorry. No, no, go ahead. No, no, no, the URL for the sync. No, that's, I think that's a different place. That's just for like a cloud event to send the mess. So this is not the URL that it's getting out from the jelly file. The sync URL is coming from the jelly, which is then it's using to send the event to Let me find where that I think it was the last piece of code you were showing maybe Trissie, is this up on your GitHub, actually? For us, but also for the other mentors, they might want to have a readover. Yeah, yeah, yeah. Yeah, so I, it has a lot of like, Debuggers in right now, but I'm like cleaning the code and I'll put it into GitHub today, right, you know, right after the meeting is done, and I clean up whatever is required, I'll put it on. Okay, awesome. Thank you. Is there, there's something that you can submit a pull request against or is this the initial implementation? Yeah. Okay. Because if you had a pull request, then it's easier to comment on specific things, I think. Yeah. Yeah, so with how he actually created a cloud events plugin. A few months back, so I'll just send a full request to it. Perfect. I actually think I, let me just go back and see what we're looking at here. Like looking into this like end point file, I like, like right now I real, I'm realizing that I hard coded it there in the end point class, which basically is configured for, you know, it has to sync your own it has the event, whichever that event is configured to be received So if I go into the jelly file. So, so all of these events, and whatever, you know, a user is going to choose. It's going to be represented here and then if I were to go back into the numerations, you know, this is the configuration needs to go here for sync URL and this is the URL that I'm sending it to and I'm not using it here. No, there's the one line line 84 was the one that I kind of I was wondering about. Yeah. But again, it's a, if I can look at the pull request, I can just make that comment and it's probably a more efficient way. Yeah, yeah, that makes a lot of sense and I'm totally going to do that is just I don't. So I did another question since there's other people I noticed for logging you use system system dot out. I'm not sure what whether that will create like the same type of Java logs that you can you can look at in Jenkins but I'm not super familiar with Java so it might be fine. I think you should ship to other logging libraries. Yeah, yeah, I think you guys are you guys are correct about that. And yeah, I think I just use that use that there because I was like, I just need to print and see but yes that this is the thing that I need to log and make sure that it it's into debug mode. So whenever person is doing that would be the correct way. Yeah, but I mean this this looks this looks great it's a good start and and I mean the code is like easy to follow and understandable. Just like this brief intro so I'm, I'm pretty, pretty happy with your progress. Yeah, amazing work. Thank you guys. Learning to write Jenkins plugins is not trivial. Especially jelly. Jelly falls, or quite a hard to figure out how they work. Finally got it working it was so fun. Did have few questions actually. So the, the thing about abstraction or just maybe just having interfaces that we were talking about earlier, and looking into. So I was looking into how you'd implemented in GitHub auto status plugin. And, you know, since it has those where where those like listener implementations need to go to. I think that that was a very great place to use this abstraction for for common you know methods or just maybe variables, but do you think, or I'm just, I think that over here. I don't know if abstraction is necessarily going to help, but I do feel if we are able to think of any situations or methods where we are absolutely going, which are going to be common between listener implementations because, like for example a run list like a run listener has an on started on finished whatever queue listeners completely different. And all of them share different information so I don't like there's not a lot of common information besides, you know the name of the job, the description and stuff. So, I like I want to hear your guys idea on how or where can we leverage just maybe using interfaces and like putting common methods, or just like, if there are variables which other listeners. that can improve functionality and other different classes can use. So I mean I think I kind of trust your instincts it sounds like you've spent a lot of time thinking thinking it through. I think my guidance was just around. Let's look for you know people write write a new Jenkins plug and they tend to go find something that similar and copy it. And then so they create a copy and then that you know it gets out of date with with the thing that they copied from things have to be made into two places. And I just feel like this is the sort of thing that that that's common like I could see other plugins maybe wanting to be able to send events. So, so I don't have anything specific but but I just wanted to kind of put it on your radar to be to be thinking about those things because plugins can extend other plugins so it's possible that some plugin may want to want to call something in your plug in it might be that that it's not and that I just wanted you to be to be thinking about it I don't have anything specific but when when I start looking at the code if anything pops up we can. I'll bring it up and we can talk about it but yeah if you don't see anything common don't don't create abstractions. Just just because because of that that comment and I think like like properties are less interesting on on an interface I mean it's to me it's kind of more about behaviors and so if the only thing you see is like common properties that to me that maybe seems a little bit less interesting. So but but I mean when you're talking about it I just got the sense that you've spent a lot you spent a lot of time thinking about it and your conclusion made sense to me. Yeah I think I was just like still you know I have sort of figures on my notepad just trying to understand where you know is that commonality that we can create and leverage abstraction because it is a really great idea to you know like have that implemented through an abstraction and have different listener implementations like leverage that and but I don't know if I necessarily am able to understand it yet, but I'm still working on it and all on extending other plugins though it was something that I was trying a bit earlier again as I said you know if I this if I am to extend the plugin that would mean that I'm delegating the task of having that like that event payload and being sent over to my or like to this cloud event plugin. So, so it's it's sort of tricky, because you know, for if you consider GitHub auto status plugin for example, it has those listener implementations and that's when it gets data right. So if there's a list of patient for run listener and run listener on started, which already sort of like a method that it's overriding. So, as that as the run has started, and I'm like, I'm again extending that plugin and as a running so started I want to get a lot of status plugin to send my plug in that event data and I will send it to the sink, but do you like do you kind of see the it well that contrast was a sort of different overridden on started methods because your plugin has an on start which needs to be implemented for to send me the event details. And it's just a bit it's just a bit tricky to get around, because also like you know whenever a job is started I would want to run the on start for my plug in and then I'd want to run the auto start or like the on started for the auto status plugin to send the information over. But I think it's if that makes sense I'm not I'm not really sure if I'm making sense right now. But, but okay I have what I'll do is sort of create a diagram of what I'm trying to say, and then I'll send it over to you as well. A bit more sense. But I just think that it will over complicate this process, because what if what we only need to do is you know get implementation as soon as the run is started, or run is finalized and send it over. And which I think is it's happening right now we just, you know for job started finished and also we looked at finalized or completed. I'm sorry go ahead. I was going to say let's let's just table that that discussion I don't think you need to create a diagram. I mean, what a good good skill to have in engineering I've learned over the years is knowing when to ignore advice from people.  I mean my comment was more, you know spend some time thinking about this and then you've done that and I mean I can see that you know you're capable and you're smart and I trust you so I don't, I don't want you to be spending time worrying about convincing me. I'm good. It was a good. It was like a good place for me to think because I was just like, that can also be a way we can do it you know, like if I just had it handed over to me wouldn't have been as fun I think, even if it were just like jelly files as I was talking about earlier. It would have been fun if you know the jelly file was just like there. So it was kind of fun to figure it out. And I was also trying to extend like plugins in this particular plugin and seeing if or how does particularly the ones with listener implementation because that's the sort of the thing that we want to do. Also, you know the listener implementation on started there basically like non returning methods. So those methods don't return anything that's what we want to do is like as something is a started want to return something to another thing to you know get basically an event out of it. Yeah, I think we might have to go with a single listener implementation and then maybe like you know different listeners of course, which, which is also like sort of my another question about listeners itself right now. Well, what I had was a run listener and it will be simply. I think very easily extended to an item listener and Q listener, but we have and Karen and I were talking last time about having like computer listener, which is if a note is online or also does also work on scalable system so the note that I was talking about earlier like that the agent note which was offline. I tested and made sure that it works on, you know if there's like a distributed system and so so it was it was working. But in terms of sending events like this is this job was completed by so and so agent, or this job, or like this agent is offline so trigger this event. So you know that's also do we do we want to also have events about computers or just executors and agents, or just stay to the generic like jobs and cues and files and folders maybe changes. Yes. I mean, the short answer is that I think that events about agents will be useful. In fact, we were having problems with some of our agents at one point and I created a plugin just to write the name of the agent just so we could track because we suspected that some of them weren't working but so the long answer is, does it fit within within your project and so that's, that's what we have to decide. If you have time for it I think it would be interesting then does anybody else have any thoughts. So if I heard correctly this the question was about sending events regarding agents like agent creation agent. So, yeah, I think that's a good idea, like when a new agent is created or something but I don't know if that so are you thinking of creating that as an event that is created in the beginning, as in like one of the first events that are created by the plugin. Like for the plugin. Wait, I don't, I'm not able to follow through, like what do you mean the first? I mean, no. So, I don't, I don't think that would even be a problem really, but I was just thinking what are the first kind of events like you're going to tackle on creating.  No, no, like the first as in the first that you would start working on. So the first that I sort of worked on, like I was, we were looking at here I think you, you, you were not able to join at that moment was just like job I can put briefly share my screen again. So just so we know how like the UI and stuff looks like and how. Yeah, so yeah, I was in a little bit of a dilemma that's why I joined. But hey, Jeff, it's nice to see a happy real. Yeah, I'm sorry I missed the first couple I slept through one of them the other one I just space but I this will just become part of my routine every Monday morning and I won't I won't forget plus I saw the reminder today. I'm just like very happy that all of us can come together and work on this stuff. I'm sorry for like a little being a little late today, but hope I can also come on time and we can kind of sync on Monday. Are we're where are you located. Are you in India or are you elsewhere. Oh, no, I'm in India. Okay, so it's evening nighttime for you evening. Yep. Okay. So what is the global configuration for cloud events. And this was another thing that I was thinking. So you know how we were looking at, like, statistics, gather a plugin and how it had like things, and then it had like things for jobs things for QL things, but I mean things for Q things for bill steps. And that having it this way it makes it more modular because you know you just want to configure like a sync which receives only job started events so the thing does not will not have to do a lot of configuration. It's also one of that it's making it loosely coupled and not like super attached on single sync perceiving all of these events. When, if we are to add more events in terms of going on to other listeners like you and computer listener, I think the list will either just have to grow, or maybe we can say something like this is a bill. Like, over here we can say something like build and point and then particular ones. Anyway. Um, so this can, can you select. Do you just can you just select one one event at a time there or Um, so I can add more but here is just like one. Yeah, can you can you make this event into events and kind of like it could be where multiple events. Oh, okay. So you can, you can, you can basically configure multiple syncs is it. Yeah, yeah, I basically can send it to the end. So maybe instead of like, it's this endpoint doesn't work. So it's probably going to give me whatever it is this endpoint is not configured but this the first one is, and then I can learn some more to go somewhere else. And it doesn't matter. I don't know what spending time on this. I'm not completed. I'm done. I were to say, but at least and if I like, build it, I'm going to. So I did have a, I did have a thought about this, too. I want to know what other people think again, I mean, I don't want to muddy the waters too much but if there ends up being a lot of settings, you can actually create your own setting applet for the plugin. So rather than having it in Jenkins system. I wonder if at some point that would be useful. What would be that? What would that be again? Yeah, you can have your own settings app and I mean like with without I didn't know this but with the auto status plugin right so I just add it to the the config that has like lots and lots of settings and it's just polluted you can have like your separate settings applet. And I don't think it has to be jelly either. I think you I think, yeah, the auto status one had groovy files. That was very interesting. It's groovy is not. Yeah. Probably we could use configuration as code. I think that's a good idea. What is it? Does it use JavaScript for its UI or No, like you have to pass YAML. Oh, like you just kind of say cloud events, and then you say, what is the event sync and then probably the events you will be looking for maybe something like that. I see that that's that's a good point, but that's not necessarily UI related right that that's just we would want this plugin to support configuration as code. That's that's a very, very good point. Any modern new jug Jenkins plugins should Can you send them the chat while you were talking about also completely gave up the idea of using stapler because as I said last time it's like The binding that we like the binding with URL like object URL binding and I think that we're going to need that here. It's more. Yeah, we might need it for when we're configuring a sync. Since, you know, we need a URL to to be configured to see how events, but not not for source. I've sent them in a slack chat. Oh, thank you. So the prototype you've made is it just like a journey prototype or does it work with the sync. It works with the sync. Oh, that's nice. We're going to write some tests for it. Um, yeah, I was thinking about I have a separate a clone sort of where I was like testing with some genius, but I'm not a very adept one yet. But I think I will also get to that I was just thinking about that. I'll also get I just realized while asking that question that we are so early into this phase, like we haven't even started coding phase and already reached out. This is what this is like some good work though. I was impressed with the events just like one thing I was thinking like every time you've got to send a different event like do I have to configure the same sync on three different endpoints? Like if I want to say send the same like I want to send events to the same thing. At that point I would have to create three of the endpoints. Like you're saying that when I sort of send like job finalized job started job completed to the same endpoint. Yeah, yeah, yeah. Yeah, so there's like an all events option on there. So just in just then. So do you want me to share my screen again? I don't know. Oh, it's right here. I don't share this. So sorry for coming in late and asking so many questions. No, no, these are good. No, we need questions that makes the creative. I don't know on juices flowing in the right. Oh, anyway, I still have the most trouble just sharing my screen out of everything. This is such a big trouble. Anyway, so, yeah, but I think, yeah, even if it's all events, it'll send all of these events. So I think it still would be nice to separate like job events from Q events and Q events from a build step events. That would actually be nice like something like all job or all Q but I was thinking was you could have like a selector like you could add events like you are adding the configurations right if you could do something like instead of event, it will be like events and you can say something like under that like a button which says add events and you know you could add events from there like maybe this is this will be like two months jelly work. Like there would be a repeatable for events. So right now you are very repeatable for the entire configuration. Yeah, I think there is a repeatable for the event only like itself. The event or no event not event for the endpoints. So end point is basically like a Java class and I have a repeatable for the Java class and event like the end point has the sync URL and the event where it meets like where a particular whatever whatever event we want to be sent to the you. So that's what I have right now but you're suggesting that add a repeatable for the event. Because that it will be easier to select for like one sync if you want like a little kind of events to be sent. So I was thinking that sense. Yeah, I think that the reason I kept it like this or like the reason like I feel like this might be more modular in the sense that a particular sync is only receiving the particular event that it's configured for and I think that's if we if you're looking at EDA just patterns in general you know you you'd think that for example if you're like looking at Terraform or just maybe something like infrastructure has scored and you know as it receives a single event of a type node has failed. So that's one event so it knows whenever it receives that event it does not have to parse through a lot of information it knows that this is the event that it's configured to receive and then it will you know spin up another like agent or cluster or whatever it's configured to and similarly for if they're like the cloud bridges and in like Azure or AWS or something like that which are configured to receive one single type of an event and whenever that goes into that particular event or like whenever that sync receives that event it knows what event it's receiving and like with the state gather plugin I think it was like the UI with school but it was sending all of the events together and for everything that's happening inside a queue gets sent to a but again it was for a different purpose the purpose here is different than the purpose it was there but I think yeah we can we can also try how that might work in UI and also in inside code I think that makes sense I think for now we'll keep it this way and yeah it does make sense like to have an endpoint which might be configured to receive certain kind of events like in this case you've given like job created or like job related events so if that endpoint is configured for that then it makes sense to kind of make it you know that it only receives those events and we kind of don't load it with different events but I think we'll get to more in due time once the interoperability stuff kind of goes ahead in the event sync and you know we start seeing what kind of prototypes are being made I think slowly we will have to start aligning based on those ideas like how how we can make this work with Tecton or something like that but that's like that's like way in the future I feel like maybe three to three months ahead but yeah this looks great I think I think we are already this looks great like we've already crossed like covered so much around here That's nice We plan on working the coding phase Okay also with how I wanted to let you know a bit earlier that the coding phase actually started last week So we are coding Well I haven't been Yeah it just feels like everything is moving fast Yeah but this is great Yeah but thanks for letting me know So what is the next thing you're planning for this? So the next so the first thing is obviously extending this but that's not going to be super like troublesome or you know something that's going to take a lot of time but the next big thing I'm looking at is sync implementation and I'm looking at extension points for like webhook like plugins and obviously start with there's the generic trigger plugin and all the all or most I would say are extension points for actions since it gets tied to a URL so I just have to understand that better to sort of just make just see how the entire infrastructure and the entire system is going to work when Jenkins is configured as a sync especially because the events that we are receiving will not just be specific to a job but they can also be specific to say a node or like you know because I think I was looking at that and notice also and like a model object I think so if we have I'm still not sure I still like saying things up loud there's a model object that we connect our action to and that model object is our node and we connect an action to it so it's going to have a URL where events about that node can be sent or not about that node but like just events which might tie into that node for example some event has been received and we want to scale up our nodes and we want to delete or do something with our current system or agents and stuff so that's sort of the thing to just understand the implementation of sync and how can we use extension points if there are more extension points I'd love to just understand and read through them and see how we can do this and also tests reading tests I think you said a lot over there but I think but I feel like you've got you've got a good idea so at the end of the day this is going to be run by you so and I hope you've got the access to the repo yes I do and we were talking about it earlier and Jeff and Rohit mentioned that it would be nice if I like if the code is on on GitHub so you guys can look and comment so I'm just going to do some cleanup because you know my logger now it's not like a debugger log it's a system print and that's that's not good I'm just going to change it to a logger or something like that and also just clean up some more places and then put it there so you guys can comment and just see what's going on and I do have the access okay that sounds good yeah thank you for creating the repository it wasn't it was a hard task no but you know I was like okay if I not just exist because I think I would have I would have like I should have like I would have wanted to ask for permission to host our repository I'm not sure I think you would have got permission to host it it's just that before the project started I thought we you know better to probably we I even thought maybe we should have created issues earlier but I think I think we are in a decent place we could start creating issues if you want and then you know start working in that way but I feel like you've already you're already so driven that probably you will be creating them that yourself I guess I will sum up PR by probably in 24 hours I'm not going to say like right tonight because we're in different time zones in 24 hours it should be on there so everyone will be able to see also debugging by print statements is standard in many ways like everyone everyone does it all the time I put a link in the chat it's like a joke but you know I mean it's everyone does this Oh that's funny like I so I haven't been vigilant about these dates turns out so I just wanted to let know about the coding phases a little bit and do we have time to talk about this stuff Absolutely we have time I have time to stay on the call So the coding phase runs as you said from June 7 June 7 to July to August 16 so it's quite it's a good it's a good period of time or one weekend she's zooming ahead being awesome and this year has reduced hours actually so we're she's like is requiring less hours from the students to take into account everything that is going on in the world and how turbulent this time is which gives us a great deal of flexibility that being said would you have to meet certain benchmarks but we're way on track for that so a student could not say oh I will do all my coding in the last four weeks and it will be fine they do want to review process midway through just to ensure things are on track but that that is it within within that structure we have a great deal of flexibility and where we push and where we need some more time for for whatever reasons be it schoolwork or something is happening in personal life or just something is happening and it's really sure to who can set that flexibility schedule yes and again like you know thank you so much for being understanding and appreciate it really really important and also it's like so nice that I have such amazing team of mentors so amazing and I it's I think it's like unless you guys have like want to change the meeting hours or like change, I don't know the way we are meeting right now, I think it should be fine it works for me, currently the new meeting time is 1400 UTC it's a little bit of a faff doing these these conversions but I think that's the way it's said and it's set on the calendar that way but if anyone has an issue with that and I'll double check it with Marisha because I just want to make sure he can it's the one mentor we need feedback from and I want to make sure it works for him but yeah we can be flexible on that but that that is where it's currently set now cool was there was there something that was just that was just about timing but was there something else Rudy are you is this is this flow working for you with the Slack channel and the weekly meeting and now hopefully more more reviews on GitHub yes yes it's no I think it's perfect. And like I feel like if when we move forward, I might want to maybe do two meetings a week when we are more into you know the same phase I don't know how it's looking right now that be okay with your guys schedule. I'm fine with that. I will just say that Friday is another is the bookends of the week have the last meetings for myself. But but you know we can we can do another doodle and see what works for everyone. Thank you. Everything else is good. Good. Good you're doing amazing work. So I'm so happy. I don't have anything more to show or add right now. You'll talk more on GitHub and Slack. Yes, if you have any doubts like you're always around Slack so if you end up starting the same stuff, you can always ask on Slack. Thank you guys. That's so amazing. And if you have a question for me feel free to do add here or ping me directly because then it'll I'll get a notification. I'll see it more quickly. Great thank you all. And unless we have any more last minute advice or questions then good we can close the meeting pretty much on the hour. Thank you for showing your work for G that was really, really awesome work. Okay, see you all next week.


It seems like over the past 10 or 15 years or so, if you go to a used game store and you wander over to the Super Nintendo cartridges, you're guaranteed to see, like, 8 or 9 copies of certain games.  This game seems to pop up everywhere and it's always cheap, so what's the deal? Well, the deal is that this game, for all intents and purposes, is Battle Clash 2. It's made by the same development team, intelligent systems, and it's got the same structure, which means the game is one big boss run and just like Battle Clash, it's only compatible with the Super Scope. It does not work with the controller, well for the most part anyway I'll get into that later, and it also does not work with the Super Nintendo mouse. But it would also explain why, when Nintendo published this, they opted against calling it Battle Clash 2, since that game was not that much of a hit, and they opted instead to press the old reset button and try again with a different title, and yeah, that didn't really work out either. The thing is though, is that this game is pretty dang awesome for what it is, and it's easily the best Super Scope game out there, thanks again to Andrew Garcia for helping me out with the footage here. World Combat was released a year after Battle Clash, but the story takes place 3 years later, and Roland Dracanowicz, I mean Mike Anderson is back, and the guy that killed his father is also back, and the exact same gameplay is back, but not only that, there's another character you can choose from, and it's Carol Eugene? Really? Of all the names, you pick a name that sounds like my mother's aunt that she's ignored for 30 years. What was your second choice? Dorothy Gladys? Well anyway, I'll get to that whole situation in a bit. The gameplay is the same thing as you would expect, you point the big plastic bazooka at the screen and you shoot at stuff, one weapon does big time damage at your target, one weapon defends against incoming stuff at your mech, but in Metal Combat there's multiple charge shots here, along with a bunch of other extra power ups including a lightning bomb and a plasma bomb among other things, you have at least 10 different bosses you have to defeat this time, with each having a different weakness that you need to find, same as the last game.  Seriously, did the folks at Intelligent Systems realize they were making a super scope game that wouldn't go anywhere?  The visuals here are absolute top notch. Just based on this alone, this game is way way better than you would think it would be. And it's worth checking out based on that alone, the details in the enemy design allow you to target weaknesses that much more closely, which is a massive improvement on its predecessor Battle Clash. Plus, I mean just look how freaking cool this thing looks. You'd think this thing was a final boss in Final Fantasy Shadow Legends vs SNK with Knuckles Mobile Edition or something like that, but it's this game of all things. It looks amazing. Visuals aren't the only reason this game stands out from its predecessor, Metal Combat also has a couple more game modes. There's a training mode here, plus time trials, and a combat mode which is two player versus, which utilizes the controller for the second player. Alright finally, something without the got dang super scope, but there's more. If you enter B-A-B-Y, real clever, on the controller when the title screen just says Metal Combat you can unlock another mech to use called Tornado, controlled by Carol Eugene. It's a bit weaker, but it charges its secondary weapon more frequently, so at least it offers a different approach to the other mech. So yeah, this is a big time improvement over Battle Clash, and why wouldn't it be? It's a sequel, even if the title didn't say so much, but Metal Combat Falcons Revenge is the absolute best game of its kind. It's the best super scope game out of a whopping 12 games compatible with the super scope. Congratulations, I guess that's kinda like being the sharpest knife in the crayon box, but yeah, to tell you how much effort Intelligent Systems put into this game, they even offered two different endings, one good and one bad, depending on if you managed to kill a certain character at the end. I like that, and hey, if you have a super scope, you'll like this game. Alright, I want to thank you for watching, and I hope you have a great rest of your day.


So, now we're going to use our income expenditure model to show how a recession affects income and output through the multiplier, and then how the government can come in and spend money and help get the economy out of a recession. Now remember that aggregate expenditure, planned aggregate expenditure in our model so far is equal to consumption plus planned investment. But now we want to add another very important component and that is government expenditure G. Government expenditure is government expenditure on aircraft carrier, bombs, schools, both at the state and local level and at the federal level. Now Keynes' major insight is that in a very serious recession the economy might settle down at a very low level of income and employment. And to get the economy out of the recession the government might have to step in and spend money. It's better if it spends money doing useful things like building schools, but the major thing the government has to do is spend money, start employing workers, they'll spend money and through the multiplier process they will start buying things from other firms and that will create more jobs. So let's see how this works in this model. Let's say the economy starts off at a high level of planned expenditure, then because of some financial crisis or some other big event there's a big decline in planned expenditure from the black line all the way down to the green line. When that happens through the multiplier process income falls and Keynes said it will just stay at this lower level Y prime unless something is done. And what can the government do? The government can come in, spend money, hire workers who then will create more jobs throughout the economy and we can represent that by a movement up in the planned spending curve. And what will happen is if the government spends more money then the economy will expand and that's fiscal policy to generate more employment. Voila.