Hi, this is Sapin Bhatia from Muktware.com. It's an important day for us today because we are introducing a series of videos on YouTube called Mukcast. In this series of videos, we will cover a range of topics. We will interview a lot of people from Linux and free software world. We will give you a wrap up of what happened in the last seven days. So it's a weekly event. We will review software as well as hardware and at the same time, we will also discuss serious issues around GNOME Linux and open source technologies as well as communities. So when we were planning this video series, we were thinking about what topic we should pick to launch this series. We were at LinuxCon where we met and interviewed a lot of people. Those interviews are still in production stage. We are editing them and we will publish them as soon as possible. So we picked the interviewer of Ibn Uptan who is the founder of Raspberry Pi Foundation at the same time he works with Broadcast. If you look at Raspberry Pi, it doesn't need an introduction. It is one of the most popular devices around. But I was curious about the history of this device and the foundation. So I asked Ibn if he can talk a little bit about the history of Raspberry Pi. Raspberry Pi was an attempt by a group of us at the university in Cambridge to solve a recruitment crisis that we were having in the middle part of the last decade. So we found that we were not having it, that we had too few people applying to study computer science and we found that the range of skills that people had when they came in the door, these were still incredibly bright young people, but the range of skills they had when they came in the door was nothing like what people had in the mid-1990s. And Raspberry Pi has now evolved as a huge revolution. There is a huge open source community right where people are writing different tools for it. They are using it in a different way. Even Google released a tool called Coder which converts a Raspberry Pi device into a web server. So what I even thinks about it? Yes, what's been really fantastic about the Pi is the extent to which other people, as we've kind of pursued, because we've ended up in a situation where the amount of innovation is coming out of the community, kind of dwarfs the amount of innovation. It was really interesting to hear Gabe Newell talk about this earlier here at LinuxCon, is that if you create open platforms, then you allow your community to kind of take the lead in innovating. I think the Google Coder example is really fantastic. I didn't know anything about that until the announcement. I think one or two. Somebody else at the foundation had been speaking to Google about it, but I didn't know anything about it. So I was as surprised as anyone when I saw that last week. He's right about community and companies who deal with free software and open source should understand that even if they have the brightest and the smartest brain on the payroll, there will always be people who are smarter, who are brighter than their own employees. And the innovation from these people will dwarf the innovation and development than you do in-house. So it's always a good idea to engage with the larger open source community and benefit from their work as well. Now, if you look at the example of Coder, though he even made it clear that somebody from Raspberry Pi was working with Google, but he was surprised with that announcement. And sometimes a lot of people from free software world think of this surprise element as lack of transparency. Some people think that companies should be transparent about their projects and products from the first day when they wrote the first line of code. I do not agree with it to some extent because surprise has its own PR value at the same time. You tell everybody about a product when it's actually ready for people to see. So let's see what event says about surprises. Does it mean lack of transparency or does it have its own value? No, I think it's fantastic.  You know, like I say, somebody else at the fan, that's one of the other things that's an interesting learning experience for me is that the Raspberry Pi has now become a large enough organization that I don't know everything that's going on inside the organization. So, as I say, somebody else in the organization had been talking to Google about it, but it was a surprise to me and it was a great surprise. So, you know, fine, if I had a surprise like that every day, it would be a lot of fun. Exactly. And that is the beauty of surprises. Now, I'll go back to the hardware aspect of Raspberry Pi. It's not easy. It's actually very hard to bring open hardware to the market. We have seen how companies like Canonical struggled to raise funds for their Ubuntu Edge project, which eventually failed. We are also witnessing how Aaron Siego from KD is trying to bring Vivaldi Tablet to the market. He has not gone to any fundraising route. So, let's see how Raspberry Pi Foundation has managed to raise funds to bring the hardware to the market. The interesting thing about the Raspberry Pi Foundation is we're largely funded by our own, by the profits that we make from selling Raspberry Pi. So, unlike most charities, we don't do fundraising, we don't do conventional fundraising. However, we were extremely lucky at the start of the year that Google UK offered us a million dollars of funding to specifically to put pies into the hands of children in the United Kingdom and Ireland. And that was, I mean, it's been extremely helpful. So, I think we will have shipped by the end of the year. We will have shipped between 15,000 and 16,000 Raspberry Pi kits. So, that's a Raspberry Pi, a case, a power supply and an SD card. We will have shipped between 15,000 and 16,000 kits to children in the UK via our partner organizations. So, we don't, you know, we're still a very small organization. We don't do this ourselves. So, we have five or six partner organizations, including Koda Dojo, Code Club and OCR, who's one of the examination boards in the UK. And they're responsible for taking pies from us. We were funded by Google. We take pies, we deliver them to our partners, and our partners are responsible for identifying children who they think will benefit from having a pie in their bedroom. Google assists a lot of open source projects and Raspberry Pi is one of them. What is interesting about Raspberry Pi is that it is being used in schools a lot. So, I'm curious if the device has started to have an impact on the curriculum of such schools. If the device has started to change the mindset of students and teachers about open source analytics, let's say what Iben has to say about it. I think it's very early days. I think that, you know, it's really only the past six to nine months that these things have been available in sufficient quantity that education systems can order them. I think we do see some schools at the moment, particularly private schools and what we call academies, which are effectively like charter schools. And I think in American language, those are those would be charter schools. So we see some of those schools that have a little more freedom in terms of their engagement with the curriculum. We see some of those schools buying pies in quite large numbers. To get to that, I think it's going to be a while though before all schools, in particular, before conventional state schools in the UK, adopt pie in large numbers. That's right. And I hope as the device becomes more popular, it will start influencing the curriculum and will also change people's approach towards open source and Linux. If you look at the basic device, it has remained more or less same ever since it was launched. Yes, there are significant improvements. There are a lot of hardware modules that you can add to enhance the features or usability of the device. So I'm curious what's going on behind the scene. I'm curious what are the areas where the foundation is investing its resources in and at the same time, I'm curious, what was the next big thing we can expect from Raspberry Pi? I think the next big thing from us is our improved desktop support. So we've been spending a lot of time, energy, money on improving the desktop user experience on the pie. We're very committed to making the pie a great platform for particularly for multimedia. It's got a fantastic chip that we use, PCM 2835. It's got a fantastic multimedia accelerator in there. So we do see a lot of people already using the pie as an Xbox, as a XBMC, as a media center. So what we're trying to do is get some of that multimedia, get some of that multimedia capability into the hands of people who are just doing desktop applications. So with this in mind, we're spending a lot of money on Wayland, the next generation, compositing a desktop protocol. It's been a lot of money on G-Streamer, WebKit integration. Lots of these things to make the pie a more attractive general purpose machine. It's important, I guess, for two reasons. One, I guess, developing world applications. We do think that there's a big future for Linux machines and in particular on Linux machines in the developing world. And secondly, we've always believed that these machines have to be good fun. They have to be things that children will engage with.  So, you know, trying to make the platform a more usable, general purpose computer helps with our educational mission. That is quite ambitious and interesting, the way Raspberry Pi is evolving and it's growing bigger and bigger. I'll go back to the question of hardware, where I forgot to ask that. I really understand how hard it is to bring open hardware to the market. I was talking to Ernest Hugo of KD and he was discussing how difficult it is to talk to hardware manufacturers to bring open hardware to the market. So what challenges were there for Raspberry Pi? How did they tackle these challenges? It's been a, it was certainly very challenging to hit our cost targets. We started out, I mean, I guess we reversed into our implementation, so we started out with a cost target, we started out with this $25 target. It took us years, it took us, I think four years, from 2006 when we started, it took us four years to even have any hardware that was acceptable to us. So we're very lucky, the relationship with Broadcom has been very helpful for us in terms of getting access to a really good silicon. So a number of people including me who were involved with Pi were also involved in the BCM2835 project at Broadcom in Cambridge. So it's kind of our chip, the chip that's in Raspberry Pi is kind of our baby. So that was very helpful. But it did take us a long time, even once we had the chip, it took us a long time to put in place the supply chain, a design which was cheap enough, a design which had little enough in addition to the core processing unit. And to put in place the supply chain and business model that will allow us to operate fairly, operate sustainably at fairly thin margins. That's true and Broadcom has played a very important role in bringing this device to the market. The hardware is the USB of this device because it brings the cost of computing down. If you look at the price factor, the device makes a lot of sense in big countries like India or a lot of African countries. Though I understand that Raspberry Pi is very strong in North America and Europe, but I'm curious what plans do they have for break countries where people cannot afford computers because of their high price. So let's see if we can talk a bit about the current market of Raspberry Pi and what plans do they have for break countries. So right now North America is our largest market, I think, followed by Germany and then the UK in terms of monthly shipments. I think the UK, at least until very recently, still had the largest install base because obviously we were very big in the UK very early on. I think going forward we're putting a lot of effort into understanding how we can go to market in India more effectively, how we can go to market in South America, particularly Brazil, without currently there are large tariff barriers which push that I think a Raspberry Pi cost 70 or 80 dollars to deliver it in Brazil today. So we're trying to work out how we can not be exposed to those tariff issues. And then I think in the medium term our attention really is turning to Africa in a big way. We think Africa is an enormous opportunity for Linux. We think it's an enormous opportunity for ARM Linux and for Raspberry Pi and for other machines like Raspberry Pi. That's good news for break countries, especially for African countries and hopefully South America as well. Now I'll talk a little bit about the software and community aspect of Raspberry Pi. Raspberry Pi is not just about open hardware. There's a huge open source community around it and you have to maintain a fine balance between what you want and what the community needs. So what kind of relationship is there between Raspberry Pi and the open source community? We've always spent a lot of effort on community relations. My wife Liz runs our website and our Twitter presence. We've had a community relations person before we had anything else. She's been working on this for over two years now. So it's useful and important to us for several reasons. Obviously it gives us an idea of it can be as simple as it's a source of free engineering. You wouldn't believe the sorts of really valuable things that people have done. I think we got very early on, a couple of months in, we got an email from some guy saying we've got a pull request from a guy saying pull this 5-line patch here, SD card performance will double, things like that. So it's a source of very valuable engineering effort. It also gives us, by engaging with the community, we get a good idea where we should be spending our effort. We get a good idea of we don't like to pick winners. We don't like to have a fixed idea of what people should be doing with the pie. What we like to do is look at what the community is doing with the pie. Look to see whether there are places where we can do a little intervention where we can go in and spend $10,000, $20,000, $30,000 on a piece of engineering which will enable the community to move forward. Most recently, obviously, we've been doing this with Xbox Media Center. Before I wrap up this interview, I'd like to talk about the buzzword innovation. It really doesn't matter what you are doing today. What matters is what plans do you have for future? Will you remain relevant in future? So I would like to talk a little bit about what's going on at the innovation front in Raspberry Pi. So I think certainly in the short to medium term, it's all about the software. It's all about trying to make the existing platform better. I think we've said we're going to do a version which has better power performance. We are going to do a board revision in the medium term. Sometime next year, we're going to do a board revision which improves the power consumption of the device. But apart from that, we're not really very focused on hardware. We're mostly focused on tuning up the software stack. I said on my talk today, there's a lot of low hanging fruit available to people. We're very keen to go and hunt that down because ultimately we could chase off after another more powerful hardware platform, but what we'd be doing is orphaning everyone who'd already bought a Raspberry Pi. We know at some point in the future, maybe two, three years out, we're going to need to ship new hardware. We can't be shipping if we're shipping the current hardware in 2017. That's going to be a disaster for us. But we want to make sure that we give as much life as we can out of the platform we have. Thanks a lot Eben for this interview. I hope that the people who are watching this interview now know better about Raspberry Pi and more people will buy Raspberry Pi now. That brings us to the end of this first episode of Mukkaz. Just let us know what you think about it in the comments below. If you want us to review any product or project, if you want us to cover any topic, just let us know and we will try our best to cover it. So for now, bye-bye, see you next week. Happy hacking!


So, welcome to the next session. We will have Anmol Krishan Sarvadeva as a speaker. He will talk about guns. I met Anmol, I think, first time at Geo-Python Conference some years ago. Anmol is very active at visiting conferences, real physical conferences back in the day. So, he was also a pike on Thailand, Malaysia and many, many more. And as I said last year, I don't even remember Geo-Python or Euro-Python. And so, welcome. Anmol, he's also, I have to mention that this year he is also one volunteer of Euro-Python. So, thank you very much for volunteering. And now I give you a talk. Start your slides, please. Thanks, Martin, for the introduction. Okay. So, hi, everyone. I am Anmol Susteva. The title for today's talk is Painting with Gants. We'll be talking about the Neural Style Transfer and the technicalities and challenges of using that. So, a brief introduction about myself. So, I'm an international tech speaker and a distinguished guest lecturer. And I work at OLX Group. I have done my Masters in Advanced Computing from University of Bristol. And the specialization was in field of Computational Neuroscience and Artificial Intelligence. I represented India in various international hackathons and I'm a researcher also. About OLX Group. So, it's a group of, which contains of 20 plus brands. And it actually has around 45 offices spanning across five continents. And we serve across five continents with 350 million people per month. So, the flow of the talk will be as follows. First, we will be looking into an introduction to Gants. Then we'll be taking a look at what style transfer is. Thereafter, we'll be learning about different neural style transfer networks that are available and are popular at this time. And then we will dive into the actual NST implementation by looking into loss functions, content loss functions, style loss function, total variation loss function. And then we'll be doing kind of a code walkthrough also. The talk will be supported by a few demos also. Those are actually adopted from the official TensorFlow and Keras repositories. So, I'll be pushing the code to GitHub and we'll be sharing the link to it in the breakout channel. And then post talk, we can have the Q and question and answer session in the talks, talk painting with Gants channel and discord. So, yeah, prerequisites for this talk that you should be familiar with Python and Keras, especially using TensorFlow back end and an experience in artificial neural networks is good to have. It's also good if you have experience with convolutional neural networks and generative adversarial networks. And you should be inquisitive to learn about deep learning. So first let's start by revisiting the fundamentals of generative adversarial networks. In short, I'll be referring to them as Gants. So discriminative and generative models are the two types of models that we use in again. So first coming to the discriminative model a discriminative model forms a discriminative network or the discriminator network. Essentially a supervised learning model which tries to classify the data which is fed into it. So it is just kind of a classification model that we are using here. And it doesn't really bother about the underlying distribution of data only the quality of data matters to it, so that it can classify into categories properly. And then on the other hand we have a generative model that forms the generative network. And instead of classifying the data it's used to generate the data. It actually learns the underlying distribution of the data that's provided. And then on basis of it, it tries to generate samples that are near real looking. Mostly it's unsupervised learning but what if we want to have some conditional training done. In that case, we may support the training set with label data also so that it becomes kind of supervised plus unsupervised learning. So conditional Gants actually use label data. And there we have sort of supervised learning also implemented a bit. And then the art of actually learning the data distribution underlying data distribution which the GAN actually does is through like implicit density estimation so we don't require to calculate any probabilities externally. Everything is done internally by the network itself and it's called as implicit density estimation. So I'll be referring to these terms going forward, but just to give you a just of what GAN network vanilla GAN actually looks like in form of a schematic diagram. It's this. So the goal is to generate near real looking samples of the underlying distribution that we are provided with that's the training set. And then we have this input layer we where we feed the random noise this random noise forms the part of latent space. This can be uniform distribution can be normal distribution. We have to pass this distribution through GAN which is like again formed by two neural networks one is discriminator network one is generator network. So we'll be covering on the details of what's hidden inside this box in the coming slides, but then once we pass this input to GAN the GAN produces an output, which is of some other dimension say in dimension. And then that's maybe an image that we formed out of random noise or something like that. So training of GAN algorithm has two essential parts one is training the discriminator the second is training the generator. So training the discriminator network actually involves the following flow. So we take a sample a real sample that that's a sample from the training set. And then we pass it through the discriminator and we have the discriminator classify it. On the other hand, we have a generator network to which random noise is fed. And then that generator network actually produces a sample which we call as X star or X that that's a fake sample and that fake sample is also fed to the discriminator. Now the discriminator should be able to classify this as a fake sample, but our aim is to actually have generator improvised to the extent that discriminator starts failing in distinguishing between the real sample and the fake sample. So there will be a point where discriminator will start labeling the will start classifying the fake sample as real sample. And there's the path of generator training so generator actually uses this random noise in the second phase of this training this generator actually uses this random noise generates a fake sample. The sample is fit to the discriminator and the discriminator classifies that as real or fake. But the essential thing is that during this training of generator phase we backpropagate the errors to generator instead of backpropagating it to the discriminator. So in discriminator phase, we actually have the backpropagation done to discriminator network whereas in generator phase we have the backpropagation done to the generator. And here we make sure that the discriminator networks parameters are not trainable so in the second phase we set the trainable parameters for discriminator to false because we just want the generator to improvise. So in schematic diagram it looks like this so this is the first phase train the discriminator the second phase is training of the generator. So we have this as x the real sample that we are providing to discriminator and then we have this generator to which we feed some random noise from the latent space that we are calling as z. It can be normal it can be uniform. It can be any other distribution. It produces a sample x bar and then that's also fit to discriminator. So the discriminator classifies x and x bar as in some category maybe real or fake and then classification error are propagated to discriminator so that discriminator learns. In the second phase, the only thing that we do is we remove this training sample phase, and we just pass the x bar and then we pass the backpropagated error to generator. So that's the only difference and then we do this we repeat the cycle in iterations so that the network learns so the discriminator actually gets improvised on distinguishing between real and fake data and the generator on the other hand improvises on generating data which is near real looking and which adheres to the underlying real training data is data set distribution. So these are all fake samples generated by a Nivedia style GAN. So no one can tell that these are fake. They look pretty much real. So this is how much we have advanced in the last five years since the inception of this concept of GANs. So next comes the main concept for which we are here today. We have gained quite a lot of hold on generating near real looking images or kind of photo realistic imagery. But what if we want to now generate art instead of just generating images of static objects, we want to now design new objects or build artistic artifacts. So how can we do using GANs. So here comes the concept of style transfer. So, so as I told you earlier, we now have to dive into this generating new kind of artistic artifacts. So what if we have a kind of image which we call as content image or base image that's there of a dog. And we have a style image here I have taken image of cross. So what if I just apply the style of this cross over to the content image and say it generates this output. So that means the style of this image has been imposed on the content image. So we get some unique form of art that actually contains both the content as well as the style. You see that the content also is dominant here and the style also can be reflected is getting reflected in this image. So this image holds kind of combination of both the content image and the style image. So that's our aim. We will be generating art based on similar lines. So having a content image will be having a style image and our goal will be to transform or embed the style from one style image to a content image in order to produce an image, which is a combination of both and looks good and realistic and original. So, yeah, that's the aim of neural style networks that we'll be covering now. So you must take note of one thing that this model that we saw this training that happened that didn't learn the underlying distribution. So here comes the first difference with respect to vanilla GANS vanilla GANS actually had the underlying distribution being learned. But here we are transferring styles. So that's the first difference. So we extract the style from the style image and then embedded to the content image and the result should look like a blend of both the images. So why not simply interpolate the pixels? That's, that's because if we interpolate this with this, what we will have is blurry image that's highly distorted and the style actually dominates the content image. So it will look muddy. It will not be clear and both the pixels will lose their entity. So this, this is the thing why, why we should not use simple interpolation for doing such sort of things. And this style transfer networks have white applications in the area of gaming in the area of developing applications. A few years back we had an app called Prisma, which had this sort of style transfer made public. So people were able to apply styles from different images on their selfies and all. So that, that actually saw real boom in, in like last two, three years and people have advanced in generating new networks that are like state of the art networks and can do a style transfer. And many more applications have come, which we'll be discussing in a few minutes. And the last one is that the actual image, if you see, it's just that you have just applied the style on the content image. So the dimensions and all you can play with but ultimately it looks like it looks like someone has applied some art on the content image and we got the generated image. So popular style transfer networks, if we see, these are three networks, one is PIX to PIX and one is CycleGAN and then there's Neural Style Transfer. We'll be covering a bit on CycleGAN and PIX to PIX after we cover Neural Style Transfer, but first we will start with Neural Style Transfer. So Neural Style Transfer, as I told you earlier, doesn't require any training set. We have seen that it doesn't require any training set. It doesn't have any kind of, you can say back propagation being done because we just have two images to deal with. We have to transfer the style of one image to another image. So there's no involvement of any kind of training set. So that's another unique thing about Neural Style Transfer that it actually picks up the features from the style image and applies the features on the content image. And it creates some hyper-realistic imagery. So let's say we have this base image and then we apply a style image that's shown here at the bottom. We get this kind of combined image. Just playing around with the loss functions and the hyper-parameters. We can see highly varying resultant images. So this is one image that we see. Likewise, we can have different degrees to which this image can be exploited by this Neural Style Transfer network. And we can have different images which have the style features transferred or embedded onto the base image in different degrees. So we can generate multiple images from the same style and base image using combination of both. So the core of Neural Style Transfer is that essentially we have a loss function which constitutes of content loss. That's the loss which I'll be telling about more in a bit. So content loss, style loss and total variance loss. So first thing is that we have content loss where we would like to have the combined or the resultant image to be as similar to the content image, the base image. So that's a loss between the base image and the generated or the resultant image. Then comes the style loss in where we have the generated image compared to the style image. And then on the basis of how much degree of correlation is there between the style of both the images, we calculate the loss. And then last is the total variation loss that also we call as total variance loss wherein we have the generated image checked whether it's smooth or whether the pixels are distorted and blurry. So how do we minimize this loss? Ultimately the goal of Neural Style Transfer is to minimize the combination of these losses. So how to minimize this loss? We actually use a gradient descent technique wherein we update each pixel over the iterations and then we have something as you saw in the previous slide, the combined image. And there's a difference with the vanilla again which I highlighted earlier also that there's no training set required here and there's no back propagation concept being applied here. So coming to content loss, now onwards I'll be making references to bits of code also but before moving forward, I would like to show a quick demo of how we can actually utilize a pre-trained network to generate these art pieces. So yeah, so this is the iPython notebook that I'm using. I'll just increase the size so that you are able to see. So here we are importing libraries. This is the TensorFlow library, matplotlib. We are setting the run config parameters for matplotlib, the numpy, python image library, and func tools. So this is the import part. Next comes a function. So this code has been adopted by TensorFlow, the official repository. So the function is to convert the tensor to image. So this does nothing but uses some numpy functions like watching on the channels that are there, the dimensions of the images that are there. And then we choose the primary channel if that's if the number of dimensions like it's greater than three dimension. Herein we will be using four dimensional tensors. So it converts the image, the tensor to image. Then that's just a function which is being utilized below. We have the content part. Content part uses the utils function of keras. So get file actually gets the file from remote place and then fetches the file. So we are using image of the dog which I showed earlier. And then for demo purposes, I have taken three images which I'll be running through in a bit. So these are the images of style that we want to transfer. So this is a content image. This is a style image. Then we have load image function just to display the images. This again uses the normal numpy and TensorFlow functions and libraries. We read the image, we decode the image, we convert the image and scale the image. So it's kind of processing of image and then showing the newly formed image. So we resize the image afterwards and then we return the image. And I am show function. This is a typical function. We just are attaching title to the image. And then we have the plot function. So till this point I'll run each of the cells so that you observe what is happening. So first we will see image, the style image of bushes being applied to a dog. So last time in the slide I showed grass being applied to dog. Now we will see bushes being applied to dog. So yeah, let me print the images that we are taking into account for this. So this is the image of bushes and we are applying it to dog. And then we'll have the pre-trained model VGG-19. So I'm taking a VGG-19 pre-trained model which has all the weights set and it is able to classify into 1000 categories. The network has been trained on 1 million images taken from ImageNet dataset. So you see that the combined image is having the style of the grass and also adhering to the content image has the content of the actual base image that we passed. So nothing much I have done here. I have just used VGG-19 pre-trained model. And this is the destination to that model. We have passed it to the, we have passed to it the content image along with style image and then we have passed to tensor to image thing which takes out and processes the image and then displays it. Now let's quickly jump on and then I'll uncomment this and we'll have some other style transferred to this image. We can directly go. So I'll show what's the style image we are referring to now. So, okay, let me change the name here. So it actually took from cache. That's a style image that we are trying to apply on the content image. And then I'll quickly apply this to the base image. And we should have a result like this. This looks like a novel art piece. So that's for the demo of the first part. Now moving back to content loss. So content loss is actually kind of L2 difference or the like mean squared error between the content image and the generated image. So what if we compare apples to apples? The content is similar. So the information of the pixels should also be similar to some degree and we will have less content loss. But what if we compare ocean or sharks or say oranges with apple or banana with apple, then there will be higher degree of content loss. But we will not do pixel by pixel comparison. What we will do is we will have the higher level features compared. So how to compare the higher level features while training a neural network. Say we pick up convolutional neural networks and there are blocks or there are stages at which we train the neural network. And as and when there's any stage which progresses to the next stage, some of the features gets dropped, the lower level features gets dropped and we just are left with higher level features. So at each layer of neural network, say convolutional neural network, the lower layers represent some minute details say very, very minute details and the higher layers or the layers at the top. They actually contain the higher level features or just say broad features like this is a car. This is a building. So just to capture the higher level features will be dealing with the top layers of the neural network from a pre trained model and in our case it's a VGG 19. So it is a 19 layer the CNN architecture. And as I told you it's capable of classifying images into 1000 categories and has been trained over 1 million images from the image net dataset. So the VGG net architecture looks something like this. We have an input layer and then we have five chunks. So one, two, three, four, five, these are five blocks.  So we'll be using this VGG 19 net and then using the higher layers from block five will be capturing the higher level features or the content or the features through which we will be actually collecting and computing the content loss. So this is the code. I'll just zoom in if that's possible. Yeah. Okay, so we have this Keras library being used VGG 19 is the model that we are using the pre trained model that we are using. And we just specify the path to base image we specify the path to style image, and then we just specify some weights so content rate is the total loss total variation loss rate is the style rate is there but that will be looking out in a bit. So base image, we have the base image style image being processed by the path from the path which we have provided and then we have a placeholder. You can say placeholder image created that wherein the combined image will go. So we are having these three majors. We pass these three majors by contact on catenating them in form of a tensor and then we pass them in VGG 19, which has the pre trained weights, and then we get some resultant image from that image we actually take we actually take from the pre trained model the third, you can say layer of block five. So if you see third layer of block five, we take the, we take this as a reference for calculating the loss because higher level features are captured here. So we may take to second layer we may take third layer we may take fourth layer but it totally depends on you like taking different layers may result in varying results. So I have chosen three as a layer, you may choose to also So we choose to and then we pass the base image and we collect the output from this third layer. And we also have the combination features on the combined image taken from third layer in the network. And then we just have the L2 norm applied that's mean squared error applied to both the images so generator image and content image and then we just have the content loss calculated by multiplying it with the weight, the content weight that we specified here at the top. Coming to style loss, that's the second type of loss that we wanted to cover. So coming back to style loss, we have images which are like similar to each other maybe. So those images we call them as correlated images but there are images which are different and have different styles or different lower level details that doesn't have like common lower level details. So those images are considered as having less degree of correlation. So how to calculate correlation between the layers. So degree of correlation between two images can be can be computed by calculating the degree of correlation between the feature maps. So as we want to capture lower level features because lower level features represent the style. Just just note that higher level features represent content and lower level features or the output from the lower level layers of the neural network represent the style so we we take into consideration the lower level layers of convolutional neural networks we fetch the feature maps from there. You flatten the feature maps we take the dot product of the feature maps between the two images and then depending on that dot product of the value of dot product is greater than some say value that we have specified we consider it to have higher degree of correlation that means the style of both the images matches. So suppose this is image of grass image a is image of grass and image B is also image of cross. So you see that the orange points the dark orange points that you see on both the images. If they overlap so that's the area where the image has like similar kind of correlated style. So we can say that these images are correlated to some extent and in case if this considered this block B to be fully orange then we will say that that has a higher degree of correlation that combination has a higher degree of correlation. So in our case when we will be calculating style loss image a will be style image and image B will be the combined image the resultant image that we get after training. So in order to actually have the style loss calculated for different layers of our network. What we consider is a thing called Gram matrix so Gram matrix is dot product of all the feature maps against the feature maps. So suppose you have layer. So you have this layer. So if you take a dot product of a with a then we you take dot product of a with B. So in this image it should be clear we have image a and we have image B image C. All these images are of grasses but one contains only grass one contains pushes also one contains some brown grass. What we have is that we have the Gram matrix which you see on the right calculated so we have feature one map to feature one and then we calculate the correlation between them by taking dot product of the feature maps. Then we take dot product of feature one with feature two of image a that means this one and this one, the product of both will be taken and the common area or the result of the product will be shown by some color. Likewise we do fall do for all the features of a layer for for an image. So suppose this is image one we have first dot product taken across all the feature maps. Against the feature maps of that image and then we have this Gram matrix likewise we generate Gram matrix for image B and image C and then if the Gram matrix of two images in our case it will be the style image and it will be the result and generated image. So when the Gram matrix of both the images are highly comparable. We say that the style actually hold it throughout the training. So the style was actually transferred to the generated image. So in that case, we will consider it as success. So we have to calculate the mean squared error again. That's L2 norm error and we have to minimize that error. So coming to the code we again say that style loss is zero initially we have the definition for Gram matrix function here. This function actually does it flattens as I showed you it flattens the feature map and then takes a dot product of it. Against itself and then returns the Gram matrix. So feature map dot product is taken and we get the Gram matrix cell likewise we do for all the feature maps and we get the whole Gram matrix. Then we have this function of style loss where we calculate the mean square error between Gram matrix of combined image and style image. So this is nothing but the mean squared error of the Gram, the combined image and the style image and these are just like some parameters to have that we are passing for the function to the loss function. This feature layer actually shows the active layers that we chose. So we chose block block one. So we'll be calculating loss against all the layers present in block one the block one of image is this block one. So suppose we have five layers in image we'll be taking that Okay, so we took block one block two block three block four block five. So we have taken each layer of all the five blocks so con one one, con two one con three one con four one con five one. So each of the lower level layers from each block have been considered in order to have this style loss calculated and then then we just pass this and extract the feature maps from these layers and then we pass it to the style loss function that we have Then comes a total variation loss a total variation loss is nothing but the loss or loss with respect to the quality of the resultant image that we are observing. So in case of the combined image is distorted and is pixelated. We will consider it as noisy and the loss will be very high. So what we can do is we can take combined image and then we will shift pixels of that combined image each pixel to the right once and also we will do another step we will take each pixel of the generated image and then shift it towards down by one pixel. We'll have both of these results stored in A&B respectively and then we will just take a sum of these two and we will calculate the error. So we will calculate the error by shifting the pixels to right and downward so that will show us whether that image is highly distorted or not. So that's for total variation loss. Once we have these three losses we will just combine all these three and then we will get the resultant loss. Then is the time to start the training of model. So then comes the training phase. So we have computed the loss till now we have computed gram, we have computed gram matrix based on which we have computed style loss. We have computed the content loss by taking into consideration the content image as well as the generated image and we have calculated the total variation loss. So this loss can be trained. So this network can be trained by taking into account loss and we need to minimize this loss. So we'll be using an optimization technique here. So essentially Neural Style Transfer is an optimization technique which depends on another quasi-Newton numerical optimization technique called BFGS and L is for like a limited memory use or we can actually constraint it on basis of resources. So L is limited memory BFGS algorithm which is a numerical optimization algorithm and what it does is it finds the local minimum of any objective function based on the gradient of that objective function. So essentially what we need to do is we need to minimize this computed loss over iterations by using the gradient descent method. And what we'll be doing is we'll be updating value of each pixel by an amount which is promotional to the negative of the gradient that comes from this loss function. So let's dive into the code here. Okay. So it's pretty much the same as I showed you in the snippets. So we have the base image. We have the style reference image paths and we have the weights defined here. So total variation weight we have defined, style weight we have defined and content weight we have defined. We process the images and we specify the dimensions of the generated image that we want to have and we specify the iterations. So for this demo I have considered 50 iterations only but in real scenario you will be using somewhat like 5000 iterations or 4000 iterations to actually see the result that I showed you in the slides. So let me go back to the slide once and show you the end result that we'll see. So this is the end result that we should see and this is after 4000 iterations of. We have the pre-process image, pre-processing of the images done. So it actually just opens, resizes, applies the image processing functions and then we get the tensor basically out of it. We have this deep process image also. So it converts an image, it converts a tensor to the image so nothing much is being done here. Again, a reshape function is being used. Then we are clipping the NumPy array for any additional you can say overflow that's happening. And then we just pass the base image and style image to the pre-process function that we have created and we get the tensor representation of the two images. Once we get the tensor representation of the two images we have all the three images combined image that we are treating as a placeholder image as of now. The style image which is a pre-processed image and the base image which is a pre-processed image. So these three are good to be fed into a tensor concatenated network which will be feeding the same to the VGG19 network for training. So we'll be combining these three and then concatenating and then feeding into the VGG19 pre-train network which we imported at the top. And then we'll have the model loaded with us and also we'll have the key layers that we want to actually match against taken into form of a dictionary here so output represents the same. So now is the time to compute the neural style loss that we talked about earlier. Again, this is the same similar function which I showed you earlier. This is a bit modified from the actual one that I showed you in the slide. That's a gram matrix function which I have already explained. So it calculates a gram matrix for the fed tensor. Then style loss is still in style loss we have the combined image and style loss gram matrix generated, style image gram matrix generated and then we calculate the L2 error. And then we have the content loss which is just simply the MSE that's mean squared error between the generated and base image. And the total variation as I told you that we'll be shifting by one pixel and then shifting towards the downward direction by one pixel. So that's again the sum of both and then we just calculate the error. And then we select all of these three losses and we add them to form the main loss. And the thing which you should be seeing is this the next thing that you should be seeing is this that we have the gradient computed for this particular loss. And we fed we feed it to the evaluator, you can say class that we have created so evaluator class evaluator class actually returns the loss and the gradient value at each stage. And we have the iterations formed so my network is training as you can see over say 4000 iterations what we do is we take this loss. We pass it to the evaluator class we get the loss and gradients and then we update the value of each pixel by by the negative of this gradient thing. And ultimately what we see is this combined image. So that's all for this talk. I'll be posting the links to all these cool things. So next we have pics to pics just I'll be talking a bit about it. So it's used for image to miss translation so you can have say hand out or say a schematic diagram being transferred or translated into an image which looks real kind of a chipboard representation being translated to an actual building or silhouettes being transferred or translated into images Google map street view being transferred to map view. Likewise, and then we have cycle gas which is again an advanced scan for neural style transfer. So here we have essentially to gas one is training for the first input and one is training for the other input but both are actually dependent cyclically on each other. So that's another application but discussing this is out of the scope of this talk since we restricted it to new style transfer. So this is something which I am pointing you towards that you can actually explore cycle again also if you are more inclined towards scans. So these three are the popular. You can say networks out there to have the stylistic artifacts created and linked to each one of them and wherever I use resources I use the code references from David Foster's generative deep learning and Jacob Jacob's guns in action. So these are the two reference books that I consulted. So that concludes my talk. We are also hiring at the well script so feel free to reach out to me or just drop in at the courier section and then feel free to apply for the roles. And yeah, don't forget to follow me on Twitter LinkedIn we can get connected and we can have your questions answered in the discord too and then later on we can get connected on these platforms. So thanks a lot for listening.


A Bemidji City Council member is hospitalized in Fargo at this hour. Ron Johnson is being treated for bleeding on the brain. Two brain bleeds were found Sunday afternoon. Johnson, who represents Ward 3 in Bemidji, then went through brain surgery at around three in the morning on Monday. Johnson is reportedly doing much better today according to his wife Carol and doesn't appear to have any brain damage from the bleeds. He will remain hospitalized for at least one week for further testing. Ron is the Graphics and Promotions Manager here at Lakeland PBS. And all of us here at Lakeland PBS would like to wish Ron our best and we're hoping for a full and speedy recovery.


Part one of this series It took me 45 minutes to get two-man feed and when I say two-man feed I don't mean two-man unscreen feed unsplit unshit Collar duty no scope across map. I'm talking about three quick kills or two quick kills with a no land beyond today I am for three quick kills with a no land beyond they may think oh, that's so easy DPJ anyone could know I can't do that I Cannot do that it's so hard for me you guys probably if you missed episode one I'll link in the video description It's more so about shits and giggles your guys entertainment see me fail because that's what you're getting here I ain't no true Vanguard.    Oh, what am I using here? I'm using any gunslinger I'm no land beyond party creature. I'm gonna put on this so I can rock it people in the faces I got my I only use this club because of the increased cruise for reputation. I've got zone each walk up here with What happened today? I'm gonna kind of remember So long zone a secure Was it was it was knock on head radar?  And I'm not hitting men now. I can snipe better With no land behind and I can't with any of the sniper as weird as that is And I Anybody else? Shit, that's what you want people to do my fees already over.  Thank you. Well so far. He went good at the start But now it's going real bad. I don't want to team up. I don't want to gang up on me I didn't really read it update.  I don't even know how I'm fair to my teeth. I don't even know I'm fair to my team I'm good at slamming of course. Yeah, never Zone B neutralized See bad we captured zone B. I don't even know what it is.  I was never kidding.          That's crazy He's the lead. Oh we're gaining the lead.  We're like a few thousand points behind I'm paying attention.   Let me find another three queens. All right.  I want to do Let's you guys I don't know if I'm gonna post it before I've posted this. I've recorded the video already as a first weapon But I don't know if I'll post it before I post this but either way it's basically me Going back using them amazing year one weapons weapons that are not available to get in year two But another point in the video besides the fact you guys see me run around in pvp Which is amazing weapons which I do okay with obviously I can't do amazing because I ain't amazing at pvp But I do okay with But also showing you guys the year two equivalents to them So I don't know if I got any I'm going to demonstrate. No, I haven't I have the load load on p14 This is an amazing an amazing Also rifle for pvp. I mean I posted a weekly montage where it's me failing doing good About two weeks back in that video when I'm on the 12 kill streak with this weapon And a lot of people are wondering what this was It is the load down p14 I haven't done a video on this one yet, but I will do a video on it But it's year two equivalents. It's got this right here.  So you can't get different roles. See with this here This is a gunsmith weapon This can come with different roles and if he has got a package I would suggest you saving it until he has that perfect stability roll try to get counterbalance on it What have I done this? I've got crowd control rodeo And uh, where's frame Now this is okay But it isn't as good as this this one's got something about to see the hand on this weapon Just makes it a headshot machine and it's got almost max stability as well, but it's a beast. It really is an absolute monster Great and probably the best year one also rifle I've used and it still dominates in year two And the point of having year two equivalents because of trials and iron banner Well, yes, if I haven't posted that video already Stay tuned for it. The first weapon I'm gonna do is an amazing pull shot. I've called the one two three scissor g Incredible weapon Incredible weapon probably my favorite pvp weapon of all time Fucking That's doing on my desk.   This is gonna go so bad Controlled Okay, so this for bravo team.   I'm just gonna be trapped in here. Bye.  I'm your friend mate She's trying to hit me there with a pulse. I mean a future I don't care that here. You're in the lead. I'm not a hit marker then About 10 seconds after he landed and I got an assist with it.  I don't get it. Everyone's on green bars.   What the fuck's going on? Shade step Bang, dude. That is ridiculous man.  You can know go among me and then red definitely Don't care before that Don't see rush just now. Okay.  What's going on? That was The worst of the worst lag I've ever seen Heavy ammo on the way. That was terrible That was terrible That was disgraceful You neutralize zone C. How how do people play like that? I mean like How do people play like that?        I might have been sure. I got one kill. I've got a gun man I ain't getting all three man feeds in this game. That's for damn.  What happened to him then you put a bubble down it just disappeared Zone B secure Neutralized.   I need a big map. I need a big map like sure's a time something like that It's small that's close quarter maps. Ain't no good for me.  I see you dead.  No, I didn't even see him for a while. I just heard it fizzing in my face Fizzing towards my face and we're putting our three of coins Yes, I did I teabag with it on somebody He lost zone B neutralized I'm a scrub Zone B secure. Enemy captured zone A Still we know so it's all good. It's all good in the hood Zone C lost. Enemy captured zone C You can't teabag me or you mother-fucking-wine You have room and don't let me on Enemy captured zone B Individual achievement. What's all that about? I don't even know.  Virtual reality's got me hooked. Map by one or like Get what you need.       What would how would you want that car?  It's probably back in this plastic Actually, he's doing around my desk with a pudding if he sticks.  Let's motherfucking go What time is it time quarter two?   He's coming back for me the lead. Look at that.  I shot the wrong person there.     I'll go right give me a zombie of puppets though Zone a neutralized Switch weapon. I did a switch weapon. You can dance on my buddy.  I've hit you that's fine. I've hit him I might do the dinner should be my job. Oh, you're not.   Don't see lost Let's get see get see you're in the lead. They were good. They were good. You can be as well. Nice Zone b neutralized zone c secure.   See lost I'm dead.  Do you like it so good?  Last week's catching up. Let's go to a point Let's go to a point zone c neutralized I'm dead lost the lead should zone c Okay, just run straight past me. All right. Yep. Well for you We ain't trying to win or nothing.    I like that.  GBR house.  So that's nine packages I love one of these. I'm like accumulating about 50 packages for one massive video Spoiler, I haven't had a sniper yet Haven't had a sniper yet At all could this um Hand cannon Not bad. Not bad at all Pretty bad range, but still without looking the chamber. We've got rifle barrel and we've got outlaw It's not bad, is it? That's not bad Try that bad boy and bad! Beep! I also got this uh Herger D as well. I was going to say J then Herger D. We've got a glass half full. We've got a range finder and we've got hand-laid stock So it's going to have max stability.  What does this fall in that because It's basically the same role as a smite of the rain But I know the smite of the rain's more powerful than it should be for its stats.  I'm gonna try that. I'll level it up and try it and see how it goes See how it goes. I know Don't have to tell me.   Oh one of them. I didn't notice Yeah, whatever. All right, this is gonna definitely my last game.   That's why I like to be honest. I like to play pvp Although I ain't good at it. I think I still like to play.  I'm at zone B secure. Bitch Gains the lead Hey, just enough. How do you still like it with it?  It's coming out off another day as well.  That's have come in I will not financially name his hot moots have canoe Have to remove my fucker I like this a lot in pvp. I get so I'm gonna mess you say get a rex grub.   You're just sweating.  Where'd he go?  Here dad zone C secure Oh, here dad.  That's a good spot. Let's go. It's this fucking. Oh, this recording. Sorry dad, from every angle Neutralize.  Heavy ammo on the way Zone A secure my life. Zone held you have total control I got back to that way that was that was a good spot. Come on let's want to go. This tree is coming Don't see you lost. How did I know that was gonna happen?  Total control. Good work Shame kind of lying down. I can call the tree.    You know I'm freaking three people after him man. That should be enough That's Like I don't think you can do when you're lying down sit down zone C lost be low cancel thing Enemy captured zone C I go up there He's trying it. I don't think he's supposed to anybody else but me zone C neutralized. I don't even know where that was lost I don't even know when boy Bitch Enemy captured zone C Don't worry that's all.  I'm careful dead last I don't go far zone B lost Enemy captured zone B It's a mistake. This is easy You didn't scratch him. It didn't even scratch him. That was uh I've been noted that was sick Oh, that's that 55 minutes 50.  Give me a nice lunar looking chamber No Well, no, that is it. I don't do no more of that today. I've had enough of no land more fucking behind Uh, but guys if you do want to see more Uh, let me know down below in that comment section also drop a like I do appreciate the support And it really does help out Um, you can let me know where I'm going wrong. I know where I'm going wrong.  Thanks talking about as always and I'll catch you on that next one Peace out till next time. Bye