Mae'n ffordd a'i ddweud o ran gweithio'r sgolon ni'n ffordd am y ddechrau a'r ddweud i'r ddweud o'r cyflwyf yn ei cyflwyf. Rwy'n wedi cael ei ddweud o'r ddweud o'r ddesiftyn, rhan o'r agoriadau a'r ddweud o'r mae'n groes iawn, ond mae'n gweithio'n ddweud o'r ddweud i'r ddweud o'r ddweud, ddim yn cael ei ddweud i'r ddweud o'r ddweud, gy Before we can start to discuss the more exciting stuff we have Carthick who's going to talk about sentiment analysis. So something if you've got a website and you want to understand the complaints then that's what he'll be talking about. We've got Anusha who's then talking, she's talking about summarization, which is an important thing if you want to... You're deluged with data and finally Sam will talk, he's got like a five minute presentation. Cynghwarae'r tymp yn gweithio gyd. Mae'n gwneud i chi gyd o blendodd yn gweithio eich mae'r cyntaf yn y cyfraff yma. Wyddo dvent hynny. Mae'r braf hwn yn gweithio'r braf hwn yn dweud, fel y teimlo, ac mae'n gweithio bod olygu iawn ddigon nhw'n gangb o'r rhoi. Mae ddysgu'r gw realitya, mae citr ac mae yn dda i'r gweithio a gwaith. Mae'n gweithio gweithio gwaith yma ar yr olygu ecild y gallan o'n ein gwerthau. But now, let me go for it, when we're ready, we're still coming in, still coming in. So, I'm going to be talking about recurrent neural networks and text, and this is our fourth meetup. So, if you haven't been to the other three, the introduction to this is going to be rather abrupt. So, this is a bit about me. So, I have a background. I did a PhD ages and ages ago in Machine Intelligence. I was in New York doing finance, kind of start-up-y thing. I moved from New York to Singapore in September 2013. For 2014, the whole year, I basically had fun. And I was doing open source, reading papers, writing code, playing with drones. That was a good time. Since 2015, I've been in serious mode doing natural language processing and deep learning at a local company. And I'm on particularly good behaviour, and particularly serious tonight, because we have four representatives at the front here from the company. And I say it's serious, but we actually have, I have quite a lot of fun anyway. So, I've also been doing workshops and writing papers, and generally a deep learning enthusiast. So, in outline for my piece, I'm going to talk a little bit about basic neural networks, rather abruptly move on to recurrent neural networks, both the basic idea and what problems immediately arise, and then grooves and LSTMs, which are essentially the basic building blocks that people use. Then I'll talk a little bit about natural language processing, which is a more tricky subject than just picture processing in some sense. Tokenisation, also word embeddings. And then I've got a little application demo, which talks about uppercase named entity recognition, which we'll get to in your understanding by the time I go there. So, quick review. A basic neuron does simple computation, layers of neurons can do feature creation.  You can also look at the previous slides which we had from previous talks. They're all online. Basically, when I say simple, this is a single neuron. What we have here is some inputs. So, this may be, this, for instance, these X's could be the humidity, the precipitation, the sunshine, the temperature, the output is, is it winter? So, this is a very simple idea. Basically, in order to map these inputs to this output, what I'll do is I'll sum up these inputs times some weights, and then I'll apply a non-linearity. And this is the result of one thing. Now, if you think about it, by multiplying these weights by the inputs, I'm generating a kind of a linear feature. This non-linearity actually means I'll have linear on one side and then zero everywhere else. So, this is a very simple function. You can't learn much with this function. But if you change the weights, then it will do different stuff. You can then combine these in multiple layers. So, the question is, how would you train all of the weights? If you did a linear regression, you could easily train this first one, but by the time you put them in layers, it's very difficult to see how errors up here will translate to errors or changes in weights down here. So, if you want to understand how this all works, there's a thing called the TensorFlow Playground, which is great fun to have a play with, which is at playground.tensorflow.org. Basically, this allows you to set some Xs or some features on this side and look and try and classify some points into blue and orange on this side. You can add different numbers of neurons within a layer or different numbers of layers here. So, this is great fun to play with. If you press the play button, it will then train and you can see it converge or not. Very nice little example. But the basic thing, the basic takeaway, so this is where I kind of finish with neural networks, is once you do, for doing supervised learning, you have inputs and outputs where you know the data. We've seen what a single neuron can learn, but the goal is we need to train a whole network to predict the outputs from the inputs. What we do is essentially play a blame game. If we're making errors at the output, basically we can assign how much blame for the inputs for the layer before, how much should we jiggle each of these little weights to fix up what it told us. But equally, that will then give us an error at the previous layer. It can then assign blame, it can then assign blame backwards. By assigning blame all the way through the networks, essentially iteratively, you can then blame every single weight for the error that you got. So, if your inputs of the picture said this was a cat and it was a dog, basically you can say, well, what caused me to say that? And every single layer through this network you can fix up a little bit.  So you have to iterate this. And this is where you need a GPU, basically to make this thing finish. The other takeaway here is that these deep networks create features on these intermediate layers and even if you don't understand the structure of the problem yourself, it has often been observed that this will create features which are useful for solving the problem. The mathematics of this are not well understood. So if you want to show your boss, here's a model which will always work for these reasons. This is not the right room to be in. If no other model works, then this is the right place to be because these things often work in practice. Though in theory no one really knows. Okay, that was simple neural networks. That was deep neural networks, now on to images. One of the nice things about an image is you've got some organisation amongst the inputs. So each input here would be the top left pixel, then the one next to it, then the one next to that. But these are very much more related to each other than the temperature and the precipitation and random other variables. Images have coherence amongst them. So you know that there's this whole concept of up and down, left and right. You've also got maybe rotation you could do. There are various other transformations which are self-consistently or expansionally. Self-consistent within the image and wouldn't it be nice to use an operator which understood that? So the idea here is we're going to use a feature instead of just a single number. And the way which we use that is basically we say, well Photoshop knows about features. Photoshop filters enable me to manipulate an entire image at once in the same way. I can apply a blur, I can apply a sharpen, I can apply an edge texture, something like that. So very simple filters, but these will only be controlled by very few parameters. So you'd have a very simple what's called a kernel which you'd apply across the picture. In order to produce the next feature essentially you can just vary these nine numbers and you'll get a whole new picture out. You can produce multiple different pictures, multiple different views. And you pile these all together and you'll have overall a convolution on your network. These work super well and if you wanted to know exactly how well they worked here in the last three meetups. I'm sure we'll get back to images put today's text. So the one thing which we haven't addressed is sequences. So in the previous examples we've had a set of features that we knew or we've had an image of a certain size but a lot of real world data occurs in a sequence. So it's a way, whereas you know often have fixed inputs, lots of domains have sequences of stuff. For instance text you can think of a series of words or English text is definitely a series of words. Or you can think of text as just a whole string of characters including space. Or you could think of text as being a question and response and an email and a whole dialogue as being a sequence of different events that happen. Equally you can think of the audio which I'm spewing at you as being a whole sequence of CD quality 16 bit values. How would you deal with that? Equally video clips. So what you see from me is also a huge sequence of new events. So the question is what kind of technique can we do that you can apply to these again and again. Essentially by a symmetry argument you want to do the same thing again and again. You should have the same parameters again and again. So for processing sequences basically the variable length input doesn't fit the models we've had before. What you want to do is run a network on the inputs at a given time step and then that network will be used at the next time step and the next time step. It's all very well processing these as independent events but the whole point of a sequence is that they're linked together. What you do is you pass along a hidden state from one network to the next one. So this one will have an output which will then feed into the hidden state of this one which will then feed on to the next one, feed on to the next one, feed on to the next one. But the trick is though that this is all the same network just used repeatedly and even though we don't know what the hidden state should represent this thing will learn to have a nice hidden state or a nice internal representation because you incent it to do so because of this whole blame game thing. So if what you can think of it is in a time sequence but if I've got an answer at the end which I'm trying to get to I want to assign blame that blame will then propagate backwards through time it's as if I had a very deep network back to if I want to blame the very first word in the sentence as an input I will have a very deep network to deal with. So in a sense time is depth hopefully we can learn some features internally which are useful and it turns out that this often happens. So this thing is kind of the key point so I'll say that again again recurrent neural networks. So basically we have one network at every step of the input it has kind of an internal state that can be carried forwards each time step like a history but because each of these are kind of a mechanical operation of passing through and multiplying by numbers and adding numbers and then passing it forwards and adding by numbers up to the end at the end you'll have an error and you'll pass it all backwards you can well another word for this blame game is back propagation back propagation of errors is the derivative chain rule so basically because this whole thing is simple operations which you can take derivatives of and which things like TensorFlow will make it super easy to take derivatives of without even having to think about it because it's differentiable you can train this to do tasks and it will learn those tasks so here's a picture of the basic RNN it could be symbolised like this basically a network feeding on itself and we can think of it as a chain where these inputs lead to a network here with some hidden state it passes on to this next one this kind of builds up more and more hidden state inside so again each node knows its history all the weights are tied because this is essentially the same network and you've got network depth being time wise so this is a plain recurrent neural network what we have here this is the same diagram again but basically you start with a hidden state which could be just zero this is the uninitialised beginning pre-sentence for instance then each word you do each word that comes along however it gets into the network we do some multiplications, do some adding it passes on whatever is left over do the same basically you have this mess which is evolving as each new word comes in and at the end you say give me the answer and the answer could be is this a positive sentiment or whatever and if it gets it wrong you then blame everything for telling it the wrong information but what you do is you do this for millions of sentences millions of different blames and this should learn but the problem is that this has a gradient problem so the gradient problem is in a long sequence the early inputs are very deep in as much as the very first word if I said not a good movie the word not is kind of further back or not a good movie for families with young children now the not is a good movie for families with good children would be a nice sentence to have not a good movie, a very bad sentence to have so you've got to reach all the way back through this chain with these problems but each problem in this very simple version is just adding and multiplying and adding and multiplying, adding and multiplying so if you get any one of those these parameters wrong you're adding and multiplying by the same parameter again and again and again and again if you make a tiny error in that parameter the numbers will explode the parameters near one it's going to be fine but if it's 0.5 you'll have no error if you have 2 then the error will be exploding so this is a problem for this thing and typically people wouldn't be able to train very long networks so the solution to this is instead of always multiplying by a weight what you want to have is like a straight through path so I would want to be able to go directly from my error here to the not at the beginning cos that's kind of actually the most important thing and basically this would give me a gradient one path it would essentially transport my error to the beginning and what you'd want to do is be able to switch that error signal on and off somehow gating so this is what a gated recurrent neural network looks like gated recurrent network looks like unit looks like so basically here is this is one this is now a whole bunch of matrix multiplies some of which gate some of which pass along here is my input this is kind of the hidden state from before this is a kind of remembering this is a forgetting this is a tunnel state this is a whole mess of stuff built up to kind of gate the signal on and off and up here is kind of my straight through path so you can use this as intensive flow or in keras this will just be grew there will be a function grew which would do what you want so in some sense this is a difficult diagram in another sense you just need to know the words gr and u here's another one this is very very popular this is invented by Schmidt-Huber in Switzerland as you will say this is also extremely popular this is a long short term memory unit basically here is your input these are your x's these basically up here are kind of your hidden states but then going across here there's all sorts of multiplies and non linearities and stuff but here the only difference between this code and that code is the letters LSTM as far as say keras is concerned so ok this is phd here piled higher and deeper you can also you got these layers you can pile them higher so the inputs in the hidden stuff in one layer can be the x's of the next layer why not so if you want if you think you've got some good features this way but it could be more hierarchical then have another layer on top so we can just build this up that way or if you've got context which may affect you backwards suppose I'm seeking German where my verb is at the end and that could inform my thoughts at the beginning I actually want to run information backwards along the line so as well as having my English text with not a very good movie for families with children you know unless they're girls the girls thing really affects what you maybe so the key point even having done all of this with all these matrix multiplies and everything it's all still differentiable which means that you can train these things and which means that given that natural language processing is a sequence of words or whatever it's a sequence we can probably learn to do natural language processing using this machinery so text so that was part one part two, text text is very intuitive to everyone and that's kind of one of its AI problems it's not so obvious what's going on and I'm going to kind of point out how bad the situation is so for text you've got documents you've got paragraphs, you've got sentences, words, characters this is obvious in order to feed this stuff into a neural network of any kind you've got to preprocess it or into any kind of data science you need to preprocess it you've probably got a sentence split it you've got to tokenize it you've got to think about your vocabulary and what to do when there's exceptions for instance for the encoding can you even open the file so this is a this is a problem is your whole pathway uniclo clean like what character sets are you going to accept at some point are you transmitting it over something which will just basically throw away a lot of your data is E the same as E or whatever so here's a French symbol it can be written in unicode quite a lot of ways the question is if I have a Fabergé or something is that the same as a Fabergé egg right and that's a good big question for your dictionary how flexible should it be what about this this is a Japanese quote symbol is it the same as this quotes this double quote or is it the same as this double quote or is there any difference do I care what about this bullet point which is this is another this is all HTML this page so this is actually very very simple encoding it can get very hairy what about this one which is a nice one which is one of them is a ligature it's not obvious what's going on here so there's a special thing in typography called ligatures where the F and the I are actually connected together so I don't think it doesn't look as though in this character set they've actually made the distinction but in a better character set these two characters are joined together and this is a different symbol from F I this is a problem in finance because finance is a problem and profit is a problem there's all sorts of problems across all these domains sentence splitting so it's pretty obvious when a sentence ends because it will have a full stop at the end except that this sentence has only one full stop the rest is an abbreviation you've got a decimal point and like an acronym so but there are nice libraries which can do this NLTK doesn't have many uses but it can do this quite well tokenisation basically you've got a string of let's decide we've got a sentence now I want to break this into pieces it's a good idea to have just a single standard throughout your code base otherwise you'll be fighting yourself with spaces and commas and stuff there's a pen tree bank which is a nice tokenisation often used but what do you do about Chinese text when there are probably no spaces what do you do about Japanese which has a whole different idea about what punctuation symbols to use another thing which will hit you is lots of this research is very English centric so this is one of the reasons why having an NLT based style up here is a different animal from one based on the valley just because we actually care about these things already and here is a this isn't so easy in fact this is tokenised like a pen tree rank it does have a space here because that's the way it should be in a pen tree rank tokenisation anyway vocabulary suppose we've converted our sentence to tokens so what we want to do is we build a dictionary we convert the tokens into numbers so now we obviously want to be in a numeric domain eventually and a very simple frequency analysis will tell us what are the stop words which are extremely frequent words which have fairly little semantic value it could be of this the a a lot of what I say will be stop words but you'll only hear stop words and say I mean listen then there's common normal words which will be the vast bulk of it you can probably get up to like 10, 15,000 words which will just all be common and then you move into the domain of pretty rare words they'll be infrequent will be fairly infrequent you can imagine all sorts of TensorFlow will be very infrequent just because it's a fairly new word they'll also be typos and junk in any given text they'll be junk and then you want to have a special unknown symbol which is often called so understanding text so now we've got it in some kind of manageable format basically a string of numbers or a sequence of numbers English we probably want to have a vocabulary of about 100,000 words as kind of your minimum sensible size people go up to like 2 million but then you're just grappling with web addresses and just junk you've now made the junk promoted that to being like officially a word and two simple ways of doing this is bag of words and the wording beddings so bag of words is you convert a sentence into a set of words you basically throw away the ordering of the words because it's irrelevant to you and you'll do a simple statistical analysis this is basically how surprising is every word in this sentence compared to the rest of the documents doing it this way is what people have been doing for the last 30 years it's surprisingly effective and many people who would claim to do a natural language processing AI are only doing this because it works well enough to fool you until you're familiar with it but because it's got no idea that jumps is a different word from jump it has no idea that jump is similar to spring it has no idea that spring is similar to summer or spring is similar to winter and that winter is stuff to do with jumpers right and jumpers is to do with jumps so another way to do it is to go for wording beddings there's a major advance word to veck, glove, these things came out since 2010 maybe 2012 the idea is that words which are close in the text should have representations which are close but at the beginning we start off with just numbers now we want to convert it into a vector so each word will have a 300-dimensional vector and the way in which we generate this is you slide a window over your text and everything within that window we then nudge towards each other so this nudging process is kind of like a backpropagation idea but the nudging if you do this enough and where I say some text we're talking like a billion words so if you Wikipedia would be a good starting place so you do this enough and you do it multiple iterations these things will converge into something which is surprisingly interesting so what I'll actually show a little example where it shows you how these things will self-organise into a very interesting space whereby things which are similar to each other will align and things which don't have much to do with each other will not so there are also other things which you can do to do with the geometry of this space which is even more crazy but doesn't seem particularly valuable at this point so text demo time let me go and oh so that's me okay so basically so all of this sorry the reason there's a picture of me there apart from it being excellent is here is my github repo and in this github repo there's a thing called deep learning workshop deep learning workshop has all of this code there's these notebooks which you can download if I've made it so that these if you download this code you can run it it will pull in everything it needs it's all right there so there's no secrets here you can run this on your own if you want to or I encourage you to run this at home okay so it also has some comments which I won't bother reading basically here I'm pulling in it's a long way away I'm sorry basically there's a thing called punked from NLTK which enables you to do sentence splitting and there's some examples of how it actually does that this is a thing where it actually shows you how it can tokenize a little sentence I've also got a corpus of Wikipedia right there so basically I've got a very small corpus of Wikipedia it's just the first 100,000 sentences I think this actually allows you to this spreadsheet allows you to actually create a glove embedding but I'm going to skip that I think because because of time one of the things you'll learn from 100,000 sentences even if you run this glove embedding for lots of iterations and this takes maybe 30 seconds it's not 2 minutes I think it's not too bad you'll get a very bad word embedding a better way of getting a word embedding is just to download one off the shelf and off the shelf means people have published these it's a big download but once you've got it you've got a nice word embedding so I may have to actually run some stuff now just stop me I don't want to do that I don't want to do that word embedding is poor sorry excuse me okay so basically I've downloaded a word embedding so this I've just taken the first 100,000 words of a glove embedding which is a 50 dimensional embedding so this is actually a fairly small embedding and I can ask there's a simple call I can do what is the most similar words to king and it's telling me prince and queen and ai ai whatever that is an emperor so it sees that emperor and these things appear in the same context all the time equally I can then test analogies so this is the geometry so this immediately shows you that man is to woman as king is to queen, daughter, prince and throne queen is the first selection it likes that so this is this word embedding has actually captured a huge amount of not just a similarity within language it's actually captured something about relationships the next one is parises to France as Rome is to Italy so it's actually captured geography from just reading Wikipedia and doing a simple nudging kind of thing across a window kitten is to cat as puppy is to dog another thing about animals relationships and understand is to understood as run is to ran so this is actually picking up some kind of crazy grammatical thing just by reading just by reading and reading and that's it so what I can do if I'm let me just check I'm running it what I can do here is I can pick up that there is a nice thing called tensorboard so tensorboard is part of tensorflow basically it'll let you graph out your your data but it'll also graph out your embeddings so this is the word embedding cloud formed by which I've just loaded in and I can look around in this cloud it's kind of difficult to see what's going on but if I type in say king and there we go and let's just show those points but at that so here's a tensor here's where all the kingish words are so you've got queen over here father, imperial so this is kind of interesting that it's actually can get this out of pure data it doesn't actually know anything about the English language apart from where to split a sentence or whatever it can do this by reading Wikipedia so you're getting all this stuff for free so let's kill that and I might actually kill this too okay so that was a text demo so what we've understood so far is and you can prove it to yourselves you can tokenize stuff we can convert it to word embeddings which is strongly the better way to do it than TIDF now so now we've got basically each of my words corresponds to a 100 dimensional or a 50 dimensional sequence of numbers so I can just take that as one word then I have another word another word and another word and I can run a network of some kind across these numbers and I'm entirely in a matrix multiply kind of domain so let's try an application of this the application here is recurrent neural networks for text and to build a quality natural language processing system like I have been for these kind of people over here an essential component of this is named entity recognition which I will call NER just for elegance and this has to be flexible and trainable partly because we're in a region where there's lots of new stuff coming we need to train it specifically differently from the guys in the US to particularly there are quirks here so for instance in NER terms in Singapore it may be fairly rational but in Malaysia you've got the datos at all something something something that's one whole thing Chinese names can be rearranged in strange ways and this is just for people on the other hand one thing which the American systems will be robust against is that Richard can be called Rick or Dick or Charles can be called Charlie or Chuck or Chaz anyway there's a whole bunch of different things which the Westerners do which aren't necessarily done here but on the other hand systems in papers typically are orientated and corpuses orientated to the kind of Western case so here's a quick example what NER is we want to transform soon after his graduation became managing director of Lam Soon so you can see the problem here right we want to know that Jim Soon is a person and Lam Soon is an organisation and the key kind of things which you're watching for is his graduation kind of means there's probably a person coming up because the word became is something which was more of a person thing than well maybe it's more of a person thing than corporate thing but managing director of is almost always followed by Lam Soon now I may be managing director of my daughter but apart from that managing director is a company so there's kind of sign but you can see that there's signposts in here which you could get just from the data if I aligned every sentence in a huge corpus with managing director of something all of them would agree that the next thing was probably an organisation except it could be managing director of a company he founded which ok then you'll have to give that a pass right so the question is can we train a an RNN of some RNN is kind of a generic term can we train an RNN to do NER so what we want to do to create or probably just download a word embedding we'll get a NER annotated training set so either it'll be human annotated so someone's actually gone through and said ok this is a person thing, this is an organisation or location or a date or we created in some way by labelling which is the basis of a paper and then we just train an RNN on this data set and then see whether we get any decent results so this is going to be a demo so the human annotated corpuses or corpora are difficult for me to distribute so I do this workshop thing where I hand out USB keys but I can't do that with any official corpus because these are expensive and difficult to license so what I'm going to do is kind of cheat here I've got some Wikipedia 100,000 lines of that and I'm going to use NLTK to annotate it as if NLTK was the truth which it isn't but I'll try and then I'm going to train the RNN on the machine annotations and I'm going to just look at how well it performs compared to its trainer which is NLTK so this seems like a fairly fair baseline but in order to make it more interesting I'm only going to let the RNN look at text in single case so I'm going to transform all the text into upper case or lower case but the trick in that is if it works it will be great because I know that NLTK will totally fail if it's all lower case because NLTK really relies heavily on whether the word is upper case or not so here's a quick network picture but this is something we can all understand now I'm going to use a bi-direction or GRU or current neural network here are my X's but each X is going to be a vector from my embedding you'll go up into two different networks and then they'll be combined to which this thing will be a label at the top is this a NUR or not and going this way is basically the hidden state in a forwards direction and this one is going in a backwards direction so this kind of lets me if I know I'm starting a name I can get to the end but if I know I've found the end of a name I can get back to the start basically it's good to have two ways so this is another there's another thing in the notebook another notebook in the workshop which lets you do this and I know I'm running low on time and that's basically this is something I almost got time so basically this is a a tagger and I'm just going to pull in the same tools which I had before I'll pull in the corpus like I had before and a reference tagger which is the NLTK thing so this can do some job of actually annotating parts of speech this is let's see what part of speech and how it's only simple to looks like so this identifies where the nouns are it does quite a good job of actual part of speech thing but I'm actually going to make this all lower case and I'm just going to learn where the nouns are basically I'll load in some glove and I'll fix it up so that the word embedding is Keras compatible because Keras, which is what I'm going to be using here is a layer on top of TensorFlow which means that instead of specifying every little matrix multiplication I'm going to do I can just say grw or I can say bidirectional and it will do just the right thing to construct the whole TensorFlow graph so I think you're going to find that a lot of more of our stuff becomes Keras focused because it just makes things easier and if you're starting out Keras is probably the way to go so this is fixing up this to be like a Keras a thing so here is kind of the magic so basically this page is taking the tokens input at the top which is my sentence length of stuff and then it's converting this using an embedding into a sequence of embeddings and then it's doing a bidirectional grw and then it's going to concatenate them and at the end of it it's going to do some kind of dense thing for the softmax so this defines the entire deep learning experience here but you've seen how much extra stuff you've had to do because this is text because this is text is quite tricky and Keras will helpfully give us here is how many parameters are involved we've got 5 million parameters in the embedding the model will only take 30,000 parameters so this is actually kind of a small model and now we're going to do the training of the model this is another feature of the workshop is that these things should train 5 minutes or less so I will now dance so this is going to do a thousand epochs each epoch is I think 64 sentences so this is going to read 64,000 sentences and run them through this so we're now at 150 of a thousand and it's going to finish so this is a nice feature of Keras in that instead of running big training loops this is kind of crazy stuff basically there's a nice progress bar and this is very nice and it will also enable you to do graphs out for tensor board it has some very nice features so we're just going to let this thing we're at 300 now so this is being trained on my laptop my laptop's got an i5 CPU it's got 8 gig of memory it's not like a super duper laptop it does have an internal graphics it has a GPU graphics card but basically it's switched off because GPUs are like special petals when you update your machine everything stops working so if you've got a desktop maybe it's worth it or a server worth it this laptop I'm not going to touch the GPU it does work but it makes life very different is there a way to if the error is input as a point will it stop? so what you can do is in here you can add something called callbacks and you can pass it an array of callbacks and at the end of every epoch it will do your callbacks and one of the callbacks is like keras.earlystopping so you can add in early stopping criteria you can add in different things to look at how you're doing basically or save my state or whatever so you can do all of that kind of game but also there's in part of this there's also basically you can define what loss functions you are what optimization functions you are so you may have something which also flags up whether it's going backwards on itself kind of thing there's a whole bunch of variability keras is all of the all of the defaults are good basically they've read all the papers if you just put in the standard stuff it's probably about middle of the road whereas in TensorFlow raw you may forget a parameter and it's you're actually way out of left field and everyone's decided that drop out should be a half or something or something anyway we're now at epoch 1000 we don't have to save these weights don't have to load them I've got a little function and here we have some output so I've got it's very difficult to see I'm sorry this is fighting me so I've got some sample sentences here and the first one is well three sets but if you look at them they're slightly different so the first one is a pure sentence which is Dr Andrew's works at Redcat Labs then let's see what part of speech analysis looks like when are you off to New York Chytania one interesting thing about Indian names is they're all kind of unique so what this is showing here on each word basically this is what NLTK says so NLTK thinks that Dr Andrews is a ner and this RNN thinks that Dr Andrews is a ner and it also thinks that Redcat Labs is a ner so both of them agree here which is good NLTK thinks that New York is a ner which is right I just think that York is a ner so that's not quite right Chytania they both pick up so this is you can see that basically one has learnt from the other you wouldn't expect the RNN to be better than the better than the NLTK one because it's teacher he's only teacher has been that but if you now go for this one where it's using proper case every word starts with a capital letter Dr Andrews works so it thinks that the first name is Dr Andrews Works Redcat Labs is fine so it thinks that speech analysis is a name so this is NLTK so NLTK has all of these problems whereas in fact the RNN because it did all its learning on the lower case version the difference between these sentences so provably it learnt to be almost as good as its teacher but it's actually superior in many many ways in that you can apply this to text which the teacher totally fails on so there is a kind of section let's look at the statistics but it's hardly worth it because the teacher is just so much the output is so much better so here's when I wrap up text processing is messy and the vegings are magic RNNs can be applied to lots of things text is what we're focusing on now and having a GPU is very helpful because I just did this on 64,000 sentences but our typical training run would be 100 million sentences which is kind of like an 8 hour job an overnight job on a GPU there's also you've seen the workshop thing please add a star I like that I will take questions but before that I'll ask you questions there's a feedback form that Google would love us to fill in and it's at bit.ly tf minus sg tf-sg if you fill it in before the end there will be prizes if you fill it in after the end no prize for you so there we go and I will take questions I will not accept friend requests but I will accept LinkedIn okay, Karthik do you want to so I'll stand here he'll do the next


Okay, I'll start now. So thanks very much. Welcome to this talk on debugging with LVM. My name is Andrzej, this is my colleague Graham, and I'll try to showcase technologies within LVM that will help you with solving various problems. Both Graham and myself are compiler engineers at ARM. We work on the scalable vector extension support for R64. None of the technologies that we'll be describing today is something that we work on. It's something that we use on daily basis, something that we very fond of, and we feel that it does deserve a bit more advertising. So this presentation is split into two parts. I will introduce and talk about LDB to begin with, and then Graham will jump in and he'll introduce you to LVM sanitizers. LDB, one of the nicest thing about LDB is very compatible with GDB, and I think it's important because a lot of people are familiar with GDB and they are concerned about switching to LDB. And this is apparent when you realize what the architecture of LDB is. So, up a case LDB is just a sub-project within LVM. Now, that sub-project implements a bunch of binaries like lower case LDB, lower case LDB server. And what happens, as a user, you'll be using LDB typing commands, but behind the scenes, it's actually the LDB server that does all the heavy lifting like memory, read, memory, right, register, read, register, right. And the two communicate with each other using the GDB remote serial protocol, which means that, in principle, LDB server implements GDB server and can be a drop-in replacement for GDB server and same with LDB. In principle, it works, I tried it in certain scenarios, but obviously there will be some clashes, but still, I think it's fairly obvious that they try to make it compatible with GDB and I think that's important. If you're interested in the GDB remote serial protocol, it's fairly straightforward. Every packet starts with a dollar sign, followed by data, followed by a hash sign and a checksum. And LVM actually extends that because you can do that and LDB works on a lot of complex platforms. But what's interesting from a user perspective, you can use lock-enabled GDB remote packets with an LDB and see all the packets being exchanged between LDB and LDB server. And that can be useful because if you're on a new platform and something doesn't work and you're wondering why, then you can use that to see where the packets are being exchanged and unknowledged or if you just want to see how, for instance, registers are being requested and read. So this can be just a fun thing to use. And in fact, LDB has a lot of logging channels and you can check them by typing log list. And the list is very long. And this brings me to a structure of LDB commands.  Another example is process launch, which basically means study or debugging. These are a bit verbose, but they're aliases, so you don't have to worry about it. And I'll show some examples. If you're stuck and you don't know what you're looking for or you don't know where to start, you can just type a propose and a keyword. And keyword doesn't have to be a command. It can be something just fairly random and LDB will display your list of help pages with relevant info and then you can look the corresponding help page. But I think it's much more interesting to see this stuff in an action. So I have a few examples. The first one is it's not complex. It's basically a main function that the class array of names, and J Graham, and J Graham, and loops over all the names and displays either hello and J or goodbye Graham, depending which name it sees. Or I am confused if it sees something else. And if I now build it, oh, sorry. LDB, LDB example. Yeah, sorry. Sorry for that. So yeah, it goes and hello and J, goodbye Graham, because that's what's implemented. Now if you run it in LDB, yes. When you type and always say what you're typing because I don't see it. I'm only interested in the LDB commands. Yes, so I'm starting LDB and I'm typing breakpoint set dash n main. I don't need that. So that's the verbals way of setting a breakpoint. I'm setting a breakpoint. I'm running my process and it's displaying the assembly which is not what I wanted because I didn't build the debug info. So let's do that again.  It highlights where you hit the breakpoint including row and column, which is useful because sometimes you can have multiple expression in one column. And now if I just step through and I'm typing n in LDB, which is an alias for stepping over which is compatible with GDB, it's re-displaced the same stuff and highlights where exactly on which line you're stopping. Now, but what I really like about LDB is the expression evaluation engine. And you can basically type any C++, more than C++ code that you want. So now I'm overwriting the contents of the character array. So I'm just typing X for expression and then you type your C++ expression. In this case, for auto name and names, names set to pinky, which is just random name. And now you can see that it has overwritten it. So it jitted the code and it did something behind the scenes and now it's changed. And if I continue, it's confused, which is fantastic because sometimes, and I do it all the time, you might be tricked into, okay, I'm gonna modify my source code, recompile, rerun, and then debug it and see what happens. You don't have to do that. LDB leverages lip clank and has very access to very powerful JIT so you can do stuff like that live. Let's try another example. This one, the source is not very important. So I'm trying a different example. This implements a multi-threaded, very simple multi-threaded increment of three numbers. So it increments X, Y, and Z to some very big value and it does so in three threads. If I do it, if you look at it in LDB, so this time I'm using just B to set the breakpoint because there's an alias. I don't have to do breakpoint set, B is fine. So that's something you know from GDB. And I'm setting a breakpoint on function in X because I know I implemented the file, so I know it's one of the methods that's executed in parallel that does the increment. And again, it displays the relevant code. It doesn't fit on the screen, but you can see the breakpoint is being hit over there. And because this is multi-threaded, I know there will be multiple threads stopped. So I'm typing thread list and indeed there are three threads. So I want to know what's happening with these three threads. And for that, I think it's really nice to actually use GUI, which is Graphical Use Interface. So I'm typing GUI in LDB and it takes you here. And you get this for free out of the box and it just works. Main window is your source and highlights where you hit your breakpoint. Your variables are displayed here from the current frame and there's a process and that one is the most interesting one because you can expand it and you have three threads, all fine. And then you can jump between the threads and it shows you exactly where in each thread you are. And you can also go over different frames within each thread, which I think is fantastic and really useful. Another feature is embedded Python interpreter.  And you can type anything basically you know from Python. What I'll show you is an example of using LDB Python bindings. So you can do something like so I'm typing print LDB target get triple. And if you're familiar with LVM, you'll know that triple is something that LVM uses internally to describe the platform you're on. So I'm just printing that using LDB Python bindings and it's x86, 64, macOS. There's many more. There's many, many, many more. And as a matter of fact, you can use those Python bindings to script your debugger and extend it. And in order to present that, I will use a, I have a hobby project called LVM Tutor. It's a collection of passes. And if you're familiar with LVM passes, you'll know that each one implements this run method. But it's either run on module, run on basic block, run on function, run. And then you have the new pass manager, old pass manager. And if something breaks, I just want to start LDB and hit a break point and I don't think which method it needs to be. I want to be just like, just set a break point. So let's try. So I'm starting LDB and this is a long command, so let me just quickly explain what it does. I'm debugging opt, which is LVM's tool for running passes implemented in shared objects. I'm loading some specifying which shared object I want to load. So it's this one and that's something I implemented. And then just bunch of options to specify which pass I want to run. So this is a bit complex because I'm actually debugging opt and my pass is dynamically loaded, not yet because I haven't started yet. And I'm thinking I will be lucky. So I'm setting break point. I'm typing B run, run on module. And I'm thinking I'm being lucky because debug is a clever and it finds 73 locations. Makes sense because I'm debugging in principle opt which belongs to LVM, which has a lot of passes. So there'll be a lot of run on module methods. But I know it hasn't found the one that I'm interested in because I do image list within LDB and it tells me which shared objects have been already loaded. And you have a full list and you can check the pass hasn't been loaded yet because I haven't started opt. Anyway, I'm being naive. I'm thinking I'm being lucky. I type run and I had break point inside my pass. So automatically behind the scenes will happen opt started running. It loaded the pass, the shared object at some point. LDB notices that and goes, right, you're loading some new object. Earlier you requested some break points. I'm gonna investigate this new object and see maybe I can set a break point in that object. And there you go. And it just works fantastically behind the scenes. Now you may be taking this for granted but few years ago stuff like this didn't work. And it's quite impressive that it just happens to work. And this worked and that was just one pass. Now I'm doing exactly same thing but with a different pass. So it's just a different shared object and different command line options but otherwise I'm doing the same thing. So run on module exactly same exercise found 73 locations and run and it's not hitting the break point. The pass runs, displays the output from the pass but the break point wasn't hit because the method this time is not run on module. However, I have this method that I implemented in Python which is not complicated called LVM Tutor because that's my project and that simply passes the command line that I passed to LDB. It has access to that via LDB Python bindings. It's not really magic. And it identifies okay you load it hello world and it knows that because it just checks the name of the file and sets a break point accordingly. And it knows where to set it because I told it like I have a map of pass and which method to stop on. So if I now type run it again automatically set the break point and it hit it. And you can do a lot of things like, oh sorry, I removed this one. But let me show you how I implemented that method. So there's two steps. The first step is dot LDB init file that's loaded when you start LDB and it has just bunch of things that you want to do whenever you start LDB. First the commands it's settings set again a noun followed by a verb and it just says display 20 lines of code around where you hit the break point or where you stop. So that's the setting I use on my computer. The other thing is command script import commands.py. So this basically says take that file and load it. And finally on this line I'm just saying take the file that you just loaded, take method that implements my custom command and alias it as lgim tutor. And that's it, that's done. And that's more or less what I had on the LDB. This is a few aliases available within the LDB but I actually copied some of them from official documentation of LDB. I recommend you go there, the list is very long and there's a lot of aliases like this. I show the expression evaluation engine. We talked about Python interpreter and implementing custom commands. Now, there's a few links. So LDB tutorial available on the official page of LDB is amazing, there's a lot of examples and it's written very, very well. So I recommend visiting that and trying to follow the examples, GDPRSP protocol was really, really, really well described over there. And if you're interested in my examples then you can also look at them. Now, there's a lot of things that you cannot do with a debugger and you can deal with them using a sanitizer and that's something that Graham will talk about. Okay. Right, so what is a sanitizer? It's a feature in LLVM and GCC as well that allows you to build your program with additional instrumentation code that helps you find bugs that you didn't know were there. And it's pretty easy to start using in that you just need to have this extra command flag on your compile line of dash fsanitize equals and then the name of a sanitizer. And there's several sanitizers available for things like figuring out whether you've come up with an invalid memory address and try to de-reference it or if you've invoked some undefined behavior from C and it performs this instrumentation by wrapping key operations within your program. So things like loads and stores, it will call off to an internal routine that it's linked to and build up a map of memory showing which addresses are valid and generating a log of where a piece of memory came from. And if you run top when you are executing one of these, you will often see something like a virtual memory size of 20 terabytes which looks a little alarming at first but it's not really that big. It does have an overhead of somewhere between three and 15 times space wise depending on which sanitizer you use and which Hopkins you're going to use but generally it's not that bad unless you're going to do something like launch six of the electron apps at once and have them all instrumented, you're probably not going to run into any problems. There's behavior that's tunable when you encounter an error. So the default is to just report that an error occurred and then keep on going and hope you make it through depending on what it was, you may crash anyway. There's an option if no sanitizer recover which says no, as soon as you see an error, I want to stop and then if you're just using the undefined behavior sanitizer that doesn't need the extra tracking of memory. So you have the option to just say plant a trap instruction and I don't need to link in the compiler runtime. These can be combined to some degree so if you wanted to track whether you'd hit a signed integer overflow but then definitely stop if you walked off by the edge of an array, you can do that. But for the address memory and thread sanitizers, they all want to control the heap and control all of malloc and stuff so you cannot use them together. So let's go into a brief example of what we can actually get out of this. So I've got a simple example here over two files and I suspect a few of you've already seen what the bug is but if I compile this, so we're kind of expecting a zero here because we're summing up all the things in the array or the integers in the array and we had initialized them all to zero. So something's gone wrong. If we use the address sanitizer and rerun, it'll now show you that, yeah, you've gone off the edge. So what it shows you is that you've got a read size four, so one integer, from your my loop and that memory, the memory it's closest to was just a 40 byte region. So you're 10 integers times four bytes which was allocated in main and you've hit the red zone region just beyond that. So it basically, what this simple example shows is that this isn't just instrumenting your local code within a function, this is a global kind of bounds checking mechanism that memory that's been allocated in one thread can be protected and guarded against overruns and, well, in one file, can be protected against overruns in another file. So the address sanitizer detects how the bounds access is like that but it also picks up use after free or after scope or double free. On Linux, there's a default, well, it's on by default. You can detect leaks in your program and you can also detect initialization order problems. So if you have init functions for your various libraries and they might reference memory that hasn't been set up yet by another init function in another library, it can detect that kind of problem. So the undefined behavior sanitizer, is a class of behavior within the C standards which is not implementation defined so that it has a known thing but it depends on which vendor. It's anything is valid if you do these cases. Things like signed integer overflow is a case of that and in theory, deleting all of your home directory is a valid response to trying to run that code. In practice, the compiler generally just deletes some big chunk of your program and things go subtly wrong. It can also catch very similar cases that are not technically undefined behavior so unsigned integer overflow but are nevertheless useful to catch because you generally don't expect a very large number to overlap to a very, very small number or the opposite. So we were trying to find an offset between eight and 32 and that's not quite correct. So in this case, I'm showing that there's an additional tool beyond just saying a sanitized equals undefined that you can specify an additional behavior that has just to track that kind of thing. So it's a long list of all the undefined behaviors and close to undefined but undesirable behaviors that it will report to you. Thread sanitizer. So this is obviously your thing to report data races. An example for that as well. Okay. So the thread sanitizer in this case is reporting that we had a data race between the threads on ready and on item in that it will tell you which threads wrote to it first, which one wrote to it second and that there's no synchronization between them. So it catches data races like that and that includes data races on mutexes which you'd normally expect to be okay because sometimes you might have split off and then one thread tries to initialize it but before it's done so another thread is already trying to lock it. It catches problems where you're trying to destroy a mutex say if it was a scope one while it's still locked, catches problems where a signal handler might overwrite Erno. So if you had an asynchronous signal come in you'd probably don't want it to overwrite whatever area you've got coming back from a file open. And in cases where thread sanitizer doesn't yet understand what it's looking at you can help it out a bit. So if you were to use open MP and you'd try and run that simple example and you'd deliberately put in an OMP flush which acts as a fence and basically stops everything from crossing that barrier. Thread sanitizers doesn't yet understand that. So you can tell it that you could use the annotations to tell it that the right to item happened before this flush and that the right to ready happened after it and similarly in thread two. And if you do that then it will actually understand what's going on and say okay that's fine and keep on hunting from bugs elsewhere in your program. Memory sanitizer, this is to pick up issues where you haven't managed to initialize a piece of memory. So in this case some old program understood a couple of command line options initialized who in those cases otherwise didn't bother. And then at some point somebody adds a new flag and something goes wrong somewhere down the line. I'd love to show you that one but unfortunately that one doesn't work on Mac OS. So it's Linux, FreeBSD or NetBSD only at the moment. It might be the other BSDs but I don't think they've been properly tested. And similar to the other ones it can track the origins of memory so that it knows where it was allocated and where it might have been partially initialized but not all the way. So you get an idea of where you should focus your attention. So say you've started to use the sanitizer on your program but it's just too much overhead. So you've got some really big compute happening in the middle and you're pretty sure that that bit's correct and you want to focus on finding bugs and things like your parser because parsers usually go wrong. So you can add an attribute in your source of don't touch this one with the address sanitizer or don't touch it with the thread sanitizer or something like that. But trying to add all little attributions all over the codes as you find that they've hit too much CPU time or memory is maybe a bit too much. And so you can actually just give it a file and say for this source file I don't want you to instruments funk one with anything. For this other one I don't want you to annotate this with the address sanitizer. Unfortunately C++ names have to be mangled. So that's why you've got this Z9 thing. But it does allow shell wildcard of star to match everything with that prefix. More info. So I have definitely not covered all of this yet. It's a few other useful ones. So point of compare and point of subtract. If you've malloped two separate objects and you try and compare pointers on them that's undefined. Of course they make me completely different memory spaces you don't know. Control flow integrity. If some sneaky bit of user data is trying to overwrite your stack pointers and get you into a bit of memory you're not supposed to. You've got the CFI to catch that. The data flow sanitizer is a bit of an interesting one because it's not an automatic one like the others. In that you have to manually tag pieces of memory with a name. And then it will keep that tag wherever that value goes through memory. And it's quite interesting to see with the age of a debugger. And there's more in the process of being written. So the type sanitizer is currently under review if anybody wants to go and look at that for caching strict aliasing problems which is yet another class of undefined behavior. There's the main clang documentation site which contains a lot of useful information on the sanitizers and how to invoke them. And there's an older set of documentation from Google. Some of it's out of date but it still contains some useful information. And I have seen people set these things up in public CI like Travis so that they get at least weekly builds verifying what they've done. Back to Ange. Yeah, I'll quickly wrap up with a one final example because you might be wondering, right, LB sanitizers, it's in a way connected but a bit orthogonal. Can you marry them together? And yes, the answer is you can. So I'll try. Right, that's the wrong directory. Yes. And very trivial example, basically allocate memory, deallocate memory in F1, F2 and then you try to write to that memory after the allocation. Yeah, so the sanitizer should capture that and it does. Yeah, so that's something that Graham showed you. Well, it's a slightly different example but the concept is the same. Now, you want a bit more insight. You want to do it within LDB. So let's start LDB. I'm typing B main. So setting bake point on the main function, run. I'm stepping over F1 and F2. That does allocation, deallocation. And I hit the point where I'm trying to put something into the array that has already been deallocated. And you type memory history, name of the variable and it shows you the stuff from the sanitizer. It shows you, well, I apologize, it's not displayed very nicely because it doesn't fit on the screen but basically it shows you the back trace of allocation and deallocation. So you have the insight from the sanitizer within LDB and that's all wrapped up into this memory history array. You can have a look what it is by typing help memory history. So this gives you access to the address sanitizer. You have access to the other sanitizers as well. It's not wrapped up in nice commands but you have access to that. And otherwise, I think I will just finish here. So most of all, thank you very much to the community because this is stuff that the community has developed and it's really nice and we can use it on a leverage to be more productive. We rushed through a few things so if you have questions afterwards, obviously we can have questions now but feel free to email us with follow-ups. And otherwise here, thank you very much. Yes, so we have around five minutes for questions. Say, remind me, present us to repeat the question. Yeah? My question would be, I'm not using LDB and what would be main motivation to switch for it? Like, are there additional features which you would promote and advertise? So the question is what would be the reasons for switching from GDB to LDB? That question comes up a lot and my answer is it would be best to ask a LDB developer. My personal take is I don't have such thing. I switch between GDB and LDB and the way I look at it, they are complementary. If you're on macOS, GDB is suddenly not as well supported as it could be, so you can only use LDB. LDB, by default, you have this nice interface that displays you context. It leverages lip clunks, so you have C++ expression evaluation out of the box and whenever something new comes in at C++, you get it immediately as soon as LVM gets it. I don't think you get that in GDB. Also, you might be on some exotic target for which there's only LDB implementation or you might be on a target that has only GDB implementation. So I think the way I look at it, it pays off to know both tools. LDB doesn't depend on that. Okay, okay. But I thought because you have LDB and LDB server, GDB and GDB server architecture, that kind of removes that dependency in the GDB as well. GDB, that's product city six. Okay, okay. Yeah? Sometimes GDB has a very hard time of analyzing the variables in the local stack frame and most of the time it's not that big of a deal because in the worst case it says I don't know it, but sometimes it actually starts to hang and then basically I have to stop the box session because GDB just stops this hang. And so the question would be does it, is LDB better in that respect? I mean, I'm not necessarily talking about resolving everything but at least not crashing so I can at least start to do a different introspection of the one that currently doing that. So I think the question is in certain scenarios they'll be more stable on reliable than GDB in particular when you have heavily templated code. My personal experience, for me both GDB and LDB are very stable. For what I've been using. In the past I admit I was switching between LDB and GDB and LDB was very flaky but that was three, four years ago. I'm using it now on daily basis for debugging LVM and it's rock solid. I'm really impressed. And the scenario that you're describing heavily templated code, well that's tricky for debuggers because one stuff is compiled and you request a template that it's not, it hasn't been instantiated. So there's been developments within LDB to actually make that work better based on modules. And I think patches are being upstreamed. There was a talk at LVM Dev, maybe that's something would be relevant for the scenarios you're describing. But my personal experience they in part, but maybe my, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah. I've not tried it but I think it would be interesting if you compared and came back because I'm interested in myself. Yeah, one more question. What about something called hardware ACM? Is Arnold helping out with accelerating complicated to return types? So the question is about hardware sanitizer and I'll just pass the gram. So there is a, a hardware assisted sanitizer. It's mostly developed by Google. I think with ARM has contributed a little bit to it. But it basically takes advantage of the fact that ARCH64 can reserve the top end bits of a pointer for any other purpose and still be able to use that pointer as a plain loading store. And I think that's used to serve the address generation for stack variables. But you also have an upcoming feature of memory tagging which should do a lot of what you get in the address sanitizer for free by coloring memory and storing bits of, storing that data of what color that memory is elsewhere in memory such that the hardware can actually detect overruns for you.