Okay, thank you for the introduction, Raphael. First, I would like to thank the INCF for inviting me a second time. To get rid of my opinion. And maybe the title is a bit general. Maybe I give first a little introduction about myself. Why I think I'm qualified to talk about it. And also, to bring up the point that I was educated at an institute for neuroinformatics. At the Universität Bochum in 1993 to whatever it took to do my PhD with some intermediate stations. Then I was actually 13 years in industry. I had the luxury still to continue my research in computational neuroscience. And in 2011, I joined the Blue Brain Project in Lausanne. I guess most of you have heard about the Blue Brain Project. Why I was recruited there? Not so much because I was a strong believer in it. But rather maybe I'm a little bit of a critic to it. And some of it will probably come out during the talk. Very early on, I started together with Markus Diesmann to develop a simulation tool called NEST. And I'm mentioning it now at the beginning because some of what I will be saying will refer back to this simulation tool. Although, don't see it as a commercial now, even though you could. So what is NEST? NEST is a simulator for large neural models. Means you can plug together many neurons into networks and investigate whatever favorite model of processing you have in mind. Typically, you use it by programming it in Python, which is programming language. However, the actual computational expensive part is done in C++. So it's compiled. And one of the interesting points about NEST is that it also runs on supercomputers. So you can scale up your model to huge sizes and still be able, if you have the computing time, to execute it. NEST has undergone a long history with many changes, as you see here. And one thing that I would like to point out, and we'll come back to that point later, that if you look up NEST, there are only very few papers actually about NEST as a tool. This is very deliberate because I and also my colleagues, we don't really think it's good to publish anything about tools. Rather, you should publish the methods that are used in the tools. Because a paper about a tool is outdated within one year, two years, I don't know, it doesn't really make sense. For that, you have the internet, you can have a web page, whatever. But publish the algorithm, that was always our philosophy. So you will find a lot of individual publications, illustrate or describing the methods we use inside the simulator. Okay, cut, yeah, no longer talking about NEST now. I will go back to the very roots, David is not here. So maybe you should have seen it anyway. I will go back first and ask ourselves what is neuroinformatics anyway? It's one of my favorite questions because I grew, ah, there's David, okay. So I will go back to the word informatic as we use it in German. It was coined by a pioneer in computer science called Carl Steinbuch. So he is the founder, he coined the word computer, he coined the word informatic, which is synonymous to computer science in German. And he's also the father of one of the earliest neural network models called Lernmatrix, learning matrix, which is an early form of an associative memory. And because it was developed in the 50s, he actually proposed a hardware implementation. So he is also the founder of what later became known as neuromorphic engineering. And he also postulated a resistor with memory which he called Mimistor, without an R, roughly 10 years before Leon Shua proposed his memory store with the R with the same properties. This is an illustration from the Lernmatrix paper. So this is very much the same notation that is used today for associative memory models, some of you might know. So this is the patterns, we called it gestalten, that you could apply. So there's an eye, an auger here connected, and then you could build associations between patterns that you put in and patterns that you read out. And here is sketches of the implementation of this, using ferromagnetic elements, and he also described in detail how learning and retrieval of information was to be done. Okay, so we have the word informatic now. The question is now what about the neuro part? So in most Western European countries, except the UK, I've learned from Wikipedia, the word informatic, informatica is what the English call computer science. In the UK, a common definition is the study of the structure, algorithms, behavior, interactions of natural and artificial computation systems. This is from the University of Edinburgh, but I've read that many universities have adopted this definition. And this is what actually the Germans would call neuroinformatic. In the US, a common definition is obviously or apparently the application of information technology to the arts, scientists, and professions. I like this definition, it's so general, you can do anything with it. Yeah, or the interdisciplinary study of the design, application, use, and impact of information technology. So yeah, we can do with it whatever we want. However, at least in most of Western Europe, the word neuroinformatic long was used, what was called artificial neural networks research. And you find still a lot of universities that have institutes with that name. And there is still no trace of what we today roughly call neuroinformatics. So in the mid 90s, there was the first, the US human brain project. And they adopted the term in analogy to bioinformatics, roughly meaning data basing the brain. It's a bit broader, but from my understanding with the very general definition, this is what it boils down to, somebody might correct me. But nonetheless, the question, of course, is nowadays, what should it actually mean? And I have also an opinion on that, which I will also force down your throat. So first, artificial neural networks, they don't need this word anymore. They are called now machine learning, so they should leave us this term. And also, we don't need a new word for computation neuroscience either, I mean. It's just creating bad emotions. And from my, I'm coming back to that point, from my personal experience. There are many papers published about tools, but very few papers describing methods and algorithms. And this is, in my opinion, really a problem. For example, if you look around and you try to find, for example, a good implementation of a cross-correlation, it's very difficult to find. Spike-based cross-correlation. I mean, if you see what people do, it's a technique that has been used for 50 years and still there is not a single paper describing a decent implementation of it, and that's a pity. Likewise, so many other methods that you use. So what I think we need is a word that denotes the computational methods used in neuroscience.  Not necessarily pertaining to the results you get from that. That would, in my opinion, make sense, and now I have the response that I was looking for, okay?  There was another hand up there. No, no, I don't say it. I just said they don't need it anymore because I have transcended from being just artificial neural networks to machine learning. There is much more now, of course. You're fully right. To be cognizant that when you're talking to an audience with a lot of Americans in it, they're going to think you're talking almost exclusively about database techniques and nothing else. Yeah. Yeah. Yeah. Yeah. And they're not aware of these different points of views. So I wanted to bring this up just to make people aware that there is a, the term has a long history and it has gone several shifts in meaning. And, okay. So now I do another cut and I actually, oh, there's another one. Yeah, it's interesting. Yeah. Yeah.  If you want to go exactly what the tool is doing, like the new tool that people are using, that data is organized data, then there are very few references on that and it's even on the chart. Yeah. That's also not bad. Yeah, not good. Okay. So I will come back to this point actually later in my talk. I will now first deal with the question, what is the model anyway? And, yeah, so there are different views you can take. It's, again, a term that has been overloaded with many different meanings. And I don't start with the definition of what a model is, but rather why we do modeling in neuroscience. And there's an old quote from a Gregor and Lewis from a wonderful book from 1977. If you can get hold of it, look at it. It's also very much about neuromorphic engineering because the computers weren't up to the task of simulating nets of neurons, so they're basically sold at them themselves. It's a very nice book and if you have weekends time and you have some interest in electronics, you can build your own neural networks with a soldering iron and a bit of patience. So who of you is actually building models regularly? So how many answers? Models of neural systems of whatever, compartment neurons, neural networks in any way. Okay, so I have a list of questions for you which you can silently answer. You don't have to expose yourself to the audience. So the first question is what is actually a model? What do you think is a model? And what is a simulation? Is it different? Is it the same? How easy is it to write simulations? And how easy is it to read a network model? That is, somebody gives you a description and you say, aha, I understand what this thing is supposed to mean. Then running network simulations, can you trust your results? And if you do, why? Then running a model again, let's say a year later, have you ever tried this? How difficult, how easy is this? Well, if you're not doing models but just analyzing data, you can just replace this by analyzing a favorite piece of data that you have. It's, I think, the same problem that you have there. And then, of course, other people's models or other people's data. Why should it be easier with other people's stuff than with your own stuff? And I will implicitly, not explicitly, but implicitly try to answer these questions along the way. And I will start with a slight overview of what you could think models could be. So it's a categorization that I put up here. So there are three that are put up, which are non-exclusive. So at the first level, you have so-called phenomenological models, which basically put down a formula for some effect without actually going into the details how the things actually work. We have a lot of these models. I will give an example later. And if we actually go back to it, a lot of, basically, at one level, all fundamental laws in science are phenomenological models. Because if they weren't, they could go one level deeper and then say, OK, then I'm here again. So if I have, for example, Newton's laws of motion, they just describe how a mass is moving, but not why. You don't learn really why the mass is moving that way. It is just an observation. You can't go any more deeper. Then you have mechanistic models, where you actually use these basic fundamental models to build up something larger. And a lot of models that we have in neuroscience are of these type. For example, if we have a neural network, we compose it of neurons. You hardly find a neural network model that isn't composed of neurons. That is purely phenomenological. There are these models, but they are in comparison relatively rare. And then there is a kind of orthogonal axis, which you could make up, that is whether a model is statistical or not. A statistical model is a model where you use random variables to describe one or more of your observables. Or you use entire distributions. So statistical models can be mechanistic or phenomenological. And that's the categorization. There are some things we would like to have from a model in order for it to be good. For example, a good model should describe the behavior of the system in the circumstances where I know it. So it should explain existing data. This is I think the minimal requirement that we have. How well it should describe this existing data is of course a matter of taste and debate. And it should, of course, be able to describe how the system will behave in other situations. Sometimes in discussion this is called it should predict new experiments. A mathematician would call it should generalize. And for scientists models are often useful to separate those parts of the system which contribute to certain behavior from the parts which don't contribute to certain behavior. If I want to understand the laws of motion I don't need a model of a complete car. It's just too much. I don't need it. Any other body will behave the same so I can just use a point mask and have it move. And then a model should have as few free parameters as possible. And the free should be stressed here because free parameters are the ones that you basically use to tune your model to a particular situation. And if you look for example at nature's laws there are no free parameters anymore. They are fixed. We call them nature constants because they reappear everywhere and once I know them here I know them there. I don't have to tune and crank the model every time I go there. And optional, of course, the model should provide a mechanistic understanding of the system but that doesn't always work as we pointed out already. I will give a little example of a good model even though it is an ideal model so it doesn't describe all situations and that's the gas equation that we know from physics. It gives you the relation between pressure, temperature and volume of a gas. It's used in many situations and it doesn't have a free constant. There is one parameter, this R, which is 8.3 joule per Kelvin and mole and that's universal. That's a nice thing. And this equation is valid to a certain degree of accuracy for many gases which are not too complex and the pressures are not too high. It has one parameter and it provides no fundamental mechanistic understanding why the dependence is like this. You can, I mean this came later, of course derive this equation by using statistical models of particle movements but other than that this is a phenomenological model that has turned out to be quite useful and almost true. Pragmatic definition of a model at least from my point of view is this one here that a model is basically a function which describes what we can call dependent variables. The response of an experiment, the quantities you're interested in in terms of independent variables that are the one you can tune during your experiment and a set of parameters. That's the model. And now you have two steps usually that you have to solve. You have to choose the right equations and you have to choose the right values for parameters. These are the steps that you have and then you have to care what you want to describe and how well it generalizes beyond it. Some examples of these independent and dependent variables are of course the independent variables position of the recording site in an electrophysiological experiment, the amplitude of some injected current, orientation of a moving bar, type of visual object that you show, whatever you can control as an experimenter. And the dependent variables are the one that you expect to change when you turn the knobs of your independent variables. So it's basically all the things you can or want to measure. And I want to give a classical paper as a as an example. In my opinion, it is actually the first paper in computational neuroscience 50 years before Hodgkin and Huxley. It's a paper by Lapic where he describes the relation between an injected current and the membrane potential in a piece of tissue. So here that was the time where he measured. This was the observed membrane potential. His hypothesis was that it's a leaky integrator that you have as a membrane. Basically that's the equation here and it shows that the observed value here is closer to the predicted value of the model that was used at that time which was a simple linear relationship here. So that paper here is from 1907 and I think is one of the first computational approaches to neuroscience where you really use a model to describe what a piece of membrane is doing. It should be mentioned very often it's said that this is the original integrated fire model but in this paper there's actually nowhere a threshold. There's no spiking involved. It's just a passive membrane that's described here and I think it was Tuckwell who noted that because he computed the time to first spike he implicitly assumed the threshold and that's why he used this as the first neural model. Another example would be something like this where you actually have a hole cell recording and you inject current in this case here the random current and then you observe spike trains of a multiple membrane potential traces of a multiple trials and you try to find a neuron model that best describes the spike trains here. There was a paper by Julie Bae and Gerstner in 2006 with a single neuron modeling competition where people were actually asked to provide models that best reproduces this data. Interestingly enough there was no, in this competition you could win a considerable price. Interestingly none of the typical none participant from the group of compartment modeling I will come back to that if you don't know what that is participated here which is interesting enough. Okay. So these are examples of models where you have data which you want to describe you have a formula which you think describes your data and you go through the process of matching the data with your hypothesis that you have. So there are two steps and in machine learning they're also known because a neural network an artificial neural network can be seen as a model for data there are two steps which people distinguish one is called model selection at this point he is giving up so model selection it is basically finding the best equations to describe your system and then there's a step called model fitting which means find the best set of parameters for your equations in models where you have a lot of parameters that step is essential if you have a very good model that doesn't need any free parameters you're lucky and you're probably out for the Nobel Prize but even the Hodgkin-Huxley model I have to say has a lot of free parameters and that's also one of the prime problems of these models for my personal taste. Okay, so how do you actually go about so just from a theoretical point of view you have some pieces of data which X and Y denotes the individual pairs so the axis can be vectors of data don't have to be individual points but they can be high dimensional and usually you fit your model to the data by minimizing some type of cost function which is usually mean squared error or anything along these lines and I will use another historic paper to illustrate this process which is an old paper by Gerstein and Mandelbrot it's the same Mandelbrot that invented the what was it called in English this fractals and so it's a random walk model for the spike activity of a single neuron 1964 and I will make a little detour now into stochastic point processes because that was a theory underlying their model that they used and it was also a very new theory at that time even nowadays it's considered classic stochastic point processes are events that occur randomly in time typical examples in the textbooks are mining accidents failures of light bulbs or radioactive decay so the actual events they don't have a duration they just occur like spikes and you usually describe the random process by the set of the spike times here and this TI then is a random variable and the spike train can then roughly be described as a generalized function which is a sum of delta functions here for computing convenience and what you usually do is you define a measure which counts how many events you have in a time interval t this is basically the cumulative that you have here and it gives you at each point in time how many events have occurred since then so of course good to know for example if you are building a construction a big factory and you want to know in one year how many light bulbs do I have to exchange or how many machines will fail so the research of this point process is quite important in a lot of applications and of course you can use it here also for spikes so Gerstein and Mandelboot that started out with something of a membrane equation here so you have the membrane potential here which decays with time but what they did is they actually neglected this leak here and just said ok the membrane potential is the superposition of random events arriving so these are the excitatory ones and these are the inhibitory ones and basically what you get is what is called a random walk and a random walk is a very versatile process in statistics and it's also very counter intuitive and that's why I like again to go a little bit more into detail about this so the typical situation is this one here let's say you play a game you throw coins and whenever it's had you get a dollar and whenever it's tail you lose a dollar and the cumulative earnings that you've made in this game this is a random walk the money will go up and go down and typically you think ok because it's symmetric I lose as much as I win that I roughly if I just observes this over time I will roughly be either a little bit in the plus or a little bit in the minus the interesting point about a random walk is that this is not so the random walk is actually a process where the probability of being near the expectation value the expectation value of 0 actually decreases with time and this is now here the formal description and this is here different trajectories of a random walk so imagine that these are the earnings of different players that they play the same game and what you see is that these trajectories actually deviate quite quickly from the zero line so these here go all in the negative these one here go into the positive and what people have been looking at traditionally is the occurrence time when the trajectory will actually go back and cross the zero axis which is your expectation value and they were also interested in the mean which is zero and the standard deviation you see the standard deviation is actually growing with square root of t so the longer you wait I mean this probably most people know the more the less knowledge you have about where you are to be aware of the advantages of earnings and losers but what we clearly see is that we deviate away from this middle axis here and that's actually quite a remarkable effect and there's a nice book by two nice books on probability theory by Feller and he gives this very nice analogy here so suppose that a great many coin tossing games are conducted simultaneously at a rate of one per second per night for a whole year and on average one in ten games that's ten percent of the game the last equalization will occur before nine days have passed and for the remaining 356 days there will be in the loss or in the win not changing that's for me that was quite counterintuitive to learn they're also in one out of 20 cases the last equalization takes place 21 fourth days you can compute how that is and in one in a hundred cases two hours and ten minutes after you start gaming and then it never changes again so anyway this model apparently was a good model for single neuron spiking activity as Gerstein and Mandelbrot because before the start of the season I would say that incorrect referee calls and it doesn't I haven't an exercise in the book of biological physics about the human defects of incorrect referee calls but how do you reconcile that with the mean being zero the expectation is still zero because you have infinite time you have infinitely many games I mean that's the thing about zero the problem is in any one game it isn't and in any one game even though you move far away you can't be sure you'll stay far away the further you go the more likely it is you're going to stay far away it's all got to do with probability and it's very hard to get your head around and to get it right so this is what this is the empirical distribution of the spike times spike intervals that Gerstein and Mandelbrot found and you can nicely fit it to a gamma distribution which results from this random warp process this is of course incorrect because they neglected the leak that the membrane has and a few years later Richard Stein it's a wonderful paper actually extended this model here and he actually fixes the problem that the standard deviation of your membrane potential should grow to infinity if you just wait long enough because this is not observed that's a clear contradiction to data that you have and here you find the expressions actually for the variance in this case and you can see that it's actually fixed once this term here is saturated and from this formulation you also go very nicely to more abstract formulation even which is called the Ornstein-Ulenbeck process where you assume that you have a lot of incoming spikes with small amplitudes which is a common assumption and then you can make what it's called a diffusion approximation so you no longer have a random warp where the standard deviation grows but you have a diffusion process with a fixed standard deviation and then you end up with this equation here and this you will actually find in recent discussions about neuron statistics okay so these are now models where we are relatively agnostic about how the neuron actually works so we take Lapeak's formulation of the membrane potential and then we apply some statistical assumptions for example you have randomly arriving events whether this assumption itself or not is not really discussed mostly and now I will turn into the other realm of modeling where you try to put more and more information about what you know about the neurons into the neuron model so if you look at a neuron this is a pyramidal cell here you look that it has this nice branch structure and all of this is covered by an electrically conducting membrane so quite a long time ago people thought okay it should be possible to represent this structure here into what is called a cable model that you assume each of these pieces here is isopotential and you can replace it by a little piece of cable and then in the end each cable segment here gets its own properties in terms of ion concentration, ion channel concentration, things like this and with this you can construct a replica of a measured cell and this is also where Hodgkin-Huxley comes again in because they invented the formalism which is very useful here namely that you have these so called mechanisms here and gating variables, these are these little terms here which plug into the current part of the membrane equation here and this is of course nice because you no longer have one model depending of how many of these ion channels you put into your membrane equation and this allows you of course to describe the very versatile behaviors of neurons that you find in experiments, this is of course very good, you go away from a stochastic definition or stochastic description of neurons to a deterministic description of neurons where you are precisely able to compute the membrane potential of a cell under well controlled stimulus conditions. This works very nicely to reproduce slice experiments, it's more difficult to do modeling of in vivo experiments since there you have a lot of other factors that are not under your control. So the point that I want to make here is that because you have the ability to plug in more and more ion channels, suddenly model selection rather than parameter fitting becomes also an issue because by adding a new ion channel a new mechanism you are adding a new equation to your system which expands the capabilities of your model to express things in the end you could basically do anything with it and the question at one point of course is how much of these mechanisms do you actually need, mathematically speaking you would say you are adding yourself a class of models that become more and more powerful the more equations I add to it and in machine learning this is actually called capacity of a learning machine and I would like to use this analogy here because it's quite fitting and also each of these mechanisms usually has some coefficient attached to it so you are not only adding dynamics you are also adding some constant some free parameter to the system that you would have to determine somehow and then of course and that's a big debate in science or in neuroscience, how much do you actually need this question was answered well a couple of hundred years ago by this teller here, that's William of Ockham in Oxford and he said I'm not sure how rusty your Latin is Entitas nuns multipli canda preta necessitate Entitas should not be multiplied unnecessarily there is a more recent version of this statement by Albert Einstein, he said things should be made as simple as possible but not any simpler okay that's nice and funny but what does it mean again in mathematics we know that if you want to describe a set of data points some type of equation you usually have the choice between two situations you can either follow each data point precisely here but you are probably lacking the right relation this is called overfitting here so you have too many degrees of freedom in relation to your available data and this will usually result in poor generalization to new data which means if I add a point here which will maybe somewhere here then this curve won't be able to actually properly describe it whereas the straight line here makes an error on each of your little data points so the error on your known data is slightly higher but it will better be able to generalize to new situations and this is what is called in machine learning the bias variance dilemma it's the trade of between overfitting and underfitting and it's a simple lesson that we draw from this is that more is not always better quite to the contrary you can make your models worse by adding new degrees of freedom so very sketchy there is this relation that you can draw so if this is the capacity of your model think of it as a number of mechanisms that you have in your compartment model or the number of neurons you have in your network or the number of equations and there are parameters and this is the error that you can make and basically the empirical error is the one that I measure on my data that I use to determine the model parameter so I can by adding more and more mechanisms I can make this error virtually disappear but the problem is from a certain point on I will start modeling my uncertainty and even errors in the data or noise in the data and the actual error this one here is the error that I'm actually interested in it is how well is my model able to describe new data the data that I haven't used to tune my model and this curve this error will initially go down but once you reach a certain point it will actually go up and that's the point where you ought to stop there are some modern machine learning techniques that use this relationship here support vector machines for example they try to find the optimal point and the optimal capacity of your learning machine for your problem but of course when you are constructing a model of any system yourself you have to be aware of this problem in particular if you're dealing with non-mechanistic models where you don't have an idea about which components are actually contributing to a given effect for example statistical models are prone to overfitting or any other type of parametric model like Fourier expansions or Taylor series and things like this okay what should you do you should even if you have just a little bit of it you should split your data that's the least you can do the one half you use to determine your model and the parameters and the other half you check whether the model still performs well on this and this is a method that the mathematicians call cross validation and I advise you to read up on this in your free time and maybe a little bit of summary now regarding modeling so a model is a formal entity it's a set of equations a set of prescriptions that you write down and basically a model exists primarily up here in your head and this is very important the mathematics is just a tool to get it down and to communicate it to somebody else but often you have ideas about it that are often forgotten to be laid down on paper for example if we write down in this profession of computational neuroscience if we write down this equation here we usually assume that T cannot become negative otherwise this thing would explode we never write this down we never put a heavy style function or something around it but this is an implicit assumption that we make about our model which is rarely stated and lots of these exist and many non-reproduciabilities of models where people cannot find the effect that somebody else has found is because these implicit assumptions are not properly laid down okay you forget anything here yeah a good model helps understanding a system by reducing it to essential characteristics and one point I always like to stress is because it's almost a religious discussion for some people a model needn't be complete it can be an ideal situation and they're very useful these ideal models they needn't even correspond to an actual physical artifact examples are these ideal gas equation there are no ideal gases the Turing machine as a model of computation there are no Turing machines every computer works differently but still it's a useful concept to show certain limits of actual computers the same goes to the Carnot process everybody knows who knows what the Carnot process is okay the physicist so the Carnot process is a very trivial simple model describing how heat is transported between two reservoirs it also assumes ideal gases and lots of idealistic conditions which can never be met in reality but you can compute upper limits of for example a refrigerator or an air conditioning system with it and that's why these models are useful so if somebody tells you that your model is not good because it misses this or that relax as soon as you can do something else with it useful with it okay and another thing all models have a limited range of validity that should be checked against the physical system so you always have boundaries where your model isn't valid anymore that's true for every model and except for very few laws in physics but even then classical mechanics relates to macroscopic objects and one point you have quantum mechanics and so on so this you also have to be aware of and this concludes the first part and now I will ask to the slightly more subtle question what is then a simulation I talked about models and now comes the question what is a simulation and I would like to draw this picture here now so models are often complex too complex to be investigated analytically it starts with the Hodgkin-Huxley model which is already very tough to get into by analytical methods and then you use computer simulations to investigate such models so a simulation is a computer implementation of a model I've talked about this painstaking process here that from experimental data you have to construct a model and then you have to validate your model against your data now the problem is what happens actually here in this step from the model to a simulation and it's a in fact a fourth step or it's a three step process that you have here only two steps are often not separated mentally so the first one is you have your model so it's some equations that you write down and then you come up with a computer model now what is a computer model the computer model is still not tied to a particular programming language though often you do but it's rather the choice of numerical algorithms that you use and the choice of data structures that you use to implement or to represent your model in a computer and then you take the computer model and you implement it in some programming language so the choice of a programming language also adds some limitations to what you're doing and then of course you have a final you have a program that you can run and investigate in that sense a simulation is very much like an experiment because it gives you the behavior of the system under a very particular setting of parameters and that's different to the analytical investigation of a model where you can make general statements which are independent of a particular choice of parameters I give you a little example here of these steps that you have okay so consider a tiny model which consists of two very simple neurons and I have a neuron here and I have a neuron there and I use this Lapeak model this integrate and fire model so you have a membrane potential equation and some currents coming in I just have a constant current in my example here and this is actually a nice model because it has just one free parameter and that's the threshold everything else are basically material properties that I assume like the capacity has a physical meaning and things like this the interaction between the two neurons they are purely phenomenological because I think say whenever this neuron here produces an action potential a spike I will get an exponential decaying response on the post-synaptic side and this response has a certain weight and a certain delay that's of course not a good model because it neglects everything we know about synapses but anyway we will use this for a while and now I have to ask myself how do I make a computer actually do this so first you have to solve this differential equation here and then you have to select values for the various parameters that you have here and then you have to restrict yourself to a certain time axis and a certain temporal resolution with which you want to look at the results and then you have to sit down and write this in matlab, cc, plus plus, python whatever it you currently have if you look at what is around and I've seen a lot of simulators this equation here in this simple setting is solved with what is called an Euler method where you basically take this the value of the right-hand side if you bring this over to the right-hand side multiply it by the time step and this is the solution of your equation if you see it makes a big error you can solve this equation analytically few people do this here is an ad hoc version in c++ which I wrote and it should appear here without cranking my head so this is a piece of code and you see that it continues at 27 lines so you have to restrict yourself on a time axis so this will be 100 simulation steps and the delay will be 10 simulation step and one simulation step I denote to be 0.1 milliseconds so I simulate 10 milliseconds here I have two neurons and these are the data structures to hold the membrane potential synaptic current, the external current and so on so these are all choices that I've made at one point or another to represent my little model and this is the actual simulation loop that you have if you look at this and you probably ask yourself why am I showing this to you I show this because you don't really see much if I gave you this piece of code and tell you to draw a picture of what this thing is I wouldn't expect you to come up with a picture that I showed you before and when I was an undergrad student I received a 200 kilobyte piece of C code from Moshe Ablas he's a researcher in Israel, inventor of what is called sinfire chains and we had to reverse engineer the neuron model and everything from this piece of code it took us a good part of three months and we discovered quite a lot of I wouldn't say errors but things that were suspicious now nice can I get back my nose so if we write down a few statements about this ATOX simulator here so this program is near optimal in performance in terms of speed because the C program is very fast, very quick, that's nice but the computer model is the computer model is a set of algorithm data structures that I've used but we have to be aware that the computer model is always a lossy representation of your original model it doesn't preserve much still, nowadays you will hear a lot about model sharing and things like this, people don't really talk about model sharing or implementation sharing and that's very dangerous enterprise because implementations have errors and one is likely to be lazy not to actually scrutinize what has been done and likely to reproduce the errors very nicely one point that I will stress also to the end but I would like to stress it already now reproducibility is not the same as rerunability if you run a program twice it better give the same results that's clear but a model implemented independently by another person re-yielding the same results that's something good that's rarely been done I come now to the nest implementation, there's nest again my favorite tool it has an advantage and a disadvantage, the advantage is that the same thing here is much shorter and much easier to actually see what's happening because you have words like neurons here, at least if you read the manual you know that this is a neuron model here and voltmeter you can guess what that is you see that there are two neurons involved a voltmeter and that you have an external current and that you have a measurement interval and that these things are connected and then simulated for 100 milliseconds and then you get a nice picture of your memory potential that's the good thing now the bad thing is that we've hidden all the interesting parts from you you don't really know what's happening and this is this is Berry's point and also your point that if you have a publication about a tool you don't really want to know, I mean you can read up how it works or how to use it but you really need to know how it works what is used underneath to solve the equations and get the results because graphs are deceptive I've seen a lot of times students coming up with simulation graphs and making a philosophy about the nice effect that they found until they realized that it was a programming error that they had and all the nice theory about information processing collapses because yeah because of an error after all a simulation is just a computer program and there are certain measures in industry that tell you how many errors you have to expect per thousand lines of code and this is for industry code where the industry measure for a good programmer is 12 lines of code per day this is what an industry programmer will do if you take an average over one year the productivity of a PhD student is much higher I checked that but I guess also you won't find only 5 errors per thousand lines of code yeah without yeah I will come to that I will come to that but first I will give a slightly more complex example namely if you don't have just one neuron so this is also a very nice model of model in computation neuroscience it's a model for delay period activity so delay period activity is what you observe is if an animal has to memorize something for a certain amount of time it's not a learned object it's a cue for example and the stimulus is withdrawn but still you can measure something so this is a trace or several traces from a monkey motor cortex so the monkey was told to respond to a certain direction and the cue to respond was given here so there's a go and a no-go paradigm sometimes the cue is given the monkey should not do anything it doesn't have to care for the direction where it has to respond and then it has a go paradigm where this cue is given and the monkey has to respond three seconds later and what we see here that this particular neuron here continues firing until the point the monkey reacts a little bit longer and then it returns to its baseline firing rate and it was which still is a big discussion point how populations of neurons actually do this because it is classical result from the 50s if you plug together excitation and inhibition you basically get two states either the network is at full activity it explodes or it's dead it doesn't spike at all something in between is very very difficult to obtain and so this was a question worth investigating and the model that was proposed among others by Amit and Brunel and Brunel in 2000 investigate a bit further that you basically have an excitatory population an inhibitory population and then you have activity from the rest of the brain coming in and basically how much excitation you get from the outside and the relation or the ratio between excitation and inhibition determines whether you get this delay period activity or not and you can actually draw nice diagrams mapping out the dynamical regimes that you have so to give you a few traces here how the network can respond in these so you can get your very pathologic oscillatory states you can get what is called asynchrone is irregular which means it's irregular and the neurons are asynchrone or you get different versions of oscillations here the nice thing about this model is that it's actually quite easy to implement and it's also quite nice because it scales it's nowadays basically used as a benchmark for very large simulations so in this case here we have 10,000 excitatory neurons 2500 inhibitory and roughly 15 million connections and it's clear with paper and pencil you get a certain way by using statistical methods by using stochastic models and this is done in the 2000 paper of Nicolas Brunel where he used a first passage time derivation solution of the so-called Fokker-Planck equation to get the firing statistics of this process but of course you can also simulate it and again now I have to switch here I'll get my mouse I can write down a talk version of this simulation here in Python this time to swap the programming language so again down here is the main simulation the main simulation loop and you compute lots of stuff it's in fact quite similar to the previous model here but only now that you have to account for the random incoming spikes you have to provide something for that and you also have to put in data stride methods to actually measure what you're interested in and this gives you this nice piece of code it has 59 lines it's not bad the corresponding nest version has 32 lines is this one here it's still a lot of lines but it's clearer now how things actually happen there because random connectivity is a tricky business in the paper you read that every neuron receives a certain percentage of neurons from the other but how this certain percentage is actually drawn is not really specified and that's one of these implicit assumptions it really very much depends on the specific choices that you can do in your implementation what results you actually get and also how you implement your random noise that you put into the network which is required here for example you can do it realistically by having spikes coming onto the neurons that will give you a certain a a certain property of the noise when it comes to the membrane potential because each spike goes through the synapse and the synapse is basically what in electrical engineering would be called a filter and this filter imposes a certain spectral characteristics of your membrane potential so if you do a Fourier transform of your membrane potential you will be able to recover the shape of your post-synaptic potential however the alternative would be just to inject a random current into your neuron and if you do that you get a very different spectral composition of your neuron you might say that doesn't really matter much but it does because you have a different power component and because the membrane potential itself is a low pass filter it will throw away much of your random current signal and the firing rate will actually be lower although you put in the same noise just because you don't filter it correctly and these are implicit effects that you get by the choices you make in your implementation so try to go back that was right that was this little script here so first about some notes about this python simulator here so the code is not much longer than these two neuron examples so you might say well what's the deal the problem really is that these the data structures that you want to use to store the connectivity of a network they have to be carefully chosen because neurons in the network are sparsely, relatively sparsely connected and not all to all so if you do like neural artificial neural networks they are the weight matrix for 10,000 by 10,000 neurons you end up with a huge waste of computer memory and also in this particular model all the weights are basically the same so you would just have to store very little information but setting up these data structures would have made it impossible for me to present the code on this relatively short line the algorithms and data structures are very specific to the particular problem that you solve here and if I want to change the paradigm slightly I basically would have to go through the entire code and make changes everywhere and that's why it's a tricky business to write your own simulations even if it looks like you can quickly do it so any experienced modellers are likely to choose the wrong algorithms and data structures and I've seen this many times over so this is the same code again that I showed you and the output that it would produce so it's random dots so basically that was the main point of this paper the advantage when you use one simulator I use Nest here but there are many other good simulators Brian for example or Pine as a meter language they allow you to have an explicit representation of your model and you can hope that because many people are using it that the algorithms are sufficiently debugged that at least in the most visited places of these simulators the errors are minimized I shouldn't say gone last bug does not exist we have to be aware of that so I come to this can you run that on this laptop? do you want to see it? so this is python here and I just run so at one point it will so now it has started simulating and it also warns me that my simulation time was not an integral multiple of the simulation time step and that might lead to inaccuracies here sometimes and then the output is on this screen now here you get the ruster plot of the thing here no that runs on any yeah so ipython is indeed very useful there is also a project by Andrew Davidson called Sumatra which is the attempt to have a kind of electronic lab book that automatically stores everything about your simulation that you do allows you also to annotate it so that's also yeah it's a different concept but okay so yeah okay so this is a list of errors here that you can make so first you can make the semantic errors by just using the wrong data structures and algorithms these are very common in particular if you have little experience then you can have programming errors in the implementations they mostly show up quickly so it's not that critical but in long running simulations long running means days it's of course very frustrating if your program crashes after one and a half days in particular if you had to ask somebody to grant you computing time somewhere then numerical errors they are often under estimated I've heard simulator x may not be so accurate but it's fast I can produce inaccurate results arbitrarily fast I should say but if you have a certain accuracy measure or accuracy goal then you want to be fast and that's the difference and you might say I make a lot of fuss and I've never seen any problem here but my mouse is not where it's supposed to be but this is from science 2006 about five papers that had to be withdrawn because of home-brewed code and the errors in it and implications it actually had and in this case it was analysis code but it also transcended to simulation models that were built on top of this this here is the field of protein folding there's more money in the field than in computational neuroscience and I dare not to say how many results in computational neuroscience are not really results unfortunately if we look at the landscape of big simulations conducted in the recent years we will find that most of them actually don't use any type of simulator that is used by more than one group so this is Izzikiewicz's work here Moda is IBM the cat brain and the red brain some of you might have heard about this the use of course is IBM they have their own simulator I trust them that they know what they're doing Anders Lanzner's work split but nobody ever saw a line of split it's just as a hint now the point with unit testing this is something that at least in Nest we do for quite a while that there's a large library of unit tests where basically a lot of models and a lot of features are precisely tested and it's always interesting to see that for example with a new compiler version or a new operating system version a curacy breaks down because things in a particular programming language or particular implementation of it are suddenly different most obvious case is of course to switch from 32 to 64 bits if you have buffers which implicitly assume certain bit size random number generators for example then you quickly run into very peculiar problems in particular this faulty yeah okay yeah I should actually okay so unit tests are the one where you basically test every feature that you would like to test and then regression tests are the ones that require a little bit more bookkeeping of how you develop your software because regression test means you find an error then you write a little script that reproduces that error and whenever you do a new compilation of your software you rerun this little script and see whether there is a regression whether the error reoccurs and that's quite common phenomenon because often the error isn't actually solved by your solution you just fix the symptom but not the cause and then sometimes you need several iterations to find the root of your problem and also there I'm glad to say at least Pine and Nest and Brian I think as well they have now regression tests so that things like this are monitored there was a review last year where different simulator packages were compared for their accuracy and that was quite nice that they actually took Nest out for some of the tests because it was identical to the analytical solution so apart from machine precision rounding errors there was no deviation but I don't want to advertise too much here even though I did it already so the final point here so simulation is a computer implementation of a model and one should always be aware that translating a model into algorithms and data structure is lossy there is no way around it you always restrict from the general to a particular case a model in a computer is not a model on piece of paper never ever and to put it a bit more bluntly a model is actually a model of a model that is not as pretty version as that and we should always be very critical with respect to the results that we get and if we write a simulation paper and this is rarely done people should actually convince the reviewers and the audience why this particular implementation they chose is the correct one and not only by saying we tested this but actually showing that certain test cases behave as they are expected to behave there are always test cases that you could run and compare to the simulation with the expectation of your pencil and paper things okay but another issue of course is that this process of translating a model into software is a programming task and I would like to quote this here from Bruce he's a veterinary of IBM and he wrote a very famous article which is called No Silver Bullet Essence and Accidents in Software Engineering and he says that all software construction essentially evolves or contains two tasks one is the essential task which relate to translating your problem into data structures and algorithms and what he calls accidental task which is the implementation part which is the fighting with libraries compilers and whatever and then he says unless what we do now is 90% accidental and only 10% essential there won't be a 10 fold increase in productivity regardless of how much we make our programming environments and things better and the same I would say is true in computational neuroscience there has been a big debate or much activity in the recent years and the INCF is in the middle of it to improve and support the tools that we use but the actual task of conceiving models and translating them into the correct algorithm data structures they remain and they will always remain and that's why model building simulation building will always be difficult there is no such thing as a paper that you put up and it's very easy to understand what the model is doing a good model means that it allows you to understand a phenomenon that you weren't able to understand before and that might be a very complex process underlying it ok final slide so I said this before so re-running a simulation code does not mean reproducible research so as much as I like the help of published models in one form or the other they don't replace the need to re-implement it will always be necessary so a model description in a paper should always be that the model itself is re-implementable and that also applies to very large data driven models with that I mean for example what we do in Lausanne with the Blue Brain project the model description has two parts three parts one part is lots and lots and lots of electrophysiological data and then you have individual description files that behave and then you have the description files how a particular experiment is conducted in your simulation and what I mean with being in principle re-runnable is that given the data you should be able with appropriate resources of course to build your own neuron models and to build your own simulation software to get the same results okay so yeah of course when you write a paper you should always keep in mind with this description that I give there will somebody be able to re-do the same without asking me for the source code I mean looking at source code is ugly, it's the last retreat and personally I don't want to do it because it's very difficult to reverse engineer source code so standard notations why being models in particular physicists they like to invent notations that's a bad habit very bad habit if somebody calls a parameter v why call it x the next time I mean you standard tools to simulate your models yeah and this is so trivial but never done use tables for parameters and data okay there's a whole paper actually about this by Eileen Noorley and Hans Eckhardt-Plässer and myself where this is a bit laid down I somewhere have the reference if you ask me I don't have it here and with that I thank you for your attention I think there's some time for questions thank you