And today we'd like to talk about Bayesian neural networks. So I assume that, you know, some of the, well, many of the concepts behind Bayesian analysis, but if you have any questions yet, do just interrupt me at any point. I may not be able to see the chat, so just pick up and I'm happy to be interrupted at any point. Right, yes. So today we'd like to talk about Bayesian neural networks and to do that, I'll cover three main topics. We talk about regressions, starting from standard regressions and then going into how neural networks can be used for regressions. I'll talk about classification and then I'll give you a bias selection of examples of recent studies that I happen to have been involved in using Bayesian neural networks in biological research. So yes, as said, do interrupt at any point and otherwise I'll get started. So I would like to start with just like a basic model of linear regression, which I'm sure you have already covered in the first part of this Bayesian course. And so in a linear regression, we have a predictor which we want to relate to another variable and then we try, we fit a model basically to be able to use this predictor to make an estimate of these other values. So for example, if we have a diet here like broccoli intake in pandas and then body mass of pandas, then we can regress them and try to figure out whether broccoli intake correlates with a change in body mass. So we can write this as an equation where we have the body mass equal to a slope times the predictor, so the diet, plus an intercept. We can also show the same model using these nodes and lines here. The lines here represent co-efficient. So the slope and intercept in this case and the nodes here represent the input that is the diet and the output. There is a body mass in this case. Our prediction here in Y will be basically an estimate of pandas body size plus minus a standard deviation or plus minus an error, which we can also estimate in the model. So under this model, we have essentially three parameters. We have the slope, the intercept and the error. If we are to fit in a likelihood framework or in a Bayesian framework, this type of model, what would be our likelihood function here? What would they use as a likelihood function? So the likelihood quantifies the probability of observing a specific body mass, given my parameters and given my data. What would be our likelihood function in a linear regression? Have you seen this earlier in the course? If you want to step up. If not, I'll go with this, but then you're gonna have to tell me everything about the priors. So a likelihood function here that we can use is based on a normal density. So we could say the panda body mass here is normally distributed with a mean centered in my prediction Y in a standard deviation set equal to the error that we are estimating. So for any given combination of the slopes of the slope and the intercept and the error, we can calculate the likelihood of our data. So the likelihood function will be basically a probability density function of a normal distribution. So the probability of observing the body mass, pandas body mass that we do observe, given our prediction in a sigma is given by this formula here, okay? And then if I fit this in a Bayesian framework or in a maximum likelihood framework, I will be looking for the slope, intercept and error that in a maximum likelihood framework will maximize the likelihood of observing this panda body mass. If we are fitting this type of model in a Bayesian framework, we are going to have to set up priors on each of the three parameters, okay? So we have three parameters, we've got the slope, the intercept and the error. So we need a prior on the intercept, a prior on the error and a prior on the slope. For the intercept, we could set if the intercept represents the body mass of a panda that doesn't eat any broccoli or where the broccoli doesn't have any effect, if the broccoli doesn't have any effect. So we could imagine this being some sort of mean body mass of a panda. Then what type of prior distribution could we use here for this intercept? Prior is a distribution that should reflect what we know about these parameters, right? So if this intercept here is some sort of mean body mass for a panda, what would be a reasonable prior that we could set on this intercept? There is it in the chat, a normal distribution by Daniela. Okay, okay. Yes, I'm not seeing the chance I guess, but it could be a normal distribution. It could be also a uniform distribution. It could be actually something informed by the empirical knowledge that we have about pandas body size. For example, if we know that pandas are mammals and they are terrestrial mammals, we could define some uniform prior that spans a plausible range of body sizes, right? Body mass. Now, what's a good prior on the error on the sigma? Okay, the sigma will be the standard deviation that we use in this normal distribution. What's a good prior for sigma? What do we know about sigma a priori that being a standard deviation matter, Michelle? Maybe we could also try to set up on a range of the variants that is present on different animals that you have an estimate. We could, yes. So if we have some prior information about like, yes, about random variants in this body mass, we could use that information. What if we know nothing about pandas at all because we are botanists and we know nothing about pandas? Can we still say something knowing nothing about pandas or animals at all, is there anything that we can say about the sigma parameter? Can it take any range of values? No, I mean, it has to be in some range that is possible biologically. There wouldn't be 2,000 kilos more, something like this. Yes, that's true. Is there anything more general about this sigma? Remember sigma is this, oops, it's this standard deviation, right, of a normal distribution. What values can sigma take, animals or not? There will be less than the maximum and more than the minimum, right? Yes, even more general, maybe. Is there any value of sigma that doesn't make sense here? Minus one. Any negative value, actually. Like a sigma is a standard deviation, so it cannot be negative. Because a negative sigma doesn't make sense here. A negative standard deviation is meaningless. So what we could use here is a prior that will not allow any negative value. For example, a gamma prior or an exponential prior. So any distribution basically that assigns zero prior probability to negative values. And the last parameter that needs a prior is a slope. Any idea of what prior we could use here? A slope parameter. Do we have any null expectation on the slope? Did it do regressions in the previous classes? Yes, we did. Okay. We could, like, I assume there is some idea for what kind of effect you would expect or what association would expect. Otherwise, we could set it to zero if you have no idea or set the mean to zero. And, I mean, maybe Gaussian would be too strong and go with the t distribution to have a weaker prior and allow wider tails. For instance, yes. But generally, you would use some priors, some prior distribution that is centered in zero, which is your null expectation. And a prior that allows both negative and positive values because you can have a positive or a negative correlation, right? So you could have a normal distribution at the distribution, Cauchy distribution doesn't really matter, but essentially it will be something that is centered in zero, which is so something that assigns the highest prior probability to a zero effect size. And then a distribution that allows both the positive and the negative range. Here, for example, I'm using a normal distribution centered in zero with some derivation of one. So this would be like a typical setup for a linear regression where we have some prior on the intercept, some positive spanning prior on the error, and then usually a symmetric center distribution centered in zero for the slope. So now we have the likelihood, we have the priors, and we can run our MCMC, for example, to estimate the parameters of our model to do that we'll have a data set, which we will now start calling training set because we'll be talking about neural network in a moment. We have this data set where we have a bunch of observations with broccoli and panda body mass. And then based on these observations, we're going to optimize a model and find basically the regression line that maximizes the posterior probability basically. So we'll do this in an MCMC and obtain not only a single estimate of our parameter, but a distribution of estimates. But let's say that this is the line that we estimate with our posterior estimate of the slope, the intercept and the error. So our trained or optimized model will look like this. And it will basically tend to minimize the distance between the line and the observed pandas. So this model can now be used as a predictive model. So if we are given the diet of a panda, we can make an estimate of the predicted body mass. So if this panda that we haven't managed to measure eats this much broccoli per day, then we can predict its body mass using our regression line. Okay, so we can use a regression as a predictive model. This is what a linear regression looks like. This is what a multiple regression looks like. So when we have now multiple predictors, for example, we try to estimate or correlate body mass with three different quantities. Diet, genome, which could be, I don't know, genome size, for example, whatever range size. The equation of this multiple regression looks like this. So we have a single coefficient for each predictor here. And then we sum everything up and then we have an intercept. Okay, so the predicted body mass is equal to the product between this coefficient and the diet, this coefficient and the genome, this coefficient and range size. And this coefficient times one, which means basically the intercept. We can rewrite this equation. It's a matrix multiplication. Only these matrices are one D, one dimension. So it's a bit silly to do that, but this will help us understand the neural networks in a moment. So this is basically the same thing as this one. Only we've write it as a matrix multiplication where we basically do a multiplication item by item here where we multiply by columns. So the first item with the first column it has a single item here. Second predictor by the second column, third predictor by the third column. And then we sum everything up and this is going to be Y, okay? We see that these operations will come back when we talk about neural networks. So these are two examples of regressions. These regressions are linear by construction. So they can only infer linear relationships between the predictors and the response, right? And the body mass. This is fine in many cases. It's what is nice about these models is that these parameters are clearly interpretable because they each correlate a single predictor. But of course, there is some limitations to what a model like this can do, especially these responses in reality are actually not linear. And this brings us to neural networks. The first neural network I would like to introduce to you is a neural network used for regression. And we're basically going to walk through the steps of a neural network to understand how this is actually not a different from a linear regression, but it still has a lot more power. But the operations that are going to be needed in a neural network are actually very similar to the operations that we just did for the regressions, for the linear regressions. Only here, I'm adding here a bunch of terminologies because this is the terms that are used in machine learning. But in a neural network model, we'll have an input which is basically our predictors. We're going to have weights, which are coefficients or slopes if you want. Then we have nodes that are kind of intermediate steps that take you from the input to the output. We walk through these steps in a moment. So these are nodes, they are just numbers. There is an activation function that we will talk about in a moment. And then some more weights, a bias node that is equivalent of an intercept. And then we have the output that is a prediction. So the terminology is slightly different, but we see that the operations are not that different after all. So let's see what this sort of abstract model actually does. And to do that, we are going to walk through the operations that are needed to go from an input to an output, just like we did for the regression model. So the first operation that we have here is this thing where we go from a bunch of predictors to a bunch of nodes. These nodes are a numerical representation of the predictors in a different dimension. And it's not numbers that we aim or care to interpret directly, okay? It's just some intermediate step that takes us to the output eventually. What is going on at this step is again a matrix multiplication where we multiply these vector predictors by a matrix of weights. These are the, each line here is a weight, is a parameter that we eventually will estimate. And the operation that we need to do to go from X and W to Z is a matrix multiplication. So to explain this, I'll use real numbers, which hopefully will help. So let's say that these are starting diet, genome and range size. So we got three numbers. And these are our current parameter values. So I'm just assigning at this point random numbers to these lines, okay? These are parameters. And we are going to have to estimate these parameters. In matrix multiplication, so now we need to go basically from these three nodes to these three nodes, okay? But we go through a matrix multiplication to get there. So we first multiply each X value, so each of these predictors by a W column. So we take this value and we multiply it by these numbers here and we obtain three numbers that I'm placing here. Then I go to the next value here and I multiply them by the second column, okay? And I get this column here. And then this is the range size, which I multiply by these three weights and obtain this other column. So now I went from three numbers to nine numbers, but I want to end up again at three numbers, right? Which are these three nodes. So the next step here will be sums by rows. So we obtain these two D matrix and now we take these three numbers here, we sum them up and we obtain the first node. So the first value here. Then we go to the next row, we do the sum and obtain the second value. And this is the term. So now we've taken these three values here, multiply them by a matrix. So in this case, three by three and obtain again, three values here. Interestingly enough, I know this is a course using R, but if you are interested in doing this in Python, you can do this whole set of operations in a single line, in a single function. So this will do exactly this type of operation. Now, this vector that we obtain here, these three values of Z are a representation of our input data in an abstract parameter space that we don't have to understand. So we are not really aiming to interpret these values. But what is interesting and important to realize is that each of these value here is already a function of all of the three features, okay, all of the three inputs because they integrate their summing up effect sizes basically from all three inputs, okay? So this also means that these neural network model will be able to account for all possible interactions between inputs. So now we have this set of values. The next step that we are going to, which we can place here, the next step that we need to do is going through this hidden layer. And we go through an activation function. Again, these are these terminology using machine learning, but activation functions can be actually something, things that are really simple to compute. For example, one of the most commonly used and one of the easiest things to compute is the radial function. And this is simply a function that will take any negative value and set it to zero and any positive value and leave it as is. This radial function is one of the simplest that exists and it's still extremely powerful. The whole point of this hidden, of the activation function in the hidden layer is to break the linearity of the response because we want neural networks to be nonlinear or potentially nonlinear, okay? So the activation function will typically be something that is nonlinear. In this case, it looks like this. So everything that is negative is set to zero and everything that is positive stays where it is. There is a whole range of other activation functions that look, for example, sigmoidal. So they are all characterized by being nonlinear. Their point is really to make sure that the neural network can account for nonlinear responses between things. So going through this step is very simple. We just take every negative value, any value that happened to be negative and turn it into a zero. And then we go to the last step, which looks very much like a multiple regression as we've seen before. So now we have a vector of numbers. We are adding here this bias node that is the intercept and we do the same operation basically that we did before. We see there is a question. Yeah, I was just wondering how do you choose which activation function to use? And in your experience, does it matter which one you use or not? I mean, how important is it, which one you use? Yeah, so that's a good question. We talk a little bit more about selecting the settings of these networks in a bit. So we'll come back to that. Ideally, these activation functions will not make a huge difference. In some cases, especially for small networks, then using one activation function or another can give you a better answer. In some cases, it's only a matter of convergence. So some activation functions will make the model converge sooner than others. But again, there is no strict general rule. There isn't a single activation function that will do the best job everywhere. So it's part of the fine theory that we'll see later how to do. But generally models are not extremely sensitive to these types of choice. Yes, Elis, I think you're muted, I'm afraid. Let's have one question, sorry. I'm wondering what you have to bias nodes in every layer just and just at the last one. You can have bias nodes in every layer if you want. It's just about how you did the network, I guess. Yes, it's pretty much like a hyperparameter that you can set. You can choose to have bias nodes everywhere or you can have it only at the last layer or you don't have to have one, basically. Yeah. Right, so in the last step, basically, yeah, we let this bias node that is basically similar to an intercept in a linear regression. This node will be basically a one that you multiply by its own parameter. We do this matrix multiplication again, we sum by rows and we end up with one value that is our output and it's our predicted body mass. So you see that basically going from input to output here implies a lot of operations, but they're very simple operations. There are multiplications and sums pretty much. So nothing really very expensive to compute going through this network. And this is good because we have a lot of these operations, right? Once we have a predicted body mass, we can calculate the likelihood of our panda given our prediction, right? So if our prediction is here and our true body mass is here, this will be the likelihood based on a normal distribution. If our prediction is here, this will be the likelihood. So the likelihood will be higher. Okay, so this likelihood part will work just the same as in a linear regression. So the likelihood function is still the same. We get a prediction and then we calculate the likelihood of our data, which is the observed body mass of a panda based on this prediction, okay? So in a Bayesian neural network, we have a likelihood function that is the normal, the PDF of a normal distribution, the probability density function of a normal distribution. And then we have the priors on the weights. There is no strict rule again on which prior we should use on the weights. The standard choice is to use again some distribution centered around zero, assuming that zero means that there is no effect size. Here, the effect size is a bit harder to interpret, right? Because there isn't just one effect size linked to every predictor. But the typical choice for priors here is a normal distribution or some other distribution centered in zero. So if we have a likelihood and we have priors, it means this means we can compute a posterior. And if we can compute a posterior, we can run any Bayesian algorithm to estimate our parameters. Here the parameters will be all of these weights, right? And we will try to optimize these weights so that the likelihood of our pandas, panda body masses will be the highest, or should say the posterior. So if we have a training set like this, we are going to have a bunch of pandas for which we have the body size and the predictors. These are a training set.  We can use a posterior sampling algorithm. I'm sure you've come across Metropolis Hastings MCMC or Hamiltonian Monte Carlo algorithms. So these will be algorithms that basically iteratively update the parameters. So it will start with some initial parameters. Basically we initialize the weights of your network in some ways with random numbers. You will set them in a, if you're using MCMC, basically you will use this as your current parameters then you will update your parameters using some proposals, right? So you will slightly update the parameter values. You will calculate the acceptance ratio, which will be based on a comparison between the posterior of the new parameter values and the posterior of the previous parameter values. Based on the acceptance ratio, you will accept or reject the new parameter values. And if you accept the parameter values, you will start from that point. If you reject them, you will start back from the initial point and then update the parameters again, calculate acceptance ratio and so on. So I'm not going into the details of MCMC because I think you have seen it in previous classes, but of course if you have questions about this, do us. As we sample basically parameter values from their posterior distribution, we will collect them, right? So we will produce a collection of posterior values of our weights, the parameters of a neural network. So if this is our model, we will collect posterior samples of all of these parameters and all of these parameters, right? So these are like the posterior distributions of these parameters. All good so far? Okay. If I now have a distribution of weights or parameters, basically samples from their posterior probability, distribution, we can use these weights to make predictions. So if we have broccoli intake, genome and range size for some pandas, we can predict their body size by running these measurements through our optimized network. And in fact, we can do it many times by every time something a different weight here. And if we do that, we are going to obtain a posterior distribution of the predicted body size of our pandas. Okay, so we are gonna predict pandas body mass based on these three inputs, but accounting for complex relationships between these predictors and the output. And I say complex because it's been shown that if a neural network is deep enough and complex enough, so if it's got enough parameters, it can approximate in principle any function. So neural network, unlike a linear model, we'll be able to, in principle, estimate any sort of response given our inputs. And it will be able to account for any possible interaction also between our or among our predictors. So this is pretty awesome. BNNs, neural networks in general are very powerful. So why don't we use them all the time? Of course, neural networks are very powerful, but there is a number of drawbacks. Neural networks are basically by definition overparameterized models, which means that they have way more parameters than necessary. And this changes a lot like the whole business of model testing, in a likelihood framework, you tend to find the simplest model that best explains your data. Explains your data. In this case, you're going to have to use different methods and we see some of them. But essentially, there is a risk of overfitting using this model because there are so many parameters. Defining priors in a Bayesian neural network is not trivial because these parameters are not directly interpretable, right? The parameters themselves don't have an interpretable meaning and that means that it's not as easy as it was for a linear model to set priors that are informed by some a priori knowledge. For example, for the intercept on the linear model, we could say that the body size of a power, body mass of a panda has a certain plausible range. So we were using a priori information to define our prior. Here, for these nodes, that's not so trivial. The parameters themselves are not directly interpretable and therefore these models are not explicitly designed for a hypothesis testing. So there are ways to do hypothesis testing using neural networks, but that's not their main goal. Yes? I'm just wondering how crucial it is to have informative priors because maybe if you train your network for long enough, then you'd be able to reach the ways that are adapted to your question. Yeah, so informative priors are useful sometimes because in some models, if you have a priori information and want to use it, a prior is a good place to do that in a Bayesian model. Here, for example, if we had a priori knowledge about the body mass of a panda, it would be difficult to specify a prior that reflects that because the body size, body mass of a panda here does not get its own parameter.  So it would be difficult here to set a prior that reflects a priori knowledge. Yes, but if you update them accordingly, at some point you don't really care about the initialization anymore, right? But when I say prior, I don't mean the initial parameter values, but I mean the prior distributions that we use to calculate the posterior. So the prior here would be basically what we used here for the parameters of the linear model, which I cannot find anymore, too many slides. So for the linear model, we had these priors, right? Prior distributions on our parameters. Here, we could define priors knowing what we are doing, basically, because we know what this parameter means. This is an intercept which we can imagine to reflect the mean body size of a panda that doesn't need broccoli or something. This is a standard deviation, so we know what type of prior we want. Here, we want some sort of prior that assigns the highest probability to our null hypothesis, which is no slope, basically, so zero. In the case of a neural network, the parameters are not directly interpretable, so defining a prior on them is not as trivial. So priors here will be typically centered on zero, but if we wanted to add some extra information, it would be more difficult to do that. Okay, got it. More questions? Okay, so these are some drawbacks, basically, of doing these types of model. One specific concern is the risk of overfitting. And this, in particular, in machine learning, this is like a crucial point, because machine learning, unless it's using Bayesian neural networks, is using essentially a likelihood approach. And in a likelihood approach, the typical way to fit a model in a maximum likelihood context is to find the parameters that maximize the likelihood of your data. So that's the typical way to fit a model in maximum likelihood. The problem here is that you have so many parameters that if you did that, you would be almost certainly overfitting because you have so many parameters that if you maximize the likelihood of your data, you would find parameters that are perfect for your data, so like perfectly matching your data, but it would have very poor predictive power because the model would be overfitting the data. So what people do in machine learning, which is not Bayesian neural networks, but we'll come back to that, is typically to split your data set into a training set and a validation set. Okay, so you just take your data set and split it into two chunks. And you sample your parameters or you optimize your parameters based on the training set. In this case, you will be minimizing a loss that is basically the inverse of the likelihood. But at the same time as you maximize or minimize the loss or maximize the likelihood on your training set, you will also monitor the likelihood or the loss of your validation set.  And for a while, these two validation and training loss will follow a similar trajectory, so they will decrease or their likelihood would increase. At some point, however, the training loss will continue decreasing, but the validation loss will start increasing, meaning that after some while, the parameters are being optimized toward overfitting the training set. Okay, so the parameters are getting really good at predicting these three pandas, but they start to become better at predicting these two pandas. So monitoring the validation loss, you can detect when the model is starting to overfit and then basically stop the fitting process. So instead of maximizing the likelihood, you stop the optimization at some point. And you will basically stop at the moment where the loss, the validation loss will start increasing. So the optimization stops here and these are the weights of your model. This is in a typical machine learning framework, so not in a Bayesian framework. In a Bayesian framework, the risk of overfitting is limited and it's limited by the fact that you have priors. These priors that you set on the parameters, they are typically centered in zero and they have a regularizing effect. I'm not sure if we have seen regularization or Bayesian shrinkage before in the course. So regularization is typically this effect that priors have on like preventing overfitting. Regularization is used a lot in multiple linear regressions as well by using priors on the coefficient set, basically reduce the risk of overfitting. Here, having so many priors on so many parameters will effectively prevent the Bayesian neural network from completely overfeed the data. Validation sets will still be useful in Bayesian neural networks but they're not strictly necessary here. So you can run your MCMC as long as you want and at some point the model will reach a convergence and that convergence point is likely to be about right, so not really overfitting. Any questions about this? Yeah, can you explain the last part again of the prior use? Yes, so the priors are used in a Bayesian neural network by basically calculating for any given value of your weights you calculate their prior probability, right? Just like you would do for any free parameter in any model in a Bayesian context. So in any Bayesian framework, when you have a model like this, for any given value of B, here you will calculate its prior probability based on this distribution. For any given value of C, when you calculate the prior here and any given W value calculate the prior here. The posterior probability will be proportional to the priors, so this probability times this, times this, times the likelihood, that is the probability of your data, right? So this is the standard Bayesian construction of a model. In a neural network, you basically have priors on the slopes, right, on these weights and these priors will have a regularizing effect because they will not allow, they will constrain basically the parameters. So in a B and N what we have is a likelihood function that looks like this and then we have all of these priors applied to the weights. So the weights cannot just go anywhere just to feed the data. They will be constrained by this prior. And the prior will favor weights that are kind of around zero. Okay, thank you. Yeah, so this type of priors is used also in a multiple regression context where you have multiple coefficients and using this type of prior will do regularization in a multiple regression. And similarly will do regularization. Regularization means like it will essentially prevent the model from overfitting. It will do so also in a neural network. Does that clarify? Yeah, exactly, thank you. Any more questions about this? Now there are many parametrizations that neural networks can take. So we can change the activation functions as we mentioned earlier. Rail, Loos, Sigma, there are very many activation functions that you can look up online. There are typically simple things to compute but they're always non-linear functions. So you can change the activation functions, you can change the number of nodes, right? So this will just increase the number of parameters. You can change the number of hidden layers. This will make them become deep neural networks. The operations here will still be the same that we did before, right? It's just that you do more of them. How do we choose among these different options? Ideally these different options will not have a big effect on the outcome. So if that's the case, you don't have to worry much about it. In some cases you can have, you can use a validation set, even in a Bayesian network, to choose between different architectures of the models. For example, you can have a case where your training set will sample likelihoods that are lower for a very simple model and increasingly high for more complex models. So for most of you more parameters, this is not necessarily the case, but it could be the case. But if you then monitor the likelihood samples through your MCMC, for example, in your validation set, maybe you find that this is the distribution that you're sampling. And if that's the case, then this model would actually return the highest validation likelihood, meaning that this model is maybe overfitting a bit. So in this case, I would choose this model. This is just kind of a rule of thumb. It's not really strictly model testing, like explicit model testing. Explicit model testing in the sense of calculating marginal likelihoods for a Bayesian framework or AIC likelihood ratio tests or these kinds of things in a maximum likelihood framework. This type of model testing doesn't exist for neural networks because there are just too many parameters that you would have to basically test. But yeah, so once we choose our best model, we can do predictions and obtain predicted body masses for our pandas, given any combination of broccoli intake, genome, and range size. Yeah, so that was all I had for the regression part. Any questions? Do you need a break? Good to go. All right. Then let's keep going. The next thing that I wanted to talk about is networks for classification. This time, this will be a lot easier because we have seen already the neural network as a model. And classification is just another type of task that a neural network can deal with. And most of the network structure will be the same as we had before. So we could have the same input that we had before, the same nodes here. But our question now is not about predicting the body mass of a panda, but it's predicting whether a particular diet, genome, and range size is associated with a panda, a brown bear, and a cave bear. So you can see already from the structure here that the model itself is not very different as except for the last part. So the classification task will be basically to find the parameters here that for any given diet, genome, and range size will assign the highest probability to the correct species, okay? This first part of the network runs just the same as we have seen for the regression model. The second part of the network or the last part of the network, the output layer is now different because instead of predicting a single value here, being before the body mass of a panda, here we need to predict basically three values, which is a probability associated with each of the three species that our model includes. So our model here has to end up with three nodes. And each of these nodes will become a probability. The first one will be the probability assigned to a panda, meaning that these diets according to these parameters will be assigned with that probability to a panda. Second probability will be assigned to a brown bear, the third one to a cave bear. So here instead of having a single node as we had before, we have now three nodes because we want to end up with three probabilities. And these values here, which will come from propagating the features into the hidden layer and into the output layer, these numbers will have to be transformed into probabilities. But these values can be positive or negative, right? They can take any value. And so to turn a vector of any range of values into a vector of probabilities, we use the softmax function. There is a simple function. Again, you can write it in one line in Python and in R as well. That turns any number of, any vector of numbers into a vector of probabilities. The equation itself may look more complicated than it actually is, but what we're doing here is basically exponentiating each of these values. And then we divide each of these exponentiated values by the sum of the exponentiated values. When you exponentiate a value, whether it's negative or positive, you will always get a positive value. So then you divide the positive value by the sum of the three values. And so these three values that you obtain sum up to one, basically, okay? You divide them by their sum and you end up with three numbers that sum up to one. So they qualify as probabilities. So the only difference basically in this network is this one function here that is called softmax that turns a bunch of numbers into a bunch of probabilities. Now, what's the likelihood function in this model? Before we were using a normal distribution as a likelihood function, the PDF of a normal distribution. The likelihood function in this model is even simpler. It's basically the probability mass function of a categorical distribution. The likelihood is already given by these probabilities. So under these weights, and these particular broccoli take range size and genome, my model gives me a likelihood for this panda of 0.11, so 11% probability. By optimizing my model, I'm going to update these parameters, these weights, and I will favor parameter values that give me a higher likelihood for my panda, right? Which will mean that these broccoli, these genome and range size will be associated with higher probability to the correct species, in this case, a panda. Okay, so again, here I will have a training set, so a bunch of species for which I have the predictors and the label, so the species. And then the model will be trained to find the best answer in all of these cases. Again, we have the issue of potentially overfitting, which is clearly a problem in regular machine learning, less of a problem in a Bayesian neural network. In the case of a classification task, overfitting would mean recognizing exactly each item in your training set, like you do here. But then this would result in a poor predictive power, meaning that if I give the model a data set that is different from the training set, then the model will misbehave. So as usual, we want our models not to overfit, so that we have better predictive power. This is done using priors as a regularizing function here. Again, we can also do the same approach that we used before. So we can also split our training set and validation set. So we can split our data into these two training validation sets and then look at the validation likelihood to see which of the different models returns a better likelihood for the validation set. And if a more complex model returns a lower likelihood in the validation set, it probably means that it's slightly overfitting. The degree of overfitting here will not be as high as in a regular machine learning context. Now, if we optimize this model for classification, we're gonna get a bunch of posterior weights, right? So distribution of values here, which means that for any given diet, genome, and range size, we can now make predictions and predict whether these observations come from an individual of these pieces, these pieces or test pieces. In this case, if the model was trained well, and if the ground truth is brown bear, this diet will be linked with the highest probability, posterior probability to a brown bear. We can get a range of predictions basically by running the same features, the same predictors, we can run them through the neural network across many of these parameter values. So we get a range of predictions and we can collect all of these predictions to obtain posterior probabilities of these features being associated to a panda, a brown bear or a cave bear. If the model is trained well, it will give us a good answer, the correct answer for most of our data. One of the reasons why Bayesian neural networks are really useful is that neural networks, like non-Bayesian ones, tend to be very certain about their predictions, even when they're wrong. So just looking at these probabilities, which you will get also from a standard neural network, is not really a good way to determine how certain you are about your prediction, because there is a demonstrated tendency of neural networks to be very confident about their answers, even if they're wrong. And this is a problem if you, for example, feed a model with data that doesn't belong in any of these three species, which could happen. In a Bayesian neural network, because we have this regularizing effect and because we can compute posterior probabilities associated with each class by running the model through a range of posterior weights, when you feed the model with unknowns, so with out-of-distribution data, diets, a diet genome and range size or something that the model has never seen and that is not included in the output, then a Bayesian network will typically give you uncertainty in the outcome. Like, I don't know which of these three it is. It cannot directly tell you it's something else because it's something else doesn't show up in this vector here, but it can at least tell you that it doesn't know which one it is. And it will do that by assigning low probabilities for posterior probabilities to all of your classes. So this is a major advantage of Bayesian nets. We can then evaluate the accuracy of our predictions using confusion matrices, which you may have seen before and calculating the accuracy. Basically, we can take our validation or test set and see how many times did the model get it right or wrong. This is to get an idea of how good the model is, right? So in this case, out of 37, I got 33, they were correct and that's an 89% accuracy. We can look at the confusion matrix, which will basically tell you for every class how many of them you got right. So I had 14 pandas in my dataset, all of them were predicted correctly. I had 10 pairs here in my data set and I predicted nine correct and one wrong. And this one wrong ended up in this class. Okay, so this is what the confusion matrix shows. Here I had 13 K pairs and 10 were predicted right and three were predicted wrong. Now it's important to look at this prediction at this confusion matrix because like high accuracy doesn't always mean that your model is good. And especially this is the case when you have imbalanced dataset. In this dataset I had a lot of pandas and very few bears. And so I have a very high accuracy here, but what the model is predicting is basically that everything is a panda. And even so, like you still get a high accuracy just because the test set is biased, it includes so many pandas. So saying that everything is a panda will still give you a 92% accuracy, but maybe the model has not learned anything. So this is something to be careful of. And this is a general issue with supervised learning. So supervised learning means that you train a model based on some observations and the model will only do as good as the observations allow it to. If you have the chance to experiment machine learning with toddlers, it's pretty amazing because toddlers are basically like untrained neural networks that you can try to train. And I did that twice. And you can see that like toddlers will mistake things in ways that you would not really imagine before. And this is coming from their limited training set because they have not seen everything yet. So they will identify a big tail as a snake if this was their training set or a half moon as a banana. And so this happens to toddlers, then toddlers know better after a while. Machine learning models also have this type of problem. So they need to be trained properly with proper data sets so that they don't end up giving you answers that don't make any sense. I have a question on a previous slide. Please, yes. And I admit that I have not attended the course so maybe it just comes from limited knowledge. So one prior to that, where you have the even one prior. Yes, here, where you said, okay, if really you have data that is not from any of the three bear species, it will give low posterior probabilities among all the classes. So does this mean that they don't need to add up to one? Oh, they will still add up to one. So here they will be 33% each. Ah, okay. So you call 33 a low probability. Okay. Yeah, I mean, yes, like typically you would say I'm confident about the posterior probability if it's greater than 95% for example. Right, right, right. So they're just talking about each one but they do need to add up to one. Yes, they do. Okay. Perfect. Thank you. So it's basically up to you to define what threshold you consider significant. You can do this by simulations or you can use some standard cut off like 95%. But this will be a case where you basically don't know what the answer is. And it's a good thing because the ground truth was that the data come from a different species. More questions? Yeah, maybe one question about the training data. So how big should it be? What is like normal numbers you normally train on? So yeah, that depends on the type of model that you're running. Like if you have a model that is like simple like this one with a few predictors and kind of a simple outcome then you don't need very many training samples. The good thing of Bayesian neural networks is that they scale or that they adapt well to small data sets because they have these priors that you get posterior probabilities. So even if you have a small training set that's okay because the model will basically give you more uncertainty on your prediction. So you don't risk basically getting very confident and very wrong answers. That's why Bayesian neural networks are particularly useful for small data sets. Meaning like, I don't know, like a few tens of these observations would be enough to train a model. And it really depends on how many parameters you have and yeah, how complex you build the model. Obviously, if your data set is small you won't go for like many hidden layers and many nodes. You will tend to go for simpler neural nets. But in a BNM usually you will get a quantification of the uncertainty in your predictions whether it's a classification task or a regression task, just the same. So even in a regression task if your training set is small then you will have a wide uncertainty interval around your prediction, right? So that's good. In a regular neural network then you would need more data set more training data typically to get decent predictions because you have fewer ways to quantify the uncertainty that you have. Okay, so if I understand it correctly the main difference is that normal neural networks they really try to give you a precise answer and while in Bayesian you get basically this distribution as an uncertainty so it doesn't need that much, that bigger samples, right? So yes, that's basically, I mean it's the difference between maximum likelihood and Bayesian analysis. Like in a maximum likelihood analysis you get a point estimate and then there are some ways to get some sort of confidence intervals but you get a point estimate. That's your prediction. In regular machine learning you get a single value here or a single choice basically in a classification task. Boom, it's a panda, that's it.  So if you have fewer data this will be reflected in wider credible intervals and this is true for Bayesian neural networks for any other model basically. Little data means more uncertainty in a Bayesian model which is why we use Bayesian models and particularly so with small data sets. That makes sense. Yeah, okay, thank you. But so this BNNs are basically better for classification so if in analogy it's like a supervised machine learning right? Well, this is a regression task, right? Okay. This is a regression task. Yes, I'll give you an example of a non-supervised BNN in the last few minutes if you... Okay, cool. It's fascinating, thank you. Good. Yeah, I would basically... Or is there any other question? Otherwise I would show you a couple of examples of things that we use BNNs for. Then I'll start and then if you guys have questions just interrupt me. So one application where we've used a Bayesian neural network is in a context of conservation or computational conservation biology, you probably say. So you probably know that there is a red list that classifies organisms for their extinction risk. It's compiled by the IUCN and so the red list basically takes every species of animal or plant and tries to define whether they are endangered or not to what level they're endangered at risk of extinction. And this is great if you're interested in mammals and birds because they're all assessed by the IUCN but if you are interested in other organisms this red list is very incomplete. Okay, so there is very little known about plants even less about invertebrates in even less about fungi. So here we use Bayesian neural networks to make predictions of these classes. So this is a classification task and as predictors we use basically data coming from GBV that is like occurrence, biological occurrence data that spans like many more species than the IUCN red list. And you can take these occurrences basically and turn them into features that enter a neural network and then you can use these occurrence records which will kind of approximate the range size of a species, how many individuals of the species there are and so on but you will take these occurrence records, feed them into a network and then get as an output a prediction of their extinction risk. And so you can train the model with the red list that already exists and then apply to the other species that are not yet classified but for which occurrence record can be found. And for example, we did this on plants in Madagascar and out there where the IUCN has assessed 4,000 plant species in Madagascar using this model we were able to add additionally 5,000 species. So we more than double the amount of species that have at least an approximation of their predicted extinction risk. And in some cases this will be like qualitatively different. So for example, for firms only 3% of the assessed one were estimated to be at risk of extinction whereas with our BNN predictions we get between 38 and 57% that are at risk of extinction. So this changes like qualitatively, you know the picture and because we feed the BNN we get a credible interval because we run a Bayesian neural network we don't get a point estimate we don't get a single classification for every species but we get a distribution of classifications for each species. So we can tell that between 38 and 57% of all species are threatened among firms in Madagascar. There are also cases where we use these Bayesian modeling networks in a non-supervised approach. For example, we have a model that is used that looks at fossil data to estimate parameters of speciation and extinction rates. Basically they quantify looking at the fossil record of a clade for example, mammals. They try to ask these models estimate how speciation and extinction rates change over time. Like how fast species diversify and how quickly species go extinct. And these are models that are implemented in a Bayesian framework. But we can also use neural networks here to try and link changes to speciation of speciation and extinction rates to particular things. For example, traits like phenotypic traits of the species, temporal events and things like that. So here the likelihood is not that of a classification or that of a regression. So the likely functions are completely different. They're based on Poisson and birth death stochastic processes. So it's a completely different context but we can still use neural networks to make the connection between some predictors and the parameters of this model. The parameters of this model are speciation and extinction rates. And we can make the speciation or the extinction rate a function but a non-linear function of a number of things. So the speciation rate could be a function of climate, time, some traits, diet, overlap with humans. And so because we don't know a priori what types of interactions or what type of responses these things may have on the speciation and the extinction rate, we use a neural network here. And this is unsupervised because we don't know the ground truth but we can calculate for any given lambda here or for any given speciation and extinction rate we can calculate the likelihood of our data. And so we run this model to estimate speciation and extinction rates indirectly as a function of all these network. And what we found, for example, for elephants, elephants are an interesting case because they've been around for a long time. They have a very good fossil record but the diversity dropped very quickly, very recently. So until a few million years ago there were between 20 and 30 elephants, elephant species roaming across all continents, almost all continents except Antarctica and Australia. Now there is three species, only two continents. And so if we are to try and find out why they diversified the way they did, we can use these types of models and correlate speciation and extinction rates with specific traits and events for elephants, which in the case of elephants was in diet and traits overlap with humans and climate. And what we found is that there are non-linear complex relationships between traits and diet that explain speciation rates and that humans basically are the one predictor for the extinction of elephants, which kind of didn't survive us too much, but our model was able to quantify that 8 to 20 fold increase in extinction rate linked to humans. So this is another example of using a basic network inside an unsupervised model. And I hope this wasn't just confusing but there was a kind of what I had for today. Thanks for listening. Do you have questions? Yes, I'm not sure I really understood how you find the weights if it's unsupervised. Yes, that's a good question. So our model is using, without the network basically, is using the observed fossil record to estimate some sampling rates basically and the speciation of the extinction rates. And so for any given speciation of the extinction rate and fossil data, we can calculate the like, or we can calculate the likelihood of any of the fossil data given any combination of preservation rates and speciation of the extinction rates, okay? So that's a likelihood. It's unsupervised because we don't know the true speciation extinction rates. But we can use, so this is like a regular likelihood approach where we basically, we have the data week, we have a likelihood function that tell us what's the probability of observing this data given at this point in time, for example, given this preservation rate and these speciation extinction rates. And these likelihoods and prior functions are based on these Poisson and Perth test processes. So here we use the network to basically modulate how speciation extinction rates change over time and across species. And the network will only basically speed out a particular speciation rate for a given species at a given point in time. And given this value, we can compute the likelihood of our data. This clarifies a bit. I mean, in most evolutionary biology, models are unsupervised in a sense because we never know the ground truth, right?  And the same applies to phylogenetic inference where we try to estimate the phylogenetic tree connecting, phylogenetic relationships connecting species. We estimate them based on likelihood that we calculate, but we don't know the ground truth, right? So we hope that our model is doing a good job. And so this is similar here, but the way to modulate the parameters is given by a neural network. More questions? Yeah. Yeah, I have a couple of questions. And so they are about the classification case and they both have to do with how to best interpret or present to people who may not even be used to Bayesian thinking what comes out of such a classification task when you have, when you don't have a point estimate, but when you actually have a distribution. So the first question is, yeah, just how do you have any experience with how you would make some kind of plot or a table or output that will more guide people to think in this framework of we have, we have probabilities for each class rather than a hard assignment to a class. So this is one of my real life issues. In the end, there are people who read what I have, but they, in the end, are very interested in is it this or is it that or is it that? How to communicate better that, well, it might be this or it might be that and this is higher probability, but the other one is perhaps also not wrong because the probability is not that much lower. I don't know if you have any creative ideas about that. I mean, communicating uncertainty is not always trivial because well, many scientists I think know what uncertainties are, but yeah, I agree that it's not always easy to convey this. For the case of classification, you get actual posterior probabilities, right? So you can basically, in this case, I can tell that I'm like 80% or whatever it is confident that this diet genome and range size are associated with the brown pairs. So you actually have a number and if you want, you can use thresholds. So what I did in some cases is to set thresholds basically, or for example, using a test set, you can set thresholds under which you are not confident in any classification. And if you have a test set, so if you set aside a bunch of your data, you can say like, okay, if I just take the one, the prediction with the highest probability, whichever the probability is, even if it's like 40%, then I will have a certain prediction error in my test set. But I will do predictions for all of my samples. But if I say my thresholds for accepting a prediction is 90% for example, then maybe my prediction error drops to 1%, so I'm more confident, but that could also mean that every time I don't get a 90% positive probability for a class, then I refuse to do the prediction. Meaning that you have 1,000 data points, you make predictions maybe only for half of them, but for that half you are very confident. And then for the others, you don't do the classification. And that's something that you can do with a BNN because it gives you these positive probabilities. You can just say like, any outcome that is below whatever, 90% for zero probability, I will not consider. Yeah, and then I have another question also about this classification task. What actually happens if you have a set up where say you have three classes, you have a small number of classes like here in this example, but some of them say, two of them are very close, say they are a brown bear and a black bear, and one of them is very far removed, let's say it is a mouse. Then what I would expect is that, okay, you will probably get a lot of cases where it's kind of uncertain between the brown bear and the black bear, if it's one of those, and it's usually very certain when it's the mouse. So then if you use something like you just said with a threshold, then it will miss a lot of the bears because it's very often very uncertain and you will just say, we don't know what it is, but actually you do know something, you do know that it's a bear, but it will just, in a result like that, it will just get lost and will say, I don't know anything. Do you have any notion of how one could deal with that, to not lose all of this information that it's actually not a mouse? Yes, so first of all, you can calculate the accuracy per category, right? So we've seen the confusion matrix, you can basically look at each row here and produce an accuracy. So here my accuracy for pandas is 100%. The accuracy for brown bears is 90% and the accuracy for black bears is, I don't know, whatever percent. From the training data, what do you know? Yes. So yes, it always, yeah, from your test set, right? So you can tell what the prediction accuracy is for every class. If you have a case as you say, where like for example, here pandas are very different from the other bears, and so for the other bears, what you will see is that the confusion matrix for this part of the confusion matrix will be more fuzzy, right? But what you could also say is, or decide to do is to summarize these two classes and say like, maybe I have a 99% accuracy in determining that this, that a given item is either of the two bears here and it's not a panda. So maybe the accuracy over these two rows is 95% but then I am not sure within this block, I don't know. So then you don't throw them away, but you can say like, okay, this is at least one of the two classes. Can you just add probabilities for that or is it more complex? Yeah, or you can like, you can, you can calculate the accuracies, right? Because you can say like, here I have 14 on one side and 23 on the other side. You can calculate the accuracy, like how accurately can I separate pandas from the other two? And so basically it's like doing a confusion matrix with only two classes, where here you put both bears in one row. You can aggregate them afterwards, right? And calculate their accuracy. Yeah, and is it correct or is it not correct? Say when you're actually using the model for classification to add the probabilities. So say you have your classification into three classes and you still want to know that but then you also want to know how high is the probability that my individual falls into one of two classes or two of the three classes? Is it correct to add the probabilities that were spit out from those two classes or not? Like, I mean, maybe we don't even have point estimate so it's a little bit hard to know what that means. I mean, what I would typically do is just to do many predictions because you can because you have all of these weights. You do many predictions and then you collect them, right? And then you see like how many times they categorize these pieces or these items as these pieces, that's pieces, that's pieces. And then like you have these frequencies and then, yes, you can sum them up, right? So 33% of the times this was this, 33% of the times it was that. So 66% of the times it was one of these two. Yeah, okay. So yes, that's fine. So we didn't do it directly on the probabilities you get here but on the something, on the posterior frequency of the classification. Yeah. All right, that is very useful, yeah. Thank you.


back to JSA TV. Jamie thank you so very much for inviting me. It's always an honor to be here with you at JSA TV. Thank you and thank you for speaking today. You're in state on streaming media, opportunities, demands, always amazing and you really bring that that sub-C perspective to the table. Thank you so much. No thank you for inviting me. It was a lot of fun and I think that we've heard from different people, different participants in the value chain, their perspective of what's coming and it's exciting times for anybody in the space. Yeah it really is. It really is in 2020s. Game Buster. Can't wait. So let's get back to 2019 and how you guys have been tearing it up. I think it's seven awards in total that you've won just this year alone. Is that crazy? That is amazing and I'm so proud of the Globe and a team. The fact that in different venues we're being recognized for the quality of our work, the service that we deliver, the innovation that we have put into the market, it's great. So we're actually honored to have received those awards. Well I mean it's it's clearly deserved. I think the entire industry would join me in saying that. You of course have Malbec, not just a fabulous wine maybe you say, but GlobeNet's new 2,500 kilometer sub-C cable to Argentina with Facebook. So recently you guys won Best America's Project for Malbec. Tell us a little bit more about that. Well we did. I think the idea of Malbec is not new. It's how we're implementing. I think that that is great. We're in a joint build agreement with Facebook and it's all about partnerships right and part of our DNA at GlobeNet is to embrace partnerships and this is a great example of what you can actually do. It's a fun project. It actually completes a vision that we've had for many years to be able to extend the network into Argentina. You may recall that we did something similar back in 2013 where we extended our network to Colombia and we're in the middle of the actual construction cycle. We're gonna start laying early next year. So around mid 2020 we're gonna land the first cable in Argentina after some you know since the early 2000s so it's been over 20 years right. Yeah it's very very cool. Alright so I also hear that your infrastructure value pain if you will. Can you tell us about some projects we might see soon? Sure so let me tell you first about some of the other things that we're doing that have helped us win some of these accolades. In our Fortaleza facility which has been with us since the let's say the Genesis of GlobeNet we've actually converted a dual cable lining station into a true carrier-neutral interconnect facility and I think what what has set us apart let's say from others that have tried is really that neutrality right being carrier-friendly every single local provider is in our facility and the fact that we built a partnership again and this time with IXPR to deploy to deploy it in our facility in Fortaleza made a huge difference. Northern Brazil has been one of the fastest growing areas in Brazil from an economic perspective. There's a series of new network providers that have emerged over the last let's say five or ten years but mainly over the last five and we have them all as our customers and partners in the facility right and again it's this model of you know being carrier-neutral being carrier-friendly setting the right the right goals in where we said look this is what we're gonna do this is how you're gonna interconnect it's clear to everybody that wants to come in and we're lucky that we have several major content providers that have also selected a facility to have a pop there just because of that right so I think that's that's another example how we're going from your traditional let's say subsea service provider right international subsea server provider to look at you know what's next you know how do I grow that value chain how do I leverage the assets that I have and how do I leverage our origins of an infrastructure company right the other thing that that we did is we built a small-scale data center in Barranquilla in Columbia and so that that was our genesis to look at the data center space right where we see that that having that neutrality is important and and we are definitely looking at that space as a growth area for the company it's the neutrality but it's also having clear set rules right this is how you're gonna interconnect this is where you're gonna interconnect not hey let's be creative and just you know hook up in the in the beach man or the manhole not the beach manhole the manhole no it's but to say you're gonna come in here you're gonna set you know put your wrap here your rack here this is you know the cost for your cross-connect all that stuff needs to be clear and then you start attracting people because you're you're fair you're neutral right oh absolutely and that's why I mean we've converted Fortaleza into a major network interconnect hub for the region so all right so looking forward to the new year what is ahead in 2020 2020 for us is a very important year I mean we already mentioned my back the landing of a new cable into new geography into a new country that in itself is right so so that's let's say our number one milestone for 2020 mid 2020 so complete let's say our landing right but but beyond that I do believe that there's other things that we're working on we continue to do network upgrades throughout our system we continue to establish further partnerships you may recall that years ago you know we we looked outside the box and we said or thought outside the box and said how can we service Manaus and for that we also got an award many years ago that you may recall and so we're looking at similar regions I'll disclose them at the right time right where we're extending network services again through partnerships I think it's a combination and it's really part of our DNA that we've had for years as you partner with threshold providers you partner with service providers you partner let's say with OTT such as Facebook and you're building that ecosystem that makes it attractive for other to want to be there absolutely and that's what we're working on so globe net 2020 another year of more growth for you I'm so excited Eric as always and I'm so proud of our team and I thank our customers for entrusting in our network their services it's all about that it really is thank you so much for joining us and being here thank you viewers for tuning in to JSA TV happy networking


So we get started first with your foot inserts. I'm going to have you put these in the plastic sheaths, and then you'll go ahead and place them with the long piece I got from your shoes. By doing this, you're going to simulate a little bit of pain, discomfort when you're walking around. And for people with dementia, they have pain that they feel, but they're not able to communicate it to the appropriate people or staff.


Okay guys, let's try this problem and make it a little harder. How many grams does 1.73 moles of boron weigh? So we know again from the periodic table that the mass of 1 boron atom is 10.811 anu. And we also know that we can approximate 1 mole of boron to equal 10.811 grams. So we just use this conversion factor to figure out this problem. So 1.73 moles per boron, multiplied by the conversion factor, 10.811 grams, divided by 1 mole, and then just multiply 1.73 times 10.811 that equals 18.7 grams. Good job guys.


iscussion prior to voting, I think I would really appreciate, I would also appreciate if Suzanne's able to do it, tax rate projections like we did last year's presentation based on a couple of different values of houses, and I'd also like to see the full equalized tax rate rather than this 5% cap just for a frame of reference, what 10.52% tax rate increase would look like as part of a brief presentation prior to voting. And then our next meeting would be like, it would be the board taking ownership, but you know, after that meeting, the board will take ownership of the budget, but yeah, okay. Suzanne, can you remind me one percent was, was it 300,000? No, what was it? One percent on the tax rate and don't know right now Okay. Tax rate or the budget? I don't remember. You're talking about the budget? No, I'm not going to tell you that. Net localized spending, it was about 300,000. Okay, thank you. Can you talk about that? It was 300,000. So I thought Daniel's suggestions were good, and also we'll obviously have the CLA numbers, so we could have another round of discussion with some new information. I do have a quick reflection, Tab, if that's okay. Yes. I don't want to speak for everyone, so I'm going to practice using my statements. I'm going to practice what it is. No, sorry. And just name that there is grief in this process, you know, to what Joshua was naming, like there's so much beyond our control. There's so many emotional feelings that are coming into play here, and I think that we can't deny that there's cognitive dissonance because we care so much about nurses, and counselors, and our kids getting their mental health and physical health needs more than ever because of the pandemic and all the additional stressors and trauma that families have gone through. And to Floor's point, trusting the administrators, you know, through the rounds of conversation and looking at the data and looking at their assessment of what the needs are and how they're diversified and supports to meet the needs, I think we need to come to just reach a place of acceptance with ourselves for acting potentially and making a decision that symbolically is an hypothetical to the values that we hold. And knowing that the levels of complexity and nuances in how the system is structured makes it so that both can happen, that the needs can get met, and that we're cutting counselors and nurses potentially. It's perfect, I don't think I could top that, so moving, we are moving to our next item, if I can get each and every one of you here, thank you everybody for your hard work. Thank you, don't feel like you have to. Thank you, thank you, the rest of us. Yeah I think you should take the same option. Exactly, right? Why would you cheat, am I being disgusting? You want to burn your spoons. That's right, yeah. Well never goes, now what do you wanna do? So, the moving to the Board of Budget Communications and the steering committee, the steering committee met and ladies just want to give you, we had a meeting with fans there. So that quick update of the flyer that we talked about sending with the third committee, and then I'll talk to you again. Just real quick, one of the things we talked about is last year the board chose, and it was successful, I think, to send cards to everyone, all the member of the QR code for the annual report, instead of printing the entire annual report. I think that worked really well. We had very few, if any, requests for having mailed this year, the steering talked about, instead of mailing a card that just has a QR code to mail a budget flyer, so that when they receive that mailing, it has the CliffsNotes version of the budget. So that's right there. Everyone will get that. Still will have the QR code, so if they want to put a thick annual report, they'll still get it. We met with Ben Merrill today. He's our sort of communications graphic person. He said that sounds like a good idea. He had some good thoughts about what to put in it, so we're working on that piece. Do we also have several copies at town offices? Always. In fact, even with the reduced number of printing we did last year, we probably still printed too many. But yes, there are town offices, there are all the schools. And it's in the central office. And we, there is always a note on the flyer that if you want us to mail you one, we will. And tell them how to access it. Tell them a good idea. This gives information. Yeah, we'll have a little bit more information for people that won't go into the QR code. And we'll see that the difference between a trifle and a hateful. Yes. It gives us a little more room depending on. Yes. And is it possible to even have like a poster or a flyer that does have the QR code that could be posted all over? It will have it there. Yes, he also said that the mailer will be the trifle with the QR code and he could make different portions of it into one pagers. Like I was just thinking it's on the school door. Yes. You know what I mean? Or at the mail post office or something. You know what I mean? Yeah. Let's go. Yeah. You don't have a post office. Yeah. You don't have a post office in the mail there. We don't even relive it. It does one part of them we'll talk, we have some creative ideas today when we were meeting about ways to do, I'll reach for a community and maybe, have the bios and groups of three and go to a way to explain the budget like separating board members across communities and having other ways to share the budget with our community. That necessarily means having the entire board together once we agree on a budget, we'll have more conversations on that. The mail-in ballots for watching center, I'm just waiting for a text from, I'm assuming that by now, Berlin it said yes. I'm gonna leave that to the end of the meeting to make a motion just because I haven't checked my text right now. A Gnusky Valley data motion. It's 23. I move the appoint. I move that we allow Megan to sign on it. Yes, that's all. Amen. I'm looking for the right word. Second. Thank you. You got that, Lisa? Yes. All those in favor, please signify by saying aye. Aye. Can you both? Heading none. The motion carries. It's for general relation, page 24. I'm gonna let you leave that, Kari. Yeah, so starting on page 13 of the packet is information about our superintendent performance evaluation, one of the most important things we do all year. And so the next steps in this process will be, a survey will go out right after the new year and you'll have a little over a month to complete it online. Probably repeat for most of you, but it's not a long survey. The rubric is in here. Take a look at it and we'll give you plenty of time to fill it out. We will compile it, we will share the results with you all, with the board in executive session have a discussion and then we'll sit down with Megan and speak with her about it. That's the process. Very important. Our goal is always that 100% participation in the survey, so it's a half an hour to do that. The next one, we added a new point at Clark. You guys will receive a general presentation. Great board member, we thank him at this board meeting for his service to the board and to the community. We're not gonna look into trying to interview on a point of our member right now because we're so close to the election and we're in the middle of the project. So we will look for somebody to run for the Wisterlitz seat, but for right now we need a clerk for our board. Can I just ask a question about Jonas's sort of other roles he played for the board, committee membership and anything like that or steering committee membership? Are those things that need to be addressed or reassigned in its absence? That's what we would do once we... Once they're done. Once we have a... I was just assuming whether we appoint Clark would assume the roles for right now and we're done with negotiations which was the committee that he and Diane and Joshua and Chris are getting. Chris. Who is he? Hey. I don't know. I don't know. Especially because I see that you took your breath. So I'd like to nominate Joshua Sevenths to be our clerk. Okay. All right. Okay. Are you accepting the name? I... Yeah, yeah. See? All right. All those in favor, position you by by saying aye. Aye. Aye. Aye. Thank you. Congratulations. Thank you. There's a 15 page report. No, no, no. No, no, no, no. No, no, no. No, no. No, no, no. This isn't sure. No, no, no. No, no, no. No, no, no. The one thing that I do want to, and if you guys are okay, it would make, because he would take the position also in the steering committee. We don't have that many more steering committees before the reorganization. It would mean that Romney would have two seats on that committee, but I think it's okay. Is the clerk automatically members of the steering committee? Is that what you're saying? Yeah. So the vice chair, clerk and chair are automatic members of the steering committee? Yeah, that's fine. Let's see if that was that. Okay. I think it would be good to just make a change. Yeah. All right. Moving right into the finance committee. We're select. I want a Doty Generator bid. I move that we approve awarding the Doty Generator Project contract to local electric LLC in an amount, not to exceed $81,400. Second. Any discussion? Yay for refrigeration and emergencies. Right, yeah. Reducing stress for their community. Yes. Would this power the entire building? Yes. And if something. Yeah. So the plan is that in interim, we're going to purchase a smaller generator for this winter because the one that you're awarding this bid for won't be installed in time for this winter. That smaller generator would then be transferred to the central office once this larger generator is installed at Doty. We're going to buy a small one, like a cheap one, just to keep the freezers and- Yeah. Over that? And also keep it- Oh, okay. Yeah, yeah. I'm just thinking of emergency shelter, actually. Yeah. Because if they're powered the entire building. Yes. With the intent with the larger- This generator. The smaller generator, which is much less cost, like in the $700 to $900 range, but then be treated. It's generator cost plus installation. No. $700 to $900? $700 to $900. Yeah, it's just to- $700 to $900. It's only going to do the generating. It's only going to do the freezers. It's only going to do the freezers. $700 to $900. It's only going to support it. I know. Oh, yeah. It's just to do the freezers. Is it like a buy school hour? Yeah, but you're talking to the girls. Yes. You're actually going to hire your own cycle. You're just going to come right in. You'll just charge one phone? No. It's two o'clock in the morning. We need your service. And then the protein. Then we transfer it over to the central office so that if we lose power there, we wouldn't have things freezing up on us there to protect the asset there. And the town has potential for applying for a grant that might offset the cost of this. Our generator after it's not figured into it right now, but it might be a added bonus for us. Well, those seem to say we're missing the five I say. Hi. Hi. All right. Any opposed? Any abstentions? Here we go. We shouldn't get through pre-qualified criteria for security and project. I move the board established the recommended pre-qualification criteria that contractors must meet to be included on a select list of pre-qualified bidders for the 2024 security system project. Thank you. Thank you, Chris. Any questions? Can I just ask a quick question that is raised for me? What's the threshold for which we don't approve capital investments? Or maybe they're not called capitals at 20,000? No, that one's 40,000. We don't go out to bid for the 40,000. The 40,000, yeah. But board authorization? Board authorization comes when we want to use the money from that capital fund. You all have the authority to utilize the capital fund money. Any capital fund expenditure. Yeah, okay. Thank you. So when we're using that fund, we get your permission. And the pre-qualification comes when something we think is going to go over $500,000. And our definition of a capital is 5,000? 5,000 and a useful life of more than a year. Yeah. Well, those are my favorites. Can we clap by saying hi? Hi. Hi. Can you pose? Hearing none, the motion carries. First of all. I move that the board pre-qualifies Farrington Construction Company, Incorporated in EF Wall, and associates as bidders for the WCUSD 2024 combined capital projects. Questions? Do they need to call the criteria? They do. No. What was your favorite position by saying hi? Hi. Hi. Just pose. No, good. Got another question. Whoo. Worth it? Post-vote question. Post-vote question, if you hurry up. No, it's good. Okay, so does that mean others cannot bid or? Correct. They weren't pre-qualified. Okay, so these are the only bidders on the project? Yeah, if they choose to bid. If they choose to bid. They still could decide not to bid. Okay. Which does mean that I will have to petition the AOE for a bid waiver because I didn't get three bids. Okay. Each time then. Any time we don't get three. So we're gonna put only pre-qualified funds too. You're gonna have to do that. Okay. Okay, thank you. Fund forward, fund balance projections. So this is a report? Yes. I'm sorry. I'm sorry. I looked away. I'm sorry. I'm sorry. I'm sorry. No, it wasn't for you. I'm sorry. So the FY24 fund balance projections for the capital funds, sharing with you the balance for the capital improvement fund this year was $4,180,517.  I'm sorry. I'm sorry. That'll be hard for all of us. I'm sorry. I got it. We are projecting that the ending capital improvement fund balance for FY24 is now $1,943,950, which also includes a $629,000 reserve that's set aside for Eastmont player because they came in with money into the merger. All of the other schools have utilized the money that they brought into the merger with. So we're still holding that aside. The project to capital improvement fund balance available for future projects at the end of this fiscal year after that reserve is $1.3 million. This is an improvement of $890,000, which is based on the $1.9 million in the projected fund balance for May of 23. So in May, I was projecting that it would end much lower. Primarily the parking lot project coming in under budget was the biggest driver, but the overall budgets at EMES boiler, the U32 boiler project, the Calus boiler project, the U32 H-back projects all coming in under budget. And we also received an unanticipated efficiency Vermont grant of $25,000, which offset the boiler project at Calus. We are also anticipating more from efficiency Vermont for the U32 and Eastmont Pellier projects, but haven't received those yet, so I didn't include that. So what is the EMES money being reserved for? EMES projects. We have a few capital projects that have been identified, including some of the money for the security system upgrades that you just set pre-qualification standards for. That's one of them. They've just recently did the playground, some door upgrades, a sound system, I think. But for right now, they don't have a ton of things on their list. So is it for any capital improvement project that the reserve funds are used for, or select ones? I'm thinking the boiler, the boiler when you follow? The boiler came out of that. They came in with more than the 600. No, I know, it was like 900. Yeah, after the boiler project, it's at the 600. So the boiler project came out of the reserve for Eastmont Pellier. Yeah, and everybody came in with a little bit, it's just used it, used it all. And I would say the state of Vermont has gone through all of our facilities and done a really thorough review and identified things that they would recommend being fixed, they've done that statewide. We, as in Chris and Bill and I, we're really pleased to see that almost all of the things on our capital plan were the things that they were identifying, I think, or two minor things that they found. We're added to our list and we'll take those reports and update the capital plan for this coming spring and offset that. Suzanne, why is the 25... No, go ahead, sorry, I hadn't seen it. Why is the 2526 budget column highlighted on this? The capital improvement plan. There's, is there something special about it? Can you repeat? Oh, it was like she was... 4043. Because that was the 2425 plan and I've shifted everything over and just haven't... So is that the... Like, that's how I'm gonna outline it. And that's the first year of the general OIC. That's our next year that we'll be doing our planning for but really what should be highlighted is 2425. Got it. Thank you. And I noticed that what's in the budget is slightly less than our long-term planned allocation to it just because of what's falling in within the calendar. Yeah, with the fund balance there, yeah. Yeah. Thank you, Suzanne. So I talked to Chris briefly if policy would be our next item and we were thinking about tabling the policy for tonight. Oh. I didn't think it would be. What? That's just like... You said we need one, right? The policy number. And then there's a little bit of policy. Yeah, there'll be a... Is that okay? Yes. I'll be good to touch. I've got more. It's not cool. I'm just denying. You know, there's... There's people that have been absent for the morning. And then... The Stanley doctrine played a significant role in our policy committee meeting. What? The Stanley doctrine of making sure we match our numbers with the key of the video. We haven't already said that we're no... Open your seat. You're not. I have one. I have four. Okay, no, this is five. No, go ahead. It's the one that I didn't carry over yet. I thought so. When I read the packet, I was like, I think D34 is wrong, but... So on the red line, it's corrected. Interesting enough. But on the next one down, where it's like the one where you've accepted the clean one, that number did not come through. That's the only one I have. We're not discussing it too much. Yeah, but we'll just do it then. Okay, but so as we might as well. It was removed. So the consent agenda in the minutes of November 15th, December 4th, and December 6th. I move we accept the approved minutes of November 15th, December 4th, and December 6th. Say it. Did you get that, Lisa? I was kidding. Yeah, kidding. Any discussion? Hearing none. All those who favor placing the side by saying aye. Aye. Aye. Aye. Any opposed? Any abstain? A motion to approve the board orders. The folder run around and has enough signatures. Thank you, everybody. And now we need a motion. Do you need a number or just a motion? We need a number. Does someone have the thing? Yeah, I don't have the motion. Okay, here you go. Do you have it there? No, I don't. I do. I move that we approve the board orders for November 16th, 2023 through December 20th, 2023, in the total amount of $883,584.73. Did you get the number, Lisa? Yeah. Okay. Second. Second. Is that both of them though? That was both of them, weren't in the general election. Okay, thank you. Any questions? Discussions? Seeing none, all those in favor of placing the side by saying aye. Aye. Aye. For reflection. Thank you everybody for being here. It's late. I have two points to thank Jan Arsino who's playing the role of, it's not because Megan and Pat have to be here, of the ghost administrator from the theater. We're keeping us safe. There's a ghost light that stays out in the theater overnight and then there's our ghost light administrator. Thank you for that. I feel safe that you're here. And I quote, comments for you folks that are prepared for Christmas and so take a draw on your way out, please. Oh, thank you. Thank you, Chris. Yes, I'm Mr. Jones. I want to thank him publicly for the service and the verb that he brought to the board and really appreciate that he served with us for as long as he did. Here, here. Mm-hmm, yeah. Hi.  Thank you, everybody. Yeah. I move to adjourn. I move to adjourn. Danielle, I move to adjourn. I move to adjourn. What was the favor of placing the side by saying aye. Okay, happy holidays, everybody. Good evening. Thank you. I hear family.