Good evening, everyone. So this is a talk about a solution that we've developed here with my students on how do you improve user experience when web servers are overloaded. So this is joint work with a number of my students, a couple of BTEC students and a MTech R&D student who graduated and my current set of MTP students, Stanley, Murli and Ramli. So coming to this problem of web server overload, so I'm sure all of you have experienced it at some point. It's a fairly standard thing. As Shivram has said in the introductory email, you've booked tickets on Tatkal. You've probably seen IRCTC crash on you a couple of times, right? So what is this problem? So we have a web server that has, say, a nominal capacity of five requests per second. That is, it can process five requests per second and we are specifically concerned with web servers that are serving dynamic content, you know, not static web pages that can be offloaded onto CDNs and things like that, but websites like, you know, a ticketing portal, IRCTC, a travel website, various things that when a user request comes in, the web server has to expend some resources, you know, do some contact the database, do some calculations and return a result due to which it has a finite capacity, a low capacity. Say in the Stoy example, it has a capacity of five requests per second. So five users per second come in, they are served. But more users than that come in, in any given second, initially the queue starts to build up. The server has a small queue. The queue starts to build up. The queue overflows and eventually the users coming in beyond the capacity will not get served, right? They will perceive what is called a website crash. And note that this is not just a problem about web server provisioning, right? You could say, well, you know, build your web servers so that they can handle whatever load you're going to get. So it is not just a problem of provisioning because frequently transient overloads happen even for the most well-provisioned servers. For example, the classic case of IRCTC, so the web server, the IT infrastructure is actually pretty well-designed to handle a few thousand requests per second. But at peak load, when this Tathkal booking opens, they get a few lakhs of requests per second for a short period of time. So it does not make economical sense to actually provision the server for this very occasional peak load. And therefore, websites have to be designed to handle these transient overloads, even if the web server is actually well-provisioned to handle the average load in the long term, right? So this is a problem that will not go away with just proper provisioning. And the end result of this is poor user experience, right? People see websites crash. So there have been some solutions, obviously web server design is an old problem and people have looked at various solutions to web server overload. So the classic solution is there are several techniques to improve server capabilities. For example, if you have a server with capacity five requests per second, add another server replica, load balance your traffic across these two and your web server capacity doubles, right? So there are several examples where you can do load balancing at layer seven, layer four, layer three. There are various ways in which you can actually improve server capacity by load balancing across replicas. And most modern web servers actually run over several replicas in data centers. So this is one way to avert web server overload. The other way is, of course, put in some kind of a proxy that does admission control, right? Whatever excess load comes up beyond what the server can handle, you put something that denies admission to users. You don't let this extra traffic even reach the web server. So these are about the two classes of solutions roughly that have been proposed for the problem of web server overload. You either beef up your server and beyond whatever limit you've set for the server, you turn down the rest of the traffic, right? So now what about these users that get turned down? What happens to their user experience? Well, if you are lucky enough and if the proxy is polite enough, it will accept your HTTP request and return a response, a 503 page service unavailable response to you, right? And sometimes if the guy doing the admission control is even too busy to do that, your TCP connection simply times out. And you see this connection timed out page, right? So either your HTTP connection fails or your TCP connection fails and this guy says try again, right? But at this point, there's really no methodical way. When should the user retry? Fine, the server is overloaded, you've hit capacity and there are all these users who are being turned down. There is no guarantee of eventual service. There's no methodical way in which the user is told please try now and you will get served, right? So people keep hitting refresh, refresh, refresh and if you're lucky you get through at some point, right? So that is how retries are today. So our idea is to build something that tries to improve the user experience for in periods of transient overload, right? For example, if a web server was using our web queue system, the user interface would look like this. If you reach the server at a point where it is overloaded, you will not get a 503 unavailable or a time out. In fact, you'll be put in a virtual queue of users. You'll be actually told there's so many people ahead of you. This is your wait time in the queue. You'll get a HTTP auto refresh page. That will take you to the website after the wait time has expired. And obviously when you reach the server, you will get guaranteed service. So what we're trying to do is simple. If you've been to a bank, you've received one of those tokens, right? It's just that. You are actually, there's this burst of traffic. We are just smoothing out this traffic to match server capacity so that transient overload is averted, right? And we do this by assigning wait times. We, instead of returning a service and available HTTP page, we return a web page that auto redirects after a certain wait time. So obviously all the magic is in how do you compute this wait time and make all this scalable and all. But this is the basic user interface that the user will see. Is that clear, any questions? Right, so this is a high level architecture. I will go into each of the pieces later. So how does all of this work? So between the user and the web server, both of which don't have to be aware of, yes, yes, I will cover that. Just give me a couple of slides, I will get there, right? So for a web Q enabled website, you would ideally not have to hit refresh. You will go to the website. If it is not overloaded, you will go through.  There's no incentive. There is a disincentive to hitting refresh. I will get to that, right? So how does the system work? It has two entities, what we call a token gen and a token check. So initially when the website is, a web server will choose to redirect some of its queries through our web Q proxy, the token gen proxy, right? So how does this happen? This is pretty standard. You use DNS, I mean, websites today host a lot of content regularly on content distribution networks, CDNs. So when you know that a certain URL will generate a resource incentive, resource intensive request, say the user has filled up a page with all the details of his train ticket that he want to book. When you hit submit, that URL will not go directly to the web server. But in fact, that URL will point to a token gen proxy, right? The user's request will be redirected through the web Q proxy. So this is about the only thing that the website administrator has to do. He has to decide, these URLs generate a lot of load. I want them to go through this proxy so that I don't get overloaded, right? So once the URLs come here, token gen will assign a wait time to the request. And if the website is not overloaded, you will get a zero wait time. You can immediately go to the server. If the website is overloaded, you will get whatever you are put in a queue.  And the user's browser waits for the desired duration of time. And then it tries to go to the web server, at which point another proxy that we call token check simply acts as a simple inline proxy. And intercepts the request, sends it to the web server, and hands the response back to the client. And these two proxies talk to each other. They convey some feedback to each other about the server performance, and do some basic capacity estimation. So for example, token gen needs to know at what rate I should send request to the server, right? So token check will give it basic feedback, like now the server looks overloaded. It's taking a lot of time to respond to requests, so slow down. So they do some adaptive calculations between them to actually shape traffic in this manner, right? So a couple more slides that will answer some of the missing pieces here. So high level, there are two proxies. The first proxy that the user goes to simply gives you a wait time, and the other proxy just lets you through to the server, right? So why split them into two? Because they do two different functions. Maybe after the next slide it will become a bit clear. But the primary reason is that is an inline proxy, right? And it is easy for that to get overloaded. On the other hand, this proxy is an out of band proxy. As soon as a request comes, it simply returns a wait time does not keep any state. So the reason for the separation is scalability, because each one needs to do, each one is doing a different job and needs to scale separately, right? So that will become a bit more clear in a little while. One more slide please, okay? It will become clear. So how is this wait time computed? It is, at a high level, the idea is straightforward. A wait time is assigned in seconds. We return a HTTP refresh page in seconds. So token gen knows the capacity of the server, right? So there is a capacity estimation module running that knows the capacity of the server. How that is done again, we will come to that a little bit later. So what token gen does is it knows this capacity of five requests per second. And it tries to schedule five requests per second to the server. And it remembers how many requests it has scheduled to the server into the future. So it maintains an array where the ith element says, I seconds into the future, I've already told five guys to go to the server. Three seconds into the future, four seconds into the future. It maintains this wait time array, right? So when a new user comes in, it finds the first slot in the array that is available. So one, two, three, four, five, the six seconds from now on, I've only sent two users so far. I could potentially send up to five users, given that's the capacity. So it fills up that entry and returns a wait time of six, whatever, you know? So this array, each entry corresponds to how many users have scheduled to the server at that ith second. So since we're dealing with wait times in the granularity of seconds, if you deal with any other granularity, obviously, the way this array is structured will change, right? So how is this capacity done? And we have a way to do it in a distributed fashion. Again, I will come to that a little bit later. So the main question here is, can users jump the queue, right? What if I get an HTTP refresh page that says, redirect after 20 seconds, what if I hit refresh again? So one way is, you know, you keep per user state, you note down the IP address of every user and these two proxies talk to each other and somehow prevent users from jumping the queue, which is highly resource intensive and it is hard to scale this, right? So we would like to do this without maintaining any per user state. So how do we do that? These proxies share a secret key, whenever a user shows up, token gen, along with when it computes the wait time, it also computes a token, which is basically a hashed MAC or, you know, a hash based on the secret key of these three things, the user's IP address, the wait time and the current time that the user showed up at.  I compute a hashed MAC over all of these things, so that is my token. And this token, I return to the user and beyond that, that's it. Token gen forgets about the user, it does not keep any per user state, right? So how is this, and this token is embedded into the URL that is returned to the user, you know, your HTTP refresh after 20 seconds and you have the token, you have the wait time. So after you wait your time and you go to token check, what does token check do? It extracts whatever wait time, time stamp that the user is reporting. And it recomputes the hashed MAC, right? It knows the secret key, it recomputes the hashed MAC and checks that the time that the user showed up at is actually equal to the time stamp and the wait time that he's reporting. So what happens if I try and manipulate the time stamp and the wait time, if I'm told to wait for 20 seconds, but I actually change my URL and only wait for five seconds and show up at token check, this hash will not match, right? And assuming that, you know, you cannot generate a hashed MAC without having the secret key, and assuming you do not have the secret key, there's no way user can show up at a slot other than his assigned slot. Right? Does that answer? So when you are told to go after 20 seconds, your browser automatically redirects after 20 seconds, you better only go after 20 seconds. Because you try to go earlier or you try to go later and reuse the token at a later point, it will not work, right? So there is some granularity of one second or whatever granularity at which you assign wait times. Your, this token checking will only work at that granularity. Yes? Yes. It is pretty lightweight. Yes, it is assumed that this is lightweight and HMAX are, you know, symmetric key cryptography is pretty lightweight anyways. And so we, from what we've measured, this adds very minimal CP overhead and latency to the calculations. Yes? Like what kind of does that? You could do that even today. So I mean, the classic response to all of these questions in security literature, you're not opening up any more vectors for attack beyond what exists today, right? You can flood a server even today. So this is not a solution against application layer DDoS attacks, right? So all this will do is fine. If you're getting a DDoS attack, you're getting a spike of traffic. It will only smooth it out. Sure, if you filled up the queue into the future, you filled up the queue, right? So this is, this token helps us actually ensure that users follow this wait time that we've prescribed and yet without maintaining any per user state for scalability, right? So what are sort of the pros and cons of this design? So first thing is this can be deployed as a third party service on demand. Note that we do not much like how a CDN is there. So why are CDNs so popular today? Because you can sort of amortize the cost of this content distribution across many websites, right? So similarly, something like this can be done. So this thing can be a third party service in a cloud or whatever and websites subscribe to this when they expect overload or only at the time of overload and this does not touch any aspects of the clients or the server. So it can be a third party service sitting in the middle completely on demand. And so this entity token gen is the only one that will face overload. Note that token check, once you've smoothed out the burst, scaling token check is easy because token check only needs to be able to handle as much load as the actual server. If your actual server is doing 200 requests per second, token check only needs to handle 200 requests per second. Because if all this wait time mechanism is working fine, it should never see more load than what the server is already seeing. So scaling token check is very easy. You can scale it much like you scale the server. And in fact, if website admins so desire, they can integrate token check functionality within the web server itself. Because in the end it is a small amount of computing a hash, which is very small computation. So this functionality can be pushed into the web server itself, right? On the other hand, token gen is the one that's going to see this huge spike of load, and for this we have a distributed scalable design so that this alone needs to be robust to overload. So this is one of the reason for the two proxy design, because the two proxies have different jobs. And scaling this is a lot easier than scaling a full fledged web server, right? So these are the other advantages is of course the clients are completely unmodified and the web server is also mostly unmodified, except for the fact that the server admin has to identify the resource intensive URLs and set up redirections. Beyond that, both endpoints are unmodified, yes? Yeah, token gen has a distributed architecture so it can scale to how much ever is needed to handle that overload, yes? Yeah, so that's my next point here. So one of the cons of this is, of course, all this will fly, assuming there is user acceptance. So a user would much rather prefer to be told that you are in a queue and wait for 20 seconds rather than hitting refresh impatiently. So that is a fundamental assumption. We have no user studies to back up this assumption, but we are assuming a known wait time with feedback is better than an undeterministic wait time even though the other thing might be shorter, if you get lucky. So if token check is seen, yes, it has to check the hash and it has to throw away the request. Those requests will not go to the server, but since the user has no incentive to do that, it is assumed that the load to token check will be at a low enough rate. But yes, if the users are doing that, then you have to worry about scaling token check as well, right? So the first assumption here is user acceptance that the user would much rather be placed in a queue and this is not that, this is for transient overload. It is not for persistent overload. If your server can only handle 1000 requests per second and you're getting 10,000 requests per second for an entire day and you're giving wait times of two years, nobody is going to use it, right? So this is assuming you have short bursts of overload, which bring servers down and which destroy user experience. And this is to only smooth out those transient overloads. And for that, we're assuming people are willing to get this user experience. And the other thing is of course this overhead of redirection is acceptable. That is if you're told you go to one proxy instead of directly going to the server that people are willing to do this redirection. Websites are willing to pay this extra cost for the benefit of having a better user experience, yes? No, that is not an assumption. I will come to that later. We are not assuming request types as I've sort of informally assumed so far in the discussion for ease of explanation, but that is not an assumption, yes, yes, exactly. So we will take request hardness into account, that's a good question, yes. Yes, or whatever C users of C is the capacity, yes. So we do have a server capacity estimation module. So if we find that the server is getting faster and we find that the capacity is increased, you can fill in the slots as well. So we are not taking this value of five as an input from the server. We are estimating it ourselves based on how fast the server is clearing requests. Yes, so when server capacity changes, so yes, that's a good point. If you filled up 5555 and suddenly you've realized the server capacity is ten. So some user who has come later might fill up this extra slack in some of the earlier slots. Yes, so when server capacity changes, you are not guaranteed first come first serve semantics, yes. But we assume that, so we have a server capacity estimation module that periodically runs and after that we assume there is a certain period for which server capacity stays constant and during this transient period that can happen, yes. So you've sent that redirect page, right? I've sent somebody a redirect page for 20 seconds into the future. Now I find that some vacancy in slot one has opened up. So I cannot go, I have no control over that guy. So, yes, that is also a possibility. You can underutilize the server if, yes, if first come first serve is so important you can always underutilize the server, yes. Yeah, problem, yes, again that is the user acceptance assumption, right? We are assuming people are fine with it and they follow the protocol. If you say come back after 20 minutes that and you're automatically redirected to the website after 20 minutes, you better be around to use the service otherwise your request doesn't get used. So one way is our server capacity will, if there is a constant rate of abandonment that will figure in the capacity calculations, right? So the capacity is calculated by observing the server response time. So if some x% of the users always abandon that will figure in our calculation and you will over provision like how airlines sort of overbook tickets by some percent assuming cancellations. So that will implicitly happen in our calculations, right? But we are not specifically accounting for that, right? No, why? Because you are in a first come first serve queue, right? So if you, if there are thousand tickets and you are the 200 guy to show up, it doesn't matter if you get service after two minutes or three minutes as long as your slot in the queue is reserved for you. So waiting for two minutes is assuming nobody else is rushing to the server and breaking the queue. Everybody follows the queue, you're still, you log in at eight o'clock, ten o'clock, whatever the time is and you should be fine, right? It's not fair only when capacity changes, but when capacity is fixed. So we are not assuming in those five minutes that Tadka opens up the server, admin is somehow reconfiguring the back end and changing capacity, right? So assuming capacity is fixed, the queue is fair, right? It doesn't matter, even if- It's basically a random, it's like a queue. That's it, it's like, it's like saying you get the line to get tokens that a bank is fine, if people rush and unruly and push you, that's life. If your network link happens to be a tad bit slower and you're behind in this queue. So the alternative is much worse, right?  And a situation like Tadka, on average, nobody is going to get in, right? That's not true, you've probably had very biased experience of Tadka. It's really not that bad, right? So it does work. No, no, no, it's not that bad. So, I mean, the overload is there, but it's not, assuming token gen doesn't come down with the overload, and you know, assuming some level of sort of equal distribution of network latencies, you should be in a fair enough spot in the queue. Yes? So which is why I said not all requests need to go here, right? Only the request that take up resources, the request that has to access the database, the request that has to access the payment gateway. Only that goes through a queue. And in the sequence of requests, in the various sequences, whenever, so you can treat each of them as a separate queue, or you can just put one queue in between, and once you get in, you get everything soon, right? So but the simplest case is each sequence, each step that consumes resources, you put in a queue, depending on the extent of overload you have. In a separate queue. In a separate queue, yes. Or you could choose to have, so this is orthogonal to this, where do you decide to put the queue? Assuming all users have similar profiles, and you know, take the same number of steps, you could have one queue in the beginning, or if there are also a lot more users' requests to just see the schedule and very few users' book, you could have separate queues for both the steps. But the number of resource intensive steps, so we don't intend to put static pages and forms that you have to fill here, right, because it's only the database access, those kind of things that have to go through this. Because the other things a website can presumably handle and it'll not crash handling all of that, right? So the rest of the talk, so I'll start with how we estimate capacity, how we have a distributed token gen architecture and a few results, right? So how do we estimate capacity, right? So this will cover some of all these things of the server as, yes. Why would it go down? It depends on the reason why it's going down, right? Token gen is overloaded, it goes down. In spite of the distributed architecture of 10 replicas, all 10 of them go down, well, the system goes down, right? So there's, obviously. So how do we estimate capacity? So here is some simple queuing theory or even basic system stuff, right? So here are two graphs, a good put and response time of a server. So first, consider a very simple web server that has only one type of request. And each request takes a fixed amount of resources at the server. So on the x-axis, I'm increasing the input load to the server from 20 requests per second, 40 requests per second, 60 requests, and so on. And the capacity of the server is somewhere around 100 requests per second, right? So as I increase the input load, what happens? The good put of my server, the number of requests that it's actually serving, increases as long as you are below capacity.  You give me 200 requests, I can only serve 100 requests, right? So as long as you are below capacity, good put linearly increases. Once you've hit capacity, good put flattens out, and in fact, it might even drop if users are retrying a lot. On the other hand, response time is fairly low as long as you are below capacity because there's no queuing. But once you start to get near capacity or response time, sharply increases because of queuing, right? So this is measurements from a simple web server that does a simple PHP script in the background. So this ratio of good put to response time, this is called the power ratio. And it is known to peak somewhere around capacity. Why? Because as you reach capacity, once you cross capacity, your response time increases sharply and response time is in the denominator, this is good put over response time. So this power ratio peaks somewhere around capacity, right? And so one way to find out the capacity of the server, so one way is of course to ask the server admin or to find out the service time of request, peak into the server. But if we do not want to peak into the server, we assume no server support. Then one way to do is keep sending requests at different input loads to the server and probe this power ratio at different points. And find out which point gives you close to the peak of the power ratio, right? So this is what we try to do in web queue. So if you want to probe at a certain input load, there are two ways. One is you give that input load explicitly as capacity to token gen. And make token gen schedule C request per second for some time into the future for some observation interval. And token check, because it intercepts every request to the server, it can monitor these requests and it can tell you, this is the average response time, this is the good put of the server. And it can give this feedback back to token gen, using which token gen can say, okay, this load, this was the power ratio, at this load, this was the power ratio. And try to map this curve out and find out capacity. The other way is you don't let token gen do any operation at all. It's a no op, it simply redirects whatever traffic comes in. And if you observe for a period of time, if you have distribution in the number of load levels you see in a certain period, then if you've covered good enough amount of low load, high load, if your server is fluctuating, you just observe for a period of time. You see what the good put response times are. And based on that, you can probe this power ratio at various levels. And by just observing what the server is doing. So this is sort of intuitive, right? If you see a server is getting back to you very fast, then TK, maybe you can try a higher value for capacity, or the server response time is low, you can use a lower value, right? So all of this is sort of basic queuing theory. But this breaks down if you have a real web server that has multiple types of requests. So this is the actual power ratio sample. So each point here is a 10 second interval where we've sent a certain number of requests to the server and observed the good put and response time from the server and plotted the power ratio curve. So here the backend is actually a model server that is serving two types of requests, requests to display the course webpage for courses that have two different types of contents. So here the two types of requests consume different amount of resources at the web server, so which will be the case in any real web server, right? There'll be different types of request which consume different amount of resources. And for that, if you look at the power ratio curve, it is not as neat as the one I showed you before, right? Why? So consider two points here. So this blue point here and the green point there, they both have the same amount of load in terms of requests per second. But the blue point presumably has more easy requests, which is why the response time in general from the server is lower, which is why in that epoch the average response time is lower and therefore your power ratio value is higher. In another epoch, which happened to have a lot more harder requests for the same input load, the power ratio is lower, right? So this is what would frequently happen with a real web server. And this clean idea of draw this power ratio curve, find the peak will not work. So what do you do in such cases? Excuse me. So suppose you know a priori that these two type of requests, one request is 0.25 units of work in some units, CPU time, whatever is the bottleneck resource, and the other request is one unit. If you somehow knew this relative hardness of the request, right? And in all my calculations of input load, good put response time, I no longer think in terms of request, but I think in terms of these abstract units. So whenever a type one request comes in, I add 0.5 to my input load. Whenever a type one request leaves the server, I add 0.25, whatever to the good put. So in all my calculations, if I treat request one as 0.25 request and request two as one request. If I do this scaling, if I knew this relative hardness of the request and I redraw the same points that were there in the previous graph, this is how they would look like, right? The minute you scale request by the correct relative hardness, the power ratio values once again align neatly on a curve and now once again it's easy to identify capacity, right? Somewhere around the peak of the power, yes? Yes, they can share the same bottleneck resource, right? So this is number of units in terms of the bottleneck resource, okay? Bottleneck resource is, so we are assuming, are you speaking about the relative ordering of the request within that epoch itself? So one request consumes five milliseconds at the database, another request consumes 10 milliseconds at the database. So the relative hardness is one is to two, that's it. And sure, there will be different numbers of them. We will take that into account for the total load.  And this request consumes that many units and another request consumes another. So I'm not sure why the independent, independence assumption is there. So they could, in fact, more very frequently they will be targeting the same. And in that model experiment, they were targeting the same CPU, whatever, CPU slash database, right? So here, the capacity estimation problem boils down to somehow estimating the relative hardness of the request. So here there are two types of requests, there could be four types of requests. Between all of them, if you can somehow accurately identify the relative hardness, then you can use this simple power ratio idea to get the capacity of the server. So a simple algorithm to do that will be collect some sample data, give different types of requests, different mix of requests, that is different ratio of the type one, type two, type three requests. So the total load level as well as the relative proportions of the different request types, collect a large number of samples, bombard the server with different load levels and different mix, collect for each epoch, collect the good put response time and offered load, and guess a value of relative hardness. This is one is to two is to three, whatever, pick one value, scale all your samples by that value and see if they fall neatly on a curve. So try and fit a curve and see if they fall neatly on the curve and calculate the regression error. So if you've guessed the wrong relative hardness, your regression error is going to be high, so go back, try again. So this is sort of just searching over the space of all relative hardness vectors. And you can use any, of course, you could do exhaustive search, but that would be infeasible. So you can use some technique like, so my student has used simulated annealing here, any such idea that you can use to search over this space of relative hardness values. And how do you know you've hit the optimal value? You will find your regression error kind of goes down. So we've plotted regression error versus the relative hardness values. And when you are close to the ideal value, that is when your power ratio values will somehow intuitively make sense, because you've waited the different request by the different amounts, right? So the minute you get around that area, you've found relative hardness. And once you've found relative hardness, it's easy to identify the capacity of the server in terms of subunits, right? So does this actually work? So we've tested it with Moodle back end with three or four types of requests, right? And we found that the estimated capacity is within 6% of the actual ground truth value, and the training period, we require about a few minutes of training period. So we found that an epoch size of 10 seconds is needed to get enough data to accurately calculate these average response times and all of that. And we needed a few tens of points to fit the curve reliably. So we required a training period of about a few minutes, and we could get pretty close to the actual capacity of the server. And note that all of this was done without utilizing any server metric. So a lot of the previous literature on web server capacity estimation assumes you know the service time, or you can measure the service time of the server, or you can instrument server code to calculate queue lengths, various things, right? So this is assuming zero support from the server. So this is a black box technique to estimate the capacity of the server, and we still get reasonably close to the server capacity, which is good enough for our purpose of shaping traffic, right? We can be a bit conservative also and take a lower bound on whatever we get, leave a little slack, and this works reasonably well in the context of WebQ, right? So this capacity that is estimated, so there is a small observation period, and after this the capacity that is done is given to token gen, so that it can use this capacity to assign wait times and shape traffic, right?  So as part of future work, we plan to remove some of the assumptions that exist around this training period, and we also assume that the server capacity does not change much during this training period itself, because that would completely mess things up. And we also, once the training period is done, you've identified capacity. We can also detect changes in capacity by the same idea, right? So once you've estimated capacity, keep monitoring your response times, good ports, and keep fitting points onto this curve. The minute you find that the points are no longer lying on the curve that you fitted, you know your capacity has changed. If your server, suppose somehow got faster, the back end added another replica or a replica failed, you find that for the same input load, your response times are getting longer, this technique can detect that and recalibrate itself, right? The recalibration takes a few minutes, but it can detect a change in capacity as well, yes? Yes. So after every 10 seconds, you know these are the types of requests, these are the numbers of each type of request. You don't know the relative hardness, but you know, we are assuming you can identify, you can look at a request and identify, oh, this is asking for the schedule versus this is making the booking. Therefore, you can identify the different URLs from the request, but you don't know how much resources they consume at the back end. But you know these many types of requests are there, and these are the counts, the frequencies of the various types of requests in each epoch. Yes, so the way we've tested it is we've generated random load to the server, right? Random input load, random mix, just random number of various types of requests bombard the server, and this is what we got. So they are two separate, so they don't get routed through token gen, the guy who's assigning the wait times, he just assigns the wait times and signs off. He's not involved in any more in the request anymore for concerns of scalability. On the other hand, token check is an inline proxy, it sits in front of the server, and ideally can be part of the server. And you need that to get all these metrics. So how do you do distributed capacity assignment, right? So this is the benchmarking for a single token gen. So we've built token gen on a single machine. So it is, obviously a token gen can handle much more requests than for the, on the same hardware, if you've deployed a server versus token gen, token gen can handle a lot more requests because it's doing much less server, right? The application server is presumably accessing a database, doing some computations, whatever, this guy is just looking up an area and returning a wait time, so it can scale much better, but still, at some point, this is going to hit a limit. So on the x-axis is the number of requests we are sending to token gen, and on the y-axis is its good put, as you can see, after about 8,000 requests, this machine, so this was running on a four core machine. It was running a variant of the Apache server which implements token gen, and it cannot keep up with the load of requests, right? So it, its good put starts to drop, it starts to drop requests. So a single token gen will not work. At some point, it will become the point of failure, even though the website, you're trying to protect the website, but your solution itself gets overloaded. So we do need a distributed design for token gen, right? So you could have, you should have multiple replicas, and the user requests can, you know, through, for example, in DNS, you could return multiple IP addresses or you could randomly return different URLs of different token gens to different users, whatever it is you could decide to split traffic amongst multiple token gen replicas, and these guys together should somehow shape to the capacity of the server. So if your total server capacity is say 15 requests per second, each of them should take some fraction of that pi and shape traffic according to that number so that together, each one is below their capacity, but at the same time together, they are able to handle a huge spike in capacity, and this can be dynamic as well. The more you find the load coming into the server, you could dynamically scale up and down. We want our design to be as easily horizontally scalable as possible, right? So the key question is how do you split capacity between the different token gens? One ways, of course, equals split of capacity, your total capacity C, and token gens give each guy C by N, but clearly that is not desirable because there's nothing guaranteeing an equal split of traffic to all the token gens, right? What if that token gen on the top gets very few small fraction of the traffic and the guy at the bottom gets a larger fraction of traffic, the guy at the bottom will end up assigning longer and longer wait times, right? Which is in some sense not fair. So we would like to have this property, yes geographically distributed in different area, so think of it like something deployed on the cloud, like a CDN, right? So it's hard to have a load balancer in front of it. That is a possibility, you have a load balancer and then you worry about scaling that load balancer, which is a different ball game altogether, but here we are assuming there is nothing in front of it and users are randomly assigned one of the N token gen replicas or according to geography, whatever it is, and we are not assuming an equal split of load. If we assume an equal split of load with a load balancer, the next few slides become kind of pointless, but to make the problem more interesting, we have not assumed an equal split of load, right? So now the goal is equalized wait times. The goal should be such that even if you're distributed no matter which replica you go to, you kind of get the same wait time as you would have gotten in an overall centralized system, right? So why this goal, of course, we could have picked several other goals, but this seemed like a reasonable initial goal. So how do we go about doing that? So all the token gens, they have each one has their own wait time array. They know I've scheduled this many requests into the future. It needn't all be five, it can be whatever each guy has its certain share of capacity and it has a array of wait times and all the token gens talk to each other, exchange this information with each other and each token gen can calculate what is the average wait time that it is assigning and a sum of all the average wait times that the peers are assigning and it can calculate its share of capacity as H over H by P. So if I am assigning much longer wait times on an average compared to my peers, then I'm entitled to a larger share of the capacity pipe, right? So this is the simple algorithm that we've implemented to calculate share of capacity. So this, we still have a little bit more work to do to understand if this is the best possible way to split it and we want to explore the design spaces. That the way to calculate share of capacity is there something better? Is there something that's somehow provably optimal? So all of this is work yet to be done, exploring the design space and also the scalability of this mechanism itself. Here we need every pair of token gens to communicate with each other to exchange this wait time arrays. So in this literature of distributed rate limiting, there is this idea of using a gossip-based communication. You know, X tells Y, Y tells Z instead of everybody talking to each other. So as part of future work, we also plan to explore a more gossip-based mechanism to reduce the communication overhead as well as perform a scalability analysis of this mechanism. So all of this is to be done, but for the present, we have a simple distributed design that empirically works and equalizes wait times and helps token gen scale to a much larger load than what a single token gen would be able to handle. So what I told you about was the share of capacity, right? What each token gen is entitled to. So there is another concept of usable capacity. So note that even if you're entitled to a certain share of capacity, sometimes it may not be possible for a token gen to use all of that share of capacity because it has to respect prior allocations. Note that the other token genes are doing some allocations into the future. So if a new guy comes up, he may not be able to claim his entire share completely. For example, assume at the start of time, there is only one token gen. A lot of traffic is only coming to one of the replicas. And that guy assumes nobody else is seeing any traffic. Great, let me just take the entire capacity and he schedules at that capacity for a long time into the future. He uses up a lot of the future slots. So now in between a new token gen comes up, it starts seeing traffic and it says its share of capacity suddenly increases, right? So for a transient period of time, it will not be able to use its entire share because it has to still respect the allocations that some of the other token genes have made into the future. So we have this notion of usable capacity which is calculated as your share minus whatever other commitments the other people have made into your share. And eventually at some point, the other, the first token gen will realize that there is some other guy that's up here and I'm using up a share and it will cut down a share. But for a transient period of time, you will not use your entire share of capacity, right? So I'll just illustrate it with a simple example. So this is a simple experiment that we, sorry that we did. So the green line here is the server capacity and the blue line there is the load that was being thrown in at the server. So the load that was being thrown in at token gen. So we had two token genes in this case and the blue line was the load and the red line was at what the server actually saw. So the two token genes together smoothed the load to match the capacity of the server. So this is from a actual implementation. Now let's, sorry, let's look at the individual token genes, right? So that is the first guy and this is the second guy. So the first guy started getting load from time p equal to zero. So initially he assumed the other token gen, he saw the other token gen wasn't getting any load. He assumed he had the full server capacity share to himself. He started scheduling requests at the share of the server capacity. But once the second token gen started, that's when that guy realized he needs to reduce his share of capacity and you can see an adjustment period and you can see even here that this guy learns that his share of capacity is a little bit higher, but it takes him a while to actually start sending requests at his share of capacity because he has to respect the earlier allocations that are made. But eventually in the long term these things are stabilized about, both the token genes are seeing roughly similar load so it stabilizes to about equal share of capacity and together they can shape capacity to the server. So if you have different amounts of load, so the total load of 8,000 requests per second is coming in at different values from the two different token genes. If each one, this guy, the y-axis in the two graphs are different. This is around 6,000, that's around 2,000. If they get different amounts of traffic, you can see that their actual share of capacity that they learn is also different and they learn different amounts and adjust dynamically so that the wait time in the end is somehow equalized. So I've told you about the two different parts, the capacity estimation and the distributed design. And now a little bit on the implementation. So we've implemented token gen as a fast CGA extension. It's basically based on the Apache web server. So an earlier implementation was able to handle only a single type of request and it was centralized. Now we have a distributed implementation that can take different types of requests into account. So once your capacity is estimated for different types of requests, token gen takes this. So in calculating wait time, assigning wait times, it takes the relative hardness into account. So the two pieces have been integrated. So when it is marking something in the wait time array, it'll actually mark the number of units of work it has scheduled in a certain slot and not just it has scheduled one request. It'll also mark the hardness of the request implicitly so that even if the mix of requests is changing, you get few harder requests, few easy requests. You take that into account while you're scheduling load to the web server, right? And token gen was implemented as a lite proxy and the web server was a model backend. And it turned out that we built our own custom load generator. So why I'll come to that in a little bit. So our load generator actually required about as much work as the rest of the system given that it had to do a lot of work. It has to actually overload a server, right? So we built an open loop load generator that given a certain number of requests per second, it generated load at that rate and every request was sent to token gen, wait time, the thread responsible for that request waited for that time and then went to the web server, got the response back and it was instrumented to spit out the good code response time so that we could monitor how well the entire system was doing, right? And each request was wrapped in a lightweight Java user level thread called fibers and these fibers are multiplexed on a fixed number of kernel threads. So that the total number of kernel threads that we use is fairly small, right? So this is some of the implementation details. So why did we have to build our own custom load generator? So this graph here, so initially we started out using some of the existing load generators like Apache JMeter. So what do existing load generators do? They are usually built to bombard a server with a large number of requests, but they are not really built to handle this long wait time, right? So when we are testing our system, we take a server with a small capacity and our metric of how well we are doing is actually how much the overload is compared to the capacity of the server. So the minute you have a huge overload, huge peak compared to the base capacity of the server and each request was using up a thread on the operating system, what happened with all existing load generator was pretty much very soon they hit a limit in terms of memory or the number of live threads, right? Because each thread was making a request, it was getting this huge wait time and it was consuming resources on the machine all during this wait time. So which is why when we tried to use, so this is the graph generated by Apache JMeter, we told it to send around 700 requests per second to the server and that red line over there, as you can see it could not sustain a burst of 700 requests per second to the server for a very long time because pretty soon it hit the limit on the number of threads and this was on the most powerful machine we could find in our lab at 24 core server, right? Yes, so the load generation itself is open. So when I say 100 requests per second, I am generating 100 new requests per second. Sure, each request there is a wait time and it sends the next part but I am not waiting to generate the next request after I finish this, right? So this was how one of the best load generators did in our case, it really couldn't sustain and we had lot of issues testing our system at high load simply because our load generator gave up and that is actually, it's not a criticism on existing load generators because this is not the use case they are built to test, right? Typical load generators are built to bombard a server with a large number of requests but no server implementation makes threads wait for typical server response times are small enough that it's only in our use case of wait times that this system fails. So on the other hand, this graph here shows we've asked the same workload, we've asked our load generator to generate 700 requests per second for a lower server capacity and it could sustain 700 requests per second fairly easily and I am not showing there are other results about the memory consumption was much, much lower and so on and so forth, right? Because in the end it is a fixed number of kernel threads and that leads to much lower memory consumption and easier load generation. So now putting all the things together, just a couple more slides, this is the graph on, this is the final graph where we've integrated the capacity estimation, the distributed token gen, everything, there are couple of token gens, load going to both token gens, variable load, variable amount of requests, everything and that is the blue line here is the actual load going to the token gens in terms of subunits of work and this is the server capacity in subunits and that is the final shaping we could achieve. So we are ensuring that the load to the server is below the capacity. And finally, what does all of this gives you? It gives you better response times, right? So the red line here is the response time of the server without web queue and the green line here is the response time with web queue. So obviously once your server is receiving load only at its capacity, the response times are lower, much more deterministic and leads to overall better user experience even after you get into the server, right? So this is fairly obvious from the fact that you're always below server capacity and so your response times are much better. So this is a summary, it's a simple mechanism, does not require modifications to clients or servers, there's a bunch of lightweight proxies and our design itself is scalable and robust to overload. And assuming users are fine with this feedback mechanism of asking to wait instead of hitting random retries, this solution makes fairly good sense. So general feedback, so this work is still work in progress. So an initial version of it that did not handle distributed, not have a distributed architecture does not handle different types of requests was built by a bunch of students. Now my current set of MTech students are still working on this as part of their MTP. So any feedback, any thoughts on the design itself and also general comments on the usefulness of the service itself would be highly appreciated and who could be the potential users, IRCTC or Flipkart when they have their mega, giga, sale, whatever, right? So all of these, any thoughts on this would be great, yes? A cube built up, yes. And that could bring the server down. Yes, yes. So if we, if we are especially on the side of underestimating capacity, yes, you're again getting into the problem of server overload, right? So yes, yes. Yeah, so that is something that we've considered. But the point with that is these proxies need to still have a hook on to the user request or the user needs to come and check in or they need to maintain some communication with the user to inform him or the queue is moving faster slower. So that's, it's always a tradeoff between how much state you are willing to keep that further increases load on the system, right? So it is a conceivable design that you assign shorter than needed wait times. So instead of this total wait time, W just assigned W by to ask him to come back a little bit later. Now that he's waited that much time, maybe you get a better estimate. Yes, that is a good possibility where instead of giving the total wait time at once, you ask the user to come and check back periodically and you will have a better idea of how the queue is moving, yes? Yes, yes, that's a good idea. So not keep any state but ask the users to periodically check back so that there's an improvement, yes. So it depends on the use case, right? It depends on what is the peak load that you want to handle. So at some point, even with 100 replicas at some point, you will hit, there is a physical limit, right? So the way you provision token gen, the number of replicas is based on what is the peak load that you want to handle. Say if your server capacity is 100 requests per second, I want to go up to a million requests per second without wait time compromising, without user experience compromising, beyond that, fine, no guarantees, right? So it depends on how much that peak is relative to your server capacity that you want to provision for. It would not be unavailable but it would be filled up with requests of this particular type for a very long time, yes. That is possible but in the earlier case, you would bring down the server, here at least the server is up and running and hopefully a few other people will get through. You could integrate, you know, intrusion detection system into token gen that weeds out certain requests but yes, that point is well taken and we have no solution for that. It does spread out the load so it spreads out an attack by definition, right? So here the request was view a course webpage or any action that you would want to do on model. So this involves a database access.  Yes, yes, yes, they're all real requests on a model. Last question probably we can wrap up and take it offline. And if the user is not waiting, you can deduct that. But this requires you to maintain an HTTP endpoint, right, that we are trying to avoid here, yes. So this is sort of, yes, that is one design option here, our explicit goal was to not maintain an HTTP endpoint for the purpose of scalability. So if that option is there, yes, that is a good way to check if the user is still waiting or not. So that's what I described at the beginning, right? So if you hit refresh before your wait time, so we have some small cryptography, a hashed Mac that won't check out. So there is no incentive for users to jump the queue. So that is one of the, to wait a little bit longer than the first request. No, we are not assuming one request. And I'm not sure, probably we can discuss it offline, but I don't yet fully get the question. So one last question, which is not a technical question, unlike the ones we've seen, right? So this is an example of academic research working on a real problem. Do you want to build this out of the product? I mean, how do you want to publish this? Yeah, so that is something that we've been thinking of as well, and I would solicit feedback on what to do. But yeah, it is possible, you know, you put it out as an open source code, you start up something based on it. If there's enough interest, get people to use it. You can monetize it, serve ads while people are waiting, and just give this free of cost to websites, and you know, just generate revenue based on ads. So there are several different directions. To be honest, we haven't worked all of it out, but we hope to in the future. Yes, sure. Yes. All right, on that note, let's thank you. Thank you all.


Hi, my name is Stu Miniman and I'm a senior analyst with the Wikibon project. I'm going to talk today about how enterprise architectures are invading the enterprise and how that's affecting converged infrastructure and really how all of IT purchases their infrastructure. So if we look at models in the enterprise today, enterprise today is they will spend their dollars to save time as opposed to hyperscale players. We're talking about the hyperscale players, talking about companies like Google, like Apple, like Facebook and they have a much different scale than the traditional enterprise players do. They spend their time to save money. So, you know, you talk about some of these guys and you say they might have a thousand PhDs to figure out how they can architect something. If you look at what Facebook is doing, they can actually manage over 20,000 servers with a single person managing that environment which is just orders of magnitude better than what the enterprise can do. And what we're seeing is really, despite the fact that hyperscale is only about 10% of the spend in IT, they're really driving a lot of the innovation. There's hot technologies like server-based flash and some of the sweet spots for software-defined networking or SDN are deployed in these environments. And hyperscale, when they buy their infrastructure, they really look at these as being disposable. What we're talking about is they really want to automate things as much as possible and reduce how much their people are touching these pieces. And we're starting to see some of the best practices and technologies from the enterprise pushing, from the hyperscale pushing into the enterprise. But one of the big differences for the enterprises, since they don't have all of those PhDs, what they really need to have in the enterprise is solutions. And what I mean by solutions is that somebody needs to take all of these components, put it together and offer it to them. It's not just buying white box servers or some kind of bare switches and saying, go build your own because there are only so many Googles out there that can do that kind of technology. So when Wikibon looks at this space, this architectural change, what we think is going to happen is the deployment of what we call software-led infrastructure. And software-led infrastructure is following some of the things that we've been watching for the last couple of decades, but some nuance and some specific things that we can now do with the technologies that are available. The first thing we want to look at in this environment is that we're talking about dynamic infrastructure. And by that we mean scale out. So I should be able to buy small and be able to add pieces incrementally. We've seen some point storage solutions out there in the past that have done scale out really well. If you look at Equalogix, if you look at Isilon, all of those really can buy piecemeal and they just scale nice and simple. Secondly, automation. If you hear what VMware talks about in the Software-Defined Data Center, automation is key and we would agree that automation needs to happen. Software-led solutions and in many cases this software can manage all the services and it's more than just managing the environment, but really functionality that might have been baked and hard-coded to the hardware before can now be led by the software and should be in many cases hardware agnostic. Also, simplification is absolutely critical. Need to get rid of what we like to say is the undifferentiated heavy lifting. So if you look at what Converge Infrastructure's been trying to drive and what the hyperscale people do, it's what we call rack level convergence, which is rather than thinking about my compute and my storage and my network as distinct silos of operation or cylinders of excellence as they were often called in the enterprise, we really just want to think about what processing, what storage and what bandwidth I need and it should be able to be deployed in a simple manner. And the last thing we talk about is there's a whole lot of new architectural designs that need to be considered, web-scale applications take a different architectural approach to how they're done and there are a lot of new technologies specifically want to call out. Flash is something that is drastically changing how architectures are built and cloud. So, you know, cloud is much more than just virtualized, it really is an operational model that needs to change and as Wikibon CTO David Fleuer says, when you build your own infrastructure sometimes it might want to be in-house, sometimes it might want to be in the public cloud or more and more you might even want to be thinking, can I have my information within a close proximity to access more cloud solutions? So, if you look at what Amazon has with its direct-net technology and big hosting providers like the Switch, Supernap in Las Vegas where we can actually have my data and cloud data close to each other and we can even use technologies like Backhaul to be able to plug high bandwidth connections between them because if we talk about going over distance, you know, data takes a long time to transfer so we need to really think about that. So, when we talk about converged infrastructure, converged infrastructure really has always been about simplification and if you look at the architectures that built for converged infrastructure, most of them are driving towards this rack-level convergence. So, of course, VCE's VBlock is the early leader in this space, Cisco with both VCE as well as with NetApp with the FlexPod, EMC with the VSpecs have been driving that rack-level convergence as well as IBM with their relaunch of pure systems about a year ago as well as HP who really has done a phenomenal job of converging the server and the storage so that many times you don't really think about them as server or storage but it's just that resource that we talked about that go together. And Dell's also driving converged infrastructure solutions. Beyond simplification, most of the converged infrastructure vendors from the traditional side are looking to drive automation and that's usually some layer of software that is put on top and it's a management way of looking at things. There's also ways to look at really kind of pooling and automating what I do with my infrastructure and once again, sometimes that requires an additional layer of software or some kind of clustering to be able to pull things together where the kind of traditional converged infrastructure solutions that have been on the market have been falling down is first of all, from a scale-out architecture, most of them don't fit that model real well. I need to think of some kind of matrix or clustered configuration and if we look at things like Flash, most of these architectures were not built specifically with Flash in mind and therefore they've had to, while they can put Flash into the environment, they're not necessarily taking advantage of server-based Flash in the best manner possible and the interaction with cloud is often hazy. Now I want to talk for a second about some of the new vendors, what the big guys would call the ankle-butters of the world but others have called hyper-convergent, architectures built with really that hyperscale mentality in mind and pulling that into the enterprise. So the first one I'll mention is scale computing. Scale computing has really built for the SMB market a solution that pulls that hyperscale architecture into the environment. It starts at a real small price point and really starts below where Flash today can hit from an architectural standpoint. So it's a nice solution to look at for people that want the efficiencies and operational models of hyperscale but not ready for not driving the applications that would be kind of in the middle of the market or the enterprise market. Second one is simplicity. Simplivity is actually based in Westboro, Massachusetts which is close to our office here in Marlboro. And simplicity, really what Wikibon believes they've done is build almost a God box. It takes a lot of the storage efficiency technologies and drives it into that kind of hyperscale model but since they're trying to put so many different features inside of it they have multiple fronts that they need to fight against. The third one that I'll mention is Nutanix. So Nutanix actually does the best in Wikibon's opinion in meeting the criteria that we've set forth here. It's built with really kind of the Google type architecture from a scale-out perspective so that they've learned from what Google file system and Colossus have done to be able to scale out their environment. They are a software-led architecture. So while they do offer the hardware pieces it's all commodity-based hardware and they offer Flash in a couple of different modes including server-based Flash and they built their design so that they can be hardware agnostic and keep adding on top of it. It is rack-level convergence so that it gets all of your server, storage and network in one piece and as I said they are built for Flash and for Cloud. They've actually came up with a term called Virtual Computing Platform or VCP where they've kind of matched out how they feel they meet these kind of new architectures and even talk about how Nutanix can coexist with cloud architectures really well so that I can have applications that can span between my internal and external what they call a hybrid environment. So the action that we really talk about at Wikibon for enterprise CIOs that want to look at converged infrastructure and how Hyperscale is going to invade that market is what you can get by moving towards converged infrastructure. Of course simplicity is one of the things that all of the vendors will tell you that they're driving towards and you need to look at not only what the initial acquisition cost is but what ongoing support is going to cost you. So you want a solution that is going to be able to help drive down your initial cost and secondly you want those operational efficiencies so productivity should go up when you're buying these solutions so look for environments that can really automate your environment and dynamically scale out so that we don't have to have our people spending so much time adding and tweaking and adjusting the environments that I have and finally the solution should also bring your workforce agility so we should be able to be doing more being able to spin up more applications and not have to think about okay when I add a new application what's the bespoke environment that that's going to go in and how long will it take me to tweak that environment you want to be more like what Facebook can do which is just okay I've got five different models to choose from I go choose the one and I go move forward on it so Wikibon really believes that we can look at the hyperscale solutions out there and have that be a predictor as to where the enterprise is going to come over the next couple of years and converged infrastructure and especially those hyperscale type solutions are going to help enterprises take advantage of the next generation of IT so this is Stu Miniman with Wikibon check out wikibon.org for all the latest research on hyperscale converged infrastructure, big data and much more and find all the videos with our media partner SiliconANGLE at youtube.com slash SiliconANGLE thank you for joining us


So today I would like to talk to you about something that we all know and love and that's the Theta experience. Just curiosity, curiosity, how many of you know about Theta and experience Theta where you're floating in the tank? Oh, good. What I'll also like to talk about is biofeedback as well in conjunction with the Theta experience and I'm talking about a specific type of biofeedback called alpha Theta training and it's used in clinical practice today and treating disorders like anxiety, PTSD, etc. Just a little bit about myself, I've been in the mental health field for about 25 years. I've worked in multiple clinical settings, hospitals, inpatients, residential settings, private practice. I'm also board certified in neuro-biofeedback and I currently consult and work with mentoring and supervising individuals who want to do this type of biofeedback. I know today some of our presenters talked about biofeedback and its use. I'm going to go in a little bit more depth. But before I start, I want to kind of go over about a start with brainwaves and talk a little bit about states of consciousness and how they associate with different brainwaves. So most of you know what brainwaves are, EEGs or electrosyphalography and basically it is the electronic recording of the electrical activity on the cortex of the brain. Believe it or not, your brain produces its own electricity. It operates on about 15 watts of power or about a half a peanut an hour. Brainwaves are categorized or are known for their waveforms or rhythms. They're also named after Greek letters, almost like a fraternity, beta, alpha, beta and delta. Rhythms are identified by their frequency or hertz per second. It's the speed they oscillate at and their amplitudes or the power of the wave and it's measured in microvolts. Each rhythm or wave is associated with various states of consciousness. So what you're looking at now is a raw EEG. This is what you would see with all the brainwaves combined. And if we were to filter that out, each brainwave would look a little bit different. So we look at the beta wave. It's also known as the waking rhythm. It's cycles per second is 12 to 30 hertz. Most of us right now are in beta just to hear, listening to me speak and being conscious and awake. It's the normal waking consciousness. It's consciously, you're consciously alert and focused on the outside world. Also there's a frequency between 18 to 30 hertz that's associated with busy or anxious thinking. Kind of that anxiety realm. A lot of us, when we start to get anxious or stressed out, we start to increase that beta frequency. I call it cortical irritation and I'm sure we've all experienced that. It's also beta is also the higher arousal state. Now there's also another frequency called gamma. I believe one of the researchers talked about it briefly. Now gamma is still up for debate in the field of neuroscience. Some researchers don't believe it exists. Some do. But if I remember correctly, the gamma wave was discovered in fighter pilots where they used to monitor their EEGs. What they noticed is that they were in these high stress situations in combat. They noticed the spiking wave. So they came to be called gamma. It's also found in Zen monks as well and deep meditation. Some people feel it's an artifact wave. In other words, it's muscle tension. It's not really a brain wave, but it's just worthy to note that. Alpha. The alpha wave is also known as Berger's waves because it was the first wave discovered. The German neuropsychologist Hans Berger discovered it. Actually he was also the one that invented the EEG. If you can see, it's cycle is 8 to 12 hertz. It has a very similar, almost like a sign-like appearance to it. It's also associated with relaxation and reflective states and idle states. It's also generated when you close your eyes. So let's do a quick demo. Everyone just close their eyes for a moment, okay? Within the first one or two seconds, your brain will start producing alpha. Actually, you may start feeling a calming effect. It starts to be generated in the parietal lobe and starts spreading over the cortex. It's also associated, you may experience alpha when you're watching TV. It puts us in that kind of passive attention. It's a very feel-good type of feeling. It's also associated with peak performance and flow states. Specifically, there was a lot of research done with athletes. And there's a researcher out of the University of Chicago, Somali, who studied peak performance. And there's been a lot of research done on quantitative EEGs and recording athletes' brain waves when they're performing. And they had found that when they performed their sport or when they're in the zone or the flow, you would see an abundance of alpha in their brain waves. So really what was going on is that your beta waves were being turned off. They're in that moment, just in that moment, performing in their sports. And their muscle memory takes over and they don't think about their sport because they practiced it over again. But you see a generated amount of alpha. It's also the bridge between your high arousal to your low arousal state. So what I mean by that is when we're in beta, we're highly aroused, we're taking all the stimuli from the world. And as we slowly lower our arousal states, our consciousness starts to relax. And then alpha is now the bridge between going into the theta state. There's also a mu wave between 18 to 13 hertz SMR. I put it up there because later I'm just going to explain a little bit more about SMR and its discovery and how it's used in discovering neurofeedback. SMR stands for sensory motor rhythm. And it's a part of the sensory motor stripped here in the brain. And when you generate it, you get actually a kind of a calming effect on your body. It's also, you can see it in cats, actually. Part of it was discovered in doing EEG with cats. Cats actually are very still when they hunt. So they produce more SMR to keep their body still.  But delta is known as the sleep rhythm. It's between 0.5 to 4 hertz. It's the slowest and highest amplitude of all the waves. It's increased during sleep. And it's basically when we're unconscious. So let's go to theta. Theta is where the magic happens. Theta is a powerful rhythmic wave. It's between 4 to 7 hertz. There's two types of theta, hippocampal and cortical theta. Hippocampal is a subcortical structure of the brain, the hippocampus that is responsible for memory. We'll learn a little bit more about that. It's also found in children in all wakeful states. So when we're in our childhood and we're growing up, most children are in a theta state. In fact, as we mature and grow into young adulthood, our brain waves proportionally change. And we actually begin to wake up and we start to make more beta. So when children, kind of an explanation of why they're kind of daydreaming, they don't listen, it's in the theta state. Also, a lot of our memories, our childhood memories are formed in a theta state. I'll report and we'll talk a little bit more about that in a second. It's also associated with drowsiness. It's generated every day when we drift into sleep and when we wake from sleep and into consciousness. So we all crash through theta every day. When we're sleeping, most of us are maybe aware of it, not aware of it. We go in and then we wake up, we come back out again. It's also known as the deep state. It's also seen in deeply internalized meditative states, specifically Zen monks. They've done quantitative EEGs to show this. More importantly, it's also associated with the hypnagogic state. How many of you are familiar with the hypnagogia? Okay, so this is where we're going to get into the meat of things. The hypnagogic state generates an unexpected and unpredictable mental images. So what is hypnagogic consciousness? Hypnagogia is the unique state of consciousness between wakefulness and sleep, it's on the edge of sleep, right before we zonk out. It's basically a theta-dominant brainwave state. We all experience it, but there is often fear that it's a pathological or abnormal. I've got a personal story I'd like to share with you. Back when I was a teenager, I suffered from a disorder called sleep paralyzation, anybody familiar with that? Yeah, it's actually how I started getting and discovering the hypnagogic in the theta state, it's kind of what triggered me into really following this. And getting into psychology and wanting to learn more about it. But what would happen is when you're asleep, your body will paralyze itself in order to do that. So when you're dreaming, you're not running. But in this disorder, when you're coming out of sleep, your body doesn't wake up. But your mind starts to wake up and you can go into a theta state coming out of Delta. And when you're there, you start having hypnagogic images. And I can remember I was one night waking up for the first time. And I was paralyzed and I couldn't wake up and I started to panic. And in the process, all of a sudden, I felt this presence in the room and these faces behind doors. And I was thinking, my God, is this it? Am I going to be abducted and brought up to the mothership? And fortunately, it wasn't. But it scared me enough where I went to a psychologist. And I thought I was actually having a psychotic episode or a psychotic break. But fortunately, they were wise enough to be able to diagnosis as a sleep disorder. And neurologic explained to me what was going on. But any of us who do experience it for the first time, it can be a little frightening. Hypnagogic images. This is something that when we float that we all probably encounter or this is kind of the signature trademark of hypnagogic. Are the spontaneous vivid images that you'll see. The fleeting, they're fleeting sometimes flashing lights. Color, that can be archetypal in nature. Faces, shadows, shapes, sounds, and they're almost hallucinatory. Anybody experience any of this? Good. The hypnagogic consciousness in history of mankind has been used in shamanism and spiritual traditions. The Theta dominant brainwave state has been utilized for creative, intuitive healing, shamanism and spiritual purposes for millennia. So from the shaman who used to go into the spirit world, from drumming, rhythmic drumming, putting you into a theta state or even watching the flickering of fire lights would put people into these altered states of consciousness. And in doing that, they would feel that they would contact the spirit world. It's also associated with unconscious material, specifically childhood memories. And that's one of the aspects of the theta state while floating is a lot of us will have that kind of memory recall and start remembering childhood memories. And remember when I was talking about earlier, when you were growing up in your childhood, a lot of your memories were formed in theta. Here's one that I think we can all relate to is the hypnagogic consciousness was also, brings on a very creative state. A lot of scientific discoveries and creativities, inventions that were discovered. William Blake, Albert Einstein, Thomas Edison. In fact, Thomas Edison is a story that he would sit in a chair.  And right before he would fall asleep, his hands would open up and the ball bearings would fall on the floor. And he would be startled back into the theta state. And while he was there, he claims that's where all of his information and creativity came for his inventions. Also Albert Einstein used to do thought experiments. One of his things, he would enter an ultra state of consciousness and he would take an experiment that he was working on and he would just kind of go with that. He also claimed that when he was in this state, he was opened up to a state of consciousness that provided him the information and the knowledge that he needed to create the formulas today. The photoelectric fact and the theory of relativity. Also, the benzene ring was discovered that way. I don't know if anyone knows organic chemistry, but the benzene ring was discovered by the scientists. He was sipped by the fire and just watched the fire and the flickering of the fire light would put him into a theta state. And he had a dream of a snake coiling around and grabbing its tail. Thus giving him the image that he needed to create the benzene ring. Also, I didn't want to mention intuitive purposes. Louis Ryan was a prominent investigator of spontaneous psychic functioning, confirmed early observations that intuitive functioning is facilitated by drowsy states. Furthermore, modern research into ESP conducted under controlled laboratory conditions has shown that the GANS field technique which is partial sensory deprivation is effective in facilitating ESP. I don't know if anyone has that experience in the tank as well, but also out of body of experience and feelings of peace, joy and connectedness. Now, I just realized I don't have much time left. I was afraid this is going to happen. So I'm going to talk a little bit faster and try to speed through this. Just wanted to talk real briefly about the history of theta pioneers and the research because this kind of gets into the meat of the research that was done using theta states and training people into alpha theta states to for clinical purposes. So Thomas Banzinski developed the concept of twilight training. He was a researcher who was doing research in muscle tension and he found that people would enter into these theta states. He basically at that point developed what is called twilight training. It was a device that monitored individual brainwaves and while they produced alpha theta, it triggered a playing of relaxation tape and that tape would be scripted, suggesting in the voice of the individual. Also known as hypnagogic visualization. Just want to use a quick example of how powerful this is, because this is also the meat of alpha theta training. In learning how to restructure or re-script our childhood beliefs and changing who we are. I was also a professional runner back in the 80s. And as I was running on the race circuit I had to learn to run with the Kenyans. So actually early on I started using theta as a way to relax but I also used it for visualization and would play a tape myself. Not knowing any of this research at that time of watching, rehearsing my races over and over again. And after about six months of doing this, I ran in one of the races. Let's see if it plays. Abraham Hussein and Joseph Kipsang's third and fourth place finish comes of little surprise. But the sight of Brian Neerstad chasing Joseph Zau across the finish line delights thousands with local pride. So it made my day. But it also showed me the power of the scripting as well. Jay Gonkelman, very quickly. Jay was one of the first researchers back in 1973, developed a twilight training facility in North Dakota. He used the feedback technique involving placing patients into a flotation tank for 90 minutes. And he produced patient brainwaves or monitored with pink noise audio feedback and provided for alpha and dominated EEG outputs through an underwater speaker. When patients became theta dominant, the pink noise was muted in a tape loop with a short prerecorded message in the patient's voice, rescripting early childhood beliefs. Jay's actually a friend of mine. I was gonna have him speak a little bit today. We don't have the time, but next week he would love to come. He's one of the leading authorities in EEG and quantitative EEG research and neurofeedback. And actually has experience running doing alpha theta training in a flotation tank. Yeah, very cool. Elmar and Alice Green real quickly, they also worked on using the twilight training. They found as well, they kind of not discovered, but also worked with hypnagogic imagery discovering that. And some of the students seem to have paranormal experiences. And they discovered a new type of psychological well-being. This is where the rubber meets the road. Eugene Peniston, while on a theta state in a training seminar, he came up the concept of, and you see the idea of using alpha theta training for chronic alcoholics in the VA Medical Center. Now these vets also were suffering from PTSD. And in his study, and designed back in 1989 in which they used three carefully matched groups, an experimental treatment group, which received the brainwave training, an alcoholic group, which received conventional treatment, and a non-alcoholic control group which received no treatment. Results showed much higher abstinent rates in the experimental group compared to the alcoholic controls and significant normalizations of various personality measures in the experimental group. In other words, what he did pre and post is he used an MMPI, which is a psychological tool that evaluates personality structure and psychopathology. And what he had found is that it brought it back to normality. He did further studies using the same protocols in which he took beta-dorfen levels, and it found that those increased as well, which is kind of the painkiller, but also gave an effort feeling. And a lot of this treatment today is also being used in addiction study. Part of the theory is that the effects of alpha and theta would increase this and give the person another sense of a high. Versus seeking drugs that they found that in doing and practicing this, this kind of overrode the original addiction. And the key factors in preventing relapse was changed in the personalized. So I am over time. I guess I got two quick minutes. I was told I can get extra time in. Nor feedback is trains your brain. Think of it as aerobic training for the brain. It's based on operating conditioning. In other words, there's a reward and threshold reinforcements that are given through a computer system. In alpha-theta training, what you're doing is you're hearing alpha tones. When you begin training, the clinician actually monitors your EEG, and then you can actually train your brain and coax it into alpha, and then down into theta. Once you're in theta, you can teach an individual to keep that theta state in a wakeful state. Yeah, it helps stabilize the mind, increases functioning mood. Space-H technology was discovered back in the 60s, and I can't talk about it now, but it actually, a lot of this technology was developed in NASA to working with astronauts. Alpha-theta training, I just kind of gave you an overview. But really how it's worked is when I do alpha-theta training, I have someone in a room, they're sitting in a chair, they have headphones on, they're listening to the music, as well as tones. Electrodes are placed in the back of their scalp in the occipital area. And then they're hearing the alpha-theta tones. And then as they go into theta, you develop a scripted tape of the participant's voice, and then usually they're hearing something they want to change about themselves. That's what Peniston used when he worked with the vets, is instead of drinking beer, he actually taught them thinking about reaching for orange juice or taking a non-alcoholic drink. And in doing that, that's changed their behavior. Okay, so at this point, I just feel that the natural thing to do is bring alpha-theta training into the tank. This training would not only augment the theta state, but will enhance the flow practice and the post-float experience, which has the capacity to change one's own life. And that was Jay, I was going to talk a little bit more about him. So, an ending, let me read this real quick. Floating along with alpha-theta training will augment the experience of the theta state, and with that, the hypnagogic images and the awareness of various states of consciousness. It is in this awareness that gives rise and insight into the realities that we have access to. Hypnagogia suggests the possibility of evolving and expanding our consciousness. It poses a serious question concerning the nature of reality. The power and magic of the hypnagogic realms brings us closer to our own intuition, creativity, mystical insight, and unconscious mind. We can engage in the self-hypnosis or hypnagogic visualization and re-scripting our own behavior so we can achieve things thought impossible or too difficult. We are explorers in traveling and together we travel into this new world. We are in fact following the footsteps of some of the most influential and intuitive human minds in history, and we'll continue this historical path longer to the next millennium. After all, Einstein couldn't be wrong. Thank you.


So here's a headline from CNN, right? The headline from CNN. This is funny. Biden says border walls don't work as administration bypasses laws to build more barriers in South Texas. So it turns out that, you know, that walls don't work. And therefore what we need in the Southern border are not walls, but we need barriers, because barriers work. The Biden administration, which has been anti-Trump wall, has said that they, you know, that they would not, that the walls don't work, that they will not build one foot of walls, they told us, not one foot of walls. That, you know, the walls were brutal, destructive, whatever. The Biden administration announced this week that they are indeed expanding Trump's wall. They are going to build more sections of the Trump wall.  So, you know, as long as you don't call it a wall, then everything's good. We're all good as long as we don't call it whatever it is. Evasion, evasion, evasion on steroids, that's what these guys are particularly, particularly good at. So there you go. Biden is building a wall. Now, this all comes as the Biden administration is under enormous pressure, enormous pressure from Democrats, from leftists in places like New York and Chicago that where the cities cannot cope with the influx of illegal immigrants, that is people who cross over the border are granted a granted to stay in the United States while their asylum case is being reviewed, which could take years, not allowed to work, not allowed to apply for work visa. And then by the grace and wonders of the government of Texas primarily, put on buses and shipped to New York and Texas, dropped off in the middle of cities and it's cold, by the way, and then left because they can't work left to become basically welfare recipients of the cities that are receiving them. This is all a ploy to get Democrats upset about illegal immigration and surprise surprise, it's working amazingly well. It's not like Democrats care that much about illegal immigration one way or the other. It's a problem far away. It doesn't affect their life until now and suddenly it affects their life and suddenly they discover the Democrats do in Chicago and in New York that they don't like, they don't like illegal immigrants, not in Chicago. And indeed, moreover, they don't have the money to fund these illegal immigrants, their welfare. So what you're getting is particularly in New York, we've heard of Mayor Adams, but now in Chicago and in Chicago, the problem is deeper because if you remember in New York, like Adams is not exactly a progressive Democrat, he's quite a centrist, a realist, I mean, he's pro-police, he's pro, you know, being tough on crime. Adams is more of a centrist, he's not one of the, but Chicago, God, Chicago is dominated and overrun by crazy, progressive Democrats on the far left.  Of course, these are also union people that don't want them to work or people to support the unions, they're not probably union members themselves until suddenly it's in their backyard and suddenly the mayor of Chicago, who is one of these teacher union, super progressive Democrats, he doesn't have the money to house and they're actually building tents with heaters for these immigrants. I'll tell you in a minute where they're building them. And suddenly he can't afford to build the tents and give them food and do all the stuff. He's calling for Washington DC to allow them to work. Wow. I mean, work permits.  And he's talking about traveling to the border. I'm not exactly sure what he'll do when he travels to the border. I don't know if you've heard about this, but Mayor Adams is going to, I can't remember, two cities, two countries in South America, Ecuador, maybe Mexico, and he's going to tell people in those countries not to come.  And basically they're just as well off in Ecuador as they are in New York, which is not true, by the way, because if you read about the crime statistics right now in Ecuador, some of the worst in the world, so of course they want to leave and get out of there. Other problem Chicago's having is they're trying to find places they can build these 10 cities for the illegal immigrants. So the only places they can build big 10 cities, flat areas with open space, is in the South of Chicago. That is in the Hispanic and Black and poor areas of Chicago, away from rich Chicagoans. And of course, the people who elected this current mayor of Chicago are those Black and Hispanic neighborhoods.  How come you're shuffling this problem onto us? Chicago's in a, I mean, Chicago's in deep, deep trouble anyway. The conflicts within Chicago are horrific. The violence in Chicago much worse than in New York, but this is tearing Chicago apart, supposedly. This is making things much, much worse. And it hasn't stopped and it's only going to increase. And this little bit of wall that the Biden administration is talking about building is going to make zero difference. The solution to illegal immigration has always been and always will be to legalize it, basically to allow for easy legal migration. And indeed those countries where the Biden administration is allowing relatively easy asylum application in the country have seen a significant decrease in people trying to cross over the border. But the harder you make it for people to become legal, the harder you make it for people to come into this country legally, the more illegals you will have, the more illegals you will have. Anyway, the Democratic Party is basically apoplectic. And here's the thing, Chicago's going to host the Democratic National Convention next year. You could see massive demonstrations and real challenges in Chicago. Rome was not destroyed by legal immigration.  Rome was destroyed by Christians who lived inside of Rome. And most immigrants were legal, by the way, in Rome. The barbarians that ultimately sacked Rome were legal immigrants. They weren't even illegal immigrants. And they were Christian, interestingly enough. So Chicago, the National Convention is going to be there. And if they, you know, Democrats have one issue going for them in elections so far. And that is abortion. Americans are generally pro-warms right to have an abortion. And they're beating Republicans all over the country on the abortion issue. But the one issue that is really in the hood, Democrats, that Republicans have huge leg up on them, that Trump knows how to capitalize on and has in the past, is the issue of immigration.  Whether it's Trump or Biden, it doesn't matter. Or Bush and Obama, it doesn't matter.  You're going to get Democratic constituencies voting on immigration as a major issue against Democrats. And it's going to make it very, very, very difficult. Very difficult for, you know, for Democrats. They've got abortion going for them. They've got immigration going against them.


It was a decent day to take a stroll in the park. But if you were at the Bemidji City Park, you probably heard the buzzing of chainsaws. Members of the Bemidji City Parks Department and the Conservation Corps were busy removing noxious weeds that choke off native plants and trees. Since Monday, the crew has been removing buckthorn and honeysuckle plants from Lake Boulevard, North Country Park, and City Park. Now by removing these plants, the hope is that more native plants will spread and grow throughout the forest. Just to help the native plant communities thrive, it's something that you really got to keep after. We got to keep the seed from dropping and creating a new generation of buckthorn and honeysuckle. So getting after it now is in the hopes of keeping it manageable. The Noxious Weed Removal Project is set to be an ongoing project for years to come. If you've enjoyed this segment of Lakeland News, please consider making a tax-deductible contribution to Lakeland Public Television.


So in 2016, I learned a money skill of copywriting and I would put copywriting in both the marketing and sales category. So I first started using copywriting to help me grow my e-commerce brand and sell more of my products online. After I sold those e-commerce stores, I got paid to use copywriting to help other businesses grow. Today, I use copywriting to sell my own products, to sell clients' products, and I also teach people how to use copywriting to grow their business as well. This one money skill of copywriting has helped me generate over eight figures in online sales and it's a skill I'm gonna use every day for the rest of my entire life. Now over the years, I've stacked on more of these money skills on top of copywriting, but now you can see just how powerful even learning one of these money skills can be.