is a case study of Mali and the general question was what was is the impact on me when I when I went there The for the last time to do some interviewing and Visiting around the country for the for the project was about a little over a little less than a year ago was in July of 2011 and Mali was a stable quote-unquote democracy and Everybody I met said it's a stable democracy. So the first draft of this paper was It's a stable democracy. What has been the contribution of aid? and I took too long to finish the paper and Busy with other things and finally I sat down to work on it in February And I was more or less finished ready to send it to Danielle and then there was a put it down In Mali and suddenly I Had to change everything. So so now I've sort of it It is quite handy. I think for a case study Our book of case studies because this is the case where you know things went wrong We didn't have a case like that. That's very hard to say You know establishing that kind of counterfactual is very very hard to say but in retrospect, I was really struck by The sort of complacency of donors And there's sort of assumption that things were stable and for Reasons linked to my other research. I was very aware of the to our egg issues in the north of Mali And so the risk of instability Struck me as very real. I met some to our eggs and in the case of some other research And so so I was very surprised that the donors Basically didn't think the two eggs are an issue You know, they thought it was a minor little annoyance in the north and And certainly I I mean The dough I so I do think the donors could have done a lot more to address both the two to our grievances and generally imbalances in the country that that had I think Very deep But you know at the same time you don't want to be too critical of aid and I don't in general I'm not a person who thinks aid Can play more than a minor supportive role in the development process So since I believe that when things go wrong, you know, I don't when things go, right I tend not to give credit to the donors or when things go wrong. I'm not going to blame the donors but so there there are some Country like Molly has certain very specific structural problems Regional imbalances very high level of inequality Very very weak institutions deep deep Urban rule differences And what striking is is just how these haven't changed, you know If you look at donor reports from the 60s, they say they say they give this litany of issues You look at donor reports today. They continue to give this litany of issues. So I think that aid is very good at Providing services MDG campaign, but I think one of the real I think disquieting implications of This case study Is is that aid is not good at addressing the strategic issues, which may undermine I think that's and that you know, maybe it's too ambitious to expect aid to be able to resolve these things but They in some cases it's clear that they can't really play a role One of the issues in Molly is very very unusual for a democracy is the language of government Language that's in Parliament Is a language that only 10% of the people speak? I mean the language of government the language of the state is French but 85% maybe 90% of Molly install Don't really speak French so so that to me symbolizes this real Creevage between elite and masses So could aid have lessened this major cleavage Well in theory, yes, but then the mdg campaign has in fact spent an enormous amount of attention and resources on improving literacy Which would be a way to have get people to learn French but nonetheless if you look at that process of literacy and the conversion to French It's really slow here. We are 60 years after independence and it's still true, you know, so that's a case where Can you? should the donors Look at themselves and take blame for this. It's tough to say but another case may which strikes me as as clear every Report on the to our issue has said one of the problems is just how isolated and Marginalized the to our region is we need to build a road from Bamako to Kedel the road actually goes about halfway This has been true since 1960 since the first to our rebellion, you know in 1962 The two are one of the key to our grievances is build the road to Kedel It's still not built It's still not built I'm sure there are good reasons not to build it, you know, it's Kedel is a tiny little town. It's the deserts It's probably You know, I'm sure the World Bank economists have said that there are five roads that make more sense from an economic point of view But in terms of national unity in the political stability in the country, it's been a key investment And it's not been made Despite Six billion dollars of foreign aid in the last 20 years like the some debates now are, you know, how important is democracy or Should the donors impose conditionality about elections and those strike me as as secondary issues Mollings want democracy and they showed it again this time and and I don't think democracy Makes a huge difference, but I think it makes a positive difference. That's another conclusion of our project. So those debates that are still very present in the donor community strike me as second-order Issues on the other hand addressing these strategic underlining Dynamics that really threaten national unity in these countries That strikes me as as more urgent and and and the donors need to talk more about these issues Because ultimately if Molly's going to be an imperfect democracy for a long time There are going to be issues of corruption It's kind of inevitable but The current civil war could set Development back in my 10 years One thing that's really striking is is how centralized it is in in the capital and How how much the imprint of aid is Is urban bias that it's and capital bias and I and I do think That again a part that's inevitable, but I do think the donors need to look at themselves and And really push to Decentralize more to get out to the countryside to get out to the provinces The the parastatal Designed and funded by the donors to help develop northern Molly To address a Tory grievance that parastatal was in Bermacol a thousand miles from the north Because presumably because the people who work there wanted to live in Bermacol The but the failure of that parastatal to make any kind of difference Was a source of great unhappiness on the part of the party. I do think that in fragile countries They're probably this failure to address strategic issues and structural constraints, I think is Is probably something that a lot of good, you know, it's true in a lot of countries One one other I think Finding that I think is generalizable has to do with democracy assistance So not development assistance, but specifically aid that tries to improve democracy The the the numbers from Molly living in the numbers suggests that the donors only work on one dimension of the democracy problem and that is the enhancement of vertical accountability so the accountability of the government to voters to people right, and so so there are a lot of assistance goes to a Elections, but strikingly a lot of assistance goes to human rights to gender imbalances to civic and political rights voter registration, you know virtually no aid goes to mechanisms of horizontal accountability so and by that political scientists mean the judiciary and the legislative and Their relationship with the executive branch So there's very little money that goes into party support. There's very little money that goes into legislative support and Very very little money that goes into strengthening the independence of the judiciary But the last is there's a bit more money, but strikingly there's nothing very very small amounts of money Going to Parliament so in Molly out of you know, 800 million dollars a year going to Molly One half a million went to the legislative branch And and I think mechanisms of horizontal accountability are key to the survival of democracy and to the quality of democracy just as much as elections and and horizontal of her vertical and so and my senses and Danielle agrees and this comes out in the other projects is I think this is generalizable It has to do with we don't like political parties. We think in legislatures there, you know, as we say in America It's like a sausage factory. You don't want to look at how the sausage is made so So I think there's just a natural distaste for for parties Poor barrel politics in the legislature and so the donors don't they just are not present there and I think that's a mistake in general, I mean, I'm not even broader finding I think is is that The aid to democracy should be increased In the case of Molly as far as I can tell it's One or two percent of total hate flows the data is not good But I think it should be five six seven fundamentally the problem of democracy in low-income countries Is that the executive branch is too powerful? vis-a-vis the other branch of the government and vis-a-vis the people and Both both horizontal and vertical accountability are very important you do leave you have a real cleavage between elite and masses you you you don't want the elites to all be it, you know together in a kind of elite cartel and not have Not be accountable to the masses through elections and vertical accountability, right? but that at an institutional level the ability to check the power of the executive branch to make sure that There's not corruption to make sure that Decisions are thought through and act upon to give a voice to opposite viewpoints You really need the legislature So in this case of Molly The the the read one of the main reasons for the coup was just the lack of a response by the government Two events in the north and mollions were very unhappy You Took the pulse of the street in Bamako Friends were telling me six weeks before the coup the government has to do something This is deeply unpopular that that they're losing all these battles in the north and I think if If that unhappiness have been voiced in in parliament It would have led the government to act more aggressively and that might have then staked off the coup Hard to know but that's what I mean by why you want the system of checks and balances You know within the national political institutions of the country I feel very very strongly that Part of this complacency about strategic issues has to do with this myth of Of non-interference and an apolitical the apolitical nature of politics every donor decision ever in a country like Molly has political implications and Maybe I mean maybe that's the most to me the most obvious implication is donors need to be aware of the political implications of what they do and as we move to democracies So as we move to systems in which there are elections in which there is civil society in which there are oppositions Then It is even less if you in an authoritarian system For the an authoritarian system to be legitimate It has to make a claim that it represents a hundred percent of the population You know in a democracy it doesn't it has to claim that it represents 51% of the population And so in a democracy If if you're assisting the development you're assisting the 51% you're not necessarily assisting with 49 you know unless we protect the rights of minorities and So I'm worried about a horizontal vertical accountability So I think this mythology of a kind of technocratic apolitical foreign aid is dangerous dangerous But I'm a political scientist So You know I think politics is important


Okay, it's been about another minute, so I think I'll go ahead and get going. First of all, thank you all for coming. I know it's like 5.30, pretty close to dinner time.  I'm feeling pretty hungry myself soon, so looking forward to eating after this. I won't try to keep you forever. So, this is gonna be a fun topic, I think, at least fun for me, because this is squarely at the intersection of two talk tracks, I guess. There's the ML and AI track, and there's the observability track, and the two are very firmly intertwined in this talk, so if you're familiar with one, but not the other, then hopefully you'll find some stuff that's useful here. If you're not familiar with either, you might be able to bookmark some of the stuff after you go learn a little bit more about some of the others, but regardless. Let's get going. I wanna talk about, I call it observing an element in production, but this is really about ways that you can make large language models more reliable in production. And the impetus behind this is, you are likely aware, pretty much every organization in the planet right now is looking to build with large language models to some extent, and really, really powerful, but they're kind of hard. It's not magic. There's a lot of problems that they bring, and a lot of challenges that a lot of teams may not have necessarily been prepared to handle. We were one of those, and so I wanna walk through some of our case study at Honeycomb where we learned a whole lot, but I wanna spend a little bit of time on why large language models are hard in production. So I would posit that your average engineer who knows a thing or two about the product that you're working on, if given an afternoon, could probably cough up some sort of prototype that does something pretty useful with an LLM. That's some of the incredible power, and I think it, I just wanna underscore how amazing it is that we have literally the world's most powerful machine learning models just available from a single API call. Like, I could never have dreamed that we had this kind of power available to us, but with that comes a whole set of problems that we may not have knew that we would have. And to extend that, I would say it's actually fairly easy to take a feature that uses a large language model to market for something that's experimental or alpha quality or something like that. If you give a team about a month, they could probably put something together that integrates with the rest of your product, solves a specific problem and does a pretty decent job at it, but the problem is not getting something to market, it's keeping it to market, because that initial launch that you might have, you can create a nice little marketing moment, you can have a nice splash, maybe solve a problem or two, and then your users are gonna start getting used to whatever it's doing. They're gonna expect it to do more, they're gonna wanna do things that you could have never possibly predicted, and they're gonna want it all to be fairly reliable or at least what they think reliable means. Now, unfortunately, you just put a nondeterministic, undebuggable black box into production, and so that kinda goes against the whole idea of trying to make software more reliable. Well, it's not really an either or, and I wanna talk a little bit about that. It's my hypothesis that it's very hard for your typical software engineering team to test and debug LLMs in a way that they would normal software, and that's for a couple reasons. I think the first and probably the most important one is that when you give your end users a blank box and you tell them, type whatever you want, and we're gonna try to make this work in our product, that opens up your product to inputs that you cannot possibly predict.  Can't write unit tests proactively that are gonna be as good as they can be. Now, maybe given enough time, if you spent an entire year building something, you could approximate something that your users are gonna do, but a year in development is nonsense and no business owner is gonna actually accept that as a proposition for building with large language models. Once you're actually there, though, like it's not just about the user inputs because these models, depending on what you're doing, can sometimes be non-deterministic. Now, there's a bit of nuance to that, like if you're using OpenAI, like the majority of us, they are literally non-deterministic, but there are open source models where you can set the temperature value to zero and that can actually be deterministic, but that's a piece that may change in time. But regardless, there's infinite expressibility on the inputs and not necessarily a reliability on what the outputs are gonna be with respect to that same input. You could have somebody input the same thing twice and get two different answers. And that's by design. That's literally why we use this. There's the phrase that large language models hallucinate is something that you'll see quite a bit. I don't really agree with that characterization. I would say that every single output from a large language model is itself a hallucination and it's really just a matter of tuning the hallucinations that you get every single time to be useful to the use case that you actually have in mind. And if you kind of make that mental shift, you can sort of accept that, okay, this thing is kind of inherently unreliable, but we can rein in it a little bit. And the last piece of that, which is there in tax, you can pretty much read it, but with my own experiences with prompt engineering, with something that's live in production for the better part of the year, it's extremely easy to cause a regression because kind of going back to that first point, your users are doing things that you may not necessarily predict. And you're gonna have things that are actually working that you're really not aware of yet. And if you fix something that somebody said it's not doing the right thing, you may have caused a regression with something that was already working and because you weren't aware that it was working, you've now fixed one problem and caused another. This is extremely common in prompt engineering. This is something that anybody who's in production right now with this stuff probably knows and feels quite a bit. And I wanna tell a little bit of the Honeycomb story and how we managed to make our own feature, at least our first feature that used large language models be good enough. It's not perfect, but it's pretty good. So in May of 2023, Honeycomb is an observability product. So if you're not familiar with observability, the idea is that you have a bunch of data from your systems sent to our backend. And if there's something going wrong or if there's something that's particularly slow, you can query that data and you can say, okay, is this, if I'm running an e-commerce site, is my checkout service the one that's particularly slow or something like that? There's all sorts of questions that you can come to our tool with. And what we found with our new users who were coming in is they had heard Honeycomb is a great observability tool. All right, cool. But I don't know how to use it. They would come in and they would know what they wanna ask, but we had a UI that didn't let them actually express that in the first place. And they would walk away and say, okay, I don't know how to use your UI. So one of our hypotheses was, well, what if we put a text box in there, have a large language model do its best to translate that into something that we can get structured output in the form of a JSON object and parse and validate that thing and then turn it into a query that we execute on behalf of the user. We launched. It was a great marketing moment. I definitely do wanna highlight that if you have the opportunity to build something with language models right now, your marketing team is gonna be very happy with you and that will provide good business to the company that you work for. So I recommend doing it. But more importantly, we iterated a ton after we released. And we really, truly stumbled our way into what might be some best practices. There were frankly no documented best practices out there. Prompt Engineering is still kind of the Wild West, but it was even more the Wild West earlier that year. And we think we have a methodology and the beginnings of a tool set that can be used to rein some of this in to make something not just a good marketing moment that your marketing team is gonna be happy with, but actually turn it into reliable software that your end users are going to enjoy using in the long run, rather than just a fun little experiment that you eventually have to shut down because it's too janky. And so specifically, I didn't have the numbers in here. I completely forgot to do that. But we did see actually some good results from doing this, in particular with net new users who come into the product. We see a lot more retention from them using the, if they use the natural language querying feature, they end up querying their data a lot more in the long run and ultimately converting into paid users more than people who don't. And in particular, our enterprise sales folks really love this feature because instead of having to handhold someone through the process of learning how to use our products, they can just say, see that text box? Just plug in your question and see what comes out. And it'll create a honeycomb query for you and then you can manipulate that thing afterwards. And in a way, even if it doesn't do what you want, it kind of teaches you how to use the product such that you can be successful. And our sales folks have basically said, this is great, we love it. This has basically shortened the cycle that it takes to get somebody over that first hump in using the thing. So how did we get there? We practiced observability. And if you're thinking that a representative from an observability company got up on stage and told you that they did some observability for a feature and it worked really well, try not to be too cynical about that. It actually is legit. And when I say observability to the max, I really truly mean it. So whenever Honeycomb releases a new feature, we practice observability for it and see how it goes. But we really don't do it to the extent that we did for this feature, not even remotely close. Very few times in fact, are we leaning on observability as sort of like the primary way to improve a given product, right? Very often, frankly, a lot of our changes are front end changes and we can just directly ask the user like, hey, did this chart thing improve stuff? And they say yes or no and they sort of say, hey, this would be a little bit better.  And it would have been way too much effort for us to actually try to talk to all the different people and let alone understand if we had accurate sampling of their experiences or what everybody wants, so we had to use observability. So starting from scratch, we captured everything as tracing data using open telemetry. So if you're not familiar with the trace, it's actually, it's kind of distributed trace, it's a fancy word, it's a $25 word for a 10 cent concept.  The idea is that if you have a given transaction, a process system, you can say, okay, user input this thing, it hops through whatever subsystems are there, maybe it calls another service, maybe it calls a database, eventually it sort of finishes off and the user gets a result. So there's sort of this pathway and what happened along that pathway, that's what tracing enables. We capture absolutely as much contextual information as we can between the point from which a user hits the get query button and they see a query that has actually been executed on their behalf. Specifically, it's actually when we enqueue an object to run against the querying engine, but it's pretty much the same thing. We do retrieval augment a generation, or rag as it's often called, another $25 word for a 10 cent concept, but it's a large multi-step process and at every single one of those steps, we capture relevant information about the decision that we made and the data that we gathered that we ultimately contextualize into a prompt that finally makes its way to the large language model. And then also on the end of that, when we actually get a request, we parse that request, we parse the value in this case, it's a JSON object, we turn it into what's called a honeycomb query specification that then gets validated against a set of different rules and there's a couple other things that we do programmatically to fix things up sometimes depending on if they need to be fixed up and all the way until a query gets executed and there's a little feedback mechanism and somebody can say thumbs up or thumbs down or not sure on the result that we gave. It's a lot of data, but it's also really important because every single one of these things are a part of the end user experience and especially on the input side, every one of those things directly influences the behavior of the large language model. And so if we wanna systematically understand how this model is actually doing out in the real world, we have to capture this kind of data, otherwise we're gonna be blind as to how, why it's actually doing what it's doing. And so in particular, I'll have a visual of this a little bit later, but I say there's pretty much four primary things, the user clicks the get query button, there's a pipeline of context gathering via a rag pipeline that happens, it's fully instrumented, we make a request in this case to open AI and then we parse the data and validate it against a set of rules and then we submit to our querying engine and that whole thing is 48 spans inside of a trace, so 48 structured logs, each of which representing a particular piece of the end user experience and critically, each of those representing something that could potentially go wrong and we wanna make sure that we understand, okay, if something is going wrong, is that because the model is at fault or are we making the wrong decisions when we gather context? Did we validate something incorrectly? That actually happened. There's all kinds of things that can happen. And then we also monitor this thing with SLOs or service level objectives. So if you're not in the SRE space, a service level objective is arguably the best way to monitor software today. You're not monitoring things like, my CPU level is good or my memory usage isn't blowing up, you're monitoring an end user experience. In this case, you're saying, okay, what matters here? In this case, we wanna make sure that the user hits the get query button and a query gets submitted to our querying engine. That needs to succeed, like that whole thing needs to actually happen and it should happen within a certain amount of time because if it takes like a whole minute to do this, then the user's probably just gonna walk away in frustration anyways, so it doesn't even matter if we were correct. And so these are actually two service level objectives where we measure, first of all, that entire time slice and it needs to take less than 10 seconds. And then we call this an error SLO, what's really kind of like an availability or success SLO maybe? The idea is that regardless of what the user input, we should, for the vast majority of times, always have a query that gets submitted against our querying engine that can ultimately result in query results that somebody can look at that it did their job. Now, of course, the key thing with service level objectives is they bake in a budget of failures. So if somebody enters in, like my favorite color is blue, well, we can't really produce a honeycomb query based off of that, that's kind of nonsense. And so that's fine, the large majority of our users are not going to be inputting nonsense, they're gonna try to get some actual value out of this tool. And so that's what service level objectives allow you to do is they allow you to establish what your success rate over a target period of time is and then what your budget for failures are. And critically, it measures every one of those things, what I said, those 48 pieces, those 48 spans, each of those can represent an event that could potentially fail this service level objective and we want every single one of those to succeed. So every step needs to happen and it needs to occur in less than 10 seconds. And the other piece to this, which I think is not really talked about in the LLM space is if you want to iterate with this stuff in production, you need to be able to deploy your code very rapidly. I would say that anybody who's worked at Big Tech probably doesn't have this problem, but the majority of enterprise organizations out there are constantly trying to reduce their cycle times because regardless of if you're using LLMs or not, you want to be able to ship bug fixes pretty quickly. This really dials that up if you're using large language models. In particular, you want to do things like monitor that end user experience. Are we doing the job and are we doing it in the time that we said we want to do it? Okay. But what if we fail? What if there's a case where something is not very good? Okay. Well, that's fine. We can fix that, but we want to be able to deploy that and immediately look at the past 24 hours and make sure that, okay, we're still doing the job that we want to do, but we've also fixed the problem that we thought we just fixed without introducing something new that could have gone wrong, right? And that's why those SLOs are really important because those allow you to make sure that you're guarding against those regressions over time. And kind of as I said before, this is something that you will find is extremely challenging to put into a unit test suite. You need to use an observability tool that's monitoring all of this data that you've captured as traces at a very high level of granularity. And once you can set up sort of this flywheel where you're successfully sort of getting into this mode where you are prompt engineering every day and fixing very specific problems, you're identifying a specific thing, you ship that fix, you monitor over the next 24 hours, okay, did this actually do it? Rinse and repeat again and again and again. This is exactly what we did with our feature from May 3rd all the way up until July, I believe, yeah, all the way up until July. And we literally shipped pretty much every single day. We would isolate a very specific problem and we'd be constantly looking at these SLOs and looking at this trace data. So that was a bit abstract. There was a lot of stuff in there. I wanna give you a walkthrough based off of this feature and actually how it's doing live as of last week. So the screenshots that I'm gonna have in here are actually live from what's actually happening right now with our query assistant feature. So what you're looking at here is our SLO page. So there's a bit going on here. I wanna make sure you can actually have some time to absorb it. It's called natural language query generation availability. So we're saying that the proportion of requests to generate natural language queries that complete without an error. And so we're basically identifying the success criteria where we're saying, okay, that entire 48 step process was able to succeed. And if it failed, for any reason at all, doesn't matter what the error is. It could be something completely unrelated to the large language model. It could be that we expected a JSON object and we didn't get back JSON. It could be that we just had a bug unrelated to any of this stuff that was somehow affecting this feature. Doesn't matter. We wanna capture any of those reasons why something could possibly fail. And so you can see that we have about a 95.6, about 96% or so historical compliance with this SLO, meaning that about 96 out of 100 times somebody hits get query, they actually get a query. And then we have a budget burn down here. You can see we have about 78% of our allocated error rates. And this is something that's calculated automatically by us saying that we want 80% of our requests that we make or 80% of that people hitting that get query button, succeeding over a period of seven days. So this is just sort of an overview page, but it's too much to sort of fit on a screen, which is why I split this into two screenshots here. And so the second one is where it gets a little bit more interesting. So this is Honeycomb's SLO tool, but there are other service level objective tools out there that can show you similar kinds of things. This is starting to get into very specific reasons why certain things may fail. So what I'm highlighting here in this screenshot kind of at the bottom left here is a particular error called MLResponse does not contain valid JSON. So what that meant is when we got a response back from OpenAI, we expected a JSON object, but we didn't actually get one back. So we kind of can't create a query when we don't have the object that represents what the query should be. Oops, okay, well, let's dig into that a little bit. So what we can do is go directly from one of those failures into a querying interface, in this case, it's Honeycomb's querying interface, where we're scoping everything that we're looking at down to exactly those times when that error occurred, when we did not get a JSON object back. And we've also gone ahead and grouped by the user input and the response, so we can see, okay, sort of in these three cases, in this case, over this time span here, we got this is the user input and this is the output that resulted in that error. And so indeed we can see there are three such cases where somebody entered something and there was no response given back by OpenAI at all. Well, that's not a JSON object, that sounds about right, but that last one is really bothering me because somebody input a number of requests where HTTP.Route contains this thing and this thing and this thing. It seems like a pretty valid input, I feel like that should result in a query. And indeed it looks like I have a JSON object there and the response that kind of looks like it's a JSON object, so why did this fail? Okay, well, that's something that I might want to investigate. So what comes next is digging into a specific request, going into each of those 48 steps and being able to understand exactly what happened when we got there. And so this is part of that. Again, because it's 48 pieces, it's a little hard to throw up on a screen there. And then in fact, I've collapsed over 20 of these spans, but you can see that there's a lot of operations that we're performing. They're named in very particular ways such as find all suggested inquiries for dataset, most relevant columns, fetching some embeddings from an embedding model that we have, create chat prompt, all that kind of stuff. All of these things are relevant here. But in particular, in Honeycombs UI, when you select one of these things, this is one of those spans in that trace. Spans contain a bunch of key value pairs that have rich information that you can capture at each point. And so in particular, I actually highlighted the response itself, the actual raw data there, because I noticed that it didn't fit in my table view. And indeed, that does look like it's a JSON object, but it doesn't look like it's finished. There's order.external underscore, and then it's done.  That sounds like a bug. We should probably fix that one. I'm the owner of this feature, so when I say we, I mean, it probably should be me. But basically, what I want to highlight about this is I didn't start by knowing that there was this case where sometimes it produces part of a JSON object. I started with a very, very high level. These requests should succeed over a period of time. And then started to narrow down, okay, where are the ones that actually fail? All right, let's now dig into that. Let's look at inputs and outputs and just sort of see what's going on.  Let's dig into that trace, sort of walk through each of those spans, and yes, I actually did walk through each of these manually just to sort of make sure everything was going on. It took me a few minutes. And then I found one of these, and I said, oh, all right, well, there we go. It's an incomplete JSON object, can't parse that. Oops, that's a bug, and kind of did the job there. Now I can take this into prompt engineering and say yes, we have a very specific problem that we can go and fix, this wasn't nonsense that somebody input, and in fact the model seemed like it was trying to do something useful.  But now I can start forming hypotheses about why is the model not doing what it's doing? And critically, I have the direct user input, and then I have all data in every single piece of context that was assembled as a result of that tracing. And in fact, one of the fields on here is something that we call full prompt, where after we've sort of gathered all contacts and parameterized everything into the prompt, just literally that full string itself is actually available there. And so what that allows me to do is just quite literally copy paste that field, go into prompt engineering, and just replay that request. And be like, okay, is this reproduced? Like what is actually happening here?  And the insight that I sort of gleaned from here, and when I say I need to fix this problem, I mean, I literally do need to fix this problem because I've not submitted a bug fix for this yet, is due to a variety of reasons, we limit the response that we get back from OpenAI. It would be a whole other talk to describe why you wanna do this kind of stuff, why things like rate limiting matter, why things like tokens permitted matter, and various ways that you can guard against prompt injection attacks and that kind of stuff. Limiting response size is one of your mitigations against a variety of problems that can go wrong when you're building with large language models. And in particular, we say 150 tokens is sort of the max we assumed that that would probably be enough for most responses that we get back. And that's true, but now we're at the point where I actually do wanna fix this thing, and 150 is actually not enough. Furthermore, we've been in production long enough that I can probably be confident that we're not going to have this be the attack vector that finally takes us down. So maybe I can think about bumping that up, or maybe I can instead be, okay, well, what if that output should not have been more than 150 tokens? What if it should have been less? All right, well, that's another path that I can explore there, and I think I really wanna highlight that this is based off of real user data and real interactions that we're getting in our product. This is not something I had to divine from somewhere and hope that it actually represented what users were doing. So the last thing that I wanna highlight is, what I sort of showed you is there's really nothing magical about this, there's really nothing terribly special about it either. You can do all of this today. All of this is powered by open telemetry. If you're not familiar with OTEL, I highly recommend checking it out. It's quite a bit, so set aside some time to make sure you really can understand what it gives you, but it's an extremely powerful instrumentation framework that is also an open standard that works with pretty much any tool. So you don't have to use Honeycomb, you can use pretty much any observability tool out there, they all support open telemetry, and you can do this today. Whatever language you're using that's actually running your application, you can just pick the SDK that supports it. Right now, OTEL has stable support for 11 different languages, so chances are what you're using is already supported. You can add what's called auto instrumentation to your whole system, and you start with that first.  That gets instrumented for you automatically, so you can really just focus in the context of your own application. Okay, when we're doing retrieval, this is the operation that we're performing. We're gonna create a span, we're gonna capture a bunch of data at that point, and that's just gonna be a part of this trace that's been created for us, or rather the skeleton of that has been created for us automatically by open telemetry. And then you can just have your dev team in the course of, I'd say, maybe a day, maybe a little bit longer, kind of depends on OTEL setup. Get open telemetry setup and start identifying these are the relevant operations that we wanna instrument and instrument those things. I've seen this many times, lots of engineers, like a senior engineer and a team somewhere could probably get this stuff wired up within a day and start to give you even just the most basic insights about what your users are actually doing. Lastly, I think you can expect the way that you accomplish observability with LLMs to get a lot better over time. Right now there's a lot of really rich automatic instrumentation for all kinds of back end technologies. Those back end technologies have been around for a lot longer than the constellation of technologies that are sort of emerging when building things with large language models. That's gonna change. We are sort of en route, I guess you could say, to building auto instrumentation packages for things like requests to LLM providers, frameworks like LangChain and LamaIndex and all the different C of different vector databases that you might be using and just making sure that automatic instrumentation that captures these are the requests, these are the inputs and these are the outputs that came back can get wired up for you. Similarly, there are standards being developed right now in the open telemetry side of things. Come talk to me, I'm the person who's proposing them. And we're basically identifying naming conventions for things so that for example, if you wanna capture a prompt, the name of that should probably be LLM.prompt or something like that. And what that does is that provides a specification that additional libraries can be built up against so that eventually we can have a pretty rich ecosystem of automatic instrumentation sort of at every level of application that uses large language models. And finally, observability tools are, we're identifying this as like this is a legit use case for observability tools, period. And so Honeycomb is one, I know Datadog is another, there are other vendors out there that are starting to build out better support within their own product once you have that data to support this use case quite a bit better. So that's all I got. And we got just a few more minutes but it's also near dinner time. Feel free to ask me any questions you have, I'll stay here as long as you want me to. But if you leave to go grab food, I will not be offended. Thank you. Thank you.


All right, a very good evening to you. Welcome to Chichat Live and here on YTF4 TV, your favorite youth channel. How are you doing? I hope you've had a great day. You've had a fantastic day. Thank you so much for joining us. We're gonna have a great show. Now, if you're joining us for the first time, now at Chichat, we're just all about gossip, trending topics that have been making news all week. That's what we talk about. We do a roundup, all right? Now, my name is Felene at Felene Jeanne and all social media pages. And guess what? I have the Gang Gang in studio representing today. Now, let's get started. I'm gonna start with our very own C.J.C. as we call her. What's up? Hi, how are you? I'm good. How are you doing? I'm good. Thank you. We're... What's your name? I don't know. My name is Coincidence. I'm from Coincidence. Coincidence. I'm from Coincidence. I'm from Coincidence. I'm from Coincidence. No, it's not there. It was Coincidence. I think it means feelings where it's said to us in a connect. Yeah. That was three weeks. Three weeks. It was in a connect. You need to waste the akibila to remember me. I need to come back. Hi, so guys, it's a saying to me. I get to my office. It's another Thursday. Come back to Waida. And stay tuned to my C.J.C. I think it's a drama to come home. So, since you're not after me, I'm gonna zip out. Of course, of course.  Dennis, a headmaster. What's up? How are you doing? How are you doing? I'm doing good. I'm doing a headmaster for Danny. And the intro here is Sambamba Vizuri. Look at Danny. Look at Danny. I'm Danny's headmaster. Okay, fine. I'm gonna tell Danny that I'm a dad.  I'm gonna remove you from the house. Because I'm a cutie. I'm a kid. Ladies and gentlemen, the room is officially open. All right, that's what's up now. Let's get started with our first story right here. And American actor and singer Rotimi, better known as Drey Besan, he's role in their series Power and Houston's onion fiancé, singer Vanessa M. Day. On Tuesday, they revealed to the world that they were expecting their first child together. Now, speaking to People Magazine, the two revealed that they were expecting a baby boy sooner. They said, we are super excited to welcome our son as first-time parents. Rather, everything about it, about this experience, has been a brand new challenge and we love it. End of quote. Now, the two got engaged shortly after they started dating in December 2020 in Atlanta and have since moved in to get there. There. I know. All right, now, here's another thing. Vanessa's ex, that is Jux. Guess what he did? Ngoma. He released a song, just to wish them well. Let's have a listen. That was maddened. So many things to talk about. First of all, do you know Kim and Kanye? During their whole relationship, they had that one factor. And that is RJ. The one constant factor in their relationship. Up to none. RJ. The RJ is that one ex, who never goes away. We all have that one. Is Jux going to be the same for Vanessa? I'm not what is happening. Like, from a man's perspective, for all those two MBA, what is happening? Jux has a job. I'll go and add you to one. You can't just leave it to just go. It is... You have to write on that one. Yes, imagine. You know what I mean by Jux? If he's not working in public, I mean he's working in public. Why not? It's October, October. You know what I mean? Yes. We've got to get things paid. No, it's true. Jux, I really hope this thing works out for you. If it works out, if it works out well, if it works out, we'll be able to make money out of it. You know what I mean? What is the other thing you want to get from this? Imagine you're with Jux and you're holding feelings. No, but this is the second song. There's a guy who had a collab together with, not so long ago, when he was with Jux, he was like, I'm the one with Vanessa. I'm the one with Vanessa. I'm the one with Vanessa. I'm the one with Chana too or something. So now that we're pregnant, he's like... He's the one with Vanessa. He's the one with Vanessa. Like, I wish... Like, I don't know. JC, is he that one ex we are talking about? Yes, we all have that one ex. You know what I mean? If you move on, if it works out for you, boom, and reappear from nowhere. Okay. And you can't go from... Boom, boom, boom, boom. And you can't go from where you are. Jux, you know what I'm saying? You know what I'm saying? He's like, Why? He's like, he was pregnant two weeks ago, before he was pregnant, and I said he's pregnant. But he's like, I'm the one with Vanessa. No, he's like, I'm the one with Vanessa. You took your time to do this bra. So, it's probably new. It's a hit, yes. People knew. No. So, you know what I'm saying?  Now you know what I'm saying? Jadi. Are you excited for Vanessa? Are we excited for Vanessa? You see I saw everybody was Even trending on social media was trending Just the perfect time because he released his album the other day Are we excited for them? Yes, I'm really excited How do you feel about that? I'm so happy because I'm looking back at her relationship with Jukes I'm saying that she and I have been dating for 6 years That's a long time There is nothing that he is giving her to assure you that he will be there forever That's when you move on and you are happy because you are at the studio You are at the studio and you are happy because you are at the studio I'm so happy for them I'm happy for the fact that men are trying to do now what we thought our grandparents did They are going to be there forever That's the point of the generation The more you date with someone, the more you are sure I know You see, people never used to move in together They used to be brothers and sisters They used to be brothers and sisters Exactly It's true They are brothers Imagine if you were to date with a girl, you would say that after 3 years Let me get this straight Men when they meet a woman They always see if this is going to be the one or not Let me just give you a few examples So when ladies are out here in the streets on social media They say, tell us how you prayed I don't know Jordin Sparks tell us how you prayed I got a very heavy You see, you see I know they are like, they got up and they are somewhere I call upon you So is that true? It's actually like men who have been married for 6 years Didn't get married, didn't marry Never got pregnant, nothing And now they are here They are here Engagement, they are together They are going to be together They are going to be together You see too, after I compare He is so happy He found her So I am trying to understand What is happening I love her You see, men always know First date When you meet a woman When you meet a woman It's a pass First date I will tell you briefly By the ladies out there She is the first date I don't remember what she said At your first appearance She is the first sighted She is the wife When you meet a woman She is the first date She is the first date She is the first date She is the first date Let me tell you Every man has his own Type The moment that type shows up She is the first date She is the first date She is the first date She is the first date She is the first date She is the first date First date So you mean According to People magazine I was reading what they said They met They met in the US Vanessa was there for an event She attended an event That's how they met When I came back to Africa They stayed They started talking He was like He is the first date Then they said He was like You know what You are moving No They said There is a woman in the US She is the first date She is Africa He is like You are coming to the US And you are going to be my wife Just like that I will tell you for me That happens They said I know so many friends of mine When I was in the center of attraction Every time he disappears From that circle He is the first date Every time he comes He is like He is the first date He is the first date I know I know I know Look He is like Who him? Two women are once They are once He is like Let me give you something very funny Men are very jealous of creatures They are very jealous We protect what we love If you ask me How will I go? How will I go? Who you are amending missing Or what is missing? Missing These are the real statements I met his friends, I met his boys, nothing. Yo, that doesn't mean nothing. I met your mom. I'm going to say goodbye. I'm going to say goodbye. If you, as a lady, may introduce you to your guys' friends.  Where are you at number 20? In your list, where are you at number 20?   You're the one who disappeared.  You're the one who disappeared. You're the one who got my friends from sugar.  Ah, will you make it? I'm not acknowledging how.  What's your name? It's just me. I don't think it's an accident I don't know what I'm saying. I just want to give you a word. Jesse's boyfriend is here. I will come back after I say. I am not saying it. It's impossible, it's not possible. I can't believe it. She said, how can I be so blessed with 22,000 words per day? She said, how can I be so blessed with 22,000 words per day? She said, how can I be so blessed with 22,000 words per day? I don't understand. That's the thing, if you want to know where you are blessed, Now, what I want to introduce you to, I will introduce you to here in order, I am not a beast. I know them from the order of dangerousness. Before you come to know it, you've met like 10, 15, but you've not met them one, so the day you're going to meet them, They were like, oh my God. Okay. But you can realize that they were two and a half years old. Yes. I was very surprised. I said, I want to introduce my besties because they might be hotter ones than him. Yes. You know, so you have to wear your options. Yes. That's not it. That's not it. That's not it. Yes. The besties must wear it. They should wear it. Oh, so they'll wear it. Yes. That's for the besties. That's for the besties. They're like a creature. They're like how we used to live in a house. That's what Jacky. We were born in a house that was much cooler than we know it. We were born in a house that was much cooler than one and a half years old. We grew up in a house that was much cooler than the other two and then we decided to get we're friends. There is no friend who can meet us. I love you. You look great. Yes. Yes. I was trying to hit on you. I don't have the. All right. We start receiving calls. Yeah.  I'm so sad that he had to go through that and that some people in this country would go so low that now they're just going after my family and it just makes me sick in my country. Yeah. All right. Yeah. Yeah. Yeah. Yeah. Yeah. Yes. Yeah. Yes. Yes.


Everybody forced here ripping a box of twenty one twenty two court Kings basketball Hobby box here for David. Hey, thank you David and good luck That is a kiss burnt level one, right? Yeah level one to kiss for get Davey on Mitchell level two Drew holiday that's out of 149 red foil Ruby foil Katie Cunningham Jersey card the pistons rookie there nice, Kate The auto is a mark Jackson impressionist ink Number to 25 on the mark Jackson got a work-in-progress rookie Trey Murphy the third and Jalen Suggs level four very nice That's awesome great Suggs there good box for you David. Thank you again. We'll get those cards right out to you


Hello, everyone. In my paper, I'm going to present an international project in which my university, Adam Mitskewicz University in Poznań, participates. The project is related to managing big data in archaeology. And it will be rather brief presentation due to limited time, so my presentation also will be around eight minutes. And I will do that in the three short following steps. So let's start with a short introduction to the project. Sejshat, the Global History Data Bank, is a massive international database of information on past societies. It was founded in 2011 as a response of necessity of organization of vast amount of data collected by historians. So it was born on the ground of history first. Here you've got the project website, so I encourage everyone to just open and you can find every information on the project. So what is Sejshat? It's a very comprehensive data bank that comprises different variables, different aspects of past societies, like social, economic, ritual, warfare, and track their development of these aspects through time in a selected areas. It's therefore designed as a general tool for identifying long-term trends and asking big questions. And moreover, this is a tool for testing various hypotheses and theories about the social evolution and historical dynamics. In the future, it will become a very powerful tool that allows users to ask a very broad set of questions about the past. Sejshat extracts data from different sources, websites, academic publications, human experts. And the long-term goal, it's very ambitious, actually, because the long-term goal is to incorporate the data of all human past societies. And at the moment, the Sejshat database is running on a wiki engine. So similar, the same as Wikipedia, which allows great flexibility and a large number of users working on the database. However, as I will show you in a moment, it will change in the future. So the plan is to move to some very advanced IT scheme. So on the ground of Sejshat, the Global History Data Bank, archaeological Sejshat was born. When after a long discussion and the practice of collecting data, we realized that actually Sejshat lacked some important archaeological variables. And also, we lacked some proxies that could translate the archaeological data into this Sejshat coding data collecting template. So actually, the archaeological Sejshat developed in a few stages. I don't have time to describe this in the test. I need to mention the Oxford team. So the team from Oxford University initiated the discussion. So people like Peter Turchin or Harvey Whitehouse, they started this initiative to create archaeological Sejshat. We also have a couple of meetings on which we discussed the archaeological Sejshat. And I have to mention that now and in the future, quite a number of people around the world, teams from different institutions, volunteers, have been working on it. And we still are looking for some more volunteers. So here are just a couple of examples of the key archaeological categories that were generated as a result of those discussions. So just examples, what set of questions we are asking, we are looking for while collecting the data. And just we don't have much time, but I want to show you just a quick look what we are doing in Poznań, how we employ archaeological Sejshat in practice. So we selected around 70 areas through Near East, from the Near East, through Anatolia, Balkans and Central Europe. And we are tracing the development of Neolithic way of life. And the collected data in the future will allow us to test different hypotheses concerning the spreading of Neolithic and tracing different changes in different aspects. So later on, if you have any questions about the practice, I'm happy to answer that. So in the final part, what is the future of Sejshat? So Sejshat is one of the use case of the bigger European project called ELITE. So ELITE is a European Union project that began in 2015, and it will end in January, 2018. So pretty soon. So the main focus of ELITE project is developing methods, tools for engineering information systems. And in terms of Sejshat, so ELITE will help us to develop a platform for Sejshat database. So we are going to transfer Sejshat from wiki format into the RDF standard, and the Sejshat data will be stored in a triple store database. So... Okay, one of those tools is Dakura, so created and managed by Trinity College Dublin. So it's a great example of a tool who will allow us to make this data-gathering process more quick, more effective, and... Well, yes, and actually I need to mention that we already managed to... Because the program is in the making, but we managed to release a first portion of data in this new RDF format. So, yeah. Anyway, those tools will allow us to move away from manual data harvesting. They will speed up the gathering process and later also querying and searching for the information. Those tools are based on the idea that the tools allow us to go over a large digital text amount and search for relevant information, and also those tools would structure the data-gathering process which would keep track of data quality and information quality. So to sum up, in the future, Seishat will become an extremely powerful and one of the... Well, let's face it, one of the largest data banks on human history and prehistory. It will incorporate tools for automatic harvesting of data from internet-accessible sources, databases like, for example, JSTOR. It will allow us to look through vast amounts of data, and test different theories on social revolution. Okay, that will be it. Thank you.


Hi everyone, I'm Joe for jaspyscasewrakes.com coming at you with 2019, Panini Prison Football Hobby Pack Break number 13. This is actually a pack from a fresh case for whatever that's worth. At the end of the break, we'll re-randomize your names. The top five will get teams in this order right here. Name on top, we'll get the Steelers, second name Houston, third, fourth, and fifth. Pretty good price point too to try to chase some of the nicer teams. Those pick your team four, we'll be coming up in the next video. Very big thank you to everyone here for rallying around this pack, for making it happen. All teams are in. Let's roll it, let's randomize it. Four and a one, five times. Different dice roll for the giveaway part at the end, one, two, three, four, and five. Four and a one, five times for the football teams. Two, three, four, and fifth in final time. After five, we got the Colts down to the Giants. All right, no trade windows here, but here's the first half of the list. There's the second half of the list right there. It's alphabetized by a team. Print and rip. Got it. Josh Norman, one, two, three out of 175 for Washington. I'm going to go to Gary. Pretty quick. Nothing too crazy in that pack, but let's see who's going to win these teams. Let's gather all your names again from one through 32. New dice, new list, top only the top five. So fingers crossed, top five after eight, three and a five, eight times. Good luck. One, two, three, four, five, six, seven, and eighth in final time. After eight, a lot of sad faces here, but I appreciate everybody giving this a shot. Very close cane, no team though. Thanks for trying. Thanks everyone for giving this a go. Appreciate it. But there can be only five. Top five will get teams. Gary, you got a team, Mark, Austin, Gary, and Gary with the top two spots. After eight, put a little rooftop next to your name so you know you won that in this filler. With the Pittsburgh Steelers and the Houston, I almost said Oilers, the Houston Texans. They haven't been the Oilers in a long time. Austin with Tampa Bay, Mark with Philadelphia, Gary with Buffalo. Thanks for watching everybody. We'll see you in the next video for the next break. Jasper's casebreaks.com.


7 Activities for Teaching the Simple Present for the ESL Classroom ESL teachers often encounter difficulties when teaching grammar patterns to students abroad. The following 7 activities for teaching the simple present for the ESL classroom will help your students pick up the tense in an easy and efficient way. 1. Daily Routine This is a very good activity for when you start discussing the simple present and how it's used for routines and regularities. You can place the students into groups or pairs and have them discuss their daily routines. What do they do in the morning? In the afternoon? In the evening? You can then also have them write up a schedule of their daily routines and share their findings with the class later on in a class discussion. 2. Cultural Norms This activity is great for introducing adverbs frequently used with the simple present. Have your students share things that people do in their country, culture usually, normally, typically, etc. For example, in my country, people usually eat rice for breakfast. The next step is then to encourage your students to make negative statements including adverbs like never, rarely, infrequently, etc. 3. Classmate Interviews This activity is useful for new classes where the students aren't too familiar with one another. Set up your students in pairs and have them ask each other about their families, hobbies and interests using the questions you like. And what do you like to do in your spare time? After a certain amount of time, have each student introduce their partner to the whole class. Here, they practice using the simple past in the third person singular saying things like Bobby likes to play football in his spare time. 4. ESP, Extraordinary Sensory Perceptions Sensory observations are always expressed using the simple present. Therefore, have your students observe their surroundings, at school or at home, and have them take notes. Hereby it's important for them to use and describe all five senses, sight, hearing, feel, touch and taste. 5. Where am I? Similar to the previous activity, students are encouraged to use all five senses to describe a place they would like to be in. They make five statements or more describing that place to the rest of the class. The other students must then guess the place. This is also a great opportunity to review the uses of that, on and in when describing places. 6. Present Simple Bingo Mingle For this type of bingo game to work, brainstorm a list of get to know you questions with your students and write them on the board. For example, do you like to listen to music? Do you drive a car? Do you play the violin? Next, give each student a 5x5 grid whereby the center square is blank. The students then fill in the questions from the board into the remaining squares on the sheet. When done, the students mingle around the classroom and ask each classmate one of the questions. If someone answers yes, they write down his or her name into that square. The first person with five names in a row wins. 7. Weekend Party This is a great activity to play before the weekend. Ask your students what they usually do on the weekend. Have them share their routines with the class. If you like, you can even have them write up a paragraph about their ideal weekend and then read it aloud to the rest of the class. Are you ready to teach English abroad? These seven activities for teaching the simple present for the ESL classroom are ideal to help your students understand the structure and usage of the simple present tense. Speak with an ITTT advisor today to put together your personal plan for teaching English abroad. Send us an email or call us toll free at 1-800-490-0531 to speak with an ITTT advisor today.


Hello guys, it is crypto grounds here. Welcome back to another video. Um, yeah, 500 subscribers. Woohoo. I have a I have a 500 subscriber special coming out soon. I have finished recording it, but I decided I wanted to make this video first Since it's been highly requested throughout my entire channel and I Finally got to making my own Server-based leaderboard and unity and that's what we're gonna be doing today. I have my Document right here. It's got 50 freaking steps.  So make sure you make an account. I will kindly go through the I kind of go through the process of making it count So you're gonna want your put in your email address your password and your count name Which is used to sign in I'm not gonna. I'm not gonna go past here, but So it'll ask you for a professional or a personal user I use professional I don't really think it matters But I would recommend it's professional and then you're also gonna get this picture right here You're gonna get this makes you choose the basic plan and you should be good to go Yes, so you will once you make your count you will go to the dashboard. Let me close this real quick So let me sign in real quick So you will be brought to this screen right here The AWS management console. So what you're gonna do is launch a virtual machine with EC2 You will get this dashboard right here What you're gonna do is select the Amazon Linux to AMI, which is the first one make sure you choose the x86 archetype whatever it's called and For testing purposes, I would choose t2 micro All these settings should be good Storage that should be good actually here. I Think this is good Yeah, this looks good. So The storage should be good. This honestly all should be good to go and yeah, so Once you launch it, hopefully I'll be able to remove mine Let's see. Okay, perfect. So you're gonna get the screen So you want to create a new key pair after all this is set up and once you click launch Oh, and yeah, create a new key pair Choose a name and download it and save it to a folder on your desktop Now I don't want to go through this process again because I already have it see I have mine right here It's called CG access that PEM. So make sure you download it somewhere You'll remember it never lose it. You're gonna need it maybe in the future Do you want to make another instance or something like that? So anyways, I have mine set up already?  Oh, yeah, so you want to download putty so search a putty online you will get this right here and And Download the 64 bit. It's the installer msi. You will run through the installation process And that should be pretty self-explanatory Then you want to launch putty gen What you want to do is load Your dot PEM file so you won't see it here at first But make sure you choose all files and open up this dot PEM And then once you load it, you know what I'll load it here since I'm probably not going to use it Okay, so once you open it in you're gonna have a key passphrase. So make sure you just type in two passwords and That should be good to go you want to save as a private key saving the same spot as what you have your dot PEM Folder and you should be good to go Next you want to open up petty. I meant putty I think that's how you pronounce it. So what you want to do here is after you make your server you're gonna want to go to your instances and Click on your main one and copy this public DNS right here and That's gonna go right here Before that make sure you type EC to dash user at and then your public DNS Make sure you keep the port for 20. It's make sure you keep the port to 22. I will be important for later and Go to this SSH right here. Make sure you expand it by pressing the plus sign click on off Keep these settings except for allow agent forwarding and Then you're gonna want to open. I've already done all this before so Yeah, we should be good to go so you open it you'll get this kind of console right here and If you get a pop-up that says it's just like a some random alert may she click ignore or just click Yes to ignore it and what you want to type here is run. That's you want to do Want to type in pseudo Yum update Why for yes? Okay, we're good to go for that and then what you want to type is pseudo yum install Tmux so what this does is that it basically allows you so I already have installed it Which why I didn't show anything from you basically allows you to close everything You don't have to touch it ever again. The server will be running endlessly on the back end at AWS Very helpful.  So you want to take new you want to do tmux New dash s and choose a name. I'm just gonna put Name I'm just gonna put serve serve for server Okay, so this will be our window or our console that we will be interacting with After that, let's see. Where are we? I am looking at my instructions because there's no way in hell I can remember how to do this from scratch Um, so when you do close this in the future in order to resume back to this you can do tmux A dash t and then serve and then or serve and you'll be back to where you were So that's kind of convenient Let's see. Oh, yeah. All right. Next. So you want to go to cyber dot cyber duck Dot io Get it from my download for mac os windows do not do this one because you have to pay for it And you'll click the cyber duck for windows right here, which I already have it Hopefully I won't install. Yeah, so you just run through the installer And you should be safe there Um, what's the next so, um, open cyber duck. Yeah, allow. Okay Okay So what you want to do is click open connection And I want you to choose sf tp In the server you're going to put Um, why did you put oh, yeah the server it's your dns. So just put that in there And for your username It's ec2 dash user And keep the port is 22 You don't need a password And for your key go click choose Go to your whatever you saved it and choose that private key. So this is mine right here new key with password Okay. Oh one very important step. I totally forgot to do so close all of this real quick You can just do control c actually, you know, just close this Totally, I've totally forgot to do this one step. You see this app right here pageant right here Double click it And you'll see this list right here.  I have my default settings right here. So I can just load it. You could save it too Just in case you do need to load it again Go back to off agent forwarding Open there we go. We should be safe to go And we can also do tmux a dash t serve There we go. See we're back to where we were before and had our previous commands I think that's just really cool and honest and in my opinion um Okay, so Opening a connection. What was I doing? I think I stopped in the middle. Oh, it's right here. So, yeah, make sure you choose your dot ppk that you have created Um, and then you click Connect. Yep. So I'm not going to connect because I already have one connected So on here You need to install node. So this is quite a Combination of things to do. So I want you to type When you paste it in here, you will need to right click it for it to, you know, continue to paste Um, I'm dumb and I kind of Okay, yeah, so just Right click the pace. Oh, so copy or no paste it Control v. Why isn't it working? So Copy or just type down what? Yeah, just ignore what I'm saying. Just type down this command Enter So I've already installed it. So I'm good to go and then what you're going to type here Is to activate node type that Okay, should be good to go and then search and then type nvm install node I'm not going to go through the installation process because I've already done it But that's what you type and now we need to install something called the yarn in order to access our server So we're going to paste that in here. It's the curl With the dash o make sure you copy everything Exactly right because it will mess up And then you'll press enter, but I don't need to do that. Sorry. I haven't installed And then what you need to do You need to close this after you installed it So do control b and then let go and then type x and then y And then you need to do tmux a dash t serve Oh, um, it's it's I forgot we close the session. So we have to make a new one Don't worry. It's not a big deal if you have to make a new one or not So you do new dash s serve Here we go. We're back to where we were The control b and accept completely shuts down the Our app so we have to create a new one every time And okay, so now we're going to go back to our instances where I scroll down to security groups Click on our launch wizard one Right here go to inbound edit And I want you to copy these two right here this one you should have already But I want you to make these two right here. So I'll take a look for a second while I get ready Okay, and then you want to save And then we are going to do yarn install I already have it installed. So I'm just gonna yeah see it's already up to date And then what we're going to do we're going to create our strappy project Which is what we're going to be using our server for Um, so we're going to go here. I want you to type yarn create strappy App my project I don't want to make a new project because it's going to take a really long time to make it in here because you have such a Slow server it took me at least an hour I don't I mean it took me an hour to convert mine on my desktop to the server So I don't know if installation will be the same But I'm not going to go through that process because I don't want to wait an hour While I'm recording, but you type this command You should be good to go um And then exit out of the project. So you're going to do the control b Control b x control b x y and then you're going to do teamux new as serve again and then You're going to You're going to go to cd Whatever your project name is that's mine And then I'm going to type yarn start So the reason why I wasn't able to start the Start the server properly is because I already have one running So I don't so we're all good. Let me actually check just to make sure I actually have my server so running And other things you don't want to share this url with anyone I've I've been blocking it out because I don't want anybody to see it Oh, right. You need to do I type in the port as well Which is 13 37, which that's strappy's port and there you go So here is where our stuff is being stored just for an example But yeah, so that's where mine is once you start it you should be able to do slap You won't go automatically, but I want you to put your dns here Put the port and then type admin And then you should get a login here mine's mine's different because I already have an account, but I want you to create your account So let's I'm going to log in for me because I already have an account Remember me I mean already Yeah, um, okay So your screen is going a little bit different because I only have one content type But first thing you're going to go to users And I want you to add yourself so add your username and your email that you used to log in You should be good to go And then go to roles and permissions go to the public one right here I also sorry if I'm going really fast. I'm trying to make this a really short tutorial So it's not an hour long. So you got to wait for me to explain something I want you to be able to pause the video continue and I'll be at the next part And there's no step. I'm skipping here. I'm make sure I get everything Okay, so for the events I want so it'll look like this So make sure you select all And then you're going to have to add a group So make sure you add a user if that doesn't show up just add your username right here So you're able to create events and stuff like that Um cancel because I've already done that so go to content type builder I'm going to make a new one. I already have my events one right here Well, let's create a new type and call it tutorial Continue So what we're going to add to this tutorial is right here So we're going to add a text. This is going to be our username Make sure you keep it capitalized and make sure in your script it matches this too Just just so you know ahead of time Uh add another field we're going to add a number And I choose decimal since I use doubles You could use strings if you wanted to if you want to do if you want to convert your big number If you do big double you convert that to string and submit it here It works exact same But I'm going to do a number because that's how I do it Um I'm going to do it money Finish so we have our two fields right here And we should be able to refresh and we should see that at the side right here I don't know where to go. Oh shoot. I actually did something. I forgot to save it Let me create it again User name Add another field number coins Finish ah shoot decimal finish save Okay, it didn't save for some reason. I don't know why I think I'm allowed to have one content type only But this is what yours should look like it should be empty. So just ignore this right here But this should be empty right now Um, you can create a test one if you wanted to you just go to add new events type in your username Hi, and I put my number in here. I'm gonna put 10 Save it up that will be showing up on my leaderboard the next time I play my game So yeah, it's cool it updates automatically and yeah, so So the thing I'm going to show you what to what to do next is go to your link So when you want to see your data as a text For me since I made tutorial I would search up tutorial, but for some for some reason it didn't create I'm just gonna use mine as events, but yours would be your name. Make sure it's all lowercase And see here. There's my leaderboard. There's the high that I made And oh another cool thing so you see this id right here 56 you can go to slash offense 56 oops And you'll see this individual one right here. This will be used really useful for replacing uh Very replacing numbers like updating the leaderboard instead of having to add one every time which I will explain later on But yeah, so don't forget that this is lowercase right here. See mine's events. I would use one word also um I think that's all I need to show you there So let's now we can hop on unity Keep this open. You're gonna need it So let's make a new Let's use my idle game tutorial series even though it's not part of it Ah, shoot. No, I don't I don't want to use it. This is connected to get hub Rats, I didn't I didn't want to do that. I wanted to start a new project I'm gonna make a new one called leaderboard making two projects now oops Yeah, while we're waiting my 500 subs special I am almost done with that It's gonna be it's gonna be a special cookie. I'm gonna tell you guys that it's gonna be a good video It should be fun. It's gonna be long, but I feel like that It's it's a different video. It's something I haven't done before but I feel like you guys should like it Uh, you saw my script. Oh, no Okay, so all right, let's create a canvas real quick. All right. I hate this is the part I hate ui canvas ui text we're just gonna make We're just literally just gonna like make a very basic project because I yeah, we don't we just need us We need a little giant text So this is how we're gonna format it. It should look like this one name number 100 Now the thing I'm not going to show you here is how to set your username because I've done that in one of my clicker hero Tutorial, so I'm gonna have a a name set already. So every time we load We're gonna, you know show it Show what it looks like. So here we need to make a script call We can just name it anything. I'm just gonna call it leader leader leader b because I don't I don't really don't care um Open up this script So I'm kind of I'm gonna be kind of all over the place. So just be just be ready for this Okay, so I'm gonna copy and paste a class that I've made on the side called leaderboard entry We can get rid of id because that's not really relevant. Actually, we will need id never mind. Um, so Yeah, so this will be our coins right here um leaderboard entry That's be good Now we need to copy and paste This should be capitalized This should be capitalized. We don't need id We are going to Have two methods. I like starting from scratch public void start. We're gonna have a username be username be crypto grounds to Doe grounds 2.0 And our coins will be 154 146.7 just some random number um public void updates Okay, so now let's let me hop down to my server right here I'm at my leaderboard um Okay So we need to have So we're gonna we're gonna make a few things We need to have two more classes above this Oh, we also need a using system as well in order to serialize this stuff Okay, so Screw that we're removing this to our main class. So we could put classes inside Other classes if you didn't know that so here you go. We should get no errors here replace this with Coins and actually we don't need this leaderboard entry. I don't know why I still have that Yeah, we don't need that anymore. Okay, so just get rid of that Sorry about that. See I told you I'm gonna be a little bit out of place. So just don't skip anything And we need a string here And I want you to put your dns right here So I'm gonna paste mine I'm gonna block it out make sure you have the slash events too And I'm gonna I'm gonna copy and paste method by method explaining like those So here's one right here This is post to leaderboard Ah, we need another one using unity engine Dot UI for the text that we're going to use in the future and unity engine Dot networking I'll give us ability to use unity web web web quest um replace some of this data so leaderboard data Ah shoot coins. This should be coins Okay replace coins with That It should just be username username username Okay No, let's see what's wrong here. I'm gonna copy and paste all these up here. I think we need system dot link as well So let's see. What do we not need here? Yes, we need system link. I have a feeling That gives us ability to use dot count as well. So this should be the four things you need Uh accidentally do accidentally deleted UI So Honestly, I think we can delete this but I'm just gonna keep it because I I don't want to retest that honestly So basically what it does Is it grabs it creates a new leaderboard data It sets this so it's an empty it sets coins Right here It sets coins to zero All right. No, no, no. Yeah, so it starts with everything starts with zero or blank um It grabs the coins which is going to be for 146.7 which will be set here and We don't set username yet because We don't need to do that yet Yes, okay, perfect. Um, so then we're gonna um use json utility To json to convert this um to convert this whole thing into json into a json string And the reason why we don't want our username here is because we want to it's because we want to do something here Which I'll explain in a second and then we're going to convert this json string to a bunch of bytes Which is what unity webcrest will be able to Use to send that those bytes to the to the server And it's really cool how it works, but so we're going to create a unity web request called request And basically this right here it checks if the leader board event leader board um if Trying to think okay Trying to think what I did here Basically, it's supposed to check if leader board already has Yeah, so it checks if username already exists in this array and If it does then we will go to this for state for statement right here which basically This event leader board Okay, I okay. I got myself confused for a second. I understand what's going on now, but this right here This this will be grabbed later on So you'll see where this comes from but basically There's a method that will grab the entire leader board and store it into an array okay So it's going to check my entire leader board to see if this our username crypto grounds 2.0 exists If it does already then we need to update that variable instead of creating a whole new one Or else we'll see two crypto grounds 2.0 So we so that means it'll go to this for loop where It'll start from beginning which is like the first entry to the last one It'll check if any one of them Is equal to use it has the same username if it's the duplicate Then basically we will we'll just completely delete it from the leader board So we do that by doing request equals unity web request dot delete. We put our dns right here We add a slash and then the leader board id And that's cool because that's where we see this right here. See this slash 56. That is exactly what we're doing right here So another thing I also want to share before I continue these id's they never reset they don't go back to one So every time I make a new id 57. Oh, it would be 57 if I delete 57 I create another one It'll just jump from 56 to 58 so that never decrements it always increments Which is why I made this id thing Because I don't want to I don't want to go from one to something like I really wish I could I don't make the things a little bit easier, but that's why I put this id here And that's why I have The id right here anyways So it will delete it Now we'll go to the actual part So if this doesn't happen, this is basically ignored It's gonna complete it's gonna make a new so basically deletes that duplicate And then it doesn't exist anymore And then we set username to our username We we set json we convert it to json again. We convert the bytes again, and then we do Um a post instead So we actually post it again So we delete it, but then we post it with the updated coins Which is which is what which is our goal. It's took me so long to figure out It was actually kind of fun to make it But yeah, so we had to do that And then Um Yeah chunk transfer no idea what that does So we want to set the content type to application.json or application slash json But there is a website I've used where you can set like push data to it to your dns And basically there's different ways to do this hdm you can do html and stuff like that So I did a application.json since that's what we're using up here And then we ended off with request send or request dot send web requests Which basically updates it and we're good to go That was a butt load. Oh my gosh So I'm gonna put this in the updates really what I do for mine Is that I do post to lead board whenever someone completes a challenge or an event for crypto clickers So do it my advice is do it as Less often as possible because you're paying you're paying money for that t2a.micro Server that I bought that you bought or you didn't really buy yet really I haven't spent any money yet I'm sure that will come after a year but you will I mean You don't want you want to use as much data as possible Because if you're gonna have 100 000 users is it in posting every frame It's it's just not gonna look good your server will crash consistently You'll be spending a bucket load of money. So use this as Use it as least as black Use it as less as possible. Okay So we should be safe here You know, I just realized I might actually have to make a new one. You know It's okay. It's okay um I'm gonna come back to my old script. I'm gonna Re-pace time tokens this week Even though it's supposed to be coins, but since I don't want to make a new one. I'm using time tokens this week Okay, so it's the same thing just replace yours with coins Uh, I'm gonna make a button Actually, we can just make a button that post is it. So yeah, we're good there. So next we need to get our scores And this is this method right here unity web requests. Oh, I numerator great So what are you using here now? I think it's system.collections. Yes, it is Okay, so now we should be able to use i enumerators, which is required in order to forget for this to work um, so basically we're just we're just getting um, we're treating our data from the dns And we yield return. I don't I'm not really sure how to explain how this works, but there's an error It's gonna show an error and it's not gonna update Um, but basically we're converting this download handler right here That's basically our entire like what I showed you earlier. It's the entire website and then we convert it to json Actually, no, it's already in json So we're going to convert it from json into this wrapper method right here So we're basically converting it back into data right here And then this line right here is very important basically It takes this entire invent invent event data and it And it orders it from the highest number to the lowest. That's how we get that leaderboard ranking Okay, so that's how we get Let's see. What am I missing? Yeah, we need a leaderboard text So just put that up there So next Yeah, sorry if I'm not making any sense. This is I'm not very I'm not very good with this So paste is here So this right here Ah shoot, I did not copy copy at all Okay, so basically this star core chain get scores will run this get scores right here So another thing you should really run this once a minute or so Or make it once every five minutes But have a button that autumn that that people can click to update it with a max with a minimum time of one minute Just so you can save space or speed and stuff like that Um, so yeah, I'm just going to put an update for now just for testing Um, see I said the leaderboard text to blank Um create a counter setting at one and for each Why is this like this? I don't I don't know why it's doing that. That's supposed to Oh, right. There we go. Okay, so For each data inside this, um event leaderboard right here We're gonna grab We're gonna start with the first this is the first place So remember we're going from highest to lowest so it's already sorted right now So we're going to go to even data counter. We're going to put a dot We're going to put the username and we're going to put our coins right here And then we start a new line and we add to the counter and it continues And I believe we are done Um So yeah, um, okay We can test so this is where things can go wrong Okay, so this is our leaderboard text and our event leaderboard. We're going to save And we're going to cross our fingers and play So here's our leaderboard right now What I want to do real quick, I'm not going to save it because this is just a quick button thing I'm going to make so we're going to create a button right here Make sure you're not in play like I am I'm going to add my canvas to this go to leader B And post the leaderboard ready There there we are right there. So now if we post again see how it updates like that Instead of adding us twice. It just replaces it So I'm actually going to add this button like legit so we can see So canvas leader B post the leaderboard So just put that right here see how it saves and it always updates and Nothing can change it either Okay, so we're going to post again nothing happens Now let's make this time tokens 500 See how it updates now if you put one now we're in last place All right, so once again Make sure you Create an aws account make sure you create an instance how like I have like I did download putty download cyberduck You're using strappy You make sure you have your dns with you at all time Because you're gonna be using a lot make sure you don't show it with anyone else Um, yeah, I hope you guys enjoyed that tutorial is a lot shorter than I thought anyways have fun Which you got I'm wishing you guys luck if you guys have any questions I probably can't help you because I had someone else help So anyways a shout out to number engineer who made incremental unlimited Incremental unlimited Let's go here. I don't want to go to the reddit Where is the play store right here who made this the game so make sure you install the game on android only And yeah, thanks for watching download or donate donate to me on kofi download my um Download my Download my project on patreon if you're interested this project will not be included just because Um, you gotta watch the full video in order to understand what you're doing and just that's just you just gotta watch the full video Yeah, anyways, thanks for watching. Have a great day. Subscribe like the video. It's from vacation news. Peace You