testing one two three all right here we go chat up and I'm all set no idea how many people are in here yet and I have to my first thing order business here is to decide what to work on a lot of people in my viewer list but no idea how many of them are actually here I want to move these up as possibilities instead of being in the not right now because we've got enough going now that stuff like that becomes not implausible do check Twitter all right now I guess actually while I'm waiting for people to show we'll just run the game quote unquote game so one of the reasons why I mentioned that something like guns is feasible is because it would make it start to become more of a game as would you know putting in an object system and stuff like that and I should really smooth this stuff out and I probably want to turn off the third-person camera yeah smoothing the perspective would probably be good making your speed yeah that was an interesting thing making your speed be more plausible to the to a running speed given the scale look at how long it takes us to change I guess that's because this is sort of this is still the old flight model just with collision and forcing you to the ground but the actual physics is still the suit the pseudo flight model that I used for creating the demo so I probably should work on that too well yeah so let's get rid of the third-person camera put that on a toggle this is not how you want to in the long run like the long run the player object is an object and it's part of the regular object rendering and then you have to do a special check for is it this particular player because we have multiple players but for now and I have not started this is me this is my pre-stream work just cleaning up the old states we're not counting this is really starting yet I mean I'll still put in the archive but this isn't actually being rendered with the voxels right well see that's all a definitional thing all that a voxel is is you have pixels which are points in 2d space that we have as a color and we often treat them as little squares you have textuals that are very similar to pixels and you have voxels and so a voxel is just a bit of data in a 3d grid and the bit of data that I have here is some solidity and some texture info rather than a color but there is in many senses of the word it is still voxels however there is this whole contingent of people who are like voxel rendering is when you have a 3d grid that you're you know in some way dynamically visiting or for some people it has to be an iso contours generated from a 3d grid or something else so yes this is Minecraft style that's why I call it a block game obbg is open block building game because there are people who get pissy about their terminology in a pretty much not actually supported way for what terminology really means so now I want to support flying again so this is work physics move walkable is this still all allowing up and down angling I think it is so if not flying and we do that else anyway so hopefully that me and during rents answered your question and here we have the flying mode back and it seems to feel about the same as it did so you one of the things this does is it smooths the rotation although you can't tell because you're not controlling the rotation and now that I have a flying flag we put in a title for that want to keep it's more remote that's okay really like to be a control key but I don't know SDL to do that instantly so I'm not gonna bother right now so then where do I smooth the rotation pending views a impending view so if flying flying we want this craziness it's not flying and let's just instantly update it wait what you X how does any of this work because the thing it's this pending view X seems like it's a wrong sign though I'm not sure what let's just move that whole thing in and I think this just is that we'll see if that's right shake your head I actually kind of don't okay now my rotation is super spastic in the Z so it might have a some kind of weird feedback but it's also just not being scaled and I'm not clamping it X is the one that was too much but I can do this right so what that's exact in both you know my X and Y yeah this just needs to I need to actually track through what the math is actually doing because it's just feels terrible but maybe it's just because I haven't scaled it correctly where does this come from I stepped right by it okay it is just that so it should just be this so it's just a matter of getting these skill factors right it just feels there's something weird about it it's weirdly jittery maybe I need to do it a little smoothing mouse direction that doesn't seem like it makes sense it's interesting that my velocity doesn't actually follow that at all right right because that was the old flying physics yes where that was what I wanted it to do all right so that's enough cleanup of the old state so let's go ahead and tackle a new task so here's our task list is this is that font readable I might have put it back down I don't remember might have set my font size back down I think I had had to bump this up so when this game have shadows I think jazz is an important thing in a game well it's a trade-off since one of the things that I'm doing here is trying to push as many triangles as possible for geometric complexity so if I draw shadows I start having to actually draw all the triangles twice once for the shadow map and that's got a cut down on the the view radius on the other hand it's worth doing in the long run the open part of this is the idea that you could be steered towards different game designs and so you could choose to have that or not so that's really part of add better lighting that's down here so then the question of when is of course that there's no order for any of this thing there is no and this is doing it in software rather than real shadows there is no roadmap here like I have no idea what to do what order things are going to get done so fun is fine all right well if that's true and you're actually talking about the old font was fine then I'm gonna put it back and then if it's too small again tell me and I will bump it back up so obviously there's tons and tons of stuff to do here so maybe let's categorize so look prettier so lighting even the terrain stuff is dynamically set view distance kind of is better terrain other train fixtures better lighting performance did I did any of those things where any of those things actually performance this kind of is so let's call that performance even though it's not really features call this voxel features so this one is the actual voxel feature and this is a feature we need to make that very particularly useful and the system rendering and physics is part of that raycast is part of that use more block types is part of in some sense look prettier shoot guns is a gameplay feature networking is a Minecraft feature so we'll put it under voxel features just because it's not games not game specific it's not really gamey it's just part of having the basic yeah it's hard to it's you could argue it either way but I'll just put it there for now so and you know the reason shoot guns is there is just because that makes it a little more distinct from on craft even that that's not really that exciting a gameplay element so networking it's gonna take a long time but it's pretty important my friend Nick mining was just posting about Star Citizen and commenting that you really have to put networking in from the beginning if you don't put it in from the beginning trying to retrofit it it's a pain in the ass I think it won't be that bad if we don't put it in right away because I am always keeping in mind that it needs to be my networked so I have I try to avoid doing things in ways that will be a networkable except to the extent that I do just put things in all the time where I'm like well this can't be this way in the long run but that's how I'm doing it just for now so there are some just for now is in the engine right now that are not how it needs to work when it's networking which is part of why putting networking in even now will be a pain because yeah those things have to get fixed so user block create destroy is obviously pretty big for looking like a minecrafty thing and the thing about that is I pretty much have to do just save load of some kind for this because while I could do this all by itself without any just save load not only does it not persist from run to run but it doesn't persist if you just move away and come back you know I'd have to re keep that data in memory and rebuild from it and how much of that data can I keep in memory like that really wants to get flushed out to disk let's see so anyway so I have a whole lot of look pretty ears and I've been working on those little bit here and there as you've been going but they don't really matter to being a game smooth camera height when stepping up fix walking physics what do we have to fix in the walking physics we have to better handle losing don't stick to walls when falling a one block and then push smoothly along walls and then the physics or physics model should steer just a velocity momentum to the direction instead of Tony so those are the things I need to fix in what I was just when I was just walking around just now those were the things I saw that were pretty obvious and you know for this to feel any good like you know if I wanted to make an alpha that people could play with which you know wouldn't even really be an alpha because it's not going to be much really have to fix that it's not ready for anyone to try to even stand in that world without that something fix it feels terrible and you know game feel having your character feel good when you move your character around is actually pretty huge and it's you know it's not just professional it's like unplayable sort of when you don't do that right so that definitely have to get fixed before anyone else tries to touch this but but we've been working on that recently so I'd kind of like to do something else shooting guns really requires some objects out there and it's probably more interesting if you're already multiplayer I mean I can make NPCs that you should add but so that kind of implies that networking would have to get done before that I think networking is gonna be pretty boring though so I don't know I don't know if anyone in the chat has an opinion and I'm not gonna take your opinion necessarily but I'm interested in hearing your opinions if you have one let's see make sure the bottom of this is visible and check for memory leaks is of course just sort of maintenance quality are any of these really code quality camera height when stepping up is kind of part of the fix walking physics but yeah I'll go ahead and move that down even though that's can be seen as just looking prettier I'm exciting to see networking a million yeah well so the thing is the whole thing won't get implemented today but certainly take a stab at doing some of it I guess yeah why don't I go ahead and pretend I'm gonna do networking and see just gonna make a first pass and sort of think about what needs to get done and I'll talk about it and I'm just thinking to myself so now let me be clear oh I should probably this shirt color okay yeah I guess it's different enough from the black of the background I should probably be clear that I have not ever shipped a network game so I have even gotten a network system I don't think I've done a real-time network oh I did one that I get it the actual thing working I think it did I think I did a little toy thing at looking glass that didn't turn into a real game but so I'm not an expert at networking specifically so both in the low level like how I handle packets and in the high level like what the design for dealing with networking stuff there's not necessarily gonna be the ideal thing I'm gonna be feeling my way as I go I mean I've thought about it a lot I know some of the standard practices I've opinions about some of the standard practices but those opinions have not been tested against reality so so yeah so sure let's go ahead and start thinking about that so of course one of the things I have to do here is hopefully SDL has wrappers for the socket stuff I mean it's just sockets everywhere so but you know the syntax is a little different in windows from everywhere else so is this not a core not part of core SDL then I guess it's a separate thing and do I only get SDL one documentation probably so I need to get command shortcut that can get to all the time so where do I keep my SDL libs I don't remember options it's all the way back here now directories I have it in here app SDL to net why is it down multiple level I see on that okay so I already have it set up so hopefully I can just link it live except I want to add this to release as well and then I probably have to copy I have the how does it get the DLL do I have it on my path maybe I have it on my path just checking the why this Windows version this is Windows 7 I just have my have it set to Windows classic theme and I mainly turn off the Windows gradients and this probably needs to go into the fact notifications all right so I can do the low-level networking and then there's all the high-level networking questions so it's a few things to think about one is transmitting the map across so one of the goals here is that you're making a game with minimal building then the client can just generate the map itself without you having to send it across the net which reduces the load on the server a lot because now the server only has to keep track of enough stuff for physics it doesn't need to be able to send you the data you need to render out now if you're making a game that's Minecraft like where people are building lots of stuff all the time that's not viable and you will have to send all the data across the wire and that's fine and if you wanted to have something where you don't you're making some game where you don't want people cheating and exploring more of the map than they can legitimately then you don't want the the physics you don't want the clients to be able to generate it so you'd want to have a secret seed and the clients again wouldn't generate the data it would have to get sent to them you could do something where the map is the client can generate the map but that doesn't include info about you know in Minecraft terms it would be like where the village it villages are or where the strongholds are or things like that so you might have two separate seeds one for sort of the core map stuff and one for the sort of yeah NPC like you could call it like I think of it in terms of traditional game design you might think of it it's like a map seed and a mission seed where the mission is also procedurally generated all the stuff and that would allow you to keep that stuff secret from the player while still letting them while still offloading the the problem of generating now if you we're thinking about optimizing this in terms of like how much power you're using globally and you'd rather have the server generate it once and send it out to the clients probably rather than they have the clients regenerate it from scratch but I don't think that's a valuable way to look at it probably who knows how much power you're using transmitting that data having to compress it and all that stuff so it all depends on how efficient is to generate so anyway so those are some of the things to think about for that side of things as I said like we have to think about this from both the high level and the low level so I mean I just set up the low level SDL but the high level is sort of more interesting so that's all down to like what do we need to transmit and depending on the game you might also have a game that just has a fixed fixed map you know like you know a 2k by 2k map or something like that and that will be nice for physics like if you want to make a game where you're just simulating everything all the time everywhere you just close off the world to a fixed size and you know rock on and get rid of all the caches that I have currently for that kind of stuff so I want all those things to be supported and so I can while implementing a new feature like networking just pick one of those things and pick a sort of a simplified model so for example networking I could have no transmitting any map data across to start with and just rely on the clients to generate it so I'm independently and then eventually that stuff needs to get added so you know when then once you get to use your block create destroy hey you have to transmit to create destroy messages but also when you have somebody who joins late they need to find out about all the previous create destroys so so we'll see we'll see how that stuff goes eventually so what we need to do we now need to start thinking about everything in terms of client server where we didn't before so the biggest thing right now is that we are generating this data we're actually currently storing our physics representation in the mesh stuff and that's gonna be bad with a server split the server just wants to store the physics data independently but the player does still want to have the physics rep as well so I mean all that stuff subtracted as I've said before so probably this will be okay so what do we want to do we want let's get a document up let's just go ahead and say new text server dot c location so let's break this down server's responsibility rotative physics rotative physics does that have to do anything else at this point clients responsibility predictive physics generate to train rendering physics yes so this is one of the things I was gonna get out a second ago is that so we're gonna split the client that's the single player the single player needs to have both the server and the client running and I'd like them to not have to redundantly generate the train but that's gonna be even grosser this is something that you know you don't have to worry about when you just have a fixed map you can even if you want have two copies of the map loaded one for the server one for the client but I'm sure games don't I avoid doing that but here like the what you're responsible for in physics with the server is it wants physics data around all the NPCs and all the players and if it's if it's single player you know it's only only one player but those to be NPCs if you want to simulate NPCs you are far away on the map the server will need that meanwhile the client only needs the physics for what it's doing the predictive rendering which is our predictive physics for which is the stuff around the player so there's going to be this weird thing where they want different physics terrain so there's a few ways to approach that we can just not worry about it and have redundant copies of everything although I'm running in 32 bits so I memory pressure is actually kind of significant and we can try to get that stuff right or we can just kind of hack it and as you know my approach to these things is generally to hack it until I need it correct and generally do meshes render meshes pull user input now in the long run I think a server would have to be 64 bits or if it's only 32 bit it would probably have a limited player count and one of the things I'd like to do is push the player count a lot higher if you have a 64 bit server in other words make this fast fast and make this fast so jarring terrain I'm always going to be keeping in mind how do I keep the terrain generation efficient so that the server can do that but on the other hand it could be the case that you also like well the dynamically generated terrain the server has certain limits but if you just pick the 4,000 by 4,000 fixed terrain then that and the server 64 bits so it can just have it all loaded into memory all the time then the player counts go up significantly and that would be fine like if that's you know there's no specific problem I'm trying to solve here it's just one of those a lot of the times you hear about how something works and I have the reaction of it doesn't seem like it should be that hard sort of you know I don't know you know why does 32-bit Minecraft only render out a max distance of 256 when I'm in 32-bit rendering a max distance of 1200 well partly it's because it's Java and they're open gel capabilities in Java are limited but partly I think it's just not well implemented well designed well then there's the programming there is not as good as it could be and so you know that's there's always sometimes for me a goal of sort of maxing things out and so you know like I was talking before about the shadows like in trying to really push the triangle count as high as I can with an eye towards large view distance I hate to then sacrifice that triangle count back to doing shadows even though obviously with shadows it'll look a lot nicer but as always with OBBG the goals here are to be flexible to support many directions and so as I was saying like I'm comfortable with the idea that maybe the high player counts are only with a static world and that then boils down to just trying to make that part be fast you know and I have no idea what that involves I've not spent any time or thought about it but looking at what we've done so far the obvious things are that the collision detection needs to be made efficient the current walking physics does a lot of these probes and is that the best way to do it the old voxel engine that I did with this everything was actually in fixed point because you need to convert between fractional coordinates and voxel coordinates a lot you want to take some fractional coordinates say which voxels this is overlap and look them up in arrays and so it's a float to integer conversion which has in the past sometimes been slow and in VC6 is certainly still generates a bad code for doing that although you know eventually I'll be compiling this with other compilers that aren't as slow so I've already lost what my throat if I just went and looked at the chat and now I forgot what I was talking about so oh yeah the fixed point coordinates so that was how ZMC worked and which is the previous voxel engine that I was working on and I don't know that it's necessary to go that far but I could certainly go back and change all this to use fixed point if I felt like that was what it took to get high player counts I would make that change absolutely but hopefully it won't go that far I mean the advantage of fixed point is that as long as you're using floats you know floats only have 23 bits of precision I guess I have 24 there what 1 8 23 so because there's a implicit leading one bit they get the 24 bits of mantissa and what that means is as you get further from the origin you get less precision so you end up you know if you're at 6 5 5 3 6 that takes 16 bits to represent so you only get 8 bits here of precision so in the fixed point stuff that I was doing I have 32 bits and I would do that actually the same thing I would only get 8 bits of precision within the boxes but that let you get up to 2 to the 24 your your you know a single voxel coordinate could get actually it's plus or minus to the 23 I guess and we just before you hit the edge of the world and it would just have full precision out there so the thing you can do to if you the problem here is that you get more precision when you're close but as you get further away the precision goes down so you know once you're at 2 to the 20 now you only have so this is fractional this is that so now you only have you know 1 16th and you can see the guy who's walking to the Far Lands in Minecraft Kurt J. Mac on YouTube Far Lands or bust you can see how the world is very jittery because he's now gotten so far that he's into something like this area the 1 16th or 1 8th precision and you know the way to combat that if you're using floats is to use doubles so that you have way more precision far away from the origin but I don't want to use doubles everywhere for performance reasons and you can do stuff where you don't use doubles everywhere use doubles for the core coordinates and then you switch to a local reference frame and use floats and that may be how it has to get done in a long run but again since I only really care about high player counts on a small terrain a relatively limited terrain I could don't really care about how that works I guess that doesn't I have to write the code once I don't want to write the code twice once fast code for small terrain and once the slow code for large stream so never mind all right so these are all the things that these things have to do of the things that we're currently doing once we add more features which things have to get done where may change I don't know or maybe not maybe this is really all the server ever is really responsible for doing so what do we have to do so the the interesting part of networking is the communication between these two but I can't actually do that until I've actually split these and the splitting is what's going to take a while and is why I don't think this is going to be that interesting task is because I'm not going to actually get to that interesting communication stuff because I first need to split this they keep the single player clients split this and have them communicate in the single player client directly without using networking just to refactor it into what I can do the networking on so that may take a while so what do we do so how do we do this refactoring so one thing is that the current caches that I have need to be made explicit about whether they are the client cache or the server cache and as I was talking about the need to share that would be nice to share the generation and it would be nice to share the caches but I think that might be too hard so I think what I'm going to try to do is share the generation but not share the caches so the first step of that is to make the mesh building really separate from the rendering currently it's a little bit driven by the rendering I think get the next built mesh get from Q non-blocking no I guess that's okay that's already split out well okay so yeah so the thing right now is that the client and the server kind of need to be able to issue independent requests now for a single player they are going to share all the requests so they don't need to generate independent requests but eventually they will so do I need to actually fix something there I'm not sure basically the problem is like so they you get this return thing you get this data out of the queue and you need to do two different things with it you need to hand it to the client you need to hand it to the server and if those are separate threads that's gonna kind of be a pain the other thing is what do we do with this oh that's the built mesh that's the other thing that's not the genshark that's the mash although it has the physics data in it but that's sort of an accidental hack I don't know I want to that I want to rely on that in the networking but maybe I can so if there's no reason that the server and the client need to be separate threads in the single player in single player you can just sit here and alternate the server and the client and that may be cleaner you know it's not going to be an effect as an effective use of threads if of multi-core if there's useful work that could be doing independently but I think at least to start with that may be a cleaner way to go so so we start by refactoring the server and the client to be independent data structures they don't share any data directly but they are still called from the same thread so when we do this get next built mesh right here we can say client that server save and server physics data and the client can do all that stuff which implies that all the current data structures will be client data structures which I'm fine with so let's rename all the current data structures so I think that's called replace is my command for that yeah so let's mesh cash and why it's gonna become let's do the old quake thing of saying see underscore for client s underscore for server I don't know if that's a good idea or not remember I said early on I have no idea what's a good idea for this stuff or not I'd have a recursive version of this I guess I don't I guess I have at work I made a recursive version but I haven't done it here so all right it's fine that all the sources here for why replace mesh chunk mesh cash with mesh cash and then what other data structures do we have is yeah the thing is like this is a thing that is on token boundaries so I have to actually do all these variants by hand and of course it's gonna turn out that like the mesh chunk oh yeah the mesh chunk size I don't want to change it's only the cash darn it all right let's make sure it's still builds that I didn't accidentally rename anything to the same thing as something else okay good I'm curious to see what comment this word here or is the entire thing just talking about this other person's thing no okay nothing that I care about all right so these ones I have to change so that's mesh chunk cash x log 2 c underscore for why let's make sure it builds again because I'm so likely to type out something here I want to like that in mesh chunk x for world x come see you mesh chunk oh darn it chunk x for world x see mesh x what a terrible name right so that hopefully is all that stuff all right and cash radius maybe only appears in that file too yep all right so that's that stuff what else do we have that is client only have another we have the gen chunk cash but the gen chunk cash we'll say is part of so what we're going to do is we're going to have three conceptual processes here chunk generator the terrain generator and gen chunk cash generate gen chunks and currently it actually does the meshing as well mesh is this is no longer this guy's job and it's going to trigger generating the train for rendering but it's not going to actually do it we'll just leave that for now and these this does live in a separate thread where's these don't so this guy these guys can actually share data structures because they're in the same thread if I want this guy which means that whole thing I just did with making that mesh now it's got to have to be different so that's fine this guy makes its own data structures in its own and accesses from its own threads so as long as the other guy never touches gen shock which I think is true not sure if VC6 has an option to compile and see of course it does it dates from 1998 when C++ was becoming popular but certainly not as popular as it is today this is back when they actually took the C compiler seriously so yeah so now you see these are static to this file and this file is owned by what I called the terrain generation responsibility so we're going to say that this is the mesh builder process I need a better name for that server and clients are clients of this that's terrible our customers process so the server needs to maintain some kind of data structure how this can work the server just needs to keep physics data right now so let's just go ahead and the server define I could this for now log to though why did I call these what was the terminology for this gee I just cash acts okay I wish you said auto repeat better mark fine grains it's auto repeat is really not very good in the windows for programming so each data so then we have mesh chunk which has a fist chunk let's go ahead and just use mesh chunks for now mesh chunk server is cash what we're gonna do is we're gonna say static I'm just gonna call player X and clear why for now this is the idea is that the physics cash is gonna maintain data centered around the player so we need to know where the player is to know how to update the cash so then we say avoid server process mesh chunk if it's the player bounds update it so this in we really don't have any public functions yet do me the whole point of this is that for now it's just you know leave with the render so here we do we say s whatever that function was process so now the server and the client both are seeing all the measures you know now the server still needs to request them because it's gonna keep a different size cash from the client and it's gonna have to force it to build some that the client doesn't need which I mentioned before is a terrible idea because then we end up generating more chunks but whatever we'll do it that way and see what happens player again this is all kind of hackery because eventually the player the server needs to support multiple players obviously and this will all need to work in a different way but again we're just trying to refactor from what we've got to what we want so so MC chunk X so what do we do this the player player is there's a macro for that no because the this is a different set of macros so I shouldn't use this macro although it would actually work because it's not actually dependent on cash size it's only dependent on the chunk size which is always gonna be the same so I didn't actually need to climb up by this so we can actually use this so let's actually do that even though even though their name is wrong and we could catch this but we don't care all right so if let's say chunk X it's greater than equal to PCX minus S physics cash X over 2 so if it's in with it within the square around it then this is cash this is player cash player chunk chunk Y I just have to use words here otherwise it's gonna get too confusing fizz chunk fizz cash X equals MC chunk X and S physics cash X minus one we don't even check what's currently in that slot we're just gonna overwrite it up fizz and let's just chunk asterisk this cash chunk equals I just have this cash X MC chunk X equals actually we just copy it we don't care about a lot of the data that's in here but rather than bother to explicitly zero it out I'm not I'm actually gonna zero it out all I care about is the fizz cash the chunk in that well so let's be explicit about that since I know that's true it's only copy the data we care about might be more robust just copy it all because then in the future when I add stuff it'll go but I think this is all still kind of hacked so it's probably what a terrible line look at that totally makes no sense PC MC because of that PC MC is fizz cash MC and then chunk all right so there we go now we're keeping a little square around the player with physics data how big is that square it's eight chunks so four in each for a radius of four around the player and each of those chunks is 64 so four times 64 is 256 so we're giving a pretty good region around the player but not nowhere near as far as we render even though that could it's actually really getting called and if it is getting called what it's doing we never set the player XY so that part won't work and we're not telling it to rebuild that stuff so he doesn't actually manage to store data so let's add a visualizer so we can actually see that what so we could wait to visualize this I think we'll just store because it's small enough it's what's eight by eight which is what 256 numbers that's maybe kind of too big you can make it smaller but if that's a small bit there so update server feedback to feedback the job of this is to set some global variables that we can print out so what we're gonna do is store this data into this global variable what is that called is that the right name mesh chunk size x log 2 right and I only do now a left shifts on right shifts if it's a left shift I'm sticking with times plus minus player x so this is telling us where the cache where the thing is relative to the player assuming we ever call that function which we don't currently so let's add that real quick box will render just do it with the camera right here why what else do I need to do I need to display it but did I do everything I need to do here I think I did cash feedback where we print print percent 4d I'm not printing both x and y I don't even know what that's printing has this in the address so I was probably just garbage I don't know I guess I need to know if that's gonna be readable enough yeah my space character is not the same width as the digits and maybe the minus character isn't also which causes this stuff to not quite line up okay it's that's a little hard to tell what's going on so I think what I want to do is code this differently I want to use player x that's the same thing here I want to just measure my chunk offsets rather than measurement coordinate offsets because I don't get enough value out of using world coordinates for that okay now when I move around just when I cross a chunk boundary it'll update am I sending the player x player y I thought it was up there goes and now you can see I'm getting pretty far away it should always because I'm not actually telling it to generate these things so what it wants to do is always be sent around zero zero that's not working actually why is that wrong but that didn't matter why x y why is it letting it go why oh I see it's yeah yeah okay right so it goes from negative two negative two up to one comma one it's just centered weird so as I move around what should happen but is not coded yet is that it should always keep that zero zero somewhere on here keep the negative two negative two to one comma one range and instead they're getting arbitrary big and that's because I'm not actually generating the chunks that are in it so I need to actually generate those so we need to be able to request the chunks that are missing which we would do here but we don't want to request a chunk we've already requested and this gets into that whole giant mess that we already have in the current requester here where we sort the list and figure out which ones we already have requested and all that stuff and I don't remember does that code live here or does that code live inside the doesn't look particularly messy I don't remember where this code lives this is the single-threaded version no this is the single-threaded version what is oh no they're both part of the single-threaded version okay how does this work does anyone remember how does it decide which chunks it needs to do is there just a separate call somewhere request mesh generation yeah there is yeah so we iterate over all the chunks to build the consider list which is prioritized then we put that all into a list and then we tell the other side about that list and what we need to do here is we need to also throw in higher priority the physics mesh requests but they need to know about the meshes that are already requested and how do we do that we let the other side do that the other side says oh I know that mesh is already pending so request mesh generation see and this will be much simpler non-single player because the server and the cash and the client don't have to share this process so what are these called requested meshes so let's find requested mesh and then this is int needs mesh because the server only wants the fizz chunks does not actually want the meshes so for now consider mesh and we'll consider mesh is not a requested mesh I state equals state dot needs mesh is true mesh builder does this swap requested meshes where it processes them get chunk status alloc I have no idea what any of this code does anymore get junk status I understand but what would alloc mean get chunk status all right so we keep a whole separate cash I forgot about how this works right there's a cache of what ones are processing now the problem is that they we have two kinds of requests like if we requested a mesh for the physics we could have a request that says I don't need mesh geometry I just need physics data but then later after that gets requested we come back in with a new request which is now I need the mesh data oh I hate this stuff is any of this stuff for me I see a huge amount of conversation that's not for me to do prefix stuff like me because currently that conversation has gone far afield from OBBG which is fine I'm not complaining just so if we get two requests like this we kind of the easiest thing to do in one sense is to just serve them separately but the problem is that we have this whole mesh cache mesh chunk status cache all right and we can't really have two things because we assume that they're I guess I could make that have a third dimension I mean the obvious thing that you want to do is that if you get a needs mesh request and you already have a pending request that doesn't you mesh you want to replace it and that's just gross because it's all threaded summary of the current problem is we have a client and we have a server and in single player they are running together possibly on the same thread and share the mesh builder and the mesh builder builds terrain geometry and then takes that terrain geometry and builds meshes triangle meshes and builds physics data and when the client in the server are sharing it the server only needs the physics data and the client needs both the physics and the mesh data so it would be nice if when the physics when the server part requests a chunk and it only needs a physics it would be nice if the system did not go ahead and build the mesh data and then I have to throw it away because building the mesh data is inefficient and the server is doing the server requesting this physics data is making us do extra work beyond what the work we're currently doing in the single player version in the single player version under the model that I'm talking about the thing we'll be doing extra work so it would be nice if it didn't have to generate this throw data that gets thrown away but that now means there are two different kinds of mesh chunk requests ones that want triangle data and ones that don't and although it would not normally happen you could get in the situation where the server requests and the client both requests the same chunk one of them requests it with a server with mesh data and one request it not with mesh data and if they come in in a wrong order you would end up the two requests are currently I just added a flag to distinguish those requests but the way the system keeps track of pending requests it doesn't have any facility for dealing with that and there is no good way to make sure that to deal with what happens when you get two of those requests at the same time you don't currently ever get two requests at the same time because it sees that there is already a pending request and discards the new request but that is not the right thing to do anymore if the old request said I don't need triangle data and the new request says I do need triangle data then you can't just discard the new request so you need to discard the old request or replace it with the the other thing that's all possible it's just a pain because it's all threading gobbledygook that is the summary which is not short which is like exactly what you know back when I wrote this whole the this thing not work manager the swap requested meshes this was the big painful thing back when I was doing the threading and it is painful and people were like okay wait what is the problem why is this so hard and the problem takes a long time to explain because it is actually a hard problem and yes I jumped straight to the hard problem when I do these things because yeah I I mean even just making it so that the two requests are independent is itself a pain I'd have to change the way we track pending things to add a third dimension mesh status I would now need a third dimension of whether you requested the stuff or not and that would propagate all the way through everyone that uses this data structure you would need to know about that and why are these four by four oh right let's put stuff within and then if I were later to performance optimize it I would have to put all that stuff back so it seems kind of dumb to not just do it the correct way initially but it is not clear what the correct way to do is I can make the request happen at the same time the problem is that the server may decide it needs it this frame and the client doesn't decide it needs it some number of frames in the future it shouldn't this shouldn't ever actually happen the way things currently work but I don't I I need to handle like that specific case shouldn't actually come up when I'm just having them both process a square around the player but I don't want to put build in the system that's actually has that lurking bug when I later go and change how they what their policy for what they're keeping track of is how much stuff would I have to change if I just treat them a separate requests mesh status has to get a third dimension so let's see how many places use that so get chunk status needs to take another dimension get chunk status Alec needs to take another dimension that is debugging that's the only one for that get chunks status who uses that no way get chunks status is for mesh chunks wait that for Jen chunks I'm confused no that's for mesh chunks I don't understand what this code is doing what is a mesh chunk status oh right it tracks that right it caches that it has no it can't be a synchronizing updated but it caches it yeah okay right so this would have to look at the question mesh state that would come in get chunks as they're I guess that can be done it's going to involve changing like 10 or 20 places so this is only talking about right now I'm not wearing this is not a problem once you have a separate server in a typical line this is only for single player where the client and the server are running in the same process so yeah none of those questions actually have any bearing on the actual problem for what I know that's not a very productive or helpful comment there but you are misunderstanding what the actual problem is so yeah I guess I can just go ahead and it's sucky because I'm gonna have to back it out probably at some point and it's definitely not needed once you're actually multi-threading but let's just let it break where it breaks this one is my we shouldn't call it mesh should we because we're calling it a mesh junk and it's the meaning of mesh junk is overlaid that's part of why this is so terrible because this meaning is already so overlaid needs triangles let's call it it's triangle mesh is the intended meaning but let's just call it that for now bull needs triangles and this one has to actually check that yes I yes I have a job the problem is that I didn't want to do the good that I'm doing right now they seem to dumb way to do this we can be iterating through true and false by going from 0 to K would that be better to go false to less than equal to true what a terrible terrible terrible thing I'm just blindly assuming all this stuff is gonna be correct without me writing any more code no idea if that's true at least they all have the right variable handy and don't even know why I had that in there all right because I was gonna start generating right so now I need to actually generate those requests and let's just do static let's do this we're called requested meshes remember requested mesh return the number requested and we'll just request them all so the x coordinate we need is player cx minus s physics x or 2 plus plus j and end on state equals I think that's ignored and these triangles is false all right so function yeah that bull looks x me out to that's one of the several reasons why I didn't really want to do this this way all right so now request much generation we'll make that handle this what is max what is our where is rm is that so that's now message six chemics can why so you can see how this is has almost nothing to do with with actually doing multiplayer that was just why I said this was gonna be kind of boring is because I'm trying to solve all the these problems to do with having a separate server and client in the same thing efficiently like if they both just built their own terrain generation totally independently this wouldn't be a problem but of course to have them do that given that I already have all this this mesh builder system that was global would actually have been a pain all I guess I could have done it on its own thread I guess the server could have just built things directly instead of going through the mesh builder so that does that and that gives us M and then we say I equals and the state has to be RMS requested actually okay and let's see what I've forgotten to do okay so we'll catch feedback redefinition oh I have to make Harley close curly braces okay that built no idea what it does so now for now we're having them do things explicitly so okay it should not request the mesh if it already has it so this should be int Rx equals that that right I'm pretty sure that's how this works if we have the mesh chunk then we don't put it in consider mesh so yeah if we already have it we don't request it that makes sense so here we need to say if we already have it if server cache not server cache feedback anyway it was fizzcache so x-1 so if we already have it of course we need void init physics cache server function to rein this so it won't be valid fizzcache of jmi.chunk x equals i plus one it's never valid I gotta call that for our nits caches oh sorry it was that one s init fizzcache fizzchunk physics cache of course that was sarcastic of course in case you couldn't tell okay so then now we built it okay so now when we get in here we can say if fizzcache mc arrow chunk x does not equal rx or fizzcache mc chunk y does not equal ry then we need to generate the request all right so that all maybe works and then when we come back from here did I do this right m didn't do that mm mm and then here when it comes back we want if bm dot let's see those triangles then it's for the server I want that find my has triangle definition there we go so now we need to actually set has triangles and not generate the mesh data and all that stuff so that goes in mesh builder so mesh builder I think it's worker manager generate terrain so can start proc gen is that the only place we do that probably start proc gen let's see what that does proc gen in progress oh so right there's a separate cache for that right so that's this spot so if some chunks that wasn't valid and it became valid wait proc oh proc gen generate train so I know this is where we did okay gosh once we have 16 ball joints then we're ready to go so we say all that and then we say if rm meets triangles then we have to start another process that does all that stuff if rm does not need triangles however we're actually done so now we have to go see job build mesh see how it deals with cleanup at the end so here's it when it's done it adds to the cube built meshes let's look at how the cube meshes is dealt with get that's built mesh is that called directly by the thing I think it is it is right to get built mesh not a thing okay right so so we look at where that gets generated again add to cube built meshes out mesh MC has all this other data so the main thing is that it has a mesh chunk that got mallicked alright so basically what we want to do is instead of starting a new process here we want to do some small subset of this we get a mesh junk set the mesh junk coordinates need an out mesh no I just copy that did I delete it what after I copied it I guess I did out mesh these are gonna be null so I probably need to set them oh but it's gonna try to clean them up so yeah they probably do need to be once we're done we do that so where do we make the physics probably and generate mesh for chunks that build this junk okay does this actually use any of this data here I don't think it does so let's do that you see do I have a WC I think that got to find yeah what is T T is a task which gets built right here so it's just those okay RMX on chunks chunks is not the right name where did chunks come from go back to that thing again we're not very trained built mesh rather so where does chunks come from there is no chunks here right build fizz chunk oh that's just the chunk set right yeah chunk set okay so that's part of we just built this thing so MCS chunks at valid refers to MCST so that's T.cs don't know if I need to take the address of that yeah I do okay I think that's it's how much is zero and then we can release the chunks how much of MC is MC cue it these don't need to get freed you don't need to wake them that because that's part of the other thing probably crash build fizz chunk MC is allocated MC chunk is set chunks looks okay MC PC column does not why off X where do those get built are those separately allocated MC Alex fizz chunk this column should alloc itself gen chunk seems like it's valid I guess it's not though yeah that's pretty valid you see you see not valid J chunks but that's interesting this is forcing it to build chunks it doesn't need it's forcing it to build four by four and it only needs a two by two so that will be something to change eventually okay so we have a bunch of chunks those seem fine but when we went to access this which chunk is that chunks chunks of chunk J of I this looks bogus why is that bogus we tried to use T.cs which is MCS error CS let's just run again in case somehow that changes hey look it's probably reaching the same thing okay so it's clearly wrong because that table on the left has totally wrong data and I don't know if it's updating or not all right so not sure how this can be totally wrong like that let's go ahead and check what we've currently got and just to be on the safe side Twitter messages since I've been doing this for a while okay so we're getting the server cache callbacks here and the why coordinates don't seem right so am I like and they used to be right I mean they're all wrong but the Y coordinates are more clearly broken right they're all Y coordinates are all changing at the same time the X coordinates seem to mean ignoring one of the might be the same property right the X coordinates should only vary along the X and the Y coordinates should vary along the Y so though some kind of bug with the Y coordinates which seems odd because this code should or this code should mean it can only update one of the slots so I don't know how it can update the other slots okay so let's make sure this is still doing the right thing so J sub I sub J sub I chunk Y mice player why you know I think the feedback is valid is showing us what it's supposed to show so then the question is see even if I were requesting the wrong thing here which is certainly possible it shouldn't get written back into the cache the fact that the cache used to work makes this really odd of course the cache used to work being fed the old data I mean sort of pseudo work so what happens if we do this we're also not freeing it free we don't free it in this path because we store it it's not how it's called mesh chunk free free was it called set much chunk only lives in mesh builder which I should move out free mesh chunk oh is it not global it's global free mesh chunk what was the actual error undefined not in this I guess it's not in this I expect this to still yeah now I switched it back to the old thing and it should just do what the old thing does and there's a bug with the physics chunk request overwrote the render request even though it shouldn't have because they're supposed to be separate but it obviously caused some kind of problem that made the physics the render thing break and this looks like we may have to use the wrong chunk here for at least some of this because this stuff no longer meets correctly so that happened anywhere else yeah that's broken out there so we're broke the mesh chunk rendering the mesh chunk processing in other ways that looks wrong as well yeah so there's some kind of thread race condition and this could be an out of memory or this could be a bug are we not cleaning up our chunks correctly yeah we don't free the chunks probably no we do release gen chunks we did it right here worked our part out mesh we don't set that flag what was the flag I remember what the flag was oh has triangles that would probably be good to to do correctly so we don't accidentally receive a mesh that has no triangles and try to process that by the this thing so that should fix that is it still closing yeah there's still we're still having some kind of bug here with getting the wrong chunk data somehow or the wrong mesh is a wrong chunk data I think the mesh this looks like it's seamless seamlessly built from the wrong chunk data and now we're back to the physics being broken but now we can switch the physics call here back to where it was and see if that works for physics data no now this is still that busted data here in that the matrix that's being this way that the top left still totally busted so we're not requesting the right data and then somehow when we process it we're allowing it to be set wrong which I still don't understand how that could happen so I guess I should check here I hadn't even hit this code yet it was wrong oh wait so it doesn't seem to actually be updating are we requesting we are requesting we're requesting 16 meshes which is all the meshes around us because they're all wrong and then this is requesting a whole bunch more meshes whoops I meant to step in I think it's that initial data there is the initial data that said is always invalid that's what that is showing that's why those numbers are all the same yeah so like one of them got updated but the 1 2 3 4s should be 0 1 2 3 so that's just showing that invalid data state and which means this is never getting called with useful data looks like one is getting called when there's one getting updated the m dot mc chunk x well that isn't good why is that not correct mesh builder we're setting it here oh we set it from T let's uninitialized why do we initialize this task outside of this I don't know I don't think we need to I think this should go in here and then we can't accidentally reference it still working so is that world x or is that chunk x I thought was maybe chunk x except it's being multiplied here but why are these called C they're not chunks no okay those are world chords CX and CY are world chords and they're just RMX is unskilled so RMX and RMY must be so that should be right build this chunk maybe takes chunk words it takes world chords and doesn't actually ever use them anywhere it looks like see what we're getting here see why the same coordinate multiple times yeah it's just they only tried to generate four of them which you can see in the are the four corners of the matrix actually have the correct coordinates in them I guess what I should do to make this even easier to visualize is have this just store the chunk coordinates instead of all saying and relative to the player in the long run of saying relative to the player will make more sense but in the short term this will let me move around and see whether they update at all or not that was just me yeah so I'm not requesting the correct coordinates I'm only requesting two by two around the player it looks like there's cash X and Y there were a bunch of that so that's four this was 16 right let's look at what we requested yeah that looks plausible certainly it's requesting out to negative two so then the question is why when it generates them these coordinates were coming out wait I'm setting chunk coordinates in the requested mesh is that correct world coordinates hopefully this all works and we are correctly generating around the player so we'll see if this update as I move yeah okay correctly generating around the player some physics data we're doing it a little bit inefficiently because we generate the same data twice and the way it's generating is a lot more around it but finally got all that working how long did that take it's now 920 so I didn't start on this right away but it's been two hours since I started the stream almost so finally this there is room for now the physics to have a separate physics cache on the server and so we'll be able to run the physics separately on the server and the client and you may remember that the physics is all designed to pass stuff in and then calls gathered collision geometry and so in a traditional client server we would have the client and the server the implantation of gathered collision geometry would access different things but in single player where the client and the server on the same machine those are going to actually need to know which machine it is and maybe I'll have to pass into this some kind of context handle that's used for gathering the collision geometry and it will use that and of course initially the server and the client will just generate the exact same data all the time it won't actually be possible to see any deviation so I'll have to introduce some kind of intentional mismatch between the client and the server so that the authoritative server will override the client but I think I'm gonna stop there like I said didn't really actually get to any networking at all just started teasing apart some of the stuff that is combined probably the hardest individual thing here that need to be teased apart everything else is already more or less ready to go run separately so probably next time oh I forgot to put this up so probably next time I'll be able to actually start working on running the physics separately in the server and the client and I still won't do it networked it'll still just they'll just communicate directly but I'll go ahead and have them do it in a single player you don't really necessarily want to work that way you probably just want the server the client to just always use the servers thing but I'll go ahead and implement it that way explicitly before trying to actually send that data over the network just so that we can see that it's synchronized or see that they're doing the same thing so what are doing the same today stream refactor mesh generation so single player server all right so yeah that's it and I will look at questions in chat thanks for watching


Welcome to the dark side of enrichment analysis. The dark side is a new series in which I focus on major problems that are rarely talked about in bioinformatics, and I'm starting with gene-set enrichment analysis. If you're not already familiar with enrichment analysis, I strongly recommend that you go watch my introduction to the core concepts first. The first problem I want to focus on is confusing term names. If you do genontology enrichment, you will very often find that the most significant terms are high-level terms with very long names. Examples include G06796, phosphate-containing compound metabolic process, or G051171, regulation of nitrogen compound metabolic process. If you're a biologist, you would be excused for thinking that this might relate to processes like phosphate homoestasis or maybe nitrogen secretion. But if you take a look at the child terms, you will find terms like G06468, protein phosphorylation, and G06355, regulation of transcription, DNA templated. At this point alarm bells should ring. What you're looking at is protein kinases and transcription factors. In other words, you're looking at signal transduction. And you might ask yourself, is this even metabolism? Well, when you attach phosphates to proteins or form RNA, you are making and breaking bonds, which is the very definition of metabolism. So even if this is a confusing usage of the term, but may lead to misinterpretation, it is correct. The second problem is overlapping pathways. If you do pathway enrichment, you very often find surprising pathways showing up as enriched, often involved in various diseases that you're not studying. So it's not rare at all that you see asthma, inflammatory bowel disease, malaria, and tuberculosis all coming up as pathways enriched in the same study, which may seem absurd. It turns out the explanation is that the same genes can be in many pathways and are thus not at all pathway specific. This is why if you do enrichment analysis on a list with many cytokines on it, you shouldn't be surprised to see enrichment for pathways involved in every disease that has to do with inflammation. The last problem I want to talk about today is study bias. We live in a world where disease genes attract more funding, and highly expressed genes are simply easier to study. For this reason, both of these classes of genes are going to have more annotations than the average gene. If your gene list thus comes from either a disease study or was produced using techniques like transcriptomics or photomics that have abundance biases, you will have the same biases on your list as you have in the database that you're using for doing enrichment analysis. Consequently, you will get inflated significance estimates and possibly even artificial enrichment of terms that are not enriched if you did the analysis properly. You can address this by using a custom background, but this is only a partial solution. I want to end on a positive note. It's not all bad. Yes, there are confusing term names. Yes, there are overlapping pathways in terms of genes. But there's a very simple solution to both problems, and that is that you should simply look at the underlying genes. If you look at which genes are responsible for the enrichment of a given term, it's immediately obvious if it's protein kinases or transcription factors. Disease bias is more difficult to deal with, but when it comes to abundance bias in transcriptomics and protomics, there are correction methods that can deal with this, and these are implemented in tools like, for example, a Go tool. The most important tool, though, is critical thinking. You should always look at your results and think very critically about why the terms you're seeing come up as significant. That's all I have to say about enrichment analysis today. If you want to learn more about how to actually carry out enrichment analysis, I suggest you take a look at this presentation. Thanks for your attention.