what's going on with Plex it's plenty for direct play um setting up WireGuard for VPN setup that was really useful to do that before I put it on my main server because I don't really want to disrupt my main server because that breaks stuff for the rest of the network for the rest of my house and when I break stuff like that people do notice in my house it's unpopular um sometimes it's nice to have uh somewhere you can set up like QMK building or some other development maybe Rust or something else before you put it on your main development box because you may not want to disrupt whatever you're doing on that development box or make sure that it's not going to disrupt what you're doing and that is that's turned out to be pretty helpful um there's also been some instances where having a dev node I can just spin up and throw away it's surprisingly useful besides the services like why is my particular version not building correctly I can put on a specific version of GCC I can put on specific versions of make I can do whatever debugging I may need to do that of my build environment itself without disrupting other stuff that is working on my main dev node and that's saved me in valuable in invaluable amount of time learning from what I did so my biggest lesson I think is that the CPU's mattered much more to my use cases than memory like I haven't had any real problems with memory other than I don't think I really have it's really a matter of the CPU and not having enough cores most of the stuff I'm testing isn't isn't really memory bound it's usually CPU or disk bound and that said the Raspberry Pi is for the hardware I have it was comparable once I put the put the storage in a denadicated actual USB 3 hanging off of the Raspberry Pi not trying to serve off of the SD card those just aren't fast enough once it was in its own once it was off a uh off the USB 3 the performance between when I had the storage on like one of the nodes I started with to Raspberry Pi is almost unnoticeable I really didn't make much difference at all especially with the new Raspberry 4s Raspberry 5s I expect to be even better as I was mentioning memory just hasn't been as critical for what I'm doing I'm not doing really memory intensive testing in there now if you are doing something different that is something to consider maybe you do want to do GPU sharing which Colossack does support in some instances to do LM AI development or something like that or you're doing large databases or you're trying to do memory caching or something like that where you do need more memory for doing some of that stuff it's perfectly reasonable spend the money then on getting more memory instead of more CPU I just found that for most of what I do 90 99 percent of what I what I've done it's really been limited by the CPU course not the memory something else like your mind which is going to be kind of repetitive because I'm sure all of us have heard this before is back up your stuff whether that's your template that you spent all that effort building in the cloud back that up somewhere outside the cloud so that if that way you can treat your cluster as what you can reinstall it you don't care you just reload the templates it's not a big deal also keeping any like isos or additional templates that you're not actually using outside your cloud somewhere else we have more storage especially when you're limited with the storage space it's like if you're building this in as cheap as you can and you don't have that two terabyte drive sitting extra unused keep those out of the cloud too once you once you build your base template with from an iso don't keep the iso on your cloud once you build a template and then you update it to the newest release of that version don't keep the old versions around it just it just waste space you're not going to go back unless you have a specific reason you're just not going to go back as many of you are going to be familiar with this if you worked with the cloud at all keeping your templates your machine images your images as small as possible helps a lot and it's not just in how much storage space it takes up it speeds up your instance launching a lot the bigger your template you're starting from that has to be copied and that slows stuff down and the smaller the cluster you're running on the more noticeable that will be there any questions so far comments sure here he's that one first then i'll get to you so cloud stack seems to be like us so i use proxmox at home yes so i kind of do the same workflows with the proxmox i just whip up a virtual machine install bunch of server on it and do that testing that i do in there proxmox can do a lot of the same stuff clausack does the the advantage of cloud stack over proxmox is that clausack can scale bigger than proxmox can so i can have a full cloud like it's not like it's not just vm's like there's image management there's auto scaling of some sort you can do there's migration between hosts there's um there's now kubernetes managed kubernetes clusters since 418 i haven't run it i haven't run that particular stuff because i haven't had a need for it yet though that's what i'm gonna look into next okay but it's a full cloud so proxmox can meet a lot of these got a lot of these things especially with some of the nice um i think they have some more base interfaces now yeah yeah no there's there's definitely overlap with some of these is just i don't think you're gonna run a thousand instances on yeah no proxmox and you're not gonna do migration between zones you might not because you can set up multiple zones kind of yeah the clausack uses some different terminology but you basically have different regions and different zones and you can migrate between them and there's a bunch of that kind of stuff does that answer the question yeah um we'll actually have a second part here um so i assume so the the raspberry pi is i assume are running containers and then your x86 machine that can run vm's so it depends so on the raspberry pi management storage node it's just running the os it's not running containers i'm not running any virtualization or anything on that it's just running java most of clausack's actually in java so that um that's pretty easy to deal with it's just the compute knows that do stuff uh cpu definitely matters and a ton of applications but there are some applications that are extremely memory intensive i encountered while i was building a uh self-hosted map and routing server oh yeah oh routing 100 lives in the ram so oh yeah especially perform it if you want to map for all of north america you need like a hundred gigs of ram and there's no two ways about it so uh some applications read the docs guys i'm curious is how you hooked up the storage to the raspberry pi is are using like a usb hard drive interface okay yeah i can talk some more about the storage on the raspberry pi um so what i did is i have an ssd that's sitting in um the most performant external device i could find for raspberry for a 3o usb 3o uh you're not going to get the best performance but for the hardware i was using to begin with it really isn't any different it's it's it's been more limited because the hardware i started with i mean you saw that i spent less than 500 dollars shipped and that hardware compared to the raspberry pi is not much performance difference now if you're optimizing your hardware for storage more you can get better performance than the raspberry pi can provide but for if you're doing this on the cheap for as cheap as you can it may not be as big of a difference that you think it was it sure turned out to be the case once i started doing it i expected to be a problem and it was a problem when i ran the storage off of the sd card but once i moved it to its own usb 3 it wasn't really a problem which kind of surprised me um and someone was mentioning about the networking yes depending on the type of stuff you're doing it will make a big difference memory may be a much more limiting factor than anything else uh particularly with routing or if you're doing some uh software defined networking which cost act totally supports but i would expect that those routers you'd need to have configured and running are gonna take quite a bit more memory than what i may have set up but you have to take that in consideration when you're what you want to optimize your quad for and if you're billing a cheap you can always start with the last five dollars or much less if you have the hardware laying around and upgrade later get started try it see what happens um so one question is did i spend less than eight of us um at the time i did this uh definitely after about five months i was definitely way less than i would have spent for a similar set of instances it's a little harder to tell now because um some of the stuff i did in clausack you can now do cheaper in adbs without actually having instances so that changes some of the calculations but a rough idea was with what i did on average is probably about 50 to 70 dollars a month um so it takes around 10 months for my investment assuming like i'm actually spending 50 dollars in instances um advantages you don't have to worry about shutdown termination to save money you don't have to worry about any extra costs for storing amis they're effectively free i already bought the storage so i could have any custom eyes for as long as i wanted it didn't really make any difference in my cost we're in almost all the clouds there's some costs for storage um and after it's been running for so long really the main cost on my stuff is electricity especially being sdg and ease region i can see some of you shaking your head some assuming you know what that means um but yeah electricity is by far the biggest driver cost right now for me and also to be aware there's there's some testing you just i couldn't do in the cloud like if i was going to try to test plex for one example um the smallest dvd i have is 3.4 gigs to move enough of those into the cloud to be useful it's just yeah it's just not going to happen which isn't and the cost for that kind of storage is just way too high where i could reasonably test it yeah i didn't have maybe two or three dvds in a blu-ray but it was enough to test and say yes this is something i want to put on my network yes i want to start managing it so it was it was useful and ended up saving me money um some final thoughts one thing that will take a surprising amount of bandwidth especially if you're installing multiple different linux distributions is packaging i would strongly suggest something to limit your bandwidth or you will notice some issues um i don't know how many of you are running any sort of package caching like uh apcash or ng that's what i've used since i'm mostly a debby and ubuntu guy um but that is that makes a really big difference on how fast turnaround you can have on spinning up your templates and getting into a reasonably recent set of whatever the recent security updates have been um rpms have similar solutions you can configure apcash or ng to the cache rpms but yeah i would strongly suggest that you uh you have some sort of package caching it'll make a big difference it makes installing instances even vm is probably like your your proxmox having something there that you like i baked it into my template that it points at my local apcash so that when i spin it up i can run an update and it's going to update in maybe a minute or two instead of like however long it takes to download all those updates um as i mentioned before keep templates and isos and whatnot um outside your cloud you don't need once you build up a template from your iso doing a maybe a base install because you can get to the console in clausac there is support for that um you don't really need to keep the iso around just keep the template depending on what you're doing if you're finding storage is becoming a problem first of all don't use the cheapest hardware you can whether it's a raspberry pi or what i did um that will definitely make a big improvement also dedicating a storage for a switch for some of that storage can make a big improvement if you if you think you're gonna need a lot of like you're gonna move a lot of data you're gonna be doing something that's very disc intensive for your type of use case start looking at what some of the other options are other than nfs or maybe splitting up your secondary and primary server your primary storage so that your primary storage is on the fastest possible way or maybe you use local storage which will make it harder to migrate instances but you can you can specify clausac to use local storage for vms it just slows down your migrations a lot um the vlan tagging and trunking uh i'm now looking into that not for my home cluster but for a different one um definitely takes a lot more more advanced networking knowledge than i have but having at least some idea on that can make it much easier to talk to your network engineer who's configuring your switch if you have that ability um but it's also not unreasonable to buy sysco or whatever i don't know what people are buying that are full day or three that can do vlan vlands because there's a lot of power to be had there um and as mentioned previously is that clausac does support some sdn some software defined networking um that seems to be somewhat in flux not that clausac isn't gonna support it but there seems to be some talk of resurrecting some dead sdn projects now that vmware has been bought by broadcom and has changed licensing models there's some doubt for some of the sdn stuff that they've surprised previously to be continue to be available um so i just kind of if if you're interested in that kind of stuff keep your eyes open because there's definitely there's been some discussion on the list at least one or two mess one or two threads about um people talking about that they're trying to get a group together to resurrect i can't remember what the sdn tool was they're talking about but they're talking about resurrecting that so that would be useful if you're looking for that um it can also be really helpful to have an ssl cert for your clausac on my home networking it's my own network but i still try to keep everything encrypted because if someone gets in it makes it that much easier for them to get elsewhere and if it's encrypted it makes it much less likely um and more likely that i will see something that looks wrong if it's harder i did that by pulling uh let's encrypt cert and generally you can't pull certs for privately routed dns but there's no reason if you have your own dns name or your own domain there's no reason you can't set up either uh a wildcard cert of some sort that you can get from let's encrypt you do have to do some dns authentication but it's you can automate that with answell if you spend enough time or salt stack or whatever a chef or whatever you want to um but i did that and that that definitely helped um there are some addition or maybe even setting up uh an acme server for testing maybe you want to try doing this acme server with ssl sign certs and automate pushing certs on to instances that's definitely something doable with this um as i mentioned i think in passing earlier as a 418 clausac has added a managed kubernetes cluster that also supports the cluster api and i've seen someone live demo this out of presentation i was very impressed that they were able to do that where you basically just have your your base kubernetes cluster somewhere you tell it where you want to deploy and it'll just deploy and and automatically set up the whole cluster in clausac i have not looked at that yet because i just haven't had enough need need but in 419 there's some additional improvements with that just came out the end of february um that looks pretty good it's a fully managed cluster it supports capi it supports one of the other apis i think that's common in kubernetes i i haven't looked at kubernetes enough to speak to it but yeah it's there um and as promised here's some of the links you can reach me at that email either one of those um the presentation will live at that google docs address if you want take a picture as many of you already are the clausac roles um those are the ones i'm currently using for my own private cloud and for what i'm doing for work it is not you don't take it as a really good example of ansible some of that ansible i wrote in there is very early from what i've learned now and i just haven't gone back and rewritten it um there's also been some changes with how you should be doing some stuff in there namely your modules should boot fully the full name of the module some of you know the ansible there's stuff like that that's still in there um but yeah you're welcome to look at that um and then as promised the lack rack page if you're thinking about rack hardware and you're trying to figure out how you're going to put it mounted somewhere you want to do it in cheap those lack tables look really good for that and unless people have more questions i think that's it hello hello perfect hello everybody can you hear me perfect so let's get started um today i'm going to talk about a journey of a home-based personal cloud project because this is a very personal project so i would like to start from the start uh it was in 2007 when i was in high school and we were on a weekend with my dad and we were bored and we were looking for something to do and i've heard about this little thing called ubuntu and i decided to there were an event at paris that was called ubuntu party so we decided to go and it was a full day of conference around free software and it changed my mind i to this day i knew what i wanted to do then later later that year i'm the dream of the of the moment was to um i became true we went road tripping to the united states and um this was my very first trip overseas as you can see this is me we can change in 17 years and 17 years later here we are here um at the same place in las angeles um i would like to share my experience about um running a project entirely on free and opposite software without based on the idea that we don't need to depend on cloud provider to hold our own data from the bottom of my heart i would like to thanks the organizer for selecting my talk because this means a lot to me so who am i now my name is julien rio and i'm an open source dba you can find my previous talks on this website and if you want to see the slide and click on the links you can already open it they are available and you can follow me on mastodon today we are going to talk about why the context of the project and the history the timeline we will talk about infrastructure data management uh alerting observability automation and what's next and i will give you a little takeaway so why why running a home-based personal cloud storage why on earth because i want i never i've lost data in the past and i want it to never occur again uh and i also want to control my data uh as you can uh heard i'm french so i don't want to provide my data uh anywhere in the world i want to keep it with me and i also wanted to learn new stuff so a lot of solutions have um implemented maybe are already solved by uh one tool or something i just wanted to try some tools one by one and have some fun a little bit about the history it was in 2013 2013 uh i was living in an apartment and my data was stored on usb drives classic but the usb drives are uh hard to find maybe they are lying around somewhere and we used ntfs because at the time we had some microsoft windows uh laptops uh that were unable to open like xt 304 uh file systems we had to physically plug and unplug the disk and eject uh the disk before uh um before the end or something bad happened so i decided to look for some nas the network storage like this type of nas i didn't bought this this one but for the illustration so i used my desktop uh pc to create a nas uh it was on my home office with some some basher's because of windows and but there is a big problem every time i upgraded this the server uh the client were unable to to reach the shares it was a big problem for me and in 2015 i got a new job at a major cloud provider in europe ironic huh but this gives me the um advantage to have a discount price on disks which is not the case anymore they don't provide us disks anymore because you know safety reasons but at the time it was um available so i decided at work we also use open gfs so i discovered at that time open gfs more on that later that were was uh able to create nfs and cifs shares and i use gnu linux and uh on servers and desktop at work so it brings me back to 2007 when i discovered the open source world so i decided to create a small storage with all this hardware i could buy at this discount price so it must be silent and small because i live in an apartment at the time it has to to have the same design as the nas i've shown before so very small like a synology you know this this brand i bought three uh hard drive of 40 rubites it was well pretty cheap at the time used drive so they were four years old when i i plugged them but it's okay they work and i decided to buy new stuff new hardware like intel nut motherboard and pc i read card because there were not enough sata port on the motherboard and i decided to try free bsd because on free bsd open gfs is support is built in but i made a mistake when i bought the motherboard it wasn't on the right format i um there was mini etx and micro itx they are supposed to be small but they are not the same so it ended up like this i had the motherboard outside of the case the monitor is my uh my tv and it sees on um where we watch tv it's not on my office so it can give you the vibe at the time of what i was able to do anyway i've set up some things on frankenstein pc and uh okay let's start to copy some data and a big sound noise started to ring inside the room later i found out that it was because of the temperature temperature when it was higher enough there is a buzzer inside the case that were uh causing this this sound so anyway uh i have something i can't use so let's buy new hardware i decided to buy something i know so a classic atx tower with new hard drive smaller ones to reduce the price because i don't need that much of storage so i decided to and i've already bought some parts before and still use free bsd and in 2018 i had a baby so i had to put the computer away of my home office to create a room for the baby and i didn't have any time more to do this stuff and one year later i got a house a new house my house with more more space noise is not an issue anymore because we have a basement and a secure basement behind like a regular house and i have a third pc an old storage um because i reused my main computer as a storage it was my very first pc i've mounted in 2028 uh 2008 it only had three uh hard drives of one terabyte three is a good number you will see later away but i already had them somewhere but this one had some issues um for the installation the usb stick didn't work so i had to burn a cd rom on the latest version of free bsd at that time that could and whereas causing um a lua er you can click on the link the bug uh was open i tried free bsd 11 same error uh after two two or three cds been burned i decided to fall back to dbm and it worked but after that uh and running for some time the pc uh freeze 3d so i had to hard reboot it from time to time so i decided to fully replace that later on and upgrade it too with more hard drive a few years later so quick recap today i have three servers um they are not named uh chronologically because uh of all this history uh they are pretty much the same capacity except the first one that is a very big one but it's enough and i decided to use dbm only because at the end of the presentation you will see that i use automation and it was hard for me to manage two operating systems at the same time i had to choose and dbm has a pretty nice support of uh open zfs but i had computers everywhere what should i do with them um i need to find somewhere to put them and to be useful because i don't need free pc uh to run uh storage so the idea is to put one computer at my dad house and my parents in law house and at my house i have a distributed storage system as you can see we have one server in France and two servers in Belgium all in different houses all in the basement a little word about the clients because now i'm not the only one using this um this uh solution this infrastructure there are my parents so i decided to convert the windows uh uh clients the windows uh computers they had with something very similar uh i decided to use kubuntu which is ubuntu with kde interface because it looks like windows and so they are not very lost and in the end they only use the browser so it doesn't matter for them but it was better for me to do the maintenance and i decided to use uh nfs network file system it's easy to set up with a full uh linux environment or even bsd environment it's easy to maintain and it doesn't break when you upgrade um so the idea is to mount a remote directory but locally on your client so you can copy file without worrying about local storage but it's not that user friendly especially for the parents so i decided to and my dad in in reality has a new uh pc with linux he kept the old windows pc and he kind of have both system at the same time to make the switch eventually one day so i had to find a solution even for windows that's why i use i choose c file just i just picked this one because it's written in python uh not in php and at the time i was installing um directly on the host without docker uh system or something like that but right now i use docker so i could use any solution out there but it turns out c file is pretty it's okay it works uh it's user friendly there is a drive client you can mount uh as a drive uh there is a web UI so you can even share links on uh on a UI on a web link if you are not uh inside your house you can share something to the family for example and it keeps filing things there are different levels of caching but uh you can read the docs because i'm not an expert in different modes sorry one big issue is connectivity how can we connect all three hosts together because i want in the end to copy the data from one host to another first the static ip address you should know that in belgium to have a static ip address on a major isp it costs 30 per month only for one static ip address it's horribly expensive i don't want to spend 30 euros per month so that's not an option and isp modem settings uh i have three houses with different isp settings if we can take only mine uh it allows only few protocols by default you have to do port mapping and you even have to do to request the isp to lower the security level and it work at my apartment but not at my house so not an option what can we do i decided to use open vpn uh it's a virtual private network solution a vpn with a client server model uh it can authenticate with certificates using encryption like tls and it allows clients to communicate with with each other and it can even assign static ip address to the clients or the storage server and here is the settings you have the topology which you declare a subnet uh you allow client to client you can define the static ip address and the the pool okay but i can't rely to you i had to use something external to configure this vpn so i have to rent uh a vps somewhere uh in that case it's at my own company but i don't have any uh discount on that but i could use no uh every vps uh uh in the world uh the cheapest one could work you only need a network and a static ip address of course and the the network is encrypted so you don't care the account sniff or something and the certificate authority is at my house so it's pretty secure and it goes like this um all the network flows uh transit via the vps so you should uh select a location that is close to your client that's why i use a vps in France because it's the closest location i could choose so it it was working now the servers can communicate with each other and for that we can configure ssh to remote uh control them so from my house i can control everything now let's talk about data management as i said multiple time before i choose open dfs for zetabyte file system uh zetabyte is sent for uh the number of grain of sand on earth but this file system can handle more bytes than that so it gives you the idea of the powerful system of dfs it can manage your disk create some redundancy with red you can create multiple file systems um you can create snapshots for uh implementing a backup solution and the snapshots are basically free it's a copy on the right system so there is no impact as i can have seen with lvm for example when you take your snapshot with lvm there are back in the days there were a lot of io operation with dfs there is no basic little to no impact and with those snapshots you can do replication cloning and even rollback on the data there are built-in feature with uh the fs like compression uh it's transparent and encryption and it's production really even on linux uh at my work every backups are on open dfs it's like 600 terabytes of data and it's for databases so yeah production really even on linux but here we are at home no so it could be easily okay so red in the fs it's more like a red z uh there are multiple levels of red you can uh do the standard red but i choose the red z with three disks you have a parity disk and you can lose up to one disk so if you lose one disk you can replace it and re re silver the data it's the fs can handle that for you and then uh you can list uh the pool here i have like six terabytes of data available on uh on three disks and a little less fire and a half thanks to the parity and i use pretty much 50 percent snapshot included and i care i keep 10 years of snapshot and this project is less than 10 years but i won't reach the maximum maybe in a few years but uh right now and i can enable the compression so for uh binary data it's not that important but if you have documents for example like uh text files or uh um tesis or something like this the it can be compressed uh very well and you can create file systems so you can create for example for my example i've created one file system per person so everybody has its own uh space but you can do whatever you want i even have one file system for uh one software for example and you can mount this file system directly using the fs itself and you can create snapshots so the data for example if i need to access the data in the middle i can clone the snapshot on a main point and directly access and even modify the data without touching the snapshot it's pretty magical and you can replicate the snapshot you can replicate the first snapshot by piping through ssh and you can do incremental uh send uh with the previous snapshot as a base uh snapshot but as you can see it could be a little bit complicated at the end so i used the snapshot management um system called sanoid which is open source uh it can take snapshot for you run pre and post snapshot scripts if you for example run a database and you have to do some more pre operation before uh the fun fact about this feature is uh i contributed myself to implement this feature because when i discover this uh solution this sanoid solution i decided to use on in um on my home lab and it was so better than what we used before at my company that we decided to use it at our company and it can prune uh so remove uh older snapshot all based on policies and you have monitoring capacity you can check if your zfs pools are okay or the lc or if you run out of capacity the configuration is like a nini file uh you have here two templates you can say the number of snapshots based on the howl the the the frequency like howly snapshots daily monthly yearsly and if you want to create snapshots and prune snapshots based on on that and i use this uh i keep 10 years of snapshot with um one years uh in in monthly snapshot and one 31 so one month every for every day snapshot and i have a template archive for managing the replicated snapshot so they can't create other snapshots where they shouldn't so you have two templates and uh i deploy the one i want where i want and the policies you use the template defined before so you can use one template or another with that and by default sanoid comes under debian with um system d services service that is called with a timer and this timer is like a crown job um and the service is like a one shot uh command so here is by default it's it by default it's cheaper like this and there is a timer that runs every 15 minutes it doesn't mean that a snapshot will be created in uh 15 minutes only the configuration will be evaluated at 15 minutes so it can run all the times only needed snapshot will be created based on the the date and time and if we play with some timers we can see the next time the command will be executed and um sanoid comes with another tool that is useful it's syncoid so there are two binaries in sanoid uh repository syncoid is like r sync but for uh cfs snapshots you can see like this uh it can resume on interruption because you know uh network can fail and it can control the bandwidth uh i have a pretty small a pretty low bandwidth at home and i don't want to uh make some noise uh and um interact the work when i'm working or uh slow down the tv for the kids or something even if it's bad but you can control the bandwidth especially the upload bandwidth so the usage it's like r sync pretty much the same same option not not the same option but the same principle i've added some a list of command to this script and with the same um idea as seeing a sanoid i've created a system to service and a timer so every configuration is available here if you want to on my website if you want to check this out later and one nice feature here is the um randomized delay a second for example i don't want my house to overlap with some other house so i prefer to delay the starting and with a random value so if the timer are uh capable of this can do this so it goes like this uh the local client clients connect to the local storage and the storage overnight send the snapshot to the two other storage be aware you should not do multi master or multi primaries replication um it's only to the two nodes there is no circle here um the colors don't overlap it's like a circle if you take the big picture but if you zoom only one host will send to the two others and sanoid is able to to check the health of your system and the capacity so now i need to be aware when something goes wrong so let's implement some other thing i decided to use the good old magios it can be old and ugly but it's light and it works it um it can be configured with a simple configuration file we have a simple infrastructure it's okay uh it has a web UI if we want to for example uh put some down times because we are doing some stuff we don't want to be alerted like a real uh enterprise um or or not if you don't want to use the web UI it's okay and there are lots of plugins and welcome to pilot it's like a pilot but in french um it's a raspberry pi uh a small host i had some raspberry pi uh somewhere so i decided to take one to take one and to install it to have my own magios at home magios has multiple components uh it can declare host host groups to regroup multiple hosts and services to link services to host groups and notifications so the pilot is now about able to run check commands to different hosts and for that um there are here multiple host groups and there are monitoring servers for the the monitoring server itself there are storage servers for the old group and one host here with the name so we can declare our host it's a host name and IP address simple we can define the host groups comma separated list of hosts and we can define multiple service commands the one i use there are three commands i use one is checking to ensure the host is alive the check an nrp which uh remote uh plugin executor for magios and check htgp for the web services uh there are multiple services states okay warning critical and non you can define um you can based on the services uh level you you can define the alerting uh later if you want to be alerted for a warning for example or only critical and the service configuration goes like this we have a host group name so we attach the service to host groups and we name the service of course and we have one command check an rpu with some argument here check the fs snapshot to tell the nrp uh local agent on the host okay execute me this command and uh send me back your result and the nrp agent is deployed on every storage server and it runs the command um on the host and for the notifications um a big fan of telegram i use it every day so i decided to implement a notification system for magios it's open source you you can click you you will uh go on a repo and it can be configured you can uh have it you it's written in python and it uses ginga for the template so you can basically put whatever you want into the template and you have notifications like this so here i i have problems on my snapshot one snapshot is too old so it means that the the overnight transfer has failed so i can check it out but i'm not on call i will check it out later when i have time it's home base project i don't want to to be uh a slave or my infrastructure i'm already one at work i don't want to have work at home and a recovery notification same and we have the web ui but there one issue the web ui is available only locally how can we make it available to the world for example if i'm not in my house i would like to access the web ui for that i've set up nginx on the vps that run the um open vpn server and the nginx acts as a reverse proxy so it terminates the ssl uh connection and communicate via the vpn over http which is insecure but the vpn itself is secure and it's only monitoring system we uh and it's also already included so the web ui goes like this everything is green but it's not always the time that's why we have set up an i have set up an infrastructure like this okay we have the live system alerts but what if we want to know the evolution of the system what do you want if we want to put some observability in them so i want to know the disk space evolution it's good to know the network stability because i had some stability issues on my over my network and i wanted to know where is the uh the the original the issue and the most important here was the temperature like the buzzer before i want this to happen again and the power consumption because my dad want to save money and it's kind of scary to let the computer run 24 7 so he wants to know how much we cost so let's let's do this for that i use what's called the tig tig stack it may not be the most latest uh technology here but at that time that was pretty common so tig is for telegraph inflex db and grafana telegraph is a plugin that collects metrics locally and sends them to a data store uh a matrix data store so it gathers some inputs like the cpu like disk aio uh and all those stuff um and write to some outputs here the inflex db inflex db is like uh a real time uh data analytics database okay all the solutions are in the open and the last one is grafana is like a tool that allows you to create dashboards and visualize your data and my dashboard looks like looks like this i have the disk evolution i have let's say the the capacity the fragmentation all the zfs stuff here uh i have some disk temperature the age of the disk eight years i should i should uh watch them but they don't fail and the different colors is because i have reinstalled the server at the time a little overview of the infrastructure of the observer vt1 um i decided to use my monitoring server to install inflex db and order stack but at some point the sd card wasn't crashing so much so i decided to use a real server that i also use for other projects um so now i have real ssds and but i don't have so much metrics but i don't want my host to reinstall my host every year for once in a while so i have deployed a new server that runs docker image images of inflex db and grafana and i also use the same technique to access the a web interface with a reverse proxy with a nginx so we have a lot of things going through the vps unfortunately if i had static ip address and a way to to do something at home i won't require and i will be 100 free from a cloud provider so docker image are available um but in the end if i would like to start from today i would replace this uh metrics collection with a prometheus based um solution uh i will learn new things and this is also the goal of of this project and then sensors i want to know the temperature the humidity and even the noise of where the storage are are are placed of my basement and for that i decided to use arduino uh it's powered by usb so you have just to plug it on your computer do some electric stuff with pretty ship sensor it costs it's very cheap i don't know the exact cost but it's way cheaper than all the stuff that i've already bought before and it works with a breadboard and some electric cables so the architecture is like this you have a breadboard here some cables you you can plug them and it goes through they go through the sensors and you will have to put some code into the board run it and gather the data so the software there is an IDE for Arduino when you can write code and push what's called the sketch you can upload it to the board and let it run forever so here is the code you can initialize some values for humidity temperature and sound and if you want to have more um sensors you can update the code you have a setup function for example the sensor are okay after two seconds of wait before the values are not uh not okay so you can wait here wait wait here initialize some stuff then you have a main loop so the delay is more in the main loop main loop here you you define the initial initialization part the main loop is the loop that runs all the time and again gather with the module some uh matrix and format the matrix with the format you want but there is a problem with that um if you use one process it's okay but as soon as you use two or more process to access the serial in interface you will have uh access issues so for that you should use multiplexing and it kind of it's more in the you you often see the mosquito part the the broker when you have independent systems writing to one centralized system but here the the concurrency is only on Nagios from one part and telegraph on the other part so we have to use a multiplexing system so we I use the MQTT broker called Mosquito I've created a tool that's called Serial to MQTT to gather the value and send them to this little queue and I use the check that was already existing check MQTT and the telegraph plugin already has a a consumer of MQTT so diagram goes like this there is the sensor going to Serial to MQTT to Mosquito and Nagios and the matrix could read from the queue and have the same data for doing what they are supposed to do so the graph goes like this as you can see the temperature is okay the threshold is like 25 degrees I don't know how much is infinite sorry and you can search 25 degrees it's pretty warm but it's okay the buzzer sound was triggered at 29 so way over so it's pretty cool in my basement even in the summer it's one year of data and we can see the summer is here and the winter is here so we are we are okay what about humidity oh humidity is pretty high like 90 percent but we don't care it doesn't affect the system there is no the hardware looks pretty good it's pretty new and with the temperature even in summer the humidity is not a problem not at least not at home where I live and the noise what about the noise I can't even know what does that mean in reality the sensor is activated when you tap on it you have spikes only when you tap on it if you have ambient noise you won't even detect it so I was confused I decided not to look at it because the the noise is not a problem anymore in the basement and what about power consumption how much will it cost for those who don't know the Belgium price of electricity has spiked over the last years it was like four times the price we used we had before the covid but it's it's two times now but it's pretty expensive and those are the price of Belgium here it's the fourth top expensive given to this statistics and France is over here so we have not cheap electricity first I decided to want to use the wattmeter you can plug it on the wall and see how many watts do you consume but how to parse the data because they are cheap but you unless you put a camera and analyze with AI I don't know but it's pretty hard to implement so I decided to use an ups the ups is little is literally pricey more than 100 bucks 150 bucks but this is a project I want to to to be happy so I decided to buy them and to work with them with something called apc ups demon which is a demon for this kind of ups there is a telegraph plug-in of course and the graph and a dashboard is already available and the nice little things is now I can save my servers from power outage that was not the first purpose but it's nice to have now and the dashboard is here you can see the evolution of the power consumption thanks to the battery and it costs only seven dollars a year even for an expensive country as Belgium seven dollars per year not even the price of a cable so in real life there are some photos um this is the afl tower but the server is in Belgium but just because I'm from Paris but I live in Belgium now and you have this is my fridge and this is the wall so there is a little space here and it's cool it's okay it doesn't take so much space and you have this Frankenstein electricity thing over there but it works generally the kids don't go here so it doesn't fail or they can go just before on the fridge and they won't touch it so it's okay it's the basement they don't usually go to the basement and at my dad house it's the same principle and I'm my in-laws house same the fridge the wall okay now uh we'll try to go a little faster uh let's talk about automation because failure we already have saved our data it's okay but some failures can happen you can have micro sd called that fail or flood of fire in the house oh it can happen so for that my workflow of diplomats goes like this I inside the operating system install and configure the software and restore the data or for a new server it's optional and we will focus here on install and configure the data the software and for that I use uncivil which is an open source um uh simple it automation system is and by simple it's really simple it's written in yaml but not the same yaml as you have in Kubernetes it's really simple yaml and the concept you have an inventory the inventory here is a group of hosts and uh because I don't have a lot of hosts I use static file and a playbook playbook uh uses some roles uh that groups some tasks and tasks are modules plus uh arguments so the inventory example is like this you have a name and eventually an IP address because I didn't I don't have set up a DNS server uh I just put the IP address because it's static over the VPN it won't change even if I move from anywhere in the world the IP address will stay the same as soon as you have an internet connection and my playbook looks like this you have a site dot yaml which is the best practice when you have to configure an infrastructure and I have multiple playbooks to uh configure multiple parts of my infrastructure I have storage server but I also have other server that I want to cover here I decided to write my own roles like be like the the previous talk just for fun same as you so I decided to use a common role uh for storage as the fs role an open vpn role and inside the playbook you can have optional um roles for example the you can target some subhost into your inventory and in this example I deploy on easy nfs server on the storage one because I'm the only gig that use the in nfs server at home so I want it to be deployed only on this host and a role uh is like this you you have uh your default variables uh your handlers to act eventually on services like restart some services if needed like you have your task and your templates some modules I use the uh I use the built-in modules of uncivil like apt to install packages like files to manage files services and templates templates to create the files based on a ginga ginga templates with variables loops and so I use a template to manage my sync with uh destination script as I've shown before and this is the template it's a ginga template you have like brackets here for uh every data set in the main data set I do a double loop with destinations and I do a little echo for the log and I call the the same command uh as I've shown before but template it and it creates a file shortened for the the the talk but there are multiple lines and I just need to configure variables to tell the the list of data sets and the destination and uncivil will do the rest for me without any copy past issue and for upgrades I use not a role but a task I've created a task apt upgrade that calls a module apt to upgrade and I run it with a cli uh uncivil playbook you can run for the whole infrastructure sit dot email uh or the upgrade and you can use this uh as a base uh building blocks for the rest of your potential infrastructure like at work but at home so what's next I should open source my uncivil code base but right now I count because I've put some secrets in there so I don't want to share my secrets so I have to remove my secrets my variables and put that online so I will I will do that very shortly uh I promise uh because it could be a good example of what I manage uh how to so there is something that uh wanted to it's automate certificate management for renew my certificate for the VPNs uh I have monitoring on the date but I don't renew them with your automation I do it by hand uh I probably should use the FS encryption because okay your house is safe but you can be broken by some teeth so but I'm not sure they will pick my server first but but we don't we don't know so encryption is you should encrypt by default and use primitives for metrics I should probably forward some logs because it's good to have logs at one uh location instead of going on every host and the more important is to handle the mobile phones because today we take a lot of pictures for example with our mobile phones and right now the solution is not pretty well designed for that the client for C file is not very working well so I should fix uh this issue okay so the takeaways uh self-hosting is not that hard as you can see a three server or three house um even if I've done it through the years it wasn't my free time with kids so it's okay you can do this and I I'm not burnout so you can take your time but it's not that hard uh but if you really want something fast you should consider using a built-in solution like trunas there are a lot of feature uh that I've shown that probably are handled by trunas and free an episode software is awesome this is my philosophy I try to contribute as much as you can and you even have a postgres hoodie so this is in my blood and enjoy what you are doing because if you don't what's the point I would like to thank you for being here uh it's uh uh you are a lot here so it's mean a lot to me and uh thank you if you have questions maybe just a little one or two question because I've been running late so yeah yes yeah the question is uh do you have a hot swap disk and am I able to detect which disk has died um the answer is I have hot swappable hot swappable disk but uh it's easier for me to take down the the host to manage the host to see the serial and even uh list the serial identifier and uh and do it like this because we are at home not at uh at data center we can do whatever we want yeah thank you uh what's my uh you this usage this is a question I have um it wasn't uh shown it's like more or less two terabytes sorry uh for logs only logs no logs are uh rotated so I don't keep them uh I'm sorry uh I'm out of time but if you want to ask me a question I will be around I will stay here for the rest of the day so you can ask me anything thank you beautiful yes all right hold it yes no I uh I tell people the same thing all the time so I treat it like an ice cream do you mind shutting the door all right welcome everyone uh I'm here and I'm gonna talk about running an open source hacker space my name is Tracy Homer uh by day I work for the software freedom conservancy which is a non-profit uh dedicated to software advocacy and uh open source open technology and we have a booth in the uh other room if you want to come visit us after this um but in my free time I help administrate our local maker space so Knox Makers is located in Knoxville Tennessee it is the southeast's biggest and oldest maker space we have 350 members we've been in existence for about 13 years and it's like your classic hacker space model so it's all community run we are an actual non-profit organization but we're all volunteer run we are fully funded my member dues um we don't like take any like corporate sponsorships and we are also fully open source every class that we teach every uh software that we have there is uh open as open as we can and I'm gonna talk a little bit about how we make those decisions what kind of tools we have and what software we run and then like some of the challenges and some of the places where we're not we're not quite there yet so we have a set out to have some core values which is what we envision will help Knox Makers you know keep its core solid and what we use to make decisions about our organization um you can see on here that open source is not listed actually because we feel like it fits under several different of our values um I think that people get into open source technology a lot because of the non-commercialism and privacy and it certainly does fit under there but we think that it fits a lot better under opportunity because it allows people the opportunity to run the software no matter if they have a windows machine mac or linux it allows them to have not pay licensing fees because the worst would be if they came to the space they paid their dues and then they could only use the software at the space because they didn't have the money to get the licensing fees at home you're not getting ads thrown at you all the time and it also gives people the opportunity to learn very deeply about the software they're using which is an example of we have here which is our cnc router which is in process so if you've ever tried to purchase a cnc router they're quite expensive so we're home building ours which is another aspect of opportunity it's giving us the financial opportunity as an organization to save us a lot of money by not purchasing one and having it straight out of the box but it also allows for like cross disciplines so we have a lot of people that are into woodworking not necessarily into software but they see this become this getting built and they're interested and they're like well how is it run how do I get my designs onto it how is it programmed and they can learn about the software they can learn about the stuff that we're putting into it and also when you I think that when you learn an open source software you tend to learn more than just like press this button make it go so when you do go to a different software even if it is proprietary you know more about the inner workings of it and like how it runs and not just like press this press this press this magic your stuff comes out beautifully so there's a lot of opportunities for that at Knox makers and I'm going to go through all the different areas that we have so this is sort of like our how we look at different softwares and different tools when we come across something that we want in our shop right so green makes sense fully open open hardware open software great for everybody red proprietary we don't we're just going to say no we're not going to teach it we're not going to spend money on it we're going to avoid it at all costs we don't let our members teach on it I mean our members are allowed to use whatever software they want in their own time and on their own projects but what we have in the space we're going to keep it fully open but then there's the yellow part some things you just can't get open hardware for like a CNC embroidery machine which I'll talk about later there just is not any open ones but you can access it via Linux or via plug-in or like put a USB in the side of it so as long as we can find a way to access it without downloading their web app or something like that then we feel that it's it's not the best but it's okay and we'll allow it into the space we'll allow it as long as we teach the open way of doing things right but then the problem comes well not the problem but like the challenges are is like how do we make this accessible to our members so we have like I said we have 350 members of all varying levels of tech experience all varying levels of other tooling experience some of them don't have a computer at all some of them like programmed in COBOL at one point some of them don't care about open source some of them you know only have a pine phone because they don't want to like use a proprietary phone so how do we make this accessible to everybody and really that is a really difficult problem because we want the people that have the SolidWorks membership or some other proprietary CAD software to be able to come in and still use their designs so we try and have different applications we have we have a web app that you can like load software into that will like take SVGs no matter where it comes from things like that so and asking those questions and going with our members trying to make it accessible to everyone is really at the key not just open software but open to people's different experiences it's taken a lot of thought and a lot of work on how to do that and I guess I will go more into that in the specific tooling so here is a map of our shop in the different areas that we have some of our areas don't have any software involved at all like machining or blacksmithing they barely have electricity like they're not going to have any software questions but some of ours are just fully based in software and we have plenty of members that are you know programmers or computer nerds as well as there's not necessarily a zone for that but you know we have people that are into coding and network things like that so the first zone that I'm going to talk about is 3d printing so 3d printing it feels like they they're very open prusa is like pretty popular it's a very open software and hardware you can build your own if you buy a prusa you can buy a kit you can get the plans and like print out an entire machine yourself so it's accessible that way and we also have had other we've had maker bots in the past we've had different different types of 3d printers we have prusa slicer loaded on our computers there so that anyone that brings in their own designs can just load it into the computer and then all of the the slicer is like set up to with the configuration for our 3d printers so they can design it on their own we also teach classes in free CAD we have thought about teaching classes in open scad but no one has really gotten it together because it's you know it's really difficult to put a class and a talk together like that but 3d printing I feel like is a very open discipline like even if you have a closed printer there's plenty of CAD programs there's plenty of design programs and slicing programs where you can make have open software and it's available to a lot of people it's a little bit different with resin printers so this resin printer in the picture here isn't open hardware prusa does have a resin printer it's a little bit pricey so we don't have one someone donated this one to us and we were kind of iffy on it because it has like that windows web app in order to like load your design onto it but what we found was prusa also has a resin printer slicer so and a python script or some script that you can take and use the the prusa slicer and put it into their proprietary slicer and then it will put in your open design and you never have to worry about the windows app there's still some issues because resin printing is smelly and chemically so we're still working on like getting that onboarded but that is the reason that we accepted that donation because we could access it without having to download their proprietary app software we also recently got a five head prusa xl prusa xl which is just really this slide is to show that off because it's pretty impressive we want to grant to get that printer so they're not always they're not they're not very cheap but it's really cool and i'm excited to like expand our 3d printing zone with these new printers electronics we don't really have any um machines that have software attached to them we're working on building a pcb mill that uses gerbil but it's not up yet we do however teach classes in kicad we teach electronic theory classes that we post all of the class notes on github we teach a lot of arduino classes in raspberry pi so there's a lot of open areas in microelectronics as well i've actually heard a couple talks already today that mention arduinos or raspberry pies or seen them like all over here so we also have noticed that a lot of people come in and they want to learn basic soldering so we've designed some soldering kits and sell those for fundraising and also have those schematics on our github as well just to make the learning accessible even if the and the software even if there's no software attached we still have um all of those plans accessible to everyone fiber arts is an example of our as more of a yellow a yellow area like um there's no there's not a lot of open hardware um in the fiber arts section but we found ways to make it work so we have a silhouette machine which is proprietary and has their own like little app uh that we don't like to talk about but they also have an inkscape extension so all of the computers at nox makers has that inkscape extension we teach classes on how to use the inkscape extension we don't teach classes on how to use the other uh the other part of it so then that allows people to use uh the silhouette machine without ever having to download that other stuff and actually i think that silhouette is silhouette and cricket are some examples of how um we are able to evangelize fos a little bit because recently they like shut down your thing your you know you load your design into their software and then they're like okay well now you can only have it if you pay a subscription fee so what you said is a good example about why you shouldn't use why you shouldn't use these types of software because you know don't know when they're gonna turn around and uh you know change the terms on you so you own all your projects at nox makers we also have a cnc embroidery machine so again there's no open hardware embroidery machines in existence that we know of if you know of any please let me know so when we were shopping for one we wanted to find one that would take um a usb that you could load your designs in with a usb because there is another inkstip extension called ink stitch um that outputs several different formats a lot of them are usable with different embroidery machines so the fiber arts woman was tasked with find find an embroidery machine that can get these types of extensions that has a usb option and um she found one and then she also found one that had like local uh sewing machine repair support so that was obviously the one she chose but ink stitch is an excellent extension it's really easy to use i made this uh pattern in about five minutes this morning you just load in a vector you uh click go and it will make all the stitches for you and you can load that into your embroidery machine so when someone takes the class we go through how to use the ink stitch extension uh we give everyone a usb that they can keep and leave leave their designs on and show them how to load that into the machine arts and crafts we have a couple different uh machines there this is our large format vinyl cutter so if you don't want to use the silhouette at all or if you have like something bigger you can use the uh vinyl cutter the stickers i don't know if you can see the stickers on my laptop but i cut them out on the vinyl cutter um that also has an inks inks tape uh it's not you don't even need an extension for it you just use the plot function and it will print right out of the box i think that this is like an off-brand vinyl cutter it doesn't have any uh like apps with it or anything like that but you can just plug a usb into your laptop press the plot function and it will cut out your uh vinyl for you something that we thought about uh for a very long time was a poster printer we had one that was given to us and it was used it was it was not working great when it was given to us so we sort of fixed it up and we used it for a while and then it started not working really well at all but it was used enough that we thought well it'd be a good enough investment that we're going to buy a poster printer and printers are notoriously uh terrible for not being open at all so when we were searching for a poster printer uh our main thing that is that we wanted one that was already in the cups database so we know that someone else had used it and someone else had done some testing and at least it had been successful for someone else using a linux computer uh the really sad bit is that when we got it it and plugged it in and we're like setting it up and everything it didn't work right away and we're all makers so we're like trying to press the buttons and we're like trying to fix it and we ended up having to call the helpline and the helpline wouldn't help us because we hadn't registered the serial number with their organization and it was like so heartbreaking so we had to have a repair tech come and fix it for us I mean that's just really wrong it was very sad but now it finally does work and you can use it with any of the machines at the space even if you run an obscure form of linux. Laser cutting is our most open area our first laser cutter that we had was a small co2 laser that was built by our members and then we decided that we wanted to upgrade to one with a much larger cutting bed and we decided that the most economical way to do that was to buy a laser from China you can see in the background of the picture and then we ripped out the brain and put in our own guts that were all open hardware all of that is posted on our wiki all the instructions on how we did all of that and what software we used for that so we have our own custom safety on there if you open the door while the laser is running the laser will shut off if you hit the emergency stop button obviously it will shut off if you turn off the air filtration system it will shut off I'm sorry if you turn off the water cooling it will shut off but when you turn the laser on the air filtration automatically comes on so you know we're building in these safety features to make it accessible and easy for members to use even if they've never used a laser cutter before or they're not like a tool a person that uses tools very often but it uses linux cnc to with the g code to get the gco2 the laser cutter linux cnc is a pretty old and mature project it's used with lots of different cnc machines mills, lathes, robot arms basically anything that uses gcode to move something you can load into linux cnc and again with this we added our own features to it so on the right the this side of the screen you can see the bed up and bed down button so people can just like press click this button and it will move the bed up for different things we can also we also set a bounding box so if you click on the bounding box it will outline your old project so you can you can show where your project is on your material and things like that to make it easier for members to use but how do you get your design from inkscape to gcode that is a tricky bit and inkscape has a plugin called something something to gcode but it's not very intuitive so it just it's sort of you have to use certain colors you can't change the power settings we're not you know it's a little bit difficult if you're not used to like techy stuff so we have a custom web app that our tech director developed and that will you upload your svg into it and you can see that all the different colors in my svg are on the right side of the screen and i can choose which ones i want to engrave what power i want to engrave it as which ones i want to cut which order i want to do them in and actually it's been upgraded since then you can actually load in a raster it will automatically recognize that it's a png and we'll go to the raster settings so you can set different raster settings we also have a custom extension that we built for inkscape that will do a lot of the common things that people want to do with the laser so hatch fill which is these hash marks to like color something in with the laser or make boxes or make text into paths something like that so that you can it's very approachable even if you've never used inkscape before you can go in and you can output something on the laser fairly easily we recently also got a fiber laser which the difference is a co2 laser will do wood and organic materials a fiber laser will do will engrave and cut thin metals it will engrave metals and different materials like that there is not we use meerkat for that which is just now being developed for the fiber laser like i don't think that there's any other maker spaces that use it for fiber lasers i'm not sure if there's any other people that use it for makers for fiber lasers it was originally developed for for co2 lasers which is a totally different like movement system so with a co2 laser the entire gantry moves back and forth and cuts out your material with a fiber laser it has a mirror that just moves and engraves your material that way so meerkat started developing this for fiber lasers our tech director was very in contact with them when we were working on this project and trying to debug it and try and make it usable for the common person who's not like a laser expert and eventually we have gotten it to work for us and it's been really good so far he's even thinking about moving our co2 laser onto meerkat in the future or his own personal laser we've loaded materials fiber lasers are very picky about the materials like what settings you use so we have our own materials loaded in there we sell like little coins or little steel plates that you can engrave on uh so we preloaded all those materials and then we're working with our members to like if you put a new material in and you find good settings share with us and we'll share with everybody else so it's like opening up all that as well so those are our tools internally we also use a lot of uh open stuff we use um wiki media for our wiki page we use next cloud we use wordpress for our website you know these are all pretty common basic things uh we also have computers available for members to use uh that run debbie in this one we try and this is a very boring picture a screenshot of the desktop of our computers but as you can see like all the favorites are at the bottom so if you're very used to using windows or mac you've never touched a linux machine before you're afraid of linux it's so technical it's scary well it's not really you can see all the software that you want is listed at the bottom of the screen you can just click on what you need and it's right there for you we also have different scripts that will update them all when we find that uh something that um if there's like a package that's not running properly you can run a little uh script and it will update all the laptops and uh so that they're all equal and um completely ready for members to use for whatever and those just live at the space and people can come in so if they design something at home they can bring it on a usb or if they bring their own laptops in and they don't have inkscape or they haven't downloaded it yet they don't want to try it yet they can try it out on the space laptops so it works out really well they're very helpful especially if you like forget your computer at home or something there's always one there we also use pf sense for our firewall and network in the space uh we do have cameras in the space despite what we said about being having privacy as a value it was really important to us being open 24 7 uh and people potentially being there by themselves and maybe hurting themselves horribly we thought that having cameras was important uh we use zone minder with like a motion sensor on it uh we do not uh publish it to the internet we don't use like we definitely don't use proprietary software that's going to like you know sell your face to like some corporation or something like that uh we actually have really strict policies about uh who can look at the cameras and when there has to be a board vote and it has to be a good reason it can't just be like was so-and-so in the shop yesterday I want to tell them something like no it has to be like something catastrophic um and it's like sort of a pain to look so we don't really want to it's mostly because someone leaves the door unlocked which is really bad we don't we definitely don't want to leave the door unlocked but if someone does you can see it in the background we see the last person that left and then give them a tourist phone call but uh it happens really rarely I think 11 times since we put the cameras in place years ago is the how many times we've had to like log in and look at them uh and they automatically like wipe after I don't know a month or something maybe less than that so we don't store that data we don't keep it we uh uh don't give it out to anybody for uh member management we also have another custom built member management system which we keep our member data in we try not to collect any member data if we don't have to uh we don't collect people's income or gender or addresses or anything about them it's basically contact information when did you join a little bit about yourself we do collect uh we do ask people submit their photographs because 350 people is really difficult to keep track of so it's helpful to have your photograph there but we don't share that with other members um but this is uh this is the web interface to it uh which I've been told I'm the only one that uses it because I'm the membership person so I'm the only one who logs in and makes changes to that we also have a api so it like has different scripts that run that adds you to like the mailing list automatically or checks to see if you um you can run um sorry I've made some shell scripts that will like tell me different things about members uh so we can like have a count of how many people we have at a certain time or how many people are on probationary membership or something like that uh for our internal chat server we use matter most we've used that for a lot of years already um it's for members only and it interacts with our hacks db as well so there's a special channel that just run bots where you can uh have a little slash command and you can say who was that person named tracy again I don't remember her so you can type in people slash people tracy and then the all the members named tracy will pop up and you can find out their picture so you can see if you're looking at the right person how long they've been a member um a little bit about them we also have uh different slash codes for teachers of classes so we don't have to let them have access to the website and all the people that have purchased tickets and all their information they can just type slash events the number of their event and it will list the people that have signed up for their class so we can restrict that data access to them without giving them like admin privileges on the website it's really helpful this is how we mostly interact with it um it also uh will run daily and check to see if new people have joined or new people have signed up for matter most and stuff like that so it will um we also have bots that will check our machines that will ping them once an hour and tell us if something is down or if someone has submitted a trouble ticket uh it will also pop up in this uh bot channel as well um we do live streaming once a week so once a week we have a community night a public night where people can come and show off their projects show what they're working on ask for help on stuff and it's streamed to the internet so we use obs to like control the camera versus uh screen type thing and we stream to big blue button as well so we uh we enabled big blue button back in 2020 when everyone started using video conferencing software and we went and did our show and shares on big blue button for about a year um and now we just stream now for people who aren't available if they can't come out on a Tuesday night for whatever reason they can still watch it on from home using big blue button and we also still use it for our board meetings because it makes it easier for members to come if they don't have to leave their house it makes it easier for me to go if I don't have to leave my house as well so it's worked really well for us um it's I'm a big fan of big blue button especially uh so there's some of things that we uh could improve on uh it's difficult to change after three 13 years and 350 members if you set something in place in the very beginning it's really hard to change it uh so that's that's my excuse it's not that we don't care it's just that there's maybe no better option or we haven't found the time after putting out all the other fires to uh switch over to something new uh the first one is we use github for our code repository I don't know if you're familiar with uh software freedom conservancies opinion of github but it is not good don't tell the table up there that I said that uh but you can come talk to me about it I'd rather that we move to something less corporate but it's it's really difficult it's a huge social site github I mean it's like everybody knows where to go if you say you're on github everybody knows what that is so moving off there is not is not risen to the top it's not a huge priority and it's I feel like that's okay right now we also use gmail for our email again is just what we started when we started the organization it where there was a couple people it was the only option uh we were maybe not as hardcore about open source when we started this and it was free and we were broke so gmail it was uh we've been looking every once in a while on where to migrate off but we want something that has you know has available for a lot of aliases you can use uh different bots to control what who goes into what mailing list and what uh so we haven't quite found something yet we've heard a few options but uh it would be hard to transfer over like you can't transfer over overnight it would be a multi month project uh to do that so we're still using gmail for our email we did use google sheets uh and google drive at first but we have moved away from that none of the forms on our website anymore our google forms we have them all within uh they're either like custom built like by ourselves custom coded or uh through word press the biggest one that gets to me is that we use quickbooks for our treasury treasurer stuff that makes me really sad i'm not a huge fan i'm in fact not a fan at all of quickbooks and into it they have some serious issues that i won't go into but they're as far as i know there are no open source crm and treasurer and uh financial things for like small business type things there is or nonprofits i have looked the organization i work for uses a very uh old project that is incredibly technical and would not be accessible unless you're like a legit programmer uh which is fine if you're getting paid for it but if we want volunteers to work on this and they're not like super militant hardcore uh open source it's really hard to convince someone that they have to learn like terminal commands to bookkeeping so we use quickbooks right now and for real seriously if anyone knows of any open source bookkeeping software or crm software please please tell me because we've been searching uh the best we found was udu which is open core and specifically their bookkeeping software is uh subscription based which is disappointing uh they have other services so i'm not knocking them i think that they're a good organization but what we're what we need for bookkeeping they just don't have um the last thing that we don't have open software for is the keys to our building and we actually use a physical key like a normal key that you would use to get into your house we don't like have a little passcode or a little dongle that you can use to get in part of that is because we rent so like changing the door hardware would be a big deal because our you know our landlords need to get into the building part of it is the question of privacy as well so like we don't want to collect we don't want to know who's in the building when we don't want to know that information we don't even want to collect it we don't want anyone to know that uh another reason is that not all of our members have smartphones so you can't have an app on your smartphone because that will you know enable like some people won't be able to get in the building that way then uh so eventually we would really like to have like a dongle way to get into the building um if we ever own our own building we will probably enact this and then have it uh key to the membership database as well so if someone uh like quits membership removes away we can just turn off their access instead of expecting them to mail their key back in which is what we do right now uh but um if we go to a different place where we're renting again and they have a key fob system it's going to be a lot of hand ringing because we don't want to give our data away to them either so we're not really sure how this is going to play out but currently we're using the old school way and that is how uh that is how we do that at the moment no software that is our answer to that one uh this is about the end of my talk it's been really fun to um i wasn't a linux nerd when i started going to nox makers and i've since like become like fully everything that i own is uh running open software and i really love it um i'm open to questions if anyone has any uh otherwise you can talk to me at the booth or email me at nox makers thank you uh yes question no yeah uh the question was about the space how much money comes from membership versus donations versus grants uh the the long answer is is that our nine nineties are public so you can definitely look that up the short answer is is that all of our um operating costs are covered by membership dues we have enough that we have enough members that like everything we would be fine if we never had any donations or grants we mostly use we've never used a grant for operating costs we mostly use them to get cooler tools so and what was the name of your makerspace kaneho valley makerspace go visit that cvmake.org yeah i'll totally plug anyone's makerspace i love it uh did you have more questions question uh the first question was is there ever tension between linux users and non linux users uh and if we could just do this one thing proprietary would it fix everything well which is what they ask uh and the answer is yes sometimes there is tension um sometimes if someone especially in teaching classes because you get people that are super excited about the projects they're working on and that's valid i'm excited about the projects i'm working on and then they're like but i use x y z and we're like you can talk about the whole thing except don't teach people x y z and yeah sometimes it does cause a little bit of tension uh but we try and have it so that if you do bring in your project in x y z there's a way to get it into a linux or an open software so that you can at least do the same thing that way or pass it through uh an open software like uh the web app to get stuff onto the laser cutter it will take only svgs so i've never actually tried to export out of uh like a different cad program in an svg and load it into there but i'm not sure it works i think you at least have to pass it through inkscape and pass it in inkscape svg so sometimes there is but we you know this is how we are this is how we run and you can make your projects any way you want but this is how we do it here uh and your second question was how do we do outreach and uh it was a very slow burn getting going uh we did a lot of community building things we did a lot of interesting talks we had every tuesday when we did like our show and share we'd have like a talk afterward and like draw people from the community in uh we don't do that as much anymore because our show and share sometimes like will go an hour because we have so many people showing off their projects uh but sometimes we still do have talks um about different makery projects or different things that people are working on so just having a lot of public events uh and advertising into the community and uh being a friendly group that helps a lot too um so does that help i'll come i'll come to your organization if i'm ever in the area again uh yes yeah i think that if we uh the explanation was about their RFID system to get into the building and i think that another thing that we would um another challenge that we have with that is that when we have a trial period where you're allowed to be a when you're a first start to be a member you're not allowed access to the building by yourself for 30 days to make sure you're like a nice human um so if we had a pro a program like that we would want to keep it like normally locked and uh currently if you're a trial member you can just if there's someone there they will have the door unlocked and they can come in and use the shop and whatever so there would have to be some way to for some for it to tell like i'm a trial member uh let me in any way something like that but without giving them full access so but it's a good thought to think about yes uh the question was how do we balance uh privacy and and assessing people's skills so we do have safety classes that we require everyone to take if you want to work in the woodshop you have to take the woodshop class even if you've been working in a woodshop your whole life you have never worked in our woodshop so we do require that and you it's not guaranteed that you pass the class like if you're acting a fool during the class you you don't get to be authorized um we put a very high level of personal responsibility really um we tell people that we don't fault anyone ever for asking a question we are all here to learn everyone has been a first timer at some point uh if you are caught using a tool poorly like if someone comes in and there's like blood everywhere that's obviously a reason to look at the cameras right so and then we'll find out who was doing that uh and we'll have a chat with them um but for the most part everyone that's come in really wants this place to work and is willing to learn and is willing to not mostly uh not use things that they're not allowed to or things that they don't know how to and we have lots of opportunities for people to learn from each other as well yeah uh so the question is how do you balance wanting to do all the projects and making sure that they follow through yes uh sometimes things take a really long time so that's one thing uh I mean like and the other thing is that I think it we have a really good core group of volunteers the board is really committed the area heads are really committed so even if it does take a long time um it eventually someone will pick it up we'll it's not always only one person working on a project you've got a team of people that's sort of like if one person has a family emergency someone else steps in or something like that uh as far as like doing other volunteer things in the shop we have really cultivated a uh culture of like of cleanliness honestly I've been to some maker spaces that are really cluttered and it's hard to keep them clean it really is because like like you said your own shop you can do whatever you want uh but cultivating that volunteerism like if someone needs a shelf built in the welding you just like call up all the people who are newbie welders hey you want to learn you want to do some skill building you want to come out on a Saturday and like do a project um but like the big big projects like the laser cutter um we would want to make sure that it was someone who's going to follow through on something like that if it's a smaller project and we're not sure that it's going to like end up finished any day I mean maybe it just doesn't happen for a long time it's a lot of we're really casual so uh and that's an expert that's a I mean that's what we tell our members as well like there's no customer service here things are not always going to happen in the most timely manner but it's a community it's just how we it's how we go so yeah uh the board yes but also we uh we have our area ahead so each of the zones has what we call Zars and Zorino's that run them they run the budgets they decide what projects to do they decide what tools to purchase so they would sanction something like that um the a lot of our things not a lot of our things are home built anymore unless we find like a super good deal on craigslist or something and it's like something in horrible shape and we like have to clean it but we're sort of in the financial place where we can purchase things uh maybe not software but like machines we can purchase them so you had a question that's a very loaded question have we had any drama or um antisocial behaviors so the 30-day trial membership is sort of how we get around that like if people can't uh keep it together during that time then they don't get to become members and that doesn't happen very often we do have uh what we call peace officers which is like a third party that if you're having a misunderstanding or something you can go to them and they will help you like both parties work it out anonymously if there is like uh a harassment or some other like serious issue we would just kick them out like we don't tolerate um we don't tolerate any of that stuff at all so that's really our policy is uh don't be a dick is our policy sorry it's just yes uh the question is what is the division between volunteers and members and management team uh well ideally every volunteer would be a or every member would be a volunteer like that is how we talk about it like if you come and use the shop at the very very very least you have to leave it cleaner as when you came in like everybody volunteers here everything that is done here is done by us as far as um management team we have eight board members we have one person in charge of each zone and each zone has like some of them have deputy people so like the wood shop is really big it's way too big for one person so he has like small like not smaller but like people under him that like are authorized to fix the tools but honestly anyone who comes in if they see a project on our project board that needs doing go do it I mean everyone's a volunteer that's the that's the answer that I'd like to give is that everybody's a volunteer yeah the question is do we have an issue with people leaving their projects laying around um and sometimes uh we also give the area heads like their own decisions like in the wood shop that's where the bigger projects are going to be in the wood shop electronics projects are you know this big not less of a deal um and he does allow people to leave projects there for a couple weeks at a time as long as it has your name on it and the date and it's being moved forward that's totally cool um there is each of the areas has like a little corner that people might store their stuff as long as uh it's not being obnoxious and honestly as long as it has a name on it and it's moving forward you know people actually get interested in it like oh I saw that last week and now it looks cooler um we do have personal storage everybody has personal storage if they want it so it's just some shelves with some bins on it you can leave small projects or PPE or something like that the first question was do we have multiple levels of membership and the answer was no everyone has the same uh rights and responsibilities we do have different prices like we have lower prices for students or people on fixed incomes um but they still are fully members in every other way how do we keep track if people are checked out on the tools uh that is an option within HacksDB that I'm pretty sure no one uses sorry uh I know that when I was teaching authorization classes I was not using it you can keep track of tickets in um in wordpress and that I suppose if we if we thought that someone wasn't authorized they we would go back and check on that I mean I suppose we should try start trying tracking that which is honor system right a lot of things is done on the honor system um which is another reason why the door access project never really got off the ground is because we thought feature creep we thought oh we could make this not only into door access but we could make it into tool access and then we could put them on all the tools and you'd have to key card into a tool only if you have access but then there was so many questions about safety and like people walking away and like how long does that do work if you code if you like flash your key or something and then it just never got off the ground uh maybe a couple more questions one more yeah uh the question is how do we handle physical accidents or liabilities well everyone has to sign a waiver when you come into the space we have very expensive insurance and I'm happy to say that only two people ever have had to like call for help because of an injury so is that answer your question it's I mean we as we have them sign a waiver that uh you know we basically tell people like this place is dangerous and it is you know it's up to you to keep your own self safe so which is why we require the safety classes and we teach them and we encourage people to ask questions and uh if you're if you're questioning your ability or questioning the tool then back away and wait so I think that's it yes one more the question was is the 30 day policy 30 calendar days or 30 sessions that you're under observation that second one sounds horrible to administrate uh it is 30 calendar days um from the day they pay their dues to 30 days after that and we do also have like a little onboarding session where you have to come and listen to me and one other guy talk about how you know how to be a good member and stuff like that um but it is 30 calendar days and we do but we do require them and ask them to like come and show your face interact with people we've really found that it is the best way to uh you know make sure that they're good humans is to just talk to them and you know get a feel for the space and get a feel for them and like how are you going to interact with other people so um it is 30 calendar days all right I think my time's up so thank you hello can everyone hear me okay that works let's see oh um let's see if that thing works yeah okay turn it off in the meantime um okay hello everyone um I might wait just a couple more minutes to let people come in I think we start officially at 6 15 um is this is this volume okay for the the microphone yeah okay good thank you started oh sorry about that okay um so welcome everyone thank you for coming to my talk today which is creating terrain models with python blender and second life um anyone here familiar with emacs yay anyone here familiar with org mode emacs org mode at least one person that's how I did my presentation so this is actually just an org mode uh text file in emacs with a um something called org tree slide mode so um yeah hooray for open source very esoteric text editors um so who am I and why am I giving this presentation um I am all of the things that it says here uh writer visual artist vocalist musician filmmaker scholar or perpetual student I guess I should actually add a coder to that and then long time explorer virtual worlds um I am very passionate about free and open source software and um deeply appreciative of all the amazing variety of tools that are available to us um thanks to the internet of course and also the tireless efforts of open source and free software developers um there's a lot wrong with the internet with social media and AI and all these things but open source I think the open source movement in general is one of those things that the internet has really um helped to kind of you know push into the mainstream so to speak um I am currently uh my qualifications I guess uh if you want to call it that I'm currently doing an ms in media arts and technology at UC Santa Barbara um which is a very non self-explanatory program it's a very strange program but I won't talk too much about it um and I previously completed an MA in mass communication and media studies at San Diego state uh and also a very convoluted MFA at Cal Arts right up here in Valencia so I did um vocal performance and creative writing as the two sides of my MFA plus integrated media which is the sort of tech focused concentration so very convoluted completely non self-explanatory anytime anyone asks what I did there it's always a long conversation um and uh the reason I kind of started doing these training models of course I've been a resident has anyone here heard of second life raise your hands most people it's not everyone uh but I'll talk a little bit about that and then go into blender and python and what I did with this uh but I've been a resident of second life for 19 entire years which is a little scary uh I joined in March 23rd 2005 the platform actually um began or it was released officially to the public in 2003 I had beta before that so I've been there not quite since the beginning but almost um again very scary so try not to think about how old that makes me um this is just a link to my student bio on the UCSB MAT page um I'm not gonna like read this or go through all the links here but that's just sort of a little bit what I just said um and some of the other things um that I've done there uh and then what I will actually do just to start off start us off uh especially since not everyone here was familiar familiar with second life um I did a video essay a couple of years ago just a three minute video essay um in second life that I'll show and then I'll get into the main portion of the talk my avatar goes past my camera shall not follow as it turns out this divide is at the heart of a question that has driven me for decades the question is this at what point does my presence become less about who or what I am and more about where I am is there a difference for someone else in the same virtual space seeing me on their screen the answer might be more straightforward my avatar is how they know me after all is how they see that I'm there in some worlds or in some circumstances they might even see me as no more than a name the question is far more challenging when you consider that technology has given us so many new ways to know ourselves and to experience our surroundings ways that do not necessarily match our experience in and of the physical world that we're used to in second life for example there's the third person view potentially for multiple directions and a first person mode as well called mouse look while these assume that you're looking either at or from your avatar's position it doesn't end there you can focus on anything in the environment zooming and rotating and panning as desired you can also take advantage of hardware peripherals like a space mouse similar to a joystick but with more axes in order to steer the second life flycam around as I've done quite a bit while filming various videos including my mfa thesis besides the cinematographic freedom involved what I also find fascinating about second life is its mainland of contiguous virtual land masses this contiguity presumes a sense of place and perspective that is at odds with the design of most social VR platforms where smaller worlds more self-contained appear to be the order of the day now this isn't to suggest that second life residents can't simply teleport across the grid at the touch of a button we can and often do and yet the mainland continents still possess a plethora of airports marinas roads and railways even the occasional gas station all of which speak to an impulse toward a kind of realism with many people choosing to acknowledge the virtual distance and traverse it at their leisure in a manner that is paradoxically in contrast to the platform's extremely flexible camera controls is the world itself perhaps an anchor grounding our experience when our avatars and their fickle points of view do not I could talk much more here about this and other worlds worlds which I've explored for more than 25 years and counting but I should probably save that for another video for now and by saying that one of my lifelong goals is to critically and creatively explore a variety of virtual spaces asking questions about place perspective and interactivity and how those concepts come together to define our virtual lives and how we express ourselves so yeah just that's a little bit introduction to some of the concepts in second life again you kind of saw the camera controls are very flexible and as you saw sort of the maps showing there's very large land masses in second life so I will close that so what is second life quite a few of you raised your hands and said you'd heard of it at least how many people have actually used second life or been in world one person two people a little bit a little bit okay that's about what I expected actually but it's still around a lot of people think it's gone but it's still around but for those of you who haven't heard of it where we've never been there and excuse my mouse keeps sliding down here it is a shared persistent online virtual world with many many thousands of people usually there are anywhere between like 30 to 60 thousand people online at any given time of day any given day and it is made up it's persistent virtual world like I said so it's made up of thousands of different regions that are all laid out in the form of grid hence why a lot of times when you talk about the metaphor of second life's world people call it the grid a little bit like Tron's world if you've seen that or Tron legacy and each region is a square in the shape of a square 256 meters on a side and there's actually another resident who's been a long when I say resident by the way that's just the second life terminology for a regular user they're also linden lab employees linden lab is the company that runs and maintains it anyone you see in second life with the last name linden they're an employee of the company there's also a group of independent contractors called moles like the little animal linden department of public works and they do various projects for linden lab and they'll have the last name mole but everyone else is a regular resident but one of the other residents who's been around a long time and she's actually a professional statistician in real life she's run something called grid survey this database where she's gotten information about second life various statistics for including sort of this the size of it and so you can see here as of like six days ago about a week ago the grid had almost 28 000 regions totaling this you know amount of square kilometers and then you'll see that some of them are actually directly owned and run by linden lab itself and others are private estates and you can just see here the only unfortunate thing about grid survey is it takes forever to load I don't think her machine is very good but she has a lot of data so I'll forgive her for that but it will load eventually but just to talk a little bit more about this even when it says private estates those are still basically rented or leased sort of from linden lab the company itself so this is not distributed this is not decentralized there are open source alternatives to second life called open sim or open simulator that is actually an offshoot of second life originally and those can be hosted anywhere by different people you can actually get open sim like a grid of your own whatever size you want on your computer but the second life grid itself which is the most populated again is all is hosted through aws amazon web services run by linden lab so the private estates here that number it's still sort of going through linden lab but it's regions that are sort of paid for by other people oh and here's grid survey has loaded you can see just the the data that I got here and if you kind of go through this endless list you can see regions that currently exist that no longer exist and see various information and then there's also the grid survey api which has various information so and again as I just mentioned in addition to the privately owned island regions in estates they they're called island just because they're sort of separate from you know out in the grid somewhere and I'll show you a map soon there are also several contiguous land masses which are you know thousands of regions in total there's one that's 433 regions another that's like almost a thousand and they kind of vary in that sort of range and they're what's known as mainland or the mainland continents and that's kind of where I started the focus of my project for the terrain models and again mainland is is a special case in the sense that even if no one else pays for that region it'll still be there it's part of that landmass it'll continue to exist and the way that linden lab actually sort of maintains or pays for the upkeep of these regions is that they will lease individual parcels kind of you know land within these regions to regular residents that can do different things with them and again these land masses are known as mainland there are about 10 I would say official mainland continents and then there's balasari in the linden homes which is the largest continent but is kind of a special case because that's one of those areas where I said it's developed by the moles the independent contractors as a benefit for premium subscribers second life is completely free to use but if you pay a premium membership one of the benefits is that you get a free linden home in one of these regions which I'll show you in just a second this is again I'm not the only one who's apparently obsessed with this world this is another resident run project the second life geography institute that has a list of some of the mainland continents here there's a ton of information about this but so this is a list of continents not all of these are actually mainland like I said balasari is kind of a special case the lake sea is an open ocean area not really a continent and these older premium east south south south are sort of the older versions of these linden homes but it's still kind of interesting and then you can see here this is on the the second life official wiki that's one of the senior vps of linden lab going for a skate ride but so this is basically a wiki page about all these different themes and as I mentioned balasari which is currently about 2,500 regions so it's quite a lot has these different themed neighborhoods with like hundreds of houses each in these different themes and as a premium member you sort of get you know you can get a home in one of these neighborhoods and kind of do whatever you want with them so there's I think about 12 themes currently you can just kind of I'm not going to dwell too much on this but it's it's interesting and I will say that one of the recent updates some of these might look a little dated some of them look nicer than others the lab actually just who's here who here's familiar with pbr a physically based rendering so a few of you it's basically a new way to do 3d graphics that's a little bit closer to how things are actually rendered or look in real life using like the properties of the material to be like if it's metallic how rough it is so they're actually they've just done that and there'll be real mirrors and reflection probes and things like that kind of beyond the scope of this talk but again you know kind of looks nice so and then one of the continent size models that actually did last year I started off with smaller regions and then this is actually one of those mainland continents that I mentioned heterosur which is 433 regions so this is kind of how it looks on the world map 2d and then this is all made in blender rendered in blender and you get farther down and then suddenly see you can it pops out you can see that this is sort of what it would look like again the objects that are on the world map are sort of shown there now it's just the you know solid rendering mode in blender but this is how the terrain would look if you could see the entire thing at once at least and so I just did a loop around that this is also called the atoll continent kind of can probably tell why but so this is the first continent sized model that I made sometime last year and I think if I remember correctly this particular model in blender at the original level of detail with the the data that I got from Second Life to do this would have been 227 million triangles my laptop would have exploded if I tried to render the video with that but I decimated each region's mesh so that it's only 2.3 only 2.3 million triangles instead and I've actually I'll talk a little bit more about the process involved in getting this data and using it but when I did this last year and gathered the height data across all these 433 different regions it took about 32 hours to get the data for all 433 regions now it takes much less time luckily because I changed how I gathered the data a little bit so and then I do want to show just before I move on of course it would ask me for this sorry one second as everything has to be secure these days which is good of course you might have seen in the perspective in place video the video essay that I showed that there was sort of a map museum it's called the new cadath lighthouse art gallery I don't know how good the internet is in here so it might take some things a while to load but as you maybe saw in the video there's this place which is a long-running gallery again everything that's gray is basically loading in everything's online textures sort of download on the fly so you can kind of see this is that map you can see someone made there are train stations and railways so someone did sort of a live updating map of certain little automated cars there or train cars on that map and then you can see this is the new cadath lighthouse art gallery with lots and lots and lots of maps of various parts of second life the mainland continents along with other places and this is quite interesting this is from 2002 it's in the shape of a key the original 16 second life regions as they existed back then and then over time more and more regions were added to the point where it became oops sorry someone just messaged me and I'm gonna put myself on unavailable I can't log in without some I'll talk to you later but yeah so and then you can kind of see over here some of the this is some of the bellas areas it's grown and oh thank you very much I was going to actually ask if you guys could see and that probably helps quite a bit so you can see some of the mainland content collections or map collections magic map where you can see any particular region and then I was going to show on the world map you can kind of see also so this is a live updating world map which again because it's live sometimes takes a while to load but so that this continent that you see here which has a sort of an odd shape from like up here to like down here is Bellas area that linden home with all the different themes then you know premium member neighborhoods as you can tell and this is the original continent Sinsara and then you can see Heterocera and there's others over up here so there's super continents and then if you zoom all the way out I don't know how well that's going to load but there's like I said almost 28,000 regions most of them are sort of off by themselves either as individual regions or forming contiguous land masses much smaller than the mainland of course but still maybe someone owns it and wants to make a private estate for whatever reason whether it's a school or some other organization or just an individual with the deep pockets but you can kind of see it get a sense at least of the scale but the question is why did I actually start making these terrain models and as you might be able to gather there's a lot of landmass in second life and the viewer software can to a browser basically but it's what you use to you know be in second life they're very the viewer code itself is open source luckily they open sourced it some years ago so there's the official viewer from linden lab and then various third party viewers that have different features and different UI so and is available for linux as well which is a good thing for me since I'm a long time linux user but regardless of which viewer you use you can only render a small chunk of the world at a time you can't see like that entire continent at once I think the it might be different on certain viewers but you can change your draw distance from like 32 meters away from your camera position up to 1024 for anyone you know developers you'll be familiar with powers of two and you'll see that a lot in second life in various areas like every parcel individual parts on a region is like a maximum of 16 square meters or minimum rather I should say and that goes up from there so but anyway so you can only see a few regions ahead of you maybe three or four if you're lucky depending on your GPU it's a huge world it's extremely diverse and I wanted to basically for myself and for other people both people in second life and who have never been there give a different perspective in the world and be able to see more of the world or more of its land masses at least the terrain than you're ordinarily able to do also I like to tinker and explore different things through sort of project based learning so blender's fun python is fun put them together and try using the blender python api and sometimes it's not so fun because there's a lot of testing a lot of debugging and I I'm still somewhat skeptical of chat gpt and other things and of course I started this before sort of the AI wave really took off so all of this is basically just me digging into the code looking at stack overflow which of course now is probably overrun by AI anyway but but yeah it's it's been a very interesting project how am I doing the time okay I have a lot of time and so the process actually involved in this it's a multi-step process there is a built-in scripting language in second life called lsl or linden scripting language and you know you can give put a script or multiple scripts into any particular object and do lots of different things with it and so one of the functions and you'll hear see here on the just close a few of these this is the lsl portal which is sort of the on the official second life wiki showing it's kind of a c-like language I would say but so it has lots of different functions lots of things you can do related to you know camera movement collisions you know different math you know physics all of these things people make all kinds of vehicles in a cell but one of the functions that I again used for this the main function is this function called ll ground which basically tells you the height the terrain height of any particular point in a region you might notice here it says offset there are some functions that get like partial details or details about the region where you can actually say the coordinates again the coordinates are zero to 256 on the x zero to 256 on the y unfortunately ll ground is a little bit different it takes the height below the scripted object or if you're wearing that scripted object below your avatar so one of the kind of challenges I encounter is basically having to constantly check okay where am I or where's this object as I'm getting the height so that if I move a little bit you know I try to kind of keep myself relatively still but you know you're constantly checking the position but so what I did with this function is I loop across a region and I have a python script on my computer everything by the way another kind of little function here HTTP response anything any object in second life can be simultaneously or either or or both HTTP server or client with certain limitations of course but so the way I actually gather the data in the first place is to have a python script that communicates with a scripted object in SL and basically tells it I want data from this row this x value and then the script in SL will say okay it'll loop across an entire row of x or I should say yeah the y coordinate is what it's sent in and then it goes every half a meter across the region at that particular y coordinate every half a meter in the x axis basically and gets the height and then feeds that data via HTTP to my my computer as just a bunch of float values I can show you here what it looks like this is a particular region called Cormac so this is 513 lines of just tons and tons of float values again every half a meter getting the the height at that particular point and then some sort of preliminary data about that region that helps me when I want to reposition them in blender or in world so what did I just do so there's a lot but and so that was a challenge of course in itself to try to figure out how much data can I actually send by HTTP to my computer from SL on every pass through and when I said that it took 32 hours to gather the data for heterostra last year I had one scripted object that was just doing this now I have I think one object that has what's called a link set when you have multiple objects linked together each of them can have a script so now I actually have it in parallel so the Python script is sort of doing I don't who here is familiar with Python by the way or okay quite a few do you know async or async I know there's that and I think I actually looked at that and I wound up doing I think in a threaded version instead that basically is able to I think in like seven or eight threads get data from different rows which saves a lot of time by being able to communicate with multiple objects in second life and not rely on that one HTTP connection which is sort of a bottleneck after that I can make a list that of different regions that I want to gather data for that and then the script can actually the script on my computer can look at this list and then communicate with the script in SL and say okay I've gotten all the data go to this region at this set of coordinates and then the script in SL will teleport me automatically usually it runs pretty smoothly occasionally get teleport fails in second life and then I have to not restart the process but restart from where it left off for where the teleport failed but usually it works pretty well um you can also download the those map textures that you saw in world um thanks to taiki shepherd's grid survey and there's some other people who have this every texture every image most every object actually in second life has a uuid a universal unique identifier um and sort of the textures so these are all sort of publicly available um and taiki has the uuids for these map tiles on her site um so I download them generate region models using blender's python api apply the textures part of the script can also decimate and reduce the the complexity like I said it was like down from 226 million triangles to however much I said um and then export them as mesh objects as colada files that you can then upload to SL uh and then I have another lsl script that can reposition these uh and apply textures to the water terrain and the water which is a separate um it's called face in second life but it's basically a different material uh for the water height uh and profit not really I haven't made any money off this but it's a passion project for now so uh maybe at some point how am I doing okay um so um what I'm actually going to do now just since you've seen a little bit about the process that uh is involved uh and what I've done um hopefully no one's gonna come in here and wonder what's this person standing in the middle of the floor so I'll just put myself in afk mode um where I'm out of the way uh so I'll do a little demo of blender and then I'll show a few more things and if you have any questions I'll do q and a toward the end but also feel free to stop me at any point um if you have questions about anything uh and I do not have blender open yet okay so one second so here is the blender interface it's amazing and what people usually do is they open blender and then they delete the poor default cube um poor little cube um but so if has anyone here actually tried using blender's python api not really okay good that's I mean you know hopefully this inspires you to do something with it anything I find blender um there's a guy some of you might know named Andrew Price or blender grew um I've he's like talked about how with blender there is a lot of complexity there's a lot in the software but he uses what he calls the 80 20 rule that I think uh let's see if I say this right that he uses 20 percent of blender's features 80 percent of the time so again even though blender's extremely complex you're not going to use all of it I mean for most projects you'll use a tiny portion a big part of the learning curve is just getting some of the shortcuts into your brain and I'll talk about that too but so if you have not used blender's python api you can see up here on the right I hope that's maybe a little hard to see well up here um there's something called scripting tab in the default sort of layout and you click here and then here's your text editor another view uh viewer area uh and then here's the blender python console and this is extremely useful for getting started with blender's python api I'm just going to maximize that uh and then you can zoom with the control wheel so that's a little bit easier to see um but so this is where you can test out things um see if they work see what you need to put in for commands and then you can kind of write them down in the text editor and and kind of run them so one of the things I'm going to do right now just to kind of show you what you can do um even if it's is pretty simple I'm going to delete everything so I'm going to create let's say basic text objects now I don't want to have to go into edit mode and actually you know say something there I just want to be able to change it and maybe I have a script maybe I have a text file of strings or something that I want to use um I'll just like that um and what you can actually go and do um this this is where sort of the learning curve for blender's python api gets involved because um everything in blender is basically on the back end this this hierarchical series of data blocks um so there's a lot of like dots involved so for this if I type this open so oops that's the wrong the the focus follows mouse gets a little bit uh tricky sometimes but so bpi dot data dot objects that's sort of the hierarchy under which the objects live and then if I hit tab you'll see there's only one object in the scene right now it's text and so let's say I want to actually change the string like the actual text that's showing then there's another dot another data again things get somewhat long here sometimes and then body that's currently the so if I hit equals hopefully everyone can see that um and let's say um this is only a test and of course I hit I forgot no I forgot the quote um and look it's been changed just like that uh another thing that you can do is actually change uh in it oops um in addition to that you can also change something uh data dot extrude let's say right now it's uh zero there's no extrusion it's a very it's like it has no volume to it let's make that one oops uh and then suddenly you have something that's actually quite large um and then if we delete that then uh one of the other thing oh actually let's not delete that for a second um and you can go to this mouse movement is not helpful um you can actually go to the location of it so like if we go let's say it's not data now now it's just the object itself because we're not looking at the data of the text object we're looking at the object itself um and see where its location is okay that's you know it's kind of at the origin let's change it to one zero two now it's been moved up around a little bit now the other thing that you can do of course and this is sort of where um I'm getting into sort of the the way that you can make it easier on yourself how to find things is if you go let's go to the main screen here if you go to preferences in uh in the interface tab I'm sorry this isn't bigger but uh on the interface tab in blender as preferences there's a check mark here I already had it checked but it's called python tool tips if you can't see that it's right there um python tool tips and now what happens is if you hold let's uh actually bring that back text if you hold your mouse it's over here in the properties thing and you hold it over let's say location x then you're gonna see it says this is really can is that really blurry I get yeah well I'll read what it says so it says location uh location uh of the object and then ordinarily what you don't see but what is now there since I enabled that checkbox it says python object dot location and then it'll say b pi dot data dot objects text location zero because it's the the x coordinate what you can actually do there because you don't have to remember that you just right click on it and then hit copy fold data path if you do data path it's just gonna say location but you can either do shift control all see again these like massive shortcuts but uh or just right click and then now if you go to scripting and you hit then you've gotten that and that's the x coordinate uh for that so you can do that with pretty much any sorry um any property like that or anything and if it's a menu or let's say uh we wanted to add let's say that we wanted to add a new text object um usually would go out there up to add text or shift a fo