Hey everybody and welcome to the talk infrastructure at arch making so let's go blue and where we talk about the infrastructure at arch and how to make so let's go blue and so anyways our agenda for the day is I'm gonna tell you a little bit about me then I'm gonna introduce the team to you I'm gonna give you an overview about you know our general infrastructure what tools we use and so on and then I'm gonna delve into a past kind of like some history lesson about our our infra where we come from and then we're gonna see where we currently stand and then I'm gonna see well I'm gonna basically give you some kind of roadmap that we have current projects and and you know near future and far future projects and now we're gonna try to tackle those and then I'm gonna show you how to you get you how you can get involved and in infrared arch so a little bit about me I work you know by day time well mostly anyway I work as a freelance DevOps consultant and also software architect depends on the job and I'm involved in quite a few open source projects I also do quite a bit of game dev I do quite a bit of rust and I also like seal it was very much so if you check my GitHub you're gonna see many of those and and I joined our edge in 2020 at 2010 actually as a trusted user and I think I've been using arch since I think 2007 not quite sure but anyway so I eventually became a developer and now obviously DevOps and not quite sure when I became a developer at some point I took over KDE and this is I think when I became officially developer all right enough about me so the DevOps team who are these people so you might see you might have seen some of those faces we're not only DevOps we all have packages so there's who it is Francois who's a very recent joining actually to the DevOps team he's not entirely on board as of this moment currently and we're still working on that there's Grasolini there's a heftig who maintains the market server there's yellow there's untracks and there's me and we all do different like we all have different levels of involvement in this project some have their their their pet parts we all try to kind of be be able to do everybody else's job if need be but mostly I think it's it's fair to say that everybody has their own topics that they care about like like anything else an arches it's pretty much the same and the only thing that's a little bit different I think is that we have this DevOps team which has regular meetings and I think this is not the case and in any of the other arch teams at least I'm aware I think the security people might have something not quite sure and all right let's delve into the overview so let's start with servers everybody likes servers so arches mostly hosted a heads now we have six heads not bare metal servers and 17 heads not cloud servers and total so that's 23 machines and then we also have two off-site backup servers for Borg so we use Borg backup for all our backup needs and on top of that we have five sponsored servers by Cape thanks guys shout out and I'm afraid we have to admit we have yet to make use of them because we have very recently actually got them so still still the process of setting that up as far as services go we have quite a quite a few of services right so we run like if you look at our you might see you know that the front page and the a you are everybody knows of that but there I mean also the wiki of course but you know there's quite a few services public services that we provide and and many of those services that we provide actually you know they might be presented to you guys as like a coherent like one software solution or something like like Archweb but actually there's many moving parts involved in any of these so for instance we we run the archive so you need this in order to use the Arch Linux you know way back machine essentially and it's also it's also does it also plays a role for reproducible builds and we have Seagate which you might know as projects.archinux.org currently trying to get rid of that very recently we've gotten a good lap up and running or well we have we have had it for some time but we're trying to make better use of that as as time goes on and yeah we have the wiki which is a media wiki of course we have a security tracker you guys can give the wolf and security stuff can post some stuff there we do have a patch tracker it's a good old patch work trying to get rid of that as well actually the the packages that that you might see as you know the like the packages an area or like the mirroring infrastructure that's actually just an our sync service which runs somewhere that everybody pushes the packages to and then every like all the tier one mirror sync from that so it's like the the packages thing that we have is essentially like the tier zero mirror and what else yeah we have the brick you might have seen him like it's a it's a little bot that we run to get IOC going to have like IOC moderation to ease IOC moderation actually we still use fly spray as our primary bug tracker I don't very much like that I don't think anybody really likes that but it's what we currently stuck with we are planning immigration at least a partial migration of some of our projects to GitLab but we have yet to work out how our package bugs which is the vast majority of bugs are gonna be migrated so we're still working on that and of course we also have the account service as is all but I'm gonna talk a bit more about that one later in particular and so we have also some staff only services actually we have a build server that we offer our staff so you know it is a pretty beefy machine it's it's kind of last gen but it's still pretty beefy it has 24 cores for 48 logical cores it's also sponsored by Hetzner so thanks guys and so our our staff can use or our packages at least can use that server to build their stuff we also have a Kenboard which we're also trying to actually quite quite soon we're trying to get rid of that one it's essentially like a Kenman board it's a free open source Kenman board trying to get that one into GitLab because GitLab has nice nice features for that and so we're trying to get rid of some of our services which are now super suited by GitLab we obviously have some mail service I say service because we have multiple mail servers we're trying to kind of perhaps condense down almost like a little bit our mail stack has has grown a little bit over the years and trying to get get that in order soon ish we have rest very recently we've gotten a monitoring stack that yellow has been working on so we used to use Zebix and it's Zebix is kind of old you know fashion and so we're trying to switch to Grafana Prometheus alert manager which is working quite well so far we have pretty nice dashboards if we have some time at the end I might be able to show you guys some of that and then we have a matrix that I already said that Teftik maintains and we have a Quasal and I'm not sure who maintains that to be honest anyway so over you about code so we have we we try to follow infrastructure as code where we're possible so that means if you make a change and like if you want to make a change in any of the servers you should may have a corresponding code change somewhere so for instance we use Terraform where we can and Ansible where we must so that means Terraform is a tool that allows you to write declared declaratively specify some infrastructure elements and the nice thing about Terraform as opposed to Ansible is that it actually does take care of cleaning up things after you remove them or that it actually tries to make sure that the the status that you give it is actually the status that exists that's expected in the end and Ansible in some cases has a really hard time doing that so but yeah we still have to use Ansible in many cases so for instance this is what it looks like so we have 8,000 lines of YAML delicious YAML which is pretty much all of it that's Ansible and we have 2,000 lines or lines of HDL so for instance in HDL or Terraform we specify our heads now the cloud servers we service we heads of cloud servers where we specify all of our domains our IPs all of that we specify all of our key clock things yeah all of that some some minor services as well and and Ansible we have like the vast majority of everything so we have 63 Ansible rules that's quite a bit that's that's everything essentially it's like email stuff that's that's all the services that you that you see that I that I talked about earlier except for key clock is in there so everything is in there it's it's quite vast you can take a look at that if you like and I think as time goes on we're gonna see more of those things move to Terraform maybe perhaps depends but we're actually pretty happy with Ansible it's not it's not a bad tool it's just you know YAML gets TVs sometimes and you know when you work with YAML mostly like every day you kind of kind of want to see something else but you know that's what that's what we're stuck with I guess anyway so the past let's let's continue so dark times dark times lay in our past yeah I mentioned like when I talk about the past basically speaking of arch when it was like first made till about four years ago so that that's the past that's the dark times and basically everything was mainly managed so that there wasn't like no centralized server management whatsoever right basically there was just a bunch of servers in some cases quite quite more or less literally in somebody you know else's bedroom and and all the like the devops that wasn't really like the term devops there was not like an arch devops team that there really was nothing like that it's it was more or less just there was some like people that managed some servers they had some scripts they knew how to do it they want to dox we essentially had the best factor of one for every single server so if you're not not familiar with this term basically it means you know if there's one person that knows how to do something and they get hit by a bus now we have zero people that know about how to do something and so that's a bus factor of one and it was really hard for outsiders to collaborate essentially was it was actually impossible to be honest um well because there was no no way right there was no code we couldn't just give SSH access out to outsiders so there really wasn't any way outsiders could possibly collaborate with us and there was no devops team so as I said before there's like only like a loose collection of people different accesses to different servers and there was no centralized issue management so that if somebody like had a problem with mirrors or something and somebody other that managed arch web or some some other service had to be kind of aware of that then there wasn't really any nicely defined way to to share that and there really wasn't any automations that were in place outside of cron so we did have quite a fair bit of cron doing work so automated bash grids essentially but really there wasn't anything that automated things between servers so yeah that was that was a pretty dark time so but then eventually we approached the road to enlightenment and that meant we started using Ansible so the first Ansible commit was made in May 2016 this is essentially also the like how far our infrastructure repo goes back and since then we've had a pretty recent pretty steady stream of migrations towards Ansible and we've also started using Terraform in February 2019 and I think we originally started using that for specifying the service for heads now I think we started documenting things documents documenting stuff is important and we started doing we started the DevOps team which didn't exist before and well and that brings us to the present and well now we do have a DevOps team with consistent access to all services and servers that we have and most things that we have are now in Ansible and Terraform so that's something I'm really happy with there's some stragglers here and there for instance like mail aliases are still managed manually you're not gonna find those in Ansible I'm pretty unhappy about that because you know somebody made a mistake there then that would be pretty horrible for arch emails and also pretty much a security problem but we're gonna get there on baby steps most things that we have are now documented or well most important things anyhow so we also didn't we used to not document any kinds of procedures or processes like onboarding off boarding stuff like that it was kind of just like in the moment like if somebody could remember to do something that they do it and otherwise they wouldn't really have a checklist so we started using checklists quite a bit for that kind of purpose like in the end I would prefer to be automating that but you know in the meantime I think it would be it's okay to just kind of have a checklist it's better certainly than that happening I think at all and we did start a migration I think that's what I said earlier like in February 2019 I think we started the migration from bare metal service to cloud service and so that has a few reasons first of all we like compartmentalization but that means that we want to have one cloud server that it does basically host one service and it allows us to in the end be pretty like pretty cheap about specific servers and services and we also had this effect on bare metal service where we had side effects where basically one service that existed on the server would be influencing another service so this is obviously broken compartmentalization but it was just done that way because it was easy and simple and was it was kind of like a hack and we don't really like hacks anymore so we're we did this migration or we're currently like in the middle of this migration and yeah that's still going on we're still sorting out some servers but in the end I think we're gonna have a cheaper most secure and I hope faster and much better documented um well general info after we finish this migration to cloud servers we're not gonna get rid of them like gonna get rid of bare metal servers entirely because we still need hardware features for some things and obviously like the build box would be much cheaper as a bare metal server but from the vast majority of services I think the cloud servers from hetsna are basically the best alternative that we have and then lastly we also have automations and ci which we used to not have at all right we had no ci and in any other of our patches or merger requests or anything I think we had we're like we had some very like sparsely ci like sparse population of ci files on on github at the time I think we used service for some services um but now with gitlab we we have like we can offer all projects one consistent and and well-known ci that's also trusted so that's also another important thing that we can now do trusted deployments from our own ci because we have our own runners and that's that's a big step forward um this is still kind of like mostly a work in progress but I think it's one of the most important features because arch being mostly volunteer or pretty much all volunteer driven means that any time that we as a team do not spend human resources on any of our well computer tasks like computers are much better doing like automated stuff like building stuff uh deploying things ci stuff making sure that unit is around and stuff like that uh I think all of that time could be much better spent on on actually working on real problems and the distro and it would be much less much less exhausting for everybody that that had to do those tedious tasks and also it will ensure that we just have a much better foundation for for mesh requests and and we make sure that all of those tests are being run and that'll make sure things are more stable for our uh software that we actually develop um so one important thing that I talked about that we didn't have before I think that's really probably like the centerpiece of probably I hope um our our DevOps revolution arts uh GitLab and so we received an ultimately license from um gitlab.com I think um who of our ultimate licenses to all open source projects that can demonstrate that they're open source so now we have GitLab so you and that was a pretty big step so that should be our central repository for code hosting issues collaboration um as I said before we earlier weren't really that easy to collaborate with which is obviously not great for a a project that really requires volunteers to have fun doing what they are doing because there's no other incentive so we're trying to make this a bit less um or making it make it a bit more seamless and and make more fun for everybody and uh yeah I'm used to have to send archpatches for many projects on mailing lists like it's uh 1980 and uh now we actually I think we arrived uh quite confidently in uh the 2010s at least now we have merge requests so all right um and as I said earlier we can also hopefully get rid of patchwork flight spread can but at the least um maybe more I don't know but for now these are these services are slated to be mostly or completely removed in favor of GitLab we have an overview of all our issues and um I can actually show you that so this is an overview that we have of all our currently migrated projects we haven't finished the GitLab migration obviously but um if you look at this this is essentially like the arch group in GitLab and allows you to look at all the issues we have quite a few issues um but keep in mind many are still in flight spray there's no package issues here at all so basically an arch you always like it's not just in distribution that has packages right uh obviously we mostly have packages and we mostly have package bugs but we also have many software projects of our own like arch ISO you might know arch boxes for dvms arch you know stalker um the infra project itself and and many other projects that we have and so these are all listed here um you can actually if you want to you can actually lay um filter by label for instance and see whether the label is um good first issue so some projects tried using or are gonna use are currently using this tag which is a global tag and then you can see which projects currently are by the reporters consider to be good first issues so something how you could potentially get involved um and I said before we have a powerful ci for everything now and that's also actually available to our users we hope it doesn't get abused too bad but if it does we're gonna you know come up with something we said some sensible defaults but guys please don't abuse our ci we only have this one pretty happy that we do have it at all um so um next topic uh that you might be wondering about is arch Linux sso so we have arch Linux single sign-on uh it's just a fancy way of saying that you have one account that's shared among all the services that we have and also one login that you have so that's shared so if you log into one service you should be logged into everything and that allows us to so you know you used to have like accounts for like wiki bucks so fly spray uh a ur possibly security tracker uh maybe if you were a tester you had an arch web account to to sign off packages so there were many accounts and internally obviously we have some more and so this just became really really hard to manage so imagine somebody gets onboarded off-boarded and for users was just plain annoying you know having to manage multiple accounts multiple identities and what for right and we also have the forums and it would be really nice to just have like one persistent set or one persistent account that would just follow your around everywhere and it would also mean that we have one central place to manage accounts and also to to manage access levels so for instance on our SSO currently we force two-factor auth for all users which we think is is really the way forward and to ensure that no accounts are taken away we realize that this is maybe inconvenient for some users but we decided that is probably the best way forward to ensure that uh well users wouldn't you know wouldn't suffer from reuse passwords quite as badly as if they didn't have a two-factor auth and also considering our users are mostly pretty technical uh it's I think it's better to say that this is a reasonable choice to make so we have one account for everything um two-factor everywhere and this is actually realized using the software KeyTlocked by Redhead which is well it's it's a it's a type of uh IAM which is identity uh no identity and access management and it's it's just a piece of Java software that we use which which seems to be the best compromises uh seems to offer the best compromise out of all the various software that does pretty much this thing out of there we also considered LDAP and and you know all of its implementations for a bit but LDAP really is more geared towards like uh systems logins and not so much towards uh web software logins so you can use LDAP with uh web software but um KeyTlocked which uh uses OIDC so OpenID Connect and SAML seems to give us more um let's say access control and and seems to be better geared for the web so this is what we we uh rolled with and also we got to be talked to the KDE people that did the migration to their LDAP back then and they said that we're pretty unhappy with LDAP and they're changing away from that so that's another thing um so yeah what's what's the look at like what's a roadmap like um well I said we had some migrations that we're still doing and we're still in the midst of that and since I just talked about as is all obviously one very hard topic is to get all the accounts merged somehow because we don't really want users to use uh to lose their access to any of their accounts and currently we have as is all enabled only for internal and for like for staff accounts and for users we're gonna have to get creative we're gonna have to make sure that we can really prove access so that the users can really prove access for instance for the like forums account or wiki account or whatever before um merging that into their one SSO account this is gonna be a topic that we're gonna be busy with for quite a long time I believe and and then only then can we actually connect those accounts as those services to to the SSO so currently it's mostly an internal tool but it will certainly at some point become more generally available tool especially for GitLab we're gonna open up GitLab at some point and then or well pretty soon actually I hope we really prepared most of everything for GitLab to be generally available and then hopefully we can give everybody access to that we are also migrating from amendment 2 to amendment 3 which is a bit of an annoying migration but it has to be done because amendment 3 is actually modern and supported and amendment 2 isn't so that if you guys don't know this is our mail bale server list software that we have and we need to automate more things as step before I think anytime that is spent doing things that the computer can do better it's probably time spent that we should be automating it instead and then the computer can do all things like the problem about automation like the only central problem is security because if you give a computer access to do things and you fuck up then you might be accidentally deleting all of our connects this would be quite regrettable I think I hope everyone agrees and so we gotta make sure that or if you know if somebody you know hacks the box or whatever then we wouldn't want to risk that so this is something we have to look into and eventually we get we should get rid of most of our bare metal servers so we want to finish the migration to be a host or vps or whatever and then we can get rid of the hardware service which will also alleviate some of our paints as far as hard drive changes for instance go or upgrading or something like that so that's gonna be much easier so next part is known issues of our future endeavors or things that we know are problematic in our current future up ahead is that for instance our form software fluxvb is essentially unmaintained and that you know it's php software so it's kind of like double bad so you don't want to have unmaintained php software under any circumstances but problem is currently our reality so we're really trying to get something else there instead I don't know we have no ssh audit locks so essentially that means or like if anybody of our dev ops currently look into any server they can essentially do anything they want without being a watch and also anybody can use ansible roles at any time and we would prefer if we had some kind of paper trail for that so this is kind of hard we do trust our dev ops so this is not about trust so much it's more about being able to to know when somebody did something for for general auditing so that you know if something fails you can see who did what and then maybe gather some information about the nature of the failure and as I said before our current may set up really needs innovation and innovation frankly yeah it's it's it's kind of from the very old days hasn't really changed much it's gonna change for sure when we connected to sso to ketlok so yeah that is also still up it's a known issue I don't really have any remedy for that we basically just gotta check it out and see what see what's good we need better gdr compliance automation we currently have we try to our best to be gdr compliant where we can but I think this is done mostly manually and and every team basically manages by themselves so wiki people and forums people have the different ways of dealing with this and it would be really nice to have some more general solution there's also place in there with the sso thing so if somebody has complete control over their account from one central place this would be easier for us to do but it's certainly a hard problem that has no quick fix so for the time being I think we're stuck doing this manually yeah so finally the call for action so get involved arch is 100% volunteer driven and and everything we do is public I might have shown you this before but everything we do is public so this is our info repo if you go to this link you're gonna see this this stuff popping up and and I'm happy to report that actually we've had a recent influx of active collaborators so that's really really something nice especially in the DevOps thing where you know things are sometimes complicated and kind of annoying really happy about having some collaborators join and but we always need more people helping out so make sure that you join Arch Linux DevOps in a free node RSC and just ask to help out just usually somebody around from us if we're not all sleeping at the same time which really happens I grant you that and and just ask for help and or ask for an issue or something or maybe pick an issue right you can join the mailing list it's not super active because we're mostly active on RSC but really important things are posted to the mailing list and yeah get involved we are super friendly bunch breaking servers is great fun especially you know if it's 4am and you really have to get up at like 9am amazing get involved break some servers with us we only have so many arms and legs just you know grab an issue make a PR make sure something breaks we'll deploy it no questions asked and but seriously you don't have to be an super expert to be of any help to us all right you you can basically most part of DevOps is just learning on the job I mean just look at the DevOps culture currently like there's one new silicon valley startup tool coming out like every week every day maybe I don't know some new amazing YAML based technology you can't possibly be an expert in all of these things basically just you know get involved pick something that you it sounds interesting to you and just kind of try to tackle it we're all learning on the job basically just just try your best I suppose it really doesn't matter that much we're all here to help we all make mistakes basically just just try it thanks guys for listening this far if you've made it use some links that I've also shown the presentation and also my my personal page if you if you're interested in that but apart from that thanks for listening get involved and see you on IOC Hey Sven hello let's have a look at some questions nice talk there for a thank this wrap up of all the infrastructure we're running I've split up the questions a bit in servers and services so maybe let's have a look at the servers first one of the prominent questions was basically by a lot of people is the Arch infrastructure actually running on Arch yes I'm happy to report that all of our critical and also non-critical servers are running almost entirely on Arch I say almost because we also use docker sometimes and we sometimes use non-arch docker images and we run some some minor services on other distributions for testing in fact all right cool our server maintenance handled basically updates etc that's a question by Machilano right so I'm gonna take this question as it meaning that how do we get notified about updates and then how do we perform those updates so we get notified by Prometheus yellow build a Prometheus exporter that allows us to see which servers have how many pending updates and then that is like too many then we update and we currently perform the updates manually because you know we gotta make sure that things happen nicely and we don't do any automated updates currently all right very enough um same person Machilano asked if there are any pros or cons of bare metal servers that you're currently seeing or facing basically in this infrastructure sure so the pros are they are a better bang for the buck they have greater performance and cons are they are harder to manage and if you fuck up you have to well basically reboot the same thing and then it's kind of annoying to mess around with that and they're hard into provision because you can't just upload a cloud image at least in the heads and you can't and in vps as you can so and they're also of course kind of hardened to to put into little provisions so this is also why we want to use vps in the future yeah makes sense I guess I guess following up on that there's a question by ebal in regards to vms and their provisioning more or less I can I can read the question for you I'm guessing you're using vms or vps instead of dedicated boxes which we are to some extent so are you using packer to build immutable images based on arch Linux so when you update upgrade these images every week every month or is there like a CICD automation for that right so we don't have any immutable infrastructure currently but we use packer and we use packer to provision the vpses and so the vpses are they are pretty long-lived so we treat them as if they were normal dedicated boxes and so we maintain them and update them just the same but we initially provision them using a packer image that we update a talk once we need a new machine yeah so in the vein of arches install once always update there's a question by arch guests on how do you handle security of the servers in regards to hardening and managing roles and accounts etc sorry can i can you read that again please yeah how do you handle security of the servers this is in regards to hardening and managing roles yeah sure so we use ansible vault for all of that and basically all of our stuff is an ansible vault and it's all in the repository all right cool um do we currently do uh disk encryption on any of the bare metal machines no we are currently trying to set this up john carlo has something in the pipe for that but it's really actually really hard to to really get this right and yeah yeah well we're working on this it's something we're looking into but it's it's really really hard to get right especially because we don't have any machine access directly i have an ansible setup for that if you're interested but i'm not sure if it if it works and yeah it's got the problem though isn't it it works for for my use case currently so i know um i i guess one of the more interesting questions also regards to to hardware and maintenance and so on is what made you choose heads not to host your services rather than another cloud provider that's by dr hashimoto right because well back on the day when we started doing all of this there wasn't that many alternatives really and heads are really was the cheapest by far and we've just kind of grown you know happy with them frankly they provide good services um there's really no reason for us to change yeah cool um i guess the ansible vault question was already answered there was a specific question by peter strudel in in regards to the secrets and so on that's handled by ansible vault um i guess that also leads me to a question that is kind of interesting in general for all of this type of infrastructure question what is the bus factor currently for managing the infrastructure that arge maintains so technically speaking all of the people that were in the list that i had in the other slide i think seven or eight or so they should all have access to all of the service so if something breaks we have at least have those people to be able to manage it now of course there's also the the question of knowledge transfer so not everybody knows how to fix everything and and many services we still are the bus factor of one but it's much better than it was before so still one in many cases um but we're getting there we're getting there that sounds good um let's hop on to the services um there was a question by lurst that's the same person who also asked about the bus factor actually um if we use docker containers for anything uh we do so we use docker containers everywhere on gitlub ci in fact we even built arginalix vm's since very recently we built the vm's inside of docker images using chemo so this is pretty freaky but yeah so we use docker quite a lot um we don't i don't think we use docker in any uh production deployments currently and um probably it's not going to change because we don't really have any reason to change this currently um but yeah it's going to be ci mostly and of course we use docker to build docker images so the official docker images are built inside of docker images yeah docker the docker um i guess something that i also have to look into in the future to fix my stuff with archaisel um there's a question that is probably more tailored towards uh whether dashboards are actually available publicly if i understand this correctly that was asked by kgz uh if there are grafana dashboards in a repo somewhere oh yeah we have a link for that actually so yeah we so all of our infra um or at least you know that's the kind of the idea that everything we do is infrastructure as code and so we also provision 100 of the grafana dashboards that we have as code and so they are available in the repository you can look them up you can set them up yourselves and uh we hope that in the future we are even able to expose some non-critical stats to to others so that is something we're looking into currently hmm yeah good stuff um then we have a question that is more or a set of questions actually that uh tailored around the the entire uh sso um topic uh will there be uh already created forum and accounts and so on get merged uh when we roll out sso well not when we roll out that uh i mean we we've basically rolled it out um there's going to be many steps and i think this whole process is going to take probably years frankly we want to get it right and um it's really hard for us to to make sure that users can really demonstrate that they own an account and we certainly don't want to like or on the on the side of um trusting too much um where it's not like where it's undue so uh we gotta make sure that for all of the services we can find some mechanism where the users can really guarantee that they are that they own this account and only then we will be able to merge so for the like for the time being and for some time to come uh users are gonna have separate accounts i'm sorry for that but at least you know going forward we're gonna have um you know at some point as is all for everything yeah makes sense i guess gotta make sure um are we thinking of switching the forum over to discourse it was a question by uh bittin well so um not discourse in particular we so we put up this to the foremost people and we had them try to kind of decide which forum might carry us until the next decade or something and of course like discourse was one of the choices and for various reasons some people didn't like that and we don't really want to make this like we don't really want to call the shots on that because you know it's it's a community decision afterwards after all and people gotta want to manage and like those forums and use them so we want to don't want to make that shot for them but we have to like i wear my security and maintainability goggles and everybody else like has to also like it so there's no decision currently discourse is one of the choices it's currently not the primary choice um but there's many factors that's coming to this to switch back on a bit of server related topics uh we we have uh mentioned that ansible vault is in use for secret storage and so on um there's a question by night fox or by about uh whether we want to exchange the current secret storage with for instance something like hashy corpse vault um given that we're using terraform sure um we thought about it it's currently not a priority at all maybe i don't know it certainly very much in the backlog okay um also a question by kgz and regards to our primitius setup if we are in need of uh any uh well exporters being written for any of our services still yes someone's offering here yes and we have plenty of projects that we operate and i think pretty much none of them have um like real you know business stats exporters for them so like everything that we have all of the customs over that we need uh have still needs exporters we only have some operation exporters but we really could use some more in-depth software specific exporters yeah nice so kgz if you're hearing this get in touch write some exporters please um we have a few more questions around the encryption of our cloud servers i think you mentioned earlier that none of them are actually at this point right okay so that answers that question as well um and do we apart from the docker containers use any lxc containers for instance uh there's a question by keto any lxc containers lxc yeah or lxc lxc yeah no we don't no okay yeah um we are uh moving to git lab currently uh most of our well internal projects and and so on and uh the question by orhun is if we um if github is used for anything regarding to defops or if it's just uh well our svn to gits um mirror basically for packages and other things right so currently it's only basically the mirror for all of our repositories so we mirror everything or the general ideas to mirror everything from git lab to github and not allow any contributions on github this might change depending on feedback but currently this is the way we use github so basically it's kind of like an outlet for for just giving users maybe a bit more uh kind of like view into the distribution but yeah we currently don't have any kinds of participation plan for using github and i think that is about all the time we have isn't it yeah i think so we got to wrap up i'm sorry if there were uh questions left unanswered you can still hook up hook us up on the isc channel or through the mail if you uh you want to know more yeah i'll certainly be around for any other infrared questions or stuff like that all right


And to entertainment news now, Norley Wood actor Chua-Talu Agu has regained his freedom after spending four days in custody of the Department of State Services. The spokesperson of the actor's Guild of Nigeria, Chinda Monalisa, has released a statement and a picture showing that the actor is now free. Agu was arrested in Anambra last week for wearing a flag of Biafra. It was assaulted by the soldiers who accused him of publicly displaying solidarity for the indigenous people of Biafra. The army released him after 24 hours, but he was then re-arrested by the DSS.


Hi, we're back, Dave Vellante from wikibon.org. I'm here with my co-host John Furrier of SiliconANGLE with SiliconANGLE's continuous coverage of Strata. And we're here with good friend Marco Pacelli. Marco, thanks for coming on theCUBE. Again, you're a two-time alum anyway, maybe three-time losing track here. But you know, this whole conference is all about what you guys are doing. You know, the plumbing is largely irrelevant. You know, we talked about this before last time. If you don't have applications on top of that plumbing to extract value, then you're really not going to get much out of your big data. And that's really what the theme of this conference is, isn't it, John? Yeah, I mean, Marco, you're the CEO of ClickFox, which we know, ClickFox has great IP in essentially figuring out business value.  That value proposition that you've been banging the drum for at least six months that I've known you is materialized in a full-on conference. So the Stratoconference here is about data. But basically, the theme is business value. So it's not only the geeks and the programmers, but you've got the mainstream guys here talking business value. So one, how's that make you feel? You feel valid? Hey, finally, the world gets it. And more importantly, what are the key challenges that you're seeing with the marketplaces that figures out big data is about big value? So hey, Dave, guys, thanks. This is pleasure and I'm just ecstatic about this. This is great. It just makes everyone in our organization happy about the fact of all the hard work we're putting into this past six, seven years is becoming real. And it's just a pleasure. Look, the big data tank is getting deeper and wider. You guys know that. Even on the first USA Today and the cover page of the business section, they were talking about how there's more communication than ever. There's more means of communication than ever that people, productivity levels of people are going downhill because they spend 50% of their time at work communicating to the outside world through different means of communication. Chat, SMS, social networks, emails, the funny thing is is all that data goes into that same tank and it's getting deeper and wider and how do I pull value out of that? How do I? Just total validation right now. It's not about data silos. It's about all data is available data. Correct. So it's not about, okay, this type of data is a database and we just have to see you have clusters on which helps companies scale. So no one really has that answer. I mean, your company is the one that we've seen it does. Still the other stuff, it's like BI, DW, as the guy from Jock was saying, the high priest of, you know, data warehousing. The high data priest and the data tempo. Temple of data warehousing. It's old school. From the accountants. It's totally old school. So I mean. If all you want to do is pull a report out and you're trying to find a static answer to a static issue, great. I mean, that all works fine. Plenty of tools for that. So can you talk about some of your customers? I know you have a lot of confidential customers and they're big names. Can you talk about them one? And talk about the use cases. Or if you can't talk about them, talk about the use cases where you guys actually use your, the people use your product for the value. We have a couple of the biggest financial firms in the country. One of them has had billions of dollars. Names? Is there names? I can't put names on those right now, but we have all the wireless providers. I will put a name on one of them. Sprint. Sprint, you know, was recognized five years ago as having a gigantic customer issues that they didn't understand their customers, didn't care about their customers. And to get an answer out of their Sprint to resolve an issue was, you know, it took forever. Today, actually in 2010, they got awarded Best in Class by Gartner in Customer Care, Customer Service. And it was because they spent five years pulling out the right value out of their data, tying the experiences and interactions to their presentation. How do they do that? I mean, so how do they do it? How do they go from, you know, zero to hero? They took a 360 degree view of the experiences customers were having across their data, across their touches, across their products. So they use your product and technology. Absolutely. And for the folks out there, just describe the ClickVox algorithm or product under the covers in general terms. The concept is there's a pattern to everything in life. You know, whether you're walking down a street and seeing things or walking into a restaurant or if you're looking, or if you have a very wide deep tank of data, somewhere in that there's a pattern. It's finding that pattern, connecting it together and determining if the pattern that you find is the right pattern for that customer and for that product to be serviced as. So our technology out of unstructured data or structured finds a pattern, links it together, analyzes it and determines whether that pattern has the best effect for the product and the customer to give you the ultimate return. And the return could be the highest customer satisfaction for the best experiences, could be the lowest cost to service, it could be the highest return in sales and product adoption, it could be applied to anything. So you measure and monitor that end-to-end customer experience, which is very unique. A 360-degree view of product and customer experience. The product is also a customer in the data world. So now, we were talking to Tim O'Reilly before about the whole issue of privacy, right? That comes up a lot in this world. He said, you know, basically give it up, right? I mean, there really isn't a lot of privacy in the internet. And it's, to everybody's good if we're actually sharing data. So, and there's really no way to sort of adjudicate this. And so I flipped it on him and said, well, what about my data? I mean, if I'm having a customer experience I'm clicking online, you know, shouldn't I be able to see that data? And he said something interesting. He goes, yeah, I guess. He goes, but don't think of it that way. Think about it as if you can get access to that data and it's of value, you might be willing to pay for that. Or if you might be willing to give that data up, do you see the day where I as a consumer will be able to have some kind of navigator or dashboard that shows me what my customer experience is across all these touch points? Is that a possibility? That should be available now. It should be in the cloud. It should follow you around to everywhere you go. I want that. I mean, and you should not have to recreate experiences. You should be able to click on past experiences and have them be driven for you automatically. I mean, you don't have any more hours in the day. It's 24 hours. You have no more hours left, right? So anything you can do better, quicker and not have to even do will make your life easier. Yeah, I'm already working half day every day. 12 hours. That's good, let me know how you do that.  So Marco, Marco, you're an entrepreneur.  Five, six times?  Yes. You're doing a lot of traveling, you know, and see you're well-dressing from Italy, you know, handsome entrepreneur. Is he asking me on that date? What is, you're very popular and the online numbers are going through the roof. What is your view of the world right now? I mean, we saw you in London for the EMC thing. You were doing some business out there. The data is a global phenomenon. It's like, what's going on around the world? Just give us insight from your perspective. As an entrepreneur, not as a CEO, as you go out and talk to customers in this data area. For folks out there, there are people trying to figure out what to work on. Should I solve the healthcare problem? Do I do this? So, you know, just share for the folks out there what you are seeing right now for opportunities. I'll tell you what, it all comes down to executives are tired of making the wrong decisions. And somewhere in that data exists what I call the business case engine. Is the answer to what I'm supposed to do and the decisions I'm supposed to make to drive my company to where my shareholders and my customers wanted to go. And if I can't get that answer and determine the next step and see the effects of that step instantly, not wait 90 days to see the effects, but instantly see them the day next, then I can change my business. And this is a common theme all over the world is, you know, if you look at the top three banks in this country, Chase, B of A, and Wells Fargo, right? Or City, if you want to put them in the same category, you know, each one owns one of three clients. But yet they have data on every customer in the country. The content of the information they have and what they can do with that information can change the economic status from day to day, right? The question is, how do they leverage that information to make their company better, the shareholders happy or the customers happy and provide a better product than the outcome, right? It's all there. They got to find that needle in the haystack. And it's not about reporting. It's really, the data should work for you. I mean, the guy from O'Reilly who put this whole program together, I asked him a specific question.  But he said, you know, what's surprising? And he said that most people don't know what questions to ask. Okay, so, which makes a lot of sense because you now have no available data. What can you share with folks about what that means? What question to ask? What does it mean? And have you identified any questions that people are asking themselves that they're working on? I'll tell you, if you do it right and you are able to tie and analyze the data correctly, the answer to every question you could possibly come up with is already answered for you. And that's the way it should work. You know, all the answers should be there for you. It shouldn't be a question of what I need to ask. It should be which answer do I want to retrieve out of the data. Because the answers are there. It's just getting them out. You shouldn't have to ask. You shouldn't have to generate a report. You shouldn't have to go out. So, you're an answer engine? Yeah, pretty much. Yeah, that's a good way to put a data answer engine. So, how do they know what answers to ask? How do they know what answers they want? I mean, it's almost that elementary. The answers are, what's today's issue? Is today's issue that I lost a million clients? Tell me why. Tell me why I lost a million clients. I mean, try to get that answer out of a data warehouse. And tell me what I have to do to make it better. Show me where the ROI is, right? Or better yet. In fact, why don't you just show me why I lost a million clients today? I shouldn't even ask the question, right? Just automatically tell me or show me. Hey, tap me on the shoulder and say, guess what? This is why you lost a million clients. And another big trend. So, now I know exactly what to attack, what to fix, and I'll get an instant ROI. And give me the root cause. So, if I know the root cause and I know I lost a million clients and here's the root cause, I know what to fix. I can fix it tomorrow. So, you show your customers where they can invest to fix the problem and get an ROI, right? Yes. I mean, the concept is root cause, surface the root cause and build the business case engine to make the decision on what to do. What do you not do that people need you to do that you're going to be sort of working on in the next year or so? What we would like to get to from a point of view of the value of the data is to drive the change automatically. If you, if we can get to a point in the industry where the retrieval and analytics of data becomes real time, and that's really a factor around the processing power of technology and the direction it's going in. But if we can get the data real time, extract it real time, analyze it real time, we should be able to give a real time decision back and we should create a real time change in what you're about to experience. For instance, use the analogy, you're about to walk from point A to point X, or point Z, right? And there is a certain path you're going to take. Before you end up at a certain spot which I know may be dangerous, I'm going to change your path because of the way I'm analyzing your steps, the time, the temperature, the road. So I'm going to change your pattern of behavior. I'm going to change your experience you're about to have on the fly to give you a better and safer experience. Same concept with anything you do in business is if I analyze the data real time and I provide real time feedback, I can provide a change about what you're about to do. Maybe give you the answer before you ask for it. Maybe change the next step you're about to take because I know you won't enjoy the next step you're about to take and direct you to somewhere else. But I have to be able to provide that real time and make the changes to your steps you're about to take real time. So you're going to be anticipatory. Now you've already compressed this, we've talked about this before, you've compressed the cycle times dramatically. Correct. And we said earlier at the top, plumbing is sort of irrelevant, but it only is irrelevant if it works the way you want it to work. And it wasn't maybe a couple of years ago, we talked about green plum and holly, your load times accelerated, green plum is a, we got a shout out to them, John, because they're sponsoring our bandwidth. Yeah, they're paying help with the bandwidth cost. So they've been great here. I know you're a customer of green plum, so you're very happy there. And that's about the load time. We love green plum. Getting closer to real time, right? Yeah, yeah, yeah, that helps us extract data extremely fast, load it really fast, yes. Yeah, so, okay, so that's good. And now the other thing that we're hearing at this conference is it's all about sort of the individual, from e-business to me-business. That's really the focus, and that's really what ClickFox is all about, right? Correct, correct. Is how to improve that customer experience. Right, feed it back to you.   I mean, for companies, you talk about like the wireless companies in my experience. I mean, come on, who doesn't want to know what their churn is, the behavior of the customer. I mean, consumer behavior has been this, you know, science and art, kind of like advertising, you know. It happens, there's theories, 50% of it we don't know exactly. So, but we're going to an online world, you're delivering that. Correct. I mean, do you have guys just saying, hey, you know, we love this, we want to re-up, do more? I mean, when they first taste the success, what do they do? I mean, they must be blown away. Do they challenge the data? They go, they want more. They want more. I mean, you know, you probably have favorite restaurants to go to, right? You can probably walk in and get a table and you probably get the table you want. And by the time you get to your table, maybe the host has already gotten your favorite drink there and your favorite appetizer. And you've had a great experience. The reason why is because that owner knows you and knows what you like, what makes a good experience versus a bad experience. And you always have, they can depend on your business certain times a month, right? So now take that and extrapolate it across the world and all the different businesses that you interact with. And if every business knew that you personally liked that and knew what was a good experience versus a bad, wouldn't it make your life better? Well, certainly it's great for the user, but it's scary for the business because I think about all the re-engineering they got to do now. Not much. I mean, they've made some business assumptions based upon flawed data. Actually, all they need to do is get the value out of the data and change the way they treat you versus me. And that's simple. You're in the user experience business. Basically is what you're in. User satisfaction, yeah. You know what's missing, John? I want a dashboard. I want to see my experiences so I can compare them across all my service providers. Right, don't you want that? I want a leaderboard. Yeah, me business. Right, I mean, maybe it's the JD powers. Maybe we should start that. So we can bring the log post. Seriously, I mean, that to me is. That'd be a big business for you. Really, I mean, I think that that's empowering the consumer to really understand those experiences and make choices based on those experiences. Mark Pacelli from ClickFox. Great to have you on theCUBE again. We alumni again for the third time. Appreciate it. Great to see you, man. ClickFox is really killing it providing a really innovative product to large, large clients around understanding consumer behavior and delivering solutions to keep customers. So congratulations. Thanks, guys. All right, good to see you.


Thank you. So I'm Paulo Grady. I'm going to talk about PyTorch and Autograd. Just a little bit about me. Currently I'm a data scientist at Zalando in Dublin. This is Zalando, our online fashion retailer. I'm also a member of the PyCon Ireland organizing committee. We'll have our conference in October this year if you're interested in coming to Dublin. I'm also on Twitter, so if you'd like to follow me. So just a quick overview what I'll talk about today. So I'll just give a quick background on matrix computations and deep learning, that sort of stuff. And then I'll get into PyTorch, tensors and variables. I'll talk about Autograd. So Autograd is actually a module within PyTorch. And I'll discuss gradients and back propagation, that stuff. I'll then give a simple flow world example of how to use the framework. Just simple linear regression. And then I will illustrate differences between a define by run framework, which is what PyTorch is, and a define and run framework. And I'll do that at the end. Okay, so just a quick background. So before deep learning and GPUs and frameworks around GPUs, obviously all computation was basically constrained to CPUs. And the sorts of tools you would have been using would have been stuff like, well, we still use stuff like BLAS, which basically give you simple matrix computation routines and LAPAC, which would give you routines for solving linear systems, stuff like that. And then in Python, you'd be using NumPy or probably familiar with NumPy. So all these tools are all based around performing computations on the CPU. And yeah, so all support, well, NumPy supports, say, multi-dimensional arrays and arrays. But at that time, there wasn't great support for like tensor operations and stuff like that. And so with deep learning, probably you've heard of that. Basically, the models here are quite large and you need a different paradigm for computing the models and updating the gradients and all that stuff. And so people started using GPUs that allowed that basically enable deep learning. And so basically here we use tensors and our number of frameworks, they probably know TensorFlow and Tiano as well as PyTorch will talk about now. But yeah, so we do the computation on GPU because it's an order of magnitude faster. And so as I said, there are a number of different frameworks, TensorFlow, Tiano. And another thing that the frameworks offer is in addition to doing your computation on the GPU, it would have support or features for calculating gradients, either symbolic or automatic differentiation. And that's important for machine learning algorithms because you usually have to calculate the gradients to feed into an optimization algorithm. And before these tools, you'd do that by hand. So you'd pen and paper, calculate what the gradients are for your update rule. And now there are tools that will do that for you just from the cause function. Okay, so the great thing about all this stuff is that Python's at the center of this. All those frameworks I mentioned have bindings for Python. And I suppose the new kid in the block is now PyTorch. That's what I'll talk about. So PyTorch. So I suppose the first thing to mention about PyTorch is, Torch itself has been around a long time. It's a Lua framework. So there's a lot of great work within that community that has been done to build Torch. Obviously PyTorch has bindings around us, and they still share common C libraries stuff like that. And Lua Torch is still its own separate project as well. So PyTorch is quite a new framework. I think beta release was in January. And I suppose the key difference between PyTorch and other frameworks is that's defined by run framework. I suppose to define and run. We'll discuss that later on. But to me, and one of the reasons, well, the reason I like PyTorch is, it looks more like programming as you know, but it feels to me to be more Pytonic. So I feel with other frameworks, you kind of feel that there's a bit of setup going on before you actually get to do your training, which is fine. But I think with PyTorch, it just feels a little bit nicer. So the main components here, PyTorch are the NN module, and that allows you to build and train neural networks. That's what deep learning is. So these neural networks can become very complicated with many different layers, many different types of modules and layers. And PyTorch is basically used at the cutting edge of research, so it supports all that stuff. And nothing but PyTorch is it has autograd, which has a nice support for automatic differentiation. I'll talk a bit about that. And then once you have your gradients from your AD, you can then use that to update your model parameters for your deep learning network or whatever, or linear regression using the optimization routines in the framework.  So the version of PyTorch that these examples are for is <IP_ADDRESS>, and it's running on a Python 3.5. So just give you a kind of a quick overview of the status of the project. So this is a tweet that I've seen recently on Twitter. It basically shows the aggregate activity on GitHub repos for deep learning libraries. PyTorch is a fifth there, which is quite good. It's quite a young project. It's probably obviously at the top there are TensorFlow, you have Google and big companies behind that one, but still, even after a few months, it's doing very, very well at fifth spot there. So first talk about tensors, and I have a simple example here. So just to be clear, we're talking about matrix multiplication stuff like that. You would not use native Python lists to do matrix multiplications, I'll just state that. You can model a matrix using a list of lists, but you would not implement it that way because, well, you can do, but it's very inefficient. So lists, as you know, can contain arbitrary items, arbitrary types. So there's a lot of overheads on the kind of memory handling of that that you would want to avoid when you're doing big computational jobs. So everyone, you're always going to use a different framework or module, so you might use NumPy. So the way TensorFlow handles, sorry, PyTorch handles that is, we use tensors. So these tensors can be initialized, as you can see here with lists, but the underlying kind of memory, the way they're mapped, mirrors more, say, a C programming array where the individual items are continuously laid out, but it's just, yeah, that's the foundation of these types of data structures. So if you're used to looking at NumPy, once you have your tensor or end array, you can perform many different types of operations and extract many different types of information, so you can determine, say, the size of the tensor. Obviously you're going to add, you're going to add in place, now the place. So here's just a simple example where I set up a tensor of ones, add two tensors of ones, then I do an in-place add. So PyTorch supports in-place operations, and they are identified by an underscore at the end of the method name. So, yeah. Yeah, so I mentioned NumPy, so you may be in a situation where you have your data already prepared or transformed or whatever using NumPy, and you want to get that into these PyTorch tensors, and the simple way of doing that is it supports, like, a NumPy bridge. So here I have a Y underscore NP NumPy array, which I've defined, and I can multiply it by the previous float tensor that I created by just specifying this NumPy method on the tensor, and the simplest dash. You look at the result and it's basically a NumPy array. So that's handy, so it's very easy to include NumPy using PyTorch. You can also go the other way around. So we have, we want to multiply, matrix multiply, two tensors. We set up a Z, which is a NumPy array, and then using Torch from NumPy, you can create a tensor, and then, you know, you get the result. So two by three, multiply by three by two, it gives you a two by two matrix, and because we're on short, basically you get a Torch dot short tensor. So that's all handy stuff, that's all good stuff. Let's go back from the framework. Yeah, other operations you can do is you can create, you can change the shapes of your tensors, and that'd be expected. Another nice thing about PyTorch is you can be very explicit about where the computation happens, and so obviously I mentioned the GPU, and that's where you do most of your computation. But you can explicitly say that, so you can set up your variable, sorry, your tensor, and then you can specify that should be on the GPU. Likewise, you can say, I would like to move that to the CPU. So it's very explicit, you know where things are being moved to. You'd be careful with your copies and stuff like that, but it's just nice to be explicit like that, and that's a nice feature. So another important component of PyTorch is variables. So variables are imported from the Autograd package, and basically a variable is a tin wrapper around the tensors.  So for machine learning and deep learning, it's important to know the history of how a value was calculated. So that suggests like a DAG directed acyclic graph or some sort of computation graph, and that's basically what the variable does. So the variable will keep a history of how that value was being calculated and basically gives you the structure of the graph dash was used to create dash. So that's useful when it comes to the AD stuff, and it's also, so that's a nice feature. Yeah, so as I said, that will allow you to facilitate the back propagation of gradients, and basically supports the AD stuff. So yeah, I just want to mention that, so all the slides I'll be talking about here, they're all in the context of machine learning of the learning step. Once you have your model learned, you don't need any of this back propagation gradient stuff, so you can still use the variables, but the way to specify that you don't need a gradient stuff and you just want to run in inference time or test time is to set the volatile flag on your variable. So you do that if you're in production with the model. So here's just a simple example of a variable. So it's important for a model grad. You can see that it's initialized with a tensor. It's just got two simple tensors here, X and Y. I then add them together to get Z, and then I just echo out the values starting Z after the addition. So as I said, variables keep history, so there's an additional property on the variable, the Z variable, because it's an autograd variable, and that's called creator, which is this one here. And that basically tells you the last operation that was performed to actually calculate the result. And in this case, it just shows you it's an addition. Then I can take Z and do, say, sum operation on the contents of that tensor and assign that to S. I can do the same, have a look at the creator for S, and you can see that identifies that last operation was a sum. So this is using this creator property. So that's an interesting thing. So with the creator, it gives you a reference for a data structure where you can actually chase the references and construct the graph for yourself. So I have a simple course of function here that basically prints out the different operations and the data when it hits the data. And by passing the S dot creator, you can see the different operations that have been applied to S. So S here I'm doing it in addition to sign, and you can see that force operations add constant, then the sum, then the add, and then it shows that you actually display the data that was there. So that's a nice feature. So you can imagine for big deep learning networks, you could create a graph like this, more fancy graph, saying with, say, a dot library. But it also keeps a version. So each time the variable S is mutated, it basically increments this version number, and that's used in the machinery of the autograd. So that's a nice feature as well. Okay, so I'll just talk quickly about autograd. Yeah, so autograd provides classes and functions that implement automatic differentiation of arbitrary scalar-valued functions. So scalar-valued functions, so you're usually talking about cost functions, loss functions, stuff like that, and they're usually scalar-valued. So that's why it's highlighted there. And just a quick note that the type of AD that's implemented with autograd is reverse-mode autodifferentiation. Okay, so this autograd module is actually, so obviously it's coming through LuaTorch, but that was actually originally inspired by another Python project called autograd. So it's gone from Python into Lua, and now full circle back into Python through PyTorch. So the define by run, part time is inspired by the chain or framework. Also in the manual, the docs for PyTorch mentioned that the implementation for autograd is pretty quick, which is great. Yeah, and so to implement the back propagation, the only change of the ad is just basically change your tensors to variables, and it's a very, very simple thing to do. Okay, so we're going to talk about back propagation, and we're going to talk about autograd, automatic differentiation, all that stuff. I'll just quickly discuss calculus or differential calculus, and just to kind of illustrate why we need to do this. So differential calculus. Okay, so you may or may not remember from school or college, so basically you have a function. So this equation here is to calculate a derivative from force principles, but basically you have a function and you would like to know, you would like to know the gradient of a tangent at that point. Okay, so that's where the gradients come into it. So to do that, you take your original function, form differential calculus, and you get a derivative. And then using the initial x input, using the new derivative, which is also a function, you just basically apply that into the function, and whatever value that comes out of that basically tells you what the gradient is or basically the slope of this line. So we're really talking about points on a curved function and calculating gradient for that, basically a slope of a line. And then we use these slopes or gradients to indicate something of value to us. And in terms of machine learning and optimization, this allows us to identify extreme functions or basically maximum points and minimum points. And the way we do that is by looking at the gradients for the functions. And that's why we use differential calculus. So back propagation, yeah, so back propagation, because we're talking about networks and multi-layer networks, it's difficult to calculate derivatives of those type of networks. So there's an algorithm that will do that. It's a famous one you probably know about. It's a back propagation. So that will calculate the gradients of your model parameters with respect to the last function and will push back the gradients through the network and allow you to update or change the model parameters to make it a better solution. So the rule that we use to actually achieve this is the chain rule. It's a standard rule in differentiation. And back propagation basically iteratively applies that to all the different layers and that's how it pushes the gradients back through the network and allows you to update parameters. So AutoGrad does this for us, it does the back propagation. So we're talking about AutoGrad, we're talking about back propagation and it's simply implemented using just a backward method on our variables. So the stuff that we've been doing with the variables, that's setting us up now for calculating derivatives and doing back prop. So just as a simple example or just to convince yourself that the derivative of a sine function is a cos function and I'll just illustrate or try and convince myself that back propagation will actually give me what I expect which is cos. So I just set up a simple x variable which is basically one period of a sine wave so from zero to two pi and then I calculate the sine of that signal or variable using just torch.syn which obviously goes into out. So another feature of the variables is you have an additional grad property which will echo out the actual gradients for that variable if they were calculated at this point they have not been calculated so returns nothing but at this point I calculate the gradients by calling the backward method and at that point I calculate the gradients for you. So this will basically calculate dx so the next time you look at the grad you will see that there are gradients there so first what I do is I just look at the out I convert it to number just to make it a bit easier to read and for the inputs which is zero to two pi I have a look at what the out is and we get zero one zero minus one zero and that's basically a sine then by looking at the gradients we can see the same and we can see that's changed one zero minus one zero one so that's basically the cause so we've convinced ourselves now that it's performed calculated gradients for us and we just do another check where we compare it back to cause so that's great we've convinced ourselves that it worked and we're happy with all that just another simple example actually no sorry first so how does all this work so basically deep in the code of PyTorch the sine sine function is actually an instance of this sine class which basically inherits from a function which is imported from AutoGrad so you can see two methods here we can see the sine function in the forward method but we can also see backward and we look here closely we see that it basically just on the backward method it just calls cause which is the correct thing to do we know in advance that the derivative for a sine is cause and basically that's what we do so if you've looked at calculus before you know the standard tables for different functions and derivatives we basically do that that specifies the gradient formula that gives you a gradient and if you actually wanted to extend PyTorch well then this is the pattern you'd use you'd subclass from function and you'd create a class where you'd have your forward method and then you'd define the gradient function backward method so just another simple example I just wanted to display a tangent something similar to four I create a X variable which is a linear space note that it requires grad is equal to true is here then I basically set up the equation quadratic equation and I'm only interested in looking at a gradient for one point within our linear space there and I set that up and then finally on Y I call the backward function using the gradient style set up here and so the main thing we're interested in is the value of the gradient at that point I've identified which is indicated by this index here and so I know that the value here should be minus four again minus 9.3 because of the discretization and from this information I can calculate gradient so I'm just now going to plot that and just plot the original quadratic function and the gradient and tangent so you can see just confirms again what we'd expect we have our point minus one on our quadratic function and we have a nice tangent there you can see that in the plush that's great okay so I'll just finish off with a simple low roll example to mention so just going to talk about linear regression linear regression is basically you have a cloud of data points and you're going to try your best to fit a line to that cloud of data points and try to minimize the distance between the line and all the points close to the line so the model for this is quite simple it's just a linear relationship so y equals alpha x plus beta y, sorry alpha is a weighting and b is an intercept x is called independent variable and y is a dependent variable so this model is actually already implemented in point torch you don't have to implement this itself and that's in the nn module you can see the types of models like convenets and whatever other sort of stuff like that but that's our model for this and this is what our data looks like so I've created this for I've created a noisy example so I've introduced some noise there alpha is 2 and beta is 3 so you can see the interceptors around 3 okay so just quickly so we've been talking about models that's great but you need other things as well if you're interested in learning you need a cost function or I'll discuss that but you also need to identify or specify a learning algorithm for it to update your model parameters so the cost function you use is pretty standard one mean square error so the value for this cost function would be zero when you're predicted values and your actual values are exactly the same and that's what you want this is a very common one so this is where we use our gradients so we have here the gradients of our cost function which is MSE with respect to the model parameters and then those gradients are used within the optimization algorithm or to cast the gradient set algorithm so it's a further step after you've calculated your gradients that's needed okay so I'll just show you the code so setup this is how I set up my data my x and y and basically we set up our linear regression model so this is how you set up a model in PyTorch in it you will specify the different layers or modules that you need in our case it's just a simple example one layer just a linear and then in the forward pass you would describe how these different layers are combined for us again it's very simple but you can imagine for like very very big with a lot of layers and a lot of different interactions between how the forward method is specified then finally we just instantiate it and because we're only interested in one dimensional regression we just specify that then in the model so I mentioned we need a cost function I said we'll use MSE I specified that here and then I also specified cast a gradient descent and the learning rate and then I also specified that here we have our training data that's basically the collaborative points I showed so before we do any training you can actually inspect the model parameters see what the values are so on the model there's a method called name parameters and that's basically a generator of the different parameters so here we can see there's a weight which is our alpha and we can see a bias which is our beta and then they'll be changed and then we'll have our line for the data so this is the main training loop for it to actually train the model so this is the way PyTorch does its training within loops so the first step is five steps here and the first step is to zero the gradients on the model parameters so the gradients actually accumulate and if you don't zero them they'll just continue to accumulate second step is to take your model take your inputs and then take the output so you're just basically running the model on the input then from your outputs on the towards step you can compare them then to the labels using the criterion and this will give you your loss so this loss is a scalar so when you look at convergence plots in machine learning this is basically where you're plotting this single value and using this loss you can see that there's a variation two dash by calling backward so once you have your gradients you can then update the model parameters using the optimizer and you do that by calling step on the optimizer so you can see here within the loop there's a lot of things going on the steps are very explicit and this is defined by run type paradigm so just keep that in mind we can change in our model parameters another feature of the model is that you can actually introspect on the different layers so I just pretty print the individual layers of our model which is simple in regression and then I plot the line so here we can see this line fits pretty well to the original data and we can say that our model has now learned from the data so I mentioned that in the previous example we set up that same problem to Tiano which is another deep learning framework so here in Tiano we import Tiano we set up in some way to the previous example we set up our X and Y data it deviates in the way that calculates gradients so you have to set up some bollock variables so similar to PyTorch you have to set up your cost function then you have to calculate your gradients you can see this is done in a very different way and you specify your updates and then you have this step where you take all this stuff and you basically compile this all this information into a Python function so then the final thing to do is you can see that it's very sparse it's just basically a one-liner so that contrasts a lot with PyTorch so there's no opportunity here to kind of make changes within your learning loop so there's no kind of dynamic possibility to introduce dynamic behavior here whereas in PyTorch within iteration you could have make changes within each iteration so here Tiano basically creates the computation graph before training and PyTorch does it at each iteration so just to finish up so I discussed PyTorch and NumPyBridge tensors and stuff like that also discussed graphs and variables talk about gradients on AutoGrad get a simple example and then at the end here define by run and run so that's me finished now I don't know if you have any time for questions well thank you very much Paul it was a great talk unfortunately we do not have time for questions to 30 exactly but right on time thank you very much again let's thank the speaker again


So, ladies and gentlemen, we have done it. We have created a sexually mature adult female 3D base model. Fully weight-painted, rigged and ready to go, with 148 total bones, 22 correction bones, 49 correction drivers, 59 mesh correctional shape keys, 59 shape key drivers, 49 poses and four organized libraries, with hair physics, jiggle physics, and color-coordinated custom-designed easy-to-see bone shapes for the left and right side of the body. Ladies and gentlemen, she's all yours. In the next few videos, I'll show you how to modify the rig and transfer the weight-painted data from Blenda to your own character, the same way that Rigify does. But in the meanwhile, feel free to reverse engineer it, steal pieces of it, or use the rig for whatever you want. The creation of this rig and the tutorials teaching you how to build it were all funded by the brave names you see here on the screen.  Corona hit us pretty hard last month, and a few members of the squad lost their jobs and they weren't able to support anymore. Some of you even messaged me about your situation and tried apologizing for it. To anyone in this situation, I just wanted to say, please don't worry about it. The last thing that I would ever want for anyone supporting the channel is to be starving in the background. Seriously, guys, that's not cool. So please just stay safe and do whatever you gotta do to keep your life intact. I'm more than happy to just have you guys here enjoying the show. Simply leaving a like or a funny comment for me to read is already more than I could ever ask for. And because I don't think it's fair to blame someone for being in this kind of shitty situation, this month I'm not gonna take anyone's name off the website. Thank you for everything up to now, and I hope you guys are staying safe and hanging in there till this is all over. Also, somehow, we officially had 10,000 new faces joining the last 16 days. So if you're new here, welcome to the channel and glad to have you aboard. Unfortunately, those of you who've been here for a while know that YouTube's been real rough for me. It's actually gotten worse. See this? This is a list of all the words that get you demonetized on YouTube, which means you can't have these words appear in the title, description, thumbnail, or tags without getting hit in the bank. You can't even say the words because auto-caption will usually pick it up and hit you from behind.  And unfortunately for us, the last three months I've been trying to teach you how to rig the human body. And as it turns out, teaching 3D artists how to recreate the human body is apparently wildly inappropriate. You remember the download video for Blanda? Well, that got hit with demonetization for having sexual content. Not even joking. Every other video, my thumbnails get flagged for nudity, which is kind of funny because the censored version of the thumbnails often look even more provocative than the original. Rigging the human body is hard.  That is why I started the series in the first place.  It's even harder when you're not allowed to talk about the human body or use words describing the human body in your tags. Like, let's be real here. Do you really think that anyone who's looking for this information is going into the search bar and typing in buttocks rigging or tush cheek animation or weight painting the gluteus maximus? No! Nobody talks like that. Nobody thinks like that. If people want to find the stuff then they type in how to rig ass. Realistic boobs, breast animation. That is how people talk.  But I can't use any of those words in my tags without getting flagged like a match between red vs blue. So, for those of you who are leaving a like or dropping a comment after each video, you are my hero. And you're the only thing keeping this channel above the water. Because trying to stay alive on YouTube is like trying to keep a fire alive on top of water in a snowstorm with nothing but friendship. So, thank you.  Recently, I just checked my spam comments folder and there was like a couple hundred comments there and most of them were obviously not spam. Usually it's just people trying to support the show. So, if any of you left a comment and wondered why it's not showing up, please just know that I didn't delete it.  Anyway, now that the rigging series is complete, we will now be moving to the animation series. Specifically, creating animations in preparation for game design. This is the series that I have been looking forward to all year. Character animation is my favorite part of the 3D process. So I am super excited to finally share this stuff with you. And in preparation for this series, I'm gonna spend the next few days designing the main character that we will be using in our game. So, the next time you see Blender, you might not recognize it. But I think you're gonna like what you see. Anyway, thanks for watching and hope you had fun. As always, hope you have a fantastic day and I'll see you around.


See, the dimensional levels are separated by 90 degrees. Musical notes are separated by 90 degrees. The chakras are also 90 degrees. It's a number that continually appears. Probably at this point, we should get a common understanding of what dimensions are, like third dimension, fourth dimension, fifth dimension. What are we talking about here? Most people think of dimension in terms of time and space, the x, y, and z axis, with time becoming the fourth dimension. This is not what we are talking about. What we're seeing as the various dimensional levels has more to do with music and harmonics than anything else. A piano has eight white keys from C to C, which is the familiar octave, and in between there are five black keys. Together, this creates what is called the chromatic scale, which is 13 notes, well, actually 12, with the 13th being the first in the next octave. So from one C to the next is really 13 steps, not eight. This also correlates with the seven, eight, and 13 chakras system I mentioned in lesson two.