Okay, let's welcome Martin Kraft. He will talk about recursive inventory management and this will be a subject that is interesting to all of us who do System automation so welcome Martin Hello, is this good with microphone excellent Yeah, I'm here to talk to you about recursive inventory Management, and I don't assume that everybody of you knows what I mean with that just from the title as As a matter of fact, maybe I can say up front before I go into my slides that the name recursive in this is Born out of a very very long night in Switzerland actually And I should have chosen a better name, but I guess it's stuck. So whenever you see recursive in the Following couple of slides you can think of hierarchical if you want Back then when I was implementing it I did it recursively. So that's why I called it recursive But we shouldn't expose too much of the internal implementation designs, right? anyway I'm going to be talking about configuration management and because that is also a name that was used for some time For what is known now as version control. I Specifically want to say that this is about system administration. So if you hear from version control, then actually that's not the right place I'm configuration management as I understand it as most of you will understand it I'm sorry Thomas that phi is not on there And is something to do with one of these or some other tools you will have probably seen Possibly used some of these tools at some point in time. They are listed here in somewhat order of increasing or decreasing age So see if engine predates pretty much all and one of the latest newcomers has been ansible and salt And I'm actually going to not be talking about see if engine BCF G2 and chef But only about puppet salt and ansible and actually not even about those But I can tell you that what I'm about to tell you works with those three products When I talk about configuration management, I also mean system automation. I mean that My assumption is that every single system administrator is lazy or if they are not they should be You should be able to do something once and not have to repeat the same task all the time As a matter of fact if you did the second time then you've done something wrong That is sort of the mantra of system administration and over the last couple of years I have noticed that there's a very strong divide. Well, maybe it's not that strong But it certainly divide into two separate approaches to system configuration management or automation The latest type has of course been the cloud and we see that this conference is already very cloud Centric in some ways a lot of the talks are about clouds So obviously I had to put that onto my slides. Otherwise, it would be completely off-topic But there is also and I beg you all not to forget the sort of classical approach to system administration If I identify two approaches, I better go ahead and sort of like separate them for you Because maybe I'm the only one that perceives it that way And so I just kind of just want to use the opportunity to give you an idea of what I mean the cloud is sort of What what exists when you have buzzwords that could range from unprecedented ease of system optimization to orchestration of course cloud is all about scalability And because you might have a heard an example yesterday used you might know that your television ad is about to go live So you need to be able to spawn very quickly ad hoc provision very quickly a thousand new notes The one thing that you know about all these notes is that they are going to be homogeneous They're all going to be exactly the same more or less, right? But they are all web servers that are just designed to handle the requests by your potential customers and generate you money System administration classical system administration is a lot older than this concept of the cloud and System administrators are as you all know those grumpy people that are running around in the university building Actually, they're never running around right they enter it early in the morning or leave early in whatever You never see them and when you go approach them you get a bf bofh excuse usually right when When the system doesn't work, but fundamentally What these system administrators are doing are they are maintaining systems that have some sort of longevity so these systems are installed at one point in time and Ideally run for a very long time until the hardware breaks at which point you try to move that exact same system because you don't want to Reconfigure your LDAP and mail server at this point after the hardware crash You just want to have the same system up So you're dealing with longevity across heterogeneous systems. You have your mail servers You have your web servers. You have your computation cluster. You have whatever, you know better than I do in your use cases The real difference between The cloud provisioning and the classical system administration is the following We use the classical system administrators use themed host names whereas cloud provisioners use WW001 through 999 so a Little bit of my mission today is to bring back the themed host names. I Just put that in there laziness I already Mentioned and I did have unprecedented ease on the last slide Obviously, if you want to spawn a thousand systems and you pay by the minute then it should be really easy and quick to do Laziness is somewhat different from that laziness is you don't have a thousand systems You have 10 but they give you nightmares or wake you up at night or actually cause you to stop your vacation early and All of that would be much easier if you could just lazily press a button on your smartphone And like from the Caribbean islands have it all be fixed I'm not promising that that's gonna work and There's also The concept of orchestration and classical system administration. I'm just mentioning that here because I you know I don't want to rule out that we don't orchestrate if you have a nagios Installation or munin or even post fix with client certificates You need to know about the other nodes on one hold node So in some ways you would like to make sure that the mail server will accept mail send with a certain client certificate Before you tell that client to send mail with that client certificate Obviously if it's five minutes plus or minus and some things don't work That's okay for the use case. I guess you can do something against that if you want But we also do orchestrate we as in the classical system administrators so What I'm going to be talking about today it comes very strongly as you may have noticed from my speaking from the perspective of the classical system administrator But it will Potentially if I do a good job or if you take the time to look at it suit your cloud needs as well So it is not something that is exclusive to classical system administration It might actually be something that revolutionizes the way you think about and Do system administration on your cloud? I think one of core differences between these configuration management systems that I've introduced puppet and ansible and salt Which also relates to the distinction between the classical system administrator and the cloud provisioner? I should invade actually The cloud provisioner is in my opinion the one that thinks about system automatic Automation in a way of targeting configuration at nodes Whereas I think that the classical system administrator thinks about classifying a specific host now the distinction between those two is not at all clear-cut and I'm really hoping for your patience here and for your understanding that you will just let me get by with us because I'm not willing to answer questions about terminology But let me give you examples of what I mean When you do cloud provisioning with the tools such as salt and ansible These are the latest newcomers to the game and they are very much targeted at the cloud and whereas puppet sort of like Existed before the cloud came and then try to Implement cloud-like behavior or features into its feature set When you when you come at the cloud provisioning problem from these tools Perspectives then what you're doing is you're saying my mail servers are expected to work such and such Have port 25 open on their firewalls for instance and then you say this should be applied and Obviously I have my systems or cloud somewhere. I should have made that a cloud.  I Will target my Debian specific Configuration to all of the nodes that export the fact I am Debian or has some sort of Debian related fact Fact just very quickly is something that is generally Generated on the node and then export it back to the system so that you have some knowledge about what the node is like Which system is it running which which version? What's the host name? What's the SSL certificate fingerprint? These are all facts that are exported to the central location and use there and what we're doing here is that we're actually relying on these facts To target behavior at nodes or for instance if your domain name ends in Zurich corporation Then you might be you might you might have all of the Configuration that is specific to being hosted in Zurich at these nodes that fulfill these criteria Now I think that is backwards Maybe that's because I'm limited to my classical system administration thinking but imagine these questions What classes does a specific node belong to which behaviors does a specific node or isn't a specific node expected? to have and You can only answer that by looking at all of the data generated your entire inventory and then walking it back So it's it's kind of inefficient on the other hand Of course, you will see that the other questions that you can easily ask answer with the system of targeting nodes are not as Trivially answered with reclass, so it's really just the opposite perspective of going at it but Another thing that I find very disturbing is that the behavior is actually dependent on data on the host and While I can see a lot of benefit in that because you do Infrastructure of sorts I still think about system administration as Actually being about keeping an inventory of all the stuff that you have and sort of centrally defining the deep behaviors that you want From your systems and then making sure that the software Forces those on the systems rather than asking the systems first whether this is what they want It I think it gives you much less space for errors If you can look at a central location and know that when this is enacted all of the nodes are going to be the same rather than having to worry about Data that is exported by one of these nodes so How I think about it and this is not going to come as a surprise and To many of you and this is also Probably not even that different from what I just showed with targeting as I said the differences is not clear-cut What I what I think about is that when I have a node that's called blue dot example dot work I want to say this node has a certain number of classes this node has this behavior Rather than saying this behavior should be targeted at this node So I say that this node belongs to the classes male server NTP client for instance and host it at Zurich And this is very much see if engine Inspired that's what the first one did the first configuration management system that I noted and I got to know about it through fi Actually, and I really like the idea and it just stuck. I couldn't get rid of it anymore and This is this is very much like definition of behavior and controlling the software the configuration management software that you're Dealing with and I'm not actually going to go into that at all today So no puppet no salt no Ansible specifics here And but one of the problems that comes with all of these systems is Where do I keep my data? Where do I specify the values for the parameters that these systems expose because they are not all equal? They are as I showed earlier part of a mostly heterogeneous environment. I Might go ahead and say That my NTP clients should all just benefit from pool dot NTP dot org and specify the server to eat that But then for some reason I mean ideally that's a distributed system, right? I might want to say that actually when the server is in Zurich. It should use a different value This case only the servers that are in Zurich use a different value all the others fall back to the default That's not rocket science Same example here. We also have a second node called white Just hosted in Munich now. Let's look at puppet Let's look at how we do this Approach with puppet. I'm sorry that I just noticed that the NTP server is actually not zero dot De dot pool at NTP dot org, but red at my own infrastructure I hope you will forgive me for this And my point here is that I want to tell puppet that the server at blue should be using an NTP Should be an NTP client that is why I include the The NTP class here or the module which provides this functionality But that it should actually not fall back to the default, but should use red example org as the NTP server now If you have five systems, that's fine if you have 50 It's probably still fine if you are not quite sold by that concept of laziness that I earlier introduced But if you have a couple of hundred servers, then this is not going to be okay anymore because suddenly you will find yourself with an Site description that has a lot of these stanzas and they are just all going to be the same so in puppet You can actually factor out some of these behaviors into Something that is not really a node, but sort of an include if you want to think about it like that Also programming programmers will understand what is meant here by inherits Now in the common nodes all nodes that inherit from this node definition There is a default set For the NTP server and it includes the NTP class and just by the mere fact of inheritance Blue dot example org now also includes an NTP client configured to use that server But blue dot example org is in Zurich and we want to be using the Swiss specific. They run better, you know so I go in look the NTP server for blue should be red, right? No anger That does not work. This unfortunately does not work in puppet in addition. They're multiple inheritance So what I mean with multiple inheritance the programmers might run away now What I mean with multiple inheritance is that obviously you're not just not all of your hosts in Zurich are also NTP clients ideally and you don't want to be creating a class for every single combination of potential parents that you have so it's really nice to be able to multiply inherit from from different behaviors and Puppet unfortunately doesn't do that In fact puppet actually says in their docs specifically that you should not use inheritance So what I just showed you this what what seems sensible to many of us to factor outcome in behavior Rather than having it all in one place multiply multiplied a thousand times Puppet actually discourages this in the documentation now. I'm not trying to take a stab at puppet here There is Puppet was one of the very first Tools to actually say if you want something more complicated than the simple note definitions then use this functionality that we have over here But I just showed in terms of the overriding the NTP server it does make Parametrization unnecessarily difficult simply because puppet only ever knows one instance of the variable NTP server And unless you want to go into scoping nightmare You are basically stuck with Reimplementing a lot of the data or multiplying a lot of the data and generating a lot of redundancy which as We all know is going to bite you at some point in time So let me Before I go on let me go a little bit into the what I consider to be system of optimization Principles and a bridged version because I don't want to have any questions about what I'm leaving out and suggestions that this list is incomplete I know it is These are the important things for me I want to be centrally in control of something and I want my data to be versioned with Versioning control system of your choice. I Need to have Parametrization I need to be able to say that NTP clients. That's a very simple example, right? But think of postfix. It's like it comes with 500 parameter value configuration Possibilities and some of those are actually sensible to parameterize because they are going to differ between the code and I have seen too many Installations of puppet where you actually have in the module code in the one that is responsible for Installing your NTP server or postfix special casing based on the host name and that just hurts Not right there because it makes it work But at the next upgrade or when something changes and suddenly you have all this redundancy in the code That is going to come back and bite you. I Think that system administration should be about no redundancy do it once do it in one place only and be able to always find That place where you did it very easily without thinking too much about it at three o'clock in the morning after being woken I Think that this idea of using information from the node such as the IP address or the free memory or the SSL Fingerprint is actually very good, but I don't think that it ever should be used to Steer the behavior that is going to be applied to a node Which is very difficult to say because obviously I want to describe my infrastructure independent of having to know Which one is a fedora server and which one is a Debian server and if I you know, I need to kind of ask those systems that that is at least the fundamental motivation between behind this This functionality, but what I'm about to show you Is a system by which you actually take it away from puppet you don't say puppet You know how to deal with all the different distributions. I just have to tell you to install a package and you'll do the right thing My approach is to to say I know that this host is a Debian node and I'm going to treat it as such And if that ever changes, I think I have a different problem So reclass as I said a misnomer of sorts you can think about it in terms of hierarchy, but the name here Ah has Been taken up before I realized that I misnamed mine Um It stands for recursive external node classifier external node classifier is a term that comes from those configuration Management softwares and it basically just means instead of requiring you to specify your infrastructure in note groups or in note Stands us as I've just shown in puppet. Um, you have an external data source and classifier while I try to Make a statement about my use of classes in this The CMS uses reclass But asking it questions such as what applications what modules should be applied what applications should a role have How does this note differ from all the other nodes that have the same application which is parameters and Which nodes belong to a group because essentially I want to be talking to groups of nodes and not just individual nodes and not always only to all of them at once For the system administrator It will obviously as all of these tools to allow you to deploy and manage site-wide configuration changes It will allow you to say things like upgrade all the nodes that are tagged with sable Update the message of the day in Zurich because of a power outage and fetch logs from all of the hosts tag mail Server because the BND or whoever secret service has actually knocked on your door Um Important for you to realize before I dive into the details of reclass is that reclass just assembles and provides the data It doesn't do any changes. It doesn't your hosts don't even know about it It is actually just one abstraction layer on top of all the configuration management systems I'll skip that and It is a single data source which means that if you use it with one system, you can use it with other systems There are adapters that interface between these different systems and reclass For instance mode of indication and output. Basically, it's an API definition of sorts system administrator specific API definition by which I mean We call programs and then parse output The adapters that are provided puppet, which is actually not yet provided But I get into that in a second salt and ansible as I said earlier So if you're a user of any of these and you think that you might want to use a different one This is where the slide about Remote execution and configure I'm done For instance salt and ansible they work very well together I mean ansible is very great at remote execution and is not so great at configuration management where salt is not so great at external execution and or remote execution and somewhat better at configuration management the important point is you can use both of them And you keep all of your relevant inventory data in reclass and Those tools just get the data out of reclass and do what you asked them to do So puppet. Why is it not actually in puppet? It was originally written for puppet Because puppet didn't do what I wanted and then I puppet really didn't do what I wanted and I rage quit puppet And I removed it that was the best thing ever tell the new configuration management system to just purge puppet It's the first thing you do get rid of your predecessor It's a good feeling all these Ruby packages and so I rewrote reclass since I ditched puppet and I wrote it for ansible and salt which is what I'm using at the moment and I could really not be bothered to Reimplement it now to all the puppet users who are wanting to use reclass who are interested by this They are a couple. I'm not trying to scare you away.  I'm sure you'll have it done in an hour or you buy me a beer and I'll do it for you It's trivial Salt who have you use the salt? That's almost a representative sample It provides top and pillar data for salt It has been actually integrated as a salt but not reclass itself But the adapter for reclass has been integrated in salt since 016, which is not yet in devian It is an unstable but not in back ports are stable node groups if you're interested in if you if you are Would like to Completely get rid of the top data in salt then we need node groups and so ask me if you want to help here Ansible who have you uses ansible? That's an equally big representative sample size It provides inventory and node information. It is actually implemented as an external script Unfortunately, it does not support this really new feature of batch calls But enough of the sporing stuff, right? I Think that Parameterization to system administration is key if you manage to somehow Remove all of the salient bits of information that differ between your nodes in your infrastructure and Factor them out keep them in reclass ideally then you're gonna have a much easier time Switching hosts around or doing this and that or switching configuration management systems around because all your data is independent of that Of course, we should only Do the sensible approach don't special case, please Reclass even allows you to keep your parameters modular so that you don't have to in reclass duplicate information and Ideally non-redundantly define your data in one location. So let's look at what it's like Yammel is probably known to everyone here. And if not, then I trust that you will immediately understand this Here's the definition a bridged definition of blue, which you've already met It is a post-fix node. So it has an application of post-fix defined And it is an NTP client and it has a server defined Which is the canonical default server? What this means is that it's independent What you see here is independent from what your configuration management system ends up doing Applications might be called modules. They might be called states and salt for instance, or they might be called I forget what it's called an ansible. It doesn't matter But you can understand that this is actually going to translate Into a host that has post-fix installed and a host that has the NTP client config installed and configured to be This value here, but this is not recursive What we have in reclass is basically two directories. We have Yammel files for nodes in This directory and then also Yammel files defining classes in another directory and And the recursive part is that nodes and classes may specify other classes to inherit from as many as you want You can think of these classes as tags. I very often do Because you might want to address all your systems that have post-fix installed So just simply talk to all the nodes in the class post-fix and It does smart deep merging on return from a recursive Descent walk I had to show up a little bit that I didn't computer science student here by which I mean that obviously there's going to be some data replacements going on and This is implemented in Python and also Yammel has a merging feature, but when you have a set of Values defined on one host and the set of values on another host and they all happen to be kept inside a dictionary Inside of a hash then obviously you don't want to just replace The existing hash with a new hash and forget all of your existing information But you want to merge the two dictionaries and the same exists for lists and the same exists for some special cases of scalars Yeah, what about order for off the text very good question have slides prepared for that But let me first show how the next step would be so here's blue and We have a class down there called common and we now have ntp defined for all nodes So it's in the common class and other than that I changed me. Oh, yeah It's now classes mail server not applications post-fix anymore But I'm sure you can see the direct relation between what I just showed and what this is now And obviously what you would expect this is the value that gets installed as the ntp client on the blue host In general and this is a very very important Baseline assumption that you can rely on in reclass, but that you also need to understand When you have something that is more specific that that talks about less nodes Then it overrides the data that you specify in a more specific less specific case for instance all your Debian at stable Nodes will be able to override parameters that were defined in the Debian common class. I Hope that makes sense So in this case now and to get to your question about the ordering We have a new class here hosted at Zurich and it defines the ntp Client or server to be the Swiss specific one But we also know that common This class up here already includes the ntp server and already defines the default and as a matter of fact We might have multiple of those But through this hosted at Zurich simply because the class appears later in this list it gets to override it There's a well-defined order Even if you don't want to worry too much about this because all you have to do for instance here Let's change to a different program. Let's say that in general. I want my SSH servers to have no root login allowed What about a backup client that needs to allow root login or you can find other solutions for sure But what this is one of the ways to do it? So now you define a class backup client and you set in it You override in it permit root login with a without password and because the backup client comes after the SSH server Expected behavior, but this might not be enough for you So all you have to do now is that if my backup client actually relies on SSH for transfer Depend on the classes on the class for SSH server, which will now cause when when the backup client is hit to walk through the data structure and Do everything that is necessary including the merging for the SSH server? Unless it has already done that right in this case. It would just simply say I've already seen the SSH server I don't need to do it again It would actually be bad if I did it again, right? No, it's only on first occurrence And then there's also something called parameter interpolation and that actually wrecked my brain and really made me question whether I am a CS student Because inside these this entire data structure that is being created when you merge reclass data Up the tree or down the tree should I say? You now can in reclass reference any other key So in this case, for instance, I might want to have message that is a reference to Floyd in My message of the day and then in diamond One of the hosts in my infrastructure. I simply Include a little message and that gets automatically merged as well And this is looks a little bit trivial a lot of the other configuration management systems do it themselves already But then it's either single pass So if you have two references, then you're it gets stuck with it with another reference or it converts everything to strings Which is not necessarily something that you want you might want to have a list after all So future work I'm almost at the end of this I need to package reclass I feel really stupid I come to depth cons to give a talk about reclass and it's not actually in the archive But that should not be a problem It's a Python module and you can very easily pull it from git at the moment and install it with easy install But I'll try to work on the packaging this week as well I was thinking and also during depth conf you could actually integrate this with preceding and Di so that you could provide your data even before installation and then have the same data Reused later on when you are done with installation and you pass over to configuration management In terms of cloud you if you have your name, sir, if you have your server's name WWW 0 0 1 through 9 9 9 you don't really want to create a thousand identical YAML files that all inherit from the same class So I'm thinking there should be some sort of policy classification where you can say that all of the nodes that are basically wild cards on on the node name That should be implemented membership lists. That is the concept of Postfix and client certificates what I mentioned earlier or that you want your Nagios clients to be also monitored by the Nagios server A lot of the configuration management systems address that by having some sort of wacky Communication between nodes or a central data collection called in puppet for instance store configs Which I think does not belong there Even though I agree it is nice to think about that Nagios only starting to monitor a certain Node as soon as that node has actually started the client package Yeah, sure, but if 15 minutes later through the when next time the configuration management system runs I will know that now definitely all of my nodes are in the same state. I can just statically tell The Nagios server about all of the clients, but obviously I don't want to have any Redundancy in the data set so it's somehow reclass needs to figure out membership lists of classes and be able to pass them on as parameters Haven't figured that out yet Other data sources potentially in terms of performance YAML files could become a problem They are after all opened and read and closed for every single run of reclass at the moment You could put this into a database There's actually already a plug-in infrastructure in place better unit testing without any philosophical debates and your idea here if you're interested by what I just said and you have some Usage ideas, then I'd be very very glad to hear them and with that I end my talk and thank you for the attention And I'm open to questions Can you move back? I think three slices There was where where you said there's a parameter and then yeah this one Down there if you add classes SSH server, is it important that classes is written after parameters? No YAML is basically very compatible with Python and all this causes. This is going to be read into a Python dict and The dict is unordered anyway, so it doesn't matter So so it's it's it's defined that the interpretation of this syntax is that even if the parameter is without password and then the classes SSH client which has Has a higher priority Which you define in classes SSH server or in backup client? This is it's a very good question because it allows me to also address one more time this distinction between targeting hosts and classifying them With the targeting hosts approach You would now say what's the parameter that all of my web servers should get and with reclass or the way that I like To think about things I start here I say I have blue right now in my screen and I want to configure that and so I then start a recursive walk of the tree So in this case I load the SSH server class and then continue to do my work here And it merges the this value no into my tree and then it comes back And then I go to backup client and backup client then merges this Scaler over this scaler so that at the end when I come back to my class. I now have the without path password Set if those two were we reversed if SSH server was after backup clients because you just you know Quickly set up the host and you then went out to dinner or something like that Then what would happen is backup client would be loaded. It would actually set Sorry, no, it would actually see there are classes So then it would go into SSH server. It would then read this Write no into the tree and then come back to backup client and overwrite the no with without password before it then returns to blue That example that work Okay, I'm not sure if I really understand seems to be a bit complicated It could be because it can be a very big tree. This is a this is a very classic example of where a Diagram would have been a better slide than just code On the other hand what you just said this could be a very big tree that made me stop the idea from drawing a diagram because it Was not going to be very informative I suppose I can either try to explain it again or let just let it sink in and ask me if you still have problems because I'm Fairly sure that everybody who has dealt with recursion at some point in time or who understands what it means to even walk the Directory tree on your Unix system recursively for instance using find Will immediately understand what this means When what I would propose I think If I have a very big Configuration with a lot of yeah with a big tree or a very deep tree It would be very nice to have some debug support so I can say oh Which passes are walked through the tree for a certain client? Yeah, that would be nice. So I have this to-do list and reclass and debugging is number four It would have really helped this Especially the parameter interpolation to implement that correctly because it requires you to do a topological sort of all the dependencies between parameters That really kind of would have been much better if I had debugging done first but It is very simple in the end and if you look at the code it is actually very simple And once you understand the general concept once you understand what actually happens when you walk a tree the important I guess the only important thing here that that is a little bit different from walking a tree is that what you're doing is You're doing the action at the end when you come back up. So it's tail recursion and I'm sorry. I'm failing at this It's just too slow Tail recursion means that basically you don't just add to your list when you find a new directory and you enumerate the lists but in terms of Unix file system imagine you had a sim link somewhere under a To a tree that is somewhere under under zed and now when you basically Walk the entire file system. You will see the contents of the tree Reference by the sim link first even though it only appears in zed and what what then is the important part is that you? store what you've already seen and that you go all the way to the leaf and then you do the operations when you Come back and that's when the merging happens and that's when this Specifics of I'm having more specific Data override less specific data comes in that's how it's implemented Further questions I'm happy to walk you through examples and everything Maybe you're already answered in the last slide, but I was distracted. Do you plan to do you plan to? To add possibility to use an external node classifier to a class I mean to be able to extract from a database the set of nodes with some properties which will allow to To do most of the class if I but we will still be it will still be possible to override Most of the things with the class. I mean I already have an external node classifier This is an external node classifier So I don't understand is your question about can this coexist with another external node classifier or reuse the data from another external node Yes, that is something that I recently thought about how to do it's kind of difficult because I Mean unless I want to implement an adapter here for cobbler or whatever you have right Into reclass. I'd really rather have you configure cobbler with salt or with ansible or puppet and then let me get at those data Once they are ready pulled because I don't think reclass needs to get any more complicated on the other hand There are there's a plug-in infrastructure for storage and this YAML FS Which I showed you which is the nodes and classes YAML files and so on you can actually it's two functions that you have to write You can overwrite that and put it into a database or XML RPC to your cobbler server I think that should be possible, but I don't really want to put this into reclass proper at this moment But what I do want to do is figure out how to get at the data that your configuration management system has already collected which includes things like the SSL fingerprints or Then free memory all the facts that I was talking about at the moment You can't use them in reclass. You have to use that in templates later You have to combine the values and ideally, you know, be nice if you have all the data actually available in one place Are there more questions? There's one. Yeah, it's on. Yeah, how'd you handle in? Well, I was gonna ask but I think I figured I You know in your blue example say you don't care that blue itself is an SSH server So you go down the tree backup client and that was an SSH server and then you know So you know that permit root login was overridden in backup client. So that's okay when you walk back up How'd you figure how'd you handle it if? You end up with sort of down going down the tree somewhere at the same level To overrides to permit root login There's no there's the same level, but there's still a well-enforced order in this case Whatever backup client does would overwrite whatever SSH server does and this is I don't think there's any other way to do it And this actually gives you a nice nice control over the system because I often Tag a number of my notes with test which I then append to the list of classes And that means that that actually overrides whatever I set in there overrides everything else And then later on I can remove remove that test class again and and resume normal work Of course, I do all my testing in a testing environment not a production environment never That's a joke more questions good then thank you. Thank you very much


So, who are we? Well, there's three of us, I swear. One of them is a ghost in the show. So, Long died, actually, led a lot of the work that we're talking about today. But he was in Shanghai and not able to actually get out of here. So we have a video segment, but Long is an excellent part of this process. And we want to acknowledge that as we go along. I'm Adrian, and I work at TouchRate. And I work on WebAssembly runtime call was zero, which is written in Go. And I got acclimated to DAPR much more so through this work. So I'm sort of like a community contributor. And it's Mauricio. Mauricio, I work for Diagrid. And I think that we have a couple of slides like introducing, right? That's you. That's your Twitter account? That is me. Yeah. Good. I'm code from the crypt. And so if you want to follow those things, I promise that it meets its name. Definitely. And if you're interested in these topics, yeah, I'm at Salavoya Twitter as well. I work for this company called Diagrid. We help people run DAPR in production and, you know, in multiple clouds. And I'm writing this book that is titled Platform Engineering on Kubernetes, where I'm covering kind of like these topics and how you put different projects together and how you extend platforms is kind of like the topic of today's session. That's the QR for the book. And down there, it's my name with 40. That's my age, unfortunately. But it's also a 40% discount code if you're interested in that. I'm just trying to find your book. I'm trying to find like a positive angle of the 40 thing. So this is what we're going to be talking about today. We are going to start like with DAPR just showing it in action. How many developers do we have here in the room? Yeah. That's a good 30%. 30%, I would say, maybe. And then we are going to talk a little bit about how we are kind of combining DAPR and WebAssembly together. And then we are going to see another demo showing that exactly. I think that what we are showing here is just starting points for collaborations and there are tons of other things that we can be doing. So if you don't know anything about DAPR, DAPR is a CNCF project. It's a distributed application runtime. And I will not spend so much time discussing about what DAPR is. I wanted to show you in action. But you need to know two things about DAPR. One is thriving on the CNCF community is the top 10 largest projects in the CNCF. That basically means that there is a lot of people contributing back and getting involved and extending it basically. And the main thing that you need to take away from this presentation if you don't know anything about DAPR is DAPR provides components that helps application developers to make their distributed application simpler, easier to maintain and the couple from infrastructure. So let's see that in action. Imagine that you are building this kind of applications where you have applications that are writing data somewhere, some other applications reading that data and doing some processing and some applications that are subscribed to notifications. Imagine that you want to listen every time that a new order gets created or every time that you write something to a database or every time a new customer come in. That's the subscriber app that we have down there. And if you are building these distributed applications, it's quite common that you will have a front-end application or a mobile application that is interacting with all these backend services by using HTTP or GRPC. So you need to think about how those interactions will go. But in real life, we need to replace this cloud that is there for real infrastructure, like our cloud provider services, our databases and all that stuff. So for the sake of explaining the example, I chose to mention Redis here and Kafka. If you want to write data to the database, maybe you can just use Redis for that. But this is starting some complexity on some tight coupling with the application code. You need to add the Redis client in order to write data to Redis. And that's a dependency to your application. The same for the application that is reading data from it. And the same for the application that wants to use Kafka. You need to have a Kafka client in the programming language of choice in order to be able to submit and to consume Kafka messages. Another challenge that you will face is that if you are using HTTP or GRPC between the services, you will need to deal with, for example, HTTP retries and circuit breakers and all that stuff to build in resiliency inside the application code. So Dapper comes to try to help us to solve some of these common challenges by abstracting away the infrastructure and that's the Dapper components, the things that I mentioned before. In this case, for this demo and for the application, I'm using two Dapper components. One is the state store that allows you to abstract storage. So if your application needs to store some state in a database or in this case in a key value store, you can use the Dapper state store component to abstract that away. The same with the PubSub component that abstracts away message brokers. So then, by using these Dapper components, what you can do is you can actually swap the implementation. If you want to change readies for PostgreSQL, you can swap that into the configuration side of things and keep your application code the same because what Dapper does is introduce this Dapper, side catalyze running in there exposing consistent API for your applications to use. So your applications, if you take a look at the application code that is down there in that repo, you will see that it's only doing HTTP requests against the Dapper side catalyze that it's running locally and it's interacting with these backend services, these databases and message brokers just using HTTP. In the same way, Dapper provides more components to solve different challenges like secret stores and now we have workflows as well. And because, for example, here in the frontend application, it's also using Dapper. We can actually reuse all the components and all the behaviors built into the Dapper side cat for retry mechanisms and secret breakers. So you kind of get that resiliency built in in your applications without changing the application code, without making the application code responsible for implementing these behaviors or adding libraries to implement that. And all the upgrades of the drivers. All the upgrades of the drivers, right? Like if you are using readies, you need to make sure that your readies driver or the SDK, it's the same version of the readies that you're using, right? And sometimes you want to run readies locally and sometimes you want to run readies in Google Cloud, for example. And maybe they have different versions. So that basically means that your application needs to have two different dependencies depending on where it's running. That becomes complicated and it's tightly coupled. We are trying to solve some of that with Dapper. And as I mentioned, Dapper uses that side cat approach. It's the most common way of using Dapper nowadays, but we recognize that side cats are not for everyone. So if you are not into side cats and you said, ah, they're using side cats, I'm not going to even look into Dapper, there is a new initiative in the community that is basically expanding that for different deployment models. And I think that Watson might be something else that we will look into the future. So let's take a look at the demo. Again, this demo, you can run in your own computer. Like you can just follow the step-by-step tutorial here. And what I want to show here is that I have an application running in my kind cluster. I've installed Dapper using Helm into my cluster. Just a single line, you get Dapper installed. You can actually see that I have the Dapper system namespace running. And then because I've deployed the application modules, like the one that is writing data, the one that is reading data, the one that is subscribing to notifications, we can see the pods that are running here. And because these are Dapper-enabled applications, you will see that Dapper is injecting a sidecar into each of these modules. Let's take a look at how the application looks like. The application looks like this. Not here. Here. The application looks like this. Pretty simple. We are writing messages. Hello. Here we come. Hey. Right. That's writing into the database. That's being written in Redis. Again, if you look into the source code of this application, there is no Redis code into the application that it's writing this data into Redis. It's just doing HTTP calls by using the SDK, the Dapper SDK. But you can do just plain HTTP calls or JRPC calls. And this pod below here, it's receiving the notifications asynchronously. So when I click the button, it goes to the service and fetch all the notifications that that service has received. It's pretty simple. But I would love to do something like, hello, Qcon. All right. I want my emojis in there. Unfortunately, that's not working just yet. But we will try to fix that in a while. One more thing that I want to show you here is that Dapper is running inside Kubernetes. That means that we can configure Dapper using Kubernetes resources. And for that, as I mentioned at the beginning, we have components. Right. So for this demo, you need to see here that I have two components configured, the state store component and also the pop-up component. Again, abstracting away the infrastructure that is being used to store data and also send async notifications between services in this case. So let's move to talk a little bit about WebAssembly now. So I can click again. You can click a bit now on the presentation. Let's do this. That's why you brought me here. Check this out. Click. Hello, everyone. Two clicks. I'm sorry that I cannot be there. Let me introduce myself first. My name is Daishan. You can call me no one, will. I'm a Microsoft MWP, Dapper Nintendo, and now I work in Intel. So what is Dapper? Dapper is distributed application runtime. This also uses any code or free work to write her microservices and run in nCloud or age in infrastructure. And this uses HTTP API or GIPC and GIPC API to provide many useful and amazing features. This is the architecture. As like a service mesh, it's a Dapper drone as a sidecar. We can use this directly with all the code changes. But the questions come. How can I change the logic inside the Dapper itself? For example, my company has many applications. They all need to check a circle to ensure the security in HTTP request. This is not published, and I cannot upstream this. And I do not want to rewrite this logical in every application code. So how can I do this in Dapper? A first thought, a quick thought is that I can release a new one. But this is very hard, difficult since Dapper have many reports, and we need to think the upstream version. This is a big job. But Dapper provides and think this and provides a useful feature, Midwire. So the Midwire supports users to do some changes in request or response. But this is built-in, which means this is hard code with limited configuration. Users cannot drive their old Midwire, which is not public. This is not flexible. So how to do this flexibly? We can use a Wasim. With the help of Wasim, we can do this. We can dramatic load, and with highly communicable. This is more than flexible. So how we implement this and how to use this pace continue to say. Thanks. Great. Thanks a lot. All right. Let's click. I thought it was going to click. I don't know where I am. There you go. I do need practice. So anyway, long, which we'll thank along the way in different ways, was the first person to take action on the WebAssembly kind of idea streams that came out of Dapper. Well, I didn't join into this conversation until later. It's interesting that this whole what to do with WebAssembly and Dapper started in the end of 2019. And as I was starting doing the like archeological digs in the issue list, the first thing I found was people wondering, well, should PsycharLogic itself be compiled down to this Bikove format called Wasim so that an application can embed the Psychar literally in the same process, which is kind of an interesting thing. Because WebAssembly is basically a way for you to run other third-party code inside your same process without actually launching another process or anything else. And then there was also questions of the other way around. Should Dapper allow applications who have been compiled to this Bikove format to just basically run workloads or to like how Dapper might run a container. And I think something happened in 2020 that got everybody busy. So there wasn't so much action going on in the WebAssembly front. Well, people were putting their case in virtual education and stuff. But the lead half of 2021, this whole WebAssembly, otherwise known as Wasim, just like the file extension name for WebAssembly module, that picked up again through some tension about how to handle this kind of like continual crisis in Dapper, which is like we've got a lot of community, a lot of different types of features that want to get into the same build. You have conflicts. People are like, well, okay, what about the size or the memory usage and the maintainer is going to be able to afford to do code reviews and patch maintenance on hundreds of components in the same repository or repositories. So basically you could see this trend of like what to do about extensibility, like how do we make it more flexible. And one of the options there, which I'll have a slide next on is like WebAssembly as it happened with discussion around GRPC. But anyway, timeline-wise, a proof of concept happened in design form in late 2021, but it wasn't able to be action for a little bit later. And yeah, so Dapper first had WebAssembly support 1.8, and then revised it again later last year. So let's get into it a little bit more. I know these things are kind of abstract, and I say about like embedded. So literally meaning that if you are using WebAssembly inside of an application like you have a say a Go application that's compiled with WebAssembly runtime inside, you wouldn't be able to know it's actually launching these VMs because there's no side effect, no process IDs or anything else like that happening. It's just local code. So when you look at this as far as a way to run third-party code compared to GRPC, GRPC has some side effects on your deployment, right? Even if you're doing fancy stuff like Unix domain sockets and things, there will be some side effects. At least you'll have to define a service definition and share those between the code bases and stuff. And both of them can get the job done, but one of the interesting things is is that WebAssembly can be used, and it can't be used for all tasks because it's limited. There is no deployment real task there. The only thing you have on your hands is what code am I going to allow to be loaded into this process and where am I going to get it from? Are you going to get it from a desk or an OCI repository and stuff like that? So effectively, if you can, chop pieces of functionality out. If you're in a GoLang ecosystem, you may just do that without even caring about supporting multiple languages. You may still be writing your components in Go. But of course, WebAssembly being a virtual machine, you can compile other languages to that byteco format. So it is a polyglot solution as well. So the design that Long had proposed to the community was to use a component, literally component middleware for HTTP middleware, and allow that chain to have a filter that could be implemented in WebAssembly as well as the native built-in features. And those of you familiar with Istio and Envoy will know that Envoy also has the ability to have WebAssembly in its filter chains. So it's not like a completely different idea, but it's an easy way to get through sort of like analysis paralysis, like what to do with something. If you pick a type of technology that's easy to relate to and it's one that tends to need customizations, it's, I think, a perfect choice. The problem was that this was at a point where the WebAssembly community wasn't so strong in Go yet, and there wasn't a way to embed a WebAssembly virtual machine into a Go process without relying on platform shared libraries that were like C libraries. And DAPR has sort of like a strict no on dependencies of shared libraries, right? So that was parked until long discovery of the project that I work on with originally just with Takeshi and Eda who also did a lot of the work on the Envoy and Istio WebAssembly stuff, but it's a large community now, which is a zero-dependency WebAssembly runtime for Go. And so that sort of like fit the bill. The first version long basically just, you know, handcrafted WebAssembly to prove the point, but that's not very developer friendly. Basically, I don't even know if the source was there, but we said, okay, well, if you're going to do something in a fairly difficult programming ecosystem, like WebAssembly, you're going to need SDK support.  And so in this case, the first simple version was like, okay, let's just rewrite requests and get this shipped out the doors so people can play with it. Immediately, people ask for more. They're like, okay, well, I don't want to just do rewrites. I want to change everything. And that's exactly what you want to do with the proof of concept. You want to get end users, and that's just like theoretical stuff going on. And during that period of time, people were asking, well, why don't we use the same thing that Envoy does, which is the proxy wasm, which is what they do. The problem with that was that proxy wasm was basically modeled around Envoy itself and the type of lifecycle and also the hooks were very much related to that. So for example, it had, you know, layers below HTTP abstraction and even gRPC abstraction, even some time-based tasks and things all in the same box. And that just wasn't a clean way for a component that's literally supposed to be HTTP and HTTP alone. So we actually created a different, like SDK base in WebAssembly and compiler uses where like ABI instead of API. So basically if APIs are for services, APIs are how compilers communicate. And we designed this to be a lot faster than proxy wasm, but the main thing was actually to make it more developer friendly. So for example, you can do async hooks and like offload things on the request response, but you can also do completely synchronous. So you can see this here that you have not just some URI substitution, but also the ability to do static response as straight in the WebAssembly. So that actually doesn't ever jump out to the host in those lines. It's far, far faster. And yeah, so that's how it kind of came out from a design perspective. And Mauricio is going to show you like exactly it working. Yeah, we just like see it working. Like in the example that I was showing before, we are just going to extend that. And for extending the example that I've created before, I show you the application sending some requests to the database and stuff. And what we're going to do is we want to extend that using the HTTP middleware component that was introduced by Long here and Adrian. And we are going to use the was zero runtime that it's already embedded into the that preside card to run that. In order to do that, we need to create the filter wasm file, in this case like an HTTP filter kind of thing that we are going to include in our HTTP chain. And for that, we need two resources, two configuration resources in Kubernetes, the middleware component and the configuration resource that it will allow me to connect my application and said, okay, I want the filter in this specific application. So let's jump into that. Let's do that pretty quickly because I'm pretty sure that we will run out of time. So first thing, let's take a look at the filter goal. Again, we are writing a filtering goal, and then we are going to use tiny goal to compile that into WebAssembly. So let me cut this and let's see if you can see it okay. This is a simple HTTP filter like Adrian was showing, but it's a little bit more tailored to the application that we are writing here. You can see that the handle request code here, it's basically receiving an HTTP request and an HTTP response.  You can write any kind of filter here. And what we are doing here, simple thing, we are just parsing for emoji tags and then just replacing that with the emoji code in there. And unfortunately our platform... This is code. This is business code. This is business code. This is whatever we want to do. And this is kind of like the extension that we want to build without changing neither our application or the dapper binaries in this case. We want to inject this code in the platform. Unfortunately, our platform doesn't support cats today. We had a CDE, and we will need to patch it later on. But for now, this is kind of like what we have. So as I mentioned before, we can use tiny goal, another open source project that allows us to take that goal file and transform it into a wasm file. We just run the compilation. And then of course, because now we have a file here in our file system, we need to make it available to our application that it's running on Kubernetes, right? So what we have done now, and that's kind of like the initial solution that we have, but we are looking into different approaches, is to, you know, I will wrap this into a config map, put it in my cluster, and then mount it as a volume in my pod so I can actually consume it from there. But as I mentioned before, we have two files here, two resources. The first one is called middleware. As you can see here, the middleware component allows us to say, okay, we have like an HTTP filter in this case that it's going to be written in wasm, and we are allowed here to set a path of that file. So in this case again, it's a mounted volume. Somebody will need to mount this and then just consume that file. Here's where we are looking at the options for OCI registries so we can just actually fetch these wasm files from container registries, right? Like it should be much easier to make that happen. And finally, the other thing that I wanted to show here is resources configuration, which is how we wire things together, right? Like again, it's an HTTP chain, right? Like we can write multiple filters. That's why we have an array here. And for now, we just have this simple filter that it's implemented using the middleware component called wasm, right? I already applied this to my cluster. So if I do get components again, you will see that there is a wasm component there already. And I also have like a configuration, an upper configuration in this case, that it's again the app config that I need to wire into my application.  I will show you that after applying it. So this is basically now configuring my application, just the configuration change to actually start using this filter. And let's take a look at the deployment file for the application. Resources wasm. And what you can see here is that my frontend application now, it's actually mounting a volume from a config map that contains the wasm filter. And then remember that I don't want my application that is running, the frontend application doesn't have anything to do with the filter itself, right? The application is not even aware of that. That's why I need to use this adapter annotation to let the adapter sidecar know that it needs to mount the volume under this path and that we are going to consume that config that basically contains all the filters, the array of filters that we want to define for our application. As soon as this configuration is applied, and let's take a look at the pods here and make sure that my latest version was deployed like 46 seconds ago, we can go here and do again the port forwarding. So we can access the application and let's go here to Safari. Let's refresh to make sure that it's still there. It still has the data, right? But if I say hi now, we should be able to see this working. Hey, there now we have emojis, guys. This is impressive. But if you thought that HTTP filters are boring, take a look at this. This is gold. And because it's happening at the front-end level, all the backend services now, for example, the notification service, will have already that payload. Everything else, we will actually see the result of that filter. As I mentioned before, unfortunately, we don't have cats. So if I do hi-cat, that's being censored. This is where you add your business logic, whatever you want to do with the request you can do there.  So that's what I wanted to show. How do you add things up together? This is, again, just the start of one of the things that you can do. Yeah, some interesting things that I thought about was that if you did get the cat-enabled version, right? Exactly. You just basically roll it out like a config change. The other thing is that you didn't see any pod side effects because it's embedded. It's in the same process as the sidecar. So the sidecar is actually inlining that logic within it. Excellent. So that's kind of cool. Yeah. So we saw how Dapper in general can do a couple things, which was the first part of the presentation. And this is the first experience with WebAssembly is to tune the HTTP middleware chain. And everything we talked about is not like, okay, it's wait until next year and then you can download it or do some patch, hack things. This was already out in Dapper 110. So you can just do this now. No hacks required. What's next? Well, we did notice that Istio has a little bit more experience in WebAssembly. So for how to source the WebAssembly files, we pretty much decided to just go ahead and do exactly how Istio is doing that for Envoy, to use like OCI-based paths or the HTTP graphs, whatever. If you're interested in this kind of work, it's actually not terribly difficult. So you could help us contribute there and get some experience with WebAssembly. It would be fun. We also have another component which is pretty interesting that's already half implemented, output binding. So because in Dapper, it can handle things, not just HTTP, you saw that there's messaging, subscriptions, all sorts of stuff, right? So output bindings, you can imagine, you have inbound events and then you can have a message processor that's just dangling on that and actually run with a Wasm module instead of routing into another component. So you can get like FAS capabilities in Dapper that way. And then we still have the ongoing discussions about generic extensibility. And I think the thing is, is that bringing in more experience about different well understood places of code and using WebAssembly is a great way and much easier than just going to generic extensibility first. There's too much context to learn. So let's sum it up. Okay. So we've got Dapper, it's got this distributed application framework basically. And we have a way that developers who basically yielded to the framework and said like, okay, I'm going to give all this responsibility to the sidecar. Now they can actually influence the behavior of the sidecar without custom builds, which is awesome. That was made possible in part because of Was here, the project I work on, because Dapper happens to be like a pure go project. That was quite cool. So we joined forces to try to make that and people will be able to have more flexible infrastructure without having all the work of doing custom builds and the maintenance there. Yeah, I think that's a very important point. We are adding a different level of configuration and fine tuning for platforms and for projects that are already being built. And I do see this happening more and more often in different CNCF projects. This is becoming a way of extending projects without changing them and having a way of deploying configuration code instead of upgrading the entire stack because you need to add new extension points. So we're going to hang out a little bit for questions if you have any, but thanks a lot for joining and we do want to thank Long even though I couldn't physically be here because he was a really pivotal one in all of these design and implementations. Thank you very much. Any questions? I know that we throw a lot of information at you. Yeah, file system support, that's for you. Yeah, so the question was what about file system support? So WebAssembly, by default, doesn't have any ability to access files on the host that's running it, but there is a system called WASI which allows you to basically have system calls like file open, type of commands. Was there the runtime that Dapper uses supports that? So sometimes people are using files for configurations or they want to have a SQLite database that they're using inside of their app logic and it's not actually a very difficult thing to add. It's more like questions on how to bound that and what I would suggest for folks who have a use case that requires file system to bring that use case with the request into the issues list and that way we make sure that whether it's read-only or read-write or virtualized files so that that gets into the configuration. Right now the configuration is pretty bare. We just have WASM and that's it. So basically it's an iteration and I don't see any problem being able to implement file support. It's just a matter of how and what people want to do with it. Other questions? Okay. No, when for example the sidebar starts it's going to load this every time or it's just one time and loading one of these web assemblies will take the same amount of time. Yeah, so the question is what's the life cycle of the WebAssembly module? And basically at the moment Dapper is not doing anything like file watching or dynamically looking at the file system to reload itself so the compilation phase happens before a request occurs and that's a one-time thing. So for example when it loads a module it translates into machine code that actually invokes. It's amazing. We have a page on how that works. It's really cool. But then that's just held there in memory and all requests go through it. Concurrency is actually handled in Dapper itself. So Dapper has controls for how many simultaneous requests can go through and so that's actually what's gating how many of these module instances are going on but the actual module is held static until reboot basically. Yeah, so from the Kubernetes point of view when the pod starts it mounts the volume and then it reads the file and that's the version of the file it's going to use for its own life cycle. Exactly. So if you want to update it right now we don't have any refresh functionality you basically need to update the file in the config map and then of course restart the pod, right? Yeah. But that's coming, right? Yeah, some folks were like should there be a dev mode? Watches the disk and things like there's actually threads on this topic and the main thing is we really wanted to have all this stuff user driven so if you have some specific use cases that'll make the config switches much more relevant for everybody. Yeah, and again the idea of having that repository with a step-by-step tutorial is that you can try it out. At the end of that repository there is also a much more simpler example it's like you don't really need to install all these things you can just start with WebAssembly and TinyGo but yeah, for a full version of an application running you really want to try the lifecycle operations and all that stuff, give it that a try. We're out of time. Thank you very much folks. Thanks again.


Countless arenas, resulting in a continued academic perplexment still litter Egypt's Giza Plateau. The pyramids, the Great Sphinx, the enormous megalithic blocks which made them. Not to mention the mountain of ancient, yet clearly incredibly precise, seemingly highly advanced tool marks which can still be found across the still surviving fragment of its basalt floor. 2,000-plus ton ancient singing statues, mysterious staircases cut directly into an inexplicably enormous, mysteriously concrete-like plateau. One strong enough to continue to be the foundation for the Great Pyramids themselves. The burning question then remains, who built this magnificent place? And can it be proven that, along with these structures not being the work of the currently claimed ancient Egyptians, but can be connected to countless other equally baffling ruins? A few even found in some of the most remote landmasses on Earth. Proof, if there ever was, of a once highly advanced, ocean-going, world-dominant, yet now lost civilization. Mencari, as mentioned in a previous video, the smallest of the pyramids, has undeniable polygonal casing stones, masonry in the exact same style and stone as that of ancient Peruvian sites. Yet in addition to these granite blocks, partly encasing Mencur, Aswan, its quarry with its gargantuan unfinished obelisk, and indeed its temple, are yet another series of undeniable proofs as to the true identity of the original builders of ancient Egypt. Although due to the clear similarity of the polygonal masonry, a now lost yet unique technique of placing seemingly randomly shaped stones without mortar perfectly together with that of Saxe Huaman, Machu Picchu, along with many other, the temple of Aswan, when looked upon closely, possesses polygonal perfection. Due to the same level of accuracy as that of Cusco, however, we feel, due to the medium of creation being softer, that of sandstone, their abilities in accuracy in producing this lost masonry technique, which has now been identified all over the world, really shines through. Are we looking at the proof needed to not only connect the most incredible, unexplainably ancient megalithic sites worldwide? Through the pyramids, ancient Peru, Easter Island, Ethiopia, the list goes on, all built by the same lost civilization. We find the evidence to support such claim, highly compelling. The severe undulating erosion upon the walls of the Sphinx enclosure undoubtedly showed that the Sphinx had been heavily weathered, long before the Sahara became a desert. Therefore, one must suspect that it could indeed be over 9,000 years old. Not knowing exactly how much rainfall there has been in the distant past, the Sphinx could indeed be far older than this. The most notable scholarly advocates, Robert Scotch, argues that the Sphinx may be far older than 12,000 years. Robert Baval and Graham Hancock proposed that the Sphinx may have been built around 10,500 BC, during the last age of Leo. Anthony West believes everything on the Giza Plateau testifies to an advanced, secure and long-settled civilization. Therefore, he suggests that the Sphinx may have been built not during the age of Leo, but the whole processional cycle earlier. In around 36,000 BC, a date he feels is more in keeping with the history of Egypt, as chronicled by certain Egypt kings. Regardless of an exact date, all of these talented Egyptologists propose a date set much further back within history than currently accepted, and they have provided considerable evidence to back up such conclusions. At the time of disclosure, the argument sent shockwaves through the Egyptologist establishment. Not because of the datings, Egyptologists and mainstream historians have grown quite inept at ignoring data, but more because it was realized that there is indeed no other explanation for their arguments. There is little doubt that the Sphinx enclosure was subject to severe erosion within its lifetime, and although it could have been explained away as a naturally formed enclosure, we fortunately know from analysis that the limestone blocks dug out from there were then used within the building of nearby Sphinx temple. Interestingly, no other site in Egypt shows the same type or degree of erosion. Was the evidence hidden away? Unsealed from the public in what could only be called a conspiracy? Sediments surrounding the base of the monuments under once existing watermark upon the stones halfway up the Great Pyramid's sides indicate just that. Two-inch thick salt incrustations once found within inner chambers, silt sediments rising to 14 feet around the bases of the pyramids, found to contain seashells and fossils that have been radiocarbon dated at nearly 12,000 years old have indeed slowly vanished over the years. These sediments could only have been deposited in such great quantities by major sea flooding. A watermark was also once clearly visible on the limestone casing stones of the Great Pyramid. These stones were unfortunately unknowingly removed by invading Arabs. These watermarks were halfway up the sides of the pyramid, or about 400 feet above the present level of the Nile River, 200 feet above the base. It seems the last remaining shred of evidence, the enclosure, survived due to the talented individuals that were required to spot it. Individuals who are thankfully on our side. Egypt. Undoubtedly, one of the most controversial places for modern history to try to keep the control of, in regards to its origin, its true age or original builder. When one either visits the Giza Plateau, and is lucky enough to gaze upon these three great pyramids, or merely able to peer upon them through their computer screens, the first thing that will usually cross one's mind is awe and amazement. Yet this is often instinctually followed by an air of wonder. A curiosity as to how these miraculous structures were built, who could have possibly built them? And most importantly of all, why? Yet these questions, and indeed the pursuit of their answers, has been a mission for many well-funded deceptive individuals for many years to work very hard to distract you from either asking or pursuing as personal line of inquiry. For example, the golden mask of King Tut, along with the many other undoubtedly spectacularly valuable artifacts encrusted with precious metals and jewels that can be seen littering Egypt in its many museums and in the mountains of literature, books, and touring exhibits, which are published, pushed, and permitted in regards to this spectacular area of human history. The arrival of the last of King Tut's chariots at the gem, which stands for the Grand Egyptian Museum late last month, was an exciting event for archaeologists worldwide and a source of pride for Egyptians. We moved today the sixth and the last chariot of King Tutankhamen from the military museum in the citadel, which was there since 1987 to the gem. So we were keen to show you the moving of this very nice artifact and the packing and unpacking methods, professional methods you are using by my colleagues in the ministry. The Tutankhamen exhibit comprising about 5,000 pieces will display for the first time all of Tutankhamen's artifacts in one place. Experts from around the world have been consulted on how best to preserve and display the collection when museum workers accidentally knocked off the beard of King Tut's burial mask in 2015 and hastily glued it back on. There were fears that modern chemicals would cause permanent damage to the artifact, but scholars around the world put their heads together to save the golden mask. The museum will also be a venue for international conferences on Egyptology. There is something in you always. We found out today in my talk the family of Tutankhamen through DNA. How Tutankhamen died. No one murdered him. My excavation in the Valley of the Monks that we are doing right now, important excavation looking for the tomb of Arkhiz and Amun, maybe soon a tomb will be revealed in the Valley of the Monks or the west valley of the kings. Most of the artifacts in the Tutankhamen exhibit have been relocated from the Egyptian museum in Cairo. Their new home is only about 2 km away from the place where the young Pharaoh's tomb was discovered in 1922. Modern officials say the gem will be the world's largest archaeological museum when completed and will hold about 100,000 artifacts in total. We have now 3,000 employees and workmen working inside the project. We are expecting our schedule will be ready from the engineering part by December 2018 and we are deciding now the perfect time or the ideal timing for the partial opening. In addition to King Tut's exhibit, the museum will display objects related to some of the greatest historic Egyptian kings such as Ramses II, Akhenaten and Amenotep III. The ancient Egyptians, although claimed as ingenious, were merely adaptive, just like the equally acclaimed Romans and Incas of Peru. These re-inhabitants merely rediscover the creations of a far older, far more advanced predecessor, who I believe not only constructed these sanctuaries, which these well-studied ancient civilizations merely used to enable the flourishment of their own cultures, in turn leaving a smorgasbord of architectural artifacts for funded academics to excavate and subsequently parade around, usually bombarding many individuals with deep insights into their lifestyles, culture, and death practices. Are yet, as I would have predicted, nearly always absent that which supports my posit. Any logical explanation or demonstration of how these people built these structures in which they once inhabited, like a void in their academic study, one which is not only consistently ignored and concealed by these same academics, but are unknown facts to all of modern humanity to this day. This mystery is a result of the incredible nature of these structures, the precision involved in their constructions, and the enormity of some of the stones used in the building of the structures. Many of you may have seen my recent videos, or be a keen follower of my work, and as such are aware of the fact that due to my in-depth study of the unknowns regarding these sites worldwide, and the collection and collaboration of the similarities and differentiabilities I have personally collected and categorized regarding many of these ancient structures, I have personally been able to establish a very strong, evidence-based hypothesis regarding the identity of three separate law civilizations, which I have established using signatures within their style of building, and by default, differentiations in their styles of building, to unquestionably identify them as separate yet particular groups responsible for the different unexplainable structures spanning the entire globe. Yet although these groups have indeed crossed paths, such areas as Aswan Quarry and most significant to my own research in Italy, where the polygonal civilization built upon the Cyclopians' work, allowing me to establish which preceded which, and although these groups have been established to have abandoned projects midway through, thus indicating that they came to a sudden and untimely demise due to cataclysm, the civilization responsible for the pyramids, and indeed the movement of the blocks at Baalbeck in China, which all far exceed 1,000 tons, is yet another civilization which far predated all which I have already identified. These three civilizations are the polygonal civilization, the Cyclopian civilization, and the Neolithic civilization, each with their own unique building techniques and identifiable stone-cutting signatures in their technologies. The pyramid builders were unimaginably more capable than all three, and although the Neoliths, who indeed have created some astonishingly advanced ruins, could have quite possibly been a surviving remnant of the civilization, this digression is for another time. Though at sites such as Baalbeck, the Trilithon, which contains stones over 1,000 tons, there are Cyclopian stones built atop the stones, and at other places in the world polygonal masonry has been found, such as Axum in Ethiopia, where the toppled obelisk is said by some to be in excess of 1,000 tons, I have never, and now strongly feel will never, find any indicative evidence of these civilizations building the footings under any of these gigantic megaliths, as they were not responsible for their creation or placement. Additionally, the civilization responsible for the pyramids, and these enormous megalithic blocks elsewhere, were also the civilization who created the false door. A mysterious rock-carved feature also found littering the now exposed mega metropolis found beneath the Guatemalan rainforest by penetrative radar. Taikal, part of this metropolis, the place where the plaque illustrating a past global cataclysm was once found, also has pyramids built solely leading to these false doors, with one found in Peru, built into the only rock face containing a very peculiar crystal known for its resonance qualities in amplifying radio waves. I feel that much of the spectacles found in modern Egyptian museums are merely distractions from the really important truths which we should all be focusing on instead, such as the true age of the pyramids, structures which in the past I have also independently identified as still possessing three separate identifiable stages of attempted casing stones for conservation, each significantly older or younger than each other, with the true exoskeleton of the structures made of stones far in excess of 1,000 tons. Join us next time, where I will expose the controlled opposition within the fringe fields of archaeology which have stemmed from a growing pursuit for the truths of these facts, with a focus upon the water erosion hypothesis of the great Sphinx, why it is a misdirection and the Sphinx's true, original, undeniable identity, facts and truths exposed, which are undoubtedly highly compelling. Do truly understand the astonishingly true history of the unfinished obelisk? One must first wade through a quagmire of well-financed fallacy, infested with many a false profit, incomplete or simply illogical conjecture, all of which, defended by countless academic figures of institutions of influence and power, acquired via the funding in their defense of a form of mass worship of academics' perception as if an all-knowing authority. So, with things like the obelisk, for example, one begins to wonder if this all be by design, since academic records of this monument began, no one who has described it, predictably, has ever managed to wrap their head around how such a stone could have possibly ever been moved. Ergo, all well-funded explorers, reporters and journalists alike, with the expectant pressure of their return with a deciphered mystery. It would appear this explanation never arose, yet was skillfully averted. Firstly, the rock had indeed been abandoned abruptly at some point in history, conveniently allowing academia to make nearly all those interested in the obelisk overlook this eventual intention by its original creators, a distraction made by a fault line. Chris Dunn, an independent investigator, held in varied regard, found that details of decoration were already being added to the stone as it was being hewn, running exactly through this so-called fault, disproving this so-long-held academic fallacy. Yet, alas, although the unfinished obelisk lay still attached to the strata of Earth, like that of the larger of the two megaliths in Yangshan Cori, the largest some 16,000 tons, academia is not required nor would even attempt to provide any logical explanation as to how these blocks would have been moved. Additionally, however, and perhaps most revealing, is the pregnant lady of Lebanon, a 1,000-plus ton megalith, so large that just like that of the unfinished obelisk, no attempt was ever made to explain the ancient civilization responsible could have moved such stones to their final placements. Yet, remarkably, the proverbial nail in the coffin and vindication of our claim was the excavations made around the pregnant woman, recently revealing that this stone was not abandoned on a slight incline, as claimed, but was placed atop another stone of even bigger proportions, suggesting it was part of a once enormous structure and exposing this reoccurring academic strategy when it comes to dismissing the controversial. It is a reality which we find incredibly annoying. Along with the many other currently unexplained feats of engineering present within the ancient ruins of Baalbek's temples is undoubtedly the variety of ancient stones that were somehow incorporated into the structures. Although modern academia and indeed its supporters, who are all seemingly suffering with selective research syndrome, claim that Baalbek is a Roman ruin, we feel, as mentioned, the sheer size of the ancient megaliths that were installed masterfully into its construction are, obviously, far too large for our Roman ancestors to have transported from distant quarries and who have installed into the structure. We are more than open to this proposition that they were indeed installed and built by Romans if we can be provided with one single logical explanation as to how this was done. But, as of yet, this remains elusive absent any academic explanation as to the side. As mentioned, the astonishing array of ancient stones is also an area that is rarely covered by individuals attempting to convey an air of all-knowing to the masses, as these features are, just like the enormous megaliths present at the site, currently unexplainable. Specifically, it's the pink granite columns. The reason for our focus on these particular stones is the fact that this pink granite is only available at one known ancient quarry, that being the famous quarry of Aswan, located within modern-day Egypt, an astonishing 1,500 km away. Some of these stones, weighing in at more than 10 metric tons, this achievement, we feel, is a clear indication of the fact that the builders of these ancient sites were far more capable than that of our more recent Roman ancestors. For example, as previously covered on our channel, Henri Léard brought two lamisu weighing in at a similar size around 10 tons to London. This task took over 18 months of arduous suffering for hundreds of our modern ancestors, placed a mere century ago to complete. It included several near disasters, and included loading them onto wheeled carts, complex systems of modern pulleys and levers operated by dozens of men, the utilization of over 300 men in total, a barge, and a custom built ramp to haul them up the steps and into the museum. How these same curators, historians, and academics alike can continue to claim that our Roman ancestors completed such tasks, along with the placement of such enormous stone megaliths, is to us absurd. Was the unfinished obelisk found within Aswan the work of the same civilization? We feel that these pink granite columns could, in all possibility, be a link that connects these two ancient sites, and in particular, the Great Pyramids. Was Baalback, with its enormous granite megaliths, built by the same people as the Great Pyramids? Is the giant megalithic exoskeleton of the Great Pyramids, which we have already exposed, built with the same techniques as Baalback? We find the evidence to suggest such, highly compelling.