Hello everyone. Welcome to Cloud Native Live, where we dive into the code behind Cloud Native. I'm Itai Shakuri, I'm Director of Open Source at Aqua Security. I'm also a Cloud Native Ambassador and your host for today's show. So this is Cloud Native Live. Every Wednesday we bring a new set of presenters to showcase how to work with Cloud Native technologies. They will build things and break things and they will answer your questions. This week we have Chris Tomkins from Tegera with us to talk about how to leverage EVPF with OpenShift with Project Calico. He will introduce himself and the technology in a second. Before that just a quick reminder that this is an official livestream of the CNCF and as such is subject to the CNCF Code of Conduct. So please don't add anything to the chat or questions that will be in violation of that Code of Conduct. Basically just be respectful of all your fellow participants and presenters. So Chris how would you like to introduce yourself? Yeah well the first thing I'm just really happy to hear you say that I'll break things because that covers me if I break things now so great. So my name is Chris Tomkins. I'm a developer advocate at Tegera. Tegera who developed Project Calico as part of the Project Calico community. I worked as a network engineer for many many years and gradually got drawn towards automation, large-scale automation technologies and when we started to do Kubernetes I was deploying Calico and I really liked the product and believed in it so I joined the team as a developer advocate. Cool so yeah do tell us about Project Calico I think for Yeah yeah I think that's a great place to start. So Calico is an open source networking solution for networking and network security for containers, virtual machines and native host-based workloads. So the idea is it provides a consistent experience and set of capabilities for all of those kinds of workloads in public cloud or on-prem from a tiny cluster all the way up to a multi-thousand node cluster and we can support environments like Kubernetes, OpenShift, Merantis, Kubernetes Engine and so on and the ideas to prevent present the same experience to developers and engineers. We offer a standard you know best practice security model and a high-performance data plane, set of data planes actually and incredible scalability and it's a real-world production hardened product and open source setup that is you know is heavily deployed already. Yeah it's good that you mentioned that it's also open source right? Yeah so there is an enterprise product but everything we do today will be in the open source product and the enterprise product adds on some features around observability well lots of features but primarily around observability. All right that's cool so the topic of today's show involved EPPF and OpenShift and Calico do you want to set up the context for this? Yeah definitely so I guess if you think of it as Calico allows you to implement your network policy and your networking across those environments that we discussed and it uses some technology in the data plane of the nodes to actually do that work. Now I did a separate talk which I could talk for half an hour but I won't about the you know control plane and data plane model for for networking and so on but just suffice to say that the data plane is how we implement the actual networking policy that that enables the networking to happen. Now Calico actually offers a choice of data planes. The chat window has just disappeared on my browser but hopefully it will come back. So yeah Calico offers a choice of data planes and the idea behind that is that different people have different use cases and expectations for the data plane of their of their network environment. So in order to achieve high performance you want to have the minimal data plane amount of data plane code that allows you to implement the feature set that you actually need. So our main original data plane was implemented mainly with IP tables which is great it's still in production for many users and the performance is good it's rock solid and battle hardened but we wanted to offer another data plane which has some advantages over that and so as well as the standard Linux IP tables data plane and the Windows IP the Windows data plane that we offer we have this third data plane which is the Linux eBPF data plane which is what we'll what we'll be focusing on today and then just to say as well there is actually a fourth data plane in TEP Preview which is a VPP vector profit processing data plane but it's in TEP Preview it's very interesting I encourage people to look at it but we won't talk about that anymore today.  No no I just wanted to comment simply eBPF is a really exciting technology I think and very I'm happy that we get a chance to talk about it today because it's very relevant specifically now I feel and see a lot of movement around it in the industry especially from the networking perspective of course so yeah just yeah so should we drill a little bit into into what it what it's all about? Sorry I shall we go a little bit into what the eBPF is all about? Yeah yeah please yeah so yeah as you said as you said it's you know it's a really interesting area and what eBPF basically is in the the old days if you wanted to have some code that ran in the kernel of the Linux kernel then you would need to either actually submit code into the kernel repose and get it approved and obviously that's a really long process and a very difficult process for good reason or the other way you could do that would be to write a kernel module but what eBPF actually is is a way to run you can think of it as a a way to run virtual machines inside the kernel and those virtual machines are extremely high performance they're heavily secured because they only have access to a to a very limited number of of kernel functions depending on where they are mounted in the kernel and so as well as being high performance and secure there's small bits of code that live inside the kernel and operate inside the kernel under those restrictions so in the case of of how calico uses eBPF what we're essentially doing is replacing the functionality that we previously implemented in ip tables in eBPF inside the kernel and and the the actual performance increases obviously but there are also some side effects but later on I don't have many slides but I do have one diagram I'd like to show and when I show that diagram I can explain why why that functionality also improves with with eBPF yeah I once heard a description of eBPF that I really like sorry I don't remember who said it so I apologize to the person who said it to me but they described it as like JavaScript is to the browser eBPF to the kernel basically mechanism vehicle for us to extend the kernel and to put our own code which otherwise would be very dangerous and difficult so yeah yeah exactly and it and it as you know I'm actually not a developer in my you know in my first the first part of my job is not a developer so I have to be careful not to overstep my knowledge but but as well as only being able to make certain function calls and those kind of things eBPF programs are automatically limited to execution time and those kind of things so they can't they can't get into tight loops and they can't they can't chew up the tpu that kind of thing so it's great because there's a lot of protection there to prevent to prevent your program from accidentally misbehaving and yet you get excellent performance there's also this concept of a eBPF map which is basically just a key value store that the eBPF programs are able to access which allows them to exchange data between each other to you know to record flows that kind of thing or whatever you might want to do now eBPF obviously it has a lot a lot of use cases you know it's commonly used for for tracing and for you know tcp dump uses eBPF but in the case of what calico is using it for it's using it to implement the network policy inside the kernel all right sounds good and just the the final piece there we talked about calico we talked about eBPF and then we are going to run all of these on top of OpenShift right so so like i'm assuming most of the people know about OpenShift but just to set the record straight for Kubernetes users could you just say a few words about OpenShift yeah sure and i'll be honest OpenShift isn't is the weakest part of my knowledge you know it's a it's a big beast so but but OpenShift is the way i like to think of it like you had a good analogy about eBPF but the way i like to think of it is that a kubernetes cluster you can extend it in so many interesting ways with different tools and you know for for observability and for for for storage and for cni and all these things and you could spend a lot of time both at time and money trying to figure out what a good kubernetes deployment model is for you and i like to think of OpenShift as a container application platform based on kubernetes that takes a good set of a good set of additional tools so as well as being a container orchestrator it it's an enterprise kubernetes distribution and it has a validated set of integrations it's still kubernetes it's still certified kubernetes and it's still open source but it but it allows you to build kind of like a consistent complete enterprise grade kubernetes ecosystem and and then to to run that wherever you choose to run it you're not limited to running it one place the demo we do today the demo i do today will be what we in AWS but obviously you can run OpenShift in any environment all right i think that's a very good intro so let's get to it let's get to yeah yeah great um so i think if you there's a couple of things we can come back to if we end up with spare time but i think it's a good idea to get to to flip over to uh just literally the the couple of slides i have and then a name to get to the demo um so yeah i think you can see my screen there we go yeah so um this just kind of slide it just kind of this is um red hat so in slide incidentally um which i've used um the url at the bottom there but really this just helps to understand that OpenShift is kind of a big uh suite of tools around kubernetes um or you know for observability and for um developer services and so on but i won't drill into that because it would be challenging for me to talk about everything on that slide meaningfully um but this slide is is a really interesting one and um this is the one i said that i would briefly mention um uh i said i would briefly mention about uh where BPF hooked in so this is the packet flow diagram and you can see at the bottom of the screen this is this is courtesy of um jan engelhardt which is it's uh on Wikipedia this diagram if you want to see it yourself but this is the packet flow through any linux node and you can see that the green part there is the layer three network layer and that's where ip tables and all of those kind of things uh happen so you have um a mango table and that table and um uh and you can see that the the track the the packets flow through all of that and the reason i mentioned all this is because um that's also where coop proxy is implemented um so when you run services on uh on kubernetes coop proxy uh implements those services and it exists in that green uh that green section um however down here you can see uh hopefully it's just about readable um you can see ingress and egress q disks and these two points here are where we actually attach our BPF code um and that means that the BPF code can entirely sidestep the the main packet flow um so that's why i wanted to highlight this diagram and so as a result of that um the advantages of of actually uh running ebpf um excuse me yeah the advantages of running uh ebpf in calico are performance but there's a secondary a really interesting uh secondary benefit which i'll demonstrate uh in the demo which is that uh is source IP preservation so if we have a look at this diagram uh and we'll see this for real live demo in a moment um but you can see with a traditional kubernetes cluster with coop proxy running um you can see that if an external client comes into a to a service pod the first thing they do is they hit a service uh on the kubernetes node and then that that the coop proxy uh that's that's serving that service does a destination that and a source map to replace uh the source ip with its own ip and the destination ip with a pod ip and then the traffic gets forwarded on to the service pod and then it comes back but it's important to note that the traffic when you're running with coop proxy without the ebpf data plane you can see that the traffic has to go back through the coop proxy and therefore it's destination and source knatted and the side effect of that is that the service pod down here never actually sees the source ip of the traffic uh the real source ip it sees the source ip of the load balancer uh of excuse me the coop proxy um so one one of the advantages is that once you turn on ebpf you get a flow that looks like this so when the external client comes in and they talk to the service the service is implemented as a bpf program not in coop proxy and that means that the bpf program can forward on the traffic and that the service the service pod on the destination node sees the real source ip of the external client which maybe might be good if you have a an auditing use case uh or if you have um uh you need to restrict a particular ip block by region by um by country or something like that uh and then the final the final step here shown is number five is that if the network allows it the packet can actually be returned directly um without going back via the ingress node so so those benefits are um yeah you get that's a performance lower latency and you get um and you get uh source ip preservation um so yeah it's pretty cool cool is that is that thanks to i know is it because uh you're um doing the uh network uh routing work at the lower level in the network stack yes yes exactly um uh they they're my colleague shawne crampton um i do i've done some some sessions called calico live which is just me and shawne crampton uh chatting about informally similar to what we're doing now but but he's one of the ebd f data plane developers for calico um so anyone who wants a lot more depth on this um if you if you watch those sessions uh you know frankly he his understanding of the depth is much deeper than mine but yes it's that's exactly we're implementing it at a lower level and therefore um by replacing kube proxy we can we can introduce other examples sorry other other benefits excuse me um yeah so i think we should go ahead and and show it right yeah of course let's do it cool okay so uh so just to explain what i'm going to do um i wanted to create a um i wanted to create a demo that anyone who's watching can actually do this themselves afterwards um so in order to set up open shift we need a um we need a dns name a top level dns name which obviously would cost money um but i was made aware uh by a colleague of um of this website freenom.com i can't actually recommend it you know for for production i don't know whether it's you know i don't know what their their production services are like but if you want a quick uh domain uh top level domain name um you can get that here for totally for free so if i come into into here um i've registered this domain um registered this memorable memorable domain here i mean it's not going to replace amazon anytime soon but um but i have this domain and it's registered for a few months so the reason i show this is because we need we need a top level domain in order to to be able to get open shift set up so the very first step that i did before today because it takes 24 hours to propagate is you come in here you grab these uh you you need to specify where the domains dns should be directed to um so we change it to these custom name servers so let me show you where i got those from so i'm using aws route 53 um and i set up a hosted zone and if we see the code for that when you set up a hosted zone you just it's i think it's create hosted zone and then you specify the the name that you want which is the same name that we just saw and uh amazon aws will come back and give you the top level name service that they want you to use so all i did was take these four these four names here and put them across into this GUI um so that's step one um once we've done that uh the next thing we need to do is is actually get the open shift installer tool um so i'll come back here and there are two uh open shift installer tools um so i will actually delete them and redownload them uh no i didn't want to do that i wanted to do that so actually i just realized yeah one thing hold on one second yeah good okay so yeah so the first thing to do is is to download the open shift installer tools and it's just a couple of w w get commands here so uh you can see it's hitting mirror dot open shift dot com and we're getting the stable open shift installer for linux and then the second one is the open shift client uh this won't take long in the meantime i'll just remind everyone if you have questions or comments just type them in the chat and read them and chris can answer yeah thanks it yeah okay cool so those are done so we just we just extract out those two um extract out those two uh tar files and you can see it just gives us a read me file and a couple of binaries um so i'm going to put our current uh our current working folder on the path um so we should be able to type ot yeah we can so i just was testing that my path was working you can see the open shift client is there so that's fine so i'm going to switch now it's a you and i talked about this briefly before we came on to the call i'm going to shift to a terminal recording just for a short time and i'll explain why that is once it's actually playing um so i'm going to use this tool called Askinima which is amazing if you don't use it um oops i'm going to play play the recording before you play it if you could just resize your window so that it doesn't reach all the way down because your name kind of right yeah yeah gotcha okay absolutely yeah thanks i forgot that would happen thank you yeah that's perfect how's that great okay yeah so oh goodness i'm sorry i can deal with uh Kubernetes clusters but i can't operate my browser and my window let me try again if i go too near to the edge you try to maximize that's great yeah how's that perfect tiny bit off right there we go cool okay good so um yeah so let's start that playing now and um um this is now the recording um but you can see that it's uh it's just showing you can see that this recording was actually done back in june but this is only the first part so we will be doing a live a properly live demo it's just this first part and the reason is because um you can see that i'm in the demo i'm record that i'm downloading those files that we just downloaded the problem is that the open shift install tool actually gives away quite a lot of information that i that you that we don't want to be sharing um you'll see in a moment that when i run it um it shows all of the dns names registered to root 53 for the account the aws account that you're currently logged into it also shows um the secret keys which again i don't want to share so i've actually edited this recording so that the keys you see in a minute are they're not real keys they're they're edited and similarly the dns so here we go so now we're getting to the point where we're we're moving on past so you run this tool open shift install create install config and we're not actually uh creating a cluster yet we're creating an install config i would creating you know the uh the um declarative um definition of how we want our clusters live so we have to specify what public ssh public key we want to use we have to say where where where do we want to run it so we choose an aws um yeah you can see it you can see it was a live demo because i made a mistake you choose which region do you want to run it in so i'll choose us west 2 and this is the point where if i hadn't edited this video it would be listing all the domains here um so that's why i had to edit this but you can see that we go and we're down the list and we choose the domain that we're interested in which is the test domain that you saw before and then we name the cluster and then we put in the pull secret um now i'm gonna pause there just to show you where that pull secret comes from so so anyone following along you can you can search for red hot open shift developer sandbox and if you come here to this website um you can see that you get to you can start a trial and you get a free trial 30 days trial of uh as of an open shift developer sandbox when you sign up for that um then you get this dashboard and in this dashboard you get your pull secret and you basically just need to copy this pull secret here like so and then paste it in so someone asked if you could do uh just uh recap the last 10 commands that you ran i guess maybe you can do even better and share the the script later on yeah i can um just i mean this is an script that installs open shift is it uh doing anything uh special no it's not not yet no it will do we will go into more interesting stuff but let's i think it's better i'd rather recap it here than then ensure recording just in case there's anything hiding in that recording in the file that i don't want to share but yeah so so uh that's fine so so all we've done so far really is um to recap we set up a free domain somewhere we put that into aws route 53 then we downloaded um the two open shift um uh client and open shift install packages which you can google for those uh we expanded those using tar and then we ran this uh we then we put we put uh our current working directory on the path and at that point that's when we then we actually run the open shift install tool and this by using open shift install create install config it's going to create a it's going to create an install config file um and then these are all just this this is all just an easy wizard so you can see it's finished it's created an install config file and this is uh the only the only edit we need to make to that file at the moment is this edit that i've made here so if anyone's not familiar with said what we're actually doing here is we're saying um the install config was created and we're saying search for it and replaced any occurrences of open shift sdn with calico so we're telling uh open shift that when it builds the cluster it should build it with calico networking not with uh open shift networking just to be clear though uh uh this is not going to enable calico ebpf yet this is just calico uh traditional ip uh ip tables data plan just wondering why that's not continuing did i accidentally press yeah for some reason that seems to have stormed um but that's okay i can actually from here i'll pick it up so you can see i've i've pressed control c so we're now back to a live you know actual live live demo um so you can see that the next command you use is uh open shift install create manifests and i won't press enter now because it won't work because i did this already this morning because i knew that this would take too long for the live demo so what that actually does is it consumes the install config and it spits out the actual manifests which are going to be deployed um so it tells that into a folder called cluster so if we go into this folder called cluster um in here there'll be a lot of manifests um and then there's an extra step which is documented um on uh the doc stop project calico website which is actually to download um some extra uh project calico manifests and drop them into this folder so i'll just do one example but they're about 30 or so so you just um it's a curl command like this and we drop we drop uh these manifests from project calico.org into the manual here now when all of that's done then you run um open shift install create cluster uh like so and then it will take about 40 minutes which we'd be pleased to hear we're not we're not going to do that now but um but what happens is it will then go and build the cluster and once you've done that you will have a working kubernetes cluster which is what i have now so you can see that this this cluster was built this morning um in preparation for this so this cluster if we have a look now you can see that um you can see that there's a lot of pods running on this cluster and you can see that in the namespace calico system you can see that we have uh a calico node demon set so this cluster is now running calico networking it's ready to go but the but the um calico node is still running in ip tables mode not in epf mode at this moment so i'm conscious of time time always runs away so i need to run on a bit um so i'm going to demonstrate the the the source ip the lack of source ip preservation so i have um i have a manifest called echo server and if i show you what's in there this manifest deploys um makes a deployment called echo server with just one replica answering on port 8080 and it creates a service um and the service is type nlb network load balancer so this is an aws load balancer set to nlb it's important that we use nlb because without nlb we can't do the source ip preservation um so if i deploy that manifest now created an echo server and i've created a service so echo server external um service load balancer and as well as having a cluster ip it has an external ip and it's answering on port 8385 and it's redirecting to that pod so if i grab that now this won't work yet it takes a moment before it's ready but um if i try to hit hdp and then the the new um load balancer ip and then port 8385 yeah it's not working yet we need to wait a moment um but in a moment we will see that we're able to hit that uh hit that pod but when we look at the pod's logs if you recall back to that diagram that we looked at you we won't actually be able to see the real ip of this my public ip essentially so what actually let's while we're doing that let's have a look if we check my public ip now see that my real public ip is this 31 31 something address so let's see if it's ready yet not yet so uh while this is being created maybe we can take a few questions yeah sure um so there is a question about um support in the epf data plan for ipv6 if you know um i think it's supported but i actually i'm i don't want to say 100 percent because i'm i feel like i might be wrong let me see if i can answer that right now one second um no it doesn't it's listed as a limitation actually so if your users uh let me pull that up now that's i'm glad that i'm glad i checked out i didn't give the wrong information um so if a listener wants to enable the dpbf data plane you can go to docs.projectcalico.org and this documentation is really good you can see that there's some information here about the benefits and there are some limitations and one of those limitations is that it does not yet support ipv6 the question is actually like if you can share or if you know about an eta for that oh uh no i know it's i know it's being considered but i i definitely don't know the eta but if you want to join uh if the listener wants to join our um slack calico user slack that's the best place to ask that question and then i can go away and find out exactly what i'm allowed to say officially and and i'll deliver the best answer i can yeah sounds good uh another question about uh the pros and cons of ebpf yeah security perspective um it really makes i don't think there's a great deal of difference from a security perspective because at the end of the day the policy is being implemented the same policy is being implemented regardless which mode you use uh i suppose that you could say that um the ip table's data plane has been around for longer um and therefore perhaps you could make a case that it was more secure however the um the ebpf data plane requires a newer kernel and requiring a new new kernel arguably has security benefits so i don't think there's i don't think there's a big i don't think it's a big consideration actually um i think i would say they're functionally equivalent really that's a good point actually a question that i also had about uh what's the requirements for installing the ebpf data plan from a kernel or operating system perspective yeah so that's documented uh let me give you the right information um you need a supported Linux distribution which is either um ubuntu ubuntu 20.04 or newer or um 1804 if you have an updated kernel or uh red hat version 8.2 which has a kernel version of 418 or above um or another distribution which another supported distribution which has a kernel of 5 5.3 and above so actually on that same url that i just uh that i just showed you a second ago um if you go to if you just search for project calico enabling ebpf you'll find uh all the information including the the prerequisites around the kernel there's a couple of other prerequisites as well um around mounting the vpf file system uh so yeah all of that is on there all right sounds good cool um i think we should push on just to make sure i get through this in time otherwise we may overrun um so you can see that in the meantime the the echo server is now responding on the port um so if you have a look at the logs um there you go so you can see that we're looking at the logs of the actual echo server pod and you can see that it didn't see the real public ip of the user um so just keep that in mind because we haven't turned ebpf on yet so once we have it should change now the other part i was going to do at this point was to actually demo the performance but to be honest we're getting low on time so i suggest that um on the tigera blog there is a detailed blog post which includes the performance uh graphs so i suggest that um people can go and have a look at that but just suffice to say that the ebpf data plane performance is significantly better um and if anyone you know if anyone wants to test it themselves they can they can follow along and run run their own tests um so let's actually turn on the ebpf data plane now so the first thing we need to do is to switch the encapsulation for calico from ip in ip which is the default to vxlan um so we do that with uh calico cuttle uh what did i miss looks like i screwed up that command one second i think you're missing a another um yeah did i missing one at the end oh yeah i'm missing it at the end yeah i see well done right you spotted that quickly yeah there we go you can see that i ran two calico cuttle commands here there's an ip pool custom resource and this ip it's called default ipv4 pool so what we did is we turned off ip and ip encapsulation and we turned on the x-lan encapsulation um next point because calico needs to speak to the kubernetes api but the kubernetes api is actually hosted via kubet proxy so you can see the problem if we turn off kubet proxy how can calico talk to the api so we have to do something to help it talk to the api directly uh so we do is we use this oc command pretty cluster info and we can see this is where the control plane is actually running um so i've created a um a bit of yaml a config map um called kubernetes services end point in the tigera operator namespace and it has the host name which should be the same yeah that's the same and it has the service port so what we're basically doing is we haven't applied this yaml yet but let's do that now and while it's applying we can talk about what it actually does okay so it's created this kubernetes services endpoint config map which has this configuration and uh the um i'm just going to put it so if we wait one minute now and let kubernetes detect that new config map and then we restart the tigera operator it will actually tell calico to talk directly to the api rather than talking to the kubernetes kubet proxy service um we can see if we have questions uh while we wait yeah sure um yeah i'm just gonna read this one is preserving source ip just an example of uniform software defined general purpose networking rules or is source ip preservation the primary value of calico well there's two parts to that the first part is it's not the advantage of calico it's specifically of the ebpf data plane and actually the way i see it is actually that it's more of a side effect of the benefit of ebpf i think i wasn't in the room when they originally decided to make an ebpf data plane but i think the primary reason for making an ebpf data plane is the performance advantages um but then uh i showed that large diagram of the packet flow through a linux node and because kubet proxy is implemented in uh in the layer three uh green part of the diagram and um ebpf the ebpf hooks are at the start and the end of the flow in order to implement policy in ebpf you kind of also have to replace kubet proxy that's my understanding and if you're replacing kubet proxy anyway then you can you can improve it and i think that was the side effect but source ip preservation is actually a really nice side effect um it's it's something that's surprising was not there by default being able to being able to see the thought ip of your of your user because especially if you consider a compliance type environment you know you need to be able to accurately record your logs i would say it's a benefit rather than the main objective you have another question before we move on uh just a follow-up from the same person uh the incumbent or alternative to the ebpf data plane is calico in calico is based on epitables is that correct that's correct yeah exactly um so if the if the uh if the viewer has time to watch it um i did a talk there was a kubernetes security and observability summit earlier in the year and i did a short talk i think it's about 20 minutes about why we offer multiple data planes and what the advantages and contrasting those are rather than talking specifically about ebpf i said here are the data planes that we have here are the advantages of them all and i really believe that that no single data plane is the perfect solution for all users um because they all have pros and cons um and that includes ebpf it has high performance but uh it has for example it has the kernel requirement um which obviously some some environments that's that immediately means it's not possible um so yeah all right so now that we've waited um if we restart the tiger operator so we're deleting the pod in the namespace tiger operator and it will immediately be recreated um so what we've done there is actually um told we've now told the operator that we want kubernetes uh we want calico to talk directly to the kubernetes api not via kube proxy uh so now that we've done that we can actually disable kube proxy um so you can see that oh yeah i remember now so uh there is um there's an open shift operator and in here we can tell it that we want to turn off kube proxy so right now if we um we have a lurk you'll see that there should be five no six excuse me kube proxies running and now we just patch uh we patch the open shift operator to tell it that we want to turn off kube proxy and as soon as we do that we'll see that the they're terminating already so now we're at the point where we're still not running the ebpf the calico ebpf data plane yet we're still running the calico ip tables data plane um but we've turned off uh we've turned off kube proxy and we've got the we've got it talking directly to the to the api so finally now we can actually turn on uh ebpf and we do that um again this is an operator command so we're merging this config and we're saying that we want to turn on uh bpf the next data plane bpf and the cool thing is that enabling ebpf mode shouldn't disrupt any existing connections so if you have live connections um they will continue to use the standard linux data path until they time out and when they time out they'll reestablish using the ebpf data plane which is pretty cool um so let's prove that we are actually running um it first of all so if we look at now we're looking at all the pods in the calico system namespace and you can see that the modes are starting to restart so you can see that one restarted 14 seconds ago 28 seconds ago so it takes a moment there's still one that hasn't restarted still three that haven't restarted actually so we just need to wait a moment again i've attempted to carry on the demo but i think we have to wait till all six because you know that just by bad luck you know that the workload we care about will be we'll be sitting on the wrong map um of course of course it will of course nothing has exploded on me yet so i'm still expecting something something to explode soon but there it goes we can actually see it terminating nice initializing cool okay so they're all running now so we're running the bpf data plane now there are there are lots of different ways that like we can show that we're running the bpf data plane but one of the quickest ways to prove that we are is to look at the logs for the calico node so if we take any of these calico nodes and we grab grab their logs for the phrase bpf um we should see yeah lots of stuff it doesn't really matter what we see here but you can see that um yeah here we are rescind bpf roots so you can see that it's doing bpf work there is more that we can do if we have time we might i might show you a little bit more but let's first of all check that sort IP preservation again so if we look at the services again nothing's changed this is still the same service still running up time 16 minutes now i'm never sure whether or not i need to recreate the service so we're going to find out so if we curl it again you can see that we've proven essentially that the bpf data plane is working and that remember the service wouldn't be working without kub proxy if we weren't using the bpf data plane so the fact that i got a response from my echo server proves that the bpf data plane is both enabled and working um so if we look at the logs again now yeah there we go you can see now that we have my public ip cool yeah that's good isn't it um so and obviously that means that you're you know you're Apache logs or whatever are going to reflect that and you can start to do maybe another use case actually might be for um understanding uh where your users are coming from your geo gip maybe as well um so there's quite a lot of use cases for that cool um so i think this is good timing actually because we because we skipped the performance part um we can actually we can do two things first one is for me to show you the performance graphs uh and like i say um anyone can read anyone can test this themselves using the same process um oh never mind i thought i was find the blog post um i'll try to find that in a minute um i thought i would find it more easily okay never mind yeah if you if you want to send them later and we can post them in the slack channel for the show yeah perfect yeah that's fine um actually i think i can grab let me see if i can grab it right now yeah here we go brilliant okay actually it's fine i can just show you this on um so this is a a presentation on on a similar topic um here we go so there is this caveat and this is an important caveat um which is that um traffic between two instances which is a single flow in aws can only maximum do five gig um and this is nothing to do with calico this is an aws limitation uh this this this screen grab here is actually from aws documentation just to kind of show that this is you know a real thing um so the reason i say that is because we're testing with a single flow um so this is um open shift with ip tables this is the throughput and you can see that the ip tables is the blue and more is better so you can see that tcp it makes very little difference but if you look at this udp performance it's nearly twice as much um i don't really like spending a long time on graphs anyway so uh you know people can come back and validate this themselves and you can also see that the cpu utilization is lower so but i think more interesting than that let's go back to the cli stuff because people can look up graphs anytime they like uh so let me show you one more other cool thing that we can we can we can see uh and then i think that should leave us with a couple of minutes for any more questions so one nice trick we can do is um we know my ip now so if we create two um variables so i created a variable called ebpf interesting ip and it's my ip my public ip and the other one is uh the interesting port the port number we care about um we can run this for loop and what this is actually going to do is this looks pretty funky when you first look at it but it's not too complicated so just to break it down it's 555 p.m uh my google home talking there sorry um you can see what this is actually doing is it's listing all the pods um it's grabbing the calico nodes and there's one calico node per kubernetes node then it's grabbing the names of those nodes then it's going to iterate over those it's going to print out the name of the calico node and then it's going to kubectl exec so it's going to exec onto the calico node and it's going to dump the bpf connection track table and then it's going to grab my ip and my port and the port we care about so this what this should do is there we go it will show us the flow through the cluster now it's probably timed out the the flow that we did a moment ago is probably timed out already so that's fine so we're expecting it to come back with nothing that's okay okay so you can see at the moment there's no record of my flow but if i run that curl again and then run the command again we should hopefully see that's weird what did i do wrong what did i do wrong can't save you this time no no no i can see that um that's weird i've used that command plenty of times it can't be timing it doesn't time out that quickly so even though you know i run this and then a few seconds later i ran it um okay well i guess this is this was to be honest this was just an add-on thing anyway i was just going to show you can see the flow but uh you what you what what we should have seen there is is the flow coming in on the ingress node and then reaching the workload um and we can actually see that that is not knotted through the flow i don't know why that isn't working um but i don't have time to troubleshoot it now do you have any we're almost up on time already do you do are there any more questions yeah so viewers please ask any questions if you have any um and while you're thinking about your questions uh maybe chris i can ask you about where can people go next if they want to ask more questions find out more about what we've learned today yeah yeah great so um so the best place to go is um doc in terms of documentation the best place to go is to go to doc.projectcalico.org which is is that one i showed here um and you can see that on here there's quite a lot of high quality resources and if you go to um if you search here for ebpf you'll find that there's um quite a lot of information about what ebpf is and and actually you know quite a lot of detail about how it's working and also there are deployment guides for all of not just open shift but for deploying ebpf on clusters on other platforms as well so that's great in terms of documentation um you mentioned you have a slack uh so maybe that's what i was going to say next great place to come is to calico user slack um which i think if you go to tiger there's a project calico community page and well that's taking me back to the same place yeah um if you search for calico user slack you will find oh here we are yeah um so it's this uh tiger.io project calico community and you'll find that um there's quite a lot of information here about how you can get involved um there are these uh certifications which are free which are brilliant uh these are for the open source product community meetings and um you can visit uh you can find our calico user slack here as well um i'm on there all the time and there are people on there as well um shorn and other people who you know who are deeply deeply knowledgeable on on our ebpf data plane so any question you have we should be able to answer wonderful um all right so i think we're out of time and we don't have any further questions perfect so um yeah thank you chris um some one of our viewers said this was uh provoking i can agree uh so uh yeah so thank you for that and thank you for all of our viewers um just a reminder yeah sorry i should say thank you to you though thanks for thanks for taking the time yeah good pleasure and uh yeah see you uh everyone next wednesday every wednesday on cloud native live thanks again please thank you bye bye


Mae'n bosb amser, mae'n ddaeth o'i drwsig â'r rŵl iawn. Rydyn ni'n gwybod hanc. Wrth gwrdd yn ei bod yn ei bod yn assumptions. Rydyn ni'n deall ymlaen i chi, Charles. Yn ychydig i chi, Gwylliannau ein gymneud yma, ac mae'n gwaith er fyddyn ni gondol i chi! Dwi'n gweithio ymlaen i chi. Roedd yn Bryn Peri, yn y profesor o'r unibwr o Oxford i gyfer ydiadol yn gyfrifoedd yn gyllidol mewn ysgol yma, ac yn i chi'n mewn i Kenya! ac rwyf y gallwch i'n gweithio ar y syniadau, oherwydd sy'n 20 pas yn ysgol ym 7 yn ymolod. Rwy'n dechrau'n gweithio'n gweithio a bod hynny'n ddwy'n ddwy'n ddwy'n mewn. Mythaidd, mae'r llyfr am hyn o'i ddwy'n llyfr... Fy angen! Rwy'n credu o'r cyffredig? Mae'n ysgrifennu o'r ysgrifennu o'r cyffredig. ..wys pro-hard talk. Hard talk started under Carlos' regime, and Crystal has always given me a hard time about hammering him. Jimmy has been let off very lightly, but we're going to see what we can do about that. Jimmy, let's start with you.  So my first question is, what did you build on from your predecessor, regardless of your absence of Wikipedia? What did you build on from Carlos? What was the key thing that Carlos left you? First thing, Brian, is that I like your socks very much. Okay, there's no time wasted. Serious penalties. What have I built on from Carlos? I still have most of his stuff. Almost every member of my management team except one or two. Is that a good thing? I've been Carlos' people, so I'm building on what he left. Also, we have to show what the institution has done over time, and many of the things we have to show started on Hank and Carlos, and I'm building on this. Okay, so you're not being specific, you're being very general? I'm not being like the World Bank and FAO.  My goodness, my goodness. Off record, huh? Okay, off record. Absolutely, World Bank is still my friend, huh? And so is FAO, right then and with you. Sit down. As a former World Bank person, I hope that they're very upset with you. Now, yesterday, I mean, all I heard was describing problems, but not showing the solutions, let alone not showing what Ilyri's contributions are going to be. You know, what are they, and where are they laid out? They've laid out in many impact statements. You didn't do your homework very well, Brian. You should have gone wrong and seen all the impact statements, the books that were being put out, all the film clips, all that. Ilyri has had the impact. Did you listen to the minister's speech? He elaborated. Are you just being facetious or you really want to look for facts? You probably wrote his speech, I imagine. That's what happens with ministers. Didn't you? Did you write his speech? We didn't write his speech. I didn't write his speech. I don't know if Ilyri wrote his speech, but if our communication people did not have a hand in that speech, then they should be sucked. Okay. You've answered my question, Jimmy. So this brings me to the question, are you really in control? The CRPs, these wonderful new creations are running the show. Tom Randolph is the man in charge. He's the man with the money and the direction, isn't he? So what is your job? I hire Tom. You want me to do everything myself? Is that how you run your show, Brian? I heard that from your staff. Yesterday, at the end, you presented a dream. I actually quite liked the dream. It was a little bit of a standing on the Mall in Washington. I have a dream type of stuff, which was great. I'm a loaded dream. That's right. That's why I brought the power. That's why I brought the power. What the problem is, Jimmy, is that we can all dream, but I still don't see, apart from your few impact statements and writing the speech for the minister, I don't see really what the substance is. Are you familiar with this? This is something that, this is a white paper, strategic review of livestock in the CGIR. Have you seen this? January 2014, ISPC, white paper, yes I have. Is there anything in there that... I guess you sat on it. Are you doing all those things? We progressed since then. January 2014 is a long time ago in terms of the CGIR. Two weeks ago, we released a draft of a strategic results paper. If you haven't read that, you should before you speak with me. I'm going to move over to... Sorry, I'm going to come back. I'm coming back. Hank, I want to... I've got something in here. Does this ring a bell to you? Does it? Can you make a comment? Can you be with a microphone? I'm trying to remember whether it was you or Trevor who put that on. Well, actually, it was neither, but what... When you came in, if you remember in the various talks that I used to give, and you arrived from Ilker, and you took over as director general of Ilry, I used to have this figure of Hank Fitzhugh coming out at the screen with a Superman shirt on and calling it Ilker Man. You don't remember that? Personally, I was spared from that, but I did know that there was stuff going on down in Nairobi. I wasn't the most welcome person there. Well, I'll come back to that. I just wanted to share this with you. I don't know whether you recognize the handwriting of this, but this is Brian. Since Hank won't be needing his shirt in the future, I thought it should be left to you. Don't forget the cape and the lights. This is today's best wishes Nancy. How kind she was. Hank, are you a Twitter man? Are you a Twitter person?  Well, Twitter allows 140 characters. Now, you were never famed for being able to compress anything into 140 characters, but in something like that, what did you leave behind? What is the key thing that you would like to be remembered for as director general? Hiring me. Your term will come again. An institution that was amalgamating two cultures to be better prepared to serve international livestock research. I believe that's what I left behind. Those were two diverse cultures. Do you feel you succeeded in uniting them?  When I left in 2001, and Carlos and I actually had a conversation about this, I said there are a couple of really weak points, and I won't name them. You could name them if you wanted. No, because they have to do with people. They're still here. No, they're not here. What I said was that most of the difficult people, with maybe one or two exceptions, were now gone. What is it? President company exemptives. As I think of it, there were two individuals that were still there, both about the same height, neither of which liked each other at all. Anyway, it was an institution that was ready to move ahead. There was a strategy that, I believe, was accepted within the CDIR for funding. Part of the challenge of Illry has always been in capacity building. Is that correct? In building capacity. Has Illry got it right? The problem I see is, yesterday we heard in the speech that you wrote, how many PhD students, how many master students, how many people are there. The problem is, it's not students, it's institutions that need to be built. Is it not? What is your view about that, and how has Illry done in building institutions? I don't believe that the individual center's responsibility is building institutions. I take the lesson from what Wendy said yesterday about her experience. One of the principles underlying BECA, in that it was going to be a place in which scientists from many different areas, many different areas would come in and they would have a place where they could actually build on their knowledge. That's a very good example of where you could argue that Illry's done exactly the opposite. Instead of, there are so many national institutes in East Africa, if you wanted to be built in East Africa, why didn't the money go into there? As a result, you built the capacity on the Illry campus and poor old Magugar and Cabetti and Down in Tanzania are still struggling. I take that point, but Brian, I would also have you look back at what the history of the national institutions developed in the National Agriculture Research Institutes and so on have been. Particularly here in Africa where there has been a problem of poor governance and poor support for them. Build up the capacity, I think at the risk of bringing things up. Zimbabwe is a good example. The capacity in Zimbabwe in 1990 went down because it didn't get the support. We needed to have an institution like what the CG centers could provide, sort of almost like the Middle Ages in which the monks preserved it while the chaos went on. OK, I'm going to have to move on. Carlos, is that right that we were... Hank took us into the Middle Ages in terms of... First of all, if you could just respond to that and then I'll come on to something else. No, I think clearly the concept of a beca was that precisely you didn't want to scatter a few resources across the whole continent, but that you wanted to create a capacity at one place to do science with top notch infrastructure. And I've always argued most of the developments are going to happen through the private sector in terms of new breeds, etc. But these countries need a capacity to understand the science so that the public sector can regulate, engage, and set the rules of the game. And I think that's very much what a place like Beca can do. And that was a better place there, Edilary. I agree that you can't spread the resources, the limited resources which were available. You had to concentrate in one place and being an international place allowed us to move and share and operate in a way which international programmes can't. When I talked to you three years ago, I characterised you as the politically correct DG. You were always looking at making sure that things were politically correct. And as part of this, you were the guardian of the smallholder farmer. You heard all this yesterday about how the smallholder is central to Illry. But we also heard these trends. And I asked you yesterday whether you'd seen Mario Herrero, your friend, his Africa futures, in which he shows that if you rely on the smallholder that Africa is going to be importing enormous amounts and it's going to be breaking its economies from that. Mario even said, well, maybe we need larger, medium and large scale farms. That would be heresy in your context, wouldn't it? What is your comment on that? Brian, you helped us write the strategy in 2002. And keep in mind, we had just approved the Millennium Development Goals. And I think so, we agreed that focusing on poverty was exactly what the world wanted from international organisations such as Illry. But that's changed, has it now? I think it has changed, indeed. Yes. I think we focused clearly on poverty. We still came with our CJR baggage assuming that productivity gains were the way to address poverty. I think over these 15 years we have learned that there is a whole array of instruments and that productivity is one, but it's really a much broader perspective. So these instruments, maybe you should tell Jimmy, because this emptiness that I've just described, what are the instruments that Jimmy should be building on now then? Well, I think clearly when we're talking about poverty, we're not just talking about individual smallholders out there. We're talking a lot more about addressing poverty through employment, poverty through livelihoods for some people by helping, as Jimmy said last night, helping other people move out of the sector and really working with the rural transformation. I think what we've been trying to do is to work against the rural transformation and that's such a powerful force, you will never do that. So I think we've become more realistic in terms of understanding that the interplay between social policies and agricultural technology addresses a whole array of livelihood pathways.  Paul Collier thinks about poverty and thinks about all sorts of tools. Carlos brought up this business about the Millennium Development Goals. We've now got the Strategic Development Goals or they are emerging. Sustainable Development Goals, I beg your pardon. Thank you. I've got so many bits of writing there. There are now 17 of these, Jimmy. You've got these four areas that you had everybody go through yesterday. FAO's got its three or four areas. I mean, isn't it bizarre that you're trying to create your own little... Why don't you stick to the Sustainable Development Goals? Why don't you use them as the framework? Because livestock is relevant to all of them. I see you've even given a talk on that. Yes. To address them, it doesn't mean you need to fold yourself into them completely. You map yourself to those goals and we have mapped ourselves and we map to most of them. You just said, I spoke about them. I must have done so. I usually think about what I say. That was a PowerPoint presentation that somebody prepared for you. Yes. I get a lot of help, Brian. I have to admit that and I have good health. Just look around the people I have around me. Yesterday we heard, for example, a comment saying, well, this should be addressed through the climate change gender group. This should be addressed through some other group, a livestock group. Surely everyone coming in and focusing on what their contributions are to SDGs, which is on human well-being, is going to be much more useful, bringing people together. You can't have everybody as far as you're concerned. All of Ilri should be one unit. You can't manage that way. You put things in relation to how they function. You manage in these functional groups. That doesn't mean they don't relate to each other. They relate to each other. The CGIR has picked three things, not four things anymore. Poverty, food security and nutritional security, and natural resource management. Those are the three things. If you look at those three things, they map to many of the sustainable development goals. One thing is that Ilri has got its strategy. We were a bit critical about it in this white paper. We also said that should Ilri's strategy be the CG-wide livestock research strategy? What is your comment about that? Ilri's strategy is informing the CGIR strategy as well as driving the agenda. But there isn't a CGIR livestock strategy? Well, there's not a livestock strategy, but a CGIR strategy that sets things at a very high level. Livestock is a part of that. Livestock is a part of the poverty agenda. It's part of the food and nutritional security agenda. And it's part of the environment. You don't need to extract it uniquely from that. What we need is things to work together and relate to each other and not separate them. I'll come back to that, Hank. Listening to this elegant prose here. Hank, what would you have done differently if you were in Jimmy's shoes today? And very nice shoes they are too, Jimmy. All I'm missing is a sock. Hank, what would you want? Because the environment has changed. The CG has changed its mode of operation. Have you any thoughts on what you could have brought to those type of pressures? Well, my thoughts really spin off of what Jimmy said. The livestock strategy for the CGIR is that the development of livestock fits within the overall agriculture. If you try and pull livestock out as a separate commodity, a separate entity, and just deal with it, you're not going to be successful. In fact, one of the problems of the 70s and 80s with both Ilca and Ilrad is that they did that. They said, we are the livestock centers. It's sort of like saying, cement, we are the wheat center. Instead of focusing on the fact that Acarta and others were dealing with wheat as well. So what's important, and I would build on the question that you said, the ill-re strategy, and I think it does by looking at the CRPs, should also be thinking, should be built into that, is what are the CRPs going to do? And not only the one that John's leading and Tom's leading, but also what's going on in Acarta and what's going on in the other centers. Well, we were quite critical in this white paper because we felt that there wasn't enough integration. And for example, many of the feed centers are not looking at the feed for livestock types of issues. How do you make that happen within the CG? Well, Brian, they're doing more of that now. The whole purpose of the system-wide livestock program, which I would say is informed the development of the CRPs, is that success was going to come from looking at food feed. And that meant you brought in cement, you brought in all of the crop centers and got them built in. And an awful lot of effort, Salvador, Jimmy, all of these put a lot of effort in making sure that the resources, human capabilities, institutional resources of those centers, with their partners in the national programs, fit into the SLP. And I actually think that the SLP was very successful. It was terribly underfunded. So it's one of the things that you look back on in your era very positively. Hank, you are now with biodiversity. Carlos, sorry. Carlos, you're with biodiversity now. All sorts of interesting questions come in the livestock, animal genetic resources issues in livestock. I don't know whether you're directly involved. So here we've got, not Paul Collier, we've got others in the sustainable intensification program, calling for these various activities, including getting the right genotype. And there's been this dramatic narrowing of genetic resources as a result of the intensification process. What is your view about that? How do one get this balance between retaining indigenous genetic resources in livestock, but at the same time moving this sustainable intensification process forward? Well, I think the key to it is the fact that the diversity of the environments in which you will have to produce food. This is the big difference with the Green Revolution of the 60s, where you had a very homogeneous environment, you had one package, IR8, 100 kilos of nitrogen, et cetera, and that was broadly applied and was quite successful. That was the easy game. The reality now is you've got to produce in a lot more marginal locations, more diverse environments under rain-fed conditions, and that absolutely requires a lot more diversity in what you're going to do. So I think the sustainable intensification principles are going to help us think through that, but you will have to tap into those genetic resources to adapt it. And the whole argument I guess Steve Camp and others make is with these powerful instruments we have now in genomics, et cetera, we're going to be able to customize those materials a lot more. So we will have a range across large spaces, but obviously we will specialize in individual niches. OK, and that one that was yesterday was extremely interesting, but Jimmy, in the Illry strategy we wrote when we did this white paper, we said that the genetics component was more of a shopping list than a plan and a program. What do you say about that? You jumped the conclusions. You didn't understand what we were trying to do. We were thinking through consulting with partners, looking at where the science is going, and evolving or thinking about what we should do. We presented to you yesterday our plan. You shouldn't make the same, I don't think you can make the same judgment about this plan as you were when you were writing your white paper. But Brian, I know a characteristic of you, why you're a good scientist and all that. You like to find the critical things. You like the juicy parts. Well, that's the reason why you're director general, Jimmy. You've got that great ability to give those wonderful smooth answers that don't answer the question. In the way you would like me to. I've just been given a sign about time. Communication with others. There is this very interesting initiative that the global agenda for sustainable livestock, I think Henning is around here, has been very much part of that. That has been very successful in bringing a wide group of stakeholders together. Is this something that you think is useful or is it just a detraction and a waste of time, Jimmy? Henning can tell you that I chaired the process of starting that agenda until I became DG, where I felt that I couldn't one afford the time, and two I couldn't prejudice the agenda. I was only interested in research, but I was the chair of that group. With FEO as the Secretary for the 4, 6 or 8 meetings when I was in the World Bank. So it's something I believe in. I'm a supporter, and Shirley has continued to participate, and other staff members have continued to participate. It's had a great success in terms of raising awareness within the commercial sectors, the JBS, the Nestleys, the McDonald's, et cetera, who have really taken on this issue of greenhouse gases and livestock production. How on earth are you going to bring that type of capacity to respond to Africa and Asia? I don't know. Nestle and others will respond. That's their interest. We are in the tent. It's a big tent. We created a big tent for these relational aspects that we talked to. We cannot isolate the solutions of livestock from everything else. So we created this big tent where we can try to look at what are the issues, what are the priorities and so on. For us, the issues are what are the researchable issues that we can reasonably contribute to. I don't know how Nestle and McDonald's and all the rest of them will do that. We are only interested in that from the standpoint of what sort of products they will be sourcing and how can we provide the products that they want to source? My last question to all of you. Is Ilri too DG centric? Has this harwch? Should you have a junta? Should you have a pollet bureau or somewhere where you've got a diversity of opinions? Are we putting too much into the hands of one individual in running this livestock program? I just told you I had a lot of help. They're right, PowerPoints and so on. So we get a lot of help. No, I don't know if it is DG centric. I thought so when I first came to Ilri. People like to hear the DG speak. I remember going to a fire alarm one day, soon after I arrived, and everybody had gathered in this zone where you're supposed to gather in the case of a fire alarm. And at the end of the drill, it was a drill, at the end of the drill they asked me to say something. I don't know what I did to say about a fire alarm. It was Martin who was supposed to... Carlos and Hank, do you have a last comment on that? I think you need a capacity to engage more widely. I think one of the challenges we've had is that we've been very self-referential. And I think we tried in Ilri to be much more open. And I think the management team, the way we were operating when I was here, was very much engaging everybody around the table. And I think we need even a lot more external inputs into the system. Hank, you have the last word. Well, you've heard how Ilri has evolved. Brian, what you're thinking about are the DGs of the 80s. The Empress and the King. That changed. Am I not? I would like to think that some of those changes became apparent during the 90s. And I'm really impressed, positively impressed by the changes that came on. One example that I've noted, and I think some of you would have verified it, I was always referred to by my first name. A lot of people kept trying to push it was going to be doctor this, doctor that.  Everything now is on a first name. You don't identify the DG up on a podium or whatever. Okay, very good. With that, thank you all very much indeed in the land of former Emperor Haile Selassie. Thank you very much to Emperor Jimmy Smith, to Carlos. Thank you very much indeed.


What did you think about building that foundation? Well, I mean I think a lot of this thought work So, you know, it's really like taking the time to sit down with yourself and a notebook and thinking through you know, who do I want to do work for and What do I want? How do I want them to see? See the agency that they're you know hiring so I think for me since it was me By myself for a long time A lot of that was really, you know my values and things that were That mattered to me that I wanted to make sure came across, you know I I always want integrity to be a big part of what I do want people to trust me And I don't think you can be in business with people if they don't trust you or at least you won't be in business with them very long so You know that was a big one for me. I wanted to make sure that I Was able to like handle the work that was coming into so like I was allowing it to kind of the agency to grow organically and not You know not trying to force Things in there, you know that we would be able to to take on the work and do it Well, and I never want to send anything out in the world that I'm not proud of or at least can't stand behind And so I think yeah, it's a lot of thought work. It's really like Getting into yourself and learning, you know, what's important to you as a person and and how that can translate into your company Because if you can't stand behind it, you can't ask other people to yeah That's fantastic because you're going to have a brand you're going to have a culture that your customers in the community and The industry knows about and whether it's the one you want or not Yeah, if you don't do that upfront thought work It becomes whatever it defaults to you with either how you how your customers work with you your clients work with you And your employees interact as well. So so that's great That's not something you hear about a lot someone taking the time and care to think all that through and be prepared for that workload and work Flow to really give clients all that attention that they really need Yeah, I don't think a lot of people talk about it, but I do think a lot of people are doing it So like or at least trying to do it. So, you know successful companies I think have to have done that at least subconsciously right because they're setting up cultures that People want to work for that people want to work with yeah I find that companies hit a point where they go. Oh, we need to be really conscious about this now We all and I know is my experience to I'm just gonna get out there and do it and then I hit a point If I need to be really conscious about the brand and to be really conscious about the kind of clients I want to work and then so that's almost like a new beginning of the company Yeah, for sure And I think you learn some of that stuff as you go to like I definitely You know, I jumped in both feet in like I wasn't you know I didn't know everything like I said like business was not something that I went to school for you know I went to an art school for design. So You know you learn some of that as you go and you you know hire people to help you You know I you and I had a lot of conversations and a lot of you know coaching to to help with some of those things and I think especially yeah, like just learning how to even like things like you know the time management and figuring out how to Just like doing some of the budgeting things like those weren't think they sound like I Don't want to say small things but they don't sound like huge things in the grand scope of everything But they make such a big difference when you're trying to figure out how to run so many things at once And I had to I was trying to balance two at the same time So I'm trying to build this thing up on my own But he's already done a lot of launching programs and he's got staff and so forth So he and I've been talking about partnering up lately. That's my new model approach You know, that's the way it is with startups, right? Always pivoting if you see a different way to get there the idea is just to get there no matter what you do You're always trying to figure out. What's the fastest way? So that's fantastic. It's a perfect lead-in. What's the there? Where are we taking laser tag fitness right over the next 12 months 24 months? Five years. What's what's the plan for world domination? Yeah, nice. So I'm following the kind of standard Startup entrepreneurial model right first thing I did was an MVP. I just wanted to see if this would even work I got some equipment we went out to I made a deal with the city of Las Vegas use their outdoor football fields and Just proved that it's a workout, right measured heart rate We counted 12 calories per minute, which is you know right up there with rowing and some of the most intense workouts That you can do so the MVP was a success then I went for the minimally marketable product So people just to prove people are willing to pay one is organizational structure I think there's a lot of organizations that can learn from the military and How to implement communication and leadership structure into their organizations? But more important than that and it's something that really has played a strong significant value in my life is Values and and really understanding how if you have a strong set of core values that can be Be a magnet for people to come to you or but it can also repel people away That don't believe in the same things that you believe There's a lot of playful banter between all the military branches again This is before I got into like a lot of personal development that comes with the DJ business And I know we'll get to that but like I was just a worker bee Like when I ran the the youth academy I was like the highest ranking person on on the grounds normally in charge of like 200 kids But it was a very high stress high impact like every every time my my radio went off It was normally a fight or a kid wanting to quit it was always the these crisis decisions What was beautiful about the electric thing was I had taken one class for that whenever I was in high school But it was a way that I could just go release and just I did have my boombox from the Marine Corps I would play music and I would like you know get a whole crew pumped up and we would go wire houses or we'd wire commercial residential whatever a lot of industrial stuff and It was just a place where I could listen to music and just think and get the job done And I didn't have to make any big decisions. It was like hey Here's all the stuff you need to wire this job You got two days get it done and it was just a paycheck But it the funny thing is to answer your question.  You don't want to do this for the rest of your life He goes I have this friend. He owns a DJ business. You should go talk to him and when I went and met Jason Bailey I had no idea what I was in in store for But that guy changed my life. He's the first person that like put a book in my hand and said and when I walked into their company culture DJ connection at the time was like the largest DJ business in the state of Oklahoma They dominated Tulsa, Oklahoma City and when I walked into the environment, it was like all these guys in suits But it was like all this fun crazy stuff hanging up all over the wall. Everybody's on the phone. Everybody's energetic There's people doing like, you know, big claps and all that stuff like during phone calls and I'm like, what is this place? And in short the poetic if this is entrepreneurship like this is the heart of it and that's where I just got hooked and Little would I know that everything that I learned from being a drill instructor from being in the Marine Corps I didn't understand how much that would carry over Into the wedding industry now.  I like being organized I like doing that and it just that's one thing that comes in in handy for not only planning weddings But executing weddings and all that stuff. It's like seeing the problem before it happens and just being ahead of it Well, and one of the cool things about being a wedding DJ When you're a club DJ You're a club gj night in night out Different crowd but all looking for the same thing when you're a wedding DJ You are making the most important night of two people's lives The most special night they're ever gonna have. I mean, that's there's a lot of a lot of satisfaction that can come with that besides the money Yeah, and and you know, I I that's what the purpose was for a long time It's like I I get to be a part of this But now as I sit back and I look through all the photos and I see all the videos and like I look at What I have and how important that is for me I have a front row seat to love every weekend And I get to see that's gonna be the title of your book Yeah front row seat to love Now I think that so where that delegation comes in is having an even more crystal clear vision of where I think We deserve to be And then saying here are some of the things that we need And what I've been fascinated by is that there's so many people that are willing to raise their hand Or they know the exact person That they can introduce me to that would love to step on board. So that's been one thing Uh, one of the most challenging things that I see for for young Especially veteran entrepreneurs because that's who comes through our cohorts Is a lot of times like we give them this platform in a stage when they graduate to say, what do you need? And they'll say, you know, I just you know, follow our social media and just we'd love your support and it's like No, you need an introduction to the director of operations at mgm Because your product or service has to do with that Like that's what you need to ask for and if you ask the right like for the right thing People raise their hand and be like, oh my gosh You need to and I say that because there's been a lot of times where people have asked that And I've been introduced to those people and and when I meet somebody and this isn't an ego thing It's like I'm fascinated by how many people I met during global entrepreneurship week that were like Are you Dave berlin? Four people have told me this week that I'm supposed to meet you and then they tell me why I'm surprised It was only four and I'm like here. Here's who here's who you need to call. Here's where you need to go Here's what you need to do and just amazing things starts to take shape from that. So


Hi there, I'm Sandy Alnok, artist and paper crafter here on YouTube with one of three coffee cards. This one is layered coffee cups using Avery Elle's Cool Beans stamp set. The other two will be coming soon if they are not trailing on the heels of this one. I planned a coffee week but my schedule keeps getting upended so might be some other videos in between but stay tuned for more coffee cards following this one in the near future. I've stamped my images onto some cardstock from Neenah and this is Desert Storm. This is not the same as craft just so you know that the color might be craft but the paper is still Neenah so it's going to work with your markers. Craft cardstock depending on what brand may or may not accept some of your Copic ink. So I'm going to use a couple different colors of red to make some shading on the outside. I'm keeping it really simple. If you're somebody who's just learning how to blend doing an idea like this where you're making an image that you can keep coloring out past the lines may help you to suddenly get the idea of blending because you don't have to stop at the edge of the image. You do have to die cut or fussy cut them out and there's that but if you're trying to learn how blending works this will help you because then you can just have a swash of color and not stop the marker because stopping the marker is always the hardest part trying to keep the ink from bleeding outside the lines etc. I decided I wanted my hearts to be white they will eventually and I know this when I'm doing it they will eventually be a little bit I can't even speak pink because the ink that's in the pen may absorb some of that color from underneath and a strong color like red is more likely to do that than a lot of their colors and that's okay with me because pink parts on a red coffee cup would be cute. I've also stamped it onto a piece of solar white cardstock so that I can make the lids for the cups and you could do this on a little scrap as well you don't have to have the whole cup just enough to get those cup tops on there and notice the shading that I did on the inside of that lid there look how it looks like it's receding in there really great tip to make something look like it's going down into it is to give it that kind of shading and the shading I'm doing on the cups no lids there's no science to this because in every lighting situation the lighting will be different on cups like this it's just the nature of it so make it good enough that people's eye is faked into believing that it's for real and then you'll be fine don't stress out about it. I stamped my images onto another piece of the Desert Storm so that I could do my shading without having to worry about coloring right up to the edges of the image and it also gives me a guideline for putting my cut out pieces down there so they were all fussy cut with my little detail scissors and then I can just slap them in there using some flat adhesive for parts of it and then dimensional for other parts so those were done with flat adhesive and you can use ATG guns or tape runners or be creative tape lots of different kinds of things that you can use on the back of that and I'm even gonna use that I used the dimensional adhesive for the second cup and flat adhesive for the wrapper on that one because that one doesn't need to be popped up as well because I don't want this to be too dimensional and then have to cause extra postage at the post office. Tip for cutting out when you're cutting out something with white edges around it like this I used to try to use a black marker since the edge is black because of the stamping but I found that I can use some sort of a gray in order to do that and then I don't have maybe black ink seeping into the image because I've had to recolor some sometimes. The dimensional adhesive that I'm using here is Tombow's power tabs super sticky and they're not as thick as the role of dimensional adhesive so it works really well you get a little bit of dimension without actually causing the post office issues and I'm gonna add it to a card base with two layers of die-cut cardstock as well just so I have a little bit more finesse on them not just the coffee cups themselves and it looks kind of clean and simple but has something a little fancier along with it and I love making thank you cards because I send out a ton of thank you cards especially to my wonderful patrons thank you to all who have joined in in this last couple weeks I appreciate your support and links for all the supplies are in the doobly-doo down below and you can watch more videos but most of all go make something just go make something and send it to somebody in the mail I'll see you later bye