So I'm heading in to do my first ever keynote speech. I scrub up well, don't I? Yeah, maybe, whatever. Rock up to Optus Stadium and have a look at that ground. Yeah, oh look, there's me on the TV. Yes!  How's the spread? But most importantly, how's the view? Damn, this is goals. I can get used to this. And then they brought puppies in to have a cuddle with. How's the turnout though? Over 300 people rocked up to State of Social 2 here. Some amazing people talk, including me, talking about TikTok. Okay, so my turn next. Getting kind of nervous, miking myself up. Here we go. Hey guys, how are you? Just want TikTok as well. Got a content there. My first time went pretty well, and I've got a lot of positive feedback from it. I'm going to upload some footage of it soon. Thanks for watching.


Namaskaram Kochi The queen of the Arabian sea is wonderful as always. Being among you all makes me very happy. Two years ago I went to the Kochi refinery. This is one of India's most modern refineries. Today once again from Kochi we dedicate to the nation the propellant derivatives, patrochemical complex of the Kochi refinery. This one project will help send our journey, our being, Atminderbhar. Thanks to this complex, foreign exchange would be saved. A wide range of industries would gain an employment opportunity would be generated. Dedicated of RORO vessel to the nation is special. A distance of almost 30 km on road becomes 3.5 km through waterways. This means convenience up, commerce up, capacity building up, congestion down, pollution down, transport cost down, saagrita cruise. Terminals bring both comfort and convenience for tourists. It will cater to over one lakh food gates that is so much to see, learn and discover. I urge our young kind of friends to think about innovative tourism related products. I also urge you all to use this time and travel to as many as nearby areas as possible. You would be happy to know that the tourism sector in India has been going well in the last 5 years. In the world tourism index, ranking of India moved from 55th to 34th position. But there is a lot more to be done and I am confident this year's budget has devoted significant resources and skills that will benefit Kerala. This includes the next phase of the 40 metro. This metro network has come successfully and has set a good example of progressive work processes and professionalism. The year gone by made humanity face are never seen before challenge. Powered by hundred thirty crore Indians, our nation's fight against Covid-19 has been spirited. The government was always sensitive to the needs of the Indian diaspora specially in the Gulf. As a part of the Monday Bharat mission, over fifty lakh Indians came back home. Many of them were from Kerala. It was our government's honour to serve them in such a sensitive time. Indians working in the Gulf to know that the full support of my government is there to ensure their welfare. People have shown that with the right opportunity they can do wonders. Let us keep working to create those opportunities. Together we will build an Atmanirbhar Bharat. Once again I congratulate the people of Kerala for the development work they have been inaugurated today. Thank you.


Hello, everyone. My name is Karan. I'm passionate about systems engineering and reliability. I work at Walmart, where I'm responsible for developing traffic platforms. Today, we are going to go through an overview of leaf. Leaf stands for lightweight EBBF application foundation. We acronym it as leaf because it's based on EBBF and is extremely lightweight. Let's go through what we plan to cover today. We will start with a brief introduction to EBBF and leaf, followed by an overview of the leaf that form an EBBF solutions. We would then like to talk about our open sourcing plans and walk you through a demo. EBBF runs as a mini VM inside the kernel. It essentially provides a sandbox environment to insert code into the running kernel. Let's understand this in a little bit more detail. The Linux kernel is fundamentally event driven applications that are running in user space, make system calls to the kernel. A couple of examples of system calls include reading and writing from desk, connecting to applications, reading from sockets. We also have hardware devices like network cards, disks, USBs. And these hardware devices send interrupts to tell the kernel that certain data is ready to process. Example when a packet arrives at the neck or when the data is available for reading on the desk. The kernel which you see in the middle handles all these events. In cases where we would like to extend the kernel's functionality, the traditional approach has been to build your own custom kernel by writing modules upstream and wait for the next kernel release. EBBF presents a new model which allows us to extend the kernel functionality through simple programs. These programs can be associated with desired kernel events so they are executed whenever the event happens. For example, we can run an EBBF program when a packet arrives at the neck or when an application makes a system call to the kernel. So in a way, EBBF programs are to kernel, what plugins are to proxy or web servers. EBBF has out-of-the-box integrations with low level networks such as XTP, TC, as well as probing mechanisms such as K probes, U probes and trace points. Now EBBF also provides a safe and secure way to write efficient EBBF programs and run them in the kernel. The verification step ensures that the EBBF program is safe to run. It validates that the program meets several conditions. For example, it makes sure that the program does not crash and that it always runs to completion without sitting in infinite loops, for example. The just-in-time compilation step translates the generic bytecode of the program into machine-specific instruction set to optimize execution speed of the program. This, in other words, makes EBBF programs run as efficiently as natively compiled kernel code. The popularity of EBBF is rapidly growing. There are more and more EBBF programs being written to solve a wide variety of problems. Several startups are building technologies around EBBF and large technology companies like Facebook, Netflix, Microsoft are embracing EBBF to solve large-scale problems. At Walmart, we too are embracing EBBF and using it to solve similar problems. A challenge we faced when first adopting EBBF was how to manage and orchestrate multiple EBBF programs on a large scale. We require to run numerous EBBF programs on a given node and we have thousands of nodes across many DCs in a hybrid cloud environment using multiple cloud providers. Due to the lack of an enterprise-ready solution, we decided to develop our own platform. LEAF provides complete lifecycle management of EBBF programs with the help of an advanced control plane that has been written in Go. This control plane orchestrates and composes independent EBBF programs across our infrastructure to solve crucial business problems. As we proceed further, you will hear me use the word kernel functions interchangeably with EBBF programs. We refer EBBF programs as kernel functions as they allow us to extend the kernel function. Coming back to our control plane, our control plane consists of two major components that work together to orchestrate kernel functions. The first one is deployment APIs, which a user calls to generate configuration data. The configuration data includes which kernel functions will run their execution order and the configuration arguments for each kernel function. The second one is the LEAF demon, which runs on each node. It reads the configuration data and manages the execution and monitoring of kernel functions running on that node according to the data that's available. In this way, we have simple APIs that allow users to add, remove, and reorder kernel functions on the fly. And these APIs sit on top of a distributed model to manage and configure kernel functions on a per node basis. Outside of the control plane, we have developed many EBBF-based kernel functions that help us replace proprietary applications and hardware with blazing fast EBBF code. We do have a section later that provides more details about this. Together, the control plane and the EBBF kernel functions provide kernel function as a service. In the next slide, we'll try and understand how this kernel function as a service model works. Now, as you can see on the top right corner of the diagram, EBBF programs can be community developed, third-party vendor ones, or the ones that the Walmart LEAF team has developed. We have a LEAF build engine which pulls the kernel functions, compiles them, and pushes the bytecode to an artifact management solution. Now, when a user wants to deploy a kernel function, he can call a LEAF D API to provide configuration data. Once LEAF D reads this new config, it orchestrates the kernel function on that Linux host as per the defined parameters. If the user gives a set of kernel functions, then LEAF D can orchestrate all of them in the same sequence that the user wanted. Executing kernel functions in a sequence is called chaining, and it is one of the most critical operations that LEAF can perform as a platform. This whole workflow is in line with the build ones deploying everywhere philosophy wherein we would build the deployment package once for any environment which in this case would translate to multiple kernel versions and set configuration at deploy time. But I walked you through an example of a new kernel function deployment. All the operations that we discussed in the previous slide can be triggered through APIs. In the next section, we will take a closer look at the EBBF kernel function ecosystem. EBBF programs have the capability to instrument, inspect and interact traffic while providing enhanced performance observability. This slide gives us an overview of use cases. And as you can see, these can be broadly classified into three main areas, networking, observability or tracing and security. In the realm of networking solutions, the combination of programmability and efficiency makes EBBF a natural fit for all packet processing requirements. We have developed EBBF programs for various business use cases such as packet tagging, traffic mirroring, traffic direction and layer 4 road balancing. Similarly, we have started providing deep visibility into system and network performance. While traditional solutions rely on static counters, static counters and engages exposed by the operating system. EBBF enables the collection and internal aggregation of custom metrics that are generated from a wide range of sources. We're also actively pursuing the use cases of tracing and profiling that allow unprecedented visibility into the runtime behavior of applications and system itself. On the security side, EBBF allows us to combine the visibility and control from all aspects to provide functionality that operates on more context with better level of control. Some of the EBBF programs that we have developed allow us to export net flow logs and perform connection and connection rate limiting. We are also partnering with our security teams on use cases around deep packet inspection and DDoS. In today's presentation, we'd like to talk about some of the networking and security solutions that have been developed by the LEAF project. EBBF is developed and managed by a super smart and enthusiastic community that is actively working on adding new features to it. We believe that EBBF will eventually become the modern software defined networking solution of the cloud and cloud native era. There are a lot of networking and security solutions in the market that are based on EBBF. Celium and Calico are good examples of CNIs for Kubernetes. In this slide, we're going to talk about some of the EBBF based networking solutions that have been developed as a part of the LEAF project. Hopefully these examples will showcase the potential of EBBF. I mentioned previously that EBBF provides us with a mechanism through which we can plug in our code to pre-defined hook points in the common. Here we are using a network hook called TC, which is traffic control. All of the ingress and egress traffic goes through these hook points. Once we place our code at these points, it can see all the traffic and process it. For example, we can extract the information that we need and act on it, pass the data onto higher layers of stack, drop the traffic or redirect the traffic to some other host. It can also manipulate the traffic by changing certain fields of the packet depending on the requirement. All these are done in the kernel, so it yields the best performance as we don't need to carry the packet all the way up the stack to the user space, which would then involve multiple context switches and processing resources. Let's go through the EBBF solutions then. The first one mentioned here is FlowExporter. As enterprises start serving live traffic out of public clouds, it is increasingly important for them to export traffic flow data to security solutions that provide an advanced threat protection across the extended network and cloud. Private clouds provide traffic flow data through dedicated network appliances, which are like hardware-based solutions. However, tenants in public clouds do not enjoy a similar level of access or network visibility since the infrastructure layer is shared. We considered options to address this, such as adding a network hub to process traffic flow data. However, such a configuration increases traffic latency and also adds another layer to manage in the traffic ingress stack. As the best solution, LEAF developed an EBBF kernel function which extracts and exports flow metadata directly from Linux-based edge proxy servers. This essentially eliminated a hop in the critical ingress path, reduced the site latency by 50 milliseconds, and also helped us save on the licensing and operational cost involved in managing the additional hop. Next is traffic mirroring. The best way to succeed in a business is by providing an amazing customer experience. At Walmart, we want to have visibility into how our customers are interacting with our site. We have a few analytic solutions that can operate on the data streams and provide the needed analysis. But these solutions need the data of interest, and that interest changes from time to time. There is an opportunity to save valuable time and money by automating the process of collecting this data. One of the most effective ways of collecting this data of interest in the public cloud is from the edge proxy servers. However, it is also critical hop that handles all of the interest traffic to the site and is performance sensitive. We again saw the potential of using EBBF and developed a kernel function that encapsulates the filtering and mirroring functionalities together. This solution supports one or many custom filters and also allows us to capture only header data if required, thereby limiting the bandwidth utilization. Additionally, given that EBBF is very lightweight, highly performant and safe, this solution has been implemented at the source, which is on the edge proxy itself. This allows us to eliminate multiple hops in the traffic paths, reduce latencies, and the licensing costs of buying and maintaining third party vendor solutions. The last one on this slide is packet typing that we use to ensure quality of service in the stores. Like I mentioned before, EBBF can not only perform read only processing, but it can also be used to manipulate the traffic. This is one good example of packet manipulation. The bandwidth availability can be limited in stores and so it is extremely important to use it optimally, especially in peak traffic situations like the holiday season. Since payment transactions are critical for any business, we do prioritize them over other kind of traffic. The way we achieve this is by setting the DSCP tag on the packet so that the egress routers can see the tag and allocate dedicated bandwidth to the tagged packets. This ensures that even in the event of congestion, the payment transactions don't fail. We have a few more kernel functions here that are based on XDP, which is ExpressDataPath. With the advent of XDP and EBBF, it is now possible to achieve high performance packet processing in the kernel data path. XDP allows us to attach an EBBF program to a low level hook inside the kernel. This XDP hook, implemented by the network driver, provides a programmable layer before the Linux networking stack. This makes XDP the de facto choice for use cases wherein we must drop or redirect the traffic. Let's go through a couple of XDP-based EBBF solutions that have been developed by the LEED project. The first one is concurrency and rate limiting. As enterprises increase their digital footprint, it is vital to have safeguards put against certain bursts of traffic or cyber attacks. Additionally, sometimes due to certain bursts of traffic, the upstream applications can slow down to do various reasons, which in turn can cause back pressure at the edge causing the connections to pile up and to overall maverage infrastructure. Having a connection limiting and connection rate limiting feature allows us to limit the concurrent number of TCP connections and the rate at which new TCP connections are established. Adding this functionality to our edge proxies and load balancers protects our compute resources from getting overwhelmed when there is a sudden burst of traffic that is beyond what our resources are capable of handling. There is also a recent exciting offering from LEED that is Layer 4 load balancer. Several internet companies serve millions of requests every second out of their edge network. Since the Layer 4 load balancer must process every incoming packet, the solution needs to be highly performant and also must provide the necessary flexibility and scalability required in production environments. Traditionally, Layer 4 load balancers have been hardware based primarily to suit the high performance requirement. However, taking a hardware centric approach limits the system's flexibility and introduces limitations such as lack of agility, scalability and elasticity. As applications increase in number, complexity and importance, it is vital that the infrastructure layer is app focused and not limited by the confines of hardware configurations. All the performance needs can be met with software solutions themselves using EBPF and XTP. The LEED project has leveraged Katran from Facebook to develop an EBPF based load balancer offering that is implemented in a hairpin model on a single neck. One of the key features that we wanted to enable in our production environment is DSR or direct server return so that we can send responses directly to the client. This ensures that the ELB does not need to handle return packets which are typically larger in size. To implement DSR, we developed an ELB agent that runs on our fleet of hypervisors. For other environment types, we run the agent on VMs, bare models, etc, depending on the use case. The agent can run on any commodity hardware that is based on Linux. ELB is helping us replace hardware based solutions with a modern software based solution. Enabling DSR not only eliminates centralized choke points in our network but also helps us improve the overall site frequency. Since it uses XTP, it is blazing fast and it can process the traffic at near line rates, thereby helping us reduce our infrastructure footprint considerably. Both the above XTP kernel functions can be run in a chained fashion to work cooperatively with each other. This can ensure that all the illegitimate traffic is dropped by the connection and connection rate limit functions and the ELB doesn't have any undesired effects even under adverse conditions. This concludes our section on EBPF solutions. Next, we would like to walk you through our open sourcing plans. At LEAF, our vision is to create a marketplace for EBPF programs where users and developers can share their own signed EBPF programs and download EBPF programs from others. Our LEAF control plane can then be used to orchestrate and compose selected EBPF programs from the marketplace to several business needs. In this way, LEAF provides developers with a cloud and vendor agnostic platform for adding capabilities to the operating system at runtime. We believe that the creation of such a fully integrated software ecosystem around EBPF will unleash its full potential for community adoption. A vital prerequisite to the kernel function marketplace is the open sourcing of the LEAF project, which has been a top priority for Walmart and our team. I am glad that we are open sourcing the project today and would like to thank the Linux Foundation team for helping us through every step. We would like to encourage everyone here to adopt and contribute to the LEAF project. In the next section, Brian will walk us through a demo on how we can set up our dev environment and start using LEAF. Thanks, Karan. Hi, everyone. My name is Brian Merrill and I'm a software engineer on the LEAF team. I'm excited to be giving you a demonstration of LEAF today. Before we actually drop into a terminal and start running commands, I want to give you a quick preview so you have some context going into this demonstration. So if you look at this slide, there's two main boxes on the right hand side is our virtual machine on which we'll be running LEAF. This virtual machine is all set up and configured using Vagrant Automation, which is checked into our open source repository. So you can grab this automation and within just a few minutes be running the same commands and things that I'll be showing you today. It's a very easy, quick way to get your feet wet with LEAF and give it a try. So on this virtual machine, we're running a couple of go-based web servers where we can send test traffic. We're hosting a kernel function repository. This is where LEAF can download the EBPF programs that it's going to manage and execute. And we're also running Prometheus and Grafana to show some of the metrics that LEAF provides for the EBPF programs that it's running. On the left hand side of the screen, we see our host. From a host where we'll be able to access all of these services on the virtual machine via some ports that we've configured. So let's go ahead and go to our terminal and start running some commands. So like I said, there's some ports configured on the host machine. We're showing those here. Those are our go HTTP web servers that we can access to send test traffic to. We have our Prometheus, our Grafana port, Prometheus. And we have a couple of LEAF ports that we're able to run and access APIs on. And then this is showing our LEAF code on the host, which is mounted onto the virtual machine. So we could actually make code updates to LEAFD and quickly test them out on the virtual machine if we wanted to. So to keep things consistent with our slides, the slide that I was just showing, I'm going to keep our host commands running on the left hand side of the machine, the screen. And our virtual machine commands running on the right hand side of the screen. And then you can see which window pane is active because it'll be the black one. Hopefully that'll help you follow along as I move between some of these window panes. So onto a virtual machine, you can see that we've already built a LEAF binary, a LEAFD binary. Let's go ahead and run LEAFD. For that we need to be root because it's going to be doing some privileged things and loading these EBPF programs into the kernel. We need to provide it a configuration file. This configuration file doesn't tell it which EBPF programs to run or how to run them. This is just an initialization configuration telling you which ports to use, how to log, where to store metrics, things like that. Let's go ahead and start LEAFD. Okay, LEAFD is up and running, but it's not yet running any EBPF programs, but it's ready to be configured to do so. So before we actually start some EBPF programs, let's run some traffic against those test web servers or one and see what the results are. So I'm going to run Hay, which is just a nice, easy program for creating HTTP load. Let's run 200 requests with 20 concurrent workers and let's run them to just pick one of these web servers. Okay, you see that we got all of those responses back very quickly. It only took 0.01 seconds. Most of them came in within that time and they were all 200 okay responses. So that looks good. Let's say we're in a scenario where we really want to slow down this traffic. Let's see how we could do that with LEAF. So we have a payload here. This is a payload that we can send to our LEAF configuration API. In this payload, we search by sequence ID. We'll see that there's two EBPF programs in this payload. One is a rate limiting program and one is a connection limiting program. Then we have various arguments and things we can send to them. Some neat things we could change are which ports we want these two, these EBPF programs to monitor and the actual rate limit and max connection value. And these can all be changed on the fly, which is really great. So now that we've seen the payload, let's actually send that into LEAF. You can see we can do this just with a simple HTTP post in our test environment. So let's run that. And in the top right, you should see the LEAFD log load this configuration and start these EBPF programs. So we can see it doing that now and we got a successful response back. To verify that it's actually running the programs we expect, we can look at the LEAF debug API or the NIC that's configured on the virtual machine. So if we run this, we'll see now LEAF is running with our connection limiting EBPF program and our rate limiting EBPF program. So now we should really be slowing down this traffic. We run our hay command, the same hay command we ran before and see if we've made a difference. You can already see that it's taking longer. In fact, it's going to be taking much longer. We're really going to slow this traffic down. So let's take advantage of this time to kind of take a deeper dive look into some of the things that LEAF is doing under the covers for those that are interested. So as a root, we can run a BPF tool, which is a core BPF program. It's not a LEAF program. It's provided by the kernel team, I believe. And we can show which programs are running. So you can see our XDP programs are running here. We can also show some of the maps that are running. And LEAF uses these maps to do all kinds of things. It's where it provides the configured metrics. It's where we provide the configuration values. It has these maps are how LEAF is able to chain together programs but be able to reorder them or add and remove them without having to remove the base BPF program from the network interface. There's lots of things that are going on here under the covers that we don't have time to cover right now, but might be interesting for you to dig around in on your own. So our hay program finished. You can see it took much longer. It took about 60 seconds. So we really slowed that traffic down. And LEAF provides metrics into what those BPF programs were doing. So let's take a look at that using a browser and logging into Grafana. I'm going to skip creating a new password. And then I can look at the pre-configured dashboards that are part of our vagrant automation that are already populated here. So the first program that runs in that chain is our rate limiting program. It was sequence ID one. So let's take a look at it and see what it saw. You can see that it saw a big sharp increase in the number of connections received. And you can see that it actually started dropping some of these connections. So any of the requests that got passed rate limiting would go on to the next program in the chain, which would be our connection limiting program, which limits the max number of connections that can be alive at one time. So let's look at what it saw. Again, it saw a sharp increase in connections. And even it was dropping some of those connections that it saw. So it went through two layers of aggressive limiting. And that's why we saw that time for our requests increased so much. This was an extreme case to show sharp increases and dropping the traffic. But these programs are all configurable to fit your business needs and requirements. So that concludes our demonstration. Let's jump back to the slides. We want to give a thank you. The entire Leaf team would like to say thanks to all those that have been involved in our journey thus far to get Leaf to the point where it is and running in production at Walmart and doing some great things. We're excited that it's now open source and hope to have many, many more collaborators and contributors in the future. We have an exciting roadmap planned ahead for Leaf and lots of things that we'd like to do. So if you're interested in getting involved, we would love to have you. If you're excited to answer any questions you might have about Leaf. We also just want to say thanks to the open source community in general for providing eBPF and answering our questions and helping us get to the point where we are. Thanks very much and have a good day.


Get underway. I'll apologize in advance. I will admit I have not run through and timed this so I might be a little bit under I might be a little bit over anyone who is here at 8 o'clock this morning I spent more time practicing singing than I did speaking This this will be our first time run which yes, I know that's that's a faux pas But we'll hopefully not be overly long and we'll give you some good information that you can use for understanding how Drupal CI works and How you could potentially use it in your own testing within your own organizations or how it will be deployed on Drupal.org So as we get going here and technical difficulties which happen when you Finish your demo environment 30 seconds before you start presenting So a little bit about myself I'm a Canadian Drupal hobbyist from Regina, Saskatchewan in Canada I call myself a hobbyist because my day job has absolutely nothing to do with Drupal I'm a network engineer IT consultant with the company called Sastel, which is the full-service Telephone provider that serves the province of Saskatchewan, Canada We do wire line wireless IPTV The full suite of telecom services But they do absolutely no Drupal despite my best efforts for the last four years So you'll see on my badge. I'm currently employed by the Drupal community when it comes to comes to my my work with Drupal so On top of that I also have a that's my day job. I have a dad job. I'm a half-time single father to five and six year olds who keep me busy every second week So my evenings and weekends hobbyist is actually evenings and weekends every second week hobbyist Which is when I spend my time doing a number of Drupal jobs And you can see there's quite a large list so on top of Drupal CI I was involved with the Drupal.org D7 launch. I worked with I do some work with the infrastructure team I do some work with the technical working group on the governance side also worked on some contrived projects so extended file field is probably the The most well known but not recognized one. It's actually the module that does the files table on Drupal.org So that's so I'm a contrived maintainer there Also core contributor jupacon volunteer and just whatever it's I might be happening to Discratch as far as a friends and family website module that might be needed at any given time. So that's a From a hobbyist perspective, it's a it's a fairly fairly busy fairly busy list and I'm gonna have to add Hollywood doctor to that list after today as well and For today, we're here talking about Drupal CI a little bit of a little bit of a overview of the agenda Typically my speak or my talks in this topic have been about well, here's the background. Here's here's what it's about Here's why we're doing it. Here's the very high-level architecture What I want to do today is do something a little more technical than than I have in the past and That might mean something but most of those talks in the past haven't been recorded. So unfortunately There's a I can't sort of point back and say this will give you overview and this will give you the technical details So I will go a little bit into that into that background just so that everyone's on the same page as we get into the deep dive on the test runner But from an agenda perspective a little bit of background talk about the design principles that went into the Drupal CI project Some of the components of the project and then we'll jump into the meat Which is the deployment models how you can use this and a deep dive into the test runner itself Which is the the brains of the actual testing operation and then with time Do a little bit of a demo and use it step through some steps to show here's how you might use this yourself So the goal of the modernizing test spot initiative, which was the initial initiative name for for Drupal CI It was really an attempt to go and redesign Drupal.org's automated testing infrastructure So the current the current testing infrastructure is built on Piffer and Pift Which were modules that were first built in 2006 and then refreshed in 2009 So we've been running six seven years nine years on on the existing architecture Which in software terms of course that they're they're dinosaurs Also, we really needed to bring our testing capabilities up to speed using modern tools modern Testing methods and to support modern testing frameworks Piffer and Pift were built that long ago as a single purpose simple test stack We tried to extend them in order to do things like code review with the coder project But they were really architected for one purpose and Drupal CI one of our goals is to Build a more flexible generic job distribution framework that we can use that can evolve with the community as our testing needs change over the over the coming years Another goal was to Simplify the infrastructure and maintenance of the platform and we want to do this by reducing the amount of custom code in the stack reducing the barriers to entry as far as the how much code there is to learn in order to just get involved with the Piffer or that it took to get involved with the Piffer and Pift projects and Hopefully facilitate new contributions by lowering that barrier of entry so how we got here For about two years I talked to a number of core devs saying our testing architectures getting old and we need to renew it We need to do something quick when we release Drupal 8 the testing architectures on Drupal 6 It's officially unsupported. So you're testing Drupal 8 with an unsupported platform. This is not somewhere. We want to be And I pushed that for probably about 18 months did not really get anywhere And it wasn't till bad camp in 2013 that I realized that the problem was I was asking a bunch of core developers Who are already really busy building Drupal 8 to come help me build something new and they've got they've got enough on their plate At bad camp I walked into the DevOps summit said hey, I have an idea to build a testing framework I want to make it look like this and I had 21 people at my table like that and I realized I'd spent two years talking to the wrong audience But at bad camp we took here's the architecture here's what we want to do said what do you think and had enough people say If I was building it that's how I build it. You're on the right track So we walked out of bad camp in 2013 and started a new unofficial community initiative to go out and build this project we then Next step was at dev days in Seged in 2014 and we brought together a team to add the sprint to say Well, let's see what we can build according to this picture and within three days at Seged We had a gentleman by the name of Ricardo Amaro built a fully functional prototype of the test runner In three days, which did everything from installing Docker starting the containers and running the test suites and giving you the results From a proof of concept. It was absolutely amazing What we did though what we did want though is to convert it over to PHP right at that time It was a single bash script We wanted to modularize it make a little more flexible so that we could apply it to a few other areas of the community as well And so Austin was an attempt to start a rewrite Amsterdam was a very very integral from the perspective of the architecture of all the surrounding components around that test runner and then in In Bogota we went and we did some final polishing with that About a little over a month ago. We brought the entire team together and that's folks from Canada the US Portugal Switzerland we've got a project manager in India one in Australia really an international team We brought everyone together in Portland Funded by the D8 accelerate program in order to all get together in one room for a week and try and really progress the the initiative forward and What we walked out of there was probably about 80% of the way to what we need to get this actually tied into group.org in the future So that was absolutely fantastic Our my goal and it's a bit of a stretch goal Is to release an alpha version of the test runner by the end of the extended sprints at this conference So We're we're close. We've got one major outstanding piece Which is really just the publishing of the results after the test run runs and if we can get that we'll be releasing an alpha here Within the week, hopefully So I talked about design principles design principles for this project modularity We wanted a number of smaller independent components a number of building blocks that we could then piece together in different ways To meet different use cases Second design principle was flexibility. We needed to be able to handle more than just simple test testing We want to be able to do be hat front-end frameworks a B testing performance testing We want to be able to automate tasks in the issue queue And and as well we want to be able to support the community with their own custom test scripts and testing capabilities that they want That they want to to try So so from an extensibility perspective we want the community to be able to contribute new functionality to the platform So as we'll see a little later. We've got a plug-in based architecture Which really should simplify that process the process of the community coming in and giving us their own testing capabilities Or or building out their own needs on the framework that we produced Other design principles multi-purpose once again We don't want to build a single stack simple test testing framework anymore We need a multi-purpose architecture and the last one here is local first 80% of the time In my first two years as a test spot maintainer was spent answering the question it works on my sheen my machine, but doesn't pass in the test spot why and so number one on my plate is Building a system that when someone comes in asked why doesn't pass in the test spot We can give them the test spot have them test debug and troubleshoot locally and take that off the maintainers plate So that's definitely a major goal with this initiative and I think we've I think we've done a good job on that The approach we took was building the local testing environment first and then extending it to work on on jupel.org and the surrounding architecture So when we talk about components, there's a number of different architecture components that are involved in the full N10 stack We've got jupel.org which initiates test requests Both on commit and on demand So we had to develop some integration with jupel.org that could feed stuff off to our off our environment We've got an API layer that interfaces between jupel.org and our dispatcher and This gives us a lot of flexibility in the future So what this allows us to do for example is with a single dispatcher We can take requests in from jupel.org or we can take them in from say a private security dot jupel.org site or we could Change out the Jenkins dispatcher in the back end when the next new Excellent dispatch product comes along so so we've provided some abstraction layers to make this module enough that we can change Out chunks of the product without having to change the code on jupel.org or how or the code on the test spot itself Then the dispatcher as I mentioned we've got a Jenkins server. That's running our job distribution It's responsible both for job dispatch, but also for the the Load bound or load on demand so with Jenkins we can have Jenkins Detect that hey, I need this many test spots where they're all busy and I can go to AWS and spin up more As needed right now. That's a human action. So when we get ready for a sprint We've got right now Archie sitting down at a computer for a little while building test spots in preparation for the sprint And if we're ever behind We never catch up because that that demand builds so quickly come come. I mean Portland. I arrived at the Portland Sprint I was about 20 minutes late There was already a hundred patches in the queue at the peak There was 300 patches in the queue and by 2 o'clock in the afternoon. We had 50 test spots built So so that on-demand feature is actually going to be really key to moving forward in the maintenance of the of the architecture force Then we got the test runner that generates the test environment that executes the job That's that's the brains of the actual testing operation and I'll do a deep dive into that a little later And then we've got a results API. So that's an extraction layer between the test runner and the result server Now the reason we've done this is to be able to for example have The Drupal.org results server that we're sending results to but you could also have say your own private results server that you also want to send those to or as a Deployment model which I'll get into into a sec in a second. We can do a public results server Separate from the Drupal.org results server that works kind of like a paceman environment and I'll get into that in a second So deployment models of course as I talked to we have local testing first That's you working with your local laptop. That's you with your own AWS instance doing doing testing of whatever you're working on but we can also with the a The test spot and your own results server. We can support a private stack. So this might be within a company You've got multiple devs and you've got your own results server that you want to show Just like Drupal.org would but as a private instance A slightly larger company might put a Jenkins server in the middle of that So we've built this with components in such a way that and made them all public For the community to be able to use this and implement their own stack using these components if desired as well So there's the Drupal.org stack, which of course is when you start talking public stacks That's the one that everyone's going to be familiar with But we did want to be flexible enough to meet multiple use cases. So The this pace been concept that I that I mentioned Imagine that you as a developer have done some code you've wrote a patch It gets you've tested locally and you get a fail and you don't know where that fail is coming from What we could do potentially and what we've been talking about doing with this is Put up a short-term result server where you can then type in a command that will zip up your entire environment Including your database your web directories your test results the and and and The output of the job run zip that all up load it up to a Result server that someone else can now pull it down and they have the test run that you just ran there in front of them For them to troubleshoot We have a lot of fails in Drupal that are that are sort of a random nature We have things that fail and then people can't reproduce the failure And so they upload the same patch to QA dot do a hundred times hoping that one of them fails and they happen to catch it You can debug it well now we can take that random fail when it happens locally and The plan would be to upload it share it debug it together and then Expire it off that result server after three days and you're able to upload it to jupel.org when you've got the trouble Trouble solved so So those are the types of those are the types of things that we want to types of features that we want to try and Enable for the community Something more than just upload a patch to the issue queue and have that patch Return an hour later. Sorry it failed and then having to go try and reproduce that yourself Another deployment model here, which I didn't put up on the slide is is looking in the jupel.org stack Being able to do non testing jobs So the framework that we've built and the way that we built it could support for example a job type that Automatically creates an iterative when you pass it to an issue note We've got a Jenkins server. We've got the environment. We got the code check out There's no reason we can't just go in do the get apply do what we need generate a file and spit it back up to Drupal.org So we're also looking to gen we're looking to enable issue queue maintenance tasks and take some of that manual work Away from the community as well. So I'm going to get into the the actual test runner itself now And this is the project is Drupal CI underscore test spot. It's on Drupal.org And this is the the brains of the testing operation I Wanted to provide some understanding about how it functions how it operates and Hopefully give the community some of the background on what they would need to do in order to take it forward Use it for their own uses and or extend it to provide additional testing functionality. So so everything can be found on Drupal.org The application itself is a symphony console application So for those for anyone who's not familiar with symphony console It's a it's an extension on the symphony framework that is built for Really leverage itself for building command line applications. So this is a command line application written in symphony console And in the in the director in the directory tree there on the on the right source Drupal CI Contains the symphony console code So if anyone's built with symphony console before that aspect of it should be quite familiar On top of that we've built a plugin based architecture. So If you've worked with Drupal 8 a lot of the stuff in there Should also look familiar because we pulled in Drupal's plug-in discovery and plug-in annotation methods from Drupal 8 So what we're doing in order to discover plugins on the testing framework We're using the Drupal component reusing the Drupal components in order to do that. So if you've done D8 development That part should also look familiar to you in Drupal CI Now even more specifically the architecture of that plug-in Of what we do with those plugins if you worked on migrate in Drupal 8 this will look very familiar During our Portland Sprint chicks turned to me and said yeah, this looks exactly like migrate So so the so if you're familiar with migrate this should also look very familiar to you at the barrier to entry it once again Should not be anywhere near as high as it was with PIFT and PIFT it took me about eight months to learn that code Enough to be able to support it properly So I mentioned it's a plug-in based architecture. We have three main supercategories of plugins So we've got job types Job type might be simple test might be PHP unit might be create an inner diff We've got build steps which are little which are the incremental steps that happen during a given job And we have a number of free processing plugins and I'll get into each of those So when we talk about job types The job type is really the large category over what we're doing So if we're doing a be hat test will have a be hat job type if we're doing a selenium test We'll probably have a selenium job type What those types consist of is a job definition template which step which just defines Here's the things that I need to do and a class that defines. Here's the different variables that I need to do that work Here's the default values for those variables and here's any custom not logic that I need. So so if I switch over here My environment so this is this is the job template for our php unit test and Essentially, I mean this should look familiar to folks who've worked with some CI program or CI systems before Yeah, let's bring that up. What's that? What's the zoom keyboard shortcut for php store? That gives me settings. It doesn't give me a name That was the first thing I tried. Oh, that's the setting and it doesn't do it No, I just turned on can change change font size with the mouse wheel. So apologies for the small font But this should look familiar if you've done some some CI work what what it's got. It's it's a YAML file It's key-based. It's got the the build steps and each step then has some sub steps The first the the parent keys here are actually mapped to build step plugins And then the secondary keys are mapped to pre-processed plugins So we've got environment web and then we've got inside percent signs DCI php version So what this is is a variable substitution template basically So if you were to set DCI php version equals 5.4 when you do the Drupal CI run command It'll substitute that in and it'll choose that the appropriate container for your 5 php 5.4 environment it then runs through setup step some checkouts a get check out and the syntax for defining a get check out and Then it runs in execute which is again just placeholders right now But DCI run script the default is php unit Options run options includes things like color and some of the default options that you might want on php unit And then the run target which is the the groups for example that you might want to run All of that because it's variable substitution and we have plugins for it can be customized by setting environment variables on your local machine As as you run it the class itself is is very small. It's got available arguments Just from a documentation perspective be nice to show people what arguments you can use for running on that job and The default default values for those arguments So as we're building a new job type, that's really we need to define those two files So that's the definition template and the job class wonderful state. I started this in So apparently the slide transitions are not going to be my friend today So the next plug-in type that we have are the build steps. So these are These are the build steps which are available as plugins in the system right now So configure Validate set up an environment these four will run on any job type Configure basically pulls together all your environment variables builds out your job definition validate Make sure that you have all your required variables and then set up sets up your directories does your code checkouts Does your copies gets the code base that you want tested set up and Then environment defines what containers you're going to use in order to do that So those four will run on any type after that we've got install execute complete success failure publish. These are just examples and As long as a plug-in exists and you can use Sorry, you can put whatever you want in as a build step and if a plug-in exists for it It will execute the logic in that plug-in. So when we looked at the php unit template We basically had environment and execute. There's no Drupal install step So that was all we needed for that for that particular template so then within each build step there are sub steps that are run and Not all sub steps are tied to a single build step. So inside setup. We have check out We have fetch we have patch. We have different setup steps that you might want to do to build out your environment However, something like command which runs an arbitrary script or an arbitrary command You might want to do that and inside any build step. So there's a generic build step Which contains those types of tasks you might want to do at any point Inside the directory structure These are the directories under the plug-ins directory. So if you look at plug-ins build steps You get these step or these build step directories and then the plug-ins that can operate inside that directory Are located? Sorry inside that build step are located in those individual directories And then I mentioned it's not a hard-coded list of build steps that your jobs need to fit into It's easily extensible with new build steps you can just add the plug-in that has the logic required for that and As long as you as long as you make that available We're more than happy to also contribute it back to the project so that other builds or other people can also use that logic And extend it forward so the third type of plug-in is pre-processed plugins and This is really where a lot of our magic happens There's two types. There's a definition pre-processing and then there's variable pre-processing So when I showed that template it had those build steps You'll notice for that PHP unit. There was no there was no setup build step. So there's no patching There was no fetching patches or applying patches in that by defining a DCI fetch Variable It will go out look for the fetch definition pre-processor Which essentially adds the section to the job definition that you need to do to fetch a patch So once again, you set DCI fetch with proper syntax. It will insert that into the job type for you So so the PHP unit job You may want to run on just the straight code base. You may want to patch it You may want to run it after the patch all that flexibility is handled with the same template through the use of these pre-processing plugins So a definition pre-processing plug-in adds a new section to the job definition And that's why we call it definition and that section then does the variable substitution that we that we had However, we learned very quickly that that wasn't quite enough We had variables that we might want to set that don't directly substitute into the template But actually change or augment the value of another variable So that's where we created this concept of variable pre-processors an example here is The DCI run script variable defines What is the run script that you actually want to kick off this job with so for php unit the run script variable is php unit but You may want to add an alternative argument to that and So when you run the run script, we don't want to run php unit We might want to run php unit dash xyz and so what we needed was a way of adding take an environment variable that would then Change the value of another environment variable, and that's what these variable pre-processing tasks do So as we move forward Terrible coloring there test runner execution. There's a class called run command, which is the symphony console run command It doesn't compile your definition validate your definition set up your directories and then for each build step Loops over the sub steps and executes run Very very basic logic flow from that perspective The steps that that we have the four that I talked about that will really be there on any on any job Include environment which starts your service containers and starts your executable containers Then we've got the setup build step which sets up your working directories Does your copy or your checkout of your code and he fetches and he patches? And any permission changes that you might need so it sets up the code base on your local machine The environment step Starting those executable containers then maps that local code base into the container so that it can be operated on Then the execute build steps runs the test scripts and commands and then what we'll have at the end Which wasn't in my templates because it's the outstanding piece We need to develop is the publish build step which gathers all your artifacts together runs your email notifications or or rest posts notifications or Publishing to the jupy.org results server So that last circle there is the outstanding piece that is preventing us or that is In between us and our alpha release We talked about containers. These are all doctor containers and what we're providing is a number a full container stack for both the service containers all the databases for the environments support as well as the PHP and or web container stack for different versions of PHP We've got a stack. It's it's a layered stack if you're familiar with doctor We try to take advantage of the caching But we since moved it to dr. Hub which doesn't pay attention to the caching and that so so You'll have the containers there can build them locally if you want to pull then it might which is our default mechanism It might take a little while to build it the first time and as we get into demo there I can show you some of that But the container types there's two types There's executable containers and service containers your service containers are the databases Essentially so they run as a service and then when you start your executable container to do the testing It'll connect link across to the database container and use it for for the test job the configuration for these containers is also in the code base and So we've essentially got YAML files which give you the doctor Configuration for the container that you're going to start up So if you're wanting to go and extend this and you need to add another volume mount you need to expose different ports It takes the doctor syntax right here in the in the configuration file if we look at the Database configuration files you can see the difference there is we actually exposed the database service ports for example So this is all straight Docker syntax if you're familiar with Docker those that syntax can go right into this file And it's it's then added to that container and I'm really going to have to stop context switching here so that really brings us to what I was going to do as far as some usage and demo stuff and So I'll just bring that up increase the font before I do that So of course the first thing you're going to do is pull down the test web project We package it with a vagrant box So if you're on Windows or Mac, then you can do do the vagrant up it'll run through the entire provisioning script It'll also do some initial setup for you just to make the environment a little easier to easier to use the Drupal the actual script itself is the Drupal CI script And that's that's the name of the this we're going to call it the first thing you're going to want to do after your download and install this is A Drupal CI init And what this will go through it'll go ensure you've got Docker make sure your versions are are are accurate Eventually we'll do dependency calculations make sure you've got all your dependencies are okay there and then it goes through and it Will download from Docker hub the containers that you'll need for the environment this went rather quick Given that I already have the most current containers locally that Process could take you 10 20 30 40 minutes depending on your net connection and things like that. It's a first-time build issue once they're there then it You can use them ongoing and you should not have to update them as often once we've got to an alpha state But it then prompts you What database environments do you want to support and what this is going to go is grab the database containers for the for the environments you want Just because I already have it I'm going to go with my sequel here right now to ensure we've got a quick and snappy poll and Then it asks what PHP versions do you want to support? Here we're going to go with PHP 5 4 for the moment. You can ignore that warning That's because I already have an environment and forgot to clean it up before I did the demo But that's the power of live demos right it actually leads us into the next thing that you're going to want to do After download install which is Select your configuration set what configuration do you want to do or set up the configuration for your local testing environment? so for that we set basically Environment variables or we have a configuration managed little dummy configuration management system built in so for the Drupal CI config command has a little bit of a Little bit of a settings management built into it so Just a I'll start by blanking this out and step you through the actual steps So I'm doing I know you can't see it on screen, but doing Drupal CI config load blank Okay, so I didn't quite finish setting up my environment before we started here We did create a argument to help us with these items We can force it over So I basically loaded a blank configuration set. It's got no settings Yeah, so blank is the name of the configuration set that that I was trying to load just to blank out All of my current Drupal CI settings Couple of things I'm going to do here and these this isn't documented should be eventually I'm instead of a get check out. I want to use my local code bases. So there's a there's a Variable there I can set So all the variables are namespace DCI underscore and this particular one is use local code base and Where is the local code base that I want to use? The other thing I'm going to want to do is I want to use simple test jobs by default Rather than typing Drupal CI run simple test. I want it to default to simple tests So I can set a variable for that. That's our job type variable I don't want to run the full test suite because we'd be here for a awfully long time so we want to set the test groups that we're going to run and We'll stick it to the action test group for now the last thing which I which I do want to do for here this is explicitly specify which database version we're going to test against so triple CI config set DCI DV version is My sequel 5.5 Now with those items set I should be at this point now Where I can simply Drupal CI run because it's getting awfully close to six o'clock. I did Okay, and therefore we're going to wait through a clone It is a shallow clone. So we're 30% and so essentially that's How you would go forward Matt using the environment now what I can do I set those those four parameters and working on Drupal 8 core. I can then do a config save my d8 core Let's load switch over to another project set whatever I need to retest that project and at any time in the future Drupal CI config load my d8 core set up and I can swap back and forth from different projects like that So it went through it's running it created the container instances and now it's executing run tests on that container If we scroll up here a little bit It tells us it first loaded the platform default arguments Then some job type default arguments. So these are all specified in the simple test job class And then the local overrides that I set so it's telling you here's the hierarchy of variables that we're using And you'll have variables that show up in multiple of those the further down the further down the list They are they trump everything above them Next steps it generated our code base data volume created a checkout directory Determined what containers it needed made sure that those images existed on the local machine, which was part of that in its step And then did the checkout Then created our results directory in purple these are all executed on the container itself So everything above that was done on my local machine now in purple. We're on the container Created results directory created results XML directory for us Did some permission changes that we need in order to be able to write those results on in the container? Created created our database And then this is the Drupal script run test sh line that it then executed on that container And then as we go down here Took about one minute to run through the Default actions suite fix that little typo Set our db version Postgres 9.1 miss typoed my fix of the typo, and we're now executing Against the Postgres database, so that was one of the major goals of this is to expand the number of environments that we're able to test against with Drupal.org As of this week we can test PHP 5 4 5 5 5 6 and 7 we have containers for all four suites against SQLite, Postgres, ReaDB, and MySQL So we're expanding to meet the minimum coverage requirements for the core developers for what they want to test for against as they come towards a Drupal 8 release So with that that is more or less what I had for you today I want to talk a little bit about extending Drupal CI and the tasks for that I'll leave a slide in here for how you would go about doing that. What are the steps really? It's create a new directory create a PHP class create a default template and create any custom plugins You need for that logic and that that that's the steps the four steps that you need in order to extend this for free-owned custom testing needs But with that I'll Certainly ask provide any feedback that you can on the session each of the sessions has a page on the on the conference website Where you can provide feedback on the session definitely helps speakers like myself to prepare for future sessions and and that feedback is Definitely encouraged and definitely desired So please please do that and I'll open it up for our last 10 minutes here any questions or discussions that people might have So I have a question So what is the difference between when we run PHP run test run dot sh file from Drupal folder in Our local environment whereas we don't use Drupal CI and you run it locally you're running it against your local environment You may have Modules that were installed manually on that site or you may have a PHP version that doesn't match with the test spots run You may have different Apache configurations and that's where we run into the issues where people run a test locally it passes Okay, it's good. I upload it to the issue queue and the test spot rejects it So we really want to ensure that we're providing the same testing environment on the test spot as we are On the local environment so that we don't run into those conflicts anymore. That's that's the primary reason behind it Definitely by all means as you're developing run it locally with PHP run tests, but before you send it up to the test spot Let's make sure or this is another step. We can do to validate that it will pass once we send it to the test spot Yeah, thank you So as we saw in the demo, you know just doing a couple typos or not understanding all these environmental variables can lead to some frustration What what do you think is the next step for improving the The brittleness of some of the environmental variables and in the pre-processed. I guess they're variable pre-processing Yeah, the number one key is going to be initially documentation We certainly recognize that We've been charging full speed ahead because we're listed as a critical blocking date Trying to get some function into Drupal.org that so that we can do the testing that That unblocks that critical, but as we get to an alpha here Definitely documentation is going to be a number one key thing We won't have the stability will still have that realness at an alpha release But if we can at least show people how to get around it Once that alpha is released and we've unblocked the D8 D a critical that's tagged against us Then we'll be able to move forward with the with the polish and the refinement of those variables those plug-ins adding things like Debug verbosity into the into the client adding support for items like like monologue so that we can then Actually provide Detailed debugging information that will tell somebody that yeah This didn't work because of X So that that'll be a big push definitely before a beta But probably a post-alpha push for us Hey, I noticed there is a project for Drupal CI for for a public module to deploy it And but the project is empty So I was just wondering what the status with that is or if there's a plan or just don't be touched it yet But that's somewhere you really want to go. Yeah, so that was part of the sorry That was that was part of the initial plan is that we would have because it's what we did with Piffer and Pift We have we have a puppet master for that and that's how we structured the project initially We move forward and I'll maybe if I miss anything I'll get the guys in the back there to probably augment my answer here But we did move forward. We've got some Packer build scripts that are actually building AMIs on AWS So so rather than giving you a puppet script We are creating the AMIs for folks that they can then consume and and those will be I believe Those will be public AMIs for anyone to be able to spin up an instance of the test runner that if they want Right now that AMI is the full Jenkins slave, but it contains the test runner. So so we're looking to to do that We've also set up the some some automated Building on docker hub from from our repositories So some of the chaining that we need so that when we commit a new change to production We'll automatically get that change built into the containers So once so if you're on your local environment A pole will refresh the container and and you'll have have the latest builds there. Did you want to augment that base? ready so so so for the for the recording there the The puppet mod there the puppet was initially intended as that separate module The puppet steps are now built into the individual projects themselves So packer for example has does has the puppet deploy stuff and the manifest and stuff within that particular project um Just listen to your talk. It seems that this project is not