We are waiting to be able to make our first pass and nothing's really changed except for we have the two-step installed that we purchased from HHP. That's the Stage 6 Motorsports two-step. Currently we have it set for 3,000 RPM. So our goal today is to launch at 3,000 RPM on the two-step off of the trans-brake. We're still running the factory suspension right now. So we haven't changed anything back here yet. Like I've said, I have a bunch of stuff at home to put on the back here. So stay tuned. We're going to be working on the truck tomorrow on the rear suspension to get this thing working a lot better. But today we're just basically coming out to play and test the two-step out a little bit, get a couple more logs. So like I said, we got a 3,200 DA and we haven't changed anything in the tune. All we've done is we added the two-step and we're here to check it out and see what happens. I'm thinking if we can get good traction, we should be able to be a little bit quicker even with it being hot. We should be able to launch a little quicker. So I'm hoping we can dead hook at at least 3,000 RPM and see how she runs. Like I said, still got the same tune in it from the last time that we ran 11.9 in about the same DA. It was like a 32 or 3,300 DA and we was able to run 11.9 still. We're still running, right now we're running pump 94 gas with 100% meth injection for my snow water meth kit. So we got 100% meth 94 pump gas and we're pushing about 12 degrees of timing roughly in the tune. And that's what we've been running for a while. We're shifting at 6,500 RPM. So like I said, we're going to try out the two-step today, see what goes down. Everything on the truck is still the same. We have mirrors, we have full interior, we still got factory seats, everything. We even have all the paneling in the back. Everything is still in the truck.  We are still where we've always been for a while. One thing I will mention and I don't think it's going to be a gain at all besides we've lost maybe 12 pounds and I'm going to go over why we did that. We did delete the AC. The AC is gone and that's because it quit working and I don't drive the truck that much anyways so I don't need AC but the AC quit the last time I took the truck out to a car show and instead of fixing it, I said screw it and we deleted it. So what we did is I got with Matt over at Moes and he kind of explained to me what you do because there's no AC delete kit for these things like back in the day you used to be able to get an AC delete belt. Well, that don't work on the Rams. You have to have a pulley down there to be able to run it. So what we did is we took the old AC compressor off and you know the AC compressor is basically multiple layers and what we did is we unbolted the AC compressor and we removed all of the guts on the inside of the AC compressor and all of the back plates. So we're only running the very front bracket of the AC compressor down there if you can see it and as you can tell I kind of polished it up on the grinder a little bit. So we're running the factory AC pulley with the front bracket off the AC compressor and that is it and that's held on by two bolts and the factory belt that's it. So like I said that's not going to be a gain I believe at all very very minimal if any. We took out like I said getting rid of all the compressor parts I think we took out like 12 pounds from gutting the compressor and then plus the factory lines that run over the top of the motor here and then run over to the firewall those are gone granted those don't weigh much I didn't weigh them you know that's probably another couple pounds right there in those lines. Now it did free up some space not having to work around those lines anymore but that's the main reason I got rid of it was because it took a shit I don't need it and I will enjoy working on my truck a little better without those AC lines running across the front of it so that'll make my life a little easier working on it but that is it.


All right, welcome to our talk. We are presenting today about how to speed up your Drupal code base using a code profiling tool called Blackfired.io. My name is Alex Durgachev, and this is my colleague Jigar Nathap. He and I are working at a BalvingBlob out in Montreal. It's there. It's there. It's just a little chilly. So yeah, we're happy you're here, and we hope you learned something. So just a little bit about the two speakers here. I'm the co-founder of a Balving Web. I'm less and less technical these days. 10 years ago, I was the back end developer, and now my wife, Suzanne, was the front end developer. But now we do a lot more.  And I guess I do a little bit of everything these days. And we've been doing Drupal since basically the start. Jigar, people also call me Jerry. I'm from India, and I've been working on PHP since 2008. And on Drupal, for about three years, 2012 or 13, I started. And it's been a nice ride working with Drupal. And this Blackfire thing is really awesome. I hope you like it. Great, yeah. And Jigar joined the Balving Web about a year ago. So our company has been doing consulting in web development since 2007. We're based out of Montreal. We got some Montreal bagels at our booth on the other side of that wall, come by and chat with us. And we've been doing Drupal pretty much for nine of those 10 years. And we do a lot of content migration, design, theming, all the stuff. A lot of enterprise-y kind of projects with integrations with legacy systems and data synchronization, scalability concerns and infrastructure as well, a lot of multilingual. So a lot of stuff. Here's some clients we've worked with over the years. I think these are out of date screenshots anyway. And here's what we're going to talk about today. So you guys here, because you have a Drupal website. Who here is involved with Drupal websites in some way? OK, good. You're in the right place. Whose website could be faster? Do you have any website that you'd like to be faster? OK, great. So we're going to talk about what that means, what faster specifically means. We're going to talk about what a profiler is with my Blackfire as a good one. We'll segue into doing a demo of Blackfire and what it can do. And then we'll discuss a couple of case studies, because it turns out using a profiler and doing these optimizations is kind of like an art. And so examples are the way to go. So for the motivation section, I think we all know that user experience is really tied to how fast a web page loads. We also know that how fast a web page loads affects your level of concurrency you can do, so how many users can actually visit it at the same time, given the same hardware. And in web development and infrastructure, everyone always says developers are expensive, hardware is cheap, just throw more hardware at it. But it turns out for most of us here, we're probably living most of the time on a single server environment. And if we are, it's a lot cheaper to stay in a single server environment in terms of developer time. So obviously, get a really beefy server. That's what we do with 128 gigs of RAM and SSD drives. But at some point, before you go to crazy multi-server environment, see if you can have really good caching and really good performance before you have to go to a much more complex setup. So that saves you a lot of money. And yeah, so everyone knows very well that page loading time is correlated with SEO and user behavior and how much conversions you have in your e-commerce site and so on. And then I have an anecdote that I'd like to tell from Google's history. You guys know that Google became famous because they came up with this page rank algorithm, right? That allowed every web page to rank each other by links. Like it's a vote for the quality of another web page. So they just asked the web, show me who you think your important other pages are, and I'll rank it out. Well, turns out that idea was not patented by Google. It wasn't new. What Larry and Sergey came up with was an algorithm that was probabilistic to crawl this giant graph that you couldn't really crawl. And instead of doing an exact calculation of what web pages or how they rank each other, they did an approximate calculation. So metaphorically speaking, they did a performance optimization. And they were the only guys who could do it fast and almost in real time. So that's why they became huge. So turns out performance is pretty important. And premature optimization is not necessarily the root of all evil to paraphrase a famous computer scientist. So we're discussing code profiling, which is kind of back-end profiling. It's kind of like your Drupal site needs to generate some HTML. So what this doesn't talk about and what this session isn't about is browser rendering time. So the time to actually download the HTML and download the CSS, download the JavaScript, and then render all that visually and do all this stuff. And any bandwidth issues you might have, JavaScript can be quite slow because it's its own interpreter that's running inside of your browser. So if you have too many JavaScript libraries or they're doing complex things, that can really slow things down. And of course, if you have heavy assets like images and fonts as well. In our experience, the majority of website performance problems that are user-visible are actually on the front end. So maybe start with the front end and see if there's any low-hanging fruits, like 5 megabyte JPEG files on your home page. Who's done that? Yeah, I've done that. So but this session is about code profiling. It's about why is my Drupal site sluggish? So it's kind of like taking this code base that you downloaded from Drupal.org with a bunch of core and contrib modules, maybe some of your own, and saying which part of this that's running is slow when generating a specific request. So it's kind of taking an x-ray of your little Drupal code base. So what it actually does measure for a typical request is how much time was spent for that request per function. We'll see that in a second. And how much resource it was using in terms of CPU, memory, database queries, network, web service calls, and NIO, which typically means network or hard drive access. And what these tools do, these profiling tools do, is they hook into the PHP interpreter, the PHP engine. And they do something called instrumentation, which basically at the beginning of every function call, it makes a little timer. This function makes a log. This function got called with this many milliseconds after 1970, and then that many at the end of it. And then it gives you a report. So that's what we'll be looking at today. So we're still in the motivation section. So why is it specifically important for Drupal? Well, since you're mostly Drupal developers, I'm not going to hide the truth from you guys. Drupal, as a web framework, is not the most lightweight thing out there. I mean, you can write something custom in PHP that runs three database queries and renders a web page. Or you can hard code everything using static-side generator. But a Drupal as a framework is very flexible, and it's very powerful. But that all comes at the expense of being potentially heavy, depending on what you do with it. We find that Drupal Core itself is quite optimized. I mean, it's certainly not super lightweight lightweight, but it's reasonable. And it's had a lot of eyeballs to make sure that there's no major problems. But contrib modules, depending if you only have the most commonly used ones, chances are somebody's already made sure that they're optimal. But if a site builder downloads and enables and configures 35 contrib modules, chances are five of them are going to be a little bit more on the experimental side. And they will be really affecting your performance. So, and the same thing with legacy code. I mean, if you inherit code from another developer, which happens in real life all the time, who's to say he didn't just take a shortcut last time he was developing something and move on? The other thing to consider is page caching. So Varnish is a well-known page cache tool that a lot of people are using. And what it does is it takes a snapshot of a given URL. So it's between the browser and Drupal. And so if a user comes back and asks for the same page, a different user, they'll just serve it from the cache. So page cache is great, but it's not enough. It's not enough because if you have logged in users, that's no good. Like, typically the standard setup for Varnish is that it just skips all, skips caching any authenticated requests. And also, if you have anything like e-commerce or anything with cookies, the moment you add something to a cart, that sets a cookie. So it's effectively like being a logged in user. So there's lots of scenarios where Varnish will not save your run. If you have an inefficient site, maybe not every request, but maybe just 5% of them will be really slow. And on really high levels of traffic, that will bring down your server. Or when you clear the caches, or when the caches expire once in a while from a time out. So that's one factor. Another factor is as developers, there's a selfish motivation there. Whenever we work on a site, on a Drupal site, it's if you don't have page cache turned on because we're changing the configuration and the code all the time. And working on a slow Drupal site is really painful. Like, I would quit my job if I had to constantly work on some giant slow website. So that's why this tool is great. From a business perspective or from a motivation perspective, here's five random projects we ran this Blockfire tool on and what we saw. So from McGill University website, just looking at a profile, pretty much right away we're able to spot a 10% improvement. So it took less than an hour to spot the improvement and it was a two line code in fixing it. We'll see you later today. For a client who I cannot mention for a non-disclosure reason, for a specific client, their homepage took six hours, not just getting there. Their homepage took like a second and a half to load in production at high volume traffic. And with, it was quite complex, but with a couple hours of work, a day of work, we're able to cut that down half of it and with another day of work, another by half again. For our own evolving website, when we moved to Drupal 8 for some reason it became a lot slower in D8 than D7. We didn't know why, but using Blockfire we were able to do some investigation. We'll talk about that in a second and cut off 20%. For another project for Linux Foundation, we had a view that pretty quickly we realized this view is configured with revisions and revisions aren't cast and using this tool we were able to spot that and so on. Oh, the funniest one is at the end when we have a 92% improvement in page loading time I think it had to do with the sitemap module being misconfigured or something. And the FAQ module, yeah, I forgot. And so it's just basically adding a filter for content type or something. It was just a single configuration setting. Fixed it, so, yeah. Please, please, and this was in production for a year before we came along. That was the project we took over. So please optimize, please check if you have a slow page. Sometimes it's really easy. So I came up with a methodology for profiling and of course every methodology needs a catchy acronym. So we got MAFIA, all right, M. So M is for measure, so you gotta figure out what exactly is slow and what exactly is fast, what exactly is the page that you're trying to profile. When a client calls us and says, my website's slow, believe it or not, that's not very helpful if there's any clients in the room. You wanna know, what do you mean? Which page is slow? Are you logged in or are you not? Is the site slow in development with only one user, you? Is the site slow in production when you have 10,000 people hitting it in the same day? So that's first. And then second of all, you know, you have to figure out what the URL is, if there's any cookies involved. Then you have to, when you're doing this measurement in your head, you have to keep in mind what's the URL, like what specific pages are slow and which ones are faster. You have to compare development to production because maybe you have different hardware and different server contention. And you also have to keep in mind with Drupal, one of the things we like to do is we like to disable some modules that right away just like, are these modules the culprits? So we always vary things up when we do this analysis. And of course, part of measuring is using the profile that are like blackfire as we'll do in a sec. Once you're looking at a profile, you're gonna look for low hanging fruit, things like that you consider almost bottlenecks. Fix this little thing and all of a sudden your webpage loads three times as fast. In real life experience, we find that that happens quite a bit. So instead of trying to make micro optimizations everywhere just on your first pass, look for the slowest page on your site and then look for the thing that's making that slow. Maybe you will find it fast. Typically what we look for are things that are like an expensive calculation that might be able to be cached, some too many SQL queries or an SQL query that's too slow, a web request like a web service that's walking and as a result it's not loading rate or unnecessary node loads or entity loads. So this is the typical Drupal bottlenecks that we look for. Another thing to keep in mind is sometimes you look at this report that we'll see in a second and you see overall sluggishness. Everything's slower than it should be but no specific thing stands out. There is no bottleneck, my site's just slow. Sometimes you have to like put your thinking hat on and say what might be going on? Is this a heavy site? Is this a slow server? Or maybe there is something going on like swapping, like maybe I'm running out of RAM and as a result it just everything feels slow or maybe I'm in development and I'm running Vigorant with VirtualBox and I'm using VirtualBox shared folders which is a way to have your host and your guest. If you share the same folder and synchronize that really slows Drupal down. You might actually see this once you start development and that could be the explanation. So it's not your code. Another thing could be server contention if there's like a shared hosting and there's other people using the same servers or if you don't install OpCache. Okay, so next thing in Mafia is fix. So you're gonna make an actual change. That means you're actually gonna change some code or some infrastructure. Like you're literally gonna make a tweak. And then it's very important to compare to the baseline. Keep in mind that sometimes if you in Drupal, if you remove something that's not necessary at the start of the request and you're happy because it saves time at the start of the request, maybe later on in the middle of the request when you're rendering, I don't know, the main body as opposed to the header, maybe in the middle of the request we needed the same node being loaded. So just because you're not loading that node anymore now, loading that node anymore now, you'll still need it later. So you haven't saved globally. So you have to compare. You have to do a profile before and after and you have to do a check for overall page speed before and after. Then is iterate. So like I mentioned, first, you look for low hanging fruits. You measure, you fix, and then you have to do it again. So we're actually gonna keep going and again. And at some point you have to know when is fast enough for my hardware, for my site, for my client. I don't know. It's a personal question. You have to almost keep the Chrome network console open as you browse the web and notice what pages you think are slow and what pages you think are fast and keep in mind. The number that I have in my head is anywhere around 500 milliseconds for loading or less is reasonably good. Anywhere between 500 milliseconds and a second is not ideal but tolerable and anything for back end loading time. Beyond a second, I always think, man, this site is slow. Why don't they have caching properly configured? Or what else is going on? Yeah, and keep in mind also, once you do this iteration, you wanna have a notepad or some Excel sheet with a file so you can keep track of what you've changed. Maybe use good branches too. I couldn't come up with an A for Mafia but MAFI just wasn't catchy enough, so. All right. So for the tools. So for front end performance, what we use is at the start, we can start with Chrome Developer Tools and there's a lot more but I'm not covering that. But the most basic one is this network tab. You see what's going on. You see all the little resources that my presentation has. They take almost no time because it's on local, it's three milliseconds because it's all static HTML. And there's other stuff inside of Chrome Developer Tools that I won't show but it's really great. Get to know it well. We use it every day. For more back end stuff, there's a tool called Apache Bench that lets you hit a webpage 10 times or with a concurrency level of what you set and it'll give you like a standard deviation. So it shows you not just one page, how long it took to render it, but if you hit it 10 times. Another one that's for more complex scenarios is Jmeter. It's also a command line tool. And I don't know, this is ancient. I haven't used it in years. Anyone still uses Jmeter? Is there a more modern version of it? I don't know. So but it's a Java tool where you can actually have a whole scenario. You have cookies and login and go to this page and go to that page, so on.  And then most like last couple of years everyone's been using New Relic which is a performance monitoring like a combination server and application monitoring and performance analysis tool. That's really great.  But it's not nearly as detailed to go into your actual code base and give you a call graph to show which of your functions and which of your contributing custom modules are the culprit as a code profiler. So that's XHPROF or Blackfire. XHPROF was a tool built by Facebook. It was open source and Blackfire started off as a fork of that. But eventually the creator of it decided that he changed so much. It wasn't a fork, it was a new tool. So the creator is Fabien Portentier. He's also the creator of Symphony and Twig. So and if I'm not mistaken he came up with Blackfire when he was trying to speed up Symphony applications. And it's not an open source tool but it's got a free tier that's comparable or much better than XHPROF as a profiler. So whatever you were getting for free before you're still getting for free. So it's kind of like the GitHub model where they give you a lot of stuff absolutely free but once you start using it in your organization they have little restrictions that will make you want to pay. So yeah. And the way I discovered it personally is that every year or so I would Google whenever I need to like have a slow site I would Google XHPROF, better user interface. And I wouldn't have any hits, wouldn't have any hits. And about two and a half years ago I was like what's this? And here we are. It has really great docs. It's really simple to install, much, much clearer. Like they have really, because it's a commercial product they went out of their way to give you copy and paste instructions for Mac, for Windows, for Ubuntu, for Red Hat and so on. It has a lot of visualization and JavaScript built into the tool as opposed to XHPROF, which was like ugly tables. So it's a much better UI and also just a much better sort of developer experience. It's somebody who sat there and iterated on a tool. So for example, they have native built in comparison profile.  And it has other things like that. It's actually has support for PHP 7, which as of a year ago, XHPROF didn't. And he points out that you can leave it on all the time in disabled mode, but you can have it installed in your server without slowing things down. So that's the tool. And I'll do the demo as part of our first case study. So the first case study is this McGill University listing of courses and programs. So let's see. Here's a page on my laptop for McGill site. This lists all their courses, all their programs and regulations for undergrads. It's got like, I don't know, 100,000 nodes. But here we're looking at a random node with 10 fields. It shouldn't be so complex. And this is in fact a very light page. If I'm not mistaken, it took about 200 milliseconds to load with no page cache. So super light. Remember I said 500 and less is good. But we also know that all the students are gonna hit this page the moment registration opens on the same day. And so we're also worried that maybe caching isn't enough. So we wanted to make sure it's really tight. And just in general, we wanted to see if Blackfyre can spot something. So we ran a profile. So I think I'll show you how it works. One of the things you have to install is a command line thing. You'll install the Blackfyre PHP probe and you'll have to restart Apache or PHP PM. Another thing you'll have to install is the Blackfyre agent, which is a little service that collects data from the probe and then sends it back to the Blackfyre server, like the payment or free service. And then the third thing is the Blackfyre command client. And it's actually three or four command line things you just copy and paste, like I said. And the last part of the stack is the little Blackfyre Chrome extension. You guys see it here?  So yeah, you click. And it lets you compare it to previous references you made in the same environment. For now, I'll just click profile. You see what's going on. That's profiling 0, 10, 20, 40, 50, 60, 90%. What that just did is it loaded this page with the profiler flags on to PHP and that activates the Blackfyre probe. It starts doing its instrumentation or profiling, collecting all the data and sends that over to the Blackfyre server. It did it 10 times. Any guesses why it did it 10 times? Why it loaded the page 10 times? Okay, it's cut. So sometimes we have cache turned off or maybe the site hasn't been hit for a while so it doesn't have the things loaded in its cache. So the first request tends to be a bit slower but then the subsequent request now that we have the cache built. So what happens is the page gets loaded from the cache and things are a bit faster. So only one time measure is always not sufficient to say whether a site is slow or maybe sometimes you open some site like LinkedIn. The first time it opens a bit slow but that doesn't mean the site is bad or has performance issues. Maybe something went wrong and then you refresh and it loads like super fast. So it's something like that. So doing it 10 times, it can get figures for 10 different runs and then it can do an average and say like this is the average time. So we get a better like figure to work with. Yeah, and personally that was my major pain point with XH Prof. I would get a report. I would look at it. I would optimize it. I would surely make an improvement. I would get a new XH Prof report and it would be slower.  Well, because that variability that always happens happened to be greater than my improvement and I can't even tell what's what. So when you're dealing with averages, it's just as a developer, it's a lot more sane. So it makes a huge, huge difference. Another thing that you should notice is it lets you name your thing such as Drupal North 2017. So here, when I made a profile of this page in the listing of profiles that I can access later, now it's named. XH Prof didn't really do that.  All right, so what do we see in this call graph? I know the resolution is really careful. Up. Sure? Yeah. All right. What we get here is like a list of function calls. Like if we, how do you zoom out on a Mac here? Okay. So how do I go now? So as we can see here, like this is the starting point. Like this is where we start serving the request. So it's main as we can see.  You must have seen that name if you ever edited the index.php file of Drupal. So, and then the menu execute active handler calls a bunch of other functions. And that's how like we get a functional call tree for the entire request. Like who calls whom and how things work. Like for example, here we get other figures. Like Drupal alter, the alter method function has been called around 600 times. We're serving this particular request. And we can also see Drupal load has been called 110 times. And for example, we click on one of them. We can see that the Drupal alter function in itself took around 16.4 milliseconds to do his thing. And it like used so much of memory. And we can also see other information, like 22 different functions called Drupal alter. So we can like, if you click on this big bar, it means the caller which called it the most number of times, we can see its name. Like menu link translate is the one that called Drupal alter around 22 times. And similarly, we can also see the call is, like what are the functions that Drupal alter calls? So sometimes like a function is not slow, but a function is calling a function which is slow or taking too much of time. So, or maybe it's being calls too many times. So we can click here and see that this particular function has been called 335 times from Drupal alter. We can see like the, like this one has been called twice and so on, like the smaller ones are also there. And so this gives us like, we can see who's calling whom and we can figure out which function is the one which is taking the most amount of time. And similarly, we also have, this is the time view. Like we are looking at how much time which particular function is taking and everything is like the time factor is highlighted here in this view. But we also have other views. For example, we have a memory consumption of 24 megabytes. So if you want to know which particular function is taking the most, like using the most memory or like sometimes instead of reading a file using fopen or something, somebody has done a like file get contents or something. So all of a sudden we have an entire use file in the memory. So we can find it here. Like this particular function gets called and all of a sudden we have a memory usage. So we can detect problems like that as well. And what? I can drop it here. Great. Yeah, so that's a good start. I'd also add that at the top we have the global statistics here. So here it's showing, sorry guys. Here it's showing that the whole request took 215 milliseconds. Out of that, the hard drive basically waiting was 30 milliseconds, 184 was mostly CPU running your code. We already talked about 24 megabytes of memory usage for Drupal that's quite low. I don't know if you've seen your Drupal profiles but it's probably gonna be higher. And there's no network services.  So if there's anything funky, you can see. They're also anonymized. So it captures the query but it doesn't capture the actual arguments to that query like username and so on. So the next thing that's very exciting about this call graph, right? So like Jigar mentioned, this is the call graph for your entire Drupal code base. Every function that Drupal has in it is here and when it calls another function, there's an arrow there. You see some parts are pink. What that is is BlackFat calls it the hot path or basically where the majority of your time is spec. So you can visually, when you're exploring this graph and clicking around, you can focus on the hot pink before the light pink, before just the gray. So that again saves you a huge amount of time and you can inspect the code visually. In this particular case, I also wanna make a distinction between inclusive and exclusive time. So I don't know, let's find a random function like Drupal Render. So Drupal Render here spent any time. Let's find something more pink guys. I hope you're cool with that. Okay, node page view. Spent in total 79 or 80 milliseconds. That's the inclusive time. That means inclusive of this function from when it was started to be called to when it returned. Whereas the exclusive time is, well, how about the child functions, the functions that it called. Let's measure those, add them all up and deduct that. So the exclusive time is always gonna be less than the inclusive time. It's the time spent just in that function. If there's a for loop in there or if there's a SQL query in there rather than some child function. So this is what you have to analyze.  But if you wanna see just where is time being spent, it's the inclusive time. So you look at both and black bar shows you both. Same thing with other things. So here, it's actually quite hard to see in this resolution. But if you wanted to say what's slow here to click around, I can try to do it. But I might have to cheat because I know. It turns out that I noticed this is after half hour of looking at it and trying to figure out a surprising amount of time was being spent in Moriarty process page. That's Moriarty is the name of the theme that's running here. That's the Sherlock Holmes villain. In fact, the Sherlock Holmes villain turns out one of the villains in our Drupal site performance. So 13% of the time is being spent in Moriarty process page. And most of it is in Drupal get form, Drupal build form again, 9%, 9% study search box form. So I was thinking, what the heck is this study search box form? And why is it taking 10% of my request? And I go back and the site is called study. And I think, what is this search box form? You can probably guess it's up here. And so it doesn't usually take 10% of your request to render a static form. What's not static? Well, apparently this is a list of faculties. This is actually dynamically being pulled from a database of nodes. So maybe that's slow? I don't know. Let's look at what Blackfire says. So it says it's spent pretty much all of that 9% in load academic faculty nodes, which cause node load 36 times. So this is the Drupal 7 site, in case you guys are wondering. So this is with just about half hour of exploring the profile of Blackfire. I noticed that. And we actually went and opened in our editor, load academic faculty nodes to see what the code does. Can you guys read this? There's a four each loop. That just basically takes all the faculty lists from somewhere and cause node load on each faculty and then returns these loaded nodes in an array so that the form builder can render that little drop down. First of all, we don't even need to load the whole nodes just to get the 10 tiles, right? But second of all, is there any other even more direct optimization that could be happened? Any Drupal 7 developers or Drupal 8? Yeah, no load multiple. So here, all we had to do was convert it from, instead of loading the node in the array, not loading the load, but loading the node. In the array, we do it outside of the array. We just accumulate the node IDs and call it node load multiple. And instead of running all of those queries 36 times, it runs all of those queries once. And right away, it saves us that 10%, or most of that 10%. And in case you're wondering why, this is me, I was doing this. I first wrote this in Drupal 6, like, eight years ago, where we didn't have node load multiple. And then when I upgraded to Drupal 7, my algorithm was upgraded to Drupal 7, refreshed the site and fixed errors until errors stop appearing. They didn't call the client and say, you're done. So that's how the site was in production with that my node load multiple. But if I had spent just that extra hour with Blackfire, you know, the five years ago when I did the upgrade, it would have caught it quite quickly. And it catches lots of stuff like this all the time. So this saves 25 milliseconds. It's not too much, but it was very, very small to begin with, 230 milliseconds. And yeah. So for installation, like I said, okay, that was the case study, that was it. For installing it, you have the little doc pages that they even tell you what to copy and paste. They even give you your tokens that are generated inside the doc pages, so it's seamless. Yeah, so what else does Blackfire do? I mentioned comparisons with baseline profile, I won't show it to you today. There's a scenario where you need to be able to trigger the same profile, not from the Chrome extension, but from the command line, because it's more complex than that. So for example, if you have AJAX requests, you know, you could say start recording AJAX request and then rerun it with profile. But there's an alternative method for the command line or anything that has to do with where you do post requests, for example. So that little button start profiling won't work. So yeah, so the way you deal with that is there's a command line tool called Blackfire. It has a sub-contract called run, and then you can run various PHP code in it, including this drush command. In case you guys are wondering why you didn't say Blackfire run and then drush ccl, but drush launcher ccl, because drush is actually a bash script that, or even if it's a PHP script, it forks and gets another PHP instance and that will confuse the profiler. So you've got to open the thing that it forks to. Another key feature of Blackfire is being able to share profiles publicly. So unlike XH Prof, which dumped it on your server and on your Dev server in there, that you can't throw it on Slack to your colleague, this one you just put a link. So it's like GitHub. The post request is like that for a kernel. Ah, sure. Yeah, so the next thing I wanted to show is for a different case study, this mysterious client that shall not be named. They have a home page that was really slow. How slow? Well, let's open up the Chrome's Inspectoria tab and see what's going on. In my Docker, so my rather fast SSD laptop, the request to slash, which it fills in with the host name or my case Docker, takes 400 milliseconds and then we see it's actually a redirect, a three or two. And then it redirects to the actual slash en, which is the real home page and that takes another 300 milliseconds. So my really fast laptop, it takes 650 seconds.  One, and two we noticed why the hell is this redirect taking longer than the actual page request? Can you guys guess what the redirect must be doing? It's language detection. Yeah, it just sniffs a header in the browser and so if you really want to be high performance, you probably like do this in Nginx, right? Or whatever you set up as you reject the header and then do the redirect in some C level server that wouldn't even touch Drupal or PHP. But let's see why Drupal's so slow and Blackfriars are helped. So if you wanna profile a request like this redirect, so we selected this request. Oh yeah, because the moment you enter it in the thing, it'll redirect to another URL. So you can't ask Blackfriar by clicking that button as we saw. You can't ask it to run that first thing 10 times because it already redirected, you know, you'd be on it, slash, yeah. So instead, you select the first line in Chrome, you right click and you select copy as curl. Has everyone done this? No, I'll show you with a different screenshot. So in the network tab, here's the network, take every request, you right click and you go copy as curl, okay? And then if you open your thing and you paste it, look at what you get. You get curl, the URL that Chrome is gonna open to and then all the headers, including the cookie, the cache, everything that Chrome sent. So it really mirrors the request very accurately. Okay, so you did this and then you prepared Blackfriar in front of whatever you just copied, right? So the Blackfriar command line client lets you say Blackfriar space based and then you actually get a command that Blackfriar will recognize. It'll run the exact same request, but now command line instead of in the browser extension. So it'll give you the same little 10 hits and it'll give you a summary here and a link to Blackfriar.io with your profile that you can view. So just the same and this will work for all your Ajax requests and all your POST requests as well. So here, once we open that profile, we're not gonna look at it now, but once we open that profile, here's what we saw was in the hot path. Most of the time it was being spent in this function. Can you guys tell me what's wrong with this code, Drupal 7? It's already written in the page type, isn't it? Yeah, it's all the... So this is happening in Hoke pre-process page and it does a redirect. So by the time Hoke pre-process page run, it already rendered the home page, which it'll never show because the last line of Drupal go to is exit or die.  And it's already probably rendered some blocks. I'm not even sure, but I'm pretty sure. It has already rendered blocks at that point. So you shouldn't put redirect code or access control or any of that kind of stuff in a pre-process hook. You should put it in Hoke in it at the very early part of the bootstrap cycle. You do that and now the redirect instead of 350 seconds takes 74. Like I said, that's not great because you should move that into engine X, but with a one line change, you've already saved 90% of your redirect and 40% of your home page load time. So that's something. So here's some more tips. I already talked about aggregation, but sometimes you need to disable it because sometimes you actually care about that first time for some reason. So if you know what you're doing and you're willing to deal with more randomness, there's a little checkbox that lets you say run a profile but disable aggregation. Another thing you need to consider is that when Blackfire shows you its graph, its call graph, it doesn't have like node load is called with many arguments, right? Like a node ID perhaps, 36 times. It throws all of those things into one. So it merges all those calls with different arguments into one. Sometimes though, like some functions do variously different things depending on what the first argument is. The function theme comes to mind. Theme page, theme block, all of these things are very different. One of them might be slow, one of them might be fast and they're all running different code. So basically sometimes you will want to tell Blackfire, I want you to treat a function name and for this function, the first argument as effectively the function name to be used in the graph. And it can do that. Another thing is everybody here uses XDBug, okay? Well if you're like a Drupal backend developer, you should be using XDBug. It saves you a lot of time. We use it with PHPStorm usually. And the catch is you do have to, every time you're going to run a profile, you turn off XDBug and every time you are XDBugging, like you just have to keep turning XDBug on and off. We'd like them to fix it somehow but we haven't figured out how to do it. So we were starting, we were using the two tools together all the time and we have to restart Apache or PHP VM quite a bit. Another thing is it does introduce profile overhead when you're running the profile. Remember that they're bragging that it doesn't introduce profiling when you don't click that button but that means the request time is gonna be slower with Blackfire running. And it's not 10% slower, it could be 50% slower. This could be substantial. However we find that despite the slowdown, it still gives you reasonable numbers. So it doesn't introduce so much noise that it confuses you. So I mean this is just a fact of profiling but don't be confused when you see different things in your Chrome Inspector tab versus the Blackfire report. Another thing to keep in mind is the memory versus time trade off. In computer science it's a fundamental trade off if you ever study like algorithms one on one. So you can always like add more caches and then you cache everything ever. You never expire those caches and maybe eventually you'll never have to recompute anything again but you just have to have a lot of RAM. So Blackfire lets you keep track of both but keep in mind that just by adding caches that doesn't mean you've made your assist admin's job better because all of a sudden he'll say, well your server build's gonna go up because we have to get a more expensive server with more RAM or we have to get more of them. And another thing, okay so that's that. And another thing is the Blackfire SDK which is like a PHP library. I believe it's actually by installing Blackfire probe. There's a set of PHP functions. You can just call that are magically there because Blackfire is PHP extension, the probe is there. And one of them is enable probe and one of them is disable probe. We use it all the time when we have a really thorny situation and we want to say don't profile me all of Drupal. It's way too complex. I know the problem is somewhere between this line of code and that line of code and everything it calls within it. So then you just, at the top of that line of code you say enable probe, at the bottom you say disable probe and then your profiles will magically only be filtered for that. If you use it like we always do please make sure to, after you've made your fix remember that methodology of Mafia you have to go back and actually compare the overall non-filtered profile to make sure you get better results because, again, maybe you just pushed it down to another part of the page. The last case study is that for our own site when we upgraded to Drupal 8, we love it but why was it slower than our old block? So we use Blackfire and we actually noticed it wasn't slower because when it's cached it's cached but it's great. So the page cached when it works just as fast but we noticed that when you edit any node like pretty much all the caches seem to get expired or most of them and so our blood view became way slower and so we were wondering how do we, how do we fix that? We edit nodes all the time or maybe people do other things to change the things that invalidates cache. So first of all, we had to write a Drupal 8 equivalent of a little hook I guess it's an event subscriber called this one subscribes to the kernel event request event and it says call my little function called kill block cache and that little function in turn calls the Drupal cache tags invalidator service.  And this is the name of your hook implementation and what it does is it calls an API call that says please find me whatever thing is supposed to invalidate cache tags and invalidate these two tags the node list and anything tagged with node 239 node ID 239. So this is a little snippet in Drupal 8 for disabling cache so that when we do our little aggregation we don't ever have those things cached. So then we run our profile and in this profile we noticed that 17% of our page load time again for static corporate site which is showing blogs, like a blog listing 17% was actually being spent in something to do with blocks and it said get visible blocks per region and it was inside of Drupal 8 core our corporate site has very little contrib modules almost no custom code. So we looked at what's in it and we noticed this code and we noticed that it does blog storage, blog properties, blah blah blah and then for each block on your site basically it's a for loop for each block on your site it calls block access view. Can the current user view this block? So it's checking access control individually for each block. Well our new Drupal 8 site is multilingual and in Drupal 8 a lot more things are blocks than before, that title and stuff. And in the way we built it not knowing better at the time this is a couple of years ago we made a French block and an English block for all these little elements or fields so we had a lot of blocks then a lot more than before we had like 150. So with 150 blocks the same site as we had before became from this code's perspective a lot more complex with a lot more access control to check. This is a similar challenge that we always had since to Drupal 5 or Drupal 6 days in the node access system when you wanna have a menu rendered with a bunch of nodes in it Drupal wants to be able to hide all the little menu items that aren't accessible but it'll have to call node access for each of the hundred items in your menu. And so in Drupal 6 they came up with a node grants scheme. Are you guys familiar with this? Like what is a hook node grants? And hook access for hook node grants. Yeah, hook node access. So yeah, so we realized that the exact same scheme can work for blocks and so far we have as a contrib module like an implementation that just hooks into your site and implements it.  It took a day and a half. This is definitely not in a trivial category fix but so we applied it and it saved 80 milliseconds from the page request time which was quite substantial. I believe we had 117 milliseconds in that function and now it's minus 80 so I guess there's 30 left. So it's a pretty good thing to say. This is a little screenshot of that diff profile that you guys heard me mention. So that's a little other motivation for using it to like a profiler and why black fry is a nice one. We wanna point out that what you've seen so far is all free but once you get in you start paying them like $15 a month or something you're at the profiler plan. I don't know how much it is a good one but not too much. And you get more data retention than a couple of days and you get to use recommendations which Jigar will talk about. At the premium plan you get to have multiple team members. You get to have custom metrics so you get to say measure this particular thing and you can integrate black fry runs and these metrics with your CI system and it can have like Slack integration and if you pay them enough money they'll even let you run your own hosted version because you're running for a large government or organization that'll never send their quick profiles to a third party service that you guys aren't using GitHub at all. So that's kind of where they are. So Jigar. Okay. Okay, so the thing is like Alex mentioned sometimes we want to, has anybody used VHAT tests?  So that happens per commit. So like what usually what we're as developers used to doing is before launch we test a site and we say like everything is looking good. If this is not looking good then we'll make some changes use blackfire fix it and that's it. We launched the site. Now if we build some new module after like say the site launches and then you have some new requirements you want to write some new code. So what happens is like for those like small development cycles we usually do not tend to use tools like blackfire and stuff to evaluate the scenario. Like I made a module and now how did it affect my performance? At launch the site was working good but what happened after the new module?  So say somebody makes a commit and pushes it and you see that suddenly the site has become slow like it will give you raise a red flag saying things changed in the blackfire evaluation. So that way we get to know like per commit what performance hits like performance changes that the new code is bringing. So to do this we don't have to do anything big. What we have to do is we have to have a dot blackfire.yaml file and we get to like write some tests there. Like for example the things that you're seeing here are tests which get Drupal 8 integration with blackfire which executes these following tests and lets you know like gives you suggestions, recommendations stating for example you should execute less SQL queries. So say you are executing more than a certain number of queries you get this message then you should enable APCU cache like for Drupal 8. So you can specify rules of your own and we will take a look at it here. Like by the way we wrote these Drupal 8 integrations for blackfire and there will be some for Drupal 7 soon as well. So like I said like we have a dot blackfire.yaml file in the root of our project and I am sharing this example of, first we detect the metrics for big frameworks like Drupal 8 and Drupal 7 as well and Simphony and you already get some metrics which you can use in your tests but you can define your own custom metrics like here what I'm doing actually is matching any calls which are made by this particular function to this particular function and every time this condition is satisfied this particular metric will have like increment. So you would get to know whether the session is like whether somebody is logged in or somebody is not logged in and if it's an anonymous session then we will have plus one or something more than that in this metric and similarly we're doing the same thing for detecting whether we are loading a Drupal page from cache or if you're loading it like we're building it over again instead of using page cache. So we define these two metrics if like you want to define custom metrics and then we use those metrics in our tests like this by writing a test section in your blackfire.yaml file and as you can see when this condition is satisfied like if Drupal 7 is installed and if it's an anonymous user request then we check whether this condition is satisfied as well. So if the page is being built from cache then everything is fine but if it's not being built from cache like page cache is turned off then we get to see this particular message.  So we can do things like this with the blackfire.yaml file, right? Yeah. And so my colleague Dave and later Jigar worked on creating all these metrics and assertions and we submitted them back to blackfire who has like encouraged us to do this and so now the built-in blackfire tool has this recommendation section which is just running these assertions that we list here and if you just open blackfire on your site it'll run through this checklist and we for the previous in previous projects we have actually, we don't know why this site is slow and we're like wait, blackfire says there's a recommendation we should look at and sure enough when it was the recommendation that we've written like half a year ago and it helped us. So checklists are great and checklists built into your tool are even better. So check them out and at a certain level you're allowed to write your own assertions and metrics and they're as simple as this. Yeah, so that's about it. So we have some blog posts here you don't have time to cover them on the slides. Follow me, my colleague Dave and sorry we forgot to update the slide but and also, I am Jigarius by the way. Jigarius, Jigarius like this. So, Jigarius using, yeah. So follow us on Twitter, read these blog posts with more details on some of these case studies. And also been asked by our corporate propaganda department to point out that we're doing lots of upcoming online trainings so we do some site building and interest stuff but we also are doing developer training as well. Nothing covers Blackfire specifically but if you have interest, we can actually come up with a day long course. So let us know if your organization needs it. And there's a week long five day training in Ottawa, September 11th to 15th. So tell your colleagues and check out the volume of the last training. And the last thing is if you guys sat through this presentation we were very grateful. And so the good folks at Blackfire were happy that we're giving this talk and they sent us a few books to hand out. So if you guys are interested in grabbing this book it was is written by the creator of Blackfire and Symphony. And like I mentioned, it's as much an art and a science and these are only a few case studies and he actually goes into a lot more details and he's as good writer of prose as he is of code, I'll be shocked to share. So this is a really great book and that's what inspired me to write this presentation. So grab it if you'd like. Thanks a lot. Don't have time for questions. I think you have your break, but we'll be here. So come up and ask us. Thanks.