Okay, so what's cool about it is that they are built upon another open-source tool and they're also open-sourced there too and that's not so common today so yeah thanks so welcome to the last talk in the web dev room today we try to make it as easy and smooth as possible after a long day and we want to share some experience of building the open-speed motor and what we learned the hard way during all the time and want to encourage you to also build open-source software even though you might think it wouldn't be worth it in the first place but yeah let's get into this first by explaining what we are not doing so are we product developers like no we are not doing anything for specific products that we are selling to customers so do we earn our money with open-source also no no money from that but that's fine are we researchers academic like every researcher looks like this probably but still not us and we we have some of the people from Akamai here and there are a lot of services which take performance in the center that's the core business it's not not our way so we are we are not in this business as a main purpose yes so who are we actually like this needs I'm Stefan and we are working for an IT service provider called ETERATEC and actually we're just full-stack developers and we are building sophisticated software systems for our customers which are mostly internal stuff like for enterprises so how did we get into touch with all that stuff why are we now implementing and maintaining the open-speed motor since eight years and how did we get into daily web performance work anyway so how this happened and what we learned during that way this will be the key part of our talk we hope you can have some takeaway from it so to tell you what happened let's go back to 2011 yeah so the whole project started 2011 and we came to this topic because one of our customers we did some other development jobs very large e-commerce platform had a very simple setup to measure their web performance it was basically to ping some URLs and get the the times from it then that was basically it and yeah it was really simple and it was just like a smoke test whether the page is up or not and they wanted to improve that and in 2011 there was some some ejection to web performance but not too much and so they asked us whether we can we can help them to build something that gets more insights for it if you would have these questions today you would have a lot of tools like lighthouse all that stuff you can do in the browsers we have web page tests in forever we have side speed i o we have two yeah just a broad range of tools but in 2011 just to take you back first and to give you an idea what our situation was like um a few facts like the market share in browsers was still mostly desktop mobile was not completely but mostly irrelevant and i mean it was iphone three time no care was still in smartphone business seriously like they tried and so on and if you look at it now the world changed definitely if you look at the browsers directly and look how they looked like back then it was a completely different world the world was dominated by internet explorer unfortunately we are out of this time but to be honest it doesn't look that much more healthy on the 2019 side so now chrome is dominating what's the dominating and i won't complain about chrome yeah and last but not least let's look at the web perform tooling we had available um so page page speed inside emerged in 2011 2012 the navigation timing api was not yet available there was no lighthouse no resource timing no side speed i o no things there was web page test like there was web page test forever i think i don't know i don't know a word without web page test yeah it came with the internet right so this was our yeah our setup yeah that was basically the the point where we are standing and i had to think about a system that can provide detailed results measurements because our customer was really challenging they wanted to to have something that measures all the time that measures a lot of different situations different browsers different internet connectivity and they wanted to be able to deep dive into every result so what they had before was just something the operations team can could use but they wanted the tool said that every developer could use to to um yeah to to inspect the pages and to have a get a feeling what's going on and what could be better and yeah could be improved so stefan already said it um web page test was already there and so uh yeah no no wonder we we ended up with a setup like this it was initial in a private instance of web page test measuring on different browsers with different connectivities all the different uh uh sorts of like the like like the real users uh visiting are visiting the page and we used a wpt monitor another uh php application in this time uh which could be used to automate this measurements so yeah we got it running and it worked but our customer came up with some more extended uh the crimans and one of them was derived from their analytics data so they had very detailed uh idea of their users and they have to have defined or derived different user types from the analytics data so something like the explorer guy or the the the direct buyer or something like that so these these types of users are uh yeah walking through the application in different ways and this these are different user stories user um customer journeys so to say and they wanted these customer journeys or these user types to be reflected in the measurements so they wanted the measurements to walk through the shop in the same way like the real users did and uh yeah that totally makes sense because of browser cache fills up in a realistic way and things like that and another thing they wanted was somehow to get an overview of the overall performance of the whole application so they wanted to be able to combine all the different measurements to one single number something like that so a key performance indicator for performance so and so we realized that this initial setup with webhatch test wasn't enough so we had to extend this and we had to build some things around it to get this done and this was the the development that started that yeah finally led to the open speed monitor and our initial approach was something like this so we had still uh wpt monitor with webhatch test and we derived additional informations from the data so we had a something in between in the beginning just a proxy for webhatch test which collects the data and applies some logic to rewrite this kpi and we call it customer satisfaction index I don't I won't want don't want to go too much into detail but yeah they they ask ask some real users in a lab how they feel the page behaves and they derived or we derive together with our customer informations about how page load timings for different pages translate to the satisfaction of the users of the pages so all this together gets the possibility to combine this to one single index so yeah so this was where we were in 2012 and we involved this for the customer to 2013 and 14 and as we build software as we do in our business it was closed source there was no intent to have it available for others and in 2014 there was the idea to just open source it without having a bigger picture in mind like not caring it was just a good idea everything was open source why not put it open source we developed it for two years then and it's yeah built a nice component and we could think it would be useful for others so long story short we talked to the customer and they agreed that we could open source it and maintain it and yeah so we had the naive approach of just going there and putting it open source so we had one commit by Niels or the code base pushed to github and we quickly realized well mistakes have been made they are online now and the first learning came quickly that open sourcing a project is slightly more than just pushing it to github so this was the open speed motor when you would open it with the code base we had on github that day like you you started and it was a complete power user tool I mean we had it for our customers set up private infrastructure everything and so if you would just open it from github or download it compile it get all the information how to actually well start it then you would see this which is totally unusable for anybody who has no idea of what it's about because it's about direct filter options on results and so on there was no getting started documentation no setup dialogue and it was just hard to start also our main feature the customer satisfaction index was completely hard to configure it was about downloading CSVs and uploading them and we had APIs for them because it was never intended to be a tool to be open source and just use so it was completely inaccessible to other people and we learned that the hard way because nobody cared but that was not hard enough like we forked web page test by then because web page test was not able to do everything we wanted it to do and then we weren't of course when we put our tool open source we need other people to have this fork available but not as our fork of course because we've a diverge from web page test so we simply tried to yes have a pull request if you take a closer look it's a dream of every maintainer like 50,000 addition 8,000 deletions 80 commits with yeah references to internal tickets and yeah no wonder it was not accepted it was not even commented on like yeah we can do it but I need some time to check it yeah so they never made it into web page test and there was that was yeah a hard lesson to learn for us but yeah yeah so that was not very nice because we were a part of the new features and web page test and nobody could use our feature so no good solution no good situation but nevertheless the next year we extended our services we had further customers which we analyzed applications for and we had a lot of work and our setup worked so we kept with it with the situation like it was and yeah just worked with it and we used the open speed motor for many customers in that day and evolved it but we didn't focus on open source yeah that's it that's it and there were some other challenges so we all at Eterotec are software developers and we do development projects for different companies and so we had a lot of changes in the team so it was just a small team developing this this tool and yeah most of the people were there for some weeks or months maybe and then switching and yeah it's it's hard to keep track of of all the changes that happen in this area so web development is very very much there are a lot of changes in metrics and browser metrics or in the in everything so it wasn't so easy to keep track of this all the time so and so we didn't found the time where we didn't realize the changes needed but finally at some point we decided okay that we can't go this this way anymore we we have to do this invest to get this merged back into a web page test and that's was that was really really a good decision so it was a lot of work mainly done by Stefan and we yeah it was a complete re-implementation we had to do because the fork was I don't know two years three years or something away from web page test and but finally we got it and we could it made the way to a web page test and was very useful for us and I think also very useful for others and yeah yeah and we also made lots of effort to make the open speed method self usable like a little screenshot on the bottom right we had a landing page which explained what it actually does with results for setting up measurement for setting up the infrastructure so at this point around that time you could actually just start it and you're getting introduced on how to connect to web page test how to set up steps scripts how to set up measurements you could use the official web page test instance to actually use the open speed motor so we were in a much better position so this is the next lesson we learned working together is really key so it's very important not just doing your own thing and consume some open source projects and do with it do things with it with it but also yeah also have it in mind that the changes you make you should integrate again and work together with the people developing different tools and yeah the effort we made to re-implement and integrate this multi-step feature also um yeah the advantage we had was that every web page test development since then reflects or takes multi-step measurement into account so that was a huge benefit we get from it yeah like Patrick Medin the maintainer and developer and everything of web page test he cared about it and we got actually support from him like when we introduce it and give him the work and everybody then we got it back from the community and from him and that was hugely beneficial uh benefit for us and actually some people started to care and use the open speed motor so first issues on github appeared from random people we don't know about but it was a good sign i mean they were complaining but we were happy that people complained because it means people care and that's nice but let's get to 2017 so our customers cared more about mobile performance as anybody that tries to measure synthetic performance on a real devices other than well servers or desktop PCs like mobile we tried or we do and looks like this this is basically like um a drawer from a server wreck and we have lots of stuff in there the data center is not very happy about that stuff that we put in there but still they let us do it um and we had it in the first place like a few one or two iPhones a tablet an android smartphone but we needed to upgrade it we need to measure different things and things were completely getting out of control for us like this happened for example if you can't see it the battery was blowing up and like basically blowing up the phone because no phone is constructed to be 24 seven being on power measuring websites and yeah so this was happening and we were like um is this normal can we do something about it and every time when we went to the data center it was a bit scary what happened every time we had a call we were afraid that they would say yeah our data centers on fire because of you and like this was what we needed to build to make like any kind of operations so I don't want to go into detail but we had Raspberry Pi is lying around for every mobile phone with best scripts on it running measuring battery temperature reporting it to us restarting the network doing the traffic shaping and it was yeah one best script started with it then another came then another came then we had 10 best scripts doing stuff um and it was working but it was not stable in the end so every time I went into the office I we had a few phones in the office by then I got into the room restarted every phone and get to my desk and in the evening I would restart every phone and get home like there's another good idea so we started obviously to talk to people about it and this was a really good idea because turns out they also had problems the agent running the code was not stable itself it was not fixable just by having more best scripts it turned out so we got input like we need a rewrite of the agent and we reported our struggles and we heard what other people would do and like when pet meanin implemented it at the end um he made it really better for us so all that discussion with the community and the exchange was really really important as equally as important as contributing or sharing code so the new agents running on the mobile phones was developed because we were in touch with the community all our things we needed it to do were already implemented or we could directly like talk to it and keep track and implement the stuff we needed and we knew that other people's phones would also blow up and this was good for us because I mean they still like the battery thing it happens still today our solution is to change the battery or the phone every six months it's not a good solution but everybody is doing it this way like everybody we talk to like pet meanin is doing it this way and he davis is doing it this way they all have that battery blow up problem so it's not solved but it's better for us to know that other people have the same problem yeah yeah okay so the next year 2018 uh we mainly did several things in the open suite moving toward the main part was that we realized there are again some new metrics and really important metrics because they take the user into more into account so the user they're so to call user centric metrics because they are not so technical when when is the page loaded when at the first when that's the first byte arrives or something like that on instead it's more like when is the most important part of the page rendered so when are the hero timings we heard it before a rendered or when is a page usable when is the main thread so free to handle user interaction with the page so time to interactive is a metric that can measure things like that and these new metrics were developed by different people for example hero timings were integrated to web page test by developers from speed curve things so and yeah we we integrated them into open speed monitor to to be able to track them over time so that was really another lesson we learned it's important to share your code and other people can use it for example this multi-step feature is used by others now and on the other way if people develop things like speed curve and they also open source it and put it into web page tests we can use it so together we can really get wave further than we would build everything on our own it would be impossible for us to care about all that stuff ourselves because we are still doing customer work all the day and have just a little time left for the open speed motor and just with all that work from the community and that we like exchange everything with them it was just possible for us to keep track with all the development and and the web performance area so yeah so we did a lot of stuff and in 2019 and 20 in the open speed motor was we have a new dashboard for the whole application to monitor the performance we integrated a new feature called performance aspects because people or customers always ask us how fast is the page when is it available when is it interactive when does anything happen they didn't just care about one question and we thought it would be a good idea to just reflect these questions in the open speed motor so like you can have not just metrics by now but these questions and then we have an interactive metric explorer to assign metrics to these questions to abstract from the from the metrics and having an easy solution for the customers to see like this page is interactive at this time and how it's measured they don't care about it to be honest and you can define it once per page or something yeah this is one something we also learned it isn't the same metric on different pages always so if you have the question when when is the main part rendered or one is when does rendering starts and you need different metrics on different sort of pages so we abstracted from that and yeah build something like just a metric is it usable and yeah yeah but still it's 2020 now and we need to say community engagement with the open speed motor is still low and that's okay for us actually because in a long time we tried to build a community around or thought about doing it but we really yeah learned the hardware that we have no time for it just to do it and we don't need to force people to do it because we could still provide or use it on our own and have other benefits for our customers and like it was not important that people contribute to our project because all the contributions around what was keeping us alive and what was even yeah important for us so yeah the lesson learned here is community doesn't come for free and needs love actually probably we don't know we can just make assumptions why we don't have something but we can just say in the end it's okay for us it doesn't need a community to be an open source project which is successful for you because for us it is a success because we can use it and we couldn't use that tool with these requirements for our customers if we haven't built it open source yeah so let's have a quick wrap up of the key three key things that we took away from the time and then you maybe can take away from our experience so first of all without open source software we would still be on a web performance level of 2012 like not we like you and us but we because 2012 is when we diverged from web page test and when we built our own tool set which was not open source and we could not have used all the features and all that discussion because we would have lost the rest and then we wouldn't have enough funding from customers to care about and build it ourselves so yeah probably we wouldn't be there and we wouldn't do anything with the web page test anymore yeah and we wouldn't be standing here so the next of the three lessons we or thesis we want to present is it's not that important how many lines of code anybody else is contributing to your project it's much more important what happens around this so to collaborate with other projects to build the things into other projects that you need for yourself and to talk to the people to take part of it and yeah and the last one is that you don't need to be a key player in web performance to contribute you don't need to be a researcher browser developer or being in a web performance team which are totally cool things but if you're just a developer like us and just have the chance of doing something you can still play your part you can contribute and you can do some something without being yeah without it being your major profession and yeah it's sometimes it's just enough to just share and discuss what you are doing and not just being in your black box and yeah we are really happy to be part of all this stuff because I mean for us it got a passion we are still doing it we're standing we care about it and would not have been possible if we would not have stepped in this step of putting the open speak monitor open source so in this way we can just encourage you to even if you think my tools will not be used or is not used just put it open source engage with the community and you can benefit from it yeah thanks for listening thanks for the wikipedia team to have wikipedia team to have this deaf room it was a really interesting afternoon and enjoy the rest of the foster somewhere he has a question feel free to ask or ask us later yeah well there was some discussion turned out the battery has still a bit more logic than just plugging it there there were some attempts by people to replace the battery by a DC charger did not work out to be a stable operation so yeah you've tried it sounds like yeah it's pity but sometimes you really need to replace a whole smartphone just because the battery is blowing up yeah that's why we talked to them and that was was was making us really happy that they are also experiencing the same issues and cannot solve it so yeah so the question is what was the fact that got us like people using it oh okay well it was basically when we made it available for others to use and being set up and help them to get into the project because if you run it and you have no idea how to set it up you will probably not use it so when we did develop that and we had some documentation issues arose and these issues showed us that people were caring and issues as our emails are still coming from people we didn't know like yeah we are using it for two years now can we do this and this and we are like I don't know I didn't know somebody is using it yeah but also things like docker help so we improved the documentation and then we had a docker compose file and then it's much easier to just start up and have a look at like 2017 or something we provide a docker compose file setting up everything including a web page just instance or an api token I don't know but just having a complete setup with one command was very important it really depends on the month and days and on the team like I'm out of developing it for like half a year now because I have a different project and I yeah do some fun stuff maybe and no I mean the more funny parts of you know seeing the development of open speed motor and seeing what's going on but also for me seeing what's going on and web page test because in the end we highly depend on web page test some if I have to have eye track what's up in web page test more that's different for you yeah I'm also out for two or three months now but from time to time and there and yeah it really differs a lot we have we have over the last seven years you could say we would have one or two full people caring about open speed motor but moral work was like all the operations because we have the private instances for customers but it was all the same team for us which was caring about the customers doing the perform performance analysis for different customers developing the open speed motor so that it would not die because of old old technology and yeah so it was always a mixture couldn't say so many hours okay great thanks