Welcome back to the last talk of the day. This will be Kenneth Hoss from HBCU-Gent, giving us the easy build state of the union talk. Thank you, Simon, and welcome everybody to the traditional state of the union presentation at the user meeting in a little bit different setting this time. So for the people who do not know me, I'm Kenneth Hoss, I've been working at the HPC team at Ghent University over 10 years now. And I became responsible for user support and training. And as a big part of that, I'm also responsible for the software installations we do on our HPC clusters. And as a part of that, I became the lead developer, the release manager and the benevolent dictator for life as they call it, of the easy build project. I did not create easy builds, it was already there when I joined the team in Ghent, but it was quickly handed over to me as the person who did create, didn't have time anymore to maintain or develop it. A little bit more about HPC again. So we are the HPC team that is central to Ghent University. So we help out anyone in Ghent University or by extension in Flanders who wants to do scientific computing. We have a fairly small amount of resources, about 20,000 cores in total currently with a handful of GPUs on the side. And our latest cluster is AMD Rome. So that's getting interesting. And we have a big or bigger cluster coming up, just named Hortons, just currently under construction. So that's about well over double the amount of AMD Rome cores we have compared to our tier two setup. And it will also include 80 A100 GPUs. We're also a member of the Flemish Supercomputing Center, which is a collaboration between the Flemish universities on everything related to scientific computing. So as usual in this talk, I will cover most of what has been asked in the easy build user survey. So the survey is something we do every year. This is the fourth time in a row that we've done the survey. So it's starting to get interesting in terms of trends, looking back a couple of years, what has changed recently, where things are going or yeah. It's becoming interesting to take a look at these results and see what's happening. We got good participation again, close to a hundred people as we did in previous years. And I'm not gonna cover all the questions here because there are some questions where we keep seeing the same answer. So it's not really very interesting to discuss these, but the full survey results are available on the easy build website, on the user survey part of the website. You can scroll through the raw results, including all the answers to the open questions are available there and pick PDF file. And I'll cover the highlights here in this presentation. So to start with the demographics of the easy build user community, so what kind of profile do people have? What type of organization do people work for? This is actually pretty stable over the years. We don't see big shifts here. So most people are identified themselves as a system administrator or as a user support, as a member of the user support team. And then there's a smaller part to our scientists, researchers, managers, software developers and so on. Also the pie chart on the right, which shows what type of organization people work in, pretty much we get the same distribution over the years. So this is not maybe not very interesting to talk about. The next one also in which part of the world are you located three quarters of the community, at least based on the survey are located in Europe. And that's also what we see in terms of who is active in the community. So this is also, again, very, very similar to previous surveys. So why do I mention it? Because it's interesting to take a look at the differences with the SPAC community. So SPAC has done a user survey as well very recently. So I think they wrapped it up in November or December of last year. I wonder where they got the idea to do a user survey. But if you look at the results, so many of the questions are actually very close to what we have in the survey. And two of them are also what kind of user are you? So what kind of profile do you have or what country are you in or in which part of the world are you in? And if you look at the answers for the type of user profile, you'll see there's a big difference between EasyBuild and SPAC. So SPAC is more used by, or at least a bigger part of the community as software developers and scientists. While in EasyBuild, it's more system administrators and user support team members. So you can see that it's almost a night and day differences, maybe a bit strongly worded, but you get the idea that the user communities are very different. In terms of where people are located as well, you can again see most of the EasyBuild user communities in Europe and a smaller part is in North America, which includes Canada. So that's important to know in the comparison. While on the SPAC side, the biggest chunk of people is in the USA. And then the smaller part is in Europe, which doesn't include everything in the rest of the chart because that's country-based, which Japan is separate, Australia separate, things like that. So again here, you also see a big difference between the different communities. And I think a big part of the reason for that is the impact that the ECP project has on the SPAC community. So they get a lot of funding through ECP and hence also a big part of the user base comes through the ECP project. If we look at how long people have been using EasyBuild, this looks very similar to previous years, but we do see a bit of a drift towards people that have been using EasyBuild for a long time. So two to five years or even longer. That seems to be growing. That's that ratio of the user community. But we do still see new people coming in, even though the new people are less relatively speaking compared to the experienced people. So three quarters of the community have at least two years of experience with EasyBuild. So that's very important to know also when we ask for feedback like in this survey or just on the mailing list or in Slack that the new people are getting, let's say a lower weight in discussions than experienced people. So it's very important to be aware of that. Then how did you first learn about EasyBuild? The majority is still here work of mount. So that's people telling each other about it is about one third of how people learn about EasyBuild. But maybe the more interesting thing to notice here is that it was already in use when I joined the job that I'm currently doing or when I started the job that I'm currently doing that has been increasing over the years and it's now up to a quarter of people that got to learn EasyBuild through that. So because the site where they started was already using EasyBuild. So to me, that's a very good sign. People who start using EasyBuild stick to it because they're probably happy with how it works. Looking at some of the highlights that we had in 2020 in the last year. So 2020 was a bit of a weird year of course, but for EasyBuild specifically, there were several highlights. We see the community keeps growing. We see also growth in the amount and also in the quality of contributions. That's very good, but as maintainers, we need to be able to keep up with them. So make sure we take a look at incoming contributions, make sure we process them, review them, try to work together with the contributors to get things merged properly. And up until now, it seems like we're managing to do that. And I'll get back to that towards the end of the presentation. We also saw a big increase in the number of supported software packages. So about one quarter, yeah, we have about one quarter more software packages supported than at the end of 2019. So that's quite impressive. We have a proper EasyBuild tutorial now. So I'll have a separate slide on that. We also have our first big security bug that was fixed and which has a proper CVE identifier. So the issue was that the GitHub tokens that you can give to EasyBuild, so it can interact with the GitHub API. That was leaking in an EasyBuild log file under very specific circumstances. So it was not always leaking. You had to do two or three things in a certain way for it to actually be leaking. But yeah, this was noticed by one of the EasyBuilds maintainers. We quickly fixed the issue and pushed out a release straight away. So we didn't wait with pushing out a new release. And we took the proper steps and reporting the issue and getting a proper identifier for it and so on. So that was a bit of a new experience with us. But to me, it shows that we're doing well. So we know how to go through that process. We're taking steps we were supposed to take. So I'm actually quite happy with how that was handled. Not happy about the bug, of course, but yeah. We've phased out the use of the Travis platform for running our test suites in the different EasyBuild repositories. We're now fully using GitHub Actions. For multiple reasons, Travis was being a bit of a pain. It was quite slow. Some of the tests were failing for connectivity issues or problems not related to the EasyBuild code at all. We're seeing a lot less of that in GitHub Actions. And then also somewhere, I think around summer 2020, Travis made some big changes in terms of how many resources it provides to open source software. So things got very tricky there. So we were very happy that we were already actively working away from Travis. We made an effort to make all the Python code. We have PAP8 compliance. So PAP8 is a code style standard in Python. And we also included the code style checks in the CI tests as well. So any new code or any code that we change will also be PAP8 compliant. And we have a new domain for EasyBuild. So EasyBuild.io is now the homepage and we try to make things easier in terms of the URLs we use for documentation. And so looking at supporting software, as I already mentioned, the number of different supported software packages keeps coming up, so no surprise there. And it's been fairly stable over the years. So it's almost a linear increase, which is good in some sense where we don't have many problems with adding more or supporting more software packages, but it's also a concern that it keeps going up. So there's no sign of slowing down or anything like this. Maybe even it's likely speeding up. So yeah, having to support more and more software is obviously a problem, but if the users or if the researchers keep asking for new software, then that's what we have to support. The EasyBuild tutorial. So this is new. So having a proper EasyBuild tutorial is something we didn't have up until mid last year. We have made a proposal or we submitted proposals for an EasyBuild tutorial at both supercomputing and the ISC conference several times, but it was never accepted, even though the tutorials usually got pretty good reviews. Somehow we never made the cut. That changed in 2020. So we did get accepted for the ISC conference, the European Supercomputing equivalent, let's say. So that was very good news, but then of course the pandemic hit and ISC was in trouble. They were not able to do a physical meeting. They ended up actually postponing all tutorials. So there were no tutorials at ISC 20, including the EasyBuild one. They were postponed to this year, but we figured we were already preparing things and we figured to just push through and organize the tutorial ourselves online. So we did similar to the setup of this meeting. We had a limited Zoom meeting, which was live streamed to YouTube and we handled all the questions via Slack. We ended up with over 100 people registering for the tutorial, so that was quite good. And since it was the first one, we were maybe a bit over ambitious in terms of the number of content. That we had in mind and also the amount of time we had planned to cover it. That didn't really work out very well, but at least the tutorial is there. It was fully recorded. People can get back to it. If things were going too fast, they can scroll back in the recording and so on. And we did an evaluation afterwards and we still got a 4.3 out of five in terms of score. So people still liked the tutorial, even though the most common remark was definitely having too much content or maybe going through it too quickly. So that's something we'll take into account for the next installment of the tutorial. So as mentioned, accepted tutorials were postponed to 2021 at ISC. So that means we again have an easy build tutorial or at least hopefully this time it will actually happen at ISC 21. So we'll use the tutorial that we have. We'll fine tune it a bit, maybe trim the content a bit so it fits in the timeframe we have in mind. And hopefully you'll hear more about that, about the easy build tutorial at the next ISC. Then the new domain, there's not a lot to talk about here, but we now have a proper homepage which just redirects to the homepage we have on GitHub, but that URL was a bit long and maybe hard to remember. So now it's just easybuild.io, which is very easy. And also the documentation now lives on the same domain. So docs.easybuild.io is easy to find. And yeah, we should have done this years ago, but somehow it never happened. Then more highlights in 2020 in terms of changes we've made to easy build on an additional features, the try update depth option, which is still experimental as something we spent quite a bit of time on or at least Alan has spent quite a bit of time on getting that properly implemented. We think it's a very useful feature, certainly not perfect, but it can help you a lot with updating existing easy config files to another tool chain and updating all the dependencies along with it. So that should automate a big part of that process. We also have proper locks now. So if you're running easy build on two different systems and installing in the same file system, they will not step on each other's toes anymore. So one of them will run into an existing lock file and it will just cancel what it was trying to install while the other one will continue. We have lots of speed ups in easy build itself in terms of stuff it was doing in the background over and over again without it really being needed. And that was causing significant slowdown in terms of responsiveness of the EP command. So several of these issues were fixed. Last year, we have better GitHub integration. So the new PR option, which you can use to open pull requests on GitHub straight from the easy build command line. That also works for framework and easy blocks. So not only for the easy config files and sort of vice versa, you can now also pull modified easy blocks from an open or merged pull request from GitHub. Very similar to how you can pull in easy config files from pull requests. We added support for installing software in an alternate system root environment. That there's a lot of background to this but this will become more clear later this week when we talk about Gen2 prefix and the easy project. And we also improved the Rpot wrapper script. So the script that we have to do Rpot linking of libraries rather than regular dynamic linking was improved a lot. There were some issues that it wasn't taking into account or some bugs left and right that were fixed. And next to that, we also have a new concept called easy stack files which I'll provide some more information about later. So these last three things, the sysroot support, the improved Rpot script and the easy stack files were actually a direct result of the easy project where easy build is one of the components. So we were using easy build or we are using easy build there in a slightly different context than we are used to. And that was the main driver behind these changes and these enhancements. Next to that, there's also significant improvements for especially for TensorFlow and PyTorch. So Alexander from the University of Dresden has spent a lot of time in applying patch files to TensorFlow, figuring out problems with the tests, figuring out issues with both TensorFlow and PyTorch on power systems, for example. So there's lots of changes in how stable the TensorFlow installation is when using easy build same with PyTorch. Also runtime problems, bugs in the software itself that we have patches for that are maybe not merged upstream yet or only coming in future releases. So there's a lot of these things that have happened in the last year. And more general, we also have improved support for both ARM and PowerCPUs as well as a REL-8 based operating systems like CentOS 8. So there were some issues, small issues there left and right that also got fixed last year. So looking into detail about some of these things especially the easy stack files. So that's a new concept. It's still an experimental feature in the latest easy build release. So it's not fully stable yet or at least not fully capable yet with what we have in mind. Easy stack files are a way to describe a software stack that you want to install with easy build in YAML format. So there's an example shown on the slide on the right where it says, okay, I wanna have bio conductor, easy build itself, Gromax open format are installed. And I wanna install these with this particular tool chain. So in this case, FOS 2020-A or for easy build the system tool chain, this particular version. And for Gromax, it actually has two versions with FOS and one version with FOS CUDA. So you can see this is a really condensed way of expressing what you want easy build to install. And then when you give this easy stack file to easy build and you enable the robots, it will make sure that all the dependencies, all the tool chains that are required for these installations are installed. So rather than having to give the EB command all the names of the easy config files that correspond with this, you can give it just a single file and it will make sure that everything is in place. Currently, it's a software first kind of structure in these files. So you list the software and then for each software you listed tool chain. We're probably eventually gonna support the other way around as well. So tool chain first and then software if that's what you prefer. The example file I'm showing here should already work in the latest easy build version, but we have some enhancements in mind, which make, we wanna be a bit careful there. We may have to change some of the behavior while implementing these enhancements. So that's why it's still an experimental feature. So we have to configure easy builds to allow you to use experimental features so you can use it. The enhancement we have in mind are support for also specifying easy build configuration settings. So for example, always enabling the robots when using a particular easy stack file or pulling easy config files from a pull request with Trump PR. So you can specify these things in the easy stack file. We also wanna add support for filtering where you can say, okay, grow max with Foscuda should only be installed on a system where easy build is configured with a GPU label. So where you can say to easy build, look, this is a GPU system and any software that's relevant on a GPU system should be included. Other things should not be included or the other way around. So we'll build or implement a way of expressing that and making sure you can specify that in the easy stack file and in addition, we also wanna add support for regular expressions so you can give it, you can use a wild card for version, for example, and say, okay, any grow max version you know about with this tool chain, please go ahead and install it. And then when there's easy build updates or new releases of easy build that may include easy config files for newer versions of grow max, you will get them automatically when you do another run with the same easy stack file. So all of these enhancements are ideas currently but we actually have somebody who's actively working on implementing these. Then the easy project, we have a separate presentation on the easy project tomorrow I think. So I don't wanna steal too much of Bob's thunder here but I do wanna highlight so I think this is a very important project. I was invited into the project by the people who started it, mainly from the easy build side, but I've been quite active in it and I've been helping out a lot. It's a very nice active bunch of people who are working towards a common goal where easy build is one of the key components. So I think it's a very important thing to learn about. And next to easy build, we're also using CERN VMFS, Gen2 prefix in this project, the Lmod modules tool and so on. So several of the talks during the easy build user meeting this week are actually motivated by the easy project. In some sense, I see easy as the next step after easy build where you don't just work together on a tool to facilitate software installations but actually work together on the software installations themselves. And projects like CERN VMFS make that possible in an organized and good way. So again, please take a look at Bob's talk tomorrow about that new new TC to learn more about this project. Then back to the survey results. So one of the other classic questions we ask is what kind of operating system people are running easy build on. CentOS or REL based operating systems are still the most prevalent here that has been the case over time and is not a big surprise. I think so it's pretty well known that REL based systems are most prevalent in the HPC community. CentOS 7 usage is slowly going down. So people are starting to switch to CentOS 8 and REL 8. The people that switched to CentOS 8 after the news that Red Hat will be making this CentOS 3 which is more like a rolling release maybe scratching their head a bit after that after hearing that news. So it's gonna be interesting to see how that works out. And there's already a new REL based Linux distributions being developed currently, the Rocky distribution which is not available yet. So you cannot install or use it yet but I think by summer of this year it will be usable and it may be a good replacement for people who started using CentOS 8. One thing that was striking for me here is that about almost 10% are still using CentOS 6. So this is probably on old and hopefully dying clusters. It's been going down but a lot slower than I was expecting. So 10% is still quite high, I think. People who use Ubuntu seem to be more aggressive maybe and into switching to a newer version of the operating system. And we also have Gen2 now as a new entry. So this was not listed as an option in previous editions of the survey. You could add it as an other category yourself but now we've added it here and about 5% of the people were using EasyBuild on top of Gen2. So that's probably people who are either involved with the Easy Project or who work in the Compute Canada Consortium where they have a very similar setup to what we do in the Easy Project. And overall this has been pretty stable. So REL based systems are still most popular. Next ones are Ubuntu and then there's a long tail of other operating systems that EasyBuild is being used on. Nobody answered Raspberry Pi OS. So maybe we can see that change as well next year. Then sort of related question is which Python version rather is being used to run EasyBuild. So this is not about what you install with EasyBuild or what you use to run EasyBuild itself with. So here we're expecting a shift to Python 3 since Python 2 is officially dead. It was dead end of 2019, sorry, even though they still had a last bug fix release in April of 2020. But still most people are using Python 2.7 as the Python version to run EasyBuild with. That's probably not a big surprise because Python 2.7 is still the default Python in CentOS 7 for example. So if most people are using CentOS 7 they're probably also using the default system Python which is why they're using Python 2.7. There's an increase in adoption of Python 3. So about one third of the people are using Python 3 to run EasyBuild. But that pickup seems to be relatively slow, especially if you know that Python 2 is officially un-maintained. So that's maybe a bit concerning. And as expected there is splintering across the different Python 3 versions. So Python 3.6 and 3.7 are more prevalent. But the more recent Python 3.8 and Python 3.9 are already being used and are probably gonna be increasing over time as well. So we're coming from a couple of years where basically everybody was using Python 2.7 and we're going towards a new era where there's a lot more splintering in terms of Python versions. That's a bit concerning in terms of EasyBuild development. But as long as we do a proper job in CI and making sure we're running the test suites on all of these different Python versions, we should be okay. While working on these slides, I discovered that I did some predictions last year. So I predicted that most people would still be using Python 2. So that's the case. Predicted a significant increase in Python 3 use. That was maybe an easy prediction to make. But that's also the case from going from 16% to 36%. And I gambled that Python 3.6 would still be the most prevalent Python 3 version. I got very close to that, but it's actually Python 3.7 is the winner here. So just for the fun of it, I made new predictions for this year. If we do the survey again at the end of 2021, I think we'll still see that Python 2.7 is actually the most used Python version to run EasyBuild, even though it's a dead language, it's a dead version. We're probably gonna see more splintering across Python 3 versions. And I think or hope that Python 3.8 will be the most prevalent Python 3 version, even though it's now pretty small at 3%, I think that's where we're going towards. Then related questions we ask is how troublesome would it be if EasyBuild becomes incompatible with Python 2.6? So EasyBuild today is still compatible with Python 2.6, just because it's pretty easy to do that. If you're compatible with 2.7, you're basically also compatible with 2.6. So there's very little difference between these versions. It's clear that most people wouldn't really mind that we dropped support for 2.6. We've actively deprecated Python 2.6 support in EasyBuild 4.0 already, which is September of 2019. And yeah, we're happy to see that the need for Python 2.6 is declining fast. So even people who are still using Central S6 probably have already updated to Python 2.7, even though 2.6 is still the system Python. So yeah, we're ready to let that die finally. Then how much of a problem would it be if Python of EasyBuild went to Python 3.0 only? So only compatible with Python 3. That seems like it's maybe a bit too early to do that. So people would be concerned about doing that, about 15% says, yeah, that would be an issue for me. We currently still don't have any serious plans to drop Python 2.0 support. So support 2.7 rather is not a big issue. And in addition, there's probably not a big benefit in only supporting Python 3. So in terms of code base, I don't think there's a whole lot of changes we would make if we only had to support Python 3.0. There's very little incentive from our side to go Python 3.0 only. Maybe the only advantage would be being able to run the test suite and less configuration so not caring anymore about Python 2.0. That's maybe the biggest advantage that we would see. So short-term, no plans to drop Python 2.0 support. If we do it, we'll probably do it in EasyBuild 5.0, which is currently not in view either. So we don't have any big plans to make big changes that would warrant an EasyBuild 5.0. And just to make it clear here, how things work currently, so if you run the EB command, it will pick a Python command that will work when using EasyBuild so it checks the version of the Python commands it runs into. Currently, it will consider first the Python command, so the default Python on your system. If that doesn't work for whatever reason, so if it doesn't like the version or the Python command is not there, it will try Python 3.0 first. And if that doesn't work, it will fall back to Python 2.0 in the hope that it finds a 2.7. If you don't like this order or commands that it checks for, you can define the EB Python environment variable to tell EasyBuild what to use, and then it will at least try this first before it looks at any of the other three. So you can just set the environment variable and run EB Show System Info, and that will tell you which Python version it's using. So I would recommend that people opt into using Python 3.0 to run EasyBuild through this mechanism. And if you wanna see which Python commands EasyBuild considers and whether it likes them or not, you can use EB verbose equals one to get more information about. Then back to the survey, we also always ask on what processor families people are installing software with EasyBuild. So here Intel is still king, and Intel Skylake is the most prevalent, or I should say Skylake X, I guess the one that supports AVX 512, so the server version. That's still by far the most used CPU. The older Intel CPUs, even very old ones are still around like Sandy Bridge and even Harper Town and the Halem and Westmere are still being used. And especially Sandy Bridge is still close to 30%, which is quite high, but at least it's going down. So if we compare to last year, last year about 50% of the people still had the Sandy Bridge cluster, now it's only about 30%, and also Haswell is slowly going down. No surprise here, AMD Rome is now at 36%, so it has seen a big increase compared to last year when it was only at 14%. So the second generation Epic AMD processors are becoming quite popular in the community and there's no real surprise there. So just again, for the fun of it, my prediction for 2021 is that we'll see a strong increase in AMD Rome. So I think it's slowly gonna be a good competitor for Intel, for several reasons, performance, price, all these things. And we'll probably also see a strong increase for ARM processors. So right now they are barely visible in the plots. So if you look at Graviton 2 and Thunder X2, only a handful of people actually install software with EasyBuild on these platforms, but I think that this will change dramatically in 2021 because ARM is becoming a very interesting platform to use for HPC. Then Accelerator, so same question, but for basically GPUs. Of course, Nvidia is still king here, no surprise. The Nvidia V100 and the Pascal series are very popular. The Capular, older GPUs are also still quite popular. And there's a quick adoption rather of the new Ampere, the A100, Nvidia GPU cards. So they're already at about 25%, so that's quite high. There's currently very little adoption for AMD GPUs and Intel XE, I guess these platforms are just too new currently to already be used very actively. But that's probably gonna change in 2021. We'll see more and more systems coming up with AMD GPUs. And 2021 will probably be a bit early for Intel XE, but also there we'll see systems that start popping up that use these Accelerators. Then more EasyBuild focused, so which tool chains do you use for installing software? So I tried to organize things a bit more here than I did in previous years to get a better view on it. So all the green bars are versions of the FOSS common tool chain. Blue is FOSS CUDA, yellow is Intel, red is Intel CUDA. And then the last box is a big mix of things. So patterns I see here is that the 2019 B versions of the common tool chains are most popular, are still most popular. That's probably because of the amount of easy config files we have. So we were using the 2019 B versions for a long time because 2020 A was delayed for several months. So yeah, we had the longer lifetimes, there are more easy config files for it. So it makes sense that more people are still using that. 2020 A is a close follower and at least we see pretty good adoption. So the most left hand graphs in each box are the most recent versions. So they are getting good adoption. So at least people are following along, are not getting stuck with very old, or yeah, very old tool chain versions. There's a big gap between FOSS and Intel. Again, no surprise there, Intel was licensed. That this is changing with the one API versions. At least those are freely available and free to use. So maybe we'll see changes there as well in the coming year and years. The IO-MKL tool chain. So the first one here, that's Intel Compilers Open MPI and MKL library is slowly increasing in adoption. So that's currently at 20% while it was lowered the years before. Yeah, and there's a long tail of different tool chains. Then here another prediction and we've had some discussions on this already in the recent easy build conf calls is that we'll probably see an increased adoption of client-based tool chains. This is a bit of a very easy prediction to make because most of the new tool chain versions like Intel one API or the AOCC compiler or the NVHPC compiler, which used to be PGI are basically all client-based. So this is almost a given it's not really prediction. But it does mean that we should probably start paying a little bit more attention to client-based tool chains and easy build going forward. Then which easy build version do people use? So the good news here is that 90% of the people relies on easy build, on proper easy build releases. And this is I think slowly increasing over time as well. So that's good. That means all the effort we spent in pushing out releases regularly. So about every six weeks is definitely worth the effort. And people seem to be happy with the releases that we push out since they are certainly more stable than the developed branch, which is, it can always have bugs that we haven't discovered yet. Also here, 80% is using easy build 4.3. So very close to the latest release. And over 50% is using the latest release which was released somewhere mid-December, I think. So that's also a good sign. There's quick adoption of the new easy build releases. So that means the jump to a new version is quite easy to make. Nothing breaks, usually when you update to a new easy build version. About 10% is using develop. I would guess most of those people are maintainers. So who pay close attention to what is moving the developed branches or are happy with running into a bug left or right and then try to maybe fix it themselves. People who are not using the latest release, so about four out of 10 people mentions they were not using the latest release. So that's the split up here. So some people just say the current version works fine for me. So that's about one third. Another third said I should probably upgrade but I don't have time to do that even though the upgrade should be straightforward. And then other reasons either using develop or I've made local customizations and work myself into a corner that takes time to get out of or then a variety of other reasons. Some people only update easy build when they jump to a new tool chain or things like this. So overall, this is pretty similar to previous surveys. In terms of frequency of releases, we've also asked this question every year and here more and more people seem to be happy with the frequency we have, which is about every six weeks or two months somewhere in between that range. We tried to push out a new easy build release. The hope was to push out a new version before the easy build user meeting this week that didn't happen, but we'll probably work on that either during the week or immediately after it. So in terms of frequency, people seem to be quite happy, that's good. Then another question we always ask is which part of easy build do you not like? And we usually have a pretty good view on what that is, but it's good to see that confirmed here in the survey. So about 30% of the people said they don't dislike anything. So that's pretty positive, that was nice to see. The most unliked aspect is the fixed versions that we have for dependencies and tool chains. We know that this isn't an issue or something that people don't like about easy builds, but yeah, it's a blessing and a curse. So it's good because that helps with, if somebody gives you an easy profile that works for them, it will probably also work for you.  So it makes it annoying to update to new software versions and update the dependencies along with them. That's currently mostly a manual job, even though there's things like try update depths that are a big help in automating that. So yeah, we should probably work on that and try to get this number down a bit. The error reporting. So yeah, in this survey, it comes forward as another aspect that people don't really like. It's also the main type of question we get in the Slack channel, for example, so people who tried something, it didn't work and then they're kind of stuck because they can't seem to figure out themselves what the actual problem was. So either maybe the compiler ran out of memory or out of disk space or yeah, they have trouble finding that the easy build log file. Yeah, we can probably do a better job there by being smarter about how we report errors about extracting more information from the output of the command that we were running, look for patterns. Yeah, we can do a lot better there and we have some ideas on how to do that. We need to work on it, but it's definitely in the back of our minds. Then the next most popular thing was the lack of manpower either for dealing with contributions or doing easy build development. That has been also raised a number of times basically in every survey. We currently don't have any dedicated funding for easy build development. So that's something we're trying to look into also actively looking into that but it hasn't worked out until now. So it's pretty difficult to find funding for ongoing projects so that are not research or that don't have, let's say, a huge impact on certain aspects of science. So it's quite difficult to motivate funds for that. We currently try to remedy that by making sure we are organized with the manpower we have. So one example of that is in the group of easy build maintainers, which is about 15 people, I think. We have a rotating maintainer of the weak role where we ask, okay, who has time, this particular week to pay more attention to incoming pull requests and then those people try to spend at least one hour for working day and actively looking into stuff that's coming into easy build. And that really helps. We see shorter turnaround of pull requests and more questions being answered because we know one person is paying more attention to things. Then other tools or projects that people are using in combination with easy build. No surprise here that most people are still using LMOT as the modules tool. And I think the tickle-based ones are actually going down a bit in popularity even though the 4.x version is being actively maintained. Interesting here is that singularity is, the popularity of singularity is decreasing. Maybe that's just because slightly different people answered the survey compared to previous times. So about a hundred people, it's maybe not the best possible coverage across the HPC community, but it was still jumped out for me a bit. Similar for SLUR, small increase there, but it's probably not to be overstated. Condal, Docker, OpenHPC and SPAC are pretty much a flat line in terms of how many people are using it. That's for most of these projects without naming them, that's good. That it's, the adoption is not increasing. The one that still puzzles me here is seeing about 30% of the people saying they're using both EasyBuild and KONDA to install software. So that's a bit mind-boggling to me. I'm not sure why that is or how people are actually doing that. And new here are CERN VMFS. So that's about 12, 13% are using EasyBuild in combination with CERN VMFS. Again, that's probably people in the Easy Project or Compute Canada, but maybe also other side. So like I know CSCS is using CVMFS as well. And the ArchPEC tool which EasyBuild can leverage to detect what kind of CPUs you are running on is a small but very useful library as well. Then communication in the community. So are people subscribed to the mailing list? We still have about 280 subscribers to the mailing list, but the activity is going down there. That's not a big surprise. We're seeing a shift to Slack for that. And this also comes forward in the survey. So less people are interested in actively participating in the mailing list and more people are interested in actively participating on the Slack channel. So things are shifting from a traditional mailing list to Slack and that's not a surprise. Oh, and also something that has been fairly consistent over the years in the survey is that we noticed that the significant part of the community remains silent. So maybe they're reading the mailing list or reading on Slack, but not actively participating in discussions. So that's very important when we try to pull in Slack, for example, for the opinion of people we should be very aware that a lot of people are just not speaking up at all. So it's not because we ask something there that it covers the whole community. That's very important to realize. The mailing list is going down in terms of traffic. So we still have 280 people subscribed there. I would guess many of those are not actually looking at the posts at all, but we still see it actively being used for questions that people are still getting their questions answered there. And I would say usually in a fairly quick turnaround time. So it's certainly still useful. And there's no point in stopping the mailing list. I can imagine some people still prefer it over Slack as well. And then the Slack channel has grown over time. No surprise, even in 2020, it almost doubled in the number of members. So we have 160 people extra compared to the end of 2019. And also the number of messages that were sent over time since the existence of the channel of the easy-built Slack in 2017 has almost doubled in 2020. So we're close to 200,000 messages sent now even and yeah, end of or early 2020, that was about 105,000. So it's almost doubled in a year time. So yeah, it's the main way that people interact with each other about the easy-built projects. Then as a bit of a sidestep, something new that has emerged last year is the easy-built tech talks. So that was an, it's actually an indirect result of the previous user meeting where I invited one of the lead open MPI developers to give a keynote talk at the last easy-built user meeting. Somehow he overlooked that email and then five months later, he mailed back apologizing for not responding to my email and that he was very much willing to give a talk like that for the easy-built community. This was months after the user meeting, so that wasn't possible to organize then. But we figured we could still set up a talk like this, a sort of keynote talk, but very technical and in-depth about the open MPI project. We set that up and while we were working on it, so back and forth with both Jeff and Ralph who did the presentation, it was clear that there was absolutely no way that they were gonna cover it in a single hour. So they ended up actually doing three sessions covering different aspects of the open MPI project where it is now how it is structured, where it is going and so on. So those were very interesting talks which are now also a key part of the open MPI documentation themselves. So if you look at the open MPI website, you'll see a video section if you go there. The easy-built tech talks are right there. They see this as a very good way of documentation, the open MPI community. We had a second easy-built tech talk as well since there was a lot of interest in ARM, especially when Nvidia announced that they wanna buy ARM. So yeah, we were wondering how mature is ARM for HPC and so on. So we found a good speaker for that, Chris Edsel, who works at an HPC site in the UK where they have a big ARM HPC cluster. So he gave a very interesting talk about that as well. And we hope we can do more of these tech talks in the coming months. So if people have suggestions for specific topics, I'm glad to hear them.  So nothing marketing or anything related to that, but just promoting the project, really going pretty deep in a technical way is interesting. Then also in terms of communication within the community, I wanna highlight that even though we're pretty vocal about this on the easy-built mailing list at least, is that we have easy-built conf calls via Zoom every other week, which are open to anyone to join. So anyone who just wants to listen in or who wants to see what is being actively worked on or who has some questions about easy-built is free to join the conf call. So the Zoom call is open to anyone and we share the link through the easy-built mailing list and also through Slack. We've been doing these conf calls actually since November, 2013. So I was a bit surprised when I looked back how long we've been doing these. So yeah, over seven years now and yeah, we've barely missed any of them. So pretty much every other week we have a conf call even if I'm not around as the lead developer then somebody else takes over and usually there's a good discussion going on. So we've had over 160 of these conf calls up until today and they've been very valuable. The conf calls are now organized at two different times. So at 10 a.m. European time, Central European time and at 5 p.m., so two weeks later, Central European time and then again it flips back and forth between 10 a.m. and 5 p.m. to cater to different time zones. So that was by request by some people more East who had trouble with joining the conf call because of the time of course. That has worked out quite well. The attendance has been pretty good over the last couple of sessions.  I usually try to create an overview of recent developments in the last two weeks. So it's clear that what we are working on, what will be included in the next release, what is still blocking the next release and so on. And there's always time for questions that anyone can raise. We keep detailed notes about the conf calls as well on the EasyBuild Wiki and that seems to be a valuable source of information as well. Then looking at contributors a bit and I cannot resist with including this GIF in the presentation again like I've done in previous years. So looking at EasyBuild contributions starting with the framework, the core, the heart of EasyBuild, which is in a separate repository. So how things are changing over the years. So the graph shows it's missing a proper labeling on the Y axis, but the graph shows the number of pull requests to the EasyBuild framework repository and by whom they were created. So the blue part is myself. So I've opened just above 100 pull requests in 2020 for framework. There's a small part HPC again which is actually mostly consultants. We pay to work on EasyBuild. Then another part maintainers and a significant part coming outside of the EasyBuild maintainers. So that's good. It seems that people who are maybe less familiar with the insights of EasyBuild are able to make useful contributions through pull requests. So over 50% of the PRs in 2020 was from people outside of the HPC you can team. So that's an important metric to mention. And you see kind of peak here in 2019, that's mostly because of the effort that we put into porting EasyBuild to Python 3. So that's why there were significantly more pull requests than compared to previous years and compared to last year. And similar view for EasyBlocks. So here we see an upwards trend in the number of pull requests in total. And my involvement in EasyBlocks stays more or less stable. So that's good. We're getting more changes in and I'm actually less involved. So I'm happy with that. Yeah, so about 30% more pull requests which I guess is the best unit of contributions you can get compared to the year before. And over 65% by outside contributors. So people who are not even in EasyBuild, outside contributors meaning outside of HPC again and about 40% by people who are not an EasyBuild maintainer. So they make changes, they propose changes and then one of the EasyBuild maintainers reviews, tests and branches those. And then for EasyConfie files the numbers are a bit more spectacular. Also an upward trend which is luckily a bit slowing down last year. So we have two years in a row with over 2,000 pull requests to the EasyConfie repository. So 2,000, that's about, if you think about let's say 250 working days in a year, that's about eight pull requests per working day. So that's quite a lot. So the growth you see here since 2016 is partially or probably mostly thanks to the new PR functionality we have an EasyBuild where you can very easily create a pull request from the straight from the EasyBuild command line without running any Git commands or without directly interacting with GitHub. So it makes the bar to contributing significantly lower. And also here we see a growing ratio in terms of contributors from outside of HPC again. So I'm the blue part again. The red part is our consultants for the most part and my colleague in the team. And then everything else above that is people outside of the HPC again team. So we're still very active but we're getting relatively more contributions from other sites, which is very good. And then looking at these EasyConfie PRs again but in a different way. So these are the same numbers per year but the colors are different. So this is more in terms of processing at least the left graph is in terms of how or who processes the contributions. So the dark green is myself merging pull requests for EasyConfie files. So I was involved or I merged about just under 500 pull requests last year which is still quite a lot. The light green is other maintainers. So they collectively tackled certainly the biggest chunk of pull requests. So if you look at the split between the dark and the light green in 2020, about three quarters of all merged pull requests were handled by somebody else that needs. And that's a quite good metric or yeah, you could say one quarter is still way too much. It probably is but I'm of course very active in the project. And then the right graph is showing how many of these pull requests were created via the new PR feature in EasyBuilds where you don't need to leave your terminal to make a contribution. And there we see an increased adoption. So over 85% of people are creating EasyConfie pull requests through straight from the ED command line. While a low minority is still doing that manually which is okay. But it's very good to see very good adoption of this new PR or the GitHub integration in general. Then looking at the number of unique contributors so different people making contributions to EasyBuilds. So that's still going up. So we're still getting new people making contributions to the project for EasyConfie files. We've crossed 250 different contributors for framework we crossed over 100 unique contributors to the project in the last year. So that's very good. And the lines are still slowly increasing. The increase is more is quicker for EasyConfie that makes sense because it's a lot easier to make a small contribution through a new EasyConfie file than it is to start making changes in easy blocks or framework. And then the same thing but looking per year because the previous graph was showing the total since the start of the project. Here we look at things per year. That's important because you wanna see how many people are actively contributing recently to the project. So for EasyConfie files, we see just over 100 people, 100 different people made a pull request to, or I think that's even a merged pull request to EasyBuild in the last year. And also at least for framework and easy blocks we see pretty stable numbers over the years. So that's very good, around 30, 35 people are contributing every year to framework and easy blocks. And then the final question in other than an open-ended one in the survey was how people rate the overall quality of EasyBuild. So how happy are they with EasyBuild in general? The excellent and great numbers are actually going up there. So if you look at everything that's sort of positive, so at least okay or better, we're hitting 98%, which is the highest we've been up until now. We see significant increase in excellence. So close to half of the people rate at excellence so that I'm not sure what happened there in 2020. But yeah, there's a significant increase there compared to previous years while it was pretty stable in previous years. So I guess this can only go down in 2021, we'll see. And yeah, there was another option pretty bad and nobody answered that as we go. So yeah, very happy with the overall evaluation here. Then looking forward a bit. So some things we want to work on in 2021 is somewhat fueled by the results of the survey, but we were, for most of these things, we were actually discussing already actually in comfort calls or in the Slack channel. So the better error reporting is something I certainly want to actively work on and make it easier for people to diagnose problems, actually pinpoint what the issue was, why installation failed, and hopefully that enables them in fixing the problem themselves, or at least they can more accurately report the problem either in the Slack channel or through the mailing list or through a GitHub issue or so on. So we get a more focused effort there. We want to make it easier to reuse existing easy config files and port them to newer tool chains or yeah, automate that process a lot more so it's less of a manual effort. So to answer the biggest thing that people don't like, the fixed dependency versions, we can probably remedy a big part of that by yeah, providing more features or better support in Easeable to make that more automatic. The improved regression test for Easeable, that's actually more to help myself. So before every Easeable release, we do a full regression test of, or at least we try to do a full regression test. It's becoming harder and harder to actually run it to completion. Of all the easy config files we include in the release and we try to make sure nothing is broken, certainly compared to previous releases. So if something is suddenly broken, it means we have made a change in develop that affects something that worked before. And then we try very hard to fix that first before we actually push out the release. Right now, there's some crude scripting that I do and have been using over the years to run the regression test. I basically push a whole bunch of jobs to one of our clusters, whatever is the least busy at the time, and then I have some scripting to evaluate the results of those jobs, but it's all very manual and very hacky. It's not very clean. So we wanna really start using Reframe for that. So as Vasilius showed in his talk, Reframe is the ideal tool for running regression tests, which is what we wanna do with Easeable. And maybe we can even pull that open a bit and have a public dashboard that shows these things worked with this particular version of Easeable. And maybe there was a regression, but it was not serious enough for us to block the release and we wanna make them more public, more visible. And we also want to run the tests in a more controlled environment. So using container images so we can run in both CentOS 7 and 8, for example, maybe also a recent Ubuntu version, whatever cluster we have access to, we can test on those operating systems. So containers are very good for controlling your test environments. And hopefully we can also do this on, we actually already do this, but not extensively enough, I think on different processor architectures of different generations of Intel, hopefully also AMD Rome, but also more exotic things like power and arm. We wanna get better test coverage on for the regression test. And the Easeable documentation. So even though most people are fairly happy with the documentation, and I didn't discuss that here, but you can see that in the raw results of the survey. Over time, people are fairly happy with what we have in the documentation. We know there are some missing parts that are still not covered yet. But what I wanna do is, I think one of the reasons we are not working enough on the documentation is because it's too cumbersome. So you have to write, make some changes in the plain text file which has a particular format, but you can't really preview the changes you're making in the rendered way locally, at least not easily. Even though there's a very good tool for doing that, MKDocs, which is what we used for the Easeable tutorial. That's really a dream to write documentation and you change markdown files locally. And as soon as you save the file, it re-renders the documentation locally on your laptop. You can do even offline work on the documentation. And I think we should move the current documentation into MKDocs, so in a different format, that's gonna be a pain. But once we make that jump, and maybe along the way, we can make some improvements to the documentation as well. But once we make that jump, I think it's gonna be a lot easier to make changes or make improvements to the existing documentation. So that's one of the things I also wanna work on in the coming year. Then challenges that I see in the coming year are somewhat, some of them are similar to previous ones. The top one certainly is. So the scientists and researchers seems to be always finding new tools to install. So the quarter increase in the number of supported software packages is certainly an example of that aspect. That's gonna be an ongoing challenge. I don't see an easy way of fixing that other than collaborating more and making sure you share easy config files if you have them, even if they may need a bit of cleanup before they can be included centrally in EasyBuild. Hopefully, I think getting more contributions or making people contribute more is the only way of dealing with this in a good way. A big issue I'm seeing, which is also slowly coming forward in the survey results as well as the increased diversity, first of all, in hardware. So we're coming out of an era where Intel wasking, maybe still isking, but it's slowly changing. So AMD Rome is already a headache in terms of which software, which libraries, which compiler you should use to install software. But there's also other platforms on the horizon like ARM, for example, which I think is gonna be, yeah, it's gonna see increased adoption and increased interest in the coming years. And there's other, power has never been away, but it's still a bit of a niche thing, but it's very relevant and certainly relevant enough to care about. And on the longer horizon, there's projects like the European Processor Initiative, the EPI project, which is looking at risk-5-based accelerators. So that's yet another platform that we may have to deal with soon in the coming years. And then somewhat related to that, there's also software, there's an explosion in different compilers. Intel 1API has both the classic Intel compilers, but also the new ones. There's NVHPC, which used to be PGI, is a relevant, very relevant compiler, AUCC and so on. So there's a lot of increased diversity and a lot of challenges coming there as well. So we need to find a good way of dealing with that in some way. And then as I showed, the number of contributions we're getting, we're very happy with those. Also the way in which they are being made. So through the new PR option also makes the life of maintainers easier because things are coming in in a structured way and they look similar, they're fairly easy to review if they were done that way. So if people want to contribute to EasyBuild, I strongly recommend using the GitHub integration features because it helps put yourself and the maintainers. But with the number of contributions, we're getting 2000 pull requests per year. Certainly in combination with the previous point with the increased diversity, I think we need to be a lot more aggressive in terms of automating things. So we already have a small bot running that's very helpful and we can probably make it smarter, make it more capable and give it access to additional hardware so it can fully automatically test things in pull requests after a human maintainer has at least looked at it and submitted a review that he's happy with the contribution before it's being tested. So there I think we need to do more and try to factor out the human as much as possible because that's where most of the delay is. But this dives a bit more into the increased diversity in hardware and software. So CPUs as I mentioned, GPUs as well. So NVIDIA was very popular, is still very popular but AMD GPUs and Intel GPUs are on the horizon. So that's gonna make the diversity in hardware only worse in the coming years. And tied to that, the rise of different compiler suites, different compilers, different libraries, especially for the AMD GPUs, the Rockham stack is which we currently don't support in these build yet but we'll probably need to start looking into that. Yeah, lots of challenges here in terms of diversity. Maybe we need to react to that a bit and the common tool chains we defined. So the FOSS and Intel ones we have currently maybe that's no longer enough. Maybe we need something that's Clang-based, so C-Lang-based in addition to FOSS and Intel or maybe as a replacement to one of those. So we can only make changes like that after revaluating things a bit, benchmarking and checking how many of the big applications actually work with these other compilers. And there's been a lot of discussion recently as well about merging the FOSS and FOSS CUDA tool chain. So we have the split, which doesn't make a lot of sense because the only thing where it really differs is that OpenMPI is built on top of CUDA and CUDA are rare and that's causing a split in the tool chains and lots of duplications and easy config files. So we are already actively discussing how we can merge these two together in a sensible way. And I'm hoping that's what we'll do in the 2021 A common version of the FOSS and FOSS CUDA tool chains, but that's to be worked on. And then related to the diversity at least in CPUs, certainly Intel and AMD. We've seen some reports from Ulich mainly recently that they have seen very good results with the Bliss Blast Library, which is also there's a presentation on the Bliss Library by the lead developer and Wednesday, I think, so later this week we'll learn a lot more about Bliss, but the results we've seen especially compared to Open Blast, but also compared to Intel and ML, very promising when there were a benchmark against each other on the AMD Rome CPUs. We need to look if we see that same pattern on Intel CPUs, especially compared to ML, wanna benchmark that more, but currently it's looking like we will probably switch away from Open Blast to Bliss and the FOSS tool chains, but that's also to be confirmed after a bit more of experimentation. That's what I had in terms of slides until this morning, where I got an email from somebody in the Lumi project. So I was involved in discussions or they invited me into a meeting shortly before the end of last year because they were actively looking into which software tool they would use to install software on the big Lumi supercomputer that's coming in Finland and which will be accessible to lots of European countries. It's a new praise tier zero pre-exascale system. So the two major contenders there were no surprise, EasyBuild and SPAC, so they were comparing those in-depth and also talking to both myself and to Todd Gamlin. And the final decision which I made last week was made public today, so they sent me an email that are going forward with EasyBuild for at least the primary build tool of Lumi. So SPAC will probably also be used, will be provided by their users, but most of the core installations that they provide on Lumi will be via EasyBuild, which is interesting because it's a very prestigious project, that's one reason, but also Lumi is fully AMD based, so it has a big AMD CPU partition, but also a very big AMD GPU partition. So yeah, we'll need to work on supporting the AMD ROCOM ecosystem. And hopefully we'll start seeing contributions from the Lumi support team towards that direction. So that was a very interesting news item that I was able to include last minute in the presentation. That wraps it up for me and now I think we still have time for some questions. Yes, thank you to Kenneth for the talk. If you have any questions, then please use the raise hand feature in the Zoom or raise a question on the Slack chat. Sorry, I think I was muted there. Apologies for that. If you have a question, then please raise your hand if you're on the Zoom call, if you're just watching through the YouTube and raise it on the Slack. Yeah, raise it on the Slack and we'll pass across to you. See the first question, so I will. Okay, so Kenneth, I thought that you announced that Lumi will use EasyBuild. What do you think about the support for Cray2Chains and Cray recipes that happen with EasyBuild? What do I think about it in general or? How do you think it's gonna work? That's a good question. You probably know that better than I do, since at CSCS you're the biggest user of the Cray support we have. But you guys have plans to go into Lumi and support it to change and modify it to change or do you have any plans for that at EasyBuild? That's mostly up to the Lumi people, I would say, how they wanna use EasyBuild. And I think they already know or they know their plans on how they will use EasyBuild since otherwise they wouldn't have made the decision to go with EasyBuild. But I know they took a very, very good look at what EasyBuild can do, what it can't do, what aspects they may not be happy with, like the fixed versions. They probably also took a very good look at what CSCS is doing on their Cray system with your own easy config files and how you're handling that on top of what Cray provides. So I think it's more of a question for the Lumi people themselves. I currently don't have a very good view on how they are planning to use EasyBuild in practice. The meeting I had with them was more about them asking specific questions like can EasyBuild do this or if this happens then how can we work around it or how can we fix it? Those EasyBuild provides features that will help us in this aspect or in that. And I just answered their questions. I didn't tell them what to do or anything like this. But it seems like they see enough value in what EasyBuild can currently do or maybe they have a good idea of what they would need to do to support the parts that are missing that they don't see it as a big hurdle to adopt it. Okay, thanks. Any other questions? I don't know if there's something in the Slack channel or maybe my presentation was too long or too clear. There's no more questions, I'm happy to wrap up. It's getting late also for me, but yeah, we'll certainly be around throughout this week in the different sessions in the user meeting, also in the Slack channel. So if you have any questions, don't hesitate to raise them there either in the breakout room or in Slack. And we're happy to answer them. Yeah, there's nothing else I guess we can wrap up here. So tomorrow the first session is the CVMFS tutorial at 9 a.m. UTC and then I would have to check the schedule myself, but you can find the detailed agenda on EasyBuild.io slash EUM. So we hope to welcome you back tomorrow. Thank you everybody.


Great to have you back on the breakfast on PLOS TV Africa. First major conversation for today takes us back to going deeper and trying to understand better with regards to what happened in Iqoyi, not very long ago, a few days ago actually. And of course, seeing the response from the Lagos state government, as of yesterday they were on record 44 people confirmed dead. We're speaking this morning with Isaac Idoko, his HSE manager for industry safe management company here in Lagos. Good morning Mr. Idoko, thanks for joining us. Hello, good morning. Thanks for having me. Great to have you on the program. The conversation really is to understand better how we could maybe have saved more lives at the Iqoyi disaster. We've spoken of course with other people who have you know you know expressed their opinions with regards you know the responsibility of the Lagos state government you know to those people who are victims of that disaster. But of course you know as Nigerians we understand how we normally act you know as long as the government has you know said that all is well and you know we go back to sleep but we've chosen to continue and keep the conversation going. So I want you to share your thoughts concerning the things that you saw if you watch if you saw the video clips from the section rescue operations the things that you saw that you maybe thought were inadequate or completely wrong. Thank you for this question. Before I continue I would like to say to send my condolences and on behalf of my company to the families of those that have lost their loved ones in the cause of this some tragic incidents media souls rest in peace. Also I would like to send my sympathy to those that have lost all forms of resources which humans and materials in the cause of this incident. So sincerely like you rightly said in the cause of this rescue mission like every other ones that have happened in the past who've seen a lot of lapses who've seen a lot of things that we are never meant to be. And as a nation in Nigeria I think one may be forced to conclude that we are a nation that is largely driven by emotions and one thing with emotion is that it is highly active but temporal. Okay very very short-lived. These are happened not once not twice not twice. You can recall that in 2014 we had incidents with the church guests. We have over 100 people died. In 2016 we had the building collapse in Leki where a lot of people died and even this year alone we've had a couple of 2020 to this year alone we've had a couple of other collapses in Korodoo and other places within Lagos and other parts of Nigeria. So what should we have done differently learning from the past and learning from history? First of all one of the things that I've discovered in the cause of this particular operation is lack of immediate response on the part of our emergency responders because when an incident of this magnitude happens rather you have a very very high chance of rescuing a lot of people alive when you venture into your duty immediately and I would like to even use the just one example of what happened in Miami recently. I think about four months ago we are a building collapsed in the Florida seaside the poolside. That incident happened by 130 a.m. around 130 to 2 a.m. and immediately emergency responders were on site and started work started rescue operations and all. So there were high chances of being able to rescue a lot of people alive because where these people are underneath them these brakes these rubbles and all these things they have limited access to oxygen they have limited access to air to breathe and also coupled with the danger of dust and even the cement itself because when you work in cement factories you're supposed to do a lot of dangerous that you don't even need to breathe with your the normal breath outside you're meant to put on a mask that filters some of these vapors outside so you see when this is not done immediately you now expose these people to more danger so you expose them you they can easily die and then they won't you're just being after a day to three days is just anyone who's got says your time has not come but on a normal and professional level following standard operating procedure immediately that happens because what when that's a coincidence happened I think for over four hours our emergency responders when we were to be found we learned they may be hold up on the road or no one knows even the few people that were rescued alive immediately if not for guys around and neighbors and maybe other people around I had to go in there try to get some people alive maybe we would have had more casualties than what we currently have okay so you know that there are different types of emergency now for the case of a building collapse as is with us in Lagos Nigeria I call you to be precise what type of equipment is required what kind of instruments do we need them to show up with them because you've talked about the fact that they need to show up but what kind of equipment did I need to show up with good thank you so much for this question now when there's a building collapse depends on the on the type of building and the environment but some of the equipment that you need to show with on site includes the excavators the cranes because when you're trying to make a rescue operation there are different types of operation we have the the shoring which has to do with building the temporary the temporal steps in order for you to have easy access to the particular place then we have the lifting and moving we are in the case of when you look at the mode of the building the equi building you see that they are more or less the blocks and the other concrete are stuck together so in this kind of case when you want to make quick rescue you need things like the cranes that can easily be used in lifting and moving also we need equipment for breaking and then breaking and breaching so when this happens some of these equipments are the things that should go on site but in the case of the co-e what we saw was only these excavators and that further aggravates the issues because in fact as a matter of fact when you look at what happened in the co-e you discover that most of these operations is more retrieval and recovery mission rather than rescue because the excavators all it does is to just go in with the with the rent and then try to pull down the the rubbles the concrete and all and in the process in fact we have some that we have some recorded cases that the excavators disembarked some of the people some of the victims so what that means is the person may be allied but at the collision with the excavators realize that the person would no longer have any chance of surviving so some of these equipment because the kind of building the kind of environment and the kind of operation you would like to carry depend that would determine the type of equipment that you go on in with but it's not at what time that you would only use the excavators because like that it co-e is done there are high chances that if you do lifting and moving using the cranes and some of these heavy machines it should have been easier to rescue some of these victims do you think that we could have worked faster you know with regard to co-e it took about a week you know maybe more than a week before they eventually then declared that section rescue was over do you think we could have worked faster if we if we were serious about actual section rescue not recovery like you've mentioned could we have done absolutely yes absolutely yes in fact one of the issues that happen how that keeps happening is the issue of employees competence all right we keep saying that when you leave your employees without having proper and requisite skills and and the knowledge about certain operations when it plays out maybe your family may actually be on all friends close friends may be the victim of your inefficiencies we should have responded faster and also even at that at the time we responded a lot of things happened a lot of negative things happened part of which was one you know the the guys that came on site I don't think if they have shift I don't think if they have batches but we learned at the point they had to stop work for almost seven hours this is because these guys also that are working they are not provided with your rights PPE is personal protective equipment as a matter of fact I am a few professional and I look at and I looked at some of the others they were putting on when you look at them I don't think some of them are made of high density polyethylene that can withstand for your test and when this happens when employees are giving inferior items to take to sites to work knowing fully well that that may not give them maximum protection what happens is their confidence level will be low and their marginal productivity to level below because even in as much as we are trying to rescue other people you don't also want to be a victim so we should have acted very very fast and also the the the emergency responders led by an incident commander they are meant to have shifts because you can be on rescue mission and then you leave work for us saying that you're going on break or whatever that happens as people are going on break other people are meant to venture into work into the rescue mission as others are living more people are coming so we should have acted faster and better to save more lives in this case all right so another case or another issue or that was also raised is the fact that the issue of manpower the fact that they were on the staff and we could see members relative or family members suddenly sitting that they should be recruited in the entire search and rescue and you know those who were around do you think you know that should have been done being that they don't have the necessary training absolutely yes I do agree with that because you know when you have a loved one there you would even want to jump in and try to save their lives and truly they are understaffed like I said that's why I did mention that these guys are meant to be on shifts and they are meant to be in batches so if it's shift one you can have bash one two three all right so that once you work for setting us because as a matter of fact apart from like I did mention about the equipment that they're supposed to if you're working in that place as this excavators keep working and bringing out these doors to the robbers and the conquerors these guys tend to inhale doors and all manner of things and the mask I saw them with basically are just the the laboratory mask the normal COVID-19 mask that every other person are putting on this cannot protect you against this danger this is very present this does that comes out of that place what also does is it's it makes the the staff on duty the the emergency responders on duty to easily wear out while working so they are understaffed which is the best which is a major problem and not just that the ones that are on ground do not have the required equipment and PPE materials to work with in order for them to put in their maximum productivity so yes I think our emergency system needs to do more and not just recruiting anybody they need those that have passion for rescue they need those that have passion to devote themselves to this because all we see around now is um when you know someone they're not someone when there's um a vacancy you get filled in but when the time to perform comes because most of our institutions really that is what happens and that is why whenever there's an incident whenever there's an issue whenever there's a problem you see the the real competent competent people that are meant to do the right job are not to be found but you mentioned giving the little but you mentioned that before the emergency I mean the legal state emergency management actually arrived it was true the help of those who are around neighbors that were able to I mean because of their proactiveness that's what we were able to rescue the persons that we were able to rescue alive so I mean how do we now explain the fact that yes people need to get you know the right training at that point in time let's not forget that it's an emergency and then you begin to look for who's got the right training yes I will you know one thing in HSA is you cannot try to solve one problem and create the other because when you look at the um the hierarchy right of safety when you when you are in any environment the first thing you need to do is to eliminate hazard all right to eliminate hazard then if total elimination is not possible you have to substitute the hazard with a lesser one if that is not also possible you have to create an engineering control if after engineering control you go to administrative control and then you come to the PPEs now the guys on on duty and the neighbors and the likes they were able to rescue few people that is what they've been able to do but there's in the process of rescue you know they may be using their hands to pull down some of these concrete and the rest thereby there may be other victims in there where these um uh concrete and the likes are being pushed to that can affect them and lead to their death as well so that is why um trained professionals are actually required but in this case that we know that yes it's an emergency neighbors um families um you know emotions are high everyone tries to get in there but what they should have done if um I were to advise it basically is since they are just um stopped there are some basic duties that could be given to maybe site engineers, bricklayers and some of the people on ground they can assist why the trained personnel do the main job that will help them to fasten and quicken the rescue process I had never I actually even forgot about the the um you know proper gear and proper equipment for rescue because of the environment it's filled with dust and and cement and and all of that it makes it difficult to breathe and so it must be even harder you know to to do the job when you are dealing with breathing difficulties at the same time um which we obviously didn't you know think of taking care of because they don't have enough you know equipment and they're not properly um uh uh dressed for such and rescue kitted for such and rescue some other thing I want to I want to ask about um in movies and from what we've seen you know elsewhere we hear about the fire department when there's a disaster like this you know they are first responders the fire department they rush there they have their own search and rescue team or you know you know branch rather of the fire department um and we see these things play out very very you know very interesting we don't have that here what what should you would you expect from a local government like etiosa um you know who should it really be just last summer dealing with a situation like this or showed etiosa itself etiosa local government area have its own makeshift fire department or what not that has all the equipment that you've mentioned that should have its cranes that should have its air ambulances that should have its its you know um whoever it is that is necessary and the equipment that are necessary for a situation like this is that what you should expect absolutely yes in fact I was going to get to that because um just like I'm still trying to use the Miami incident as an example because that was what happened recently and when you look at the building um the structure somehow look alike all right and then the way it happened and the the collapse also have similar model um when you want to when there's a collapse building and you want to make a rescue only the emergency in the like such as the last summer now cannot do it in fact they are not meant to do it now in safety and in HSC as a matter of fact dust is very very flammable dust that you see is very very flammable dust bonds so when there's such emergency fire fighters must be on ground because in the process of trying to rescue you don't know if there are for example this is um just um this building is just still going up there have not been electrical um um installations here and there yet I don't think there was so if not one once this kind of thing happens and um there is a spark from somewhere there is this and that there are some dust that bonds so what that means is once that happens you on the surface you may not be able to see it because um of the rubles and the concretes and the rest that must have covered it but those underneath these rubles get to inherit um this um smoke these things that comes out of the uh cables and but when firefighters are around because when they come they don't just come with ordinary um phase just as we have they have a lot of they have a lot of gears that they put on they have they have face masks they have the full face mask um with their air cylinder um some of them go to these kind of places with um scba that self-contained breathing apparatus so once they are there they have equipment that will be able to also have smoke detectors they have equipment that they deploy to this site and they'll be able to know if there is a leakage somewhere there's a smoke there's a fire underneath anywhere so they don't work in isolation when when you look at the the the US incidents you see um you see the police the military men they see the uh the firefighters you see the red cross you see so all these guys work together in emergency rescue only the emergency responders they don't work in isolation it's it's meant to be um a collaboration as a matter of fact I read I think on vanguard that the military men the Nigerian the federal government mobilized some military engineers to sites so that they can assist the legal state government um in trying to facilitate and fasten this rescue operation but it was declined so you see yeah and it's not meant to be so this is meant to be a collaboration because only you cannot do it engineering it's all about experience and is that what I'm another thing I'm trying to you know get you to speak on is you know what would expect from um not just legal state now but at the local government level would you expect that the the Etiusa local government should have these things readily available um so we don't have to wait for um rescue equipment to come from Mushi because that's what like the last mdj said to me uh that you know when disaster happened you know their staff got there you know after you know a few minutes I guess but the equipment couldn't get there uh till a couple of hours later because of traffic and the equipment were coming from Mushi so would you expect that Etiusa should have these things readily available that you have their own cranes because you mentioned cranes I don't think there were any cranes there they were just excavators and excavators are mostly used for demolition if you looked at all that demolition videos is mostly as excavators that you see um so should Etiusa have its own department its own if you want to call it a fire department have it a fire department that has a crane that has any of these things readily available absolutely yes you know um it's not just meant to be um these things are not just meant to be provided at a local government level as in in developed um world up to world level they need to have their own emergency plan and emergency team because for example something happened maybe the local government sectoria to aware this equipment are stationed happens to be in Ikoi and something happened in maybe um Ajah or Shangotedo or any other place you realize that for these equipments to get there a lot of damages would have been done so at the local government level we are meant to have trained personnel that will be able to immediately respond to emergencies why more hands are being mobilized also these equipments are supposed to be provided for each of the local government you are absolutely correct on that because you don't you can't be waiting for an equipment from motion when there's an emergency in Ajah or when there's an emergency in April or when there's an emergency elsewhere so they are meant to be provided for each local government and then each of this team need to be trained on the use of these equipments because like I said even if provided only the local government emergency responders may not be able to uh you know to um to savage the situation but their immediate intervention would have saved a lot of things and a lot of life and even the equipment provided there were situations even in this in this Ikoi incident where incidents where um the equipment broke down in many cases running out of fuel a lot of things that are not meant to be actually Nigeria is a big country so a lot of things that are not meant to be anyways um there's also I mean for me as a person I I guess would have been so carried away by the things we see in the movies and then the things we see in other country hoping to see that there would be some securing of the environment with the tapes and masking and that's what you see when there's a crime scene or you know there's an emergency so securing it having those max tapes and all of that so people do not trespass but mostly I don't think if that was really the case here but another thing that was also common as a practice was from time to time those releases of oxygen to the particular you know environment where this thing happened the release of oxygen and the release of water as well do you think that this is also a safety and a healthy practice for search and rescue and even dogs you may want to you may even want to talk about dogs and you know rescue rescue dogs I don't think we have any of that here yes absolutely correct first of all you did talk about the um the safety tips the caution tips when an emergency happens in order for you not to allow um untrained personnel or untrained people to just rot into the site and try to engage in self-help what you need to do is we call it in HS we call it cordoning you need to cordon the place all right you need to use caution tape to cordon off the place and then allow only permitted untrained personnel to have access to the particular place so when this happens you know that you're already um in charge of this region as a matter of fact this particular cordoning of the of the environment must be left so until full investigation is done and completed even after rescue operation so that is that also but in our case you know like I said when you can't give what you don't have most of these personnel do not either they do not have this training or they do not have this equipment available also you talked about um pumping of oxygen and water to the sink absolutely correct is a safety practice isn't being that as I said earlier when I when I began when this kind of emergency happens the person is more or less like in a confined space all right and in a confined space when you are in that kind of place you have limited access to oxygen you have limited access to earth you have limited access to everything that makes you to breathe properly so when you pump in oxygen because there are cracks because there are holes they find their way into uh into underneats if not completely but to a very large extent and once that happens the strength of these victims that are struggling to survive get reinvigorated and then they begin to have their energy little by little so once you do this and as you do this the um the rescue operation continues immediately while you try to call and wait for response so you must at all times have these things in place because they help when you are there when you are underneath that particular place where you have limited access to um to oxygen what that means is that the hemoglobin of the red blood cells that um that passes this oxygen through your body and then gives you access to breathe and and live fine will be denied this oxygen and then that will lead to suffocation and eventually you lose your breath so it is actually a very good practice um trying to it doesn't save the whole situation but it helps to in um in certain rescue operations and also like you said the dogs the snapping dogs where you try to use it yes these are things the miami situations you see dogs here and there everywhere trying to um uh to locate if they die if there's human if there's no human and all so but in our case it has always been the same story I did give an example of 2016 even before i mean 2014 even before then there were collapses here and there I gave an example of um 2016 and even last year and this year these things have become reoccurring the exima and like I also did say in the um at the beginning we are a nation largely driven by emotions but emotions are um active but short-lived so when things happen you see everyone cry everyone barks everyone dies but after it we who follows it up to ensure that a policy is being put in place a policy um is being designed to ensure that these equipments are provided at local government level these personnel are well trained because emergency as an emergency responder you shouldn't just be an ordinary person you should be a field professional that understand the risks for example when as an emergency responder if yours is some for building collapse because you have building collapse you have a bomb terrorism and all that you have all manner of you have earthquakes and all that be based on your department or if you are even for um all of these rescues you are meant to be trained for example as this building collapsed what are the dangers what are the risks that the cement the blocks and the environment poses the risks determine the kind of equipment that you go in their way so and as you get there you're already equipped you're already kitted you'll be able to perform maximally but when you don't have this training or you don't you don't have this equipment your marginal productivity will be low because your confidence level is low as you are working you'll be telling yourself remember your family is waiting for you at home when you breathe in this in excess you may collapse here or when you get home you may collapse so because these are actually very very hazardous dust and stores that are exposed to in a sane society what levels of incompetence and failure would you point out that the families of these you know victims can sue the legal state government or sue la sema uh for um can you share with us you know the the different levels of failure that you've noticed that should be enough for a class action lawsuit okay um like i said earlier we have late time of response when something happened by 130 a.m where people are sleeping they were um immediate response to the rescue to ensure that okay they've moved these guys out of that place and all that is one two another failure another um failure is that of um failure of equipment not being able to provide the right equipment when they need um a rose for these guys to be rescued that is part of it and then also the the excavators the excavators that um that were used or that was used you know you don't just use that for the for for the few bodies that were dismembered in the cause of trying to rescue you know it wasn't meant to be so you can't just what's happened was basically more of retrieval and recovery rather than rescue because even the few ones remember the few ones that were rescued alive and the ones that when the incident happened immediately and then maybe a few as we but afterwards what we saw was just horrible so the excavators dismembering people and then all that it wasn't meant to be so this also boils down to um the failure of the government to provide the requisite training to respond at the time because security um securing lives and properties of every citizen is a sole responsibility of the government therefore they can be they can be i'm not a lawyer but um speaking from um safety perspective it is a wrong practice because for every incident that you see for every incident that you see they are caused by either an unsafe act or an unsafe condition so government not provide not responding immediately by trying to come when um when the uh this thing happened within a few minutes to uh maybe at most an hour an hour is even too much for rescue operation to commence whereas we heard that this took over for hours before they arrived and all so this is the failure of the government because when there's an unsafe act or an unsafe condition you can be held liable for it so the government by not responding immediately actually the government by not responding immediately created an unsafe condition for these guys for many of these guys not to survive all right Isaac Hidoko HSE manager at uh industry safe management company Limited here in Lagos thank you very much for joining us this morning we of course uh we'll keep the conversation going and hope that there is some clarity as to uh what happens next thank you uh for your time once again good morning all right thank you for having me absolutely uh stay with us here on the breakfast we'll take a short break when we come back we're going to be moving into talk in politics now direct or indirect primaries that's what comes up next