Hi everybody, my name is Frédéric Wotled and I'm a system engineer and I'm in charge of the management of the HPC infrastructure at the University of Namur and I'm very pleased to show you a quick overview of what we are doing with easy builds at CECI. First of all, what is CECI? CECI stands for the Consortium de Vecimaux de Calculation Attentif. It's a consortium of high-performance computing centers of the five universities in the French part of Belgium, so in Mons, in the University of Namur, Uliège and ULB. And the CECI is founded by the scientific research and the well-known region. And all the facilities of the Consortium de Vecimaux de Calculation Attentif are accessible by all the researchers of the member universities. So, there is about 450 active users on CECI and you can see here a big quick overview of the field or application that runs on the CECI clusters. There is five clusters, each one in each institution and the CECI cluster is in fact designed to accommodate a large diversity of workload and needs for the researcher. And for example, the two Hercules and Dragon 1 has been designed to cope with sequential jobs and the three other ones are designed to run parallel jobs. And currently the CECI infrastructure is being upgraded. The metro tree has been upgraded last year and this year we will upgrade Hercules and Dragon. You will see that for Namur, we will install AMD EPIC processor. And in Mons, they will install Skylake as well as several nodes with NVIDIA Tesla V100. The CECI also provides a common storage that is visible for all the frontend and the compute nodes on all the CECI clusters. In fact, each cluster has its own gateway to access this distributed storage and the technology behind this storage is based on GPFS, renamed Spectrum Storage by IBM. The main purpose of this storage is to allow the user to transfer data between the cluster but also to store all the scientific application, part of the scientific application we deploy at the CECI. In terms of the build of the scientific application, there are several objectives. The first one is to allow the profitability of the science by allowing the user to reuse older versions of the software. Also we aim to provide optimized versions of all the software we install. We try to version all the HDPs as well as the infrastructure code we use. We want to also to install all the scientific application in an automatic way. Also we would like to have a system that is flexible enough to cope with new hardware, new node architecture and also to cope with the new needs of the user. There are different challenges. The first one is that the CECI is the multi-site consortium with five HPC teams with an heterogeneous actuar with several architecture and each team has its own workflow, its own hardware, etc. Also we plan to set up an instant federation that allows the user to submit a job from one front end to any compute nodes on the CECI. And also we have a bunch of commercial applications such as MATLAB, GOSHA and VASP and we must take account of the license, limitation and restriction. For the design choice of the setup, the first one is that all the software installed at CECI is optimized for the active texture of the nodes. We use mainly, we use compilation firms to compile other applications or the compilation is done directly on the compute nodes using SLAM dash dash jobs. So we generate packages using FPM. These packages can also be installed on the distributed storage. And the use of packages allows us to uninstall cleanly all applications without breaking something and all the packages generated are archived. So we can do great application if needed. Also, as I already said, a great part of the application are installed on the distributed storage. And of course we use EasyBuilt to install almost all the applications at CECI. In terms of the directory layout, you can see here an example of the directory layout currently on the distributed storage. The first thing to describe is that because we have a commercial application with restrictive licenses, we need to install application locally on each cluster to avoid problems with the legal teams or something like that. And the rules here are freely available applications, mainly open source software uninstalled on the distributed storage with the exception of the Intel Java and Poplan compiler. And all the restricted software, all the commercial software or software that needs registration are installed locally and are only available locally on each site. In terms of dependency, application installed locally can depend on applications installed on the distributed storage, but not the other way. Let's take a look at the first level here. We can go closer. There is a naming convention here. The first part identifies the distribution with the version and in fact is based on the uncivil fact as the uncivil is the tool we use to deploy automatically the nodes but also the compilation farms as well. So for example, you can have a Red Hat 7, maybe a 9 or something like that. The middle part is in fact the identification of the processor in terms of processor family, model and stepping. And the last one is the interconnect. So for example, currently on the common storage, the following architecture are installed, and the support will only pass in a phenomenon, etc., none is for non, it's for gigabit connected cluster. There is a special case, it's the no-arch, it's for architecture independent software. Typically at CC, it's mainly to store databases such like NCBI blast or sweet spot. Again, the second level, we use the installation framework. You see that currently we use only two installation frameworks, the easy build and manual installation. Manual installation is used for applications that we don't have yet recipes to install it or for example, to install the blast databases, etc. And we support easy build but the organization, the territory layout is set up to accommodate another installation framework if needed. The level three is based on easy build release. So in each sub-pre here, you have all the modules of software, compilated with the same version of the toolchain. And so for example here, you have all the applications compiled with 2017 B, first and Intel toolchain. There is a default release set CC wide. So all the clusters have the same default release. This default release is updated once a year. And each release here are independent of the other. The user always has access to the other release and can test future release. And there is a special case for what we call TIS modules, it's for toolchain independent software. It's mainly designed for applications that need a specific version of a compiler. For example, Gaussian is only compiled with a specific version of the Portland compiler. Also the case for software bundles has binaries, such like Matlab, Mathematica, Crystal, etc. Then we upgrade this module every three or four years. As the easy build release, the older release are always available and the future release can always be used as better. So it is the view from the storage on the nodes. Every month, no arch and depending on the architecture of the node, we month the correct subtree. This means that on every node, the path to a certain application is the same, but it's point to an optimized version of the cluster, of the other applications. So it's allowing us to, if it is possible, to run parallel application on different nodes with different architecture with the same path. You know that several users love to out code paths in their submit script. So in this case, it's also always the same path on every nodes for the application. As far the build procedure is concerned, there is no common workflow between all the CC sites for the moment. For example, on one site, the build is directly done on the HPC nodes using easy build dash dash jobs, or it is the case on Unamu. We set up a compilation form with virtual machine. Each virtual machine emulates all the architecture present on the HPC cluster. At Unamu, we have four different architecture, two in production, one for the test cluster and one for the cluster for the students. And so all the completion is done in parallel and on each machine that emulates the correct architecture. Then we generate R-Pen packages that R-Pen packages are archived, then installed on the common storage or the local storage depending on the license. Then we keep all the R-Pen. If we need to upgrade one software, we increase the version of the R-Pen. So we keep all the version of the build in case we have to downgrade the software. So at Unamu, we have one build node, a virtual machine per architecture and one install node per architecture. For the user point of view, we plan to set up MOT7 for all the CC clusters. We currently, we only support a minimal number of toolchain, namely FOSS Intel, as well as the Portland compiler, we need to compile a solution. And we use the default easy build mononaming scheme. There is no plan to use another mononaming scheme for the moment. And all the users have access to MetaModule, I will show you later how to use it. So the principle is to avoid the module available to list a big list, a big pile of modules. So when we run a module available on the cluster, you only see MetaModules, the TIS modules and the packages built with the default release. If the user want to have access to an older release or a future release, you have to run the one MetaModules corresponding to the release you want. And then the default release is unloaded and the module available only show the older release in this case. So the user can change between release and ModuSpider always shows all the modules available in every release and every version and release of this. So the user know which MetaModule need to load to access specific version of the application. It's quite important to allow the user to use the older version of the application and normally the older version are free. So we don't change anything so the user can redo his simulation. In terms of the future works, we plan to set up a generic build based on a generic architecture. So it's just a way to provide non-optimized build. For example, if we set up new nodes with new architecture, so when the optimized build are not yet available, and it is also useful to provide to the user packages to be installed on the local workstation at Udamur, the user want to have the same software installed on the workstation than on the HPC cluster. So for future release, we plan to enable the Airpath linking and the work is almost done. We plan to automatically update documentation after every build. So the user, we have on the web page all the tables that tell him on which cluster you can find which version of an application. So as a conclusion, so at CC, we provide optimized build for all the architecture on the system. We allow the user to load the older version. We give access to future release. The user, for this case, the user just have to load a meta module and we support special case for application that only depends of specific to chain. Thank you. Any questions? Thanks for your talk. Could you put back your slide on with the module available screenshot? I'm just wondering, because you use the flat naming scheme, but it seems you work on software hierarchy. No, no, no, it's only based on release. So each release is independent and when you use, for example here, module load release 2006B, you switch the release, it's not hierarchy, just based on the release. It actually is a hierarchy, you just don't know. So you only have one level in the hierarchy, which is the tool chain. And when you're loading a different release, what Elmot sees is a change in the module path. It knows it has to reload all the modules or it will change the modules later. It actually is a hierarchy, but it's probably not managed through EZ-DOTE. No, no, no, not at all, yeah, same as they have a new liquid build stages, yeah, exactly. More questions? Very nice talk and very impressive and I have two questions, actually, brief ones. One is why do you keep your license restricted software on different file systems? Why don't you protect them just with ACLs, for instance? Yes. There is some ACL that, yes, we use Unix permission to block the user for using licenses restricted software. And these ACLs are per site, there is no global ACL, so each HPC team has its own ACL to allow user to use commercial application on each site. For example, at TUNAMU we have MATLAB installation and MATLAB is only available for the local user. I was wondering because on our side we had the idea that we have a module which is available for every user. Every user can load the module but can't execute the respective software because then the module just shows an information up on loading that the software is restricted and in the case of, let's say, an MD, a user has to print out their license, send it to us and then we enable it and it's just less work. That's why I'm asking. Also, the second question, you compile on VMs and I'm wondering that's kind of dangerous because then you have to maintain two software trunks locally on your actual hardware and it always has to be in sync with the respective VMs and whether that is actually necessary because in this case we compile with the shared jobs on our machine and if it fails we know if it's pending due to the missing system library then perhaps the node wasn't installed correctly or we made a mistake and then we see it immediately. So the VMs used in the build farm are the exact mirror in terms of installation that the prediction nodes. Because we use uncivil and packages, if we can rebuild nodes, I see a buyer, just reinstalling all the RPM, if we break something we can go back and reinstall from a specific check point here. Okay, thanks. Thank you again. Okay, so hello everybody. My name is Luca Marcela and I'm talking to present the CSCS site updates. CSCS is the Swiss National Supercomputing Center located in Lugano in Switzerland. And the talk is based on the information provided by my colleagues Teofilos and Guillem as well for the scientific computing support and computer and data services support groups at CSCS. So we will talk about the usage of this build at CSCS, the timeline of this usage and the use cases that we have on two production systems that are called Cache and Escher. And they are used by the National Meteor Service in Switzerland, Meteor Swiss. And then the main production system, the usage of this build on the main production system which is called pitstain and it's a Cray XC system. Then we will talk about the usage of Jenkins in order to deploy the CSCS software stack using poor request integrations in GitHub and the use of pipelines with Jenkins. And then the final remarks with some future plans in order to adapt EasyBuild in our use cases. So this is the timeline of the usage of EasyBuild at CSCS. It started in 2015. Then with the help of Kenneth and Peta it went to providing Cray software stacks as well, Cray tool chains as well on the system in collaboration with Guillerme. And over the years it increased and now it is fully deployed after the end of 2016 when we had a major upgrade we deployed all the software stack using EasyBuild. So with few exceptions in some systems like the Meteor Swiss system for software that is somehow quite specific and still we are working on deploying that within EasyBuild. There is also the need to convince some users about the usage of EasyBuild once you actually deploy some recipes. And in our repository the number of poor requests for new recipes is now over a thousand. So it is widely used also by our users. Now the HPC systems that we have at CSCS are listed here. So on the top you have PitStank which is the main production system which consists on two types of architectures. One is a Haswell processor known with the Pascal GPU. The second type which is called Cray XC50. The second architecture called XC40, Cray XC40 consists of two sockets with broadband processors. Other systems are listed here. I mentioned already the Meteor Swiss systems, Escher and Kesher which are a Cray CS storm system with multiple GPU on a single node. And then we have here listed two let's call them general purpose system Leone which is a large memory system dedicated to a company. And Fulen which is a virtual system built with OpenStack that is based on an Intel breadwell, Broadwell and Skylake architectures for Skylake for FET nodes that come also with GPU nodes as well. And for the moment I think that Fulen is our test bed I'd say for using generic tool chains because in the Cray system we were limited to actually using the Cray tool chains most of the times. We played a bit on the Meteor Swiss system with the GMVolf tool chain but now it's Fulen is the one using the standard Intel and GMVolf tool chain. We use GMVolf because for our MPI test Mvapage perform better with respect to OpenMPI. So that's why we were speaking to that. This is the architecture description of the Meteor Swiss system as I said it's a Cray CS storm system which has two sockets with 12 cores processor and 8 NVIDIA Tesla GPUs. Now the easy build software stack was deployed initially by Guillermo in production in 2015 and initially we were forced to build also basic things like the GCC compiles because the initially provided compiler by Cray was not employed correctly the vectorization as it was supposed to. Pit stain the main production system is based on Intel Haswell processor and Pascal GPU on the XC50 nodes and the XC40 compute nodes as I mentioned before has problem processors. At the moment we are improving a bit so the number might change a little bit in terms of total scratch capacity and for the rest it's more or less the same. So pit stain which as I mentioned was it underwent a major upgrade at the end of 2016 after which we fully deployed a software stack with VZ build. It's now number 5 in the top 500 so it's filled the fastest in Europe and number 18 in the green 500 and this is according to the November 2018 list after the supercomputing and it mentions the Cray XC50 partition it refers to the Cray XC50 partition only. Now according to the in terms of software that we release on the main production system we have on the left software that is based on recipes already available on the EasyBuild repository that you can find on the EasyBuilders GitHub of Uganda. The Python recipe here it's actually based on Cray Python so it's not exactly the stock EasyBuild recipe we do that because Cray already provides two Python modules and Cray separates Python 2 and Python 3 so we also follow this separation so we will still continue separating Python 2 and Python 3 and this is just to avoid simply duplicating things that are already present like NumPy, SciPy are already present in the basic installation provided so we will stick to that. On other systems we use the standard EasyBuilder way of building the Orbit in Python. And Wolf, CP2K and Gromax Boost, GSL well CP2K actually and Gromax I think for Gromax we use a custom EasyBlock that was provided by Victor I don't know if it has been pushed back okay yeah for CP2K yeah I remember I had some discussion with Kenneth about CP2K as well I think we still use a custom EasyConfig just because it is a bit more flexible in terms sometimes we need some modifications requested by users because users are not satisfied with the single installation they always want to modify that a bit and in order to let them modify that because we have custom EasyConfig modified by users it's easier instead than modifying each time the EasyBlock just to provide them some modified EasyConfig because each of them CP2K the software that it's currently always updated from the SVM and so they still wanted to rebuild and sometimes they really need some custom features so it's really difficult to stick to a standard version and in terms of usage here you see on the on the right side the Excel statistics on the system actually CP2K and Gromax are widely used some of the most used software at CSES user code means that we actually cannot identify that so it's something that maybe it's compiled directly by the user it's some known code maybe but doesn't stick into our name frame of executable or module frame so we cannot identify that immediately and it needs some further inspection we also have software like Quantum Express or VASP that are already built using EasyBuild as well ANS is a material science software I don't think we have a recipe for that usually these users tend to build their own version anyway also users of Quantum Express or VASP CP2K as I mentioned they want to rebuild anyway so we provide a module we have the we use a standard TCL module framework but they generally tend to use their own their own build so what we provide for them is anyway a system in order to modify the recipes that three actually build for them so user when they ask something new about modification of an existing software that we provide or some new software we provide them with a modified easy config file so we give them access to our GitHub repository we point them to our local easyBuild installation and to the documentation on our user portal where we actually mention how to compile the code in general on our systems and in particular how to build within the EasyBuild framework so the local EasyBuild framework that uses a module in order to load the environment of EasyBuild and then users can use their own recipe or our recipes in order to rebuild the software that we support and this has been working so far more than a more than a year so I think it's pretty stable in terms of our software deployment that is based on Jenkins we are I'm now going to describe a bit how this is made on our GitHub repository because we also use that in order to deploy the standard software stack plus testing new recipes so Jenkins is the basic tool that we use as a continuous integration tool for testing new recipes and to deploy the software stack of the packages that we maintain and support with our easy config and easy block files in the EasyBuild framework so we have then three different main projects the testing project to test new to test the poor request the software coming the recipes coming with a poor request to our repository the production project we which continuously build the software according to the software list that we support and then a regression project on Jenkins that is meant to totally test from scratch every week the build of our software stack in order to to check for instance download links and compatibility with updates and modifications on the system recently we have moved on pipelines which means a high flexibility in order to perform in parallel the activities within the Jenkins tool and each project has a separate Jenkins file which contains the configuration of the project and its version control so every modification is tracked and then you can easily write the continuous integration tool in parallel in order to ex ex better exploit the resources that we have in order to submit a poor request to our production repository these are the rules that we advertise on the GitHub page so it's something that's a common sense plus some slight customization due to our architectures GPU non-GPU and the names of the systems I think not so many users are submitting poor request but I'm pretty sure that many use at least I know that many users are actually using their own custom files so I think it's more or less the same as it happens for easy building general so user tend to modify the easy config file but sometimes due to time restraints maybe they do not contribute back and so this is an example showing how it how it works let's say for instance here then GPU means the architecture with the actual GPU accelerator you are submitting a poor request for cafe to which is a data science software so a new easy config file that then after the poor request is submitted gets to the CI and goes to the dine GPU machine so the architecture lab that's dine GPU in the and you see on the right side how the the blue ocean interface of Jenkins is dealing with that in pipeline so in case you had multiple entries here like the GPU or DOM and see like multicore for another system name then you would see multiple multiple dots on the right side like in this case so yeah that only means all the possible architecture on this system and then you see that two different architectures of the two dots the two green dots that you see over there and for the regression test that I mentioned so that weekly tests all the recipes that we have in our software stack in order to ensure backward compatibility with modification that happened over the time in the systems you can see that all the systems are built and here it was a very lucky case all green anyway it's it's quite let's say it's quite useful to have to have this interface quite flexible the blue ocean interface and the pipeline on top of that now to conclude with the final remarks we have been testing lately to build easy build recipes within Docker containers this is the reason for that is to have reproducible builds of legacy software stacks so outdated software maybe that cannot build no more on recent updates of parts of operating systems and then it's also due to resistance to change from from users so they are used to a certain software that maybe builds only on I've no redact six and that's the reason to use a container successful attempt so far so sent OS 7.x redact with the cave that you need to subscribe to use the you package manager because otherwise you need to download yourself all the packages and it's a bit annoying because of course the containers in general do not come with what you use generally if you install yourself so I try with Ubuntu and it's much easier although it's less useful because we don't have that introduction but just for testing and the create programming environment as well so the MS group they tested that as well producing but that can only be used on a trade system with the license for software so require steps as I mentioned you need to manage yourself of course all the dependencies in order to fully start with easy build like Python or mod compilers like new Intel and then you need to create users and to deploy the docker image to the users in order to let them access the image that you have produced that is still work in progress but it's probably one of the major things that we would have to do in the near future then moving to easy build so officially at an HPC site takes some time as you also have mentioned before can it so learning curve and resistance to change in this case it's not resistance of users but resistance of the management which is even stronger so and then easy conflicts versus easy blocks so the advantage of easy blocks they are reusable so it's great for stable software but there's a hover head if you have let's say bleeding edge software something new and needing some let's say work around in order to be installed the reproducibility then if you how to keep track of changes if probably if you need to hack the installation then it's a bit difficult because you suppose that you need to upload it say a pull request every time at a certain point there is a trade-off between having a local easy build as you said if you have more than five provides too many but sometimes it's really difficult in order to modify continuously and be accepted in a pull request on the other hand easy conflicts of course let's say they provide the advantage of being self-contained so these are three pluses but on the other end of course it duplicates a lot of things because of course you need to copy and paste all the time within but you can modify that easily then wishlist I think there is already an ongoing discussion as far as I know between Alam and Victor about a new command line option for allowing a try dependency version including so dependency as well to be updated but with version still work in progress and then in Guillermo mentioned to have a extended run with some more information in terms of logs command line so each command line maybe separate separate log for each command line then backup of custom easy blocks to ensure that builds are reproducible that what do you mean with the second thing exactly what have a separate log file for each command that runs in the extended run so it is actually when you run a build actually it would be nice to have a separate log only with the commands that you run so that you can easily copy and paste and because right now in the logs you have output you have what easy build does and the output makes so you need to go and pick the things that you need and on the extended run it's not exactly what the easy build runs like you cannot copy and paste 100% sure so if you had a log with only the actual commands that easy build issue during the build someone could just take that copy and paste and run on the terminal for debugging or reusing the recipe this has this has come up before so people want to have I think that's what you mean have easy build dump a shell script that basically if we just run the shell script does what easy build does that it's possible but it's not very trivial so there's more to just running commands easy build is changing directories applying patch files most of that stuff is not done with shell commands but it's quite in stuff so you would have to have yeah actually there is already a wrapper for changing directories there's dedicated functions you can make that spit out the corresponding CD command but to get it 100% right it's probably very difficult but make it work and 90% of the cases is probably fairly doable and what dry run or what extended dry run already provides now is you can copy paste all the commands it does to set up the environment that's done in such a syntax you can just select it copy paste drop it in a shell and it will drop you in the environment the same environment that easy build does but for when it works yeah that's a special case yeah it may be more complex there because there's module swap stuff that happens there a bit of magic maybe open an issue on this with some more details and we know exactly what you want to comment on on the next point there's a backup of custom easy blocks that's implemented now and so basically there's an in an installation directory in this reprot directory there's the easy configs but also the easy blocks and patch files and there's more in there basically I remember everything but it should catch everything I think yeah it catches the hooks as well so it should catch everything to be able to reproduce but then there's something we missed okay yeah and then about external modules something that I actually also ran into sometimes so improve some error reporting for missing modules and then having versionless entries in the metadata file when you use external modules that it might be let's say error prone sometimes because sometimes but it's much easier because then you don't have to duplicate entries all the time in the metadata file but of course you need to take care because as for the skip sanity check you need you need to know how to use that of course at your own risk so latest news we deployed an HPC OpenStack cluster full and the one that I mentioned before with easy build mainly using GM Wolf also tried a bit with Intel but I had to invent a new custom toolchain like Intel with Mbappet so IMV MKL something like that which it's difficult to pronounce as well and so far it has not being is not being giving a good performance but anyway this is a problem of Intel not of easy build the deployment was fine and in terms of useful links so the on top the easy build use a documentation that we provided CSCS plus the CSCS easy build repositories on github with custom easy configs and easy box and at the bottom the wrappers that are mirrored anyway under the easy builders maintained by you again so easy builders less CSCS and then you have the common three structure and thank you for your attention let's see I have a small question not exactly ready to receive it but to exalt just curiosity when do you start using exalt well we use that already well before it was all D then it became exalt so for a long time after the latest upgrade of pitsteins in 2016 I think we migrated to exalt and Victor and my colleague young young were always fine tuning that in order to have a better report because as you know you need to match the names of the of the actual modules well when they don't use when they use our provided modules is quite easy I think and the libraries when they use custom software Victor and gg managed to have a mapping of the names of the executables in order to report also that that usage in jobs okay is the setup public or no that is not probably okay any plan of making it public well they just said that they don't know yet if that's gonna be made public that's a that would be a question for the year maybe we ever said that it's about his pay grade and related to that as well do you have an idea of what is the size of the data you have gathered with exalt 60 gigabytes of data every every few months 60 gigabytes and I assume that's already heavily tuned and filtered one question about regression testing that you did what do you actually test exactly so it's for testing from scratch the build of our software stack so in general when you build easy build downloads the package and then start building so for instance sometimes links gets outdated then you want to test that from scratch and then also we have continuously updates on the either operating system or on the programming environment that does change a little bit the environment so some build might fail so if the module is already present in the our production build does not reveal that because by default easy build does not overwrite the build so the regression will overwrite everything so in a separate folder of course and then to ensure that if you want to rebuild now the software then you are able to that with the current recipe that we have in our repository so that's


On April 29th, a mile-long asteroid will make a close approach to Earth as it passes safely by. There's no need to worry, this asteroid will not even come close to hitting Earth. Our solar system is actually littered with asteroids and comets, and sometimes they do come very close to Earth. When an asteroid or comet could one day come close to our planet, it's known as a near-Earth object. But how close is close? A near-Earth object is defined as an object that could pass by our planet within 30 million miles. NASA begins to keep close watch on objects that could pass within 5 million miles of Earth. To put that in perspective, our Moon is only 238,900 miles away. Now, this doesn't mean that objects don't pass closer to us than our Moon. But the Moon is still very far away. However unlikely an impact is, NASA wants to know about all near-Earth objects so that we can track their movement through near-Earth space and predict their trajectories well into the future. The space agency has a planetary defense coordination office that maintains watch for asteroids and comets coming close to Earth in order to warn of any potential impacts. Along with its partners, they discover, catalogue and characterize these bodies. But what if one of these objects did pose a threat? If an asteroid were on a collision course with Earth, we want to be prepared. That's why NASA is working on several techniques and technologies to help deflect an asteroid on a collision course with Earth. Join us as NASA experts discuss April 29th's close approach and what NASA is doing every day to help protect our planet.


And welcome to another episode of A Likeable Science here on Think Tech Hawaii. I'm your host, Ethan Allen. As you know, likeable science is all about science that is meaningful, relevant to all of our lives, why we should all pay attention to science, why we should care about science and support science. Today, we're going to be dealing with a subject that really gets right in that beautiful interface. We're going to be talking about addiction, drug abuse, and the work that's being done to help people deal with those issues. And we have Alan Johnson, the president and CEO of Kenamaoka. Well, thank you. Well, thank you. We appreciate this opportunity. Oh, yeah. No, very glad to have you here.  There are more people now in need of, we're misusing drugs, certainly, and more addictions to more, more compounds coming up all the time, right, that are. Yeah, you know, part of the reform that we did, you know, when we did it in our state, we were doing the reform, which we said, okay, we have to look at chronic illnesses and what the impacts are. And we, like many other states, began looking at, okay, we're treating addiction, but what about misuse? And then, oh my God, that's when the government came out two years ago, it said 93 million people have a misuse or addiction problem. It's almost 30% of our nation, if you count prescription drugs and alcohol. And we said, oh my God, and they don't get any treatment if you're misusing. It's really opened our eyes. This is sort of like a continuum. There are people who don't use it at all. There are people who use it a little bit, very sensible, there are people who sort of misuse, and then there are people who abuse and, and or get addicted to it. Yeah. That's right. So we're not talking about moderation, but we're saying, hey, if you're misusing, you're doing it to the extent that you're hurting your brain and hurting your organs. And that's 30% of our population. And we said, and they tabulated that it was half a trillion dollars a year going to our healthcare. And that's without really knowing all the, yeah, and that's not taking help on the, on the personal suffering and the problems with all your friends and family, trying to help you out and deal with these people, right? Yeah.  So a lot of suffering that's going on, and by extension to the family, too. Yeah. Yeah. And it's, it's a funny interface between sort of personal choice, medical health and legal system, right? I mean, all three of these groups sort of, well, interface on it.  But if you're not, maybe that wasn't our best choice. And how do we get medical, physical medical, talk to behavioral health? And that wasn't happening as we get them together, we go, we've got great technology, we've got great solutions. We're just not really working together and implementing them. Yeah. But that's interesting now that that middle ground and misuse is being recognized. And again, this is sort of like the trend in medicine in general these days, right? It's better, you know, an ounce of prevention is worth a pound of cure, right? You're far better sort of stopping the problem before it gets to be too severe, right? Well, we found out is why we were tasked with looking at chronic illness, we found out in the United States that 5% of the people cost 50% of the cost. And we're the most expensive in the world per person with the lowest outcomes of all the advanced nations. And a large part of it is that, hey, we have, we're not taking care of people when they first get sick. Half the population we do, but the other half we don't. And when they say, if we were to take care of everybody, that's way cheaper than not taking care of you and then take care of it with all our wonderful technology that's very expensive. Yes. That book came out from Pearl's book, Mis-Treated recently, that really lays out the whole case on how badly our medical system as a whole is doing. Basically, it's, we face this perfect storm of weird insurance laws and... Our systems are broken, but we undoubtedly have the best medicine in the world, best doctors and even the best behavioral specialists. But how we do it, how we work together, not too good. We don't have the outcomes to show for it, for sure. Excellent. So let's be sure we get some terms out here. So addiction is, I mean, how do you define addiction? Addiction, you know, by the medical field is designed as a chronic relapsing disease, kind of like heart disease and diabetes, but with addiction is that you have an obsessive compulsive need to seek and use drugs, even if there's negative consequences, even if it harms you, you would still obsessively do it. And so something like this can become very, very harmful for you. You're losing a lot of control. You're uncontrollable.  Right, yeah. Your whole risk reward system gets thrown real out of whack. And rather whack. And also you have structural changes in your brain. Chemical changes that, you know, become dysfunctional in those brain. And so when you need them most, they're not there to help you. In fact, they hurt you. They work against you. Yeah, yeah. Okay. And I miss you. And the difference of addiction is it's treatable and preventable, but if left on, you know, it's gonna not end good.  So the answer for you would be slow down. Where addiction is, you can't, you have to stop. You can't ever use again. But you could slow down, but maybe you need a little help with that. But you're losing some control, and do you know that? We think people don't really know that. Right, right. People, particularly people who are getting these, they're drugs legally, right, like how can I be out of control? I'm continuing to refill my prescriptions using prescribed. Well, the case in point is the opioid crisis that every state, 50 states has a crisis of over-prescribing like 700,000 prescriptions here in Hawaii, 30% of the people using them, and we have 66,000 people on the mainland dying from it. But it's the 50-year-olds and 40-year-olds that started opioids.  Yeah, and that gets us into the next one, talk about, well, are there ways to look at the factors that, you know, people, who's really at risk? Can you tell sort of in advance before people ever start dealing with using drugs, who's at risk for addiction or is there really, are there lifestyle indicators, are there tests to do, any of that stuff? Yes, yeah, there's what they call protective and risk factors. But we now know the genes are about 50% of the issue. And the issue is that we know a lot about genetics and we know a lot about the chemical reactions in the brain, but we don't know exactly on the genes for, you know, why about 25% of the people are going to say, you know, I don't like drinking. It just doesn't do much for me. And 25% of the people go, this is the most amazing thing ever. And then about 50% of us are in-between. We kind of like it, but is it worth the hangover and how much times? And so we manage it. So we don't know why that is that there's those people. If we did, we think we could do a lot. But protective risk factors has been geared towards the kids because usually you started as a teenager. Only recently is it middle class starting at 50 with opioids. Yeah, when you were at a rotary group, you showed a slide that showed how big, sort of, midlife bulge in opioid use with a 50 to 40 to 60 year old was basically being a huge percentage of the people who were really in trouble with it. Yeah, so before we would say, if you're 50 years old, we didn't really see in treatment because you probably started as a teenager or a young 20s. You didn't live till you're 50. Not very many people did. All of a sudden, we're overwhelmed with all these 50 year olds coming in that said, illness, injury, now I have an addiction. I can't stop. Help me. Yeah, that's it. So it's all too easy to do, too, right?  And the companies who manufacture these opioids, in particular, have been quite aggressive about marketing them to doctors, right? Yeah, and not always forthcoming or on either. But that's true, is that, hey, look, it's really good for acute pain. But it's dangerous for chronic pain where you use it over a long period of time. And in fact, it can increase your sensitivity to pain. And your doctor has to make that call, whether that's worth it or not. But you can end up into an addiction easy. Right, yeah. It's a very sophisticated drug. A lot of their doctors all the time. Well, doctor shopping gets pretty common. Getting some from your dentist. And then you're getting from different doctors. And then, of course, you're getting from your friends and family who say, I'm 50, I'm working. Could you help me open up your medicine cabinet? We all got that little stash or emergency use when I'm going to need it the most. Okay, you can have it. So all of that leads to, okay, then I'd be more. And where do I get it? Right, and they start mixing drugs in. And then if I can't get more of the legal stuff, I go off the roots. Yeah, the methamphetamine is really spiked here in Hawaii. We haven't got, we've got some hair and increased, but not like the mainland to get huge heroin and fentanyl. But it looks like they're turning to methamphetamine a lot of folks. But a lot of them are just heavily using alcohol and prescription paints, not a good combination. And then you get, I mean, you've got also these people who are in very diverse life circumstances, right? Some people who are just sort of slid into it very accidentally from, yes, they're a working person. They've been good all their life, basically. Never been in trouble. And suddenly they're in this spot. But you've got other people too who have been, for various reasons, vagrants out on the street, homeless, in trouble on and off. And they start using particularly some of the things like the methamphetamine, right? And they get, I mean, oftentimes it seems people speak of them going sort of psychotic. That's true, but the majority of people probably start with something less intensive. And usually they say half are because they want to feel good. And so if you're talking about misuse, it's alcohol and prescription drugs. So that's kind of where they start.  Today it's really huge problems with marijuana with kids and they're having troubles with it. So they start there. And they say half use it to feel good and half use it to feel better. So traumas, things not working well at home, puberty, grief, moving, bullying, all that type of stuff. It takes away those feelings. The other half are saying, oh, this feels really good. Let's do more of it. So those are kind of the two approaches that come at it. So it's about 50-50. So then it leads to bad cases if you don't manage it well. Right, right. And then when you get addiction, you can't manage it at all. It's unmanageable. A lot of people say, well, it's low morale. Or what's your morality? What's your willpower? It's not, once you reach a disease, brain disease, it's uncontrollable. Right, right.   Yeah, we like to say that, hey, if you're starting off, you're getting kind of a whisper to use more. And then all of a sudden it turns to a loud voice. And when you get addiction, it's somebody with a bullhorn screaming in your ear. And there's nothing you can do but hear that pounding, driving, screaming. People who get a whisper, they don't understand the pounding, driving, screaming in that all you can do is try to take care of that. Right, yeah, yeah. It becomes sort of an urgency to it. I must deal with this in the moment right now because that's the only thing that matters to me. Yeah, and we do know that the brain that stops working was, besides the reward functions of it that wants you to do it, how you deal with stress and it creates anxiety, but it's also cognitive understanding. You don't know how to analyze, evaluate, solve problems, and you lose your impulse control. You can't control your behaviors and you can't control your emotions.  Yeah, so again, it sounds like, as with so many things, we should probably be doing much more in the early end educating kids about that and helping kids understand their own thought processes and how they think when they get impulses to do something, how they should evaluate those impulses before acting on them, right? Yeah, I think that, and that's, so that area in the adolescent brain does not develop very good, how you analyze and impulse control. The other parts are working, but that part doesn't work. And I think we need to get the message to them if you start using marijuana or alcohol, you may arrest the development of your brain and you may never get it back.  So we've got to get better messages out there. Yeah, yeah, the prefrontal lobes of the brain where you're really, which really are the judgment that's the last part of the brain to develop often times not into late teens and into the 20s. Yeah, they were adult at 23, they're saying, yeah, very good, yeah. So it's, so those impairments, we got to get that message after the kids. Yeah, so this is the 14 year olds or 12 year olds don't start, you know, and understand this is probably something I need to be, to be very cautious about. And we don't want to have scare messages.  Yeah, of course, that's been the whole issue with marijuana, right? It used to be when back in the day, a low percentage of THC, what, 10%, 5%, I mean, it was really low and now they've got stuff that's just phenomenal. Yeah, they say it's like 300 times more powerful if you get the high grade stuff that, cool, that's, before we would say we didn't see addiction on marijuana, but now it's the third leading addiction in Hawaii for adults, you know, meth, alcohol, and then marijuana. Those are the big three.  It's really surprising. Interesting. So we've talked about the sort of the downsides of this and how it all occurs. When we come back, but we're gonna take a short break first, we're gonna look at sort of the upside. You guys at Hina Maoka are doing what you found, what the evidence that it works for is and all that, but we'll do that after the brief break here. When we come back, here on Likeable Science with Alan Johnson. I'm Jay Feidel, Think Tech. Think Tech loves energy. I'm the host of Mina, Marco, and Me, which is Mina Morita, former chair of the PUC, former legislator, and Energy Dynamics, a consulting organization in energy. Marco Mangostorf is the CEO of Provision Solar in Hilo. Every two weeks, we talk about energy. Everything about energy. Come around and watch us. We're on at noon, on Mondays, every two weeks on Think Tech. Aloha. Hello, everyone. I'm DeSoto Brown, the co-host of Human Humane Architecture, which is seen on Think Tech, Hawaii, every other Tuesday at 4 p.m. And with the show's host, Martin Desbang, we discuss architecture here in the Hawaiian Islands and how it not only affects the way we live, but other aspects of our life, not only here in Hawaii, but internationally as well. So join us for Human Humane Architecture every other Tuesday at 4 p.m. on Think Tech, Hawaii. Yeah, and welcome to Likeable Science, back to Likeable Science, I should say. Here we're talking here on Think Tech, Hawaii with Alan Johnson, the CEO and president of Hinomauka, recovery center, I guess, you call it? Yes, sir. And we've been talking, we spent the first half of the show discussing what addiction was, how people fall into it, the different sort of paths they fall into, the dangers of it, the damages it can do. But your business is really all about helping people who have gone into it, addictive or now even working with people who are just misusing as we were talking about, right? I know, it seems like a dark subject, but we're about hoping opportunities.  There's not curable if you have addiction, but misuse is you can slow down, but an addiction is treatable and we can work with you and you can have a wonderful life, maybe not using, but a life of abstinence, but it really can do help. So what are the basic principles of your treatment? So underlying ideas. You know, it depends on the continuum you spoke about. If you're misusing, you're gonna get a lot of motivational type work. If you're, now you're going too far and you need short term, now you need the beginning stages of addiction, you're gonna get this outpatient, but then if you're getting really serious, you need residential, you probably have comorbidity issues, diabetes, depression, and you gotta treat the whole thing. And then you need to be removed from your environment because you gotta relearn all those things. So what happened is you lose your memory, so you learn, you don't know how to do social skills anymore, you don't know how to do evaluation and decision making anymore, you gotta relearn that. So the recent, the research, brain research really helped us to improve our therapies because now we're focusing on changes in attitudes and behaviors and coping skills and how do you address triggers, rolling that. So there's a lot of work like that. Yeah, because this whole triggering business is really bad, right? Somebody may be doing okay and then something happens, they either see a bottle of their favorite booze or smell that, you know, a smell of good marijuana or whatever and... Once you get to the addiction, these cravings, you stop using, those cravings go on for years. Years and years, so they diminish, but sometimes they go away and they come back really strong and then go away, but come back less strong. So you need the support system, but you need these coping skills, how to do it, and you need a practice to relearn your, okay, here's our problem solved, but here's how I cope if something triggers those cravings again, and you understand it's a short duration. So a lot of that happens. Yeah, again, this is sort of getting back to what I was saying earlier about, you really need to educate, we should be educating kids from a fairly young age about being aware of how they think and what's going on in their brains in terms of, when you feel this way, like, step back for a moment and think, why am I feeling like I should go out and do this? What keeping that decision making going so you don't have no control over your impulses, and also the other part is the emotions. You lose the touch over control of your emotions, so we want to help you deal with your emotions with positive ways and begin to learn how to process your emotions, so you go to school, you don't learn that, you don't learn, grandma died, how do I deal with that? You don't learn any of that. There's some, I change schools and one kid does that well and the other kid doesn't, how are you processing that and what kind of support systems do they have? Yeah, and schools typically don't teach a lot of emotional management, emotional management, and people learn very different things, different families, family backgrounds, give very different messages about that, right? Culture and ethnicity, all that has a different way of expressing, so maybe not trying to change that, but how within your culture would you do a healthy expression of working with your emotions that comes out in a good way that helps you, not hinders you, and we're not very good at that, and so we often end up hindering, not helping. Sure, sure, because you can't just ignore the emotion and suppress it and not express it. Suppressing is the worst, but expressing it really badly, why now, I got too good for you. You have to find that happy medium where you express enough that people can see and hear, you're paying, you're frustration, what have you, right? Yeah, so a lot of the things you would learn in treatment I think would have been great for anybody to learn, you know, because you're really developing those things to a little higher degree, all your cognitive ability, your how to deal with stress, your reward system, all that is really valuable. I mean, you suspect if you taught this to five and six-year-old kids, they'd really learn it fundamentally, it integrated in their brains in a much better way. We could, you know, for one of the things we're trying to do to make it better is we're trying to say, let's get behavioral health into primary care, you know, you go see your family doctor, it's their favorite specialist in the room, so you need a little bit of motivation, so there's some great motivational techniques. I mean, doctors tend to say, well, lose weight or slow down on the drinking. We know that it helps a little bit, but for most people, not too good. But if they learn motivational techniques, they could do something more. And then we also love the idea that, hey, if you do need more, then you come back to the doctor who stays with you. It's a relapsing disease, so you're slipping. Maybe if you need a little more, you come see us for a couple of weeks, you know, outpatient a couple of times, and then if you need it again, we're there, and then if you get too carried away with it and you're gonna need us for our outpatient or residential. So I think that's a great news way. That's why the United States have developed those plans right now to do that. Yeah, because we've had this area of fragmented medical system. I mean, it was driven home to me years ago when I switched jobs and had, essentially had to then switch doctors, and it's like, why should I have to switch my doctor, just because I switched my job? I mean, I just spent two years developing a relationship with this doctor. I finally knew them, I trusted them, I liked them, they knew me. We do lose that by switching, unless there's a good reason or you think you need something different. But being with somebody and then having that network of people who are working with you makes a big difference. And if a behavioral health specialist is part of that, great. And you can use these motivational techniques for diabetes, for hypertension, all, you could really make, that's where you make the changes, motivate people to change. Like to change the behaviors. It would really improve primary care. And then we would like primary care to deal with more chronically ill things and not the less minor things. People with less licensure can do with that. So that's what they're doing in other countries that have been really phenomenal. And we think, that's the next step for us here in the United States. Excellent, so they're getting a whole sort of behavioral health workforce that really focuses on this kind of stuff. That we talk to each other. And we share the treatment plan. Because if you're in treatment and we want to know if you're diabetic or you've got this major heart condition and how do we help you if you're in residential? Now we're saying, hey, look, we need payment reform, we need a doctor on site. So that we can handle more complex patients and a nurse. Not a hospital, but a little investment in us. And we've already had pilot projects. We can make a phenomenal difference on outcomes and keep people out of going to the hospitals, which is where all your cost is. Yeah, right, right. Somebody checks into a hospital for a day and there's thousands of dollars, you probably, right? And we have this few number of people but a large number of them are going to the hospital all the time. Yeah, yeah, repeatedly. They're repeatedly. There's millions of dollars. I mean, we spend about nine billion here in healthcare in Hawaii and they say 5% of the people are doing the... If that's true, that percentage, that means about four and a half billion a year we're spending here in Hawaii for to treat a few people who didn't get treatment when they had to onsite of a chronic illness. So some investment and some good ideas really could save a lot of suffering and a lot of money. Yeah, and we really need to do that. We see that in other things besides the health system too. So in a lot of European countries, for instance, in a K-12 schooling, the teacher will move along with a group of kids for three or four or five years. You'll have the same teacher again and again. That's great. It saves a lot of time each year not having to sort of learn the rules again, right? You know as a student what you can and can't get away with, you know?  The teacher doesn't have to learn new names, new personalities, right? They're just watching you grow and working with you and they get some really good outcomes from that. Yeah, I think we could learn a lot from the European countries and also Japan and some of the others. The thing is what we've been very good at is we've been comparing ourselves to other states. Oh, I'm not as bad as that state. And I think when we were the best, that was a good idea. Who is the best of the states? But now that we're in last place, why are we saying I'm better than my teammate who's worse than me but we're all in last place? And we have the best medicine and the best technology. We could take some of their ideas and we could do phenomenally better. I mean we could skyrocket back to the best. If we just plan better. Right, but it's a very large scale, very sort of system-wide problem is a problem. And there's a lot of different factors of economics, sort of the business models are currently being used, the academic structures, the medical insurance companies of course are a big part of it. All these things sort of right now locked into a sort of pattern, right? And you can't change one of them. Well, we began here in Hawaii recently by having all these task forces and I must be on seven task forces now and I'm glad to be invited to them and with hospitals and insurances and other providers and government. And for the first time we're talking to each other. There's a great deal of hope on that. My complaint is we move too slow but we need some activism and more people saying let's do it but it's a good start. Excellent, excellent. That is a hopeful trend. So when you see an individual patient, I mean how do you define success for them? No? Well, it's a chronic relapsing disease so it's not uncommon that you come and do you complete treatment and are you absent when you leave treatment? We talked to these coping skills, how are you a couple of months later? And so we're just learning that now with support from the federal government to help us change its funding. And so they're saying well maybe you should be a case manager, we try to connect you with a primary care doctor so that we know if you're relapsing, we catch you so it's a little bit more not you're out there for a long time and you come back for a lot more. So that's beginning to happen but so we do have follow-ups for six months but a self-report but we're going for by having a doctor and a nurse and we're talking to each other, we went from about 50% completing residential to about the 75% completing. That's really good. Now how many are still there six months?  What did we do for six months? Nothing. Right. So you know. You catch me in a good moment, I'm fine if you catch me in a bad moment. You're really good, even though last week you weren't too good. Or this week you are good but you were great for six months. So that has to improve. So some case management is happening now that we have some expertise on that. So that's just at the beginning stages it developed. That's great that as a sort of a therapeutic specialty you're learning about yourselves, you're learning about better ways to manage your own businesses or to make your treatment more effective and more helpful to people. Well it's kind of like saying, hey we got all the stigma out here and we got the shame and punishment and now let's call it a medical issue and that it's treatable as a medical issue. Now if you were to do that, how would you do that better? And that could lead us to a lot better solutions. Yeah, I mean if you look to really the old model where a lot of illicit drug use was treated strictly as a legal problem more than a medical problem, there was a very weird situation where you had the police on the one side saying, let's enforce this law more tightly and organize crimes. And yes, let's keep it all enforced tightly so we can keep our monogamy on things. And it's like, well the police and organized crime are on the same side. Something wrong with power handling this. I do like that one, the police need to keep us safe no matter what. And if you're misbehaving and we could take away your drug addiction and you would stop misbehaving and we would say you're not really a criminal, you're doing criminal because of your drug addiction versus a criminal who has a drug addiction and you take away the drug addiction now you have a sober criminal. Those are two different people. But for the police they're recognizing that those who have a drug addiction they're reaching out more. Their safety is their priority. But they're reaching out to us to say, can you help us more? So we began those talks just in the last year so we see them as being, look we know we can't arrest ourselves out of this, we need you. That's really refreshing to be, we're at the same table because before we would be at the table and disagree and now we're kind of agreeing. Excellent, that's really valuable because yeah, again it's a sort of systemic view of it and let's get this person, let's get them in help. Excellent. Hey, well this has been great fun and very informative. I've learned a lot and I'm sure all of our viewers out there have learned a lot too. Alan Johnson, the CEO of Hinamauka Recovery Center and people I'm sure can just Google Hinamauka. Hinamauka.org and we do have walk-in clinics where you just show up between nine and two and we'll help you right there screening assessments. So please come and see us and we got multiple sites and at 27 sites so we're all over it. Excellent. So thank you for having me. Thank you for being here and I look forward to your continued success and I hope you will come back and join us next week for another episode of Likeable Science here on Think Tech Hawaii.


We have to create, yes, we have advantage of demography. This part of the globe has advantage of demography. I am again and again ambitious, I am citing, I am quoting that this part of country and this part of globe. That means we all have a common aspiration. We must have a common goal also. In our Gora Bamayya Balijatra, the world's largest planet, where we are preparing to meet Balijatra, the Cominsar India has kept in mind to be a replacement to the Earth's surface. And without access to any alternative to the world, the world is having a mixed impact. Balijatra is a matter of connectivity In this days, santuaries ago in these days ੎ arises  결국  preparations are carried out due to the constructing of the block. ੎  set up ੎ the process of ੎  in the process of construction ੎  split in two. ੎  after a few months of ੎  the trading ੎  they used to come back to the mainland, ੎  brought fortune ੎ got new prosperity ੎  brought new ideas ੎  paid new ideas ੎  also ੎  whenever they ੎ went outside ੎  they took new ideas and prosperity ੎  prosperity ੎ to different part of the world, ੎  the ੎ the rampul of ੎     the five years they have. The event today is a testament to the government of India's approach and commitment to bringing international focus to eastern India and particularly Odisha and enhancing their global relevance. I am confident that the insights and collaborations that emerge from this conference will significantly contribute to the PURVODE mission.


Well, our next speaker is Mike Spever. He's CEO, CTO at Active Group, co-organizer of the annual Bob Conference, the Wellerberger Conference in Expert and Functional Programming. He has about 20 years of experience in teaching programming in high schools, universities, and other contexts. In his talk, Mike will share his experiences and gives us an idea on how changes in culture, methods, and tools in programming might contribute to the future of software and the software of the future. Please enjoy the talk by Mike, the stage sewers. Thank you very much. And did she say 20 years? I'm older than that. So I appreciate you staying up this late. So I'm all grown up now, so talk about growing up. So I graduated at some point and got a degree in computer science. I got a PhD many years ago and now I'm CEO of a software company that carries some actual responsibility. But as I was preparing this talk, I thought back to the past, and it happened to be the year 1983 when I started hacking. And well, I wasn't there, but it must have been the year of 1C3, right? And so if you read what was written about the computing revolution that was to come in the early 1980s, a lot of what we're having arguments about today was already there, computer security, data governance, surveillance, all these things were already on the radar of the media back then. But a couple of things were different also. So here's a major German news magazine that had an article on computers in children's rooms. And not everybody had a computer back then. So if you were a kid like me who was interested in learning about computers and didn't have one at home, there was a place you could go to which was the local department store. And well, the picture isn't very clear but that's actually a local department store that had rows and rows of home computers set up and plugged in and ready to go for children to play with. And so back then there were, you don't remember this, but there were many different kinds of home computers. There was no internet. There was very little in the way of readily available literature on how to program computers. So we all had to figure it out by trial and error. And we didn't really have a good methodology of how to program computers. Now fast forward a couple of years. I looked unbelievably geeky in 1986. I had my own computer and that must have been even the second or third one that I had. And by the time I was an exchange student in the United States, they made me president of the computer club. You can see another picture of me right there. And that was my first experience of teaching a formal course. So that was in 1988 and that was the end of that year. So that means I taught my first computer science course 30 years ago. And I then ended up designing the intro programming course at the University of Tubingen and taught that over more than 10 years. I taught programming to humanities majors. I've done a lot of professional training by now.  Your coworkers, your friends, my children, certainly my relatives. A lot of them have had to endure one or other kinds of programming course for me. So back then, as we were trying to figure out how to hack computers, as there was very little available literature, we also looked for role models. And here's an early role models from the 80s, a man named John Draper, better known as Captain Crunch because he used the whistle that he found in a cereal box to manipulate the US phone system and then actually went to prison for it. So but John Draper also is not as well known for actually writing software. And he produced one of the early word processors that was available on personal computers called Easy Writer that I actually used. And there were reports on the prolific programming style that Draper practiced. So I don't know if you can read this. The joke is if Draper were writing math routines for addition, he came up with the answer two plus two equals five. He would put a clause in the program if two plus two equals five, then that answer is four. And that's generally the way he writes programs. Who's seen programs like this? Where, yeah, so some of, yeah, a lot of you, right? Where somebody adds clause after clause of special cases until, you know, so the special cases encountered in the wild are all covered. So of course, you know, in 1983, we all figured out ourselves, but people by 1985, the Hacker Bible came out and already worried about how we would educate children about computers in the future. And so there was an article about computers in school and I'm gonna zoom in a little bit and as I'm sure you've been involved in many discussions on how to teach programming to beginners and as always, that discussion focuses on the programming language that's being used. And back then, you know, the popular languages were somewhat different from today. There was Basic, Pascal. Basic had that reputation of producing spaghetti code. Fourth is mentioned, logo is mentioned and C is mentioned there. And back then a prominent professor of computer science in Germany said, well, as long as you don't program in Basic, you'll be fine, right? The pure fact of not programming in Basic is gonna keep you from writing spaghetti code. And as we now know, that isn't really true. And I'm sure most of you have seen really crappy in spaghetti code and production, but the bullets kind of keep hitting closer, right? I think this must have been two years ago when a vulnerability in popular IoT devices essentially brought down the internet because they were all assembled into a botnet that brought down a popular DNS provider and GitHub went down. I think that's what most of you probably remember. There was Heartbleed that I'm sure a lot of you remember due to a vulnerability in a buffer overflow because in OpenSSL, which was written in C, there was Equifax, which divulged a lot of social security numbers because of a bug in struts in the web framework that they used. And more interestingly, I think for students of the past, there was the Cloudbleed vulnerability which would divulge secret passwords and account information on web pages just as part of the web cache. You wouldn't even have to ask it secret questions as you would have to ask Cloudbleed. And that was oddly reminiscent of something, well, none of you remember, unless you're as old as I am, which was the BTX hack, which was one of the founding stones, I think, of the Chaos Computer Club where people were able to make the computer of a bank that was attached to BTX, divulged passwords, essentially a memory dump that contained passwords and then transfer a lot of money from that bank to them. And so that felt a lot. So the present, in many ways, it feels a lot like the past did. And you would think there would be some kind of progress. And I think I was as guilty of that as anybody. So in 1986, I wrote my first book and as many books now and then, it focused on a specific programming language. Of course, I picked the worst programming language in hindsight that I ever could have picked and that is responsible for a lot of the vulnerabilities that we see today. So but again, at least I can say it was in 1986. So it was a long time ago, a long time ago. So you would think things have changed. So some of you may have been at a talk earlier today about the Bob system for teaching young kids on how to program. And Bob prides itself on teaching, programming in exactly the same way as computer experts would program, but it would use an approach used based on big gamification to teach students, right? And so in 1986, we were programming in C. So Bob now is programmed in C++. So something has happened since then. So and you can look at a lot of pages, web pages and systems for teaching beginners on how to program. So here's a prominent page supported by a lot of major organizations called Start Coding DE from this year. And you can see again that a lot of the stuff that he teaches were first of all, it's focused on particular programming languages on scratch and Python and processing. So there's three programming languages mentioned there. And if you can read the fine print and if you can read German, it says Spielerische herangehensweise, which means that there is a fun-based playful approach to teaching how to program. And that's common to a lot of things. So if you go to kid space, you will find a section there called Jungtakt. Jungtakt, and you can see that. It also takes a very playful approach to teaching. And if you look at its material, a lot of it also features particular programming languages. And if you look, you will find that there's a curious absence of methodology on all of these pages. And I think part of the reason is that I think if you teach programming, you can easily get the feeling that it's very easy, right? I mean, you can teach people to write very trivial programs that will blink an LED or do something like that. And the lights, the eyes of your students, they will just light up and they will just love what you teach them. If it's trivial, if it's not so trivial, so I like functional programming, which is sometimes not so easy. But in any given group that's large enough, you will find a few Sheldon's that will just eat up everything that you teach them.  And if you've been at it for 30 years, you'll notice that it doesn't really matter what you teach. You will always find a few Sheldon's that will tell you that they love what you teach them. And so after a while, you realize that, well, there's a few Sheldon's, but then there's also a lot of students in your class who are not Sheldon's. And well, as you're chugging along on your teaching train and you look out the back, you suddenly realize that you've left a lot of students behind. They might still like what you're doing, but you really haven't taught them any significant abilities. And so over the years, I've had a lot of opportunity to think why that is. Why is teaching so hard and why is it often unsuccessful? And one of the reasons is that I think if you pick up any, almost any book on programming or open any webpage that tries to teach your programming, it uses an approach that I call teaching by example. So here's a book by a computer science professor, a professor came out this year and learned Java the easy way and the easy way is in the second chapter, right after the introduction, we do an example, right? And there's a concrete listing of a concrete program and the text essentially is just a commentary for that program. And then the expectation is that from the example that you've seen that exposes various elements of the programming language like while loops and system out print on it, whatever, you will, I think by osmosis or some other magical mechanism absorb the ability to apply the techniques to new problems. And my experience has been that that often does not work for the great mass of students who are not Sheldon's. I thought back to the great teachers that I've had, maybe not in computer science, but what methods were they using? And then when I thought about it, I realized that they were using methods that were completely different. So here's a picture of my 11th grade, English teacher and Carvalho in 1987. That was as I was an exchange student in the US. And at the beginning of the class, she handed out a piece of paper. Well, you can't really see it, so I'm gonna zoom in. That said, you know, how to write a composition in this class, right? And so composition of this class, well, here's the second section is the body, right? The body is three paragraphs with each paragraph developing one of the areas of the thesis. First area of scope, usually one paragraph, transition topic sentence. If you look at this, this is a very precise outline that explains the function of just about every single sentence in your composition. So it's a very rigid harness, if you will, for your composition. And, you know, coming from Germany, Germans are generally very, they don't accept a lot of power distance and they don't like authority that much. You know, I was an instant rebel to this concept and it took me most of the year to really accept that it was a great way to learn how to, to learn one good and working way of writing a composition and then I could branch out from there and develop other working ways of composition.  And you might say, well, we have methodologies like that in computer science and in programming as well. You know, we've been doing object-oriented programming which is supposed to be very principled. But I've seen a lot of crappy object-oriented code as well. I don't know about you. So, you know, an object-oriented programming, well, it has encapsulated state, it has polymorphism and it has inheritance. Inheritance is particularly bad because there's so many different uses of inheritance and typically your programming language will only have one mechanism and so people often get it wrong and they do inheritance the wrong way around and there's empirical studies but there's many other ways in which object-oriented software sort of ends up looking like this. So I don't, yeah, anyway. So, you know, this is not supposed to be an indictment of the hacker culture. I mean, we see a lot of things around here that look like this and they're wonderful but they're no substitute for methodologically sound development of software. So, yeah, well, many years ago I thought there has to be a different way and I got together with a man called Matthias Fell-Eisen who happens to be a fellow German bureaucrat like just like me but who lives in the US and who wrote a book called How to Design Programs which you can find online if you're interested. You don't even have to buy the paper and we kind of did sort of a German version of that working with Matthias called Deinprogram which also has a free book on it and the overarching principle of that approach is that we try to be systematic about everything so that we try to attach systematic fixed plans or system to every step of program development and these things are called design recipes. Just with a name invented for this. So I'm gonna, well I guess I'm gonna torture you with one concrete example, a classic example. So Matthias was in Texas at the time. So the classic example is that as animals on the Texas highway and we're gonna concern ourselves with two kinds of animals. So there's an armadillo on the left and there's a rattlesnake on the right and there's of course the highway in the middle, okay? So we'll start out with armadillos and for the purpose of this application you might just describe armadillos. So what's important about armadillos and here you will see why is now armadillo has the following properties. It's alive or dead and it has a certain weight and so I'm gonna greatly shorten this process of systematic development here. But what we say is well if you find yourself describing your data, this is a data centered approach, if you find yourself describing your data with words like it has the following properties or it consists of the following parts then you're dealing with something called compound data. So this is a fixed part of the plan and it has a name and it's very important to give names to things so you can talk about them. And so once you've identified that you're dealing with compound data you can model that data and you can translate that model into code and here you see a screenshot of a system called Dr. Rackett which is a programming environment that was developed for beginners. And you can see a little program fragment in a programming language also developed for beginners. In fact that fragment was developed for beginners. That says well in the interest of brevity we're abbreviating armadillo to dillo. So it says well we're dealing with dillo so we're calling the data type if you will we're calling that dillo. If we're dealing with compound data we need one meter constructed so we need a constructor and so we're gonna call the constructor make dillo. We'll see later that we'll need one way of distinguishing dillos from other things so we'll need what's called a predicate. Ignore that for now. And on the previous slide or here in the data definition the comment that is in yellow you saw that an armadillo has two components it has two parts and therefore there need to be two if you will getter functions and those are called dillo alive P and dillo wait. Never mind all those parentheses at least not for the purposes of this presentation. Oh and one more detail maybe is that if you look here it says language Dr. Rackett is a programming environment that has many different programming languages and this particular one as I mentioned is called the Martha abstraction which identifies one particular set of programming language levels for beginners. Now you know once you have that record definition you can immediately call the constructor to make examples so here's two example armadillos and so you see these two expressions make dillo and this hash mark T means true. So make dillo true and 10 means while we have an armadillo where alive is true and the weight is 10 so that might mean an armadillo that's alive and weighs 10 kilos and then there's another armadillo that well, alive unfortunately is false so it's probably dead and it weighs 12 kilos and the defines there just say well we're gonna call the first one d1 and we're gonna call the first one d2 okay. And so one way to talk about these functions that we've created is by writing down what we call signatures. They're almost type signatures not quite. Let's say what these functions behave like so the constructor function that you just saw applied is called make dillo and you know in these funny programming languages most things are written in prefix notation so make dillo accepts a boolean and a number remember that was that alive thing is it's still alive and the number that says how heavy it is and it produces a dillo object and the dillo alive and dillo weight getter functions both accept a dillo object and the first of them says whether the dillo is alive or not so it produces a boolean and the second one produces a number that is the weight of the dillo okay. And well so why is this all relevant life on the Texas highway is animals get run over by cars right. And so we're gonna write a function that simulates that process. So that thing in green is what we call a problem a short statement on the short comment on what the function does that we're about to write so we're gonna run over an armadillo then it says well we're gonna write a function called run over armadillo and it goes from dillo to dillo which means well the dillo is not really a dillo it represents the state of the dillo before it gets run over and then one comes out on the right hand side that says what the state is after it gonna run over by a car okay. And then you can write so you can see all these things are sort of required elements of the curriculum I'm going over them very quickly. The two next things are example and test cases at the same time so check expect says well if we run over that first armadillo it was still alive but when it weighed 10 kilos after we've run it over it's dead and it still weighs 10 kilos okay. And the second test case says well if we run over d2 it is already dead so after that it's still going to be dead and it will weigh 12 kilos. Now from the signature above you know we already know what the function is called and how many arguments it has it has one argument so we can write something down mechanically that is what we call a skeleton which is that thing at the bottom. So we're writing a function or something called run over dillow lambda says that it's a function and D says well that dillow up there it's gonna be called D here. This is a mechanical process and so we see how that is elaborated. We can do more things mechanically from this because dillow is compound data and whenever we accept compound data as input we probably need to look at the parts to do anything that's meaningful so we might as well call the two getter functions. So there's dillow alive of D and dillow weight of D and also compound data comes out as output and so we probably need to call the constructor in order to construct the dillow that it's supposed to come out the right hand side. So all of these things can be written down and we ask our students to write all this stuff down and also actually write down those ellipses marks the three dots and these are building blocks and they are completely mechanical they have nothing to do with the actual purpose of what we're doing and then usually it's pretty easy to then fill in the missing steps and say well we are not really interested in whether the dillow was alive or not before we're probably, but we are interested in the weight in constructing that new armadillo and this is the run over function that does that. So you can see that there's a lot of steps it's an extremely bureaucratic process producing that program and some of those steps are completely mechanical. They might be boring but they enable everybody to make progress and so you might imagine I said rattlesnakes and rattlesnakes are very similar to dillows I'm gonna run over those very quickly so rattlesnake has the following properties thickness and length and you can see there's a record definition there's a bunch of signatures that come out of that we can do a purpose statement that says oh we're gonna run over rattlesnake now not we can't just do armadillos there's a couple of test cases and rattlesnakes can run over they get flattened out so their thickness goes down to zero when they get run over but the important thing is now we might think about well we're just gonna run over whatever animal comes under our wheels next so we're interested in an animal on the Texas highway and an animal is one of the following it's either an armadillo or a rattlesnake and whenever you see that wording it's one of the following what's this or that or that or that you're dealing with something that we call mixed data and so again this thing has a name and with mixed data again there's a fixed set of steps that you go when you program to them so you define a signature that says well animal is a signature that's mixed from dillo and rattlesnake and we're then going to write a function that runs over an animal and so it doesn't have dillo to dillo it doesn't have rattlesnake to rattlesnake it has animal to animal it has test cases like as before and well remember that there were two different kinds of animals on the Texas highway and so we need to treat them differently right it's a different thing of running over an armadillo and running over a rattlesnake so for that we need a conditional and we need this predicate function that says well that animal A is it an armadillo or is it a rattlesnake and then we need to fill in the gaps and of course we've written those functions already so it gets pretty easy this is I don't know if you've seen this and I've run over this very quickly and I've left out a lot of steps this is an extremely bureaucratic process and when you read about this at first you think this is so boring I can't I can't you know this you know you go like this right and interestingly if you if you practice this on your students surprisingly pretty quickly they go like this because they realize that this bureaucratic approach actually makes them produce a working solution on their own some of them can't believe it that it's successful but then they start smiling but it took a long long time to to develop this approach so Matias has been added I think since the early 1990s I've really been involved in this process I think since about 2001 so really you need to take an approach to teaching you need to observe how well it does and I can't stress this enough when you observe it is not enough to measure the popularity of what you're doing it's not enough to ask your students do you like this stuff right is that something that's interesting you know I've seen a huge bandwidth of things where we were unsuccessful but people just liked it just people had just liked it fine and so and that's this process you know we repeated many times and we're hopefully going to repeat it many times in the future so but it has produced the following insights you need so it's very useful to have a very bureaucratic approach to programming that we call design recipes I didn't much talk about the fact that it's very helpful to have programming languages that are specifically developed for beginners or for learners doesn't have to be beginners and it also has to have a programming environment that was designed for beginners and well you can go either to a page called program by design or to dine program and you'll find a lot of information about that but it took a lot of work and it's usually it's not enough to just take your professional programming language and stick it on your kids it might be it might work for trivial programs but it's not going to scale anywhere so yeah well it's about growing up but I guess this one is about growing down so I think the progression really should lead us from you know the C and C++ like unsaved languages and all the hacking of the past and that we like to use when we make exploits to save languages and run times and finally we're seeing something like Rust and ATS coming up I really only know how to do this bureaucratic and systematic approach to programming with functional languages and if you've watched this closely you saw that what we did was type driven right we looked at this at the way our data is organized and then organized our programs according to that data whether you use a statically type language or not is kind of secondary to that but in the future I think we'll see more of that you will write down a specification of what you're trying to do as part of your types and then your system will generate for them and that's also a way of proceeding very systematically so I think when you're teaching programming it's you know being inclusive is very very hard right it's not enough to get people excited it really is important to impart practical knowledge and and competency and the only way I know how to do this is to teach a systematic approach you might call it bureaucratic if you're German but to my surprise even the Sheldon's like this after a while anyway and and we should really refocus or we need to focus you know after all the exploits we've seen over the past couple of decades on correctness and I think this is the path to do that thank you very much thank you Mike for this presentation we now have like two or three minutes for questions please go to the mics one two three or so ah we have one there hi Mike on yes so coming from a lot of the software development world and practical experience probably delivering with other customers have you got any insights into how people would often do teaching by code review direction for people to go in reading to learn more about code learn more about the structure of code in the way that software development often is typically told do you have thoughts on that specifically in this so I think it's valuable in my experience it's just not enough right so the shelves of this world they're able to figure out how to program by themselves essentially right but just saying you know this piece of code is good that's teaching by example right this piece of code is good and this piece of code is bad or we can improve upon it it's just not a constructive way it's helpful right to sharpen the sense of and it you know to help that sharpens your review facilities which is an important part of learning anything but really I think it's so in my experience it's completely crucial to insist that in your teaching methodology you teach them steps that they can actually follow right where I agree with that but in respect of how many modern software development teams work and in a code review you can go this approach will have this in this problem have you thought about doing it this way what have you considered do you have any thoughts on that process in teaching people in that process in that duration so for beginners my experience has been that it doesn't work very well right okay that's a so it's a so it's a it's a common educational point right that you need to teach people you know they're very different many different approaches and we're going to we're going to judge them and we're going to say this one is okay and this one is maybe a little better my experience has been with the vast majority of students right is if you if you use that as your main teaching paradigm the problem is they produce zero solutions and so it's a much better idea to really insist on that one correct solution and you can branch out later okay does that make sense thanks uh sorry Mike one it's okay short quick questions please well I hope it's a short question thank you very much for the talk I'm going to crash my brains or what you've told for the coming days there's one thing very popular among youngsters which is Arduino it's stick up with C and C plus plus and you get very crappy C code so how would you put this what you've told tonight into the environment of such a popular platform like Arduino so I think I would try to make the teaching languages work on an Arduino I think that's going to be very hard with the really really small ones but with the sort of medium-sized ones it might actually be it might actually be possible but but yeah as you point out there's a bunch of crappy C code running on Arduino's I can see the attractiveness of that approach but it's not one that I personally prefer could scratch fill the gap no thank you very much it was a quick answer I'm sorry we're running out of time thanks for your questions and thanks again Mike for your wonderful talk the plus