All right, thank you, Fred David, for the warm introduction. So actually, this is my second time at NorthSec, and I'm really pleased to give a presentation on this stage with other speakers. It's a really great event. And today, I will be speaking about distributed reverse engineering for large-scale malware processing. In a nutshell, we have a large collection of malware millions of samples. And we wanted to distribute reverse engineering of this collection across a small cluster. This presentation is co-authored by me and Alexander Matrosov. He is not here today. He is a security researcher with Intel in the United States. But however, I would like to give also credits to Gabriel Barbosa and Rodrigo Branco. Those are researchers with which we collaborated on this research, which was originally presented at Black Hat 2015. And at the bottom of this slide, there is a reference sort. I will start with your small disclaimer that opinions presented in this presentation does not necessarily reflect the opinions of our employers. And we are responsible for all the results or mistakes and whatever. So blame us, not our employers. And let's start with your picture of the hardware. So basically, this is a small cluster that we were using in this research. You can see everything is well set up. We had about nine machines. And there is even an AC here to cool down the system when it works really hard. But however, if you look at the different angle, there are still some room for improvement. And interestingly that we were working on this research in July, June. And it was really hot in Oregon, Hillsboro. And electricity at some point went down. So we're really thankful for fire detection and smoker system, which prevented from a disaster. So I hope Rodrigo's insurance company won't see this slide. And here is the agenda of today's presentation. So we'll start first with your introduction and you're explaining our objectives and motivation. Why did we do it this way? And why did actually we started this research? And then we will go to our detailed overview of the process. How did we process malware, machine learning algorithm, which are malware analysis algorithms we were applying to them. And then we will go to the discussion of the results we obtained. And finally, we will present some validation of the methodology and tool set to be able to confirm that our approach that we decided to take is correct or it contains some flaws and needs some improvement. And then I will wrap up with the presentation with some conclusions and acknowledgments. So basically they're one of the most, so about the motivation. I think their presentation of Olivier Bilodeau and Hugo yesterday about DevOps was a very nice explanation that the malware is growing with an absurd space and we need to have a scalable analysis environment. So that's what we tried to build because right now the process is focused per single sample analysis. And this is mostly manual work. So we decided to try to scale our analysis algorithms for large collection. And we also tried to provide the research material for the researchers in this area to be able to contribute as well, to use our results and build their research on top of them. So the objectives were just to demonstrate their possibility of in-depth large-scale malware analysis. And I want to highlight here a word in depth because right now there is quite a bit of framework that allow to do our analysis, but in most of the cases it's quite shallower and they're allowed to do some kind of filtering whether this is malware or not, but in depth we are mean a complete disassemblier of the malware sample. We also tried to distribute across the cluster IDAPRO with X-rays decompiler. So why did we pick up IDAPRO? Because this is an indispensable tool for malware analysis. It is widely used in the industry. And so we were kind of interested in our case or if we can do this automatically in a batch mode on the cluster. And another objective is to share with the community all the information that we generated, IDB files or scripts or dumped intermediate representation and so on. Let's speak about the scope of this research, the scope of the project. So we're limit our sample set to only 32 and 64-bit PE executables, that means our executables for Microsoft Windows platform, which were not packed. We were using mostly static analysis. Why static analysis? I will explain on the next slides, but we were doing only static analysis for this reason. We need to have unpacked your samples. We didn't impose any restrictions on their size of the malware. So we had a processor in our collection samples as large as their several like tens of megabytes. And we were given a preference to Microsoft Visual C++ compiler. So we're analyzed samples compiled with Microsoft C++ because of using our X-ray school explorer, which works right now only with their Microsoft Visual C++ object-oriented types. And we were using this cluster, which Rodriguez and Gabriel worked on and they presented this result at Black Hat 2012. So this paper contains the complete description of their infrastructure, but in brief, we have about nine machines. They're sharing 72 cores and 100 gigabytes of RAM. And here is the brief overview of the methodology of the process. So we started by pre-processing. So we take our samples from opener sources and started to filter out. So we filter it out everything. What's not 32 or 64-bit or non-packed malware. And I think in the input we have about almost eight million samples. After filtering out all their samples that don't meet these criteria, we started with running them in the analysis environment and applying different machine learning malware analysis algorithms. And we were obtaining results from their decompilation and disassemblier and we're storing them in the local file system in files. At the next step, we did your structuring and aggregation of the information to be able to process this, to compute some statistics or to do data analysis. That was done in phase number four. This research we were using are only static analysis. For the following reasons are generally, dynamic analysis takes longer time than static analysis. And since we had a very large sample set for us, the performance was really crucial.  We're just loading the file or in the disassembler and it works pretty fast unless there is your sample obfuscated, packed or it exploits a vulnerability in their analysis environment. Of course, there are some limitations for instance. When you're disassembling you need to make sure that there is a good coverage of your disassembled code. For instance, there is some dead code which your disassembler is not able to identify. It's not reachable from the entry point. But how we're at a proiter has a very nice heuristics and generally provides a very nice coverage for your disassembly. So that's why we decided to go with this tool. And in addition, we used the following software, additional software to get additional data. So we used our hex-race code explorer to dump C3 easier of some either recognized functions. I will explain what the C3 is later. But so far you can consider C3 as an abstract intermediate representation of your decompiled function. And we also were extracting object-oriented types from the file using hex-race code explorer. When dumping C3's, they're from their IDB file. So an IDB file is basically, I'm pretty sure everyone knows this is your file where I'd approach towards all the results of the disassembly. So I will be frequently using this acronym IDB file to refer to the results of the disassembly. So we were computing C3 depth like how far this function is from the entry point in the call hierarchy. We were also interested in seeing how many malware is actually using the newer instructions which are implemented hardware implementation of the advanced encryption standard and your get-seq instruction, which is your works with the Intel technology, SMX or safer mode extension. Additionally, we were doing study of this pointer usage. So this pointer is an implicit pointer which is used in objects-arranged languages to refer to the instance of the object. And we were paying attention to their, its actual implementation within the code. For instance, Microsoft or a Visual C++ is implementing this pointer by means of our ECX register or our CX register. If we're speaking about 64-bit platform, but however, sometimes when you're doing some optimizations, it can use other registers, ECI or EDI register. So we were really interested in looking how actually pointer this is implemented for these compilers. And we're also analyzing cryptographic functions, functions implementing cryptographic primitives using the idiscope plugin. So let's take a look at how this process works regarding dumpings and trees. So first, we're loaded a file in the idepro. We'll wait until their auto-analysis is completed. The functions are identified. And we started by enumerating those routines in their disassembled files. And we started with their dumping. So we processed a certain number of the routines because like some IDB files, they contained up to a thousand routines and more. And in this case, we don't have much space to store, first, all the data. And secondly, not all the data are really interested because for instance, there are some many small routines which are just contaminating the data set because they present in every sample. So we decided to filter them out. And as a result, we were processing first 60 routines of size larger than 116 hexadecimal. We were processing 30 crypto routines identified using their advanced encryption standard or spotted by their idiscope plugin. And we were processing the first 60 other routines bigger than 60 bytes. Those constants, 60 and 160 were chosen empirically using our experience and manual analysis and during the manual analysis. Once we find a function, we're a decompiled function using a hex-raised decompiler. As a result, we obtain intermediate representation C3 and we serialize C3 toward a string. But before serializing, we are doing C3 normalization. Normalization is important to make sure that same role functions, they have the same C3s. For instance, you can imagine two functions where they're exactly the same except in one function, the local variable is initialized with variable one and the second function, the same local variable is initialized with value two. So those functions are very similar and we want to have exactly the similar C3s but since the difference or initializers are used, C3s will be different. So during normalization, we're filtered out everything from the C3 which is not really relevant to keep the general structure of the function into dumpets or I will say a few words about this later. Next thing, we're dumped information about object-oriented types. The object-oriented types contain two interesting properties. First, the virtual table which contains pointers to their polymorphic methods. So basically methods which were overwritten in their subclass. To do this, we were finding all the references to these pointers within their IDB file and we were looking for virtual tables which were stored in the following sections. R data, data or section with any other name but with the data attribute. So usually Microsoft Visual C++ compiler puts them in the R data or data but sometimes due to obfuscation, the sections may be renamed or we analyzed all the data sections and we were finding all the cross references to virtual tables within the IDB. Later, we used some heuristics to determine the size of the virtual table and the total number of the methods in this virtual table. The code, those heuristics, they're available in GitHub account for X-ray code explorer so you can go here and see how we're doing this and once we are recognized virtual table we create structure representing this table and include this in the result. In addition to our finding virtual tables, we also did reconstruction of the types attributes. In order to do this, first we identify an instance of the object within the IDB file and then we find cross references to its constructor. So, and once we go into the constructor, this is your part from where we actually define the attributes because let's say most of the attributes that are initialized within the constructor and once we go within the constructor, we can see, okay, this is this, it points to this buffer and we're actually tracking all the references to different offsets within this buffer with the size of the reference. So basically, this defines the layout of the type. We are filtering types which contains only three year attributes or more because there are actually many types. When we first started to run this on a smaller sample set, we got many types which are contains zero, one, or two attributes and this is not really interesting for the results. So, we are filtering out small data types and left only the ones which have three attributes or more. One dump in C3 is we're very computing how far it is from the entry point in the total number of the cross references. For this, we are enumerating cross references to this routine and we used a breadth first search algorithm to find if we can reach an entry point from this routine and we limited our search to 100 nodes just to make sure we are not running infinitely and once we find an entry point, we are calculating the distance from the entry point in the total number of the cross references. When it comes to identifying our Advanced Encryption Standard new instruction set and GETSEC, we used a linear sweep approach in this case because it is faster. So, we always start processing with your section where entry points point at and your Wisconsin first half megabytes of the section. We use a linear sweep disassemblier and once we identify GETSEC and Advanced Encryption Standard instructions, we look for 15 instructions before and after to see if all these instructions are correctly disassembled because this may be the case if we are, let's say you start disassembling instruction not from the right of set, but let's say from the middle of the instruction, we get your AS instruction and then there is a summer bad instructions so that means that we're probably not doing this in the right way. Regarding this usage study, we were checking up to 5,000 calls within their HIDB file and we were analyzing five preceding instructions to a call to see if there is a loading of the E6 register. We use some heuristics to find if the E6 register is being loaded with some value, for instance, it is a move instruction or load effective address. And once we identify all their calls which are loaded in E6 register, we compute their percentage and you're including this in the results. A few words about distributing IDAPRO. We were really surprised with the performance we got using IDAPRO because actually we were a little bit skeptical about this because when you analyze your sample or let's say your single sample analysis, IDAPRO is a kind of interactive disassembler. It shows you a lot of your dialogue boxes trying to choose different parameters for analysis. It does auto analysis and once your auto analysis is completed you need to enumerate through the routines within this IDB file and do this in efficient way. But however, the performance was really nice and it met our expectations. However, there are some problems first related to our developing IDAPRO plugins. Is the key. Is the key is complex and you're quite, probably the commentator. Are there any people in the audience who developed your IDAPRO plugin or X-rays? Does it raise your hands? Okay, not too many people. And also the thing is that your signatures of the functions can change from a version to version. So that means sometimes you have problems with the compatibility between different versions. Let's say either 6.8 or 6.9. As I mentioned, we had a really good performance on commodity hardware. We didn't have a high-end systems like about eight cores per machine and 10 gigs of RAM. And another observation is that most of the plugins were written for Microsoft Windows platform. Like they were using Windows API, they were using Windows types before making them work on Linux because our environment was running on Linux. We had to apply some efforts to program. Another observation is that most of the plugins are not made to scale. Like they are targeted in a single sample analysis as well as in IDAPRO. So that was one of the observations which were made. And interestingly, when we run IDAPRO with the plugins on the large sample set, or we are identified many bugs are in either in our code in other plugins. So it was a very nice test. Sorry, if you're interested in testing your plugin, please send us and we'll try to see how it works. And now I wanna say a few words about the HexRace Code Explorer. I, Anatix, we are developers of the HexRace Code Explorer. The first version was released on Atari Com 2013 and the latest version works on IDAP 6.9. But however, some guys approached me and they told that there are some bugs in the course. Or please, if you find the bug, you can go to the GitHub page and you're reported there. I think this was the most efficient way of reporting the bug. So there are numerous features in the HexRace Code Explorer. So originally it was designed to facilitate object-oriented code analysis and your position-independent code analysis. But in this research particularity, we were using only one feature which is extracting object-oriented type information from the binary. And right now I'm going to give some low-level information because the HexRace Code Explorer works with the HexRace Decompiler, which is using the C3, the intermediate representation of the decompiled routine. And here is the explanation of what actually C3 is. C3 is an abstract syntax tree which represents their decompilation. For instance, on the left-hand side or part of the slide, you can see an expression, variable two, variable three equals variable two plus variable seven. And on the right-hand side, you can see how this was actually implemented in C3. So you can see here, so if there is a mouse here, no, okay. So there is, oops, sorry. So there is an assignment operator which corresponds to a, which, oh, there we go. Yeah, so there is an assignment operator here which corresponds to this equal sign. And so the variable three is located on the left-hand side of this operator and on the right-hand side, we have an addition which takes two parameters, variable three and variable seven. So this is the way actually how C3 looks like in the decompiler and how it works with. So each block within C3 is of type CITMT. And CITMT actually is a base class for two subclasses. One representing expressions and the other one representing statements. So statements are basically where they correspond directly to the statements in the C language. These are the blocks or E4, while expression is something what has a type. So you need to make sure that the type information is consistent in the C3. For instance, if you're calling a function and passing it in an integer, then you need to make sure that their C3 block representing integer actually has a type integer. And this is the only requirement because the hexracer SDK allows you to do whatever you want with the C3, but you need to make sure that the type information is consistent.  Okay, so let's take a look how this is used in this research. So we did a type reconstruction. We reconstructed object-oriented types and for type reconstruction, there are two essential steps. First, we identify virtual table and the virtual table is related to our polymorphic types because if there is no polymorphism, there is no reason for a virtual table because all the functions, they are called directly. And virtual tables contains pointer to the functions which are actually overwritten and they're recovering virtual tables which gives us very interesting results about the type hierarchy. Another idea which we use is a type attribute identification, reconstructing type attributes. So for instance, here we can see that there is a virtual table identified in the IDB file. So it is here. And we can see that there is a cross references to this virtual table and this function. So if we go to this function, this is a constructor. So we get a constructor, we have this pointer and we have attributes in this object. So the idea is to reconstruct those attributes. This is actually what we are doing and we are obtaining this structure representing the type and its attributes and their virtual table as well. Regarding normalizing the trace, so here is the implementation of their filter item routine which does filtering of their blocks within their C tree which are not relevant. So we are skipping some cast operations, some helper functions and some other minor blocks. And for instance, here is the complete C tree of a function and everything what is marked here with black or blue is included in the C tree and everything what is marked with red is filtered out. And here are some general thoughts about like pros and cons of using Hexrace intermediate representation. The good thing is that you are working with your not platform dependent code because your Hexrace decompiler works with your 32 or 64 bit arm and you don't have to care about actual upcodes because you're working at higher obstruction level basically at source code level analysis. But however, there are some disadvantages as well. First, that the Hexrace is designed for single function analysis. Even though there is an option you can just run, okay, decompile the whole function. But what you will get is not going to be a really accurate decompilation because the results of subsequent decompilation may invalidate or change results of the previous decompilation. You can imagine like you have two functions, function one and function two and function two is called from the function one. So you first decompile function one, then you go to decompile function two and during the compilation of the function two Hexrace identifies some parameters that were not identified at the decompilation function one. And as a result, the signature of the function is changed and you need to go back to function one and decompile it properly so that you can see a proper call to the function two. So this is one of your limitations and constraints we are faced at this research. Now let's go to your results. We are obtained by analysis of all the intermediate representations we obtained while running this in the cluster. So I included a subset of the results. The complete version is available in the Black Hat Talk and the Black Hat Presentation. So first we started with preprocessing of 7.8 million samples and as it appears like only 31% of them were not backed. So this is the sample set we were using and out of this 31%, 13% was only using our Microsoft Visual C++ as a compiler. I think that the most popular packer was UPEX and its modifications. Here is the results of the use of this usage study. So on this slide we can see a table with the top 10% of this usage. Like how many percentage of the calls that we're loading E6 register. Well actually I don't see here any interesting like a function of loading E6 and their prevalence of the percentage. Like we have a maximum at 4% of calls and however we can see also high value of 64, 66%. So from this point of view we tried to use this E6 register identification tour to be able to automatically filter out our code which is using object oriented model or not. So for this reason we've chosen a threshold to limiter which is around one or 2% to be able to filter out. If we have percentage less than this we consider that this is not object oriented code and we're not going to consider this in the research. This is the information on the top 10 repeated C trees. So in total we have eight million C trees dumped and obtained from the samples and some C trees are like very frequent. We had 40,000 C trees which is 0.4% and this most one is 14,000. And if we take a look at their unique C trees we find that 30% of the C trees are shared between different samples while 70% are pretty much unique or pretty much unique tour-specific samples. So this is an interesting result which requires further investigation because it's not sure. I was really surprised with this ratio because I was expecting to see a larger portion of the C trees shared between the samples. Maybe the internalization is not well configured and we need to filter out more results. So this is regarding the unique C trees and however if we take a look at this approach but from the point of view of the samples with repeated and non-repeated C trees we can see that 9% of the samples contain 9% of the samples that are shared between each other C trees while 91% are not sharing. When it comes to C trees reaching entry point and their average and standard deviation of their depth we can see that roughly 50 C trees that we dumped were able to reach the entry point and the average depth is 5. So that means that most of the C trees we dumped is 5 calls away from the entry point. So this result might be interesting for let's say those who are developing emulators because sometimes when you're emulating the code you need to know where exactly you need to stop during the emulation and having an average depth of 5 that means that you need to emulate up to 5 calls to be able to reach those routines which we were dumping. And here is their C trees with their cross references to them. We put here maximum 10. The top 10 C trees and they're like we have one C tree with their 11,000 cross references to it so this is a pretty high and this is an outstanding number. As you can see in this table this number of parenthesis are steadily decreasing. And you can go to the original version of the presentation to get complete results. In the last part of the presentation I'm going to speaking about the validating and their methodology we were using because we run these algorithms on a large set of malware and then we wanted to apply the same algorithms to a smaller set which we know is related to be able to validate and see how well it works in this case. So here is their timeline of the modern C++ or malware used in targeted attacks. There are different groups identified Stuxnet, DoCore, Equation and Animal Farm. So for this case we choose your Animal Farm case study because we know that this malware is written in C++ and they are all related. And there is a very nice presentation on this topic presented at ARICON last year, Totally Spice where you can get all the complete information on their analysis of this malware family. But a brief introduction to it was discovered by Canadian agency CSEC as an operations law globe and it was written in our Microsoft visual C++ so that's actually what we wanted. So we applied our algorithms to this malware and we tried to compare and find some similarities. For instance here we can see the comparison of Casper's and Dino's virtual tables found in the outputs of our tools and we can see that there is some intersection. There are two objects here the run key in Autodale which are shared between those two samples so that means our virtual table approach or identification works here and there is their definition of the run key so the run key defines how malware communicates with the registry. This is a base class which defines subclasses which define particular implementations like the malware can access register directly by API using common prompt using Windows management instrumentation classes and they're from REC command and Autodale defines how malware actually removes itself from their system after infection and there are also different subclasses related to API using Move file to remove itself or using common prompt or using Windows management instrumentation classes and the usage of this each particular subclass depends on the parameters of the configuration parameters of the malware. So we did identify the similarities and we tried to reconstruct their object attributes so here is their left hand side you can see the casper's run key constructor and on the right hand side you can see Dino's run key constructor and there is a significant difference here because in case of the casper each object is created depending on their configuration value so for instance if there is a AV strategy run key API is set it creates only an object with run key API otherwise if there is a AV strategy run key registry it creates only an object with run key registry while in Dino's case it creates one large object which contains other objects as sub-objects and if you try to reconstruct the virtual tables we will see that they are not equal and in this case our approach doesn't work let's go further and we compared also Dino's virtual function tables again we can see that there is an overlap in virtual tables so here again our virtual table algorithm works well we are able to identify the intersection of the same virtual tables so that means the share similar code and if we go to the constructors they are pretty much similar not 100% but they are very similar and we try to construct their attributes of the structure we will get the same structure so here it works well and it's very nice so here is the table which summarizes the shared types between different malware families we can see that most of them share a lot of code between each other and it's also interesting to see that there is some small difference between CASPER and Dino it contains implementations of run key VMI and auto del VMI and CASPER appeared after Dino that means it was an evolution of the Dino so they reused code in Dino by adding additional functionality and here is the same information but from the different point of view the number of their shared types are in this malware families we can see that Dino and CASPER are highly correlated so they share a lot of reused code so to conclude the presentation so what we did we actually distributed IDEPRO so we demonstrated that IDEPRO can be run in large scale in batch mode but however there are some actions that we need to apply to make them run and there is a call for plugin developers it would be a nice option if you also add a special batch mode for your plugins not only for interactive work with the user and of course if you want to test your plugin on millions of samples you can set it to us and we will see how it works and we also will be releasing a special version of your code explorer for NordStack conference we are going to release it your Control-Z yes or actually a few things like we were actually going to release it before but we need to do some code clean up so as Olivia told it couldn't take quite long but we hope it will be done sooner in the 21st century and there is no Control-Z in X-Rays code explorer so this is actually where we are going to address in NordStack edition of the plugin so this will be one of your biggest improvements and we also will add a major feature extracting type information in C3's JSON format because right now it's dumped into your custom text file which is not really good for future work which we are going to do like what we would like to concentrate on maybe you have different feature requests so please let us know like one of them is pardon matching for C3's let's say you can find in the idbeer functions by C3 pattern you say you input this C3 pattern and it outputs all the functions which meet this pattern this might be useful for the vulnerability research so not only use the idbeer and X-Rays code explorer as a base tool but also use the OpenRail project released by CRUSH and if you have any bugs and features so please submit them if you are interested in the project you can discuss code explorer on a RihindsGear channel so we are really looking for the feedback and before I conclude the presentation I would like to say some acknowledgments to our employers and to our Elfa Guilfanev and X-Rays team for their support of this research without them this research wouldn't have been done so thank you and that's it thank you for your attention if you have any questions


Hi everyone, Joe for Jasper's case breaks.com coming at you with 2021 tops definitive collection baseball pick your team number 10 From a fresh case a very big. Thanks to all of these folks right here for getting into the action Appreciate it. A lot of teams are ended up in those R&B's number blocks random number block Logan you ended up getting the official last spot mojo So here are the results of the number block So it's just a serial number break for that group of teams right there and only for one box pick your team 10 and the results of the Of the randomizer the randomizer itself actually a video is in a separate video and Same thing right here for those teams right here. All right. Now. Let's see which box we're gonna get I Mean I think definitive usually has the the usual selection of Jeters and Mariana Rivera's in this mic. So I think that's what makes them a pretty pricey team So we got box ABC So we're gonna roll the die and let's see which box we're gonna do one two three four five six on the dice roll and it's one That one right there and let's see what we got. Good luck Here We're gonna lead off with Roberto Alomar 46 out of 50 blue jays Paul with the blue birds We got Frank Thomas the big hurt 21 out of 25 White Sox are a number block team So White Sox one is Mike Sommich.  She'll like it too So his defining season I like looking these up his defining season was 1993 that must have been his MVP or he won a couple that was his first of two in fact He went back-to-back MVPs So in his MVP year He had a 317 batting average 36 doubles 41 home runs 128 RBIs Walked 112 times only struck out 54 times That's a pretty nice season all-star silver slugger in that first MVP year. Who did he beat out? He beat out he easily beat out Paul Molliter Who had hit 332? 22 home runs and 111 RBIs 22 steals. He had a great season too. I mean he was second Feel like Frank Thomas's following year was even better. He hit 353 fewer home runs but hit 353 38 home runs 101 RBIs couple stolen base 109 walks. That's pretty good. I Will ship you above. Yeah. Yeah, just Just text defining season to the number below on your screen and and you'll get a free trial bottle of eugenics Trust me. She'll like it too All right. Next is Luis Garcia Yeah, Logan makes how much his wife will like that card too I'm certain she will rookie autograph Luis Garcia Eric 7 out of 25 with the Nationals We got Max Muncie hit a nice home run tonight two-color patch and autograph That'll go to Kobe Chan and the Dodgers and There's Paul Molliter The guy that Frank Thomas beat out in his defining year five out of 50 Piece of his lumber who has a dog Kobe Chan is the Dodger who has the Royals Royals are a number of log team So Royals five that's in that big group over there Royals five Jonathan With Royals five. There you go. Nice Molliter piece of his lumber nice autograph The gold frame is gonna be Another White Sox 28 out of 30 Luis of Robert So that's White Sox eight Wow, that's also for Mike's homage. There you go, Mike It does have the odds with the numbers a couple more to go. Wow a big e One of one will the e of Wilson Contreras for Steve Locke and the Cubs Steve you almost you almost might be forced to To go on the Contreras chase right here. Oh, sorry. That was Brewer's Molliter. It doesn't matter. It's on the same list Yeah, it's on the same list still goes to Jonathan Number five There's Wilson Contreras and out of fives and under get the All aboard the Big Hit Express Nice and the final relic is gonna be Cavendizio for the Blue Jays 18 19 out of 30 Paul picked up the Blue Jays straight up nice two-color patch And Paul if you go to mlb.com slash authentication You can punch in these series of letters and numbers Which it looks like JC 715469 and then you'll be able to see exactly what date this came from. Maybe he played a game Maybe you look at the box score. See how he did could add a little extra Alexa narrative a little extra story to this card There you go folks definitive collection in the books we got another one box break In the store right now Jaspies case breaks calm keep chipping away at it and we'll make that happen next time I'm Joe. I'll see it for that one next time. Bye. Bye


All right, check it out. It's race day Oslo Half Marathon. I am super excited, quite nervous. I'm running and my dad is running as well. And he's a veteran. He's been doing this race many times in his life for four years. He's run 15 or 16 marathons and lots of half marathons. And this is like the biggest race in Norway, also half marathon. We're both excited. We both have our goals. We both have our injuries. We're very worried, but it's going to be a good day regardless. So stay tuned. All right, so we are at the race now. I'm starting in about one hour and 10 minutes. Yeah, for the next half hour, I'm just going to chill. And then I'm going to do a little bit of a warm-up and then get in position and then off we go. My dad is starting in a later wave. We're both super excited. We both have a little bit of an injury though. I'm injured in the hamstring. I don't know if I will finish this race, but I surely will try. All runners have injuries and bad feelings or wounds and pain. Small pains here and there. That's natural. Very common. That's very common. And it's a bad sign, a bad omen, if everything is perfect. Because they aren't. There's always something. So it's a couple of days after the race now. You just saw the start. And I want to take you through the race, talk a little bit about my experiences, what happened, how I executed the race, how it felt. And I've got lots more footage from the day itself coming up as well. So let's get right into it. So you saw the start. Lots of people. Great event. Great vibe. Biggest race I've ever been in. My second ever half marathon. I loved it. It was really awesome. I was super nervous leading up to it. If you saw my pre-race vlog, if you didn't, you can check it out here. Some of my goals. I was nervous. I had an injury potentially in my foot, so I was worried about that. But you know what? It all went really, really well. I'm extremely satisfied. I ran a time of 1.33.38, which is like smack in the middle between my A goal and my B goal. So I knew I was probably in like one sub 1.30 shape if it was a flat race. But knowing the hills, I was thinking to myself probably I'm not going to be able to make that A goal. My B goal though, sub 1.35, that was attainable. I felt that was realistic and that's what I did 1.33.38. So super happy about the result. It was tough. It was painful, but it was a lot of fun. I started off and sort of felt my way into whatever felt naturally sustainable for me. Like without really thinking too much about the watch or my heart rate, I was like just going and feeling like what feels like the right pace. And after a couple of kilometers, I quickly sort of realized that you know that 4.15 pace that I needed to hold if I was going to do the sub 1.30 probably wasn't sustainable. And my heart rate was up at 1.90, almost from the get-go. It peaked on like 1.98 later in the race, which is like ridiculously high if you consider the fact that when I go to the track, I do workouts like VO2 max workout at between 1.92 and 1.98 BPM. My max heart rate is probably like 2.05 or maybe as much as 2.10, I don't know. So I was like up there at near my max heart rate already. I was like, oh my God, I can't sustain this according to the heart rate. And I wrote down to myself prior to the race, I'm not going to go above 1.90 until possibly later in the race. But I knew by my breathing that I was not above my lactate threshold. At least I thought so. And it turns out I was right because I finished the whole race at that intensity. I was not breathing heavy. I felt strong, felt good. This is sustainable. So that shows that my heart rate goes quite up during a race. So the excitement of the race, the people, it sort of drives my heart rate up 5 beats or so like that. So that's interesting. And of course, if you want to see all the juicy details about my heart rate and my pace and read my race report, you should go to Strava where I have a profile and you can follow me there and you can watch all my training and etc. So there's a link to Strava in the description. Check that out. So I continued. The first hill was no problem. I was running steady. I was running good. I felt pretty good. I mean it's tough, but I felt good. And I was able to maintain like a 430 type pace per kilometer, which equates to a sub 135 half marathon. So I was happy about that thinking I'm going to maintain this pace more or less. The race went on. I came down to the flat section of the race. 10 kilometers in. I was starting to feel it halfway through. You know, I was like, there's something going on in my legs right now, which is expectable. And I had a bad experience at Krukskugun Halmarathon, which is my last half marathon. You can watch my race log from that here where I actually ran too hard for the first half. And I was worried that might happen something similar again, but I was still feeling confident that probably wouldn't happen. I felt like I was really pacing myself well. So I was able to speed up a little bit on that flat. Coming into the middle of the race where the start area and finish area is, you sort of come in there again around 13 kilometers in. And that's where Matt was waiting for me. And he filled me as I passed him by. So check out the footage. I was feeling good at this point, except for a little bit of a sting that I had. I struggled a little bit at the aid stations. I hadn't practiced that much grabbing, you know, cups of water and drinking it on the go. So I wasn't able to get as much water in during first two aid stations. I was still taking my sugar mix, maltodextrin and fructose that I had with me in a little pouch, taking that at each aid station. And I think I took the third squeeze like 12 kilometers in, a kilometer before the next aid station. And that's when my sting started developing. So probably too much sugar, not enough water. And but then as soon as I got the water at the next aid station, I took a moment to just stop and just get down some water just to really get it in. And it resolved within like a minute or two. So that was good. And I took it easy for a little while because I was really, you know, excited. And it was all the people there. And I was running hard. And I felt like, you know, now's the hard part of the race. Like I'm two thirds in. It's time for the hard part. And it's also the hardest part in terms of climb is like a crazy climb going up there. So I was like, I took a moment to just slow down the pace a little bit. You can see it on the Strava file that my heart rate goes up a little bit around the 13k mark, which is like where all the people are in our masses. And then it goes down a little bit. My pace goes down a little bit where I'm just telling myself, it's just me in the pavement. It's just me in the pavement. I was like almost meditating, trying to just like get a hold of myself again before attacking that last section of the race going up, up, up, up. And it was amazing. Lots of people cheering me on, children high fiving me. It was fun, but it was hard. It was hard. I was trying to suffer hard at this point, but I was sort of knowing, Hey, this is this is going to work. I'm going to make it. I was my, my, my average pace was showing that I was on 430 pace and climbing like up to 431, 40, 32. And I had to stay at 430 or lower in order to go below 135. So I was worried, but I knew I was going to go downhill on the other, other side of that hill because it's up the hill and then it's coming back down the same route back to the finish line. So I was struggling up that hill and then finally I was opening up on that downhill. It's not as deep down. Well, there's a little bit of a steep downhill and then there's like a, you know, like a slack downhill going into the, the finish line where I really was able to push the pace quite a lot and, and reel myself back in to get down to like, I don't know, 428 per kilometer, pace average or something like that. Here I am on the final stretch before the finish line. And so there it is, 133 38, super happy about that result. I mean, it overall in the grand scheme of things, of course, it's not like that's like, whoa, like an impressive time in that sense. But for me at this point, it's certainly my personal best, certainly impressive for me at this point. But I know I'm going to go further. I'm going to, I'm going to not only go further in distance, but I'm going to go faster on that same distance as well. So I have big goals for next year. And I'm going to go in faster. Not sure if I'm going to do the exact same marathon, probably not thinking of doing a marathon as well later next year. We'll look at that as the new year comes around, we'll talk about my plans and my race goals, et cetera. But, you know, I'm going to lower that personal best substantially over the next few years, that's my plan, but super duper happy about that.  So I feel like I was really executing perfectly, really. Could have done a little better when it comes to the hydration, no, but that's very difficult with the cups and getting the water in without like getting it everywhere and losing it and not being able to swallow. I was very conservative in the taper because I was worried about my injury. So I ran very little. I don't know if that's a good thing or if I was would be better off with a little bit more training, at least in the two weeks out type of time range. Who knows? We'll find out over the years as I experiment more and more with the taper and racing, et cetera, gain more experience. I ran the Ultra, the one 2.5 shoes. I have a review of it here. Pretty great shoes for running half marathon, really lightweight. I would probably enjoy a little bit more cushioning, I think, at least in training. But whether or not it was optimal for the race, I don't know, it did feel pretty optimal, though. I mean, I ran a great race. So I guess it worked fine. At the finish line, there were a lot of bananas. Everyone's eating bananas. At the finish line, they provide bananas there, which is just awesome. And of course, carbohydrates.  Of course, I had my own bananas with me, ripe bananas, because they weren't actually ripe. The bananas, they serve there. And after I finished my bananas, we went down to the same place where Matt's watched me pass, and we were waiting for my dad to pass. We didn't know whether or not he was actually going to pass us, though, because he had an injury pre-race. During the race, we weren't sure if he had broken down after three kilometers, but we were hoping. And sure enough, he came along sort of limping a little bit with his injury, but he was running pretty good at like 13 kilometers. Check it out. And here's the footage from him finishing the race.  He's done a lot of half-marathons in his life, and he's one of the slower ones. He pulled through despite having an injury. He had some things to say about that after the race. Check it out. All right, so my dad made it through. Yeah, I think like this. What do you call it? You call it the crown? He had like a strain in the hamstring, but it was close like this for 21 kilometers. But he pushed through. Yeah. Grinding through. Oh, yeah. And what was your time? Two and a half hours. Two thirty, fifty-five. So that's pretty good.  Slowest ever half-marathon? Slowest ever. But yes, I did finish. That was fantastic. Yeah. Next year, 220. 218. 218. Yeah. You heard it. Yeah. It's a fantastic crown for the streets of Oslo. It's amazing. Really, really nice atmosphere. 16,000 runners. So that's it, really. Mads, of course, was with us, crewing, filming this, some of the footage, and, you know, taking care of our stuff, carrying it in his backpack before and after the race, etc. So that was really cool having him with us. And I look forward to seeing him, of course, along the course and the finish at the finish line. Overall, just an absolutely epic day at Oslo Marathon. It's a big event. It's sent live on national television here in Norway.  I'm not saying that I will ever be up front on a road marathon or half-marathon. Well, who knows? Who knows? Probably not. And I'm more geared probably towards competing, eventually, in the ultra marathons and the mountain and the trail running and that sort of thing. But I do like road running. I like the purity of it. It's just, it's just you and the road. There's no obstacles. It's just about running. Feet going like that. Whereas in trails, there's more technical stuff and there's other variables and there's nature, which is why I actually like that more. But I do appreciate road running as well. And it was a really good day. I'm really happy about my race performance. I'm satisfied. As I said, check out my Strava profile if you want to know all the details from the race as well as my training. I also have an Instagram account where I post pictures. There's a link in the description for that as well. You should follow us on Facebook, link in the description for that as well. And of course, subscribe to this channel and check out some of the other cool videos that we have here from, you know, all things regarding running. Alrighty. Thanks for watching. So glad to have you with me here on this channel. Leave a comment, please down below. Tell me what you think about my video, about my race, about anything you want to talk about regarding running really. Let's keep going. My recovery is on. I'm having a couple of weeks now of recovery and then it's back on to training, building my fitness for future races. All right. Bye now.


I'm Jake. I'm Jason. Earlier in this conference, we released, is this a conference, actually? Is this a conference, or is it just like an event? I don't know. Let's call it a conference. Earlier in this conference, we released tooling.report. This is a website. It looks at all the different JavaScript bundling build tools, shows the things they're good at and the things they're not so good at. But in this session, we are going to write our own plug-in. Right. So cruising around the web, we see lots of sites with performance issues. And a lot of times, these can be solved with something simple, like a preload tag for a web font or some selective code splitting. But the thing is, when these are controlled by a build system or even deep in layers within a build system, making those changes can be really tricky. So supposedly, simple changes can start to seem impossible. Yes. But once you're familiar with writing build plugins, sure, that means you can write build plugins. But it also gives you insight into how the build tool works. It makes it easier to debug things when they go wrong. And of course, it means that you can go and help with community plugins as well and do PRs and add features and fix bugs, that kind of thing. Definitely. So we're going to create the same plug-in for rollup and then for webpack, just because those are the two most popular and growing build tools that we've seen. So what are we actually building? We are going to build a service worker plug-in. Of course, we're going to build a service worker plug-in. So the actual subject matter of the service worker plug-in itself doesn't matter all too much. We picked this because it touches on a couple different parts of the build process. So it's one of those things that seems simple on the surface, but there's actually quite a bit of complexity when you dig in. Right. So here's the app. Now, we're not going to go through the whole app, but it has JavaScript, CSS, images, and HTML, all of which are part of the build process. Right. So the bit that we care about for this session is specifically the registering of a service worker. So the URL for a service worker should never change. And this is because we need to be able to go and check for updates at that URL. But it'd be really nice if we didn't have to duplicate that URL here and in our build configuration and on disk. And instead, that was just filled in by the build system. So here's a code for the service worker. We're not going to go over all about how service workers work. There are articles for that. But here are the things we need for this build system. We need a list of URLs to cache. So this should be our HTML, CSS, JavaScript, and images. These are the things that a native app would ship as part of its bundle. Now, we can't just hard code these because the build system is going to change their file names. It's going to make them cacheable. It might also code split as well, which is going to generate whole new JavaScript files that we didn't know about in advance. So because we can't predict it, we need the build tool to fill it all in. Right. So we also need a version for the cache. And this is because when we're installing a new version of the app, we want to use a new cache. That way, we don't potentially disrupt any current version of the application that's in use maybe in another tab. And so because of that, we need the version to be unique for each given set of assets. Yes, and if we don't change the assets, the version number should stay the same. But if we say, like, update the HTML, the version number needs to change. Easy, right? So the challenge is that we really want the service worker to be part of the main build. We want it to potentially take part in code splitting. We want to minify it just like we would our application JavaScript and also apply any other plugins and optimizations that we have to that code. The thing is, it also needs to know about the final build result. And that is only available once the build is finished. OK, here is how I would do this in rollup. We need to get that service worker URL, so I'm going to import it. Now, it looks a little bit magic. It's got this sort of fake URL scheme. This divides people, but I really like this because I want this to look weird if it's a special part of the build. Because it means you look at this code, and you're like, huh, that's odd. And it's like, yes, it is odd. The build tool is going to do something special here. And it also means if the build tool doesn't do something special, it is going to fail really, really early. But like I said, this divides people. Jason, how do you feel about this? I mean, I like this for the fail early reason. Like, that's a nice thing. If you pull this code into a new project, it's going to tell you, hey, this isn't a thing, which is sort of exactly the message you would want to receive. But I also like that on the right-hand side of that colon, it's still just a module path. So it's really clear for you when you're visually looking through, this is the special part. This is the regular part. Yes. So all we need to do now is create the build plugin, I guess. So here's the config, the rollup config. You've seen this before if you've used rollup. I'm going to import this new plugin and add it to our list of plugins. Now all we need to do is write it. Rollup has a one pager on how to write plugins. But it's quite a long page. But I have to say, the API is really well designed, and the documentation is really well written. So let's get started. This is what rollup plugins look like. It's a function that returns an object. It has a name as part of that, and that's for error handling and logging. But everything else, the rest of the object, this is methods that are called throughout the build. They're essentially callbacks. Rollup calls them hooks because they're hooks into the build process. The first hook we're going to look at is resolve ID. This is called for every module that's loaded. So it's going to be called for the main file there, but it's also going to be called for this import and everything it imports, including dynamic imports. The ID you get is the raw unedited import ID. So it was going to start with the dot slash in this case. And in this case, it's going to include our special made up URL scheme as well. The second argument is the importer. This is the path to the module which did the importing. And this is the only time you get this piece of information. The job of the resolver is to return a full absolute ID that doesn't need the importer anymore. So any relative paths should be turned into absolute paths. Now, if you've used rollup before, you'll know that it doesn't support node modules out of the box. You need to add a plugin to be able to do that. It's an official plugin, but you still need to add it. All that is is a resolve ID hook. And it sees these bare modules and it goes, oh, OK, I'll try and find that in node modules. But also, if you want to create path aliases to make them work in rollup, all you need to do is create a resolve ID plugin. So we don't need to do anything with this first import. We just need to handle the second one with the special URL. So here we go. The first thing I'm going to do is just exit early if it doesn't start with that prefix. Actually, we're going to do that sort of thing a few times. So I'm going to pop that in a variable up here because it's going to come up time and time again. So there's an early exit. And it means if it doesn't start with that prefix, we can hand it off to other plugins or the default rollup stuff. Now I'm going to remove that prefix and resolve it. This resolve, this is a rollup API. And what it says is, take this ID and go and find it for me. It couldn't do it by default before, but now we've removed the prefix, off it goes. And that's going to actually pass it back through all of the plugins that do resolving, including our own. But we're going to ignore it because it doesn't have that prefix anymore. But it means if you have a service worker in node modules or if you have path aliases, it'll all just work. So if there's no match, just bail. And that will cause an error eventually because it means that we couldn't find that service worker. Otherwise, I'm going to add that prefix back on so we can pick it up later on. But now we have an absolute path to the service worker. We have achieved a thing. We can try and build it. Here we go. Wah, wah. It's not going to work. But you can see in that error message it's failed because of that special prefix, but the rest of it is an absolute path to the file as it is on my system. So all we need to do is tell it how to load that script. This is another hook, load. It gets the ID as it's been fully resolved. Again, we want to ignore it if it doesn't have that prefix and let other plugins deal with it. And what we need to do now is return the content for this module. So we could just actually make up a module. I'm going to do that. I'm just going to return hello. And now we've done that. The project will actually build. It's going to turn that fancy custom import into hello. This is kind of silly, but just knowing those two things, knowing how to resolve and knowing how to load, you can do all sorts with that. You can do all sorts of cool code generation, loading of a file, that sort of thing. Like it's not necessarily what we set out to do, but you could totally see how this is basically like a Constanced plugin. Yeah, exactly. And I've built Constanced plugins, and that's exactly how you do it. But let's improve on that. So instead of just returning hello, we need to bring the service worker into the build system and return its URL to bring a file into a build. I'm going to use emits file. This is a rule of API for adding stuff into the build. I'm going to say this is a chunk. And that's telling rule of like, treat this like a JavaScript entry point. So it should be processed by all the same plugins that deal with JavaScript, like minifiers and code splitting and all of that sort of stuff. It could also be asset. And if it's an asset, that's like an image CSS, I don't know, WebGL shaders. Anything that's not JavaScript essentially, you can process in this way. I'm going to give it the ID, and that's telling rule of where to find it. So we've removed the prefix again. And I'm going to give it a file name. And this is not something you would usually do. What this does is it overrides the naming system that rule of would usually use, because rule of would usually add a hash to this file. But with service worker, you don't want its name to be changing. You want it to be in a static location. So that's how we achieve that. Actually, I'd say like, this is something that you would offer to the user as configuration. So I'm going to do that. I'm going to add an option object to the top here and I'm just going to like change the variable down here. So it's using that. So that means the user can change the path of the service worker, the file name, all of that sort of stuff. Now what we need to do is get the URL for this file and strap in, it's a bit of magic. Woo, there it goes. So I'm returning the import meta and then this sort of magic string that ends in a file ID. And we've got the file ID from emit file. Rule of will see this and it will turn it into a URL for the resource. So here's what that looks like. Here's the input and here's the output. So it's created this URL object and it does that so it can create a relative URL from this script to the service worker because on the web, if you've got an image tag with a source or whatever or when you're calling service worker.register by default URLs are relative to the page but rollup doesn't know anything about the page. So it creates this script relative URL. You can actually configure this in rollup. So you can tell it, hey, all my stuff is actually at the roots of the origin. All you need to do is just add a slash to the start. And obviously that will clean up the code a bit. It just doesn't do it by default. But also in our build, we've got the service worker. So that bit is working. All we have to do now is deal with these two things, the version and the assets. We'll handle the assets first. To do this, we need to know the full details of the build. Because we need those hash file names and all of that sort of stuff. So we need a hook that is towards the end of the build process. We're gonna use generate bundle. This happens just before things are written to disk. The options program there, it's the output options. It's not very interesting for us in this case. But this is where the party is at. This is full of interesting details about the build. It's a JavaScript object, where the key is the file name and the object has details on the file. So this is an asset. So you can see it's got the file name and it's got a buffer representing the content. But for chunks, look at all this stuff we get. This is amazing. It tells us things like the imports and the exports of the file. I've used this in other plugins for if I'm generating HTML and I'm putting a script tag on the page. I know that that script is just going to import other scripts and I want to preload those things. Well, I can do that here. This will tell me which things it's going to immediately import so I can turn those into preloads. Anyway, loads of interesting info. We're gonna make use of this. In the generate bundle hook, I'm gonna get that service worker chunk just from that output option because that's the file name of it. And then we need to figure out which assets the service worker needs to know about the things that it wants to cache. Gonna iterate over the bundle, get the values and I'm gonna get everything except the service worker item itself. If a service worker caches itself, like it doesn't cause the world to end like you might imagine, it's just a bit wasteful. We definitely designed around that. We were worried people would do that so it's actually okay. But this is a point where people might want to configure the things which are actually cached by the service worker. So what I'm gonna do is add another option here, filter assets returns true by default and I'm gonna call it down here. A slight complicating factor, we want these paths to be relative to the service worker itself. And unfortunately, rollup doesn't give us access to its URL resolving magic stuff. I have filed an issue for that but we're gonna have to do it ourselves this time. So I'm gonna use nodes, posix, path resolving stuff. And the only reason I'm doing that is because posix paths have the forward slash and that's the same as URLs. So we can rely on that. And then down here, this is how I create the relative URLs. This is done by saying, give me the relative path from the folder that the service worker is in to the item from the build. And now we've got that. I can prepend that to the file. So this bundle object is live in that any changes I make here are actually going to be written to disk. So here I'm just taking the code for the service worker chunk. Adding that assets line to the start. I'm using JSON stringify to correctly escape everything. And now that will be part of the service worker. And if we build, there it is. There it is in the output, the assets. We've created a problem though. And this is something that catches a lot of people out when they're doing service workers. It's this, it's easy to think that the service worker is caching files but it's not, it's caching URLs. And we don't want .html in our URLs. We don't want index.html in our URLs. So we need to fix that. Back in our plugin, this is where we were before. How can we solve a problem like this? Regular expressions, of course it's regex. And I'm not going to go through how this works. I see a back reference in there, I don't know. I love back reference. I often say that regular expressions are right only. Yeah, you can't read them again afterwards. But trust me, this is removing the .html and it's removing the index.html as well. This could land us with an empty string, which is a problem, because that will resolve to the service worker script itself. So in that case, just gonna output a dot. And that solved the problem. There we go. Index.html there is replaced with a dot. Okay. Last thing, just need to deal with that version thing. Thankfully, we're most of the way there already. This is going to be a hash. So I'm going to include nodes hashing stuff and then down in the generate function, create one of these objects. I picked char one because I've heard of it before and it wasn't MD5 and I'd heard it was worse. Yeah, this is very secular. I mean, yeah, security doesn't matter in this case. It's just for hashing. Oh, don't quote me on that. Like security generally matters. JGARTABLE, security doesn't matter. Just, what a quote to take out a context. Okay. So now here I'm adding everything to that hash. So it's either the source or the code, depending on whether it's an asset or whether it's a chunk. And now I can get a hex version of that hash and include it in the old output there. And that's really it. That's it. We can see that the hash is now in the service worker. I wanna, there's a couple of caveats here by editing the source right at the end there. That was after source maps were generated. So we have just broken source maps just for the service worker. I've filed an issue with rollup because I would like this to be an easier way to do this. So hopefully they can add something. We could probably work around this by adding those blank lines in a transform step so we can fill them in later because that will, those extra two lines at the top will at least be taken into consideration with the source maps. I don't know, it would be good if those are proper way of doing this. But anyway, that is it. That is the full plugin. So, Jason, how would you do that in Webpack? Right. So building plugins for Webpack works a little bit differently. And to illustrate this, I'm gonna use a fairly similar approach and naming to what we saw in Jake's rollup version. Much like with rollup, the place where we're gonna start will be the bundler config. Now, a typical webpack.config.js file specifies some entry points, which are gonna be the modules where bundling starts, some options for the format and location of files generated on disk and plugins, which is what we're concerned with today. All we need to do is import our plugin that we're going to write and then add it to the plugins array. At its core, a Webpack plugin is really just an object with an apply method. And that apply method gets past the compiler instance. So wait, one second. So this is like, I see you're using classes here. So in rollup, we were using a function which returned an object, but in Webpack land, it's all class-based. So technically, Webpack does not care. It just needs an object with an apply method. But all of the Webpack core modules are classes. And I think as a result of that, all of the ecosystem plugins tend to be classes just because when people import it, they expect to instantiate it with new. You could totally write a Webpack plugin that was a function that returns an object. I was even actually tempted to do that for this talk to make it look more like the rollup plugin, but we'll stick with the thing people actually do. All right, so if you remember in rollup, plugins define special hook methods that get called in response to events. In Webpack, we do the opposite. Your plugin taps into events. And so what we wanna do is we wanna handle a special service worker colon import prefix. And so to do that, we need to tap into normal module factory. And just to preempt, because everybody gets confused with this, normal modules are source modules. These are the code you write, the code you get from NPM, code that you're going to put in your application. There are other types of modules, which is why there's other module factories for things like loaders. Basically, they're more on the infrastructure side of things. So what we're looking to do here is resolve our own import. We use normal module factory. And so similar to with rollup, anytime we tap into something, we wanna pass it a name, which is the name for our plugin. And this gets used for debugging and logging purposes and also when taking performance profiles. So within normal module factory, we can now hook into Webpack's resolver. And this gives us a reference to Webpack's own resolve function, which is the thing that sort of handles finding modules and loading them from disk. But it also lets us return our own custom resolver function, which gets passed a dependency description and a callback, cause it's asynchronous. And so this function, we can call the original resolve function, we can do something custom, we can combine the two. And to make things easier, we're just gonna take those three bits of information that we have and pass them to a new resolve ID method that we're going to write. And that's sort of everything we needed from apply. So our resolve ID method is gonna be called for each import specifier in the app with a description of the dependency, that original resolve function from Webpack and the callback to call when we're done. A dependency description is an object with context and request properties. Context is the directory of the module that's doing the importing. So in this case, dot slash. And request is the unmodified import specifier. So in this case, it includes our service worker prefix and also the relative path to that file on disk. So similar to roll up, the first thing we wanna do is figure out whether this is an import that has our prefix that we want to handle. If it isn't, then we can just call Webpack's original resolve method. It will do what it would normally do since there's no prefix, we don't really care about that module. But if it does have the prefix, we need to remove that prefix and then pass it through resolve. And so if the module can't be resolved, we can just forward that error up. That will break the bill. There will be an error in the console saying, hey, couldn't find module SW slash index.js. If it does work, then it'll pass us back a new dependency object where the request property is a fully resolved disk path. The thing is we don't actually wanna resolve this module to a path, right? We wanna resolve this module to a string somehow, right? This is the string that contains the location of our service worker. And so to do that, we need to sort of, we need to come up with a way of producing a virtual module, even though we're inside of the resolver. Essentially, we want to intercept this service worker prefix to request and then resolve it to a module that we create on the fly that doesn't actually exist that just exports the string URL of our service worker. And then one thing we do need to account for here is when we're working with URLs in Webpack, we need to make sure that we respect any Webpack public path magic global that has been set. And so this is either the output.publicPath configuration value or some value that has been set at runtime. Right, okay, so this is another difference from rollup. So rollup had to do some of the magic around resolving URLs and stuff. Whereas here, you've just got a string, which is like, where in the web route is it? Right, yeah, so like rollup, they have automation here to sort of give you module relative paths. In Webpack, it's a configuration value. And the nice thing is, because everybody uses that configuration value, all plugins tend to respect it. But so here, if we combine these two strings, we'll get the slash, which is sort of our default Webpack public path value. And that means that the result of this import will be the service worker's URL. The thing is, we haven't actually explained how to construct a virtual module yet. We know what we want to do. We don't know how to do it. So first we wanna start with that code that we wanna generate. And you notice here, I've parameterized the URL from that code string. And that's because we want to make that configurable. So in our constructor, we'll accept an output parameter. The developer can pass us the URL to use, we'll store that, and then we can inject it into our code by JSON stringifying it, which will escape it and wrap any quotes. And then the last bit is to take our piece of code and pass it back to Webpack. And to do this, we're going to use something called raw module. And we're gonna import this from Webpack core. So raw module is basically a way for us to provide code back to Webpack in the same module format as it would expect modules loaded from disk. And the reason why this matters is, we're going to use something packed with an object that has a source method that returns a code string and an identifier method that returns some sort of a unique identifier for that module. In our case, based on the code string. And in doing this, Webpack will actually grab the code that we send back to it and avoid ever going to disk to resolve this module. We'll just use whatever we pass. So now we have our import resolving to the URL of the service worker, but that URL doesn't exist because we have not yet generated our service worker. Let's do that. For this, we need a new plugin hook. And so when we need a new plugin hook, we have to jump back into apply. We need information that's gonna be codified into the service worker, those version and assets globals. And to get this, we need to tap into something that happens at the end of the build once that information is available. So for this, we can use the emit hook. You'll also notice we're using tap async here. And this is because our emit hook is going to be asynchronous and we want to hold back compilation until we're done working. So the emit hook gets past a compilation instance and a callback, which we're gonna forward onto a new method just to keep things clean. In Webpack, each compile pass is referred to as a compilation. And the main thing that we're interested here is the generated assets, which are an object where the keys are file names and the values are asset descriptors. I recognize this. This is the same as the rollup bundle thing, right? Like we had in generate bundle. Yeah, it's really similar. I don't think I actually realized until we were sort of looking at these things side by side, they contain really similar metadata. So it has, you know, the dependencies and is this an entry chunk and all that information. It's also for both JavaScript and non-JavaScript assets. And in both cases, there's a source method on the asset descriptor that will return either a buffer with the non-JavaScript asset contents or a string with the JavaScript code. So we need a list of files for the service we're gonna cache. And to get that, we could just take the keys of the assets object. Similar to with the rollup plugin though, it's generally a good idea to make it possible to filter these assets before they get embedded into the service worker just to avoid a huge list. So we'll add a filter assets constructor parameter that a developer can set and provide a function that takes a file name and returns a boolean indicating whether that asset should be in the list. Now we have our list of asset file names. The next step is to calculate a version based on a hash of their contents. And so to do this, we need to pull in the create hash method from nodes crypto module. Then we'll loop over each file name in our filtered list and add its corresponding asset source, which will be that code string or buffer to the hash content. Then our version string is just going to be a hex digest of the combined contents of all of those assets. So that's the magic version global done. Now we can do the assets magic global. So as Jake explained in the rollup walkthrough, our assets array contains file names, but our service worker needs to work with URLs. And so to fix this, we need to pretend any configured public path value to the names in case we're deploying somewhere that isn't the root domain. Right, so this is again, this is gonna be simpler than it was with the rollup case because you've just got the string that you can add to the start. Right, so because everyone knows to configure that public path, we can basically count on it being there. If it's unset, we'll just use slash, but we're gonna be able to resolve these things at build time. So I'm showing some of my biases here, but I think this is way more complicated than the rollup solution, but it's interesting seeing these little details that I'm like, oh, I wish rollup had that. I mean, I know why it doesn't, but that would be so much simpler for me if it just had a public path string or whatever. Yeah, and maybe that's part of rollup adding that exposing their built-in resolution is maybe there's an option you can pass that's like, oh, by the way, like resolve it all against this public path. That might be a convenient thing, who knows. So while we remap all of those file names to prepend that public path, we can also remove any trailing index.html because that doesn't appear in the URLs. We wanna catch the URLs, not file names, and we will use that dot workaround to safeguard against an empty string actually catching the service worker, which we don't want. Instead, we'll have it catch the directory the service worker is in. So that's both of our magic globals prepared for the service worker. Now it's time to generate it. And here we're gonna create one last function to handle compilation of the service worker. So to build our service worker module, we're going to use something called a child compiler, which is essentially a nested webpack compiler. And we'll configure this to output these sw.js file. And one thing to think about when using child compilers is they can have a completely different list of plugins compared to the main compiler. So we can pull in a plugin that generates worker compatible code because we have a service worker target. So there's gonna be a difference here as well, isn't there? Because like, well, actually you tell me because you're using a child compiler and it's like a whole different pass, does that mean that this service worker bundle is not going to be able to share code with the main bundle? It's not gonna be able to code split there. Correct, yeah. So any split points that end up happening in the code generated by this child compiler are gonna be totally independent from the main compiler. They'll be a separate set of files. Also. Okay, that's an important difference between the two. I mean, it's not common to share code between your service worker and main thread, but the rollup version could do that. Yeah. Exactly. So the other thing that worker template will do in this case is if there are split points rather than using a script tag to load those chunks of code it will use import scripts, which is available in workers. Cool. And so the last thing we need is a second plugin also imported from webpack core. And this just lets us specify the entry module to start compiling from, which is in this case our service worker source module. So finally we run our compiler as a child of the main compiler. And this just ensures that if our compiler takes a while to run the main compiler won't finish and terminate the process before we're finished. And then once it's built, the callback will be called with our compilation and also any error. So just like with the admit hook, we are gonna grab the generated service worker asset from compilation.assets and then we can pass that back to admit. If there was an error in the child compiler, say a syntax error in our service worker code, we'll just bubble that up, that will fail the build and print it to console. If there wasn't an error, we do now have our compiled service worker asset. The thing is it doesn't yet have version and assets magic global variables. So to inject those, we need to pull in something called concat source from the webpack sources module. And this lets us concatenate strings essentially, it's very similar to string concat, but with the added benefit of it is able to produce source maps. Ah, yes. Okay. So this is interesting because I broke source maps in my solution, but this is going to just work. Now I could have done the same like in rollup, I could have created essentially that it doesn't support child compilers as such, but you couldn't just call rollup as a sort of separate layer. That would have been a lot more complicated. So it's interesting now, like some of the complexity that you're using here with webpack is actually paying off. Right, yeah. So like the initial bundle of service worker was harder, but as it turns out, it's the same number of lines to support source maps as it is to not support source maps now that you've opted into some of that complexity. So that's sort of an interesting tradeoff between the two. So yeah, so the last step, we've constructed this asset, but it's only just an object in memory. So what we need to do is merge it into the compilers generated assets, like with rollup, this is a live object and anything that we merge into here will get written to disk. And so with that, our plugin is done. So we can call the callback, compilation will finish. And what we get is our generated service worker source. We can see here that we have assets with the generated array of URLs. And we also have version, which is that string based on the hash of their contents. So we did it. Amazing, and that's it. Yeah, and I guess that's bringing this talk to a close, provided that everything has recorded properly and we don't have to do it again, because I can't tell you, this recording has been absolutely cursed. We have done so many takes due to just Bluetooth stacks failing on my machine, recordings not happening on Jason's machine. So who knows, maybe this will be the final record. If you're watching this right now, then I am so happy because it means we don't have to do all of this again. Oh, right. Okay, if you want to see those implementations, they are both on tooling.report, along with how to handle assets, code splitting, hashing, transformations, all of the stuff. And not just webpack and rollup as well. We also have parcel, and we also have browserify. Remember browserify and gulp and all of that sort of stuff. I mean, you might laugh at that, but this is, if you look at the stats on NPM, that is still absolutely huge and climbing. And we did actually find some cases where it really holds its own and was maybe better than some of the more modern things that we deal with. So go on and check all of that out. Thanks so much for watching this, hopefully final take of this talk. Thank you very much and goodbye. Bye.


It was great. I met the Prime Minister of the Indian Republic, Mr. Mandi. We discussed about the educational interrelationships between Greek and Indian universities and we hope that we can promote this field further. We've discussed about the historical relationships between these countries since the ancient times. Mr. Mandi is a very well-known politician. India has world power in the last decades and always India has played a very important historical role in the global and that role continues, I can say.


Today, Wednesday, November 18, 2020, the Ministry of Health and Wellness received confirmation of five new cases of COVID-19. This brings a total number of cases diagnosed in country to date to 183. Case number 179 is a 49-year-old female from the Castries District. Case number 180 is a 12-year-old male from the Castries District. Case number 181 is a 53-year-old male from the Castries District. Case number 182 is a 20-year-old male from the Castries District. Case number 183 is a 59-year-old female from the Castries District. All of these individuals, after being assessed and tested for COVID-19, were placed in quarantine by healthcare practitioners while awaiting their test results. Two of the cases were identified through the contact tracing process and an epidemiological link has been established for these. Case number 181 is linked to case number 162. Case number 183 is linked to case number 164. Investigations are ongoing to determine the existence of any additional epidemiological links associated with these cases. Case per protocol arrangements have been made to place these individuals into care. We once again ask every citizen to embrace the efforts to minimize the threat of COVID-19 nationally. It is important to make the protocols part of your daily routine to protect the health of every individual and to minimize the impact of the virus on our country as a whole. In addition, it is important that we also take action to maintain our personal health and well-being. These measures will assist in building strong immune systems which will improve resistance to the virus and prevent the severe forms of the illness. Eat a healthy balanced diet, including fruits and vegetables daily, regular daily intake of water instead of sugary drinks and beverages, daily physical activity, limit your intake of alcohol and avoid its misuse and abuse. Do not smoke as apart from its risk for chronic illness, it also leads to lung damage and exacerbates respiratory illnesses like COVID-19. Doing these alongside the infection prevention and control guidelines of hand hygiene, use of a face mask in public places and maintaining a separation of six feet when in public will help to reduce the spread of COVID-19. If experiencing flu-like symptoms, keep away from others and seek medical care immediately at the closest respiratory clinic. The Ministry of Health and Wellness will continue providing updates as new information becomes available.