And he's now embarked on an IoT startup called Sound Sensing. Today, he'll talk to us about a topic related to his thesis, Audio Classification with Machine Learning. Hi, thank you. So audio classification is not such a popular topic as, for instance, image classification or natural language processing. So I'm happy to see that there are still people in the room interested in this topic. So first, about me, I'm an internet things specialist. I have a background in electronics. From nine years ago, I worked a lot as a software engineer because electronics is mostly software these days, or a lot of software. And then I went to do a master's in data science. Because IoT, to me, is the combination of electronics, sensors especially. Software, you need to process the data and the data itself. You need to somehow convert sensor data into information that is useful. And now I'm consulting on IoT and machine learning. And I'm also the CTO of Sound Sensing. We deliver sensor units for noise monitoring. In this talk, my goal, we'll see if we get there, is that you, as a machine learning practitioner, without necessarily prior experience in sound processing, can solve a basic audio classification problem. We'll have an introduction about a little bit of a sound very briefly. And then we'll go through a basic audio classification pipeline, and then some tips and tricks for how to go a little bit beyond that basic. And then I'll give some pointers to more information. The slides and a lot of my notes and machine hearing in general, a little bit broader than audio classification, is on this GitHub. So applications. There are some very well-recognized sub-fields of audio. Speech recognition is one of them. And for instance, there you have, as a classification task, you have keyword spotting. So hey Siri, or hey Google. As a task in music analysis, you also have many tasks of genre classification. For instance, can be seen as a simple audio classification task. But we're going to keep it mostly on the general levels. We're not going to use a lot of speech or music specific domain knowledge. And we still have examples in across a wide range of things. I mean, anything you can do with hearing as a human, we can get close to, or many, at least in classification tasks, with machines today. So in NICU acoustics, you might want to analyze bird migrations using sensor data to see their patterns. You might want to detect poachers in protected areas to make sure that no one actually is going around shooting where there should be no guns fired and so on. So using quality control and manufacturing, especially because you don't have to go into the equipment or the product and your tests, you can listen to it from outside. For instance, used for testing your electrical car seats, the auto motors run. In security, it's used to help monitor large amounts of CCTVs by also analyzing audio. And in medical, for instance, you could detect heart murmurs, which could be indicative of a heart condition. So these are some motivating examples. And so in digital sound, I'll just go very briefly through this. First thing that is important is that sound is almost always or basically always a mixture. Because sound, I mean, it will move around the corner and like an image, for instance. And you will always have sound coming for us. It will also transport in the ground and be reflected by a wall. And all these things make it so you always have multiple sound sources or the source of interest and then always other sound sources. In audio acquisition, OK, we have, of course, sound is air pressure. We need to go have a microphone, convert it to electrical voltage, ADC, and then we have a digital waveform, which is what we will deal with. Then it's quantized in time, for instance, with the sampling rate and the amplitude. And we usually deal with mono, primarily with mono, when we do auto-classification still. There are some methods around stereo but not widely adopted. And also make the channel you could also have. We typically use uncompressed formats. It's just the safest. Although in real life situation, you might also have compressed data, which can have artifacts and so on that might influence your model. So after we have a waveform, we can convert it into a spectrogram. And this, in practice, is a very useful representation, both as a human for understanding what is the sound and for the machines in order to do detection of this. This one is a frog croaking, very periodically. You can see a little gap. And then it's hard to see, but in top, in the higher level, there is some cicadas that are going as well. And this allows us to both see the frequency representation and the patterns across time. And together, you can often, this allows you to separate different sound sources from your mixture. So we'll go through a practical example just to keep it kind of hands-on. Environmental sound classification, so given auto signal of environmental sounds. So these are everyday sounds that are around in the environment. For instance, it can be outdoors, cars, children playing and so on. It's very widely researched. We have several open data sets that are quite good. Audio set is several, I think, tens of thousands or even hundreds of thousands of samples. And in 2017, we reached roughly human-level performance. Only one of these data sets has an estimate for what is human-level performance, but we seem to have surpassed that now. And one nice data set is Urban Sound 8K, which has 10 classes of 8K samples, roughly four seconds long, and nine hours total. And state of the art here is around 80%, so 79 to 82 accuracy. So now we have the spectrograms. And these are the easy samples. This data set has many challenging samples where the sound of interest is very far away and hard to detect. And these ones are easy. So you see the siren goes very up and down. And jack-hammers and drilling have very periodic patterns. So how can we detect this using a machine learning algorithm in order to output these classes? So I will go through a basic audio pipeline, skipping around 30 to 100 years of history of audio processing, kind of going straight to what is now the typical way of doing things. And it looks something like this. So first, in the input, we have our audio stream. It's important to realize that, of course, audio has time related to it. So it's more like a video than to an image. And in a practical scenario, might you do real-time classification? So this might be an infinite stream that just goes on and on. So it's important for that reason and for also the machine learning algorithm to divide the stream into relatively small analysis windows that you will actually process. And however, you often have the mismatch between how often you have labels for your data versus how often you actually want to prediction. It's known as weak labeling. I won't go much into it. So in the urban sound, there's four seconds sound snippets. So that's what kind of what we're given in this curated data set. However, it's usually beneficial to use smaller analysis windows to reduce the dimensionality of the machine learning algorithm. So the process goes that we'll divide the audio into these segments. We'll often use overlap. And then we'll convert it into a spectrogram. We'll use a particular type of spectrogram called a male spectrogram, which has been shown to work well. And then we'll pass that frame of features from the spectrogram into our classifier. And it will output the classification for that small time window. And then because we have labels per four seconds, we'll need to do an aggregation in order to come up with the final prediction for these four seconds, not just for this one, a little window. We'll go through these steps now. So first analysis windows. As I mentioned, we often use overlap. So this is somewhat specified in two different ways. One is an overlap percentage. Here we have a 50% overlap. So that means that we're essentially classifying parts of the, or we're classifying every piece of the auto stream twice. So if we have even more overlap, we will maybe a 90% overlap. Then we're classifying it 10 times. And that gives the algorithm a kind of multiple viewpoints on this audio stream. And it makes it easier to catch the sounds of interest. Because the model might, in training, might have been kind of prefer a certain sound to appear in a certain position inside this analysis window. So overlap is a good way of working with that. So I mentioned we use a specific type of spectrogram. So the spectrogram is usually processed with us called male scale filters. These are inspired by the human hearing. So our ability to differentiate sounds of different frequencies is reduced as frequencies get higher. So low sounds, we're able to detect small frequency variations. However, for high-pitched sounds, we need large frequency variations to work. And by using this kind of filters on the spectrogram, we obtain a representation more similar to our ears. But more importantly, it's a smaller representation for a machine learning model. And it captures, and also you'll be able to merge kind of related data in two consecutive bins, for instance. So when you've done that, it looks something like this. So the top is a normal spectrogram. You see kind of a lot of small details. The bottom one, we've started the male filters at 1,000 hertz. This is bird audio, which is quite high-pitched. A lot of chirps up and down. And in the third one, we've normalized the data. And we usually use log scale compression in order to, because sound has a very large dynamic range. So sounds that are faint versus sounds that are very loud for the human ear is a factor of 1,000 or a factor of 10,000 in energy difference. So when you've normalized log-scaled, applied male spectrogram, and normalized, you look at something like the image below there. So in Python, this feature process is something that is I'm not going to go through all the code in detail. We have an excellent library called Librosa, which is great for just loading the data and doing basic feature pre-processing. Also, some of the deep learning frameworks have their own male spectrogram implementations that you may also use. But this is a general thing. In streaming, so when people analyze audio, they often apply normalization learned from the mean, for instance, across their whole samples, for our seconds in this case, or from their whole data set. That can be hard to apply when you have a continuous audio stream, which has, for instance, changing volume and so on. So what we usually do is we normalize per frame. So the hope is that you have enough information in our roughly one second of data in order to do a decent normalization. And doing normalization like this has some interesting consequences when there is no input.  So you'll sometimes need to exclude very low energy signals from being classified. It's just a little practical tip. So convolutional neural networks, they're hot. Who here has a basic familiarity, at least gone through a tutorial or read a blog post about image classification and CNNs? Yeah, that's quite a few. So CNNs are the best in class for image classifications. And spectrograms are image-like. And they are to their representation. They have some differences. So the question is, or maybe it was, is will CNNs work well on spectrograms? Because that would be interesting. And the answer is yes. This has been researched quite a lot. And this is great because there is a lot of tools, knowledge, experience, and pre-trained models for image classification. So being able to reuse those in the auto domain, which is not such a big field, is a major win. So you'll see a lot of the research lately. It can be a little bit boring in audio classification research, because a lot of it is like taking one year ago image classifying tools and applying them and seeing whether it works. It is, however, a little bit surprising that this actually works, because the spectrogram has frequency on the y-axis, typically, shown that way, and time on the other axis. So movement or scaling in this space doesn't mean the same as in an image. If I have my face inside an image, it doesn't matter where my face appears. If you have a spectrogram and you have a certain sound, maybe it's like a chirp up and down, if you move that up in frequency or down, at least if you move it a lot, it's probably not the same sound anymore.  The shape might be similar, but the position matters. So it's a little bit surprising that this works, but it does seem to do really well in practice. So this is one model that does well on urban sound. And one thing you'll note, compared to a lot of image models, is that it's quite simple. I mean, relatively few layers. This is smaller than or the same size as a Lee net. And there are three convolutional blocks followed by Mac, with Mac's pooling between the two first blocks. And that's the standard kind of architecture. This one, using 5x5 instead of 3x3 kernels, doesn't make much of a difference. You could stack another layer and do the same thing. And we flatten, and we use a fully convolutional end. So this is from 2016. And still is close to the art on this data set urban sound. OK. So if you are training CNN from scratch on audio data, do start with a simple model. I mean, there is usually no reason to start with, say, VGG16, with 16 layers and millions of parameters, or even mobile net or something like that. You can usually go quite far with this kind of simple architecture, a couple of convolutional layers. So in Keras, for example, this could look something like this, where we have our individual kind of blocks, convolution, Mac's pooling, Rayleigh non-larity, same for the second one, and classification at the end of the fully connected layers. Yes. So this is our classifier. We'll pass the classification through this. And it will give you also a prediction for which class it was. So 10 classes in the urban sound. And then for each window. And then we need to aggregate these individual windows. And there are multiple ways of doing this. You could do the simplest kind of thing to think about is to do majority voting. So if we have 10 windows of our four second spectrum, we could do the predictions on each and then just say, OK, the majority class wins. That works rather well. It's not differentiable. And so you kind of need to do this post-processing. And you're making very rough predictions on each step. So mean pooling or global average pooling across those analysis windows usually does a little bit better. And it's nice with deep learning frameworks is that you can also have this as a layer. So for instance, in Keras, you have the time distributed layer, which is there are sadly extremely few examples of online. So it took me like it's not that hard to use, but it took me a little bit to figure out how to do it. And so we apply a base model, which is in this case an input to this function. We pass it to the time distributed layer. And which essentially, it will use a single instance of your model, so we'll share the weights for all these steps in the, or all the analysis windows. And then, so we'll just run in multiple times when you do the prediction step. And then we'll global average pooling over these predictions. So here we're averaging the predictions. You can also do more advanced things where you would, for instance, average your feature representation and then do a more advanced classifier on top of this. But this is called probabilistic voting quite often in other literature when you do this mean pooling. Yes, so that allows us, so this will give us a new model, which is what will then take not a single analysis windows, but it will take a set of analysis windows, typically corresponding to our four seconds with for example, 10 windows. So if you do this and a couple more tricks from my thesis, you can have a system working on like this. So this has, in addition to building model and so on, which I've gone through, we're also deploying to a small microcontroller using the vendor provided tools that converts the Keras model and so on. So that's kind of roughly standard things and I didn't want to go into it here. So a little demo video. See if we have a sound. So here are the 10 classes. And we have various sound samples. This is a children playing. I think in Spain, since they said hello. And what we do also here is we threshold the prediction. So if no prediction is good, we'll consider it unknown. And this is also important in practice, because sometimes you have out of class data is drilling. Or this is actually the sample I found said jackhammer. And jackhammer is also a class in drilling. They are, to my ear, hard to distinguish sometimes. And the model can also struggle with that. There's a dog barking. And so in this case, all the classification happens on this small sensor unit, which is what I focus on in my thesis. There's a siren a little bit louder. And actually it didn't get the first part of the siren. There's do do do do do. Only this undulating sound later. And actually these samples are not from the urban sound data set, which I've trained on. So they're out of the main samples, which is generally much more challenging task. Yes, so that's it for the demo. If you want to know more about doing sound classifications on this sensor unit, it's very small. You can get my full thesis. Both the report and the code is on GitHub. It's also linked from the machine hearing repository. Yes, so I won't go into too much details there. And some tips and tricks. So we've covered the basic audio processing pipeline, the modern one, and that will give you results and generally quite good results with the modern CNN architecture. And there are some tips and tricks, especially in practice where when you have a new problem, you're not researching an existing data set. Your data sets are usually much smaller and it's quite costly and tedious to annotate all the data and so on. So there are some tricks for that. The first one is data augmentation. This is well-known from other deep learning applications, especially image processing. And data augmentation can be done on audio, can be done either in the time domain or in the spectrogram domain. And in practice, it both seem to work fine. So here are some examples of common augmentations. The most common and possibly most important is to do time shifting. So remember that I said that when you classify an analysis window, we made one second, the sounds of interest there and what the individual convolution kernel sees might be very short. If you have bird chirps, they're like, and those are maybe 100 milliseconds max or maybe even 10 milliseconds. So they occupy very little space in that image that the classifier sees. But it's important that it's able to classify it no matter, or it's desirable that it's able to classify it no matter where inside this analysis window it appears. So time shifting simply means that you do, you shift your samples in time, left and forward and backward, left and right. And that gives them the algorithm has seen that, oh, okay, bird chirps can appear many places in time and any place in time and it doesn't make a difference to the classification. So this is by far the most important one. And you can usually go quite far with just time shifting. If you do want precise location of your event, so you wanna have a classifier that can tell when did chirps appear, not just in the 100 millisecond range instead of just that there was birds in this four or 10 second audio, then you might not want to do time shifting because you might want to have kind of that the sound always occurs in the middle of the window. But then you need to, your labeling needs to respect that. Yeah, time stretching. So many sounds, if I speak slowly or I speak very fast, it's the same, this is same meaning, it's certainly the same class, it's both speech. So time stretching is also very efficient to capture such variations and also pitch shifting. So if I'm speaking with a low voice or a high pitched voice, it's still the same kind of information and the same carries in, for general sounds, at least a little bit. So a little bit of time shift, pitch shift you can expect, but a lot of pitch shift might kind of bring you into new class, for instance, the difference between human speech and bird chirps. It might, that might be a big pitch shift. So you might want to limit how much you're pitch shifting. So typical data notation settings here is like maybe 10 to 20% on time shift and pitch shift. You can also add noise. This is also quite efficient, especially if you do know that you have variable amount of noise, random noise works okay. You can also sample, there's like a lot of repositories of basically noise where you'll mix in noise with your signal and classify that. Mixup is an interesting data augmentation strategy that mixes two samples, by a linear combination of the two and actually adjusts the labels accordingly. And that has been shown to work really well, also in combination with other augmentation techniques on audio. So yes, we can basically apply CNNs with the standard kind of image type architecture. This means that we can do transfer learning from image data. So of course image data has, is significantly different from spectrograms. I mentioned the frequency axis and so on. However, some of the base operations that are needed, you need to detect edges, you need to detect diagonals, you need to detect patterns of edges and diagonals, you need to detect kind of a blob of area. Those are common kind of functionality needed by both. So if you do want to use a bigger model, definitely try to use a pre-trained model and fine tune it. For instance, most deep learning frameworks including Keras have pre-trained models, pre-trained on the ImageNet. The thing is that most of these models, they take RGB color images as data and it can work to just like use one of those channels and zero fill the other ones, but you can also use just copy the data across the three. There's also some paper showing that you can do multi-scale. So for instance, one has a spectrogram with very fine time resolution and one has one with a very coarse time resolution and then you put them in different channels and this can be beneficial. But because image data and sound data are quite different, you usually do need to fine tune. So it's usually not enough to just apply a pre-trained model and then just tune the classifier at the end. You do need to tune a couple of layers at the end and typically also the first layer at least. Sometimes you fine tune the whole thing, but it is generally very beneficial. So definitely if you have a smaller dataset and you need that high performance and you can't get it with a small model, go with a pre-trained model, for instance, mobile net or something like that and then fine tune it. Audio embeddings is another strategy inspired by text embeddings where you create a, for instance, 128 dimensional vector from your text data. You can do the same with sound. So with a look, listen, learn L3, you can convert a one second audio spectrogram into a 512 dimensional vector which has been trained on millions of YouTube videos. It's seen a very large amount of different sounds and that uses a CNN under the hood and basically gives you that very compressed vector as a classification. And I didn't finish any code sample here but there's a very nice, the latest work is OpenL3, look, listen, more is the paper. And they have a Python package which makes it super simple, just import it. It's one function to pre-process and then you can classify audio basically just with the linear classifier from scikit-learn or so on. So that if you don't have any deep learning experience and you want to try an audio classification problem, definitely go this route first because this will handle kind of the, it'll basically handle audio part for you and you'll just, it can apply a simple classifier after that.  Audacity is a nice editor for audio and it has a nice support for annotating, adding a label track in annotating. There's like keyboard shortcuts for all the functions that you need so it's quite quick to use. So here I'm annotating some custom data where we did event recognition. And the nice thing is that the format that they have is basically a CSV file, it has no header and so on but this pandas line will basically give you a nice data frame with all your annotations from the sound. Yes, so it's time to summarize. Oh yeah, it fixed me, okay. So we went through the basic audio pipeline. We split the audio into fixed length analysis windows. We used log mail spectrogram as a feature representation because it's shown to work very well. We then applied a machine learning model, typically a convolutional neural network and then we aggregated the predictions from each individual window and we merged them together using global mean pooling. And models that I would recommend trying first if you're trying some new data, try audio embeddings with OpenL3 and a simple model like lino classifier or random forest for instance. And try a convolutional neural network using transfer learning, it's quite powerful and there are usually examples that will get you pretty far. If you do for instance pre-process your spectrograms and save them as PNG files, basically you can kind of take any image classification pipeline that you have already or if you're willing to kind of ignore this merging of different analysis windows and use that. Data implementation is very effective, time shift, time stretch, pitch shift, noise add are basically recommended to use. Sadly there's not such nice like or like go-to implementations of these in for instance Keras generators but it's not that hard to do. I just, yes, some more learning resources for you, the slides and also a lot of my notes in general are on this GitHub. If you do want to get hands-on experience, TensorFlow has a pretty nice tutorial called simple audio recognition and it's about recognizing speech commands which could be interesting in general but it's taking a general approach, it's not speech specific so you can use it for other things also. There's one book recommendation, computation analysis of sound scene events. It's quite thorough when it comes to general audio classification, a very modern book from 2018. So that's a nice one also. So I think we have questions maybe. We have 10 minutes for questions so please go to the microphones in the aisles to ask them. I think our first is there. Yeah, thanks John, very interesting application of machine learning. I have two questions, small questions. So there's obviously like a time series component to your data. I'm not so familiar with this audio classification problem but can you tell us a bit about time series methods, maybe LSTM and so on, how successful they are? Yes, yeah, time series is intuitively one would really want to apply that because there is definitely a time component. So convolutional recurrent neural networks do quite well when you're looking at longer timescales for instance, there's a classification task called audio scene recognition. For instance, I have a 10 or maybe 30 second clip. Is this from a restaurant or from a city or so on? And there you see that the recurrent networks that do have a much stronger time modeling, they do better. But for small short tasks, CNNs do just fine, surprisingly. Okay, and the other small question I had was just to understand your label, the target that it's learning. You said that this is a very mixed, the sound is a very mixed data set. So are the labels just like one category of sound when you're learning or would it be beneficial to have maybe a weighted set of two categories when doing learning? Yeah, so in the audio classification task, the typical kind of by definition is to have a single label on a some sort of window of time. You can have multi-label data sets, of course, and that's a more realistic modeling of the world because you basically always have multiple sounds. So I think audio set has multi-labeling and there's a new urban sound data set now that also has multiple labels. And then you apply kind of more tagging approaches. But you're using classification as a base. So I had with tagging, you can either use separate classifier per track or sound of interest or you can have a joint model which has multi-label classification. So definitely this is something that you would want to do but it does complicate the problem. We have over there one person in the mic. So thank you for your documentation and you mentioned about data augmentation that we can also mix up to separate our cases and mix them and then the label of that mix up should be like weighted also because it kind of concludes with previous question. Should we like 0.5 and 0.5 for the other and how will that work? Yes, so a mix up was proposed things like two, three years ago. There's a general method. So you basically take your sound with your target class and you say, okay, let's take 80% of that, not 100. And then take 20% of some other sound which is a non-target class, makes it together and then update the labels accordingly. So it's kind of just telling you, hey, there is, we're basically creating a lot of, there is this predominant sound but there's also this sound in the background. Okay, thank you. Yes, we have a question. You mentioned about the male frequency ranges but usually when you record audio microphones you get up to 20,000 hertz. Yes. So have you any experience or could you comment on when you have added information of the higher frequency ranges does that affect the machine learning algorithm or other features that one could use? Yeah, that's a good question. So typically recordings are done at 44 kilohertz or 48 kilohertz for general audio. Often machine learning is applied at lower frequency. So with the 22 kilohertz or something that's in just 16 or in a rare case it's also eight. So it depends on the sounds of interest. If you're doing birds, definitely you want to keep that those high frequency things. If you're doing speech, you can do just fine on the eight kilohertz usually. Another thing is that noise tends to be in the lower areas of the spectrum. There's more energy in the lower area of the spectrum. So if you are doing birds, you might want to just ignore everything below one kilohertz for instance. And that definitely simplifies your model especially if you have a small data set. We have more questions. You need to go to the mic either here or there. Quick question. You mentioned the editor that has support for annotating audio, could you please repeat the name? Yes, audacity. Audacity, okay. Yes. One more general question. Do you have any tips if for example you don't have an existing data set and you're just starting with a bunch of audios that you want to annotate first? Do you have any advice for strategists like maybe semi-supervised learning on something like this? Yeah, semi-supervised is very interesting. There's a lot of papers, but I haven't seen like very good like practical methodology for it. And I think in general annotating a data set is like a whole other talk. But I'm very interested to come into a chat about this later. Thanks. So yeah, we have two more and I think we're done. Very nice talk. My question would be, do you have to deal with any pre-processing or like white noise filtering? You mean to remove white noise? It's exactly because you just said like removing or ignoring certain amount of frequencies. Yes. And you can, I mean, scoping your frequency range definitely is very easy. So just like, just do it if you know where your things of interest are. Denoising, you can apply a separate denoising step beforehand and then do machine learning. If you don't have a lot of data that can be very beneficial. For instance, maybe you can use a standard denoising algorithm trained on like thousands of hours of stuff or just a general DSP method. If you have a lot of data, then in practice, the machine learning algorithm itself learns to suppress the noise. But it only works if you have a lot of data. Thanks. So thank you for our talk. Is it possible to train deep convolutional neural net directly on the time domain data using 1D convolutions and deliated convolutions and stuff like this? Yes, and is this possible? And it is very actively researched. So it's only, but it's only within the last year or two that they're getting to the same level of performance as spectrogram-based models. But some models now are showing actually better performance with the end-to-end trained model. So this, I expect that in a couple of years, maybe that will be the kind of go-to for a practical application. Can I do a speech recognition with this? Is this only like six classes? And I think you have much more classes. You want to clarify words. If you want to do automatic speech recognition, so the complete vocabulary of English, for instance, then you can theoretically, but there are specific models for automatic speech recognition that will, in general, do better. So if you want full speech recognition, you should look at speech-specific methods, and there are many available. So, but if you're doing a simple task like commands, like yes, no, up, down, one, two, three, four, five, where you can limit your vocabulary to say, maybe under the 100 classes or something, then it gets much more realistic to apply a kind of speech unaware model like this. Okay, thanks for an interesting presentation. I was just wondering from the thesis, it looks like you applied this model to a microprocessor. Can you tell a little bit about the framework you used where you transferred it from a Python? Yeah, yes. So we used the vendor-provided library from SDM Microelectronics for the SDM32, and it's called X-Cube AI. You'll find links in the GitHub. It's a proprietary solution, only works in the microcontroller, but it's very simple. You plug in your, you throw in your Keras model, it will give you a C model out, and they have examples about the pre-processing with some bugs, but it does work. And the firmware code is also in the GitHub repository, not just the model, so you can basically download that and start going. Thanks. Okay, yeah. Do join me here if you wanna talk more about some specifics about auto-classification. I'll also be around this day. Thank you. Thank you, John, Nordby.


Alrighty! I have a construction update for you guys today. It's been quite a while. There is not much going on at Cannons Wonderland, but some mystery things have appeared in the park, and we'll get right to that in a second. So there are definitely some hints at to the location of the... Hold on, I think we're getting some... Alrighty! So for those of you that don't live under a rock, you already know that Leviathan has a little bit of its track segment at a very important part of the drop... removed. And a lot of people think this is a lot more serious than it actually is. Now I do want to express that everything I'm about to discuss in this video is the opinions of myself, Brendan Cross, and nothing more. So Leviathan, in a similar-ish, in a much smaller scale from my opinion, has pulled a fury and had a little bit of a crack in a very important part of the ride. Now, Wonderland is a park that takes maintenance extremely seriously, and they were able to catch this very early on. So it needed repairs, and all that I can tell is that they are waiting on some parts to do some welding, and they should have the ride back up as soon as this welding is done. It does not need the similar 1000 tests to fury as it was not a ride-related accident. It was regular maintenance, and it was caught. So this is very good news for those that are really hoping to get on Leviathan. It could be even open as early as tonight, if all goes well. I could even see this being open tomorrow morning, Sunday morning. It all depends on how the welding goes. But yeah, it is a very key part of the ride. If you've noticed, it's one of the track segment connector pieces, so definitely going to need some extra care. But I just wanted to explain exactly what was going on. Because a lot of people were sending me DMs, being like, what happened? What happened? Nothing serious. This is actually a lot more common than you think. And for parks that don't normally have people that fly drones or are there on the daily, these things go unnoticed. In fact, I can tell you a lot of your favorite attractions at Canada's Wonderland and other parks around the world have this kind of thing happen pretty regularly. So expect to have Leviathan open very shortly, and you can rest assured that they're going to try their best to get this as open as soon as possible. And your safety is not at risk. Again, this is a very normal maintenance, preventative maintenance kind of task that is going on. Anyways, have a good one, guys. Bye.


In some cases, data is collected from several sources, a treatment may be applied to different cohorts, or data is collected over several years, or by different research teams. But when we aggregate the data, we lose information. We no longer know which cohort a person came from, or what year the data was collected, or which research team gathered the data. What could possibly go wrong? For example, suppose an intervention is applied to the students in a school district. The aggregated data is shown, should the intervention be applied generally? Since we want to decide whether or not to use the intervention strategy, we'll consider two quantities, the fraction of treated students who improved, versus the fraction of untreated students who improved. Of the 121 students with an intervention, 46 improved, so the fraction of treated students who improved is about 38%. Since the 1073 students without an intervention, 391 improved, so the fraction of untreated students who improved is about 36%, and so it appears that improvement is more likely with an intervention. So we should recommend using the intervention. Or should we? Well, suppose the data was collected from two different schools. The raw data is shown, and this is the data that was aggregated to form the table in the previous slide, and let's look at the schools separately. For school A, of the 38 students with an intervention, 11 improved, so the fraction of treated students who improved is a bit under 29%. Of the 681 students without an intervention, 206 improved, so the fraction of untreated students who improved is a bit over 30%. So it appears improvement is more likely without an intervention, and so we would recommend for the students at this school, no intervention. This may be a little surprising since we recommended an intervention earlier, but ignoring facts and evidence is only something you can do if you're a politician. We should re-evaluate our decision when more information is available. In this case, the additional information that's available is which school we're looking at. And a normal person would stop here, but mathematicians are not normal people. Let's take a look at school B. Of the 83 students with an intervention, 35 improved, so the fraction of treated students who improved is about 42%. Of the 392 students without an intervention, 185 improved, so the fraction of untreated students who improved is about 47%. So it appears improvement is more likely without an intervention, and so we would recommend no intervention. Wait a minute. In both cohorts, the group without an intervention did better, and so in both cases we would recommend against an intervention. But when the data was combined, the group with the intervention did better, and so we would have recommended an intervention. This is an example of what's known as Simpson's Paradox. It's named after, no, no, but not a bad guess, Edward Simpson, who first described it in 1951, though Carl Pearson and others noted it as early as 1899. Now Simpson's Paradox isn't really a paradox. It's a consequence of information loss as we aggregate data. In particular, the aggregated data lost the information about which school the students came from, which means more detailed information is useful. For example, suppose school A's student could be divided into those who play sports and those who don't. The disaggregated data is shown. Of those who play sports, of the 17 with interventions, 5 showed improvements, so the fraction of treated students who improved is about 29%. Of the 473 with no interventions, 123 showed improvement, so the fraction of untreated students who improved is about 26%. So those with interventions did better, and we'd recommend interventions. Of those who didn't play sports, of the 21 with interventions, 6 showed improvements, so the fraction of treated students who improved is about 29%. Of the 208 without interventions, 83 showed improvements, so the fraction of untreated students who improved is almost 40%, so those without interventions did better, and so we'd recommend no interventions. It appears we can get any decision we want by choosing the right statistics. That's how it should be. We should re-evaluate our decisions every time we gather additional data. If we know nothing about the students, the intervention is warranted. But if we know that students are from school A, the intervention is not warranted, unless we know the student plays sports, in which case the intervention is warranted. There is no paradox. Simpson's paradox is an important thing to keep in mind whenever we try to make decisions based on evidence. Suppose you want to decide on whether to undertake a course of action. The more you know, the better your decisions. In this example, we'd want to know what school and which students. Simply knowing the overall rate of success is irrelevant. Conversely, if someone has the raw data but refuses to share it, they don't want you drawing your own conclusions. Remember, the more you know, the more power you have, so be suspicious when someone refuses to share data without a good reason. And the data has some private information is a good reason. The data is proprietary is not.


Non-lethal weapons destruction gives that commander, especially for the Marines, gives that commander the option to go less than lethal. Our main mission is to provide these host nations with the knowledge and the capability set for taser, OC, ride control, and instead of lethal force, less than lethal force. Basically, it feels great knowing that I'm going to pass on some knowledge to them and some capabilities to them that could possibly save their lives in the future if they're dealing with their civilian counterparts out here. I hope that they just take away the fact that they can have this less than lethal option and not jump straight to lethal force with every adversary they've come across. I hope they take away, especially my MPs, they take away the importance of teaching these host nations non-lethal capabilities.


Hello, thank you to Astrodog from the Snestrunk Discord for this idea. Here's a video about some games that are still worth playing today that are featured on consoles or computer systems that time may have forgotten for one reason or another. Let's start with an obvious example. The Nintendo Virtual Boy wasn't exactly a sterling success, and only 14 titles were released in North America. One of them, however, was actually pretty good, titled Virtual Boy Wario Land, and it fits right in with the rest of the Wario Land series, which normally appeared on Game Boy and on Game Boy Advance. So if you liked those games, you'll dig this one too, even if, you know, it's a friggin' Virtual Boy game. Thankfully, due to stuff like emulation, you can play this one on PC or a Raspberry Pi device or whatever. And no, you're not beholden to the dreaded Virtual Boy Red. There is a grayscale mode available if you play through the Beetle VB Core on the Retro Arch emulator. I just stuck with the red for this video, just to hammer home the point that this is a friggin' Virtual Boy game. This is what it looked like originally, and it's still actually pretty good. Like I said, it's like the other Wario Land games, where it's a side-scrolling action platformer only with a 3D gimmick shoehorned in, where you can explore each stage in both the foreground and the background. It doesn't really add a whole lot to the gameplay, and it comes across as kind of gimmicky, but hey, at least the 3D effect makes for some cool boss fights. You've got four lives and a handful of continues, and the typical Wario-style gameplay is here, featuring three power-ups in the form of hats. The bull hat increases Wario's charge attack, and allows him to butt-stomp. The eagle hat allows him to fly, and the dragon hat gives him a flamethrower, but it disables his charge attack. You can even combine hats for even more power-ups, which is pretty cool. There's 14 long levels here, there's multiple endings, and if you 100% this game, you unlock a second quest that's much tougher, which is definitely a good thing, because let's face it. This game isn't the most challenging out there, but still, it's a surprisingly good playthrough on a platform that went absolutely nowhere. Next let's take a look at the Amiga, a PC brand in the 80s made by Commodore, and they were hugely successful all across Europe, but particularly in Germany and the UK. The thing about Amiga games, however, is that, well, they had a certain kind of look and feel to them, especially side-scrolling platformers, but then there's stuff like Rough & Tumble, which is like the Amiga version of Metal Slug, only two years before Metal Slug was made.  Thanks again to Astrodog for pointing this one out.  And you got your typical slate of weapons here, lasers, missiles, flamethrowers, and the level design here is excellent and keeps the game interesting. The boss fights, once you finally get to them, are also a lot of fun, and provide a cartoonishly huge explosion. You gotta love that. So yeah, if you want to make stuff go boom on the Amiga, play this one any way you can. Here we've got another surprise on the Amiga. It's Yojo, made in 1993, and yes, in the visual department when it comes to art direction and layout, this is what I'm referring to when I talk about the typical Amiga look and feel. However, what stands out about this one is the music. It is freaking awesome. The game itself is pretty good as well, but that soundtrack, damn. When I was a kid, I didn't know a single person who owned Anatari links. It was supposed to be Atari's answer to the Game Boy and Game Gear, and while I'm sure it's a perfectly decent piece of hardware on its own, it just got lost in the shuffle, mostly because there weren't that many good games for it. There is one, however, titled Blue Lightning. The links is answered to Sega's popular Afterburner series. And man, oh man, this game seriously looks amazing for it being made on a handheld console in 1989. It's a pretty traditional flight sim, rail shooter type hybrid with nine missions for you to complete with the usual machine gun and a limited number of missiles you have to use to bring down enemies. And yeah, I admit the gameplay gets a bit repetitive here, but really, I just want to give both the Atari links and this game its due because this looks impressive as hell, and it plays pretty well too. It's surprisingly forgiving, providing six lives, so you've got plenty of time to get used to the controls and actually get through the first few levels and get a feel for the action. This one's a bit more of a curiosity than a must play, but I just thought it made for a good fit for this video because it's an out of nowhere title that really surprised me. Before the iPhone and the iPod, Apple was experimenting with stuff like the Pippin. No, not that Pippin. That's right, Apple got themselves into the home console racket back in the mid-90s, creating the wildly overpriced Apple Pippin in 1996, launching at a price of a whopping $600 in the US. It flopped, and it only produced 18 games, but there's one I want to point out thanks to Coolor from the SNES Drunk Discord. It's Super Marathon, a compilation of two games originally made for the Macintosh, made by a dev team by the name of Bungie, who went on a few years later to develop a little series you might have heard of called Halo. Here we've got a story-driven first-person shooter taking place in the year 2794, and your colonizing planets, blasting aliens, and yep, it's pretty much just Doom with a new coat of paint, but still a lot of people really dig these kinds of games to this day. There's a certain simplicity about them that I can appreciate, and to the Apple Pippin's credit, this is pretty decent for a home console first-person shooter. I mean, if you compare this to the original Macintosh version, they look almost exactly the same, and at the very least, this is still just an interesting look at what Bungie was up to before Halo. Another failed home console from the 90s was the Panasonic 3DO, but unlike the Apple Pippin, this one actually had a few good ports and some decent games on it, like Return Fire. And again, just like Rough and Tumble earlier in the video, this is another game where you just make stuff go boom, only in this case, it's vehicular mayhem. You can control a tank, a helicopter, a jeep, and a mobile rocket launcher, with each having their own varying stats when it comes to firepower, armor, speed, and fuel consumption.  When I say make stuff go boom, stuff goes BOOM! What else could you ask for? Plus one odd wrinkle here is that this game uses public domain music for its soundtrack, which on the surface sounds pretty lame, but the music they picked is perfect. It's everything from Ride of the Valkyries to Flight of the Bumblebee. It's tremendous. Return Fire on the 3DO is a fun time all around. Let's go back to more familiar territory, like the Game Boy Advance. Now Doom has received about a gazillion ports, as you probably know. The Super Nintendo even got one, but it's, uh, not good. The analogy I used when I reviewed it was, if Doom was a Ferrari, then playing Doom on the Super Nintendo is like driving a Ferrari underwater. Yeah, it's impressive that it can run, but that doesn't mean you should play it. The Doom games on Game Boy Advance, on the other hand, believe it or not, they're actually pretty dang good. Sure, the games aren't much to look at, and the blood is taken out, but the sound design is surprisingly fantastic, and the music sounds good, which is saying something for Game Boy Advance, and all the content is here, all the weapons, items, enemies, bosses, levels. They're all here, and the game runs really smoothly. However, they did have to make some sacrifices in order to make it run smoothly, like once you kill an enemy, it disappears, it doesn't stay there, but that's fine. And yes, visually, the game is kind of a mess, but the name of the game here is Performance. Both of these games play so much better than I ever would have expected. Anyway, the point is, I would not have imagined either Doom or Doom 2 to amount to much on the Game Boy Advance, but they're both well worth adding to your collection. They're surprisingly faithful ports and very good playthroughs. Let's stick with PC ports, this time with the Sega Genesis and Sid Meier's Pirate's Gold. And man, oh man, what a surprise this game is. It's one of the earliest examples of an open-world game executed properly on a home console, and the resulting game here is maybe one of the 10 best Sega Genesis games ever.  You can trade goods and earn money properly, you can capture cities and command fleets into battle, you can pillage and plunder other ships, there's tons of missions to complete, you can duel other characters, seek out buried treasure, and tons more. So yeah, on the surface, you may not think a PC port on a home console from the 80s would be any good, but Sid Meier's Pirate's Gold is absolutely worth playing on the Sega Genesis. Finally, let's go back to the old Super Nintendo, and it's funny, I made a video on Super Nintendo Street Fighter games like four years ago, and I still get the occasional comment from someone saying, wow, I had no idea that the Super Nintendo had Street Fighter Alpha 2. And yes, while it does require the help of the SDD One Chip, one of only two Super Nintendo games to do so, this game actually does exist on the Super Nintendo, and not only that, it actually plays surprisingly well. No, it doesn't touch the PlayStation or Saturn versions, not to speak of the arcade edition, but still, it features 18 characters including Akuma, different turbo speeds, and tons of combos, including special moves represented by the charge meter at the bottom. I wouldn't think so on the surface, but it is a good playthrough, and this game is as impressive a piece of work as you will see on the Super Nintendo. And I want to thank you for watching, and I hope you have a great rest of your day.