So, hey guys, can everyone hear me okay? Awesome. Yeah, so as our counselor introduced me, I'm Anish Astana, I'm a software engineer on the data hub team in the AI center of excellence at Red Hat. And I'm Daniel Wolfe, senior data engineer within the products and technologies organization. Yeah, and we're here to talk to you about machine learning with open source infrastructures. More specifically, we'll be talking about the open data hub and one of our internal users, like the rocket team. So, the open data hub was originally started to address some internal problems we were facing around machine learning at Red Hat. Instead of having a number of different teams manage their own infrastructure, develop their own support mechanisms and security policies, we figured having a centralized location called the data hub where teams could share data and run their ML and AI workloads made more sense. That way we could free our end users from having to worry about the complexities of managing these systems and then focus on what is really interesting to them, like experimentation, getting inside, building cool stuff. The open data hub is a natural evolution of that. It is a meta open source project that brings together a number of open source technologies in like data and machine learning pipelines, all running on Kubernetes or in our case, OpenShift. One important thing to note is that it's cloud agnostic, so you can run anywhere, you can run OpenShift. So, if you wanna run AWS or on-prem or on GCE, feel free to do so, right? Or on all three of them at the same time. You can find a community at opendata hub.io. So, this is a reference architecture diagram for the open data hub. I know it looks like a lot, but I'll walk you through it. The components you can see in this diagram can be broken into roughly three categories. The first category is technologies or projects we are looking into and vetting to see if they would really fulfill the needs we have. So, things like Red Hat Data Grid or Three Span would sort of fit in there. The second set of components is technologies we have proved work for us internally and meet our requirements, but we haven't integrated yet into the open data hub operator. And the third set, and that's like technologies for that would be like Elastic Search, for example, or HUE. And the third set of technologies and projects is stuff that we've proved works and is a part of the operator. So, if you draw your attention to the bottom of the image, you'll see OpenShift. So, OpenShift is Red Hat's enterprise Kubernetes distribution. It's a container orchestration engine. So, and that's what we're running open data hub on. This is what lets us scale up to meet any requirements or run on any cloud. Moving up a little bit, you can see like the data engineer persona on the left. So, data engineers are responsible for building out big data infrastructures. What this really means is that they're responsible for developing and like planning out systems that can incorporate data from different sources and store it in one location or multiple, I guess, for your users to like play with, so to speak. Data engineers are generally dealing with two main types of data. Data in motion, which is like data that may be flowing from outside your system into your system, right, like from point A to point B. And then data in rest, which is data that's already in your system, how are you storing it effectively. So, with those like two sort of categories defined, we are using Kafka, LogStash and Fluendi, primarily internally and as part of the OpenData Hub operator, you can use Kafka. And then from a storage perspective, we settled on using Ceph as our data lake for unstructured data. We're looking into Red Hat Data Grid as an in-memory storage option. And then for your structured data, you can really use any database you like. We use Postgres equal for the most part. Is it, you need one from the data motion layer and one from the storage layer, at least to do the source? Yes, yeah, yeah, you can have more. Depending on your infrastructures, you may have multiple. So, some of our internal users, they have, oh, their systems have integrations put in with our SysLog. So, it made sense for us to incorporate that. But yeah, you can use any, you really need only one from each layer. So, once you have your data in your data lake, you're not done, right? You need to actually get some insight or value out of that data. And this is where your data scientists and your business analysts come into the picture. If there's just some life visualization work they wanna do, you know, just get a very rough idea of what the data may look like. And you can use projects like Hue, Kibana, or SuperSet to look into it. But if your data scientist wants to do something a little more involved, right? Like with processing data or transforming it or starting to train models on it, they can use Jupter Hub, alongside Spark with whatever, you know, Python or whatever language of choice you like with whatever machine-earning libraries you want to use. Now, once your data scientist is done with creating a model, they probably want to deploy it somewhere so that you have end users sort of testing it out, right? So, for that, we're using Selden to serve models which our data scientists have to develop. The final bit I wanna talk about in that portion is actually about the Open Data Hub AI Library. The AI Library is a set of pre-built statistical and machine learning modules that any user can download and can quickly get started with for rapid prototyping. So, say if I'm not a data scientist but I have a lot of data and I want to sort of test out what I can do with it, you can download the AI Library and sort of get started with some prototyping work very easily. Now, if you look over to the other side of the diagram, you'll see the data steward. So, these folks are responsible for restricting access to your data or to your platform, right? They're interested in just authentication and credentials. So, for them, internally, we have, yep. So, Jupyter Hub is... Yeah, Jupyter Lab, yeah. Yeah, so, we actually have it running and you can deploy it as part of the operator. So, just to clarify, we have Jupyter Notebook and Jupyter Hub, Jupyter Lab. So, your data stewards are concerned with restricting access to your data. So, for them, we have, like, currently we're using Red Hat, OpenShift OAuth, and the Ceph Object Store for credential management and, in the case of OpenShift OAuth, you can integrate with LDAP. So, it's very easy to restrict access to certain components which are sensitive. Finally, you have your DevOps engineers, right?  So, they're interested in monitoring the health of the system, right? Seeing if something is down, how the system is performing with, like, metrics. For that use case, we have settled on using Prometheus and Grafana, since they're quite a pretty easy way to scrape metrics and visualize them. The last thing I wanna talk about is jobs, right? You'll have jobs, your users may have jobs they need to run on a semi-regular basis, right? And they can vary in scope from something simple, like backups for, like, your elastic search indices to something a lot more complicated for, like, some data transformations for, like, a multi-stage data transformation for your machine learning modules. You could use cron jobs for that, but they're not very robust and not very reliable, right? Like, if they fail, you have no idea what happened. To that end, we settled on using Argo and Jenkins to manage our jobs. This is not just revision, right? We do have some components already integrated in, and if you were to go to the GitLab repository and download the operator, here's what you can deploy out of the box. So, you can deploy Prometheus and Grafana for monitoring, obviously. Selden, Spark, and Jupter Hub for data processing analysis and then model serving, and then finally, Ceph and Kafka for your data storage, data and motion needs. To answer that general question, you don't have to deploy any component here if you don't want to. So, if you like Jupter Lab a lot and you don't care about Jupter Hub, you can just tell the operator not to deploy Jupter Hub, and Jupter Lab will be at a slot in there perfectly, right? The operator just makes it easy to deploy, like get a prototype going. Next, I'll be talking a little bit about some of our practical deployments right now. The first one I'm gonna talk about is for the Massachusetts Open Cloud, or MOC for short. The MOC is a collaboration between a number of universities in the greater Boston area, as well as some industry partners to create an open public cloud for researchers in academia or like non-profits or in an industry to collaborate on and innovate in. We have an open data hub deployment in the MOC to provide a platform for these researchers and non-profits to develop AI services on and again collaborating on and driving value for them. As a side note, if you are a researcher or a non-profit in the greater Boston area and you're interested in working with the MOC, feel free to reach out. We can put you in touch with some folks there. And then the second bit I'm talking about is the internal data hub, right? Which is where it all started. We have three main goals internally. The first one is somewhat linked together, so I'll talk about them together. We wanna serve as customer zero for any new open data hub components and we want to prove that the open data hub can run in a highly available manner at scale. To expand on that, all of these new technologies and running and installing all of these new technologies and projects on OpenShift isn't always the easiest thing to get started with. Working through all of these, those kinks that come with getting started and sharing that knowledge upstream is gonna make life easier for everyone who's in the community, right? Going forward from there, running at scale requires, again, very specific configurations which may not be obvious initially, right? There's specific things you have to do. So as we have processed large volumes of data and stored it for internal customers, right? We've learned a lot of lessons and we've been contributing them upstream to the open data hub community. Finally, we also want to help drive teams at Red Hat to be more data-centric. And to that end, you know, we're creating blog posts, talks, videos, demos. I'm sure you've seen some talks about some of the work people have done. You've probably seen more of them over the next few days. And some of these use cases can be like, I don't know, hard drive, failure prediction and stuff, for example, right? These are useful things and there's a lot of data but not every of things that they can do these sort of applications. On that note, I'd like to talk a little bit about some of our internal customers. The first one I want to touch on is the products and technology DevOps team. They have applications in their build and product release pipelines that are generating a lot of logs. All of these logs are stored in the data hub. As I'm sure most of you know, not all logs are created equally, right? Like some of them actually really matter. Most of them are just silly debug things no one cares about, right? So detecting these. You don't care about it until that particular component break. Yeah, right? So detecting these important logs is like, anomaly detection is something that some of the BND DevOps teams are actually engaging with the AI center of excellence on. And if you're interested in learning more about one of these, there's a talk later this afternoon by Michael Clifford and Zach Hassan talking about their experiences building an anomaly detection model for that. We also have cluster operational metrics for OpenShift clusters flowing into the data hub for the telemetry project. This is all information about how the clusters are behaving once they're deployed, what the health of the system looks like, and it's helping like OpenShift VMs or tech leads make decisions on what future work they need to do or what features they need to prioritize or what people don't actually really care about. On a similar note, somewhat similar, we also have a lot of customer insights data in the data hub. This is data from things like Red Hat Insights or source reports for the customers. And we have a number of teams internal to Red Hat using that data to improve the decision making, right? This way they can know what work to prioritize, what maybe what customer partnerships do, or ISV partnerships to post you, and things in that vein again, right? Making smarter decisions with that data. One of these teams, one of the teams working with the Insights data is the Grocket team. So I'll hand it over to Daniel now to talk about that. All right, thanks Nish. All right, that was great. So yeah, let's see. So again, my name is Daniel Wolfe, senior data engineer, and you're kind of gonna get a two for one talk because we heard about the open data hub infrastructure from Nish, and now we're going to hear about an actual use case that we're building with open data hub. So I'm gonna cover some slides with some background and get into a little bit of detail and then transition to Jupyter Notebook and look at some code. So the name of our internal application is Grocket, G-R-O-K-K-E-T. And this comes from the word groc, which means to understand intuitively. Well, I wanted to ask by a quick show of hands if who has heard of the term groc. It is an actual word relatively recent. Okay, I'm impressed, I'm impressed if so. So what we are interested in understanding better with groc it is data on workload adoption across Red Hat products. And by workload, I'm referring to categories of applications like database or software development. And there's various key workloads that we're interested in tracking. And so then we can have a better understanding of how customers are using OpenShift and RHEL. So why are we doing this? Well, here's an example business question of how we are justifying our value of this effort. So for example, what software vendor partnerships should Red Hat create or enhance? The traditional approach would be interviewing customers and asking what they're running or looking at online, it's a Gartner research. But with Grocket, we have qualitative data that helps us answer this question. So we know what vendor to prioritize for OpenShift certification or run a co-marketing campaign. So how does it work? The one line response is Grocket works by grouping running processes on a system into clusters based on their similarity. And then ideally, each cluster corresponds to a running software application on that system. So our data is coming in through customers that I've opted into Red Hat Insights, which is shipped by default with RHEL 8. And we get data from thousands of systems and any particular process or any particular system can have thousands of running processes. So with that multiplier effect, this really makes it ideal for a machine learning approach as opposed to a manual review. So that's what I'm going to get into next is take a step back and give a brief overview of the K-Means clustering algorithm, which is the approach that we are taking. So the K-Means clustering intelligently groups data into K clusters based on the similarity of their features. And it is a fairly popular algorithm. And one of its more well-known use cases is with recommendation engines. So for example, recommending movies or songs. So let's say you're a streaming service like Netflix and if you were just watched a movie and you want to keep them engaged, how are you going to figure out what to recommend? So in this example, what you can do if you're the streaming service is take all of your movies and all of the attributes about those movies feed it into your algorithm and it is going to automatically cluster those into similar movies. So for example, if you watch Avengers Endgame, you might also like The Dark Knight. Now that's somewhat of a trivial example because those are both really popular action movies but the algorithm can show relationships that are not familiar to the naked eye and it can do it faster and at scale. So I'm not going to go into too much detail about how the algorithm works under the hood. You might need to be a mathematician to truly do that but it is based on calculating the distance between those numeric features and based on how close or how far apart those features are, it will group those into similar clusters. So back to our use case with the rocket, we are intelligently clustering processes belonging to the same software application. So here I've got a visualization, a K-means clustering visualization and this has K set to three which means there are three clusters and they're color coded here with a beige, a blue and a green. And so each black dot, if you can see in the visualization is a running process on a system. So again, like I mentioned earlier, we can really feed in millions of running processes and the algorithm will group those into similar clusters. So I'm gonna advance here and let's take a look at this beige cluster, a deep dive into it as a simplified example. I've got four running processes shown here and so the end result based on the clustering is that all of the processes have similar words and similar terms. So you can take a look here and see some of the similar terms for these processes. We've got forward slash bin, forward slash fluent D. You can see it's common across all of these as well as Ruby. So the algorithm has determined that these processes, they have this differentiator of having these terms and so it puts them into the same process. Question? Literally just the process strengths. Yep, so it's just the process strengths and when we flip it to the Jupyter notebook, I'll show a little bit more about how we do that. And so now that we have this cluster, we can do a little bit of manual research and with some domain expertise. We know that these processes all represent Fluent D unified logging software application. So then we know that if we scan a system and we see a pattern worth a process has Ruby and then bin slash fluent D, we know that that system and that customer is running the fluent D application. So we can label the beige cluster with fluent D and a couple other examples of a pure Splunk and MongoDB. Is there another question? Yes. Well, so think about the fact that we have, we're getting data from thousands of systems on a daily basis. And so if we were looking at a pure frequency count based on these processes, there's enough variation in the process strings that it would not become apparent. But the clustering can actually group these together so that we know that, okay, this is showing up on a ton of systems. We don't know what it is, but it's one of the most important clusters and then tie it back to the fluent D application. And then from there, then you have a string that you can search all future systems on a daily basis for fluent D. Does that answer your question? Right, right. The data is huge and there's enough variation in the process strings that it would be impossible to do it with naked eye or with frequency count. So the end goal is to be able to model the presence and the prevalence of workloads and software applications running on customer systems. Whereas before we'd have to just interview a customer or do online research with this way we have a data-driven approach of understanding the software and workloads that customers are using for RHEL and for OpenShift. Thank you very much. Okay, so yeah, just wrapping up this slide. Yeah, all in all we have over 200 clusters that we've tied back to software applications and we can then do the modeling from there. Another question. So the question was, I was told to repeat the question for the video. How do you choose the value of K? And there's several approaches to do that and we use one called the elbow method and I can show that to you in the Jupyter Notebook which we will get to in about just a couple minutes. Okay, so before we transition off the slides I just wanted to tie it back to Anish's Open Data Hub presentation. So again, we're building this with Open Data Hub. We use Ceph for our data storage and Jupyter Hub is our coding environment. Both of these are deployed on and orchestrated by Red Hat OpenShift and since his team has set it up for us, it's behind the scenes to us. We're just using Jupyter Hub and we pull the data from Ceph into Jupyter Hub and we do that using the S3 file system protocol which I think is a key point because it reduces the barrier to entry for using the Ceph storage because the S3 protocol is the same as you have with AWS S3. So you don't need to learn any new packages. You can use the most popular packages like BOTO3 or reading in through PySpark S3 paths just like you would from AWS. And that's what we do is we use the PySpark data frames to read in the data, run some preprocessing steps and then we use the scikit learn package which comes with the K-means to run the ML algorithm. So that's what I have for the slides. So without further ado, we can switch over to the code. Forgive me while I get it. So it creates a, it spins up a new pod every time you, and yeah, you can see it through the OpenShift UI. So it spins up a new pod every time you start your server. So how it kind of works is like you have your Jupyter Hub instance running and that's like your master. Okay. Yeah, and then every time a user tries logging into that interface, right? And then they say like start my server. You select what sort of like Jupyter image you want to use.  And that spins up a new pod based on that image. And it's, yeah. Okay, so I'm just going to step through some code here. It does take a little while to run even with a fairly small sample size. So I'm not going to run it real time, but I will just scroll down. So first we import our key packages and then here comes the Spark context setup. And this is where we can put in the S3, S3 like credentials, but for stuff storage and point the endpoint URL to open data hub and access key and secret key. And then we can set our file or a key path and read in the data. And it's partitioned by year, month and day. So we can use wild cards with the or a basic regular expressions for capturing a certain range of months or wild cards for capturing all months or all days. And then read it into a data frame. And then this is a sample of what the raw data looks like with an attachment ID for the insights upload and then the process command here. So these are all the running processes. So it's a very small sample of what we have out there. Obviously some of them are short like at CD and sleep, but then most of these others are just truncated here or are much longer. So we do some pre-processing steps to bring it into a list from data frame format. And then we question. Yes, that's a unique, or does the attachment ID act as an index? And yeah, that's like a unique identifier for that particular upload. So yes, if we did want to tie that back to an account, yes? So, okay, so with the, I guess it's not really acting as a key index. We don't really have a key index for this data frame. We're just going to take these processes out of the data frame and put them into a normal Python list. So yeah, those attachment IDs are not unique. And the process list is not unique. But yeah, we don't have a unique key index for this data frame. So yeah, back to the pre-processing steps. We removed some special characters that are not relevant to the analysis. And we replace all the forward slash with spaces so that we can have each term as a separate word. And then we removed trailing and leading white spaces. And for printing out all the processes, this is how it looks after those pre-processing steps. So just a lot of process strings in there. And back to the questions from earlier, we weren't looking at memory or cash or anything like that, just purely the process strings. So I also mentioned earlier how K-means has a, calculates its distance, or calculates the clusters based on the distance between numeric features. So we do need to convert the process strings into numeric vectors. And for that, we are using the TF IDF vectorizer. As you can see here, it's just two lines of code. And there's various approaches to doing this as part of the scikit-learn package. We settled with the TF IDF because some of the other ones are more geared towards natural language. But with this one, with this use case, we're using machine language. So we prefer one that does not focus on natural language processing. So then all of the process strings are then converted into numeric vectors at this step. All right, so here's the question earlier about optimizing K. So you do need to have K specified when you're training the model. So this is kind of a brute force approach for optimizing K, as referred to as the elbow method. And you can choose how many clusters you want to go up to and arrange for that value of K. So in this example, we go from one to 200 clusters. And so this is the step that takes a while because it trains the cluster for every value of K, as you can see here. And then it calculates the sum of squared distances, which is a way of calculating the air or the impurity of a cluster. And so as you can see in the chart, if your K value here on the x-axis is small, then you're naturally going to have much higher impurity of your clusters because you're putting a lot of different types of processes into a small number of cluster. And then out to the right is your K value increases, then that's where you can potentially overfit the model. And you may have similar processes put in different clusters because it's going to return 175, 200 clusters based on the value you give. So the reason it's called elbow method, as you may suspect, is the optimal value for K is going to be somewhere in the elbow of this curve, which in this example is somewhere in that 50 to 75 range. So after running that visualization, we then train the model with K set to 50 clusters. And that's the step that's done here. And once we've done that, then we've trained the model and we can start running some exploratory analysis on the model results. So for example, we can get the top terms for each cluster. And it's not necessarily the most common terms for each cluster, but it's what's at the center of that cluster from a numeric standpoint. And I've already got it scrolled down here, but the cluster number is labeled here and it's just printing the top 10 terms for each cluster. Those labels are not necessarily significant. It's not like cluster zero is more important in cluster 40, but you can scroll through all of them. And then in this training run, cluster 31 has some of the terms I was referencing earlier with fluent D and with Ruby and USR and bin. And so you can do this for all of your clusters and see if there's any key terms that jump out at you that you think, oh, that could be a running software application. And then continue on down. So the number of processes for each cluster does not have to stay consistent across all clusters. You can see the cluster labels across the bottom and different values for K. And the number of processes shown on the Y axis. So it certainly fluctuates. And we thought it was interesting that there was this big spike here for I think it's around cluster 19 or so. And when we looked into that one, it was kind of like, we refer to it as a junk cluster where it just put a bunch of processes in there that didn't really fit well anywhere else. But that can open up additional opportunities to then run the clustering algorithm on the junk cluster itself and see if you can get a little bit deeper into that. So, seeing or something like that, it's gonna, where does that fit into this, make use of that wider array? Specifically, this example that you have here, like where are you saying, oh, here's gonna be kicking off into some larger thing, and how does it know to do that? Right, right. So for example, in our scenario using PySpark, so all this is an ODH instance with the Ceph storage and with the Jupyter Hub. And then if we really need to scale up our compute resource, we can, from within this Jupyter notebook in ODH, we can connect to a remote Spark cluster that can spin up more nodes and have as much compute power as needed. So it's more seamless for us. And by the way, I forgot to ask the question, but the question was about how does this example demonstrate the use of being able to scale up, right? Scale up the compute power. Does that answer? Yeah, I mean, did we pass that already? Is it explicit? So towards the beginning portion of the notebook, we do specify the Spark instance that you're connecting to. Okay, so it's by explicitly importing Spark. You're saying, okay, that is gonna be making use. You have a URL in there, so there's nothing that really says, well, I'm writing this notebook here and I'm using the Spark instance here. It's almost explicitly going off to something else. Something else just happens to be the open shift implementation you have that this notebook is running on. Yeah. They're just kind of implicitly paired at this point. Well, in this example, this is the Spark master URL right here. So yeah, Anish, did you wanna add onto that? Couple of things that Open Data Hub does behind the scenes. One is if you selected in Jupyter Hub that you have a Spark-enabled Jupyter notebook, it actually spins up a Spark cluster for you behind the scenes automatically and it's your own personal Spark cluster. You can size that to whatever you want. So as you're doing your data discovery or your data exploration or your machine learning, you can size that independently. If you have a Spark cluster, let's say it's out somewhere else on a Hadoop cluster, you can also point to it, but by default we add in some environmental variables that make it automatic so that your Jupyter notebook knows how to use that resource. The other thing we're not showing here that Open Data Hub does is if you have a GPU enabled OpenShift, then we implicitly allow you to run your workloads on a GPU by just selecting a certain parameter when you spin up your Jupyter notebook. That's not something that we'll show here today, but those are the types of things that Open Data Hub does for you. It's automatically configured to use those resources for you. So that's outside of the code here. Yes, yes. So the code here is gonna take, make use of whatever resources you create when you spin it on up. If you start doing something like, oh, I need GPU, do you have to tear on down and reload, or is it like? You would basically, so you would click on that control panel and then there's a stop my server. You stop the server. You spin up a new one with whatever new parameters you want if you wanted to select the GPU. You'd select the GPU and now your workload would run on a GPU enabled container. All right, great. Yeah, so it's getting to the end of the notebook here, showing the number of cluster, the number of processor cluster. Yes, that's true. So the Jupyter Hub, that's exactly what we're doing. It's a multi-user environment. Different users, maybe there's a power user that has access much more data than someone else that's just looking at exploratory smaller data sets. We have the ability to control how many resources, how much CPU, how much memory they're allocating per user, or we can give a default that every user gets. So that is something that we've kind of lost over. It is a multi-user environment and it's allowing us to control the quotas for each data scientist. Yes, yeah. Okay, great. Well, that about wraps it up. I was just on this last cell code cell here. I was just going to tie it back in with the Fluent D aspect earlier, highlighted cluster number 31 as being Fluent D. And then here we can see the example processes. Let's just ignore that it says 10 there because it should be 31 from this latest run. But based on the preprocessing steps that had taken out the slashes and forward slashes and replaced them with spaces, but it looks pretty similar compared to the example processes from the slide earlier. And that concludes our presentation.


Being able to spend time here and be able to keep them right, how beneficial is that to you throughout the season? Well, I mean, I think we're all trying to do the same things. Obviously, we're trying to win. We're trying to make our teams better. We're trying to improve them, and it's competitive. There's great ideas. I think there are best practices, and the data, and the health and safety, and a lot of those reports, and being able to have that data, I think can really help us. So this is a great group. It's a great fraternity to be a part of. We understand how competitive it is, but it's a great group to be able to get to see and work and visit with. Well, I mean, I think I have interest in everything that goes on in that committee. I mean, if there's something specific that you want to discuss, I mean, I would love to. But I think I've tried to take it all as serious as I possibly can and put the time in with the other committee members, have dialogue, listen to all the ideas, listen to everybody's thoughts, and try to come to a consensus on what's best for the league. And you're not going to get everybody to agree on everything, but I enjoy the process. What do you think about it, at the end of the process? No, I don't think that's the, I mean, the injuries in the kick play are significant. The injuries, the rate in which those occur on those plays are significant. It's space, it's speed, and I understand it, you know? But it just makes sense. Like, I don't know if that's going to eliminate, that's going to decrease punts, maybe a few teams go for it. If their punter isn't, doesn't excel in that, you know, it's five yards, but we put it at the 25 for, you know, the kickoff. So if the ball goes in the end zone on a kick, on a free kick, we put it at the 25 yard line or a scrimmage kick, make it just consistent. Punting's gotten so good, you know, they've really far exceeded, really in the last few years, I mean, just the distance and how good some of these guys can be in a plus 50, you know, the different kicks that go and used to stand on the 10 yard line with your heels on the 10 and it's like, don't back up. And now if you don't catch it at the seven or eight, it's going to stop within two yards. That's how good some of these punters are. So, you know, I just like the idea just from the consistency standpoint of putting it at the 25 and just making it just like the kickoff. We've looked at those and we've studied those. You know, then the ball is in play, the ball of the return is in play. So, they can't tell us what the injury rate is or what that looks like on that format, even though that that's what they're doing. We do know that the ball's in play. You know, what are the impacts of 10 players standing five yards across from each other? What's the helmet contact? You know, what is all that? You know, those are great ideas and we'll continue to explore those and look at it. Yeah, I think maybe starting with Daniel, you know, Brunskill, just some versatility. You know, somebody that I think we've always appreciated from afar. Just look at his versatility and his ability to play across the line. I also like the fact that he's self-made. This guy spent two years on a practice squad and this is a player that went and played in an auxiliary league to come back and make it. You know, and so I try to use him as an example to my own son, to Tyler.   Andre Dillard, you know, is a player that we feel like his best football is in front of him with a new opportunity, very skilled, pass protector, which is something that we have to improve on. We have to be able to protect our quarterback. Bad things happen, we've all seen it, we've always talked about it. So I was excited to be able to add him. Sean Murphy, Bunting, he's got really good length, good athleticism, good ball skills and again, I had a situation in Tampa. You know, we've all seen it even myself. It's sometimes a change of scenery and a new opportunity can do players well. So excited about that. Arden Key was a player that has factored and has seen him grow, met with him. A lot of these guys, we've referenced our 30 visits of them coming in and I said, I'm just proud of them. I mean, the guy had a lot of question marks, people had a lot of concerns about him and told him I was proud of what he's done in the league, you know, to come in and be able to figure it out and move from the 49ers to Jacksonville and now be excited to come to our team. And, you know, he's got a lot of great energy. I love his juice, but also his ability to play at different spots across the front on third down and his ability to impact their quarterback. Aziz really just, you know, just getting to know him in his background and his maturity. Really liked the way that he plays, but also just the person and who he is. Am I leaving anybody out? Okay, and Luke, you know, we're just able to add some length and some speed to the linebacking core and a premier special teams player in the past and thought that he had some versatility to play inside, potentially play on the edge in some packages and, you know, he's got some length and has some speed. Well, I mean, I think that, yeah, in those positions, you know what I mean, and there's still a lot of positions that we need to address, whether that's through, you know, multiple ways. But do I think that Luke showed ability to run on special teams? Yeah, do I think that Aziz plays fast? Yes, on film. Do I think Art and Key showed a level of speed and violence when he rushed? Yes.  Just stay between their guy and the ball and finish in the quarterback and finish longer than the guy with the ball. So, you know, we'll see it. That's continuing to develop. At the receiving position. Yep. Sure. Yeah, you know, I mean, I think we're always just, you know, we're trying to put the right pieces in place, right, whether it's a big fast guy, a small fast guy, it's a smaller quicker player, it's you're trying to find what the fit and the value is at every position. And so, we ran an eye and everybody on the coaching staff and personnel side, we all understand that we're gonna have to find some guys that can get open and catch the ball that are hard to tackle after they catch it. And so, we're gonna continue to do that. With Kevin, I mean, Kevin's under contract. Kevin's a valuable, valuable member of our football team. I've said that, his durability and his leadership, you know, and so, we'll see where things go. But as of now, there's nothing to report. The communication has been really good. I tried to make a great connection with him as a player and a captain on our team for five years. And, you know, those are conversations that, you know, will happen between Kevin and his agent and Ran and myself and, you know, try to find ways to continually improve our football team. Well, I don't think that that's, you know, those are hypothetical questions. I don't know the defense. He's been a part of our team for five years, so I can't hypothesize what that would be. Yeah, you know, nobody really can identify anything and changes are made until we go out there and start to play games, meaningful games. You gotta win, you gotta score points. And you gotta take what personality that you have and put them in the best position. So, excited to where things are now with Tim and the conversations with Charles London and Justin Outen, Tony Deuce, our offensive line staff, Rob Moore. I love where it's at, but to say anything more than that, you know, would just be a prediction. I think you have to spend some time with them. You have to be able to figure out who they are, you know, when things aren't good, you know, when things, when there's adversity, how do they respond to adversity? What kind of teammate are they? Can you trust them? Can you trust them to prepare? Can you, are they accountable? And if they're not, are they willing to address it and own it and fix mistakes? You know, it's really just looking at whether you can trust everybody that you wanna have a part of the organization. Well, I don't know if they love a football and, you know, their actions.  We wanna invest in the right people, whether they're players, coaches, you know, anybody that's a part of our organization. Like, we wanna hire great people. We wanna coach great people. And yeah, they have to be talented. But I just know that it's too hard when there's inconsistencies. Yeah, he is, he is. And so, you know, I was happy for Chase to be able to get that opportunity with the Rams, you know, knew that probably when we hired him didn't think it would be long.  And he kept coming back to Anthony Levine, you know, just talking with him and hearing him what he was doing in Baltimore and the type of player that he was. And excited it is to get rolling with him. And he's got a lot of passion for coaching, Tennessee State guy. And he's excited. I mean, he just, and he's like, he was a little unsure, just like a lot of young coaches and former players that, you know, they think they wanna do this. And then finally he's just fully committed and give credit for staying with him. And then Tom Quinn brings that veteran presence. So I was really excited to be able to add both of those people to the special teams, you know, match just an O-line guy. You know, he's kind of, you know, loved the interview, loved his energy, loved his ability to teach and what he wanted to communicate to the players.  Let's say we signed 16 guys, are we going to the training camp with 16 offensive linemen, could easily see each one of those coaches just taking a group and saying, hey, you work with this one, you work with this one, or you have the tackles, you have the guards, you have the centers, whatever it may be.  Had a real, you know, maybe I didn't explain this quite as well, but, you know, trying to bring Anthony along and really have a vision for him and believing in him, but just needed a little bit more experience. And I'm excited that we were able to add both of them. And Tom's going to be able to help Anthony and Tom's going to be able to assist with AUK and Anthony's going to be able to help AUK and our gunners and some of our special teams, our safety, DB, wide receiver, skill guys. So I think they all bring a certain level of a certain skill set to coach and special teams. Well, I think that he's got to earn that ret, earn it, and, but I think that that's, you know, that's why you're making some of these moves in free agency. You know, he's got to go out and earn it just like everybody else, but I think he's going to have the opportunity to do that. I don't really recall last year's draft evaluation. I think that you won a lot of games in UNC. It seemed to be a great leader. You know, I just, that's a long time ago. I'm sorry. In who?  Their toughness, you know, accuracy, you know, how do they take care of the football? You know, there's the turnover margin and the rate of success in our league is, those are two telling of a stat to not look at the way that a player protects the football, whether that's in the pocket, whether that's scrambling, or that's when they throw it. You know, if they're putting it in harm's way, that's a tough way to win in this league is, you know, 75% of the teams that win a turnover margin win the game. Yeah. No, yeah, no, it's, we've decided that we want to make a connection, a personal connection with every player on our team with every coach. And some of those connections and those relationships are stronger than others. I've always said that. And we want to get to know them. We want to be able to help them on the field. That's our first job, but off the field as well with anything that they have with what we have. And we're together a lot. And so that personal connection, that means something. But there is a business side. There is a professional side that we all have to work through. And those, all these players that, you know, I would say that have been here for five years since I've been here really have meant a lot to our program, to our team, to our organization, to our fans, you know, to our coaching staff. And so we'll continue to work through all those professional conversations, the best that we can as we continue to build, you know, a football team that we feel like can win a championship. I don't know what damage control is. I mean, this is just having conversations throughout the roster with players. You know what I mean? Contracts are up or they have years remaining on their contract or players that are in restricted free agents. I mean, you're always trying to have honest conversations with them and understand that there is a business side of this in trying to do what's best for the team. Of course, you know what I mean? Of course we do. That's, you know, we're in March. We're continuing to build a football team, the best football team we can. And Ryan's healthy, he's getting healthy. I would say that, you know, he's putting work in. It's been good to see him around the building. So yeah, you know, we always expect that. But to make predictions, I think I've been through this last year, I'm not gonna commit to anybody being on our roster on September. You know what I mean? I've seen it change too quickly. And of course that, you know, we want Ryan as our quarterback and everybody else that's helped us win. That's what we want. Some, great, I've always enjoyed money. I've always enjoyed our relationship and our conversations. Appreciate his viewpoint on player evaluations. It was always easy to work with and happy for him for this opportunity that he has in Arizona. We got an opportunity, we coached him. He took the coaching, he's extremely hard worker, I'm happy for him to have this opportunity to go and get a significant contract for himself and for his family. I mean, he played hard. He was coachable and he played hard and he was able to produce when given the opportunity he did. So that's a credit to him. All the credit goes to him and what he was able to do. And I always thought, these guys, whether we can bring you back or whether you can go somewhere else, like that's why you do this, is to support your family and to take care of your family and everything else. He was versatile and so he was able to place spaces and spots where we needed him due to injury. But I'd say predominantly he played more inside than he did outside. But in a short week, he played outside for us and he was always into it. Anytime that we gave him a new task or job or role, he was all in. Well, I'm not sure it was, yeah, let's go. I'll give it a try and love his energy at practice. Happy for him. I would like to. You know, I mean, we do have some conversations, I think with a few teams. Can't really commit to anything right now but because of the schedule. But I think that they will probably travel to somewhere and work and then probably host somebody the third week. There's some data that some of these injuries that we can try to mitigate after the first two weeks of training camp. So the recommendation is that the joint practices occur after the first regular pre-season game. So we'll try to do weeks two and three potentially. I don't personally support those proposals. I don't support using replay to address flags on the field or potential flags. We tried that disastrously. And so if that's a word, Ashley but consistency is something that we're all striving for. And hopefully the points of clarification that we've discussed at length with the officials, with the officiating department can hopefully start to close the gap of some of those inconsistencies. I was striving to make it more consistent. Those are my conversations. And the commissioner's right to evaluate the officials. My goal is just to try to make it more consistent so that everybody sees it as closely through the same set of eyes as possible. That you see it as a foul, that Tehran sees it, that Ashley sees it, that Sean Hockley sees it and that the player sees it. Because then everybody, then there's no questions. Speed. I mean, this is a fast game.  And it's like, was that a hold? And if you're not looking at the right place in the mechanics, you know, DPI. You know, I talk with this to Tehran all the time about this. Like we want to not necessarily reward the defender, but we got to give them some leeway for playing the ball. Like 50-50 balls and we throw it 40 yards downfield should just be that. It shouldn't be 65 for the offense and 35 for the defense because there's a DPI in there. So if we're teaching them to find the football that some of that contact, you know, or play the football, we should all see it the same way. And if he's not playing the ball and there's contact, we should all see it the same way that it's a flag and not, uh, it's, no, was he playing the ball? Was there contact? There should be a flag. Well, I mean, I think that you have to, you know, you have to have a foundation for what you believe in. You know, I think we do. You know, I think you have to figure out how some of these players learn what they're capable of and not giving them too much and not overloading them. You know, whether that's using Roger McCreary as an example, like how much do you want to put him inside? How much do you want to play him outside?  Here's my responsibility. You moved 10 yards inside and now you have a completely different responsibility.  Like I went from corner to nickel or I'm a safety and I went from safety down to money or I went from safety to nickel. So that may seem like it's not a big deal, but you know, in the heat of the moment and the speed of the game, though those two job responsibilities based on where you align are very different. So if we do have younger players, then we have to identify what they can do and what they can understand. And the more that they can comprehend and understand and can play more than one position, then we'll do that. But I think that's our job first and foremost. You know, you have a guy like Daniel Brunski, right? He's going to know every position along the front because that's what he's been able to show. We may get another player that may just be able to play one position and understand what his one position is. Sure. Yeah, you know what I mean? We've had communication with both those. We've seen chance back in the building as he recovers from the knee surgery. Jamarco, you know, disappointed. I know he's disappointed, you know, kind of in the injury and coming back and trying to compete at training camp and you know, didn't work out like any of us wanted it to, but he will have an opportunity, you know, and we'll just have to see. But, you know, he's got to come back healthy. He's got to come back ready to go. And I think he understands what the competition is going to be. We have to put, we have to have competition, you know, throughout our roster. That's something that's critical. That's the only way that everybody gets better. Absolutely. As of now, you know, I would anticipate us adding another kicker, not closing the door and bringing Randy back either, but we have to provide some competition there and everywhere else. Just being conscious, and this wasn't, you know, whether, and you can include David Long in that.  It's, we're all, you know what point in time is there, are there warning signs before? You know what I mean? Are you starting to feel something and are you trying to work through it? Or what is the load management? You know, I mean, what are those signs? And just use those two players as examples. I mean, we have to figure out how to get Elijah Molden on the field. I mean, Elijah improved when he played. He helped us and he got better at nickel and Shane and the staff, you know, they were finding ways to blitz them, to have them cover and reroute. I mean, but then, you know, then he goes through a process and he tries to come back with, you know, whether it's, hey, the return to play, they're coming back, hey, I feel really good. Okay, well, just because you feel really good doesn't mean you do more than what we've asked you to do. But that's a good thing, but it's also a bad, like that happens sometimes with Elijah. He's like, coach, I feel great. I'm like, I know Elijah, this was only the plan for today. And then I turn around and then you're back out there, like, and I appreciate that you want to do that. But we got to figure out ways to get you on the field and keep you on the field. So whether it's Christian Fulton, it's just an idea of how do we work past some of these things that have continued to happen? And we've tried to track the mileage, we've tried to track the speed. You know, the acceleration is something that we have to track. I know that was, you know, something with David to start, stop. He's a sudden player. But, you know, we have to find ways to eliminate and try to mitigate some of the soft tissue injuries. Well, no, just being conscious. No, just being conscious of making sure that they're training, that we're all training the same way that we're gonna play. Like, I've asked Frank and Brian Bell and Tyler Rouse to meet with our coaches extensively about the drill work that our coaches do and what we're gonna ask our players to do on the field. It's not about running 40s. It's not about doing one 10s. It's about how do we put these players in similar drills that can recreate the acceleration, the changes speeds. Like, we talk about the special teams play. I mean, we train the release. We train a staggered release, a sudden movement that opens up their hip, that crosses over. Like, if you don't train that and all of a sudden you do that, then you're gonna have some groin issues or we try to train the speed phase where they're running and they got a guy tugging on them. Like, that's not good for hamstrings. So we have talked about that in the rules. Like, can we watch a guy that's tugging a guy to start calling penalties there? Maybe, maybe not. But then now it's the deceleration. It's to come to balance from going 20 miles an hour to coming to balance, changing direction. So we've trained all those in the off-season. Like, Frank meets with Ock and they do the release on the hoop. Okay, they're doing that one day. Then the next day it's just straight speed and then it's straight speed to come to balance, to go from 20 miles an hour to come to balance and then redirect. So we're trying to put them in the same positions. So we just ask them to do that when they're away so that when they come, I'm all set with the 40-yard dashes and the 110s and that. Like, we gotta keep training in the way that we're gonna play and explain that to the players. And the trainers have been great. Like, they all have trainers and we communicate with them and if they go to Florida, they want our feedback. And it's been really cool to reach out to those guys. Hey, here's what we're doing. This is what we talked about at Area Focus for this player and they've all been on board and I'm glad they're all going somewhere and working. If they're not gonna be with us, rather than be somewhere who somebody's willing to communicate with us and help us. Good, yeah. I think it's very good. I mean, I think it is. I mean, we just, our trainer, Todd just won a national award, the NFL for a trainer. You know, whatever the Players Association's anonymous evaluation of our program, we addressed pretty much everything that we could. But I don't want ever want it to be, I don't want a program that needs an anonymous survey. I want our players, like I want to make a connection with them enough and them to trust me enough to say, Coach, I'm not really sure. And I've had conversations with players. Like guys, you have to talk to us and say, Hey, here's how I feel. This is what we would normally do after a game, but maybe you feel a different way this week. And we modify to every single player, whether that's the plan on the practice field, that's the plan in the weight room. This player doesn't squat. This player doesn't do this. This player, this is how they do it. And they have a menu to pick from. And then the return to play. Like these strength coaches, you guys watch, they're with the trainers and they're running from this guy to this guy. And they're doing those same drills that we talked about before a player comes back before we put them back onto the field. So we're always trying to find ways to be better. We're always trying to find ways to keep our players healthy. And we'll continue to do that. And I don't see it and it's not all on the player and it's not all on the strength coach and it's not all on the trainer. It's all of us working together that if there is an injury, how do we get them back as quickly and as safely as we can so that they don't re-injure it and when is a good time to, I always go through this. Can you make it worse? Can you do your job, right? And can you protect yourself? And that's how I evaluate whether a player should come back and play in the game or be ready to go. Yeah. I think it's certainly you can. I don't think you can avoid the position. A lot of these times, these tackles are in space. And if you've ever been, with one of them kitty cats out there in space, they're hard to get to it. Sometimes when you have an idea that you want to take the hat out of the game and a helmet out of the game, sometimes that ends up on the back side of them. And so you either end up in a gator roll to get them down or players drag them down. We have to be able to identify these and say, hey, take care of your fellow player. Can you get them down without coming down on the back of them? But I don't know if we can put that in a rule right now. And we'll just have to continue to identify it. I know the injury rate is significant on some of these behaviors where you get the full body weight on the back of the legs.


In this video, we're exploring the origins of everyone's favorite Christmas present, coal. We're digging into its history and significance while attempting to mine my own for the ultimate stocking sufferer, the fuel of the Industrial Revolution. Everything we use comes from 8,000 generations of collective innovation and discovery, but could an average person figure it all out themselves and work their way from the stone age to today? That's the question we're exploring. Each week I try to take the next step forward in human history. My name is Andy and this is how to make everything. Coal is produced by the conversion of a plant matter of dense forests that used to populate the earth. Coal has been formed from most geological periods, but 90% of it came from two periods 360 to 250 million years ago. Mostly composed of carbon, it is formed from plant matter that decays into peat and then is converted into coal by heat and pressure by being buried. The earliest records of its use by humans dates to 3500 BCE in China and was widely used by many civilizations wherever it was available. Most early use was just easily accessible surface deposits. Coal and coal share many properties, but coal has a few advantages. The first was it was ready made. Coal requires vast forests to source for wood and then a laborious burning process. Coal could be dug straight up from the earth and then burned. So it was used only when mining was cheaper than charcoal production. But it is also more energy dense, producing a hotter fire more consistently. The real value of coal really peaked during the industrial revolution, while many forces combined together to allow the invention and proliferation of the coal powered steam engine. While today quickly shrinking in relevance, coal still produces 27% of the world's power. With the steam engine as my primary goal of the reset series, coal is most definitely crucial. Eventually it may power my steam engine, but now first in the early iron age it also has uses as a more effective fuel for smelting and gorging iron. So I want some of that. A constant goal for this series since the beginning has been to explore the origin and sources for the ingredients we use every day. But when it comes to metals and other minerals, that's always been proven to be a challenge as rarely can many of them be found just on the surface of the earth. Needing to go deeper underground into mines then becomes necessary but also proves difficult for a variety of reasons. At this point I've made it to around 7 mines. Some are mines that have been closed down and now offered tours, most often not allowing you to actually mine them anymore, however there was one exception with AdventureMind where they let us mine our own native copper and even operate some of their mining equipment. I've also made it into one active mine, the Redmond Real Salt mines in Utah where I got a tour and was allowed to harvest a small sample. Beyond that the best bet has been abandoned mines which require a fair amount of risk and off-roading, as well as some questionable legalities. And when even that fails I can sometimes scrounge some of the sources from their discarded tailings outside the mine. In the end getting access to a large supply of minerals has always been next to impossible. That's why I made it a caveat since the beginning of the reset that I only needed to source an initial sample of the mineral before unlocking a larger store-bought supply, as anything more is basically impossible in the current modern world. And even with that coal has remained a challenge and our requests to several mines and sites have gone pretty much unanswered. So when we heard about an abandoned coal mine right next to San Francisco I wanted to check it out and see if we could collect even a small amount. So after a very fruitful trip around California a few years ago, where we got to collect a large variety of minerals and plants, collaborate with several YouTubers, and check out a few variety of power sources for upcoming inspiration, on our very last stop of the trip I wanted to get some coal. Discovered in 1891 off the bluffs right next to San Francisco this mine was tunneled around 200 feet but then never really developed into a full mining venture. Since then most of the tunnel has collapsed and the entrance is pretty difficult to access now. To make things even more difficult the access is only reachable during low tide. We'll linger too long and you might find yourself swimming back to shore. We talked to one adventurer who had recently explored it for suggestions of how to get into it. They recommended only attempting if you have experience climbing knowledge, which I don't have, or to get a ladder. So equipped with the biggest ladder I could rent, I tried to make it in during the one window we had while there was low tide. All right so I'm here at an abandoned coal mine. We have a beautiful view of the Golden Gate bridge right over there and this was supposedly prospected back during the gold rush early days of San Francisco and has been abandoned since and it's only accessible from up there. Inside there's a rope you can climb your way up. The rope climb was never my strongest here in school so I brought a ladder so I hope I can get up there and get some coal. See if I can knock some loose, take it home with me. I hope it's big enough. Seriously. All right well that's too short. It was a 10 foot climb it's about an 18 foot ladder it's more like 20 feet more than that even 25. I don't know if I can reach it. To try to get to the actual cave was a complete logistical failure. I should have brought a ladder or at least someone who's a professional climber probably could easily made it up there. I am not. Also doesn't help that I cut up my hand getting bamboo the other day. Sorry I can't even grip that well. I might have been able to make it into some of them but I'm not confident enough to risk my life. Over I found a cave nearby here and it's got what it looks like might be a few pieces of the coal they've been sticking through and it might be able to harvest at least a few pieces of coal. Never actually seen it before so I'm gonna guess this is probably coal. Take it home see if it actually does anything. It smells like coal. It's a little bit here. Let's take a little bit. Kind of like charcoal which I think is a good sign. It comes out nicely. That has to be cool. Is this the clean stuff? Early bites. All right so I didn't make it up to the mine but I did manage to get a small quantity of coal. All right got my crystal shopping done for this year. I just walked straight into the water. While I was able to collect a small sample of coal from the cliff surface in San Francisco, a larger supply of more pure coal would be ideal. I was fortunate though to have a friend who was able to gain some limited access to the Falkirk open pit mine in North Dakota and graciously collected a bucket of coal for me straight from the mine. Hopefully I'll be able to make a trip out there myself soon in less turbulent times. For a quick comparison we burned some of the charcoal and then some of the coal to compare the different burning intensities and see if we can measure some of the temperature differences. Charcoal max is out at around 2,300 degrees Fahrenheit where coal is able to reach temperatures up to 3,500 degrees Fahrenheit. All right so here we have the coal and next to it for comparison we have charcoal and visually they're almost impossible to tell the difference. The most noticeable difference is that this stuff is heavier almost like a rock which it is a rock. It's a lot more compacted just by millions of years of pressure that turned into coal. So with this we should be able to get an even hotter fire that would be very useful for forging, smelting and eventually a steam engine. Ultimately they're both pretty much the same largely carbon but there is a lot of just trace elements in coal that could eventually be used for things like some of the first forms of plastic. So this is some of the uh higher grade stuff Martin got for me in North Dakota. This is the stuff I got in San Francisco just on the surface there um probably not the highest grade but that's what happens when you're just on the surface. Just by doing a little flame test I can tell some of it is not actually coal but uh but some of it definitely is although obviously not the best grade of coal. So this may have become known as the uh the gift for the naughty children but as a gift to humanity itself it was a crucial building block for the eventual industrial revolution and the steam engine. Now having this crucial fuel the next steps will be building a brick forge to use the actual coal in as well as needing to build the more efficient bellows to provide enough oxygen. Merry Christmas. Merry Zegmuk. Thanks for watching. Zegmuk. If you enjoyed this video be sure to subscribe and check out other content we have covering a wide variety of topics. Also if you've enjoyed these series consider supporting us on Patreon. We are largely a fan-funded channel and depend on the support of our viewers in order to keep our series going. Thanks for watching.