Hi, everyone. Welcome to the SIG instrumentation deep dive. We're going to start by introducing ourselves. Hello, everyone. So I'm Dimang Rezanit. I work for Red Hat as a software engineer. I've been working on Open Chief Monitoring for two years, I guess, and now I switched to the API server. And I also am a core tech lead for SIG instrumentation. Hi, everyone. I'm David Ashpole. I'm the other tech lead for SIG instrumentation. I've been working on CUBE for five or six years now, and I work for Google. Hey, everyone. I'm Han. I've been chairing SIG instrumentation for about three years now, working on Kubernetes for about four years. Yeah. And let's go into SIG instrumentation. This is our agenda. We're first going to talk about what SIG instrumentation actually is and what we do. Then we're going to talk about our SIG sub projects, things that we hope you guys may participate and contribute to. Then we're going to go into some of our main themes of work streams that we work on, like logs and traces and metrics. And then we'll talk about how you can contribute and where you can find us. First, let's go over the charter. Yeah, the charter is basically to cover the best practices for observability across Kubernetes components. So basically, this means while we don't own individual metrics, we own the practices and sort of the format and the tools that people use to instrument Kubernetes components like the CUBE API server, the scheduler, controller manager, all of these things that you are familiar with. We have a large number of sub projects. Here we have listed CUBE state metrics. You guys probably have heard of it. It's quite popular K log, which is used basically in all Kubernetes components, metric server, which is used for auto scaling. And we have many more sub projects. We also work a lot on metrics. So permit these metrics, which are instrumented across all of our Kubernetes components and logs, which are also in all of Kubernetes components. We're working on structured logging. That way you can get structured format instead of dealing with free form text and events as well as traces. So how do we do it? We have bi-weekly triage meetings where we go through all second instrumentation related issues and PRs. We triage them. We assign them to contributors that join us. And we hope that you guys will join us. We review all code changes for metrics and we develop new features enhancements and we'll go over some of the things that we're working on in this talk. And we also maintain sub-projects are the large number of sub-projects that we have. All right. For sub-projects. Yeah. So thank you for the introduction. I will go through the sub-projects, some of which I'm more familiar with since I'm also contributing to them. So the three most popular sub-projects that we have and that the group kind of govern as well as help to, like, better at the community in general are Cubestate Metric, Metric Server and Prometheus Adapter. Even though, like, you may not have heard of some of them, I will explain, like, their purpose and why they enter our scope as well. So the first one is Cubestate Metric, quite the most popular and the one that you are the most likely to have running alongside your Kubernetes cluster. So Cubestate Metric is a simple Prometheus exporter, which is like a piece of software that is able to convert any third-party data into metrics in the Prometheus format. And in the case of Cubestate Metric, we are getting metrics from all the Kubernetes objects in general, since those can be quite insightful in the cluster. But the reason why this is not part of Kubernetes directly is because the metrics can be quite verbose and maybe out of scope for, like, the normal Kubernetes setup. So we have, like, this exporter to expose this kind of data. And for an example, like, we have metrics about pods, about deployment, about stateful data that are useful in order to get any insights on your cluster or even, like, start building alerts when, I don't know, a deployment is failing or whatever. Some of the example that we have is that the Kube deployment spec replica, which allows you to know at one point in time how much replica your deployment had, as well as the status of the replica dated, which means that if there was some failures during a rolling update or whatever, like, you would be able to know it via this metric. So it's really insightful to investigate some failures and how does Cubestate Metric does that is. So Cubestate Metric is watching for events on the API server on a regular basis. And whenever there is a new object that is created, updated, or that there is any, like, mutation towards an object in Kubernetes, Cubestate Metric will get that information and convert it into a metric that would then be able to be collected by Prometheus and you would be then able to create alerts on it. Another project that is, that you may not have heard of but, like, might be running in your cluster because, like, it's essential nowadays in a Kubernetes cluster, which is metric server. It implements one of the API that we support as a group, which is the Resource Metrics API, which is, like, the metric that is in the common such as kubectl top, that essentially, like, allows you to know the resource usage of your pods and nodes as well. And some other use cases of this API is for autoscaling. So most of the basic autoscaling is done via the Resource Metrics API, where you can specify to the autoscaler at which point in terms of CPU usage do you want a new replica of your application. And this is done via the API that we own as a group. And it does that, like, essentially, it's what we call an aggregated API server metric server, which is, like, some kind of third-party API server through which the API server will send autoscaling request to. And then metric server itself will grab the CPU usage or memory usage in the kubelet and then return all this information to the kubectl API server. So if you have an application that's a kubectl that tries to get CPU usage info, then it will go through the API server and then metric server. Another project that is quite similar, but has a more bigger purpose, I would say, is Prometheus Adapter, which supports the three autoscaling API that we kind of own as a group, which is the Resource Custom and External Metrics API. What it does is that, well, there is some limitation when you are doing autoscaling with metric servers, since you only have, like, usage metrics. And you might want, I don't know, like, to autoscale your application based on the number of requests that are currently being processed by your application. And that is done via Prometheus Adapter, for example, but there are many other adapters. As long as they support the custom or external metrics API, they should be able to autoscale based on any kind of signal. And in the case of Prometheus Adapter, what it does is that when it receives a call from the API server to get, like, I don't know, metric server, like the number of items in your RabbitMQ queue, then it will convert that request to Prometheus Adapter and create the Prometheus server directly and return you the results so that your autoscaling pipeline will be able to directly, like, contact Prometheus Adapter to get any kind of information that would be related to your application. So now that we've gone through, like, a couple of projects that we own, I will go through the, like, we, as I mentioned in the beginning, we own the three major observability signals as part of our SIG. And I will now go through the logs. With the logs, like, we own the infrastructures around logging Kubernetes as well as some parts of the logging output that is produced by all the Kubernetes components, like, we don't know what the other group are doing with the logs and how they produce it, but we are responsible for helping them writing, logging into their code as well as, like, writing a good, on how to write a good log line in general. And we've invested quite a lot the past couple of years in logging because, well, before we had no real structure in our logging, the format was essentially inherited from the Kellogg, which is like the logging tool that we are using to produce log all over Kubernetes. And it was just a basic string. So whenever you wanted to correlate that with any other signal or try to do some pattern matching with it, you would have to go through, like, some complex regular expression pattern matching, so it wasn't ideal. So we came up with what we call structured logging, which where we try to have, like, build some kind of format or run logs in general so that they can be easily correlated and aggregated by third party, like, a logging platform. And in general, we try to have fields that are strings that are constant so that we can have patterns that are reoccurring between logs as well as key value pairs that can be used in later on. And we have two format for those today. The first one is text based, which is mostly used by developers because it's quite easy to go through and being able to investigate it is way simpler than JSON. And then we also have JSON, which is what most of the logging platform today ingests. And then it's easier for this platform to build querying on top of these log lines since like JSON is more formatted in general. So that's what we've worked on for the past couple of releases. It was fully integrated in the Qubelet in 121 with the help of a lot of various contributors because it was a very big effort. As well as later on added to the Qubes scheduler, which took quite some time because there was some e-cups, but we've done that in 124. And in general, there are still like many communities component where we would want to add structure logings. And that's why we would need even more contributors to help us build that in Kubernetes. And one effort that we've also found while doing that was that the previous logging tool that we are using in Kubernetes component, which is K-Log, has a lot of flags which might not be meaningful nowadays and flags that you might more see like on the logging platform rather than the actual logging client. So we've initiated a deprecation as well as a removal in all Kubernetes components. And these flags will be removed in 126. So if you are a component owner that are still using these flags, make sure to be worried of it because we will remove it in the next release. But that won't be a problem for any normal K-Log users because we are only doing that for the Kubernetes components, like any normal user of K-Log wouldn't have the flag removed. And we've done that to reduce the maintenance burden that incurred with K-Log in general because there was way too many flags. And if you want to have a look at this particular effort, there is a cap attached to the slide that you can check. But like now that we've had like structural logging, we've noticed that it wasn't complete like we still had some way to improve logging in general, which was still at context in what we called contextual logging. And this is new in Kubernetes like we've added it in alpha in 124. And there is only a couple of components that have it today. The idea behind contextual logging is that if you were to do logging in Kubernetes today, you would have to just inherit from like a global logger that you would share like between all your processes and all your components, which is not ideal if you want to have very specific behavior depending on the scenario you are in. So what we did is that now we will pass the logger via the context, which means that along your call chain you will be able to modify the logger and add some new information to it. The most common use cases that we would have is to make sure that once you reach a certain point in the call chain, you had key value pairs to all the logs that will be produced via this context. So let's say you are in the scheduler and you want that all the calls related to a pod being scheduled attach the name of the pod. Then you will be able to do that via contextual logging just by enforcing this particular level and value to all the logs. And yeah, this is implemented via a new API in Kellogg that was introduced by the group. And there is like many cut changes that will need to be made in order to spread that over all the Kubernetes component. And we really need like some additional contribution for that part of the code. Yes, come please help us. And we will need additional contributors, the working groups, structural logging. So I haven't talked about the group sig instrumentation, but in Kubernetes there is also the working group, which has like kind of smaller groups that are like related to a sig, but are very dedicated to one particular area of the code. Like in the case of structure logging, it was first meant to introduce a way to build structure logging Kubernetes. But now it has a bigger purpose since it's also to cover like contextual logging. There are two organizers, Marek and Patrick, you can reach out to them via the Slack channel. And you can go through also the bi-weekly meetings if you want to contribute. And this is one of the best area to contribute, I guess, as a new Kubernetes contributors because you will be able to discover various parts of the codebase. Since like everything needs to be changed in terms of logging, you will be able to have your hand into many aspects of the codebase. And it's also like well documented how to do the change. So that's a really good area to start your contribution career, I guess. Cool, let's talk about traces. Traces are exciting and actually pretty new to the Kubernetes project. So first, what is distributed tracing just really briefly? If a user makes a request to something, it often passes through a variety of services along its path. And we'd like to have an idea of what path that particular request took. And maybe it would be great if we got a cool graph or something that showed that to us visually. And that's what tracing provides. So there's a thing called a trace context that's propagated between components in your microservice architecture. And that has a trace ID and a span ID. And with that, each component can produce telemetry that we call spans that have a single unifying attribute across all of them. And that way, some trace back end can stitch those all together into a graph like you see there below. So in Kubernetes, we have a couple components that look very similar to this diagram here. The main one being the kube API server and etcd. But also we've gone ahead and instrumented the kubelet and container runtime as well, which serve requests back and forth to each other. A few things to note, we use open telemetry. The open telemetry go project has been stable for more than a year now. So that's what we use for tracing. And we export our traces in the open telemetry format. Okay, so what's the current state? So API server tracing was introduced in Kubernetes 122 and is planning to go beta in 126. So now is the time to try it out. Now is the time to provide feedback. Feel free to hit me up on Slack if you have any thoughts or if you've tried it out and want to see anything different. Kube tracing was just released as alpha in 125, and that's really exciting. And another big thing to call out that many of you may have been impacted by before 126, which is like now. We used an unstable version of the open telemetry library. So if you were trying to use open telemetry and Kubernetes client libraries at the same time, you might have had some problems, but don't worry, those are resolved now. Yay. Okay, so let's dive a little bit deeper into API server tracing because that's been a fun project I've been working on. So this is what it initially looked like in Yeager, which is another CNCF project as of Kubernetes 122. It's pretty bare bones. You can see the top span is for the user's call to the API server and the API server serving that. And then the middle one is the API servers at CD client making a request at CD. And the bottom one is actually coming from at CD and is responding to the API server. So as an aside, there's been this thing that's existed in Kubernetes for a long time, which is called I refer to as log based tracing. So basically, there's something that looks very much like trace instrumentation already sprinkled across the Kubernetes code base.  And so one of the improvements that landed, I think yesterday is that we were actually able to sort of sneak in and change those calls to also integrate with open telemetry based tracing. So now you can see the red boxes there highlighting some of the new things that have come from using existing text based instrumentation and migrating that to also use open telemetry. And actually I have a really quick demo. Let's do that. So first, I'm going to start up. I'm going to start up a Yeager pod locally. And second, I've written this. It's actually just a modified version of the API server integration test. But basically I'm going to configure API server tracing like so and set it to 100% sampling rate. I'm going to pass that to it. And then I've also turned on at CDs tracing here just for the integration test. And I've got it going to a local OTL PN point, which is supported by Yeager. And this will take 15 seconds. Oh, how do I clear the test cache? Do you remember? Let's do it. Okay. There we go. So this will take 15 seconds. We pop over here. So Yeager's up and running, but it doesn't have any traces yet. Okay, there it went. So now we should be able to pop in here and go in. Oh, the other thing I meant to mention is the API call that this is making is it's creating a node object. So that's what we've done. Set up an API server, create a node, and let's see what kind of traces we get back. So I'll do API server and I'll go ahead and look for a create operation. And then I'll click on the first result. And there we go. So this is actually the same as that screenshot, but it's much more fun because I can actually go in and look at all of the interesting attributes and stuff that have been added here. All that actually existed in API server code probably has existed for years, but hopefully is now much, much more usable. And so if I were trying to debug some issue with a create call, it would be much easier now because I can see a lot of details about exactly what was happening. Okay, cool. Let's talk briefly about the cubelet tracing. So this is a cubelet trace from the trace feature that landed in 125. And it's pretty basic, to be honest. So what you can see is the top line is from the cubelet. And that is when the cubelet started the request to create a container. And then the bottom line is from the container runtime, which is where it's serving the request to create a container. So as you can see, it's basically the same amount of time. And there are useful attributes, but it's almost like we could have just used a metric for this. So what's really missing from there? One of the things that's missing from the cubelet is that we'd really love to have some way to tie together a whole bunch of container runtime operations. Like in order to create a pod, I need to make a sandbox. I need to pull images. I need to create containers, right? And I'd like to see how all of those are related. So if I have a pod creation taking a while, I can go figure out which aspect of it actually was responsible for that. As far as general tracing, both in the cubelet and in the API server, we really want to add exemplars to the Prometheus metrics that are associated with traces so that someone using a trace backend and a metric backend like Prometheus can link back and forth between those two. And the same thing goes for logs. You can insert those common IDs like we were talking about earlier into the log messages, and that way you can jump from traces to logs, or even just find all of the logs associated with a particular request. And while I was at KubeCon, I made a proof of concept for what the cubelet tracing would look like if we had that unifying parent span. And you can see the span that we were looking at before is actually like this tiny one right here. So what before looked not very useful. Now that it's put in context, in the context of creating an entire pod, is actually much more valuable. So it just goes to show that it may just be adding a span, but it actually changes the nature of what you can figure out from your trace telemetry. And that's it. Cool. Let's talk about metrics. This is probably one of my favorite topics. I am sometimes referred to as the metrics guy. Let's go over some boring stuff. Yeah. Kubernetes uses Prometheus. Prometheus uses a client server architecture. So basically your software is instrumented with a Prometheus client and exposes a metric endpoint, which is text-based. This is scraped and stored in a time series database. So there was this one time I was on call and basically I was debugging some issue and I had to look at charts except one of the charts that I was wanting to look at had no data or it was blank. And I looked at the underlying data and there was no data. And after doing a bit of digging, it turned out that open source or upstream Kubernetes had basically renamed a metric. So I don't know how much you guys have thought about this, but one cannot simply rename metric because when you rename a metric, the metric actually just ceases to exist. You end up with a new metric. So all of your charts, which reference a metric by name, no, just don't do anything, right? And so there was this other effort going on at the same time called metrics overhaul. And what metrics overhaul was doing was it basically took a huge set of metrics in Kubernetes and tried to conform them to best practices in upstream Prometheus. And this is obviously like a noble effort. We should definitely have, we should adhere to standards. We want to do that in Kubernetes. The problem is this basically broke the entire world's ability to use charts continuously, right? Because you basically just deleted all the Kubernetes metrics from the control planes and cubelets, the scheduler, whatever, right? So as a result of this, we launched one of the larger initiatives in SIG instrumentation, basically something called the Kubernetes metrics framework. And what this does is this introduces stability classes to metrics and allows us to guarantee a certain API around metrics, right? So how did we do this? Well, like Prometheus doesn't natively allow you to add stability levels to metrics or metric definitions. So we wrapped all of the Prometheus descriptors and we added a metadata field called stability level. And so you can annotate metrics with a stability level and then we created a static analysis pipeline. So I don't know how familiar you guys are with static analysis. It's basically like compiler level stuff where we go through the entire Kubernetes code base. We analyze it. We look for all metric definitions and we look for stability classes. Then we parse that information and we guarantee that the structure for a stable metric does not violate the guarantees we provide the community. And so we have built that. We have been improving on that over time. And not only that, but this requires a lot of code, right? Like we don't want to break the entire world's ability to operate Kubernetes. That would be terrible. And we basically nearly did that with the metrics overhaul. We created a repo in component base for all of this code because this is a lot of code to prevent, you know, obviously tremendously bad breakages. And if you want to read the original cap, this was a few years ago that we started this effort. It's called it's a bit.ly link metric stability. We are extending this framework. So when we started this framework, we actually only had two stability levels. We had stable and we had alpha and stable metrics had guarantees to not change for at least one year. Right. So you can guarantee that your charts alerts will not break for a year. Alpha metrics don't have any such guarantees. We are now introducing beta, which is experimental. We have not yet defined what guarantees these have. And we are working on this and we would love community input on what would be reasonable for a beta class metric. Right. We also have internal metrics. These are likely to not have stability guarantees. Some metrics basically describe code or in implementation detail of Kubernetes and therefore changes when the code changes. Right. So these are like highly variable mutating type metrics. Those would be classified as internal would probably not have stability guarantees and you probably don't even want to chart them or define SLOs for them. So we may introduce stability guarantees for alpha metrics. We may this is on the table. You should come to our meetings and we can discuss it and, you know, we would love community input on what would be reasonable. So, yeah, we're establishing the guarantees for new stability levels. The static analysis pipeline that we built basically parses the entire Kubernetes code base, all of the metrics. And it basically allows us to auto generate documentation for every single Kubernetes metric defined in the code base. And this basically recently landed. And in 126, we will have documentation. You will be able to see every single metric that the Kubernetes, the core Kubernetes control plane outputs, the scheduler, the Kubelet API server controller manager. We even have metameter metrics about registered metrics. So we have a metric called registered metrics total. And how many stable metrics there are, how many alpha metrics there are. And the cap. There's a bit link extending stability. We're still working on this. This is an alpha. So if you want to contribute, come to our meetings, we are, we are like, you know, pretty friendly open, open seg. And I think one of the easier six to get involved with. Yeah, it's a little bit easier, I think, for newcomers. So we would love for you guys to get involved. Yeah, you should come to our sig meetings. We have a sig meeting every week. We basically alternate triage meetings where we go through PRs. And a regular sig meeting where we go over like more high level issues and concerns about sub projects and, you know, caps. And, you know, we are also open to any ideas that you guys might have on how to improve observability about Kubernetes in general. Yeah, we welcome you to participate. We're also looking for contributors for our sub projects like cube state metrics and permit use adapter. And in fact, we have been discussing deprecating some of our sub projects because of a lack of contributors. Like basically we can't support the large number of sub projects that we actually own. And we are in dire need of people to come and help us with it. So please, please come. Metrics server and structured logging are also in need of contributors. And you can contact Merrick. He has been driving the working group for structured logging. So these are the sig meetings that we have. They're basically weekly and we alternate like I said before. We have a slack channel sig instrumentation. We also have a mailing list on Google groups. And you can hit all of us up on slack.  We might not respond right away because some of us don't check slack like that. But it's not because we're ignoring you. It's just whenever we check slack is whenever we check slack. I don't use slack day to day for my job. So it's kind of intermittent for me. But yeah, I will respond to you as soon as I see it. Thank you for coming to our talk. And yeah, I hope to see you guys in sig instrumentation. Yeah, so if you guys have any questions, we'd be happy to answer questions about our sig. Hi. Thank you for the presentation. It was very insightful. I was wondering for the tracing of the cubelit if you've ever considered using kernel traces to collect more information. Because I'm actually doing like a research project on that. That sounds very cool. I had not thought of that. So happy to talk. Yeah. We also accept caps. So if you want to write a cap on how to integrate that stuff with Kubernetes, we are like definitely open and amenable to any ideas to improve. Thanks for the talk. So my question is around instrumentation libraries, right? So sounds like you moved to a buttonometry for tracing. And I'm maintaining client from to use a client goal lines. I'm interested like what's the future of, you know, metric instrumentation. Are you also try to kind of maybe converge on using open telemetry go for metric as well or how we can collaborate? Like what's the future here? David and I were actually just talking about this. So we did a lot of work wrapping the previous libraries. I have been suggesting to David that if open telemetry were to actually natively support stability classes, that there would be a path for us to move towards that. Otherwise, we can still get exemplars through the Prometheus library. So, yeah, those are the options on the table, actually. Nice. Thank you. I think, I mean, this, the stability kind of framework, like it's a good feature for client goal lines as well. So, yeah. Thank you. I would say as well, like we're clearly very committed to metric stability and we already have a lot of Prometheus endpoints. So whatever we do, we would make sure that we maintain, you know, the same metric names and attributes and stuff. So in some ways, this is just an internal implementation detail of how we, you know, how we manage code and Kubernetes. Thank you for your talk. So I'm wondering for the tracing part. I have instrumented, like, several operators with native open telemetry, like, client directly. I'm wondering if we will provide such, like, framework or library to instrument for common, like, operators, like, in Kubernetes. Yeah, I think operators are really tricky, right, because they get, they're driven based on, like, Kubernetes watch events. You know, something changed in Kubernetes and now they're reacting to it. We haven't quite solved the context propagation puzzle there, but it's something, I mean, I gave a talk in 2018, 2019 on a proposal that unfortunately hasn't come to fruition. So it's something like a lot of people are interested in and there has been a lot of excitement historically about. There's another project called K-Span, which addresses this partially that you might be able to look into. But, yeah, right now there isn't anything today. Oh, okay, yeah, thank you. Yeah, I was going to ask the same question about K-Span because it makes Kubernetes events to trace it, right? Does Kubernetes event also part of the SIG instrumentation? Yes, so we... I mean, so Kubernetes events are part of SIG instrumentation. So, and that means the, like, definition of an event, not the events produced by components. So you're correct that that is part of the code that we own within Kubernetes. Okay, yeah, sorry. I have one more last question. So I think Han, you mentioned there's a documentation for auto-generated, like, metrics, right? Can I ask how is that implemented? Because I think a lot of open source projects, we need such capabilities. We have a very elaborate static analysis framework, which basically runs through every single file in the Kubernetes code base, analyzes all of the metric definitions, and then outputs a YAML file with basically the entire metric definitions. And then we parse that into basically a markdown file, which ends up in our, the Kate's website directory. Okay, thank you so much, thank you. I guess we can close it here. Thank you everyone for attending. Yes, thank you everyone. Happy keep coming.


Can't Kitty just stop adding new features and focus on fixing bugs? Okay, I actually received that question quite a lot. So let's try to address it. Even via emails, you email me. Can't Kitty stop adding new features? Like I can't decide. I do contribute to Kitty though, so I should be able to answer this one. My personal opinion, personal opinion is that no, that's not actually feasible. It is a very naive idea. And let's actually try to understand why. So we do have something called the feature freezes. And that is roughly one month before releases, which means that you have released one month before you have a hard feature freeze, which means that no new big feature will get inside that you do from that day on will get inside the next release. If you do a big new feature, you cannot merge it to that release. And that is quite relevant because it actually gives us developers one month to make sure that everything works with quality assurance, this kind of things. And it also gives us the promotion group of Kitty, because I mean both the time to prepare the announcement and the video lately, Harron, a contributor for Proma has been doing a nice job in that. So this is extremely important. And it is very rightful to ask, okay, why can't we have that hard feature freeze where you cannot merge new big features for, I don't know, six months, four top a year? I don't know, every time I hear somebody, it's a different number. But okay, let's say that we do it, it's not going to work. And let me explain why. If right now, how it works is that if you do a very new big feature, during this hard freeze time, it will get merged, but not for the next release, but for the one after that. So we branch at the hard feature freeze, and we only merge the new features, new big features to the next release, the one in the future, whereas the closer release, we only do bug fixes. That's kind of how it works. So in order to have a hard feature freezes, that lasts for like four months, six months, you would require to have new merge request about features, and you would have to go to them and tell, no, this is not going to get landed until six months from now. Couldn't you just tell people not to make new merge request about big features? Yes, you could. Nobody would, nobody would listen to you, honestly, because listen, a lot of new features that we receive merge requests for are coming from people that are not very, you know, there are new commerce to the KD community, which is very nice. You come to the KD community, you propose something new, and you actually implement it. And those people will, we can't actually effectively tell those people to stop making new features. They are still going to come in, they won't, you know, really care about that, and they will do the thing for themselves, and then they will try to upstream it to KD community. There's also a lot of people with small distros that work on making KD Plasma better for their distro, and then upstream the change to KD Plasma itself. In order to have an art feature freeze, we would have to look at all of this merge request, which again, I can bet you would still be coming even if we publicly announced that we don't want them anymore. We would have to look at them and say, no, this is not going to happen until six months from now. Then there's core contributors. And again, I can bet you that even if we internally decide to have an art feature freeze for six months, core contributors will still do new features. The only thing that will change is that they won't propose them. They will keep on their local computers and just wait for the right moment to actually do them, because KD Plasma core developers are KD Plasma users. And whenever they see that they are missing some functionality that they do want, they are going to implement it if they have enough time. And keep in mind that most of currently KD does not hire any developer, which means that we are either doing this as a hobby or by being paid for other companies. And if we're doing this as a hobby, you can tell us what to do, you know. Yes, core KD developers are still going to make merge requests. The only thing that's going to change is that they will wait for the six months period of time to pass, which means that we would have like six months of not big new features, just bug fixes. And then after that, a ton of merge requests, all of the people that waited for that moment put all of the merge requests right there. And of course, you do also have to rebate some changes because six months is a long time. They could can change and you have to rebate your changes to the latest version. And that also takes more time. So you're making people lose time, in fact. And of course, as soon as that happens, like 100 of merge request, well, you're going to get a lot of new bugs because new features, new bugs, and okay, you've gone six months of bug fixes, just bug fixing. And then hundreds of merge requests will come anyway. So if you do a very long hard feature freeze, sooner or later, you have to finish that very long feature freeze. And when it finishes, you get all of the changes that people waited until that moment to submit, which makes the whole thing pointless. As always, this is my personal opinion on what would happen. This is my intuition. You can disagree. So is everything lost? Like is KD always going to implement a lot of new features that they don't have the time to actually bug fix? And I strongly disagree on this one. There is a way to tackle this issue. And I do think that there is currently an issue of too many bug reports. But the solution is not to just say, oh, they just have to focus on bug reports. And that that's it. It's much more complex. So firstly, we need more testing. And it's very easy to say, it's very complex to implement. And you might say, what do you mean? Doesn't KDE have tests on all of their code? Well, you know, some libraries does have like test, which makes sense. But most of KDE Plasma is UI. And it's not super easy to test UI. There are interesting projects. Like as an example, OpenSusa has this OpenQA tool, which basically runs some, um, runs some, you know, activities like pressing on buttons and such, and then takes a screenshot and then compares a screenshot to a reference one to make sure that everything aligns correctly. However, this is very hard to maintain in the long run. There has been attempts at bringing OpenQA to KDE, but those always failed. There are other ways. As an example, recently, there was a proposal to use accessibility accessibility tools to check for UI, you know, to do UI testing. That is also interesting. But to this moment, nothing like that is implemented, which doesn't mean that it will always stay like that. It means that we have to improve. And in fact, at the last KDE Academy, a new goal was selected, which was the automatize everything goal by Ned Graham. And one of the goals is to make sure that testing is better. Then there's also a lot of things to say about quality assurance. It's not just about testing. It's also about making sure that there is a team of QA people that test things manually to make sure that they work right before releases. And currently, there is no such team. Like there isn't, there is one, but nobody is in it. So it's not doing anything. Once we have a QI team, we can make sure that every merge request is approved by that QA team before actually merging it. And keep in mind that those things are all things that sounds very exciting that will actually improve KDE Plasma stability and reliability, regardless of how much KDE developers focus on backfixing on adding new features. Because whenever they want to add a new feature, there will be more testing, more QA and that new feature will be more stable when it actually reaches users. So this is extremely important. And really, this is the way to go. Finally, I want to end this with my impression is that KDE developers, myself included, know that there is more focus to be put regarding making sure that KDE Plasma is stable. And to be honest, if I take KDE Core developers, 95% to 100% of their time depending on the developer is spent on backfixing. Most of the stuff they do is backfixing. Whenever there is a new feature, usually it does not come from my KDE Core developer. The refactor, the backfix, they make sure that everything works nicely. So KDE developers are extremely focusing on backfixing and such. That is my impression. But of course, more people that come from outside and just joined the KDE community, they don't quite maybe have the skills yet to do larger factors. They don't quite understand the code base as a whole. So they add new stuff because it's very easy, it's very exciting to do, and it's easier. So usually you start off by adding new features and then you have the burden to backfix them. Again, this can be improved, I think, as a part of the automatized things, making sure that whenever there's someone new, maybe even young, that proposes new changes, new features, they also have all of the help from a QAT team to make sure that these new features are free from bugs and also that we can try to onboard people directly onto backfixing. But if you look at the last release of KDE Plasma, 5.26, that was extremely focused on backfixing. And if you look at the release before that, 5.25, that has some major new features that were very exciting. But we're all about making sure that KDE Plasma is usable for users. As an example, touchpad gestures. They're so incredibly necessary. You can just live without them. Or I don't know, the new overview screen is actually part of a rework of all KDE K-Win effects. So there has been a very large factor of effects and now they're much more simpler to maintain, which means less bugs. So again, we are incredibly focusing, from my point of view, on backfixing instead of adding new features. If we look at past Plasma releases, most of the stuff is backfixing, making stuff more stable and so on. However, I do want to say that there is always a need to add new features that make a product exciting. We cannot go maybe one release, maybe two, and that's really risky. We can't go a whole year without adding any kind of new release. Because I know that for KDE Plasma users, they think that we'll use KDE Plasma for their whole life. I don't think that's true. But you need to realize that in order to onboard other people to KDE Plasma, which is a goal we have, and in order to keep KDE Plasma users excited about KDE Plasma and wanting to use KDE Plasma, some new features you have to implement. You have to bring new things that users are asking for, such as touchpad gestures. And so often, I see people that say KDE Plasma should focus on backfixing without new features. And then they ask for features like they say KDE Plasma should be more stable. By the way, touchpad gestures are super important. KDE Plasma should have them. That's a gigantic new feature. There's a contradiction there. So new things are necessary. Just like final example, and then I'm out of here. Plasma 6 or whatever it will be called, there will be most of the focus on Plasma 6 will be on transitioning to Qt6 to frameworks 6 and making sure that everything works nicely, reliable and stable. This will be the main focus. There won't be any new shiny super big features. That will be the main focus because we know that's what KDE Plasma needs the most, I think. At the same time, it's Plasma 6. And if you go and announce Plasma 6 without any new features, any, well, that's a bummer from a promotional point of view. And we do need to keep in mind the promotional side of things as well. So KDE Plasma 6 should have some new features. Some maybe not like meant for KDE Plasma 6, but something that will excite people and make them use it. And then when they actually use it, they will see all of the focus that we have put into making it more stable, and then they will stay on it. To make an example, currently, there is a big rework from KD Core developers going on about multi-screen setups. We very much know that there's issues regarding multi-screen. We do see the bug reports. And there is indeed a rework on actually trying to address these kind of things. They're just not that easy to address, so it took time. So hopefully Plasma 5.27 or whatever is next, I don't remember exactly. We'll have this rework and will be more stable from this point of view. Me personally, I've done a lot of bug fixes. Just in the last week, I think I've done like five, six. That's a lot for me. So there is focus on bug fixing. But to say that we just should focus on bug fixing instead of adding new features, that's not going to work. That's not how it works.


In this step, we're going to look at importing static measures into our project so we can really start adding some detail to this level we've created. So the first thing we need to do is a little bit of organization and we're going to make sure that we put all the things that we're bringing all the assets into the right place. So the first thing we're going to bring in is the cabin. So I'll just plop a picture of the cabin on screen so you know what we're working towards. And what we're going to do is put that into the geometry folder because we're going to bring in the FBX file first of all. So this is our geometry folder. So if you go into measures here, you can see there's some stuff already in there that's provided by the start content. But we just want to put it in the top level geometry folder so that we can kind of differentiate what's our stuff and what's the the epic stuff. So I've already got these files prepared. So the cabin here is the FBX file. So if you've got access to my files, then you can absolutely drag that in. If you want access to my files, follow the link in the description below for instructions on how to get to them. If you don't have access to mine, just use your own. That's fine. So I've got cabin.fbx. It's a really simple mesh that I just created myself for this exercise. So I'm just going to drag this into the folder here. There we go. I'll just minimize if it lets me minimize that. There we go. So this opens the FBX import options and we need to make sure that we get these right so that the mesh behaves how we want it to. So the first thing that we need to be aware of this isn't a skeletal mesh. You use skeletal meshes on things that are going to be animated. So characters would be a prime example of that. But what we are going to do is turn on auto generate collision because I built this cabin. I know that there's no collision mesh included in the file. So we're going to let Unreal auto generate one for us, which is fine. And then for what we're doing, everything else can just remain the same. So we'll just click on import. And there you can see that's now appeared in our folder. So that is pretty much that done. What we could do now is just save it so that that kind of saves. You see you've got the little asterisk here. So if we save that, that asterisk will disappear. And then if it crashes, which is done a few times, then that cabin will be there when we open it. OK, so the other things that we need is there are quite a few textures associated with this. And we need to get all those important to the textures folder as well. So let's go to back to content and into the textures folder. And you can see we've got quite a few in there already. So what we will now do is go back to this folder and we'll just select everything. Apart from this cabin because that's already imported. And I'm just going to drag those in here. And this is absolutely the same as just importing texture files by clicking on import. But we're just dragging and dropping now as a different way of doing things. And there you can see these have all now been imported. The new ones have got the asterisks next to them. So we've got, you can see this beam one here. There's a burnt wood one which we'll use later. I'm not sure why that's in that folder actually. We've got door diffuse roof tiles. So everything's there ready to go. And again, we're just going to click on save all so that all these textures will appear back in the project should this crash. So save selected and it will go through save all those. And then we're ready to move on to actually getting this into the level. Okay, so now that everything's imported, we'll just go back and have a little look at the geometry for the cabin. If we open it up, so if you double click on any piece of geometry, any static measures, it will open up this, which that's not the best view of it. Let's just zoom out a little bit. This is your your static mesh preview window. There it is. Beautiful, isn't it? So as you can see, it's a really simple mesh, but we kind of need to know what's happening here. So there's a few things that we can look at. So the thing that up here that I'm most interested in at the moment is this collision. So we've had an auto collision created for us, and that's represented by kind of these purple lines just here. And as you can see, it's okay, but it's not entirely accurate. If we wanted to stand up on the roof, any reason we'd have to stand on here, which wouldn't match up. So we'll change the way that collision works, but we'll do that a little bit later on just so that we're concentrating on that when we need to. So we'll just turn that collision off for a second. And what we also need to look at are these material slots here. So the way I've built this is I put multiple materials on it. It's a lot needed to just build this whole thing with one material, but I've got a lot of tiling of different materials going on. So I wanted to do it with different materials. So you can see at the moment, because we've not imported the materials with this, it's just got the world grid material attached to everything. But it does tell us what materials go where because of the way I named them in Maya. So you've got M underscore slats, M underscore window, M underscore beam, a default material. I don't know what that is. We've got M underscore tiles, M underscore chimney, M underscore door. If we build all of those and slap them onto here, that will then make this cabin ready to import into the level. And that's what we're going to move on to in the next step. So we need to fill all those materials slots with the correct materials so that this looks beautiful and sexy and then we can drop it into the level. So I'll see you in the next step to build some beautiful materials. Thanks for watching. If you really want to take your learning further than I can cover in this series, then I highly recommend checking out PluralSight. They have loads of really detailed video courses covering game art and game development using Unreal Engine 4. When I learned how to use Unreal a couple of years ago, this is where I went and I log in regularly to take a new course and improve my skills. I recommend checking out the introduction to Unreal Engine 4 course by Joshua Kinney. This is really good and offers a good overview of what you can do in Unreal. You can get a free 10-day trial by using my link in the video description and you get full access to all of their courses for that time. At the end of your 10 days, you can either subscribe for more or cancel, totally up to you. It's got to be worth a free trial though, right? I'd like to say a massive thank you to my patrons. Your support helps me to keep making videos like this one and I really appreciate each and every one of you. It really blows my mind that people will support my channel and my work by pledging their money through Patreon. So again, thank you all so, so much. If you aren't already a patron and you'd like to offer your support, then please go to patreon.com forward slash Shane Whittington.


Well, big hand to Karen, that's a tough act to follow. I hope he didn't scare you too much.  And that is a difficult pitch. My name is Javier and I'm a Spanish civil servant. I am the co-leader of the technical team advising our permanent representatives in Brussels. I will first dedicate a few minutes to explain the Byzantine implications of how the legislative process is going. So we are under the Spanish presidency of the commission. That means that the permanent representatives of Spain have a function of secretariat, of owners broker. They represent the whole council. In this part of the legislative process, there are three key players. There is the council representing the member states, which has the representative legitimacy of their member states. There is the European Parliament, which has the democratic legitimacy of the votes of the people. And there is the European Commission, which has an advisory role and proposes the laws. The European Commission proposes a law and both the European Parliament and the council, the member states, draft different text proposals. They have different mandates. Then they both sit together and hammer out a common text which is then voted in the parliament. We are now in that process. That process consists of several readings, but the readings were a very cumbersome tool. So they invented the trilogues, which are a more flexible tool. But the trilogues weren't flexible enough so they invented the informal technical meetings. And those we have almost every day. So in the position, in the point we are now, we have the mandate of the council, the mandate of the European Parliament and the different proposals that come up that are confidential, although they are leaked sometimes. What Karen has explained to you right now is the product of a leak. I will refer to it because it's out there, but I cannot tell you what is going to be said today or what was said tomorrow. So, well, recital 10 of the CRA is the one that states whether you are inside the scope or you are outside of the scope. In the mandate of the correper, the commission of permanent representatives, the council means they had the exclusion of non-commercial entities. You have to not monetize at all. They speak about indications of commercial activity because there are some non-profits out there that are very lucrative, that sometimes they charge supplies, that sometimes they charge for tech support and they monetize in many creative ways. And they also take into account the circumstances of development of financing. The European Parliament also thought of other forms of monetization, such as recurring donations by commercial entities or indications like having a single entity or generating revenue, or the fact that all the contributors to a certain open source project just happen to work for the same company. And now, well, there we have some possible additions that are being put out there, taking into account the open source software, like, well, it's a unique nature of being collaborative and the role of the stewards that I will point to in the future. I want to make a stop here and insist the CRA has a reason for existing. It is very important. We are not in the 90s anymore. The computer at home which could have a virus is now connected to your wearables. It's connected to the wearables of others. It's connected therefore to the network of your employer. It's connected to the whole value chain of the employer. And each of these elements, each of these goods and services can be a vector for infection. And we live in a pandemic of infections, a pandemic of ransomware and malware. And it's creating a lot of problems for companies and for citizens. That's why the European Union has two big laws, one for goods and one for services. One is the NIS directive, the network and information system directive for services. And the other one is the Cyber Resilience Act for goods. As Kieran said, we had the red directive, the radio equipment directive, which ended up covering all the radio equipment, including Bluetooth, Wi-Fi, mobiles. In the end, it covered like half the world of IoT. Now we have included all the products that can be connected by cable and also software because software was indirectly in the red directive, but it's a product too, and it's a vector for infection too. And all these laws rest on a specific imperfection of the market, which is that the incentives for the seller are not to take into account, wholly, the welfare of the value chains the seller is contributing to, because what the hell, I have my legal liability, but nothing more. So we want to create a better environment for the whole value chain.  I give my thanks to Benjamin from the European Commission because this is based on a scheme by him. So, as we have said, we have two criteria. One is the money, where does the money come from? Does it come from a single contributor, Mr. Maniwax here? Does it come from several contributors each pitching in, or is it the project not monetized at all? And also we have how much the excess of control of collaboration, there can be a single vendor, a single shop, which would be compatible with the model for the new legislative framework. As you know, the new legislative framework governs the whole CE markings and governs everything that is sold in the EU. If you buy a pharmaceutical product, it has a CE marking for pharmaceutical products. If you buy a toy, a children's toy, it has a CE marking for children's toys, saying it's safe. If you buy a piece of metal protection for a road, it has a CE marking too. Well, you can have managed collaboration. The key word here is governance. Does your collaboration have some governance? Have someone who is the captain? And you can have total energy. You can have a non-managed collaboration with no governance at all. And that is how we envision the playing field. Of course, the real world is far more granular than what a law can say. We know, but we have done our best. So the initial purpose of the commission was to include, not to include anything that is not monetized, of course, and not to include anything that had zero governance, but to include within the scope of the CRA, everything else. Of course, this was thought to be too much because, for example, in this part here would be things like some parts of Red Fat, for example, because they have one responsible entity and one stop shop. The Linux kernel would be around here because it is managed collaboration and it is paid for by contributors. But I mean, we heard bad opinions about this. And we have also what is in the mandate of the core part, the council mandate, which is one of the two texts that are now to be debated. So within the scope is only in this text one single responsible entity and one single vendor. And in these fields around, you see some question marks here, on a case-by-case basis, the market surveillance authority of each member state could say, yes, but in your case, you have to be included here because you have many contributors, but you look more like one responsible entity because of your specific model. You have a managed collaboration, but it's so managed, so heavily managed that it looks like a single vendor. So it has good sites, because it lets and also the cybersecurity pledges of FOS, a part of this model. It's good sites and it's bad sites. This is all a trade-off. There's no model that is best in our Goldilocks point where everyone is happy. You buy advantages with disadvantages. One of the disadvantages is that it is more unpredictable because the market surveillance authority may make a decision in your case, which you don't like. And it's also black or white. Either you have the full responsibilities of a responsible entity, of an NFL, an LF vendor, or nothing. And it's very difficult to go from one point from one to the other. Then there is the parliament mandate. The parliament mandate was very broad. They wanted to include everything. But what they've had as of late without going in too much in the discussions, which is confidential, is that they have come around to lessening this a bit. And we have some possible image of a compromise which has been floated in what is called the October text, which has been leaked, that will have two figures. One figure of the NFL vendors, the manufacturers, people who are putting their stuff on the market. And the others would be called stewards, for lack of a better word. These stewards wouldn't have a very light touch supervision regime. Their obligations wouldn't be very much because they are just intermediators. Their obligations would be to intermediate between the market surveillance authority and the pool of contributors from different places. And the integrators. They would also be a way, a step up, between not having anything and having something. And some stewards may in due time come around to be full NFL vendors if they get to be successful enough. This is the proper solution, which I personally think would be best for several reasons. One of them is that, well, in every sector, companies come to us saying, we don't want separation. Leave us out. But it's called that. Having a CE marking is a very good thing. There's a reason why China puts a China export symbol that looks very, very much like the CE marking. And that is that if I'm a manufacturer of this remote and I have to integrate chips and software in it, I can have two options. I can buy chips that have already a CE marking and I don't have to worry anymore. Or I can buy chips that do not have a CE marking and then I have to comply myself with all the duties of certification, of revision, of managing and notifying the vulnerabilities, et cetera. And if you are an open source provider and want your software to be used out there, you will be astute, your producers are not going to choose you because you are too costly and besides, it's not a very manageable cost or a very predictable cost. The cost of an economic vendor is much more predictable. But through this model of stewards, you may in a very lightweight way comply with these requisites and have more market opportunities. And that's what is, I hope I could tell you more but this is currently being negotiated over and I also concur with Kieran. I sincerely desire that we would have more time to give it our good, another go and revise a better possible compromise but time is on us, the Spanish presidency ends very soon and after the EU elections, all these negotiations would have to start from scratch. And the parliament, both parliament and council are very, very keen on getting this over with as soon as possible. Thank you. We do have 10 minutes for questions. Oh, as many minutes as you like. As many minutes, probably 10. First question there. Thank you so much. Is this working? Yeah, for this interesting presentation, the situation seems to be quite complicated and after seeing all those metrics, I'm wondering whether you have created any sort of taxonomy reflecting what would be the implications for European businesses or for European citizens of all the different options there if open source is in some way impacted by CRA according to the current text of the law. So I don't know, is that information? That's a very good question and a very broad one too. So Karen harped a lot on the basic fear of most EU regulators, which is let's not give the final killing blow to our industry. Let's not scale producers away, but frankly, cyber security is here to stay. Cyber resilience will be required, not only in the EU because the problem I have touched upon, the rationale for the CRA is a real one. We live in a cyber security pandemic and you don't want to be the guy who says no, no, no mask for me, I'm asthmatic because you may get away with it, but then you will not be making many friends and influencing people and you will not be commercially successful. So what will be the impact on businesses? Well, the good side of this layer of regulations is that the impact all businesses, all software producers and all producers of goods shall have to comply with the CRA. Of course, we haven't touched this because the CRA is very broad. Depending on the criticality of your product, you will have to comply with harsher requirements or with less harsh requirements. And like 90% of the products out there shall make do with presenting a self-evaluation. So in the end, it will not be so harsh but will this kill off the European businesses? I think not because in the United States this is going to be implemented and in China, in other nations too because this is being demanded. It's being demanded by citizens. This is another part of the thing. We get a lot of pressure from industry lobbyists, from all kinds of manufacturers of equipment, you may imagine, but consumer defense associations are much less effective in loving us and we want to listen to them too. And it's for the sake of the citizens that we do this. And also for the sake of value chains. I mean, the harsher requirements and things like machining tools and things that are used in critical value chains. Cool, yeah. So one of the questions that remains is that you said that, well, CRA is of course on the EU side and then China and the US are also working on their own CRAs like loss. How is the, how is the parliament or how is the EU? We live in a very broad and connected world, right? And most of the, at least of the open source software comes from engineers all around the world. How would this apply to software that's maybe, I don't know, started in the US and then continues in China? And also how do you plan to, not you of course, but the EU stays competitive in terms of productivity and software, whereas the other CRAs around the world might not be as restrictive as this one. That's a really very good question, thank you. Well, as you have said, I'm not planning anything. I'm a small cog in the machine and humble civil servant but I'm inside of the machine so I can see things. The CRA applies to all kinds of goods. I mean, you see your own specific part of the box but it applies to all kinds of industrial goods. Everything that can have a data connection and in the modern world that includes shoes. So yes, that is the case in many places because value chains are all around the world and the things not made in Indonesia and not made in China and not made in Europe they are put together everywhere. So this is not only the case for software, this is the case for everything. And I mean, the EU has a lot of experience in this because the new legislative framework substitutes the formal legislative framework and this is the whole business case of the EU about having a customs union and having free transit along that customs union. What differences does software have with all the physical goods? Well, for one, software is immaterial. So when you talk about physical goods you talk about things that come in containers that arrive in a harbor and are put into a train and they arrive to a very specific member state and it's the members of various authority in that member state that guarantees the CE marking or things that are fabricated in some member state and they are guaranteed the CE marking in a member state or in, for example, or in other countries that have a customs union with the EU as in Turkey or in countries that have a specific kind of a treaty of mutual acceptance. He is looking at the United States and where this is accepted. But in the end, you have got a manufacturer and this manufacturer puts together things as long as the things he manufactures, CE compliant and have a CE marking and the things he buys, CE compliant that have a CE marking, he's okay. And if they don't, he's responsible for that path in software too. A very good question is, but what about the physical presence? Where is the software sold? Well, I personally wanted the software of the CRA regulation to be like the digital services in the NIS directive, which have the main establishment or main representative policy, which is if you sell digital services in the EU, you have to answer to the NIS authority, to the cyber security authority of the place where you have the main establishment. As things stand now, this is not the case here. If you sell in Spain software, you'll get to answer to the market surveillance authority in Spain. Of course, pulling together the supervision resources of member states makes a lot of sense and it makes more sense in software because it's not people standing at a specific harbor and watching things come through, it's more centralized. And there are already initiatives by the European Commission, like the Cyber Solidarity Act, where they talk of pulling all resources, of putting together resources. So I see in the future, some future model coming of unification or pulling together of these supervision actions by member states for the sake of efficiency. But of course, the EU legislation comes more at a glacier pace.  Data protection started in the 80s and now we have it's more or less ripe. So every six years, we will be getting a new CRA. As in every six years, we are getting a new NIS directive iteration. And this is one first shot. This is a button, the start shot. In six years, we will hopefully see how this has been working and perfect it.


Everybody Matt here for late in sports cards ripping 2023 panini select wwe hobby six box break number two random box style I'll random the names and the box numbers Here in just a sec, but these are boxes one two three four five six left to right All right. Here we go first the names in the box numbers and we will randomize five times. Good luck One two three four and five Jim J down to Tom K And five times on the boxes One two three four five two down to five. All right. We have Jim J to Josh G3 Sean p6 Alvin S One and Tom K four and five These posted and we'll get started. All right. There's a link to the results. I know that's not posting over in YouTube Let me uh drop a link in the YouTube chat as well Try to shrink that down to where I can see what I'm doing here. Yo Alvin S with box number one Good luck. Good luck to everybody. There's Jake the snake Roberts Blue that is to 249 Morgan Torrey Wilson Silver and Nice that is tie-dye patch auto Shamus the 25 there you go Alvin. Nice. Shamus patch auto very Ripley hyper There's Andre the giant tricolor Biggie and that is Santos Escobar orange 49 and Jorton Give me you so Orton Timmy, how's it going? RVD and to 60 Brutus beef cake. Nice Brutus the barber beef cake There's an orange Emma Also, or that one's to 49 the rock Hatterton Kyle. Oh, are you watching late last night? Oh loss blue Grayson Waller Stone Cold Bautista and Eddie Guerrero tricolor Cody and Brock Arcado for your roads and blue Ray Ripley system 99 shirt relic red Alexa Bliss Bautista diesel like blue Joe coffee William Regal and DDP Last pack for you Alvin and coffee Cameron Grimes tricolor Oscar and Randy Orton, right? That's it for box number one Jim J. Has box number two Trish John Cena tricolor Ray on cross and John Cena axiom and that is tie-dye to 25 JC Jane So there are tie-dyes that aren't autos 11 of 25 Prince Ozawa Omos tricolor Lita And the godfather Sonia Deville Eddie Guerrero silver And to 35 Joe coffee auto red or auto blue So a little bit of red Becky Lynch We have revolution WWF after this then we'll get into legacy football Stone Cold blue Oscar and Roman Reigns Scott Steiner tricolor Dolf silver la night Carmella silver and a redemption Cruz del Toro That is gonna be a legendary signatures red prisms card number 17 Stone Cold Steve Austin congrats Jim J You go Jim congrats on the stone cold silver Trish and Finn Balor mankind and true gulag Purple to 99 Ricochet Shirt relic red and carry on cross. This is Matt Riddle Orange 49 Roman Reigns and taker something Prince maroon Rock Last pack for your Jim Stone cold. There's Randy Orton silver Kane and Shotsie. All right. That's it for box number two Josh G has box number three tricolor the rock Rey Mysterio blue Sammy Zane and ronda Rousey Mark coffee Die cut Latista and to 10 that is gold signature selections auto wolf gang There you go. Congrats on the gold auto Josh G will make that one for you through your luck blue Bailey Aliyah Sammy Zane and another redemption. Good luck Josh autograph memorabilia red prisms card number 29 Charlotte flair There you go great box for you here Josh Roberts, that's edge white 125 rookie tricolor Winther and Trish Stratus Yeah, you bet Alvin Roman Reigns Keanu James tricolor rookie butch and Draganov Russo and to 99 Relic red Tomasa Champa Zeus Greg Valentine tricolor China Lois John Cena red and Roman Reigns Sensations gold to 10 Congrats Josh great box for you. There is a little thumbnail in this Right next to my thumb We'll send you an empty box and wrapper if you're gonna look you get in that Carver Place But nice Roman Reigns gold to 10 Carl Anderson your prosuna Giovanni Vinci Silver and That'll do it for your box Josh. Congrats on those hits Tom Kaye has boxes four and five Oh, dude, that's cool TPO Bam Bam live Morgan Silver and to 99. This is Relic Otto Gunther Roddy Piper Blue Bray Wyatt warrior Yokozuna Bruno ultimate warrior Jagger Reed Silver And to 25. This is tie-dye ringside action auto Papa Shango Here you go. Nice one Tom. Back to Jack and a little Dawkins tricolor Bam Bam tricolor down to James and sins carrying cross tricolor Emma Yeah, not much of one mark. But yeah, it was one Sonya Deville white Tegan knots and LA Knight diesel Silver Undertaker X pock tricolor And the Miz Relic Lexa Bliss Roxanne Perez and rookie blue Hayden Paxley Triple H Rich Holland Last pack of box number four Ulyssia Leo And Scott Scott Steiner. All right, Tom Kaye also has box number five What's up, William? Doing great here. How are you? Yeah, two golds in the same box Nice Rick Steiner silver There's stone cold. I'm driving the I'm in the Zamboni silver angel and ringside action auto Drew McIntyre nice drew Masso Jerry Lawler silver and to 99 that is uh So maybe a skirt relic some some kind of gear relic there Becky Lynch 99 there's a leathery uh Look to it razor Ramon Tricolor blue Mr. Perfect Amy Impriest and Bailey Loomis and Valhalla Tricolor This is ringside action auto silver Dakota Kai Carmelo Hayes Light green Giovanni Vinci Butch Roxanne Perez Ivar Tricolor Loomis and awesome. That is Ricky the dragon steamboat ringside Satamara mace blue And Montez Ford Mr. Perfect and maroon Montez Ford Alexa All right last pack for Tom Kaye in box number five maroon rear Ripley Chad Gable All right Shanti is going to close us out here with um box number six We have revolution wwe up after this and then four six box legacy football pick your team style breaks We are sold out today. Grab up your spot sports tonight I think we have more of these uh select and revolution breaks tomorrow during the daytime I go live at 9 a.m. Eastern tomorrow with them Bianca Belair boogie man silver And nice this is to 25 ringside action tie-dye auto big pop-up pump Scott steiner We have a zebra coming Brock lesnar zebra There you go, Sean P. Congrats on the brock Rick rude Keanu james rookie silver rock sam Perez Keena lion Scott steiner tricolor Oh, this one's awesome ringside action auto silver Dilo brown. Love me some Dilo in the late 90s There's a Jerry Lawler axiom blue John Cena Rick Steiner stone cold Bobby Lashley Finn Ballard coffee and tie-dye to 25 set break and Rollins biggie sup metro Oh, which which break is that ton of chance? Xavier Woods? Tricolor or a jade The Miz blue, uh, which which product is break 61 oba fire madcap moss silver Very Ripley here. I'll drop you a link for our schedule. Josh Briggs and Finn Ballard Relic that is not numbered Miz All right last pack up select. They're gonna be for the day. There we go. Good luck, Sean Junther Bam Bam Bigelow silver Zoe Stark and Tilly All right, that'll do it for the break. Thank you everybody. We have revolution coming up


Hey, what's going on guys, welcome back to another video. So today I want to share with you a Shopify drop shipping store that made a little bit under 700,000 US dollars actually drop shipping dog jewelry. In these videos, the typical format is to show you the sales, show you the history, show you kind of a bit of a backstory behind the store. And then we'll jump into the actual product itself. I'll show you where to source it from, how much profit you can expect to make per sale, show you the actual Shopify store and setup and product page itself. And then I'll also go one step further showing you the actual ads that these guys are running to produce these sorts of sales. So essentially you see all the fundamentals, the whole setup, pretty much everything there is to see in this business. So you guys can take it away and replicate it yourself in your own business. Now to tell the truth, I was kind of in two minds about showing you guys this particular store because I want to try and kind of shift the focus of my content away from this stereotypical drop shipping from China, from AliExpress selling cheap products with really small margins. The reason being is because it's a lot more difficult to do that today versus when it was say, when I started back in 2016. So if you've watched some of my earlier videos where I document the very beginning and show you the results and break everything down, you can see that achieving purchases of like three, four, five, even six pounds are pretty consistently was actually fairly easy to do and it's pretty normal. If you compare that today, then it's a completely different playing field. So number one, consumers have a different mind frame. There's different things or different levels of kind of professionalism in which they expect to see before they trust the store. They're a lot more vigilant nowadays. And number two, Facebook ads are a lot more competitive too. So CPMs are a lot more expensive. If you look at the average CPM from sort of 2016 to 2021, it's increased because the rate of advertisers has increased more than the rate of actual users which translates into more expensive ad costs. Basically it costs more money to reach the same amount of people. What a lot of people forget is that Facebook ads are an auction system. Everybody is bidding against each other to reach that select audience. So the more people trying to advertise to that audience then the more expensive it's going to be and essentially it's going to drive costs up. Anyway, I'm going a bit off tangent. So let's stick to the actual topic of the video which is this Shopify drop shipping store that made a little bit under 700,000 US dollars drop shipping dog jewelry. Thanks for tuning into the video. Hope you guys enjoy it. Hope you learned something new that you can take away and implement into your own business and let's jump straight into it. So the store in question is listed on the exchange marketplace. Quick background if you don't know what that is. It's basically where people go to sell their Shopify stores. So you can see the sales have made. You can see the business set up, their actual store, Facebook ads, absolutely everything. One of my favorite websites to go to for product research. It's called finderdogs.com. We can see it's another private listing so we can go straight to their website. And if we scroll down to the numbers they're verified by Shopify so it's all legit. They did a total revenue of 700,000 US dollars. Now in my opinion this can be improved and bumped up quite a lot if they make a few minor adjustments and improvements which I'll be showing you later on in the video. To give you some more background information the business is very simple and easy to run. It uses a drop shipping business and sells a durian from Facebook ads. So it literally is the stereotypical setup. Facebook ads, Shopify and drop shipping from China. This is the website in question. Finderdogs is the custom theme if we use our Shopify inspector tool we can see it's the Shopify booster nine custom theme. In my opinion I'm not too keen on this theme to be honest I think there's better ones out there but at the end of the day they have a nice product. They've targeted the right audience and that's been reflected in the results that they've had. Scroll down to the photo we can see it's not particularly pleasing on the eye looks a bit old fashioned in my opinion. So I think for somebody who wanted to invest in this business or replicate it go for a more modern theme with a kind of traditional layout and I think that could improve the results. Using our Shopify inspector tool then let's take a look at their best selling products. So we go to products best sellers and we see their best selling product is in fact this limited edition pure silver I doubt it's pure silver it might be it might be mistaken poor and bone char bracelet which they're retailing at 26 pounds I did go ahead and find on AliExpress now it did take me a couple of minutes to be honest to actually find this product so whether it's a new product or not I'm not actually 100% sure but we can see they're looking at about $10 delivered to the customer further that into British pounds looking at about seven pounds delivered so the profit margins is give or take about 18, 19 pounds which is doable on Facebook in today's market. There's a lot of things missing from this product page though in my opinion. Number one is the use of a custom font. This is the default font or one of the very kind of standard bog standard fonts which doesn't look very pleasing to the eye in my opinion. There's no kind of highlights of features or benefits or images or gifts nothing like that. It's very just kind of bland if you ask me and yet they're still been able to do some pretty decent numbers. There's no dynamic checkout button either which I definitely think hinders the results and the success that these guys have had and we can see they're also using a countdown timer as well. One thing they do make use of nicely is these quotations from I'm assuming existing customers social proof is such a massive thing regardless of what product you're selling. It's definitely something you need to implement and build on as much as possible for your own products and own stores. Something else worth mentioning too not to be underestimated is their mailing list. They have 17,000 subscribers on their mailing list. This is definitely something that is worth a lot of money in itself especially for somebody looking to invest in a business that has already selling dog products. You'd be able to produce a pretty healthy kind of turnover every month purely from a mailing list of 17,000 people. Let's take a look at their Facebook page too. Again, not to be underestimated. You need to make sure your Facebook pages are up to scratch. You've produced some content and essentially everything just kind of looks in the right place and you come across as a legitimate business with the proper links and the proper support emails. If we take a look at the kind of content they're posting we can see it's all really relevant and it's all kind of like have a high spirited nature type of content to encourage engagement because when people come onto a Facebook page they want to see that they're not the first initial ones. People like to follow other people so you need to make sure that what you're posting gets as much engagement as possible as much social proof as possible and then people will feel at ease with trusting you. So we can see again a nice funny picture 20 reactions which is pretty good to be honest for an organic post on a Facebook page 40 reactions 10 reactions 32 reactions 165 reactions so all the content they're kind of posted is reposted but it's funny high spirited content that people like to see and again it just gives people a reason to actually like the page follow the page and then they're more likely to come across your products and of course you can use that information that list of people to actually retarget on Facebook. Last but not least then let's take a look at their Facebook ad that produced the results for their best selling product and this is it here. So it's very simple actually very to the point starts off with quotations of people talking about the actual products whether they're real or not I'd have no idea to be honest. One thing I will say is the names and kind of like the quotations and words used a very kind of tailored towards the ideal market for particular products like this. So as somebody who has experience selling dog jewelry on Facebook I can tell you right now for certain that somebody the person most attracted or most likely to buy this would probably be in the late forties or fifties and female and this is reflected in the comments from their customers. So Julie Caroline my husband gave me this for my anniversary got this love it don't take it off. So it's definitely everything is kind of wrapped up into kind of targeting that ideal kind of demographic of customer. So before we end the video just a quick recap then and kind of key takeaways that you guys can use for your own businesses. This business does on average about $12,000 per month and it takes the owner approximately seven hours per week. So not a lot of time you could quite easily run two or three businesses of this particular size. The average profit is in and around $2000 US which isn't too bad especially for a beginner who could potentially or quite easily in fact do this alongside a nine to five. The website they used is okay. Definitely be improved with a custom theme such as venture. I really like the venture theme. I really like the new debutify point three as well. So definitely check those out if you're thinking about doing something like this. Product page itself is okay but again could definitely be improved. In my opinion going over the points I mentioned earlier in the video could definitely improve the conversion rate of this particular store and in turn increase the profitability. The Facebook ad itself again it works it's proven to work but could definitely be improved. Where I would improve and add for this particular product myself is I would go to TikTok try and find some small time influences young people send it out. It's going to be super cheap just send a bracelet out to them.  TikTok is such a fresh platform right now. I think products like this could potentially do really really well. The other place I would go to as well is actually go to Fiverr try and find a actor somebody who does products reviews for a living in the age range of say 50 plus female again send this out to them and get them to record a quick 10 15 second video of unboxing it and just talking about how sparkly it is and how much of a nice product it actually makes and I definitely think with those improvements that you could increase the success of your store and have yourself quite a nice little business that just kind of ticks along doesn't take much time each week to kind of maintain and just makes a nice little side income and with that being said guys I think that pretty much covers all aspects of this particular business if you're still watching the video I think very much I really do appreciate all the support on these videos I hope you enjoyed it I hope more importantly you actually learn something new that you can take away and implement into your own businesses one final quick message then before you go if you are looking for a program which will take you from essentially staff on scratch day one all the way up to having a fully fledged and functioning business covers absolutely everything you need to do in between make sure you check out my e-com academy there'll be a video link in the description below thanks again for watching I'll see you in the next one


What's going on everyone? Sam here with late in sports cards. We're in 22 Panini. Flawless, football hobby, 2 box, full case break, number 28, random team style. Thank you all again for filling this up. Thank you for your patience as well. Here is a look again at your random results. As always, you can find those on the website under the results tab, as well as the link in chat. Zonghan, L, you're up top with the dolphin, scroll down 1, Barry G with the Jags. So good luck here everybody. Let's do it. Come on, let's see a shield. See some some sort of one of one here.  Gem shield of some sort. Ooh, that'd be cool. All right, here we go. Good luck everybody. Oh, that's nasty. That's nasty in case card. That's nasty. Oh, that's nasty on top too. Good lord man. Got a laundry tag of Malik Willis. Congrats on that Titans. There you go, Tennessee. One of three. Very nice. We don't, unless a card's damaged unfortunately, James. Fortunately not. Wow, 24 of 25 Christian Watson. I don't know how that's not encased. That's massive. Yeah, got you Shane. That was also in the fine print. How about seven of eight, Dual Auto, Garrett Wilson and Breeze Hall. That's a huge card, man. Wow, congrats to the Jets, Jeffrey S. That's kind of massive. That's kind of wild that that's in the inner. That makes the inner very good. We'll get that one mag for you. Going to the bills. That is going to be five of 10, Thurman Thomas on the two-color patch auto. Now if you see with the Braves in a full case of Tops Chrome. Nice one there. I can make this one too. Three of 25. Going to the Steelers. George Pickens. Congrats on that one there. Steelers. Yeah, Thurman was a beast, man. He was really good in his prime. Holy. Congrats Pittsburgh. Going to Barry G. Nice one there, buddy. Next up, going to the Commanders. That's going to be Daryl Green, 10 of 15. And it's from this year. It's been very nice. Yeah, I wish Pickens did a little more on his auto. Congrats on that one there, Commanders. 13 of 15, Jake the Snake. Jake Plummer. Going to the Cardinals. Nice one there, Arizona. Three-colored patch auto. Michael S. Congrats, Michael. Empty. All right, in Caste Cards, we've got Joe Namit, three of 20. Going to the Jets. That was a really good inner box. Three mags in the inner box. It's kind of wild actually. Congrats on that, Jets. Next up, we've got going to the Bears. Montee, three of 10. David Montgomery. Only one Jake the Snake and it's the top loader. Jake. Now he's here today with Bob. He is here. And then who has this team? This is going to be going to Brandon M. Four of eight. Jalen Hertz, A.J. Brown. Dual auto. That is so sick, man. Congrats on that bolt on card as well. Very, very nice right there, man. Hertz and A.J. on the dual. Big time for sure. All right, Empty. Yeah, Wilson Hall and the Hertz Brown. Two duals. That's wild, dude. That's wild. Like you could have made a case that the, you easily could have made a case that the Brown, the Wilson and Hall should have been in case for sure. Should have saved those keys, too. All right, good luck here guys. Let's see what we got. Oh, dude. It's kind of gross. Empty. Nice one here for the Steelers. That is Jerome Betis. Two of five. Two color patch. Jerome Betis got a quad for, or not a quad, a triple for the Cowboys. Seven of 25. Svabak, Romo and Deck. Nice one there. Oh, player worn on that. Very nice. Very nice. Very nice. What's up, creative? That's pretty sick. We've got, this is a nice one right here. Going to the Jags. That is 12 of a 15. Trayvon Walker. Congrats on that, Barry. The rookie stuff will not be. It'll be like the enclosed memorabilia that's not associated. Vet stuff. It's like half between player and player in game. This is sick. Three of 15 going to the Broncos. That's going to be Terrell Davis. Nice TD. Congrats, Denver. Going to Derek A. Nice one, Derek. I can make this next one. Six of 10. The guy who is on the track, Drake London. Going to the Falcons. Dude, this is a really good case overall. Like the next one's nasty, too. Yeah. No, I agree on that, Moose. Trust me. You are not alone on that. Not alone on that front, for sure. Congrats on that there, Falcons. Six of 20. Going to the Titans again is going to be Malik Willis. Congrats, Tennessee. Tennessee's making out. Travis. Travis B. I agree, Pete. Absolutely. I'm very in on you. Doesn't shock me, to be honest, Cody. Does not shock me at all. And then we've got going to the Jaguars. Tony Bacelli. Two of five. Nice one there. Congrats, Jags. Tony Bacelli. Hall of Famer. First Jags Hall of Famer in history. Jags only Hall of Famer currently. Empty. Encased. We'll start here. That's 13 of 20. Going to the Rams. Kurt Warner. There you go. Rams. Philip G. We've got a Tyreek Hill. Nice one there for the Dolphins. Seven of 15. Cheetah. Going to Lee P. 14 of 20. Jamar Chase on the RPA. Again, this is from last year's Falcons. 2021. Congrats on that. It's a full-sig Jamar Chase as well. That's awesome, dude. Congrats there, Lee. Again, that is just a mark on the jersey itself that's not on the card or anything. So nice one there, man. Yeah, fax, moose. Always open, man. Always open. Not against the Ravens, you weren't. Oh, bang. You weren't, buddy. All right. Hey, have a good one, Mark. Thank you, buddy. A lot of mags in this one.  Oh, yeah, that's big. We have a triple for the Cowboys of all the quarterbacks. Jerome Bettis, Christian Watson, Daryl Green, Troll Davis, Jake Plummer, Thurman Thomas, Trayvon Walker, Tony Bacelli. Start off with our gem cards. We had Joe Namath to 20. We had a Kurt Warner to 20. Had a Tyreek Hill to 15. And then a David Montgomery to 10. We had Malik Willis Laundry to 1 of 3. We had Drake London, 6 of 10 on the Glove RPI. George Pickens, 3 of 25. Rookie Otto for the Steelers. Had a Malik Willis, 6 of 20. RPI for the Titans. It's a duel of Garrett Wilson and Breeze Hall going to the Jets, 7 of 8. We had a duel of Jalen Hurts and AJ Brown, 4 of 8. Came out of the same box as the other duel. And then a Jamar Chase RPI, 14 of 20. That will do it for the break, guys. Thanks again for the filp. Up next is nothing.