Alright, so we are recording now. Hello everyone. Welcome to another meeting of our CNCF End User Research Group. Today we'll have a nice presentation from Diego about using the interlink setup to schedule pods on Slurm. He will give us all the details about that. We will just start with a quick round of presentations. So if you have, if you're in first for the first time with us, please, we'll give you the time to present yourself. So I see that one out of first time. Do you want to say a couple of words? Yeah, thank you for having me. You can call me Norvo. It's quite different. And I'm a master's student in Grenoble, and I'm writing my thesis at Eviden, producing knowledge at us on the topic of integration in Kube flow with HPC. So I came here to like to get some ideas to meet with people. Raising. Raising, welcome. Thank you. And we have Alan from SkidMD as well. Yeah, hello. It's the first time I just started at SkidMD about a week ago. So thanks for having me. Awesome. Thank you. Thank you for joining. And welcome as well. So I will wait, Diego, is it the first time? Second time, okay. I don't know if you did the round last time. I was checking the names. You want to present yourself as well, Diego, just in case? Oh, wait, I need to give you a voice. Let me fix that. Diego, do you want to say a couple of words? Can you hear me? Yes. Okay. Hello, it's Diego Bakin. It's the first second time for me that I participate to this talk. I participate to me with other two colleagues of the system to the last talk. Okay. We are very interested in this topic. Thank you. Amazing. Welcome. And is there anyone that we don't have in the list yet that is here for the first time? You come forward. Me, I'm here for fifth time. Jose. Jose. Okay, cool. Yeah, feel free to add your name in the agenda, but go ahead. Okay. Okay, I'm Jose from Spain. I was working at a company called HPC Now. And I'm here because in the past months, we are researching a little bit about batch scheduling against Kubernetes. And I have a small experience playing with CUBE and some of our components. And I researched a little bit the topic on different projects, etc. So I'm here because I'm quite curious about what you have to Awesome. Welcome. And we have two more. We have Camilla. Can you hear us? Then we can go to Rohini. Always first question. Can you hear me okay? Yes. Yeah. Okay. Okay. Rohini. Yeah, technically my first time as a format here, but maybe some of you may have seen me at the rug booths at the last CUBECon. I recently was working for the SK Observatory now working on the Swiss part of the SK project and rationing with an HPC facility here in Switzerland at CS. And so we have a good amount of interest in this area that sort of aligns with the SK Observatory's interest as well. Easy. Welcome. You know each other. Yeah, welcome. And I see Pablo. That's a familiar name. Pablo. Hey, hey, Ricardo. Are you? Yeah, my first time here today, although I know some of you already. I'm also I'm working with Rohini actually on the SK project. Maybe. Well, welcome. I think we are okay for the routing of introductions. If we miss someone, just feel free to speak up. And before we pass to Diego, I was just going to say we have already one more session scheduled in this area for the 15th of May, checking the agenda around volcano and patch scheduling with volcano. We will schedule another one. We're still preparing around Karmada for sort of multi-scale multi cluster support. And we have an empty slot for May 1st. So either we take Karmada there, but it's probably better to present after volcano or we might have an empty slot. So if you have ideas, please reach out. It could be interesting also to have an update on Q. So we could we could ask them to give a quick update on May 1st. If you have other ideas, just just propose them. And with this, I will stop sharing. And I will pass the word to Diego. We'll talk to us about interlink. Thank you very much, Diego, again, for presenting and go ahead. Thank you for hearing me. So yeah, okay, the idea with this presentation is to propose a bit of a summary of what can be achieved with interlink and why we are evaluating and developing such a tool that I'm presenting just in a few minutes. Just one disclaimer here. I'm being focusing on HPC, how to deploy on HPC centers, let's say, but I have some detail at the end where this can be extended also to any remote resource. So this is the reason for this title referring to remote resources. But okay, let's dive in for the moment. Starting from where we in this development and investigation, you can see I have to thank a bunch of people first. And also notice that it's not really just a single institute working on this. It's kind of a teamwork made possible through the project that I'm presenting very briefly here in the first slide. So starting from where I'm employed, actually, I'm a technologist at INFN and ICSC, meaning that there are two kind of domain of science that we need to serve. The first one is INFN, that is the National Institute for Nuclear Physics in Italy. And you can see that we have different kind of area inside this domain. But every single one of them need to access resources of some kind. These resources are not only distributed all over Italy, as you can see in this small plot at the bottom, but they are also different in how they are managed. So there are cloud resources, there are on-prem resources, there are also batch resources that are not Solorum, for example. So they are HTCondor and used for more classical, let's say, distributed computing at LHC, for example. And also we have a supercomputer in Italy, and we will have more kind of HPC centers. And this is pushing the challenge to the next level. So how we can manage to have a common interface for people developing their stuff and doing science, and not only physics, but science, meaning we have here for the ICSC, that is the National Center of Supercomputing, a lot of different chemical, geophysics, quantum computing, whatever, needs to share the common infrastructure. So we need kind of an abstraction that allows us to say, okay, this is our entry point, and all the other magic appears, it's transparent to you. You don't need to care about how your payload is distributed over this infrastructure. Yeah, this is the beginning of the challenge, and what really put a boost in this direction was the participation to the inter-team project, that is an European project that aimed to create a digital twin engine. Digital twin engine meaning a platform that's as to serve as a distributed computational hub for creating and developing digital twins. So in different areas, that's starting from physics, for example, replicating a twin, a digital twin of a detector, for example, but also to replicate model for fluid disaster or whatever kind and so on and so forth. So multi-disciplinary area and that needs to access HPC centers that are also, fortunately, part of this consortium, part of this activity. And this is really the key for what we achieved so far. All these infrastructures that I'm mentioning, so starting from the grid one, so the batch on HD Condor, but also passing to this supercomputer area, often enough, produce this kind of problem for us. So produce a restricted area where we cannot envision to deploy anything that is similar to a real Kubernetes node. And we need Kubernetes for a simple fact, as you may imagine. The frameworks that are being developed for the future of the each one of the science domain that I presented briefly before needs, it's compatible with Kubernetes APIs. So we really strive to have an abstraction that starts below the Kubernetes API layer. So the idea essentially is to serve this black box that does the following. So we started with these three main scenario in mind. So starting from the first one, we envision, for example, machine learning data science pipeline in general, arriving to the point where you interact with Kubernetes API. And then this black box should be able to afloat what you request from that framework into something running on the GPU node, for example, on the HPC center. And then return it back the results that you need. Same for serverless or event-driven computing, if you want. I know new data are available. I want to trigger a pipeline that goes into specialized hardware or into specialized node for a new training of my model, for example. And the most challenging one is also the one that for seeing a backward connectivity from the node. It's a bit tricky. We have approval concepts for this and I show you BAs. But the idea is that, again, I have a Jupyter hub, for example, or IML flow hub, whatever, that delegate the payload that in this case is a simple Jupyter hub instance, for example, and it distributes this into a remote node and connect back to the hub for the user to access them through the web interface. These are the three scenarios that we have in mind. And all in all, what Renga about in the past, where we were thinking about this, is that somehow we have this kind of abstraction already in place, thanks to the virtual cubelet components. And so the idea was, okay, let's give a first try to this kind of approach, where I have my Kubernetes cluster with a virtual node that behaves like a normal node, but is configured to submit, for example, in the case of Slurm, a Slurm job with a container run inside an obtainer or whatever container runtime is disposed of at the remote site. Very often we never experience with two, three centers, obtainer or X singularity, let's say, is the runtime that is most used in the HPC side. But okay, so I offload in this way the payload to the node with this kind of magic, how good is it? So we gave it a try to all this flow, where this HPC GPO node was actually a real euro HPC center, that is Vega, I'm talking about it in a second. And the idea, it was okay, let's give it a try to what is already in place, and it worked quite well.  So the first blocking item for us was that really we have an heterogeneous set of providers and also when you pass from one HPC center to the other not every Slurm is equal to the other one. So you have somehow to configure and let the provider be in charge of managing how the offloading of the container works. So, but to do that with a full vitro-cubular implementation, they should be aware of Kubernetes internals. And this is a showstopper in many cases. So we ended up with three main requirements for our approach and for our work on top of vitro-cubulat. So creating an abstraction that allow for providers to easily extend the vitro-cubulat machinery, allowing also a central maintenance of all the Kubernetes related stuff. So we maintain all that inside a core component and we envision plugins that as just to describe how the container life cycle is managed. And it will be in charge of the providers. So with this in mind, with this structure in mind, we can summarize this as providing tools to streamline the process of having a vitro-cubulat. It's not that it's not possible to do it right now, is that our concern is about letting the providers do what they want in terms of managing container at their sites. So in a certain sense, we want to lower the barrier for new providers to enter and say, okay, you can use microservices through a Kubernetes API. And so to achieve this really from 10 kilometers far, we develop a vitro-cubulat, a particular provider, if you want to call it like that, of a vitro-cubulat that is split into components. The first one is the Kubernetes core, so where you maintain all the state machine for reports, logs, requests, delete, apply, and so on and so forth. And then this part will talk through a standard, or let's say, a fixed API spec to a remote counterpart. That is the one that will be called plug-in or sidecar with interchangeable terms in the next of the presentation. So yeah, this remote part should be in charge of managing the container lifecycle. All the Kubernetes part is separated from that. So that is the idea, but before diving into the components and what they does and take a look quickly to what we achieve, this is not a silver bullet, of course. We are interested for our use cases in very particular payloads that can avoid to rely on intra-cluster network communication. So for instance, you should not assume that you can access a Kubernetes service that is internal to the Kubernetes cluster on the remote payload. And also the only volumes that are supported are config map secrets and empty beer for your pods. You cannot share PVC from any other kind. That's the limitation and here of the schema this work. On the last side, there is the characters if you want to play the game and on the right side there is the technical implementation. So let's start from introducing the components. The first one is the fact to the receiver, right? So the one that says to Kubernetes, I am a node. Give me work to do. I'll take care of that. But at the same time, whenever you receive a request, it contacts its friends interlink API server that you can consider like the man in the middle because it's responsible to expose the standard APIs that I anticipated before. And then it contacts the plugin that has been installed by the provider and where all the logic for some meetings, loom jobs, for example, is concentrated. And that is really the discovery. So the one responsible to execute your payloads. And if you go on the right side of the slide, you see that essentially you have the Kubernetes API server and all the scheduler and whatever, all the components of the Kubernetes part are talking to this virtual node, just like any other virtual kiblet that is out there. The only thing that changes is that the virtual kiblet itself is speaking a particular language, if you want, that can be plugged to different kind of plugins. So you can see here the virtual kiblet authenticate through a token with the API server and then the plugin does the jobs. We have three implementations in the moment. As I mentioned before, with a local Docker component, you have a single machine you want to run that into a Docker. It's more for demo, but also for real cases where you have a big machine. We have Zlure in this box here, and that is the one we are going to look at. And then we have HTCondo for normal grid system. You have here the schema. I'll let this more for you as a reference to understand the flow. But again, I don't want to repeat myself. You start with a node that is designated to, with a pod that is this designated to a particular node, to a virtual node, and then from there it proceeds in a chain. So the virtual kiblet contacts an interlink and say, okay, I need to run this payload do whatever it takes. And depending on the plugin that is installed, the interlink part does these different things. Where did we test this implementation? So we decided to start from volunteer sites and we were lucky enough to have, as a first volunteer site, the Vega HPC Center. It is one of the biggest Euro HPC Center. And we were able to play with quite a success at that stage. So it was really the fact that they enabled us to convince other sites, other people that this could actually work. And related to the discussion of the last time, the last meeting, for us, this model worked very well. We have a champion, if you want, where we demonstrate things that works. And then, for example, Euro HPC Center that is also participating to intertwin. Come and say, okay, it's quite interesting. We have our way to manage container on our center. Can we plug our work onto this? So we can plug, for example, Unicore, that is their framework, into a plugin serving as a backend for interlink. And this was the second step, second fundamental step, that demonstrated not only that the workflow can work for us, but also that the principle of easily extensible is a valid argument in this case. And yeah, so now, unless there are questions, the plan is really going through demo is a big word. I mean, showing directly what it looks like, some meeting pods on this kind of infrastructure, and the notebook part that I mentioned before is working. So yeah, I don't know if anyone has any pressing question that they would like to raise. I'm afraid to come forward. I have just a quick question. What kind of, what provider are you using from Virtual Cubelet? Yeah, the interlink is, if you want the provider, the provider, so the implementation of the Virtual Cubelet is done by us, and it leverages the interfaces provided by the Virtual Cubelet software. So was this, sorry, I saw KSOC also, was that kind of an inspiration for the interlink provider or? Yeah, sure, yeah. We took that from the architecture perspective, and then we also, of course, look at around how people manage Virtual Cubelet and yeah, but the class itself is fully Virtual Cubelet-based. It's nothing fancy, let's say. All right, thank you. All right, so let's start from what it looks like for the user in case of interactive analysis. Well, I'm sure most of you knows the interface for GPTRAP. What we do here is selecting the image that is actually in the Vega URH PC center, and the user can select, okay, give me a slot with CPU or GPU and in Vega, and here I'm into the login node, into the edge node where we have interlink configured, and you can see that I have nothing running here at the moment, but as soon as I'm going to create a pod that is pointing to that particular node with this flag, you should be able to see, yeah, now it started, you should be able to see that a job, a slurm job is arriving, and it will actually do a singularity exec. Let me show you. Yeah, now it takes too much time. Let's see the job running, the notebook is up and running again, so I'm inside a node in Vega with a job on slurm running a Jupyter lab, and I have NVIDIA, as am I, and I have a GPU available to do all my development stuff. What looks like into, if I go into the node, into the Kubernetes cluster, it's actually, oops, it's actually the following, are you still with me? Okay, yes, it's not my connection, fortunately, so I can take a look at this and say get pod jup, you see here that my notebook is essentially a normal pod that's a solid one constraint, actually, and the constraint is that I'm asking for node selector, so I'm asking to go directly into the Vega node, so you have to be explicit, of course, but if you select the node that is called Vega-NewViewToQueueBlood, you will be spawned into a slurm queue with a singularity container running on top of it, and this works in this case with a trick, the job is connecting back through a tunered connection, but in case your payload is not that complicated, you have, for instance, the possibility to submit any payload that is going to download that data, collect information, train on that, and bring back your results, so I started with the worst case to show you the whole chain, but again, you can be as fancy as you want unless you require pbc that are not supported, you can bring with you config map, secrets, and empty-dir, but the connection to the internal or the remote Kubernetes cluster is not allowed from the inside of the HPC, you can think of VPN or whatever, but it's not the target of this at the moment, at least, so I don't know if you want to lead what comes next, otherwise I'm just pointing you to the, I'm just showing you the documentation, and oops, so in documentation we have two scenarios that you can try on your own, the one is that is local, you have both Kubernetes cluster and a Docker compose on your laptop, and you can check how the components are talking to each other, and the other one is actually what we use to deploy all the machinery, so we use, you have a step-by-step demo where, for example, for the authentication between the Vista kubelet and the remote counterpart, we use the OAuth chain, but in particular in this case is GitHub, but we have also instruction for everything that is well too compatible, and then you install through the client and through some configuration, both the components, the one that goes into the Kubernetes cluster and the one that is responsible to run Slurm job, for example, remotely, and so yeah, it takes literally five minutes to go through this, but the nice thing that I want to stress is that we also, since the specifications are documented through Open API, you can implement whatever plugin you want with quite some ease, and we also have example for this, if you want to extend, and we have people coming and say, okay, I want to extend and they get started pretty easily with this kind of approach, and yeah, just keep in mind that we are at the early stage of this project, so the API, it's something that we want to fix quite soon, but we are not there, so at the same time, there is a good time if you want to come on board to lead what the API will look like in the future, but if you want to rely for production ready stuff, it is not at this stage. All right, there are questions, otherwise I go back to the presentation. I see Pablo has a raise hand, go ahead Pablo. Yeah, thanks. First of all, thanks for the very interesting presentation. I had two questions actually, so the first one is, I'm not sure I understood, you mentioned the limitations, is that you only get like a config map, empty there, etc. So what are your, what is the approach here, if you spawn a notebook that needs access to some, you know, data that is somewhere else, and then I have a question that I can maybe ask. Yeah, the first one is, there was partially a lie inside that, because you can still access common and scratch area, not scratch, common area that you can decide to mount at singularity time, so for example you have your scratch area or your data area inside HPC, you can bring data there through whatever machinery, and then you can still access that via a pod annotation, you can say, okay, mount this area because I'm allowed to. Okay, because you're inside the HPC and you can just mount it, okay, I see. But for instance, at PVCs, is that a design limitation or is it just the fact that it's not implemented? Well, on one side it's linked to the fact that we have no internal access to the network of the cluster. Okay, yeah, I see. That is limiting a lot what we can do, what we can do there, and I suppose it will be many other issues that we need to take care in order to allow for that. Okay, thanks. The second question might be a bit brief, hopefully. I was just wondering, so I'm not an expert on the internals of the notebook, but it was quite interesting that you showed an interactive session. I was just wondering how the connection is established since obviously you don't know which node it's going to end up running at, which worker node, so I was wondering how it works behind the scenes, how you can establish that interactive connection. Yeah, with again, vision from a bit far away, we are instrumenting the Jupyter Lab image with an initial script where you connect through with your Jupyter Hub token back in a SSH session with the Jupyter Hub, so you open a tunnel basically to the Jupyter Hub and then you start communicating with the proxy on the cluster, the Jupyter Hub proxy on the cluster through this tunnel. So yeah, not every image can work on that, but it's really a script that you have to put in an initialization of the docker to adapt any Jupyter Lab image to work in this way. Okay, thank you very much. Thank you. Awesome, I have see Rohini, go ahead. Yeah, thanks, Diego. That was really a good presentation, an overview. I had a question. Yeah, no problem. Sorry, just to clarify, I have just a couple of slides to conclude, but yeah, go ahead. I had a question about the O-off chain in the sense that is it something that's implied here that the IDP that you've configured your Jupyter Hub with, there is an implicit level of trust with the HPC facility that the JHub tokens will be trusted or is that established? No, the only trust that the only trust that comes from this implementation is at the moment you can have finer grain, for example, in the Julia implementation we have the grain of authorization that is on the pod level, so the pod has to present an annotation with a user access token to be trusted and to be executed on the slur node, but generally speaking, the general implementation relies on the fact that the Kubernetes cluster that is talking to the interlink API is trusted by the HPC, so the HPC trusts the interlink server as a group account if you want to to put it in this way, so the administrator of the Edge machine trusts whatever comes from the Kubernetes cluster, so there is trust between the cluster administrator and the provider administrator. If this is not enough, again you can ask at the plugin level for whatever additional information you need to allow the user to submit to the queue, for example, if you want to allow only certain people, you can plug your mechanism for say, okay, if this pod arrives with valid credentials, it's allowed to run. Okay, yeah, that makes sense, and I guess that's true independent of what the entry point is, so here in the demo the entry point was jupiter hub for the pods, but even if it is coming from some other data processing way, the fact that the trust is between the interlink server and the HPC means that, okay, yeah, okay, thanks. Awesome, Dennis, do you want to go next? We'll give you the chance to finish after Diego. Yeah, yeah, no problem. All right, thank you, Diego, for the great presentation. Just one quick question, this is maybe a little bit use case specific, but do you support having multiple containers in a single pod when deployed on the HPC infrastructure, and what will interlink do with networking and shared memory in that case, because we have a use case where we would really benefit from deploying essentially two separate components at the same time in a single pod, sharing resources? Yeah, so in that case, again, it's not really interlink internal because it was the information of multi-container pods is transformed into into what is managed at the plugin level. So currently there's Lurem plugin that we use at Vega, for example, that's the following. Allocate a Lurem job with the sum of the requests for the container and then kickstart two singularity processes, one for each container in the single job. This grants you at least to land on the same note and do some stuff. It's not the cleanest possible way that you want to work with. Again, if you're interested in doing better or less some use cases, you're more than welcome to join the efforts. All right, thank you. I think you can go ahead, Diego, and wrap it up. All right, yeah, so I wanted to share also that we were a bit worried about the scale, as always, but it ended up in being a nice experience here all in all. You can see here a plot where there are 1,000 pods deployed. It's not enough, but it's not too less, I suppose, resulting in 10k cores consumed by, in this case, we leverage some example workflows from CMS collaboration. All of them, all the 1,000 pods, were managed by a Kubernetes cluster that is really a common one, if a common one exists at all. But yeah, there were no dedicated hardware or special stuff that we did to reach super high performance from the control plane side, let's say. That was a good news for us. Also, this is to mention that we are developing and we developed this tool also with this in mind. We are trying our best not to create a lot of requests to the, for example, plugins, so in the case of Slur, a lot of SQ command, but rather trying to catch whatever possible and be on the lazier part more than on the aggressive part in terms of synchronizing the statuses. So this will allow, for sure, to scale as an important role on this achievement, if you want. So all in all, we have this deal in mind. We keep and provide maintenance as a community for the core part of the virtual kubelets, so communication, authentication, authorization model, and the API that goes through the communication, the remote part to the entry part of the edge node, the caching of the state and whatever. And then we hope for interesting party to give a stab to providing the implementation that they want to in their use cases. And yeah, talking which we have experience also outside HPCs that are interested in this, so for instance your services for which will be out for you to give kubernetes access to them, but with this trick you can create your plugin and say, look, with this you can make a call to a kubernetes cluster, to a kubernetes counterplane, and that your things deployed on my SAS, for example. So that's one model that is taking a bit of attraction, and that is bringing us to thinking, okay, how we wish it would evolve from here, if other parties are interested in maintaining and collaborating. And also a paper is in the writing and for reference the sign and the implementation and very first experience that we have so far. Yeah, I think we touched most of this point, let me stress just again, we are not using all these lume for implementation, we have a lot of other use cases that are running on the heterogeneous set of resources that we have, INFN and in Italy in general, we are also working on that level. If you are interested in making questions or whatever, you can join the slide channel for that we have for the community and for the project inside Intertween. And last but not least, the link to the documentation is here, but it's also in the description of the event so you can go whatever route you prefer. And yeah, that's all. Thank you very much, David. That was awesome. I was going to ask you if you can share also in the agenda of the group the links both to the presentation and to the to the documentation. I think that would be very useful for people checking it later. We have time for more questions, so I see Kevin. Yeah, it was just an interesting project. I was curious if you all have considered adding this as a provider to the VirtuaCubelet website to kind of bring more awareness to how you can use VirtuaCubelet? Yeah, that's a good point. We haven't reached yet the level which, okay, we knocked at the door and say we want to be published there, but yeah, I think it's worth appearing there to be to be aware of this possibility. Yeah, I mean, if you look at a lot of the providers, part of it is I think a health of the project, because when I look at VirtuaCubelet, a lot of those providers are pretty old, so it makes it seem like that project is not really continuing. So I think like at least having it showing in the open source community that you are using this kind of shows that it's still actively, people are still actively using it. Yeah, good point. Even if like it's, you know, you quote-unquote think it's rough. I was actually thinking about this a year ago, and I wasn't aware of this project at the time, so it's just like having that information, because when I looked at VirtuaCubelet, I saw there was a CRI provider, and I was like oh, if I want slurm and CRI, I'll have to implement my own, and I stopped. No. And it's a similar story I heard also from other people, so yeah, okay. Yeah, I think that's a good point. That's a good way for people to find the project. Thank you. Other questions? All right, well, people think I'll ask one. What are the resource requirements on the interlink cluster in the origin cluster? Is it just like the HCD resources, or is there anything else? You mean from the control plane perspective? Yeah, I can say that for the scale test again, it's not a lot, but it's something. Really, we had ATCD running on not even, I think, not even SSSD, let's say, so on whatever kind, really normal control plane reached this kind of performances, because again, it's lighter for what we with Sozofa is from the intensity of the work is really less intensive than having real nodes and real pod. But yeah, we need to measure this, but so far, if you want to reach at least 1000 pod, you don't need any particular requirements. This is assuming, I didn't get it completely, if the interlink metadata on the virtual kublet node is that stored in HCD, or do you have your own storage? So the state, all the information are stored inside the pod resources, so eventually are in the ATCD of the kubernetes cluster, so it's nothing dedicated. What you have inside instead is the cache that is done on the remote part, so the interlink part does inside a cache of the status is allowing for not going each time the Kubernetes cluster requests a status to go ask the SLURM. If it's running for a certain period of time, I cache that information and then eventually I update the cache later. So just to avoid bombarding the SLURM services with too many requests. Okay, thank you. And I had one more, I don't see any other questions. So there was a question already about the network requirements, but I had another one which is at the scheduling time, because I understand this is at the pod level. So for example, if you want to do some sort of, say, distributed training for ML and you want to collocate pods, this would mean, like, collocation is just basically putting the same node name in the other annotation, which is the virtual kublet, and then you just assume you're running on the same HPC farm, and somehow you'll have, like, fast interconnects, all these things. Is this correct? Yeah, there is something that is interesting, but we didn't have practical use case to test with, that is the MPI configuration. So in principle, you can pass to the annotation to the pods whatever option you can pass manually from the HPC login node to the slur submission command. So in principle, you should be able to configure your job, your different pods to leverage MPI of the sites of all the machinery for that kind, but we haven't yet have a use case that is willing to test that. So it's just a wide guess. All right, and I'll ask the last one and then I'll pass to the others, but if I would use a library for, say, machine learning or anything else that does distributed pods and training, how does this translate to, have you tried this, like, for example, doing TensorFlow, distributed and then seeing what happens? Yeah, with the other try the distributed part with TensorFlow, for instance, different thing is if you want to have, if you have some service that you want to set up, for instance, I'm thinking of MF flow tracking server, you can expose that and in case of Vega, you can contact that for tracking your machine learning performances, but yeah, doing actually the real multi node, if you want, training is not something that we test, but if you are willing to, with some use cases, that would be good. But for example, the virtual kubelet, how many resources does it advertise to the scheduler? Is it like unlimited and you assume there's unlimited quota on the back? So the node itself can be either unlimited if you want to allow for any kind of slur submission to the virtual node, but otherwise you can instantiate a flat static resources when you install the virtual kubelet. So you can make that node appear like allowing for 10k pods and for 20 terabytes of RAM and so on and so forth. So typically you would have like one virtual kubelet per queue to have control like this? Yeah, it's up to the use case and the topology you want to achieve, but yes, if you want the easiest case is, okay, I'll let slur limit my usage so the virtual node is unlimited. Okay, yep, like this. Other questions? I'm watching for raised hands. Follow up on that one. So go ahead. If you are submitting a pod and you hit slurm, I guess would the pod show up as just running at the kubernetes level and then in slurm it would actually be in queue? No, no, no. The status is one of the things that is managed centrally by the virtual kubelet and it relies on the status retrieved from slurm. So until the pod is not running, yeah, the container is not running inside the slurm job, the pod status remains pending. If for any reason the submission failed, you have provider failed or something like that and yeah, but you do not see running if the job is not running. Okay, thank you. Very, very nice. So we are getting to four minutes from the end, so we have time for one more question if someone has one. Otherwise, thank you so much, Diego. This was really good. I suggest if people have additional questions so we can also follow it up in the Slack channel for the research user group. I know Diego is also there, so feel free to to ask questions there, follow up on the discussion. So I don't know if anyone has any other thing to raise. Otherwise, our next schedule meeting is currently May the second one in May, so the first one we don't have a speaker yet, but it's very likely that we'll find one. So yeah, see you in two weeks and thanks again for participating. Bye-bye. Thank you very much. Bye-bye. Thank you.


What's going on y'all welcome to another video today? We're doing a market analysis on exactly what's been going on and what levels I'm looking at for entries for breakouts To take profit looking at some supports and resistance and just some ideas and opinions I have you know with with whatever questions Some of you guys were asking so I hope you guys enjoy this video. Let's make this money Let's get this face. We're gonna do the giveaway for this video because we have not done it yet So we got 30 likes on the video and 23 comments now we asked for over 30 likes, but we'll let it slide this time Let's go ahead and do the giveaway already put in the link over here So the giveaway whoever the winner is gets a free month in our trading room, which is $70 to get in Let's see who the winner is Good luck The winner is Axel Brie Axel Brie Please send us a DM. We will get you set up. You got the right answer dollar cost averaging That was the question was the what's the best way to invest in Bitcoin the answer is dollar cost averaging So Axel congrats my boy Make sure to send us a DM either on YouTube or on IG at Bitcoin daily and Claim your free month in the trade room Appreciate you man. All right y'all so let's check this out man It has been a pretty slow few days. I think last time we streamed was last Tuesday, I believe I think after that Bitcoin took a dip below 12k. I think it was around 12k when we last streamed I think actually the same day That we streamed at night It dropped under twelve thousand it dipped to about eleven thousand four hundred ish Maybe eleven thousand three. I'm not sure exactly where But we did buy some at the eleven point four support And we've we've wrote it up to eleven thousand eight hundred So that's currently the position we're in we also entered ethereum at three hundred and ninety Currently at four or six. I think it did go to all the way to four ten ish Are in any other trades we are oh, yeah, we are in it in the chain link trade I don't know if you guys remember our last stream We're speaking about the levels for chain link where we would like to enter it was between 13 and 14 dollars That was actually the question of the day on that video If you guys were paying attention chain link hit that entry 13 or 14 dollars We entered it. So that's a trade out that we entered and we're currently in and it's currently at 15 So we're ready up on that. I Think a touch of 16 Which is our 16 is our first target. I think it touched I'm not sure what we'll look at it you guys So let's look we're still looking at the same levels that we talked about you see these levels right here We're struggling if you look back. This is literally a year ago last year. We're in the same spot We had the same issues in the same levels So that that's the main thing that we're looking at right now We need to break through this you see how these wicks are just going up and getting rejected right back Down look at the same thing here happened last year got rejected rejected Rejected went down try to go back up rejected went down. That's when That's when I went all the way down to 7,000 try to go hit 10k again and then when I hit 10k, that's when the whole pandemic started and Yeah, we went all the way down to what was the low here. The low was 35 96 man, I remember at 35 96 I couldn't buy at that price because it moved way too fast, but at 4000 I told everybody to buy I was like guys This is your opportunity anyone who was saying that they missed out on it that they didn't get to buy You know at a good price This is an opportunity to buy and a lot of people didn't want to listen to me So I mean I definitely bought and have basically tripled my money since then But yeah, if you bought at 4,000 you would have tripled your money over here So at 12k is a what is that 200% 300% profit some of that But we're so we're stuck right here in this range man, we need to get over this if not If we don't get over this and we keep getting rejected and we break past 11,000 we're definitely dropping to 10 if 10 can't hold right here You see this this area right here. So this is definitely a support is a strong support if 10 can't hold We're dropping to 9 and then we're gonna hope that 9 can support us if 9 doesn't support Then we're just gonna keep dropping. I don't think we will drop that far But that's just so that you guys have an idea of what could happen if we can't break through 12 That's definitely a possibility there. So we have to we have to be you have to watch it You know, just make sure you're using stop losses and all of that So that you don't you know get caught with your pants down if you know what I mean So now we talked about Bitcoin in the on the weekly Let's fall. Let's zoom in a little bit and As you can see, we're still good on this bull flag Both like it's still there We're still good there. Remember what happened down here. We had a very very similar thing where we took off we Consolidated for a while and then we took off So we've taken off. We're consolidating now the thing is the range is pretty wide You have about a thousand dollar range It is if you notice it was very very wide then it got a little bit tighter Then it popped over here. It's a little bit tighter. So if it keeps getting tighter and tighter and tighter Then I'm definitely seeing another another pop like this and look look at this on how wide it was wide And then it as it went on it kept getting tighter tighter tighter tighter. Look at this right here How long was this from basically starting back here? this was June 25th all the way to July 20th so almost a month of no movement whatsoever just very very small moves very very tight and Then it popped so we can definitely see Something very very similar to what happened here over here because the moves are getting tighter and tighter and tighter Which we're still trying to get through But the moves the movement's getting tighter. So that's a that's a bullish thing That's a good thing to see here that when it gets the tighter it gets that means that the market's getting ready for Impulsive move and an impulse move. So either a big pop like this how we saw here And I mean it could go either way But because of the pattern the overall pattern that we're looking at here This is a bullish pattern. So the probability is to the upside So we're playing the upside on this because that's where our Edge lies. That's where our probability lies You always want to play the high probability setup and a high probability trade. So get out So let's jump into the four hour See if there's anything different here. So we're basically seeing the same thing here Very very wide range here. You can see this this actually dropped all the way to what ten thousand five sixty and The top of the range was twelve thousand one sixty three. So that's a very wide range right there But that's the range so you got to make sure to be careful when you're playing these if you're day trading this if you're just holding long term Then you don't it doesn't really matter all this nonsense doesn't matter What matters to you is what's going on in the big picture. So Does when I zoom in like this to you know four hours or anything less This is more intended for anyone trying to day trade it like what I do is I day trade it So for example when it when it comes up here when I see it getting rejected I'm selling then when I see it bouncing like this.  Boom I'm writing it up and I'll write I'll write through all these little ups and downs and I'm taking profit this whole time So like all these red lines. These are just to remind me to take profit there So for example here I took profit then I went down if it hits this yellow line again like it did right there I'm buying again. So boom bought again When it when it hit this I'm taking profit when it hits this I'm taking profit When it started going down I'm buying again now now once they go over the my my markers and it hits them again I'm buying that those turn into buy areas.  I'm selling You just keep doing the same thing see when it came down here buying Just writing that up. Look how far up. So I'm taking profit when I bought here I'm taking profit taking profit taking profit then as it came down here. I bought but I have a stop loss Remember you always got to have a stop loss. So as soon as it went under that I'm selling because I don't want I don't want any of this down here I want to get out of there and I want to look for a better area to buy So when it dropped here drop look back to my marker. I'm buying again, and now I'm writing it up right here So if I see it start dropping again, then I'm selling because I want to get out of the position and I and I zoom in more if you zoom in more Then I'll have other targets that I that I don't have marked here like for example this area right here 11,800 I'm taking profit there So yeah, man, that's pretty much it on Bitcoin I think we pretty much covered it from the weekly the daily in the four hour anything else is just for scalping Do you guys have any questions on Bitcoin? I think we're good here Let me tell you the levels I'm looking at for entry and exits Um, so if I'm looking to enter In this range, I'm not really looking to enter where it's currently at Because my entry was 11,400 So my entry was down here 11,400 that was my entry This is another entry. So 11,400 the entry I've been using here and here and here Um, so that was the entry My take profits here were 11,600 which is around Here This range right here and 11,800 which is here. So we hit both of those already My next uh take profit mark is 12,000 Um, so if I'm looking to enter I need either a dip back down to 11,400 You could buy at 11,500 between 500 to 600 It is viable But you need to have the thing if you if you buy there you need to have your stops under 11,400 If not, you'll probably get stopped out. Um, so the stop Uh, your risk has to be a lot wider. So that's why it's a that's a riskier play if that's what you're looking to do Um, so the only other entry It's tough, but it has to be a breakout entry and it has to be above 12,000 not only above 12,000, but it has to be above like 12,100 Because it's been hitting it that 12,000 marks so many times and just stopping out It just keeps hitting stopping hitting stopping it So I if you want to be safe you have to enter above 12,100 And then your targets are probably going to be like 12,200 Um, and up here 400, 12,400 And the next entry would be above this high. So above 12,500 would be the next breakout entry But I like I like to I mostly trade with leverage And I'm the type of trader that I like to focus only on a few set of coins Uh, I know most people especially when they get started they just go down the rabbit hole of outcoins And they invest in like 50 different outcoins And like you can't it's it's impossible to learn the pattern of so many different coins you have to You have to stick to a set number of coins and just trade those and you'll be very profitable It's like you have to specialize in a in a certain coin Um, yeah, I do the same thing with the stock market I only like to trade blue chip blue chip stocks and um, and any companies I really believe in Um, and you but I only trade, you know I started with five then once I get really good at those five and I add another five Um, and I that's what I do and I max my my trading range at around 20 different options Um, I feel like if I can just learn these 20 coins or 20 stocks really really good I'm gonna make more money than someone who's just playing the lottery basically investing in 50 different coins so um Just learn just pick like a handful start with five Learn how to trade those really really good learn all the levels and learn the supports the resistance Learn the patterns how it moves and then um, you'll you'll be very profitable if you learn how to Do that and then from there once you get that then you could add on another five coins and and so forth, you know and just And you know put a limit to how many coins you're trading because it's impossible to watch 50 different coins Like it's just impossible to do So my main thing that I trade is bitcoin Um, I trade bitcoin. I use leverage when trading bitcoin so that I I get a bit more volatility in it um and um And then I trade a few a few other of the top coins I don't I don't go off to you know too much to the to the left too much to the deep end You know, I don't like to uh to play the lottery too much. Let's see So I use binance bit max and buy bit I also the way I buy with fiat To to crypto is using either coinbase or gemini Next thing you want to focus on when you're starting out is how you're diversifying and building your portfolio um, you're so your crypto portfolio So the main the main coin in crypto is um bitcoin right And that's kind of the benchmark for every other coin. So anytime you look at any other coin It's either it's one it's following crypt of bitcoin. So depending what crypto does is what every other coin is going to do and um And two it's what you know, you compare every every other coin to so what I always suggest To people I treat bitcoin like the s&p 500 if you don't know what the s&p 500 is is basically uh An index fund of the top 500 companies in the stock market Um, so I any starter When you're first going in I always suggest Putting at least 50 of your portfolio in something like the s&p 500 It doesn't have to be that but it could have to be something like that Um, because it's the main thing that drives everything else So I recommend doing the same with crypto keep at least 50 It doesn't have to be 50, but that's just um A starting point for you and then you can move that around between 30 to 50 percent of your portfolio Should be in bitcoin because that's that should be the foundation of your portfolio and then uh, then you could use like, you know, 30 for outcoins, uh 10 for day trading if you want to be going in and out and then like another 10 for Having in cash, you always want to have cash. You always need to have cash In hand so when there's a dip you want to be able to buy it All right So the the whole idea with with bitcoin and crypto is to get more bitcoin the more bitcoin you have the better So you have to have cash in order to jump back into bitcoin and and make that You know make more bitcoin overall All right guys, so let's look at ethereum Now if you guys watched our previous video, then you know that the way we like to start Is to zoom out. It's called the top down analysis Here we go. So we're currently at the 400 level right now If you zoom out, you guys can see the potential of ethereum. Look at this The high was 1419 $1419 any of you guys, uh Have ethereum back then the first time I ever bought ethereum was like down here somewhere around $100 This is what 2017 is So 1400 was the high right now. It's at $400 That means that you can what double you can triple your money This is another bitcoin you can triple your money, but it's not going to happen overnight. So chill out So going back real quick just taking a look you can see this 400 area Was a support here it was a support here during the breakout So check this out. We broke out. We broke out through We got rejected at 400 right here This is what august 2017 We dropped down We dropped down to 200 Then we jumped back up and this time we broke 400 So that was in november After we broke it We tested 400 again, and then we continued up. So it took one two three four Here we tested 400 we jumped we tested again went up, but then eventually dropped down Um, so 400 is a very very big level for ethereum There come any time to test it here one and it dropped Two three four And then finally Right now we're at one two three four and we're we're battling we just tested it one time right here We're battling to get through that 400 level Um, if we go through four if we do get through 400 420 420 is and it's the first target then for 40 and 450 Then 500 pretty much I have to zoom in more to see um a little bit deeper, but that's around the range that we're looking at If we zoom into the daily You see you see how how much it's respecting that 400 number It's a very big Support and resistance zone um by and ends And youtube rd to rd r2 d2 and youtube There's a lot of uh tutorials look up a finance tutorial All right, so here we're looking at the 400 level You can see 420 440 And it got scared of 450 so remember remember if you guys have have been here before you know I love playing the whole numbers because uh The market always reacts to whole numbers. So 400 huge whole number It's a psychological Support, it's a psychological resistance people just like whole numbers It's a psychological thing. You guys have to learn the psychological aspects of the market Because they play a very big role in trading. Um, so 400 Big level And then the other level was like 4 440 another whole number 450 another whole number people just love psychological Those psychological whole numbers. So you just have to make sure you're taking profit right before those numbers Hold on. Let me Change that sound real quick. No problem r2 d2 If you guys aren't here make sure to join the chat Let us know If you have any questions I'm happy to answer it and uh, give us a follow We're trying to get these follows man. We need 50 We're at 37. Let me see. Let me update it real quick. See if we got any more. Nope. We're at 37 guys. Come on. Let's get it Let's get it y'all. So all right. So in ethereum Guys, I am long term on ethereum. So if you're not in ethereum I do suggest getting in it if you look at the pattern. It is a bull flag. All right So it's a bull flag from this angle It's almost like a bull flag from this one, but either way it's a bull flag All right Um, the levels i'm looking at to enter is 400 dollars for like we said 400 is psychological support level So you want to enter as close to 400 as possible In the market everybody always asking me for exact numbers Guys, the truth is there's no exact numbers in trading. There's only zones. There's areas There's no no such thing as an exact number. All right, so You have to get used to that 402 is a fine price to enter if you want um But you know The closer to 400 the better so 402 is not bad Um, if it drops under 400 380 It will you have a minor support at 390 than the the bigger supports at 380 And then you have a drop off basically to Like the 350 range probably um, but 380 Is the the main support here? As you guys can see look how the price has respected it even though it's waked down You know further it still has always closed within this price So if you're entering at 400 your stop has to be under 400 um Give yourself some room Depending on your risk tolerance Your first your first target is probably 410 You're looking at 410 and 420 as your two targets before going to 440 um, so if 10 dollars is your first target you want to give yourself a 2 to 1 risk reward ratio So if you're risking One you want to be rewarded at least two right? So if if your reward is 10 dollars Your risk should be five dollars So if you're entering at 400 then your stop should be around 395 Does that make sense guys? Risk it's trading if you want to be profitable you need to learn and know how to calculate your risk reward ratio Because it is very very important If you're taking trades that don't make sense as far as risk if you're taking more risk than reward then when you're Talking about long term you're you're not going to be successful Because your losses are going to be too big and your your winners are going to be are going to be too small So that's pretty much it on ethereum. I'll zoom into the four hours, but the four once you get under daily Uh, it's mostly for scalping if you're looking for day trades So this is I go into four hours and below when I'm I'm looking for day trades So as you can see here 380. That's the level we spoke about look at it right there Look at it right here right here right here Like there's the market is literally telling you the zones You know like the areas where it has trouble at where there's either buyers or sellers There's always either right here. There were sellers And then right here. There are also buyers. There are buyers. There are buyers right here buyers So if it's dropping to 380 you should have orders set already to enter there so 400 Stop out below 400 380 is your next entry you enter there. You can write it up back up to 400 and you could have uh Let me see what this low is I would keep the stop around you keep it around 375 ish. So you're risking five dollars for 20 dollars So see that's a good risk reward ratio And then here look see how i'm telling you 410 is your first target. This is why look how it's Struggling right here. So 410 you want to take profits And then look for look at 420.  So you want to you want to be paying attention to all the whole numbers All right, let's take a look at uh bitcoin cash Let's zoom out Look how crazy this this uh chart is Like this is just like a straight drop The high here was uh 4290. I think that high was uh The day it got listed So this high 4290 that was the day got listed on coinbase Yeah, look, yeah, because it's coinbase So it went to 4290 that day. I remember that day was my christmas party For I was I worked in real estate. I was a real estate agent and my christmas party Um, I was at my christmas party And then news came out So the whole christmas party I was literally on my phone like trying to sell my positions because I had positions in it so I was trying to sell my positions And uh coinbase went down And I couldn't sell anything and I ended up selling it at around probably like 2000. I think something like that But hey, I couldn't sell at 4000. It didn't let me I was very upset about it. See look, I ended up selling around the 2500 range by the Very sad day when I couldn't sell I was very upset at that christmas party All right, but let's zoom in Now you see what the potential could be I think it could get to a thousand The high of last year Was up here $500 the $500 range big whole number big surprise They're called the lowest thing when that's crazy $80 The high here look at the high again 500s same thing. This was when This was earlier this year we hit 500 again drop down. So I see this going at least a 500 again And then that's gonna be a big test right there We're currently struggling right here. You see this this right here. It struggled in the same area before struggling there again so Once we get past we need to get past this so that we can Go up to continue up to 400 Um good levels to enter here.  So you have a $20 $20 reward $10 risk So that's a that's a good trade Zoom into the daily You can see here. We do have a bull flag But it's it's pretty wide. So you have to be careful trading this because it's still pretty wide It it has the last few days has calmed down a bit But it's it's a little wide. So be careful Um If you're looking for a breakout trade because you can't really 288 it's a little too high already So you can be looking at around 290 ish is a risky breakout trade Um 300 is a little bit safer because you could but If you if you look 300 it's been Just bouncing around there, but you can use 300 as a breakout With a stop below it 310 and 320 are your two targets So five dollar use a five dollar stop So you can have a proper risk reward What's up guys? Welcome back to the question of the day. Remember we do this on every video We give away a free membership. So let's get right into it question of the day every video will have a random question Uh, the question will always be in a random area of the video Find the question and find the answer post your answer in the comments for a chance to win winners will be picked randomly randomly once the Video receives at least over 35 likes for this one Good luck guys. Remember to Uh, post your answer in the comments Like the video subscribe to the to the page. That's it. Peace All right, let's look at uh ripple guys xrp It's just kind of been chilling It's kind of been in this zone right here Not really much going on Let me move this but it is a bull flag If you're entering Probably look at 2,400 to enter So i'm looking at a 2,400 area as close to 2,400 as possible Um And 2,450 is a target 2,500 is the next target Probably 2,550 is a target 2,600 so just all those whole numbers basically going 50 to 50 Is if you're trading it against bitcoin if you're trading it against usd Chart looks exactly the same Except here you're going to be looking to enter around 28 cents. You want to you want to enter at 28 cents 30 cents is going to be your first target whole number Actually 29 It's had a little bit of trouble at 29 not really that much if you want to be safe you could take profit at 29 But uh 30 is the main number 31 32, you know every cent Take a little bit off Then you could use a breakout above whatever the high is so above 33 cents would be a breakout entry you could buy again there So strategy would be to take profit on the way to 33 cents by the time you're at 33 cents You only want to have 25 percent of your position left And then as soon as it breaks 33 cents you want to enter again. That is at least what i'll do um If you're just holding long term then you can just hold You don't gotta worry about that whole message is in It's a bull flag so long term looks good Um, let's look at litecoin. I know somebody asked about litecoin. We're gonna check it out Let's zoom out to the weekly It's finally Been getting some life. Let's look at it against uh usd so The usd looks better than the bitcoin against bitcoin um So 64 You can see it right here And here again So it's definitely an area where people are taking profit There's profit taking going on here And then above let me see The high has been what 69 Another whole number um, if i'm looking to enter I don't really I don't there's no real patterns going on here If i'm looking to enter Um, i'm looking to either i'm waiting for a pullback Which is not very likely to like 60 But it was 60 60s here. So 60 is just risky I wouldn't enter it because look how wide this drop is right here to like 50 So 50 would be maybe a level i'd look at If it dropped that low, but it doesn't look like it's gonna drop that low Um, or I look for a breakout above 70 is what I would prefer. So if I'm trading this I'm trading above 70 Uh breakout and I'm looking to sell around 78 and 80 between 78 and 80 dollars is where I'm looking to sell Let's take a look at a da Same basic thing here Long term we have a bull flag Looks like this level 1500 Satoshi's Is where it got rejected Um, the support Is 1000 So I think I spoke about this last stream Last stream we're talking about about this And I said how people are taking profit up here at 14 and 15 And I wouldn't enter unless you got down to a thousand So it's almost there I'm not entering this unless it gets down to a thousand. That's my basically my entry point for that And I'll have a stop below it So it's a pretty good trade though if you can if you can get enter anywhere near a thousand right now It's at 1044. So that's pretty close That's already in the range of where I where I'd enter So if I'm looking to enter this and I really want to enter it, I would start entering here 1044 Then I have another entry like a thousand twenty And another one at a thousand and then I have a stop underneath somewhere Because your target your target's gonna be 1100 So that's already 60 away from where you're at 1200 1300 and 1400 15 yeah, so I mean it has a pretty good reward This is this is a pretty good trade to enter at a thousand as close to a thousand as possible So see like you don't have to enter look for new coins or New the new bitcoin you could just look for profitable opportunities where you can enter so if you look at this look how it bounced here boom it bounced from a thousand To third to about 1400 So that's that's a pretty good profit there Then I went back down it tested the same area and it bounced from here so The same area to 1400 again went back down and it bounced all the way up to what 2000 probably close to 2000 That's a that's a hundred percent profit there So if it did it here, it could definitely do it here again So went that as close as possible to a thousand satoshis is a good entry Let's see it against usd You have a bull flag Basically the same thing here you're looking to enter it as close to 12 cents as possible So 12 cents is where you want to enter You want to have a stop at around 11 cents and a half um and your target is going to be every every one cent so 13 14 15 16 17 So you could definitely enter here at 12 cents and make a nice profit Long term we're talking about, you know, it was all the way to 35 cents All the way to a dollar So there's a lot of potential here There's cardano Take a look at link Look at this big red candle See how there's not that much History to go off of so this is since it's a new coin. It's not it's not enough To really go off of you know, like right here like we're just up in the air right now We don't know where this is going because history doesn't tell us anything So we just have to look at immediate history, which is usually not as uh, Not as good as you know information But uh, we talked about these levels Last week I told you guys around 13 To 14 And look where it dropped to 30 exactly 13. Look at this So I know we we got in around 14 ish But it did drop all the way to 13, which is exactly where we told you guys to enter We told you guys not to enter up here. We spoke about this So that was this is a perfect call Hopefully you guys entered did anybody get in on that trade look at tezzels For some reason, uh, youtube loves tezzel So we're gonna be looking at tezzels as well. I need some water run out of water All right, so we're at 1362 right now This is the same thing we told you guys not to enter up here. It was too high already. He's got to wait for a pull back Um, I think we told you guys around this level. I don't remember exactly where we told you guys last time But 330 was definitely an entry and that's exactly where it went down to Look at it 330 right here 330 right here 330 right there 330 so history you see all history told us A lot about that number when a number keeps popping up like that in a chart That's a number you got to pay attention to So 330 is the entry if you're looking right now. So at 362 you're looking to enter enter 330 ish Is your entry? um other than that I wouldn't really enter anywhere here Maybe if you want to try a breakout entry above around four dollars. That's riskier Um, but viable it does have somewhat of a bull flag form But it might be invalid with this big drop here We have to we have to let it Consolidate a bit more if it can keep consolidating in this range Then it'll be a higher probability trade and we're also currently at the 50 day moving average Oh wait, there's a 20. I think yeah, there's a 20 day moving average 50 is this one right under So if you enter at 330 you're looking to sell Around four dollars close to four dollars is possible. Your next target is going to be around 420 ish Your next target we're talking about 450 Look at your three targets Your stop has to be around See that's why I want to enter at 330 So I can have a stop under 320 If you enter here that's a wide stop So I don't recommend entering here Need a pullback or a breakout Let's see what we got here. So I'm already looking at this level. Let me go See what else so this is definitely an area See it's look all this This wide range right here And then it's kind of basically doing the same thing here very wide range So we're talking about A low of about 17 dollars And a high of 30 dollars So that's a it's a pretty big range Um, if I'm entering this so let's let's see down here. Let's see where it went down to After last time I hit this area Drop down to eight dollars So we're in the eight seven to eight dollar range Went up to 20 dropped back down This one dropped way down This was in march. Okay, everything dropped in march Dropped to five and it's just been up since then So anybody who bought down here At five Five times their money at 25 So they bought the they bought at five. They're definitely something at 25. That's five times their money. Um But the most likely spot was eight dollars here So so check out this big support right here, right? So I'm always telling you like to pay attention to where where professionals are are buying where that support is So eight dollars is where a lot of people probably entered And the rejection was that Let me see This first rejection here was at 16 dollars. So that's already 100 profit if you entered at eight 16 Then it finally got over but then it got dropped right back down People probably bought around eight again It's also for you So 16 it went over it, but then it dropped right back down. These are 100 profit takers Dropped it back to 16 People bought it back at 16 What is this over here? The high is 26 and the high is 30 It's like Same weird stuff All right, so and then we're currently at 23 so I mean The target's here Well, if I'm entering If I wanted to enter this trade right now Let's zoom in So when creating your plan you always want to Find out where you would enter, right? You want to know where your entry is going to be. That's the first step So I'm entering at 20 dollars. Look at this Strong support 20 dollars a good range Um, and this looks like a another a minor another support right here. So which is 23 dollars So 23 dollars is the first support 20 dollars is probably the next one looks like 21 has some support Each whole number has some support. So these I would say these are minor supports Uh, so 23 22 21 all kind of minor the main one you really want to get into is 20 And even then so at 20 if you enter at 20 You're still looking at about a three dollar risk Because you want to make sure you're not gonna get whipped out It's already gone down here twice It could do it a third time Um, or you can if you want to play it safer You don't want to have a risk. You don't want to risk that much. You could do under 18 50 which is a dollar and a half risk Or if you want to be even safer you could just play below these So 19 dollars. So that's a one dollar risk If you're entering at 20 with a one dollar risk, um, you're looking to hit at least 22 So that's a two to one ratio you could definitely then you're looking at 23 24 dollars 25 They're all like miners, but these are all going to be profit taking spots And then this is all the way to 30, of course 30 is a big whole number So that's another spot where they're going to be taking profit It's also 100 from if you enter that the 15 mark Which is like around this area here that people probably bought Close to that 15 mark. So that's another hundred percent Yeah, man. So 20 is where I'm looking at to enter Your targets are gonna be if if you're currently in a trade I don't know what your entry is at. What's your entry chipmunk? Because 25 is probably the next Big area that you'll be looking at and then I'd be aiming for 30 after that But where was your entry entered today at 23? Yeah, I mean 23 is a is a minor support. So, you know, right here it shows that there's definitely buyers there I just don't like how wide this is. You know what I mean So you have to be careful if you enter at 23 You could probably have a stop under 2180 and you might be all right And you want to hit at least around 25 ish You could hit 30, but Look at this huge Wide thing like that's the thing like it went literally from 20 to 30 And it just dropped so because it it was such a big move It could drop back down to test 20 again So that's why I'm I'm not looking to enter at 23 If you're already in it, I mean you're already in profit Even though it might be small You just set a stop What's this number here 22 22 So there's just also an area you could set a stop under Which was yesterday's low So maybe set a stop under 22 22 It has it's under yesterday's low and it has a 20 day moving average So that's giving you about an 80 cent Risk for a two dollar reward So that's a pretty good risk reward Looking you'll be looking to hit $25 will be your exit I would take profit on 24. Honestly, I'll take Some profits at 24 profits at 25 and then let the rest Ride to see if I can hit 30 30s Basically your your own shop That's pretty much it for that one. You got any questions on that? It does have a very very solid, you know Higher high higher lows going all the way up and it's been it's been pretty healthy just bouncing on that 20 day moving average Except here this word broke it Went back up and it looks like it might be coming back down to test it again The important part is not always Looking it's not always about your entry like obviously your entry is what makes the difference As far as how much profit you can make That that's what the end that's where the entry starts to matter. So in the beginning The entry is not as important as your trading plan Your trading plan is the most the single most important thing you can do If you don't have a trading plan, you can't be profitable If you're if you have a trading plan And if you're even making bad entries as long as you're consistently following your trading plan You can still be profitable. You just won't be that profitable. Does that make sense? You'll be more like uh in the break even to you know, barely profitable area so you could Just by having a trading plan you could you won't go broke though. You'll never go broke. So you'll you'll be break even to some profits, you know um Which is great, you know, when you're when you're when you're starting That's a great zone to be in um Then what what When that gap comes in between, you know, the the difference between someone who's very very profitable and someone who's Just break even to barely profitable Then comes in, you know, the entry and the more the more, you know intricate details But uh, but yeah, the trading plan is the most the most important thing So as long as you're approaching it with the trading plan every time um, you're fine Just just work on adding your you know, that those perfect entries and no entry is ever is ever going to be perfect There's no such thing as a perfect entry Yeah, man simple is simple is the best way to approach this I see a lot of a lot of people like to to over complicate it Just because it looks a lot cooler, I guess um Also the guru gurus or whatever They like to make it seem way more complicated than it is so that they can sell you a you know $10,000 course or whatever so What's up guys you guys made it to the end of the video appreciate you guys if you made it all the way through man Thank you again. Uh, this is it for the video We will be back with another live stream and market analysis in a few days Thank you again remember to like subscribe and leave your answer in the comments for the question of the day For a chance to win that free membership. Let's go


Around the world, every day, thousands, if not millions, of people are experiencing great violations of the fundamental freedoms and the human rights. We cannot close our eyes and we have to take decisive action. And to be crystal clear in this room, the universality of human rights and the achievements of the World Conference in Vienna 30 years ago are not subject to negotiation. We need to be aware of the transversal nature of human rights and it really needs to infuse everything that we do. Technology is very important but at the same time technology can become the demon if unleashed in a proportion that cannot be controlled. I think we can still find ways and means to get the right guardrails in place for the development use and regulation of AI and again human rights comes in as a solution. Young people are organizing for the challenges of the experiencing, be it climate change, be it issues around water rights, be it issues around participation, representation and are willing to actually take the action, organize and actually be part of the solution. Testimony of two women in court is equal to testimony of one man, which means that women are considered half a human. If this is not the gender apartheid, what is this? So I have a clear proposal that I hope you will take into consideration my proposal that gender apartheid should also be introduced into human rights concepts. When you need to breathe and you don't have to beg people for that, the same goes for education, the same goes for voting, the same goes for existing as a person. I think that's the freedom of existing that way, the right to have that and having that with dignity.


Alrighty, so let's play this and I will be checking out the notes So you are thinking in parts here the flight to the landing the walk then the interaction with the baby dragon and then Jump and finally the roar. I try to keep it short. 400 frames is not short That's a it's a lot of work You're hoping that it's clear enough to be able to tell each action in the story I am going for yes I think it's super clear. It's just also super long You're also asking about the camera at the end. There is no problem. I can give you little tips and tricks about cameras looking at this And you have a pretty quick translate back with almost a counter rotation Giving this a bit of a pivot off of here almost. I mean we can definitely tweak that I think you can easily just leave it like this and then the dragon just enters frame and Then we can follow that's all okay. We can talk about camera shake a bit more So my thing is Let's pretend we start, you know flat and dragon comes in Super bomb This given given the camera shake right in the way you're going for those seem kind of fast and skittish a bit of a quick quick quick quick so I would say if you're going for the dragon is You know, it's that little thing here a is also really really fast. This seems more like Eating so do you want to do this where it's It's the parents right father mother or whatever dragon. It's they creature And you want it to be caring so that action would be a lot slower And then there probably would be a moment of yes, my kid is dead And a bit of a bigger moment and that whatever acting you want to have and then there And then the angriness But to me it wouldn't be To me it wouldn't be this it will be in place. It's almost like there. I say a cling on roar and By roar I mean the cling on death ritual Where it's more your in place you realize, you know the character whoever it is in relation is dead And then it's kind of a I think so I personally to be honest and this you can absolutely do whatever you want To me it would be coming in but faster It's it's and you can always have maybe maybe the baby dragon is still, you know breathing or not It doesn't really matter.  I'll be able to bleep that out then dragon does a quick Quick hop step step step over to here, which will give you the head here the body here Wait, does this have a tail? Yes. Okay, so Here Legs why and the horns why because then you have a silhouette because right now This is there's a lot of staggered stuff and given of the given the textures in the colors the hue and everything It's just not very clear and I want you to do Where is the nuzzling here? You mean you have a leg then you have the head over the leg you got the head here the horn Silhouettes missing so all that if I do my squinting silhouette test. I see a blob with the horn Versus imagine. It's like this head is down Horn and the baby's here. Obviously we flopped the whole thing, right? But this would be a much Clear silhouette. So again to me coming in fast Bum-bum-bum blend Or I get away from my child whatever it is then hop back quickly to this But we're much more profile and then instead of doing this it would just be like I said here Bob-a-bop and just going up and again because it sideways You would have such a stronger silhouette where the body can be lower, you know Wings can be slow or can be low and then go high up But you have a really really clear silhouette clearly my drawing is is not going to convey any of that But like imagine this would be your last war of type of thing of pain But the other way for silhouettes anyway That would be it and in that way, you know You can you can have the dragon swoop in like that could swoop in like this so it's not just straight It would be more interesting to have a bit of a swoop coming in just so it's not just a One-axis this seems very straight one-axis. So I would make that a bit more interesting You can come down up like this again swooping this way just whatever you have just more of an arc and Then this again just feels too simple Bum-bum just comes down and that's it and then quick steps, you know unless again I mean you can totally switch whatever you want to do and come in here and then I would do maybe one or two bigger steps to really show that it's heavy And then maybe the last two steps are much more slow in the liver where the the paw or the hand Whatever you want to call it comes down and just kind of goes forward and then slowly comes out It's like the closer it gets to the disease child It would be slower and softer and then maybe puts the head forward and then puts the head away And then really forward and then they contact and it's a slow contact of just rubbing cheeks and not Not this cold What is this meat, right? So it's kind of up to you.  So I don't know that's kind of up to you to decide but these are my My thoughts on that. All right. Thank you All right There's an email you can sign up you can start whenever you want you can submit whatever you want You get 16 submissions either way a like and subscribe would be awesome. All right. Thank you


keep it clean I was going on the same grave in here with another video and let me know if you heard this before the Saints are over the cap now of course you've been hearing about it all this offseason but this is nothing new with the New Orleans Saints they're always over the cap this is what they do but they continue to still find ways to get under each and every year and they still find ways to make all these crazy moves each and every year so if I'm a Saints fan I'm not really worrying about none of this but at the same time it's like wait a minute so manual Sanders manual Sanders was a free agent last season and the Saints he was deciding between I think the Saints and I forgot what the other team was but anyway he obviously chose New Orleans they had a good season there they went to the playoffs there and one thing that they did for manual Sanders which I absolutely loved in the last game of the season because you know that's when players will not in the last game of the season but throughout the season players can earn incentives based off of playing time based off of amount of catches amount of touchdowns and so on and so forth but they gave him they threw a bunch of passes to him so he could earn his incentives but unfortunately today for him which he announced on his Instagram the Saints are releasing him to get under the cap now a manual Sanders he's gonna find a job no problem we already know he's gonna find a job great route runner good hands good teammate it ain't gonna be no problem excuse me for the wind that you hear out here anyway um the Saints they they also getting ready to release Juan Alexander so say Saints have been for any they release of course a couple weeks ago Thomas Morse that so they've been very busy they made some other moves like us and people on the trade block they they want to extend some deals I believe Marshawn latter more they want to give him an extension and they want to make some other moves so the Saints have been very very busy very very busy and Drew Brees status is still up in the air I know we saw him the other day in the parking lot pushing pushing the weights on the wheels and whatnot not getting that workout in so does that mean he's gonna come back I don't know it doesn't mean he's just staying in shape I don't know but it saints got a lot of questions right now they have a whole lot of questions right now but trust me that they'll be more than fine finding the answers to those questions because this is what they do they do it every year every single year this is an annual thing for the Saints annual thing so even though they are getting rid of a manual Sanders even though Kwan Alexander's getting ready to be cut they just they're doing what they need to do to get under that salary cap because when free agency starts that's the deadline you have to be under the salary cap by free agency and oh boy this is going to be this is probably gonna be one of the most wild free agencies ever because teams you may really start to see like quite a bit of super teams because a lot of good players are going to have to be cut because the official salary cap came out today and so now people know like okay it's official now it's not a rumor is not oh it might be this it might be no it's official now so teams are doing what they need to do to get under that and again Saints they've been they've been a lot of action going on in New Orleans so we'll see how this whole thing shakes out shout out to Mayo Sanders again you gonna get scooped up quick and now this gives him an opportunity and an advantage over other free agent wide receivers because he can find a home now he doesn't have to wait for the market he can find a new new team to go to right now because he'll be he's a free agent right away he doesn't have to wait till next week he doesn't have to wait till free agency officially starts or the legal tampering period which is one of the things they just need to get rid of that it's a waste of time but he can go to whatever team he wants to go to right away so anyway team keep it clean appreciate y'all a turn of notifications on cuz next week is going to be crazy the craziness has already started but next week is going to be crazy it's gonna be insane trust I promise you it's going to be insane have the notifications on follow on Twitter on Instagram everything is engraving vids I love y'all I appreciate you watching we just dropped a new episode of the team keep it clean podcast just came out link is in the description appreciate y'all team keep it clean thank you for rocking with us and y'all will be seeing a lot of me especially next week so just be prepared to see this ugly face so anyway like Emmanuel Sanchez when it comes to the Saints I'm out