So hello everyone, we just remind you that you can still propose lighting tools and give a vote on these that were already proposed on the board that's in front of B105. Also, don't forget to give feedback for speakers on DevCon website on DevCon.cz. Okay, hi everyone. So yeah, in the space team, we have to make some windows build and to distribute some windows components. And yeah, usually people think it's very complicated to do. So I wanted to use this talk as an opportunity to explain how we are doing it and to actually show you that it's not that difficult. So my name is Christophe Fergero. I've been working at Red Hat in the space team for nearly five years. And so Marc-André will be giving a short demo a bit later. So first start, I'm going to just explain a little bit what SPICE is and which components are making up SPICE and all this stuff. So what is SPICE? Well, it is a protocol which allows you to connect remotely to a desktop. This is usually used to connect to remote virtual machines. But we also have some driver, which you can annex in the driver, which also allows you to connect to an X server running on bare metal and not in a virtual machine. So yeah, this protocol was initially designed by KumaNet, which was bought by Red Hat. Red Hat opened the protocol, opened the client, opened all the components of the SPICE offering from KumaNet. So what are the various components of SPICE? So since it allows you to connect to a VM, so it's related to QMU. So actually we've got a shared library which is used by QMU. And so the shared library is implementing all the remote stuff. So it is the shared library which is taking care of exporting the display so that remote clients can access it. So yeah, this library is called SPICE server. So yeah, if I mention SPICE server later, I'm referring to this library which is running on the host, so server side, which is used by QMU. And then, well, you've got your QMU running with a virtual machine using SPICE. And then you want to connect to it. So we also have to provide some client applications. The one I'm going to talk about in this talk is a Vaguar. But there are actually several different clients because we're actually providing a library which is called SPICE GTK and which is used by different applications. So GNOMEbox is also providing SPICE access. Virtual manager is using it to solar you to access your virtual machine using SPICE. Vinagre is using it and maybe a few other projects. And so in addition to the server part, in addition to the client part, we also provide some drivers, some demons or agents to be installed in the guest. They are not mandatory, but this is useful to provide better integration and to get better performances. So yeah, we've got plenty of components. We cannot only support Linux. We also need to support Windows. So we've got people who want to use the SPICE clients from a Windows machine to access, for example, a Linux VM. And we also have people who want, this is the main use actually, I think, for SPICE. We have people who want to access a Windows virtual machine using Linux clients. So this means we need to have a client running on Windows. And this also means we need to have the guest addition. So we need to have Windows drivers and Windows agents to run in the Windows guests. And so yeah, so about the features of SPICE. So yeah, I said this is remote connection to remote desktop. So what does this mean? So we've got the video, which is exported remotely. So you connect and you see your desktop. So this is fairly obvious. And so when you're connected to this desktop, you can also like change the size of the window of the viewer application. And it will trigger a resize actually on the guest. So the remote desktop will always exactly feed the window of the client. You can also have multiple screens. So either like multiple windows of the client, each corresponding to a physical screen, not physical, but a screen on the guest. And if you have, for example, three screens plugged in your laptop, you connect to a VM where you enabled four screens. And you put it in full screen and like on all the screens on the laptop screen, you will have like all parts of the remote desktop. This supports audio. So both sides. So if the virtual machine is playing some sound, this is going to get out on your laptop. And also you can record some sound on the VM using the microphone from the laptop. So with PICE, you also get USB and smart card through direction. So what I mean by USB redirection is that you have your laptop which is connected to the remote VM. And you plug some USB device. So this can be a USB key. This can be the laptop webcam if it's plugged through USB inside the laptop. This can be basically anything you want. This could be a keyboard or whatever. So from the clients, you can say, OK, I want to reject this device to the VM. And then instead of being available locally on your laptop, you'll be able to use the USB device remotely inside the VM. So you will see your face. If you're using the webcam, you'll see your face inside the VM. If you're using a USB drive, you'll be able to access this to see it remotely on the VM and so on. And we also provide a copy and paste between the client operating system and the remote guest operating system. So this is much better if you want to interact between the two systems. It's much more convenient usually to be able to do that. You've got a URL where you want to paste in your Windows machine because this is a Windows only website, some crazy stuff like that. And we also have some kind of file transfer. For example, dragging a file from the client on the viewer windows like the file is going to get copied to the VM. So how are we doing all these builds for Windows? So it's actually not that hard. I'm going to explain to you how we are doing it in the, not the rest of the talk, but the next few slides. So for the client, as I was saying, the client is a remote viewer. We've got different ones, but I'm going to focus on this one because I think this is the only one that we ship on Windows. So we are using a project which is called MinGW, Minimally Gnu for Windows. The goal of this project is to provide GCC builds on, I mean, the whole GCC tool chain, subinuities and whatever you need to build an application. So this is a version of Windows of GCC which is going to generate Windows executables rather than, I don't know, Linux executables or whatever. So yeah, this is a compiler. This is the tool chain. You've got like some libraries. You've got some headers in order to be able to use these libraries. Yeah, like a whole package of a lot of stuff. There's another project which is also aimed at building Windows executables which is Siegwin. The main difference between Siegwin and MinGW is that Siegwin, they have their own library for like some kind of C library when MinGW is trying to reuse the existing Microsoft libraries. So it's a bit easier to deploy because you can reuse what is already available on the Windows system. So yeah, MinGW, you can target either Windows 32-bit version or 64-bit version. And the important thing is that you don't have to run it on Windows to generate a Windows binary. You can use it as a cross-compiler and this is when things get much easier.  And this is actually how we are using it to be remodular because this means that you've got all the usual environments. So you use your development environment on Linux, your usual text editor, your usual everything like even your tool chain, the auto tools and everything. You can just use that and then you just build these sources using MinGW instead of GCC. Instead of the standard Linux GCC and in the end you will get a Windows binary. So yeah, in addition to MinGW, so yeah, it's nice to have it but you know if it's only available as a terrible and you have to build it and everything, it's much more complicated. But luckily this is also shipped in at least Fedora, I think Dbian has also some, at least some parts of it. So what does this mean? This means that in Fedora you can easily install the compiler. So you type human install, I think it's MinGW GCC or something like this. So you do this and you get the compiler and the various tools which are needed to do Windows builds. And so it means it's already available from Fedora repositories. But in addition to that, there are also quite a few libraries which were built for Windows. So some DLLs, for example, the Spice Cloud GTK library which I was mentioning, GTK Plus, plenty of libraries. They are also built for Windows and available as Fedora packages. But you will get then a tree with some DLLs and some Windows stuff in it. And then you can just reuse those libraries to build whatever executable you want. And then you can copy the libraries on the executable and everything on Windows and this is going to work. So this means that by using the Fedora MinGW infrastructure, you can get plenty of dependencies already built for Windows and you only have to reuse them to do whatever you want. So this MinGW project in Fedora, this makes things much more easier to use because you just type human install whatever you need and you don't have to build a lot of stuff before being able to start on what you want to do. And so how do we compile the client? This is what I was saying, it's very easy. So if it's from Git, first we need to run autogen.sh and make sure it's not going to run Configure. These days, usually on many projects, you can also use just Autorecon rather than a script and you are done. And then MinGW is providing a script which is wrapping Configure. So in the end, this script is going to call Configure, but it will be called with a lot of environment-variable set and all the parameters that you need in order to do a cross-compilation from Linux to a Windows executable. This command is also working if you run it from a subdirectory.  In the third one, you run the regular Configure and you make some builds in different directories and so you will have in parallel the various versions of your program which are going to be built. And so after running this script, it's like the only piece of magic I would say in the whole build process. You just run make and after it's done, if you run file, you will see that you have a binary in the Windows format. So this is very nice. So once you have that, one issue you are going to run into is that you need to distribute it because you have this executable, but it's using some GLL, some external ones, most likely it's going to need some data files, some microns, some tons of stuff. And so if you just copy the executable, give it to your friend. This is not going to be enough. So you also need to have a way to have some kind of package that you can install in your user system. So yeah, this is called an installer in the Windows world. And so in order to do this, we use a program which is called MSI Tools. This is used to generate MSI installers. This is a Linux tool, so you can still keep using your Linux environment for that. And so MSI installers, this means Microsoft installer. I think they change the name to Windows installers. But the main goal is just to have some kind of executable program. You double click on it and it's a regular Windows installer. It asks you why do you want to install the program. Then it copies all the files and does everything which is needed for installation. And so yeah, in the MSI Tools, the program which is used for generating the installer, it's called Wixel. It takes its inspiration from a Windows program which is called Wix, WixToolset, which is something, some free project actually from Microsoft, and which I think the recommended way of building installers upstream for Windows. So it's basically Wixel. It takes an XML program, an XML file with a list of files to install, the registry entry to create, the dependency like you can say, okay, I also need to install this DLL and this other DLL and this kind of things. So you're in Wixel and it's going to read the XML file to pick everything which is needed, put all of that in a big file and installer. And the nice thing is that this installer is transactional. That is, it knows how to undo everything it's doing. So this means you get a program to uninstall for free. So this means if you use this MSI installer to install your application, you will also have in the configuration panel when you ask to remove applications, you will also see the, yeah, an installer entry for your application that's doing anything. And so, yeah, we have been using this installer for a while to distribute the viewer and remote viewer for Windows. So this is working nicely. Another tool which I wanted to mention is an SIS. It's also used to generate installers for Windows. It's something different, something a bit older. Initially, it was used, it was created by NerdSoft in order to install WinOp. So fairly old music player on Windows. But it was free software. It has been ported to Linux. You can also use this on Linux to create Windows executables. The main difference with Wixel, MSI tools, is that you are not going to get MSI file. You are just going to get an executable file. And also, you don't provide some XML description of what you want to do. But it's more like a script language. So you can do many more different things. But this also means that to have a standard and basic installer, it can be a bit more work to handle everything in the script language. The reason I'm mentioning it is because for the distribution of the guest drivers, so we talked about the client which is built using MinjW, which is packaged using MSI tools. For the guest drivers, we use NSIS actually to distribute everything. Well, the main reason is that it's not very clear how to install, if it's possible to install drivers using MSI tools. When I say it's not very clear, I mean, it's no one tested it. Maybe some features will be missing from MSI tools and this kind of things. So for the guest tools, we ship some drivers. We ship the agent which is used for copy and paste. We ship the QMU guest agent which is used when the guest goes to sleep or when you want to take a snapshot and things like that. We ship the QXL driver and audit things. They need a bit more sophistication for installation. So for now, we rely on the older installer because since the script language, we have a bit more flexibility on what we can do. But it would be very nice if we were able to also do that using an MSI installer. A few issues I wanted to mention regarding having Windows applications or shipping drivers on Windows. It's easy to build your application for Windows. It's easy to distribute it. But yeah, you also will encounter some Windows-specific issues. We still happen for us actually from time to time for a virtual. So for example, in virtual, we had some issues. We wanted to pass as many keys as possible to the VM. But for example, Windows, it intercepts always control-alt-delete. So we had to do something in order to be able to send control-alt-delete to the VM. Recently, we had a bug about a Japanese user saying, oh, there is this key on the keyboard which is used to switch languages when we type. And it was also not being passed to the VM because we need to call some special API in order to do it. Yeah, among the various things to pay attention to, if you want an icon in your application, well, you will have to add what Windows calls some resources, which will be a small icon embedded in the executable file. So you need some special code if you want to under that. And also something that can be a bit tricky to get right is that on Windows, contrary to Linux, on Linux, you want to install your package and basically you know where everything is going to be. At compile time, you know where you will find your data, you know where everything is going to be, and your application can just use that. On Windows, when you install an application, the first thing that you get is you are asked, okay, where should I install the application? So this means all the data for your application is not going to be in a known location, but it's going to be in some sub directory of the installation location of your application. So you've got some API in nglib to handle that, so basically you have nglib, okay, give me this pass relative to the pass of the executable which is currently being run. So it's not that hard to handle, but you still need some special code to do the right thing and be able to find your application resources. For the drivers, well, yeah, Windows drivers, they are not going to be portable between Windows and Linux, so they are like very Windows specific stuff. A few issues we have with them that so far, I mean, so far. Last time I tried, they were not buildable with MinGW, so we had to rely on the proprietary Microsoft compiler. The main reason for that is that Microsoft is providing a special framework for drivers, and this framework was not available in MinGW. Maybe this has changed recently, but I haven't checked, so yeah, it's a bit sad that we need a special proprietary compiler to build these drivers, and then you have one very specific issue we have with Windows, it's for the graphics drivers, the video drivers. We have one for improved performance, and we need a new one for Windows 8, so they changed the API, we need to use some new API, and unfortunately, finding Windows driver developers in our Linux world is a bit complicated. One also potential issue is about the format of debugging information, which is actually, when you use MinGW, you will get draft debugging information, which is not the format that Visual Studio is using, so this means you cannot use Visual Studio Debugger, you have to use GDB or some related tools. This might be improved and changing since Microsoft has been opening more and more sources. There is some reference code which could be used to write some converters from draft to Microsoft debugging format, even to output this natively, but I'm not sure that this work has been done so far. So that was it for the Windows building of Spice. If you have some questions about this part, I can take some questions now, otherwise we can get back to questions to the end. So any questions? So regarding the Spice Landside improvements, I'm going just to do a short one to slide on this and then Mark Andre will be doing the rest. So recently, or not so recently, which changes did we have in Spice on the client side? So I wanted to mention WebDef sharing. So actually this has been present for a while. What does this mean? This means that the goal is to be able to share a local directory on your laptop and share it with the VM which is running remotely. This makes things much more convenient when you need to exchange documents or files between your laptop with all your everyday files with a VM where you want to run some tests or do some development or something. So the reason for using WebDef is that in Windows and Linux usually it's supported natively so you don't need to install a lot of things in the guest in order to be able to accept the shared folder. And what is currently missing actually is just more advertisement which is what I'm doing now because it's available in VAT Manager. You can add the necessary things to your VM using VAT Manager. It's documented how to do that on the Spice website in the documentation but it's still a little bit hidden and maybe lacking a bit of integration so that it's more widely used. Like having this integrated by default in Ignom boxes for example could be very useful to improve the experience of people. I'm mentioning the SSH agent work because Fidentia was supposed to come and talk about it but he could not attend. Basically he has been working on some path through of this path through but his goal was that if you had some keys unlocked on your local machine that he wanted to be able to use this as well in your guest so that you don't have to type again the password and everything in the guest in order to access remote SSH sessions. The main issue with that, he has some working patches so when he has SSH he needs remote VM he does not get asked for any password because it's already unlocked on the client. His main issue has been upstream integration because he does not want to take over the agent on the guest so he wanted to have some fallback mechanism between the small agents he needs running on the guest in order to do the space communication and he wanted some fallback on the agent which was already running before on the guest which can be for example Ignom Kering and like doing this upstream and getting this integrating upstream is a bit complicated so it's not clear if you are going to be able to provide that or not. And now I'm just giving the microphone to Marc Andre who is going to talk to us about virtue. Can you hear me? Yeah, I guess. Okay. So yeah, initially I was not... I replaced credential basically and Christophe asked me if I could present quickly and demo quickly the status on Virgil. For those who don't know, Virgil is a solution to do GPU accelerated rendering. It's developed mainly by Deverelli and basically how it works is unlike hardware accelerated VGPU solution from Intel or Nvidia and what's not, this one relies only on software components more or less and requires your host to just have open GL I think 2.0 something minimum support so in theory it could work with any hardware although it's not been tested a lot so the way it's implemented it's using Mesa so it's the core and in Mesa you can implement driver in various ways and the coolest way and maybe the easiest way is to use Gallium so it's using Gallium and the way it works is it's kind of translating like the internal Gallium state and commands back to open GL and that's basically the idea behind and then all the rest is like gluing everything so you get on the guest you get a Virgil GPU device that will basically pass all these commands back to the host and if there is like queries or data to send back to the guest and it's also implemented as a regular, very regular like Linux DRM driver and that makes things much easier because then for supporting like Waylon or X even today we don't need to implement another driver in user space we just use the X mode setting driver and thanks to Glamour we get open GL accelerated rendering for X so this is very nice because this is less code for us I mean for like maybe in the future for Spice because Spice is using your QXL driver so it's a different model so this one you know if tomorrow, if today, even today you can see like there is a lot of work going on in Glamour and it will benefit us basically for free so that's really nice and the current status is in the guest the complete GL 3.3 support and Dave has been busy doing a lot of like the next extension so it's actually pretty complete I would say and soonish 4.1 and yeah and there is already 4.4 extension ready and so it's pretty complete and if you run like Piglet test which is like the test suite from OpenGL like the open source OpenGL test suite with like 10,000 of tests and last time I run it was running like more than 95% of tests successfully on my laptop so pretty good so status has been merged in QMU 2.5 in kernel 4.4 it's released and Mesa 11.1 so it's basically more or less released for the past two months and so the remaining stuff basically you can use it today but you know it's not convenient so I don't know if all of this is going to be in the next Fedora release but I hope so I think that's the goal but it's still not so convenient to use because you need to configure everything yourself more or less and also it's only like the QMU command lines sort of and so it would be nice if you know you could just create a VM with boxes and it would be accelerated and you wouldn't even have to think about it so I can... yeah that's it so I can show you basically quickly if you want to see it I can show you we're running today with Pyce because we have patches ready for review they should be soon there so I will... yeah I will show you my laptop if you don't see anything here okay so it's very much my... it's my like development environment more or less so I didn't really like prepare too much this talk okay I will zoom in let me don't see it okay you can see here I want to show you like quickly the command line so command line you know it doesn't change much so we will... of course you need to have like a Verd.io device with Vergeal Enable and then you need to have like Pyce parameters to enable OpenGL 2 and currently you can specify the unique socket like this but in reality all of this you wouldn't have to care you wouldn't have a unique socket listening it's using Nivert somehow so... and then and then I can connect to it currently with just a client like this I give... yeah I can give the unique socket and yeah we are in the desktop so everything work as expected more or less you know so it's hard to see like the difference between a regular Spice client but a Spice driver I can run some OpenGL stuff my favorite is GenMark because I want to see like the performance so it's accelerated and and one thing I really like is it's intensive using shaders for example this one ShaderToy if you don't know it's really putting your driver to the knees because it's compiling like complicated shaders and the driver are not so good at compiling shaders actually and they can take some time minutes even freezing your pages and so it's... if I have network maybe I don't I mean I can run something so someone favor it's game like yeah so and you can see like people like to do that so you can disconnect and reconnect obviously yeah it's running okay so we won't see the shader I guess oh yeah okay so it takes a while to compile but if you test it on your like natively it's also that long I mean sometimes it's really long in general I think the performance we are you could say on average maybe 50% from native in the best case close to native and in some of the worst case like 30% from native but it's pretty decent like if you run this on your laptop without VM you get like 30% more performance but it's still quite close I mean and the benefit is I mean you get like accelerated OpenGL anyway so it's an even like I don't know what's this some bug I guess so yeah that's it if you have any question now okay so I didn't talk about that yeah it's currently only local and there is plan how to do this remotely using GPU encoding but currently it's only local and yes you can run several VMs it's no issue some of the issue will come up like quickly because like GPU scheduling and so on they are not but it's the same with traditional application so it's not clear how like things are being shared between a VM and you cannot like you know allocate that amount of resource of GPU to a VM and so on so so this this kind of things will come up I don't know this is more like low level and it's it's true for any OpenGL application anyway so yeah so for spice in general I mean spice in general doesn't really care much is it's complicated like I mean it uses I mean large screen are like using a lot of memory and and it's raising now and QXL so what was the limit with QXL again I don't remember because I've not been involved with that for a while yeah so I can just say a word for Virgil it's not I mean there is no limitation basically it's whatever your hardware handles as far as I know it's like if your OpenGL context can create like really really large frame buffer then no issue it's just a matter of like how big your frame buffer can be off-screen file buffer yeah with Virgil or with QXL with QXL it's all a matter of the amount of memory you want to dedicate to it usually we put the limit at four simultaneous monitors which is already quite a bit of memory if you want to go with a full HD monitor with four full HD monitors maybe you can use more than that but then you would need to no? okay so what is the question exactly I mean like the type of scenario basically if you have all your VMs running on a corporate network running on internal IPs and you want people out of the corporate network being able to accept the VMs like people in the hotel wanted to connect to the remote VM then they would just connect to the VM through the proxy for example I guess even internally you may want to isolate the VM network from the rest of the of your corporate network or things like that so we had one last question here and then we'll be running out of time so this was developed at the time of USB2 and I think most of the standard was supported for USB3 we might be missing a few small bits but yeah so USB2 devices are going to work and I think some USB3 devices are also working nicely yeah so we are running out of time so thanks everyone


So predictors are data element values that are automatically generated from samples of current and past data depending on the periods you have specified to be included in the calculation. So we need to define what is our period that we will select for generating this outlier or a predictor value, what will be the current and the past data for what periods will be used them to calculate the predictors. So for example for reviewing immunization data quality predictors can be used to analyze previous data in order to generate outliers for future data. So this is a little complex and for this academy we will mainly focus on what a predictor is and how it is used but we will not go into the details on how do we create a predictor. But there is this link in the presentation for the manual which will be helpful if you want to further create predictors yourself or learn more about predictors. But I can show you one example on how these predictors work. So say for generating a predictor in DHIS2 we say that we have to use the average for last 11 months and then three times the standard deviation. So suppose this is the way how we are calculating a threshold value. So then the periods would be 11 months that precede the last month. So for example I am generating the threshold for December 2020 then I will skip November 2020 and then last 11 months. So 11 months before November 2020 will be considered. So then we'll use December 2019 to October 2020 in my calculation. So this is the logic that we use for the threshold creation and we'll just exclude any value that is an extreme outlier. So if we look at our predictor. So suppose I have to calculate the predictor the threshold value for December. I do not consider the date of November. I consider the last 11 months which is December 19 to October 2020. The average of these 11 months is 122.8 and standard deviation is 16.4. So then my outlier calculation would be average plus three times the standard deviation which is 173. So this we've already predefined in our application the thresholds for the semenization example that we show. So then in this scenario if my threshold or outlier is 173 that is my predictor's value. Then any value if I enter above 173. So in my December data entry if I enter 180 or 200 then it will show a valuation alert that this is outside the threshold value. Using the generated outliers. Once we have generated the outliers we need to check our data to ensure that our data values do not exceed the outliers that have been generated based on the formula. So we can either use these threshold values which we are generating through predictors or we can import the values from outside say subway data or thresholds generated from some other software. Those values we can import in DHRs too and we can use those values for our validation rules. So the process is same we need to have a data element where these thresholds are stored. These can either be stored by calculating from predictors or these can be imported from external software. And then we perform a manual check by reviewing the actual value versus the generated value. So we'll see them in the application in some time. This is just an example what we'll see. So suppose for Beggar Hospital we see that RV2 doses given, rotavirus doses given to children is 6757 while our threshold that we have set up based on the last 11 months is 700. So then we can say that this value exceeds the threshold value and we need to go back and check this data. So it looks quite obvious and maybe looking at these values maybe there's an additional digit which has been put in by mistake. So this will just help the user to see if there is any mistake, a manual data entry mistake in the data or they can just cross check and correct it. So how do we use these generated outlier values if we go to each months, each periods data entry screen and look for these validation errors? These might be difficult because you have to look at many time periods to assess the threshold values. So this may be difficult however if you want to look at many time periods, organization units and data elements together at the same time for consistency. And it's always easier to run these outliers through validation rule analysis. Though we can, we will see an alert in the data entry as well but since it makes sense to compare these outlier values for a longer period of time, it's more effective if we do this validation check through the validation rule analysis app in data quality. So we stop here to look at how do we run these threshold values in the application while how do we configure these we look at them tomorrow. I think we have a request to describe the predictors again. Next question why no considering normal data for the predictors calculation? So I just understood we have to describe the predictors again and what was the question? Why no considering normal data for the predictors calculation? So I mean that's how these threshold values are configured. These are the standards that are being followed and based on these guidelines, so if you look at the manual, there are these guidelines what periods to skip, what periods to take in the count. So for our example our logic is we for predicting a threshold value, we don't take just the last month's data, we take the 11 months data before that and we predict the values based on those 11 months. That's how the guidelines which is followed for this indicator but I might not be able to drill down further on the conceptual thing of what periods should we consider for what kind of thresholds. I think there are few details in the manual but then we need to refer to guidelines for setting a threshold for the different type of data fields. And just to explain the predictors again. So basically we create a data element and we predict that what should be the threshold value. So say my data element is RV two doses of rotavirus given to children and I have to set up a threshold value just to adjust the quality. Yes please. What if you want to calculate the predictor was a period of math or beyond the event math like we want to create to calculate the predictor for its math well and we succeed by doing so. Yes we can do that so when we look at how we create a predictor tomorrow though I will not go into detail on I mean we'll not practice how our predictors being added but I'll run through on how a predictor is created. So therein we can set up these values that how many periods do you want to consider how many periods do you want to skip in between. So we'll cover so you can define if you want to use just six months you can define that while you set up your predictor. So it means that if you want to calculate a predictor for six months we will not consider the last month maybe if it's from January to June it means that you will not consider June. So if you have to calculate predictor for July so then your last six months will be the months preceding June. Okay. So this same to May yeah okay so just a quick recap on the predictor so this will help us assess what should be the threshold value so we predict these values so we can do a calculation take an average of last eleven months data add on the standard deviation to it and show me what is the threshold so based on that it will say that okay based on your data average and what are the standard deviations maximum should be 700 your doses should not go above 700 otherwise it's an outlier and so you could compare those values with the values. Now these threshold values we have for this example calculated through predictors but if you have these auto generated through some other tools we can import them in the same data elements. So if that's okay to die move to data entry so now we'll see an example of how do we run these predictor generated threshold values and related validation rules. So for this example I have selected Svegel hospital gateway PHC my immunization program and now when I run validation so I see for my DPT dose it shows an error that DPT dose administered is less than or equal to DPT dose threshold. So now if we look at these monthly validations they though they show us that your value is exceeding the threshold values but it's not compared easy to compare them on the data entry app it makes more sense to go to the data quality and run validations there. So if I go to data quality and try to run this validation step go to validation rule analysis and data quality then select animal region for like last three four months and I take the immunization thresholds validation rules. So here then if you've made these rules for all it will also run if there are zero values but let me show you one where we have these thresholds. So if you look at this example here I will look at the details. So rotavirus 2 administered is less than or equal to rotavirus threshold wherein it says that the dose is given is less than I think I've clicked on some other I'll just show you one more example which is more clear than this one. Okay so I think we have data for January here let me just go back two more minutes. So here I think this is not showing value there is some error on the message but otherwise you can see that there's an error in the validation rule my threshold is 700 and my value is seven one three which I've entered if I go into the details it says this is a validation error for rotavirus 2 administered wherein it should be less than the mean and three standard deviations of last 11 months and I could check for this data if this is an outlier or a data entry error and I can correct this data there. Since there are a lot of questions on predictors I will although we discuss this in detail tomorrow but I will just show you how are these set up. These are the predictors which we can define in the application based on these formulas that we have so if I open this predictor for DPT threshold values so I can say that every per every month I am calculating the threshold so the threshold values keep on changing for every month based on last 11 months and here in we define that how many months do we have to calculate so sequential sample count means I have to take last 12 months and I have to skip one month so that's where I'm saying sequential skip count so for this formula I just take the 11 months preceding the last month so take 12 sequences and skip one and then if I go to my generator I have the same formula I'm taking the average for the doses given in the last 11 months preceding skipping the last month and three times the standard deviation from the population where dose is given we'll come back to this tomorrow but then this is where you could set in what are the number of periods that you have to take how many periods you have to skip if you don't want to skip the last month you can just set it to zero there and the steps are same so if you look at your exercises now where the exercise 4 and 5 are for the thresholds so you will be able to see where are the errors in the thresholds and which will be further clarified once you go to the validation rule analysis and generate these so are there any questions if we take an example of where we were ourselves adding the value so even then we'll have these data elements where the thresholds will be stored we can just store those thresholds and once we run the validations it will show the validation based on those values so predictors are populating for each one but you could also manually push the values for these threshold data elements if you're using some external data source if there are no questions maybe we can give 10 more minutes for exercise because there are just two exercises here just one to check from data entry and one from the validation rule analysis and after that we could start with the next session the notifications here so please practice through the learners guide and let us know if there are any questions


OK, so we should start another talk. The topic of this talk is scaling up aggregated logging and metrics on OpenShift container platform presented by Ricardo Lorenzo and Alvel Couric. So welcome to our talk. So we are part of the performance and skill team. Me and my colleague, Elko, we work, we are more from the performance side.  And we all work together to, let's say, improve scalability of OpenShift. So we test on AWS, KVM, OpenStack, different kinds of setups. And we try and squeeze every ounce of performance from the product. So as I was saying, our approach usually is to go for individual components, testing them for limits. Then we try installing them at larger scales in bigger clusters, like 100 nodes, 200, 500, up to 1,000. I think it was the limit, 1,000 nodes we did. So first, we have tools to stress both the control play, like the API server, the ATCD, the scheduler, and the cubelets running on all the nodes. And then we kind of build our own tools as necessary. Like logging these cases, a throughput test is more to test the underlying system, which is another distributed system. So we have OpenStack. Then we have OpenShift on OpenStack, for example. And inside OpenShift, we have Elasticsearch, which is the logging stack, which is itself like memory database, analytics engine, with a lot of things to analyze as well. So like I was saying in the components, like for logging our Elasticsearch, Kibana for the web UI, and Fluendee, which is the pod collector and send the data to Elasticsearch for analysis. Then on the metric side, Elko will explain better how it works, but it's Cassandra Hipster. We also have, we do more focus benchmarks on the components, like Java and Ruby, because these are processes running on the containers. And we try to understand what's happening inside the containers. So just a bit of background on the logging stack. It's used to collect statistics or information log and information, either security information, like access or whatever is happening inside the pods in your cluster. And like I said, it's already a distributed system by itself, so it's usually generating traffic between the nodes themselves. Like Elasticsearch has a master nodes, data nodes. It has to deal with the replication between the data, like shard replication and routing and making sure the data is consistent, et cetera, et cetera. So it collects logs from node services, all containers in the cluster. And it's deployed where usually the Elasticsearch pods are deployed in the region in for nodes. So these are the infrastructure nodes. These are deployed using labels and node selectors and most of all, daemon sets as well, which I'll explain a bit better later. FluentD gathers log entries and feeds them into Elastic. So this is the part which triggers the daemon sets and uses a different kind of label. So this is how in a large enough cluster, you can, doesn't need to be large, but for larger clusters, this is more important because then you can define where exactly the pods are going to land in your cluster. Which region? Primary, infra. In this case, infrafluentD. It's the one used. And Kibana is the UI where you have web browsing. You can visualize your logs. You can build dashboards, try to make sense of the data. So usually people don't run into issues until they reach a certain scale. They don't run into problems. If you have a 20-node cluster, for us, it's quite normal. It's quite small. I mean, it's where when we start to have our baseline, we need also always a kind of a baseline to test against so that we know the system is behaving like in a normal cluster. And we get performance data at the quiescent state, like at rest, just with the logging pods up and running and the infrastructure pods up and running. We do this for one hour as a recommendation by Elasticsearch because it's a way of checking. One hour would be enough time to check for any indices operations, like some garbage collection cycles, or some other things that can happen in memory in the cluster. One thing that we run into a lot, once we reach 200 nodes, it's very common, or used to be, because we already mitigated this, was the rate limit, by default, in the OpenShift master of max requests in flight. Used to be 400, then it was bumped to 500. This basically, this is a software throttle that takes care of rate limiting the long-running connections to the API server, which, in this case, for logging, specifically, the FluentDs are the main connectors to the API server. The FluentD talks to the API server to have to get metadata about the pod, like in which host is running its name, et cetera, its project, its namespace. This is why, I mean, this is the bulk of Kubernetes, right? Metadata about what's happening in the cluster. So the nodes themselves also talk to the API server for command control, networking, the networking also. I don't know in, I just placed it there, I don't know specifically what the SDN does, but, I mean, how it consumes these, how it uses these watches, the API server, but so one thing we notice is that when we really want to scale up, like, let's say, 800,000 nodes, we just have to disable this. We have to get rid of this throttle. We set it to zero, and we start, like, with our tests, like any kinds of tests. So I just have this here for, like, as a reference. So replication controllers are important for when you want to make sure that you have your system in a desired state, right? It's like when you do OC run, and you run a pod, and it pulls the image, this pod is running, but it's not being controlled unless you use a replication control and the deployment config, right? It's the replication controller. Make sure that, you know, if you want, for this example, you'll have three exact replicas of NGNX running. And this is, the replication controller is handling this. I've added a note about demon sets because most people confuse or there is some confusion about what are some of the difference between demon sets and replication controllers. The demon set tends to go aside the scheduler. So it runs on the nodes. And so it binds directly to the node and bypasses the scheduler, which is what it's meant to do. But it's bad for, I mean, we're running to weird issues at 1,000 nodes, for example. I have here a note like the, so why is this important? If it's bypassing the scheduler, you have, let's say, deploy like kinds. It's a demon set. And demon sets also ignore, they can ignore the unschedulable nodes.  It will be deployed in these nodes as well. So you just define the spec of the demon set. It's running. It binds to nodes before any other service is running. But there is a software throttle as well here in the code. So there's a limit performance requirements in Kubernetes in the replication controller. And the demon set and in the replica set, which is, let's say, it's being next as the replication controller, which is like 500. And this was already like a corner case. One of our scheduler experts, Timothy Sinclair, found out that we were constantly having issues. And where the, let's say, OC command would just stop responding in a timely fashion. It was just super slow. And so we noticed because of this throttle, we were having a lot of contention. And we were basically swarming a cluster. So if there's a limit like 500 burst replicas, you will have like a constant time swarming of a huge cluster. So like step from 0 to 500 in a constant time, which was bringing the cluster to its knees, basically. So as I said, demon sets are used to run a demon or a pod on every node. In this case, the logging pods. It's a perfect use case for this, for the logging project. You have your sharded data store in your cluster. You run a logger on every node. And you have your applications tied to the API server, Kubernetes node services. This always, there is always watches involved, right? So persistent connections to the API server. And this tends to get more important at a really large scale, so it's a point too. Since logging is deployed using labels, you label the nodes you want for this kind of project and for using the FluentD node selector. This will be important. Well, when it's deployed already in use, you can. So from OpenShift, if you're talking about the OpenShift cluster, it's usually in the configuration, like in the master config or in the node config, master config. So this is still kind of a control plane. But we develop tools to go to the underlying cluster and try to stretch elastic search. We work closely with the developers, like to try and understand what's the best way of doing it. And eventually, at a large enough scale, so I will explain better what this is later, because I need to talk about the test itself, the throughput test. But at 50 logger pods, close to 205 nodes running, logging 256 bytes per second, we reach the internal queues capacity of elastic search. So this is something that's, well, it's normal, because there are really a lot of pods logging. But we need to know that this happens and try to talk with the developers and fix it somehow. Most recently, we also hit the Fluentee buffer chunk limits, which was kind of taking the problem of elastic search and placing it in Fluentee now. So elastic is now without any limits being reached, or most of the limits, like there is no thread pool or active pools during the test. But then we fill up the memory buffers in Fluentee. So the logging test that we developed, it's kind of an all-in-one, because it's not using open shifts scheduling and it's not leveraging what's important, like having metadata about all of these pods running. It's kind of doing stress testing and embedding P-bench to get the data from all the components. Like we get data from the Java processes and from Fluentee pods. We get all this kind of data. First, if you don't know P-bench, it's a distributed analysis performance tool, which basically wraps FIO, other common tools like stress and perf. It's extensible and it can be run remotely. You can point it to three different computers, three different nodes, and collect data from those nodes. And then P-bench sends the data back to your central node. Then you can review, like generates graphs as well. So you can make comparisons and fine-tune things. So this test is used to load up a cluster of logging traffic at certain rates, at certain scales, and wraps P-bench while it's running. And it's also possible to pass other drivers as well, because underneath it's just Docker. It's a Docker run, mounting dev log, passing the dev log to the container and using a particular image and a particular rate. So still like focusing on Elasticsearch, we delete. This was also a recommendation from developers, like we should start from a clean state, delete the indices in the start of the test, get all kinds of data statistics through Elasticse API, REST API, capture disk usage. Then it generates the load as we want, as I explained before. By default, it registers these tools, IOS-stat, MP-stat. We tend to look more at PIP-stat, just to keep track of what's happening in the processes. SAR is also useful for disk data analysis, like more fine-grained. And after one hour, which is the default duration of the test, it optimizes and flushes indexes. I think this is a recommendation from upstream, well, not upstream, Elastic, in this case, where you keep the, you clean up your shards and indices. I don't know the details, but in the end, you end up with more organized, with the shards more organized. This is what P-bench gives us in the end. So this is kind of a micro-benchmark, which we are really stressing. So it's just 10 nodes and 2K per second during 300 seconds. And as you see the Java process, it's really consuming a lot of CPU. We see system D also a bit stressed, but the Java process is really the... Although it's a short test, but it's already meaningful, you can see what's happening. This is a longer test. So at 205 nodes and 30 worker bots per node. So like during one hour, we can see what happens, like the amount of writes in KB per second. So I just have a couple of short videos before passing to Elko to talk about metrics. So this is like we have videos of this. I mean, we don't have the big cluster available now. So we are just... This is logging and running three elastic search bots, 200 flu and Ds. Meanwhile, the test harness is running, saving all the logs and flooding the cluster. And you can see what's happening in the scheduler and the API server. Actually, there are 250 daemon sets because this is a split flu and D configuration, something we've been testing recently, where we have in-reacher flu and Ds and dispatchers. So this is a 250 node cluster. This is the cluster actually being loaded with bots. And since there's a big amount of nodes, it takes some time. But you can see more or less the output of the testing tool up where it saves everything I described before and collects the disk users. I just pointed to the region info nodes because we're interested in knowing what's happening with Elastic. So this is basically P-bench registering against the info nodes. And there is 20 enrichment flu and Ds and 200 flu and D normal bots running. And you can see the cluster stage. Elastic search is in green state. So it's like we gather this data across time. So just some takeaways from this testing. If you ever have to install OpenShift in a large enough cluster and you start reading that in some project in particular, you start thinking that you should use labels and node selectors. Be careful because this can cause a lot of traffic in your cluster. Just imagining 500 pods at once pulling the images from a registry and being scheduled and everything running at the same time. So it's recommended like you pre-pulled all the images, in this case, flu and D images. And you set the image pull policy to if not present. And you should label your nodes in batches. Never use label nodes all, dash, dash, all. So this will not cause that burst that I talked before. So things are done more, let's say, smoothly, let's say. And you can also keep track of what's happening. Like if there are some image pull errors or some pods going to failed state, or you need to know what's happening and debug it as you are scaling up. So this crash loop backoff issues happen at scale quite often. Match node selector. Match node selector, it's related to the scheduler predicates, but it's not so common, which will be fixed. So still you should use labels and selectors and demon sets if necessary to differentiate, like your infrastructure from your application nodes. And use persistent volumes for your infrastructure nodes in this case, elastic search and metrics if you're going to deploy this. I will pass now word to Elko to talk about. Ah, sorry, excuse me. No, no, we didn't test that. Like using a different, no, we didn't test. Right, some more questions. So thank you, Ricardo, for logging thing of OpenShift. So I'm going to show the stuff regarding the OpenShift metrics and what to pay attention when configuring it for quite a bigger clusters. And that's our limits. And as Ricardo in the performance team, my name is Oleg Kuric and yeah, that's in short about me. So OpenShift metrics, at the time, there are three components in OpenShift metrics. Cassandra serves as a data store. So everything that is collected from the OpenShift pods is written in Cassandra. How cool are these metrics are changing? And heaps, there is a thing which is collecting data from the pods, from the OpenShift nodes actually including the pods. So how are we as an overview could be like this? So heaps that reach the kublets, plus that information to the Hauklar, and that is written to the Cassandra. It also provides the Hauklar console, which users can later on use to see information over the web interface. Actually, that is a sub-web interface of OpenShift console. And when everything works, it shows you graphs like this. So I just wanted at the beginning to show you what is the end result, and now I'm going to go how we reach to this position. So at the time, if something has not changed recently, it is showing on the graphs memory, CPU, and network throughput for last hour, last 30 minutes, last four hours. It collects more information, and possible to parse it somehow over the API. But on the graphs, it shows something like this. All metrics components are delivered as pods. So Cassandra, Heapster, and Hauklar just pods. And these pods are released quite often with different versions or sub-versions of specific images. And the main so-called place where is specified the most of these configuration options is Mestrix.yaml on upstream. In upstream repository, you can find it. It is quite long file. I will point out a couple of values from it, which one needs to pay attention. So now I will tell you a bit more about all these metrics pods. So Cassandra pods, as I said, it uses this main data store for all data collected from the OpenShift pods. And by default, it uses persistent storage through. That means it will look for persistent volume on the available to use it for the Cassandra data. It's going to mount it to the root Cassandra data. And we recommend to use that because that ensures that you have a data persistence across Cassandra pod lifetime. It is possible to scale out more Cassandra pods. And one can say, start three Cassandra pods. I said, yeah, these pods are actually nodes. Cassandra nodes, that means when one starts more Cassandra pods, or when you have more Cassandra pods in the background, they form Cassandra clusters. So the technology you have, and which is related to the usual Cassandra clusters, can be deployed here as well. If one decide to use persistent storage false, then that means no data persistence. And if you are using older version of OpenShift or Kubernetes, then this option can lead that the varlib origin pods is and if it's, let's say, not separate partition, it can cause you troubles because it's going to fill that over time. And that problem is reported and already fixed upstream and downstream, so now it should not happen. But previously, that was, however, it was good to monitor varlib origin. But for the metrics, it would be good not to use persistent storage false. That's, however, our recommendations. So heaps are component, gathers metrics from OpenShift cluster. It gets metrics from every pod across all namespaces for what it needs, read writes to do that, and send these data to Haukula metrics over the API. That is from GitHub without any corrections. I cannot say it better. So how to set up OpenShift metrics configuration is quite easy. And in my work, I think starting the OpenShift metrics worked in almost all cases. I think I put here the link, not correct link. But anyway, configuration is quite easy. Just entered a couple of commands, which I intentionally left here because you can read that in documentation as well. And I don't think we have enough time. That's the first option. Second option is using Ansible Playbook or Advanced Installer. So what of these are documented in this link? We can add more Cassandra pods and more Haukula pods. At the time, I think only one heaps that is recommended and can run from cluster. And that is a particularly problem. OpenShift metrics supports dynamic provisioning of storage. That means if OpenShift cluster is configured to use cloud provider, no matter is it Amazon or we supported time, I think, for the cluster as well, it will automatically allocate storage back end and attach to the Cassandra. It is recommended to have dedicated pods, dedicated nodes for the OpenShift metrics. And the thing is these pods usually should end on the infra nodes for better planning and recommendations to put them on the infra nodes. But it can be also different depending on decisions. When OpenShift metrics pods are running, then in my test, I didn't notice that Cassandra or Haukula or heaps that are imposing on the OpenShift node when they run some excessively high load. And I noticed that, I mean, I proved that one set of metrics pods, that means one Cassandra, one heaps there, and one Haukula, are able to handle without any problems 10,000 pods. So this can serve as just guidelines for deployment. This worked fine. Also, so if it's necessary to monitor more than 10,000 pods, I would recommend to scale out the number of Haukulas and Cassandras. And also, for Cassandra, it is not a good player with NFS or network storage. That is not official. The official thing I can say not to run it in NFS because people run it in NFS, but it can lead to the problems with the money price. At the time, there is no limitation in the sense or because if there is a persistent volume available on OpenShift system, it is going to be used for metrics if we use persistence.js through. And no matter what is from there, that persistent volume is coming, so it can be also from NFS. Just heads up with high load NFS might not be the best option, but I know that I think quite a lot of people, some estimation says around 50% of users are using NFS as a main network storage, network files. OpenShift metrics was successfully installed and run on 210 and 981 nodes cluster. That means it works fine with such big clusters. It is able to collect the data from the pods scattered across all these nodes. And we have noticed that for 1,000 pods, Cassandra storage requirement across one day, during one day, is approximately 2.5 gigabytes. And if we keep default metrics duration at 7 days and it metrics resolution of 50 seconds, that would be safe to say to allocate for 1,000 pods around 20 gigabytes of data per week, approximately, or giving just there, for example, 10% buffering case something is not correct in this my calculation. For the 10,000 pods, it requires a bit more space. Everything this is for metrics duration 7 days and metrics resolution 50 seconds. You mean, what, can you repeat, please? Yeah, Hips that is reaching out to the OpenShift nodes and I think 50 seconds every 50 seconds is reached. And this is possible to change. I think so. And 50 seconds is, at the time, value there, but someone may need to increase that. This was everything up to 10,000 pods, 11,000 pods, or even 12,000 pods. But if we add more pods inside the OpenShift cluster, then at one point of time, it is going to cause the troubles that actually that point is around 12,000 pods. It could be even higher, a bit, that Hips will not properly or will not be able to handle all these pods and data collection will not work as expected. We have bugzilla for this problem and issues identified. We have some improvements which needs to be tested soon based on concurrency and in the Hips that itself. So that's still not the expected behavior. Hopefully in the future releases, we'll have better information. There is a recent project, a recent blog from the Hauklar team who specified this monitoring microservices on Housa, Hauklar OpenShift agent. And the idea is to rework the way how metrics is collected at the time. This is possible to test. If you go to this link, you will find instructions how to play with this and see what is new in the Open. What's going to be new in the OpenShift metrics?  I had 80 nodes. The only small trouble at the time is with a quite high number of pods. OK, that's all slides I have for this presentation. I will now stop and give you a time to ask a question if you have something. Yes, please. Yes, you said you can only have one Heapster pod for cluster? I think it is possible to start more. But then they are going to operate on the same bits. So you can't say I want this Heapster to talk to pods that are related with this label. I mean, I want this Heapster to talk to pods I have had such ideas, but also good in discussion with the guys who are developers for this metrics that actually one Heapster is the way we can do it now. And the Heapster is, at the time, or bottleneck in all this story. Hopefully. Yeah, go ahead. Heapster communicates with Kubelet, not anyone that she knows. Yeah. And the Kubelet is responsible for gathering the metrics? Heapster reached the Kubelet because there's C-advisor exposing. OK. So yeah. Yes, please. I cannot answer you. I didn't compare. The question was how OpenShift metrics compare to the Prometheus. I would like to do some testing regarding that, but I did not myself. Is that for me or for him? As far as I know, no. I mean, we didn't do a lot of testing on the NFS. So mostly because it's not recommended, but we will have to do it because there are many customers with large sends and the net apps. And they have really fast, really fast NFS back storage. And this could be an idea, but we don't have testing data for that. So just recommended is just to try and pull the images, either using an ansible role. Are there any more questions for either logging or for metrics? If not, we can always talk after. So I think we can just conclude. And thank you for attending. Thank you very much. Yes. Thanks, guys.


The Appalachian Advantage plan is the simpler, better way to get the phone you actually want instead of paying hundreds of dollars upfront.  It's called the Appalachian Advantage and is available at Appalachian Wireless. Kids Fishing Day will take place at Fishpond Lake on Saturday June 4th at 9 a.m. The event is free to attend and you do not need a fishing license to participate. Yeah, so Kids Fishing Day we do every year on a free fishing day in the state of Kentucky. So that means anybody that comes, they don't have to have a fishing license to attend. Every kid needs to be accompanied by at least one parent. We will have a Department of Fish and Wildlife there is bringing free gift bags for the kids that have fishing stuff in it like sinkers, bobbers, hooks. Lunch will be provided at the event by Community Agricultural National Enterprises. A bushcraft meetup will also be held during the event at which children can learn to make primitive poles and baits. There's a bushcrafters spring meetup every year at Fishpond Lake and what that is is a it's a group of people who are really into just outdoor activities and primitive camping. They teach classes on how to build your own fire and make your own shelter, how to forge your own knife. It's actually really cool but on Saturday they have two people there. One person is going to be teaching the kids how to make primitive fishing poles and and lures and there's another guy who is a competition kayak fisherman who's going to be talking about kind of what he does and how we got into competition kayak fishing and things like that to the kids. While the event is for kids 14 years and younger to come out and fish adults are also welcome to enjoy the day. Just any any age of the kids you know are welcome and adults also like on June the 4th is statewide free fishing they don't have to have a license anyone can come and and fish and enjoy their sails. Children participating in the kids fishing day must be accompanied by an adult. A fishing gift bag will be given out to the first 50 children to arrive. For more information about the event call<PHONE_NUMBER>. For Mountain Top News, I'm Kelsey Dean.


Hey Virgo welcome back to the channel. This is Kelly from House of Virgo If you're new please smash that like button and do subscribe if you're a returning subscriber Welcome welcome welcome. I am here every Sunday through Thursday and 30 p.m. Eastern time readings generally begin at 1025 Eastern they go live at 1030 so you can pop in the chat five minutes prior to the reading beginning To chat with everyone if you want to get a personal reading with me There's a link in the description box below where you can book a reading You can also join my only fan go ahead and click that link and that will take you to my socials and everywhere else You need to go can also become a member of my channel. So thank you for being here now Let's get on to your reading. Hey Virgos welcome back to the channel This is Kelly from House of Virgo. If you're new please smash that like button and do subscribe If you would like to get a personal reading there's a link in the description box below Where you can book your rest of October. I am running a special for personal reading 30-minute phone call special So do go over and check that out. You can also follow me on my other YouTube channel called moon pie tarot and I upload daily readings over there as well Alright you guys so keep in mind. This is going to be a general weekly reading Take what fits leave the rest. I hope you guys had a great weekend Hello, everyone in the chat. So let's take a look and see what the messages are for the week coming up for you Wait, let's get into this. Oh, how apropos for October the grim reaper The relationship is over. No second chances Grow and transform your life. Okay, so something Oh my gosh, like I have all these skulls on my table. By the way, I did switch my my format around a little bit Look at this is crazy. Like it's just so fitting for Halloween. Maybe somebody's dressing up like the the grim reaper For Halloween as well. Oh my gosh. Okay, something is ending here Let me see something or something has ended or something is over Wow, it's shock shocking sudden change shocking news surprise epiphany up evil Transformation, okay, so there's gonna be some news about a relationship ending Finding out about that this week coming up here and the dragon of fly now this fell out in the reverse Okay, I feel like This is being light-hearted. This is finding out Things coming to light. Okay So it did fall out in the reverse I feel like you may not know about this You may find out about it later Honestly, oh, I forgot to turn my little well, probably you won't be able to see it But I feel like you may find out about this later Or you might like you might find out about it Some other time I don't know if I mean, I don't feel like this has to do with you I feel like you're gonna find out about a relationship coming to an end with someone that you know Someone that you're connected with or you're connected to Someone that may be there because obviously I mean if you're in a relationship with someone You're gonna know if the relationship is coming to an end or it's ending. Yeah, I kind of feel like You're gonna find out about something like this something this week That's coming to an end. Okay. Let me see. What else is going on? So that's really interesting information It's gonna surprise you that's for sure Some of you might find out about it and you might just be like holy. Oh my god I can't believe it Okay All right, I feel like some of you might kind of already know that this is happening, right?  They're making a mistake. Look at the doggy.  You're about to step off that cliff. You don't know what you're doing Look at you.   It's like that that kind of energy or you hear about it or you do find out and it's like It's shocking, but at the same time it's not So it'd be a connection to an Aquarius that could be an Aquarius involved in this Okay, so yeah, that's the energy I'm picking up this week as far as this situation going Now it doesn't have to just be about a breakup It could be something to do with someone at your career like at your work where Somebody gets hired and some stuff's going on.  They wind up like Like getting fired or something like that. You know what I mean? It's like some type of grim reaper energy Connected to you or in your circle in some way, you know, somebody trips up or they make a mistake Yeah, this is big time. Let me see what else is going on here. Wow. Yeah the moon Shady shady shady behavior some kind of shady behavior. I'm not centered. I'm not centered This is like my OCD is just not having it. So yeah, there's some kind of shady behavior with this moon Definitely Pisces energy here.  I knew it. I Knew something was gonna come out eventually. It's like Virgo. You're cut this week.  I can sense you sense it It's something's coming to an end or that there's like two people That something's not working out or you're finding out about someone and they're not gonna be able to Maintain some kind of facade.  They slip up something happens. It's just a matter of time so somebody It makes a big mistake the trip up They get caught they get found out. There's a huge Are you find out? There's a huge fallout that happens this week as far as Like a breakup or separation or something happens Okay, let's see what else is going on Wow Pizza Cups. Oh, you're getting apology from someone this week as well. Yes Oh Virgo, mm-hmm. There's some of you. It's from a water sign. Oh Yes, mm-hmm Here in an apology Well, there's that Aquarius card again. I'm just this right here I'm not kidding is some strong Pisces Aquarius cusp energy. Now, I know it's not gonna resonate for all of you So don't force it to fit. It doesn't mean it's not your reading. It could be anybody It's definitely an apology coming to you from somebody, but if it like is more specifically for some of you It's either a water or Aquarius So cancer Scorpio Pisces or an Aquarius or Pisces Aquarius cusp, right? But either way, there's definitely like I'm sorry. I want to fix this. Can I make it up to you? Can I heal this? Can we fix this thing? Can we renew it? Can we try to you know? Heal this thing. Yeah, definitely Oh my goodness.  It's definitely gonna be shock. I don't feel like I Just don't feel like you're gonna expect this coming in Okay, the Empress Think this person is like really gaming a little bit here not in a negative way, but I just feel like This oh gosh. Okay. They just said this person is is like All right, I'm just gonna have to put it listen.  They actually what they were the one that started it and you know They got to come in and confess and they got to apologize That's kind of what I feel like is going on here and like can you please like don't don't put me time out It's kind of like what I feel this person is coming back like it looks like kind of coming to you and Just being in that energy like you know, yeah, I got in a fight Doesn't actually have to be a fight But it is kind of like it's symbolic of like I got into some trouble like a bit out there and yeah I got into some trouble You know, it's not really someone who's coming in and being like You know, they started it and you know, I got dragged into this and then another I feel like they're kind of tight Taking like accountability like actually I was stupid, you know, I made a mistake here and that's yeah, there's judgment Yeah, so that's the energy here like taking accountability. I know I made a mistake. This is like coming clean Yeah, okay, so You know, if whether you want that or not, I mean, I just don't know Virgo, but I do feel like You know, why is this person doing this? Like why are they coming in with this energy towards Virgo? Like why what do they have what what is the? Because you know, even if this is like a work thing if you're their boss or like you're significant to this person for some reason Are they gonna lose something here? Like why is this person coming to Virgo? What's the significance? Why is this so important? What is the relation? Why is this person? Coming to Virgo with like this apology Wow, it is sorts. Oh my god.  He's so loud So this person you like this person really Okay, so they were all up in their ego I will tell you that like I don't feel there's any kind of Game-playing or manipulation or anything like that going on with this person This person was fear-based and ego-based In the connection with you, okay, that's that's mostly what I feel like even with the eight of swords I'm sorry six swords in the reverse feel like this person just Was stuck there's a lot of stuck energy a lot of pushback energy and They did not it feels like they just could not come out of their comfort zone. They just would not make change They weren't ready to transition. They just wanted to stay where they were and receive all kinds of attention and make all the You know receive all the rewards all the recognition, you know, even though they had this Partnership or this connection or this contract or this, you know, this union here with you I mean whatever it was like there was some kind of deal There was something there was some kind of connection between the two of you whether it was like a work partnership or romantic You know your colleagues your friends whatever it might have been there was some kind of connection between the two of you that was quite significant and Still is like it's valuable And for some reason I feel like this person Somehow felt like maybe there was something Better for them Or something that Could be more transforming I just kind of feel like you know that grim reaper vibe that really gives me this energy of like this person kind of Thought that they were looking at their future and it was like They saw like dollar signs or they saw like They were attracted to the glitter That's what I'm hearing like they were attracted to the glitz and the glam But what they really like after all that faded They actually met the grim reaper and What and like I always like I can't help myself, but The grim reaper is like pig on a lips lipstick on a pig right the grim reaper is like the ghost of The future right In that Christmas Carol story right he comes in and he shows you what your future like in that story That's what the grim reaper it you know, he just is like look this is what's If you don't straighten your shit out I feel like that's what they that's what they kind of like faced death in some way You know, um, I Just feel like this person realizes or your person like in whatever kind of connection you have It feels like a real wake-up call. That's what judgment is You know, and they're kind of like looking back on their past In a lot of ways here with judgment is judgment is like when we kind of Take a look at our past and we look at our mistakes and we kind of like get clarity and Realize how you know, we were Self like kind of putting ourselves in our own prison Making ourselves like it was no one else to blame. We were doing it to ourselves You know that self-imposed prisoner energy We were trapping ourselves in our own mental beliefs in our own Mental prison and we were defensive and pushing back all the time. We weren't willing to change Make the necessary transition to change We just wanted all the appra all the appra the praise and we just wanted people we were seeking approval We were seeking recognition. We just wanted to be the hero, but deep inside. We were really like Not we at not very good. I'm saying like this person. This person was just really egotistical like just this is what was important to them Seeking everything else all the validation But you know like pretty much a mess inside. This is what I feel like this person is kind of coming to themselves like this just this They weren't authentic. That's what I feel like is going on here. This person was not authentic So they went out there into the world doing whatever they were doing and they pretty much got Like a lightning bolt to the head like right to the crown chakra right to the ego, you know and whatever some kind of way and Not it got like they got knocked back Whether that was in another relationship with someone or that was just whatever they were trying to do They got knocked back like pretty badly and It kind of woke them up. That's what I feel like and Like so this week I feel like you find out about it in some way and some of you are just kind of like yeah I just figured I knew I tried to warn them You know Why would you give everything up? Why would you give up your family? You're this great job?   There's a lot of that energy playing out here So I feel like for you Virgo This is a little bit of an I told you so kind of reading because I just kind of get a sense that You already know you already knew it's it was just really for you. It's just like okay when is When was it it was just a matter of when not if and some of you might have been kind of like well Maybe it's never gonna happen, but that's just kind of you losing your patience or losing your faith, you know It's a little bit of a karmic retribution. That's kind of what I feel. It's not revenge or anything sometimes I feel like you might have been in this place where it's like I Went you know, you want to teach them a lesson, but that never works You're not bigger than karma people people don't people learn their lessons by Life teaches people Life lessons are taught best Not lessons taught by other people. Don't you agree? I believe that I get this really cool Thing kind of deck. I found this at a good wheel store So I thought let me try this. Let me just check this out. Look. It was two dollars and 49 cents That's cheap for a deck of cards and I said, okay, I'm gonna check it out So I'm just gonna pop this open Little haul here and try it out when you guys I'm just gonna pull a few carts See what comes up now it is Signs and sky mates. It's called it just has like Astrological compatibility deck. I have no idea If it's even sold anymore, I was just bruising the goodwill, you know, like we do as Virgos We're always looking for a bargain just checking things out and I saw this sounds like cool All right, maybe I'd like to say I found this but I maybe it found me All right, let me take a look and just see What it tells me I want to ask to see what signs What sign this might be this person? Well, let's take a look and see what it says Excuse me coughing a lot lately. I think my asthma has really been Well, okay, let's see. Oh We got a Libra here Okay, can you tell me another sign that this person might be? Virgo, who is Virgo dealing with? Okay, I've got Aries Leo Cancer Leo, that's the compatibility part Definitely double Leo there. Let's see who else Cancer Pisces It's interesting Scorpio and Sagittarius. Okay, that's interesting. So it is just a card of Yeah, it gives you all the planets. We've got Gemini showing up here That doesn't surprise you, right? I feel like Gemini is always in the readings And I'll get one more Let's see Somebody's definitely Fafo definitely Fafo, okay, and Pisces, okay All right, let's move on. Let's move on. Let's move on. Let's check the Chinese signs and see What Chinese signs? Who is this person? Chinese signs. This could be you. This could be them. It could be both of you or none. Okay You're starting with the year of the ox, okay Year of the tiger Year of the rooster Year of the goat Year of the rabbit Monkey Rat And a pig. Okay. Yes. Yes All right, what other messages are there for Virgo? Other messages for Virgo. What else does Virgo need to know here? Oh Okay, you've got reflect Virgo. It says go into your cave for a while You need to regroup and you need to recollect a lot is going on and you need some alone time Stay in the stillness in the quiet But don't get lost tell people you're taking a vacation and pull out for a little while You need a recharge Virgo. That's your card. Anyway, that is the hermit Okay You've got putting in some effort So I feel like this is about putting in some effort to just kind of go in especially when you find out about this When this comes to light this week you're gonna really need to like kind of and Especially if you get an apology or you hear from this person coming in this week To talk about what happened Because you might be They want to talk to you about this situation And you might not want to talk to them or it might be like oh here we go that kind of thing and You might want to stop you might want to pause to try to figure out how you can handle this or what you're gonna say So put in the put in some serious effort to think about how you're gonna respond Rather than react Okay, to this because this is gonna need some careful attention from you Especially if they're coming in to apologize or if They get in some trouble and now they're turning around and coming your way with some sweet text messages and stuff like that Yeah, you got spying and deception here You may need to find out because there might be a little bit of like hmm. What's going on? Why are you reaching out to me?   So that's what I feel like, you know, you need to reflect on this Kind of thing here. Okay. How's the logical male? That's you male or female energy like you need to be very logical about this situation not not have an emotional reaction to it you need to If you're going to respond to be very logical and objective and reason this thing out here Okay, and Then you've got take a risk what you've got to lose starting over can be tough But it's time to spread your wings don't overthink it. Okay. So that's what the hermit does, right? the hermit goes within and Spends a specific amount of time to reason things out the hermit does not Isolate too long. Okay Does not shut himself him or herself off from the world forever The hermit does come back out eventually And the hermit just goes in long enough just to reflect and ponder on The hermit just takes a time out. That's all Hermit puts Themselves in a time out For as much time as needed to sort through a Dilemma right so I feel like For some of you the question is Do you want to start over with this person if they've learned the lesson? And and the question is you do you want to find out if they've like did they go out and learn their lesson? You got to find out if they're really coming in because they've like Okay, now they've learned their lesson and now we're ready to talk and now we're gonna figure this out And you got to find out if that's what they're doing or they did they you know, did they? Or they just coming back in because they got nowhere else to go. You know what I mean that kind of thing So I feel like that's Don't overthink it. You just need to Try to kind of find out and ask questions really if you're gonna talk to them like ask questions Like well, what did you learn?  So go play.  You're going to the The Lotto machine and get something you got the Wheel of Fortune here I'm telling you that's the material gift. That's a little bit of money coming in Hell anybody could you look listen?  All right, travel time I Love the Ralph is like ooh scrabble time because that's what I'm calling it now. It's too funny. Oh look We've got an Aries here. All right, so there might be a connection to an Aries Let's roll that again, and there's Libra So we did see a Libra And then all Aquarius Okay All right, let's get some numbers here and see what's going on so someone is 45 All right, we've got the number 87. We've got 71 and We've got 50 Okay, hmm September 19th September 12th significant Nope, that just moved. Okay, September 2nd significant Okay All right, give me a second here y'all, somebody's got jury duty coming up this week Sorry for you, although I know some of you do like jury that go on to be in the midnight. You know what? I don't blame you I Would not want to sit in jury duty though. I just just show me the crime documentary. I don't want to sit in jury duty all day All right Not much going on here, so I'm just gonna give you We have the initial F as in Frank.  Yeah, if were you in the area, did you live in an area where you could see a solar to clip the solar eclipse? It's like in reverse. I guess the solar eclipse Super super cool. I Did not see it. I don't live in an area where it is, but did see videos of it. Okay, so Here we go This is a very straightforward message for me that they're giving me and I'm just gonna show this to you and it's just one charm So we're gonna keep this very simple right now All right, you guys So you do have what I what I believe is not sure if it's an octopus or a squid I Can't really see it all that well my glasses and then I have to switch glasses. Yes, I Think it might be oh Yeah, looks like um, well, I guess it's an octopus Whatever it is the point is it's got tentacles so I have to tell you the message that I was getting about this thing is I Hate saying it but it's true There's an energy around you. There's an energy Like some it's somebody or some type of energy. That's like pulling at you this week You might feel that like you feel like you've got Cliche to say but tentacles on you like somebody Had or wants to get their tentacles all over you. They're saying there's like this pulling energy It's almost like But it's not bad. I feel like somebody just is trying to keep it hold you back. It doesn't want you to go It feels like no, don't do it. I'm hearing don't go. Don't do it. Don't go Don't do it. Don't go. That's what I they keep saying. Don't do it. Don't go Come here. I want to hold you. I want to hug you. Just stay with me. That's what I hear Yeah, so I don't know like if it's an energy if it's a person Yeah, I'm feeling that real strong I'm not exactly sure if it's this person or if it's other people who are trying to tell you Don't Don't make a mistake or don't you know Or if it's this person trying to say don't leave me. I'm not sure what's going on here But yeah, that's the strongest message that I feel With that octopus. I don't know if squid if squid do that If they do squid have tadpoles, I have no idea. I know people eat them. So Yeah, I don't know some of you might just be having some octopus Or octopi octopus for dinner or having some or enjoy that maybe that's a favorite food that you like to eat Not me but that's okay. So yeah, I'm gonna leave that reading there for you. I hope that that message Gives you a little enlightenment and I hope you enjoyed that again if you want to get a personal reading There's a link in the description box below. I'm gonna try to do the members only Q&A this week I will upload The live stream for when you guys will be able to join in on that Yeah, and so I will see you all back here tomorrow All right. Have a great night. Have a great week going forward And I will see you all very shortly


Hi this is Ella from When I Work. Today I'm going to show managers about shift acknowledgement on a computer. Shift acknowledgement ensures that your employees have confirmed they've seen their shifts. This is an optional feature, so to use it let's make sure that it's enabled in your account. Hover over your name and select app settings. Then click the scheduling tab and scroll until you see require shift acknowledgement. Enable this feature then hit save. After hitting save, the next time your employees log into their When I Work account on the web or on mobile, they'll see a pop-up asking them to confirm that they've seen all of their upcoming shifts in the next two weeks. Now as a manager, let's see what happens when your employees acknowledge their shifts. Click the scheduler tab to view your schedule. As soon as employees begin to acknowledge their shifts for the week, you'll see a green thumbs up to the left of their name. This means that they have seen their shifts for the specific week. If you don't see an icon, then certain staff members haven't seen their shifts yet. If they only have acknowledged some shifts, you'll see a yellow icon. You can hover over this icon to see which shifts they haven't yet acknowledged. This usually happens if a shift was added to their schedule after they've acknowledged their first round of shifts. Use shift acknowledgement to better communicate shifts with your employees and ensure employee accountability.