The next, next to the stage I'd like to welcome, Naraaza Muhammad Noor. Naraaza's from the Anu College of Asia and Pacific and the title of her talk tonight is Challenging the Voya Gods of Urban Spaces. Let me tell you about a street in my hometown. Jalan Ampang is one of the oldest and longest streets in Kuala Lumpur. The part within the city center is three kilometers long and there are three distinct zones which you can't see when you Google it. The part nearest to the central business district is busy during the day but dies down after office hours. Further down, it's quiet, always quiet because there's lots of abandoned buildings. At the farthest end near the old town area, it's a bit more lively, especially at night. The way we inhabit our cities creates ambience that's invisible to maps. But surprisingly, Kuala Lumpur residents don't see it either because we move around the city in our cars. From a car, it just seems boring and constantly crowded. You might expect one group of people to be able to see this ambience, architects, but my research shows that they might be as blind as the rest of us. My research involves walking with architects and people of the city to understand how they see and experience the ambience. This is important because architects have a responsibility to the street. The street is for the people, but architects at like their building facade is where their responsibility ends. I think one reason for this is in their training, specifically how they are taught to draw and think. The architect's plan is a God's eye perspective where people are mere decoration. I think architects should think more like anthropologists than voyeur gods. When I walk with architects, I watch how they move through space. I listen to how they describe their experience. To them, people are lateral disturbances or soft elements. My research will help architects see themselves more clearly, but more importantly, this can help us make better cities. Architects educated more like anthropologists might pay more attention to how their building affect the street. No longer voyeur gods, they draw people into their plan, not as decoration, but as information. So in the future, when I walk down Jalan Ampang, I might not feel sandwiched between concrete and traffic on a very narrow pavement. I might enjoy the street at night with those abandoned buildings, lovingly rejuvenated by architects who see all the dimensions in the city. Thank you.


Hi everybody, and my name is Raaz, this is Natasha. We are both from there, and here it's in the front of your eye. And we love GPS. Now, before talking about GPS, please raise your hand if you use GPS. Okay, now keep your hands raised, please. If you have been done. Great, so I see that you guys came down, and you are still up. So, trust me, once I always had a feeling that we had enough to do the sooner or later, you might have a feeling that you are not doing it. Okay, so just so we're on the same page here. GPUs are highly optimized for deep learning processes. They can dramatically speed up the computer processes. So, that's why our people choose to do their deep learning and AI model on GPUs. So, we want more and more GPUs because we want to do more AI. But GPUs are so expensive, we can't just keep buying more and more. So, that's why we need to try to bring the most out of our existing GPUs. And that's a big challenge. So, in this talk, we're going to talk about GPU utilization and how it's often known, how we can detect it. And we're going to show you some tools that are built that can help you in this situation. So, now, Raz is going to talk a little bit more about it. So, it's true that when you use GPS, it's not that it's only doing it as a way to do it. But it's really treating it as only one aspect, a bigger problem. And the problem is how to efficiently manage and provision GPU licenses. Now, in the road, GPU provisioning is not a term that we deserve to be able to exist. But the truth is that if you use programs, it does that for you. And we will see what programs does and also what it does. But first, let's try to understand why GPU provisioning is not an easy task. And we'll do this by example. So, let's say that you are a manager of a team that uses GPUs. Now, this is true both for small teams of older team members and for large organizations in hundreds or thousands of teams. In our example, we have 14. And we have 4GPUs. So, each one of the members has its own GPU. Alright, sorry. Can I hand you a mic? Your attached one doesn't seem to be working. So, people can't hear you online. Thank you. So, should I go back? Can you hear me now? Great. Alright. So, let's say we have 14 members and we have 4GPUs. Now, each of the members has his own GPU. And then comes a new member. Now, you as the manager probably think that you need more GPUs. And let's go through all the things that are happening while you are out there on the journey to getting these more GPUs that you need. So, first of all, you are sure that you are letting the new member down because you remembered only last minute that you need these GPUs. You start fighting the IT department to get these more GPUs, even though you already have a few. You have to explain your boss why the new project is being delayed. And all this time, the new member is waiting for her GPU resources. Now, trust me, this is how I look when I wait for my resources. And the truth is that you don't always need more GPUs. When the new team member comes, instead of getting more GPUs, you just... I'm sorry. So, the truth is that you don't always need more GPUs. And we will see why. So, first of all, not all the tasks require all the resources that a user gets. Some tasks are just smaller. Now, secondly, some of the work is being down out of the GPU, leaving the GPU unused. For example, writing code or building the model. Also, some GPU workloads have heavy CPU and IO tasks. In addition to that, people take breaks.  During this time, the resources could be unused almost entirely. And generally speaking, not all the resources are being properly used. So, instead... When the new team member comes, instead of getting more GPUs, what you want is just to use these idle GPU resources. Now, until now, I was talking about a team of people sharing GPUs. But this is exactly the same case for a Kubernetes cluster running different AI workloads. Because very similarly to people, different AI workloads are not identical. They have different loads and different resource consumption. Now, it took us quite some time to fully understand why it's so hard to fully utilize a GPU. But I think that now we can say that there are two fundamental reasons for it. The first one is that many GPU usage patterns are underutilized by definition. For example, when you do remote debugging with VS Code or PyCharm, when you write code interactively with a Jupyter notebook or just a Python shell, when you have idle AI workloads just waiting for new requests, and when you have just not-so-heavy tasks like small models or small batch sizes. The second fundamental reason is that GPUs are being provisioned statically and coarse-grained. This means that most of the time you will see a GPU being provisioned for either a user or an AI workload. In order to overcome these challenges, GPUs must be provisioned in a smarter way. They have to be provisioned dynamically. And this means not a GPU per user or per single AI workload. Additionally, they should be provisioned in a finer grain, for example, fractions of GPUs. And they also be overprovisioned. This means that you should run more workloads than the available GPUs, very similarly to other resources like CPU memory and storage devices. Now, unfortunately for us, Kubernetes does not support overprovisioning of GPUs. It also does not support fractional provisioning of GPUs, but luckily for us, it does provision our GPUs in a dynamic way. Now, if you're not using Kubernetes just yet, for example, if you're running on the host machine directly or over SSH, we created the right tool for you. We call it GNV, and it stands for GPU Environment Management. It's 100% free and open source. It is highly inspired by existing environment management software, like PyAnv, Konda, and others. And it works in a very similar way. And it helps people to provision their GPUs in a smarter way. Now, GNV is a terminal-based tool just like PyAnv and Konda, but we did create integration to common IDEs. We have an extension for VS Code. We have an extension for JupyterLab, and we are working on creating a plugin for Python as well so that every GPU user can use their GPUs in a smarter way, natively, in his or her working environment. GNV and all these extensions are open source and are available on our GitHub page, and I recommend you to go check them out and see if it can be useful for your team. Now, going back to Kubernetes, we said that Kubernetes does provision GPUs in a dynamic way, but the problem is that it does it in a pod granularity. This means that most of the time you will see one GPU provisioned for one pod. And when this pod gets scheduled, a GPU is assigned to it. And as long as the pod is alive, this GPU cannot be reassigned to other pods, even if the GPU is idle and not used because any of the reasons that we saw earlier. Now this must be changed if we want to fully utilize GPUs. Furthermore, even with the smartest provisioning mechanism, the provisioning, smart provisioning is not enough because GPU resources should not be an afterthought. The point in time in which we deploy an AI workload to production cannot be the first time that we ask ourselves how many GPU resources do we need? The GPU resource requirements should be properly defined in the development phase of the project, just like any other project dependency, for example, Python packages or a Docker base image. And I truly believe that they should be specified as infrastructure, as code. Now, if you use GN for your project, it allows you to save the resource requirements as code as part of your Git repository. Okay, so the truth is that there is no magic button for these problems. We have to change our mindset, and now after we have a better understanding of it, I'm going to pass it to Natasha and she will continue with the second part. Thank you, Rod. Okay, so now that we understand the problem, our responsibilities and the power we have in our hands. But where does it lead us? What can we actually do about it? Well, you cannot improve what you don't know. So the first thing we need to do is to get a better view and to better understand the GPU utilization in our cluster. So we want to know if we have some idle GPUs, see if they have pending workloads. So let's drill down and see. Are you really out of GPUs? Now, going back to the case of AI workloads in our Kubernetes cluster, we know that different workloads have different loads and different resource consumption. But how can we actually measure it in reality? The guess is right, it's GPU utilization. So this is the official definition of GPU utilization, but the factor just means how much the GPU will be used. So if utilization is low or even zero, it means the GPU is not being fully used. So in order to detect low utilization in a cluster, we want to detect all theizations that... Let's see how we can do it. So as I said, if you have a cluster with server GPUs but you have not installed Kubernetes yet and you're considering it, you can use our job, which is my AI open source top-like tool for monitoring GPUs in cluster over SSH. We don't have time to look at it today, but it's really cool so you can take a look at it. But for our test case now, we do have a Kubernetes cluster. So let's say we have two nodes for GPUs each, so let's get a better look of what's going on on those GPUs. So this is the dashboard we built. We're going to go over everything here in just a few minutes, but what is the first thing that I want you to do? So we have eight allocated GPUs out of eight total right now. And what do we see on the right? The problem. We said before we have a problem, now we can actually see it. We have two pending pods that request the GPU and they're not allocated due to lack of resources. But at the same time, we have two GPUs that are allocated by pods that are not using them. And they are AI. So this is a big problem. Now I want us to go a little bit back and I want to explain how I built this dashboard. And then we'll go back. So to identify our GPUs, we need to know the GPU utilization of each GPU at a certain moment. So if the GPU utilization is zero and there is a running pod on the GPU, it will be idle. It depends on the GPU that you have in the cluster, you're paying for it, and you can't even use it. So we need some sort of component to export this data to us. Those that can help us with this. So we have a video CCGM exporter. It's a demon cell that exports metrics about GPUs and the loop. If you're working with GPUs and credits, you probably already have it installed as part of Convigate's GPU operations. We also have Prometheus, which I'm sure you've all heard about. It's a monitoring system that collects and stores metrics. We also have Varfana, which comes both in Prometheus. And it allows us to build the dashboards like the one we saw earlier. So they're all easy to install and easy to use. They have a pretty UI and everything. I installed them on my command cluster, and I opened the Prometheus UI. Now take a deep breath, we're going to deep dive into some topology for a few minutes. So this is a screenshot from Prometheus UI, just draw metrics without the font. This is the metric we are interested in. DCGM FIDET GPU until. It comes from the DCGM exporter. Each record here stands for a different GPU, and the value is the utilization. It's a percentage, so the value will be between 0 and 100. And the labels of the metrics give us more information about each GPU, so we're able to identify it. So for example, we have Kubernetes node of the GPU. We also have its UUID and GPU index on the node. And there's a pod running on the GPU that will have the pod name and name space. And if there's no pod running on the GPU, those labels will just be empty. Now, out of all the records here, we are interested in the ones that actually have a pod running on them. So we can use from QL, which is Prometheus Query Language, to perform some queries on the metrics. So this is how we filter on the desired results. Since we use a non-empty pod label, and those records here stand for the GPUs with a pod running on them. And also, we would like to filter on the idle allocated GPUs. So we want the utilization to be 0, which means the metrics value equals 0. So this is also from QL. And there you have it. Those are the idle allocated GPUs. They have a pod running on them, and the utilization is 0. So I used some more from QL to aggregate all this data on this metric and other metrics as well. And then I put it in this Grafana dashboard. It's also open source, of course. You can take a look at it later to see if Prometheus Query is there and how it would be in Grafana. So now that we understand where this data is coming from, we can analyze what we see. Here we have the amounts of idle allocated GPUs and below the pods that are running on those GPUs. And here we have requested GPUs. So we can also know how many GPUs each pod is requesting or has idle. Now, if each user or team will use a different game space, which is a good nice practice, then we will know the owner of those idle GPU pods. And we can kindly contact them and ask them to delete their pods or maybe delete them by ourselves if we have the permissions for it. And we can also know the owners, the users who submitted the pending pods so we may know who's going to be angry with us about their not running the pod pods that we want to execute it. We have 60 GPUs allocated out of each total. Now, some organizations might not even know this information that they have available GPUs. They're not idle allocated because there's no pod running on them. They're just available. So it's an easier problem. And I know it sounds funny, but you should just execute more workflows. Use your available GPUs. Pay good money for that. A good visualization of the GPU civilization. So here we have the GPU civilization over time. Each line stands for different GPUs. And you can adjust that time span on top of the good time page. So it can be like the last five minutes or the last 24 hours, whatever you want. GPU utilization out of all GPUs in the cluster. I'll look at the GPU with this civilization. So now that we have all this data, we understand the GPU utilization and can see it and visualize it. What can we actually do about it? Well, the first thing and the easy thing and we said that already, we can delete the idle GPU pods. So either the user or the applicant will do that if they both have access to the dashboard. Another thing we can try to do is to avoid the idle GPU pods in the first place. So for example, when you run a process inside a Jupyter notebook and you press the little right button and then the process finishes and the notebook stays open and allocates in another GPU. So instead, you can finish working on your model inside a Jupyter notebook. When you want to execute it long-term, you can run it as a Kubernetes job. That way, when the process inside finishes, the pod and the job will terminate. And the GPU will be available for other pods. And that way, depending on the problems so earlier for example, can start running and this is our goal. So let's analyze. We understood the problem and presented some easy steps we can do to improve the situation. But it takes more. You need to look behind the scenes. Maybe think of a better work process. One that suits you and your organization. We have all this data presented in the dashboard that can help you get a better understanding of the GPU utilization. So you can use the dashboard, deploy it, change it, add more information to it, adapt it to your needs, maybe use profana alerts. The power is in your hands. After you do all that, you can continue to the next steps, which is for example, smart provisioning, smart scheduling, increasing visualization of existing models. But we don't have time to talk about this. Thank you everybody for joining us today. As we said during the presentation, everything we mentioned, every tool is open source and available on our Github page. We'd love to hear more from you about the way that you use GPUs and the problems that you encounter. So here are our email addresses. Feel free to contact us. We will also be at the RANA AI booth, booth number S63. So just walk by in the next couple of days and we'd like to hear from you. Thank you. Thank you, Raz and Natasha. Excellent talk. Yeah, I'll open up to questions. Oh, there are loads. I was actually going to ask one, but I better let the audience do them first. You go first. I just had a question about it strikes me that if you're going to do it dynamically, do you need to know the length of time somebody's going to use the pod in order to so that you're not switching them, right? Or if it's idle and then that person who owns it, the way you've described it is very like you take it back. So now that first person no longer has access to a GPU. I just wondered if you could elaborate on that. Yeah, so that's a great question. Yeah, I can try to answer. The pod is idle right now. Don't hurry up and delete it right now. Models have some idle time. Then you go back to being not idle. But if you see, like, a pod being idle for a day, for a long time, and maybe you don't have to know that this person went on vacation, you just see it's idle on the dashboard, then you can delete it. Yeah, the person will lose access to the GPU, but, well, they shouldn't, in the first place, execute workloads that have idle GPUs for 24 hours. So it's like, our approach is like there are problems. If I may add, so what you suggested is a great idea. It's a bit too simple and naive. I mean, it would be beneficial. You can specify the amount of time that you think that a pod is going to run so that the orchestration system, as a hint to the orchestration system, but this is too simple when you're talking about big clusters with many different AI workloads. One to have is a system that would automatically detect idle resources and idle pods and will take these resources and bring them automatically to other pods. So this would be a great idea, but it's too naive. I'm sorry, it might be too naive and we're looking for more advanced ways to even improve it without specifying hints primarily. What we've had with using the GCGM metrics is that utilization is actually defined as if you've done one CUDA operation in the last one second. So you could do a CUDA mem copy and you get utilization. I'm kind of curious if you've been able to develop anything else deeper to actually understand how many CUDA cores are really left and how much you can push the GPU more. Yeah, so that's a great question because the GPU utilization might be misleading, because as you said it might be also too naive because if you do something very small on the GPU every sampling window, then it might seem that you have high GPU utilization. Now this is an inherited problem with looking at GPU utilization. There might be other metrics that could be created, for example to understand the size of the CUDA cores that are running. There are technologies of NVIDIA for example MPS that allows you to have a bit more understanding about them and you have also profiling tools from NVIDIA that will help you to further understand how much you actually do use your GPU. Now GPU utilization is just a very basic metric but it gives you at least 80% into the direction that you're heading. So I'm curious what you see as the future of the roadmap of time slicing or context switching of GPUs. That does seem like the holy grail I'm a system operator not a data engineer so what I see as a system operator is why aren't we treating our GPUs like CPUs with multi-threading and what do you guys see as the future of this? Is this like around the corner? Is it decades away? Where is the technology going? This is another great question. I think the way we see it is that we believe that we can have a system that automatically detects the idleness and the resource consumption in real time and then just like any other more mature resources that have been here for decades just like that we can develop technology that would treat GPU resources in a very similar way in which you just submit your workloads and the system underneath realize, understand in real time how much resources every workload needs and if a workload can give it up temporarily if it gives it up intentionally so for this temporary period of time we can give these resources to another workload this is the Holy Grail and the Holy Grail is to make GPUs very similar to CPUs because nowadays when you submit CPU workloads you don't really care about what CPU core is going to run on the operating system does it for you so we believe that you can achieve the exact same thing that GPUs are going to use and no, it's not decades away it's here and it's going to happen real soon I believe so earlier you had described a little bit about looking at batch workloads and jobs in order to be able to relieve the pressure on the GPUs it's the primary spot that you're looking for to try and relieve this GPU pressure like looking at being able how do we move some of what we like what does that continuum look like I'm not sure I understand the question I think I understand if you're working interactively in the GPU then you do need the notebook and you're doing interactive work on it but I meant like if you're debugging or building a model and then you finished it and you want to train it for a long time that's where it's better that you switch to a job and not a notebook what is this from does that answer your question? I think that's part of it even during that debugging cycle I have a day scientist working for 8 hours a day they take that break but they're still interactively looking at that to me even as you described in the presentation that's a period of time they went for lunch for an hour and a half I could use that GPU somewhere else but because it's nominally interactive job that's still holding that GPU can I relate to that? this is a really great question here we presented some basic stuff that people can do nowadays but if we're talking more advanced stuff then your point is really correct because we believe that you can have a system that detects this GPU idleness and when you decouple CPU resources from GPU resources then you can leave the Jupyter notebook running the user can still use the GPU notebook for visualizing the data for writing code interactively but as long as nothing is running on the GPU our system underneath can take the GPU resources to another pod that is sharing in a controlled way by this system and it's sharing the GPU resources so you do the holy grail is that you will not have to stop your job entirely you will just have a system that manages it underneath does that answer? alright did you get a chance to compare schedulers like MISOS versus Kubernetes MISOS has a good scheduler called dominant resource fairness and it has a very decent scheduling algorithm and you get a chance to compare and so that we can you can see which is a better scheduler for resources like GPU alright so yeah we believe that we understand AI workloads in a very intimate way because we go through all the stack we go through the Kubernetes layer and the Linux layer and the GPU CUDA layers themselves as well as the applications and frameworks like TensorFlow and PyTorch so we believe that we can create a Kubernetes scheduler that is fully dedicated to AI workloads and we believe that we can build the software that is needed in all these layers instead of relying on others to implement them for us so I'm not familiar specifically with the scheduler that you mentioned but we have a lot of thoughts around existing schedulers and we do have our distinguishments between them and I'm sure we can talk about it later cool time for one more question if anyone has one have you found that the provisioning and metrics differ by workload type a lot of what you've explained is really good for batch processing and there are a lot of workloads on the same cluster that want to hold on to a GPU longer and that might be appropriate for that type of workload yeah exactly I mean there are many types of workloads for example interactively building a model training the model and inference servers as well and inference servers can be idle as well because they are waiting for new requests to arrive until these requests arrive you can take these GPU resources to another workload so this is true also for inference clusters as well great thank you again let's have another round of applause for Raz and Natasha


TESOL PARIS Paris, France is an incredible destination for a TESOL course. What can be said about Paris that you don't already know? Paris assaults the senses, demanding to be seen, heard, touched, tasted and smelt. From romance along the St. to Landscapes on bus-sized canvases to lounging and cafes discussing the finer points on the use of garlic, Paris is the essence of all things French. TESOL course trainees can gaze longingly at its breezy boulevards, impressive monuments, great works of art and magic lights. Savor its gourmet pastiche of cheese, chocolate, wine and seafood. Feel the wind in your face as you roll or play through the steel or a sound can appear in pleasure at the top the Eiffel Tower. Paris is a city to discover before, during and after your TESOL course. So see the sights, visit the museums, they're part of the experience. But then jump on the metro or a bus and get off at a place you've never heard of, wander through a quartier where French mixes with Arabic or Vietnamese, poke your head into mysterious shops, have lunch in a local restaurant or just perch on a cafe terrace with a vign blanc and let yourself fall in love with your very own Paris. Blessed with enchanting and beautiful landscape, Paris is one of the most popular tourist destinations in Europe. Paris has something to offer everyone and attracts people from all walks of life in a respective of age, culture, gender or nationality. The charm of the city lies in its warmth and romance which is exuded in the attitude and outlook of the Parisians. If your soul is seeking the right place to express its identity, your heart looking for love, then Paris may be the TESOL course location you are searching for. TESOL trainees can see Paris coming to life when the whole city is lit up at night. Discover yourself in the rich culture and heritage of the city. Paris is famous for its world famous landmarks, parks, museums, galleries, institutions. Nowhere will you find so much to see and do in a single place. Today, it finds a place among major global cities because of its influence in politics, education, entertainment, media, fashion, science and the arts. Paris is one of the world's leading business and cultural centers. Paris has all the essential qualities to make it one of the most popular tourist spots starting from its rich history to being the real cinema capital. The city boasts of world famous intellectuals and artists. It is also home to numerous cafes, bars and restaurants and has been rightly referred to as the city of gastronomy. Paris City will make you feel lively and enthrall you with its music encompassing jazz, avant-garde, salsa and, currently, EUROPE back quote s most vibrant African music scene. The beauty of Paris lies in retaining its countryside charm despite the influx of tourism and tourists in the country. It is the only place in the world where you can hope to get away from the hustle and bustle of city life and find peace for yourself and inner soul. Come and be captured by the charm and spontaneity of Paris during your Tessal course in France. Tessal Paris Paris offers a wide range of accommodation to suit different lifestyles and budgets. Our training center suggests one of the following options, home stay. In order to offer students the best possible choice of home stays, we have formed a partnership with a private tea professional hosting company. With this option you will have a furnished room in central Paris with a shared kitchen and bathroom. Prices for a room with breakfast start at €760 for the duration of the course. Private residence. In the vicinity of the school, one to five minutes walking distance, there are several college run private apartments for rent. Most of them are twin studios, one room with two single beds plus kitchen and bathroom. Prices are around €800 per person per month. Double occupancy or €1200 single occupancy including electricity, heating, internet and insurance. Private room. You also have the option of renting a private room in a Parisian apartment for the duration of the course. They usually come with internet access and shared kitchen and bathroom facilities. Prices start at €850 per month. Student housing. Located close to the subway and many shops, this very affordable option comes with hot water, fridge, hot plate in bed with linen provided. Shower and toilet are shared. Student houses are completely self-sufficient and start at €300 per month including heating and water. Hostel accommodation. There are many hostels in the area which offer self-catering facilities, usually with shared kitchen and bathrooms. Hostels usually provide a TV room, internet access and some have phones in the rooms. And suite accommodation is available in a number of hostels at an extra charge. Prices start around €22 per night in a multiple bed dorm. Many TESOL course participants prefer to organize their own accommodation for their stay in Paris. However, if you would rather have the help of a local expert, we can provide a housing coordinator to assist you. The housing coordinator charges €135 and guarantees the choice of at least three different options. If you have any questions regarding your accommodation for the Paris training course, please don't hesitate to contact us and we will do all we can to help. Rome is one of our most popular TESOL course locations. The combination of culture, history and the large number of English teaching jobs throughout Italy make it an ideal choice for a TESOL course. Our TESOL training center is in a perfect location, close to the Vatican in public transport and offers very modern training facilities and IT professional staff to ensure you have a memorable time during your course. It is easy to imagine sitting outdoors on a pleasant evening enjoying a refreshing ice cream while sitting on the world famous Spanish steps or overlooking the Colosseum and just watching as people stroll on by. This wonderful mix of the old and the new of culture and history in the unique delights that Rome offers makes it, unquestionably, one of the most beautiful, fascinating and famous cities in the world. Rome is widely known as the eternal city and is certainly one of the most important of all European cities in terms of the sheer wealth of history, culture and religion and has been so for thousands of years. It is hard to imagine that any other city could rival the overall depth of history and beauty that can be found here with wonderful architectural structures such as the Colosseum, Treaty Fountain, Spanish Steps, the remains of the Roman Forum the imposing presence of the Vatican and the masterpiece that is Sistine Chapel and that's just beginning. However, Rome isn't only a place to experience the past. There is also a modern lively feeling to the city and it is a wonderful location in which to witness modern daily life and see how it fits in comfortably against the backdrop of ancient culture and history. The many delightful old squares with their old fashioned cobblestone streets and courtyards are favored meeting places for both locals and tourists and are fantastic places for sitting under the warm Italian skies and people watching. Rome has a great many restaurant terraces offering authentic and sensational Italian cuisine yet at quite reasonable prices. Rome is situated along the Italian west coast and while the city is very large and spread out over an extensive area running down to the coast, the famous historical center is quite small and localized and it is certainly easy enough to visit most of the better known attractions on foot. The city, however, also has an extensive, efficient and modern public transport system for visiting further afield. Due to its close proximity to the beach, the city of Rome is ideally located for people wishing to experience a major historical city and yet at the same time have easy access to the lovely beaches of the Mediterranean. As there is a wide range of opportunities for teaching English in Italy, it is the perfect choice of location for your Tezal course. Tezal Los Angeles Santa Monica, although surrounded by the huge urban sprawl of Los Angeles on three sides, is actually a separate city in its own right and is situated on the fantastic California coast. More than 300 days of sunshine per year mean that there are plenty of opportunities to make the most of the beaches on this beautiful stretch of coastline. The city also provides the perfect antidote to the frenetic pace of Los Angeles life and draws a wide range of people from celebrities, artists, students and surfers all looking to enjoy the city's laid back atmosphere. Long a major draw for visitors from the wider Los Angeles region, Santa Monica's beachfront also brings in tourists from all over the globe. The connection of the city to the national railroad network brought in large numbers of day treppers and the beginning of the 20th century saw the arrival of the first amusement piers. In its heyday there were several of these piers housing all manner of rides, arcades and dance halls, not to mention roller coasters. Although many of these are original attractions have since disappeared, the famous Santa Monica pier still stands as a monument to the city's golden age. The beachfront also has 3.5 miles of sand with parks to wander around, as well as plenty of beautiful spots to have a picnic. Santa Monica is also home to the original Muscle Beach, the famous hangout for bodybuilders, acrobats and gymnasts since the 1930s. As mentioned, Santa Monica is a city in its own right and though it has plenty to offer, it cannot be denied that its proximity to Los Angeles is major factor in the city's popularity. Hollywood is an ever popular draw for visitors from all over the world with the iconic sign overlooking the city, tours of major movie studios and the famous Hollywood Walk of Fame, just some of the star studded attractions. There are also a wide range of museums and galleries to visit including the Museum of Contemporary Art and the Getty Center which houses an extensive art collection and beautiful gardens offering stunning views over the city. Whatever reason you have for choosing Santa Monica as your destination for earning a TEFL Tezal certificate, you can be confident that you have made the right choice. Our excellent facilities and dedicated IT professional trainers, combined with the beautiful beaches and everything else Santa Monica has to offer ensure that you have an educational and fun experience on our TEFL Tezal course in Santa Monica. Tezal Los Angeles Trainees attending our TEFL Tezal certification course in Los Angeles will find there are plenty of accommodation options to choose from due to the city's status as one of the most popular tourist destinations in the USA. You will find everything from low cost hostels to high end hotels in every part of the city, however, as LA covers a vast area and getting around can be very time consuming, we recommend that you choose accommodation that is located close to the training center. Although you are free to choose any accommodation that suits your preference, we can recommend one local option that many of our previous trainees have found convenient. This option is based on apartment style accommodation that includes a kitchen area in every room. This great feature allows you to cook for yourself and therefore helps to keep your overall budget down during your stay. The well equipped kitchen includes essentials such as refrigerator, microwave and stove top, as well as other extras including a coffee maker. The accommodation also includes complimentary breakfast and access to laundry facilities and Wi-Fi internet. Full details will be provided once your course booking has been completed. Tezal New York City As the biggest city in the United States, New York's influence stretches far beyond its home country's borders to regions all around the globe. Indeed its influence can be felt in finance, commerce, media, entertainment and fashion while its importance on the world stage is embodied by its role as headquarters of the United Nations. On a more local level there are a seemingly never-ending number of things to see and do in the city and it is a fantastic place to take your Tezal certificate course. Although originally settled by Europeans 400 years ago, it was not until major upheavals back in Europe in the middle of the 1800s that New York City really started to grow in size and influence. The city's population swelled with immigrants flooding in from Ireland escaping the famine that was gripping the nation, others were fleeing social unrest on mainland Europe. New York's population has continued to grow since then and it has now become one of the largest and most ethnically diverse urban populations in the world, with over one third of its residents, born outside of the United States and around 800 different languages spoken by the city's inhabitants. With such a diverse population drawn from all over the world it is easy to see why it was chosen to be the home of the United Nations. As with any large city there are a huge number of things to see and do in this thriving metropolis. With a skyline as famous as the city itself any visitor will be sure to visit the icons of 20th century art deco architecture, with the Empire State Building and Chrysler Building two of the most well known. Once the first sight that greeted new arrivals to the city, the Statue of Liberty is another must see as is the hectic Times Square. Escape the hustle and bustle and spend a quiet afternoon in Central Park with its large number of attractions including boating, horse carriage rides and even rock climbing. Shopping is of course another popular pastime in New York City and there are options to suit all budgets from high-end fashion stores to street sellers. For theater lovers all the latest and most famous shows can be caught on Broadway while music fans are spoiled for choice with countless music venues all over the city catering for every possible taste. Wherever you are from there is something for you in New York City whether you like to spend your time seeing the sights or watching the latest Broadway spectacular you will not be disappointed in New York. When it comes to study time our trainers are here to help you finish the course feeling well prepared for your first job as a qualified English teacher. With all this in mind it is easy to see why our New York City TEFL TESOL course is one of our most popular courses year after year. TESOL New York City The popularity of New York City with people from all over the world means that it has a huge amount of accommodation to offer its 50 million annual visitors. You can choose to stay in any style of accommodation that suits your preference or you can choose one of our two recommended options. Both options are well situated within the immediate vicinity of the training center and offer a comfortable and affordable stay. The YMCA guest house has a choice of rooms as well as excellent facilities including swimming pool, fitness center, on-site cafe and Wi-Fi internet. The surrounding neighborhood is also a great place to hang out and enjoy the vibrant atmosphere of the city. We also recommend a range of one or two bedroom apartment style rooms in the same area. All the rooms have a private bathroom and a well equipped kitchen area plus access to a fitness room, laundry facilities, free Wi-Fi and daily housekeeping service. All our trainees are eligible for discounted room rates at both of our recommended options when booked through our New York Training Center. Full accommodation details will be available once you complete your course registration.


Good morning everyone. My name is Greg Pixley. I'm the spokesperson for the Denver Fire Department. We're here today to talk about fireworks and the upcoming Independence Holiday weekend. Now the Denver Fire Department often gets phone calls from people that are asking about fireworks. Fireworks in the city and county of Denver are illegal as we are going to hear shortly. But oftentimes we get questions as, is it not my right or privilege to shoot off fireworks during Independence Day? And again, the Denver Fire Department re-emphasizes that any fireworks in the city and county of Denver is illegal. It's not a right. It's not a privilege. We as a community must work together to ensure that our family, our friends, our neighbors are safe. And the Denver Fire Department can't do it alone. As we see an increase of injuries due to fireworks, as we see fires increase due to fireworks over the Independence Holiday, it takes all of us working together to ensure that the community is safe. We ask and we plead for your help to ensure that illegal fireworks are not used in the city and county of Denver. Interestingly enough, we have seen a market change in weather patterns here in the state of Colorado. And we know that the use of fireworks can only compound the dangers that we face. Our neighbors to the north of us in Lafayette suffered from the Marshall fire due to these devastating winds and these arid and drought conditions that we experience. We in the city and county of Denver hope not to experience something similar. So you as a community can work with us by ensuring that illegal fireworks are not used. We have to work together. The Denver Fire Department cannot do it alone. The Denver Paramedics cannot do it alone nor can the Denver Police. It takes a community effort to ensure that we are all safe. Help us. Help your neighbors. Help your friends. And help your family. Always do the right thing and leave fireworks to their professionals. They can do it far better than anybody that has bought hundreds of dollars of illegal fireworks in a local or neighboring state. Last thing I would like to leave you with, anytime there is the possibility of a fire or emergency, please call 911 immediately at the first sign that there might be an issue. Again, the Denver Fire Department thanks you for helping us help this beautiful city. I would like to introduce Ms. Julie Ariano. Denver Health Paramedics would like to invite you to enjoy this 4th of July holiday season but to please enjoy it safely and responsibly. The festive summer season brings happiness and many celebrations but it's important to take safety precautions. Safety when you're grilling, when you're enjoying fireworks, safety when enjoying water activities like boating and swimming, and safety when consuming alcohol including being diligent about sober driving. To remain safe this holiday, we encourage people to leave the fireworks to the professionals. Fireworks including sparklers are dangerous and can cause serious injuries and burns. Over 15,600 injuries were related to fireworks in the U.S. in 2020. This is up significantly from 10,000 injuries reported in 2019. The hands, fingers, face, and eyes are the most common parts of the body that are injured by fireworks. Many fireworks injuries are from carelessly playing with fireworks or lighting them while holding them. Please keep your family and children safe and do not light fireworks at home. It is also important to remember safe and sober driving this 4th of July. After fireworks shows and barbecues conclude, there will be increased amounts of cars on the road and impaired driving puts you and those around you at risk for serious injuries and accidents. Be safe on the road and find a ride of drinking. Stay hydrated and stay cool this summer and have a fun and safe 4th of July holiday. Next up we have Jennifer from the Denver Animal Protection. Good morning. Every year thousands of pets go missing during the 4th of July holiday. As much as we love our pets, please keep them safe. The best ways to do that are to keep your pets indoors, with music on, things to drown out the fireworks. If they're stressed out, put them in a quiet room. Even if you normally leave your pets outside, the safest thing that you can do for your pets is to actually bring them inside. Dogs, ears are super sensitive to the noise of fireworks and thousands of dogs get lost every year because of it. Make sure that your pets are microchipped and that that information is up to date. Make sure that your dog has tags on it and all that information is current. Please do the best you can to keep our animals safe this year. Thank you so much. Thank you. Good morning. Thanks. Thank you everyone for being here. On the police side of things, fireworks are illegal as Captain Pixley has said previously. And violation of this could result into about a thousand dollars, up to a thousand dollars in fines and court costs, as well as up to a year in jail. So it is important that you do not like like these fireworks on top of them being illegal. They are dangerous. People get injured year after year. So it's important not to not to do that. Other things to keep keep in mind, as has been said before, is also impaired driving. If you are going to plan to consume alcohol, marijuana, marijuana products, it's important to plan for a safe and sober ride, wherever you're going. And then the best way to report fireworks is by one, you can do it online at denvergov.org four slash pocket gov. The other way is calling<PHONE_NUMBER>. That phone number is specifically for reporting fireworks. But also keep in mind, if you do want to report fireworks, we do need a complainant to issue a citation to the violator. So it is important to be available for officers if you do make a complaint. With that, we'll just open up for questions. If an officer does catch the violation, the person can see a fine up to a thousand dollars and spend up to a year in jail as well. Manada de reportar esto, puede hacerlo por línea en denvergov.org, guion pocket gov, y también puede hacerlo llamando al<PHONE_NUMBER>. Cuando están reportando eso, es importante entender que también necesitan una persona que firme por esa violación para que le puedan dar una multa. Pues, como cual la información? Si pues es ilegal vender, poseer o prender estos fuegos artificiales en denver. Anyone else? Alright, thank you.


What is up guys and welcome back to the John and Q channel. Thanks so much for watching. In today's video, we're gonna talk about a few tips, three tips on starting a YouTube channel. Let's do it. So this video is intended for those who have just created a channel, are creating more content and have little subscribers and want to grow even more for people in that realm, like people like me. I've been slowly growing, oof, you all right? I've been slowly growing on my channel because of these three things. And number one is this. It sounds pretty normal and kind of like, oh that's a horrible tip, but it's what's worked. It's like common sense, exactly. Start creating content, start creating video. You don't have to post those videos, but whatever you're passionate about and you like doing, start creating that video. Start making videos on that specific thing. I'm a mom of two, I don't have any hobbies right now. Let's say Sarah likes to knit, okay? She's a big knitter, she's in the knitting community. I would say this, start posting videos of you knitting and showing people how to knit. You can practice and if you're not comfortable being on screen, practice doing that knitting and being on screen. Wouldn't you say that's a good tip? Yeah, a lot of people wait until they're in this perfect place, but really you just gotta start making that content, even if you're not the world's greatest knitter. So create. Just like, there's so many mommy bloggers who, they're first videos that they make and it's like, wow, that is not a good tip. But they were just learning. And they grew and they got better and more knowledgeable. Do you just wanna take over this section? I think you're doing so well. That's exactly on point. Which brings me to tip number two. Whether you have this or whether you have this, it does not matter. Or if you have this, it doesn't matter. Create those videos with what you have. Really started on my phone, did I not? Yes, you started on your phone. And then you got a Sony. We would literally go on walks just like this and I would just film and it was exhausting. There was a bunch of photos or video on my camera, but that kind of sucked. And then I finally switched to the best YouTube camera on the market and that's the Canon 60 Mark II. Solid. That's a different video we'll get into. It doesn't matter what you have. You just gotta use it. And again, it goes back to tip number one. Just create and get used to it and get used to your camera.  They use their phone or they use like a small camera that does video and photo. So use what you have. Okay, Sarah, tip number three, tell them what it is. Consistency and what you do and how you do it. Tip number three is being consistent with what you're doing. So if Sarah is really passionate about knitting in the knitting community with other moms and other people who love knitting, I would say post three times a week. Don't force yourself to do seven times a week because that's just a lot. If you can only do two to three or even once a week, keep with that schedule. Make realistic goals. Don't say that you're gonna daily vlog unless you're gonna keep up with it. Dude, are you kidding me? I did that. You did. For like a month straight.  Because you're going to be consistent, you have a better opportunity for people to come to your channel and view and subscribe. For those who are posting once every two weeks or once a week, you won't really see that much growth because you're not being consistent. So if I'm trying to get really ripped, I have to go to the gym like every other day. I'm doing three to four times a week. I don't get ripped by going once a week. So those three tips, I think those are going to help you not only grow on your channel but actually help you learn what it is and what it takes to actually create videos to post to your channel. I think there's one more tip. What's one more tip? Always be a constant learner. So if I'm super passionate about knitting, I should try to always be learning about knitting and staying up on trends. I mean, that's one thing on YouTube where if you're keeping up with the trends, that's one way that you can get some momentum and get noticed.  Why, why? That's great. She's on point. I love that woman. Okay, now we're gonna go and get some food because the kids are getting cranky but I hope this helped you because- He's getting cranky. He's getting cranky. I'm getting, no, I'm not getting cranky. Anyway, I hope this helped you because I've been getting a few. I'm gonna come back here because the water is, it's too loud. I got one last night on my Instagram profile asking, dude, what are some tips that I could use or you can use to grow my channel and to actually to create a channel? So I hope that this helps you. So please take it. Let me know if it does. Let me know if you have any other tips and I would love to have the conversation going on the comments. So I'll see you guys later. Peace.


Hi everyone, Joe for jaspyscasebreaks.com coming at you with 2020 contenders football two box picker team number nine With the chargers randomized to the break if you followed the instructions here if you got at least two teams so There's everybody right there. Thanks everyone for getting in so let's actually Alphabetize this by your first names really quick and if you bought at least two you'll get an entry so Adrian bought two. That's an entry Brian bought two. That's an entry so did John and John and Joseph and Matthew and Robert That Adrian Brian John Jonathan Joseph Matthew Robert Sean Got eight teams. That's one two three four entries Tim got four teams. That's two entries and William got four teams. That's two entries nice There you go. So there's a total of 15 entries here. Good luck. Let's get him into a blank list Let's roll it randomize it name on top after seven good luck One Two and good luck seven then final time chargers going to Joseph last bot mojo Strikes again. There you go Let's put your name right there Now we can get rid of this column. Let's shrink this back here again Let's sort by column B and now Let's print and rip we got a Naomi Osaka and Serena Williams playing against each other right now semi-final Nice Brooks Mullen is here saying I collect 1492 tops looking for Christopher Columbus discoveries inserts Think that you you'd probably find that in an Allen and Ginter set tops Allen and Ginter you can look there They'll have some stuff from around that era Happy hunting. All right primo you saying I missed it. Missed what? What did you miss? There you go basses loaded guy. There you go Brooks Happy hunting. Oh, yeah, I Supposed to use rich. Hi rich on the other account says sold out All right, so someone must have snagged it away from you Looks like it was Levi. Well, I appreciate you trying primo. All right, so Now the baseball mixer is official. That's coming up after this. Please check the schedule right there because that pretty much Is our night unless we can fill that triple case break and all its fillers I might if I'm ahead of schedule. Oh Nice Jalen hurts rookie ticket on card autograph for the Eagles Tim Chung with the fly Eagles fly nice Now if that triple case break doesn't sell out by the end of the baseball mixer, and if we have a little more time And we if that mixer goes faster than I thought it thought it would then Maybe I can squeeze in another quick little short break These are the same players, but this will represent the left-right randomizer There's a rookie of the year to 149 de Andres Swift That'll be for the Lions Joseph. Yeah, we got a rookie ticket autograph Zach Bonn won Adrian and the Saints. We've got a Kari Willis for the blue horseshoes. That's gonna be for Brian Crouch and the Colts Where's that guy gonna go? The Sean Watson and there's Jamaical hasty autograph Rookie ticket auto for the Niners Joseph last spot mojo 60% of the time last spot mojo hits a hundred percent of the time Colby Parkinson Rookie ticket autograph for Seattle Adrian with the Seahawks the Hawks of the sea All right second box. I'll post more of these the second half of this case later tonight Never pretty decent at tennis. I could just never serve serving was just my worst Worst part of my game never got that right. All right next box. Oh Justin Herbert Rookie ticket on card autograph Justin Herbert and that's for Joseph Rainor Who got last spot mojo by buying the last two teams the Lions and the Niners got randomized the Chargers and Gets his Justin Herbert on card autograph. He's got a nice auto too Nice congrats There's a numbered card and another Justin Herbert So since that round numbers card is is in serial numbered it will be randomized between the Chargers and the Packers We've got a John high tower the fourth Tim with the Eagles Yeah, the Hertz in the first box too. It's a nice break Kari Willis another one for the Colts Brian Crouch It's to a Tungo by Loa for the Dolphins Matthew Shearer John Grinard Rookie ticket autograph for the Texans that'll be for Sean Mattick James Frosch Going to the purple birds Aaron with the Ravens Kind of a butterfly Right and there you go ladies and gentlemen All right couple randomizers to do nice Justin Herbert once again so we'll do the left right randomizer and then the Packers Chargers randomizer. Let's flip this back to oh that was for the Chargers. Let's get some new dice and a new list There's a left-right randomizer And there is the Los Angeles Chargers Green Bay Packers round numbers card randomizer. Let's roll it randomize each list two and a one three times one two and Three easy left side. We'll get those round numbers cards All right, and three times here one two and Three and it stays with Green Bay. So the Packers Brian Crouch We'll get this round numbers card and some good spots for autographs if you ever run into Justin Herbert and Jordan Love And there you go ladies and gentlemen, I'm Joe for jazby's case breaks calm I'll see you next time for the next break. Bye-bye


When we have a data frame in R, and we're working in the tidyverse, it's a common practice to group by one or more variables and then use a summarized function to generate a column of summary statistics. But what do we do when we want to summarize, but instead of generating one value from a function, we want to generate multiple values. That's exactly what we're going to talk about in today's episode of Code Club. Hey there, I'm Pat Schloss and this is Code Club. In each episode, I try to apply reproducible research practices to an interesting biological question. Over the past couple of dozen episodes, we've been looking at the sensitivity and specificity of Amplicon sequence variants. Now, hold on. Even if you don't know what an Amplicon sequence variant is or what a 16s RNA gene is or even what bacteria are, I know you'll get a lot out of today's episode. This problem of how do we generate summary statistics where we have perhaps multiple output values from a function is very common in data analyses. So instead of going and looking at some tutorial using some silly example, let's look at an example using real data to answer a real question that at least I'm interested in, even if you're not. So stay tuned and I'll show you how we get this done. In our most recent episodes, we've been building up to try to build a receiver operator characteristic curve using our Amplicon sequence variant data as well as the species of bacteria that those sequences come from. So we're working in issue 37. I'm going to go ahead and navigate to my project root directory. You'll see from my tag here that I am on issue 37. I'm going to go ahead and fire up our studio here. And we'll get right into the analysis. So again, the file we've been working on is getrockdata.r in our code directory. And you can look back at the previous episodes, the last two or three episodes to see what's going on in all of this. I think it's pretty well documented. Where we're at is in the last episode, we generated a function written in C++ using the RCPP function. And this runs rather quickly. It took about 10 seconds to run the whole analysis. And what it's doing is it's comparing everything to everything. I think we've got about 20,000 operons that we're comparing to each other. And so that's about 200 million, yeah, 200 million total comparisons. So if it takes a 10 seconds, we can live with that. The challenge, though, is that we have something like 56 different comparisons. I mean, previously I said 52. Forgive me 56, I was off by four. And so what we need is a way to apply this function across all of the different regions we're looking at, as well as the different thresholds that we're using to define an Amplicon sequence frame. So I am going to go ahead and highlight all this stuff and run it so that it's loaded in my current R script. Hopefully there won't be any error messages that come up. Right now it's compiling my C++ code, and we're in good shape. One of the first things I'd like to do is perhaps organize my code a little bit better. I like to put my functions at the very top of my R scripts. So I'm going to take this C++ code.  Perhaps right below my library calls and see what else we have here. We've got the RCPP. We've got all that. This is kind of going off the edge of the page. So I'm going to go ahead and clean up the code a little bit. So I've got nice wrapping. And again, if I run this, it should compile with no problems. Great. So we've got our set seed, our desired threshold, the desired region. In today's episode, we're actually going to get rid of these two lines. So remind me to come back and delete those. And you'll see that we read in our EASV file, our metadata, which tells us what genomes are in our data set and what species they belong to and what EASVs they have. We then assembled these to generate that metadata EASV file. And if we look at metadata EASV, we'll see it has two columns, genome ID and EASV. And I pause there for a second because I'm like, where did all the data go? And I'm remembering that up here where I define my EASVs. And so that's for exact sequence variance as well as amplicon sequence variance. I recall that I had this filter function. And so I want to be able to look at all of my regions and all of my thresholds. And that comes back to these two lines that you're supposed to remind me to remove. I'll go ahead and remove those now. And I'm going to remove this filter line. And I noticed that before we had removed the threshold column and the region column. So in here, I want to be sure to keep my region and my threshold. Right? And so if we run that code block, as well as this next one to read in the metadata, we can then regenerate metadata EASV. And we see that we've got our genome ID, the EASV, the region, as well as the threshold. I'm going to go ahead and move this comment way back up top to my C++. The number corresponds to where in my data analysis plan this had fallen. It's probably not that important to have the number, but I'll leave it there for now. Okay. And give myself a little bit more breathing room to look at the code. We see that we have our metadata EASV. We have this confusion matrix stuff. One of the things I noticed when I was editing the last video, actually, was that I had the summarize function still in in my code. And that's a legacy to maybe two episodes ago or three episodes ago. So I can go ahead and remove this. But for this to work, I need to have true pause, true nag, false nag, false pause as the names of the columns outputted from my get confusion matrix C++ code. So that was back up here. And so this here, I'm going to put true pause, false pause, true nag, and false nag. And again, I'm going to compile this. So one of the things to notice that C++ code is compiled, whereas R code is interpreted. And that's again, part of the reason why C++ is faster than R. Again, it's not a big deal. And we come back down here. And we've got get confusion matrix. And then we look at the region, the threshold, the sensitivity specificity on all that. And actually, I'm for now going to come I'm going to come back to this and set this aside for now. But what we're going to do is we're going to take this metadata ESV. And I know I just told you that we wanted to get rid of that filter function, but I'm going to insert a new filter function to make our data frame a little bit smaller so that it's easier to work with. You know, if I ran everything all 56 combinations that might take, you know, 40 minutes or so to run. So let's let's only look at maybe one region and two thresholds. So I'll do region equals equals v four. And threshold equals equals 0.03 or threshold equals equals equals, let's say, let's do 0.00. Okay, so make sure that works. We see that we've got our region v four and our thresholds of zero and 0.03. And what I'd like to do is, you know, if we're thinking about the typical group by summarize idiom, we might do group by our region threshold, right? And, and then summarize, and we might then say confusion, or confusion matrix equals get confusion matrix, get confusion matrix, and then we pass it some, some value. The problem is that for summarize to work, this function needs to return a single value, not four values, as is our case, right, the four values of the confusion matrix. So we need a slightly different, actually, it's not slightly different, it's quite different approach, but it's the same kind of idea. So let's remember what happens when we look at metadata filter and group by. And so what we'll see is that the output tells us that it's a data frame that is grouped by region and threshold. Okay, and then normally what summarize would do would take each of those groupings. And so there's two groupings in this case because of how I filtered it. For each of those groupings, then it would run the summary values. Okay, so again, hold on to that thought. What we're going to do is we're then going to do what's called nest. And so we will then nest the data frame. So we'll nest everything that's not region and threshold. I'm going to remove that pipe because it's going to annoy me. And so what we get is a really interesting new data frame, right, we get a column for the region, a column for the threshold, and then a column for the data, right? And the data is a column of type list, which we really haven't talked much about. That contains two data frames, right? So each row of the data frame in this column is a table is a data frame that contains 20,041 rows, and two columns, right? So that's a bit different. And what we can then do is we can then pipe this to the next step, which will be a mutate. And we can then say mutate confusion. And so we're going to create a column. So remember, if you see mutate, that means create column. And so we will then create the confusion column. And we will populate this column using a map function. And we've talked about map before. I believe we used it previously when we were reading in a bunch of data frames that we wanted to concatenate together. We used map underscore dfr. Here we're going to use map. And the input to this is going to be the data column, right? And so we're going to take the data from the data column for each of these rows. And we are then going to give that data to our function. And so a little funny syntax, I can't remember if we've talked about before, is a matrix. And then we do get confusion matrix, open close parentheses, and then period x. And so period x is the value of data. So that's the variable that data is representing. And you can figure that out if we do question mark map, if you look at the usage, you'll see that map takes dot x and dot f as its argument. So dot x is the data, and dot f is the function. We're being a little bit lazy in not putting in the argument names. But this works pretty well. Now, if I run this, hopefully we don't get any error messages, we should get a new column that comes out called confusion that will also be of type list. And that each of those rows will contain tibles themselves. And so actually, their data frames are not quite tibles. And so we see is that confusion is a column of type list, like data was, and it's a data frame, each cell is a data frame with one row and four columns, again, those four columns correspond into true positives, true negatives, false positives, and false negatives. Great. So I'm going to go ahead and remove this summarized line because we don't need this anymore. And what we can then do with the output is that we can then do unnest. So we can unnest, and we give it the column we want to unnest, which will be confusion. And this works really nicely because it's a single row data frame column that we're unnesting. And we run this and it again, it's going to take a little bit of time, but we will then get output that is region threshold data still in this list format, as well as then can for columns, the confusion will go away, but we'll have the true pause, true neg, false pause, false neg, as well as the values for these two thresholds for the v4 region. And sure enough, that's exactly what we get here. Okay, isn't that cool? We can then pipe this to get rid of that data column. So I can then do select hyphen data. And we run that. And again, this is going to take a little bit of time, but what we'll get out is a really compact data frame containing the region, the threshold, and then the four values of the confusion matrix. So that works really well. We can, of course, then do ungroup to remove the grouping variable by region and threshold, and we will be in good shape. I'll go ahead and call this confusion matrix. We'll assign that run that and we'll be ready to run to the next step. Alright, so that ran. One thing I want to remember remind myself to do because I always seem to forget this is I'm going to remove my filter line. And I'm actually going to cut this code and paste it down here where I defined confusion matrix before. And so this is exactly what we want where we want it. I'll go ahead and remove some of this extra space. So I'm curious how long this will take to run for all 56 combinations. I think of each rep takes each condition takes 10 seconds or so to run. And we've got 56 combinations, divided by say 60 seconds per minute, it should take about nine or 10 minutes. Let's go ahead and run this. And let me see if I can run this within system dot time. For some reason, I think it won't like that. But you never know, we'll see. It likes it. Alright, so probably in about 10 minutes, we'll check back and see we'll see where this is at. So that completed and that took about 628 seconds, which I believe is yeah, it's just over 10 minutes. So we're fairly close. Now the thing to keep in mind is that we're going to repeat this 100 times to get 100 iterations, because if you noticed up above, we're taking one genome per species.  So if we run this 100 times, that's going to be 1047 minutes divided by 60, it's going to be about 17 hours. It's a long time, certainly not as long as it would have been if we hadn't rewritten that code and see plus plus, but it's still a long time. So stay tuned for the next episode. And this reminds me, go ahead and subscribe to the channel and click on the notification. So you know when that next episode is dropped, we'll see how we can use the map functions and maps from another package that will make this a lot faster to run. So but before we move on to that and close out today's episode, want to briefly go back through this code, I'll go ahead and remove my system time function. And to remind you what we do did right, we had this metadata, easv data frame that had our genome ID or easv region and the threshold, we could use group by to segment our data frame by region and threshold. We then use the nest function to create a data frame where each where there's a column called data, where the data frame each row of the data frame was a data frame, right? So it's very inception esque. We could then use mutate to create a new column called confusion. And remember, we had to use mutate because the summarize function only allows you to use functions that produce one value to populate a column. So the map function is used to create values when you're going row wise, right, or group wise. And so what map does is it takes the data from the column data, and for each row, which is basically its own data frame, it then used get confusion matrix on that rows data in the data column, probably, you could probably call this something different in nest, but who cares. So that then creates a new column in this case called confusion, which is a type called list, again, where each row is a different data frame, we could then use unnest to remove to unnest the confusion columns that we then got four separate columns for the four elements of the confusion matrix, we can then get rid of that original data column from the nesting procedure, and then we remove those grouping variables and wallah. In the end, we have a data frame confusion matrix, where we have the true false true pause false pause true neg false neg, for all combinations of the region and threshold. Now, again, this is only for one iteration one sampling of genomes per species. We'd like to repeat this 100 times, and then get some say mean or median representation of these values. Alright, this is going to close out the code for today. Make sure I save that. And I'll go ahead and commit this change. I'll get add code get rock data, and then get commit to let's see apply get confusion matrix to all regions and thresholds. Great. So again, the next step now is to deploy this for 100 iterations, and to find a way to do it quickly. And we'll use another map function to do that. And as I alluded to, we'll use a package called future that will allow us to parallelize this process. So if my computer has 16 processors on it, and I want to do 100 iterations, that it'll divide those 100 iterations across those 16 processors. So stay tuned for that. In the meantime, please be sure to tell your friends about Code Club. Keep practicing with this material. Have you used the nest map mutate unnest idiom before. See, see what you think about it and see if that might make your pipelines a little bit more streamlined and coherent. It is a it's similar but different again, by from using the group by and summarize idiom, but it's another useful tool to have in your collection of of tools in using our and the tidy verse that those map functions again come from the per package p u r r r, which is encapsulated within the tidy verse. All right. So keep practicing. And we'll see you next time for another episode of Code Club.


Good day, May 40th here. So I'm walking around Sydney Harbour, looking at Sydney Opera House, the beautiful Royal Botanical Gardens listening to my Apple news plus audio Productions, this is what the half Asian half white protagonist Reveals about Reveals about America Very scary stuff here guys. I need to provide warnings for you I'm not sure you're gonna you have to handle this. Why does the half Asian half white protagonist make us so anxious? Yeah, written by Andrea Long Chu for New York magazine narrated by Emily Wu Zeller Please be advised this article contains racial slurs. Oh, no It only takes a few years an economic catastrophe brings on the partial collapse of American society as The nation recovers an ascendant right-wing blames the crisis on China In the years following the United and for absolutely no reason whatsoever It's not like China did anything wrong with regard to COVID or any of its policies just for no reason whatsoever An ascendant right-wing just blames China sad It states is rebuilt as an authoritarian nation under the preserving American culture and traditions act Wow, that sounds awful the preserving America Cultures and traditions act well the reflection It's an act that probably sounds good for some people and bad for other people so who would have thought that Different legislation might impact some people's positively and other people's negatively So this article seems to be saying that some of politics at least is a zero-sum game Who would have thought that I certainly wouldn't So an ascendant right-wing taking power in America with a kind of a Anti-China bias and tries to preserve American traditions. This is very scary stuff Colloquially known as packed an expansive law that allows the government to ban books monitor private citizens and disappear political dissidents All in the name of preventing the spread of un-American views Yeah, I'm really sure that's just right around the corner Man, the Republicans are sure are scary, right? Wanting to intrude on our private speech like this a category that grows broader by the month Appearing sympathetic to China. Well, look, there are situations where you do want to restrict speech there are situations when you want to enhance the power of tradition and community Where you want to world people together rather than encourage people to divide up The diversity means you know, unless we have in common the better Then anyone though is calling for this kind of legislation appearing insufficiently anti-China Having any doubts about anything American having any ties to China at all. No matter how many generations Right, so there would be some outside context, right that would play a role here This would depend on how China operates At the reason that this anti-China sentiment is because of concrete things that China has done that are for other people China is very unpopular in the world. It's not just some right-wing racist thing to hate China There are plenty of people on the left hate China It's past This is fiction Obviously, even as it clearly brings to mind Japanese incarceration and the rise of McCarthyism as well It doesn't bring to mind Japanese incarceration. There's nothing like that Japanese Americans who lived on the West Coast were told that they either had to move or that they would be put in Relocation camps. People's private conversations were not monitored to this extent. There wasn't Any kind of overall social crackdown anything equivalent to what this novel describes So it's a pretty normal response when your country goes to war with country B members of your country A Who have ethnic, racial, religious, cultural ties to country B are going to come under suspicion Well as the wave of racist attacks on people of Asian ancestry since the pandemic began Okay, these racist attacks have been part of America's huge boom in crime since George Floyd died And there's been one primary group of super predators who's been carrying out these crimes and Asian Americans are still the safest of any immediate racial ethnic group in America Right, they are less likely to suffer from violent crime than any other racial group in America but in America today, we're strongly incentivized to Maximize a sense of victimhood and grievance So the American thing now is to proclaim how much of a victim you are To proclaim your grievances against the white Christian majority The book in question is Our Missing Hearts the third novel from author Celeste Inge About a 12 year old boy named Bird Gardner whose mother a Chinese American poet Abandoned him and his white father three years before Inge's little mixed-race hero doesn't speak Cantonese and doesn't seem to eat Chinese food or know any Asian people So I am one sixteenth Chinese, so I have not suffered to my knowledge from You know being stereotyped as Chinese my father was one eighth Chinese the other kids would call him chinky My father suffered enough From these slurs that he was very sensitive about them All right, he was very sensitive about people knowing that he was one eighth Chinese so to my father and growing up in Australia in the 1930s and 40s Being known as part Chinese was not a positive thing and then his mother who was one quarter Chinese She wore extensive makeup did absolutely everything she could to try to disguise the fact that she was one quarter Chinese But that was then this is now all about the love and tolerance radical love and inclusion Multicultural diversity, but his appearance alone the tilt of his cheekbones the shape of his eyes is Enough to subject bird to the unifying existential threat faced by anyone who might seem Chinese This spectacularly anti-Asian version of the United States betrays a new more openly political ambition on Inge's part Whereas her previous work focuses on the experience of Asian Americans She is now trying to write about Asian America itself The problem is that such a thing may not exist It remains a very open question whether okay, does Asian America exists? right, I Think it's a category that Has some use some utility All right, a category doesn't have to be you know 100 percent Accurate, you know profound and life-changing to have some advantages that the disparate immigrant populations huddled under the umbrella of Asian American a term coined by student activists at Berkeley in 1968 have enough in common and think about how grateful we are that we get all these hyphenated American identities I think about how much better off we are Now with Asian American black American and Jewish American Dividing America up into more and more disparate groups so that we're less and less able to have a public to have You know a communal sense But instead There we celebrate that we have so very little in common. Wow, that's really helped this country Thank you Thank you so much Come on now to justify a shared politics or even a shared identity Nobody most of all Asian Americans really believes that Asian America actually exists Contends the journalist Jake Okay, so Jewish American that doesn't really exist for some people because they if they're living up from the inside They think oh, there are so many different kinds of Jews, you know Jewish just has no meaning If you're Asian American, I can imagine that there are so many different types of Asian identity that you think our Asian identity just Just doesn't have any objective meaning But from an outside perspective, all right, it does does have meaning for people, right? We only have so much energy I have so many mental resources and so we develop categories and stereotypes Kind of economize on our thinking to you know have broadly useful ways of understanding the world as being keng in his 2021 polemic the loneliest Americans How how they suffered keng Asian American Loneliest Americans gosh, that sounds really tough I mean Asian Americans the least likely to be victims of crime They earn more money than any other racial group. They save more money They live longer lives They're more educational and social success But poor dears. They are the Loneliest Americans how awful identity is a fantasy created by striving Asian Professionals eager to reap the spoils of full whiteness while hiding behind a relatively mild Disorganized form of oppression that pales literally in the face of the systemic violence Okay, so everyone in the west today wants to grab on to The title of oppression. They owe how we've suffered And that's the way to get status and prestige and power Is to compete in the suffering sweepstakes Everyone's got to do it man Is it on black Americans There are still only two races in America black and white he declares Everyone else is part of a demographic group headed in one direction or the other What interests me here is not keng's argument per se He is not the most persuasive writer on the subject only the loudest But rather the fact that both he and in Arguably two of the most prominent Asian American authors working today End up placing their ideas on the shoulders of a mixed race child All right, this is from New York magazine What about the half Asian half white protagonist in novels what that reveals About Asian American anxiety


Hi everyone, I'm Matthias, and today we will have a look at a new feature which was recently released to the Transcribers app, our publishing solution called Transcribers Sites. Now we are at the homepage of our Transcribers account, and what you can see here is first of all the recently used documents, but also the recently used collections. One of those collections is the publishing collection that we want to put into Transcribers Sites. So, first of all, let's have a look at it. First I go to Collections, and as you can see here there is a collection called Transcribers Sites Publishing and Flemming, that is the one that we prepared. If I click into it I can see the one document, that is the content of it, and if I click into the document I can see all the pages that are in this document. This green line up here basically shows you that we set the status to Done for each and every page of this document, and this basically means we are ready to publish. Right, so how do we publish now? First of all I can change from Desk to Sites by clicking on the button here in the top right corner. If I do this I end up on this particular page, and here I can either click on this button, create my first site or create new site if you already have some sites listed here. If I click on this I end up in that dialogue field, and now I can give the Transcribers Sites instance a title, so for example Flemming Diary. I can also customize the URL, so for example Flemming Diary as well, and as you can see the URL down here has now changed, and then of course down here we have to select the collection that we want to publish. As I just showed you we called it Transcribers Sites Publishing and Flemming. This is the one that we want to publish. By simply clicking I already connected the collection with Transcribers Sites, and now what's left to do is click on this button and then we can start editing. As you can see our CMS is divided into two sections. Here on the left you can do all the editing, and on the right you basically see how these changes impact the looks of your Transcribers Sites instance when you're done. So for example I can change the title to Research on Flemming Diary, and you can see the change right here on the right above the search bar. We can also add a description. I have one ready right here, and this changes basically the description under the search bar. And I can also upload an image, but please make sure to save beforehand because uploading an image needs refreshing. So if I now click on Upload Image I can simply drag and drop an image of the actual diary right here, and if I click on Submit you should be able to see it in the CMS system on the right. And now let's have a look how we can navigate through all the pages that you can change in your Transcribers Sites instance. There are two ways to go, either you can use the nav bar up here in the top right, so you can simply click on Explore, click on Search, or click on About, and then you're basically in the page that you want to work on. Or there is the other possibility of using the drop-down here on the left, where About is written. Here you can change to Home, you can change to Explore, or you can change to Search. Right, this is how you basically navigate your Transcribers Sites instance, and now let's jump to the About page. This is how it's done, and now we can of course do some work on this as well. We can add it again the heading, so for example, Hello from the Research Team, and then we have all these sections which we can change. As you can see they have different formats, so it can be just text, it can be the image on the right, or the image on the left, or it could also just be an image, and you can basically just edit them by clicking here on Edit. I can change the section here by simply saying again, Hello from the Research Team, for example, then I can say, okay, I just want to have the text and I want to have it centered, so this has now changed here on the right, and of course I can also change the description again. And now what you can see here, we have our text the way we want to have it, of course we can do some more edits, make it bold, make it italics, etc. And what you see is that you have these additional blocks, and all these blocks can basically be changed as well by clicking on Edit here, but if you just say, I want one block, that's all I need, then you can simply click on Delete section, and now the one in the middle is gone. Good, let's upload an image, let's take the diary again, drag and drop it here and simply click on Submit, and there is one more addition that I want to share with you in the About section. Of course you can delete, but you can also add sections, and this is simply done by clicking on this button, then clicking on Edit, do your changes, and then you have the additional section right here. We don't need this, so I will delete it again, and our About page is done. Well, we have already come quite a long way. Your website is already quite customized, which is great, but of course you also want to see how the search itself looks and how the documents are shown to the users. Let's go. If you're now willing to have a look how your pages look in transcriber sites, there are two possibilities to do so. First of all, you can use the Explore page, and second, you can use the Search page. So let's click on the Explore page first. What you can do here is explore and browse through your corpus. So what you can see here is the document that is in this collection. Of course, if there were more documents, you would see a list of them. You simply click on it, and then you get an overview of all the pages included. If you now want to have the side-by-side view, you simply click on one of these pages. It takes a little bit of loading, and then you see your document, the original, and the transcript that you hopefully have done automatically with transcribers. If you click into the lines, you always see also the highlights. So there's always this connection, and you can always check where you are in the original document or in the transcript. The second way of looking at your results is using the Search page. And here, of course, you have a search bar. You can put in a search term Isabella, for example. And if I do so, I get all the search results right here. You always have a snippet of the original, and you have the text that you get from the transcript, which is related to this snippet. So you can simply click on one of these results. And again, the side-by-side view is loading. And what you can see now additionally to the explore side-by-side view is basically that the search term Isabella is also highlighted in the transcript. But of course, also in the original down here and here as well. As you just saw, showing your documents and transcripts with transcribers sites is quite easily done. But you might be wondering what the tags that you added and the metadata that you put into transcribers are good for. Well, the metadata can be used for filtering, whereas the tags can either be shown in the side-by-side view or also can be browsed in the explore page. Here is how it's done. First, we have to go back to the explore page by clicking on this button, and now you can see here on the left that there is a toggle called Browse Tags. At the moment, you just see the documents right here, but if I click on this toggle, the tags appear. So I can simply click on them, and now I can see all the tags that are in my corpus. For example, I can click on Person, and then I find Marjorie, I find Isabella, and if I now click on one of these tags, the side-by-side view is being opened, and I can see Isabella here, my tag that I was looking for, directly in the side-by-side view. So this is basically how you can enable and use tags, but of course also the metadata that you have in your corpus can be used. For this, you just have to click on Search, and now you can see here on the left that there are some filtering options. So a year filter or any other filter that is basically based on your metadata. So if I now look for Isabella again, I can simply click on Show Advanced Filters, and as you can see, there is not much there. There is just a title filter here, but what I maybe want to have is a year filter. So I click here, select this one, and now you can see that here in this section, the year filter has appeared, and of course it can be used on all the search results that you have gotten. And of course from here, if you click on the result, you get straight into the side-by-side view, and you see the search term highlighted. You might want to tweak your Transcriber Sites instance a little bit more, and of course what you can do is use the settings for this by simply clicking on this button. You can, for example, change the project title, but what you mostly would do, I think, is add another language. So for example, we can add German like this, and you also have to set translations then, of course. So for example, back in English is Zurück in German, and we simply have to do this for the whole list and then save it, and this way we get the translations into our Transcriber Sites instance. You also can add an imprint and please do so because of legal reasons by simply writing it into this box. Then the imprint is basically added to your Transcriber Sites instance and you shouldn't have any sort of problems. If you want to delete your Transcriber Sites instance, you can do here by clicking on this button down there, but there are also some more settings that you can use, for example, the theme. And this basically means that you can add a logo if you want to and you can change the primary color. So for example, to forest green. And there is a little bit of a user management there as well. So as you can see, our learn at readcode.eu account is here as an owner. I could also make it an editor instead or also remove, but normally what you use it for is to add other users. So for example, m.sorg at readcode.eu, which is myself, I have a Transcriber's account, so I can be added to this Transcriber Sites instance and then work on it as well. I simply have to click Save in the end and then I'm done with all the settings. And then there is simply one more thing to do and then you're done. Click on Want to Publish up here in the top right corner and then you're ready to go. Well, that's it. You successfully published your first collection of documents with Transcriber Sites. Congratulations on that. Thanks for watching this video and please make sure to like and subscribe. And see you in the next video. Until then, happy publishing. See ya.