Good morning. Good afternoon. Good evening. Wherever you're hailing from welcome to another episode of cloud tech Thursdays here on open shift TV I'm Chris short executive producer of open shift TV We are joined by a bunch of friends here from red hat as well as a special person from Amy will tell us Amy. Hi. How you doing today? Good. Let me go ahead and introduce my compatriots first We have Josh Berkes who is the kubernetes community person here at red hat We have Mike Perez who is the Seth storage community architect and myself Amy Marish Who is the open-stack community person here at red hat and we are very pleased to announce that today? We have Belmira more era from CERN to talk about scaling the open-stack cloud at CERN Bill Merrill So, hello, my name is don't be more area I'm a computer engineer at CERN. I joined around 12 years ago Initially I was working server consolidation and all to virtualize the batch service. That is a huge service at CERN But rapidly move my focus into cloud computing and how to deploy and manage a large-scale infrastructures I'm also a member of the open-stack technical committee and also co-share of the open-stack large-scale seed and I'm really happy to be invited to it to be here to talk with you About the CERN infrastructure awesome, so cool You said you have a short presentation Is not that short Okay, just lines. Okay. I think it will be enough for the hour I can start sharing it's CERN. We're talking large-scale here, right? All right, can you see it? Yes. Yeah, come on. They accelerate things to the speed of light. That's true So that's why I'm gonna through very fast for this All right So Yeah, I can start so this session it's about how we scale Open-stack at CERN So currently we run a thousand of nodes and thousands of virtual machines And we go through through the steps from the beginning in 2013 to today but Maybe the audience is not familiar with CERN so in the next couple of slides I'll give you an overview about the organization and the role of the CERN cloud infrastructure in the organization So CERN is the European organization for nuclear research. It was established in 1954 Initially with only 12 member states And this number has been growing over the years currently. They are 23 member states Love sits in the border between France and Switzerland and it's very very close to Geneva The mission of the organization is to do fundamental research in the particle physics field CERN is the biggest international scientific organization in the world more than 10,000 scientists from more than 100 countries work in the organization Not everyone is a staff member. It's mostly people from different universities around the world So to help understand the universe CERN provides a unique range of particle accelerator facilities the accelerator third complex at CERN is a succession of different machines accelerators that accelerate particle beams to higher energies very close to the speed of light The LHC is one that you can see clearly in the satellite picture Is the CERN largest accelerator? It's also the world's largest accelerator with a 27 kilometers diameter and it crosses two countries France and Switzerland For comparison, you can see the Geneva Airport here and to have an idea about the size of this machine Here is the CERN main site and These are other accelerators where the particle being straveled before being injected in the large other collider So the LHC that's where it's not only one but two beams of particles that travel in opposite directions And they collide in very precise points that are the experiments so atlas, ALAS, CMS and LHCB So what is quite fascinating to me is that all of these is 100 meters and the ground and This is how the tunnel looks like you can see the magnets So these big blue pipes that you see here. They are around 10,000 of them in the altarno Each magnet can measure between 5 to 15 meters and can weight up to 35 tons Inside each magnet. They are two pipes where the particle beams travel in opposite directions These are electromagnets and superconducting. This means that they conduct electricity without any resistance However to achieve superconductivity states they need to be At very very low temperatures so minus 271.5 degrees Celsius Which is even lower than the temperature of outer space As you can imagine the cooling process takes a few weeks and requires tons and tons of helium. So these are the experiments atlas, CMS, LHCB and ALAS These are the particle detectors where the collisions occur. These machines are huge They have up to 45 meters long, 25 meters in diameter and more than 12,000 tons And of course everything is 100 meters on the ground Mike visited this some time ago A detector is basically a digital camera, but they can take up to 40 million picture Seconds this produces up to one petabyte of raw data every second Of course, we cannot handle all this data Our source system doesn't support this so what physicists do they have Triggers in the experiments that try to identify in real time the interesting events and everything else is discarded So at the end we end up with the few gigabytes per second that are stored for analysis Even though per year we store around 90 petabytes of data that needs then to be analyzed So this is how it looks an event after reconstruction And with all these pictures physicists can have a representation of the collision events The analysis of all these data gives physicists insights how the particles interact But the detectors are not only undergrounds at the CERN site. They are also in space So this is AMS, the Alpha Magnetic Spectometer that was installed in the International Space Station in 2011 to measure antimatter and cosmic rays and to search for dark matter All this data that AMS generates is transferred to Earth to be analyzed and most of it is analyzing the CERN clouds Okay, so this was a very brief introduction about the mission of the organization and what it does So now let's start talking a little bit about the cloud infrastructure So to process all of this data and to support the scientists all around the world CERN also provides compute resources to the scientific community Over 90% of the resources in the computer in the data center are provided through the CERN's OpenStack Private Clouds And to understand the motivation Why we build a private cloud? We need to go back through the beginning. So 29 2011 and Then we will see the evolution of the cloud infrastructure over the years and some of our architecture decisions If you have any question, please interrupt me at any time so this is the Data center in Geneva. It's a building from the 70s It was designed to have a prey super computer in the middle at that time Of course, it's evolved. It was upgraded over the years. Now. It's a normal data center This it has two floors. This is one of the floors But one of the limitations that we have in this data center is the power capacity currently it has a power capacity of four megawatts and Is not easy to extend data center. So that's why if you visit now the Center, you will see that most of the racks are not completely full Usually they are all full and power constraints is one of the reasons So this is another data center that the CERN operated From 2013 to 2019 over six years and there we Only run compute nodes for the OpenStack Clouds all the compute nodes for computing for processing were for the OpenStack Clouds, so this is in Hungary and It was a huge challenge for us because when we launch in production our cloud infrastructure We had two different locations one in at CERN Geneva and the other in Hungary So the challenge was not only to the play open stack at that time But also to run these different locations transparently and we're gonna talk a little bit more about this later And these are another location when where we run our cloud infrastructure. These are now compute containers with high density for computing and And you can see some when they were installed and this cooling Hardware being installed Alright, so this is one of our dashboards for monitoring You can see the size of our current cloud infrastructure So we have around 300,000 cores in the clouds 3,400 users More than 4,000 projects around 30,000 virtual machines. This changed a lot. We are in the process that we are commissioning a lot of hardware because We are replacing it. So that's why you see these big drop in Compute nodes and number of pms that we added beginning of the month We also have a lot of services in the cloud. So we have ironic to provision bare metal you we have Around 8,000 compute bare metal nodes Magnum clusters Usually most of these clusters are Kubernetes more than 600 And also volumes for senior you can see that we have a lot of block storage more than three petabytes So quick question from the audience Kind of unrelated, which is Can you direct our audience member to where they can look at jobs that are available at CERN? Jobs yep, I've already sold everybody Right Everyone who wants to work at CERN with y'all Um, there is a web page for jobs. I think if you search jobs turn you will immediately find it It's I'll go ahead and look it up Just do it Job CERN. Let's see. Oh, here we go careers Careers dot CERN. That's all it is. Yep. That's all you need. No, I'll drop it in Are you already there? Thank you. Okay. Y'all are faster than I am Yep All right. So going back to 2011 So that was a period of change So we are in a period where our We had a lot of computing requirements. They were increasing a lot The LHC was running There was a need for more computing resources and we had these Power constraints in the computer center in Geneva. So we need to expansion options in the different side Also, we need the business continuity It's nice to have different locations because data center doesn't run only Processing jobs batch jobs for to analyze the LHC data But also it runs all the services for the organization So having a business continuity plan and on top of that the disaster recovery So that's why CERN opened an international tender to For all the member states to have another data center in the Hungary one So that's why we got that data center In Hungary. So the project started in 2011 and the data center was ready in 2013 Just in time for the launch of our private open-stack cloud infrastructure Where in Hungary is it? So it's very close to Budapest Okay When I say Wigner, Wigner was the exact location, but it's Budapest Okay And then we are in the situation that we know that we can have a new data center also more servers But we are still using our Managing managing tools from the beginning of the 2000s tools that were developed in-house It was a time that there was nothing available actually to manage a data center of our size in 2000 so we needed to build all these tools However, to reality in 2011 was completely different. There were a lot of open-slash projects that Probably were doing definitely they were doing a better job And we did much more functionality than the tools that we made in the house and then other problem was when An organization build their own tools Attracting people to work on those tools is actually very difficult because they only have value inside your organization And when you arrive there, you need to learn them and if you leave That knowledge will not Well, it's not interesting for other organizations and companies So it was a time that we needed to really adopt opens the open-source tools available to manage our data center And so we start looking to the all the options available So why to build Also cloud infrastructure at that time everything was running on physical machines and it's a completely shift Not only out the way we managed data center, but also for all the users that's That's what we needed to tell them. Well, now we need to transfer these workloads to virtual machines And as you can imagine a lot of them were server augers They liked it to have their machines in the center and control them and this was a huge cultural change but As everyone knows so cloud infrastructure brings a lot of advantages improve operational efficiency The resource efficiency so the what the possibility to consolidate a lot of servers in one resource it's it brings a lot of advantages and Something that was quite easy to sell was to we improve the responsiveness because at that time if someone needed a machine they needed to feel a lot of forms and Maybe after few weeks or even months they will get to the physical machine to work on Having a cloud infrastructure with an API as all service. They could immediately take that machine So we start to identify a new tool chain and the things that we clearly need was a configuration management tool So there were a few options at that time At that time we decided for puppets and this what we've been using since then but that is not only used to configure your open stack infrastructure, but is worldwide used through the Organization to configure all the IT services for example Monitoring tools and there are a lot of projects open source projects that were in the define and do that we are using today so Kibana search collect the Fluent the That we are using to manage not only the monitor not only the open set resources, but all the resources in data center under the cloud manager tool, so it was a time that There was not much available When we started really looking to this this was 2009 and I start looking to open ebola Open ebola was an open source tool. Actually, we start looking to open ebola to visualize the batch system and we We were quite successful We did huge scalability test because one of the concerns at that time it was these open source tools were able to scale to our needs So we are able to create more than 15,000 virtual machines on open ebola, which was amazing at that time That was one of the options but also certain beginning of 2006 Was managing a small virtualization tool that was built on top of Microsoft systems center virtual machine manager where the CERN team basically build a web interface on top of it and it was a Basic web interface where the CERN users could go and create virtual machines selecting an image and basically Was it so there was no API interaction. It was only that web interface It was a virtualization tool, but it was quite popular so it had in 2011 it had thousands of virtual machines running in that Microsoft infrastructure But that was a time where OpenStack was released 2010 and that was a game changer. So with all the industry support to these new Clouds tool I Think it was clear from the beginning that was the right choice for us to invest on OpenStack To understand this tool And to join the community So we started investigating basically OpenStack from the beginning This is a presentation that I gave in January 2011 To my management to basically describe my findings about OpenStack So this was based on the first release of OpenStack Austin And you can find the presentation in this link. It's quite funny now going back all these years and see this presentation I'm glad that I did it Given the timing did you consider eucalyptus or cloud stack at all? A cloud stack was not there yet. Yeah, it was only after Eucalyptus it was an option but it There was not a lot of deployments running eucalyptus at least at our scale. So It was not it was never considered as a good option for us. Yeah, I started with Grizzly. I can't imagine starting with Austin Well, it was quite a challenge going back now into these slides and Revisiting all of this is it's amazing. So in this slide, this is the architecture of Nova at that time and yeah We believe that Nova was complicated at that time with these diagrams. Well, when you see it today, it's Completely different. It was a time also where there was only two projects that was Swift and Nova Nothing else. So glance. Well, I think it was only available on Bexar or Catus So everything was no even even to create users. You needed to do Nova manager user create something like this Nova was doing more than just virtual machines at that point, right? So it was doing storage even at that time. Hey, we even had networking that was being done Some people would call that the good old days of a flat network before we moved on to quantum and then moving on to Neutron So no network still works for us actually But the Texan and me has to just clarify even though it is spelled B E X A R is bear because we pronounce things funny Yeah Texans do or just We have some weird city names Bear County is one of the Thank you, Amy to raise that all right, so This was 2011 and So we needed to to get our hands dirty on this So we created Several prototypes and the goal was to adding functionality in these different prototypes So we started with with what we call guppy because it was a very small and fragile animal And you see that the animals over time get more stronger with more functionality Um Well, this was deployed the first prototype with the fedora 16 um Why because at that time there was the fedora cloud 16 that released the rpms or fedora They were not available for rel yet centers So we use a fedora because of that Also, it was a big change for us because this it was using kvm And we wanted to use kvm because we thought it was the future because until then in our open ebola Tests we are always using zen um, but that was a pivotal moment also for kvm. It was integrated on Rail six, I believe and it it was the only one supported there We started from the beginning using the open stack puppet modules um Actually, we helped them to to develop the initial puppet modules With the puppet labs at that time those funny times And yeah, that was just initially initially tests. We went then with different release And this was always closed on it for us to test and you see that this time we already moved to center six our scientific linox six Um and also hyper v so you remember that infrastructure that I told you that we had at cern running on top of these microsoft infrastructure um So we wanted to continue to support that and we at that time believe that the easy way to do it was also to have hyper v in the open stack infrastructure And then move those machines into open stack But using the same virtualization layer So you see the challenge is it's completely new product trying to scale this or to move our own infrastructure To open stack to data centers and then two different virtualization technologies um Keystone held up integration of what was also tried during this version. You can imagine that we have a huge that directory um Then the the last prototype, uh, we opened to some of our community We try to put all of the services into aj and we add in this prototype more than 600 compute nodes This was already beginning of 2013 And basically we launched our cloud infrastructure in july 2013 From the beginning it was clear um If we get serious with this we need to engage with open stack community because there is no way That we alone will be able to solve all the issues will meet the the help of the community So you see that from early on we started attending meetups Also helping the community and the organizing meetups. This one was at CERN in end of 2013 Um And this was my first open stack summit. This was 2012 I think in san francisco So a long long time ago and this was the keynote room. Is he all small? It was at that time right So the CERN cloud infrastructure. So we started using scientific linux 6. So that was in 2013 um later we moved to centos Cento 7 Um from the beginning that we are um using the rdo packaging. Um, it's great To have all these bad packages and all the testing that redact does Uh, however, we still have projects that we need need some internal packages. So we need to rebuild all of this And from the beginning that we are using the upstream public modules for for open stack So some some considerations When we start to building this so the number of compute nodes Um, we started very small with only a few under the compute nodes But we knew that we wanted to move all the data center to to open stack So at the end is a few thousand compute nodes So is this tool able to to scale to these numbers? At that time, if you look back, there are not a lot of big sites using open stacks So that was always a concern Uh different locations. So the data center that I just mentioned At the group number of open stack projects That was a time that every week was a new open stack project popping up Um, it was very hard to to follow all of this and then there was all this split up functionality that also was happening So for example, nova volumes moving to cinder Um nova network moving to quanto So this was very difficult to when we are trying to deploy a series infrastructure very difficult to manage Also, then we add a large number of users and projects. Um, a lot of users every month Leave the organization So around the 100 users leave the organization and the 100 users come back to the organization or join the organization There is always this constant move of people at CERN Um, so we needed to automate all of this Um automation of creation of the projects when people leave The organization all the project removal all these automation needed to be invented And then of course, so when having a large infrastructure All the automation that is needed to manage the infrastructure itself All the procedures that needs to be figured out because everything is new So the kind of workloads that we run in the infrastructure Um, mainly it's physics data analysis. So all the data from the LHC experiments and many other experiments IT services Uh, and then many other infrastructure that is required for the organization the Experiments services to run the experiments itself engineering services to develop different tools for the experiments Um, and also the personal VMs. So any user at CERN has the possibility to to have a project and run their Dextops their personal VMs in the infrastructure So most of our virtual machines are ephemeral They They consume more than 80 percent of all the CPU cores available in the cloud and this is what is used for The LHC data processing And for because this is a very specific use case. We have special configuration for this virtual machine. So we do CPU pass through We have no more flavors But also they are ephemeral because they are only processing jobs. So things like line migration We are not really interested for for this kind of virtual machines Then we have the pets that are all the service VMs um Where performance is less important, but what is really important is that we can keep these virtual machines running and Line migration is a huge requirement for for all of this Um, it's a multi it's a mix of operating systems that we have here. We have a lot of the windows VMs as well So are these mostly like people's personal environments So these are people personal environments, but also a lot of services most of the services of the organization so 2013 is when we finally Open the private cloud infrastructure to our users Uh, we started with two cells and cells at that time was a quite new concept. It's not a lot of people were using that um We decided to use cells instead of regions even if we had two different data centers Um, mainly because we wanted this to be as easy as possible for the users to migrate their workloads to the infrastructure Uh, I've these were users. Um, most of them, um They they are not computer service, but they need to to manage their their applications Physicists that they have their projects Uh, we wanted to be as easy as possible for them to move their workloads from physical service to calling for the structure So that's why we wanted to reduce it maximum the number of concepts So that's why we decided from the beginning to use nova cells and these were cells v1 at that time So we had one cell in Geneva and the other cell in vigner At that time we wanted to have a j everywhere because we believed it was with architecture We didn't want to have a single point of failure We use salometer um Bad idea we moved back sometime then after um Glance was safe Backends, but at the beginning really in 2013 It was actually all the images were stored on afs because the safe cluster at that time was not ready So we had for a few months all the images stored on afs And we were still doing some cinder tests HAProxy That continues and then we had We had three master nodes per per cell because this i availability ribo team q again Ribo team q with cluster with the Miller qs I think nothing special very very common architecture. So this is a diagram that you can see that we had two cells Geneva vigner And this is all the services that they're running at at that time um For salometer, we are running MongoDB for the database There was no gnocchi at that time um stack tack we we started running that at that time it was It was very good to have a perspective of the service Um, and then we we keep more or less the the same architecture. This is the top cell um The architecture of cells v1 is completely different from cells v1. So that's why you see You may not not recognize the service that we have here from the current architecture enough open stack So this was the vm growth since we launch For our users in in july 2013 And you can see that this is the cumulative number of ems that were created in cloud. This is only until april 2017 but to the pattern Keeps the same after 2017 And this is the number of ems Growing so you see that this was very well adaptive by our community and also we Work quite hard to basically move all the compute all the physical nodes from the data center to the infrastructure There are weeks that we are moving more than 100 nodes into the Cloud infrastructure and this was not new order. This was converting the existing servers um into compute nodes For a while. So when did you get Have all of the You know standalone servers been converted or is that something you're still working on? Um, so all the servers that were that are dedicated to computing uh are converted However, not everything in the data center was converted to compute nodes and not everything runs On top of the open stack one example is storage so This doesn't doesn't make sense uh to run the storage servers. Um on top of open stack so They continue Bare metal machines, uh managed by the storage team Okay, so those bare metals aren't ironic, but yet you do have ironic as well as magnum clusters in the system Right. So ironic is a quite recent product in in our clouds. So I if I remember well in 2019 it's available since 2018 2019 um Ironic started to uh for us, you know It was a requirement because a lot of people a lot some of the use cases that that we had um didn't fit well virtual machines. So people that really needed uh huge virtual machines full node virtual machine So there is not didn't make a lot of sense to virtualize Uh that environment for them because they were losing a little bit of performance So we deployed bare metal the ironic service and initially Our goal was to To have an api for for For the users to interact with the cloud with bare metal The same way they interact with the virtual machines But we had bigger goals for ironic not only that not only having a pool of bare metal nodes for people to use But also to change all the workflows In the data center and what our goal at that time was to manage all the resources in the data center using bare metal Including the compute nodes So Currently all the compute nodes in the infrastructure are managed by ironic We have this kind of inception and we have a lot of this in our infrastructure Was that your Amy? Yep So Yeah, so another follow-up question I kind of want to ask is what determines whether it's a vm A container or on bare metal? So what determines where the workload goes? So it's quite ironic that we are talking about all this inception on all of these concepts and I have the slide scale quite simple We'll go ahead and go through and and we'll we'll get to it um No, so that that is a decision of the user Uh, so the user have all these apis and if he has kota he can create a bare metal nodes Or he can create a virtual machine or he can create a container depending in the In the application that he wants to deploy Okay, uh, I I think I still have one slide about ironic. I can go then Deeper on it Okay, so as I said scale implies simple, uh, because if from the beginning, you know that we're gonna manage nodes, um The architecture the architecture needs to be simple so this is an overview about our architecture Um, something that we decided from the beginning was to isolate the open stack services. So we don't have a Few physical nodes like most of people three physical nodes and we run all the open stack control planning those nodes So we just we try to distribute as much as possible So we have machines that only run keystone. We have around 16 of them. We have machines that only run plans api Nealtrend and so on Why because all this isolation, uh, allow us to upgrade all these different open stack components independently And allow us to focus in one problem at a time Then for nova, uh Also, we have a this kind of architecture We run nova apis completely isolated Then we have the level one that is the main control plane or the for cell zero Where we have the schedulers and conductors Uh, no the no vnc proxy And placement as well is completely isolated from nova And even the databases We have completely Isolated the instances for each open stack project. So we have a mysql instance for keystone one mysql instance for Glance and for nova we have one independent mysql instance per cell And again is to have this isolation And then we have the the cells itself. Um, they are very very simple. They have the control plane Again is what it is and then uh all the compute nodes So with which this represents is that we have one control plane or only one server acting as a control plane for around 200 compute nodes And in total we have around 80 cells Can you speak about the benefits of Isolating things into different cells Can you repeat mike? Yes, can you uh talk about the benefits of uh taking the cells approach of isolating different services within child cells? Like uh, how did you reach that requirement in your infrastructure? And and also for for my sake for cells we're talking about sort of groups of physical servers. Yeah Exactly. It's a logical partition of the nova deployment So it's all can be a rack. It can be a data center But it's a couple of servers that are together Yeah, I think this slide is good for this to explain this Um, so we decided to to go through the cell routes. Um, because we didn't have at the beginning to have these um region concept concept to our users But actually cells have a lot of benefits um Basically, they can act as the failure domains Um Also, they can allow you to configure those servers in that cell in a particular way And we are Those advantages, um, basically to to deploy our infrastructure For example, you can see here that our availability zones are basically sets of different cells Meaning that if the cell goes down the availability zone is not completely down. It's just graded Okay And because we have so many cells means that the control plane that we have for each one It's only one server meaning that if that server goes down The workloads continue to run because it's only the apis for this particular cell for this particular resources that will not be available And also It has the advantage to distribute all the control plane meaning that for each cell We're going to have a completely independent rabbit in queue Meaning that the scaling rabbit in queue will never be a problem for us because The cluster it's very small. It's just that maximum of 200 compute nodes So that's why we never reached a scalability issues for rabbit in queue in over I think I have I have a slide where I go through the advantages of cells and compare it with regions That's a real question, Mike Yeah, yeah, I just wanted to give the sense to the audience of the benefits of it. Thank you um But what I would like to show with this slide Is that we also run all these services that I show you in this slide His stone glass new turn all the open stack control plane. We run it on top of the cloud itself So the control plane doesn't run in physical machines dedicated for the control plane or doesn't run in a different clouds Only for the control plane It's running the cloud itself that it manages um So that's why we have this inception Again, it's like ironic as I mentioned early that manages also the compute nodes Even if it's an open stack project So you see that in the server first side by side with the user VMs, we have keystones We have glances VMs and so on and Keystone for example and all the other servers are distributed between the different availability zones The same availability process that we give to our users We use it and also distributed between the different cells. So that's why we reached our availability for the control plane It's nice to see using availability zones. I don't think people take advantage of it as much as they should We are exposing availability zones to our users from the beginning Yeah, because you you started out with two widely separated data centers Yeah, yeah I actually weird techie question here What what's the actual line lag between uh, cern and hungry? Oh, exactly just in time, right? Yeah all right, so Between cern and vigner, it's uh, 1,600 kilometers and that translates to around 24 milliseconds of latency Um And this was at the beginning. We are trying not only to figure out How to set up the new data center, but also to Set up open stack on top of it. Then we have this legacy this latency issue as well um, what you see in this slide as well is the connections between vigner and cern So we had two network links 100 gigabits uh, bandwidth Uh between the two centers completely redundant. That's why you see two and After a few years, we also added a third one. So we had a connection With uh with a total of 300 gigabits per second between the two centers And this is basically for us. This was like connecting It was a cloud interconnect um With appearing networks because it was the same network For people that are used to public clouds Of course, uh having the data center there, um That had some architecture implications for example Um, we run the databases. We started to run the databases in Geneva, but the latency was so high And that was a time that we didn't have nova conductor So all the compute nodes were connecting to the databases at that time Also because we are using cells v1 the scheduler was for cell So we had a scheduler in vigner that were connecting to the databases in Geneva So the user experience at the beginning in vigner was not that great because it was sometimes very slow So we continued to iterate on this. So databases were deployed in the vigner data center Um Another thing was sef because the sef cluster was in Geneva Uh, so at the beginning because the latency It was a very bad experience for users to have Work storage in vigner. So in 2015, um A sef cluster the storage team deployed a sef cluster in vigner for these use case to have block storage for the cloud um and other things for example, um A glance cache Initially, we thought that 24 milliseconds was a lot of latency and we needed to transfer a lot of images to vigner So we decided to implement glance cache Uh at vigner But that turns out it's not really needed because after sometime the images are cached at compute at the compute node level So it was just an overhead that we had in architecture So we removed that and that was a case that glance was the nodes were actually contacting glance in Geneva without any issue So you see that there were different and the several architecture constraints that we were figuring out over the years So the data centering vigner was operational between Or cern was using that data center. It continues to run between 2013 and 2019 We knew this from the beginning. It was uh A contract for only four five years that was extended one more year um By the end of 2018 we were running there 17 over sales. Um 3300 compute nodes and the For the availability zones there we had the 78 nodes So early 2019 we started decommissioning the clouds in the vigner data center and as you can imagine that was Another challenge. So we needed to remove all these sales from the infrastructure And this was completely in november 2019 and one interesting part is that 2500 servers actually returned to Geneva because um they were late purchase and It was still very good servers. So and this service was the ones that were added to the computer containers that I show you at the beginning in that picture Now is wigner totally going away? Are you just moving some servers around because you mentioned so now we don't use vigner at all anymore Okay. All right. So sales versus regions So I least here are some of the advantages uh of sales and regions to try to to make it more clear why we went For sales at the beginning Basically sales is to shard the nova deployment. It's it only applies to nova. There is no other service that has cells And that is actually an issue Isolate server domains and it's complete transparent users and one of the things it's uh also logical partition for operators and allow us to to have different configurations for A particular cell does and distinguish the different cells with different configurations Which is important for us for example for the batch use case That has a complete different configuration for when comparing to the service cells Regions, uh, they are completely independent open stack environments Um, so we need to basically deploy all the services all the open stack projects in the point independently in that region but you have that Fault tolerance. It's a completely different environment that should be managed in completely isolated way. So that is the big advantage And actually now we are running multiple cells by multiple. I mean three cells so in 2013, um for us it was more simple to manage one small cell than two small cells However, when the infrastructure, um grow to to at this point It's more simple actually to manage Two or three small clouds than one big one And one of the main reason reasons that also we move into regions is because neutrin um, neutrin doesn't have this logical partition with cells and It was a big point of failure. So if neutrin was down or our anything was affecting neutrin Uh, it it was visible in all the clouds. So all the users will see this And partitioning neutrin between two three different regions Allow us to to improve the liability of the cloud a lot Neutron agents are quite chatty with ribed MQ cluster So needs to be a very big ribed MQ cluster and that is always an issue to maintain also the new The regions that we have now They are per use case. So one region is focused in the it services and user VMs The other two regions is more for or it's only for LHC data processing. So they only have that use cases Which for us logically also makes sense the way we manage this So the way you're describing neutrin it sounds like a downgrade. So what are you getting out of upgrading? Well, we we were forced to upgrade to neutrin because nova network Is not supported anymore However, for the old regions, we still we are still running Nova network. So we have six six cells Where we still run nova network And because we still we are still evaluating how we're gonna migrate this is quite scary migrating from nova network to neutrin without interruption of the VMs Just thinking about that the VMs could Lose network connectivity for some time. It's very scary for us And those cells are running important service for the organization. So that's why we are still trying to figure out how to How to do it in the right way So simple doesn't always Um plays well with the availability, but we try to To work around this. So that's why we have multiple cells for availability zone which allow us to If a cell goes down to for the availability zone not to be completely down Which is a very good feature um The cell control plane is not highly available. Um If it dies There is no workload interruption just people are not able to connect To to their VMs using the open stack apis or to open stack operations Using the the apis in that particular cell. However, um These simplifies a lot the the deployment Because we have around 80 cells If we had availability through all these cells control plane We will have a lot a lot of control plane to manage And then as I already mentioned, we also run the The control plane on top of the cloud itself Right in queue. It's very challenging to scale and maintain um So we try to not run right in queue clusters at all and what we find is that um If we have very small clusters like we have per cell right in queue is quite stable Um, and not having the complication of right in queue clusters simplifies a lot of the deployment and operations um Do you want me to go faster? I don't know if uh We have a Hard line to finish this. How are we doing? I mean, we're fine on time. You can go over if you want no problem Okay I'll I'll try to go faster now So my sequel databases, um again, like rabbit in queue um We don't have a cluster for my for the mysql databases So we have independent mysql instances and the the funny thing is most of them run on top of the cloud infrastructure So the storage is in a net app solution But the instances itself most of them run on top of the cloud infrastructure except few exceptions that we have to For bootstrap issues that we run at them on physical servers so open stack And when we have a cloud infrastructure with where we want to have a lot of functionality that translates to a lot of open stack projects And these are the current open stack projects that we run And you see the version that we have Now in our cloud and you see that they are they are in different versions And is this isolation of the deployment that allow us to have this So nova is still in stein one of the main reasons that don't be still in stein is because we still have those cells running nova network And upgrading now. It's very very risky. We have a lot of patches for nova network to continue to run um But you see that most of the services are running very very um recent releases And that is one thing that we always try to keep up with the open stack release cycle um Having so many opens many open stack projects and managing all of this and Most of them manage Thousands of resources For example, cinder manages thousands of volumes with the petabytes of storage behind It's always tricky. For example, ironic. We have now around 8 000 nodes that are managed in ironic And we started reaching scalability issues in ironic um So again, that is one of those things that you you get if you're scaling the infrastructure And um, fortunately, there is this functionality conductor groups that is more or less like nova cells in ironic And now we are taking advantage of this. So splitting logically the ironic deployment Scale also means staging Even if we try to upgrade most of the services every six months The configuration is always changing. Um So the configuration through the models is always changing. So trying to have a CICD testing everything before we deploy it to production is quite important And we have a staging process to have everything first on pre-stage testing In a small number of nodes and qa and then going through different master levels until reach everyone in the infrastructure Test stack is what we call to our testing infrastructure very few few nodes to test upgrades and new configuration options Scale also translates into automation Um, we are using several projects for automation. Uh, for example, rally to grow the infrastructure every day Rally deploys thousands of virtual machines in infrastructure Just to make sure that every cell is okay and um Everything is running as intended Uh, run deck is um a project that we use a lot for operations For example, we have different teams Um, the repair team, for example, doesn't have access to the open stack resources Uh, however, we have all these procedures and run deck jobs that they can trigger For example, when a node is uh needs a repair They can trigger a job that will basically try to lie migrate all the instances in that node notified users If that is not possible that that node needs any repair intervention And then we have mistral mistral is also an open stack project that we use for workflows. For example Uh, for all the projects creations and all the projects removal when a user leaves organizations So going through all the resources from users and make sure that they are deleted And all this is automated Scale also means permanent changes So upgrades through the open stack release cycle is every six months. Um, so we run 15 open stack projects So as you can imagine every It's almost an upgrade day for us Um, and then also we have the open operating systems distributions upgrade So we started with scientific Linux six Uh, at some point we needed to upgrade to a center seven. This is never easy um There is no easy way to move from six to seven It's a required reinstallation in our case And now we are facing again the move from center seven to synthesize and stream Um, and we are working on this Hardware commissions so every around five years Copy notes need to be the commission um And as you can imagine a lot of live migrations need to happen To try to do this transparent to the users. So recently, uh, we just Removed around Um Or we migrated around 900 virtual machines because we are commissions some cells and we continue to do it Um, this is a lot of work. We wrote a recent blog post. You can follow our work here Security, um As you all know so meltdown specter a couple of years ago created a lot of fuzz so we needed to actually reboot Um, most of our cloud infrastructure because of this Um, also disable hyperthriding Uh, reducing the number of cores available And these operations when you have thousands of nodes, it's a lot of planning A lot of work Currently for kernel upgrades We are trying to automate this because when we have all these these compute nodes running they run for years and Operating the kernel is quite difficult without Resurrecting the user. So we are trying to automate this to basically having a tool that continues live migrating instances in the infrastructure And when the compute node is empty Just reboots the compute node for the kernel upgrade Scale it's of course teamwork. Um So at CERN the core open stack team, it's, um six seven people But over the years we added the participation of thousands of Different students fellows project associates um that joined the team for Few years of time and contributed a lot for for this project So these are my slides. I Didn't intend to take so long to to go through them No worries No, this was great Yeah, no for people who didn't know what CERN was I think they got a really good understanding of it And you know the infrastructure the fact that you are running different versions of open stack based on what project Is something that most people don't do and y'all have really good reasons for why you're doing it Right, so I'm happy to to answer your questions. Um Also, the audience they they can follow me on twitter. Ask me questions there. Um some questions through the email I'm happy to answer them Wonderful great Thank you so much Belmero for joining us today. This was really good Very informative. Thank you Never would have expected the number of problems CERN has with infrastructure Uh question just came in though. Yep, I got it Have you ever considered any other bare metal provisioning other than ironic? So Ironic was a quite easy decision for us because we had all this Infrastructure based on open stack. So it was the natural choice for us Also having the possibility to have exactly the same api to create virtual machines Um And bare metal nodes it was quite attractive and it's I think is a real advantage for us So the user doesn't need to Learn a different api a different common line to this So ironic was always I think the The most attractive solution to us I know we're gonna have a little bit of a lag, but go toss one. Did that answer your question? Well, he's responding. I do have one of my own since I'm coming here from container land, right One of my questions is Are you already running some workloads that are distributed as container images? And if not, are you planning on it? So at CERN there are different teams that are using containers for To deploy their applications. So as you saw in the in one of the slides We have more than 600 Magnum clusters and most of them or almost all of them are kubernetes clusters So there are a lot of applications Using containers as their deployment method. We also start playing with the containers to deploy open stack We recently we are experimenting with one region trying to deploy open stack On top of kubernetes using the elm charts Open stack elm charts and We are experimenting with it currently for example in production. We have All of the glance requests going through a glance that is deployed In a kubernetes cluster That answers your question trash Yeah Out of just sheer curiosity. I mean how many maintainers of Open stack do you have like I'm always curious about team topologies and size and stuff like that So we are seven core members that um But then we have all these fellows and project associates that joins our team But usually they don't do operations and they do more Investigative work like looking for different projects Evaluate different open stack projects or kubernetes associated projects And then if we think It's worth to to invest in those projects So then we go further and we try to to implement them to deploy them in the clouds Awesome, so another Go ahead. So let me just Continue in that in your question. So currently we have Some people doing work on GPUs trying to understand how to to have GPUs in the clouds Other people looking to how to have functions as a service in the cloud for example We have all these different projects That are always going on I'm I can only imagine how many different projects are going on at any given moment given the resources that are available um Have you collaborated with any other Scientific organizations about your infrastructure specifically Oh, yeah, sure. Um So at the beginning we collaborated a lot with nectar for example that is um scientific scientific research network in australia And at that time they were using open stack. They are still doing And they were quite big And we changed a lot of ideas out to deploy open stack More recently we collaborate with the Ska that is the square kilometer array Basically is the biggest or it will be the biggest telescope telescope in the world The sites in um south africa and also australia for observation And we did interesting projects with them for example, um preemptible instances um So they are not available in open stack by default. So we collaborated with ska to develop this um Also running Kubernetes clusters on bare metal. There was a lot of work that was done in collaboration for example with ska In this area cool Awesome. All right. So I drop links to both Nectar and ska in the chat if folks are curious about what those organizations are all about If there's Anything else feel free to reach out to me question wise short at redhead.com. I can pass along to bill murrow and team here But without any further questions, I think we'll wrap up here. So thank you very much bill murrow. This is awesome presentation Thank you so much. Yeah Thanks bill. Thank you so much for having me Yeah, nice seeing you All right, take it easy and stay safe reminder folks red hat is having a recharge day tomorrow So we will be off the air completely and monday is a us holiday.


What's up guys? I'm Brian Sekawa joined once again by special guest Robin. You're watching Key Spoke Style and today we're talking about the best first date colognes. So when I started thinking about this video, I thought it was going to be pretty cut and dry. You know, I'll pick five fragrances that I think would be good in a first date situation. I'll talk a little bit about them. Robin will give some commentary on each one from a woman's perspective and at the end we'll all come out having learned something. So when I brought up this idea to you, you were like, it's not that easy. Hold on. And I was like, yes it is. And she was like, no, it's not really. So we're going to start off with the female perspective on first date fragrances.  So when you first said something to me, I said the first thing is that most women don't care about the fragrance on its own. It's mostly about if I like you and we're on a date and I could smell your cologne, I'm going to connect the cologne with you versus have an experience with the cologne by itself. So the cologne will like sway you one way or the other? I mean, I think if I like you and I get a whiff of it and it's a good cologne, it'll maybe sway me. But I don't think on its own the cologne would be like a make or break. You also mentioned something that I could have like potentially like a negative connotation to. Well, if I think if I don't like over time, if like we were dating and we broke up, like your scent, if anyone else had that scent on a date, I might have a visceral reaction to it. Okay. All right. Like an old high school boyfriend or something that used to wear drug car or like sometimes you wear something and I like the scent, but it reminds me of like a family member and I don't want to like connect with you and think about a family member at the same time. You know? Never a good thing. Yeah. All right. So now that we've laid down the foundation to start with, fragrance is not the most important thing on a first date, but could have some sort of effect. I did actually go through my collection and choose five fragrances that I think would be good picks for a first date. I chose a bit of a range all the way from fresh to what I would call sexy. And I did try to choose some that were a bit off the beaten path without being too kind of wild and out there. You know, just so it maybe be something that a date might not be familiar with, have a little bit of intrigue while still being kind of, you know, inside the box. So let's get to them. Okay. So now the first one I chose is from the Zenya Sensei collection. This is Italian bergamot. It's very clean, very fresh. To me, this would be like a very safe, very conservative option for a first date. What do you think? I like it. It's very clean, it's subtle. I feel like this would be a nice way. I think this is what I was wearing the first time we went out. Really? Yeah. Okay. Good thing I said I like that. All right. So now keeping in that kind of fresh category, we have an all-time classic Creed's Green Irish Tweed. It's also very fresh, very clean. To me, this has more depth than the Italian bergamot, and it's not quite as light. I think you know this one. Yeah, it's clean like the other one, but I feel like it's, I don't want to say stronger, but there's definitely something different and more unique about this one. Right. It has ambergris or ambergris, which is something that helps the fragrance have more of a staying power, so it won't wear off quite as quickly because of that. It's not overwhelming, but they last all day. It's nice. I like this one because it's a little more unique than the Italian bergamot. I mean, this is like nice, and like I said, it's safe and conservative, but this one has a little bit more, so if we were on a first date and I wore this, I'd be into it. Yes. Maybe you. Maybe me. Okay. Next one here is by Amouage, and this is called Dia for Man. This starts to go more towards what I would call the classic man kind of fragrance. I wouldn't call it a fougere, but it has some of those qualities. It's kind of like aftershave-y, but like a way more elegant and light version of that. I know when we spray it on here that that. Let me smell in the bottle first. Okay. Well, when I spray it, I know it's going to seem a little overwhelming, but on the skin, in my experience, it's much more subtle. I think I even mentioned it in a previous video, being a very good first date fragrance because of that. In the bottle, it had like a powdery smell to it. Is that right? It smells very different here. Just like when you go in the department store and you smell everything. I feel like it's kind of that smell. I mean, I like it. I don't know that it's as, I like the lighter ones, I think, a little bit more. I think this one, when I wear this, it's when do you wear that? It's definitely lighter. I think it's a good, it is a good date fragrance because like you're going on a date, like you don't want everyone to be smelling you. I think it's like, fragrance for me is like something you might, you should be like sharing with the person you're with. And this one, I think really invites like you to like lean in a little bit. Not too far in though. It is a little stronger than the others. Right. Yeah. All right. Next up is a more classic Fougere from Roja Parfum. This is Scandal. Super classic, very old school. This one to me is kind of a no-brainer. It's very masculine and it's a great take on this style of fragrance. It's almost like spicier. Yeah. I like that because that is stronger than the other, the first two, but it's not overwhelming. It's still subtle in a way. I like that. Okay. Yeah. It's like, for me, this is like that classic kind of after Shavey, like your grandfather, but not your grandfather. Yeah. That's nice. I like it. All right. First date approved. Winner. Finally, we have the Sexy. This is Tom Ford Tuscan Leather Intense. The name speaks for itself. It's very leathery. This for me would be the one that, you know, if I thought we had a really good connection before the date that I would totally wear. Like, this is the one if I really wanted to get in your pants. Well, that's nice. You could get in my pants with that. I know what I'm wearing tonight. I don't know what that is. It's really leather. It's more rugged, manly. Would you say, so would first date, how would you know you're going on a first date trying to get into someone's pants? I mean, I don't know. I think either of these are a good option. It kind of depends on the season, of course, the time of day, where you're going, which one I would choose. All right. So those are my five choices. Now, there is a very important question that often comes up in the conversation about first date colognes. And that is, should you even wear cologne on a first date? That's a good question. I'd say, well, if you're, if you know you're getting in their pants, maybe that one. But if it's something classic and very light, I would say yes. But I'd also say first date, you don't know like how small of a room you're going to be in. If it's going to overwhelm, you don't want it to take away from you on the first date. You want like you to shine, not me to be thinking about like this guy's cologne versus what you're saying. You want to be memorable, not the fragrance. So I think we agree on this point that if you are going to wear cologne of some sort that you need to wear it tastefully. This is more. Right. This is not a time to be too much cologne guy. So we're talking about maybe one spray like, you know, right on your chest, on your bare skin, right before you put your shirt on, just to have a little bit of something, but not anything that's going to overwhelm or be a turn off to your date. And on the chest versus your clothing, because you know you're going to hug the person at the beginning of the date likely and say hello. So you don't want to like just sprayed a lot of cologne on your clothing and then you go to hug someone and it could kind of come off on them. Right, right, right. I always think that fragrance has the ability to be like mysterious and sexy in some way. And those feelings that it could potentially bring about with the lady you're on the date with only happen if it's kind of on the more subtle side of things. So as I mentioned, these five fragrances are from my personal collection. I would love to hear some of the fragrances that you all think are great first date fragrances. Leave those down below in the comments. I will put links to each of the fragrances I mentioned in this video down in the description. Big thank you to Robin once again for sitting down and lending a woman's perspective to a very important topic that a lot of guys ask about all the time. Thumbs up if you like this video, hit that subscribe button and until next time. Thanks for watching and stay tailored.


I'm Rajni, like the host just introduced. Today we are going to be doing a discussion on the impact of Gen Z on brand building. Do we have a lot of Gen Zs here? I think I see some. Maybe you all can wave out, show your enthusiasm a little bit, because that's what we're going to be talking about today, right? About how y'all are impacting brands who've been here for so many years, and they have to change their marketing strategies to sort of reach out to you, make sure you consume their products, buy their products, et cetera. So here for all the Gen Z in the room, guys. Of what we know as Gen Zs, we are all digitally native, right? We've grown up with internet in our lives. We adapt very, very quickly to technology. And if we can sense an influencer collaboration from miles away, if someone is trying to sell us something, we're just like, please, guys, stop it. I know you're trying to sell me stuff, right? Marketing to an audience like that who's so aware is quite a bit of challenge, especially for brands who've been here for years. And they are all learning newer ways to sort of reach y'all and understand the audience better. So today, that's what we're going to do with the panel here. They're going to break down what they see all as consumers. What are the challenges we face in sort of marketing to Gen Z audiences? And then what are the good parts about doing all of it together, right? I'm going to open with Pradhania. I think let's delve a little bit in understanding Gen Z first. So how does Gen Z differ from millennials, boomers, all of us sitting on the stage in terms of values and priorities? And how do you think that affects the purchasing decisions? So I would like to call out the three things mainly in the Gen Z. One, that they are digital natives, right? So they are much ahead and they are more savvy than a million years. Number two is I think they are a silent generation because they are discovering, learning, revalidating, and being on a social platform, online platform so much that they are making judgments also. So therefore, whatever you have to really understand what they are up to, to really make a difference and start having an impact. And the third thing is they just don't have patience.  And the last study was 2015, which is eight seconds. But I don't think it's eight seconds. It's just a split of a second. So it's really challenging stuff. Eight seconds is what Meta's trying to sell to you. And the thing is Gen Z's are from 12 years to 27 years. And 12 to 14 will behave differently from 14 to 16 and 16 to 18. So there are so many cohorts within that Gen Z. So it's like you have to have a different strategy. Just for an example, for Samsonite, it would be mainly 24 plus. And therefore, what is it that we are doing? It's basically Gen Z is all about authenticity. So we are taking the route of tell what you actually mean, walk the talk. And we are taking this realistic approach and our authenticity to show what we are. While for the American tourists from, actually, we have a primary age group also. So we are doing a lot of stuff, which is at the last mile, how we can make a difference. Simple like a 12-year kid can get to know, scan a QR code, and check what they can pack in a bag. For a 20-year-old, especially those who are looking for going abroad, we have a different type of content and the products introduced so that they can pack more. Like a volume is their best concern. And that is what we are providing them. And all that communication is provided to them between July till December so that they can reach out to. So there are like 1,000 ways. And we are doing a lot of, it's never enough. But yeah, we're trying to put in our efforts in each way to have us impact and get their mind space. That's good to know. In fact, purchasing decisions of Gen Z is something, it's very, very difficult to predict. So it's good that you're trying multiple things. That's how we get it. My next question is for Sanjana. What are some common misconceptions about Gen Z that brands should avoid? We've seen a lot of examples of force-fitting Gen Z into this certain type-casted audience. What are those misconceptions if you can highlight some? Thanks for the question, Rajni. So I think a lot of us believe or a lot of brands even believe that Gen Z has this fleeting attention, like we just spoke about, even in terms of, let's say, sticking loyal to a brand as such, I've found that to be not true. I think if you are being authentic, right, like was just mentioned, if you're being more authentic, more real, walking the talk, like she mentioned, and on top of that, if you're providing them the right value, they're just very value-conscious. They're constantly seeing and evaluating your brand that if you're giving them their money's worth or they'll definitely switch, right? How we try to go about that at Triya is essentially that it's not just that, okay, we made a sale to you of a hair treatment plan and then, okay, you're on your own. We are providing a lot of value in terms of free follow-up consultations. Is there a doctor intervention needed at some stage if they are not seeing results with the brand, right? And all these are additional complimentary services that we try to provide to keep this audience hooked to us.  If you tell them you have to take this treatment for eight months, 10 months, they'll do it, right? They'll just stick to it. Gen Z will need a lot more hand-holding and support and constant reassurance that this will work, constant motivation that this will work. You know, why don't you just stick to it for another month and you'll see the positive outcomes for yourself. So I think it's just about providing value and being real with them and they'll definitely be loyal as well. That's quite right because I think very recently I heard that boomers had a lot more income at the age that Gen Z are right now in terms of disposable. So definitely they need to be value conscious about everything and I think Gen Z also has a lot more options available, right? Which we didn't or boomers didn't have in their times. So great, thank you, Sanjana. Next one for you, Tashar. How is Gen Z shaping expectations for brand communications, you know? Like, for a brand like Bislady who's been around for so long, how do you align yourself to a newer audience, a newer messaging and sort of attract and reel them in? So, I mean, I've kind of lost my voice because I was at Lola Polosa last week. So I'll try to make myself heard. So yes, did what the Gen Zs do, right? But for any brand today, you know, especially a legacy brand, there is a transformation of the brand which happens between generations. And it's important that you understand the next generation as a brand as well. One of the things about Gen Z and the role of water is what we've kind of researched with them is what does hydration means to Gen Z, right? And they would ask a question like, what do you do to keep healthy? So there's two things. We exercise and we have water, right? So hydration is a very, very important point. It's beyond the functionality of water just to quench your thirst. It's a way of life. It's about staying fit. And that's when our journey began with sports marketing, you know, a couple of years back where we associated first with the pro-cam marathons, right? And then we launched this campaign called Carrier Game where we had four award-winning national athletes on board and we made a DVC, right? Very high octane talking about what the role of water is in a sportsperson life. And since then we've forged many sports associations. We have four IPL teams as hydration partners. We have five ISL teams. We did the national games. We did table tennis. We did Commonwealth. Now we've just done Dubai marathon as well and we've got five teams in the Dubai IoT20 as well. So we are changing the role of water to something which enables peak performance, fitness, health. And in the post-pandemic world, that's a concern with everybody, not just Gen Zs, right? And we've taken that mantle and there is a lot of exciting stuff coming from us. Now we went a step further. We are saying that, you know, how do we make everyday hydration fun? And that's when we got Deepika Padukone on board as a brand ambassador, right? She's a global youth icon today, right? And who better to spread the message that hydration is fun, hydration is cool? Water is the only beverage, right? You don't need anything else. And we got this fantastic video which was shot by Prakash Verma as well. So we are going all about Gen Z as far as brand communication is concerned, right? And we're getting really good response as well. All our online metrics have got a 98, 99% approval metrics as well. So I think for a brand to be future ready, you have to also talk to the consumer of the future. That's very interesting. I think what I took away from that is more of aligning with where your audience lies in their interests, like in sports, in finding the right key KOL and sort of getting there to meet them is how to attract Gen Z. I think for Chirudha and Ravi, open question, Gen Z is an important audience, right? For both cosmetics, L'Oreal Paris in general and for Tim Hortons considering the hottest international brand in India, right? What are your biggest challenges in reaching Gen Z? Considering you are a brand that's already Gen Z-friendly, but what are the challenges that you face if you can highlight that for us? I think the biggest challenge is learning the language and the slang, but jokes apart. I think this is a generation that's grown up on a lot of awareness and a lot of acceptance. So I think these two factors present both a challenge and an opportunity. Why I say awareness is because the world is connected. They are ahead of beauty trends even before we can tap into them. They know the 20-step Korean skin regime before we launch any products in that zone. So they constantly keep us on our toes and we have to reinvent in terms of innovations, the new products that we get into market, the claims that we have on our products, which influencers are we collaborating with, how digital first are we? And I think our entire strategy then is revolving around that because while this generation is just coming into adulthood, it's very critical for us to portray them and keep them at the forefront of, as our core brand TG, when we design communication. That's one. The second thing is about acceptance, right? Because it's a woke generation. There are a lot of different kinds of cohorts within the generation. So you will have extremely opposite cohorts and both coexisting, right? So people who have maybe never read books to those who are complete geeks and all the different types. So you have to cater to the vegans, you have to cater to gluten-free requirements and maybe even more so for you. But the challenge and the opportunity is how do you keep abreast of all these new trends, have your products integrated and have those claims on packs, build your credit as a brand. But at the same time, through the use of digital marketing, you can very much shop shoot these different cohorts. And I think that's a beautiful opportunity for us and we do that at L'Oreal. Yeah, I'll just add to what Charuta said here, right? So when we all talk about what is our TG and for a brand like Tim Hortons, right? When we define in a typical marketing manner, right? Oh, 18 to 35, urban audience, you know, upper NCCS and that's the general demographic that one speaks, right? And like Charuta said, right, within this 18 to 35 also, there are a lot of sub-TGs and one of them is Gen Z, a higher TG or higher consumption, cohort is still millennials. But even in today's day, there was a time when we used to say that coffee starts maybe at 24, 25, your teeth will get black or we are a T nation by large. But now, I mean, I've seen my cafes all the time, even 12, 13 years old are having a frappuccino and black coffees and that's how the habit has changed. And I think the challenge and opportunity lies in this single statement that they are brutally honest and authentic. You can't, as a brand, you can't bluff around them. And let me give you an example, right? In our cafe, we generally do a very quick and dirty test with our consumers, right? When we're coming with a new product, we just go around sampling it and we take a response. And my brand manager was doing that and I was sitting there and observing what responses are coming. And there's this group of millennials who were, and the drink was popcorn latte. So we were trying something different. We wanted to come out with a winter offering and there is a coffee with a popcorn on it. And there's this millennial cohort who said, yeah, it's good but I would still prefer my latte and cappuccino or hazelnut americano or whatnot, right? And then there is this group of young, maybe 13, 14 years old and we went to them and they were like, I don't like it, what is this? I don't want popcorn in my coffee, man, give me a coffee. So they are brutally honest and that was pushes the boundary to the brand as well that you can't, there were days where things were brushed under the carpet if I can say that, but you can't do that anymore. All this habit of the back of the pack coming at the front of the pack, reading every label, being conscious about every calories that they are having, every interactions that they are having, that has to be brutally honest. Let me give one more example, right? We speak about sustainability and Gen Zs are very sustainable and everything, right? From the product to the packaging to the experience.  What is the carbon footprint that you have recovered? What is, okay, your cups are paper, but what about the straws, right? And they're observing all this and that really pushes the boundary to the brand to go more authentic, more truthful in our communication, in all our approach. So I think that's both a challenge because overall the industry has not evolved to that stage as yet unfortunately India, but there are ways in which all the brands are working towards, but it's also a great opportunity because going ahead, the products, the brands, the services will get much more truthful, much more honest in their offering. And I see that as a huge opportunity lying ahead of us. I think Gen Z is really pushing brands to do better. And I think that's a wonderful thing for our world in general, at least for the generation that's coming. We'll quickly move on to sort of understanding now what are some effective ways that you'll have seen, you know, opportunities or things that you all have done to attract Gen Z and how that's worked. So Pradneya, Sanjana, if y'all can add on a little bit on this on some effective strategies that y'all have used from your business lens because I think both of y'all come from very different backgrounds. So the first and foremost would be social media and mainly the Instagram, which we have used really well for the Gen Zs and all about social media is all about two way communication and we're trying to be true to that. Having many, many, many interactive formats so that there is a lot of interaction engagement with the Gen Z. The second thing is, you know, using the creators to generate the content because anyways, the Gen Zs are from the creator economy and who best to entertain them if not the creators themselves, you know, themselves. So we use a lot of these creators to have the smaller form of content and the third thing is tech, you know. We are still work in progress. We have not used it to the fullest. We are not yet experimenting here. We are because that's require a good amount of capex but a smaller tech like, you know, the interactive platforms, the filters, the remixing on Instagram. Right now we are talking about the sensory tech, you know. So all these platforms we are surely exploring so that we give them the better experience of the communication than just a piece of, you know, 10 second communication. Yeah. Quite an interesting one, this one for me, have built a lot of learnings on this lately, right? There's enough been emphasized about how this generation is very aware, right? Now what that leads to is that they question every communication, okay? I have so many Gen Z customers and even we want to question testimonials because they're like every brand has these on their website now, right? These are shot by you, okay? So then we said, fine. We stopped shooting them in a studio. We said, we asked our customers to, you know, send them, send us their videos self-recorded on their phone in their natural home backgrounds, tried those, they said, no, it's still, you can hire anybody to do that, right? Then we started talking more about Google reviews that, okay, this is not now our testimonials, right? But now they also question that, that it's not so tough to plan fake reviews by your own team on Google, right? So now we are having to take steps beyond that, like doing clinical studies with third party authorities that are, that listen, here is our customer base. You do a study on how much results they've seen, what not, what's the efficacy like and you publish it on your portal. So we're having to really evolve in terms of just making them believe that this works, especially in an industry like hair loss, because people have tried everything before they come to try out. They have gone to a dermat, they have tried homopathy, they have tried mommy can use hair, right?  So now this is not a generation that fall into the trap of claims like, okay, within seven washes your hair fall will reduce by 80%. So it's also the fact that I feel we are refreshingly honest at Triah in terms of saying, paanch maheena lagega. You have to stick to this routine. It is going to be tough. You will need the patience, but this is the only way to kind of go about it. Talking beyond that, I think with Gen Z it's also experimenting with a lot of ad formats, like going beyond a face to camera. It's not enough to grab their attention for long, especially when you have long format explainer videos about a treatment like solution, right? So what we realized is that things like podcasts work really well for us.  So it's a lot of formats and then the finer details. I think it's with them, it's not enough anymore to just focus on the right media and engaging content. It's so much in the finer details, like the thumbnail design for your video, having click betty copies on the headline and description. Look at brands experimenting and getting so creative with just emojis in the push notifications you guys get on your phone now. Because insane results in terms of spiking your CTRs, right? So it's a lot in the finer details like the hashtags also you're using for YouTube SEO and stuff like that. So yeah, I think we try to inculcate a culture of a lot of creative experimentation at Draya and even A-B test creatives in this audience to see that what works and what doesn't. I think all the Gensis here, you all are really making the brand's work hard. This is like really tough. A-B testing creatives and copies is like the most excruciating task for an agency, trust me. Great, moving on to Tushar. Can you highlight some emerging trends, tech, that you see that brands are connecting with, especially with data, personalization, everything that you're seeing, expecting, they're expecting even more of that to happen. Any of that you could highlight for us? I mean, when you think about water, the first thing which comes to your mind is distribution.  How are we gonna, you know, transport it around? And one of the things that if you look at Gensis today, everything is online. Everything is required at the click of a button. Everything is required yesterday, right? How do we address that as a brand? And as a water brand, right? So we started our journey with our own D2C app during COVID, in fact, where water became an essential requirement for everybody. It's called Bisleri at the doorstep. But when we delve deeper into the lives of Gen Z and what it's all about, and we came across with a campaign called a Bisleri apne app aega, where a Gen Z spent a cat, actually, it's so easy to order water online, right? And that business for us has grown manifold year after year after year. It's really taken off, and that's where most of our gallon or 20 liter jar, a lot of our sales have shifted off to, where a lot of Gen Z are ordering that. The second growth lever for us when it comes to these online spaces, quick comms. So when you look at the Blinkwits, the Instamarts, and Zepto's of the world, right? That's where a lot of our smaller packs are going, foreign home consumption, the parties, and the digitalization trend for a category like water, almost three to 4% of our overall volumes have now transferred digital for a category like water. So when you look at the other trends, when it comes to tech and Gen Z, of course, digital is where Gen Z are today, connected TVs on the mobile phones. So you will see a lot of shift of media expenditure also happening over to digital bit, and but what will also happen is that for, especially for an FMCG mass brand, it has to be an integrated campaign. Campaign cannot just run on one channel. The same message has to go now on TV. For your consumption audience, you're 30 years old, 40 years old, especially if you look at mass Bharat, right? That's where your TV is there. But Gen Z Connect, it is digital. There is a lot of experiential play which is also going on today with events coming on to four, everybody's out on the street. So one campaign has to have multiple legs which run across different media vehicles, right? It's just not a question of being present in one media vehicle today. That's great. Would any of you would like to add on to this question in terms of trend, tech? I think it's a relevant question for all brands nowadays. Yeah, just building on to what's been discussed, I think it's about integrated media and integrated communication. So even the pieces of communication that we design, and I have seen this in my last 15 years, how things have changed from a 42nd one ad that we would make, a 40, a 35, a 30 at best, we'd ended there. Now, 30 seconds are something that we don't even think of, even on TV, because it's just that nobody has that kind of mind space. So it's getting shorter and shorter in terms of the average commercial duration as we speak. And the messaging also is much, much more mass personalized. So there is a recent campaign that we have done on hair color, on L'Oreal Paris hair color called My Hair Color, My Expression, with four digital stars, Kirti Kulhari, Bani Jai. Yeah, with the four more shots, please. Yes, yes, yes. And the content that we have created with them is not just the basic TVC, IMC, but even their individual stories on hair coloration, their authentic stories on what is their relationship. And for somebody like Kirti, she's always been very, very experimentative with her hair. But for somebody like Shaiyani, she never colored her hair before the shoot. So we then used that communication we created to tap into those cohorts of people, people who buy hair color online, people who have never bought hair color online, extremely experimental people. So currently there is a huge trend happening in the hair color space in terms of bright, vibrant colors. So you have the entire gamut of audiences and that's how we are really tapping into the different cohorts. Like I said earlier, it's a huge opportunity. They keep us on our toes really there. So like going from one ad to now making for different cohorts, I think that's been the change. Anyone else would like to add on? I'll be moved to the next question. I think this one's open to all of you. How do you see yourself, future-proofing yourself to stay relevant? Because it's not just about making one big moment, catching their attention because the span of attention is like, so how do you stay relevant throughout? And if you are future-ready in terms of your strategies to stay in touch with Gen Zs, I think we can start with you. So I think the number one is we all have spoken about it, but I think one is the relevancy to be relevant all the time, and therefore you have to continuously know it or continuously offer something which is relevant at that point in time. There are many occasions which are happening, there are stay occasions which are happening, therefore we coming up with a smaller day bags or the smaller holiday bags, you know, that's how you stay relevant in terms of the product offering also and the communication also. Second thing is about authenticity and credibility that you have to have. Whatever you're talking about, a brand promise and the values needs to be showcased and needs to be put up there so that they, and on a continuous basis, the third thing is about, you know, being there in a shorter format, you know, because and therefore all those teasers, which even I never would like to, you know, watch teasers, but doubt is I believe in teasers, therefore if at all they want to watch more, they can always go on a longer format. And the lastly is today, even from the first standard, kids are, you know, kids are taught about the social responsibility, inclusivity, diversity. They have a global warming subject in physics also in geography also, you know, so therefore how in the future, we can have a lesser carbon footprints, you know, and move to the sustainable product development, I think this way we can be relevant to the audience always. Yeah, I'll just add to what you said, right? I have a bit of a cheat code here. The way to stay connected with Gen Z is to have a team which is Gen Z. So that's what I do, my brand managers and, you know, my team is all Gen Z and they explain what it's like. You know, I would never get it, right?  And I think all that is important, but unless, you know, the team is really thinking, feeling and doing what your cohort, which is Gen Z, are doing, you know, you won't go far as a brand. So that's my mantra. And the team, this is on? Yeah. Sure. And that's the thing. I think I learn a lot from my team in terms of how to go about it. And that, you know, like the Gen Z, the team is also pretty vocal. It's more like a creative and I've proved it. They'll say to my face, no, it's not going out. And that's what I appreciate about them, right? So it's about, you know, you have to have a team who really, really walks the shoes and not just, you know, gather insights from a research or just from a secondary thing. You have to really be at it. I think just one last thing to that. I think within Gen Z also, of course, they have different preferences. They don't want to feel that you're spamming them, right? We're also evolving in terms of, you know, checking with each user, would you like us to do a follow up consultation on call? Within that, would you want it through call or through chat or you want us to not bother you at all? What's your way of going about it? And then truly personalizing the experience for them. And even in terms of data privacy, if you're asking for any data, we are now starting to tell them why we are asking for it and exactly where it will be used and going through more and more opting models, making more and more things optional in terms of what they are comfortable sharing or not. So... I mean, if you see the Insta Bio of any 20-year-old today, there'll be three things written over there, right? There'll be advertising executive in the day, poet by night, tracker in my holidays, right? So if your consumer today has multiple personalities, does not want to be identified by one personality, as a brand, you cannot just have one conversation with the consumer. You need to have multiple conversations with the consumer. There has to be a many-me aspect to the brand as well, right? And they're very important. For example, if you look at water, we've got the sports marketing program going on called Carrier Game. We've got Drink It Up going on, Hydration Fund. We've got Greener Promise, which is going on for sustainability as well, right? We've got a Bisseri App Aiga for our digital footprint, right? So we're having four different conversations with the consumers on four topics relevant to them, right? And you've got to keep them engaging. And what is also important today is that in the moment marketing, right? It's extremely important to take because this is a work audience. This is an audience which is on digital, Twitter, Insta, every day. You need to be hyper-local with your approach, right? I mean, we do about 12 to 15 limited editions a year. Whichever movie, you know, sort of kind of resonates with us. We've done from Javan to Ekta Tiger. Now we've done Dunke, we've done Fighter, we've done RRR, we've done Vikram, Jailer. All of these limited edition short birds have got us a lot of traction, especially with the Gen Z audience who's spoken about it, right? So as a brand, you need to have multiple and exciting communication with your consumers and that's going to bear rich fruits to everyone. I think that gets us to work news. What I understand from and what my takeaways from today's discussion are that Gen Z is not taking any bullshit. They're making you work for their worth. And it's also the most purpose driven generation, right? More than millennials and boomers have ever been because that's their point of focus. It was different for millennials, different for boomers for Gen Z. It's more about getting their worth, more being more purpose driven. And they sort of expect brands to do the same, right? They, I don't think you're like being sold to, so I'm not going to sell this panel also. But I think that's what is one of the key takeaways that they want to be co-creators, involve them in it, make them feel that they are part of the process of creating, bringing about that change, even in terms of sustainability, even in terms of the measures that you're taking to reduce the carbon footprint, et cetera. I think that's very key. And I think if we all in the room think that marketing changed when millennials became the target audience, I think Gen Z's in, it's giving us a tough time. So I think that's going to be worth it when we get out towards the end of it. Closing that for today, thank you so much for being a lovely audience. Thank you so much for being a lovely panel.


Now, I'm here with Trong Nguyen, who we hope will be the first person of Vietnamese descent ever elected to the BC legislature. Her families and my families and your families all had the same hope and desire when you decided to leave your country of origin to land on the shores of this country, Canada. And that was you wanted to live in a place where you could build a better future for your children and your grandchildren. You understand the importance of family. You understand the importance of community. That is why so many Vietnamese Canadians have found their way here and made such tremendous contributions to our country. Canada's story is one of immigrants who have come from around the world wanting something better for their children, working shoulder to shoulder to build the greatest country in the world. We are blessed that the Vietnamese, so many members of the Vietnamese community have chosen to make Surrey, Vancouver, and British Columbia your homes. All of you are every day when you think about your children and you think about their future working to make British Columbia an even better place. We are so lucky you've chosen us.