go ahead. Good afternoon guys so first of all thank you so much for allowing us to come and talk to you although from far away about our work here at Acre with EasyBuild. I'm David Evans of most of you probably already know me I'm an application developer here at Acre at Vanderbilt University in Nashville, Tennessee. I'm and I'm also one of the EasyBuild maintainers so that's only how you you know you have seen me. Here with me today there is also Eric Cappold which is one of our senior system administrators that is one of the two guys that also worked on setting up all the CVMFS file system and moving the stack to it so that I he you know he accepted my invitation to come and join this com call to talk with me about the work that we did with CVMFS. So Eric go ahead. Okay thank you let me move this. Okay yeah and just to add a little bit about me with this slide I was a large collaboration physicist when I was at Vanderbilt before joining Acre with the CMS collaboration at the Large Hadron Collider and then I changed career career tracks to go into software engineering and systems administration and so the point of this is saying that is that previous to this I was used to having CVMFS around and so that's that's where this comes from. The team of the three three of us that worked on this was first of all Andrew Mello he's currently a physicist at Vanderbilt and he's really in charge of CVMS deployment and synchronization scripting which will describe. Davide already introduced himself develops and maintains our cluster software stack and really I've worked on some of the configuration manager for this testing and benchmarking this system. So when we originally wrote this talk the target audience was someone who's a user of Elmod an easy build which I believe is everyone in this room but somewhat unfamiliar with CVMFS and the key problem that that we encountered at Acre and that we're really discussing here is that we want to deploy the software stack that's been built with easy build that uses Elmod to an HBC cluster and really outward from that cluster to the cloud and do that at scale and to be able to transparently execute users job on an elastic cloud with the minimal setup and maintenance footprint so it's not too much of a struggle for us to keep keep track of this. At Acre our software stack all the software is installed with only via easy build the environment modules are managed by Elmod we have 130 unique software modules visible to users and several hundred easy config files in its own GitHub repository with several tool chains. So the question we have is how to deploy this classically I think there are two general options one is to put it on a parallel file system IE NFS, a GPFS, HDFS, cluster at Acre we use GPFS. The good part of this is it's easy to do you build the software you put the file on the file system and point the users at it and there's no additional maintenance required in theory. The downside is this is heavily affected by parallel file system performance and when there's a problem with the when there's heavy write contention when there's nodes that are leaving the cluster this can really cause a problem. Additionally this can be hard to mount on cloud instances so if you want to burst out into the cloud and like us you're using a commercial GPFS file system that's that can be pretty much a non-starter you don't want to add random cloud instances to your file system cluster for security reasons and there may be licensing costs so this is difficult. The second option for deploying the stack is to replackage and deploy to the local disks on the node and this means you have to do the extra work to produce RPMs, devs or whatever sort of packaging you're using so that's a lot of work in scripting for cluster administrators and if your software stack is large this may mean you're taking a significant disk requirement on the local nodes themselves so if you're even using something like diskless nodes this could be a non-starter. So we've been pretty happy with option one in the past but the new problems that have come about in the last couple years that have caused us to rethink this model is one as I said trying to burst into the cloud it's not feasible or legal for us to extend our network file system into cloud instances and we also have the problem that our software stack contains licensed software and open source software. The licensed software we can't legally put into the cloud but we can put the open source software and we would like to do that. Another new problem is graphical interactive use. Traditionally large-scale computing our large-scale computing facility has been used for batch processing but more and more we have research groups that want to run graphical applications directly on compute resources and softwares become available such as open-on-demand from our house state, the super computing center which we we've started using which makes it easy for users to request a compute node create a desktop and use it in their browser. Once you do that software startup times become extremely important in a way that they're usually not with batch processing. For example MATLAB can depending on right contention depending on the state of the network file system take several minutes to start up on a on a cluster and the reason the reason for that is it has lots of tiny files it needs to read them all and reread them all several times and while in a batch processing when you you know the users are submitting jobs that will run for several hours if that software takes three minutes or ten seconds to load it makes little difference to the user but when the researcher wants to make plots and use the computing resources in this way then waiting three minutes versus ten seconds is the difference between usability and having a system that's not considered usable for them. So with those new problems in mind we really looked at CVMFS this is used extensively by the high-energy physics community it was developed CERN as part of the CERN virtual machine project and what it is is the file system that's optimized specifically for deploying software and nothing else and this is really how the CERN large CERN physics collaborations are able to deploy their custom analysis and event reconstruction software to a variety of clusters compute clusters around the world and now what what CVMS does is it acts as a read only file system and using an immutable data model and this gives it a lot of advantages in terms of distributing software over a general purpose file system because the data is immutable you can get away with some tricks such as content addressing if two files have the exact same content then effectively they can be considered the same two links to the same file and you only have to distribute that data once to a compute node and you really only have to read it once into into the system page cache. You can use off-the-shelf technologies that's a standard fuse mount to create a block device on a Linux system and everything's distributed using HTTP and for this you can use squid proxies to to scale up the distribution of the software stack to many systems so a little bit of a confession I used CVMFS for years behind the scenes as physicist I never really knew what it was or what it did I just knew that you had to your software was in the CVMFS directory I was in charge of writing comprehensive testing of our workflow on the cluster to make sure all of our components worked and I didn't know what CVMS was and that's that's something to say that CVMS worked really well it worked quietly and reliably for the CMS collaboration for several years on our system and I who is in charge of making sure it worked never had to do anything so that was pretty cool so so based on that history and the new challenges we were facing we decided to go see if we could make CVMFS work not just for CERN software but for our own software stack now the basic architecture of CVMFS CVMFS is you have a what's called a stratum zero this is an authoritative system which new files are written to this then talks to one or more servers which form a stratum one and this is where people will read from the files this product provides redundancy and high availability if the stratum zero goes down you cannot make changes to the file system you cannot write new files and directories but as long as there's at least one stratum one up then everyone can all clients can can read files as necessary then on top of that squid proxies are added these are standard HTTP proxies and they will just cache stratum one data and provide scale of the scalability so if you only have a handful of test machines you can read directly from the stratum one servers but as you scale out and want to deploy you could put any number of squid proxies out front and scale to thousands or tens of thousands of machines very easily so yeah go ahead yeah I'll take over I take over here so I think it's worth to also give you an overview of how how how our stack management pipeline is designed here at Taker because in some ways is one of the approaches so how do we deploy the software not only on the class but also on CVMFS so we adopt the classical three stages development with a dev QA and prod so the dev is the our let's say our contribution to the easy build community so to the easy build project so here is done by a developer like me or like fenglai or whoever else in our group wants to do it where you know we simply develop on our local machine the easy config files and then we contributed back to the easy config repositories the advantage of that is well first to contribute back and then to have a feedback from the easy build community where you know sometimes you're building software that you've never seen and you know let's try not to reinvent the wheel somebody may have good hands on how to avoid issues once the next one once the the easy config files are then merged into the easy code easy build easy config repository we simply have a single QA machine that mimics the cluster that contains a copy of the cluster stack and where we pull the dev easy config repository and then extract whatever easy config repository easy config files we need and then we can modify them depending on internal needs and then inject them directly into our own easy config repository in this way we have the flexibility of adapting the installation to our needs and at the same time to keep constant track of all the changes that we do on easy config files to assure reproducibility and avoid future errors the advantage of this is and also we use issues internally to chat to keep track of problems we had on a software it was typical before we adopted this system this way of deploying software oh the user has this we're having issues with this software yeah I remember we I saw that a year ago or something like that but I don't remember how I did it so we use issues internally on that repo to keep track of what is going on so now the easy config files are ready that are now merged into our internal repository already to be brought into broad and the way in which we do that is we have a single user for all the admins so there is dedicated user to build software on the stack via easy build so he's the only one that is allowed to use easy build on the cluster the advantage of this is that it simplifies the stack they use the file permissions on the stack so independently on which admin is actually building the software there is no problem because it's always the same user so he's the owner the advantage of the other advantages is that it prevents accidental stack modifications so for example if I as myself build a software on the stack and then at a certain point I do a pp install and I forget to use the dash user or you know turn on a virtual environment the problem is that I would end up polluting the software stack the other advantage of having a third entity that builds software is that it's easier for user support team to debug and troubleshoot what the users come out the problems that users have with the stack because there is we have the same permission as users with respect to the stack so it's it also simplifies that thing so as Bob the builder we then pull from our easy config repository and we use the slurm integration in get in easy build to then launch the job launch jobs that build on our debug queue one of the problem that we have on our cluster is that we have multiple CPU architectures so in our debug queue we have multiple nodes of all the different architectures and we build the software one time on each architecture so if I'm building something I'll build it four times for the four pictures and the debug queue is connected directly to the stack on GPFS so here so the debug nodes can read and write from the stack on GPFS once the software is built on GPFS we have an automatic system developed by Andrew that it might leave sinks GPFS with our CVMFS and once the sink is done at that point both the software is available both for the cluster and for the cloud so that's that's in a nutshell how our pipeline works man yeah so the next thing is how to how easy is this to do it's it's actually really easy because we started with the pipeline of I'm correct where we we only we only wrote to GPFS and then the whole cluster would read it back from GPFS so in order to make it transparent to the user and have it so that the debug nodes would be looking at the stack as a g under the GPFS file system and the rest of the cluster would would pull the stack from CVMFS basically all we need to do is some clever sim linking so what we did is we created some directory a directory acre arch and that was just a sim link to the network file system when you're on a build build node and we'd set that sim link to the node architecture depending on the node so the software that was built against acre arch would would naturally if you were on a yeah let's say an Intel Sandy bridge now at an older node then that would automatically link to that architecture and then that the software would all just get naturally installed on acre arch now the compute nodes what you do is our configuration management would simply relink acre arch to CVMFS where the directory structure would otherwise look the same allowing us to also do a fallback to GPFS if the configuration management software noticed that C of AMFS is down and again the sim link would be set to the node architecture so this would mean that the software is available in the same place transparently to the user in terms of the file system hierarchy but it's really being served on CVMFS or GPFS depending on where you are on the cluster so just to show you an example here I am on a standard node and I can look at some sim links acre common and acre arch so acre arch is symbolic link to a system within CVMFS we use the Oasis part server on the open so science grid to serve our open soft source software and you can see it links into a directory which has sandy bridge in the in the path as this is a sandy bridge machine we have a separate common directory which we didn't mention serves for certain binary distributed software where it doesn't make sense to make different compiled different versions on the different architectures that all goes to the same directory okay now if I go to one of our debug nodes and I look at the sim link acre common points to GPFS 22 acre common so if you're the builder user you can then write and update that directory and overnight that will be synchronized so that the other nodes see the exact same thing again if I look at acre arch this points to the internal of the GPFS rewrite file system but here you can see where we're linking to the Skylake architecture the other issue we had to solve and in distributing this is differentiating between open source software and license software for the open source software we actually didn't need to set up our own CVMS FS server as we're part of the open science grid so we were able to use their Oasis system which provides this as a service to other clusters and again what we do is in GPFS everything works normally all software is built in the same to the same directories and then we can use sim links on CVMS to separate out the private and the public software so the module directories in CVMS FS will be in the nightly sink we just flag certain certain directories as private modules and those become sim links to another location if you're out on the cloud these private links will be ignored by Elmod if they're broken so all the license software we host on completely separate CVMS instance this is only in turn available on the internal network there's no external question yeah how do you solve the problems with some software that are resolving those sim links from a common or a car down to the GPFS file system into their bash files or whatever the sim links are on the top of the directories but some softwares when they install do sim link resolution and put the actual path into their scripts yeah do that yeah no no I see what you mean that's a good question as of now we haven't hit that issue you haven't no have you looked closely well everything seems to work when no I mean yeah that's that's definitely a good point it's it's definitely it's something that we haven't hit yet so early we are just lucky yeah so going on with that good luck so I can I could show you to this sim link if I take a package like I take an open source package anaconda 3 oops okay this is a directory on our build node but if I take one of our license software modules like Matlab okay on the build node these are both directories now if I exit the build node and go back to the cluster you could see that anaconda 3 is still a directory but if I look at the license software this is a symbolic link and it goes to a different CVMS server which is our private one so what happens is if we go to a cloud instance that will Matlab will just appear to them as a broken link and it will be ignored completely by Elmod well if you're on the cluster you'll actually see that when you run a module avail command so here's the big chart of our overall distribution model the internal everything highlighted in blue is something internal and on-premises to acre so for acre we have our we have our GPFS file system and nightly that gets synced to our private stratum 0 servers and then that can go forward through an internal architecture where we can set up as many stratum one or a squid proxies as we want inside the internal networks cluster nodes can then read from that the open source software will send out to the open science grid oasis server now if you're not an open science grid member you could also set up a separate server internally to be your stratum 0 let's see that sorry about the typo there and then that architecture has stratum ones and external squid caches so what happens is that our internal squid caches are reading from both the internal and the OSG stratum one and collecting together all of the software if you're on an external user work station or a cloud instance you could then read from the OSG and external squid clash really that just reads from those stratum ones and just gives you the open source software okay and we have a little demo for the cloud instance so I've created a cloud in AWS a couple I've created an instance in AWS a couple minutes ago before I started this presentation I haven't logged into it yet so I'm gonna do that now and what I'll do is I'll run a simple script to enable enable L mod and connect it to to our open source software stack now I've never logged into this just to show you I've never logged into this before and this is a stock instance of CentOS 7 from the AWS collection so this is right off the shelf so I'm gonna grab this simple script and then this could be properly you know from this script you could create your own AWS image or containerize these instructions so I'll just run it and this will set up and in just a minute or two here set up and get the software stack ready to go on this new image any questions while we're installing software no questions so far okay now we're just populating an entirely empty environment that's right yeah this is so this is just demonstrating that this is pretty simple and fast you can this is this is a stock core CentOS 7 image okay so now everything's installed I should be able to log out log back in and type module avail and then again this is gonna have to read do an initial read from CVMFS and populate the internal cache and there we go so I can see some compilers and binary distributed software and I'm ready to go so I can I can load whatever module I want and I can start doing work what what's really nice about this in my opinion is that a researcher can use a create a container that links into CVMFS and that sort of dynamically gets updates to all their scientific software stack in a cloud image whereas if you were try if you were to try to put all of this software into one Docker image that would be kind of a pain and you'd have to do updates this sort of short circuits that step just to add just what a similar use case another one is for example we have now we have a few users that they have some pipelines that are half on their local work stations and half on the clusters so the the problem for them is how to have a consistent software stack that allows them to seamlessly move their the part of their pipeline from the workstation to the classroom vice versa so one of the thing that we are doing now is for example to allow users obviously with with our supervision to allow users to mount the part of our stack so the GCC the false part of our stack directly into their workstations in that way they can have they can build the software and they can have the same software both sides and build on top of the same stack yeah so this is this is an exciting new way we can distribute so the other the other issue is with this issue of interactive startup time and here's here's where we really had a problem with a cold cache and this this is one data point that I'm showing right now this shows the startup time of some packages Python with loading I think I had a NumPy and scikit trying to just import those modules and load Python trying to load MATLAB and trying to load R and just run an empty script when we use the GPFS file system which again this is on a regular day there's lots of workflows going you know that the the file system is under some regular stress you could see it could take as many as nine ninety seconds to load up all the Python modules several minutes to load MATLAB and when we switched over to CVMFS those load times really shot down by an order of magnitude now once the cache is warmed once the data for those files is on the internal on the compute node loading this software then the difference doesn't make as much it doesn't matter so much in fact GPFS maybe a little faster here due to how it uses user space caching versus the fuse mount for CVMFS but in terms of the user experience this is pretty similar and just to demonstrate what our users are doing is we have an open-on-demand system which gives us gives users in browser desktops that they can launch and ask for oops I shouldn't clean this up and a user might want to use this desktop to let's say load and run MATLAB and here's the step where we really had a lot of users complaints if there were any significant difficulties or stress on the file system this could take several minutes and again with CVMFS all the little files that go into loading MATLAB can be shipped put together as a block shipped over HTTP and and you know if there are multiple files that have the same content these can also be red ones and so this will this will show up for the user in less than a minute pretty reliably now which is still slow for any application these days but it's a lot better than when you're trying to load MATLAB over a rewrite file system and there we go so we did that so the basic conclusion before we go to questions and further discussion is basically we found this is a reliable and simple reasonably simple method to distribute scientific software that we can easily extend out to the cloud or to other nodes in our researchers work groups that aren't we don't want to connect for whatever reason to our cluster file system and it's also very nicely suitable for interactive use thank you very much yeah go ahead no no I was just saying that that's also if you have any questions or anything on that side we are happy to help to answer you said you had dedicated you have dedicated build nodes correct how large is that system what are you or how few users do you have no it's just so we have we currently have one two three four five CPU architectures but also because we have one that has GPUs so so we have four for GPU architectures we have four nodes that are our debug you so what happens is that I understand your point in a sense of well you know just dedicating nodes just for building this kind of a waste of money and cycles so the way in order to you know use them as much as possible is to put to make them available through our debug queue on the cluster so they are both acceptable by the users for debug purposes and at the same time available for us for building that does not slow too much our process because the maximum job that can be launched by users on the debug queue is 30 minutes so you know if we have to build a software or worse case scenario we have to wait 30 minutes and that's that's that's the way in which I justified to our managers how to to give me four nodes one per architecture and if that's a problem with virtualization you could reserve you know create VNs of just two cores on a one or two cores on an ad and release the others for general purpose computing too yeah okay any other questions for the V date and this is John day I'm curious about how you are using squid in AWS is that a standard image that you're getting from from Amazon or oh no the squid servers are dedicated machines on premises so that's on-prem and you you have a dedicated connection the you know the AWS cloud so so that the squids so that so for for the software that so so we we have a compute node on the cloud here basically that that can access the software stack it's using the public-facing squid proxy and that's good proxy we could have our own this one's actually hosted by the open science group and so literally anyone in the world can talk to the squid proxy you could we could conceivably make restrictions based on IP address or you know other qualifications but yeah as it stands this is basically exportable to everyone thank you so yeah so the the cloud node doesn't need anything so any specific credentials to talk to the squid server we just we just stand it up and it immediately begins communicating okay great thanks very much thank you guys okay I'll stop the screen is the streaming briefly so we'll set up the next talk


Jeremy, there was another issue I wanted to ask you about. Starting with Margaret Thatcher and Ronald Reagan, we've got a neoliberal economic policy that has transferred enormous wealth from the people to a small elite. In the 90s, both Tony Blair in Britain and Bill Clinton here in the U.S. moved the labor and the Democratic parties to the center right and absorbed those policies. Is there any hope, after 40 years or more now, of these policies that we can emerge from this? Do you feel of the Labour Party ever resuming its original roles as defenders of working interests? It's not so much whether the parties do or they don't. The issue is really where the politics of our society goes. We've now had 12 years of austerity, which has led to falling living standards, frozen wages, cuts in public services, and that's likely to get worse. But what's interesting is the levels of pushback against it by working class communities are now strikes in all of the major services and most of the industries in the U.K., and those strikes have been joined by lots of social justice campaigns as well. And so I am very, very hopeful that a generation is growing up that was mobilized by everything that we did in trying to change the direction of the Labour Party and was also what the Democratic Socialists of America achieved in changing the approaches of many people towards issues of socialism and social justice. And so I think that I'm very hopeful for the future and the parties, Democratic Party and Labour Party, need to recognize that triangulation towards the center, triangulation where you have one clever move after another to box off this, box off that and so on, doesn't always work. I'll give you the finish on this. I was in Brazil for the second round of the presidential election. It was very tough. Bolsonaro, very well funded, supported by pretty well all of the media in Brazil against Lula, who had been in prison only three years ago for more than 500 days on trumped up charges. Lula's campaign did not triangulate. They did not vacillate. What I saw was the most brilliant organization of poor, working class communities, homeless people, landless people, and people who were desperate for change. And I saw the strength of opposition against them. They won, albeit narrowly, but they did win. And I tell you what, that win gave such confidence to people that when Bolsonaro's henchmen came after Lula and came after the presidential offices, hundreds of thousands of people came on the streets straight away to support Lula. That's confidence and that's hope. The issues remain the same. The issues remain the same. Are we going to face the same issue? The new generation faces the same issues. One generation after another faces the issues of social injustice and social inequality. It's the system that has to change. As Bernie Sanders once said, America is very wealthy. America can afford anything, except inequality. Thank you, John. You're very welcome.


Hello, and welcome to this presentation of the STM32 nested vector interrupt controller. We will be presenting the features of this controller. The nested vector interrupt controller embedded inside of the STM32MP1 series microprocessor provides up to 150 interrupt channels served with low latency. One of 16 priorities can be assigned to each interrupt source. Interrupt levels can benefit from dynamic prioritization of the interrupt levels, fast response to the requests thanks to low latency response and tail chaining, and also from vector table relocation. The nested vector interrupt controller provides a fast response to interrupt requests, allowing an application to quickly serve incoming events. Most of the peripherals have a unique interrupt vector, making development of the application easier, with less need to programmatically determine the source of an interrupt during processing. The interrupt vector table can also be relocated, which allows the system designer to adapt the placement of the interrupt service routines to the application's memory layout. The nested vector interrupt controller provides several features for efficient handling of exceptions. When an interrupt is served and a new request with higher priority arrives, the new exception can preempt the current one. This is called nested exception handling. The previous exception handler resumes execution after the higher priority exception is handled. When an interrupt request with lower or equal priority is raised during execution of an interrupt handler, it becomes pending. Once the current interrupt handler is finished, the context saving and restoring process is skipped and control is transferred directly to the new exception handler to decrease interrupt latency. When an interrupt arrives, the processor first saves the program context before executing the interrupt handler. If the processor is performing this context saving operation when an interrupt of higher priority arrives, the processor switches directly to handling the higher priority interrupt when it is finished saving the program context. Then tail chaining is used prior to executing the IRQB interrupt service routine. When all of the exception handlers have been run and no other exception is pending, the processor restores the previous context from the stack and returns to normal application execution. When accessing the nested vector interrupt controller registers, ensure that your code uses a correctly aligned register access. Unaligned access is not supported for nested vector interrupt controller registers as well as all memory mapped registers located in the Cortex-M4. An interrupt becomes pending when the source asks for service. Disabling the interrupt only prevents the processor from taking that interrupt. Make sure the related interrupt flag is cleared before enabling the interrupt vector. Before relocating the vector table using the VTOR register, ensure that fault handlers, NMI and all enabled interrupts are correctly set up on the new location. For detailed information, please refer mainly to Programming Manual PM0214.


I hope you guys are ready for a good rankings video. As you can see by the title, we are doing my Hood of Beauty palettes. I have 20 palettes here today to rank from worst to best. I just excluded the palettes that honestly are irrelevant and forgotten at this point. Like I have some of the very first jewel obsessions and the neon obsessions. I was like, nobody cares about those. Yeah, we have 20 left. So if you wanna see my rankings, then just keep watching. Let's start off with number 20. The current least favorite palette in my collection and that is the Toffee Brown Obsessions Palette. I did not like this whole collection from Hood of Beauty. I really feel like the quality just is not up to par here. This one, it comes in last because I don't really care for the color story. Like it's fine. I actually think it's pretty unique within the Hood of Beauty line, but the quality is really not good here. This shade hardly shows up. The shimmers you have to press really hard to get any pickup and the masks are just okay. Just knowing the quality that Hood of Beauty can put out, it's like why didn't she do that with this? Because this is not good. So yeah, this is my least favorite palette. Number 19 is another palette from that collection. I just think overall this collection just wasn't it. This is the Carmel Brown Palette. So this one is a little bit too warm for me. I personally tend to go towards more like cooler tone shadows. And I think the quality of this is a bit better than the Toffee Palette, but it's still, it's like this swirl shade right here is really, really bad. The shimmers are okay. Not the best that Hood of Beauty can do. The mat's a little bit of a struggle to blend out. So it's just not the best quality wise from Hood of Beauty. And it's also not my favorite story, color wise. Number 18 is the Sand Hayes Palette. The quality on this one isn't terrible. It's definitely better than the Brown Obsessions Palettes. But I really just can't get down with this color story. I don't really like it. I never really want to reach for it. I don't think I've really used it since like my last video that I did on it, which was probably like two years ago. And it's just because I don't like this color story. The quality is like pretty average. It's not the greatest, but it's also not the worst like the Brown Obsessions. I just don't like the color story on this one. Number 17 is a really old one. I actually didn't expect them to still have this in stock, but they do at Sephora. So I threw it in here. This is the Ruby Obsessions Palette. Now I like this palette because I don't have really another palette like this in my collection, but quality wise, it doesn't really show up on your eyelid how it looks in the pan. So your look kind of tends to be like a hot pink with this. And that's just not what I'm going for. I want it to be more red and ruby just as the name would suggest. So it doesn't translate as good on the eyes, but it still is quite unique. In my opinion, it just is that translation thing that I don't love about this. So quality wise, it's not the best, but I do definitely keep it around because it is very unique in my collection. And this I believe is from her first round of launches for these Obsession palettes. So this is quite old at this point. I'm pretty sure eventually she'll get rid of these for good, but I'm surprised they still have that. Number 16 is the last from the Brown Obsessions collection. This is the Chocolate Browns. So this by far is my favorite color story. And I actually, I don't want to say I reach for this a lot. I definitely don't, but of the Brown Obsessions, this is the one I reach for the most because I really do enjoy this color story. It gives you like a deep brown, smoky eye, which I think is really, really pretty. I love really dramatic neutral tones kind of eyes. And I think that this one gives it to me, but just along with the other two in this collection, the quality is not the greatest from put it here. These shimmers, I feel like you gotta press kind of hard. It can get a little messy on the eye. Just nothing in here is like that good. You know, I just like the color story. Number 15 is the Purple Haze Obsessions palette. This one, of course I like it because it's purple. If you're new here, I love purple colors. The shadows here aren't as intense as I would like them to be. The Haze Obsessions, better than the Brown Obsessions. But I don't know, I'm just not in love with this as a purple palette. The quality is good, but it's not great. I don't know, this is just where it felt. I like it. Okay, from this point on, I actually really do like the palettes. They don't have too much to complain about. This is why I don't have much to say because the quality is good. This is just where it landed. Number 14 is the Kaki Haze Obsessions. Originally, I think I placed purple above the khaki, but khaki is such a trendy color now that I've been reaching for khaki shades a lot more. And I think this is a really great khaki palette here. I find some of the mattes can be a little difficult to blend out, but it's definitely worth it because I do find this to be a pretty unique color story and I love the looks that I can get with this. So with this and the other Haze palettes, generally speaking, they're not the most pigmented. They're a little bit softer, but they're very, very easy to work with. So this isn't like the best quality khaki palette that I have in my collection, but it's definitely workable. It's very pretty. I enjoy this color story a lot. I do recommend this if you're looking for a really good khaki color palette. Number 13 is the Amethyst Obsessions palette. So this is an older palette. Again, I'm very surprised that they still sell this, but I absolutely love the color story here. I think one of Huda's strengths is her color curations. Her palettes are so well thought out. Love this purple palette. It again doesn't translate on the lids like I would want it to. Sometimes these pull more pink than purple, but the colors here are just so pretty. I like this one better than the purple Haze palette because I just simply like the colors in here better. They're a little bit more punchy and they're more true like grape kind of purples. And I recommend a brand that will come out with the grape colored purples instead of playing it safe because those can be harder to work with and harder to just master that formula. No, but I really, really like this. There are some flaws to it, but color story wise, I think it's one of the best curated purple palettes on the market. Number 12 is the Wild Jaguar palette. We're coming up on a new kind of obsessions palette here. How beautiful is this one? I like cool tone neutrals. This screams cool tone neutrals to me. Now the reason that this isn't like not my number one because this is my kind of color story is because there's a little bit too much glitters in here. So it's hard to get a really intricate look with this. You don't have too many options. It's kind of like you put two eyeshadows on the lid and you have to head out. You can't do too much with these glimmery shades. It can get a little messy, but so beautiful. Love this color story. These Wild Obsessions palettes, the quality is really, really nice as well. So if you've been into these kind of colors, I really like this one. I think it's really great. Number 11, also from the same collection, is the Wild Python palette. Now this one is just simply fun. So I'm ranking this where I'm ranking it because it really is such a unique palette in her line. You can get some pretty crazy looks. The one flaw to this is that the shimmer shades don't pack as much pigment or as much of a base color as I would like. So for example, if you put these on your bare lid, you're gonna see your lid underneath. I don't really like that look. So in order to get this palette to work, you definitely wanna put the mattes down all over the lid and then put the shimmers on top, more so as an eye topper. I don't love that. I want these colors to be able to stand out on their own and they just don't. But the looks I get with this are so pretty that I can't complain too much. Number 10, oh, I didn't even realize they placed these right down the row, but I guess I did. This is the Wild Chameleon palette. So this one is my favorite of the Wild Obsessions palettes. And surprisingly it is warm, but there's something about the quality in this that I feel is just a little better than the others. And there's a purple in here. So easily it's okay. And I actually really like the combo of very warm shades with purple. I find those tones to be very complimentary towards one another. And yeah, the quality of this one is like better than the others. And I've loved every look that I've come up with this. So I ended up liking this one a lot more than I thought I would. Number nine is the Nude Light Palette. This by far, if you're looking for the best obsessions palettes, which tend to be a little bit more affordable, you need to pick up from the nude line as long as you like the colors. So this is the Nude Light. This is my least favorite of the Nude Obsessions, but it still is really beautiful. And I don't think I ranked this quite as high last time because I wasn't as into pastel-y-type shades because this is pretty light. It's almost a pastel palette. So the reason why I'm putting it more towards the top today is because this last year, I really learned to appreciate looks that don't necessarily have a lot of depth. And I think this just has a lot of really pretty lid options for just a light, airy, wide, awake kind of look, been enjoying this and the quality is really nice. Number eight is the Desert Dust Palette. So this is the first big palette. And we have to say, with her big palettes, typically I prefer the formulations in them, which is why you will see that they are ranking higher than the obsessions palettes. The obsessions palettes can be a little bit hit or miss. These almost always a hit. So this is a Desert Dust Palette and this is one of my favorite color stories because it does have that warm and purple mix here, but this is not her best quality of the big palettes. I found that this palette does not have the longest lifespan and that the shades almost dry out, which I don't love because these are quite expensive. So while it is a gorgeous palette, I just can't rank it at the top anymore and it used to be ranked at the top. But that is because the quality definitely does go down over time. Number seven is the Nude Medium Palette. So this is the one that I'm wearing today. It had been a hot minute since I'd worn this palette. So I thought I'd wear it today. And I love this palette. It is so beautiful. The only thing is I wish this shade gave a little bit more depth. I know it's supposed to be the medium depth palette, but I did want something kind of darker to put out here. But other than that, the quality is really great. The mattes blend out beautifully. The shimmers are on point. Look at my makeup today. I love the look. So these are really great to work with. You'll enjoy the quality a lot if you like the color story. I love this color story. I think it's pretty having the pinks and a little bit of copper there. Love that copper pop. Number six, I have the Hood of Beauty. Of course, rose gold remastered palettes. So she had an older one. This was years ago. And it was really, really dry. And she reformulated it. And I wish she would formulate more palettes like these. So I look at this. It really is kind of a very classic color story. The time that it came out, it was the era of ABH Modern Renaissance. So those tones very much speak to that time. I love the way that these shimmers feel. They feel super duper creamy. Now I did just repurchase a new one of these over the holidays because I got a pretty good deal. But I do think they do dry out because my mom's one hers is older. It's just not as creamy as this one here. But when they're creamy, absolutely beautiful. But just make sure you take care of it and be aware that the shadows may dry out. But they're so creamy that I let it pass because they're so beautiful. Number five is the Nude Rich Palette. So this is the last of the nude palettes. This one is my favorite. I don't care if you have a light medium or rich skin tone. Love this one so much. It gives the prettiest kind of cranberry burgundy type eye. I'm just gonna put a little bit of this burgundy shade like right in the outer corner. Like this is what I wanted right here. Just something like this in that medium palette. It's like a touch. You almost just need a dot. Love this one so much. The quality is really great on this. The value is phenomenal for the price that you pay. And I love the varying tones here. If you like these kind of rich cranberry burgundy eyes, you will love this one. Number four is a very similar palette. It's kind of a step up. This is the Naughty Nude Palette. I absolutely love this one. This one has those deep, rich, burgundy tones, which typically aren't the type of colors that I reach for on an everyday basis. But this is a beautiful one. I truly do not have anything bad to say about this palette other than this weird guy right here. But other than that, it's great. It's just not a color story that I reach for constantly. So based on sheer number of use is why it's not number one, but I don't have anything bad to say about it if you enjoy this really rich color story. And by the way, I did notice that that palette looked so good on rich skin tones. If you don't have it, I recommend it. Number three is the New Nude Palette. Now this is a personal favorite for me. If you have more of a rich skin tone than me, I definitely recommend Naughty Nude over New Nude. But because I've learned to more so appreciate like the lighter, more pastel-y kind of eyes, and I love mauve-y looks, this one has been perfect for me. Now what I don't love about this is how they translate on the eyes. I think this palette definitely pulls more pink than they look in the pan, which I don't love that, but I still love the looks that I create from this palette and I just reach for it more than the Naughty Nude Palette. I enjoy the quality of this one and color story is pushing this one towards the top. I just think it's so pretty for an everyday palette for myself. Now when I tell you number one and two were in battle, it was tough. I was like really weighing out the options here. Ultimately, I decided that number two was the brand new Rose Quartz Palette, but just know it's almost number one. So this is the one that came out this holiday season. I love it because it has more cooler kind of purple tones. I love every single look that I come up with. When it comes to this palette, it's unique, but it's still not too bold. It's still wearable, but you still can have a lot of playroom with this and experiment. The quality is phenomenal. The shimmers here, oh my goodness, extra glimmery, extra reflective. I love it, so I don't have anything bad to say about it, like at all, but that leaves the reigning champion to win once again this year, the Mercury Retrograde. Now these two, in my opinion, go hand in hand. So take a look at Mercury Retrograde. It's just a little bit more experimental with the colors. You have the purple row, you have more of a blue row as well. I just think that this palette is a little bit more fun, which is why I'm putting it at number one over Rose Quartz. It's interesting though, Rose Quartz, I'm more likely to grab for Friday. So this one is probably getting more used by me, but it's all about the inspiration for me with this one. This is such a unique palette. There really isn't a palette like this on the market like it, except for Rose Quartz, but Rose Quartz is more wearable, I would say. Still, reigning champion, I mean, it deserved to be re-crowned as number one because it really is a special palette. And I do believe it's on sale on Sephora for like $33 right now. Totally worth it if you're thinking about it. It's my favorite. And there we have it, you guys. Those are all of my hood up beauty palettes ranked from worst to best. Let me know what palettes you have in your collection and how would you rank them? Give yourself a little bit of a challenge, comment down below. With that being said, thank you so much for being subscribed to my channel and liking this video. And I will see you guys in the next one. Bye, guys, have a good one.