OK. Good afternoon. There we go. Microphone working, yes it is. All the other speakers have been saying how bright the lights are on the presenters, and it's absolutely true. It just cannot see a thing with these blinding lights. So I'm Neil Levine, Director of Product Management for Red Hat. I work in the storage BU. I've been working on Cef for a few years now on the product side. And I'm going to try and get through my slides as quickly as possible, which are very much the sort of Cef one-on-one. And hand over to Will Bogey from Target, who's going to give you the experience of a real-life Cef deployer, and some of the experiences he's had getting Cef to run in an open-stack environment. So very quickly, how many people are running Cef already in your audience? About half, and how many are thinking about running Cef but not running it currently? So not many. OK, so I'm going to bore a lot of you for the first 10 slides because you guys are running Cef already. But my marketing team like me to go through this stuff. So I'm just going to go through a little bit of introduction to Cef for those of you who don't know it, how it fits within the Red Hat portfolio. And there's my screensaver kicking in too early. And then hand over to Will, then I'm going to cover a little bit of the roadmap at the end, which I suspect will get a lot of questions as Cef dual, the latest version, has just come out with lots of good stuff. OK, so very quickly, let's go through this. What is software-defined storage and how does Cef fit into this ecosystem? So there's a lot of buzzwords. Software-defined storage doesn't really mean anything. It's a marketing phrase, but we still have to use it. It's a part of a general change. And we like to slide a lot. It sort of explains what's going on throughout the data center, both in the application development, moving to from waterfall to agile to DevOps. You can see containers as being part of this move to sort of make things go faster, get better velocity. And we really see storage as fitting within that software defined storage, specifically fitting into that same change that's happening throughout the data center. So we had to scale up with our appliances. We then had to scale out, sometimes still with hardware appliances. And we're now moving to a software-defined storage world. And again, this is all about velocity. OK, so what does this actually mean? Well, the old world was big silos of storage. You would provision them for individual projects, individual applications, sometimes individual users. And it was hard to scale, because when you ran out of space, you had to buy more stuff and move data around. You had a new project. You had to go through a whole procurement process again. With software-defined storage, much simpler. You buy standardized computers and disks. Layer-on just software on top of this commodity hardware, present to author a nice API. And suddenly, the users could provision their own storage, decide the performance they want, the hardware characteristics. And ultimately, everything scales, both the software and the hardware layer underneath. So what does this matter? Well, you can start buying your hardware from a single company. If you're already buying Dell or Supermicro for your compute, you can also use those vendors to buy your storage as well.  And you can also really pick the hardware you want specific to your app to your environment. Obviously, we also get the scale out. Architecture, shared nothing, distributed, no point of failure, no single point of failure. Architectures, which are really sort of the intrinsic goodness of Seth in particular, is around the robustness of that architecture to really allow you to get up to petabytes of storage. And importantly, you move all of the intelligence up into the software layer after the hardware. And so you can really have high innovation in software much faster than you can when you're tied to the hardware. And of course, being a Red Hat person, we've preached the open source way. And you get the open development process with that, too. So with an open stack, this matters, because if you're scaling open stack, which hopefully many of you are, you want to be able to scale the storage in the same way you're scaling the compute. You're using provisioning extra machines because you need the extra VMs.  So you get that agility. You can tie it together both in the actual buying process, but also in the deployment cycle as well. OK, so let's get out of the way, software-defined storage. OK, so what is SEF for the three people in the room who are not familiar with it? It's a unified storage system providing object block and now, as well, a stable file system, too. At its heart, it is an object storage system, which we refer to as RADOS. And this is where the 10 years of R&D and peer review have been going on in a SEF project were really at that RADOS level to get a highly resilient, robust architecture for storing data. If you want to interact with that object store, we have a nice little set of development libraries called Liberados, which allows you to talk directly to the object storage interface if you're obsessed about performance and you really want to get rid of as much abstraction overhead as possible. That's what you're going to be using. Most people, they want to be using that. They want to consume more common APIs. So we have the RADOS gateway, or RGW, which is the component, which allows you to talk S3 or Swift. We also have RADOS BlockDevice, or RBD, which allows you to consume SEF as essentially a virtual disk image, which is what the majority of OpenStack users use SEF for. We now also have the file system, which allows you to interact with RADOS as if it were a POSIX compliant file system. So that's the one-on-one of what SEF is. It's a incredibly vibrant community. In fact, we have many of the SEF committers here, not just from Red Hat, but from other vendors and even users of SEF. And the SEF development community has actually had a very similar trajectory to that of OpenStack. The interest and excitement, sort of the curves have been going up in the same way. So if you're interested in participating in SEF, we have some great mailing lists at sef.com, ILC channels. We invite you to come and participate. OK, now the heavy sales pitch, which I have to do in one slide. We have a distinct product called Red Hat SEF Storage, which is what Red Hat offers. This is where we take the upstream community addition. We apply our own QE, our own integration, add some extra bits and pieces on there, additional patches, some extra tooling, and we distribute it out. So Red Hat SEF Storage is based typically on one of the stable releases with a little bit of lag to actually allow us to do a little bit of extra QA. And then that's what we offer, particularly with Red Hat OpenStack, the Red Hat platform. OK, sales pitch over. OK, why OpenStack? And why does SEF fit into this? So we use this slide a lot, because it's very cool. And it shows us winning. We like this. So SEF RBD is the leading block solution with an OpenStack. And it actually has been for quite a while. It's a user survey, I think, from April? Yeah, so a few weeks ago. But actually, SEF RBD has been leading as a deployment technology, a storage technology within OpenStack now, I think, for about three or four years. And indeed, it sort of came out around the Cinder project sort of evolved alongside RBD becoming almost the de facto block implementation for OpenStack. OK, very cool graph, but why? And what does this mean? So the obvious thing here is it's unified storage. You get object and block. And now you're going to get file too. So this has some very obvious benefits in terms of buying hardware. You only have to buy one class of hardware for a SEF cluster and tune it and so on. It also means you only have to train up your team on one storage technology. Supply-driven storage is complex distributed systems. You have to train people how to use it. But if you're going to need a variety of different storage services for your OpenStack platform, you want to train them up on just one technology. And crucially, it's tightly integrated with all of the OpenStack services. A lot of the drivers you see for things like Cinder and so on, they offer a basic iSCSI interface and things can be connected through. Whereas we actually tightly integrate directly with the Cinder and Nova and so on services directly. So to explain what this actually means in practical architectural detail here is we have a library called LabRBD, which is something that's actually running inside the control services of OpenStack themselves on the block and VM side. So on the right-hand side here, you can see both with Cinder and Glance and actually through the hypervisor with KVM. This is where you run this little lightweight LabRBD library, loads into the process. And effectively, it turns the OpenStack control service into a native surf client. What that means is you get a one-to-many IO relationship. It's not a single point-to-point relationship with traditional storage technologies.  Again, this is where the RADOS cluster has got lots of individual components. That's what allows the scale out. That's what that's a cool architecture. These clients can access all of those nodes at the same time to get access to the data. Been using a RADOS gateway. We obviously present the Swift as well as the S3 API and connect that into Keystone for authentication too. So it really is tightly married deep inside the OpenStack architecture. So why does this matter? Well, from a practical benefit, you're able to boot lots of images really, really quickly. Whether it's one or 100 VMs, they're pretty much booting up simultaneously. And this is mainly because of the sort of the copy and write benefit we provide down in the SEF service. So when images are being, say, booted from glance or you're snapshotting something from either, say, from Cinder or even from Nova, it's all happening down at the RADOS layer, down in the SEF layer. There's no data movement happening up through the OpenStack control services. So you can boot things very quickly, back up and snapshot images very, very quickly. So that, again, it comes back to that velocity. The speed of operation is much, much faster with SEF. And you get all the benefits of live migration and sort of all the important things you need from a network storage service. Also, within a SEF cluster, as you've mentioned, you can pick and choose your hardware. So if you want to have a hot tier, running, say, all SSDs for your block, maybe you want to have a slow tier running 7,200 RPM HDDs or hard disks for, say, your object storage, you can provision them on different hardware, allocate them to different pools. But it's all a single logical SEF cluster, so you're only having to provision and manage one system. And then also now we're starting to have some very good multi-site replication services. We've had sort of fairly OK-ish snapshotting and exporting of the backup that we've now got with the latest version of Dual, which I'll cover later. We've actually got some pretty automated backup capabilities, which allow you to push images between clusters or to snapshot and export things off to remote clusters even running in public clouds. It's very cool. I have no idea what this clip art is doing. Daniel, the marketing guy, I've always wanted to know why you've got somebody being chased, and I'm totally sure that is. But I've left it in there anyway. OK, so quickly to finish off here, SEF is a complex system. It has a lot of knobs and levers and buttons and tunings and everything else here. And getting the most out of a SEF cluster is something that a lot of people have developed expertise for over many, many years. And we tried to distill it down to just sort of a couple of basic sort of rules of thumb, and then hand over to Will to sort of go into the details here. The first one is select the correct hardware. I think Will's going to talk about this in detail. This is the most important thing you have to go through when building your SEF cluster. A lot of people think, great commodity hardware, I'm going to buy the cheapest server I can find. I'm going to pack as many disks in it, and I'm going to have myself an exabyte server sand. Super cheap. You will, but it will be unusable. There's only a certain amount of CPU you have in these boxes. There's only a certain amount of IOPS you can push through these machines. So you need to make sure that you're picking the right blend of processor, disk, disk type for the workload that you have. If it's IOPS optimized, you probably want to go for the SSD approach. If it's real about cost and capacity, you want to go for spindle, it has a huge impact on a number of disks, even a networking here. So we publish a lot of reference architecture guides, and we've got more coming out. We've had, I think, one with SuperMicro. We've got one coming out of another vendor soon. We've got a bunch of these. And readies, these are 40-page documents, lots of amazing detail, which really helps you understand what kind of hardware you should be purchasing. Because once you bought it, you're not going to be able to swap it. And you want to make sure that you really are going to get the performance you need out of your SEF cluster. So this is just an overview slide again on the site. We've got this broken down. If you want for an individual square in a cell, you can see sort of the specific SKUs that we have. Many vendors now have dedicated SEF SKUs that you can buy. OK, so hardware, very, very important. The second thing is topology. Do you want to go hyperconverge or dedicated tier? That is, do you want to co-locate the storage with the compute, which may make sense in very small sort of pops, or sort of limited deployments, where you really are sort of constrained by space? Or if you're running a large public cloud service, you want to go dedicated tier. Again, it has different networking implementations, implications, sorry. So make sure you know which one you're picking for the growth and the use case that you have. And the final one, and probably one of the most important one, but also one of the hardest to distill in the PowerPoint slide, so I didn't, is the tuning. And again, Will will talk about this in a second here. There are a lot of settings in SEF. There are a lot of settings which will squeeze performance out of your SSDs, out of your hard disks, out of your network. Knowing which settings to make based on your environment is a little bit of a dark art. This is definitely an area where Red Hat help and can help. So I think even one of our solutions architect was saying, customer border, all SSD array, they installed it, complained the performance sucked. The solution architect went in, made four settings, and bang, the thing was sort of doing forex on a profile. Just do some basic kernel settings and some SEF settings. So tuning is all. And with that, I shall hand over to Will to go into some details. Will Bogie, Hi everyone. My name is Will Bogie. I'm an architect at Target Corporation. I work for our enterprise private cloud engineering team. And we are basically the private cloud team at Target. We deal with mostly open stack. We do responsible for object store and as well as scale out block storage, which we use SEF for. We're a full stack ownership team at Target, which means we kind of untraditionally for an enterprise we deal with pretty much every facet of our service that we own. So just to give you a little history, our first SEF environment and Target went live in October of 2014. It was based on Firefly. And SEF was the storage backing for our first official open stack release. And basically we use SEF for RBD for our instances and volumes within open stack. We use RADOS gateway for our object storage. And that's instead of Swift. Eliminating that component was kind of added some simplicity to our implementation. And also we use kernel RBD for a few different things. But the main thing is backing salameter for our Mongo database volumes. And when we implemented SEF, we replaced the traditional array based storage solution that was in our prototype Havana deployment. And we kind of discovered that especially back then a lot of the storage vendors didn't really have those integration pieces to open stack. You desk them, open stack, what's that? We don't integrate with that. If you can make it work, great. But what did work really tight was SEF. I mean SEF has a really tight integration with open stack. Always has. They've kind of grown up together, so to speak. So what do we do with our open stack cluster and our SEF? Our development environment is probably our largest. We use it for small internal workloads. A lot of Chef cookbook development goes on in there. A lot of people like to use it for just kind of scratch infrastructure. Like if you're going to test a new piece of software application, it allows them to get infrastructure very rapidly and reliably. And also a lot of our CI CD pipelines are hosted in there. Jenkins, Drone. We have a lot of different pipelines for a lot of different teams, so they're typically hosted there. We also do have two production regions. They host a lot of internal marketing workloads, business intelligence workloads, as well as some customer facing applications. Our initial SEF rollout, we had our development plus our two production regions. The requirements were to build out a object slash block storage solution to support our open stack environment. Pretty standard. Three monitor nodes. We used 12 core blades. We had 12 OSD nodes in each site. And they used the large form factor SATA drives. They were four terabytes. We had 10 OSDs per server. The journals were co-located on each OSD, which yielded us about 400 terabytes per location. We had a pretty basic LSI mega-rade, whatever they call it in, whatever your server vendor. It's probably the same card. And 10 times RAID 0. We had 10 RAID 0 devices on each OSD. After we rolled that out, it became evident pretty quick that the response times were too damn high. So we'd get a lot of anecdotal evidence, young transactions taking forever. Instances would boot. Sometimes cloud and it would time out. That general user griping, hey, it's slow. What would he mean it's slow? I don't know, it's just slow. But if we would add some instrumentation to our deployment, we'd see things like, and in the graph down below there, we'd be getting about 40 milliseconds of IO weighted, around 350 IOPS, which is not terrific. Also, just doing some benchmarking inside of the instances themselves, we would be getting pretty poor performance, especially right performance. So what could we have done differently? There's a few different tasks. And Neil kind of lose some of them. Hardware selection, monitoring, tuning, and understanding what our users are going to do with our environment. So first, you've got to understand the hardware requirements for your workloads. Just like Neil said, fewer disks that are better almost always will outclass more cheaper disks. That depends on your use case.  It all depends on how you're architecting your solution and what it's going to be used for. So don't be greedy if you're building to have a performance block solution for OpenStack. 4 terabyte SATA disks may be not going to cut it for you. So in our case, when we were designing our environment, we kind of overestimated the customer desire for object storage solution. And we underestimated their need for block, which kind of skewed the use case more towards having a more performance storage solution. So what we did was we ended up redeploying our OSD nodes, 24 core 2U chassis, the 24 small form factor. In our OSD nodes, we have 20 10 case F spinning disks, as well as six SSD journals in each box. I'm sorry, four. That's the type of that wrong, sorry. Because that equals 26, not 24. So a couple of tips on SSD journals. You need to pay attention to the SSD vendor that your hardware provider will give you. There is a great deal of disparity in SSD performance in the industry right now. So definitely do your research. Understand what CEPH requires for an SSD and make sure you're being proactive with your hardware vendor and not just accepting what they give you based on a SKU that says SSD disk. Talk to them, understand what that disk is, do some research about it. Make sure what you're getting is going to work for CEPH. We also had some additional improvements. We added a Mega RAID controller that had actually a write back cache as well. The next thing we could have done better is have better monitoring our CEPH cluster right from the outset. We kind of relied on using traditional monitoring tactics to understand what was going on there. And it's kind of a unique animal. CEPH has a wealth of data about what state the cluster is in. But a lot of it you just can't see from if you type CEPH-S. I mean, it says health OK. But the pro tip that I found is if you form, export it in JSON, it gives you way more verbosity about what's going on inside your cluster in a very easily parsable format that you can parse and ship up to monitoring solution, be it Nagios or Patrol or Datadog or a solution like that. It makes it so you really don't need to. There's no excuse to say my hardware or my monitoring vendor doesn't have a SAP integration because you can pretty much write it yourself in about two or three lines of Ruby. Monitor the user experience within the instances themselves is not good enough just to look from the outside in. One thing I actually do is periodically stand up a heat stack that runs FIO within the instances and pipes the output to a dashboard. So I understand the type of IO the user is seeing and not just what it looks like from the outside to the bubble. We initially stood up Calamari. It was a decent start, but we kind of outgrew it pretty fast. It provides some rudimentary metrics. But one of the things that we didn't like so much about it was it can only get really a week of history in the Calamari interface, which we wanted to do a lot longer trending over time for capacity and performance and things like that. One other additional thing, Seth, like Neil mentioned, you really need to tune your workload. And it's not something I can really stand up here until you hit these switches and buttons. Seth has lots of buttons and parameters. As far as which ones to engage, Red Hat Support would be a good tool to engage for that. As well as your x86 hardware vendor, I would say most, if not all of the major hardware vendors have some sort of Seth competency now, and they can give you some best practices around, hey, you need to set this to that, and you should set this to that based on what your intended workload is. But in general, there's a few things to be mindful of around tuning Seth. Default recovery priorities, especially when in our old Seth environment with SSDs, any recovery would basically just destroy client I.O. It would just grind to a halt. So you really need to understand how to tune those recovery priorities, balancing speedy recovery with the client experience. Deep scrubbing, there's the maintenance deep scrubbing tasks at Seth performs. Just by chance, occasionally, they can converge on the same node and cause a kind of noisy neighbor type effect, ensure that you're tuning that to minimize that impact.  A lot of times what you'll see in a Seth workload, maybe that's poorly tuned, is you'll see spikes of I.O. as the journal fills up, and then it crashes while it flushes a journal and spikes that sawtooth I.O. pattern. And that really tells you that you need to really pay attention to your journal sizing and your flushing. Also, this is more on the open stack side, but there are a lot of settings for LibRBD and Nova for caching. Maximized performance, you want to ensure that you have all the proper caching settings set for your compute nodes as well as your Seth nodes as well. The results, after redoing our design, we have improved latency in an extreme way. We understood the impact of our changes because we had the instrumentation in place to understand that as we go, we could see, is this having a positive impact? Is this having a positive impact? That's not OK. Take that setting out, take that hardware out. We were able to see as we go and understand. Because a lot of times, anecdotally, I don't know about you guys, but we could try things in a sandbox environment, but until we really start putting it out in the wild with actual user workloads, actual user patterns, you don't always see the problems. So you have to have an approach where you can monitor what you're doing as you go. We've improved our IOPS significantly. User feedback, extremely positive. We kind of got hailed as a hero a little bit for a lot of our open stack users because their storage performance was so poor that, and now it's quite excellent, actually. And that's the example of our IO weight, and you can kind of guess when we implemented the new hardware. So just wrapping up here, some of our next steps as part of the target cloud team around stuff, we need to start being proactive about our physical world, specifically our data centers. And typically, till now, our data center strategy has been kind of, we're going to put the server in the next available slot, which doesn't always work well when you're trying to optimize like availability through crush. So what we're trying to do is create a whole rack unit of cloud expansion. It includes the compute nodes, the monitor nodes, the CIF OSD. So when we fill up, we basically just wheel in another aggregate unit of our open stack slash CIF storage. That allows us to use crush for better availability, at least on a per rack basis. One of the things that I kind of see as a misstep when we put it in that I'll be working to rectify is, when we spin up an instance, we don't really limit the IO. So basically, it's all you can eat IO buffet. So your performance in your instance is basically whatever CIF can give you at that particular moment. So we're going to be working to implement some limits around that or reasonable limits. And we're going to, the approach I think I'm going to take is be using KVM, QEMU settings, and set that in the flavor. So in a particular instance, a flavor will only give you so many IOs. And also that will allow us to leverage sender to add different tiers of IO. Normalizing that will help with the fact that if you don't do that, typically you're going to run out of IO in your cluster way before you run out of capacity. So you need to kind of understand that curve and size your instance appropriately and give users a proper amount of IO. Experimenting with different drive types, SATA, SSD, just trying to maximize that economy within each OSD node so we can get the most capacity, yet still being performant to our user's needs. And also just working to trying to broaden the sef beyond the cloud use case. So in conclusion, form barking, have a good idea what your objectives are. If it's capacity, object, you can go for those higher capacity drives. But if you want performance, you have to really understand how to architect that. If you make decisions poorly from the outset, what it can do is it can negatively impact anything that relies on it. So in our case, our OpenStack offering was kind of got a black eye just because of the storage performance. So you need to understand what's going to be leveraging sef and ensure you're meeting your customer's needs as a storage solution. Unless you're architecting for raw capacity, use SSDs for your journal. That's pretty much without exception. But if you have to for some reason, then I would recommend using a write back cash card within your OSD notes. Research, experiment, break stuff, let red hat help you when you break it, cry. Monitor, monitor everything. Monitoring is super important with sef. That's kind of one of my main themes. If you know what's going on for your customers do, chances are they won't notice it. But if something does happen where the customers notice, make sure that they have a short feedback loop to talk to you. A lot of times, sef errors can be sort of a snowball. Like a pebble rolling downhill, gathering up snow. So if you catch it quick, then a lot of times, you save yourself a lot of headache. What was that? Pass it back to Neil. So I'm just going to finish off quickly. We have a little bit of the roadmap. And then we've got five minutes of questions and answers. So the sef upstream community released dual or version 10.2.0 a couple of days ago, which is the latest stable long term support release by the community. So we'll be taking that into the Red Hat sef storage product and releasing that in the coming months or two. The main features we're going to have here, so going from the stack from the bottom up on the Rados level, this is a tech preview feature only. It's one of the biggest changes to hit sef actually in quite a number of years. Sef was written before SSDs were really very common. And so it assumed you're going to have a hard disk. You have a file system on top, XFS, and so on. And that's what it was optimized for. That's what it was designed for. Obviously, nowadays we have SSDs. And the file system itself can be a bottleneck. So we've now got a new back end or backing store called Blue Store, which replaces the old file store. There's no file system anymore. It's subtracted out. It does raw block writes. And as you'd imagine, that improves performance hugely. So indicative testing is showing orders of magnitude. We're looking 2 to 3x on different types of workload, but pretty much all of them, sequential, random, small, and large. So you'll particularly notice this on SSDs, but it's true of a hard disk as well. So this needs some time to mature. When you come out with something this fundamental and low level, you need to give it time to bake and draw out potential issues. So we'll have it in tech preview. We'll see the feedback we get, and then hopefully we'll be making out the default pretty soon. But this is beginning to open up use case opportunities for Cef to really back you a sort of hot or a sort of database-centric workloads. And actually we had a Pocono Live last week. We did a presentation of Cef running MySQL or backing MySQL, an open stack environment with some very impressive performance numbers. So we should be publishing. I think we may have already published a paper on that soon. The other thing, and we'll mention this as well, which is that traditionally it's been, I think it was at the IO Buffet. You said Cef has sort of been first come first served in terms of IO. And you can tune it, you can say give, don't do so much on backfilling and scrubbing and rebalancing, try and do more on a client. And, but we've also now implemented a sort of a unified queue. So as these actions are happening, we can sort of really ensure that client IO is really always getting the primary sort of CPU attention. So you should now see the technical way of explaining the 99th percentile latency should be much lower, right? The issue is not necessarily that the screaming IOPS, we have that now, it's about getting the consistency and we've made a lot of changes is to ensure that your client consistency is better over time. For a block user, the biggest one that a lot of people have been waiting for is what we're calling RBD mirroring, which is essentially volume replication. That's the ability to take copies of all of your RBD images, your disks and copy them over to another site for disaster recovery. We'll be wiring this up into the Cinderuplication API in as of now basically, but the first capability, even if you're not running OpenStack, you can still as an admin go and set this up and just say, you know, back up everything somewhere else, don't worry about the user sort of triggering that through the API. I just want to back up all things everywhere. We also have an iSCSI implementation now, not particularly useful for OpenStack because as you saw earlier, we sort of have our own native protocol wired up into these services. But for people who have non OpenStack workloads, you still want to use Cep as their storage system. iSCSI is obviously the link with Franca. So if you're running Windows or something like that, you can start to use this. VMware, we sort of do a little bit more testing, but we should have support for that eventually too. On the object side, lots of features, completely re-architected multi-side experience. You can do global clusters, have active active and sort of read and write affinity everywhere. So, you know, three distinct clusters, users reading and writing to any of them, single namespace, everything syncs up. So, very simple to use compared to previous implementations. Lots of Swift API fidelity around Keystone V3, we've got Swift Object Exploration, static websites, and native LDAP and active directory if you're frightened of using Keystone. Finally, from, then this is specific to our product, we're moving to an Ansible based deployment technology. So, a lot of time, you know, setting up Cep is the initial sort of investment of time and effort. And obviously Red Hat has a strong interest in stake in Ansible now. So, we're moving our deployment tools to that. And we're also gonna be having a new UI tool sitting on top, which we should be announcing soon, which will allow you to do full end-to-end lifecycle management of your Cep cluster, both to provisioning and the management and monitoring as well. And then the big one, and this is, again, this will be a tech preview feature. Similar to BlueStore, it's a big new thing, you need to give it time to bake, but it is now stable, that it is, it stays up, won't lose your data, is the Cep file system. So, we're gonna be working on a minute integration for that. So, you can use that as a shared POSIX store within OpenStack. And we should have the kernel driver available if you're running a REL guest inside OpenStack to be able to access that. Hopefully, an update to REL 7.2 in the coming weeks, in fact. And that was it. So, we have three minutes for questions, and I've been asked to invite people to their microphones on either side, if there are any. I see a lot of your presentations, I always talk about the SSD to HDD ratio, or the OSD to journal ratio. What I never really see talked about is the reasoning for that, and the failure domains that you are creating by doing these, you know, if you do a really high OSD to journal ratio. I never see that talked about. Even though you create this huge recovery failure domain, if that SSD dies. So, you're talking specifically around the journaling where you're using the SSDs as a journal rate. Yeah, I mean, most people, and he said it too, SSDs used for journals without exception. So, the rough guideline is use one SSD journal for every four OSDs, for whether those OSDs are backed typically by HDD. But is that a recovery statement or a performance statement? That's a performance statement. It's also, the challenge is a lot of people say, well, I'm gonna raid my SSD journals, because I wanna get some resiliency there. The problem is SSDs fail because of wear and tear. And so, if you're doing the same right to both SSDs all the time, which you would be even as the statistically, you're likely to lose both SSDs more or less around the same time. So, you don't actually get that much improved resiliency by raiding the journal, as well as a result like performance implication as well. So, yeah, you do create a certain failure domain risk, but as I said, our guidance, I mean, it's a failure domain thing as well. You really wanna limit it to no more than four. You can do more, but it's also a performance thing. So, it's a bit of both. But losing, if you've got a most normal-sized Ceph clusters from two, five, six up, losing four OSDs shouldn't have a huge impact on your cluster. I can help you too with the reliability side, so. I think we have time for one more question. Yeah, just a quick one. Just wanted to have your view between Swift and Ceph for objects. What's your positioning there? So, very quickly, difference between Swift and Ceph is one is architectural. Ceph uses a strong consistency rather than an eventual consistency model down at its lowest level. So, we only confirm a successful write-back up to the client once we've written it to say three copies or do you raise your coding if that's what you're doing? Whereas Swift is lazy, it's eventual. It'll do two and then confirm back. So, that's one. The second one is that we actually do a lot of work around the S3. It's actually almost in a way our primary protocol, whereas I think the S3 support in Swift is, I mean, there is some there, but I think it's fairly rudimentary whereas we're trying to keep good parity with the S3 protocol, which for a lot of customers is actually more important. They've got tools that are written for S3 and so on. I mean, the big thing, of course, is unified storage. We do the blockers and the file as well as the object. And so, if you're looking to simplify your deployment and you've got a limited amount of space, you don't wanna be using too many storage technologies and so that's really the big win that we have with Ceph. Thank you. Okay, all right, well thank you very much.


I'm Rusty Komori and this is Beyond the Lines on Think Tech, Hawaii. I was the head coach of the Punahou School Boys Varsity Tennis Team for 22 years, and we were fortunate to win 22 consecutive state championships. This show is based on my books, Beyond the Lines and Beyond the Game, and it's about inspiration, leadership, and creating a superior culture of excellence. My special guest today is the founder of the very popular award-winning Aloha Beer Company, and he's also the president and principal owner of Cushman and Wakefield, Cheney Brooks Real Estate Company. He is Steve Sombrero, and today we are going Beyond Aloha Beer. Hey Steve, welcome to Beyond the Lines. Hey Aloha, Coach Rusty, how are you?  You have one of the best smiles and the best laughs. And I'm so happy that we're friends, but Steve, I wanna ask you about your family.   I mean, it's such a great family affair, right? That's correct. And you know, a lot of people ask me, you know, why do you start a business? Why do you have your family working with you? And you know, my answer is nobody will hire us. I have to create my own jobs. See, you see, there's that laugh, there's that smile that I love. I wanna ask you, Steve, how did the Aloha Beer concept happen? The Aloha Beer concept actually was a project at my executive MBA program. I was at the EMBA program in band number eight at the University of Hawaii. And in 1993, I was asked to, we were all asked to write a business plan for a business concept that isn't in Hawaii, but should be. And so, you know, I had just moved to Hawaii in 1991. I didn't know too much. So I said, what am I gonna write about, right? So like most people do, they go to the nearest bar and I went to the Maikai bar in Waikiki. So I sat there and I said, oh boy, what am I gonna write about? And so I was observing the entire time a customer would walk up into the Maikai bar and the waiter will say, Aloha and a customer will say, oh, beer, you know, it's typically a Japanese region and the beer, so it's okay. And then the waiter would bring him a bottle of Budweiser and it just kept happening over and over again.  I said, wait a minute, there's something wrong with this picture. Hawaii doesn't have its own beer, right? This is back in 1993. And I said, okay, I'm gonna write a business plan for a beer for Hawaii.  So I said, okay, Hawaii has to have its own beer with this many visitors coming to the island. It's a shame that they cannot drink our own beer. So, and I said, what would I call it? And it was Aloha, beer, Aloha, beer. And sure enough, that was the name I gave it. And I said, I need to do some research. There was a beer called Pronto. We don't wanna make the same mistakes. So I researched about Pronto and I learned that it tried to come back, but by the time it tried to come back, the new generation didn't know what Pronto meant. It wasn't even Hawaiian, as you know. It sounds Latin, so Italian. And so I said, Aloha, beer is a perfect name. So I wrote a business plan. My professor gave me a double A and he said, Steve, this is such a great idea. Why don't you do it one day? And I said, okay. So I trademarked the name in 1994, DCCA. And I worked towards global trademarking. And I said, I'll do it one day. And then I did one of my big real estate transactions. It was the Don Quixote deal when I first moved into Hawaii and I had money available. So I asked my wife, I said, hey, I'm thinking about executing on my business plan. Do you approve? And she said, why don't you ask Pastor Wayne first? Because we go to church at New Hope Church, right? I said, we wanna make sure it's okay for you to be in the liquor business. So I reached out to Pastor Wayne and I said, hey, Pastor Wayne, I'm about to start a new business. It's gonna be in the beer and liquor business. We need to know that it's okay for me to be in it. And then he goes, okay, but before I can give an answer, can I try your beer first? So he tried my beer and he said, okay, I approve. Steve, I love hearing that story.  I wanna congratulate you on winning the 2023 Ilima Award for the best brewery. I mean, it is absolutely awesome. I mean, how does that make you feel? Just wonderful, you know, and I know it's not about me. It's about my team, my managers, my daughter Candice and Yusuke that have been leading it and our brewmaster Jeff and all the managers there and all the staff, you know, and they made it happen. I was just there drinking the beer and tasting the beer and paying the bills, but they did all the work and they deserve all the credits. So Steve, can we do a toast right now to celebrate your winning the Ilima Award and just the spreading of Aloha beer? So really cheers to you, Steve. Yes, thank you. And that's a blonde you're drinking, right? One of my favorites is your Aloha beer blonde ale. So absolutely love it, absolutely love it. It's one of my favorites. And Steve, you have so many various beers.   I mean, what do you feel about, what are your thoughts about having so many of those varieties? Well, you know, we make 15 different styles of beer and at any given time you'll have a dozen beer available to you when it comes to our brewery. And the naming of the beers really goes to my daughter. She has a very creative mind.   And she asked me saying, dad, what do we call this new vehicle? You know, it's gonna be made using real hibiscus. It's gonna be pink, it's gonna be great. And I gave her some three names. And so I go to the brewery and I said, so did you choose which one out of the three I gave her?  I said, what?  That's not even proper. He said, okay. She goes, yeah, no, don't worry. We asked all the women with that work here that waitresses and they say it's a cool name.  So my daughter has a very, very creative mind to come up with all these names. Steve, I completely agree with you. And Candice is brilliant.  And I absolutely love that beer.    And how important was that to you to really make it dog friendly? You know, we're in Kakako, right? And we're surrounded by a condos and residents. And I figured, right? If a man wants to go to Aloha beer every day, he better have a reason and that better be as he's walking his dog. So people walk their dogs and they come by, have beer and their wives don't yell at them because they're walking the dog. But in all honesty, I wanted to make Aloha beer a very dog friendly place because we have two dogs at home and we love pets. We love dogs. And I really think dogs are a gift from God and you wanna hang out with them.  And we also have, you know, Aloha beer logo products for your dog.   That's true. And I completely feel the way you feel. And when I'm with you at Aloha beer company, we love looking at all the dogs that come in and everyone's so happy. The dogs are happy, the owners are happy, everybody's so happy. I mean, it's such a great environment, such a great positive vibe that you've created there. Also our workers, our staff, they're all happy to be there because they love dogs, you know, and who doesn't like dogs, right? Cats are welcome too, but, you know, I'm personally allergic to cats. So I'll probably keep at this. In fact, we have a resident cat that lives under our purple house and his name is Hopps. That's funny, Hopps. And Steve, let's talk about your Aloha beer team, like you mentioned.  I mean, they're so happy to be there. You've created a culture of excellence with your team members. Well, what does Aloha, the letters Aloha, really truly mean to you? You know, when we interview people that wanna work for Aloha, that's the first thing we ask. What does Aloha mean to you? Cause you know, we're using the probably the most sacred name in the Hawaiian language. And so I wanna make sure we respect it and we honor it and that everything we do is respectable. And it's a great representation of Hawaii. And so that's one of the first questions we ask all our employees, what is Aloha? So if you ask me, what is Aloha? You know, it's really, I break it down into the acronyms. A is to acknowledge. You know, you need to acknowledge the person in front of you, whether it's a customer, whether it's an employee, whether it's your wife, whether it's your husband, acknowledge this first thing. The next letter L is to listen. Listen very carefully what they're saying, right? And they have something to tell you. And so you pause and you listen and all is to observe what they're saying and what their actions are. Cause if it matches then they're speaking genuinely to you and then H is to hold, hold that thought for a moment and then A is to appreciate them and show appreciation by expressing your appreciation. And that's really Aloha. It's acknowledging, listening, observing, holding and appreciating that is Aloha to me. I think Steve, that's gonna have to be Aloha for everybody. And Steve, you met the sumo grand champion.  And your beers, the Aloha beers are spreading throughout Japan as well, right? That's correct. You know, our mission is to spread the Aloha spirit. One bottle, one pint, one can at a time. You know, as they say in Hawaii, right? If can, can. And I say, if can, can. If no can, we have bottles and pints also. By the way.  One can, one bottle at a time. And I tell you, I just came back from a trip to Italy and they heard that I was an owner of Aloha Beer Company in Bandai or just embracing me. And they're like, I wanna go to Hawaii. I wanna go to your brewery and I wanna drink beer with you. And I say, sure, come. So I am the ambassador of Aloha through the Aloha Beer Company. Steve, one of my favorite things to do is to be with you drinking beers at Aloha Beer Company because the next day my cheeks are sore.  I'm thinking, what happened? Oh, I was with Steve. And Steve, you have both of my books. I wanna ask you, what did you like about the books? You know, I love both books. And, you know, the first chapters of each book, right? The first chapter of beyond the line, in fact, I have it right here, okay? Is the choices yours. You know, that is a very, very important thought because we actually have a choice to not read the book but we read because, you know, it's a great start. And if you really ask yourself a lot of people that fight with anxiety, depression, they don't understand that they have a choice to get out of it, right?  And then the first chapter in your other book, Beyond the Game, is make an impact, right? So if we choose and we have a choice to make an impact, I think you'll be living a pretty amazing life. And, you know, I've always wondered, why did you call the first book beyond the lines? Because one of the things that I asked is, you know, if Rusty is a coach teaching tennis, wouldn't you be teaching to hit within the line and not beyond the line? Because beyond the line is a fault, right? But then when you look at your cover, it shows that the ball is just enough that it's about to go beyond the line but it's still in the line. And so I said, okay, that's what it is. It's pretty profound. And then when I think about, you know, all the words, the terminologies in tennis, right? It's when you get the game started with what? Service. And when you make a mistake, it's called a fault. And when you're tied, you're called a deuce. And then when you're ahead, it's an advantage, right? It's not life. And then it's also love, right? The word love is in there. So I really believe you're onto something there, Coach Rusty, and I love your book. Every chapter I read, it's almost a book that all MBA students should have read. And if they did, if we did, we'd probably become better leaders and better managers. Well, Steve, you're so kind. You are my promoter.  Tennis, you got to hit it inside the lines. Yes. And so for me, I'm just thinking, so many people through the years were asking me, why are you guys winning? How did you win 22 in a row? And it's doing all those things that you should be doing so that it allows winning to happen. Winning is a byproduct. Making money is a byproduct. Results, performance are byproducts of going beyond the lines. And you do that with your companies.  You focus on creating a positive environment for your guests. You're always looking at creating win-win situations. And Steve, I want to ask you, you're the president and principal owner of Cushman and Wakefield, Cheney Brooks.  Why are you guys so successful? Well, you know, it's really hard to explain why, but it was done through a lot of trial and error. And I think you lead by example. And that is something I truly believe in. And as you say in your book, we're all coaches. We have to be coaching our employees and doing our best to give them a good example. And I always also believe that a fish, I grew up in Okinawa, right?  And so that I applied that in my business. The head is so important. I am the head. And if I am not, if I don't make sure that I'm okay, the company is not gonna be okay because the fish, the company always spoils from the head down. And then if you have a problem, if you want to know really how the character of the person, look at his eyes. Look at his eyes. Look at the eyes. Yeah.  What are some things that you feel the greatest leaders do? You know, and that's very true. I get to meet, and that's what I love about my business, I get to meet all kinds of people, good people and bad people. And I learn from their successes and I learn from their mistakes. You know, I have several billionaire clients and I try to understand how they became so successful. But at the same time, I see that the more wealth they accumulate, it doesn't necessarily equate to happiness. In fact, I have some of the most miserable billionaire clients that, and I think Calomoto, Genshiro Calomoto was a billionaire client that was truly unhappy. He accumulated wealth. That's him right there.  And so he became a problem. He was probably the most disliked Japanese investor ever to come to Hawaii. And so I basically approached A&B, Alexander Baldwin, and said, hey, you know what, let's see if we can buy everything he owns and if we can do this, you guys are gonna be heroes.  So I should probably buy more stocks, right? Anyway, so I got them to engage with me as a buyer representative. And for three years after that, I constantly pursued Calomoto. I went to his office in Tokyo, Ginza, went to his headquarters. I tried to deliver a proposal to buy everything he owned and his security guard would always meet me at the front door and say, get out. You're not welcome here. But he took the offer and presented to Mr. Calomoto. I did this for three years, hoping that I would one day get to meet him. And then the fourth year, he got arrested for tax evasion and the government took away his passport. And so he had no way to come back to Hawaii. And so guess what? He calls me and say, Steve-san, I'd like to meet with you. Can you come to Japan next week? No, you've been trying to buy my property and I'd like to meet with your person. So I went to the new Otani hotel, which is where he was staying. It was a, what do you call it? I guess he was forced to stay there. It's better than going to jail, I guess. But we went there and he shows up and then he pulls out a file and said, Mr. Sombaro, you've been trying to buy my properties for the last three years. It's always been the same buyer. It's always the same price. And so I think there's something about you I need to, I think I can trust you. I can believe you. So that's where it all started. And so in 2013, A&B was able to buy 30 homes in Kahala, 100 acres in Kihe Maui, and a mountain out in Waimanalo. And we got it for less than half price. And the day, so the day A&B bought Kahala and Mr. Kalmodo's estate is the day they made money. We got a heck of a deal.  What did you do? Well, I actually went to his blue house. It's called a blue house. It's his biggest property. And if you recall, if anybody remembers what was going on, he had statues put up in all his properties. He was allowing people to move in for like a $1 rent. And he was trashing the property and making it very uncomfortable for his neighbors. And so he was trying to buy more homes. And so the day I closed, I was so happy that we finally resolved and solved this problem. And so I went to the blue house and sat on a statue. And that's the statue right there. I sat on the lion's statue. There was a lot of other statues available, but that's the one I liked that I said, all right, this is it, it's a victory dance. And so I felt like I finally tamed the lion of Kahala. Wow, I mean, that's something, I like seeing you on that statue. You did tame the lion. And wow, Steve, I remember seeing Kawamoto on the news because I mean, he bought 30 luxury houses in Kahala. And then why was he trashing it? You know, that's a question I asked him. I asked him. Kawamoto-san, you know, why are you doing what you do in Hawaii? You're not really appreciated. And why are you making very, very uncomfortable and miserable for people, your neighbors? And he said this, he said, I came to Hawaii with good intentions. Like any investor I came to buy real estate and make money on real estate. But then over the years, I met some really bad people. And so I was taken advantage of by some of the contractors, some of the professionals and advisors. And so I said, I can play the same game. And so he basically became an angry or not so happy investor trying to get more property and to make a point. And then he realized that every time he did something weird, the media would come to him and interview him. And so I guess he really thought that he wasn't being rewarded for strange activities.  I mean, that's interesting insights right there to see what the reason was behind his actions. And I mean, it was so bad because Kahalas are premier luxury neighborhood. And I have to say, all of Kahala, all of the people that own homes there, they must be super thankful and super appreciative of you, right? Yes, especially Richard Turbin, right? During he came to see me and he embraced me and said, man, Steve, you're a hero. We love you. And then I was encouraged to apply for membership at the Waila country club.  And so in my interview, I can't believe this. I still can't believe this, but the membership committee all got up, gave me a standing ovation and hugged me and said, thank you. We want you to be a member of Waila. So that's why I became a member of Waila. I was welcomed with open arms. And so you should be big time. I mean, talk about statues. They should have a statue in Kahala and Steve. I mean, it's unique what you do because you're not a typical real estate company. And what would you say are like three or four of the biggest transactions that you've closed? Well, you know, we did the biggest transaction during COVID, during the pandemic. Everything was shut down. And I was able to sell the Waikiki Galleria Tower for 270 million. And then after that, very recently, actually this year I was able to sell the Tauna Pali Beach Hotel for more than 300 million. So these are mega deals. Deals that realtors typically don't do but these are more like investment banking deals. But you know, I go for the big one. I'm the big whale hunter because I know that when a whale is hunted the whole village eats. You see all that wisdom that you have, Steve. It's incredible.  Valuable lesson and I'm still learning. I mean, I'm still working progress but I know that my faith is what carries me. My faith is what moves me. In fact, in our company every Monday we have a prayer meeting amongst a special group of people our staff and employees and we pray for our employees, our clients, our customers and that's what's kept us going. Well, I love that you said faith right there. And Steve, I have to say I'm so proud that we're friends and I really, really want more Steve Sombro's in the world.  What you've done to help Hawaii and beyond is incredible. And I just want to thank you for taking time to join me on the show today and please come back, okay? You're always welcome at Aloha beer, okay? Okay. Thank you, Steve. Okay, cheers. Cheers. And thank you for watching Beyond the Limes on Think Tech Hawaii. For more information, please visit RustyKomori.com and my books are available on Amazon and Barnes & Noble. I hope that Steve and I will inspire you to create your own superior culture of excellence and to find your greatness and help others find theirs. Aloha.


The Nordstrom anniversary sale preview is available for you to take a look at what is going to go on sale when the sale starts I'm going to have all of the details of the sale down below It looks I would like to take a look at the beauty exclusives So I thought we'd pull it up on my computer and take a look. I'm gonna do a good old bargain or bust I haven't done one of those in a while So I'm gonna go through and tell you what is a bargain and what is a bus for the Nordstrom anniversary sale Beauty exclusive page did take a look at the Purses as well. There is one person particular that I have an eye on and I'll share my favorites with you as we go But let's get into it. I'm pulling up the link to the anniversary beauty exclusive sale, which I will have linked down below So right now I am on the entire anniversary beauty exclusives page We have a lot so there's makeup skin care fragrance tools and devices hair care bath and body So I'm seeing if you use this Kiehl's body moisturizer You get this ginormous refill, but I got a pretty good price And I'm looking at these slip pure silk white king pillowcase sets 220 dollar value for a hundred fifty four dollars. Oh, that's really expensive I use Lily silk So cases and I'm perfectly happy with those but slip is a really really great quality brand if you want some new Silk pillowcases lots of Kiehl's. I'm seeing I'm not as familiar with Kiehl's They have a few products that I enjoy like they hand salve is great. You save six bucks I mean hey if you need some this is pretty good T3 has the 1.25 inch ceramic curling iron for 113 though I will say normally around the end of the holiday slash black Friday T3 We'll have some really really good sales at least they did last year So I would hold off probably until the black Friday sale So I would almost say this is a bust, but if you really need a good curling iron T3 is a really great brand Okay See there also is a hairdryer here. Oh my gosh I'm not even gonna try and pronounce this but this is pretty good a fragrance discovery set still 220 But this is a very pricey fragrance brand. I don't spend that much money on fragrances. So that is not for me Whoo, isn't this a hefty price tag for La Mer? Moisturizing cream and $1,600 value for $1,330. It's still a bust. I mean that cream is beautiful I've tried it before but it's still a bust highly skin. Why did I not know they sold highly skin So these are lots of beauty tools. I'm not as familiar with them. Which is why I'm passing by. I there's not much I can say about it But just so that you can take a look Right, I do like the Kate Somerville exfoliate for $50 originally 68 That's a decent sale price. Nothing crazy. Now This is what I strongly dislike and I think it is technically bargain, but it's a bust For the Nordstrom Beauty exclusives. This is my bone to pick in orgasm blush duo a $60 value for $40 Unless that is the only blush you're using Why are they selling two of the same thing? That is such a bust not a bargain. We don't want the same thing I hate when companies do that Nordstrom come on man Like skincare, you know what I can get behind them doing that for skincare, but makeup No, mm-hmm, especially with dusty old orgasm blush I mean don't get me wrong gory orgasm is beautiful But we all have so many other blushes than orgasm. It looks like Nars has this velvet matte lip set for $49 that's cute, but I'm not really attracted to it If you like the lip injections, which I think kind of went retrending on tiktok $42 for a set again Nordstrom Okay, I will give them this Fix plus is a little bit more essential than two orgasm blushes But why do I need $42 of fix plus? Oh my goodness Trish McAvoy? We have a brow set $42 nobody on YouTube uses Trish McAvoy, but my mommy's is Trish McAvoy YSL has this mascara set for $39 if you like the YSL mascaras I find that kind of dumb so this is actually a really good bargain right here the ABH Deluxe brow kit You get two different eyebrow pencils though I don't really like the one on the most left most side because I don't know. It's too big I like a fine point. They do have the brow is and the brow gel, which is really good Honestly, this is quite a good value. So this right here is a bargain. We have a co-parry set I really like co-parry. They're one of my favorite skincare brands for like body care Now what I do go after for the Nordstrom beauty sales. It's often time not beauty. It's fragrance I love getting the Veluspa candles Veluspa and Nests are my favorite like high-end boutique candle brands And you know, I use my bathroom body works because I'm cheap But I will normally treat myself to at least one candle set during every Nordstrom sale So I see they have this Veluspa set of four petite candles for $39 This is something that I actually might end up picking up because I live in a smaller apartment I don't need a big three-week expensive candle. So I like getting minis for my apartment So this is something that I actually do have my eye on and then there also is a candle trail from Nest I'm gonna have to go in person to smell everything. But yeah, I'm excited to take a look at the candle deals Super goop has this glow sunscreen and glow stick duo for $41. I know super goop is really popular I have a few of their skincare items or sunscreen items that I enjoy but You got me messed up if you think I'm spending $42 on a shower gel I love Nordstrom, but sometimes I'm like I can't relate. Okay. I do know that better than sex is a holy grail for some people but why why I don't even think it's that good But you can get two mascaras or it looks like a travel size as well for $44 It's a deal, but I think it's a bust This also looks like a bust to me an extremely boring the mac attire I set an $83 value for $45 This looks really uninteresting. I know we can't get a super good look at the colors here But it it doesn't look that interesting to me. These are basic colors The mascara is okay. The eye cool is pretty good. No, that's boring. We don't like that I'm not seeing anything that's really calling my name. This is good. The super goop sunscreen for $48 set I do like that one. That one looks good Nars afterglow lip balm trio for $49 I've never tried that formula. So ysl I've never tried this formula either, but $50 for two if you like the colors, that's good This your attic lip set for $58. This is nice Dior is such a trendy brands right now And I think that this is actually a good deal It looks like you get a lip maximizer a lip serum and a lip glow for $58 If you like the hydrating Dior lip Set, I do enjoy this one. I think that that's a bargain right that well bargain is strong And then this one is also good if you do not have The pillow talk lip stuff from charlotte tilbury or you're running low. This is really good $59 for a lip sheet Which is an essential in my makeup collection this color right here. You'll use it for everything A lipstick and I don't know what this last one is The name at the top of my head or are these minis? I don't know. I hope they're not minis as long as they're full size This is a good deal. Okay the hyaluronic happy kiss pillow talk But yeah, I mean, I think this is a great set for a gorgeous bridal lip if you're getting married This is a good set. I love pillow talk for brides As well. Okay, so that's good. I do like that We have a wild mint and eucalyptus candle duo from nest. I don't think I'd like that Oh, this is a good one too charlotte tilbury always brings it pillow talk cheek set So you get the pillow talk cheek to cheek and The beauty light wand. This is a good set $54. I recommend this one. That one is nice Both of those products are gonna leave your cheeks So stunning. We have a sigma brush set. You can always get sigma on sale I'd hold off on that if you use the bifasail from lancôme. This is a good time to pick that up I used to use that for years boy smells. Hmm. I think I looked at this last anniversary sale I'd have to smell these because that's $65 and I need to see what size those candles are Westman a tillie a have they been in the sales before maybe I Didn't have my attention to the brand in the past, but they have these squeaky clean liquid lip balm duo and for $57 I don't think I would end up getting this but I'm intrigued because I've never tried this product from westman a tillie That color looks so beautiful. I mean $57 is expensive But I I'll take a deal on westman a tillie a They're a little Spancy and then we have the lancôme mascara sets which They've had for forever Okay, let's see hourglass So we have an exclusive palette which normally eventually ends up coming out at hourglass anyways It only stays exclusive at Nordstrom for a specific amount of time But this is a tree of powders to define enhance and finish the complexion from a lid from within glow Let's I hope I don't lose my space, but I don't know the colors Do we know if these are new colors? It doesn't specify. Okay ambient lighting powder in luminous bronze light that exists Ambient lighting blush in diffuse heat and in ambient lighting bronzer in luminous bronze light I believe all of these are repeat shades I will have to fact check that and like you guys know, but I probably own these So that's a little underwhelming, but I take it back if those are all new But I don't think they are but the hourglass naming system could use some work They just like reverse words and switch them in and out It's hard to tell some Trish McAvoy eye base, which oh gosh, it's been here since I've used that This one intrigues me Always I never end up doing it because who needs this much But the vitamin enriched face base duo set from bobby brown for 64 dollars If you like it if you use it every day This is the time to pick this up because it is expensive, but I've noticed this going on sale a lot So there are other opportunities, but this is a bargain of the bobby brown luxe essential eye palette Bobby it looks pretty it looks very very wearable, but that's something I'll be passing on These joe malone sets are always so cute The cutest I don't own any joe malone fragrances, but those always catch my eye I love the keel's ultra facial set or the facial cream is really good. That's a good set Only thing else interesting a couple perfume sets. So I tried The hourglass veil primer today actually I've never used it before I did a full face of hourglass today That primer is so good I am so late to the game on that and then the powder of course is really nice This is a good deal of really great staple products from hourglass 72 dollars. So it's like a 30 discount. It was a very good products from hourglass I do like that set, but nothing is like Exciting me. Oh, I do like this fresh skincare scent fresh has some really great products Okay, let's go on to the next page new brow. This is always a good time to get new lash I wonder if we'll see they usually have it on sale here another Joe Malone clone set that one is for men's that could be a good gift They changed the packaging on new lash. This looks nice. I get this every year This is the first year that I'm not going to pick up the new lash set because I've stocked up so much that I have so many on backup down But this is definitely something that is a bargain. These are really expensive and new lash really works I always get asked what lash serum I use. This is the lash serum that I use. It is amazing And yeah, so this is the first year. I'm not picking it up. I do like the new packaging, but I highly recommend that replica jazz set That's cute. I don't really like that scent though. It's not my favorite. It's not sweet enough for me A lemaire hydration set. This is actually a pretty good value given the price of lemaire skincare This is a bargain for sure. So you it's a five piece skincare set for 95 dollars, which is Crazy. You got got the cleansing foam and I'm sure these are really tiny treatment lotion, renewing oil and creme de la mer which I love, but I like that set. That's a bargain. It's gonna last you about two weeks, but it's a bargain Okay, this is 110 dollars for those little things Boutique candles man. I love them, but I hate them. Chloe. That's cute. I love the aesthetic of Chloe just in general Um, I know the shantakai focils mascara is very popular 103 dollars for two if you use that regularly that could be a bargain for you It just depends if you're actually going to use the products or not shantakai has a couple of skincare sets. That's nice I love the perfume sets. I think they are so beautiful The Gucci ones always stand out to me as well new face new face is really trending It really does like I've seen it work instantly where people will use it and their draw line is more snatched So this is a good deal 135 dollars for the set and the cream. I'm kind of interested I don't know that I'll actually bite the bullet and get it because that is not something that I need But it's something that I want so I I can see myself adding this. This is on my wish list. Okay. I like that Oh, I like this. What's minotillier the weekend set? 140 dollars. It's still expensive. It's not that good of a bargain But you get a cream highlight a cream bronzer. It looks like and cream blush They're all good products still like kind of expensive though. So in terms of value, maybe not the best This is kind of good For hair lots of perfume sets. I'm not a perfume expert to wall sue first care serum duo I do really love this serum. I actually got to do an ad first to wall sue with this It is very very nice. It's a bit excessive to have two But if you love it, it's a nice serum ghd Love ghd strainers. They last forever charlotte tobury magic cream set 196 dollars apparently a 280 dollar value.  I'm over that 22.  Oh, okay. This is the set I always get getting this one the valuspa Set of five tin candles for 36 dollars. That's going to be mine. And so I know I love this deodorant from kupari That's a good deal.  This is a good deal too. I like these sets from briogeo bargain Um, this is cute. I'm always tempted to get these from mac this lipstick set You get five mini lipsticks for 35 dollars and it's just a really great way to get a bunch of colors It's a great gift to give if you have a teenager. It'll give them a nice collection I like my home too. Um, these two lip sets from mac. I think are a bargain. I do like those My college roommate used to wear this stila eyeliner exclusively. This would have been perfect for her I haven't used that in years and this mac set actually looks Kind of valuable Okay, okay This is nice from benefit, right the gimme brow and brow pencil for 33 dollars. That's good. I like that That's a bargain The bobby brown all these are all boring to me I could never get into these eyeshadow pencils from charlotte topary. They're good. I'd have to what colors do these come in light pink 35 dollars. That's good. Actually, that's a pretty good value if you're into that kind of product Bobby brown also has a similar one curl liqueur Ooh cure liqueur. I thought I said curl And that's it for beauty. Um not into it. Normally charlotte topary has a palette Or something an hourglass has a palette, but it looks like the hourglass is a repeat Checking to make sure I didn't Miss anything. Yeah, I don't like the beauty deals this time. There really is not that many that interested me There's only 44 Makeup deals and they're like the double orgasm like who needs that No, I'm not I don't know that I'm gonna pick anything up. I'm gonna pick up the candles for sure But I'm not screaming for the beauty products. I really am not Let me know your thoughts. Do you agree with me? Now? What I did see on the anniversary preview that I was interested in I haven't looked through the clothes and shoes. I usually find an item or two that I'm interested in but in terms of handbags The only ones well, okay the weekend travel bag. This is a really good deal from base. This is shea mitchell's brand $67 for the weekend travel bag. I just bought this for my mom for her birthday and I got it in black and I'm Super angry that this deal does not exist because macha and sakura look at these colors $67 I already have so much luggage and I also don't go anywhere But if I did travel I would buy this. This is such a good deal and the bag The bottom part zips off and you can put shoes in it. I really like that This is a good time to get base stuff And then what else did I see? And then the only other item that really interested me were ones from coach And they have a few colors on the norsham sale. So like this one is $199. I think that that is so cute This is the one i'm looking at either the black or the This one. I really like the nude. I think that I might get this one right here But it's so beautiful and it's $200 originally. It's 350. That's what people are normally paying This is not an outlet bag. This is a retail bag. So these don't go on sale So for me, this is the biggest bargain of the sale in my opinion that I I want to get $150 off Oof, I love that. Okay. Yeah, I think I'm getting this I limit myself now to buying one pricier bag a month and this I think is going to be my july bag I Love this taupe color. I think it is so cute. Yep. That's going on my wish list. That's what I'm getting anyways, you guys That was my nortrum anniversary preview sale thoughts bargain or bust for the most part pretty well underwhelmed Um, I'm gonna take a look at the clothes and the shoes. I usually like that but in terms of beauty I'm not feeling it. Let me know your thoughts down below. What are you interested in? What are you passing on? I would love to hear it And with that being said, I hope you guys enjoyed this video and found it helpful Thank you so much for spending your time with me and I will catch you guys in the next one. Bye guys. Have a good one