Let's see. Thank you for coming today. Let's see. My name is Andy Robb. I am the technical product manager for the big fast data team at Walmart e-commerce, sort of the e-commerce side of Walmart. Couple things. Our presentation is in two parts today. I'm gonna go through some introduction stuff and then hand it off to Mingming who will talk about some more things in depth. I will try to get through my portion of this relatively quickly, so that Mingming will have as much time as possible and to get it to any questions. I believe today's session's being recorded, so if you do have any questions later, please make your way to one of the microphones. All right. So Mingming, Rae, and I are on the big fast data team. We are actually one of two teams at Walmart that run large multi-tenant Hadoop infrastructure within Walmart globally. Our team has been running shared infrastructure since about 2012. Our clusters are in the dozens of petabytes and tens of thousands of cores range. Those are shared amongst all of our users and generally for research purposes. As a result of that being research targeted, somebody will come on, run a research job, come up with something that they really like running and then decide they wanna run it all the time. Of course what happens then is that it needs to run with a deadline and it's a research cluster. Somebody else can come on and run any job that they want at any time and it's very hard to help someone get their job to end at the right time. So that is one of the leading reasons that we started a project about a year ago to be able to deploy single tenant clusters in our open stack environment. These clusters might be yarn clusters, they might be Spark standalone clusters or they might be Facebook presto clusters. In addition to that there were also a lot of teams that had varying software version dependencies. Somebody either wants to try out something new or stay on something older. We also don't like using streaming applications on our shared infrastructure. Those jobs essentially never end. Once you start one you're always going to use those 24, 60 or 100 cores until the end of time, until your job changes and you're told to do something else. As a result of that in a shared environment that's not really fair. You are essentially taking cores away from other people's use cases. The idea with the shared infrastructure is you use it for a little while and then you give it back. All right, also we wanted to be able to independently scale, and this is sort of a very traditional thing, scale CPUs and data storage independently of one another for isolated use cases. In some of our cases we did in fact build dedicated Hadoop clusters for teams that had very demanding or very revenue impacting jobs.  And so we realized that we were able to cover a bunch of these use cases by using our open stack infrastructure. A different team from us actually started building both the open stack infrastructure itself, that's not us, as well as the SEPH storage infrastructure that really enabled us to do this work. They started that back in 2016. So for this talk, we will be talking at a very low level. So I apologize if we go over a couple of people's heads at one point or another. We are hoping to be able to talk to essentially the contributors and operators of SEPH, SWIFT, and open stack installations. Anybody who runs Hadoop ecosystem technologies that talk using the SWIFT API. Community members from Hadoop ecosystem that deal with the file system interfaces and then potential operators and highly technical users for the Hadoop ecosystem components. That's anything that can really talk to the HDFS interface. So that's MapReduce, TES, SparkJobs, even Presto. Anything that you can implement something underneath those to talk to another system. So what specifically we're talking about? Well, this is the layer that does that interface. So in Hadoop file system land, there's an object interface that you can implement essentially an arbitrary storage mechanism underneath the HDFS API. And so that's the software layer that we're talking about that we're working with here. The thing that actually enables a Hadoop component to talk directly to a SWIFT API. Thank you to Comcast for providing the diagram for this back in their talk at OpenStack in Tokyo in 2015. So this is built on a lot of existing and very good work. First and foremost from the OpenStack Sahara team that has a subproject called Sahara Extras. In that subproject is essentially the canonical implementation of this driver that we just sort of refer to generally as Sahara Extras, but it's really a small piece of that overall project. Also, there is a SWIFT driver built into Hadoop itself that we believe is actually a Sahara Extra. And then Comcast has also done some really great work with the Sahara Extra implementation patching that and helped guide us in our early work with this. Thank you to them. The general architecture that we're using for all of this is that persistent data is stored in object storage. The clusters that we have on our OpenStack compute infrastructure are totally ephemeral. Even when you've got large ephemeral disks attached to your worker nodes, the idea is that those clusters can be blown away at any time and don't lose anything in the process of that. We accomplished that both through the use of object storage and a shared hive meta-store that all of the ephemeral clusters can talk to. So if somebody loads a table in the data stored in the object storage, the metadata stored in the shared meta-store, you spin up a new cluster, whether that's yarn, spark, or presto, and you can immediately query data from that. There's no weighting, there's no loading data into that new ephemeral cluster, it's just there for you. The other thing that was really nice about the way that the API was implemented, the way that the Sahara team did this, is that it allows you to plug it into an existing cluster, and so we're actually able to add this invisibly to our users onto our existing persistent infrastructure so that we can actually load data into object storage. We only do it on the same data center, but we are able to do a distcp from one of our persistent clusters out to object storage so we can have jobs that may run primarily on a persistent cluster because we haven't migrated them over to using ephemeral clusters yet, and copy data over that way. That does not include the meta-store, so we would have to have some job that would in the ephemeral land go and add, say partitions to a table over on the ephemeral side in the shared meta-store, but the bulk of the work is able to be done by some large infrastructure. So in Ceph, you have the option to use Swift or S3. Both APIs are supported from Ceph's object storage implementation. On the S3 site, well, I guess I should start with, we actually spent several weeks when we started this project, trying to figure out which one we should use. We really were interested in supporting both. So there are lots of projects that support S3 and we wanted to be able to take advantage of those. Swift was also interesting to us. There are some projects that primarily use Swift and so we were interested in the notion of using both. Why can't we do that? The software supports it, right? If you are loading basic objects, you can use both. So at the binary level, you can load a file with Swift into a Ceph-backed storage system and pull it out with S3 and it's fine. The problem is with pseudo-directories, they're incompatible. So as soon as you start doing something with any complexity, especially with a Hadoop workload, you're not gonna be able to use the two of them together. So that was a point we realized we had to pick one and stick with it. So on the S3 site, broad client-side support, awesome. Unfortunately, when we did try to start using some S3 APIs, we realized that a lot of clients are built only to talk to the canonical implementation of S3. I'm actually pointing them at a URL in your infrastructure sometimes is impossible without a patch. They just don't support the notion of a URL that doesn't end in Amazon.com. Then there's the general concern around a closed standard. The community doesn't own the S3 server-side implementation and so if we need to make changes to it, we don't really have the option to do so. On the Swift side, client support's not universal. Unfortunately, well that won't get better without adoption. So if we go and start using this and request that tools that we want to use on top of our system use Swift, we can influence the community to more broadly support Swift. And then in theory, tweaks and changes can be made more quickly with Swift because the community does own the spec for that. In the end, we did decide to go with Swift. Hopefully that's relatively obvious. All right, so we are using a relatively older version of Sahara Extras. There are a couple of different branches of it. We're using one called Ice House with patches that we had some help with. Some of the issues that we ran into immediately. We tried to run some Hive queries on ORC stored data. They essentially fail if you're doing anything with really any moderate sized data. There were an uncontrolled number of HTTP connections. We'd run a query and we might get 10,000 connections per node. You're gonna overwhelm a couple of RGWs really quick with that. There were really slow meta operations. So deletes, renames, copies, especially with eye object counts. Those just took absolutely forever. When we did run larger jobs with lots of objects being returned from a list operation, they were being truncated.  Well, it turned out as far as Hive was concerned that data didn't exist because the files that were storing that data just weren't being returned. They weren't being made available to the system. Well, we did run some tests with long running processes, so in this case, Presto, where the application actually stays resident. This would probably also be applicable to Impala. We would run some queries one day, we'll go home, come back the next morning, run the same query again, up arrow, enter, and the query would fail. And we had to restart the cluster to fix that. The API wasn't re-aughting against Keystone properly. And then finally, we couldn't get large object support, which was at least partially implemented in that branch. We couldn't get that working for us. We couldn't get it to correctly break a file that was larger than five gigabytes. Okay, so why did we go off on our own a little bit? We spent several months patching the existing Sahara extra code base such that we could actually return those patches to the community. We realized within several months that that was taking a really long time. And that there were some fairly dramatic changes that we needed to make that we couldn't do just with pull requests. So we did essentially an experimental side project where we said, all right, let's just try making a bunch of changes and don't worry about making it look nice for a pull request. That ended up being very successful for us and we were able to add some performance features and make some modifications to fix a bunch of the issues we were running into very quickly. We also changed the name. So Swift A is intentionally slightly different. It's a bit of a nod to S3A so that we were more easily able to test. So even the class name that we're using is Swift A instead of just Swift so that we can load both jars into our application and just change configuration to switch between the implementations. Specific features that we implemented, bounded thread pools for things like listing, copying, deleting, rename. In some cases that adds parallelism. In some cases it limits the parallelism, multiple write policies that adjust how the driver uses local storage for the local worker that you're on as well as the upload behavior. Redesign the range, seek supports that we could run high queries against our C files, implemented pagination so that we could get more than 10,000 objects back from the server side. And LRU cache to limit header calls and stack calls against the API which depending on if you're reusing a particular set of files can be themselves overwhelming and slow some things down. And then something we call lazy seek to adjust when we actually issue a rest call to the server side that really sped up our presser queries. Along with all of this, we added a small patch to Ceph's implementation that adjusts a performance penalty that we ran into with large objects that we'll talk about in a little bit right now. The large object support that we ran into was that, well, originally, we couldn't get the client side support to work. And so we built upon that to make that work correctly so we can split essentially at arbitrary size. And then we also ran in, once that was working, you would run Hadoop FSLS on a directory that has subdirectories that has large objects in it and has files in it. It doesn't really matter as long as it's got the large objects in it, this happens. So the directories are returned correctly. They look like directories. The files come back correctly, they look like files. The large objects, unfortunately, look like directories. That's how they're implemented, that's how the system works, but there was no indication in a list, the object you get back in a list call, that the subdirectories are actually large objects. And unfortunately, that's what you base a lot of work on. And so there were cases where jobs would fail or user scripts wouldn't work properly. That was itself problematic. The problem is to fix that, you have to issue a stack call for each of those subdirectories to make sure that they are or are not, in fact, directories. Doing that for a few is fine. Doing that for 10 or 20,000 subdirectories is really a drag and could seriously degrade performance on something like a Hive query where you're doing recursive, sorry, where you're doing recursive runs down a directory tree. So we actually added a patch to Ceph and some, a little bit of code in Swift A to adjust that behavior so that we could get back a little bit of information just to let us know that the subdirectories were these large objects and that dramatically improved the performance of queries and just general behavior of the system. We are hoping that that essentially a hack can be better implemented in the community as part of the Swift standard, which is something that S3 supports so that this particular use case can more effectively run. All right, so caveats to what we've done. We've not tested against a Swift proper cluster. We've only been running against a Ceph cluster with the Swift API on it. Because of that inefficient list mechanism, oh, we just talked about that. Sorry, I didn't go through these slides too closely just before this. Okay, the patch that we applied to Ceph, you can see in a pull request, 14.592.  So performance results, do you wanna take over? Sure. So, Meng Meng Lu is going to talk about the performance that we were able to determine from the system. Yeah, so we did a couple of performance evaluations to test our performance against Swift API versus Sahara extra against our Ceph storage clusters. So one thing we want to test out is how well our bounded thread pool perform. And we evaluate a couple of file system operations such as deletion, renaming, and even uploading. And we also evaluated the difference of our several write policies we implemented in Swift A. And we look at both file system operations as well as a map reduce benchmarks such as high bench and pick one of the jobs called word count. And note that here is the SPAC that we did our experiments. We ran in a couple of open stack VMs each with local SSD storage. And we tested against a Ceph HDD, a couple of HDD storage clusters. And they are implementing a Ceph version and with RGWs and HA proxy on top of that. The first result as you can see, this is comparing Swift A and Sahara extra on a single operation Hadoop FSRM. This is deleting a large directory generated by high bench word count. This is the largest scale big data. It's 1.6 terabyte of 6,600 objects multi-splitted into 256 megabyte chunks. And we can see that since Sahara extra inherently for this operation is a single threaded implementation, but Swift A is able to add multiple threads to essentially doing the deletion. And we can see some performance gains over here. The next operation renaming essentially, we are doing it on the same exact same settings as the deletion, but rename is essentially a copy and delete. We do this again comparing Swift A and Sahara extra. And we can see that since Swift A can enable multiple threads doing the renaming, essentially with only three threads, we're able to see three X performance gains and with more, you can see that we bring down the renaming performance from essentially about an hour to a few seconds. Now, here are the three right policies we implemented in Swift A. The first one is called multi-part single thread. And here we are dividing very large files into small splits. And this is called multi-part split. And in this single thread implementation, we only require a very minimum amount of storage on the local desk. We essentially only write to local storage one split at a time and upload one at a time sequentially. Now, the second policy we also enable is called multi-part no-split, where we essentially can save the whole file to local storage, assuming we have that enough local storage, and then we are able to upload via like byte ranges in multiple threads in parallel. And this is making uploading faster. The third policy is really a combination where we enable buffering into local storage only several chunks, several splits, subject to the number of threads we enable, and then upload them asynchronously from the local writes to the object storage. Here is the result of uploading a single 100 gigabyte file. We split them into several chunks, and this is doing a single Hadoop FS put on a single SSD compute node. As we can see that single thread one split is the slowest among the three, but it requires the least amount of local storage. No-split requires the whole like 100 gigabyte on local storage to do this uploading. And essentially a multi-part split here has the best performance regardless of the size of the split. And it tells us that asynchronous uploading in multi-threads really has a lot of performance gains. We also compare the three policies in running a MapReduce job, a high-bench work count. This is the 6.0 high-bench work count job and we had three scales, huge, gigantic, and big data. Know that the big data one generates 1.6 terabytes of data with 60 mappers and 60 reducers. And we set four gigabyte per mapper reducer, ran it on 10 compute SSDs. The spec of each compute node has 52 gigabyte memory. And we use the default settings within our Swift A thread parameters. We can see here that the split policy has the best performance among the three. LazySeq is also an important feature enabled in Swift A. It seeks only when necessary to read the data and we see a huge number of, it reduce a huge number of connection overheads to the input streams, which are common in Presto queries. And note that a feature similar to this has been implemented in S3A API as well in this Hadoop Genre ticket. Okay, so as future work, we plan to open source this after internal workload validation. It's mostly likely within the Walmart Labs GitHub repository. And we are also looking to investigate using local tier storage for buffering before multi-part upload. That is leveraging memory as well as local disk. We are also planning to look at multiple read policies to improve downloading speed. That is to fetching the objects from the object storage. We are also very interested in supporting both Swift S3A protocols at the same time, meaning that we can have the Swift client read data from S3 client generated object. And this requires a rewrite to essentially how the two protocols generate pseudo-directories because currently they put a zero byte file to different places to indicate whether there's a directory or a file. All right, I'm going back to Andy to conclude the talk. So first of all, so we were able to get Swift A2 scale for us and we were able to run some very large workloads with it internally for our testing. And we're using actually production workloads that we just happened to have snapshots of. We would like to merge this work. We don't really want to maintain a file system driver for all of time. So we'd really like to get this work merged back into the community as well as some of the changes that we've discovered are necessary for really performant operation. Get those made to the standards at some point. We would love folks help to do that merging and to just make the code better in general. And then again, for the large object support when it comes to pseudo-directories we would, there's still a little bit of work to be done in the community for that. You may have noticed a couple of oddities in some of our testing data. We noticed them too. We haven't necessarily isolated the exact reasons why some of those numbers were off a little bit or in some cases a lot. Those are some of the work that we've got left to do to figure out what those outliers were. So with that, are there any questions? Hi, did you exclusively test Swift Day against Swift provided by Ceph or do you have any knowledge of how it would perform against Swift, Swift, so to speak? We really only have Ceph clusters available to us so we weren't able to test against Swift proper. Okay, follow up. Does Swift on Ceph implement atomic rename? I've used disccp and noticed that of course disccp keeps temporary files and then moves them to a final file commit or staging location which cause massive shuffles because it's changing keys essentially. I'm wondering if Ceph has a similar problem there. So Kyle Bader from Red Hat on the Ceph team just said but no it's essentially the topic operation. Thank you. Yeah. Hi. So I work upstream with Sahara and we're definitely interested in your work. So let's talk about going upstream to like Sahara extra or something like that. That would be awesome, yes, thank you. Thanks. Appreciate you coming to the talk, thank you. Good talk and excellent performance improvements. A quick question though. Yeah. You talked about three issues, right? I can understand the performance improvements one to three to pull. You also talked about a large number of HTTP connections with Swift FS and I think a small performance with large object count. Can you talk any improvements in those aspects? Yeah, so specifically to this. Yeah, so we found that in Sahara extra a lot of times in some cases it enables thread pooling but it's uncontrolled so you cannot actually bound it. So for instance I'll give you an example that in Hive you have this MACK repair table statement where like for a very large table you would essentially read all the directories and file system metadata and try to load them into the Hive metadata and in that call we would essentially create an uncontrolled number of HTTP requests to the RGW side and RGW would be essentially overwhelmed and you cannot get any results back in that case. The other question was about the large directories. The large directory performance issue was directly related to the ability to get back metadata information about whether a subdirectory that you were listing is in fact a directory or a file, a large object. And so that required a tweak to both Ceph's Swift server side implementation as well as Swift A. Yes, good talk. Thank you. So in real case, for example, for your workloads large objects is more important or massive small objects is more important? Or both? It depends. So in our case we've noticed that if we break even moderate sized objects into smaller objects it's a little bit easier to parallelize the reads for them. In cases with large data sets to keep those large data sets in single containers just for logical organization purposes it's necessary to tailor the object size to get as close as possible to the limit without going over it so that you can keep the total number of objects in that container minimized. So it just totally depends on the workload that you're targeting or the particular data set that you're talking about, at least in our case. Yeah, the second question is when you bring a lot of threads in the client I guess in the client or in Swift APIs. It will help. So it's a when you issue a call like for instance for a MapReduce job these threads are not only on the clients but also on all the mapper like all the worker nodes of the. Yeah, so the question is, is there any overhead like brought by this design? Because now you bring a lot of extra threads it may take some CPU cycles from. I think in my understanding since object storage would only allow HTTP requests to fetch any object and hence those threads are inevitable and the only way that we can make it better is to have a better control of it. So we know how many threads are initiated from the client and on each of the worker nodes of Hadoop to the server, essentially.  We've seen a lot of problems, yes. One of the pieces of work that we've spent time doing is figuring out what the right number of threads to use is. I think even with some of the testing you can see highly diminished returns when you start really cranking up the number of threads. There are a number of reasons for that but yes, at some point you just get to a state where it doesn't matter how many more you add. You may actually degrade your performance. Okay, thank you. You talked about using dynamic large objects or the large objects support. Did you investigate using static large objects and what was that? Why did you choose DLOs instead of SLOs? I think we went with DLOs mostly because they're, well, they work a little bit better with our use cases which are always within a single container. And so there's a little bit more straightforward. There's nothing that crosses the container and the reading the manifest isn't really necessary for a dynamic large object. So you don't need to figure out where you're supposed to go to grab things. And so it was just a little bit more straightforward. I had a question about, you mentioned the impedance mismatch in S3 and SWF doing the kind of fake directory listing. Can you expand on that? Like in a lot of libraries, like for example, JClouds, you'd often use a similar approach of creating an object with a trailing slash of zero size which works on both providers. So what specifically did you run into that? I'm just curious. There's a zero byte file created on the SWF side along with the directory. So at the level of the directory, right? You create a zero byte file to indicate this is a directory, not a file. This is the SWF standard. On the S3 side, there's also similar but that zero byte file is placed at a subdirectory level. So that would make the income, I think that without code changes, these two clients are incompatible. So a lot of cases, we create data using one client, right? We create data using Swift client. Then those directories are arranged that way. But then the S3 client cannot read data directly. Sorry, you may need to explain a little bit clearer. I guess I was getting at the point where, say I'm trying to create a directory full in the subdirectory bar and create, in both places, I'd imagine creating a foo slash and a foo bar slash. And these are just objects, right? So because of the flat game space. So there's no actual notion of directory necessarily. So could you explain one more time? I must have been missing something here. I mean, it's essentially hacked in, right? If you're using just the Swift CLI, that's all you have to do for whatever reason. And I'm not totally clear on the details with, in order for Hive to see those as directories, you actually need to add a zero byte file that doesn't include the slash. And so you end up with two files, one, or two objects, one that is bar slash foo, no trailing slash, and then another one that is bar slash foo slash. Without that, Hive doesn't see the subdirectory as a subdirectory and you end up with a bunch of very strange data. On the S3 side, there's a similar syntax, but it's just slightly different. I think there may have actually, there's actually a string that you're supposed to append, like the literal word directory or something that normally gets hidden by the API. And this is all from by memory from a year ago from Google searches when we were trying to do this. And so it's not that they're totally, they're only incompatible in very superficial ways. And it's just that they're bad enough that it's frustrating to work with. Yeah. And I have to say, these are implementations of the Swift and S3A driver code. It's not something that created by, you know, the Swift or SAP backend. Looks like it. If you guys have any more questions, we'll be around afterwards. Again, we are really looking forward to working with the community on this and thank you very much for coming.


Wednesday I am currently as always in the dark in my car at boot camp it is a Monday that I'm filming today's video so it's my first day back to boot camp when I pulled in the parking lot nobody's here which is kind of strange because it's four minutes before boot camp starts I'm hoping that there's boot camp today after Easter I'm filming this the Monday after Easter I guess worst case scenario I go to the gym if there's no boot camp but that's really weird because nobody's here which is kind of uncommon so we shall see it where the day takes us but I did have about half of my protein coffee this morning while I worked on my computer a bit before heading out to boot camp I always need a little bit of coffee before I work out it's just it's a requirement I'll finish that when I get home we have a busy day today so if you're excited for what I eat in a day I give this video a thumbs up again subscribe if you're not I would love to have you here and don't forget to check out the description box down below for nutrition coaching for personalized macros and calories one-on-one coaching I'll link everything I share with you in today's video as always and don't forget come on over join our Facebook group we'd love to have you so I guess I'm gonna wait a few more minutes see if who shows up and if boot camp is not happening today I am going to head to the gym to get this morning's workout it dedication consistency is key good morning and an official welcome to today's what I eat in a day I actually just got home from boot camp got ready for the day just finishing up my protein coffee and then I'm going to head into the kitchen and we'll make some breakfast but I wanted to share my OOTD or my outfit of the day with you my entire outfit is from a clothing brand that I recently discovered called S Deer and let me tell you how fun and trendy their clothing is you know that I love a good high quality staple piece for my wardrobe something that I can wear long-term that doesn't break the bank and I've been getting into a little bit more trendy fun clothes as I've continued to lose weight so when I discovered S Deer I immediately had so much fun shopping on their website picking out a few staple pieces to add to my collection I ended up purchasing five pieces from the S Deer line starting with this adorable top that I'm wearing today this is the ruffled round neck chiffon shirt it comes in the color black and is in sizes small through extra large now I picked mine up in an extra large I will tell you right out of the gate their clothing runs small I normally wear a medium or a large and they let me know that their clothing sizes do run on the small side so go ahead and size up for everything on their website so I did pick up an extra large and this fits me perfectly I really like the ruffle detail on the sleeves and I love that it has this flouncy design so it conceals the stomach area and the back area and you know that the back area is a problem for me and it does cinch in at the way so it does create that really beautiful sleek silhouette it is very very high quality and very very comfortable this is a piece that you can dress up you can dress down I love the ruffle the ruffled neck it does have a zip closure in the back and like I said the quality is exceptional so I decided to pair them with these fun camo pants I used to live in camo when I lived in Washington and I ended up having to get rid of all of my camo pieces because they just didn't fit me anymore so I wanted a nice solid staple piece to add to my collection to add to my wardrobe so these are the personalized mid-waste camouflage cargo pants they do have a waist cinch pockets and they come with a belt they come in the color I have here which is dark gray green in sizes small through extra large now again I did pick up an extra large I'm glad that I did they fit me perfectly and I am able to cinch the waist in a little bit with the with the belt that's included with the pants they're really really lightweight material I love the cargo style pockets and how they cinch in at the ankle they are more of a cropped pants so they are perfect as we move in to spring and summer I'm pairing it with this fun a little bit more dressy shirt so it goes to show that this shirt is very versatile you could pair it with something a little less dressy like these fun camo pants or you can pair it with a pair of pants that's a little bit more dressy or even a skirt speaking of a little bit more dressy pants let me tell you how incredibly comfortable these next pants are they felt like I was wearing pajamas and they are so sleek so fun so trendy these are the casual elastic waistband zippered cuff sport black long pants they come in black sizes small through extra large of course I ordered mine in an extra large and they fit me perfectly I probably could have even got a large in these they are a little bit more oversized than I would say the rest of their pieces run that I purchased but I love these they have an elastic waistband but they are elevated it has the front that looks like there's a zipper so they can be a little more on the dressy side there's pockets in the back which also add a little bit of class and elegance and I am obsessed with the bottom of these pants how they have that silver zipper with that crossover black piece so trendy and so fun and like I said these are so comfortable I could see myself wearing these all the time these will definitely be a staple piece and then I wanted to pick up a skirt I love a good dress I love a good skirt and this black pleated skirt immediately caught my attention on their website and I thought that this would be perfect as we move into the warmer months this is the college style double breasted pleated black a-line skirt it comes in black sizes small through extra large I got mine in an extra large and I just have to tell you the quality of this blows my mind it is so thick it is double lined it does have the zipper closure here on the waist and the fun buttons in the front with some faux pockets this is so cute it's absolutely adorable it's long which I appreciate it does camouflage a lot of the loose skin on my thighs and the quality is so heavy weighted I think it looks really cute with this top as well and this is another thing because it's black it's definitely very very versatile I could see myself wearing a more fitted tank top with this in the summer it's so cute I'm so glad that I picked it up and last but not least I needed a new spring jacket it is still cool here in Arizona we have monsoon season where the temperatures drop and it and it rains and I do not have a single solitary spring jacket not a rain jacket not a spring jacket so as soon as I saw this one I loved the pattern the style the trendiness of this this is the hooded short jacket with drawstring letter printed cinched waist it comes in gray which is the color I have sizes small through extra large now I picked mine up in an extra large it is over size probably the most oversized of everything I definitely could have sized down to a large but I'm glad that I have the extra large just because it is going to give me a little bit more room could put a sweatshirt underneath and be able to layer because the weather does change on a dime here so it'd be nice to be able to layer this layer under the jacket it is kind of this camo style print and I love that it has the letters on the sleeve it just adds that trendiness that edginess it does have a hood and the quality again it's very weighted it's very very high quality I also very much appreciate that it cinches in it the way so that you can draw that in it's not big and bulky and boxy you can cinch it in and actually have a nice shape with a jacket like I said I can't believe the quality of these pieces every single one of them is really nice and I am so incredibly excited to have a high quality stylish trendy jacket for spring so those are the five pieces that I added to my wardrobe from S. Deer I can't wait to go back on their website and do a little bit more shopping because the pieces are so fun and trendy if you like black neutral colored pieces staples that you can mix and match with different colors if you would like or other neutrals S. Deer has you covered because most of their pieces are on the neutral side which again is great to have a curated capsule wardrobe S. Deer was so kind to offer a discount for you you can use the code Gen 15 for 15% off site-wide or you can use the code Gen 20 for 20% off all orders over $199. Have fun shopping have fun looking around their website you will not be disappointed with the quality and again just a reminder to make sure that you size up because it does their clothing does run a little bit small but I'm really excited to wear these fun camo pants in this adorable shirt today so it's time for some breakfast I am super hungry so I'm going to make a scramble I actually have some diced ham so I'm going to do a ham and cheese egg scramble so we'll do some diced ham eggs obviously I have some light shredded cheese from Trader Joe's and then I'm going to pair that with some blackberries I've shared a few times how important it is to keep your fruit in mason jars in your fridge these black berries are two weeks old and they are perfect it definitely keeps fruits and vegetables a lot longer if you store them in black meat in mason jars so I am going to put together my scramble pair it with some fruit it's going to be a high protein good source of carbohydrate breakfast so here is my breakfast this looks so good so I ended up doing two full eggs two ounces of diced ham and two tablespoons of light shredded cheese and then I did 60 grams of blackberries I'll go ahead and put points calories macros here on the screen for you look at her it's like 80 something degrees and she's just happy as a little clam out there I'm just finishing up my motivate amino which is my post-workout recovery drink and I have some very exciting news with motivate you know I love them I use a lot of their supplements I actually just stocked up I'm heading to New Orleans next week and I need to take my supplements with me you know I take my motivate supplements when I travel so I picked up some travel staples but they have a new product and a new product that I am absolutely ecstatic about motivate now has protein powder and you know I live on protein powder I use two protein supplements every day to reach my goal so when I found out that they had a protein powder I was so excited that I bought both flavors so they came out with vanilla and chocolate so one scoop is a hundred calories 20 grams of protein less than one gram of carb non-gmo 100% whey protein really clean ingredients very minimal ingredients it does have natural flavor which you're gonna find in a lot of products but it is sweetened with stevia I am so excited about these I haven't even tried them because I wanted to test them out with you first impression let's see how amazing they are I have no doubt that they're going to be great because everything from motivate is great but for my morning snack it is 90 degrees today so we're gonna make a protein shake so I think I'm gonna do chocolate I might do vanilla we'll see we'll head into the kitchen in just a moment for that but like I said I'm so excited that they have a hundred percent whey protein oh and side note super super affordable and then I stopped up on some of my favorite to-go things that I like to take with me when I travel and I've shared these before but I just wanted to remind you that motivate makes full-sized tubs of their burn their no-stim burn their Amino their greens but they also have little small packets so not only is it a great way for you to try motivate out but if you travel it's a great thing to take for traveling so I ended up picking up a five pack of the no-stim burn so this is the motivate burn without caffeine in the shade in Miami lemonade so good tastes like religious amazing lemonade I love these little packets because what I do when I get to the airport is I ask a restaurant in the airport to fill my water bottle up with ice and then I add in my favorite pack and again you can keep these in your luggage in your carry-on in your purse so I got the no-stim burn and then I also got the burn variety pack which is another thing that I really really like because you get five different flavors of the motivate burn so each day I can have a different one to kind of change it up and this is the one with caffeine so some days I want caffeine some days I don't so I made sure that I picked up both the regular burn and the no-stim and then of course I grabbed the Amino which is the post recovery which is what's in my water bottle I am doing a 5k on my trip so I need a recovery supplement and this is the pineapple mango so so good that's what's in my water bottle I absolutely love it so I grabbed a five pack of that and then of course of course of course I had to pick up the motivate greens packet I always like to take greens with me as well it's something I struggle with especially when I'm on vacation or out of town or out of my normal element I don't always have greens available so I always have a supplement and that's two packs of tangerine two packs of peach greens so I'm going to take those with me as well so again the sample packs are not only a great way to test out the motivate products but also take them on the go I keep a few of these in my purse if I work to regular nine to five job you can bet that these would be in my office desk drawer so there's the sample packs and then let's head into the kitchen and try out the motivate protein and make a protein shake and as always I will link motivate down in the description box for you with 20% off so I think I'm gonna do chocolate I'm kind of feeling a chocolate mint kind of vibe for my protein shake so we're gonna test out the motivate chocolate I'm gonna have this one tomorrow so I will put here on the screen what I thought of the vanilla protein as well and we'll do a true taste test of the chocolate so the first thing I'm going to do is measure out four fluid ounces or half of a cup of Fairlife milk I have a half of a frozen banana peppermint extract I'm going to just do a tiny bit a little bit goes a really really long way and then one scoop of motivate chocolate protein I will tell you it smells very very chocolatey if I need more liquid I always just add some water I'm going to pop the lid on and blend it up okay cheers let's test out the motivate chocolate it's so good it is so so good I put just the right amount of peppermint in there the banana adds a little bit of sweetness but the chocolate it just comes through it's a really organic chocolate taste not a fake chocolate taste I have no aftertaste of any sweetener it's really really good can definitely see myself putting this in my Greek yogurt making protein putting out of this making protein shakes 10 out of 10 really really delicious I will like I mentioned link motivate down below with 20% off pick up the protein powder amazing quick adventure out I actually just finished a coat finished a coaching call I have another coaching call in 20 minutes I am getting another coffee I am moved I was pretty tired after my workout and I'm just feeling extra tired I was at my neighbors last night for a little while for Easter and so I went to bed kind of late and whenever I go to bed late on a bootcamp day like a get up at 4 a.m. you have to get up at 4 a.m. I'm usually a little extra tired so I definitely need a coffee right now at Duncan they have their cold brews for $3 for the month of April so I ordered a cold brew with two cream and two sugar-free vanilla and extra ice I love that I can get extra ice because you know how I feel about Duncan in the ice situation and then I've got to run to the post office I need to mail a couple bills and a package off so I'm going to go do that while I'm in the car though I wanted to give you guys a little bit of a update kind of a sad update actually if your friends with me on my regular Facebook page not in my Facebook group but on my personal page you would have saw that my mom lost her dog on Friday it was pretty sudden for her and if you didn't know my mom is all alone by herself and her little dog Lucy was her whole life and so she's of course very devastated as you know I can relate to so I've been like an overprotective parent checking in on my mom several times a day I'm literally text her like three times a day so I'm actually gonna text her right after this but she's just devastated and she feels just really lost without Lucy so we sent her flowers from me Troy and the dogs and I feel like every day she's getting a little bit better but I don't know what is going on in the world but it's just been a really rough go around with dogs I mean first our neighbor lost their dog a few months ago and then Lola's diagnosed with cancer then we lose diesel then my mom loses her little Yorkie now Lucy was almost 14 and she ended up having a heart related issue kind of like a congestive heart failure it was really sad and I mean I'm sad for my mom because I know how much she loved Lucy I loved Lucy so I just thought that I would give you guys that little bit of an update so if you have dogs if you have fur babies in general hug them a little bit tighter because I don't know what is going on in the world but it has just been a rough go around with dogs I have a pickup order for Jen Clayton yes just coffee okay thank you there's not a line for the first time in a million years like I'm literally pulling right up to the window it is my lucky day hi thank you have a good one thank you coffee in hand medium cold brew extra ice let's test it out there is lots of ice in here super grateful for that the only thing I have to count points and calories for is the cream I usually count it as two tablespoons of half and half because I do two creams I don't know exactly what Duncan puts in but we know that counting macros and calories even counting points isn't an exact science so for me I counted is two creams equaling two tablespoons and that seems to really work that's actually really good and I am not a huge cold brew person I've actually shared that on my channel before I'm just not super into cold brew usually has a weird taste to me but this is really good and I almost added an extra shot of espresso I think I'll try that next time but three dollars you can't beat it all right post office check I missed my coffee while I was in there it is 1050 I have 10 minutes before my next coaching called just enough time to get home and I got a notification actually when I was in the post office that Palmer's bark box will be here today and you guys just love seeing that little turd get his bark box because he's pretty much the most in patient dog ever so we'll do the bark box a little bit later today as well I've even been thinking kind of about lunch and what I want to do for lunch I'm gonna wait till after my next coaching call and then I have another one a half an hour after that but that should give me enough time to eat lunch I've been really into these little like adult lunchables lately for lunch so I think I might do that again it just keeps me full and it gives me a good variety of foods kind of like a snacky lunch so that's what I'm thinking so we'll see what ends up for lunch bark box is here Palmer's already at my feet I don't think he quite knows what's here yet they're both staring at me though but this is the bark buster movie night box okay this is super cute so inconspicuously let me try to show you first the baby so we have the bark buster oh I know you see it just a minute please we have the bark box buster VHS tape this is so cute just a minute please you need to learn patience and then we have the enjoy the show popcorn bucket with a ball okay that is super duper cute and then you always get two toys two bags of treats just a minute honey we have the gobble a set soft base which is a turkey treat and we have the clucky chicken treats and then you always get a type of stick can you hear a little turd there's two chicken sticks in here so we'll give those to the dogs and then you always get a little card that shows you all the other toys that you could have got in your box so super super cute um don't jump up just a minute honey do you want to give in the popcorn bucket or the VHS tape oh look at how cute the inside of this is I didn't even realize that drool intentions what okay Palmer mommy will decide okay we're gonna do the VHS tape the drool intentions easy easy oh look at that that's a big toy for a little Palmer bring it back let me see I don't care about the little toys so I'm gonna give her one of the chicken sticks we'll do half of one you can have the other half later honey there you go Palmer do you want to treat would you like a chap the chicken stick and then you can come back to your VHS tape there you go bud so I just finished my third coaching call for the day I have one left and I'm gonna make lunch like I mentioned I want to do kind of a snack plate I'm gonna use my uba home plate because it has pre portioned areas you guys know I talk about this pretty much every single week I eat off of these every day multiple meals a day if you don't like your food to touch this is the plate for you plus it kind of helps you manage portions this would be your protein this would be your carb and this would be your veggie so it helps with portion control so I'm gonna put together my snack plate I'm going to do half of a cup of cottage cheese a string cheese some fresh raspberries for a crunch I'm gonna do Chetties I really want my multi grain crackers from Trader Joe's but I'm out I'm going to make a Trader Joe's run this week so stay tuned for that in a grocery haul but we're gonna do some Chetties and then for a sweet treat I'm going to have one of these Navitas organic power snacks these are so good nice clean ingredients they're 35 calories a piece and they're really really good so let's put together my little adult Lunchable lunch all righty here is my lunch I can't tell you how amazing this looks I'm super excited so I have my string cheese I have 50 grams of raspberries half of a cup of cottage cheese one serving of Chetties this is what was left at the bottom of the bag so we got a lot of little pieces and then my sweet healthy treat for dessert so I'll go ahead and put points calories macros here on the screen and I'll link Uba home down below with 10% so I just completed diesel's shadow box I shared this with you guys and last week's what I eat in a day that I had purchased this beautiful shadow box off of Amazon and finally everything came in for me to complete it so I have a picture of him I have the lock of his fur from lap of love I have his collar and then his paw print I just love it it is really really nice quality you can hang it on the wall or set it up on display like I did here just makes the perfect memory and it sits on this little table in my living room so I see it every single day I did link it in I will link the shadow box right off of Amazon I think it was like $25 I'll link that down below for you if you are in the market but it turned out absolutely beautiful and it makes my heart so happy also I want to show you guys this that I ordered off of Amazon to it says in loving memory of diesel 2012 to 2023 and it's a faux rock that you can customize again super super affordable we did go ahead and put that out here in our yard kind of next to our palm tree diesel love to lay outside and just enjoy the sunshine so just another fun memory piece for him also in law gave us this for Easter isn't this beautiful it has beautiful tulips and such a cute little Easter bunny basket I do need to take down my Easter decorations today but I wanted to show that to you in this little turd yeah we have another coaching call so this is going to be my afternoon snack I'm going to have a serving of these butter bean butter boon these are really good they kind of remind me of corn nuts it's 110 calories for an ounce and you're going to get six grams of protein they're really really good so I'm going to snack on some of those for dinner tonight I am making chicken alfredo spaghetti squash so basically we're subbing pasta with spaghetti squash and I'm going to make a little bit extra squash because we love it parmesan and butter on it is so good so let me show you what's in the recipe you're going to need all purpose flour low fat milk salt and pepper I will link my gravity fed salt and pepper grinders down below fresh parmesan cheese minced garlic I made my life easy and bought the pre-cooked pre-shredded white chicken from sprouts actually had to buy a second one because somebody not naming any names try Clayton ate a lot of this chicken so I just went and bought a second one fresh parsley lemon juice shallots light mozzarella and then the recipe called for one medium spaghetti squash and like I said I'm going to cook up two so we have some extra so I went ahead and cut the squash in half cleaned it all out I'm going to drizzle just a little bit of avocado oil into the middle of the squash that's going to help it brown and keep it nice and tender and then I'm going to add salt and pepper it's going into a 400 degree oven until the squash is cooked you'll know because it'll be browned and it'll look like little hairs ready to be dug out of the squash so I'm going to get those in the oven while we put together the chicken alfredo so I have a pan warming up sprayed with some non-stick cooking spray I diced up about two large shallots I'm going to add those to the pan and then I'm going to add a big scoop of minced garlic and we're going to let that saute for a couple of minutes I'm going to add in two tablespoons of all-purpose flour and kind of whisk that in with the shallots and the garlic and then I'm going to slowly add my two cups of milk and whisk as I go we're basically using this as a thickener between the flour and the milk we're going to let this simmer for about three to four minutes and then going to add in my lemon juice quarter cup of parmesan cheese and some salt and pepper and then whisk in two cups of cooked shredded chicken the chicken will warm through just by sitting in the homemade alfredo so I shredded one full squash into a bowl I'm just warming up the chicken in the alfredo we're going to mix it in with the squash and then I'll plate it up we'll top it with some mozzarella and some parsley so here is dinner so this is the chicken alfredo spaghetti squash that whole bowl only makes for serving so it is a big serving especially for the points and calories so I will go ahead and put everything here on the screen for you so for dessert tonight I'm having a serving of my lesser evil popcorn this is our favorite it's the Himalayan pink salt and you actually get three cups for 120 calories couple grams of protein some good fiber so that is going to be dessert thank you for joining me for today's what I eat in a day on WW I hope you enjoyed coming along with me today seeing all the good food that I ate tonight's dinner recipe 10 out of 10 recommend super healthy low carb absolutely delicious don't forget all of the recipes are on my recipe website which I will link in the description box along with nutrition coaching links and discounts to my favorite things links for everything I shared with you today and my Facebook group definitely come on over and join us there thank you so much for watching and I'll see you next time


We are hydric and struggles. We are hydric and struggles. We are hydric and struggles. Asia Pacific and Middle East. One region. One team. One region. One team. Diverse and inclusive. Executive search and consulting. Executive search and consulting. We are the future.


Everyone, Basel Chapman here on this Friday, 25th of March, and we're looking at the Dow up 200 points at 35,908. That means it's made that prerequisite leg D from a buy signal to a buy mode. It's achieved everything that we were looking at from the time that we went long back on the, I think it was the 15th of March, right about here, about 33,000. Now we'll see what happens. This is going to be very interesting because it looks great, but in fact to get to a leg D in this particular environment is important. But normally in a chart I would want it above the 35,824 high that was made back on the 9th of February. So this is just the start of something I would love for it to hold all the way through to the end of the month that goes to Thursday. What we're looking at very short term, we did on the one minute chart, we just made a nice peak F top and we're pulling back. I wouldn't be surprised if 45, 25 is going to be the key metric to watch on the short term as support. Now what we're looking at is the S&P also went to the peak D, so all of these achieved what they needed to do. Now maybe they're going to have to pull back some. We'll watch closely up 17 and 45, 37 in the daily chart. Weekly is improving and even that monthly is going to improve. QQQ not so good. Down 1 at 358 because it's still lagging and got stuck at the 200 period moving areas. IWM stuck in a range and the rectangle range it is up 47 cents. Now 206 gold, let's see where gold is right now. Gold is down just a little bit, down 9 at 1952. It's already in the consolidation area. It's done fabulously up until now. Now it's digesting these huge gains and crude oil also stuck in a range pulling back a little bit now on 75 cents at 111. I'm just saying that if at any point there's a close under 106 in the next three, four sessions, that says gold is going to take a more important the rest period, but it's still up in higher territory. The big thing that we didn't spend too much time on today is that bonds plunged down to 260 at 127.83. This means that yields, I showed the yield chart, my triple yield chart earlier on and then the five is over the 10 year. This is usually a recessionary talk and I did show a chart that we might hold okay, but let's go one step at a time. We're looking at yields moving much sharper. I'm going to hand you over to Larry Perzevento and great program for the rest of the day. We've got programming all the way to Tom O'Brien wrapping it up three o'clock to four o'clock. Have a wonderful weekend. Check out my opening call my daily newsletter. We've had some really nice positions. See you on Monday. Thank you.