On Sunday, May 3rd, 2020, the Ministry of Health and Wellness recorded its 18th case of COVID-19 in St. Lucia. In response to this and as protocol, a team has been dispatched to the residents as well as the workplace of the individual. This is part of our normal protocol whereby we initiate contact tracing. Contact tracing is essentially trying to find all the individuals who have been in contact with a confirmed case. These individuals will be assessed based on their level of risk and their level of exposure and a determination will be made as to how best to proceed with these individuals. Persons who have been exposed and at high level risk, these individuals an assessment is made of their homes to determine whether it is possible to remain at home in quarantine or whether these individuals should be quarantined at an isolation facility or our quarantine facility. The teams are working on the ground as we speak. Individuals have been contacted, individuals have been brought in based on the level of assessment. These individuals have been brought in so that they can be very closely monitored by our health team. These individuals are not confirmed cases of COVID, but as per our protocol, they have to be monitored until proven otherwise. This is one of our measures and the measures taken worldwide to break the transmission chain and to ensure that COVID does not spread into the community. So the Ministry of Health and Wellness would like to reiterate that these cases, these persons who have been brought in are only for monitoring. We are following up these individuals, it is part of our normal and standard procedure where we assess individuals based on their level of contact, on their level of exposure and if they are not able to remain at home within the community for follow-up, then they are taken to our facility where they can be monitored by the medical staff there.


Hi friends, welcome back to my channel and a very special welcome. If you're new here, my name is Jen I'm a certified weight loss and nutrition coach. I'm on Weight Watchers and I count calories and macros Happy Friday. It is Friday. So it is another weigh-in day I'll share with you guys how my week went and we'll talk about this week's WW workshop topic So if you're excited give this video a big huge thumbs up Subscribe if you're not because I do a weigh-in every Friday and I upload five videos every single week down in the description box I will link nutrition coaching where I offer personalized macros and calories This is how I lost 90 pounds in 2022 and almost 140 pounds total You'll also find links and discounts to my favorite healthy things and we'd love to have you come on over and join our free Facebook group It's wonderful. It's supportive and we would absolutely love to have you So I'll make sure that is linked down below for you as well So let's go ahead and jump in to my week my weigh-in and the WW workshop topic Totally amazing amazing week I actually just got home from boot camp if you follow me You know that I do boot camp three days a week and Friday is a boot camp day for me I just picked up a few more workout pieces from Yvette sportswear. You guys know that I love Yvette I have quite a few pieces of their workout and at leisure apparel Yvette is a professional female sportswear brand Curated for females and everything is such high quality fashionable and last for ever through multiple washes There are some staple pieces from Yvette that I love I'm going to share with you my all-time favorite piece that I picked up from Yvette as well as some of the new pieces that I Recently purchased one side note about Yvette that I love is all of their pieces come in these little zip top reusable bags These are great for travel. They're great for what bathing suits They're great to throw socks and underwear in when you travel. So I always keep these They just have that zipper closure and I love the little added touch of these with each of the pieces that you purchase So this is the first new item to my collection and I am Literally loving this. This is the shift strappy fitted padded cammy tank. I have mine in a size large It is a slim fit with a light support built-in bra. It hits right about your waist with a scoop neckline I love the thin straps in the crisscross design in the back It is so beautiful and so flattering. I love the built-in bra and then again I love this whole strappy sexy design in the back. You could definitely wear this as a light support sports bra You can wear it all on its own I plan on just wearing this with jeans or leggings It is so buttery soft and like I said, I love that there's a built-in bra So that I don't have to wear an additional bra with the top and I get that fun sexy strappy design I think it's very very flattering and I love that it hits right about the waistline I also grabbed another sports bra. I feel like I can never have enough sports bras And with spring and summer coming. I fell in love with this color. This is this beautiful Peachy orange color. This is a front zip closure sports bra And again, I love the back of this with this mesh panel and the dual strappy design This is the limitless front zip mesh padded running bra This is in a shade coral red and I picked mine up in a L plus So that's a size large. It's designed for high impact sports This is actually the sports bra that I wore to boot camp this morning because that is definitely a high impact activity run Train working out. It is definitely the perfect high support bra. There's non removable molded cups It has that front zip closure, which is easy to remove and put on It has double shoulder straps that breathe breathable mesh panel in the back and it has the classic racer back I also love that it has the wider waistband so that it doesn't roll up I feel like it sits perfectly. It is extremely high support Friday's at boot camp is circuit day So we do a lot of running jumping burpees push-ups Sprinting walking. We do a lot on Fridays and this held everything in You know that this is a problem area for me and they do flop around a little bit So I really appreciate a high impact sports bra. It does have that snap closure on the back with three different levels I love it. I love the color and I love how soft it is again in that buttery soft material Fantastic high impact bra. You also know, I love their leggings and like I've mentioned I've really gotten into purple so I saw that they had these leggings in purple I have a couple other colors of these and I love these I call these shrink a size leggings because they are so flattering on your body and they just suck everything in These are the ship mesh yoga leggings. I have a size large in purple They deliver soft and breathable comfort with a professional design I love the mesh panels towards the bottom of the legs They are perfect for yoga pilates and daily stretching as well as any low-impact activity like walking These are full-length leggings with a supportive waistband a four-way stretch and the mesh panel the inseam length is 26 inches I wear these to boot camp even though they are a lighter support legging They still hold everything in you also know that I have a lot of loose skin on my lower back and my stomach And it holds everything in they do not have a pocket That's a question I get asked a lot these particular ones do not have a pocket But I really really love the whole mesh kind of sheer pattern at the bottom I think it just adds a little bit of flair to these They are so supportive and so buttery soft the shift leggings from you bet You can't go wrong with any of the styles We just talked about how much I love out leisure wear and how I pretty much live in pajamas or at leisure wear I saw that you bet had these pants These are a velvet kind of these are a velvety type of material with a stretchy waistband that has the event logo These are the shipped light velvet jogger pants. I have mine in black size large These are made from a soft velvet fabric. They feature an elastic waistband and ankle cuffs perfect for everyday wear I love the event icon on the waistband they are comfortable and they kind of give you that whole vintage vibe. I also love that they have the Tapered in jogger style bottom. They're so incredibly incredibly comfortable. Like I said, I got a size large They're a little bit oversized on me, which is okay for comfortable at leisure wear But you could definitely size down if you're in between sizes. I love these I'm so excited to have these in my collection. I wouldn't these are something I will definitely get a lot of use out of another thing. I love is a good jacket I wear a jacket every day to boot camp and I've really gotten into more of a cropped jacket Can we just talk about how beautiful the color of this is? This is the shipped cropped running jacket I have mine in the blue color in a size large.  It doesn't ride up I actually wore this jacket to boot camp this morning worked out in it and it is Perfection. I love it. I love that it zips all the way up for those colder mornings It is so soft. It is a nice thick material I am 100% getting on to the event website and buying some other colors I mean, I love the blue, but I definitely need a few more colors of this jacket.  This is the shift light zip oversized workout hoodie I have mine in the color black in a size medium It offers a relaxed training style for workouts the zipper front lets you move freely It is a little bit of an oversized fit has two side pockets Comfortable fabric and one thing I just love about it is the detail. It has the event logo down the sleeve It is so comfortable. It has a hood. It zips up. It is so warm. It is a thicker jacket This is one that I wear a lot when I walk to the gym. If you follow me on Instagram I'm constantly in this jacket walking to the gym. I wear this to the grocery store I wear this at home. Like I said, I own two of them. I absolutely love love this hoodie It's definitely a go-to for me. That's why I had to grab a second one I will go ahead and link you bet sportswear down below for you They're offering up to 40% off on their website. You can also use my discount code to save even more money So you can use the code gen 15 for 15% off all purchases or the code gen 20 for 20% off on orders of $79 or more I will link all of the pieces that I picked up as well as a link to their sports bra leggings tops and they have an entire Plus size collection as well. I will have all of that linked down in the description box at the very top for you Definitely recommend Yvette. I can't wait to get on their website and do some more shopping So now let's chat a little bit about my week Overall, I had a pretty good week as you know My dog Lola was diagnosed with lymphoma a few weeks ago and is currently undergoing Chemo treatment. So she goes every Tuesday for four weeks then takes a week break and then goes for four weeks It's a process because we have to drive all the way to Tucson So it's a several hour process every single Tuesday, but she's doing really well A lot of you have reached out and asked how she's doing. She's doing fantastic The swelling in her lymph nodes has definitely decreased pretty much nothing. It seems that the chemo is working as it should Unfortunately the day she gets chemo. She generally doesn't feel that well at night So Tuesday nights are a little rough for us. We're up a lot of the night on Tuesday But this week when I took her to chemo I gave her an anti nausea medicine before in hopes that that would help with some of that sickness that she feels The day she receives chemo, but once Wednesday rolls around. She is back to her normal self. She's happy She's playing we're just really really happy that it's working and that she's feeling good and that her quality of life Is what it was prior to being diagnosed So fingers crossed that we go into remission and that she lives a long healthy life and again Thank you guys for reaching out regularly. I promise to continue to update you on a regular basis Other than that my week was great I got in all of my workouts this week finally the weather Cooperated and I was able to get to boot camp all three days got to the gym to lift weights Tuesday is now an official rest day for me. I take a rest date every Tuesday, which means I don't take a walk I don't do any activity other than day-to-day activities because that is Lola's chemo day It's my filming day It's an editing day for me and honestly I have to say that I really enjoy having a full rest day It's something that I never incorporated before But I feel like my body needs it because I am sore a lot and I work out pretty heavily the other six days of The week so I've been loving having a rest day if it's not something you have in your life Highly recommend. My water was good. My food was good. I feel really really good about my week before I jump into my way And let's chat about this week's WW workshop topic and this is an interesting one I feel like this is something I would discuss on my nutrition channel If you aren't subscribed to my nutrition channel definitely subscribe I upload two videos every week all about nutrition debunking diet myth It's a really good way to get additional information for weight loss and a healthy lifestyle and this week's workshop topic pairs really well with my nutrition channel and that is what happens to my body when I lose weight Obviously the scale goes down your body changes. It gets a little bit smaller But what else happens with weight loss the number on the scale only tells a very very small bit of the story We talked about this in my life update video where I shared my weight with you for the first time and how The scale is just a number and it does not matter. It's more important how your body looks and feels how your health is I will link that update video down below if you missed it But it ties hand-in-hand with today's topic as we lose weight as we become healthier overall We feel better mentally and physically this can happen within a few weeks and research shows that losing five to ten percent Of your excess body weight can lead to huge health benefits and just feeling better overall There are some other huge benefits for losing five to ten percent of your excess body weight Those are improved mood and sleep. It can feel easier to get around You can have lower blood sugar your a1c and cholesterol and a lot less joint pain Decrease risk of heart disease type 2 diabetes and hyper tension and all of these are NSVs Otherwise known as non-scale victories that have absolutely nothing to do with the scale I talk a lot about NSVs on my Instagram on my channel in my Facebook group because I think those are even more Important than the number on the scale and like I mentioned that is such a small piece of our overall success There are some things you could think about when it comes to NSVs list three things that have changed in the last month or so For you maybe your clothes fit better You started a new activity routine or you feel less stressed and irritable other than your weight If you need help if the scale didn't exist, how would you measure your success? Then register the full impact of each of these things is getting dressed easier Do you have more energy to play with your kids? Of course recognize your efforts think back to when your journey felt tough and how you got through it And of course reflect on the bigger picture. How do you feel celebrating these NSVs? How is it different than only focusing on the number on the scale? I don't focus on the number on the scale at all I really honestly could care less what the scale says I care more about how I feel in my body How is my body looking? What results am I seeing from my workouts? Am I getting around easier? One thing that I really noticed is that my knee pain Substantially decreased with just a small amount of weight loss if you didn't know for every pound you gain or lose It's four pounds on your knees So think about if you lost 10 pounds you've removed 40 pounds of weight on your knees if you gain 10 pounds You add 40 pounds of weight just to your knees That's why our knees hurt when we're overweight and that was one thing I noticed pretty quickly that substantially increased and now my knees are living their best life I have a zero knee pain and I can be extremely active and not suffer from it for days We can become laser focused on the scale and that is not where our focus should be We should really focus on these non-scale victories are these other things that we notice along the journey You have to remember that there are a lot of health benefits for losing weight Like we talked about losing five to ten percent of your body weight can really decrease your risk of health conditions You can feel better your clothes fit better You can move around easier again that was something I struggled with over 300 pounds was moving around I had to hold the handrail to climb stairs and now I'm up the stairs and down the stairs with no problem And my knees don't hurt anymore going up and down stairs So those to me are way more important than that number on the scale Don't focus on the scale like I said, it's a small piece of your overall success I want to share with you three fast facts losing weight can lead to both mental and physical benefits including improved mood and sleep Lower blood sugar a1c and cholesterol and better mobility and reduced risk of heart disease type 2 diabetes and stroke I don't know about you, but these three things are way more important than any number on the scale So remember take your laser focus off the scale and focus on all the NSVs are non-scale victories along your journey Love this topic love everything about it And it is just resonates with everything I share here on my channel And I hope that it helps you just take your focus off the scale and focus on other more important things when it Comes to losing weight and getting healthier. So now let's go ahead and jump in to my way in like I mentioned I feel like overall I had a really good week. It's a little stressful with everything with Lola But it's definitely better than it's been the last few weeks I feel like I was on point with my food and my water I will say I was a little bit more hungry this week. So I did go over my calories by a couple of days It doesn't matter. I'm still in the calorie deficit. So when I stepped on the scale, my weight did not change at all I'm exactly the same weight as I was last week, which is fantastic. We know that I'm trying to figure out if this is maintenance for me I'm trying to get as lean as I can before I have plastic surgery in me So for me maintaining my weight is a huge win just because I don't have really any weight left to lose or fat left on My body to lose so if I can maintain my weight That's a huge win for me and that's coming off losing weight for the last two weeks and about two pounds in the last two Weeks which is fantastic. So a maintenance for me. That's a huge win So let me know how was your week when you stepped on the scale Which we know is just a small piece of the overall picture. Did you gain? Did you lose? Let me know what non-scale victories you're proud of actually share all the non-scale victories in the comments Crossing your legs Buckling your seatbelt on the plane taking the stairs without huffing and puffing or your knees hurting Let us know all of your favorite non-scale victories I cannot wait to hear them all because like I said that is what we need to focus on versus the scale And of course if you enjoyed today's video, I'll give it a big huge thumbs up Subscribe if you're not I do a weigh-in every Friday and I upload five videos per week I will link the vet sports were down in the description box at the very top with all the discount information Definitely check them out highly highly highly recommend as well as nutrition coaching links and discounts to my favorite things and Come join my Facebook group. We'd love to have you happy Friday friends I hope you have an amazing day, and I'll see you in tomorrow's grocery haul. Bye You


Good evening, everyone. My name is Vijay Samuel. I am a principal architect at eBay, and I help build the observability platform. You might notice that my co-speaker is missing. Aishwarya just had her baby boy, and she's back in the US right now. But yes, congratulations, Aishwarya. We did manage to get a recording of her portion of the talk, which I'll play when those slides come. We've been using OpenTelemetry Collector, and in the past we have talked about metrics in previous KubeCons. This time we're going to talk about tracing, how we have been playing around with various configurations of the OpenTelemetry Collector, and where we are at right now. So various configurations and the conclusions that we were able to make while fine-tuning OpenTelemetry Collector for tracing. That said, we'll do an introduction of what tracing is, how it looks like inside of eBay, what's the problem we faced, how we solved it, some of the lessons that we learned along the way, and if time permits, let's do some questions. So what is tracing?  So why is it important inside of eBay? We have call chains that can have like tens of databases, several microservices to the point where knowing what exactly the customer saw when a request was being served, tracing is very critical for that. What is our scale? We have a 14-day retention with 190 billion spans that we process every day, which roughly translates to 2.2 million per second, which is served by 3.6k provision computes and 150 terabytes in storage. And from an architecture perspective, every Kubernetes cluster has three kinds of applications that run. Some of them are what we like to call generic applications. So these are typically, I wrote a program in an arbitrary language. It doesn't, I just deployed on Kubernetes. It's basically free flow. On the other hand, we have managed applications that are there. Managed meaning they have full support in terms of developer lifecycle and whatnot. And they have SDKs into which we bundle the open telemetry SDK. So generic applications would need to bundle it themselves. They need to hand roll instrumentation, whereas managed, the instrumentation comes for free. They write their spans into open telemetry collectors. And from there, we have an ingest layer which receives in OTLP, writes it into our TraceStore, and whatever metrics that we are generating out of the spans, it goes into our metrics platform. And we have the Sholokio console. Sholokio is what we call our observability platform inside of eBay. There is a trace square here, which uses the Eager protocol, and a metrics square here, which uses the promql semantics. What is the problem that we faced? Let's be honest, tracing is not easy. It's quite hard. Tracing can become very expensive depending on the amount of HTTP requests that you're serving on a day-to-day basis. Organic growth means that you're going to spend a lot more. Every span has an arbitrary number of span metadata attributes that are going to be there. There are span events. People can abuse it like a logger. People want 100% sampling. To give an analogy, we have an internal logging platform, which is very similar to tracing, which generates 15 petabytes a day with 100% samples. This is without any of the span metadata type overhead that we have with the more modern version of tracing that we are using. There's a high barrier of entry for end users. Every app needs to be instrumented in the call chain, and context propagation needs to be done right. If not, you're not going to see the complete structure, which means that if there was one collector that dropped one span, it can lead you through a wild goose chase if you land on that exact span or that exact trace. Open telemetry is not that easy either, especially with the collector. There is a very famous lego analogy that's there. There are many building blocks to the collector in terms of receivers, processors, exporters. You can put them together in any order. It doesn't really limit you. You can either create something that's really beautiful or you can create a monstrosity. You have the ability to have one tier of collectors, two tier, end tier. Where should the Kubernetes enrichment in which stage should it be? There are so many things that are left to the implementer. There are challenges adopting as well in the sense that tracing is relatively new. There are many people who have talked about their reference implementations yet, unlike things like metrics where there are so many reference implementations out there. The community is also extremely vibrant in the sense that what is knowledge today can be completely obsolete tomorrow because this is one of the faster moving communities out there. What are the things that we tried but never stuck? So first, we tried a single tier approach where we have all our applications writing to the open telemetry collector. We first put it through the KX enrichment, then we generate span metrics. We do the tail sampling processor and from there, we write the traces into the trace platform and the span metrics into the metrics platform. The next approach that we tried is a two tier approach where we moved the tail sampler out alone because what we saw with the tail sampler is that all the spans for a given trace ID always need to land on the same open telemetry collector for it to be able to hold it in memory, make the decision to sample or not sample and then flush it out. So that being said, we have applications that write to the first tier of open telemetry collector where we were doing span metrics and Kubernetes enrichment. Span metrics go into the metric store and the raw trace goes directly into the trace store. But we also write a copy of that into the second tier where you do the tail sampling and then you ship it into the trace store. Finally, we also tried another where we moved the span metrics processor also into its own tier. So on the first tier, we just have the Kubernetes enrichment that's there. On the second tier, there's a path dedicated for metrics that are coming from the span metric processor. There's a dedicated path for tail sampling for the traces and the raw traces get written into the trace store as well. So how did we go about solving this? There are multiple things that we did. The first one being we solved for mass adoption. We did pipeline performance tuning. We did some scaling related tweaks. We optimized on storage and then we went for sampling. So the next few slides, we're going to talk about each one of them. Mass adoption. The first thing we did was we wanted to go for the highest common denominator. During the architecture, I described our managed framework. 90% of all the applications that are deployed inside of eBay use the managed framework, which means that we can ship instrumentation through the managed framework and immediately you get 90% adoption of tracing inside the company. And we have this concept of a monthly mandatory site-wide upgrade where every application that's on the managed framework automatically gets upgraded to the most recent versions of their dependencies, which means that any instrumentation updates that we want to do, there is a one-month turnaround time to get it into all the applications. And there's around 8,000 applications that we are talking about. We are working on getting tracing support on a service mesh. We deploy Istio internally, which means that if the mesh also emits spans, any application that does not have tracing instrumentation on it, the bare minimum, you would at least be able to get the client and server side spans through the on-voice sidecar. And finally, we wanted to make tracing instrumentation optional. So end of the day, if no developer knew how to instrument spans inside the company, at least I would consider that to be a win. Developers, what they would need to do is just use, in case of Java, just use the SLF4J, do logger.log. Under the hood, the open telemetry SDK would make sure that trace IDs and span IDs are being tagged to each of these loglines. And they should just be able to use those to get the logs. They don't necessarily need to instrument spans because the framework is already taking care of that. So within the managed framework, what we did is that we shipped in the open telemetry SDK, and we made sure that there is instrumentation for all the client calls, all the server APIs that are there. For database calls, like depending on the database, some of them have instrumentation already. We are working with the database providers internally to make sure that the remainder is covered. And we have instrumentation for iOS as well and write coming in the future. We had sampled at 2% by default. And basically what this allows us is that we can do what is the three-click RCA, Asterix on the three-click because it's a little more. But the idea is for a person to be able to do an alert to metric to trace and then decide if they want to go to log our profile later. And exemplars play a big role in all of this. And for all of this, we make sure that the red metrics are computed for every kind of span that's there. The next part is pipeline performance tuning. With performance numbers are everything. We broke down the pipeline into independent chunks. We evaluated how the numbers looked like, tried different combinations, rins and repeat to the point where we can figure out how to place these Legos in a way that works for us. So that being said, the first thing that we did OTLP in, OTLP out, memory limiter and batch, which we require anyways, identify how much throughput we were able to get out of that, the memory limiter and batch processors are very low overdasses, require some memory, but not too much. And they more or less don't impact your throughput. The next thing we tried adding the span matrix processor to see how the throughput looks like. It's fine for a single instance. But for multiple instances, we'll get to it in a couple of slides. The next one is basically the Kubernetes metadata enrichment. This is an interesting one. On a single instance, we saw that pulling the entire cluster's worth of part metadata along with some namespace metadata had a 5GB overhead. But when we add multiple instances, even though we were shedding the number of spans evenly across various instances, the overhead of each Kubernetes enrichment processor remains the same because every instance has to anticipate every pod's data that spans that are coming in. So it has to hold all the pod metadata in memory. So across each one of them, you basically see a standard memory overhead that's there. So the final configuration that we ended up with, the first stage only does the Kubernetes enrichment. It does an OTLP exporter into the second tier, where we only do span metric processing. We completely dropped the idea of doing the tail sampling because the way we see traffic inside the company, it doesn't really make sense to do tail sampling on the open telemetry collector. And the simple reason being that there is always going to be the case where a request flows from one region to another, which would mean that we would have to load balance all span traffic across regions to make sure that the entire request spans are being processed by a single open telemetry collector. And this was simply not feasible for us. And depending on the kind of application, you would have to hold the spans for a longer window, which means that you need more memory. So we went for an entirely different tail sampling methodology, which Aishwarya will talk about. And even on the load balancing exporter, Aishwarya helped add a feature to load balance based on service name instead of trace ID. Because if you were doing the service name, sorry, the trace ID based load balancing, what's going to happen is that every pod in the second tier will need to see every kind of span metadata combination. But on the other hand, if you do just by service name, all metadata combinations for a single service is always going to be collocated on a single collector, which means that the memory requirements are substantially reduced. It's like everyone needs to know everything versus a collector needs to know about only a given service. So that change on the open telemetry collector also made things quite efficient. So if you are doing a two tier approach where the second tier is going to do span matrix, it's better to do the service name based load balancing rather than the trace ID based load balancing. On on scaling, we did two things. The first thing is we started leveraging CADer because like I said, logging and tracing have very seasonal throughput. So at least for us during the holiday period or during weekends, when there is more shopping that's happening, you're going to see more volume, which means that doing auto scaling can greatly improve our ability to handle higher traffic without having sustained deployments of large size installations all the time. And the second thing that we did is to make sure that we are not using the Kubernetes service object to do the load balancing because the weighted round robin approach that it uses for GRPC requests, it is not very efficient and ends up sending the open telemetry collector pods out of memory. So we basically use the mesh which can handle GRPC load balancing a lot better for that. With regards to storage, from a consumption perspective, we implement the Yeager APIs because that's more or less a standard right now. And we use click house for storage. Click house is cheap. It's fast. It really it works really well for us. But the one improvisation that we do is that we don't have a single large click house cluster, but we have smaller click house clusters that are backed by a ring based discovery. There's a routing table that we maintain saying that this service name needs to be routed into this particular Kubernetes cluster and our ingest APIs, they can basically use that to make decisions on where to write the data into. And we also make sure that you don't need to do a global scatter gather on all the queries by implementing a global index. So every trace ID, we know exactly which are the shards on each of the clusters to basically query to get the data back rather than trying to query all of them. This is how our storage architecture looks like. We have a traces table, which has the standard parameters that a given span requires. We supplement that with a bunch of materialized views, one for service operation, which has the service name and operation name. We have the trace ID and service name in another materialized view. And we have an index saying that this is the index ID for this service name plus resource attribute combination. And this is basically what a combination of all three is what we basically use to make routing decisions. And we also have a sample table where all the sample traces which are retained longer are stored. Both these tables and materialized views have different retention. We retain the route traces for two hours because we believe that a usual site triaging event should not require more than two hours worth of spans. And the sample table is what we retain for 14 days after the two hours has expired. Let me move to Aishwarya's. Hello everyone. This is Aishwarya. I'm working as a senior software engineer for Observability Platform at eBay. And I've been working on the tracing project. Now coming to the sampling part of tracing, we have a lot of applications at eBay that seem traces. And there is a huge amount of data to process and store on the tracing platform. So with that we are raised with two fundamental questions. These every trace really useful. If not, do we really have a room to be more efficient while processing and storing data on the tracing platform? Now moving on to the next slide. If you see this diagram, this shows the high level representation of how tracing data looks like. So we have a lot of traces that ended up without any issues. And we have a small set of traces that ended up with high latency and also a small set of traces that ended up with errors. So we don't really need to store all this data on the platform. So to be more efficient and effective. So the ideal representative sample would be to sample a small percentage of traces that ended up without any issues. And we can always sample 100% of traces that ended up with high latency and also with errors. If we are really interested, then we can also take samples of traces with specific attributes that we are really interested in. So with that we can be more efficient on the platform side. Now moving on to the next slide. We have adopted two types of sampling mechanisms. So the first one is SDK-based parent head sampling. So when applications generate traces, so even before sending those traces to the platform, they're sampled at 2% by default by the framework. So the type of sampling that we are using here is parent trace head based. So that means when a root level generates a trace ID, a root service decides whether to sample a trace ID or not. So when the decision is made, the decision is passed on to all the services in the call chain. And all those services obey the decision set by the parent, whether to sample a trace ID or not. So with that, we also generate exemplars. So exemplars generate trace IDs that are only sampled. So basically, we attach these exemplars to a latency metric that is present on all the applications. So this latency metric has a latency dimension bucket and we add exemplars to each dimension bucket. So when there is any issue, it's easy to debug because we have at least one exemplar present and we can easily jump from metrics to traces to see the end to end distribution, to see end to end call chain and see where the issue really exists. Going on to the next slide. So the next type of sampling technique that we adopted is exemplar based tail sampling. So when we receive traces sent by applications, we store all the raw traces for two hours and then we apply exemplar based tail sampling. So basically, as I already told, we record exemplars for metrics on the framework. When a trace ID is recorded as an exemplar, that means it's really important and has to be preserved for longer duration of time. So all those trace IDs recorded as exemplars are received and then they are sampled and stored for longer duration of time. So not only exemplars are exposed by framework, we also have span metrics processor running and all the traces that are received on the platform use span metrics connector to compute red metrics, meaning request error and duration. And we also have exemplars attached to each and every red metric. So recently, we also added a new metric called events total, which record metrics for error labels that are emitted as part of spans. And since we have exemplars recorded for all these metrics, so we collect all the trace IDs from these exemplars and they are also stored for longer duration of time. Now apart from these two sampling techniques, we also have one percent of random sampling across all the traces that are received on the platform. Now moving on to the next slide. So coming to the tail sampling architecture that we follow. So we have traced tracing hotel collectors and framework sends all these spans to the tracing hotel collectors. And we also have span metric processor running that generates red metrics. And these metrics are sent to our metrics ingest pipeline. And this metrics ingest pipeline also send these metrics to metric aggregator. So metric aggregator during rollup, it also preserves exemplars and they are not dropped in the pipeline. So once aggregator receives all the metrics and aggregations are done, so these metrics are again sent back to the tail sampler ingest that we have. So this tail sampler ingest collects all the exemplars and it records all the trace IDs and stores in our click house table. So we also have a tail sampler job that is running. So tail sampler job picks the trace IDs that are needed to be sampled and it gets all the trace IDs and it gets the tracing data from raw tables and samples all the traces that are being exposed as an exemplar. So a couple of more things on that on the section that Aishwarya talked about. So exemplars are really important to us in the sense that think of latency metrics that are there. You basically decide if you're not using the new native histograms, you basically decide what are the buckets that you're defining for your latency and each one of them is going to sample exemplars and then emit it into your backend, which basically means that if you take those exemplars and you keep sampling all of those trace IDs to retain longer, what you're essentially going to get is that for every bucket that you defined, you get sample traces for every host that's emitting these metrics, which means that when you sample, it's almost a representative sample for your entire ecosystem. And what Aishwarya did also is that she worked with the open telemetry community to make sure that we are emitting events total as a metric where if you record every event that has a unique exception type, that also means that every error scenario also you are making sure for every host, every error type, you have at least one sample that you're retaining much longer. This greatly improves our efficiency while at the same time not impairing the customer experience too much. So what did we get out of all of this? One, we got a cheaper and easier platform to operate, not having to retain all these spans for a greater window, not having to do complex sampling algorithms on the open telemetry collector or even after the fact when it has landed on storage. Now that we have distributed tracing without any of the developers having to lift a finger, we can create dependency graphs at both a per request level or at the service to service level. And we have an industry standard in terms of open telemetry, but at the same time, we are gearing it heavily towards what eBay actually needs. What are the lessons that we learned? The first thing is observability is a team sport. And we should use all the pillars effectively and not try to say that, okay, use this one pillar and try to derive everything out of that. In our case, you saw how closely knit the metrics platform and the trace platform are being knit together because the metrics now provide the signal on what needs to be sampled and what does not. And the same time, all our P1 observability or P1 detection functions are strictly done just by metrics. We don't try to use the other ones. Exemplars could offer a representative sample. We just talked about that. Numbers really matter. Fine tuning took a lot of benchmarking, profiling, and we finally were able to land a pattern that was good. Sometimes the community also offers some prescribed patterns. It's better to just stick to them rather than trying to improvise. And finally, no one should need 100% sampling to do effective observability. At our scale, we do 2% sampling, which is plenty enough for us to be able to look at all kinds of issues because end of the day, one in 50 requests need to have the issue that the customer is facing. And with the kind of call volumes that we have in a day, one in 50 is fairly easy to achieve. With that, I'll take questions. Thank you. Thank you. Very great talk. Very familiar themes and very familiar journey in that, maybe for other people. I'll be interested. I've got some questions. First, your deployment model for the collector, it sounds like you didn't specify, but it sounds like you have a central pool of replicas for your gateway and you haven't got sidecars or agents per host or Kubernetes node. Correct. And the reason that we do that is that there is a severe problem of resource fragmentation when we go down the route of either a demon set or a sidecar. Sidecar, you also have the problem that if we need to do upgrades, then we'll have to restart the parent container or we'll have to have the managed platform do a rollout across the site, which is more than 8,000 applications. Like I mentioned, so the cluster local gives us the flexibility to do independently rollout. We can have bigger parts for handling more throughput and we can scale it at will. So that's why we do that. Okay. Also that you didn't mention application metrics. So you mentioned span metrics, but you're not collecting metrics with your connectors or logs either. So logs, we do the file based approach. It still uses the demon set pattern. We are slowly but surely now that the logs API is stable, trying to get people to adopt the SLF 4J based bridge for open telemetry sometime this year that should happen for metrics. There is some free instrumentation similar to tracing where the four golden signals are instrumented on a dedicated Prometheus endpoint and we scrape it for free for all the applications that use managed stack. And there also we emit exemplars for the latency metric as well. There are actually, since we have the managed framework has a lot of framework providers as well, like the database team or the messaging team and whatnot. And each one of them have their independent Prometheus endpoint, which we scrape and then we ship it into into our metrics platform. And a lot of free observability is provided to them in terms of security dashboards, curated alerts, being able to do health aware rollouts and things like that. Okay. Just one more very quick one on your ingest. Do you have any buffering or asynchronous read write buffering and storage before store? We only do the batch batch processor, nothing nothing more than that. So everything is pushed straight through the pipeline to your storage? Yeah, if the question is, are we using something like Kafka? No, we do not. We just receive it on the gateway. We pick a pick a click out shard to write and that shard will also have a buffer table. We push into that and then it periodically flushes into storage. Thank you very much. Thanks, it was a great talk. Thank you. I think my question is on the application level sampling that you did. Can you talk a bit about that? Miss?  Because my I'm coming from a point where you might be filtering out something which is a high latency trace. So on the application, as in the SDK based sampling, yes, we, we just do head sampling, like probabilistic sampling at 2%. So basically it's as good as flipping a coin and deciding if something needs to be sampled or not. And we are okay to do that because when we look at all our latency patterns or the number of errors that are being seen on a given application, it's well over 50 per host, which, which generally means that like a 2% sample, the lowest volumes should still see at least one request that's coming, coming into the platform. So that being said, like, we just go, go the, go down that route. One follow up last from my side.  The logic was the one that I mentioned earlier in the sense that we have cross region traffic, which, which necessarily means that like if you want to do a proper tail sampling on the collector, all the traffic that was seen across regions needs to come into the same trace collector. And we try to make sure that we are not passing log or trace information across regions unless we really have to. And that being, and the other reason being that doing a true tail sampling, you can always have a five minute or 10 minute offset or even a one hour offset for something that's really long for whatever reason to come in and then do the tail sampling decision. But we, in the case of the tail sampling processor, you had best can do maybe 20 seconds, 40 seconds. And if your throughput is really high, then you need a lot more memory to make sure that you're really doing a good job in the tail sampling. Thank you. Great talk. Two questions about tail sampling. The first one, is it based on open telemetry or is it something that you implemented yourself? Second, you mentioned that it's a tail sampling job. Do you mean that it's a crown job that how frequently does it run? And could you touch a little bit on how it is implemented? So the exemplar based tail sampling is something that we built in house, but it is something that is reasonably easy to implement on your own. All you would need to do is you have a processor that can basically look at all the metric signals and basically write them into something like a database table so that whenever there is an exemplar, it's making note of that. And the tail sampler, like you mentioned, is just a crown job that basically selects that table that has the trace IDs and then uses that as the subquery to basically pull all the raw spans into a different table. And that window can be configured. So we do two parameters. One is the offset after how long it needs to kick off and what is the interval. So typically we do five plus five. So at the fifth minute, we make sure that the 10th minute is being sampled. So the same query that I talked about, it just keeps getting applied for that time window.


This afternoon, we are very fortunate to have with us one of the first scholars who arrived at Erie, not the first scholar, but one of the first who arrived at Erie. First, let me read an introduction about him. Our guest speaker this afternoon graduated Kung Lao Le, B.S. in Agriculture, majoring in agricultural engineering at UPLB in 1959. He took his MS in agricultural engineering major in power and machinery in UPLB in 1964. He finished his PhD in agricultural engineering in 1968 at the Iowa State University in Ames, Iowa, USA, creature in power and machinery, minor in mechanical engineering and statistics. He was an Erie MS thesis research scholar from June 1, 1962 to June 1, 1964. He was a visiting scientist in an interim head of the Erie's Agricultural Engineering Division from 1994 to 1996. He has written, presented and published some 50 articles and reports on agricultural machinery and mechanization. He was a consultant of Agronomics, Incorporated, UNDP, FAO, UNISCAP and UNIDO. He was an expert in farm mechanization and transport for the Sisi, Boris and Delayo from 1974 to 1976. He also served as a professor of agricultural engineering and dean of UPLB from October 1, 1976 to December 31, 1983. He was also an agricultural machinery expert for UNISCAP from 1984 January to 1991 December. He has a host of other achievements and I don't think we will spend the whole afternoon reading all of this. Well, anyway, we are very fortunate to have with us this afternoon one of the first scholars who arrived here at Erie. So, ladies and gentlemen, without much further ado, please join me in welcoming our guest speaker this afternoon, one of our first scholars here with Erie, Dr. Rinaldo M. Gentian. Well, thank you Achu, I'm thinking with this wonderful gadget here, which you are familiar with, I am not. It stays here, it's a tablet, which has a use. It senses that somebody is sleeping while I'm talking. Well, thank you Achu for that nice introduction. As an agricultural student at the UPLB, when I back in the 1950s until I graduated in 1959, I witnessed the development of Erie. So, I'm privileged to have an institutional memory, so to speak, from its founding, from its first state work, because I am from engineering, I was curious about what's going on. And you know this, where we are, and the whole of this little complex here around the pool, was a hill about the height of the dormitory right there in the cafeteria. So, this was level down, and this building rose. But the first building that was set up was the service building, which housed the Department of Agricultural Engineering. And the whole experiment station was banana and coconut, essentially, with some rice, palace around. And so it was the hug engineer Lloyd Johnson, who was my mentor when I was here, that developed this experiment station. And I was sort of part of that, as a scholar, and sort of on the job training during its expansion. Right there, more coconut plantations and so on. It had a wonderful setup of irrigation and drainage, never seen before in the Philippines. It was a road network, the same road network that we see now, except that there were no buildings right there in the rice paddies. This is what the material goes off. So, the flow rate of construction activities during the late 1950s, that was 58, 59, the inauguration of 1560, coincided with those of the McKinney Forest. Part of the McKinney Forest, where the Jamboree site, you're familiar with that, is now located. So, the Jamboree World Jamboree was first held in Asia in Los Minos. So, it was a Jamboree city. And you could imagine that that was a forest, sort of a virgin one. And that was clear. So, they must state it, if you would say, that it could be seen. The activities there could be seen from the farm machinery building at Pewry Drive, where we had classes. So, with these two constructions going on simultaneously, and if that happened today, there would, I'm sure, be some protest of destroying the environment, because trees were really created. But there were lessons. First of all, EV, the first international institute in cut research, developed into a world-class research institution, as we all know. While the Jamboree site was restored to the original forest, as you see now, in about 10 years only. And the lesson is, if you don't intrude on it, that the Sweden farming and local keying in the Philippines, it will restore itself. Of course, there were interventions by the boy scouts who planted those trees. But this lesson is to lose our display, that we can never reform our mountains again. I've always been in contact with EV also, aside from seeing the physical development, because we were just neighbors. Our building is the nearest building at UPLB with EV, practically neighbors. And we were very close in partnership with the Department of Agriculture and Engineering. And they are, in other words, the EV staff was in fact appointed as affiliate faculty with the UP College of Agriculture, along with the besting professors from Cornell University. So the Los Baños was really strengthened by both Erie, Cornell, and the scholars who had graduated from Erie scholars were created everywhere through various fellowships. Now, why was Los Baños selected as the site of Erie and the site of the World Jamboree? Because at that time, late 1950s, the Philippines was a burgeoning economy, only second to Japan, as they say. Second to Japan, but now no more. It's the last generation. Now, what caused this leadership in the economy? Second to Japan. I consider industrialization as the correct policy that made it. It was President El Pijo Kirino, who in 1948, he was president, had a policy of industrialization. And that's when these industries like the petrochemicals, the textiles, and all the others emerged. And this brought about more jobs and really a product economy. But that is my surprise. Why are Philippine economic planners ignored industrialization? In favor of agriculture. I'm not saying I'm against agriculture. When all the countries like, what you can name, our neighbors from Taiwan, South Korea, Thailand, Malaysia, and now Vietnam, and Laos, even, and Indonesia, they pursued industrialization to be prosperous, so that they have more jobs and so on. They pursued it deliberately. But how were all the divisions of our planners to be, oh, the Philippines is an agricultural country, and so we have to develop agriculture first. But they have been talking about this for the last 50 years or more. And nothing happened until we were overtaken by these countries pursued it. Now I have a story, and I use Bangladesh as a model. Bangladesh. A country, our critics sarcastically compare with in order to make the Philippines look good. Because, yeah, we are second at least, not the poorest, but the second to the poorest. Bangladesh, according to one writer, is the most mechanized, the most mechanized country in the South Indian continent. Why so? Well, there's a story. In 1988 and 1990, it was ravaged by two successive cyclones. I think the demagnitude like Sunday in Europe, and many massive loss of lives and devastation of property, as well as the war animals, they were decimated. And so President Ershad at the time, I had a problem. There was surely to be famine, because farmers would have worked their fields. There were no war animals. So the report is to import, this is my field, farm machinery, from either of the two, Japan or China. Well, the engineers and the scholars said, we import from Japan because of high quality. It's for the use of the farmers. But there is another reason, but we need more machines with a limited budget. But if we import from China, we have three or four times the number of machines, and we need those. So President Ershad abolished the committee and continued to import. At the risk of his political career, he imported engines and tractors from China, because, well, they are not durable, but surely they can last one season, which we need. And fortunately for him, there was rice production and there was no famine. And fortunately also, that decision ensued a kind of industrialization, a localized industrialization, because my men's scene, it was the seat of a family industry, an indigenous family industry. And true enough, when the machines and engines failed, because the parts were easily worn out, they had to replace it, these family industries and machining works in my men's scene decided to make parts themselves. And I found out that they could make it better than China. So that was a driving industry. It created more jobs and my men's scene is a prosperous town in Bangladesh. Dr. Mangur agrees with me because of that. Nobody almost is jealous because of that industry. So that is a lesson. With a happy outcome, and it is because of industrialization, little as it is. Now, perhaps Erie too may consider undertaking activities, maybe probably as a advocacy at this time, in pursuit of the goal of, I quote the goal of Erie, finding sustainable ways to improve the well-being of poor rice farmers and consumers, as well as the environment. There is the environment. The emerging culture is that if rural folks could help them, they would rather get away from agriculture and work off farm to get more income. But the policy of making agriculture the sink of abundant labor condemns the rural workers to poverty because they are poor employees of poor employers. So how can they be rich? Both of them will be at a loss, but are losing. So these workers must find a way to augment at least their income. And this is the model in Japan. The workers there, the farm workers are aging. The average age is much higher than the buildings. Our average age of farmers is 57, and up to 57, we're not that strong anymore. So what do farmers do? They hire liquiders, but unlike in Japan, the farmers there use machinery and they can work by themselves. Also women have a greater role in farming. They can work by themselves. So in the Philippines, they hire unproductive, manual labor, and that is the difference. So based on these salaries, wages of labor, laborations, be augmented somewhere from off the farm. And that's why there's an exodus to Manila because the jobs are there. Unfortunately, this has caused urban life. It has been my privilege to have been in the first, I don't know, I was not in the first bunch, second bunch or third bunch, whatever it is, but the difference April, May, June, that's the interval of the first, second, third bunch. Where was I? It does not flash where I was. And I thank IRI for this, for giving me the opportunities for doing granite work. I hope that through the years I have not worked in vain because of the noble objectives of IRI in giving such scholarships for increasing the capacities not only for the Philippines but also the other rice producing companies. Now here's the story. After finishing my MS, or I was not finished yet, at the time, in agriculture engineering, UPLB and UPD demand, you see, at that time UPLB engineering here was not that developed. So I had to take courses in UPD demand and we were a counter-duck. My colleague who joined me later were fetched by car to Dilman and brought back here every Friday evening, every Friday afternoon or evening because we had to attend the Saturday, the ERI Saturday seminar and a course by engineer Lloyd Johnson. So before I was finished, there was an offer of the Rockefeller Foundation Fellowship for PhD and Dr. Chandler interviewed me on the eve of my birthday and one of the most difficult questions you asked me was how old was I? I said 55. I was 55 then because he had bio-data, short-term data how would you strengthen your bio-data when you are just a young man? Before him was my birthday and I thought he was just testing me and it was a catch-22 question. What is catch-22? Whatever your answer is, is wrong. I said my answer was 55 and he said you are still 54 tomorrow is still your birthday and in America you reckon your age on your after your birthday. Why? Why? I did not disagree. You are not supposed to disagree with an interviewer. I realized that it was a catch-22 question. I had in mind a reasoning that in engineering 24.99 years is already 25 but if I answered 24 he would say the answer would be 25. So it is catch-22. This keeps on escaping me. Okay, next. Really my last word there is that this month Chander was really smart. This reminds me of a story.  They were given a goal, just like they gave a goal, of touching the lips of a sexy lady precisely 10.00 meters away. But precise. From the lips. The question was give the minimum number of counts to have your lips touch the girl's lips according to the following rule. The first count is half of 10. That's right. The second count is half of 5. 2.5 And the third count is 1.25 Right, half of 2.5 and so on. Always holding the remaining half until you reach the lips. So what is the minimum half? Then you have to explain your answer. Well, the scientist answered very quickly. He was a mathematician. Infinity. He said since my lips will never touch the lips because there will always be a half remaining to be half. No matter how infinitesimal it is. Why try? We call that in mathematics an asymptotic to the line. Those of you who knows this. It is quite approaching but not quite touching. That's asymptotic. So that was a fine answer. Well, what was the answer of the technologies? He was practical. The scientist was theoretical. He said I will settle for 15 counts and that is after mentally computing, holding everything. He said by that number of counts my lips will be 0.3 millimeter. Less than a millimeter from her lips and by phenomenon of thermal expansion our lips will touch. So I am practically with the practicality of the technologies and the preciseness of both are correct but in real life you have to be practical and upstream members or the scholars and trainees and the fellows when they get back to knowledge of their institutions will be solving real life problems based on science. I have just cited my experiences and observations as research scholar of IRI. I am sure that many of you have been either trainees, scholar, fellow of IRI and have present memories and rewarded experiences of staying at IRI and Los Baños. Don't forget Los Baños. Perhaps as IRI alumni we can join the UPLD in this is all what they call a UPLD Los Baños spirit a type of band that is unique and felt more intense intensely not in Los Baños or IRI but when you go out away from IRI or Los Baños that's what they call Los Baños spirit instead of saying if you can call it we can call it IRI spirit or Los Baños spirit although we are not there and it works it feels at least when one visits a place they feel comfortable because of renewed bringing of acquaintances and banding and so since we always meet I start with being in the upstream somewhere in the partner institutions you will feel at home our IRI spirit let's say something to think about and what have we to give IRI in return I suppose that in recognition of the efforts, time and resources that IRI has spent on us a scholar, fellow or training it is but fitting and proper as a show of gratitude that we do our best in applying our knowledge gained to be productive researchers in discovering societies or innovating as technologists on various aspects of rise in our home institution or NARES our countries that have sent us to IRI would expect of us no less also we owe also to the poor rice farmers consumers in the environment no less when our best efforts have been made when we have reached a breakthrough when we in adopting the idea of the famous English pro-Satirist he is a famous English pro-Satirist Jonathan Speer who wrote in 1726 it is of my opinion that who so ever could make two ears of corn or two blades of grass to grow upon a spot of ground where only one grew before we deserve better of a time and do more essential service to this country than the whole race of politicians put together that we made early in just substitute two panels of rice instead of two ears of corn or two blades of grass with success when we have improved the lives of our rice farmers we will have to pay IRI by helping it approaching its goal it is the partner countries of IRI and in particular the NARIS and institutions to which we may belong that battleground is what in the world is poverty and blunder of the environment even of that the approach to the goal is at best only asymptotic


Hi, I'm Alex Rackner, and I'm a two-year-old specialist based out of Decatur, Texas. Yeah, in my mind, Gerald Alexander is probably the best two-year-old guy in the business and has been for a long time, and I think that Alex has the ability to be the next two-year-old guru if you want to, you know, you could say that. Just the way it gets his colts broke and ties them to a cow, at the same time, to me, that's key. I think you can do whatever you want to do with a colt as long as you tie it to the cow. It works, and that's what I like about Alex's two-year-olds. As you can see, they can stop them, turn them, do whatever he wants to do with them, and they're still hooked to the cow, and that makes all the difference in the world following somebody. It makes it a lot easier when I get them as three-year-olds. If they're broke enough, you can do what you want to do with them, and they know how to work the cow. I grew up on Kauai, a little island in Hawaii, and grew up on my dad's ranch just riding horses since I was able to walk, and my dad got into cutting when I was pretty young and got introduced to it that way, and got to spend some time on the west coast with some really good trainers out there when I was a kid, and was lucky enough to spend a few years around Carlos Benuelos when I was younger and learned a lot about colts from him. And eventually, here in Texas, I was really lucky to get to be Ronnie Rice's assistant for a few years, and Ronnie helped me with a lot of the same things, you know, just focusing on training a cow horse. And that probably got a lot of my work ethic from him. That horse needs to be tied to a cow, and that's the most important thing in our program. I mean, I feel like anybody can get along with a horse behind me, but I really try to get them tied to a cow and make that kind of their foundation, and that's probably been my biggest thing. I mean, I try to approach them all as individuals, and I try to get them real broke and all that, but I try to base everything on a cow. I like starting them on cattle pretty early and making that kind of the common denominator, but treating them all as individuals and just making them the best they can be, whether they're going to be open horses, non-pro horses, amateur horses, just kind of finding their spot in the big picture and making them the best they can be. The thing I've been most impressed with is the level of communication, and like Lisa, their work ethic. There's not one thing that comes up that Sindel doesn't contact me. We're in constant communication about the horses, even so much as, you know, the vet appointments. We're kind of working together on that to keep horses on their maintenance programs and on schedule, so it fits, it rolls right into our program. And she calls me about anything that comes up. There's never a surprise if they're having an issue with anything. I hear from her immediately, and I'll talk to her a lot of time. On Sundays, they're working horses. I never have any doubts that the horses are getting exactly what they need. Hi, I'm Sindel Rackner, Alex Rackner's wife. I graduated from Kansas State. I have a bachelor's degree in animal science, and that has a lot to do with the program when it comes to just noticing small characteristics of a horse. I learned this from Ms. Teresa. I don't let horses sit in stalls. A lot of our two-year-olds are in pasture, but there's some that are in stalls, and I don't like them sitting in stalls for one day. I like to get them out, get them moving. I'm big on communication, whether it's with Lee, Holly, my husband, customers. I'm big on updates and communication, and I don't like to leave anybody in the dark on anything. I treat all of these horses like they're my own, and any small attention to detail, I give it.