When I was 18, I came in contact with Elijah, who stole cars for a small profit. The thing is, I didn't have a job and I sure as hell couldn't go to my parents for money so I started joining him on these ventures. He must have been at least 15 years older than me. We met at night and would circle big parking lots trying to score a car. We were fairly successful. For about two years, one night we were both able to score a car and were celebrating in a bar. It was then that Elijah told me he knew of a job, but it wasn't going to be our usual MO. If we play it right, it'll be an easy score. He told me while chugging his beer, all we have to do is break into some old dude's place and steal a painting. My curiosity rose up and in a hushed voice I asked him, how much would this painting be worth? He let out a small chuckle and lowered his voice to a whisper. Millions, my friend. I don't consider myself a greedy person, but the thought of making millions in one night was enough to get me to agree. So the painting was rumored to be destroyed long ago. Currently it's in the possession of an old man on the outskirts of town. He whispered, looking back on all this, I should have asked more questions, but the thought of making millions in one night was enough to make me forget all reasoning. Curse our foolish minds for ever thinking we're going to find such riches without paying a price for it. Two nights later, I was waiting in the parking lot where we'd met lighting the cigarette. Judging by the needles lying around, we weren't the first to realize this place was abandoned. My cigarette was almost fully smoked when Elijah arrived. Why did you bring your car? He just looked at me and shrugged. How else are we transporting the painting, huh? I guess he had a point there. After we reviewed the plan one last time, we decided it was go time. We would go in and find the old man's bedroom where I would stand guard making sure the old geezer wouldn't wake up while Elijah searched the house. He promised me there would be no violence involved whatsoever. Worst case scenario, if the man woke up, I'd tell him not to move and if anything went wrong, we just rush out of there. It was about a five minute walk from the lot to the old man's house. We walked in complete silence. There was an uneasiness in my gut already, as if I subconsciously knew what was going to happen inside of that house. We got there and climbed the front gate. This wasn't a house as much as it was a mansion. I kept thinking to myself that I wouldn't even find the bathroom in a house like this, much less the painting. Elijah got the front door opened in almost no time. The entrance hall was as big as my entire apartment. We took out our flashlights and scoured the place. This man was an art lover indeed, as the hall was filled with paintings. They went from giving me a slight uneasiness to me just not being able to look at them. My God, the things that were portrayed there. I remember this painting of a teenage girl posing in front of the Eiffel Tower, but there was this gargantuan silhouette in the background. I had the strangest feeling that I knew this girl from somewhere, but I chalked this up to my nerves. I was entranced by all of these images until Elijah's voice pulled me out of it.  He whispered. I was glad that at least one of us wasn't too taken aback by what we'd found here. The painting that we were looking for was supposed to be of an old lady sitting next to a vase filled with tulips. I made sure to scan every painting in the faint hope that it would be displayed in the hallway. As you can probably guess, there was no such luck. As we made our way through the hallway, there was an open archway leading into what I assumed was the living room. This place was big indeed. There were two doors on either side. The living room's contemporary furnishing stood in contrast with the esoteric artwork that filled the walls. I took slight glances at the paintings not wanting to look too long. I don't think I looked at any of them for more than five seconds, and some of the images still haunt me to this day. One such was displayed right above his fireplace and freaked me out beyond belief. I just stood there, gaping at it. The painting depicted an old woman standing on a crossroads looking up at the sky. Her eyes displayed madness and there was a wide smile on her face. The sky was a mixture of red with black clouds and there seemed to be looming shapes between them. In the sky flew a wild array of impossible creatures. They were a brownish and yellow color, and some were dragging people in their gigantic claws.  The familiar voice of Elijah pulled me out of it again. Uh, I'm sorry, I whispered. But don't you think these paintings are just weird? Weird as some paint, huh? He said, but I could see he didn't like looking at them any more than I did. Let's just find the creeps bedroom so you can stay guard. Let's check the rooms in here. We had no luck on the ground floor, so we took the stairway up. We came out into a long hallway with more doors on either side and one at the far end, slightly ajar. I went to examine it and saw the old man sleeping in his bed, illuminated by a glimmer of moonlight. Okay, okay, good. Elijah whispered, you stay here and keep an eye on him. If he wakes up, just threaten to hit him with the flashlight. I'll check in with you every 10 minutes. I checked my watch to 30. Elijah checked the rooms in the hallway while I kept my eyes on the old man. After a while, Elijah came out of the last room and seemed confused, more than being confused. He seemed broken and in a state of shock. I approached him and asked him what was wrong. Everything's fine. Just make sure that freak doesn't wake up. I'm going to look upstairs. I really wanted to believe everything was fine indeed. But the look in his eyes told me otherwise. That was the last time I'd see Elijah alive. Time seemed to have frozen. As I stood there, trying not to freak about how insane this all was, the old woman from the painting kept popping into my mind. I'm going to try to explain this next part the best I can without sounding like a complete nut job. The thing is, whenever I thought about the painting of the old woman on the crossroad, I could never remember the painting itself. It seemed like some kind of memory.  The flying creatures screaming and hissing, the people walking around and ignoring the monstrosities around them. And that woman, laughing at me, almost like she was mocking me for not understanding what was happening.  I must have gone in a trance there. I usually wasn't like this, especially during a job. I felt my heart stopping for a second as I checked my watch, 255, where the hell was Elijah. My knees almost gave out as I made my second discovery. The old man wasn't in his bed anymore. In a state of panic, I shone my flashlight in his room, and it was completely empty. I didn't know what to do. My instincts told me to get the hell out of there, but I didn't want to leave Elijah behind. I snuck through the hall and up the stairs, opened the first door, and found myself in a library. Again, the strange paintings were all over the room. Only this time, they seemed to be telling a story. In the painting closest to the door, there was a child standing in an open field, hands raised to the sky, and clouds opening up. Behind him stood the old woman, her hands on his shoulder, like she was guiding him. In the second painting was a gigantic UFO descending down towards the child. The next painting filled me with dread. There was the same child, but taller in an alien landscape dancing with these creatures around a makeshift campfire. The things must have been three meters in height compared to the child, and their limbs were positioned in poses that were all but natural.  In the last painting, the child was all grown up, standing in that same landscape, surrounded by other children. They seemed to be smiling, raising their hands, but if you looked closer, those smiles didn't seem real at all. It was like an expression of pure panic in their eyes. Far away in the background stood the gargantuan shape that I'd seen before, only in more detail now. It stood at least 80 meters in height, and had a snake-like body. The thing had no legs, but rested on its tail and hands. Its muscled arms were long and disproportionate to the rest of it. There was an outline of its wings folded on its back. I felt the grip of these paintings coming over me again and left that room. I stood very still, listening for any form of sound. It must have been my imagination, but I could hear slow breathing all around me. I wanted to get out of that house as fast as I could, but I didn't want to leave Elijah behind. God knows he always came back for me in bad situations. At that moment, I heard Elijah calling out my name from underneath me. I ran downstairs, and barged into the last room Elijah had come out of. I found myself in some sort of study room. There was a desk at the side and an armchair in the middle of the room with a coffee table next to it. If you were to sit in the chair, you'd have a perfect view of the painting that hung neatly in the middle of the wall. It displayed an island with a big pyramid in the middle. In front of the pyramid was a lake. People were swimming in the lake whilst objects stood around watching. Two of the people seemed so familiar to me.  There were tears running down his face. The other was a young man looking from a distance at the scenery, his eyes filled with fear. Suddenly it hit me where I'd seen the crying child before. It had been on the news for weeks. The child had been abducted while playing in the forest and was never seen or heard from again. I remember vividly because of the mother's heartbreaking plea on the evening news begging her son's kidnappers to return her child to her. It was because of this that my mind drifted back to the girl in the painting of the Eiffel Tower. She disappeared while her parents were visiting Europe. Only she looked years older in the painting than at the time of her abduction. I shivered at this realization. None of this could be real, right? How could such things even exist? How can light still exist while such darkness opposes it? There was a name plate under it that read, Day Paradiso Die. I almost felt the breeze of the island. When a voice pulled me out of it again, only this time it wasn't the familiar voice of Elijah. I turned around. The old man stood in the door opening. I froze with fear at the sight of him. At first glance, he was just a regular old man, but there was something about him. Something sinister. He placed himself in front of me, smiled and asked, You like my works? He had a proud look in his eyes as if he was expecting me to applaud him. I tried to speak, but couldn't. The man beckoned me to sit down in the chair, and I obeyed him without any question. There seemed to be static in his voice as he spoke, Ah, where are my manners? It's been so long since I had guessed. I sometimes forget the formalities. Can I offer you a drink?  It took all the courage I had to ask him two simple questions. Where's my friend? Who are these people in the paintings? The old man chuckled, I am afraid your friend had grown weary. He decided to let his eyes rest for a bit. As for the people in the paintings, you can consider them as my private collection.  He turned around and looked at the painting of the island. I saw this as my chance. I got out of the chair and tried to make a run for it. Before I knew what was happening, the old man had grabbed me by the arm and made me sit down again. What's the rush?  His words were cold and stern. I could feel his strong grip around my arm long after he let go of me. I'm sorry. Let me just take my friend and we'll leave immediately. He looked at me with disgust and started hissing these strange words. N'gazalame t'junga andrafande andahunja t'jung. These words are still burned inside my mind. I started feeling dizzy and felt I was drifting away. I couldn't have closed my eyes for more than a second, but when I opened them up again, I was standing on the island. I looked behind me and saw the pyramid. It mesmerized me. My body felt so heavy and all I wanted to do was to lay in the grass and let myself drift away. I nearly did until the young man who I recognized earlier in the painting started to yell at me, Wake up, you idiot. He yelled. It was Elijah. Only he seemed ten years younger. None of this made sense. Where are we? I asked him, fighting against that suppressing feeling of sleep. It doesn't matter.  He screamed. I found myself closer to that warm embrace of sleep. As I closed my eyes, his fist came towards his punch felt like a hammer hitting my face. But when it connected, I was back in front of the old man. His sinister look was gone and was replaced with utter shock. How did you get out? I use this moment of confusion and adrenaline and kick the old bastard right in his chest. He stumbled backwards and I made a dash for it. I ran out of the house and kept running towards civilization until I threw up from exhaustion. The guilt I've felt for leaving Elijah in that cursed place never subsided. Weeks later, I saw a missing person article in the newspaper with Elijah's picture. Only his name was displayed as Morgan Fletcher. Guess I never knew him as well as I thought. I thought about going back to that cursed house ever since, 35 years later. I'm still thinking about it. 10 years ago, I got as far as the parking lot, but fear always got the better of me. I never went to the police.  I would have taken this story to the grave. But yesterday, I saw a news report of three teenage kids that were last seen hanging around the area of the house and disappeared without a trace. I have the worst feeling that the old man is behind it. Curse our foolish minds forever thinking we were going to find such riches without paying a price for it.


Okay, well welcome everyone. Thanks for coming. I know this is the end of a really long day Last day of the conference. We're all a little tired And we do have to save our energy for Aaron's keynote. So it's always a lot of fun. So don't miss it But I'm glad you're here. We do have some good things important things to talk about so So to start off who is this person up here my name is Daniel I've been a ruby developer for about 12 ish years or so other languages for longer than that Spent most of that time in early-stage startups really small companies Currently though I work at a different kind of company Yeah, my current company is such a small company So it's a bit careful about its messaging. So some quick notes. I'm not speaking as their representative This is my talk and my views are my own not my employers That said my employer does own my code And for whatever reason that actually includes examples on slides So I should mention that the code samples in this talk are copyright Google and license under Apache 2.0 So that's stuff out of the way I have to say that I was really impressed by David's David's opening keynote. I Like what he was saying about the importance of belief systems Those underlying values that should anchor us and anchor our our decision-making The importance of recognizing what those values are and how they define us So yeah, I really appreciated his message and one thing that it made me think about Was what are the values that have rubbed off on me? Through all the various small companies and not so small companies that I've worked for One of the common themes I think across all these organizations Big or small especially the good ones Has been the importance of measurement of data. I remember my my first rail startup was back in the beginning of 2007 or so that we launched our first rail sites and And So 2007 we were on rails 1.0. So good old days and I remember a few months after we launched Our CEO was on national TV live on national TV for something totally unrelated to our company So he was he was on this on this interview and he happens to Accidentally drop the name of our site on On national TV, so it was totally unplanned. He didn't intend to do it but it was on national TV and The show was actually pretty popular at the time. So as you might guess within a few minutes I get a page from our monitoring system and Yes, our site started to get really sluggish Took a look at our logs and we were seeing multiple orders of magnitude traffic spike so This was This is of course, you know, it's both a good thing and not so good, right? but it was in 2007 and We didn't have these these nice auto scaling cloud services that we do now We were on physical servers in the Colo at the time. And so I Spent the next couple of hours logged into those servers. I was trying everything I could think of To increase our capacity. So I was tweaking our load balancers I was you know disabling features. I was spending it more Mongrel's per machine We were running under Mongrel 1.0 at the time. That was the state-of-the-art at the time nothing really worked we were just we were just extremely laggy and I Guess eventually after a few hours traffic kind of settled down a bit and in the site recovered But in the meantime, we were flailing around in the dark Now later when we did our post-mortem investigation We determined among other things that our Mongrel's were actually memory constrained So we had we had these in memory caches within each Mongrel process and as traffic went up these caches would get squeezed and we start thrashing so, you know, we just You know, it just started pulling over so my attempt to fix the problem by spinning up more Mongrel's Was just squeezing memory even more and making the problem even worse The worst part of this was I couldn't even tell that this was happening We had a caching strategy. We thought it would work, but I Didn't really have good data on its actual behavior in production And as a result, we weren't really able to respond well when we ran into a production issue So it's crucial to have real data On what your app is actually doing in production when it's under production load This is true Whether you're just starting out as we were or whether you're one of the largest most successful companies and have the largest Products some of the largest products in the industry. You have to know what's going on So at these small company that I that I work for now, we we measure everything And if you're if you're just starting off if you're just launching your first app, you also need to measure everything So how do you do that? Well to begin with the obvious There are a number of excellent services out there, obviously that do system monitoring application monitoring This screenshot here shows Stackdriver, which full disclosure is actually developed by my current small employer But there are a number of other companies and a number of other products that are very very good And if you spend some time down the hall at the exhibit, so you've probably met some of them performance monitoring services error monitoring Who here uses some kind of monitoring service in production? Okay, that's that's good. This seems like a majority Yeah, those services generally do a really good job of collecting data and providing visualization and analysis tools That said There will be times when you do need to customize when you need to measure something that a general purpose tool won't give you out of the box So this afternoon what we're gonna do is we're gonna take a peek under the hood at some of the techniques that these monitoring services use To instrument your application and you'll see how you can use some of these techniques to perform your own monitoring Or you can customize the existing monitoring for the services that you're using to fit your applications needs So here's what we'll cover We'll learn techniques for instrumenting your app We'll look at how to gather data without disrupting your running apps behavior in production Then finally we'll discuss what sort of things you should be measuring So let's talk about instrumentation This is an instrument It's an old electro retina graph machine. It's used to diagnose a variety of retinal problems Modern versions of this instruments are a lot smaller. Maybe a little bit less scary But old or modern these machines do actually have one thing in common There they have electrodes and those electrodes need to be in direct contact with your eyes in order to measure things So generally a patient is given anesthetic eye drops So a little bit scary But necessary and similarly when you're collecting data from a running application Measurements do need to be in direct contact with the code that's being in that that's involved and that's the job of an instrumentation API It it's To to collect data from running application It just it gives you the ability to inject Measuring code at key points in your application to put that measurement code in direct contact with the code that's being run Rails apps can use instrumentation API called active support notifications And see how this API works. Let's take a look at an example So remember my imploding cache How would you know if your cache is working the way that you expect? In my case caches were running out of space and when that happens you'd probably see a lot of cache misses Indeed your cache hit rate is a good indicator of the health of your cache in general So let's measure it and we'll do this by using notifications To count the cache hits and the cache misses. So here's what that would look like Whenever rails reads from a cache It calls this method active support notifications instrument This call takes a measurement of the caching code. It notes that the cache was was read records how long that took And it records other information such as the cache key that was read and whether it was resulted in the cache hit or a cache miss It gives all that measurement data a name in this case cache read dot active support So rails is actually already doing all of this for us. We say it instruments. It's cache code So in your app What you can do is you can subscribe to this measurement and you do this by calling active support notifications subscribe So you give it the name of the measurement that you're interested in and you give it a block and whenever that measurement is taken Notifications will call your subscriber block and give you a chance to do something with that measurement data Now I won't go into all the details of the API. That's something that you can look up on your own In this example All we're doing is we're taking whether the cache hit Just right Yeah In this example, all we're doing is that taking whether the cache hit or missed and we're logging it So now we have a log that might look something like this After all afterwards we can run simple tools like grep or word counts on our log and analyze that data and get useful statistics Now rails actually already instruments a number of things for you in addition to cache read So for many measurements, all you have to do is subscribe to them But you can also call instrument yourself And instrument your own code and this is actually particularly important when you're writing your own rails plugins It's a good idea to instrument your plug-in so that applications that use your codes can measure its activity and its performance All right, so so far we're receiving notifications of all our cash hits and cash misses We're logging them so we have an overall measurement of the cash hit rate This is already interesting data and as you can see it's very easy to get just a couple lines of code But to make it more useful we sometimes need to collect a bit of context So let's take a look at another old medical instrument This is a vintage x-ray machine actually from a from about 1900 or so That time x-ray machines were a little more than a tube of radioactive material that the doctor was kind of positioned over the patient so Obviously many of the early experimenters with x-ray imagery weren't really aware of some of the hazards Radiation exposure There so there were some illnesses and some deaths in both patients as well as doctors and researchers around this time nowadays of course When we x-ray we're very careful and we're very specific about targeting exactly That the part of the body that we need to measure and this is something that you also want to do when you're instrumenting your app Most apps have a number of controllers and actions Actually was just talking to someone a few days ago whose company had a monorail with around 400 controllers most of us don't have apps that large but Still you often want to focus down your measurements a little more than Then measuring that entire application So one particular controller or maybe even one particular action So what if I wanted to measure cash hit rate for just one particular action that was interesting So the first option that might come to mind is okay Let's let's turn notifications on at the start of the action and turn them off at the end Trouble is notifications are global They apply to all threads at once and so that will include threads that might be running other requests A lot of us are probably on multi-threaded web service at this point and so this won't work in those cases So what do we do? So here's the technique that you can use Let's start with our existing cash subscriber. So right now it's logging on every cash read Regardless of which action is being executed Now we can determine the action by subscribing to a different measurement In this case the the start processing event This is a measurement that's taken at the start of processing a request by action controller It captures various information such as which controller and which action is going to be executed So we can determine here whether to take a cash measurement Now we need to communicate that information to our cash read block All right, and we need to do that on a per thread basis. So we can't use a global variable So for that purpose Act of support provides a per thread module attributes So you might have seen module attributes like matter reader matter accessor Normal module attributes are basically just global variables and they're attached to a module But there's also this this version that can have a different value per thread It's actually just a convenience wrapper around the thread local variable if you're familiar with that from Ruby But using this pattern we can now communicate between our subscriber blocks on a per request basis Obviously, there are still some caveats. These are still globals even if they're thread scopes. So use them with care For example number of web servers. I think Puma is one of them Will actually reuse threads across multiple requests So make sure that this kind of data actually gets cleaned up or reset to between requests It's not a perfect solution, but it's good enough. I think for this purpose So now we have a technique for measuring hash hit data and for doing so for a particular action Let's take it a step further Here's another interesting looking instrument. This one's from about 1960 It's hard to see here in this photo, but the subject is wearing contact lenses And they have miniature lamps connected to them. So it's actually able to to capture eye movements and eye reflexes At the same time the whole contraption moves and the motion causes visual illusions So the device is actually gathering multiple different sources of data Combining eye measurements and machine motion and it's using that to study some of the mechanics of visual perception Combining information from multiple sources Correlating that information something that you need to do quite often when you're instrumenting your application For example, it's helpful to know your cash hit rate But it would also be good to know how much a cash hit actually buys us is the request latency Actually any different if you if your cash hits or misses So again, here's the code that we were just looking at we're determining which action is running And then measuring whether there's a cash hit or a cash miss and then we log that information We can get the request latency in a number of different ways In this case, so we can subscribe to get another event the process action events This measurement is taken at the ends of an action by action controller So it can provide information about what happened with the action such as the HTTP result or the latency So in this case in this example, we've added the controller. I'm sorry We've added the subscriber to the process action event And we're logging the request latency. So we have two log lines now Logging the cash hit or miss logging the latency To make this more useful. We should combine these two pieces of data cash shit or cash miss and latency Log them together and once again, we do that by using a per threat attributes to communicate between subscribers so now those two pieces of data are logged together and When we run this in production, we might get log data that looks something like this So you can see there's kind of a clear latency win when we get a cash hit But it looks like our cash hit rates might be a little bit lower than we would hope So now we have actual useful information that we can apply towards Improving our cash behavior or as my PM friends at my small company like to say we have actionable data So we've gone through an example using notifications There are several other instrumentation APIs that might be useful There we go. One of the simplest ways to instrument an app is to use controller filters Here's a simple around filter that just measures the latency of the action So this is the easiest way to gather really simple information about a request as a whole You can also write a rack middleware This is useful if you want to measure the behavior or latency of other middleware because you can insert your middleware At any place in the middleware stack You can also use this to install instrumentation code in other frameworks Sinatra, Padrino, Hanami Any non-rails frameworks that might not use active support Finally this trace point Now I think of trace point as kind of the sledgehammer of all instrumentation APIs It's a it's a bit different that it's not part of a framework not part of a web framework It's actually part of the Ruby virtual machine It works similarly. It's you you provide code that gets run when certain events happen in this case at the language level so events like an exception was thrown or Method was called or a return from a method or even move to the next line in the source code So it's it's a very powerful API. It's a bit specialized Probably more commonly used for debuggers than for monitoring and my small company We actually use it pretty extensively to build a cloud-based debugger product Because it lets this instrument down at the source code level, but it's probably not something you use that often for monitoring It is extremely powerful our however, and that's brings up an important issue So here's another image this device is from 1940 it actually measures brain waves. I think in this case it was measuring brain waves for someone who had gone through some PTSD due to wartime wartime experiences But you know we look at images like this and some of them are a little bit scary, right? We wonder if these machines are safe.  So this is an important question when instrumenting in production. It's critical that we can take measurements In production against real traffic But it's also critical that we don't change the behavior of our app in the process We don't reprogram our app because we're taking measurements Safety is incredibly important when instrumenting so we're gonna talk about that a little bit one major component of Safety is that of keeping the latency effect to a minimum So here are some tips for for going about that First of all as we said as we saw Isolate and spotlight the interesting use cases Oftentimes you don't need data from the entire app, but maybe just a few Particular controllers of interest or a few particular actions So isolate those I encourage you to experiment with instrument instrumenting new things gathering new pieces of data However, always go circle back and reevaluate If it turns out that some measurements is not really giving you interesting information after all don't be afraid to delete it I tend to treat instrumentation myself like tests Many of them should live on indefinitely because they're monitoring critical systems But there are some that really are only useful temporarily Maybe because they're part of a one-time investigation that you did Or maybe because you put them in and turns out they they're not actually as useful as you thought they would be If you leave them around too long, they'll just slow you down just like your tests might Might just take longer than you really want So practice making those judgment calls Don't be afraid to spin up new instrumentation, but don't be afraid to delete them as well Sample your data if you can Often you can get away with measuring only one in a hundred instances or one in the thousands Trace points can be particularly dangerous Because many of its events can fire extremely often remember move to the next line in the source code So use it only when you have no other choice If you do need to use trace points, here's a pro tip It's global by default Just like we saw with active support notifications. It applies to all threads at once However, there is an alternative trace point API that you can use That's lets you instrument just a single thread at a time. The catch is it's only available as far as I last time I checked as a C API you can't call it directly from Ruby So it's harder to use but it is available and it's worth investigating if you want to use trace points temporarily for specific requests Finally, of course pay attention to how you're reporting your measurements If you're just logging to the file system That's usually pretty fast But as your app grows You might want to send start sending data to a remote analytic service. You might want to start sending data To your application monitoring service that you're using And so when you do that make sure you don't block on those calls Usually the the API Gems for the for the monitoring services will have a non-blocking Version of those of the clients that you can use if you have to use Just straight HTTP calls be careful of it net HTTP posts because it is blocking it It waits for the for the the response to come back You don't want to do that So use an asynchronous client if you can spin up a background thread Send your data in batches if that's allowed by your API in general just be very careful about how you're reporting your measurements Another element of safety is avoiding side effects Those are changes to your app's behavior. So on one hand, it's kind of obvious, you know, don't modify your application state In in your subscribers But side effects can take many forms a database query in addition to potentially adding latency You should also consider the side effect Might not nominally change the state of your application But it does change its behavior because you're invoking parts of your system that you otherwise wouldn't So be careful about those sorts of things in particular Calling methods on active record models. Some of them might initiate additional database calls. So you want to be careful about that So we've talked a lot about how to measure We have a few more minutes. Let's spend or spend those on what to measure This is an early ECG machine Electrocardiograph machine. I think some people say EKG This is from 1901. It was actually the first Machine that was sensitive enough and practical for medical use I think scientists had had experimented with some of this similar technology as early as the late 1700s But it wasn't until around 1901 That it became practical Again, it's measuring electrical activity the electrodes here are actually in these metal basins Next to the patient. So you would have one foot in both hands immersed in salt solution in these basins ECGs are of course still widely used today Not just to diagnose heart conditions, but also to monitor patients who are critically ill or undergoing general anesthesia This is of course because the heart is an indicator, right? Something is going wrong. It often shows up in the heart's behavior So what are the indicators for your application? What are those things that can tell us when things are not healthy or when something unexpected is happening or Something has gone critically wrong So here's some of those things that you can measure first results The responses that you're sending back from your requests make sure they're in line with what you expect One indicator that sometimes really important is how big are your responses? Are those sizes what you expect? Also pay special attention to error responses and obviously if you 500 then there's there's problems going on But don't ignore your 400 levels 400 error levels as well When I was younger and less experienced I tended to make the mistake of ignoring my 404 error rates Because I was thinking oh, it's just client error. It's not server error. So it's not my problem You might not necessarily want to get paged if you see a 404 spike but you do want to know that it's going on because It could mean that there's a broken link someplace or it could mean someone's trying to hack your site It's an indicator that something is going on. You should at least know about it Your final responses are important indicators, but not the only ones sometimes you have intermediate results That can be useful Another thing is pay attention to rendering Rails templates are really not the fastest thing in the world. I've just come to realize that over you years of using them Rendering can have a significant impact on your app's latency. So measure it not too long ago. I was actually working on an API that Occasionally just occasionally ran really slowly and as we dug into this Turned out that it was actually the JSON serialization That was taking up the bulk of the latency some weird Some weird interaction between what my data looked like and some issues with my JSON library So if something like that is going on, would you be able to tell of course measure your interfaces with external systems? That means your database external API's internal API's micro sorry M word Services API's your caches File system Now out of the box monitoring products will capture many of these things for you obviously, but not everything They typically po focus on the performance of these external dependencies Like how long your data face queries tend to take But what's often missing is your usage Is your application using the external system in the way that you expect? Are you hitting your cash as often as you think you should be? Finally errors and exceptions are important.  That includes expected errors Includes errors that you handle internally in your system and you don't bubble up to your users They can still be indicators an important example. That's often gets overlooked is retries Often when you call an external API you implement retry logic, right? Because network can be flaky Various things can be flaky, but if you retry don't throw that information away Instruments your retry code make sure that that information shows up on your monitoring dashboard Your retry rate is an indicator if you get a retry spike something's going on you want to know about it So we've covered a lot here and really we've just scratched the surface on the number of things But I hope I've communicated the importance of measuring again It's something that my small company right now. We just we we just do out all the time out of habit Getting started is really simple Subscribed to an active support notification as we saw just a few lines of code measure something and log it or Many of the commercial products that are out there have free trials. Go check one out. It's free However, you do it start measuring Nothing else data will be interesting to look at could also save you a big headache in the future So that's all I have. Thank you


Thanks for putting this together. I'm a little bit behind Frank, so I'm glad to have this opportunity to share my project. Yeah, so my name is Leah Sinkowski, and I have a very small herd of goats in Ada, Michigan, right outside Grand Rapids. So the title, Designing a Mobile Milking Parlor for Educational Goat Farms, is quite a mouthful. And I think it has lots of applications for all sorts of goat people. You can go to the next slide, I think. So my little farm is located currently at the back of my elementary school where I went as a kid, Ada Christian. So they have about 37 acres, lots of brush, lots of woods, plenty of space to graze goats all season long, or all growing season. I started out originally before they could be milked. I did a few different grazing projects throughout the county, and that had its challenges. The goats got out, etc. But we've come to a good place now at the school. Well, we got kicked out a couple of times there, too, with the escape, but we figured it out. So then we actually ended up expanding to another, this time a private middle school, and I really enjoyed working with that group of students, too, but currently are just operating as a one herd, one school farm. So as my dreams have changed, as my situation has changed, this like mobile milking parlor, mobile milking space has always sort of been in the back of my mind. Initially, because I was like I want to graze different locations and charge for my grazing projects. And so the parlor would allow me to milk my grazing goats. And then as I expanded to a different school, I thought, well, this is perfect. I could service the different goats in different locations. And then I realized I'd just be milking all day. So now kind of my newest sort of realization is that I think, similar to Frank, using it as sort of a mobile education or as an educational experience, but being able to move it to more like places that don't really have a lot of agriculture, places more inner city or more, more suburban, things like that. So I applied for a SAIR grant in 2019 and I'm slowly working through it. I still have some sharing to do yet this month. I have another talk coming at the end of the month, but it was a pretty smooth process applying and yeah, I've been really enjoying digging into the design and design and engineering of it. My grant is a little different in that it is more of a theoretical grant. It's not. I didn't actually build this thing. So we'll learn more about the plans on the next slide. So conveniently, my father owns a design studio that he just recently sold, but I worked with them to design and engineer put together a floor plan for this project and they did some renderings. I also had a really enjoyable time meeting with meeting for the first time, my local dairy inspectors. Currently, I operate as a herd share, so I don't have any inspection, but it was nice meeting them and getting to know sort of the standards in Michigan for the Pasteurized Milk Ordinance, which is one of the hardest things, hardest states to get a dairy certified in. So that was kind of interesting to learn. I created a bill of materials and priced out a custom trailer from our local hallmark and did a little bit of reaching out to cheese makers to gauge sort of goat milk market in the area. Next slide. Awesome. So here are some different views you can see of what we ended up with so far. So there's actually three rooms to our trailer design. So as you can see, yeah, it has a hook up on the back so you can hook it up to your van or truck or whatnot. Yeah. So the front area there you can see with the goats. I think those are actually adapted deer, but they're supposed to be goats. They are in the parlor area. So that silhouette is kind of showing on mute. Sorry. Join audio. Did I lose something? No, you're good. Keep going. Awesome. So behind the parlor area, there's two other rooms that you can't see in this drawing. But one would be the washroom, which has the sink and some storage space on the other room is the bulk tank area, the milk house where you would cool the milk. And yeah, keep it cold back there. Yeah. So we designed it in mind to meet the pasteurized milk ordinances of Michigan. So we designed it to be clean and safe and certifiable, but also in mind that people could use it sort of in a lot of different applications. Applications. Here we go. So yeah, there's a lot of different ways that I see this parlor being utilized. Obviously for me, it would be kind of a way to diversify my income and be able to travel to places of learning, nonprofits, camps, things like that and do sort of teaching demonstrations. So I do a lot of that now, just at the school that I'm at. The school actually pays me currently to be at the school key and I go there. So it's a very sweet gig. And yeah, I could use in the future the ability to diversify that and being able to bring it to other places. But originally my dream being to do some land clearing, in addition to doing the milking, that would be one application. I could see different goat breeders. There's so many people that breed goats that I imagine it's more of a hobby, but could turn it into a business by putting on demonstrations. And different agriturism farms expanding their offerings to include milking as well. A goat hobby farmer expanding into a business. A new farmer accessing shared land, free land, grazing another farm's cover crop or doing some crop rotations on a different farm, clearing out brush for your neighbor, things like that. A school or a 4-H educator using the parlor to train their students to milk or to teach other things. Or a farmer could use the trailer as a multipurpose transport vehicle. So if you're doing a trailer anyway, and this is the kind of way to do a dual-purpose transport. Oh, I forgot to mention that I was starting to say I do a lot of milking demonstrations. And for some reason, I don't, if kids just think it's like weird or something, but I always find they're very mesmerized by the milking demo. So I think it's hitting on something. They eat a lot of dairy in their lives and experience that a lot on the eating end, on the consumer end, but getting to know sort of the farm to table, the background, the aspect of how it's produced. Kids do really seem to enjoy. Sorry, next slide. Awesome. So here are a few, a few different views you can see. The one kind on the left, the lower left is sort of what the audience would view. Really, it's viewable from three sides. So it's like a little more than 180. So maybe like 200 degrees of viewing area. So in terms of like an outdoor learning experience, COVID safe, everyone could definitely spread out and be able to see. Yeah. So it's designed so for like a bucket milker. So you can see that's that silver thing on the bottom left. So that's really the simplest milking system you can implement. Modification, of course, would be like a more sophisticated pipeline system that you could use that would be quicker and less hands on. But the buckets can fit behind the goats there or between the goats. And the way with only milking two goats at once, that works really well for the system, for educational purposes, because you can walk all the way around the goats. Of course, if you were trying to be really efficient, you could milk up to four goats. You probably need a little bit of fencing on either side to kind of keep them in. And then you would need back access. Right now, you can't really see it, but there's a ramp that comes down. So this back, the back, the view we're talking about on the left, is covered, of course, by a back. But instead of rolling up, it pulls down into a ramp. So that's how the goats access the parlor. But in another modification, they could come up the side, and then you could stand in the back and milk the goats from the ground. So that is an option. And then to the right of that, you can see the side view with the goat and the stanchion coming right out of the ground. And then the back room there, which you can see a little bit of the silhouette of a sink. And then that gray thing is like the bulk tank. I just put in a 14 gallon bulk tank, which is about the smallest you can get. I think there's a little bit smaller possibly, but depends on how frequently you sell your milk, obviously, and how many goats you have that you're milking. But you could manipulate that as you wanted. So that would be the milk house, the bulk tank cooling room. And then that iron up above, that bird's eye view there, you can see from there, you can see the drain in the floor and the parlor. That's one thing for the pasteurized milk ordinance, you need to drain. And then in the other two rooms, you can see from up above, there's a hand sink with the bulk tank, and then a two compartment sink in the top room. And that's where you would wash and store the milk can. So we priced this all out from Hallmark. So actually, within the parlor, there's like cabinets and a sink. And that's kind of where you'd keep all of your filters or cleaning solutions and stuff. And that area is built right into the trailer. So that's included in that price. You can kind of see there's some stairs, it's kind of built similar to a camper where the stairs pull out from those rooms. So you access both of those rooms from the outdoors. So after you're done milking, you'd walk around the trailer to enter either the, either the cool tank where you pour the milk into the bulk tank or walk the other way around, go up the stairs in order to wash your milk can there. So if this were to, if I were to use this parlor as a sort of demonstration parlor, the process that I would take is I would drive to, I would trailer my trailer on my truck and drive to whatever location I was teaching at. I'd have to park and stabilize the trailer, unload the goats. I didn't really mention that, but seven goats could theoretically fit across. Probably that would be like an eight and a half wide trailer. We have, I think it's like seven and seven, seven by seven feet, seven inches. So probably more like six goats tops can fit trailer in this area. Another modification would be you could have another kind of a wall that would come up so that you could bring younger goats since they tend to like lay down when they travel. So there's a little bit more room for that if needed.  But anyhow, so that's kind of an option if you were to trailer, if you were to travel with your goats inside there as like a hauling area. So anyhow, you'd unload your goats. We haven't really come up with a better portable fence than like a electric net kind of fence. Those aren't great for petting zoos because they electric shock children, but that'd be something to look into. So yeah, you need to hook up water. Another pasteurized milk challenge is that you can't have a clean water tank. And so you need to have a water hookup, which for a school would not be a problem. If you were trying to access a really far outfield in the middle of nowhere to milk your goats, then that would be an issue. So there are some limitations we'll get into. But yeah, so you hook up to water, you turn on your generator, assemble your milk machine, you get your goats in position, connect the vacuum, begin the milking process. And like I said, you'd have to walk the milk around to empty it out. And then because there's water in your trailer, you'd be able to use your tankless water heater to heat that up, totally wash down anything that needed to be washed. And then to exit, you'd have to reload your goats, reconnect your trailer, and be on your way. That was a lot of information. You can move on to the next side, I think. So I got into some of these challenges already. But obviously, there's a certain cost, which I'll get to. And that would have to be recouped by grazing or education. If you were a bigger farm that already had a parlor, and this was just kind of a second one, you could, yeah, you'd have, well, we'll get into that. But you'd have to somehow recoup the cost, of course. The test of the water thing, we got into a little bit. Tanks are not allowed for a certified parlor in Michigan. So you would need to hook up directly to a source. Unless, of course, you're doing a herd share.  Also, you need a nearby bathroom, like in order to have this normally on your farm, you'd have a bathroom somewhere nearby. That would be you could utilize. So if you're out in the middle of the field somewhere, you wouldn't have water, you wouldn't have a bathroom, obviously. So there's also a limited capacity. So like I said, six to seven goats could be transported, two or three could be milked at once with some modifications. You need a place. There's like a, it's in the front, near to the truck. There's like a space for a gray water tank. So you would need a certified spot to dump that in an approved location, like a campsite or something like that. You'd also have to get some certifications, which would you would have to get if you had a mobile or an immobile parlor as well, milkhouse as well. But you'd have to have your farm be graded certified, your parlor be graded, raw dairy certified. And then you'd also have to get like a milk hauling license because you'd probably use your trailer to haul your milk to wherever you're processing it. Also, it's hard to do a random inspection when you're moving around. So you'd have to work with your local dairy inspector to report your milking locations or submit some kind of calendar. Okay, sorry. I should have probably put the costing first, but that was asked about. So we'll get to that next, maybe. Is that the next slide? Oh, sorry. One more slide, and then we'll get to cost. So we talked about most of these modifications, I think already. But if you wanted to be more efficient, you could have a pipeline system for the milking. And you wouldn't necessarily need a bulk tank if you were herd share, because you're not certified. So you could use some kind of modified freezer system, which would be cheaper. That's one of the more expensive parts. And then, yeah, so there's actually a farm in Indiana, Greta's goats, the Winkle Bowers, and they have one of these, except for it's just a parlor. So the back two rooms don't exist. So that front parlor, they're able to drive back to their farm, their main site, and wash it down and clean it up.  So that's another modified option, which would make it a cheaper trailer to produce. And obviously, you wouldn't need necessarily electricity or water, things like that. So actually, you need electricity, unless you're milking by hand. But you wouldn't need the water. So that's another option. And then I think we'll get to the price on the next slide. So yeah, we got a quote from Hallmark, and it's $16,795. So that's the buildout, which includes the three rooms. It includes the sink, the one sink in the parlor area, along with that kind of built-in cabinet to keep everything. You probably need to buy a couple more storage spaces in the back rooms. And then here, as you can see on the right, is the readout of all the other stuff you need. So generator, the tankles heater. You need washable walls and washable floors. The drainage, like I showed you, is built in. But you need some kind of vinyl flooring, or people use RFP panels for the walls for them to be washable. You also need good lighting in each of the rooms. So you need to make sure there's some pretty solid lights. The cooling tank, so the one I priced out was like a little over two grand. That's like the smallest one I saw. You need another hand wash sink in the other two rooms. And what else? Well, I use the kitchen sink to wash my hands in the washroom. And then I put in there a milk bucket washer, which is just to wash the tubing and the claws that hook up to the goat teats, because I think that saves a lot of time and makes things more pleasant. But you could get away without that, of course. And then if you don't already have some kind of milking system, like the milk bucket assembly, then you need that as well. And I added a filter right in there so you can just filter it in line. And then I also added the headpiece in the tray where the goats are locked in. Those that cost, you could also buy those to insert into the trailer, which of course you need to remove during travel and things. I think that covers all the pricing. So the total cost of this system would be $24,742 approximately. I have some unlisted items like the portable fence. If you were to keep your goats in a portable fence while you got ready to milk and did the demonstrations, you'd need like a hay feeder and a water trough, other things to store your goods, feed, cleaners, et cetera, in the trailer, in the cabinets.  Awesome. So I think that covers that slide. So obviously I had a hard time doing the financial feasibility because everyone's farm is so different. And this would be, this would interact differently depending on what you were doing, whether or not it's like a main source of income or a side source or you're not even caring about the milk and you're just dumping it and you're talking more about the milking demonstrations. So there's a lot of different ways this could go, but I sort of priced out, did like a sample business plan was sort of a combination of using the milking while you graze goats and then using the milking at sort of an educational service. So I kind of priced out a combination. So if you're a goat farmer, you probably know generally goat farming is somewhat of a lifestyle choice. And it pays for you spending time with goats and it pays for your farm that you live on and your phone bill, but it's not like a super, I think the median income for goat farmers is between like 25 and 47,000 a year. So, you know, yeah, depends what you're looking for. But I have seven goats and that's kind of what would fit in this system. And so I know that they pay for themselves, but they don't necessarily pay you a living wage. So this would allow you to do that theoretically. If you could operate your grazing service, grazing slash education for nine months, you probably need to take time off the grazing or education during the really cold months of Michigan at least, and also take some time for kidding next season for having your babies. So if you worked about nine months of the year, you got paid 750 a week for grazing and did that for 16 weeks a year. And then adding the educational experiences on to that the other four months of the year. I think you can make more money doing the education, but and this is kind of just based on looking and talking to people who do these things. So coming out with an annual income of 46,000 as a guess estimate. Yeah, so what did I learn? Well, my ideas have been evolving so quickly over the time that I started this grant till now. So, yeah, it's sort of sort of still continue to evolve in my learning. Even since when I started I had a different idea of what I would do with it. So I think I don't think I mentioned, but part of this whole thing was I made a website that the dairy inspector recommended that if theoretically if I was going to start a educational service out of a mobile milking parlor and sell that milk to a cheese maker, for instance, I wouldn't want to confuse that with my herd chair because of legal reasons, you just don't want to mix those things. So I started kind of a website to sort of showcase all these items and that was a big outreach point of my grant. So I'm still kind of reaching out and sharing that and sharing sharing the design of people. So I'm excited to see what I still have to learn over the next few weeks. I think we're doing questions at the end. So thanks for having me and listening to my presentation. Also, Leah, thanks. That was fantastic. I have a couple questions for you before we switch to the next presenter. Folks wanted to verify you've already built your trailer, correct? I have not. This is a design and an engineering grant. So, yeah, I think it was like a $9,000 grant and like $5,000 of that went into paying for the design and engineering. And then the other portion was for sharing the design and doing. Okay, great. And that leads into our next question was whether or not this was under a farmer rancher grant. And if so, because that's only a $9,000 grant, how did you fund the remaining balance of the project? Oh, I see. Sorry. Yeah. So I didn't build it. So that probably clears up that question is a theoretical project in which I'm sharing the floor plan, the bill of materials and sort of my research with farmers and ranchers. So, yeah, it's just a farmer rancher grant. Great. And then the last question we have for you is if the cost of the milk parlor included labor to build it, or if you were planning on doing that yourself? So the like 17 grand or whatever it was for the trailer included like includes water a place for water hookup. It doesn't include like the plumbing or like lighting. So that is correct. I did not include the cost for those things in the thing. So the basic trailer itself, water hookup, electric hookup or electric hookup for the generator. And then the yeah, the basic rooms and windows and ramp and that kind of thing all is included within that quote from Hallmark. But the other aspects would have to be added in. Yeah, that's right. Excellent. Thanks so much. That's really exciting and has so many applications to even like mob grazing when you're off site and you have to be far away from a barn. Really excited for you for sharing that with us. Thank you so much. Thank you.


Good afternoon everybody I'm Tommy O'Brien comedy live from T F N N headquarters in St. Petersburg Florida three PM Eastern time on Thursday final trading hour of the day we got markets right up there near record territory Dow Jones up 187 points trading at twenty nine thousand two seventeen S&P's positive by more than 18 points trading at thirty three oh seven Nasdaq positive by sixty seven trading at ninety three twenty five in the Russell up almost nineteen points more than a solid 1.1 percent the Russell trading at seventeen oh one start things off checking in on the VIX as you would expect very muted action in the VIX actually surprising it's ticked up a bit as the market has just traded a positive territory but on the day that you have the chief justice of the Supreme Court getting sworn in for an impeachment trial nonetheless the VIX twelve thirty five start things off we'll jump over the charts we'll start it off with the Dow twenty nine thousand one seventy and it's been a trend in one direction to the upside since last night we're at basically session highs we were up there at twenty nine thousand one seventy two twenty eight we're about to tick right above that level as we speak Nasdaq one hundred little bit of a different story off the highs we made earlier today of ninety one nineteen currently trading ninety one oh five S&P's almost at the highs as well we made it to thirty three oh nine and change almost thirty three ten and we're within two S&P points of that level right now crude oil volatility to the upside for the first time in a wild and crude we trade from fifty seven fifty five at seven a.m. this morning all the way up to almost fifty nine dollars we make a high of fifty eight eighty five we're slightly off that level at fifty eight forty four there's your gold contract trading at fifteen fifty one and the euro U.S. dollar one eleven thirty five terms of what else we have happening in the market earning season kicking off this week with the banks and Morgan Stanley closing out with quite a number their shares up really eight percent I think last time I checked as they beat across the board fourth quarter profit surging to two point two four billion dollars or a dollar thirty a share that's well above the ninety nine cent estimate excuse me and the big number that they had revenue for the quarter climbing twenty seven percent to ten point eight six billion exceeding analyst estimates by more than one billion with the B dollars and Morgan Stanley up about seven point one five percent right now pairing some of the gains but still quite a week for the bank stocks to kick off earning season stay tuned folks we got a treat Larry Pezzavento filling in for Tom for the hour live from three till four as we wrap up the trading day markets near record territory stay tuned folks Larry Pezzavento coming up right now