Tom here from Lawrence Systems and the 3CX VoIP desktop app has been compromised to deliver malware via updates from 3CX. So yes, this is a supply chain attack. Next, currently known as of March 30th, 2023 as of the making of this video, but there could be more news later. But currently known affected versions are 18.12.407 and 18.12416. If you have those installed on your system, or you're an IT person like ourselves and you're managing IT for companies, please remove them now. Don't keep watching this video. If you've already done that part, great, let's continue. Now this being a supply chain attack, we're not exactly sure of the how. Right now, we're dealing with more the what got delivered and what happened. This is really interesting because it's taken a lot of community and I've been watching kind of on the sidelines, I'm friends with many of the people at Huntress and watching all the not just Huntress, of course, but all the other people collaborate. Matter of fact, this was sounded off as an alarm by the people over at Crowdstrike. Crowdstrike sharing their details, then more and more companies sent a one having a write up on this Huntress having a write up. And matter of fact, I'm going to be citing the hunt straight up because they did a good job of listing all the other write ups in there. And you know, I know John Hammond and many of you maybe follow him on YouTube, great security researcher. If you didn't know, he also works for Huntress and he's the one that wrote the blog post. And we're going to dive into kind of taking this apart. One of the tricky parts about this and why so much community effort is needed is just because you have one of these infected versions, we're going to talk about the mechanisms on there. It appears to be a very targeted attack. So having the version that has the potential for the compromise does not necessarily mean the compromise itself was activated. It was probably only activated for certain targeted individuals. This goes back to the SolarWinds attack. A lot of people didn't want to take the time to really understand what happened with the SolarWinds attack, but it's very similar. Even though they delivered the payload or the infected system, I should say, to all these different SolarWinds partners, only very specific partners was it actually activated on. So you could actually have this in your environment, but it was never activated from a second stage. And that's appearing to be what this is here. So just because you have it, maybe you weren't the target, but we don't know that is still kind of speculative. We don't see it activated everywhere. So at least that's knowledge they have, we're just not sure at who it was activated at. And that's probably what had CrowdStrike tripping the alarm going, Hey, we seen it. So it took the action of it, not just the install of it. And I believe the install date was 322, 2023 for these versions. And then the activation or when we started seeing the indicators of compromise or at least CrowdStrike specifically called them out was on 329 of 2023. So there's actually kind of a seven day dwell here. And this is kind of the tricky part, because if you think about how a VoIP desktop app works, it's going to integrate deeply with the system, make a lot of calls to other things like calls to the actual servers, but also maybe integrate with other services on the desktop. So it itself can very much look malicious. And then that's a really tricky part from a security standpoint is it is an app that's going to have a lot of privileges to talk to a lot of different things internally and externally. And deciding if those things that it usually does are malicious is very tricky. So let's run over to here to the right up and kind of walk you through the steps that this goes through. Because I think it's very interesting and that blog post is kind of being actively updated. Now they have a whole timeline here. So the 1140 AM EDT on March 29th, Hunter received an inbound support request from a partner concerned with a new advisory and discussion on Reddit. This is the post that was from CrowdStrike. So CrowdStrike posted this right on Reddit, they got the news out there, they're sharing this information. This is an important component is security does not happen in a silo or just by any one individual and CrowdStrike, hey, they sound the alarm awesome. This sends a lot of security researchers going, let's start digging into this, let's start taking this apart and analysis. And when you see the complexities of the analysis, this is where it takes a lot of people to put this together, because there's a lot of components that go into compiling a piece of software. So on March 29th, numerous CDR providers and Antivirus Solutions began triggering and flag on a legitimate sign binary from 3cx desktop app. This application had begun an update process that ultimately led to malicious behavior and command and control communications to numerous external servers. Unfortunately, in the early time of this community's investigation, there was confusion of whether or not this was a legitimate alert. This is one of those things where false positives happen a lot. And they're constantly being researched by people and whether or not they're actually false is so hard to determine because it looks suspicious. But as I said, this is an app that has a lot of hooks into things because it needs to to get its job done as a desktop application. So it actually started a little bit beforehand and it looks like some people had flagged it as not malicious. There's a few forum posts I'd seen on this. I'm not a 3cx user. So I didn't have to make that decision. And I don't, you know, fault the people who said this doesn't appear to be doing anything because at the time it technically wasn't. And this is what's talked about how it actually gets the full payload in there. The 3cx download available on the official public website had included malware installations already deployed will update and ultimately pull down this malware that includes a backdoor DLL and an anomalous D3D compiler. And this is that mechanism. So we have the desktop update updater. Backdoor is invoked retrieves payload. So it's actually not from but was triggered from that 3cx install. So 3cx has a binary that has an extra call out to do something else. This is how we know what 3cx was compromised here. But the payload's not in the actual download. It's an evasion technique that they're using where when they realize it's on somewhere and then they say, Hey, go ahead and grab that secondary payload. So you're adding something to an already downloaded binary. This is just from an evasion standpoint, really interesting. So C2 command and control service. Here's a list of them. And this is where people say, Well, won't these next gen firewall solve my problem? Well, Azure online storage.com. Is that legit? Actually, it's not. But we know that now would a tool that knew this registered domain. And you're probably thinking, Well, Tom, newly registered domain shouldn't be called out to really these were registered a while ago. Some of these are actually being registered. I believe I didn't check the date on every one of them, but a couple of them weren't quite old. And they, when there's a lot of careful planning and attack, this is something that's done. So these look like somewhat pbxsources.com, gcloudservice.com. If you're watching from a firewall standpoint, this question comes up a lot, you know, firewalls, they can block what's known, but if these are new attacks and unknown, you know, you're going to have a hard time blocking them. And it's really interesting. One of these callouts was actually the three CXCOM blog slash event trainings makes you wonder if something was there as well. Once again, we're trying to reverse a lot of this, where I should say security researchers are. So the how is still going to be long time coming probably to really get this sorted out massive kudos to our security researcher and resident binary ninja, Matthew Brennan. So once again, there's a deep dive here, talking about the different pieces that are in here, the back door FFM big DLL acts as a primary loader for this file right from DML entry point. It enters a new function that we have renamed MW main function for our reverse engineering purposes.  And this is what's important is to want to really highlight the complexities of doing this. I have a feeling John Hammond being the fun YouTube presenter that he is will probably be doing at some point in the future, a deeper dive into this because this is his write up on his channel. And he's better at using things like this. I I'm a casual observer in awe of their skills. This is not goes a little outside of where my skill set is of reverse engineering binaries. But I love reading these write ups because I understand them from the concept part of once you've reversed it, okay, this is how it's doing it. They go through each piece of the component. They went through all of this. We'll scroll down a little further. Here's those domains again that they elicit. And John actually reached out to GitHub, because the report of this GitHub earlier today and what John did was report that yeah, some of these are malicious hosted over in GitHub. Once again, you asked about the firewall where people always ask about the firewall, I should say, GitHub's a legitimate place, you're going to pull things from. So someone compromises the GitHub and then uses it as a delivery mechanism, your firewall is not going to flag that or your sim tool may note. Yeah, they went to GitHub. But you know how many things go to GitHub every day to grab something. And if this was legit, prior to it's being compromised, there's no reason to actually flag it until we know. We'll scroll down here. There's a few other things. Huntress has a way to detect it, as has many other people will just kind of jump down to the bottom here. This will be ongoing for a while. So attribution, while definitive attribution is not yet clear current consensus across the security communities, this attack was performed by a DPRK nation state actor. I found that really interesting because generally attribution is something really hard to do and just not done in the early days of an event like this. CrowdStrike came right out and set it in their very first report. We think this is DPRK. So that's definitely interesting. The latest recommendations from 36 co is to uninstall a desktop client. They're preparing for new releases and update three TX desktop to be made available. Huntress assistants fully aware of this ferdy assistant. We realize our efforts are just one pebble in the pond. With that said, our goal is to always keep partners safe and as much as we can to help the broader, small and mid-sized community. Matter of fact, Huntress has a free sign up if you want to scan your system for it. They're offering this. So if you not a Huntress partner at all, you can actually test this out for free. Pretty cool. And once again, resources and references, I expect this to grow. So as of the time of the video, here's a list of them. I think they did a great job here. CrowdStrike's original post. And I've got this link down below as well. CrowdStrike's formal blog post, a company called Total. They're reporting on this, set in the ones reporting, set in the one calls this smooth operator. I thought it was a cool name. Hats off to them for the naming at that. Thought it was neat. Discussion of three CX4 in public. Bulletin board. There are several posts and forums in there. The CEO first official notification. We'll talk about that in a moment. Nextron systems, Sigma and Yara rules. Actually check those out. That's a pretty cool thing if you want to play with it. Alienbolt pulse. Actually, subscribe to Alienbolt. It's great for getting some of these security updates. Kevin Bowman's commentary, Patrick Wardle's commentary on the Mac variant, because yes, this appears to be a Mac variant in there as well. And they're going to go through more of the attacks, compromises, hashes you can look for. Now, let's talk a little bit about the three CX posts, because this is the one part where I think we deviate from a win with the security community to a fail on three CX's part of not really getting the news out there soon enough in my opinion. You know, link to this down below, but threat alerts from Sentinel one per desktop update initiated from desktop client. This actually is on March 22nd of 2023. Anyone else seeing this issue with other AV vendors, I'm glad this is all public because, you know, there's a lot of discussion back and forth. You can read this is where people, the end users, so to speak, of three CX are going, why is this getting flagged? This is interesting. So they're seeing these notices, but then we fast forward a little bit here. And there's a lot of confusion in here because three CX to not get ahead of this at all. And this forum has just been kind of interesting because CrowdStrike sounds the alarm, but then there's people still saying, well, they're my three CF rep didn't really tell me to install this. So there wasn't a response right away. And I get some companies may for legal reasons want to remain silent for a little while until their legal department clears what should be said. But it's really, you should at least say, yes, we know it's compromised, uninstall the app. That is a pretty simple thing they could have said, but didn't say there is some statements from the CEO. And thank you for all those supporting in these difficult times, you know, the usual stuff. I don't know, I don't feel the response was very fast enough. I'll leave that really up to you because that's more of an opinion on my part here. Either way, the actions are still the same. You should be uninstalling this. And I'm really looking forward to figuring out what actually happened. Because as we secure things more and more from the front end, we talk all the time about better 2FA mechanisms, good password hygiene, et cetera. And the harder you make it to get in, the attackers have to go to different areas. I talked the other day about session token sealing when they can get on your computer and copy session tokens so they can just be you bypassing all that heavy security on that front end. And this is the same thing with supply chain attacks are one of the ultimate hard to defend against things until we get things like better software bill of materials, SBOM, if you're not familiar with it, where we understand everything that's in there, we understand the sources of it, where it can be examined faster as opposed to waiting for John Hammond or other security researchers to reverse engineer everything. If there were really good manifests, and this is one of my big advocates for being an open source advocate, you have better manifests generally an open source of what went into building something the source is listed. This gives you a really good understanding of where things came from. And then you can start hunting it all down and go, all right, I know this came from here, this came from here, this repository, let's check out these different repositories. Oh, look, this one had a recent poll request that looks suspicious. This is something we just really need in the industry, because these attacks are not going to go away. They're going to keep being there. There's a lot of dependencies when you build software and understanding all those dependencies is what ultimately is going to help us have a more secure future. But for now, that's it. Keep checking and probably refreshed at Hunter's blog post. I will list as many links as I can down below in the description, which will then copy over to my forum as well. So you can kind of go in there if you want to have any discussions on it. But hey, this is a breaking, interesting event. If you're thinking about gaining a security, we could use some more people that are good at reverse engineering things and understanding this better. So hopefully this piques your interest a little a little bit and thanks.


Colloidal gold is something for an IQ boosting effect. It kind of relaxes your muscles a little bit. It has a lot of anti-aging effects. It's very, very good for the skin. I think it can penetrate the skin, but it's just very good for the skin overall. A lot of people have found my channel because of Oramus or have seen me post about Oramus. And I kind of find Oramus to be the more effective version of this. And I really like colloidal gold. It's phenomenal. I've always had really good effects on it. For some reason, the dream effects have been crazy on both of these, almost like too much for a little bit there. Streamly, vivid, long lasting dreams. There's almost like longer than any movie that I've ever seen. Every time I go back to sleep, I'm into dreams.


Hello everybody, my name is John Hammond and in this video I want to do something new but not entirely new kind of wanted to go back to the channel's roots where I do a longer video Series on one specific kind of technology or programming language And it's a tutorial series and learning thing rather than just a capture the flag video right up like I've kind of doing lately So this video I want to kick start the Windows PowerShell video series tutorial. I'm really excited about it But before we get started. I want to talk to you guys about CircleCI So you probably heard the terms continuous integration Continuous delivery and continuous deployment and that is what CircleCI is all about CircleCI offers an awesome platform that lets you automate your development of any software code and projects that you're working on in a Managed safe and test environment. They're working with some big players They have Facebook Spotify GoPro listed here And I'm sure there are plenty others and they offer a free trial version If you're interested in it, you can check out the cloud implementation or the server implementation And they are all about helping you develop the next best thing One thing that I especially wanted to mention was that they offer support for Windows They have incredible documentation and they offer some fantastic stuff Everything that they test within the CI pipeline is working on a virtual machine So it guarantees you have isolation of what you're working with it's using server core of Windows Server 2019 the latest stuff and they're using PowerShell That's a default shell, but you can obviously customize that to bash and old-school CMD.exe Offers a little bit of integration with Docker and it's just awesome The syntax is in a super simple YAML configuration file And it could not be easier to test and develop your software within a CI CD pipeline If that all sounds awesome to you, and I know it does Please go check them out on their website CircleCI.com Take a look through that documentation and see how easy it is to set up this pipeline And give them a shout, let them know that John Hammond sent you They're going to make your development process so much easier Okay, so let's dive into some PowerShell I'm working in a flat vanilla Windows 10 image I just updated it to the 1903 update And I need to put forth my usual disclaimer that I do at most of my videos In that I am not by any means a professional I'm not a Microsoft MVP by any stretch of the imagination So if I ever misspeak or if I ever relay some information to you that is wrong And some of you guys are veterans and rock stars And totally know, whoa John, you messed up on that Let me know, please say the word Constructive criticism in the comments We're all here to learn and that's what I'm all about So please do just let me know Hey John, you're being an idiot But let's get started This is going to be an introductory video It's meant to be an introduction video So it's going to be some light lifting What I first want to cover is just for one thing Opening up PowerShell So I hit the start button Or just the Windows key in my keyboard I'm kind of a keyboard guy I like to be on the command line So I would just type in PowerShell In that search bar there And you'll see it pops up Okay, Windows PowerShell is the app we can work through There are a couple others listed though You have Windows PowerShell ISE Which is there I don't know that acronym Integrated Something Environment Software Engineering It's a... Tech-setter is not the right word But it's an IDE Normally that integrated development environment For creating PowerShell code and PowerShell scripts We're going to get into a lot more PowerShell scripts Later on in the series For now, just baby stuff Trying to get our feet wet You also see Windows PowerShell With that parentheses x86 So what is that? That's that 32-bit rendition of Windows PowerShell Doesn't really matter which one we start The 64-bit one is just normally what we'll be working with Just that we're running a 64-bit PC Very, very likely And of course they have that 32-bit version of ISE You can just click on that Or hit Enter to go ahead and start it up To note, sometimes When we're going to be doing some things Later on in the series, hopefully We'll be working with commands That will need administrative privileges So if you ever want to open up PowerShell With administrative privileges You can obviously just right click on that And hit Run as Administrator What I like to do personally Is just CMD or PowerShell That's what this video series is on And I just hit Control-Shift-Enter on my keyboard And that will go ahead and ask me With Administrator privileges, do you want to run this? So here we go Windows PowerShell starts up And we are ready to rock With the Administrator PowerShell Note that it puts us in Windows System32 As our path If I were to open that up in simply my user account It puts me in Once it gets there, see Users-John So that's my user profile Or kind of my home directory equivalent in the Linux world And that's the prompt, right? That's just what's going to tell us That we are in a directory If you ever want to change directory You can move with that CD command We've seen in every other system shell Between Bash or CMD.exe, et cetera, et cetera You can CD to Period Which is that symbol for the current directory So see, our path had not changed If I were to CD Period, Period That'll move us up a directory That means our parent directory So now we're in see Users If I were to run a command like DIR What you would normally expect in the CMD.exe world You'll note that, okay, here's a directory listing Of the current directory that we're in So I can see John is an option there That's that folder I was just in And we can CD all the way to that And using that as a relative path That means from the directory I'm currently in We can move to the directory that is next to us Or in the same folder entry that we're in I slide into John and that works just fine for us And that was a relative path If I wanted to use an absolute path Or going from the very, very root of the file system Or any drive that I'm working with You could specify all of the path's subsequent So I'd say CD, C, colon, backslash, Users And I'll just go straight there And that's the absolute path Rather than going up with the two periods Using a relative path So that's that easy simple navigation Around the PowerShell command line You guys know the CD command But you'll notice that I ran the DIR command And some of you PowerShell purists Might already be angry at me Because the DIR command in PowerShell Is not so much a command And me even saying the words command When I'm discussing PowerShell Is already inherently wrong So PowerShell refers to everything That you end up typing at the prompt As a command let So if I were to say something wrong Like please subscribe Cool, yeah PowerShell will try and track that down But suddenly it'll get angry at me It'll get that black background and red text My screen gets bloody PowerShell yells at me That says, please subscribe is not recognized As the name of a command let And that's what PowerShell refers to Everything that it's going to be working with as Nor is it a function or a script file Or operable program Check the name Maybe you had just a stupid command You mistyped something Obviously that's not something we can run Please subscribe is not unfortunately Unfortunately it's not something we can just type Into PowerShell and know to do things automatically It's not a command let And that's what PowerShell wants to get across Is that everything that you type in that prompt Should be a command let And command let in PowerShell Normally have the syntax verb hyphen noun Normally there are two words or three words It depends on the lengths get the commands Let's command let's get more and more complicated Or you're doing more advanced stuff Maybe there's more to what that syntax And looks like But generally you can base The look of a command let base off of that Verb noun structure Verb hyphen noun And they like to use the kind of camel case structure However PowerShell is case insensitive It does not matter if you type in A command let with random capitalization Or that expected kind of camel case notation there So I'm using control L by the way to clear my screen CLS works just as fine just as well Clear might also do it Yep And the reason that CLS and clear work the same way And the same way that DIR works just as easily LS also works if you are kind of more of a Unix or Linux guy But those aren't the command let that I was just talking about The command let that we're looking at Or what we're actually running when we run DIR or LS Is actually get hyphen child item And that'll list out the children of the current directory But commands like DIR or the syntax here like DIR and LS Those are actually aliases for that get child item commandlet So if you want to see wow, okay It's an alias right, it's a nickname It's a convenient shorthand thing If you're more used to being a Linux guy working with bash And you just type in LS or RM or MOV And those commands are different from the usual CMD.exe, copy or MOV or DEL stuff like that If you want to see those aliases You can type in the command let get alias And that lists out a bunch of stuff, right We could go through this, we could take a look And there is a lot of information that's returned to us You see all of these kind of nicknames for commands CLS is in there just as we saw Cat might be used to that in the Linux world CP, copy, just as I was talking about And those all refer to copy item As the real commandlet that you're running there But those are just aliases that you're running Now notice that the output of these commandlets Is kind of in a interesting format Where they are kind of formatted like a table Where they're given kind of a header here With a piece of information that might vary For each thing that's returned to us The same thing that we saw when we ran DIR or LS Again, I should be running the full get child item commandlet That's more PowerShell pure I am kind of inherently a Linux guy though As you've seen probably see from other videos So if I end up typing those LS or Cat and said Please don't get angry at me I know I should be using the PowerShell side But that habit will grow Hopefully the same way yours will As we do more of this So I was just talking about though These kind of headers or the information title For each of the entries that's returned from us Or returned to us when we run a commandlet So what are those things? Well, this is where I wanted to talk about Kind of the fundamental shift and what happens When you're using PowerShell versus Bash Or CMD.exe or CShell or KShell or any other Any other system shell that you're probably used to PowerShell works with the input and output That's like coming out on the screen Not in the normal text based stream orientation You know on Linux you have standard output Standard input and you can pipe and redirect those As you would normally in CMD.exe PowerShell turns that on its head We're no longer working with just plain text But we're working with objects We're working with C sharp and dot net framework Kind of the back end of Windows objects Each of these is returned to us as some specific thing So when they go down the pipeline Or when we're working in PowerShell And we're passing objects from one command to another As you would normally in a pipe in Linux or in Bash You're piping objects into each other So let's try that Let's see if we were to run get child item Here are our things Here are the current directories and the users folder And let's say I were to pipe that And pipe is that vertical bar It's kind of above your enter key The shift rendition of the backslash key And we can pipe it to something else We can pipe it to, let's say, select And select object I think is the actual full title name So select object doesn't look like it did anything To our output, it didn't really specify anything at all That's because select object needs another argument Or some other representation of what to do with the data Or some configuration tweak as to how that program Or command line is going to operate So the way that we can do that is by supplying an argument An argument is meant to be the information and data That follows a command or command line The command line is the very, very first thing You type in here Get child item or select object Because we're piping them They look like they're kind of put together But they are, in fact, different commandlets Or commands The arguments or the parameters And I'll say those words often interchangeably Are the things that you supply after They are separated and tokenized by a different space That separates them apart So let's say I wanted to select the object name And that will return just John or public Because that is the name of what I'm actually receiving here Now that's something that we supplied to select object name As if that's what we're focusing on That's what we want to filter and zoom in on But we can actually supply more information To select object I'll clear the screen here Let's say we wanted to select hyphen first one Now that makes sense, right? I'm selecting the first object And I'm able to do that because of that hyphen first And that's a parameter and argument That I'm supplying a new value for First one And once I hit enter, now all I see is John Public isn't there anymore You could do that very, very same thing with last Or you could index And when you index, it's zero based So index zero actually refers to the very first entry John in this case And if we wanted to zoom in on that We can keep piping more and more To grab a specific property out that we wanted to We can select object name and just get John returned to us Note here that PowerShell is still gonna give us That kind of header Oh, sorry Ubuntu, my virtual machine is kind of Getting in the way there So we can drill down and figure out More and more properties or information Regarding that object If you wanted to get more regarding that You could actually check out get member Pipe it what you're working with Pipe the object that you're working with into get member And you can see more information As to what that object actually entails Talks about properties, which are like variables For that object Or methods, which are like functions That that object can perform Like deletes If we wanted to end up deleting that Okay, that's getting really annoying Sorry You could get files If you wanna see, okay Those are new files in there or move things to them Or just convert that to a string, et cetera, et cetera And you can see, okay These are the arguments that that method Or property might expect Or what kind of type that property or variable is Is it a Boolean value? Is it a string? Is it just text or stuff like that PowerShell is really, really awesome though Giving you kinds of types Or structures for information Like if you wanna have an IP address There's a PowerShell property In a format for an IP address You don't have to try and colluge that together With your own regular expressions, validation PowerShell will do that all for you And we'll play with that in the future For now, I just wanted to get these ideas across And that you can explore an object more and more And get more information out of it That you wouldn't have seen before So we didn't see a root or parent displayed In that regular output When we just simply wanted to grab The first item here Grab first one You don't see root or parent in there But if we wanted to select that Select object parent Then it will return it to us Note that select is the alias Or shorthand notion for select object So you can just use that If it's a little bit easier to type or quicker for you But when we're returning that information We're returning that in a very PowerShell-like way There's a different way to do that If you kind of just got the object That you wanna work with here Let's say we got just this directory Because we're selecting the first one In our current directory listing What we can do is actually encapsulate that In parentheses here An opening parentheses at the very start And a closing parentheses at the very end That will just say Now I'm gonna make this object As something that I can work with So we can run those methods Or access those properties And kind of a more programming-like way That you're used to Maybe more like a C-sharp style or syntax Rather than the PowerShell pipeline We could if we wanted to Write .name And we'll retrieve just that property Just that string here Remember if we were using that with select name It gives us that PowerShell header Maybe that's not the output that you want If you're trying to avoid that One option, one way you could go about that Is just using that kind of C-sharp style By wrapping the object that you're working with In parentheses and then using a .selector To work with things Or you could do methods, right? You could try and run some of those things That we saw in GetMember GetMember is fantastic GetMember is awesome Because it'll let you explore and track down Some of the properties or variables Or values or information that you're really looking for I think a lot of using PowerShell Is a very exploratory action Because you're just kind of discovering Where in this web of windows stuff Can I find the information I'm actually looking for That might be how you play And get to know PowerShell a little bit more So that is what I can think of For everything that I want to talk to you about And show you in this video We talked about what aliases are We talked about what a command Or a commandlet and arguments and parameters are And we took a look at what The GetMember commandlet would do for us Let's just explore an object But before I go, actually I thought of it now I want to show you how we can get help Or learn a little bit more about a specific commandlet If I were to run GetHelp on select object That'll give us some more information As to what we can do with this It's like the man page for PowerShell Right, the manual This is what we're actually working with Here's the commandlet there And it shows you some arguments and parameters You could pass through it Just as I talked about last Or first or others here And we'll explore these more in depth But for now I just want to show you Basically what GetHelp can do for you Because it'll give you even examples If you wanted to get more information This is a shot in the dark here But let's run GetHelp on GetHelp Okay, yep Cool, it will tell us how we can GetHelp You could actually ask for more information By using stuff like hyphen full Let's say, let's get the full help On that select object commandlet It tells us what it outputs What it inputs It even shows us more information On what all of these parameters do And it could give us some examples in some cases If you don't know the name of the commandlet That you're looking for Or what you particularly want to be investigating Or how you want to work with stuff You can try and GetHelp On things like printer And you can separate these by stars Or the asterisks Because it'll act like a wild card Anything surrounding the word printer on either end Or if we were to add one Only a star or asterisk at the very, very end It says everything that starts with the word printer Let's try with the asterisks around there That will showcase These are all the commandlets that you could use Maybe you wanted to run set printer Or get printer Or rename or add printer, etc Or you could do that with stuff like networking Right? GetHelpOnNetwork And then you can grab more information By finding those different commandlets Also PowerShell has a GetCommand commandlet That will show you everything that you could potentially do Within PowerShell And if you wanted to search or filter through that You absolutely could If you wanted to say GetCommand on printer With those asterisks and stars around it To track stuff down So that's how you could navigate Or explore or poke around in PowerShell We'll do more of that in the coming videos But I wanted to get this introductory stuff out first I wanted to get lay the foundation And kind of start tilling the ground for What we'll be doing in the later videos So thank you guys for watching I really, really hope you enjoyed this I'm super excited about the series I hope it'll be a lot of fun I hope it'll be much higher quality Than anything I've done before So please let me know Leave a like, comment on the video Hit that subscribe button, the bell thing I don't even know what it does It doesn't do anything Join the Discord server There's a lot of incredible people there Shout out to all of you guys I cannot thank you enough For all the support that you gave me And I'll see you in the next video


Built in 1828, the Providence Arcade, designed by Russell Warren and James Bucklin, was America's first indoor shopping mall. The Greek Revival Building boasts ionic columns at both entrances, plus granite, iron, and marble features. The arcade survived fires, hurricanes, demolition threats, and economic downturns over the years. Sadly, it was shuttered in 2008, prompting Providence Preservation Society to place it on the organization's 10 most endangered properties list. In 2012, Northeast Collaborative Architects unveiled a $7 million adaptive reuse project to help restore the landmark. The renovation created 48 microlofs and 17 micro-retail spaces in the arcade. The arcade brings much-needed affordable housing to Providence, while respecting the building's historic design and old-world features. The project proves that a historically significant building can be repurposed for live, work, and play.


Hello and welcome to the session. I am Arsha and I am going to help you with the following question that says, prove that cos 9x minus cos 5x upon sin 17x minus sin 3x is equal to minus sin 2x upon cos 10x. Let us now begin with the solution and the left hand side is equal to cos 9x minus cos 5x upon sin 17x minus sin 3x. Now simplify it and show that it is equal to the right hand side.  So applying these two formulas on the left hand side it can further be written as in the numerator applying the formula cos a minus cos b where a is minus and b is 5x we have in the numerator minus 2 sin a plus b is 9x plus 5x which is equal to 14x upon 2 gives 7x into sin a minus b upon 2. So 9x minus 5x is 4x upon 2 is 2x so we have sin 2x. Now in the denominator applying the formula sin a minus sin b and where a is 17x and b is 3x. So now applying the formula we have 2 sin and we have a minus b. So on subtracting 3x from 17x we get 14x and dividing 14x by 2 gives 7x into cos a plus b is 17 plus 3 into x is equal to 20x upon 2 that will give us 10x. Now on simplifying it further sin 7x cancels out 2 also cancels out and we have minus sin 2x upon cos 10x which is the right hand side of the given problem. Thus we have LHS is equal to the RHS or we can say that cos 9x minus cos 5x upon sin 17x minus sin 3x is equal to minus sin 2x upon cos 10x hence proved. That completes the solution hope you enjoyed it take care and have a good day.


In the world of technology throughout the early to mid-90s, there existed a version of Windows that was never released to the public. It was known as Windows Chicago, the final incarnation of an operating system that was meant to revolutionize the digital landscape. However, the project was abandoned under mysterious circumstances, leaving behind a dark and eerie legacy. The legend has it that those who managed to obtain a copy of Windows Chicago would experience strange and unsettling occurrences. It was said that the operating system was haunted, cursed with a malevolent spirit that would infiltrate the very core of a user's computer. The first signs of its haunting were subtle. Users reported hearing faint whispers emanating from their speakers, as if the system itself was trying to communicate. Whispers grew louder and more disturbing with each passing day, as if the spirit within Windows Chicago was desperately trying to convey a message. But it wasn't just the auditory hallucinations that plagued those unfortunate enough to possess Windows Chicago. The graphical interface would glitch and distort, displaying and settling images and distorting icons into grotesque forms. Some claimed to have seen shadowy figures lurking in the corners of their screens, their presence sending shivers down the spines of even the most seasoned computer users. As the haunting escalated, the operating system's behavior became increasingly erratic. Files would vanish without a trace, only to reappear in unexpected locations. Programs would launch on their own, and the system would freeze and crash without warning. Users were left feeling helpless and powerless, as their computers became vessels for this malevolent force. Rumors circulated that the spirit within Windows Chicago was that of a disgruntled programmer, whose life had been tragically cut short during the development process. It was said that this tormented soul sought revenge, using the abandoned operating system as a conduit to haunt those who dared to use it. As the legend of Windows Chicago grew, so did its allure. Some brave souls sought out the cursed operating system, driven by morbid curiosity. They were willing to risk their sanity and the stability of their machines in exchange for a glimpse into the darkness that lurked within. To this day, Windows Chicago remains a cautionary tale, a reminder of the dangers that lie in the unexplored realms of technology. The final version of Windows, forever trapped in the realm of the supernatural, serves as a chilling reminder that not all creations are meant to see the light of day. So, if you ever stumble upon a dusty old hard drive containing Windows Chicago, remember the stories that have been passed down through the digital realm. Proceed with caution, for you may unlock a world of horror that you can never escape.


Alrighty, so let's get these smaller construction updates off to a quick start. So I don't know what was quite going on in this tunnel, but they had some, I'm going to guess, either just a red light or infrared light. I don't know. You tell me down below. I don't want to make assumptions on that, but yeah, there was some, like, smaller work going down in the tunnel. It looks like not much went on here, but actually a lot went on. We're getting this transfer shed, sorry, the storage shed all set up, and the transfer track all set up. My guess is next week, they'll get the brake run installed slash storage area finished, which you see there. That's for the brake run, holding up the catwalks along the brake run track. That is not for the storage area. They have literally three flatbeds on site now for Fox Construction, so they are planning to work quick. Again, remember, the lift hill is supposed to be done by Labor Day. Run or started, something around there. Labor Day, the lift hill is supposed to be going up. That's insane. As you can see, that's a massive piece of brake run track right there, and that's probably going to be installed next week at some point. The storage lot is actually looking a lot less full, so they can fit a lot more objects in there. As they empty out a lot of this stuff really quickly, Fox is working really quickly. The crane operator and the iron workers are working so hard in every video I try to do. I try to give them a quick shout out because I don't even remember them working this quickly with Leviathan, so I don't know what it is, but the crew on this coaster is just phenomenal. Every day I'm there, there's more to film and more done, so it's awesome. It's working out for me. It's working out for the park. It's great. Another huge shout out goes to the truck drivers. The truck drivers delivering the track are awesome as well. The amount of information I've learned from them. Nothing secretive, nothing that they're not supposed to be saying. They don't know that stuff, but just the amount of information we've learned about their favorite projects to deliver, which projects they're going to be delivering this year, so I know there's no B&M's going anywhere in North America this year, just to China and Wonderland. Just cool information about that. Again, nothing secretive just about current year stuff, but it's just awesome. They're great people. Everyone on this project is phenomenal and has been treating us so well, which has helped us to be able to provide the best content for 2019. Anyways, on that note, I wanted to let you guys know that we're going to have some pretty up close and personal shots from one of the people on the construction site. The crane operator has told me that he's allowed to actually send us a bit of footage here. He's going to send us some exclusive footage that is not showing too much, but showing some great shots that we normally wouldn't be able to get. He's on Instagram, his name is aboriginal905, I'm going to link it down below, go check him out because he's going to be able to give us some exclusive shots like this that doesn't reveal too much, so nothing that we're not supposed to not see, so we won't get anything we're not supposed to see, but we'll get some cool shots that we wouldn't typically be able to get, which is super cool. Anyways, give the video a like, subscribe if you haven't, and stay tuned for more.


live moods here absolutely mortified that my MVP choice was Aaron Rogers for this year, considering his despicable performance in week one. But like we all expected, the MVP odds have moved significantly after week one, so let's take a look. So very early on in the Fandall Sportsbook, we had odds for MVP. Patrick Mahomes is the favorite at plus 500. Aaron Rogers behind him at plus 900. Josh Allen in third plus 1300. But I want to talk about Matthew Stafford at plus 1500 and more importantly, Kyler Murray ahead of the regular season was valued at plus 2300 for MVP. Let's check out how those odds moved post week one. Following week one, Patrick Mahomes is still the favorite at plus 600. But Matthew Stafford jumped from plus 1500 to plus 800 just behind Patrick Mahomes and Kyler Murray from plus 2300 to plus 900 odds. So I want to know after week one, where are you laying your money for MVP?


As you're being seated, please turn with me in your Bibles to the book of Exodus, the book of Exodus chapter 12 verses 21 through 28. Exodus chapter 12 verses 21 through 28. And as always, we desire to worship the Lord in the way that he desires to be worshiped, amen. And there's no better way to do that than to look into God's Word to consider how we ought to worship him. And this morning we'll be looking at Exodus chapter 12 verses 21 through 28. And it reads, that Moses called all the elders of Israel and said to them, go and select lambs for yourselves according to your clans and kill the Passover lamb.  None of you shall go out of the door of his house until the morning, for the Lord will pass through to strike the Egyptians. And when he sees the blood on the lentil and on the two doorposts, the Lord will pass over the door and will not allow the destroyer to enter your houses to strike you. You shall observe this right as a statute for you and for your sons forever. And when you come to the land that the Lord will give you as he has promised, you shall keep this service.  You shall say, it is the sacrifice of the Lord's Passover, for he passed over the houses of the people of Israel and Egypt, when he struck the Egyptians, but spared our houses and the people bowed their heads and worshiped. Then the people of Israel went and did so as the Lord had commanded Moses and Aaron, so they did. Amen. If we're not careful, it can be easy to lose sight here of who's righteous and who is unrighteous. And some people, they'll easily boil down their thoughts to this. They'll look at Israel and Egypt and they'll see Israel good, Egypt bad, Israel righteous, Egypt unrighteous.  Not by any means. And if they were inherently good, they wouldn't need a Passover lamb, would they? They wouldn't need the blood of a lamb painted on the doorposts and lentils of their doorways. Instead, what we see here is God has shown Israel favor. God has shown Israel great favor. There's nothing in Israel here to commend them to God, except God himself having chosen them. And in this favor, God has given Israel the Passover. They were to select a spotless lamb, a lamb that is without blemish and without spot. And the blood, this lamb was to be killed. The blood from this lamb was to be painted on the doorposts and on the top lentil of the doorway of each Israelite's house.  They were to keep their sandals on their feet, staff in hand. They were only to eat unleavened bread and bitter herbs with this lamb. They were planning for a hasty exit. And the way God fashioned this, he, he was, he was going to look and see the blood of this lamb on the doorway. And he said that he will see it and he will pass over their houses when he came in judgment upon the Egyptians. And you see that God, God, God is the one who delivers Israel here. If they had not put the blood of this lamb on the doorway, they would receive the same judgment that Egypt would receive from God's hand. Israel is not righteous. Israel has received favor and Israel has no boasting that they can do. They can't say that they planned this out. They can't say that there was something inherently good within them. In fact, even to this point, they have displayed the seeds of much of the rebellion that they will show going forward, even in the way that they treated Moses when he came to them. And God had told Israel in this command for the Passover that they were to remember this and they were to remember this in worship. He told them beforehand, before he delivered them, that they're going to celebrate this as a holiday for years to come. Every year they were commanded to remember the Lord and worship of his deliverance. Now think of Israel and think of Egypt and think of the Passover lamb. Which one of those is righteous? It's the Passover lamb that is righteous. Now consider us. Have we not received God's favor and Passover? We have Christ, the spotless lamb. Christ who is without blemish and without spot. And not only is Christ spotless and without blemish, he has every bit of righteousness, every bit of good deed that was required from him. He has done. His blood was shed for us on the cross.  He passed over you in judgment. And just like Israel, we have no boasting, no boasting at all that we can make before God. We cannot say that we are saved because we made a decision. We cannot say that we are saved and we have been passed over because we have prayed a prayer or because we have done some sort of religious right or some good idea or some method of planning that we have put together.  He delivered you and there was nothing that you can do to deliver yourself. And just like Israel was commanded to worship the Lord after being delivered in the Exodus, we are commanded to worship our Lord. In fact, one day we will worship in the innumerable multitude and what is the innumerable multitude in Revelation chapter 7? Who do they praise? They praise God and the Lamb, the Lamb who was slain, the Passover Lamb, our Savior Jesus Christ.  You would receive the judgment that the rest of the world would receive. So let's worship him and let's take joy in our union with him. Let's pray together. Father in heaven, we praise you for sending your Son, our Lord Jesus Christ, the final Passover Lamb, the real Passover Lamb, the one that Israel's lambs pointed to. And we praise you for him. We thank you that his blood was shed for us. May his praises be continually upon our lips. May we never cease to speak of him. May we never cease to rejoice in him. May we love him with all of our hearts. Who of us Lord would be here but for him? So we praise you and we ask you Lord to help us as we worship. May the thought of our Passover Lamb, the Lord Jesus Christ, well up such rejoicing in our hearts that it cannot be measured. And we praise you Lord. In Jesus' name we pray. Amen.


So what if you lived in more than one locality? If you lived in more than one locality during 2022, look up the table amount for each locality using the rules stated earlier. If there is no table for your locality, the table amount is considered to be zero. Multiply the table amount for each locality you lived in by a fraction. The numerator of the fraction is the number of days you lived in the locality during 2022 and the denominator is the total number of days in the year 365. If you lived in more than one locality in the same state and the local general sales tax rate was the same for each locality, enter the total for the pro-rated table amounts for each locality in that state online too. Otherwise, complete a separate worksheet for line two through six for each locality and enter the pro-rated table amount on line two of the applicable worksheet. Okay, let's take a look at an example. Similar kind of process. You lived in locality one from January 1st through August 31st, 2022, 243 days and in locality two from September 1st through December 31st, 2022, 122 days. The table amount for localities one is $100, the table amount for locality two is $150. You would figure the amount to enter on line two as follows. Note that this amount may not equal your local sales tax deduction, which is figured on line six of the worksheet. So same kind of process here. We've got the $100 and the $150 and we're breaking it out using our fraction, the number of days in locality one, 243 over divided by the 365. Line three. If you lived in California, check the no box if your combined state and local general sales tax rate is 7.25%. Otherwise, check the yes box and include on line three only the part of the combined rate that is more than 7.25%. If you lived in Nevada, Las Vegas is not a capital of Nevada. Check the no box in your combined state and local general sales tax rate is 6.85%. Otherwise, check the yes box and include on line three only the part of the combined rate that is more than 6.85%. So what if your local general sales tax rate changed during 2022? So now obviously, you're subject to the sales tax, they're trying to figure out the average sales tax. We're trying to use a table to get a general sales tax for the locality that you're in. And now the rate has changed in that locality. Again, hopefully the software that you're using, if it's a sophisticated software can help to figure out some of these changes and do some of this math for us. But you can see how kind of messy this got because remember what this came from? First, they decided to deduct the sales tax, which I think was or state taxes on the federal tax return, which I think was a mistake in the first place. But then they needed to still give the states leeway to tax however they wanted without favoring certain people that happen to mirror the tax system of the federal government, which is an income tax system. Government wasn't taxing to be Jesus out of them. So that means that they had to include the sales tax. And obviously the sales tax is quite messy to track because then you'd have to track all your actual sales. So then they said, well, do the average sales tax to make it easy. But now we could see even that isn't quite that easy because now you have all these different kind of scenarios that can come up. You have two localities, two states, and now of course they changed the sales tax rate and the state you're in. So if you check the yes box and your local general sales tax rate changed during 2022, figure the rate to enter online three as follows, multiply each tax rate for the period it was in effect by a fraction. So you have a similar kind of thing you'd have to do, right? Because now you've got a fraction of the year that it was at one rate versus another rate. So the numerator of the fraction is the number of days the rate was in effect during 2022. And the denominator is the total number of days in a year. That's 365. Enter the total of the prorated tax rates online three. So example, locality one imposed a 1% local general sales tax from January 1st through September 30th, 2022, 273 days. The rate increased to 1.75% for the period from October 1st through December 31st, 2022. That's 92 days. So you would enter 1.189 online three figured as follows. So they have the one rate times the fraction, which is 273 divided by 365. And then they've got the 1.75 times the fraction of the 92 days divided by the days in the year 365. You can see how this method is quite this kind of thing comes up quite often when you're doing different types of problems, this kind of ratio fractioning type of thing. So it's a good tool to keep in mind, although hopefully your tax software can help you to apply it in practice. So what if you lived in more than one locality in the same state during 2022? Complete a separate worksheet for line two through six for each locality in your state. If you lived in more than one locality in the same state during 2022 and each locality didn't have the same local general sales tax rate to figure the amount, enter online three of the worksheet for each locality in which you lived, accept a locality for which you use the 2022 optional local sales tax tables to figure your local general sales tax deduction, multiply the local general sales tax rate by a fraction. The numerator of the fraction is the number of days you lived in the locality during 2022. And the denominator is the total number of days in the year 365. So example, you lived in locality one from January 1st through August 31st, 2022, that's 243 days. And in locality two from September 1st through December 31st, 2022, that's 122 days. The local general sales tax rate for the locality one is 1%. The rate for locality two is 1.75%. So you would enter 0.666 online three for the locality one worksheet and 0.585. So same kind of thing. You've got one times the fraction of the year and so on. Line six, if you lived in more than one locality in the same state during 2022, you should have completed line one only on the first worksheet for that state and separate worksheets for lines two through six for any other locality within the state in which you lived during 2022. If you checked the yes box on line six of any of those worksheets, multiply line five of the worksheet by the amount that you entered on line one for that state on the first worksheet. Line seven, enter on line seven any state and local general sales tax paid on the following specified items. If you are completing more than one worksheet include the total for line seven on only one of the worksheet. So one, you've got a motor vehicle including a car, motorcycle, motor home, recreational vehicle, sport, utility vehicle, truck, van or off-road vehicle. Obviously a larger purchase like that means that you're going to have more sales tax applied to it than you may have if you didn't have a large person purchase in that year such as that. So also include any state and local general sales tax paid for a least motor vehicle. If the state sales tax rate on these items is higher than the general sales tax rate, only include the amount of tax you would have paid at the general sales tax rate to an aircraft or a boat, but only if the tax rate was the same as the general sales tax rate. Number three, a home including a mobile home or a prefabricated home or substantial addition to or major renovation of a home, but only if the tax rate was the same as the general sales tax rate and any of the following applies. One, your state or local imposes a general sales tax directly on the sale of a home or on the cost of a substantial addition or major renovation. So if you purchased a home or renovated the home then is it subject to sales tax? If it is you would think once again like the car and the motor vehicle and the boat those would be big you know purchases they have an impact on your sales tax. So B, you purchase the material to build a home or substantial addition or to perform a major renovation and pay the sales tax directly. C, under your state law your contractor is considered your agent in the construction of the home or substantial addition or the performance of a major renovation. The contractor must must state that the contractor is authorized to act in your name and must follow your directions on construction decisions and this case you will be considered to have purchased any item subject to a sales tax and and to have paid the sales tax directly. So in other words you could see kind of the problem there you hired some a contractor you hired someone to purchase to build your home they went and bought the stuff that means that they were subject to the sales tax right when they purchased the stuff to buy your home but you're the one that's purchasing them to ultimately to buy the home and if you purchased the materials directly then you'd get to deduct possibly the sales tax. So now now the question now you there should be a system or a way that you can deduct the sales tax even though the contractor is the one that bought the stuff because you're the one that told the contractor to buy the stuff so it gets kind of messy. So don't include sales taxes paid on items used in your trade or business so if you received a refund or state or local general sales tax in 2022 see refund of general sales taxes earlier. So if you if you paid for the sales tax in on your business you bought something for the business then you would expect that you would expect that you might get a deduction but you would be deducting the whole thing including the sales tax generally you would think or putting it on the books as a depreciable asset and then deducting the depreciation as applicable under the depreciation schedules all in one lump sum because because it would include the sales tax that you'd get to deduct the business expense but if you don't get the business expense then you can only possibly get a benefit for the portion of the purchase that was for sales tax. We talked about the whole refund situation earlier so I won't dive into that again.


Thank you for coming to my talk. I, that's very kind and generous of you to listen to me talk to you about things. My talk is called, don't forget the network, your app is slower than you think. I'm going to talk about, I guess, things that you probably haven't thought about yet about how people use your application and about ways that people using your application are having a worse time than you think that they are. I'm sorry. I don't really know of any good way to talk about this except by probably making you feel bad for your users. So brace yourselves and you'll be fine. Before I get to that, introduce myself, my name is Andre Arco. I'm indirect on almost all the things. That is an avatar of me that, now that I'm looking at it, that's one avatar old. I'm sorry. I'll get it fixed by the time I post the slides on speaker deck. I wrote a book called The Ruby Way, the third edition. I co-authored the third edition of The Ruby Way. It's actually pretty great. I learned Ruby from the very first edition of The Ruby Way and it was my favorite book except that I couldn't tell anyone to use it because it was about Ruby 1.8. And so I updated it and it covers Ruby 2.2 and 2.3 and if you buy it in a couple years you can use it to prop up your monitor and make it higher like I do with my copy of The Ruby Way second edition. I work at Cloud City Development. We do mobile and web application development from scratch but mostly what I do is join teams that need someone really senior to help with their Rails app or their front end app. I've done a lot of Ember stuff. And I guess if listening to this talk makes you feel like you could use someone to help you feel less bad, talk to me later. That is literally my job. I work on something else you may have heard of called Bundler. I mean I've worked on Bundler for a really long time but it's been a really great experience to work on open source and to kind of interact with every aspect of the Ruby community. People do things with Bundler that I would never in a million years have imagined that people do with Ruby and then I get to help them try to solve their problems. And if we've put a lot of effort into making it I guess easier, I don't know about easy, but easier to get started contributing to open source through Bundler than a lot of other open source projects. And if you're interested in contributing to open source, definitely talk to me later or tweet at me and I would love to help you start contributing to open source. The last thing that I spend time doing is called Ruby Together. Oh I'm even wearing a shirt. And Ruby Together is a non-profit trade association for Ruby people and companies that pays developers to work on Bundler and on Ruby gems so that you all can run Bundle install and it actually works. And without companies and people giving us money, RubyGems.org just wouldn't stay up and you wouldn't be able to bundle install because we have to work on it every week to keep it up. It's servers. It's software. It all breaks all the time and the only reason that we're able to keep it working now that there are so many people using Ruby and using Ruby gems is because companies like Stripe and Basecamp and New Relic and Airbnb are willing to give us money so that we can pay developers to make sure that it all works. We haven't let RubyGems.org go down in the last year, which is super great, but at the rate usage is going up, we need more people to give us money. If you are a manager or if you can talk to your manager about Ruby Together, that would be awesome. So the network and how your app is slower than you think. I guess routing is a thing that your app has, even if you didn't think that it does. I guess at one point there was a very widely shared article on RAP Genius's blog about how Heroku's router was a sham and everything was awful. I guess unfortunately, whether you're on Heroku or not, your app has a router and it's probably making things worse than you think they are. So let's talk about how that is and why that is and what you can do about it. So routing, what I mean is the part of your application's infrastructure that takes the request from the outside world and load balances it or forwards it or somehow gets it through your infrastructure until it finally reaches your Rails app server. And then your Rails app server does some stuff and tells New Relic, hey, this took 45 milliseconds and then it has to go back through Nginx or HAProxy or Nginx and HAProxy or whatever it is that you use back to the outside internet and then across the entire outside internet back to the user who was trying to find that thing out in the first place. So how exactly does this work? Like maybe you haven't thought about this. I totally don't blame you on your laptop. This is a non-issue, right? In development, this is routing, you talk to your app. It's great, actually. Unfortunately in production, you need more than one app server and people are coming from a lot of different places. And so this is just like a generic Rails app, not every Rails app will look like this, but almost every Rails app looks like this. You have some outside level load balancer, you have some inside level. Here's how we split requests up across all of the Unicorns or all of the Pumas or all of the whatever's. And every single one of those lines adds time to what your users see that you never saw while you were working on the program on your laptop. So question time. Raise your hand if you know how long your routing layer takes. I've asked this question in various different talks about eight times. I totally expected no one to raise their hands. I've literally had one person ever raise their hand. Eight talks, that's probably like closing in on a thousand people now. I once asked this question at a DevOps conference and zero people raised their hand. Like I don't expect you to know the answer to this question. But it's actually a really important question to ask because your end user's experience is 100% directly impacted by this. Like someone who goes to your production app and tries to use it experiences 100% of your routing layer twice for every request that they make. And like, is it a long time? Who knows? None of us. And then on top of that, not only is there this question of how long does it take in the perfect case from the time they make the request to the time your app is processing it and then from the time your app stops processing it to the time they get the response. None of that time shows up in your nice new relic graph that's how long this took. Zero of those milliseconds are included in that number. So you can look at the number and be like, yeah, we answer all our requests in, I don't know, what's a good RAILZ number? 250 milliseconds. I feel like that's a pretty common one.  How do you even find out? And then once you find out, what if too many requests come in at exactly the same time? Just having that routing layer where all of your requests come to one point and then they fan out across other points, this was like the main point of that rap genius article was that Heroku uses, and honestly, like there's nothing else that you can really do that makes sense. You just kind of randomly assign them. Like, well, here's one for you, here's one for you, here's one for you, here's one for you. And the problem is almost all RAILZ apps have like some requests that take 10 milliseconds and some requests that take like a second and a half. And when you're just throwing them out at random to every server that can possibly service them, unfortunately, statistically, it is very likely that you will end up with two horribly slow requests stacked up behind each other, and then the really fast requests start to stack up behind those. And it isn't very long before you like see a 30-second timeout and you're like, that makes no sense. That request, New Relic says that request takes 10 milliseconds. Why would it hit a 30-second Heroku timeout? And so it's not perfect, but you can at least start to get a little bit of visibility into this using a New Relic feature called QTracking, where you have your load balancer set a header that says, I got this request at this exact time. And then your app server says, well, I didn't get this request until this much later time. And then New Relic can add a thing to your graph that says, well, your requests are spending about this much time just sitting around waiting for a server to have availability to answer them. And that can be a completely separate thing that people don't measure that is adding 50% sometimes. I've seen that to the total user waiting request time. And it wasn't even measured. No one knew it was happening. Everyone was just like, that's weird. It seems to take a lot longer to get a response than New Relic says it takes to make the response. I wonder, you know, so, so ultimately what I'm trying to impress on all of you all is that the overall request time is not the number that Skylight New Relic pick a service. I don't really care tells you your request takes like that's a good number. Measure that number, pay attention to that number. If that number changes a lot, you want to know why that number changed a lot, because that's really important. But don't think that that number means that that's how long people are taking to, like, get the results of your app running. Don't, right, exactly. It's not the time that you measure that your app takes to run. And honestly, even that queuing track that I was talking about with New Relic, that requires that those clocks on the load balancer server and Ruby app server be, like, synchronized so precisely that they can measure milliseconds accurately.  And so what you want instead is a holistic measure of how long does it actually take to be a person on the internet, say, hey, Rails app, I want to know a thing, and then for the Rails app to say, okay, here's your thing, and then it arrives back. So the strategy that I have actually had that is really successful here is to deliberately create a Rails controller that returns an empty string, and then set up a service like RunScope or ThousandEyes or one of Pingdom even, like, there are services whose entire reason for existence is so that you can make requests to your own stuff from all over the world and find out how much delay your overall infrastructure adds to your application. And if you have a Rails app that returns an empty string, I guess, honestly, you could even do, like, a rack middleware that returns an empty string because New Relic measures the, like, Rails framework overhead, right? So you just want to know about all of the time up to the time it hits your Ruby app and all of the time after it comes out of your Ruby app. And so you can use one of these monitoring services to say, this is, like, the weather report for our users around the world. And honestly, like, I've worked at companies where, like, 60% of their traffic was the U.S., but for no particularly apparent reason, 35% of their traffic was from Brazil. And, like, then you really care a lot about network conditions changing and meaning that traffic to Brazil got a lot slower today. You should figure out why that happened and maybe think about setting up a CDN in Brazil. Because, like, if your traffic numbers are relevant to your business making money, they almost always are, this matters a huge amount. And right now, chances are good, like, nobody has any idea what they are. Are they bad? We don't know. Are they great? We don't know that either. Maybe they're great. Like, honestly, if all of you go home today and start monitoring these numbers and their fantastic numbers, I will be extremely happy for you. Based on past experience, unfortunately, they're probably not going to be that great. But knowing what they are is way better than having no idea that they exist. Very closely related to things taking longer than you think they do. Let's talk about servers. So I'm assuming that if you have things deployed, you have servers. This seems like a good bet. Let's talk about what's happening on your servers. Stuff, right? Like, you buy them and you rack them or you rent a fraction of one or, I don't know, you rent a fraction of a fraction of a virtual machine that is a fraction of a physical machine. It happens, right? Like, you end up with a piece of a computer and some stuff is happening on that computer. And even if you bought the computer yourself and racked it yourself, it's still running a ton of stuff and you have no idea what that stuff is. And I'm not going to tell you that you need to know what all of that stuff is, but I am going to tell you that it's really important to know how that stuff is impacting the thing that you do care about, which is your users and experience. And so a big thing that impacts this, whether you use Ruby or Python or Node or Go, you have a runtime for your application, right?  And what that means is your application sometimes isn't running while your program is running. And when that happens, your code isn't running, your instrumentation isn't running, and you have no idea how long that took. So if the garbage collector runs and your entire application just stops for a while, how do you know that that happened? How do you know how long it took? Like, you can't write. It's really hard to write code that measures time that your code wasn't allowed to run.  Okay, that's good, you can keep going. And Ruby, I guess, recently has added a thing called GCProfile that at least reports after the fact how long garbage collection took, which is awesome. But there are more reasons than just garbage collection that your code could end up paused. And so what you actually want is some way to say, I can tell that my code stopped working, stopped running, right? It was still working, but it stopped running for a second. And then it started running again, and how long was that? And there's this, I learned this trick from somebody who works at PaperTrail, Larry Marburger. And I think he got it from some of his colleagues at PaperTrail, it's super clever. What you do is you start a new thread with thread.new. This is like Ruby specific, but you can do this in any language. You start a new thread and then you say, what is the time? Sleep 1, what is the time?  And if your code stops running, Sleep 1 will take longer than one second. Little known fact. And so by monitoring how much wall clock time passes while a thread in your application is calling Sleep 1, you can accurately graph how much overhead the surrounding interpreter is adding to your overall execution time. And I have definitely seen this happen where you're running a Ruby program and you're like, that's weird. It seems kind of slow. And then you check the like, how long does it take to sleep for a second graph? And you're like, oh my God, we're spending 150 milliseconds doing something that's not running my program every second. And sometimes that means you have a memory leak. Sometimes that means that machine just got into a really bad, weird state. But at least then you know, and at least then you know that it's exactly that app server that's having this problem and all the other app servers are fine. Super useful. I guess very, very closely related to this is the virtual machine that you're, now that we're talking about interpreter lag, right? You're probably also running that interpreter inside a virtual computer. And Amazon, DigitalOcean, Paroku, Engineer Art, OpenStack, you're either running on a VM or you're running on a VM inside of a VM or maybe even if you use Docker, a VM inside of a VM inside of a VM. Hooray. And as you can imagine, this is yet another way to have weird times where your code doesn't run and you don't actually know it because your code literally couldn't run. And it's even worse than that because sometimes you'll end up with resource-specific contention, right? Like in a VM, what if you're on a VM and one of your co-tenants suddenly, like your co-tenant is running a memcache server? And so it's just like all the memory I.O. is going to your co-tenant. How do you even know if that's a problem? What if they're doing something really like storage heavy and that means you can't get I.O. anymore? And so there's like, at a minimum, the resources that you may care about are CPU and memory and disk, right? And network I.O., I guess, flavor number two of network of I.O. And you don't know when you get a shiny new empty VM, like maybe everything is great and maybe that machine has basically no network I.O. available or has basically no memory I.O. available because of co-tenants that you don't know exist. And so Netflix has a really clever way to check for this, I guess. They've written about it kind of at length and what they end up doing is Netflix spins up a new EC2 instance and then before deploying to it, they shove a giant pile of benchmark suites onto it and they run the benchmark suites and then they compare it to what they've decided is acceptable performance for that price point on EC2 and if it's below their acceptable benchmarks, they throw away that VM and get a new VM and then try the benchmark suite again and then throw that one away and then get a new one and eventually they hit an instance that meets their criteria and they said in their paper that they have observed almost an order of magnitude in difference in performance at the same price point because Amazon sells both the newest generation of hardware and two or three or sometimes even four generations old as the same VM, very large air quotes around same and then you have to deal with like co-tenancy issues where you may have a VM on a very old, very heavily contended physical machine, you may have a VM on a brand new, uncontended machine and like so for Netflix, they said that doing this and I'm probably gonna get these exact numbers wrong, I'm sorry, it's been a while since I looked at that paper, they saved something like a third of their overall server costs by doing this benchmarking and only accepting VMs that met their minimum criteria because of the amount, like they have a static amount of traffic that they need to serve but they got machines that were more capable to do it at the same price point and so they just needed to spin up less machines and pay Amazon less money. I guess, so you're probably not Netflix, this probably doesn't matter to you that much but it is at least something that you can be aware of when you're like man, 10 servers seem to be enough to serve this traffic last week, right?  So now that I've convinced you that you need to measure all of these things that you weren't measuring before, let's talk about metrics. I guess good point about the Ruby community, they're pretty good at measuring metrics. That's great. New Relic makes it really easy. You're gem-installed New Relic. Hooray, metrics. Metrics are really important. Tracking things, as we were just discussing, is really the only way to know what's happening. Without metrics, your production is kind of a black box and you're like, oh, things aren't as good as they were before, I don't know why or probably even how exactly because I wasn't able to measure, didn't know how to measure the things that matter. So I saw really, the first time that how important metrics are really hit home for me was in 2009 at GitHub's first code con. I saw a talk by Coda Hale called Metrics Metrics Everywhere and kind of the underlying point of his talk was that the reason that all of us have jobs and the reason that all of us write software is to deliver business value, whether that's to our bosses or to customers or to clients. Like most of the software exists for the purpose of delivering business value, especially if you're getting paid to write it. And if you can't measure it, you can't tell if that is what you are doing. So having said that, you're probably not super impressed by me telling you that metrics are important, right? So you do need to know what's going on. There's a catch. Once you have metrics, you have a tendency to become convinced that you now understand what is happening and I don't blame you. I do this too, right? It's like a human thing.  Now I understand it. Just like being able to see the speedometer does not tell you how the cars transmission and engine work. Being able to see a metric on your application does not tell you about how and why it is working. It just tells you something is very different than it was before and now you need to figure out what it is and why it is different. And a very common problem is that having metrics, having some visibility makes people think that they have total visibility and that just isn't how things work, unfortunately. So at the end of this little bit about metrics, this is probably going to be you instead. I'm going to talk about some ways that metrics actively mislead you and the biggest thing that causes this kind of misunderstanding driven by metrics is averages. When you have a lot of metric information, especially if you have a bunch of app servers, the easiest way to distill that down into something that you can quickly communicate is to take the average. A super good example of this is the way that New Relics dashboard, when you first open it, it's like here's a giant number. This is the average of all requests across all app servers. So you see those graphs, you see the numbers going up and down, you're like, great, now I know what's happening with my app, right? Unfortunately, no. Brains are really highly developed, carefully tuned pattern matchers. This is how humans can see Jesus in toast. This is how you can see an average and think, I know what that means.  There we go, normal distribution, right? You think, oh, the average, it's gonna be right at the top of that. This is often called a bell curve, and it's what happens when all of the inputs into the graph are generated by a random function. Tell me if you think that your app is a random function. I mean, maybe it feels like it's a random function. But your app is not actually a random function, and the practical upshot of that is that it doesn't look like this at all. This is a more realistic graph of what might be producing an average that's right at the zero point on that graph. To kind of drive home how wildly misleading averages can be, let's look at a bunch of real life graphs at the same time. So this is a whole bunch of different measured metrics from a real life thing. It was a MySQL benchmark, doesn't really matter what it is. From left to right, it's collecting the number over time. So near the left, it's like the things that were fast, and then as you go to the right, it's things that were slower and slower and slower during the benchmark. And so the small black vertical lines that you maybe can't see very well represent the average for that particular line. So to make it easier to see, I'm gonna line up all of the averages of this same graph. So not only do any, none of these look like a bell graph, right? Bell curve, not a single one of these looks like a bell curve. Worse than that, most of them have zero actual data points at the average line. It's really characteristic to have a very large number of points either clustered together in the fast zone or spread out over the long tail of slow things. But if you look down near the bottom, some of these lines don't even have a single result that's near the average line. And so if you're looking at New Relic, you might not even have any requests that take the amount of time that is the number of milliseconds that you're seeing in giant font on your dashboard. This is the problem of averages, right? Unless your metrics are being generated by a random function, the average is going to actively mislead anyone who sees it. There's a great quote about this from a tweet by a friend of mine, that's a faric. Problem with averages, right? On average, everyone's app is awesome. And so again, averages, I guess the single good thing that is really great about averages is that they can tell you that something changed. You can say, oh, my average was this before, but my average is this now. That's weird. The problem with averages is that they can't tell you what changed or how it changed. And it's actually possible to get that information out, and so I'm going to show you how to do that. So while averages can tip you off that something changed, so here's a graph of an average. And as you can see, things are taking about less than 100 milliseconds, but that effectively means there could be tons of things happening that take about 100 milliseconds, or there could be tons of things happening that take 10 milliseconds and tons of things happening that take three seconds. It's an average, so there's literally no way to know. One way to get around this is to graph the median rather than the average. The median is the number that was bigger than half of the numbers and smaller than half of the numbers. The great thing about the median is that you are sure that it actually happened, right? The average may or may not have ever actually happened, but the median definitely happened. And so if we add the median to this graph, you can see on the purple line, we now actually know more information than we did before. Most half of the values are actually very, very fast. It looks like around in the 10 millisecond range, maybe 20 milliseconds. So even though the average jumped all the way up to 150 milliseconds at one point, at least half of the requests were happening still equally quickly. They didn't slow down. That tells us that since most of the requests didn't slow down, this wasn't like an application-wide change, right?  We didn't see really, right? This wasn't a really a network switch problem where all of the traffic was impacted. The next thing you can do is graph other places. The median is the 50th percentile, right? Half is below, half is above. Start graphing the 95th percentile. 95% was below, 5% was above. Here you can see that the slowest 5% of requests got dramatically slower, more than 10% slower than, 10%, more than 10 times slower than the median. And that's what dragged the average up. Oftentimes even better than the 95th percentile is the 99th percentile. This gives you a good idea of what one, it's one out of 100, right? So this is actually a pretty good indicator of what the occasional slow request looks like, right? Well, I had to rescale the graph. And the slowest 1% of requests are now clearly the entire reason why the average tripled. The median stayed exactly the same. The median is now a flat line. And that slowest 1% is probably some single specific controller action that you now need to go find and figure out what exactly happened to that specific single thing. And so just by graphing the percentiles rather than the average, we can immediately rule out about half of the possible problems that made our average slower. And it works the other way around too. If you look at the graph of the 99th percentile and it isn't dramatically different even though your average is higher, then you know not to look for a single controller action. You know to look for a systemic problem. Aggregate graphs, this is another really common thing where aggregation is a fancy way to say, I got the many versions of this metric from many servers and then I averaged them. So here, again, here's an average graph. This one happens to be taken from the actual Bundler API. And it is a graph of the trick that I mentioned where you call sleep 1 and then you see how long it took. So the number that we're tracking here is milliseconds and it went from taking two milliseconds to, you know, one second plus two milliseconds to taking one second plus five milliseconds.  We don't know, this is an average. And you can improve this with breakout graphs. If you are collecting a number from 25 machines, put 25 lines on your graph instead of one that will mislead you about what all of the different machines are doing. Here is a breakout graph of the same data. Kind of like I was mentioning before, with the breakout graph, we can see, holy crap, this had to be rescaled. One of the machines started taking 35 milliseconds per second to sleep. But all of the other machines were basically fine. And so we wound up resolving this issue by just killing the one dino that was having trouble and restarting it as a fresh dino. But we didn't have to nuke all of our dinos. We didn't have to write this narrow down the problem immediately just from having a breakout graph. So, do it. Visualize your data. Here is an example of why visualizing your data is so, so, so important. These are some different data sets. Each orange dot is a single entry on that data set. Can anyone guess what the blue line is? So, that average. Average. Average. Average. It's actually even worse than that. The average of why. It's actually even worse than that. The average of why is exactly the same on every crack. It's actually even worse than that. All four data sets have the same average of x average of y variance of x variance of y correlation of x and correlation of y and the same linear regression. Actually graph your data and then look at it because the numbers of the averages and the variances and the correlations and the linear regressions don't contain any of the information about what is different in those graphs. One final note. A lot of people that even talk to me about how awful averages are, I then am like, oh, hey, so how do your alerts work? And a lot of people have alerts that are set up to only talk to them after the average is bad. And as you can maybe guess, by the time the average is bad, it is too late. Definitely break out your alerts as well as your graphs. Right? You want to know when the first server went down, not when the average of the servers is a down server. Right. So ultimately, I really just wanted to let you guys know that the network is a part of your application. Most people don't think about it because they don't have to interact with it in their day-to-day development on their own local machine. And after you have deployed your application, it is really user experience that matters not how many milliseconds your Ruby app spends running code. That's it? Sure. So the question was if you don't alert on averages, how do you prevent continuously alerting, getting alert fatigue and then not noticing that something actually bad happened? And the question included a note that there is no silver bullet for this. And unfortunately, the answer is there is no silver bullet for this. So the best plan that I have ever seen from the best operations people that I have worked with is figure out what the baseline of your system when it's functioning is and alert when your system is not that. That means figuring out how many requests you're successfully serving per minute and alerting when it deviates from that more than 50%. Figuring out when it's normal to have that deviate and then not alerting on that. And it's actually a ton of work because every single application has a completely different norm. Some Rails applications, they serve like 50 requests a minute and that runs their entire profitable business. Some Rails applications serve hundreds of thousands of requests a minute and they're not profitable yet. You need to figure out how it is that your metrics look when your company is functioning both software-wise and company-wise. And you need to kind of alert when it's not the thing that gives you the indicator that things are okay. That's really the best advice that I have for you. Five seconds. Any more questions? Alright, I'm happy to talk about this stuff later.


You know, like some people were just in back again, sir. The Democratic Party was the party that spread tariffs, retreated. That's true. It was the Republicans that put in this new holy tariff, spread the red, the repression worldwide. And today, the Republican Party is the party that is fighting to maintain both tariffs, retreating, and the leadership of the other party at once. Yeah, I'll give it to somebody, to me. No, I'll just give it back. Two rotations. Money! Can we sit down here? Yes, sir. Mr. President, could you lead forward just a bit? Getting a shadow on your face. Oh, I...