Okay, it's 2.20, I'll get started. So my name's Dan. Just before I start my actual talk, I wanted to promote this, that at the end of the day today we're going to have a BOF, SefOperators BOF. So please go to that pad.sef.com link. I don't know, it's faster to just memorize it or scan the QR code, but just go there and if you have some questions or topics you'd like to discuss, we can all hang out at the end of the day and compare notes. But yeah, this is my talk. So I'm going to talk about some lessons learned operating Sef for science. My name's Dan, like I said. I work for a company called Kleiso, previously I worked at CERN, and I'm on the Sef upstream in the Sef Foundation and the Sef Executive Council. Just a little bit more about me. So OpenStack summits and OpenInfo summits are like my favorite conference because this is my hometown. It was a chance for me to come back and visit, but this is also where I think it might have been the first place in 2015 that we got to present the cool work that we were doing at CERN with Sef. So we presented like a petabyte scale Sef back in 2015, and then we presented some work with HPC that we were doing at the OpenStack summit in 2018, or maybe it was already OpenInfo, no OpenStack summit, there we go. And this talk that I'm doing today is actually part two of a talk series. So I'm kind of talking about what happened. We had a decade of Sef at CERN, and I'm kind of going through the different things that we learned. The first one was actually at Sef Day in New York in February, and then I repeated that at Sef Lacan. But this was like at a time of sort of transition, I was really busy, so I couldn't even actually write the talk. So I had chat GPT write the talk for me, and it did a really great job, actually. Yeah, so that's my new best friend. At the end of the talk, it gave me like, I asked chat GPT what were my key takeaways operating Sef at CERN for 10 years, and it gave me a very good overview of everything that I learned. It's very accurate, actually, you know, planning, monitoring, understanding resilience, understanding how to test Sef. This is all very good stuff. I started to wonder that maybe I'm out of a job, actually, now that I'm trying to enter the business world. But then I remembered something, maybe this is a quick side quest. Maybe you've heard about Noel's Law of Media Accuracy. Have people heard about this before, like everything you read in the newspaper, okay, maybe you haven't heard of the newspapers, but that's like the web before.  But I think this is the same for chat GPT, like everything that is generated by chat GPT is perfect until it's something that you know very deeply, right. So I wanted to actually test it on Sef. So I asked Sef, I asked chat GPT to generate me a sef.conf optimized for performance, and this is GPT-4, right, you have to pay for this. So here's what it looks like. It's a properly formatted Sef.conf file, that's already impressive I guess. But then it gets a bit nonsense, I guess this is not a laser pointer, is it? The first and foremost we want a fast messenger, it tells you, okay, please don't use anything but this. It's the default and like, oops, I went too quick, I mean this is not, this is already getting to be nonsense, right, so this is the default. OST memory target auto-tune equals true, okay, this is also the default where it matters. Then it starts getting to OST, OST optimizations. And it tells you some things that you should not be manipulating. It tells you to manipulate some file store settings which are obsolete. It gets into some OST settings which are definitely not interesting. It tells you to complain about ops after 10 milliseconds, which is going to cause a disaster in your cluster. Some other stuff. Actually OST max back fills too, that's not a bad one.  Some let's make sure we're using the best possible allocator, I mean this is all nonsense, you should not be changing this, then you auto-tune it. Okay, then my favorite thing at the end is to optimize the mon.  So I put some, there's some risk that if you put some config on a presentation somewhere someone's going to use it, so I'm trying to put some warnings. This is very dangerous, like don't use this, so whatever. We still all have a job, I'm very happy, okay. So but the recap from part one of that whole talk was that the reasons why we all choose CEPF have remained the same, it's all of these adjectives, reliable, modern, integrated, flexible, open, enterprise quality storage for a good price. I think that's what we can agree. And over the decade in the first part of the talk, which you can watch on YouTube, I guess if you're interested, I talked about how we improved resiliency and performance, all of the scalability inside the OSDs, but also cluster-wide improved, and then we also improved all the daily operations stuff. So now operating CEPF gets more boring, thanks to all the contributions from different authors and organizations. So then I get into this talk, which I want to talk about some very specific, best practices for the different phases of a CEPF cluster. So planning, implementing, operating, and debugging CEPF. So this is just my ideas about how you plan for a CEPF cluster. This is very high level stuff, okay? But maybe some people are not familiar with the different orientations of CEPF. So this is just my very high level summary. Okay, so the first question, and I think Steve Oshel pointed this out at the beginning. You need to understand what's your main use case? Are you performance or are you capacity oriented? And that's going to push you towards different flavors of hardware. Performance orientation, okay? Always use all flash and can be these days, and six to eight gigabytes of RAM per OSD is the target that you need to aim for. If you're capacity oriented, use large HDDs and aim for, okay, there's different recommendations about percentages of metadata, but generally around 150, let's say, 150 to 200 gigs of NVMe flash per HDD for the Rocks TV is good and aim for four gigabytes. And you do, I'm glad to hear that this is now generally recommended, but you do want to for CEPFS or S3 use cases use all flash for the bucket indices or the metadata pools. Now, buying hardware for CEPFS MDSs, this is where it's hard to give general recommendations. The rules of thumb in CEPFS are to start with a single MDS. Don't have the idea that you should start with multiple MDSs, because it's actually very complex to run multi-MDSs. Scale out metadata requires lots of experience, I would say, with your workload. You should also target something like up to 128 gigabytes as a memory target for an MDS. Beyond that, I think that's when you should start looking at multiple MDSs. But also when you're buying your machines, 128 gigs would be the memory target, but plan for at least 50% or even double the amount of round, just in case the MDS overshoots what's happened, which happens quite often. The other demons are less demanding, so I think people understand what to do there. Architecturally, you should consider early on, if you're just starting with CEPFS, you should consider early on if you're going to use, you probably have the idea to have one CEPFS cluster, but maybe you should already think about multiple CEPFS clusters from the beginning. Because a single cluster can be a single point of failure. My friend and colleague, Enrico, described, he hinted at a little failure scenario that I'll actually elaborate on in later in my talk. It could also be a bottleneck, right? So why a single point of failure? There's risks during upgrades. There's possibility of bugs, there's possibility of infrastructure failures taking down the whole cluster, and also operator error happens quite often. Well, not that often, but it happens. And it can also be a bottleneck. It can be a bottleneck because if you have multi, I say here like, let's say different users, suppose you have different use cases of S3 and CEPFS or block storage and FS and S3 and all of these different applications in the same cluster. There's no real pool to pool fairness in CEPFS. So you could have one of those applications overloading and impacting the other users. So it's better to separate these things in different clusters. Also, if you start with one cluster and think I'll solve it in like three years when I grow, like migrating data, that can be very painful. This can take, it's definitely not like live transparent to the end users. And it can be very labor intensive and take very time consuming. Also, the other thing which is like these type one decision you have to decide early on is you should decide if you're doing replication or erasure coding because you can't change this on the fly. You can resize replication pools, although normally we just do three replicas. You just go with that. But you can't convert replicated to erasure coded pools. And also you can't reshape erasure coded pools. Some people think you can go from like four plus two to eight plus three or something, you can't do this on the fly. This would be nice if CEPFS could do this transparently. There are tricks to do it, but not generally an easy thing to do. And the user recommendations are like three extra application or four plus two and then two plus two, maybe eight plus three. Wider ECs, okay, they can be interesting for some use cases, but performance can really start to suffer. So now you're implementing, you bought your hardware, you want to install CEPFS. I'm not going to go into detail into installing CEPFS, but basically everyone here that's maybe using the community release of CEPFS, you should just use CEPFSADM or ROOK, full stop. Anything else is not supported. I would love to support other ways to install CEPFS, but these days it's, these are the two supported options. Once installed, okay, make sure you do some burn in validation and get to know your CEPFS cluster. Start to start with a single rados bench on a test pool, have plenty of PG's, make everything set up so that you're using your disks. And you should definitely be able to saturate the client network wherever you're running that test from. Then coordinate some, like several rados benches in parallel and make sure that you can saturate the OSD hosts. I don't say like what I mean by OSD hosts because maybe you're going to saturate some device limits first or network limits on the OSD host, but you should be able to saturate something other than CEPFS. If you can't, it means I think something is broken in your cluster. This happens often. Also, don't just run like two minute tests and then go home and say job done, like run something over the weekend, run something one week long so you can really see that your components are stable in your cluster. And then at the end of this, make sure you just delete everything and start again and then go into production. Also, don't forget to do some kind of validation of availability, practice rebooting the hosts, practice understanding what happens when you're recovering hosts or you have multiple hosts down and PDs go unavailable. Just generally like don't be shy and try to break your cluster. This is a very good practice to understand actually how CEPF recovers from failures. Yeah, so now something about configuring CEPF, you have it installed. I think maybe you want to start already playing with the configurations. In this year, 2023, CEPF's been around a long time. And as like AI systems have indexed the web, I mean Google indexed web too. There's thousands of recommended tunings online and they're almost all out of date, okay? Almost everything's out of date. If you're using the latest version of CEPF, the defaults are very, very optimal, they're pretty ideal. So, of course, anyway, I have some suggestions which are not the defaults. But some of those, I'll go through them kind of quickly. It's too bad I don't have a pointer. So, upmat balancer should be on by default, but it doesn't do a very tight squeezing of balancing the PG's. So you can configure that by setting this max deviation to one or two. This will squeeze the utilization to be very uniform across your cluster. The max backfields and max scrubs is not high enough. These are defaulting to one. These should be higher. Especially on Erasure-coded pools. This one's not enabled by default, and I don't really know why, but if you're getting inconsistent, if you start to get inconsistent PG's on your pools, and it's due to, and you should look into it. And if you find that it's just due to weak rights, so it's reading data out, and actually if you write it back, the disk is just fine. You should probably enable OSD scrub auto repair, save some headache. If you have large high capacity HDD nodes, you should use this, oops. You should use this option monOSD down, out, sub tree limit equal host. This will prevent if a whole host goes down at once. That scenario, a whole host go down at once.  If that's the experience that you want from stuff, this is the option you use. After your pools are created, PG counts are correct. I mean, people in this room should still try to understand PG's. I don't like this idea that PG's an abstract concept. Try to understand the number of PG's that you should have on an OSD. And once the PG counts are good, consider disabling the PG autoscaler. It's not that pleasant to have it resplit PG's, merge PG's in production when you're running a real system. This is not recommended by me, at least. There's some nice flags, no delete, no size change that you could set on pools that you want to prevent, especially operator error from changing things by accident. And then you can also set no PG change to really make sure that you don't start any splitting or merging. Yeah, so I mentioned about PG's. You can just use SEPH OSD DF and SEPH OSD DF tree. Count the PG's per OSD, make sure it's around 100. I think in the near future, we'll probably start considering increasing that recommendation. You can get a lot more performance now, especially on NVME with more PG's. Mark Nelson, who I'm working with now at Kleiso, he's working on this as a general recommendation at the moment. Some things about the MDS. So this normally requires a lot more attention tuning. It's very workload dependent. You can start by increasing the default debug level to 2 slash 5. Then you'll get a kind of heartbeat every one second from the MDS. It prints the memory, the status of the inodes cached and the status of the memory usage, and you can then see, okay, this workload is like, you can start to get a feeling for how the MDS is working. There's a nice option. If you're having MDSs using lots of memory and then they restart and they take a long time to reload all the cached inodes. There's this nice option, MDS OFT prefetch dearfrags equals false. And this thing will let the MDS restart very quickly without having to preload all those inodes. If you're getting overloaded MDSs and lots of warnings about cache recall and stuff, there are some options with these names, cache trim threshold and recall, various recall options that you should look into. There's good documentation on those, although it takes a little bit of experimentation, generally just increase those numbers a little bit. That's what I would do. And if you have misbehaving clients and your environment allows you to evict clients, you should maybe consider even evicting clients automatically after maybe say, on some clusters, this might maybe 15 minutes is a good number for this. If a client doesn't revoke its caps for 15 minutes, then you can evict it. You also want to configure your hardware. This is my favorite thing to talk about this year, which is that modern devices have a media cache on them and that media cache is enabled by default. And this is the experience on a Cef cluster that we had at CERN. We installed some new hardware and the latency, this is a small probe running four kilobytes every minute, four kilobytes objects of the cluster and it was suddenly going up into like tens of milliseconds territory and then the users were also complaining about the performance and then we found that actually you need to, for BlueStore to be able to access modern HDDs efficiently, you need to disable this cache. So I've shown this plot before, but this is the impact of disabling that cache on their latency. So this is now a general recommendation at the Cef.com docs in the hardware recommendations section. So there's some recommendations there about how to benchmark your devices and how to know if you want to disable that cache. OK, that's configuring Cef. We have some advice, let's say, for operating Cef. So if you've planned and validated and configured stuff well, then these days operating Cef is quite boring. It should be quite uneventful. Device failure should be transparent and replacing devices should be as expected. All the tooling is quite good for that now. My preference is to, in that previous plot, I mentioned that's like a Rados bench. I just like to run just like the kind of like sysadmins thing on the side, just have one probe running independent of your Cef systems, just so you can have an external metric about the latency or performance of the cluster. Since for 10 years, I've been running this on some clusters. Just a 10 second write some small objects with one thread and store it in a database somewhere. You can do that similarly with an RBD bench or some S3 commands, if that's your application. Also, pay attention to have whatever you're using, page or duty or whatever monitoring system you're using, pay attention to health warns, don't ignore them. Like if you're getting slow requests in your clusters, this is not normal. If you're getting them all the time, it means something's wrong and something to be investigated. They can escalate in severity and then you lead to more pain, especially for CefFS. If you're having warnings from the MDS or warnings from CefFS clients about different types of issues, those should be investigated and understood. If you ignore those, those can lead to outages. That's happened to us in the past. Another kind of practice, I don't know if people do this, but if you're, I'm a CLI guy, are people CLI or dashboard people, CLI?  Make sure you have a watch Cef status in the corner, that you can watch what happens immediately and maybe undo what you did. You don't want to like do a change and then see, oh, what happened 15 minutes later? That's kind of weird. So watch Cef status is fundamental. There's a couple of other commands like that. There's one that I think, maybe people don't know about it. I didn't put on the slide, but over lunch, someone was asking, how do you monitor CefFS? That you should, on an MDS machine, there's a command called CefDemonPerf, so D-A-E-M-O-N-P-E-R-F. If you run that like you run CefDemon, people will know how do you use CefDemon to interact with any kind of CefDemon. You can run CefDemonPerf and it gives you like a every second kind of top like output that you can get from MDS. That's useful. You should also, if you'd like to dig deep into this topic, you should probably look into CefPerfDump and CefReport and start instrumenting some of the metrics that you get from there into your custom dashboards. Here's one of my or a couple of my favorites from over the years. This plot here, these set of four plots here were related to OST maps. So from the CefReport outputs of the square on the right here, from the CefReport output, you can understand what the cluster is doing with OST maps, how many are outstanding because an OST map is something you don't want to accumulate by default Cef will keep around 750 OST maps and it's like in the Mons database and then distributed across the cluster. Whenever there's backfilling or other things going on where the cluster's degraded or doing some work, then they just accumulate. In the past, there were scenarios where you could accumulate millions of OST maps which was getting, leading toward a disaster. So it's nice to monitor that kind of information. So in this one, yeah. So I'm showing these four things because you see that there were maps accumulating up to a few thousand, but that's because there were degraded objects and placement groups recovering or backfilling. So this is all normal, but it's something fun to monitor to start to understand Cef. And here's another one. This is a CefFS set of plots. So there's in CefPerfDump of an MDS it tells you lots of latency metrics and one nice metric that you can monitor to get the, to understand the performance of an MDS is the create latency. So it tells you how long it takes to create the inodes that it's creating. On one of our systems, we had, so we have a Grafana plot that's like hard-coded not to exceed 10 milliseconds, but we noticed that it was above a second actually for a while, oops, going back. So there was a problem here where creating inodes was taking more than one second. This is a kind of a known issue related to Kubernetes clients where after recovery, if a Cef client had a file open and that file gets deleted that it will try to like reopen that inode even though it doesn't exist. And if you have a multi-MDS system it will say I don't have that inode try the other MDS and then it just like iteratedly goes through all the MDSs in a loop but none of the MDSs say you've tried them all like you can give up now. So it just keeps going in an infinite loop. Now that happens, it's fixed now in latest Quincy and probably latest Pacific, but this is what was the issue here. So you can see like the latency show that there was a big problem. There's high CPU usage on the MDSs and higher number of, this is HCR slash S that's the client requests to the MDS. So this is also showing something is wrong. So those are some of the metrics, some of my favorites. So one of the like FAQs and Cef if your community user is like, when do you upgrade? So of course, if you're supported by a vendor like follow their advice but otherwise there's no simple answer. I think by now people have understood that you should not upgrade on day zero to avoid regressions. Upstream we do all of the kind of testing that we can but it still doesn't catch everything. But you also definitely don't want to lag too far behind. I think going beyond more than a year of the major releases is you start to, you start to run into risks of like, if your community supported you need to ask for help and the majority of the community users probably forget the issues that happened with those old releases and the upgrade. There's always little nuances. So you should try not to try to stay a little bit trendy. But yeah, so if you are self-supporting definitely follow the community mailing list. There's a Slack now that's quite active with actually lots of channels, that's cool. So you can dig into different topics and look for news from early adopters. Also try to test in pre-production environments and share the results with the community. We're now, Yuri's doing a lot of work to publish release candidates. There is a release candidate out for Reef now. So if you do have the opportunity to test that kind of pre-releases and share the results that's really valuable for everyone in the community. And yeah, also understand your rollback options. It's normally not possible, but sometimes for minor releases it can be possible. Yeah, so now the last part would be about debugging stuff. So what to do when it fails. It's totally rock solid, but it's not infallible. Buggers do happen. It's weird, like now that we moved, okay, it's not even that recent, but we moved to a containerized world. I have to talk about log files. Like I guess I'm too old. I don't even know how to access log files properly in containerized setups, but it's essential to be able to access your logs and know how to manipulate the debug levels of Seth Demons if you want to debug stuff yourself. So know how to access the logs, know how to manipulate the levels.  I'm embarrassed to mention this, but like when you're under pressure, if things are down, you don't want to be looking for where the log files are and how to access them. Where is it journal D or is it in var log Seth? Like you better know that beforehand. The default log levels are not, if there's a problem, the defaults are not verbose at all. They won't tell you what's going on. So you'll want to, unless it's like a major failure, okay, then it's gonna give some abort or some assert trace back. But if it's just some misbehavior, then these are good places to start. So debug MS equals one. What that does is it just turns on the lowest level debugging for the messenger. And with that, you can see the operations between demons. You can see from OSC to OSD or OSC to MDS or clients to the MDS. So there you can already see, for example, if you have an MDS overloaded, debug equals MS, you turn that on for a few seconds, turn it off and you can see, oh, this client is like doing get, it's it's statting a file like in a loop. And you can learn immediately that you should evict a client. With similarity OSD gets interesting around debug level 10 MDS gets interesting on debug level seven. Yeah, so as I mentioned, you want to kind of turn this on for 10 seconds. It's very verbose, turn it on for 10 seconds, switch it off and then start to have a look. But what happens when it really fails? So very rarely things can go very, very wrong. In the past, in the past decade, I'm only really aware, there might have been other cases, but I'm aware of two scenarios that were like really bad where workarounds couldn't, there were no workarounds to solve these two issues. In the self community, we started calling these the bug of the year. Okay, so the first one was just at the start of the pandemic, it was this, it was just before we all got to go work from home. We had a major outage at CERN actually, and there were like rumors about this bug. It was, there was already a bug tracker about this, but nobody understood what was going on. And then it hit us at CERN and Rico mentioned, we had a cluster wide outage, like all the OSDs all stopped saying that the OSD map was corrupted. This was a disaster, right? We had the head of the department and lots of important people in the office saying like, how's it going? We're kind of stressed out. We recovered the cluster by, you can like extract old OSD maps from the OSDs themselves, even when they're not running or from them on when they're not running, and you can inject them back into the backing store and then get the, you can get the OSDs to boot this way. And that's how we managed to recover it, but that was not, so we just saw, you know, these like naive guys, we had the OSDs not running CRC errors, so we just put them back into, and it started, but then for a week, we tried to understand what had happened and through various levels of debugging, and you can watch the YouTube video, it's kind of interesting, I think. Okay, I mean, I shouldn't say that, but I was, I gave the talk, but it was a fun experience. We found that there were four flipped bits and those were caused by a bug in LZ4 that in a very, very extremely, like one in 100 trillion scenarios, like if you have non-contiguous memory, it would corrupt the thing that it was compressing. So the solution was, so CEP is now resilient to this and also LZ4 is fixed in the operating system. So that was 2020, and then just in the last year, we've had another scenario where lots of users were reporting OSDs just bloating up in memory, using hundreds of gigs. People were throwing swap, people were compressing memory, like no one knew what was going on for several months. Even restarting the OSDs, they would like right away start up and use all the memory again. So this turned out to be a bug in PG splitting and merging. So the PG's keep a log, it's called like the duplicate operation log, PG dupes. And there was like a bug in splitting and merging that was causing these logs to get like, normally they were ordered sort of alphabetically and you could just pick the first, like you want to trim a thousand of them. That's nice. You could just trim the first thousand of them, but because it violated this ordering, you were never trimming them anymore. So they just accumulated the solution. Okay, this took some, this took a few iterations to find the stable fix. It's now fixed in all the latest versions. This shouldn't happen. There's an offline for anyone. I hope no one is still suffering from this, but there is an offline trim command that can fix your OSDs if they won't start. And then now online, this doesn't happen anymore. Okay, so that's the end of my talk, but I don't know how to end talks. So let's ask ChatJPT how to end talk. Tell me, I asked ChatJPT last night, tell me a joke about Seth and OpenSack. Okay, here's the jokes. Why don't Seth and OpenSack play hide and seek together? Because every time they do, OpenSack says, you can't hide. I always find you in my block storage. I mean, that's pretty good. It's true, right? Okay, yay.


and welcome to the session I am Deepika here. Let's discuss the question which says show that the given differential equation is homogeneous and solid. X square dy by dx is equal to X square minus 2 y square plus X y. Let's start the solution. Now the given differential equation is dy by dx is equal to X square minus 2 y square plus X y. Let us give this equation as number 1 or this can be written as dy by dx is equal to X square minus 2 y square plus X y over X square. Now if we express the right hand side of this equation as a function of y over x which is a homogeneous function of degree zero then the given differential equation is a homogeneous differential equation. So we will try to make the right hand side of this equation as a function of y over x. Now this equation can be written as dy by dx is equal to 1 minus 2 into y square over x square plus y over x. Let us give this equation as number 2 since right hand side of above equation that is equation 2 is of the form zero y over x and so it is a homogeneous function of degree zero therefore the given differential equation that is the equation 1 is a homogeneous differential equation. We will solve equation 2 by putting y is equal to vx so put y is equal to vx therefore dy by dx is equal to v plus x into dv over dx. Now we will substitute the value of y and dy by dx in equation 2. So from equation 2 we have v plus x into dv over dx is equal to 1 minus 2 v square plus v minus 3 or x into dv over dx is equal to 1 minus 2 v square plus v minus 3 or x into dv over dx is equal to 1 minus 2 v square or now on separating the variables we have dv over 1 minus 2 v square is equal to dx over x. Now integrating both sides we have integral of dv over 1 minus 2 v square is equal to integral of dx over x or integral of dv over now let us take 2 common so we have this is 1 by 2 minus v square is equal to integral of dx over x. Now we can rewrite this integral as 1 by 2 integral of dv over 1 over root 2 square minus v square is equal to integral of dx over x. Let us give this equation as number 3. Now the left hand side of equation 3 is of the form integral of dx over a square minus x square and this is equal to 1 over 2a into log of mod a plus x over a minus x plus c. Now by using this integral the left hand side of equation 3 is 1 over 2 into 2a now a is 1 over root 2 so 2a is 2 into 1 over root 2 which is root 2 into log of mod a plus x that is 1 over root 2 plus v over 1 over root 2 minus v the right hand side is log of mod x plus c now on replacing v by y over x we get 1 over 2 root 2 into log of mod 1 over root 2 plus y over x over 1 over root 2 minus y over x and this is equal to log of mod x plus c or this can be written as 1 over 2 root 2 into log of mod x plus root 2 y over x minus root 2 y and this is equal to log of mod x plus c. Hence the general solution of the given differential equation is 1 over 2 root 2 into log of mod x plus root 2 y over x minus root 2 y and this is equal to log of mod x plus c so this is our answer for the above question the solution is clear to you and you have enjoyed the session bye and take care


Tiger Financial News Network. Okay folks, very emotional today for me. All of you folks, it sent white light out to me, to my friend Dr. Steve, Dr. Steve called me just a little while ago, he's out of his coma, doing actually pretty good, he said. So I'm going to talk to him a little later, but the doctor told me, you've got to love doctors folks. My grandpa used to say, you've got to really believe doctors because they're the most honest profession ever invented by man. And I said, how can you say that? He says, well, you look at the shingle, it says Dr. Tommy G. Gordon, practice of medicine. And he says, they never tell you I've got it right. He said, find the doctor that says, Thomas G. Gordon, I finally got it right. Anyway, that's what it's all about, just like trading, it's practicing anyway. He's doing a little better, still not good. And I'm sure, you know, this is just a little bounce. But it was really great to hear his voice one more time. Stock market, we got three of the indices down, only the NASDAQ is up. The Dow Jones is down, it was down about 350, it's come back about 100 points so far, S&P's rallied back about another 10, 15 points, but it's also bouncing back. The key today, folks, is the fact that we have those perfect numbers. Do it yourself. If you go and look, you'll see that the Dow Jones made another 61% retracement of the May 1st high. And if you take a look at the number on the S&P, it made a 78% retracement of the May 1st high. The NASDAQ didn't come close and the Russell is still lagging badly and continues to lag badly. Don't be concerned about this debt thing, folks, because this is a politician's play and they're going to draw it out to the very last minute. And at the very last minute, President Biden will pull a rabbit out of the hat and save the day, just like all the other 45 presidents before him have done in the last 115 years. Let's take a break and then come back with your calls<PHONE_NUMBER>.


When dams retain reservoirs, they establish a new flow pattern for the water beneath the ground surface around the lake. As the depth of water in the reservoir increases, the pressures that cause flow increase. These flows occur in the soil and rock materials that contain the reservoir and under the dam. If it is an embankment dam, water also flows through the dam. These conditions exist for the entire lifespan of the reservoir, which may exceed 100 years. Dam designers provide filter and drain zones in an embankment dam and between the dam and the foundation to control these flows and protect the dam structure. The use of these zones has long been recognized as essential for safe embankment dams. This is because they control the possibility of piping, the movement of small particles resulting in erosion within the embankment or foundation of the dam. If the potential for piping is not controlled, it may result in failure of the dam after years of apparently satisfactory operation. Not all older dams contain these zones. Even in dams where the designers included filters and drains, not all designs have been effective and some sinkholes and failures have resulted. In some cases, properly designed filters have failed to function because they were constructed improperly. Also, it is economically prohibitive to locate and construct filters to protect all the natural materials that contain the reservoir. In this program, the Federal Interagency Committee on Dam Safety is presenting Mr. John Lowe III, who will review filter design criteria and present two-case histories where sinkholes developed in modern projects. Mr. Lowe, a member of the National Academy of Engineers, has worked in geotechnical engineering his entire career. He did his graduate studies at MIT and in 1945 he joined NAP Engineering Company, which later became Tippets, Abbott, McCarthy, Stratton, commonly called TAMS. As head of the saw mechanic and rock engineering department, he established their first soils laboratory. He eventually became a partner, retiring in 1983. During his career, he has worked on numerous large civil works projects, including ports, highways, airports, and dams. His experience with dams includes active participation from early feasibility stage through design and construction into operation and surveillance. In fact, his role in Tarbelladam in Pakistan, the world's largest embankment dam, is an example of his long-term involvement and broad experience. Also, the use of an excess of one million cubic yards of roller-compacted concrete at Tarbella exemplifies his role in technology development and application. He has served as secretary of the U.S. National Committee of the International Society of Saw Mechanics and Foundation Engineers and as president of the United States Committee on Large Dams. For many years, he was a member of the Saw Mechanics Advisory Board of the Corps of Engineers, Office Chief of Engineers. Contemporary members of the board were Arthur Casagrandi, Ralph Peck, and Stanley Wilson, among others. His experience includes working on dam projects in the Mediterranean and Middle East, the Far East, South America, and the United States. For his contributions in Morocco, the king decorated him as commander of the Order of the Alluit. Since his retirement from Tams in 1983, Mr. Lowe has been a consultant on a number of projects around the world. Some of his assignments included working in Bulgaria, Chile, Malawi, Jordan, Greece, and the United States. May I present Mr. Lowe. Sinkholes have developed in the impervious cores and blankets of what presumably were well-designed earth dams. Projects which have experienced such problems include Baldur Head Dam, England in the 1960s, Tabela Dam, Pakistan in the 1970s, Luddington Pump Storage Project, Michigan in the 1980s, and W. A. C. Bennett Dam, British Columbia in the 1990s. In his ICOTS video and type of seepage and piping, Professor Ralph Peck showed examples of sinkholes and piping tunnels. To create such features, material must be eroded from the dam or from its foundation. Filters are the key measures incorporated in the dam project to prevent such erosion. Here is a typical cross-section of an earth dam. Water seeps through the core of the dam into the chimney drain. The chimney drain must be designed to be much more pervious than the core and, at the same time, have interstices fine enough so that material from the core will not enter them. Similar design is required for the blanket drain with respect to seepage from the foundation. Also, filters are frequently required between the upstream shell of a dam and the overlapping rip-rap slope protection. Upon drawdown of a reservoir, water will seep out of the upstream shell. Filters are required to prevent washing of upstream shell material through the rip-rap and into the reservoir. Filters may consist of one layer or two or more layers as shown in this sketch. The material being protected is termed the base material. If the material is the core of a dam, first a fine filter would be required, such as sand and then a coarse filter of gravel. The design of filters has been well understood since about the 1940s. First, I will review briefly the development of filter design. Most of my talk, however, will be devoted to a discussion of examples of widely graded base materials for which standard filter requirements have not worked. We use the term non-self-filtering for these materials. I believe that this non-self-filtering behavior was first recognized in 1974 at the Tabela Dam Project. I will describe the Tabela Dam Project situation in some detail and also a similar situation at the Luddington Pump Storage Project. After discussing these two projects, I will summarize my conclusions regarding filter design for both self-filtering and non-self-filtering base materials. Let us start with a brief review of the development of filter design. From the early days of earth dam design, engineers have generally appreciated the importance of proper filter design. Many early dams included filters, although they may not have been designed as we would design them today. In his textbook Fundamentals of Soul Mechanics, published in 1948, Taylor discusses filter design. One of the points which he makes is the following. Given a stack of tightly nested spheres as illustrated by these tennis balls, the largest-sized sphere which can pass through the interstices of such a stack would have a diameter somewhat less than one-sixth that of the nested spheres. In 1940, George Bertram carried out laboratory investigations of filters under Professor Arthur Casagrandi. His investigations were on materials of uniform grain size, both for the filter material and for the base material. He found that there was not any appreciable washing of the base material into the filter until the grain size of the base material was finer than one-tenth the size of the filter material. For comparison, the gradation curve for a widely graded or well-graded material is also shown. Tasagi advanced the criterion that the D15 size of the filter should be less than four to five times the D85 size of the base material. The concept is that the D15 size of the filter is an approximate measure of the size of the pore openings of the filter. Also, that if the D85 size of the base material cannot enter the pores of the filter, all of the base material will be held back from entering the filter. Tasagi also proposed that the D15 size of the filter be four to five times greater than the D15 of the base material. Purpose of this was to ensure that the filter had appreciably greater permeability than the base material. We need to recognize also that over the years, considerable research on filters has been carried out by the U.S. Corps of Engineers, the US Bureau of Reclamation, and the Soil Conservation Service. Now let me describe the sinkhole problem which developed at the Tabela Dam project. Tabela Dam is the largest hydro project dam in the world. It is located on the Indus River in northern Pakistan. The dam was completed in 1974 and the first power went online in 1976. A geologic section along the baseline of the main embankment looking downstream is shown here. As shown, the central 6,000 feet of the dam is founded on river alluvium. The left 1,000 feet and the right 2,000 feet are founded on pond rock. On the right side, the alluvium is as much as 700 feet thick. Because of the great depth of alluvium, it was not feasible to make a positive cutoff. To control under seepage, an impervious upstream blanket was connected to the inclined impervious core of the dam. The impervious blanket extended about 6,800 feet upstream. At the toe of the dam, it is 46 feet thick. It then tapers to 3 feet at the upstream end. A horizontal drainage blanket was provided under the downstream half of the dam and relief wells were provided at the downstream toe. These are typical seepage measures as discussed by Professor Peck. The only impervious material available for the blanket and core in the vicinity of the site was flood plain silt. Such material would be difficult to compact, and unless well compacted, might be subject to liquefaction under earthquake loading. The site is located about 60 miles from the Himalayan thrust vault, and ciliary faults occur much closer to the site. Thus, the dam had to be designed to withstand severe earthquake loading. In order to obtain a suitable impervious material, the design required that the flood plain silt be mixed with well graded slope wash found a short distance from the site. The grain size distributions of the silt and of the slope wash are shown here. The slope wash was separated into the fraction between the six inch and the three-quarter inch screens and the minus three-quarter inch material. These fractions were then combined with the silt in proper proportions. The resulting core and blanket material is shown on this grain size plot. The average material and the 80% limits are shown. The resulting material is well graded, has about 35% passing the number 200 sieve. The blended material is similar to a well graded glacial till. Such glacial materials have low permeability, high shear strength, and low compressibility. Engineers have generally considered them to be ideal for use in impervious zone of a dam. The blended material was used in the core of the dam and in part of the upstream impervious blanket. The portion of the blanket between the upstream toe of the dam and the upstream coffer dam was required to have 50% of its thickness of such material. The remaining blanket material had to have low permeability, was not required to be so well graded. The foundation on which the blanket was placed consisted of alluvium deposited by a fast flowing river. The slope of the Indus River at the site is six feet per mile. The alluvium is composed of two materials, cobble gravel and fine sand. Here are their gradations. The cobble gravel consists almost entirely of 12 inch to 3 quarter inch material. The sand consists of about two-third fine sand and one-third medium sand. The cobble gravel was carried downstream by the river as bed load and the fine sand was carried down as suspended load. Strata in the foundation consists of three possible combinations of these two materials, that is open work cobble gravel, fine sand strata and cobble gravel choked with fine sand. The cobble gravel does not at all meet filter requirements with respect to the fine sand, that is the d-15 size of the cobble gravel is much coarser than four to five times the d-85 of the fine sand. The fine sand can readily pass through the interstices of the cobble gravel. On the other hand, the blended impervious material readily meets filter requirements with respect to the cobble gravel. The d-15 size of the cobble gravel is definitely smaller than four times the d-85 of the blanket material. The first filling of the tabella reservoir took place the summer of 1974. As the so-melt runoff season progressed, the four tunnels being used for diversion were progressively closed. A problem developed with the last of the three gates on the last tunnel to be closed, tunnel two. The gate would close only about halfway and became stuck in this position for a period of about 25 days. Severe cavitation damage occurred to the tunnel immediately downstream of the gate and a 200 foot long section of the tunnel collapsed. During the closing of the tunnel gates, the reservoir had risen about 340 feet above riverbed. The gates of the other three tunnels were opened and the reservoir lowered as quickly as possible. When the top of the upstream blanket was exposed, everyone was surprised to see that it contained 362 sinkholes. About 140 tension cracks were observed also as well as several linear compression zones. The sinkholes were generally located along the cracks. Here are three photographs which give a panoramic view of the blanket after a drawdown of the reservoir. The first view is along the upstream toe of the dam and was taken from the left abutment. The upstream face of the dam is evident on the left hand side of the picture. In the far distance are the intake structures for the four tunnels through the right abutment. The rectangular black area is the cut face of an excavation made on the right abutment for the approach channel to the tunnel intakes. The cut is black because it was made in carbonaceous schist. The second picture is of the central portion of the blanket. In the foreground, black clay silt deposited during the filling of the reservoir is evident. In the distance, the placement of additional blank material over the original blanket is evident. This is the area where the alluvium is the deepest and where most of the sinkholes occurred. The third picture is of the left hand side of the blanket. An individual sinkhole is evident in the foreground. The blanket had settled appreciably under the difference in water pressure between that on the top of the blanket and that on the bottom. This difference in pressure varied from zero at the upstream end of the blanket to close to 300 feet of water at the downstream end because of the non-uniformity of the foundation strata and undulating settlement pattern developed. Tension cracks developed along the centerline of the ridges in the pattern and linear compression zones along the valleys in the pattern. The compression zones were evident only in the soft clay silt covering the blanket. This photograph shows an elongated sinkhole following a presumed tension crack. A backhoe trench has been dug across the sinkhole. The layer of silt which was deposited while the reservoir was up is evident. In the dried out state, the silt has a great color. Another example of an elongated sinkhole is shown in the next photograph. In this instance, there seems to have been a branching of the tension crack. Here is a photograph of a deep exploratory trench across the previous sinkhole. One can barely make out vertical cracks extending downward from the vertical sides of the sinkhole. The next photograph is a close-up of the upper part of the sinkhole. The cracks extending downward from the sides of the sinkhole are more evident in this photograph. This is a photograph of a crack extending down to the bottom of the blanket. A pencil was placed at the interface of the blanket and the foundation. The material along the crack is coarser than the adjacent material, presumably because some of its fine fraction has been washed out. Openwork type alluvium is evident at the bottom of the picture. Here is another photograph taken of the bottom of the blanket. Only coarse blanket material remains, just above where the blanket material contacted the foundation cobble gravel. Void space exists between the washed blanket material and the intact blanket. Washing out of fines from the blanket presumably will continue with time and a sinkhole will become evident at the surface. This is a photograph of a sinkhole which developed in the material surrounding the casing of an instrumentation drill hole. Apparently the installation of the casing disturbed and loosened the surrounding blanket material making it susceptible to leaching out of fines from the blanket. There are two basic questions to be covered. Where did the fines from the blanket go and to why did they get washed out of the blanket? The river cobble gravel on which the blanket was placed was choked with river sand at least for the top two or three feet. However, as mentioned earlier the sand fraction can readily pass through the interstices of the cobble gravel fraction. Water seeping under the blanket could wash the sand fraction out of the cobble gravel immediately under the blanket to open work cobble gravel nearby. Also water seeping down cracks would assist in the washing down of this fine fraction. We need then to look at the possibility of the blanket material washing into the open work cobble gravel. As I mentioned earlier the open work cobble gravel readily satisfies filter criteria for the blanket. The D-15 size of the cobble gravel is much less than four times the D-85 of the blanket material. The comparison of the gradation of washed blanket material which has been found in the top of the foundation cobble gravel and the gradation of nearby intact blanket material is shown here. Practically all of the silt fraction was washed out as well as a very high percentage of the sand fraction. To analyze the situation it is interesting to divide the blanket gradation into two fractions and to check whether the coarse fraction satisfies filter requirements with respect to the fine fraction. The separation into coarse fraction is made at a grain size slightly coarser than the one-tenth millimeter. The D-15 size of the coarse fraction is about 12 times the D-85 size of the fine fraction. Normal filter requirements are required to be less than four or five times. On the percent finer scale the separation between the D-15 of the coarse fraction the D-85 of the fine fraction is 15 percent. To satisfy normal filter requirements the slope of the gradation curve should be no flatter than four times change in grain size for this 15 percent change in percent finer. Professor Cameron Kenny who carried out laboratory tests at the University of Toronto however has found that some materials were not self-filtering unless their gradation curve was flatter than a four time change in grain size for a 22 percent change in percent finer. That is the same as a two and a half times change in grain size for a 15 percent change in percent finer. It would appear that anyone using widely graded non-plastic materials for impervious sections in an earth dam should check the self-filtering characteristics of the material in the laboratory. The sinkholes found in the tabella blanket after the first filling and emptying were repaired as indicated in this drawing. Filter material was placed in the sinkhole and a blanket two feet thick and five feet out from the sinkhole. Then this was overlained with 15 feet of regular blanket material. Since the upstream blanket had not been replaced the top of the blanket has been monitored each succeeding year of operation using side scan sonar. Here is what the results of the side scan sonar look like. No picture is obtained directly underneath the side scan unit. This is the white band in the picture. The side scan unit is towed 10 to 15 feet above the bottom of the reservoir. It takes a picture at a flat angle out from both sides of the instrument. The three white dots evident in the lower part of the picture are sinkholes. From time to time the depth of sinkholes was determined by towing a special sounding device called an ORE profiler across the top of the sinkhole. When a sinkhole was found about 50 barge loads of wet self filtering well graded gravel sand and silt were dumped on it. The barges used for this operation were the bottom dump type as shown in this picture. The barges had a capacity of 105 cubic yards. Side scan sonar pictures taken after dumping on a sinkhole confirmed that the dump material stayed together as a mass as it sank through the reservoir. Here is a picture of one of the barges being loaded and here a picture of two barges as they traversed the reservoir. It is significant to note that the open part of the sinkholes penetrated no more than about one third the blanket. This is the result of the loss of the fine fraction from the blanket. The remaining course fraction occupies the lower two thirds of the sinkhole. Leakage through a sinkhole certainly was much more than leakage through a corresponding zone of undisturbed blanket material. Nevertheless as Arthur Casagrande who was on the board of consultants pointed out the sinkholes amounted only to the equivalent of pinpricks in the blanket. Upstream of the blanket the reservoir is in direct contact with the alluvium. This area is vastly greater than the total area of all the sinkholes. Thus the contribution of the sinkholes to the quantity of under seepage is negligible compared to the quantity of seepage coming from upstream of the blanket. Nevertheless it appeared prudent to monitor the development of sinkholes and to treat any that developed with barge dumping. During the filling and emptying of the reservoir in 1974 362 sinkholes developed. During the filling in 1975 440 sinkholes developed and were treated by barge dumping. The reservoir was filled to a higher level in 1975 than 1974 and then held at that level. In 1976 29 sinkholes developed in 1977 9 and in 1978 10 11 developed in 1980 and since then only an occasional sinkhole has developed. Now let us consider the sinkhole trenches which developed at the Luddington Pump Storage Project in Michigan. The upper reservoir of the Luddington Project experienced many trench type sinkholes similar to those found in the upstream blanket of the Tobelodam Project. As shown on the plan view the upper reservoir was located by a dike. The reservoir is located on a bluff about 250 feet above Lake Michigan. The sinkhole trenches are indicated on the plan. The sinkhole trench at the southern end of the reservoir is about 3,600 feet long. Most of the bottom of the reservoir is covered with a five foot thick blanket of impervious material. In some areas impervious glacial till was present and no blanket was required. The blanketed area is underlain by outwashed sand. The groundwater table in the outwashed sand occurs at elevation 750 feet which is about 90 feet below the bottom of the reservoir. The sinkhole trenches developed primarily through the blanket. In one or two instances however where the outwashed sand contained an appreciable percentage of minus 200 material the sinkhole trench penetrated several feet into the outwashed sand. During operation the reservoir fluctuated from 100 feet to 30 feet deep. This is the range in gradation of the outwashed sand. Material coarser than the number 20 sieve practically always is less than 10 percent. Generally material finer than the number 200 sieve amounts to 5 to 35 percent. Occasionally larger amounts occur because of its very small percentage the plus 20 sieve material occurs as float in the rest of the material and has no significant effect on the characteristics of the material. Generally the gradation curve of the minus 200 material is very flat. At least a 10 time change in grain size occurs for a 15 percent change in percent finer. Because of this flatness the coarse fraction does not at all act as a filter for the fine fraction. Under the 100 foot water load imposed upon the blanket the blanket settled due to the glaciation which occurred at the site and the foundation had linear features of higher and lower compressibility material. Thus the settlement pattern which developed had ridges and valleys. As at the tabella blanket tension cracks developed along the ridges. At both projects the tension cracks developed first at the crests of the ridges. They then readily penetrated through the blanket by hydraulic fracturing. The reservoir pressure which entered the initial shallow crack was definitely greater than the horizontal earth pressure in the blanket. Because of this unbalance in pressures the crack readily penetrated to the bottom of the blanket. Considering the porosity of the outwash sand about 25 percent by weight of minus 200 material is required to fully choke the outwash sand. A large portion of the outwash sand contains only 5 to 15 percent of minus 200 material and thus is not fully choked. A very high gradient is imposed upon the outwash sand foundation where crack completely penetrates the blanket. Under this gradient whatever minus 200 material is in the voids of the outwash sand in the upper part of the foundation is washed downward into not fully choked material in the lower part of the foundation. Thus progressively ample space is made available for minus 200 material from the blanket to be washed into the foundation. The blanket material generally consists of over 80 percent minus 200 material. It is interesting to note that the sinkhole trenches which developed in the tabella and Luddington blankets both had more or less parallel vertical sides. Probably two parallel tension cracks developed a few fruit apart along a settlement ridge. Material is first lost from the bottom of the blanket adjacent to the bottom of the cracks. In the upper part of the crack water is seeping into the blanket. This seepage stabilizes the sides of the upper part of the crack. At the bottom where material is lost the minus number 200 fraction of the blanket is washed into the voids of the outwash sand. The plus 200 fraction which is only a small percentage of the total blanket is left behind. A void space developed above the plus 200 material and with time works its way toward the surface. Also there would be a tensi for the blanket between the two cracks to settle into the void space. A sinkhole trench then becomes evident at the surface. The upper parts of the trench walls tend to remain vertical because of seepage into them from the trench. The treatment proposed for the Luddington sinkhole trenches was to fill the trenches and to cover the area adjacent to the trenches with a special filter material. The gradation of the 30 and 200 fraction of the outwash sand is shown on the left hand side. The D85 size of the special filter material should be no smaller than 1 fourth the D15 of the 20 to 200 fraction of the outwash sand. A reasonable band for the special filter is shown shaded. Gradations of several blanket samples are shown on the right hand side. The D15 size of the special filter readily satisfies filter requirements for the D85 size of the blanket material. Tailings for several nearby mining operations were found to meet the special filter requirements. This concludes my remarks on the sinkholes which have developed in the tabella and Luddington blankets. Now let me summarize the key items in the design of filters. One, materials whose minus 200 sieve fraction is composed of non-plastic silt and rock flour require much more stringent filter criteria than materials containing plastic fines. The fines for both the tabella and Luddington blankets were non-plastic fines. Two, a widely graded material containing gravel, sand and non-plastic fines must meet self-filtering requirements to be stable. Gap graded materials are especially suspect. Applying standard filter requirements, one would expect that a material whose grain size curve had a slope which was no flatter than a four-time size change for a 15 percent finer change would be stable. However, laboratory tests performed by Kenny indicate that sand and gravel materials may not be self-filtering when the grain size curve has a slope as steep as a four-time size change for a 22 percent finer change. Three, in view of Kenny's test results, it appears desirable to lean toward somewhat more conservative zine of filters. Thus it is desirable that the D-15 size of filter material be no more than four times rather than five times the D-85 percent size of the material being protected. Four, when well-compacted, non-self-filtering material is much more resistant to loss of fines than when poorly compacted. The additional compaction reduces the size of voids through which the fines can pass. Thus, such material should be well-compacted whenever possible. The development of sinkholes through non-self-filtering impervious blankets is greatly facilitated by the development of cracks in the blanket and by loose zones that may exist around instrumentation installations through the blanket. Six, transverse cracks frequently occur across impervious cores of a dam where the dam overlies a steep abutment. The filtered downstream of the core must be designed to prevent loss of fines from the core if the core is not self-filtering. As a second line of defense, a layer of filter material can be placed on the upstream side of the core. Then, if a transverse crack developed through the core, this filter material would be washed into the crack and greatly reduce flow through the crack. As a result, there would be less tendency for the loss of fines. For materials containing fines which are plastic, fine concrete sand makes a suitable filter even if the grain size curve indicates that the impervious material is not self-filtering. To avoid segregation during transport and placement, filter materials coarser than concrete sand should have narrow gradations. The hundred percent size of the filter should be no more than two to three times the zero percent size. Concrete sand can have a gradation much wider than this because when placed damp, it has enough apparent cohesion to keep it from segregating. Generally, our current knowledge is adequate for us to design proper filters. The principles need to be thoughtfully applied. Care must be exercised to ensure that segregation of materials does not occur during construction.


Welcome back. So, yesterday we were discussing about similarity matrix and let me begin with the advantage of this similarity matrix. So, this is our… So, A is… So, introduced notation yesterday. So, this is N N by N real matrix. We call A is similar to B. So, suppose there exists non-singular C belonging to M N R such that C inverse A C equal to B. So, this is… We say that A and B are similar. So, we are aiming at finding a suitable C such that this B is either diagonal or in the worst scenario an upper triangular matrix. We will come to this little later and we also saw that. So, this implies C inverse E A C is equal to E to the B. So, the exponentials of A and B are also similar. In the context of O D, so let us consider this x dot equal to A x. Suppose this is the given system and we know that the general solution is given by A to the T A and x naught. x naught is an arbitrary vector in R n and now through similarity transformation, we transfer this given system into y dot equal to B y and similarly, y is T is given by E to the T B sum y naught. And when B is the simple either diagonal or upper triangular, this is very easily computed. Not only that, we can easily analyze the qualitative behavior of this y t and how x and y are connected it is very easy. So, you just you put y is equal to C inverse x and so this y naught will be just C inverse x naught. And once we know y, so it is easily we can also analyze x since x is C y. So, that is the advantage. So, that is why this all effort is being done to find a suitable non-singular C such that this C inverse is equal to B and B is quite simple one. So, that we can compute the exponential of T B very easily and then we analyze the solutions of this reduced system and then we get information about the original system. That is the advantage. So, let us now begin with this thing. So, before going to that thing, let me again make a few remarks. So, by the definition of the norm, it follows that norm of A x is less than or equal to norm A norm x for all x in R n. We also saw that also we saw this A is less than or equal to this Provenius norm. So, this is Provenius norm and from this if I replace x by x minus y and by linearity of A. So, we have A x minus A y norm in x minus y and that proves this function x going to A x is Lipschitz continuous. More is true. So, suppose A t is a matrix valued function of T in some interval. So, that means for each T in AB, A t is in M n R and suppose and assume the elements of A t are continuous and then this for each T I have this thing and since this Provenius norm is some of all squares of the elements of A t and since I am assuming they are all continuous. So, this is just a some let me put M for all T in using continuity of the elements of A t. So, this implies. So, we have this useful inequality. So, less than or equal to M x minus y for all T in AB. So, that means this matrix the x going to A t x is uniform Lipschitz in this interval and this is we use this to use this fact in studying the linear system x dot equal to and because of this global existence the solutions exist in any interval where A t is continuous. So, with this little remark. So, let us again go back to linear algebra more linear algebra as said yesterday this is not a course on linear algebra. So, I am just recalling certain facts that are used in our study of differential equations. So, suppose M and n are subspaces R n such that. So, since they are subspaces 0 element is always there and I just want their intersection to be just 0 we call subspaces we call M n disjoint. So, though the intersection is not empty, but it is the trivial subspace namely the 0 subspace we still call it disjoint and assume R n is m plus m. Let me explain that little bit. So, this is called direct sum of M n this means each vector in R n is written as a sum of a vector in m and a vector in n and this condition M n are disjoint. So, we can easily check that R n is equal to m plus n m direct sum m if and only if for all x in R n there exist unique y in m and z in n such that x equal to y plus z that should happen and this is easily extendable to any finite number of subspaces and that is what we are going to do. So, in addition suppose M n are invariant under A. So, remember A is given matrix n by n matrix. So, that is again definition A m. So, you act A on every element of m and again that should be in n and similarly for m.  In this situation suppose we choose a basis in m and a basis in n and if you put together that will be a. So, I will just write this and this will form a basis in R n. Now, you take those basis elements and form this matrix C. So, this is C and these are you put the basis elements of m here and then you put basis elements here and then you act A. So, since these are all basis vectors C is non-singular and by this assumption that m and n are invariant under A you can easily check that this C is A 1, A 2. So, this is the sub matrix. So, this is corresponds to m and this corresponds to because if I act A on this basis elements which are in m and the result is again they are back in m. So, each action is again a linear combination of vectors of these things and that is expressed like that. So, in other words so if you have two subspaces invariant and R m and who directs some is R n then the matrix with respect to that basis decompose into a block matrix. So, in other words what we have is C inverse A C is A 1, 0, 0, A 2 and this is easily extendable to may finitely many subspaces. So, let me just state that thing. So, let m 1, m k be subspaces of R n such that each m j is invariant under A that is one condition and then I have m j intersection m l is just 0, j different from l. So, they are mutually disjoint and the third condition so R n is direct sum of m k with these conditions then there exist non singular C such that C inverse A C is A 1. So, in other words in such a situation we can find a similar matrix which is a block diagonal matrix satisfying this condition. So, this is important in this case immediately we see that C inverse E A C is E A 1 etcetera E A k this we saw yesterday. Once the matrix is block diagonal it is very easily computed and now we proceed to find some special subspaces of R n which are related to A and which satisfy this hypothesis of this thing. So, that we have a similar block matrix which is similar to A. So, that is our next aim this. So, yesterday I introduced the concept of spectrum of A. So, this is just spectrum of A. So, denoted by S p of A. So, this is set of all eigenvalues of A though A is a real matrix an eigenvalue of A could be real complex. So, let us split this into two parts. So, let us lambda 1 lambda r. So, these are real eigenvalues counted with multiplicity and then this we have mu 1 mu s. So, these are non real and since the characteristic polynomial has real coefficients we also have this mu 1 bar mu s bar also eigenvalues of A. So, these are also non. So, they always occur in pair. So, the total number because we are counting multiplicity. So, therefore, we have r plus 2 s is equal to n. So, this is the order of the A and this is also the degree of the characteristic polynomial characteristic polynomial. So, now just pick. So, let lambda belongs to spectrum of A also real. So, complex case is similar. So, let K be the algebraic multiplicity of lambda. So, this is the multiplicity of lambda as a root of the characteristic polynomial that is. So, there is another notion called geometric multiplicity. So, for that we have to introduce a subspace. So, let A n 1 be kernel of A minus lambda i. So, this is the null space null space or kernel of this A minus lambda i. So, this is referred as Eigen space of A corresponding to lambda. So, if lambda is an eigenvalue we just form this and this is a subspace of. So, n 1 is a subspace of R n. So, in case of complex eigenvalue we take the real and imaginary parts of the eigenvector and still we get a subspace of R n that is important that we keep in mind. So, the dimension as a subspace it has a dimension of n 1 is called the geometric multiplicity of lambda. So, let me just state a fact that geometric multiplicity is always less than or equal to algebraic multiplicity and the difference is referred to as the difference is referred to as deficiency of lambda for any lambda. So, that is always lambda in fact. So, the good case is when geometric multiplicity is algebraic equal to algebraic multiplicity then we will have sufficient number of eigenvectors spanning this subspace n 1 and a difficult part is when this is strict inequality. So, in that case what we do is now define in general. So, define n j kernel of a minus lambda i to the power j. So, you do it j equal to 1, 2 etcetera. So, n 1 is the eigen space that we have defined and we have this j it is very easily checked that this is ascending chain of subspaces. And since we are in a finite dimension this chain cannot continue for long. So, there exist. So, there exist smallest d bigger than equal to 1 such that n 1 n 2 n d after which they are all equal. So, there are no more additions of new non zero vectors. So, this is smallest it has a name and this is called index of lambda. So, everything is we have fixed in real eigen value lambda and we are just discussing about that. So, it again one more fact. So, these are all subspaces this n 1 n 2 n d are invariant under a. So, let me not stress that again and again. So, there is invariant subspaces of R n and more importantly and dimension of n d is equal to algebraic multiplicity. So, when geometric multiplicity of an eigen value is strictly less than the algebraic multiplicity we have to go for some more linearly independent vectors in order to get the full dimension namely it is algebraic multiplicity. We want to reach the algebraic multiplicity and this is the way one reaches that thing. So, you know you do this for every so if. So, let mu be a non real eigen value and again you form the same thing there is this one there is no problem kernel of a minus mu i j j equal to 1 2 n and it will have some index. Now, you just consider the real and imaginary parts of vectors in these n j's all you have to do because remember we want to find only a basis for R n. So, we want only the real vectors and when there are complex eigen vectors you take the real and imaginary part they are not eigen vectors, but they are linearly independent as we saw yesterday and that will do our job. So, in this way so let me now just put this together. So, for each lambda in the spectrum of a real or non real does not matter we thus form invariant subspace. So, let me call it n lambda of R n that is important this subspace is in R n such that dimension of n lambda is algebraic multiplicity of lambda if lambda is real otherwise it is 2 times algebraic multiplicity if lambda is non real because that count has to be proper because when lambda is non real lambda bar is also an eigen value. So, it has to be counted twice. So, that is why this twice and they are all invariant under a and by very choice. So, we also have that n lambda intersection n mu is just 0 if lambda is that is by very construction you see that. So, therefore, so R n is written as so let me just write lambda 1 lambda R. So, this corresponds to real eigen values and then I have n mu 1 is n mu s. So, I am not writing the mu 1 bar mu s bar because they are already included here. So, this corresponds to real eigen values and this corresponds to non real eigen values and the this dimensions perfectly match and that is why we have this inequality and now you just use the fact we already stated since all these are invariant. So, these subspaces are invariant. So, therefore, there exist a non singular c such that c inverse a c is now a 1 a 2 a r lambda 1 lambda mu 1 and the next task is to find a suitable basis in each of these subspaces. So, that this block matrices have a very simple structure that is what we are going to do next time next to it. So, instead of doing for a general thing so let me just illustrate by a case how to do that illustrate this by a case. So, how to choose a specific basis in n t. So, suppose lambda is a real eigen value algebraic multiplicity 4 and geometric multiplicity. So, I have n 1 n 2 n 3 say and this dimension is 2 dimension is 3 and this dimension is 4. Suppose I have this situation there are other possibilities. So, let us take one such possibility and now I would like to construct a basis where a is simplified to an upper tangular matrix. So, choose x in n 3 such that x is not in n 2 this is possible because this dimension is 3 and this is dimension 4. So, there is at least one non zero vector which is in n 3 which is not in n 2 that is fine.  So, this is in n 2 by very definition and this vector is n 1 and these are all non zero. In fact, these are linearly independent check that are linearly independent. And now this is a non zero vector in n 1, but dimension of n 1 is 2. So, choose. So, let me call it u 1 u 1 in n 1 which is linearly independent now. So, let me put some name. So, this let me call this as u 2 and this one u 3 and this one u 4. If you change the order the matrix will change. So, that is that is what we will see now.  So, this is just becomes lambda u 1 and u 2 is also in n 1. So, that becomes just lambda u 2, but this is not in this u 3 is in n 2.  So, let me write this same for u 1, u 2, u 3, u 4 and now I write this matrix lambda 0, 0, 0, 0, 0, lambda 0, 0, 1, lambda 0, 0, 0, 1. So, it is very easy to check. That is what we get just writing this thing. And we see significantly here there is one block here and another block here. See these are referred to as Jordan blocks Jordan blocks. So, in general the number of Jordan blocks corresponding to an eigenvalue is equal to geometric multiplicity of that particular eigenvalue. So, in this case we have the geometric multiplicity 2. So, we get 2 blocks and if mu is non real. So, just a plus i b b non 0. The Jordan blocks are bit complicated because the eigenvectors are complex. So, we have to take only real part and imaginary part the Jordan blocks corresponding to mu r of the form. So, remember they always appear in pairs. So, here also we get 2 by 2 blocks. So, let me write just b 2 i 0 b 2 i i 2 let me write it etcetera. So, diagonal blocks. So, b 2 is 2 by 2 matrix this a b minus b a and i 2 is just 1 0 0 the identity 2 by 2 matrix. So, here in case of real thing we have just scalars on the diagonal, but in the complex case we have this 2 by 2 blocks. So, this is the only difference. So, let me summarize in detail now. So, this. So, summarize. So, given a matrix M and r there exist a non singular c such that c inverse ac. So, let me just write j 1 j 2 j k where each j l is a Jordan block corresponding to an eigenvalue and we have. So, if the eigenvalue is real then j l would be very simple lambda 1 1 and each j l might contain several blocks. So, one of the I am just this is not the only possibility there will be several blocks and if the eigenvalue is non real then this will be just b 2 i 2 that is the only difference. So, in particular in particular. So, we have this c inverse ac not c inverse ac exponential e to the j 1 etcetera and in the next 10 minutes I will just show you how easy it is to compute the exponential of each of these blocks. So, let me just again concentrate on that. So, let lambda be real j is equal to. So, I take one of the blocks. So, this is a square matrix of some order. So, what is e j? So, that is the question. So, again a fact. So, let me just. So, if a and b commute that is a b equal to b a then exponential of a plus b is equal to exponential of a exponential of b. So, just like exponential of real number and in general if you have non commutativity then you may not have equality in this e to the a plus b may be different from e to the a and e to the b. So, since addition is commutative we also have this and this easily follows from this binomial theorem just like again real numbers. So, if a b equal to b a then a plus b to the k is summation k j a j b k minus j j equal to 0. So, this you can easily prove by induction and which in turn give this. So, that is the only thing involved. So, now we apply this to j. So, read rewrite j as lambda i plus b. So, b is this matrix 1 1 1 1. So, everywhere 0 except the super diagonal which is just all once and since this is the identity. So, these two matrices commute. So, we have this e to the j is equal to e to the lambda i e to the b and this you can easily check e to the lambda i is nothing, but e to the lambda into i into e to the b. So, this is identity. So, we have just e to the lambda e to the b and now let me just explain this. So, this is by definition i plus b plus a j. The important thing about b is b is a nilpotent matrix b is a nilpotent matrix that is there exists some R integer such that b R is 0. So, this is a finite sum. So, it only goes up to b R minus 1 by R minus 1 factorial and it is very easy to compute even the powers. So, this the diagonal containing once will be just shifted above and above. So, it is very easy to compute. So, and we have this thing and in case lambda is non real. So, mu suppose mu is non real then we have this j as block i 2 i 2 etcetera till we can use. So, this will write it as this b 2 b 2 plus again same thing 0 i 2 0 0 i 0 i 2 etcetera. So, again this is nilpotent nilpotent and these two commute. So, it is very easy to find therefore, e to the j is again e to the b 2 b 2 into i plus. So, let me call this matrix as some d. So, I have d plus d square etcetera. So, this is finite sum. So, there is no problem about that finite sum and leave as an exercise. So, remember b 2 is a minus b b a just 2 by 2 matrix. So, it is very easy to compute that. So, compute e to the b 2 is equal to e to the a minus let me just write that e to the a cos b minus sin b sin b. So, it is not difficult to compute that. So, finally, let me just take this theorem. So, just combine all these things. So, suppose spectrum of A is a subset of lambda in C such that real lambda is strictly less than 0. So, then there exist constants k positive and sigma positive such that the matrix norm of e to the ta is less than or equal to k e to the minus sigma t for all t. So, this is important. This is not valid for all t only for t negative and just you have to use is you have this Jordan form j 1 j 2 j k. So, therefore, you have e to the ta is less than or equal to norm of C inverse norm of C and then you have norm of this matrix e to the t j 1 e to the t j k and this is just a constant you leave it and this we have already seen that this is less than or equal to maximum of e to the t j l because this all block diagonal and now if you use the previous representation you see that sigma can be taken as minus half maximum of lambda in the spectrum. So, this is an exercise you can just find that this is can be estimated less than or equal to e to the minus sigma t some constant I will put another constant here constants will come. So, there will be some polynomials here polynomials in t coming from this term and that will be killed by half of this another half. So, that is how you get only half any fraction will do. So, with that we conclude this preliminaries on linear algebra you can work out several things in the some good text on linear algebra and fill the details that are left out in these two hours. Thank you.


It attracted me to Jersey Shore's pediatric program. This program pushes you to work and pushes you to learn and become autonomous. Coming from Jersey Shore, I feel like I'm prepared to function on my own as a resident, as a fellow, and eventually as an attending. Our ICU attendings, both in the PICU and the NICU, are in-house, but they really try to be hands-off and let you run the unit in a way that you're comfortable. If you call in the morning and say, you know, these are the things that I did overnight that I'm comfortable with, these are the questions I have, they're there for you. So they really are allowing us to practice medicine in a way where we feel that autonomy but we have a wonderful safety net. So we get a lot of exposure to the pediatric sub-specialties. We always have specific weekly rounds with our sub-specialists to go over their patients that are on the floor, to go over high-yield topics, and I've never had a problem calling any of them at two o'clock in the morning with a question and they're always very willing to help out. The pediatric department, I think, does a really great job prioritizing resident wellness. We have a Thursday lecture block every week to meet the academic requirements for the ACGME, and then every couple of weeks we have a resident wellness activity. The one reason I picked this program was when I went to my interview dinner, the residents all seemed to enjoy each other and get along and seemed like a little family, and so when I started here, that was actually true. We're always at each other's house as we hang out with each other all the time. When we work together, we're able to be honest with each other and be able to help patients better because of how close we are. Our attendings are phenomenal, they're here to support us, we're here to learn. It's been a tremendously great experience so far, and I could not have done any of it without my co-residents. If you're looking for a small community program that will give you a really excellent balance of autonomy and education, I think this is the program for you.


Happiness is one of the most sought after things in life. We also have money, houses, cars, and sometimes women. But why is happiness so important to people? Happiness is important because people don't feel complete without being happy. In many ways, people seek all the things they desire in the hope that getting them will make them happy. But as reality often shows, this is seldom the case. According to Forbes, happiness comes in so many different forms that it can be hard to define. Unhappiness, on the other hand, is easy to identify. You know it when you see it, and you definitely know when it's taken hold of you. Unhappiness is not something anyone should indulge themselves in for too long. It can lead to depression, and depression leads to health challenges.  In today's video, I'm going to share with you 16 habits of chronically unhappy people. If you are new here, consider subscribing to this channel for more awesome videos such as this one. 1. They wait for the future to be happy. Do you often catch yourself saying, I will be happy when… Stop it! You are merely postponing your happiness to another time. It means you can't be happy now, only later. Why wait if you can have it now? 2. They complain. It is easy to call out people who complain, but it isn't easy to avoid complaining yourself. The world can be an interesting place with interesting experiences. Sometimes you will feel like switching roles with other people, so that maybe you would be happier. But life doesn't work that way. The only way you can truly be happy is being grateful for the things you do have. 3. They don't have hope. Hopelessness is a recipe for unhappiness. Take a good look at yourself right now. Do you think you will be rich? What's your response? Be honest. Is it yes? Is it no?  You are your greatest asset. If you can't believe in yourself, you can hardly convince other people to believe in you. 4. Acquiring things in hope it will make you happy. Have you ever spent a year saving and making money to deal with a breakup? What did you gain in the end? Did it make you happy? Happiness isn't something that material things can bring. You can't buy your way to happiness. If you are not at peace, you will never be happy. You will, in time, come to realize that buying things and giving them away will make you happier than keeping them without using them. The feeling that you are making someone else's life better is a comforting one. This may explain why billionaires give a lot. 5. Exaggerating events. Everyone goes through tough times. You have every right to mourn in your own way if you lose someone, but you need to try to keep it simple for your sake. When you start to think about all you could have done with a person and all the memories, you may not recover at all. Getting fired too is a sad event, but try to treat it as one of the things in life. You will preserve your happiness that way. Some people take bad news very badly. They can feel so distraught and stay silent for days. If you want to be happy, don't react this way to bad news. 6. Staying home. There is nothing wrong with staying in, but when you do it too frequently, it can create challenges. Going out means you mingle with other people. You interact and feel less alone. Social people tend to be happier than lonely people. Human beings are social animals by nature. If you don't socialize, it can have an adverse effect on you. When you are in the company of your friends, you can also share what is going on and receive useful insight and feedback that can solve some of those challenges. 7. Striving for perfection. If you keep striving for perfection and everything, nothing will be done on time and you will be very unhappy. People are not perfect. They make mistakes all the time. Having this type of habit will make you impatient and intolerant.  According to a positive blog, setting the bar for your performance at an inhuman level usually leads to low self-esteem and feeling like you are not good enough, even though you may have had a lot of good or excellent results. You too are not perfect. You are human. Hard work may not always give you the results you want but decide to make it a way of life. It ultimately does. People who are successful today are very experienced in feeling. They have a patient attitude to life and try as much as possible not to worry. Everything will not always work out. Making peace with this reality is key. People who make up their mind to try nonetheless are usually very successful and happy. 8. Hanging out with sad people. Anyone who desires to be free from unhappiness also has the cut of ties with people who are unhappy or at least make them unhappy. Some people are toxic.    Such a friend is not suitable for you.  Our enemies don't have this power over us because we expect them to attack. Only your friends can really hurt you. 9. Overthinking about the future. Let the future worry about itself. Worry about your present only. It makes no sense but it can save you from unhappiness. How many people do you know who worry and are happy at the same time? The answer is no one. Worries and happiness are mutually exclusive of each other. If you have one, you can't have the other. Spending much of your time in the future and imagining how things could go wrong at work in your relationship and with your health can create self-doubt and build into horrifying nightmare scenarios and playing over and over in your head. Be in the present. Enjoy the presence of your friends, your family, your colleagues at work and your youth. No minute that passes ever comes back. 10. Comparing yourself with others. This is a destructive habit. What do you gain from doing this? Nothing. Absolutely nothing. People are always going to be people. There is always someone who is going to be richer than you. There is always someone who will be smarter than you. Taller than you or more good looking. Learn to love yourself just the way you are. 11. Recalling the past frequently. Another habit that keeps people unhappy is continually thinking about the past. The past is gone. What you have is before you. Why not make the most of it? Sadly, not everyone gets this. To be able to make progress into your future, you need to make peace with the past. If you used to be the least in class, let it go. You create your own reality. Don't let the past get in the way of your great future. 12. They accept other people's negative opinion about them. No one can impute on you what you don't accept.  You only start becoming unhappy when you accept what they tell you. This is why your mindset is so important. If someone says you are incapable of doing a job, prove them wrong. Don't argue. Let your results speak for you. The more you assert yourself, the more confident you are going to be. Don't ever let anyone's opinion put you down. What they say is their business. And not yours. Smile. Relax. You have it all on the control. 13. Being selfish Unfortunately, some people think this is supposed to make you happy, but it doesn't at all. You see, when you focus on yourself, you also exclude others. You concentrate on gratifying your desires and in the process hurt your loved ones when they withdraw. You will be left alone and unhappy. Being selfish is self-destructive. Selfish people don't keep friends for long. If you want to be happy, you should avoid selfish tendencies. People hate the feeling of being used that may not forgive you if they find out you exploited them. 14. Holding back because of what people may say You have to realize that people don't care too much about what you do. They have their hands full with worrying about their own lives and what people may think of them instead.  You will be happier and more fulfilled this way. It is far easier to gain acceptance with your success in your area of interest than to justify mediocrity. 15. Making everything complicated Try to simplify things. Life is not a war zone. No one is out to get you. It's all in your mind. Sleep well, eat well and rest. You are going to be just fine. 16. Having too much stuff to do at the same time If you haven't had a presentation to prepare in one hour before, you will not really know the stress of working really hard and feeling lost. My advice, don't take on too many things at the same time. Take a break, rest and continue. Always say no when the deadline is too close. If you lose one client, you will always have another one. Trust that you are making the right decision every day and reflect on them. Continue to smile and be happy. Life is not that serious.