like to now look at the, we will focus on the CO2 in more detail. So, a Japanese scientist Kaya proposed a simple identity to try and see what our options are with respect to CO2. The way this is done is, you take the total amount of CO2 either in the world or depending on if you are talking about a country, let us look at either a country or the world as a whole. If you look at the total amount of CO2 emitted in the year, we can write this as CO2 per unit of energy into energy per unit of GDP into GDP per population into population. This is called the Kaya identity based on the Japanese scientist who proposed it. Now, the advantage of this is if you look at this, this is called, this is the carbon dioxide per unit of energy is the carbon intensity intensity of the energy sector. So, for instance, if I have a state like Karnataka or Kerala, which has a large part of its electricity coming from hydro, the carbon intensity of electricity for that state would be lower than that of let us say Maharashtra which is predominantly having a large part of its electricity from coal. So, this is the carbon intensity of the energy sector and the other factor this energy per unit GDP is called the energy intensity of the economy. So, the question we need to see is that when we want to, if you want to stabilize the CO2 emission, now we have limited amount of things that we can do in terms of population stabilization. GDP per population is usually a measure of how we would like the affluence, we would like the services and the goods, the quality of life typically increases with GDP. So, we do not want to reduce the GDP per population. So, if you look at this and this, these two are things which are sort of outside our control, we basically will focus on reducing the energy intensity of the economy or reducing the carbon intensity of the energy sector and these are the two things that we can focus on as we go ahead. So, as we said, this is the CO2 carbon intensity and the energy intensity. Now, the question we can ask ourselves is does a country that has a lower energy intensity per unit GDP imply that it is more efficient. Think about it, there are two countries, let us say India and Singapore and India's energy use per unit of GDP is higher than the energy use in Singapore. Does that mean that India is more inefficient in terms of energy use than Singapore? So, the answer to this is that it is not necessary. Energy intensity depends on, if you look at GDP, GDP is the total value add of the goods and services produced by the economy. So, if you look at the economy, different economies have different kinds of output. So, if we look at an economy which has a large amount of agriculture, economy may have a large amount of industry and economy may have a large amount of services. Now, industry by its very nature and if you look at heavy industry is energy intensive. So, if you want to produce steel, you want to produce cement, you will need a minimum amount of energy to do that. While if there is a country which is only importing steel and cement and is mainly focusing only on the service sector, it will have a lower energy intensity. So, if we look at this graph, this is a graph from the energy after Rio in the 80s, you will see that as over time if you look at the energy intensity of GDP, if you look at US, initially there has been a rapid decline and this is because the country develops from traditional energy use sources which were very, very inefficient to modern energy use and then it again keeps on declining. A lot of this, some of this decline is because over a period of time it does not require many of the energy intensive materials. So, steel, cement, it saturates the demand for steel, cement and energy intensive materials saturate and most of the growth and the GDP then starts coming from the service sector which requires much less energy and of course, there is another trend where basically industry has moved out from US and Europe and it is predominantly now in China, a little bit in India, a little bit in Brazil and other developing countries, but China has taken the brunt of all of this. So, if you look at the energy intensity, instead of the energy intensity, if you look at the energy used per person and we look at the GDP per person, you will see that beyond the point with increasing GDP per person, the energy used per person stabilizes because beyond the point, there is no additional amount of energy that you may need and so this kind of there is a growth and then there is a stabilization and this trend also, if you look at energy intensity improvements, this is a more recent graph, you can see that all countries are going through reasonably rapid improvements in energy intensity and energy intensity per capita also going in that fashion. But just to reemphasize the question, the reduction in energy intensity has two components, one is because of efficiency and second is because of the structure of the economy. So, if the economy changes where more of the GDP starts coming from the services sector, this will result in a energy intensity reduction. There are methods by which using decomposition analysis, we can see the effect of both these parameters. So, just to give you a snapshot in time, 1985 and 2010, we can see the over the 70s and 2000, you can see that agricultural share in our GDP has been declining rapidly, services share has been increasing and industry share more or less has been remaining sort of constant. And so this services, so with the result that we have had a decrease, significant decrease in the energy intensity, we have also had significant improvements in efficiency. So, if you see this, this is a tri-plot which shows agriculture is zero here and agriculture zero to 100 is on this side, industry zero to 100 is on this side and services zero to 100 is on this side. So, if you see different countries, you will see that China has grown where it has increased its industry share and in the Indian context, the industry share has been more or less constant. But we have actually, we have the option of either going high services or high industrial growth and this will have an impact in terms of the kind of energy use that is there.


I feel like when I first got here, I wasn't as confident I guess in myself, but like with the professors I've had, they've pushed me and made me realize that I'm good enough and I can do anything.


How to actually isolate those terms and talk about what each one of those means to us, at least, in a certain way. So first, I just want to introduce both of us. We're consulting that algorithm. My focus is almost entirely on infrastructure performance, scalability, so complete projects, standing issues, infrastructure tuning, really the whole range there. AQUIA for a little over three years now. So my background, maybe to point out specifically to some of you guys, is actually more from the systems side. So I've done group development for a little while, but I actually come into this from the systems administration side, automation. And I'm Jeff Beeman. I'm also a senior technical consultant with AQUIA. My focus is not performance and scalability on a day-to-day basis. And sort of the reason I'm here is to talk about why that doesn't matter, because I have to care about performance anyways. I joined AQUIA in 2010, so a little bit after Eric, I've been working with Triple for quite a long time. And my primary role on projects is usually as a lead-army type. So in that role, I have to care about everything related to the triple site, including how I've been helping just for systems, or just for developers, or just for kind of that final long-shape. It's really something everyone, everyone's going to write tests, but everyone's going to be a part of it. There's a lot of things that go into the plan, and kind of go into these a little bit more, but really important things, like making sure you're representing what the users are doing. Not what you think's important, not what's first or last to the project, but really what is actually going to make the site money if that's your goal, or get the most information out if that's your goal. Whatever the purpose of your site is, that's really what you want to make sure people are able to get to as quickly and efficiently as possible. Making sure that those tests that you're writing, you're actually scripting that in a way that really similar to what a user is. It will come back to a kind of variability in kind of this, but making sure that if the user is performing some kind of path for you to site, they're not going to perform the exact same path every time. Make sure that you're seeing things a little bit differently.  And lastly, what are we going to do to actually analyze those results? So it's really easy to look at those numbers and say, well, it looks good enough, or it was better than before, or as soon as you can actually look at those results and know what the biggest thing is, performance testing is for everyone. So I actually want to say in the kind of abstract for this session, I actually kind of put a call out to all those different roles. So I'm actually curious, how many of you guys are, would you say you're developers first? How many of you guys would you say are like system ends back in? I think that's actually, the PMs, you actually want to really kind of feed that from at the end and kind of reach out to. Because I feel like developers, they want to learn the tools and the learning processes, but ultimately it's all about getting the client project help as a whole. Before we get too far in the discussion, it's really important for us to level set on some terminology and talk about the various things that go into performance testing. Because like Eric was kind of hinting at, performance testing isn't just load testing. It's not just loading a page on your site and seeing how fast it is or finding slow queries, things like that. There are lots of aspects to performance testing. So we're going to talk about some terminology and some of it you might use different words to talk about these things, but it's important to have clear lines in your minds about what some of these things are. Performance is about the speed or the time it takes to do an individual request. This can be both from the time a user requests a page to the amount of time that it takes for that page to come back, but there are also other things that go into measuring performance on a page. How long does it take to render? How long does it take for all the assets to come back? Are there JavaScripts executing on the page that make it take a long time to actually fully render? So that's performance. Scalability is the ability of your site to go from one user to 100 users or 100 users to 1,000 users and still have a predictable performance threshold. And predictable being like not the same, but 100 users hitting your site is probably going to perform a little bit differently than 1,000 users hitting your site or 10,000 users hitting your site. But it's acceptable and it's predictable. That's what scalability is about. There was a lot of discussion when 777 came out about how 777 was slower. And the whole idea with 777 though was that it could scale much further than higher versions of Drupal. So just because an individual page request is slower doesn't mean your site is performing worse. You might just have to give up some speed on an individual page request in order to be able to handle an enormous volume of the page request. Next term is scaling. So scaling is probably a concept most of you are familiar with. It's probably the area most people start with when they're looking at performance. Scaling is saying we want to take our two small servers down in the corner there and we want to make them bigger. That's scaling vertically. If you scale horizontally, that's taking your two small servers down there in the corner and just adding more of them. So load testing is about focusing on real-world traffic problems and seeing what happens to your site for measurable amounts of traffic. So in this slide here we've got 10 users getting our site and our server is happy.  It's having a little bit of a hard time. Load testing is not about breaking your site. Load testing is saying, we're going to throw X number of users at the site and we're going to measure and see what happens. Stress testing is about seeing what happens or how far you can push your site before it breaks. Other people call this like a smoke test and stress testing is about saying how many people can we throw at the site before it crashes and burns.  But a stress test is taking a scale and putting people on it until the scale breaks. Last one that we're going to talk about and we're going to talk about a lot, we'll use this word a lot, is contention. Contention is a fight over resources and there are lots of places that you can end up having contention in your site. We found this to be the hardest thing to draw a diagram of and it's probably the hardest thing to talk about but it's also the most important concept to understand when you're thinking about performance. I think one way you can think about contention is to think about wallmarks on black writing. So a whole bunch of people, a mass of people line up at the doors and then they open this tiny doorway that's about four people across into the store and everybody wants to get in at once and there's a fight over the resources both to get in and then there's a fight over the resources inside and there are many points of contention inside that store. Contention is a big thing that will scale at some kind of measurable rate. You go from one server to two servers, theoretically you should be able to take twice of the request rate. Contention is a place that comes in to say, well you could do the first server just because that's where your contention lies and then the second server doesn't actually help the problem at all. So that's why it's so key that if you can identify where the contention is, that is what is going to block or reaching those goals from reaching that next level of performance. We've got a couple slides, a little bit later if contention is the concept you don't understand. That tries to explain it a little bit more, but for right now just try to think of it as a fight over resources. Hopefully help you understand some specific examples. So let's talk about what your tests are actually meant to do. So we kind of are breaking up this presentation a little bit into what you should do and also kind of lessons learned from projects we've worked on in the past and what we've actually done that we've failed on our projects or succeeded and did really well. So certainly coming out from both perspectives. So the key is that if your tests aren't good, the numbers you get from them are essentially irrelevant.  The quality of the test is really the valuable piece. Make sure you're setting technical goals. So don't look at this as what makes a user happy and trying to guess what that is. Actually try these things over time. Look at when you add each of your ads. What percentage of users is there? Is the site getting, is the low-test showing it's getting faster? Is the low-test saying it's getting slower? What's actually happening over time? And what are you doing to actually monitor that change side-by-side? If you're not looking at a side-by-side, it's really easy to get three-quarters of the way into a project and really have no idea what that model and that's usually where you find the biggest amount of time spent on two of these things is you're so far down the line of a project, you end up throwing more human resources to solve the problem that realistically you could have caught much earlier in the process by just taking it step-by-step. There's no regressions that are actually happening. So regressions from the QA, from the typical QA sense, so features aren't breaking other features. It's just as easy that new features could worsen the performance of other features. So anything that you're thinking about in the QA of what makes a good test, how much you're testing, how wide that opportunity is, you should go right into performance testing. It's really not any different. I think this is one area where my sort of non-performance experts opinion into the conversation is important. My role on projects is typically leading architecture, working with partners who are building features for our clients and we've found, I've made mistakes on this stuff just like everybody else, we've found that if we don't do performance testing throughout a project it can be any weight until you've built a massive project that's very complex and has lots of moving pieces. It can be really difficult to impossible to identify where your site is slow. But if you're constantly monitoring performance and as you add features to your site, you're constantly testing them and seeing how they perform. You can set yourself up for a lot more success and also reduce risk of having to spend a ton of money at the end of your project. A ton of money or time or resources at the end of your project trying to figure out why your site crashes and burns when you launch it. Another good thing to keep in mind is when you build a feature and this goes back to sort of the data build a feature and like let's say you're building an article content type. It's really simple. Everybody probably does it on their sites. You don't want to just make a few of them and refresh the page and see how long it takes for a page to load. What happens when you make a lot of them? What happens when you make the number of articles that are probably going to exist in this thing? Those are the kinds of things to think about when Eric is talking about monitoring changes to your projects and thinking about ways that users are actually going to use your site. The system inside of this, the system's engineer is a lot of these tests that run early on in the project and maybe they're bad tests maybe they're kind of exaggerated whatever they're showing. I've seen a lot of projects create all kinds of crazy complexity in their infrastructure because they have one bad test early on and it's really easy to see that and just start scaling out horizontally doing crazy things starting databases early in a project or these really complex ideas that it's so early in the project you can actually fix these things periodically. It's much easier to do that than actually create this complexity and not only at the end do you not have tests to verify what's going on you have an infrastructure that you probably can't even maintain so you've actually created twice that complexity just by not keeping up with things that were talked over the whole course of the project. And also when we look at this one thing that I always do going into client sites I might do a lot of one-week short-autotype engagements because I'm usually kind of committed against on the one side the developers on the one side the systems because those of them are always ready or the other ones are always wrong or it goes always wrong if you ask someone else to get those numbers it's amazing how that changes these two groups because the infrastructure folks can make a quick change see what it does if it's good they leave it if it's bad they flip it back developers the exact same way so just having these kind of indisputable numbers to work off of it makes the entire working relationship of bringing together these very different teams makes that whole process much easier. Talking about the quantifying aspect of this one of the worst things I see when it's actually a stress test we want to see how many thousands of users we can handle if you're only ever going to have a thousand of those you're going to be making all kinds of crazy changes for a situation that's not going to happen what are you actually trying to achieve by these low tests is it a certain number of users is it making sure that your servers aren't calling over is it making sure that each request is only taking a certain amount of time really think about what it means to actually do a successful performance test it's really probably the most common thing is X number of users well that's great but if X number of users are performing badly is that a success or not if they're getting all the results they want but it's taking 5 seconds they're not going to be happy but you haven't defined them you don't know what's a success and what's a failure so it's really important to look at each one of these levels and actually find out what you can do to look at the results of a test say that was a success or that was a failure and move on from there if you don't have that you get lost in the numbers that's where the blame comes in we support the users but it's slow to cause that really set these up at the beginning and I think this is a good example of where the project manager role comes in is maybe it's not to load on maybe that's not what's important to the project but you want to be able to support 5,000 users at a time with under 3 seconds per user and no users over 5 seconds something like that where you can put in a full solid sentence and every time you do a load test go back and just say yes or no and everyone's under the same understanding the developer is a systems everyone can look at those results and know if they're moving ahead on the project or they've gotten ahead of themselves one thing that I want to throw out here is make sure you're gathering all the data so if you're the client don't trust yourself if you're the consultant don't trust the client if you're the client also don't trust the consultant that's the whole point of getting numbers if you're getting numbers that's what you're arguing against you're not arguing against people or roles or responsibilities the whole point is to make sure that whatever the goal is at the site that's what you're accomplishing does it feel slower? that doesn't really matter does it feel faster? that's not really what that defeats all of us doing the test and getting the numbers one thing that's key is if you're actually looking at value so one thing where performance is very different than something like security is one small security issue can really escalate very quickly one small performance or two certainly has that risk but it's a little bit different 80% of the traffic is one user path that's a great place to dive in and try and find the most the most important change you can make in a short amount of time making sure you understand the full use case so when you're not sure what that full use case is if you think it's doing a search, maybe it's doing a search and going all the way through your purchase don't think about individual user actions think about what you're trying to get to because if you're building an e-commerce site you want to know the time when someone gets to the site to the time they make that order if they oppressor it anywhere in between you've probably lost that order so make sure you understand what the entry point is, what the exit point is and what's really making a user happy and successful in this site and also I think this applies to anyone anyone, any of you guys that have taken the time to comment or put on comment this session you're all smart people don't think that you've found numbers that aren't right if you find numbers that you don't think are right write a different test and see if it comes out the same way and don't write one test question it you're all smart people, write the good test talk to the right people see what you can do about getting those numbers where they need to be and then finally move on you're all smart people just make sure you're actually utilizing I think one thing that we try to do it's funny but it's true like the first bullet point never trust the client but there's also always trust your intuition and never trust your intuition always trust yourself never trust yourself and sometimes you do need to trust the client it's kind of like when we were talking about it there is no screwing right or wrong you have to constantly be questioning your assumptions and your intuition the client's assumptions the client's intuition but then you also want to always trust it too if the client is constantly telling you saving an article page it's slow and you can't replicate it there's something going on there you just may not be able to replicate it you may need to dig deeper you may need to go watch them create an article and see what they're doing they may be doing something crazy or they may be doing something that you never tested because every time you test creating an article you just go ASDF we just say we all do that right the other thing I wanted to say about this particular slide is around the important metrics that Eric was talking about where if you've got if your goal is that anonymous users are able to view pages in less than 2 seconds no more than 5 seconds or at least 2 seconds no more than 5 seconds and that is your primary goal if you're focusing all your attention on why saving article pages is slow you're not putting your effort into the right place if there's like 10 authors on the site and they have to deal with the pain of saving an article page that's not really like the business case you have to solve you need to solve the business case for the end users if that's a more severe problem I can't mention e-commerce just because that's kind of the easiest place to see a visiting user has a particular value to the site every site has some value from a user whether it's getting them into some sort of business just making contact with someone e-commerce is a great example of your goals for someone to buy it buy whatever your product is buy whatever your service is it's really easy to monetize that and look at the actual value so I think e-commerce is just a great simple case but I'm sure each one of your websites has something that you're trying to give to clients or get the other thing is don't give up until it's done this is kind of a mantra I use but most importantly don't think that maybe you're newer to Drupal but you know the system stuff but again you're all smart people it's a system stuff that looks good dive into Drupal see what happens get your hands dirty if you need to write a core patch just fix something I was on work on a client that has 5,000 nodes and I think 5,000 taxonomy there's a type of chocolate taxonomy access control and they were building out these huge queries where the query from their perspective each query is taking two or three seconds so they spent probably a month tuning the database putting on ridiculously big hardware doing whatever they can to try and fix it what they didn't do is realize the query was very fast it was taking Drupal 4 or 5 seconds to build the query so it was really easy for them to look at oh it's a slow query let's go one off from here but if you really get your hands dirty and realize what part of the page is slow not just trying to articulate what the big component is that's slow you can actually dive in and look at the profile of the page and save weeks of effort don't get lost in here it's part of a larger ecosystem in your company and Drupal is just one piece of it you might not run into issues that the actual in-user would see so one example of this is a consumer electronics company in New York they were running load tests on a Drupal site no problems at all barnaces working perfectly fine no issues at all what was happening was when the users were coming in they were going to the main.com site that was a Drupal company that happened to match the configuration they were using in varnish for this actual Drupal site so the only way to recreate the problem was to actually go back and think what's the entire user story they go to the.com they hop over to the blog they click around that's the user story it's not just getting into Drupal so their entire fix was to change the cooking their site went down multiple times on the block Friday I mean every horrible time you can think of for a consumer electronics company and it was about 2 or 3 hour change I mean don't get lost in thinking it has to be Drupal because Drupal is slow it might not be Drupal itself it might actually be some external service it could even be a web service whatever it is but Drupal site has other issues and those are all those other people out there make sure everyone's checked the easy stuff so everyone has a great infrastructure team everyone has a great development team but you're going to forget little stuff you're going to realize that if the servers are slow maybe varnish is just missing things you don't think it is because when you click around it's fine but maybe it actually is causing an issue and running tests throughout the day throughout the course of the project you'll actually catch these things and last and this is really key at least from my perspective don't accept anything less than perfect so we talked about the article node type entry that's certainly talking about priorities could you launch with that? of course but in one case I've seen a lot and this was a non-profit education customer was their site would just crawl when qualms were running just absolutely crawl and their solution to that was to see us thinking out why? why don't you ever need that so needless to say they had some slight side effects on that but that's because it wasn't user-facing so they said whatever we'll turn on so these are kind of things that if you really start shipping away at what you're considering perfect what's really making your site work you're going to end up with this just kind of down on a spiral and it's going to take you months to come out with some cool stuff I think along those lines as well something might work but there might be better solutions to it I worked on a project last year that really used feeds and lots and lots of feeds in that site and theoretically those feeds could be processed through ground that's how feeds does its thing but when we were talking about that literally thousands of feeds we had to be more creative about how Drupal processed them because if we ran Drupal Cron every minute we get all this overhead of other stuff that happened so we were able to run those feed thru thrushes through thrush itself and that was one case where it was acceptable but it wasn't perfect so we wanted to make it as good as it could be so we needed to speak up a little so this set is talking about infrastructure so one very common mistake we see is using your dev environment for testing so dev environments have a lot of overhead like you're running additional things that aren't being run in production like xdbug or xhdrop you might be using watchdog and dblog you might be logging to the database so that you have easier access to log messages in dev dev environments are typically on congested networks as well so your dev environment might be on the same server or in the same sort of network as a whole bunch of other dev sites and it's not going to give you a realistic view of what performance looks like so the other thing is to not extrapolate results so in this diagram this is a typical testing setup in the next step we're adding more users and we've scaled up our hardware a little bit and the db seems fine last time but in this case we went back to Eric's earlier example where things were fine with a small number of users but when we add more users we start getting contention and it doesn't matter that we scaled up our servers it doesn't matter that we made our web servers bigger that wasn't the problem in this next step we doubled the db size and we added more servers and now we're better so the whole idea here is that don't assume that the results you get in one environment can be extrapolated to the next step similarly, and this is again related to contention we do a small load test that might be really hard to see so I'll try to talk you through it the blue box on the bottom is the locking query a query that does something that causes a lock in the site it could be locking a table it could be Drupal putting a lock using a lock itself something causes a lock in the process and then we have a couple queries waiting but in the small load test we get about 20 milliseconds of overhead for that last query but as we add more people or we start waiting for that lock to clear stuff piles up and you get basically a traffic jam on your server and that's contention again so our originally what was a 20 millisecond overhead for a query, you start piling additional queries on top of that and you get more and more of wait time if you want to say that if you look at the first example you're really going to look at averages and begin with your test and you're not going to look at how much the slowest query was and how much the fastest query was if you're going to look at this test and realize that this was a third of what and as you actually stack these up because the other locking queries we're going to constantly add that overhead 20 milliseconds in 30 then 40 then 50 again it doesn't matter how many requests are in there, they have to take at least that amount of time so no mad ones, there's a lock and you have to start understanding what that build-up of overhead is to actually realize what the extrapolation would look like and when you really do a couple of these the extrapolation just doesn't work unless there's no points of contention which if you solve that problem I'd love to solve the other ones a good example here is when you clear Drupal's cache you have to start understanding what that buildup of overhead is when you clear Drupal's cache and the next page request that comes through let's say you cleared the menu cache the next page request that comes through has to rebuild that menu cache it doesn't matter if just one request is coming or 5,000 requests are coming you have to build the cache and you only have to build it once but all 5,000 of those requests need that same thing that's being built so they're all waiting on that one thing to be done so this next one your numbers are only as current as your last test so you want to write smart tests don't just write a lot of tests saying that you want x% of test coverage isn't always the best approach what you want to do is write smart strategic tests that truly target your use cases on your site so talk about user scenarios work with your business work with the client to understand what is the path the actual path through a site like Eric was talking about that consumer electronics company if you didn't know that users started over in this other part of the site before they got to Drupal you would never run into the problems use data from analytics this is one area where you never want to trust what your client is telling you about their scenarios for the business get real data from analytics that tells you what the actual most visited pages are or what the most common path through the site is it's not that you don't want to test the scenarios that the client gave you you just want to use real data to inform what you're testing you want to ensure variability so change what you're submitting in forms change the path for a user, change how they navigate to a particular page so you want to also measure test coverage so both components on your page as well as the total page count and as you build new tests or build new features build new tests make sure that you're testing the things that you're about to launch gets to our next big point this is one we see all the time all the time is that your tests aren't about launching your site you don't write tests and run tests just to launch your site you want to be building tests after you've launched your site because nobody's nobody in here I imagine nobody in here launches the site and then just brushes their hands and walks away from it one guy yeah every time you have a new feature you are changing your site and you need to test performance again you can have dramatically unintended consequences when you have new features and you could add any feature over here that does something you didn't think would affect at all user performance over here but it does so testing is about QA and QA never stops so you're testing to either way the one other note here is that when you launch launching is about scale about meeting the scale of the site when it goes live doesn't mean that your test patterns change so continuous integration is for performance as well as continuous integration tools like Jenkins provide mechanisms for you to do all kinds of things that aren't just about building and deploying your code so these tools like Jenkins a lot to do testing so ongoing changes to your site ongoing content entry into your site things like that can affect performance over the long term and by using a continuous integration tool that tracks results and measures results from tests you can see over time trends in your site performance and that's really important because manually running tests every once in a while you may completely miss the fact that every day for the last three months performance has gotten a little bit worse you may not be able to see that but if you can go into Jenkins once a week and take a look at that graph and see that it's going up you can proactively take care of the issue before it becomes a problem also when you're really thinking about more of the kind of pure sense of continuous integration not only are you seeing it over time but usually you're also seeing it when new code is created so you can actually have that correlation of the performance got worse and oh yeah, these are the last three code commands so you're not having to trace back your whole site and figure out where that happened you actually have all that data in one place and you know who to go talk to looking at these milestone tech tests where so much has changed and it's really hard for you to keep that bottled up and understand all those different steps so the last thing and this is really kind of the one piece where we'll get a little bit more technical it's talking about the different types of tools so we talked about load testing and that's kind of what most people think of but when you're really talking about performance testing that bronze a little bit there's a little bit more than just one that you can talk about so make sure you're doing multiple types of testing so it's really easy to look at that final in number of a load test and just use that as your solid single number that you're going to work on it's great and ultimately that's what the real end users are probably going to experience but there's plenty of different levels that are actually going to speed things up for you to work through the process so everything from request profile so for the developers to do things like exit drop or XD bug where you're really looking at the individual pieces of a page load and trying to find those problems that goes back to the the customer I was talking about that had the massive tap light queries they went to a certain page and saw bad performance they jumped on the page they didn't quite jump all the way down in the profile to realize it was the one so maybe use this for investigation and not really as a benchmark but it is the first step while you're doing development service testing so service testing is the way I look at this is if you getting a Drupal request a single page you're touching a patchy or an X you're touching my SQL you might be touching the memcast you're touching the network you're touching all these different elements but it kind of manifests as the time of one request service testing is thinking about how long does it take for a patchy or to run this query on my SQL maybe there's some network lag between my SQL and the web server you're not really going to know that unless you test it separately looking at something like Mib Slack or Mib Cache D all these different things where you're really testing each level of the stack and even testing that work itself these are the kind of things that are going to make sure that you don't make some false assumption about that one and the last two I think are a little bit harder to sort of compare but when I look at a simple response testing meaning something like a patchy bench or siege or services like Condo where you're going to get one single raw number of what the performance is the amount of the amount of opportunity you have to react to that and sort of simulate what it really is you're seeing can be sort of in this so I look at this as something that you really use for monitoring and kind of real quick gut checks but ultimately of course load testing is the plan so we really are starting to jump in quick anecdote I worked with a client who we were seeing very slow performance on queries to the database but we couldn't explain it because we would go look at the query a lot the query that was supposedly slow and we'd go log into my SQL run the query and it'd be fast and Eric kind of hinted at it but some places that we probably never would have checked until we sort of stumbled upon it was the latency between the web server and the database server the time to make that connection and then send the request over and get it back was what was slow and that's what something like that is what service profile is about as well where you would even develop query log may not tell you that that's a problem or you'll see weird things like this query like a cache get which cache gets should be very quick it was slow, slow, slow fast, fast, fast, fast, fast and then one of them was slow and another one was slow that can hint at things like so load testing we split into two different types and I think this is really a key thing from every level to understand what the end goal is first I look at what is usually called virtual user load testing and the goal here is really just to send a request out, get that response back and made a reactive usually it's just a simple transaction a lot of complexity to it or a lot of really interactivity the cost that I have down here is why that's relevant at the minute but these sorts of requests are cheap it's designed to be efficient you want to send out 5,000 requests for the least amount of resources possible that's your goal here maybe this is for stress test or whatever the role is it's really that's a specific purpose secondly I look at what I look at as a real performance real youth load testing so this is using something like Selenium using something more browser based where it's actually loading the page as if it were an end user performing actions with mouse or the keyboard actually doing what the user is doing and getting those results and if you look at the cost difference the amount of resources required to do it is a massive difference and connect some more is usually a common gap a full browser full user kind of full sign off of if this is going to work or not versus I sent out an HTTP request I got a bunch of texts back and now I'm going to do something with it they're naturally very, very different just for instance for virtual users using something like Jmeter you're sending out a bunch of these requests maybe using expat there's some kind of regular expressions to pull something out you read that, you make an action on it Selenium, you're really looking at what is a browser going to do to load this, right? Is there blocking the JavaScript, right? That's not something you would see in a virtual load test or in a virtual user load test you really have to look at this and understand what the actual user is going to see and how browsers can interpret it and maybe how you slowly actually see differences there as well based on the client and I think this gets to a really important part of performance testing that doesn't get talked about a lot except by, you know, people who are sort of at the cutting edge or do maybe like a lot of mobile work is that front end performance is really critical and it's something that as developers, like back end developers and sysadmins we don't like to think about because it's like, well my server is serving up the stuff just fine, it's no big deal, right? My job is done but front end performance can dramatically affect the end user experience. You can get a report from a client that tells you that the site is slow and everything on the server side points to the site being fast if they're waiting to download giant images or there's JavaScript that blocks that essentially blocks rendering of the page for a certain amount of time or a bunch of analytics stuff that's going on that's usually the most common problem if you've got like 10 JavaScript libraries making all kinds of analytics calls that can make your site slow but you wouldn't see that on the back end look at this as kind of a holistic effort of developers have their role in it QA testers have their role in it sysadmins have their role in it but each one of these really fits the whole process so it's really key that everyone's involved and everyone's under the same understanding and use those numbers to make everyone's life make the communication a little easier to hope some people have some questions that put the mic here just while people are going up there I want to mention we are hiring but I think specifically the kind of stuff that we're doing here this is what people like me and Jeff do on an everyday basis so if this is the kind of stuff that we're really interested definitely come talk to us after this session we'd love to meet your assistant if there's any questions left over or things you think of later just reach out to either of us on Twitter we'd be happy to continue Hi, my name is Albert I have a quick question about the tool you're using in Jenkins to create those graphs those performance graphs what expression are you using in Jenkins for that that particular example is using Jmeter and getting the X unit files out of it or X also Java Jmeter Jmeter and Jmeter so it outputs kind of a standard data format and that's what it uses to graph that so it can actually pull selenium results the same way and a bunch of other services it's kind of a common format I want to test the IE so what I have is one machine that's testing with selenium Firefox and Chrome and then it sets if that passes it sets a different branch so that another instance of Jenkins running on Windows is going to test the IE does that make sense to you guys makes more sense than running IE at a certain point that is realistic right at a certain point testing that testing on IE and all these combinations that's what users are going to do so that's going to be the most accurate way to do it this was really nice validation for sort of the direction I've been trying to scale what it came toward I'm really curious what tools you're using to generate the numbers you talked a lot about it about evaluating the numbers but I didn't see too many examples of each testing there's obviously a lot of things and we sort of danced around it I'm just curious if there are a lot under your list that you can basically say here's the top five or six things we're using to generate those test numbers if you're there to talk about evaluating sure so for me request profiling XH process is the biggest thing that's what I use to jump into a project for service testing using just a basic static file for Apache for varnish, I'll actually set up basically a URL of varnish that turns directly and test that using Siege or AV or all those tools for memcash which is a tool that comes with pretty much any installation for memcash for the real user load testing there's things like SOAS though there's things like Keynote Africa there's tons of service out there really a lot of this comes down to almost end there at a certain point so as long as it's performing this role and that's why I kind of avoid it using too many product names or anything as long as you get what it's trying to do and you can sort of map that to a features list and really knowing what you're getting out of the product that's really the key here because there's plenty of tools that will give you essentially the same number that's exactly what I was hoping using because there's just so many options out there and it's nice to know that basically it doesn't really matter the tool you're using it's creating that measurable metric that you can see the trend and maybe even those testing tools you compare the two of those you kind of look at what one's doing versus another and try to see the difference that's the beauty of numbers is you know which one worked better so don't even think that one tool is the the all or false solution so you talk about your numbers are only as good as your tests and it really takes good tests how do you determine a test that's good versus a bad test I think a lot that really comes down to understanding some of these things of what's happening maybe a little bit outside of them or what could a user do or what kind of randomness is there right variation like if I go to the home page am I going to click on the top one every time no I may click on the top one the bottom one the third one the fifth one having that kind of variation that simulates what a user would do so as a good test or smart test I really think of what would a user do what am I trying to understand what would users roll out the site what they're going to do use analytics use these other tools to understand what that is and that's what a good test is so it's about figuring out the realistic actions that's where you're going to make money that's where you're going to get visitors returning that's the sticking point right like if you're running an e-commerce site and your typical user adds a few products to their cart removes the product adds another product and then checks out your test shouldn't be user goes and adds a product and checks out you know you want to test a realistic scenario and you can put randomness in there and that's the beauty of the automated testing is going to someone and saying well today do two products and tomorrow do 46 not going to be a real advocate for testing but if you can automate that and say I wonder what happens if there's 150 items in there or 30 is it scaling linearly is there some pop there and those are the kind of things that say if it's a good test or not is could a user go in or accidentally click buttons or maybe they've done this over a course of time and added all these to their cart really think about what the user's doing because that's what makes that user valuable to your site like I mentioned if there are any follow up questions feedback for us personally feel free to reach out to our either of us on Twitter look forward to hearing from you guys, thanks


What about the the run you took him in the mood needs we're going to walk up or something Yeah, we there was a run. There was a run that we used to do in my back with the Istan Mountain and I thought to myself He was on a ketogenic diet at this point, and I thought to myself.  I thought In the last five minutes in and we start walking anyway Keeps going five minutes long thinking won't be too much longer keeps going won't be too much longer keeps going The whole way up then starts sprinting halfway through what I'm thinking this fellas off Got a few young lads around him as well Then he's pushing them on till I was like right rear We were at the top now once who's there and he went is this where everyone stops you I went Yeah, he went the gypsy King don't stop here Then he went we're going all the way up to the top See just had to do that a little bit more than everybody else and I thought there's something about him not boxing ability But it's something about him in there and up there. Yeah, that's what separates the week from the strong I believe though that that's how he's made the combat is done to be the most spoke about boxer on the planet just now


Well, good evening. This is Patience on the News, another edition we'll be going on now for 14 years. We've had many of the most famous public figures in Maine and also beyond Maine on this program. And tonight we have one of the most famous and talented figures in the Maine political world and beyond. William Cohen, former Secretary of Defense William Cohen, former Maine Senator, former Maine Congressman, Mayor of Bangor, Maine, and member of the School Board. We won't forget that. Welcome, Secretary Cohen. Harold, it's so good to be with you. Thank you for inviting me. And did you like the fact that I remembered you were on the Bangor School Board? Yes. And I forgot to mention I was an assistant county attorney. That's the only thing, that's the... Oh, I did forget that. Well, I want to just, for some of the younger people that are watching tonight, I just want to briefly summarize in a minute or so a couple of minutes your history with this state and this nation. Secretary of Defense Cohen is a graduate of Bangor High School and Bowdoin College at both places. He was an outstanding student and an outstanding athlete and an all-state basketball player.  He's a Republican and he was elected to the U.S. House of Representatives in the Congressional Campaign of 1972, took office in January of 1973. And among his other assignments, he was assigned to the House Judiciary Committee. Everything is important in this world. And the House Judiciary Committee a year later was in the midst of impeachment hearings. Question was whether President Richard Nixon should be impeached. And so the House Judiciary Committee was considering that question. He was, the committee did vote to recommend his impeachment and Secretary Cohen was among seven Republicans in those days and it was quite different than it is today. In those days, party had less precedence than the party had less importance than the country and what's one's conscience told them. So political party wasn't everything in those days as it seems to be today. In any event, he was one of the seven Republicans voting to recommend impeachment and he made this statement. And I think all of the young people watching this program should keep this in mind. He said he was asked to assess the conduct of the President of the United States, a President who he believed to be the best man to lead his country, President that he voted for. And he said he was turned out to be a President who allowed the rule of law and the Constitution to slip under the boot of indifference, arrogance, and abuse. So keep in mind what Senator Cohen said nearly 50 years ago, someone he voted for, someone of his own political party, who he said allowed the rule of law and the Constitution to slip under the boots of indifference, arrogance, and abuse. In 1978, he was elected to the United States Senate, serving as one of Maine's two senators and he served there for three terms. And those were the days when there was much more comedy and perhaps he'll talk about that a little later in the U.S. Senate. Our other senator for most of that time was a Democrat, George Mitchell. The two of them are friends. They've written a book together. These things don't happen anymore. But they did then and maybe Senator will comment a little while about where the difference lies and why. Among other things, while he was in the Senate, he was selected to be a member of the committee that invested the Iran Contra scandal. In 1996, President Clinton, a Democrat, appointed William Cohen, a Republican senator as U.S. Secretary of Defense. One of the two most powerful positions in the United States government. Mr. Secretary, many in our audience are familiar with Bill Cohen, the famous Republican politician from Maine. But don't know about your life over the past 20 years since you've left the Senate. Maybe actually more than that, 24 years since you've left the Senate. Tell us about the Cohen group, which is your business. Now a little bit about your post political life. Well, Harold, thank you for that very elaborate recitation of some of the things that I've done with my life. I appreciate the generosity you've shown in that regard. Since I left the Pentagon, I started something called the Cohen group. Oddly enough, the name came quickly to mind. But it was something I actually had planned to form as soon as I left the Senate. I had made arrangements to lease a building or office building downtown Washington. I had just two of my staff members who were going to join me. I thought about linking up with Senator Sam Nunn on a part-time basis. He was going to devote a third of his time to academics in Georgia, a third of his time to public service, and then a third of his time with me. Well, that got interrupted when President Clinton asked me to serve as the Secretary of Defense, but I had always planned to have something called the Cohen group and to deal with international law policies. And so today, I left the Pentagon, I guess it was January 20th, the new time, all the Secret Service personnel, those who were assigned to me, pulled out all of the equipment out of my apartment, and I was on my own. So I took one day off, and the next day after that, we opened up the Cohen group with Bob Tyra, Jim Bodner, and H.K. Park, and that was it from my days at the Pentagon. And we just started not knowing what we'd be able to do. And since that time, we have grown from the four of us. I think there are roughly almost 75 people in the firm now. We have offices globally in China, in Australia, also a small office in London, and a main office here, and so that's about another office in India. So we're in India, China, Australia, London, and basically here in the United States. What we do is we represent major companies who do business globally. We help them to understand the nature of the countries that they're either already in and want to expand or want to enter, and one of the barriers to entry and try to understand them. So all of my travels during the time that I was served in the House and the Senate, I was able to establish relationships with various government officials, many of whom rotate out of power, so the Rolodex gets old pretty quickly in this day and age, but as a result of all the travel I've done over the years, I have been able to build friendships. And so as a result of those friendships, I'm able to understand the countries in which these multinational corporations want to do business in. And that is what we do at the international level, and we also represent American companies here. In the process, we represent foreign companies who want to do business in the United States, whether they were from China, India, Australia, the EU, wherever they are, we have a pretty active Latin American practice right now as well. And so I have many, a number of Spanish and Portuguese speakers. And in each of the countries we are, we have nationals who speak the language in the country that we're in. So it's a really, it's a wonderful firm. I've got some of those talented people, I think, in the world. I've had four and three star generals in the firm have them now. I have four of the best former ambassadors. And I chose generals and ambassadors because they had a world vision and understand the dynamics of the world, economics, trade, and strategy. And have had major budgets, certainly as a four star general, General Joe Ralston, former Vice-Chair of the Joint Chiefs of Staff, former Supreme Allied Commander. We've had Nicholas Burns, who was number three at the State Department, and Ambassador Mark Grossman, number three at the State Department. And so I picked these individuals who look at the world and understand that we need to build alliances. We need to have friends. We need to develop strategies. Both at the strategic level, but also at the economic and diplomatic level. And so that's what I basically am engaged in. And I do commentary from time to time. I have an arrangement with BBC, and I do weekly commentary with BBC. And I go on MSNBC and CNN quite frequently as well, but mostly dealing with international affairs and military affairs. What do I think about what is taking place in Syria or Iraq, or the tensions we now have with China? How do we build a better relationship, a strong relationship with India and Australia? And these are the issues that I deal with, both on the economic side of things, business side of things, but also on the strategic. And you just hired somebody that our audience would be very familiar with, just come to work for the Cohen Group, General Mattis. General Jim Mattis has joined us. He was a young colonel when I was at the Pentagon. He was a young colonel in my office. I could tell how talented he was then. I also knew he didn't like being at the Pentagon. He wanted to be out on the field. And so he is a man of action. And I know he was chafing at the notion of being in the Pentagon, dealing with all the issues that one has to deal with from presenting matters to Congress, just the day-to-day management of affairs and the Secretary of Defense's office. And so I met him then. I admired him then. I watched his career grow. And then when he left service in the Trump administration, I asked him if he would join me. And he did. And so it's been a great addition to the firm. But he has joined General Paul Kern, who's the former head of the command dealing with all of the equipment and resources needed by the Army. I have three stars from the Air Force. And so we've had actually two stars, three stars, but four stars who are dealing with us now, including Jim Mattis. And you have Ambassador Tom Pickering, also a very famous ambassador, who's like you, a Bowdoin College guy, right? He is. He's not with me. He is with Madeleine Albright. Tom and I are great friends. And we've worked from time to time on issues together. But he's with Madeleine. Okay. Here's the competition. All right. Well, I have some other subjects that I'm sure all of the people in the audience want to explore. From 1972 to 1996, you will remember the United States Congress. And it wasn't in those days, those of us who remember, so contentious. And it certainly wasn't so nasty as it is now. So any thoughts on what's changed, why that's happened, a lot of people wonder about that. Why can't these members get along? Well, you have to remember that, you know, this is not always Shangri-La back during the days that I served. When I went to the house in 1973, I didn't go through politics as a background or profession. I went directly from the city council and mayorship of Bangor directly to Congress. And as a lawyer, I was really interested in the law, the subject of the law. And I wanted to be on the Judiciary Committee. And interestingly enough, everybody tried to say, don't go on the Judiciary Committee. It doesn't do anything. It will deal with abortion and prayer and public school. And those issues are volatile. It's not going to help you get reelected. So stay off that committee. But I felt I wanted to be on it. And I went to a very new program that was put up by Harvard. And they selected four new members of Congress. And we went to Harvard for about two or three weeks. And there I had Professor Moynihan, who then became Senator Moynihan. But Professor Moynihan is one of my professors. And John Kenneth Galbraith is another professor. And really, I don't recall anything of substance except the person who was running the program who has since passed away. He said, look, when you get to Washington, they're going to ask you which committees you want to be on. Make sure the committee want to be on a list that you put down some four or five committees, put it last. And I thought that was totally counterintuitive. And I said, why not?  So I listed all the committees that were very attractive. And I put judiciary last. And that's how I ended up on judiciary. And that's a long way of telling the story that even on the judiciary, it was not really geared to contentious issues other than the politically hot ones like abortion and prayer and probably school. But suddenly, we had the Saturday Night Massacre. And then the firing of a resignation rally of Richardson and Del Ruckel's house, the firing of Archibald Cox. And suddenly, we're in the middle of impeachment. During that time, it was very contentious. People may not remember this, but we had bomb threats during the course of our hearings. We had threats on our lives during those hearings. We had real partisan victory during those hearings. But we also had a core group of people. There weren't many, but a core group of people who said, let's just look at all the evidence and see whether or not we can agree on a couple of issues. And we came down to abuse of power. And the other one was obstruction of justice. And so it was very contentious, but approved one thing, that if you're going to impeach the president of the United States, it has to be bipartisan. You have to have members from both sides who agree that something egregious really has been done. But let me just interrupt you. Would you agree in today's atmosphere, there are no Republicans who would ever vote to recommend that the president be impeached on those two things, abuse of power? I would say possibly one or two. But what has happened is that we have moved, both parties are moving toward the end zone of their respective football field. You've got the Republican Party moving way to the right. It used to be the party of the rule of law. And now it's law and order. But when you say law, there must be justice included in that. So it's law and order and justice. And that is what's missing from this dialogue right now, saying law and order, let's crack down, let's crack heads. Well, why are they in the streets? They're demonstrating for the rule of law. And that means justice for those who have been denied it for the last four centuries. And you see it play out day after day in our lives. And some people are protesting against that. And now what we're seeing is something quite different, something you should be troubling to every American, conservatives as well as liberals or moderates. It's not conservative to send people who are not wearing camouflage people who are in military garb as such, camouflage uniforms with no identification of who they are serving, what service they're representing, Homeland Security, United States military, nothing to indicate who they are, whisking people off into unmarked vehicles, taking them to jail without charging them. That's not the mark of a democracy. That's a mark of a dictatorship. And so I think we have to be very, very careful what we're seeing take place under the rubric law and order and you have, I would compare it to what Russia has done in Ukraine. Russia sent military men who had no identification that they were Russians or that they were Ukrainians. They had no insignia, but they were Russian soldiers. They sent them into Ukraine, into the Crimea area and they helped to actually annex Crimea from Ukraine. And so I think they call them the little green men because of their outfits that were green in color. And I think there's a real comfortability here. We've finally identified these people as being federal employees from Homeland Security. But what they have done is they have now, they are dressed as military personnel with high power weapons that they are prepared to use, shooting rubber bullets, which can kill and firing tear gas, which has been outlawed in most cities and countries into peaceful protests. Some of them have been violent and they have to be dealt with seriously, but most of them are peaceful. And when you saw the White House and the president come out of the White House and go into Trafalgar Square and over to the church and clear out peaceful protesters and clear them out with violence for a photo op, that should be troubling to everyone. If you take away the right to protest, then you're really not living in a democracy because fundamental right is First Amendment, the right to protest, to bring our grievances to the government. And as long as it's peaceful, as Martin Luther King has shown, John Lewis who's lying in state, they went against the established law and order to demonstrate for the right to vote, the right to be seen as an equal basically. So we have to be careful where we characterize this as conservatives are full of law and order. Conservatives believe in the rule of law and they should be worried that any one person, and this is in our constitution, the constitutional founders knew that power has to be entrusted to someone, but no one can be trusted with power. And that was the whole purpose of having checks and balances to make sure no one person, no one agency, no one branch of government gets too powerful. And that's a conservative principle. It's not a liberal principle. And so I know that conservatives standing strongly behind President Trump on this, but we need to go back to the founding of this country to talk about what does the rule of law really mean? And the founders resented and were really worried about one person becoming so powerful that democratic checks and balances were completely eroded and we ended up with an autocracy or I would call it a dictatorship. So you were a very close friend of the late Senator John McCain. Yes. I think you were actually the best man at John McCain's wedding. Right. And very close to him. And people in his own party led by the president have insulted him, have said terrible things about it. You must get upset to hear those things about a man you admire. I was upset and frankly, that's one of the reasons very early in the game of this politics I should say, I became upset with President Trump. I became upset when he went on television said that John McCain was not a hero. Now, I was very close to John. In fact, I included him in a number of novels I've written. I used a character based on his character. It was one of my main figures in these writings because I admired what he went through. He was in prison for five and a half years. He was beaten almost daily. He was shown in the prison with a cracked hip and broken arms and a cracked collarbone. And he was tortured during that entire time. And yet he refused to be released early. His captors discovered that he was the son of the Admiral, Admiral McCain, who was conducting the war against them from the city. And they thought, well, if we let him out early, perhaps that will solve our problem with Admiral McCain. And John said, no, I'm not leaving until my time in order comes. And so he stayed there enduring all of that punishment years longer than he needed to in order to do it the right way in his mind that he would not take advantage of his father's position to gain any favor ahead of his colleagues. Now that to me is the mark of a real patriot. That's the mark of a character that we don't see very often in life. And for the president to say, well, he's no hero of mine because he got shot down. Well, what does that say to all the persons of war? What does that say about people who were wounded?  And so that was the first thing. And then the second thing that bothered me that he diminished and I think denigrated a Gold Star family. They were from Pakistan. They had moved to the United States. Their son fought for us, died for us. And yet the president just dismissed them. The third thing was I saw him mocking a disabled reporter. I said, this is not the conduct of a person who's going to heal us. A whole, this diverse country together. And I felt just seeing that take place it told me what I could expect. I think everything I expected and suspected what happened has happened. But Mr. Secretary, the interesting part about all of this is that all the polls show that 85, 88, 90% of people who self-identify as Republicans are big supporters of the president. They're the people that voted for you. Voted for you. And then we're your base too as a Republican member of Congress for me. What is this all about? I mean, this guy is not a guy of character. The president, he's mean, he's nasty and they love him. What causes that? Well, a number of things have happened. I just wish the Republicans who continue to identify as conservative members of the party would look at what's happened to our country not only here domestically.  What would a wartime president do? A wartime president would say, let me gather all of the governors in this country. Let me gather all of the federal agencies in this country. Let me gather FEMA. Let me gather DLA, which is the Defense Logistics Agency. Let's marshal all of the forces that we have, be able to distribute what we need in terms of PPE and other ventilators. Let's do it together as a nation rather than saying each state is on its own, when in fact a virus doesn't recognize state borders. A virus doesn't recognize continental borders. A virus is only a handshake away. A virus is only a plane ride away. And so the notion that somehow states you're on your own isn't a wartime president. It's an abdication of presidential power. It's a dereliction of duty in my judgment. Now, why do they still follow the president in this manner without question? It's because I think he operates out of fear. As far as the political figures, people in the country don't necessarily fear him. They mostly agree with the image that he projects. I think it's a false image as far as what he's doing to this country. But in terms of the political figures, I think they fear him. They fear his power that he can put out a tweet and suddenly they've got thousands, if not hundreds of thousands, of attacks coming in over the internet. So I think he governs by fear. And as far as the people are concerned, they see him as a strong figure. Then they think that he is redressing grievances that we've had that no one else has addressed. And he's doing this by pursuing an America first, or I would say America alone. And then I look around the world and other countries will look at us and they're shaking their heads. They're saying this is not the America that we came to love and respect and admire. If you look now what they say is they're pitying us. I mean, whoever thought the United States would be a subject of pity by anybody. And yet that's what the surveys are showing. They're saying how could a great country like the United States be overwhelmed by this virus that other countries are contending with because they're taking an all government national attitude toward it. So I think part of it is the Republican support him because they think he reflects the resentments they have. He reflects the notion that there are, there's a culture war taking place and there is. He reflects the racial war taking place and it has been going on for a long time. I mean, we really have never ended the civil war. It continues to be played out. The whole notion that it was a war of non-aggression. No, it was a war over slavery. That's what the war was over. And those in the South continue to say, oh no, it's about state's rights. State's rights to do what? State's right to keep people in chains. State's rights to keep people working with free labor under extraordinarily harsh conditions. So I think we need to have more education on who we are, what our history is, and then start to look at a leader. I think President Trump could have from the very beginning. He could have said, look, I campaigned in this fashion. Yeah, I'm a little bit crude at times. Yes, I can be nasty, but I want this country to heal. I want to heal the division between the right and the left. I want to find common cause. That's what the art of government is compromised. I mean, anytime you have more than two people in a room, you have to compromise. I write to swing my fist stops where your chin begins.  We have to reach compromise because we have so many people and we have so many of diverse backgrounds, which has been the energy and the vitality of this country. So compromise is not a bad thing, it's a good thing. And that's what I did, what Senator Mitchell did, what Senator Muskie did, what Olympia Snow did, Susan Collins, try to find the middle. Find that you can live between the 40 yard lines. If you start to put your flag in the end zone, each party does the same, very little gets done. It's when you come together, you say, okay, I can't have everything that I want from my party, but if I can get 60%, 70% or whatever the formulation is gonna be, okay, we start from there and we'll continue to work on it. But I think the danger is that we become so polarized and we become so ugly in the process of name calling, of diminishing people, of reducing them. And I mean, the names that are given out for people and again, it's a racial conflict. It's a virus that we're dealing with, combating. It's economic and it's international at this point. We have a trade war with China. We have the economic war with even with the EU. I mean, so you start looking at all the conflicts take place, you say, we need to have somebody who's going to try to heal this and John McCain believed in alliances. He and I traveled the world together trying to make sure that we had friends. We said, make your friends before you need them. Make our friendship so that when we really need them, they are there. Now, when I was in the Pentagon, working with President Clinton, a Democrat, his mantra for us was multilateral whenever we can, unilateral whenever we must. Now that's been totally turned upside down. And so now it is unilateral whenever we can, multilateral whenever we must. So the other countries look at us and they say, you know, we can't count on you anymore. I guess we'll have to pursue our own interests. I guess we'll have to develop systems on our own or link up with countries who we will trust. And that is the key element that I think we have lost track of. If countries don't trust you, they will not support you during times of strife, during times when you really need them. And so I think we're in a dangerous period for all of the reasons I mentioned. And I hope that in this next election, we say, let's select the person that can bring us together as best we can. I think it's gonna be hard. I think if you had to look at it, Biden is ahead more than how in some of these states, but I think the President of the United States still has a very good chance for a winning. And let me take you back a step to something you said about the cultural war in the United States. I know so many people, decent people who get very offended at the slogan, Black Lives Matter, and they always say, and you've heard them, we've all heard them. Wait a minute, what about white lives? White lives matter. And so there is this defensiveness among people.  I think they'd like to think they're not racist, but they are. Now, would you comment on that? Sure, just think about Amy Cooper. Amy Cooper was the young woman in Central Park who didn't have her dog on a leash. I take it that she's a liberal, and you would think that as a liberal, she would be racially sensitive, so to speak. And what did she do?  That's the law. And what did she do? She picked up her phone and she called the police and said, what? There's a black man and he's threatening me. Now, take that phone call in the context of all of the young, unarmed black men who are being shot by the police. And I could list them down the line how many there have been. They have, and so what she knew in her heart of hearts that if she called the police and said, there's a black man and he's threatening me, they would come and they would take her a version of the story as opposed to his. His life would be put in jeopardy because we've seen that take place. From every, the gentleman got shot while he was jogging. George Floyd, who was crushed to death under the boot in the heel of the police officer. You start going down the list of what's happened in this country from the time that this country was formed. They said, well, we're not racist, but we are racist. We formed this country. What did Tom Cotton, Senator Cotton say just the other day? Senator Cotton said, slavery was a necessary evil. It was an evil to be sure, but why was it necessary? Why was it necessary to bring human beings, put them in chains, drag them all the way across the Atlantic in chains in the bowels of ships where half of them had to be thrown overboard because they couldn't make the trip and keep them in chains for several hundred years? Why was it a necessary evil? Why weren't white people willing to do the work? So these are the things that we have to discuss to say we treated human beings as animals. We treated them as a burden of beast, beast of burdens I should say. And that has happened for hundreds of years in this country. And so when people say, well, I'm not a racist, no, but everything that was built in this country was built on the backs of this quote, necessary evil. And that's what people are speaking out against. They say, okay, stop killing us.  Think about the woman who was driving in Texas. The police officer didn't like the way she came too close to the center line, pulled her over and he didn't like her attitude. She was angry or she was smoking a cigarette. He ended up threatening to taser her, hauled her out of their vehicle, put her in solitary confinement where she's found dead three days later.  Kids as young as 12, Tamir Rice, 12 years old, shot when police arrived at a playground. He had a toy pistol and within two seconds, three seconds of the time they arrived they shot him to death. I mean, so there's a lot going on and I think we just have to be very sensitive to that. This has not been an equal country. It did not treat people equally in terms of their race, in terms of their religion, in terms of their ethnicity. And what's great about this country is that we can speak about it. We can say this needs to change, this is not fair. So what we believe, we pound our chest with the greatest country in the world and I believe we are. But we're great because we're allowed to fix our mistakes. We're allowed to become better than we have been in the past. That's what makes this country so great. So we have a president who disagrees with everything you say. And in fact, all of the bad things that you're talking about, he stokes.  If you look at these rallies that he had, just look in the faces of the people. He knows that they hate people, they resent people and he likes the fumes, he throws the gasoline. And that's happened before in human history. I've been reading over the past six months, every book in my bookcase about the third right. And I read something recently, the end of democracy, Germany 1933 to 1945 and how it happened because the Germans are decent people. I've spent a lot of time in Germany. Nice people, they're like us, we're all human beings. And they went for this and it destroyed their country and it was in rubbles. And in 1933, they didn't have the slightest idea that would happen. There were very few people that thought, hey, wait a minute, this is bad. And then when Hitler started talking about the grievances, they said, gee, he's smart. He's gotta figure it out. Now, do you think that there's any thing to history and should we be looking at that? Sure, well, what did Hitler really want? He wanted to purge the Germany of all Jews. He felt that the Jewish people had co-corrupted Christianity. And therefore, the only way to have a redemption, a spiritual redemption was to get rid of every Jewish person, not only in Germany, but throughout Europe. That was his goal. Now, what's happening in this country is President Trump doesn't say it exactly, but he's doing it through his actions.  Why are they thugs? Because they're out demonstrating in the street. What if they were white people as they are more and more becoming?  When he criticized Colin Kaepernick, who took a kneel during a football game when they were playing the anthem. Now, was he kneeling against the flag?   But what happened? President Trump goes to Helsinki and he takes a knee to President Putin. I'm not talking about this. You think about this. Here is President Putin that our entire national security team, my friend, Ann Coates, Senator Coates, a former ambassador. A conservator from Indiana representing the intelligence community. He said, we have it across the board here. Unanimously, Russia interfered with our election system. President Trump goes to Helsinki, he meets with President Putin privately, then comes publicly. He said, oh, I asked him and he denied it. And I have no reason to not believe him. He threw our entire intelligence community under the bus for President Putin. And he's done it systematically. He's done it in Syria. He's done it everywhere. We have an interest in President Putin as a different interest. He yields to him. Now, you have to ask why? Why won't the president release his tax returns? I asked this question a week after he was elected. I was on CNN at that time. There was a wrap up with all the coverage of the inaugural. I said, the president, here's what I would like to know. What does the president own? What does the president own? And to whom does he owe it? Those three questions would relieve a lot of doubt in my mind in saying, are we really indebted? Is he really indebted? Is family indebted to the Russians? To the point that he doesn't dare say a word of criticism to President Putin, who has caused the mischief of all of the globe. He will criticize the French, the British, the Australians, the South Koreans. He'll criticize everybody but Putin. He even goes easy on President Xi. He's making China the object of our contest right now. But Kim Jong-un, my best friend, my good friend Kim Jong-un, one of the worst killers in the world history. So something is going on that we don't know about and his refusal to release his taxes tells me something, he's hiding something. Oh, I don't know what it is. But then my imagination starts to speculate. Exactly now, I couldn't agree more with what you just said. He's hiding something. But why is it that those 85% of the Republicans who love him aren't at all concerned about the fact that Putin is the only person in the world that he's not insulted. Nobody else is insulated from his insults except Putin. But I have a couple of quick questions. We're running out of time and one of them is you talk to people, you know people in Washington, you know politicians, and you know a lot of Republican politicians, say what, your colleagues, and you serve with them. Don't they see what the president is trying to do to this country? Don't they, are they just oblivious to it? They don't understand it. They disagree. No, they think he's doing a wonderful job. What is it? Or is it just they keeping quiet? Well, I mentioned it before, there are two things. Number one, there is fear. Every Republican who has spoken out in any way being critical of President Trump has lost. They've either lost or retired because they would be facing defeat. And the other issue is that they have to agree with it. If they're unwilling to speak out, then it's either fear or they're complicit, one or the other. Now, earlier you asked, what's the reason for this? I think social, the rise of social media has contributed to it. And the internet, social media, it's all really extraordinary technology, the ability to communicate on a ubiquitous basis instantaneously. And so I think of the internet as being a river of ideas. It's also a sewer. So you have the river that's flowing with new ideas, new concepts and good things. It's also a sewer because you have information that's going out over the internet, especially social media, which has no basis in fact, which are complete distortions, which are used by the Russians, the Chinese and other countries in order to distort our system. So if you look at, if I went down Putin's list, if I had Putin's bucket list, what would I want? I would want to undermine confidence in NATO, okay? Check. I would like to undermine the US confidence in the EU. Checkmark. I would like to undermine the US confidence in the integrity of the media. Checkmark. I would like to undermine our institution, checkmark. Those are all things that Putin wants for the United States and we're doing it. We're allowing it to be done. So I can't account for the 85% or even 90% of the American people, the Republicans, who think that's a good idea. What I think is a good idea is we come back to our original basis that we believe in the rule of law. We believe in equal justice for all. That we believe that no person stands above the law, including the president of the United States, that we're all subject to the same rules. And we get back to that and then start talking about fiscal responsibility. This is a core argument and theology for Republicans, fiscal responsibility. And we're just racking up the national debt to a point where our kids and grandkids will never be able to pay it. So fiscal responsibility, having a respect for our institutions, rule of law, fairness for all. Those are the issues that we need to bring us back together. So you've touched on financial matters, the management of the government and so forth. The president is certainly not a deficit heart. He could care less about it. He doesn't have any ideology. Would you agree? I agree. Listen, the president is, he's an opportunist in the sense that he's a, he believes in transactions. I don't believe he has a coherent strategy in terms of a worldview. I think his worldview is, I'm gonna do whatever benefits America and the rest of the world can take care of itself. Well, I want to stop you there. I would say whatever benefits Donald Trump. Well, okay. He does look after his own interests and that gets back to how he's benefiting in office, while he's serving again, talking about how much money he gives up by being in the office. His family is benefiting enormously. And when they start attacking Joe Biden, Joe Biden will have to answer for himself.  So he's looking out, he wants to send, have the golf tournament in London, at his golf course in Scotland. I mean, the amazing thing's going on in terms of the enrichment of his office. So you were on CNN with Christiane Amanpour recently and she asked you about your view of Donald Trump and so forth. And you talked about you as Secretary of Defense, of course, you spent years looking at the top secret reports about our enemies and Russian spying and so forth. So you're very realistic about that stuff. But you said some tough things about dictatorship and Donald Trump and you said, you were not sure whether America, as we know it, could survive another four years. You still feel that way? I do. Look what's happening with the Justice Department. The president has intervened and arranged for pardons for his friends and punishment for his enemies. That's not the mark of a president acting on behalf of the United States to intervene in judicial matters, to take things away from prosecutors at the last moment saying, oh no, I don't care whether he's pled guilty. I don't want him to be guilty. So let's just cut that out. Firing professional prosecutors, getting rid of people, Sally Yates, who called attention to what Michael Flynn was doing by contacting the Russian ambassador and dealing with the Russians while he's representing a Turkish government. She got fired. Then it was Jim Comey, director of the FBI, who said, I'm not gonna pledge fealty to you. I pledge my obligations to the Constitution, not to you personally. So he gets fired. Then the assistant gets fired. The deputy gets fired. And then you take these individuals who are career professionals and just get rid of them all. You had the career professionals in Ukraine. What did he do? He fired the career professional who's an expert on Russia, gone. Who's he been dealing with? Igor and his friend, Lev. So Lev and Igor were the ones he was using. Using Rudy to go around. Rudy Giuliani, Rudy Giuliani. All I wanna do is get back to saying the United States believes in the rule of law. We believe in treating people equally. We believe in fiscal responsibility. We believe that no man or woman stands above the rule of law, that we all abide by it. And we do not use the presidency to enrich ourselves or our families. That is what I think we have to get back to. So that's what I've lived by. I've tried to live by during my career, long career in public office. And remember, an elected official, the president of the United States, he's a fiduciary. He's a trustee. And what the courts have said over the years, the trustee of fiduciary owes you something higher than the morals of the marketplace. We expect you to be very high in terms of your ethics and your obligation and not look out for your self-interest, only that of the country. That's what I wanna get back to. You've done a lot of politicians in your days. You served with them, colleagues. After all of these decades in public life, do you think that the character of leaders, of political leaders is important or just the transactions, what they do? I know, character is everything. Character is destiny. And I'll tell you why I am still optimistic, notwithstanding everything that I've said, I come into contact with students and young leaders. In my office this summer, for example, we have 15 students who are either in undergraduate or some possessing graduate degrees, some whom had served in the military and come back out. And we unfortunately have to do it through kind of distance learning, but I bring them in every summer to associate with General Ralston, with General Jim Mattis, with General Kern, with Ambassador Grossman, I've said Ambassador Burns, to bring them in and to see how government works or at least should work. And when I see the talent that's out there, when I see the idealism of these young people coming up, we have several, a couple of them from Maine, Colby University, you've had them from Bowdoin, but we have them from all over the country and even international students. And when I see the talent and the intelligence and the ethical standards they have, you know, I really feel good about that, but that's why it's really important that we look to them and try to inculcate in them what the history of this country has been, where we have fallen short, what mistakes we have made, where we have failed to measure up to these noble words that we keep insisting upon that we represent, so that we have life and liberty and pursuit of happiness for everybody, not just for a privileged few. Secretary Cohen, I can't thank you enough for agreeing to talk to my audience and these folks in Maine, many of whom are old enough to have been your supporters, voted for you at one time. You were in fewer, they're still with us. Pure and pure. I hope, I don't want to put you in the spot, but I hope we can get you back on at another time to comment on what's happening in this country. Thank you very much. You're welcome. I'll be there. Thank you very much. Okay.


Now, coming to the femoral hernias, as we know they occur inferior to the inguinal ligament and they lie medial to this femoral vein. So, this is femoral vein, this is femoral artery and these are the femoral naus. So, V-A-N and hernia occur medial to it, they are more common in the woman and compared to inguinal hernias, the obstruction is more common in femoral hernias, why? Because they have a narrower orifice. This is one case of femoral hernia and this is the hernia and we can see the femoral vein and then artery and then laterally will be the, this will be the naus. This is another case where we can see this is the femoral hernia which is causing displacement of the femoral veins laterally and this is the inguinal ligament and this is the femoral hernia which is going below it. Now, the question is how to differentiate all three types of hernias which are femoral hernia, direct hernia and indirect inguinal hernias. The first thing to be seen is relation with the inferior epigastric arteries. If you can see this is the hernia and inferior epigastric arteries seen later to it, okay. So, these are the direct inguinal hernia, this is the inguinal hernia and inferior epigastric arteries are seen medial to it. So, this becomes indirect inguinal hernia and this is the femoral hernia and the inferior epigastric arteries are seen later to it. The second thing we have to see is the relation with the inguinal ligament. This is the inguinal ligament and this hernia is occurring below it. So, it becomes femoral hernia, but if the hernia will be seen above it then it will be called as inguinal hernia. The third thing to be seen is something called as lateral crescent sign. Now, how to see this is the level of the pubic bone and we can see the scrotal cords anterior to it. This is the normal scrotal cord appearance which is rounded. In case of direct inguinal hernia, it projects medial to this inguinal cords and the inguinal cord is pushed laterally by this hernia and then scrotal cord acquires somewhat moon like appearance or crescent like appearance and this is called as lateral crescent sign and they are only seen in direct inguinal hernias and they are not seen in indirect inguinal hernias. So, that was the third thing to be seen. The fourth thing which is usually seen is femoral vein. Now, if we can see this is the femoral hernia and this is the inguinal hernia. This is also inguinal hernia. Now, this femoral vein is significantly or moderately compressed by this femoral hernia whereas this is even larger hernia than this hernia but this is inguinal hernia even though this hernia is so large it is not causing any compression of the femoral vein. So, this is the one criteria to differentiate between femoral and inguinal hernias. And the fifth thing is to be seen of the hernia sac with relation to the pubic tubercle. So, this is the pubic bone and this is the pubic tubercle. We draw two lines perpendicular to it and it is said that the inguinal indirect hernias may cross this pubic tubercle. So, these are the indirect inguinal hernias dilaterally and they can be seen crossing over the pubic tubercle. Whereas, direct inguinal hernia may not cross this pubic tubercle. This is the direct inguinal hernia and there is that is not crossing the pubic tubercle and femoral hernia may pass behind the pubic tubercle. So, this is the pubic tubercle and we can see the hernia passing behind the pubic tubercle region among all the five signs this is somewhat less reliable. Now, this is a case of right sided inguinal scrotal hernia and this is a case of left sided inguinal large inguinal scrotal hernia. This is a case of inguinal scrotal hernia with also bladder projecting within it. In this case with femoral hernia because you are seeing the femoral vein and in this femoral hernia we can see appendix lying within it. So, this becomes degerringout's hernia whereas, this is inguinal hernia with appendix within it and it becomes amiens hernia.