So a couple of things that we're going to talk about today with Professor Joan Roloffs really focus on and relate to a lot of the work that we do at Code Pink. So I just wanted to talk a little bit about that because I think it's so important that we, you know, not only talk about some of this great research that Professor Roloffs is going to talk with us about, but actually how can we change the fact that the US war machine is so embedded in our local communities, right? It's something that we do every day at Code Pink through our campaign to divest from the war machine and our work to build a big tent of organizations to cut the Pentagon for people, peace, planet, and a future. So our campaign to divest from the war machine is extremely important because it offers us a way to build the kind of grassroots resistance to war profiteers in our local communities that Professor Roloffs will be talking about today. So our campaign calls to divest invested assets from city pension plans, state pension plans, university endowments, religious institutions, you name it, right? And earlier this year, actually in July, Code Pink alongside World Beyond War, Veterans for Peace and Women's International League for Peace and Freedom were successful in getting the city of Burlington, Vermont to divest their city pension fund, which consists of millions of dollars from weapons manufacturers. And it was through this coalition with World Beyond War, Veterans for Peace and Wilps that we were able to demonstrate the importance of cities in divesting from war profiteers at the local level, right? And I think importantly too, this led to a really great conversation about the influence that war profiteers have not only at a local level, but also on our congressional representatives. You know, as we speak, companies like Lockheed Martin, which is one of the largest war profiteers in the world, are lobbying against the corporate tax scheme that would help pay for the Build Back Better agenda to invest in healthcare, education, and green jobs, right? So both in our local cities and at the national level, war profiteers have this outsized influence on our politics, and that's why we're here to talk about that today. So, you know, if you're interested in getting involved in our campaigns to divest from the war machine and also cut the Pentagon for People, Peace, Planet, and a future, you can always email me. It's Carly, C-A-R-L-E-Y at CodePink.org. I'll also post it in the chat for everyone who's watching. So with that being said, and a little background of some of the campaigns we're working on, I'm really excited to speak with author and Professor Joan Rolofs today about her work into the influence war profiteers have at the local level and in spaces that we haven't yet had a chance to discuss. So Professor Joan Rolofs has been an anti-war activist ever since she protested the Korean War. For this cause, she has been associated with the Pledge of Resistance. Monodon, how do you pronounce that? Is it Monodon? Ah, thank you. Greens. Greens in Green Party USA, a New Hampshire peace action, World Beyond War, the International Campaign to Abolish Nuclear Weapons and CodePink. She is the author of Foundations and Public Policy, The Mask of Pluralism, which was published by SUNY Press, 2003, and Greening Cities, Building Just and Sustainable Communities, which was published by Roman and Littlefield, 1996. And Professor Rolofs has translated Victor, Considerance Principles of Socialism, in 2006, and was the co-translator with Sean Wilbur of Charles Foyer's Anti-War Fantasy, The World War of Small Pastries. Sounds amazing. Professor Rolofs is Professor Emeriti of Political Science at Keen State College. Since her semi-retirement, she has been teaching in the Cheshire Academy for Lifelong Learning, including a course on the military industrial complex, and writing for a variety of scholarly and political publications. So very, very excited to have you with us here today, Professor Rolofs. Thank you so much for joining us. Well, thank you. And I admire the work of CodePink. I think it's probably the best anti-war organization in the United States. Amazing. My perspective. Thank you so much. We really appreciate that. And, you know, it's because of people like you that I'm so excited to be able to take on the war machine in our local communities, the research that you're doing, the activism that you have been doing. And just before I invite you to speak, just a few reminders for everyone on the call. So we, you know, Professor Rolofs will have a chance to talk more about your important research into the military industrial complex. While she's speaking, if people have any questions, any thoughts to ask Professor Rolofs, please go ahead and type them in the chat box or in the Q&A function. And then after Professor Rolofs has a chance to present her excellent research, we'll have 10 to 15 minutes for Q&A. And we really want to make sure to get to everyone's questions. So if you have any, let us know. In addition, this webinar is being recorded and live streamed. So I'll be checking on YouTube. We'll have people watching there, commenting there. And, you know, if you want to send this to people after it's over, we will send you a link so you don't have to worry about writing down everything. So without further ado, Professor Rolofs, welcome. I will let you get started with your great presentation. Thank you. I'm not going to be talking about the horrors of war because you will know about that. And also, of course, the devastation that's caused not only all over the world, but to our own people on our own environment. I am, however, very concerned about militarization. And that's happening all over the world. The blurring of the boundaries between the civilian and military activities, which I have found so much in my research. And what motivates me in doing all of this is because despite the horrors, there is so much silence in this country. The Arizona anti-war movement, it's very small, though. And some of the silence can be explained. But the silence of supposedly progressive citizens is very disheartening. I'm talking about liberals, environmentalists, civil rights activists, academics, and religious leaders. They protest about many problems, but war and militarization rarely comes up. And also during election time, hardly anyone of the public, the press or the candidates, talk about war. There's been very little discussion of that. Now, I believe that the silence is caused by fear, by propaganda, by distractions, and also by interests. And what I'm going to talk about is about the interests that have created the silence. And I'm also, of course, very interested in this blurring of the boundaries. Now, I will show you the outline of my talk so you know where I'm going. So it's up on the screen there for you, John. Okay. Well, the obvious interest, you can take it away now. The obvious interest is, of course, the military establishment, the obvious reason for silence. We can expect that. There are some people who actually enjoy violence. There are officers who want action. There are scientists who want to try out their inventions. But that may be a very small part of the military establishment. The troops may consist of many people who joined because that was their best opportunity for a career. And the advertising for recruitment always emphasizes that you join because of skills, you get skills, leadership, benefits. There's not a lot of talk. This is deliberate in the Department of Defense's advertising.  No, it's always leadership and benefits and you learn skills and all of that. The part of the military establishment is the Army Corps of Engineers. And they want to be constructive. They've been involved in civilian infrastructure and flood control for centuries. The National Guard has performed vital services in disaster relief and recently in the vaccination process. None of those people, however, are free to criticize the military and its objectives. And according to the statements from the Department of Defense itself, the goals of the U.S. military are greater lethality and world domination. Again, for pausing the silence, there are many, many veterans and their families and they are mostly either supportive or silent. And there are many non-governmental organizations that aid veterans and all the people involved in those are not going to criticize the objectives of the military. There is even an organization called Comedy Boot Camp, which teaches veterans to be stand-up comedians. Now, there are, of course, some notable veterans groups that are anti-war, such as veterans for peace and about face, but it's a very small proportion of veterans involved in that. Another large group that is condemned to silence are the civilian employees of the Department of Defense, which is, of course, the largest department in terms of personnel in the federal government. And these people are not just office workers, but they're scientists, they're architects, they're engineers, meteorologists, political scientists, information technologists. Some of them work at the Pentagon, which is, as you know, quite huge, but many of them work at military colleges. There's a whole network of military colleges and institutes in this country, or think tanks, education and training institutes, not only in the United States but throughout the whole world. NATO itself has an entire infrastructure integrated with civilians in all the countries that they operate. They have think tanks called the Centers of Excellence. And they are deliberately interested in integrating civilian politicians with military personnel in all of these places. Again, something that there should be more research and publicity about. Then many other departments of the federal government work very closely with the Department of Defense. Most notable is the Department of Energy, which occupies large areas of the country to prepare the ingredients for nuclear weapons. Even the Department of Agriculture is involved in Department of Defense programs. For example, the creation of a dairy cow industry in Afghanistan. Joint participation of Department of Agriculture and Department of Defense. So the jobs for civilians and the rust belt polishing makes for huge support for the military budget. These are real jobs. It isn't just profit seekers. It is people who are job seekers, people who need to earn a living are silence. That's where they get the jobs. And whatever they think about the objectives of military, they are not free to protest. Another source of economic support in this country, the support for our economic system, are bases. And there are about 4,000 military bases in the United States according to the base structure report. Now, some of these may be small and insignificant, but certainly there are several hundred that are economic hubs for their region. Some like Fort Bragg in North Carolina are the size of small cities. And Fort Bragg is actually the largest base in this country. They not only employ many civilian workers from the region onto the base, but they support a whole host of off-base businesses. Car rentals, supermarkets, entertainment, restaurants, motels with housekeeping suites, and even local museums and tourist attractions. And all of these are actually advertised on the bases website. The public schools outside of the bases are subsidized by the Department of Defense. And I was surprised to see that one public school system that was subsidized was in Lincoln, Massachusetts. It's one of the wealthiest places in the United States.  That keeps some people involved in public school administration quiet. Even the remediation of bases and almost all of the closed bases are superfund sites. That brings workers to the area to remediate the bases. They also look for restaurants and shopping and motels and other things. Now, surprisingly, the bases in this country offer hunting, fishing, and recreation. And some of the bases offer this only to military personnel, but there are also quite a few that offer it to anyone. And we have a picture of this. We have a picture to show this. Yep, I'm sharing it now. Thanks, Jim. Okay, I didn't see it. Yeah, here it is. Okay. That is, yes, hunting and fishing on a military base. And this base, I forget where it is. This one offers to civilians. There's a way you apply. And that's one of the ways in which the military and civilian are intertwined is a very interesting way. Okay, I'll go on. You can take it. The new Boston Air Force base in New Hampshire, which is very near me, is a former bombing range. It's now part of the space force. Part of it was considered, oops, I've lost it. I've lost it. Okay, part of it was considered clean enough from unexploded bombs to be used for recreation by military people. And they can rent cabins, kayaks, RV sites, party supplies, bikes, and bouncy castles. I hope everyone knows what that is. But local people and businesses aren't the only ones that are supported economically because major construction and electronic communication companies have huge contracts to build, maintain, and upgrade the bases, such as Buckley Air Force base in Colorado, Aurora Colorado, by the way. And that is also now part of the space force. But U.S. businesses, large and small, both build and maintain bases and are military all over the world. This exports military Keynesianism to other economies. It is our bullet and rod initiative. So now we look at the testing and training lands map. Yep, I'll show it now. There it is. No, yes. That's it. Yes. This is a map produced by the U.S. military. And this shows the testing and training lands in this country. And that includes, of course, along the coast and in the coastal waters. Now, there's a problem with some of those because the U.S. military, you can leave it there, sees two main threats to their bombing and testing ranges. One threat is incompatible civilian activities. And the other threat is restrictions that protect endangered species. These are actually, they actually use the word threats. And to deal with these threats, the military has a program to create buffer zones with using partnerships with private conservation groups, universities, and local governments. All of these people involved. But the largest of the conservation groups that is involved is the Nature Conservancy. They have the most funding from the Department of Defense. Its grants for Fort Benning, Georgia alone included one for $11 million and another one for $56 million. This is for Fort Benning. And there are many, many more. You have to add up all of the individual contracts to find out the total amount they get. But it probably is in the billions. And that's the Nature Conservancy. And many other environmental organizations are in this program, such as the Main Audubon Society and Ducks Unlimited. And in Camp Lejeune in North Carolina, which is a Marine Corps base, and it's the size of a city, it has 11 miles of amphibious operations beachfront. Among the Student Conservation Associations, many, many grants is one for $11 million for assistance at Camp Lejeune with a special concern for the protection of endangered sea turtles. So that is just part of the experience of non-profit organizations. You can take this one away. Okay. Again, quite obvious for silence. The weapons makers, their parts suppliers and nuclear fuel provisioners, these are of enormous importance for jobs. And there are also good opportunities for women and minorities at these places. Marilyn Houston, a woman was the CEO of Lockheed for seven years. And I have looked at the web pages of these big weapons makers and their staff, their CEOs, and as well as their boards of directors. And they're very diverse. They're very diverse. People very high up in managerial positions, many women of color. Yeah. In Nashua, New Hampshire, a business journal reported in 2017 that the F-35 program supports 55 suppliers, 35 of which are small businesses, and over 900 direct jobs, much of them located at BAE Systems in Nashua. Money Magazine has deemed Nashua the best place to live in the country in several years when it has done this. Well, of course, there are well-paying jobs, and there's also very fine tax receipts. This not only supports local businesses like restaurants, but also the arts and social services provided by the city itself. But New Hampshire also has an aerospace and defense export consortium that promotes foreign sales of weapons. Its members include not only manufacturers, but universities, banks, and politicians. And then actually I looked up and it included the bank I have that I use. I've found out. And it's very hard for anyone to get away from this. Yeah. And of course, this is one reason why we don't hear about the rust belt so much anymore, because this military has polished up the rust belt in many, many parts of the country. In New Hampshire, the rust belt occurred a long time ago when the chairs and the shoes moved south, and then out of the country. But New Hampshire will say, well, high tech is now our thing. But as far as the manufacturing industry goes here, and in most places, it's very much military supported. There are other industries. Probably larger industry in New Hampshire is healthcare, also government supported. And education is very big. But when it comes to manufacturing, there isn't a lot that is not involved with the military. And of course, the government says we can't allow our manufacturing techniques to disintegrate by importing everything. We have to keep up with technology. And this is the way we do it, funding the military. Okay. Another reason for silence is investments. These individuals and nonprofit organizations, including churches, have investments in mutual funds. And the portfolios have huge returns because of the weapons, not only the weapons makers, but also the information technology firms that are part of it, like Microsoft Accenture. They are also very profitable. Now, the Art Institute of Chicago was challenged on their military holdings, and they said they were required to maximize returns on investment, and they would not divest for social, moral, or political reasons. I think one of the great things about the divestment campaigns is it informs people where their money is invested. Obviously, Lockheed is not going to go broke if people divest or organizations do, but it will point out how much their own money is in, and some people decide not to do it anymore. And that's very important, especially pension fund. Workers' pension funds are very heavily invested, and I was really surprised to see that the New York City Teachers Union was invested in military funds because in my day, this was one of the most radical organizations in New York. Can I add one thing about that? I think that's really important, and that's why we're working on a divest from the war machine campaign at Code Pink, and we're working to divest, like you said, workers' pension funds of the city and state level, because it's true, right? It's outrageous that some of these, you know, maybe more progressive meaning unions, right, should be invested in, their workers should be invested in the war machine. So, absolutely. And you see, a lot of people don't know when I was teaching the course in the military industrial complex. I discussed this, and one of the students said she was going to divest because people have investments in fidelity or vanguard and fund, and they don't know what's in the portfolio. I'm bothered to look it up. It's very hard on the eyes, the research I do, but yeah. Okay, so in addition to weapons, there are so many other amazingly diverse Department of Defense contractors. The construction, information technology, intelligence, cybersecurity, guard duty, remediation, food, transportation, housing, clothing, cleaning, even landscaping. Goodwill Industries has total receipts in the billions, and they provide for the military, security and janitorial services, mowing, recycling, clothing and furniture. And other organizations that employ disabled people have contracts like the lighthouse for the blind. The other thing about these contracts is that preferences are given for businesses owned by women, minorities, Native Americans, veterans and disabled people, and at least cost considerations are waived. So I found a Native American-owned firm in Montana supplied six microwave ovens to a military facility in Concord, New Hampshire. Every kind of service or product has contracts for the military. In New Hampshire, this includes window shades, restaurants, Velcro, fat analyzing kits, disposable double handcuffs, mountain rescue training, childcare centers, baby cribs, heating oil companies, landscapers, carpenters, appliance dealers and religious retreat centers. Okay, another source of the silence is universities. The military itself has a very large network of training institutes and there were some state and private universities that have always been heavily militarized like Virginia Polytech and Norwich University in Vermont. Many universities have centers that are affiliated with the military. Non-profit think tanks like the Center for Strategic and International Studies employs academics. The universities and colleges have lucrative contracts and there are also contracts and grants to worldwide universities, institutes and laboratories. This is a very interesting thing that I found. Now it's quite normal for universities to require professors to get outside funding for their research and the contracts with the military are not just for physical science and engineering. Biotechnology and information technology are well endowed. Also environmental studies, social sciences, philosophy and religion, cash in. The military has something called the Minerva Initiative especially for social scientists and they have grants for social scientists worldwide. DARPA and other divisions of the military have a lot of in-house scientists. They also create consortia incorporating civilian universities and research institutes. Now the religion professors, one reason the religion professors have been enlisted is because DARPA concluded that some of its planned technology would be challenged on ethical grounds by religious people in the United States and abroad. One of the things they were concerned about was their idea, their plan were in planting electrodes in soldiers to make the process of attacking targets quicker and more effective. So they contracted with a religion professor at a university to see how they can dispel this religiosity or whatever because it was a threat. It was a threat to the military. Public schools are another place where the military is very well instanced. The Department of Defense itself has Junior Reserve Officer Training Corps and they advertise this and publicize it as a program to motivate young people to be better citizens. That's what it says on their website. That's the purpose of the program. It's training for character development, leadership, cooperation, management, skills, and sense of accomplishment. Nothing about military at all. And very few of the children enrolled in these programs become officers in the military, but all are exposed to military values, including the glorification of Native American extermination. And there are six public high schools in Chicago that are military academies. Parents send their children to these because they're well funded, they're clean and they're disciplined. Again, they are not necessarily interested in their children having a military career, but those parents and the children are silenced. Now philanthropy is another channel of militarization. Some of it is not surprising. Grants from the weapons makers to policy planning think tanks like the Hudson Institute, the Heritage Foundation, the American Enterprise Institute, but also museums and orchestras, universities, charities, and health organizations are generously supported. Environmental organizations are supported by weapons makers. Pratt and Whitney produces engines for the F-35. And in October of 2015, its employees volunteered to clean up the Connecticut River. So the philanthropy is not only in terms of donations of money, it's also labor. The employees are encouraged to work in all kinds of organization, all community organizations, environmental organization, get in there. Yes. And the contract of benevolence especially touches many members of the general population. Minorities, low-income people, social justice activists, liberals and civic-minded volunteers. The people who you might expect to be part of an anti-war movement have been silenced. And women donations are especially generous to organizations supporting women and minorities. Lockheed, Boeing, and Raytheon are important funders of the Congressional Black Caucus. The Association of University Women has a program to entice girls into STEM careers. And that is funded by Boeing, Lockheed, and BAE systems. Many other minority higher education is funded, boy scouts, girl scouts, make a wish, juvenile diabetes, big brothers, big sisters. And our public schools are very grateful for the enrichment provided by contractors. Employees of Lockheed, Boeing, Raytheon, etc. serve as tutors and mentors to students that actually go into the schools. And especially enthusiastic, a program such as one called FIRST, which is funded and led by the contractors. It introduces children to the fun of rocketry and robotics. And there are similar programs all over the country. And maybe you could show a picture of that. Yep, on the screen. See how much fun it is? And often the employees of these firms come in and they're wearing t-shirts with the logos on them. And the children remember that. They're very keen on logos and things. So, yes. So I'm going to just conclude now and then we'll look at some of these research techniques. The anti-war movement is going to have to figure out how to transform our economy, education, and culture. And I can't even talk about the culture. That's another huge subject. The TV, the video games. Because not only is the military so entrenched in our culture, but it is so often invisible. Local newspapers don't write about it. I don't know how to change it. All I can do is I try to make militarism more visible. That's really my aim. And I'm trying to enlist more people to do it. So maybe we could look at the research, how some ways to get started researching the state complexes. And if people could do it for their state, it would be great to have a compendium online of as many states as possible. So there's a publication by the military called Defense Spending by State, which is one thing you can start to look at. I want to show you the map now, the interactive map. First, the defense spending and then the map. Okay. Yeah. Sounds great. I'll show the defense spending now. It should be on the screen. No, that's not it. That's not it. That's number eight. We want number five. Let's see if I have that. If you don't. Let me share my other screen that I had up. No. It wasn't the map. Okay. This is an interactive map that I made. Yeah. And you get the link. You will be able to open it up and see because it's all jammed together. But you have to you have to open up and then there are what, 131? Oh, you've got it. Okay. Yeah. Okay. And anything you click on, you find out what it is. Right. Yes, I have a list of sources that Carly will send to the participants, I think. Yes, I will. Absolutely. You'll get and the link would be on it and other sources. Yeah. So I made this with Google Maps and it takes a long time, but it was no way it was fun to do it. So anytime you click, you find out what it is. I made some symbols. Yeah. Shark behavior. Okay. Here's one funded by the military. And how did you identify these different things? How did, how do, how do I identify it? Yeah. Well, there's a legend in the sea. I see. There. Yeah. That if you have to expand, if you expand that, if you can enlarge it, the map. And then the other thing is that you click on, on anything and it will tell you what it is. You click on each point, it will. Great. Of course, this, there's much more. I couldn't put everything on here. There are many, many more businesses with contracts, but this just gives you a rough idea of, of New Hampshire and many other states are quite militarized. Yeah. Excellent. It's on my website. There's a link on my website also to this. Yeah. Great. Okay. So now you can put on the searches. All right. I will put up the searches next. I just posted the link to that map as well in the chat box for everyone. But everyone will also receive a link to that right after this. So the searches, you want to start with the USA spending searches? Yes. All right. I will. Great. I will share that first. Okay. All right. Here it is. Okay. This is, this is how to do an advanced search. There are so many filters. I didn't even include all of them on here. And you don't have to use all of them. You can just use a few. You can do a filter to find out foreign countries too. You can filter universities in foreign countries. The name of a recipient. And anyway, this is just, just to get you started. Don't be afraid of it. It's quite logical. And that's where you just do an advanced search on USA and you find you could use a date range. You can limit by location. I sometimes have limited to government because a lot of money comes from the Department of Defense to local governments, state and local governments, enormous amount. Even the Department of Environmental Services of New Hampshire gets grants from the department, from the military. Yeah, that is so interesting. So if you go to USASpending.gov and you do this advanced search, you can see different entities that receive funding from the department. If you want to do that, or you can just do it by your county. You can see what gets funded in your county or all sorts of things. But anyone, people have questions, they can send them to me through my website. Yeah, that will be sent to everyone too, my website. Yeah. Great. And do you want me to share the last one? Yes, we'll do right now. Okay, this is just a sample search results. One of the searches I did, I used the time period just four years. Yeah. And I got 11,000 results. And the state was New Hampshire, awarding agency Department of Defense. And the recipient type was all businesses. And this is what I get. This is just the first page, of course, it goes on and on and on. And you can see BAE systems really gets next gen that firm changed names and moved out of this country. But BAE is the largest contractor in New Hampshire. And it gets almost about half of all the DOD money. And it was also considered to be the largest firm in Europe, the largest weapons maker in Europe. And it's a British owned system, British owned corporation. And one of the things that I found on some website was the percentage of military production of the various corporations. So that BAE has 95% of everything it makes is military. Whereas Boeing, with Boeing, it's 44%. Boeing also makes civilian things. But BAE hardly makes any civilian. It's an enormous contractor. This is so interesting because you can see, like you said, obviously, you just search for the awarding agency.  But then it's listed by size of contracts. So right away, you can see BAE is getting the harvest contracts. That's right. That's right. The disadvantage is, in order to find out how much they got in those four years, I'd have to go through all the things and add all the numbers up. And that's a lot of work. If a lot of people are doing this work, they can they can do that. Yeah. Yeah. Yeah. And other foreign owned companies are very prominent in New Hampshire. Elbit. Elbit is Israeli owned. And they are one of the largest contracts. That was one picture you couldn't, you didn't show from the Department of Defense of State, Military Defense Spending by State. The Elbit systems is one of the top 10 and Raytheon. But also, some of the information technology companies are also in part of the top 10. Yeah. Yeah. Great. That's that's really useful to know how to actually do that kind of research, because people we work with want to know in their local communities, right? Which which companies are actually getting these contracts because the Department of Defense is, you know, the Pentagon budget is notoriously very opaque. So really, really want to know. We do have a couple of questions, Joan from the audience. And also, I have a couple of maybe follow up questions about some of the really interesting points you made, because, you know, thank you also for that really extensive overview of, you know, the different ways in which the military industrial complex is really embedded in our communities, right? You've talked about, obviously, from from education to philanthropy, really, really interesting. And I guess, you know, one question that I had, and then I'm going to go to a couple of from the audience was, you know, I know that you wrote a book about sort of just transitions, maybe to a green economy, right? So I, you know, obviously, a large part of the influence or the main part of the influence that that the Pentagon has is is the amount of money that they receive. And you make a point, obviously, that people who are employed by these companies or these contractors aren't doing it out of malice, but out of, obviously, just a material interest, like they need a job, right? So what what, I mean, we need to just transition away from from these kinds of jobs, not only literally the military, but also these weapons manufacturers.  Is there any sort of great model for that that you've seen or anything we should keep in mind when we're talking about a just transition with people? I think I think more people have to think about it and figure it out, try to figure it out, because what I see like from the National Priorities Project, they say there you can get more jobs in housing and civilian infrastructure than yes, of course, we know that the question is how to move how to change what's what's embedded. And I don't know the answer. I just wish more people would work on that question. And part of it, what I can do is make it a little more visible to see what we're up against. And, you know, because so often people say we're up against the profit tiers. And it's it's much more than that. It's much more than the profit tiers. It's it's I call it military Keynesianism. I think I think the capitalist economy is very weak. And it's supported in so many ways by government. It's, you know, we have a mixed economy. Most countries in the world have mixed economies, including China. So, so this is a backhand way of having industrial policy. We have to change that. I would like to see first of all, a Department of Infrastructure, or whatever it's called, that the army engineers would be absorbed into so be a Department of Civilian Infrastructure. So we could at least get those people away from the military, you know, doing their own thing. Right. But I I can't say I know how I think I think it's a very tough thing that we're up against. And we have to unravel that and not just rail against profit tiers. Yeah, I think I unravel this. I think that's definitely extremely important, you know, the details. And, you know, we have we have had conversations with people who have talked about transitioning from making weapons to, say, green infrastructure, sort of like what you said, right? Like, we actually need that in the year now, actually. Right. Excellent. It may be a way of doing that. It could it could be. Yeah. And that would be that would be great. Because without even without the pandemic, we need the government to keep the economy going. Yeah. And yeah. Yeah. So it may be. Right. Something. Yeah. I think that's a good point. I mean, we have large amounts of government funding. We're just our government's choosing to funnel it into the military when that is precisely not what we need. Okay, great. So we have another question also. And this person was asking, Joan, can you also speak to how media at every level is supported by the war business? And also their commitment to war spending is largely invisible. So how how war spending is reported on in the media? Oh, well, I don't think it's reported. It's there isn't a lot of report in the media on on on this particular subject. There are some I mean, I think that I'm thinking about the what I would call the anti war left media there. There are there are some very good reports by people like Bill Hartung, or Nick Ters, and that's fine, or or Christian Sorensen. And I've listed these things in the thing. And they're they talk mostly about the weapons makers. But it's all the other things that Nick Ters wrote a wonderful book called the complex. But it was years ago. It needs to be updated. It was very influential. He talks about how all the companies, Dunkin Donuts and various other companies, how how much they're involved in the military. But we need more more exposure of this in the media. Yeah, I think that's that's a really great point. And sort of obviously the work that you're doing is investigative, right? And we need also more investigative journalists to do this work. Yes, I've written to Nick and asked him, please update your book. But I haven't seen him doing that. He's, you know, he does wonderful things about the special operations forces all over the world. That that's important, too. But I have this little corner that I want to expand.  Yeah.   And I'll just say also to that question, right? It's been difficult even over the past few months, six months or so with Congress, even deliberating on the Build Back Better agenda, the reconciliation bill, right? Media reports on that has spent as costing $3.5 trillion. But when it comes to speaking about the Pentagon budget, they report on that as costing $778 billion. So the discrepancy there where we talk about the Pentagon budget as they're the yearly funding, but social spending programs as the decades-long spending, it seemed larger than it is, right? Just that is just a little example, I think, of how the media really distorts this conversation in a lot of ways. One thing that of course should get more publicity is the idea of the upgrading of nuclear weapons because our country still has nuclear war as an option. Yeah. And that should be shocking to people, you know? Yes. It's still an option for our military. Yes. And the fact that we're spending, we want to commit to spending billions of dollars to, like you said, rearm. And the danger to the whole of the world, yeah. It's hard to comprehend why, you know, that would be even on the table. Absolutely. So I think we have another question. This is from Phil. It says, I believe that as indicated earlier, America does not question the need for, quote unquote, defense or any money for it. So as long as people feel fearful regarding our security, the military will have sort of a publicity edge, right? People will want to fund it. So Phil's question is, don't we need to change our thinking about our security and that we need to work for peace, not war as a basis for challenging military spending? Certainly. And there are certainly people doing this. And we have to hear about it more because is the question, are we safe? Is this military making us safe? And obviously, there's so many hundreds of ways that it isn't. Even our 800 bases all over the world become targets and likely to something could be inflamed in one of those places. You know, if we're if our bases attacked in some place in the world or our troops, our special operations forces are now very happy in not only in Africa, but also in the Arctic. I mean, is it making us having nuclear weapons or proposing nuclear war? Is that really making us safe? And what are we defending against really? No one's, you know, people aren't interested in attacking us. Right. Okay, we have these aircraft carriers that don't even work. Right. Right. Or are they F-35s that have blown up? Yeah. Oh, yeah. Oh, absolutely. Yes. Yes, this is a comment about hybrid war. Absolutely. Now, yes. Dirty tricks, assassinations, overt action of all kinds, cyber warfare. Now it's an attempt to change our cognitive ways. That's the latest thing. Not just to change the information, but they're trying to figure out how to change the way people think all over the world. I mean, yes, definitely. It's called asymmetrical warfare or yeah. Yeah. But they're still making those aircraft carriers. In addition to this, yes, they're doing they're doing both sides. They're not giving up the other stuff. Right. I think it's this, it seems like almost a contradiction, right? They want to make wars as invisible, I think, to the majority of American public as possible, like the costs of war, civilian deaths, obviously, but also invest, you know, close to a trillion dollars in war every year. And those things that we're investing in, like you said, right, they're not making us safer. They're making companies profit, right? And so I think, you know, holding those two things at the same time is really important. Yeah. So, and obviously, like you said, right, making these connections visible is also extremely important because the Department of Defense certainly doesn't want that to happen. Wonderful. I'm not seeing any other questions. I think what you gave us was extremely comprehensive. And also I know that people will be interested in looking at that map that you showed us and doing some of their own research. I personally am going to go look up where I live in California, just the University of California alone, right? And people are asking what is the website, we will make sure you have the link to that in a follow up email. So, but Joan, before we hop off, I wanted to see, are there any last sort of thoughts or things that you wanted to add about this conversation that we've had? No, I'm very happy with the turnout. And I hope people, I really hope that people follow through and it would be really nice to have a website with every state having a, not only a map, but a description of some of the interesting contracts. And so we see how tremendously militarized the country is physically, and the state of Washington, for example, and California too, California has the most bases. It's one of the highly, highly militarized states. But then throughout the country, there are all kinds of places that have survived places in Michigan and Wisconsin towns that otherwise would be rusted away. And I have looked up zip codes of some of those places that are thriving and found very often it's because of military contracts. Yes, exactly, right. And I think a really great way to also end this is to say something that Phil Nichols posted in the chat, which you think is really just on point, which is transparency is a powerful thing. Look at how our government is handling Julian Assange and Daniel Hale, absolutely right on point. And I also just also wanted to end by again, thank you so much, Joan, for joining us and for that really great presentation and look into how we can actually do this research together. And also just to remind people, right, we've talked quite a bit about the role that not only war profiteers, but the entire military industrial complex has in our lives and almost every facet of our lives. And, you know, over half of our Pentagon budget goes directly into these military contractors, like you said. And really that won't end until we cut the Pentagon for people, peace, planet and a future, which is why it's really imperative that we create a big tent of people and organizations who are working towards this goal together. So I invite everyone to join this movement to cut the Pentagon for people, peace, planet and a future by going to cutthepenagon.org. And otherwise, I just want to thank you so much and people are thinking you also in the chat, Joan, and really appreciate your time with us today. Yes. Are you going to send me the chat to print out? Absolutely. Thank you very much and thank you for moderating so well. With all our work together. And I'm happy to work with Code Pink, too, in the future. Absolutely. In all sorts of ways. Yeah. Absolutely. And thank you so much, Joan. I really, really appreciate your time. And thank you everyone for joining us. What a great, what a great time.


Hello everyone. Today we shall talk about how we tackle some operational and capacity utilization challenges at Workday. For those of you who haven't heard of a Workday, it's a software-as-a-service company that offers a single solution for finance, human resources, planning, and analysis. Our presenters today are Bogdan Katinski, Silvano Wubak, and me, Imtias Choudhury. Here is our agenda for today. First, I shall give a background of capacity management and operational challenges that we face, which comes with growth. Then Silvano and Bogdan will discuss some of the technical solutions that we came up with to address these challenges. Now let's get to the story. OpenStack is continuing to grow within Workday. Over the last few years, we've seen more and more applications running on OpenStack platform. As of now, we have approximately 39,000 VMs running on this OpenStack platform. Over 45 clusters spread across five different data centers in the United States and in Europe. Just as with great power comes great responsibility, with bigger clouds comes tougher challenges. One of our challenges is to come up with an accurate capacity usage and growth forecast. We need accurate capacity information from all of our clusters, not only at the cluster level, but at each hypervisor level. It is important for two reasons. First, to be able to accurately predict whether in our weekly maintenance window we can support any potential changes to the number of VMs we are launching. Second, we need a reliable growth forecast so that the platform can support any growing capacity needs of applications. Accurate forecasting is important because ordering hardware takes time and even more so during a pandemic. Bogdan will explain how we address this. Another challenge we face is to optimize resource utilization. We run different types of workloads on our OpenStack clusters. Some are memory intensive, others are CPU intensive. With such different types of workloads we noticed our hypervisors were underutilized. We needed to come up with an optimization scheme which maximizes capacity usage while reducing blast radius in case of a hardware failure or a hypervisor failure. Silvana will elaborate more on this. We also need to improve operational efficiency by reducing the time it takes to apply patches to a server. Servers often require periodic firmware and software updates. These updates often require servers to be rebooted. Since rebooting a server affects any running workload, we wait for a long maintenance window, usually quarterly, to do such operations. Trying to reboot thousands of servers in a short maintenance window is non-trivial. It's risky and it slows down rollout of important firmware updates. But what if we could tweak the scheduler so that the VMs avoid hypervisors that require maintenance, thus allowing us to apply changes in smaller batches of hypervisors more frequently? And Silvana will elaborate more on that. With that, I hand it over to Silvana. Before I start to talk about changes in the scheduler, I want to talk about two metrics related to VM allocation. One important requirement for our VM placement algorithm is maximize the high availability of our applications. In short means avoid deploying all instances of a project in the same host, so in case of the host dies, the impact is minimized. This is measured as blast radius and I want to show how we calculate in Workday. First, I want to show how to calculate for a single host. For example, for a given host X, if that host dies for project A, there is a drop of 20% of project A capacity because the project A have 1 out of 5 VMs in host X. This is the blast radius for project A in host X. But for the whole host X, the blast radius is the maximum blast radius for any project running on that host. In this case, it is 0.4 or 40%. In Workday, we don't calculate the blast radius for a project with 4 or less VMs in the cluster. This is why Project D doesn't have a blast radius. To calculate the blast radius for the whole cluster, we use the average and the maximum blast radius for all individual hosts. Note a lower blast radius is better since it measures the drop in application availability. Fragmentation is also metric that we aim to reduce. Fragmentation could be for any resource, but in Workday, our main concern is about memory. Fragmentation of memory measures how the space available in the cluster is spread over the host. For simplicity and make this meaningful to a broader audience in Workday, we measure fragmentation as the space available to deploy a VM that requires 240 gigabytes of memory. When comparing different strategies of VM placement, the solution with less fragmentation is the solution that provides more space to deploy those big VMs. Note in this diagram, in the deployment case 2, despite the hosts have the same capacity as in the case 1, we can't deploy the second VM in the Project B because the space available was split between two hosts. Also, the blast radius for both hosts is 1 or 100. This example is to emphasize how the order that we launch the VMs can change those metrics for the worse or for better. To give a perspective about how fragmentation is very important, in the past, in Workday, we were in a situation where across with around 300 compute nodes and average memory utilization of around 80%, and we couldn't deploy a VM with 240 gigabytes of memory because all space available is too fragmented over all hosts. Note 20% of capacity for a cluster of this size means around 30 terabytes of memory available, but no single host with 243. Out of the box, OpenStack already provides a solution to deploy different flavors, balancing fragmentation and brass raider. This solution is host aggregates. In Workday, we have used the approach of one host aggregates per flavor, a separate flavor with memory below or equal 64 gigabytes. Those flavors belong to the same aggregate. We developed two to adjust the host aggregates dynamically, based on the number of VMs per flavor. Using a host aggregates is easier to achieve a good packing. That is a big pro, but there are limits with host aggregates. For example, when a flavor being used for only one project and doesn't require much capacity could have all VMs in one or few hosts increasing the brass raider. Another problem is when the capacity of one host aggregates reduces or projects change flavors. Look at this example. There are two flavors and two aggregates, one for each flavor. VMs are the orange box and host the blue box. Letters are the project name. VMs for project A and B are in the same aggregate because they have the same flavor. But if project R decides to change their flavor to flavor 2, let's see what happens. Since the capacity needed to deploy flavor 1 reduced and the capacity needed to deploy faithful 2 increased, I need to move few hosts from one aggregate to another. But moving the host doesn't change the VMs run on it. In our example, project A was redeployed as part of their capacity change, but projects B have no requirement to do a redeploy, and those VMs keep landing the aggregates for flavor 2. Host aggregates work well if you just grow capacity. If your capacity changes frequently, they are hard to maintain, requiring a lot of operational work and a lot of resource to keep migrating. This means that we need to move from one host to another. But if we plan to change the VMs to host, how can we ensure there is no capacity issues? How to ensure we are not increasing fragmentation and brass rate? We develop a simulation tool that mimics what the scheduler does. That makes it easier to load the data from a cluster and simulate different placement strategies. The simulation tool was developed in Javascript to be more interactive. Just a static page with Javascript doesn't require any back-end. Filtering and waiting are simple to write in Javascript. A Python script extracts the cluster information in JSON format. The tool receives as input a cluster definition. That is the list of all hosts, all flavors and all VMs running on the cluster. Also, the scheduler modifiers. There are x-axis containing the host, three in this example. In the y-axis is how much resource are in use without overcommit, so can go over 100. The bar is the amount of memory consumed by VMs in the host and the line is the amount of CPU. Bar have different colors, depends on the flavor. In the border, we can simulate, create and destroy VMs. Any create will execute using the new multipliers. You can simulate deploy one or a few VMs, but also have deployed the entire cluster with new parameters. After many tests and simulations using our simulation tool, we found a way to balance the brass thread and fragmentation in Workday. No more host aggregates management and VM migrations. Note one important goal of this force implementation is to have an algorithm that is simple enough to understand why a host was chosen. In our proposal, all suitable hosts are available in each deployment. Different from host aggregates, where just a limited set of hosts are available. This maximizes the capacity available and reduces the brass thread. To achieve this, we use a feature in the scheduler called Wagers, where hosts can be preferred, but not limited. Because we want to minimize the fragmentation, Workday uses a memory wager with multiply minus one to stack the VMs. Note that alone increase the brass thread, because the scheduler will try to stack all VMs in the same host. To reduce the brass thread, we combine the memory wager with our own implementation of soft and toughened wager. Different from what OpenStack offers out of the box, our wager is configurable to accept the four VMs in the same host before starting to spread. Also, it doesn't require a server group. We use the project ID to know if two different VMs should be placed together. This way here, have a higher priority and has a multiplier minus two, because the development is actually soft affinity. The negative multiplier always inverts the order. I will explain why we choose the scale of multipliers later. Here are the results after change the scheduler. Not a huge improvement in capacity due to the reduced fragmentation. For most cluster, the average brass thread has increased a little while the max brass thread has reduced it. And because the solution is tacky, the number of hosts needed is lower than before. Just to clarify the capacity around the 240 GB VM are after we have executed all deployment. Periodically, some pets need to be applied on the host, like security pets. As the number of compute nodes increase, this is becoming harder because our maintenance time does not have increased. But this procedure could be fully automated if the schedule is also aware of the maintenance. Because this is not specifically maintained, the decision is based on hosts uptime. Hosts running longer get less priority on the deployment. After some time, they will become empty. Weekly, all empty hosts are patched and hay start. After hay start, the uptime is zero and the host gets higher priority on deployment. And the second group of hosts with longer uptime will become empty. The uptime information is cached on the scheduler to avoid performance issues. The uptime on blast radius and fragmentation is negligible because the uptime can assume just few discrete values. For a specifically host maintenance, we still use the regular procedure to disable hosts. But this solution gives us the maximum number of hosts empty to be patched without compromising the capacity. The first idea how to wait the host based on the uptime was to give a plus 1 wait on every 60 days, for example. So, there is no difference between a host up for 10 days and up for 50 days. But after we analyze the distribution of uptime among all compute nodes in all clusters in Workday, we realize is necessary a more sophisticated formula. To ensure the uptime waiter can assume no more than three distinct values. Those three values divide the host in three groups based on the uptime, minimize the impact of rath rate and fragmentation. Next slide, I will explain that better. This slide explains why we choose multiplier 1, 2 or 3 for the way here. I'm oversimplifying here, but hopefully that gives everybody an idea how to make a choice. First, note that all waivers have their values normalized between 0 and 1. In these diagrams, focus on the host with wait 0. Because the uptime wait is multiplied by 3, the winner host is in this group. But in the group with uptime wait 0, the uptime is no more relevant since all hosts have the same wait in uptime. Same idea for the anti-finity. Few hosts have more than four VMs of semi-project running, and the remain have not. So, the memory multiplier will make the final decision in the last subgroup. Let's talk about some of the challenges and requirements for capacity management at Workday. First of all, multiple independent microservices run on the platform and each may need to increase their capacity at a different time, for example when releasing a new feature. A central deployment plan controls the service deployment. In the past, the deployment plan was stored in a SQL database and requests to update it were submitted by JIRA. This process was error-prone and, moreover, before updating the deployment plan, engineers responsible for capacity management would only check if the total amount of RAM, CPU or disk is enough to increase the footprint of a service. And they didn't take into account how fragmentation and the order in which services are deployed affects the capacity. In the past, we had issues during our weekly maintenance window caused by a too large increase in the capacity of one of the services. Because of those issues, we made a requirement that the new capacity management process needs to validate if there is enough capacity taking into account fragmentation and also how the deployment order may affect capacity. On top of those issues, when a capacity issue occurred during a maintenance window, it was difficult to find which services updated their capacity most recently. So we also made a requirement that the new process has to make finding and reverting the most recent capacity changes easy. Another good-to-have feature was to make capacity management self-service. So instead of submitting JIRA requests, which had to be copied to the deployment plan by the infrastructure team, we let service teams make changes in the deployment plan directly, but still give the infrastructure team the power to approve or reject them. At the beginning, we were considering building a new tool with a web UI, but then we realized that we have the solution already available. We picked Git with Gerrit as the code review tool and Jenkins as it fits the business requirements perfectly and most of the users already know and use Git every day, so we didn't have to reinvent the wheel. Additionally, treating capacity changes like code gives us all the benefits of a CI pipeline, including tests. I will walk you through that later. Okay, so let's dive deep into how the new Git plus Jenkins workflow works in our capacity management system. This diagram shows the process of changing the capacity for one of the services. It all starts with a member of the service team updating capacity in a YAML file with the deployment plan. This change triggers the validation job running on Jenkins, which checks if there is enough capacity to increase the footprint of one of the services. Finally, the end of Ops team, which is the gatekeeper of capacity requests, gets a chance to review and approve the change. Merging the change to the main branch triggers another job on Jenkins, which updates the SQL database used by the deployment tool, so the next service deployment will use the updated deployment plan. Let's now have a closer look at how the capacity check from the previous diagram works. For this check, we use the same simulation tool that we use to simulate the changes in the scheduler which Sylvano talked about. Because we use Kube Jenkins, we can package the simulation tool and all helper scripts needed to run the capacity check in a Docker container. The data on the physical capacity of our clusters is stored in surf and is periodically refreshed by a Chrome job which runs on each cluster. The validation test pulls the cluster capacity data, reads the updated deployment plan from Git, reformats the plan to meet the input format of the simulation tool and runs a series of simulations. We simulate the best case deployment ordering, the worst case ordering and the number of random deployment orderings. If all simulations are successful, the job gives the change a plus one approval, which is a required but not a sufficient condition to merge the change. If, on the other hand, at least one of the simulation fails, Jenkins returns a minus one to Garrett with an error message explaining how much capacity needs to be added to the clusters to make room for the new deployment plan. In addition to approving or rejecting changes, the simulation tool generates data which can be used to visualize and summarize the state of all clusters after deploying the new deployment plan. The data is again in JSON format and we use Jenkins archive artifacts plugin to archive this data together with a static HTML file and JavaScript code which contains an application to browse and visualize the data. Since the simulation tool is written in JavaScript, we can reuse the same modules to generate graphs and render tables with the results directly in the browser. The graphs are interactive with a dynamic legend and labels which appear after hovering over a data point. We also present tabular data with some key metrics related to capacity and the deployment. So what are some of the key lessons that we learned? First, we learned that capacity optimization is a tough problem deciding on what to optimize itself is the biggest challenge. In our case, we defined the notion of last radius and fragmentation to determine what to optimize on. Simulating against real data from all our clusters helped come up with a good optimization algorithm. Second, when you're optimizing, account for all types of scenarios and resources, for example IP address. If someone requests to increase the number of VM they're launching on your cloud and you don't have enough IP address for that project, that itself can be a problem. Also to take into account is the deployment model for an application or microservices. For example, if someone uses blue-green deployment, then they may require for a short period double the capacity when they're rolling out a new version of the application. Finally, we learned that NOVA scheduler itself is very extensible. We gave you examples of how we managed to optimize resources by writing a plugin without requiring any changes to the main NOVA code.


I think some of the sort of international trends that are noteworthy is that it's sort of due to the conditions, but folks in the Middle East are really hearing themselves towards private production. I think what the other things that you find in West Africa is that they are really interesting care models that amazing entrepreneurs looking at fintech, especially in new ways to get primary agriculture going in a private sense. There are all sorts of development agencies working across the continent, very much focused on primary production and food security. So yeah, I think the big thinkers across the world, folks that face a crisis in a couple of decades from now have really started to invest in agriculture in a big way, so South Africa differently lags behind in that as well.