Good afternoon, Team Kings Bay, eddies. What a great day it is to be here in the Navy and the Marine Corps team in Southeast Georgia as we are fortunate to have the Secretary of the Navy, Ray Mabus, here to visit and witness firsthand the incredible amount of talent displayed by the collective Team Kings Bay. I'm proud to have the opportunity to show the Secretary the important contributions that each of y'all make every day to our nation and our military. Today has offered a wonderful opportunity for some of you to share your stories with Secretary Mabus and show them the extraordinary effort and pride you put forth in your profession. For those of you who did not have a chance to meet the Secretary during his tour, the Secretary has graciously offered the opportunity afterwards to say hello to him individually. But before we hear from Secretary Mabus, allow me to share a little bit about his background. Ray Mabus, 75th Secretary of the Navy and leads America's Navy and Marine Corps team. As a leader of the world's only global Navy, he has traveled more than 800,000 miles to over 100 countries to maintain and develop relationships with national and international officials and visit with sailors and Marines for deployed or stationed around the world. He has traveled to Afghanistan on 12 separate occasions in recognition of the sacrifice and service of Marines and sailors deployed in combat zones. Before his appointment, he held a variety of leadership positions. From 1988 to 1992, the Secretary served as the governor of Mississippi, the youngest elected to that office in more than 150 years. He was ambassador to the Kingdom of Saudi Arabia from 1994 to 1996, and later was chairman and CEO of a manufacturing company. Shipmates, it's my pleasure to introduce the Secretary of the Navy, Ray Mabus. Sir. Appreciate you. Thanks. Admiral, thank you and thank all of y'all. I take these trips for a couple of reasons. One is to get out and see the amazing work that you do. Day in, day out, the level of skill, the level of expertise, the level of competence, level of training, the level of education that every one of you has, and the work that you do for the country. The second reason that I come is to try to underscore for the people of this great country that work, because so much of the work you do is not seen. So much of the work you do is away from the American people. The Navy and Marine Corps, when we were forward deployed, our America's away team. And a lot of times when the subs that leave here, when sailors and marines go around the world, we're a long way from home. We're a long way from home, a long way from family. And people just don't get to see what you do. People don't get to see and understand what you do. And the third reason is to thank you for what you do. Thank you for that skill level, that dedication level, that level of patriotism, that level of sacrifice, regardless of where you serve, regardless of what you're doing, it is hard and it is a sacrifice to wear the cloth of the United States. In my time as Secretary, I've tried to concentrate on four things. Four words that start with P that gives you the fifth P. The fifth P is the one thing that the Navy and Marine Corps uniquely give to this country and that's presence. The ability to not just be in the right place, at the right time, but the right place all the time. And those four P's, the first one is people because you are our, you're the edge that we have. We have the most advanced platforms, most advanced systems, most advanced weapons in the world, but without the sailors, marines, to operate them, to maintain them, to make sure that they are doing the job that America means for them to do. It takes this level of talent. And so we're trying to do things with you and for you, things like 21st Century Sailor and Marine to put all the programs that we've got for resilience of the force in one place.  Things that we can do as a Navy family to make sure that after more than a decade of incredibly high tempo operations that we maintain the best force the world has ever known which is what we have today. And unlike any other service, as we come down from two ground wars in the Middle East out of Iraq coming down in Afghanistan, there are no permanent homecomings for sailors and marines. We go back out, we deploy, we give that worldwide presence. And that puts an amazing strain on you and on your family. So we're trying to do things for sailors and marines like increasing sea pay.  If you've been at sea for more than 36 months in a row, you're gonna get the next level of enhanced sea pay. And if you're E5 to E9 and you've been at sea for eight years in your career, you stay at that high level when you're at sea. And our deployments we know are getting longer. They are getting more complex. And so we're also gonna be rolling out things that if you're at sea for longer than a certain number of days that the higher sea pay will kick in then. The second P is platforms on 9-11, 2001, US Navy at 316 ships. By 2008, after one of the great buildups in American history, we were down to 278 ships and sinking, that number was falling. The four years before I became secretary, the Navy put 19 ships under contract. And one of the things I'm proudest of is in the first four years that I was secretary, we put 60 ships under contract. We are growing the fleet again. We are gonna be at 300 ships by the end of this decade. And we're gonna stay at 300 ships because even though these ships are incredible platforms, even though they're the most advanced platforms the world has ever known, quantity has a quality all its own. You've gotta have enough ships and there's gotta be enough of the right kind of ships to make sure that we have that quality presence. The third is power and that's energy. Energy is a national security thing and we're moving the Navy onshore and at sea to using non-fossil fuels. Right here, these submarines, all nuclear powered, all our carriers nuclear powered. But the reason we're doing this is to make us better war fighters. In two years, 2011, 2012, the Navy got a $2 billion bill, $1 billion a year in unbudgeted fuel increases, just the cost because of price spikes. Because every time something happens in the world, price oil goes up. It's called a security premium. Now, there are not many places to go get a $1 billion that you have not budgeted for.  Or if the bill gets too big, you buy fewer platforms because you cannot afford the platform and you can't afford to fuel the fourth partnerships. And that's partnerships of a lot of different kinds. Partnerships between Navy and Marine Corps and private industry, like there are great examples here in Kings Bay, partnership with the American people whom the sailors and Marines defend on a day-to-day basis, stand the watch, and partnerships with our country, countries around the world, with our friends, our allies. Because no matter how big we are, no matter how good we are, we can't do this by ourselves. And we have to be interoperable with other people. We have to have that cooperation with other people. And that's the reason, besides seeing sailors and Marines where they are forward deployed that I travel so much to see these other countries. Because we do work, those 111 countries I've been to, we do something with every single one of those countries. I'll give you a quick minute about Washington. As you know, we got Congress passed the bipartisan budget agreement that gave the top line for fiscal year 14 that we're in now in fiscal year 15. We're right in the middle of budget hearings, C&O and the Commandant and I have had one, we've got three next week to talk about the budget. Now, it was good that Congress did this. It gave us a little stability, a little ability to plan. But one thing to remember is that at the end of 15, if something's not done, we go back to the sequester level, to these dumb cuts that we're gonna have to make. And coming out of two wars, the American people have a right to expect that we're gonna spend less on defense, but we ought to be smart as to how we do it. And we ought to, as we draw down, we ought to make sure that we do so in a smart way and a way that puts money against strategy. And if you look at our national strategy, the three big areas, focus on the Western Pacific, focus on the Arabian Gulf, and focus on partnership building, that's a maritime strategy. That's a definition of the United States Navy and Marine Corps. Nobody can do it better than we can. So 14 and 15, although there's some debate about individual things, we have a little stability. Past that, we need to get it fixed in order to keep the readiness in order to keep the force that we have today going forward. Now one of the things that every one of you is right about and sure interested in is what's gonna happen on things like benefits, compensation. What's been proposed is that we slow down, we slow down the growth of personnel cost. Nobody's gonna get cut, but slow down the growth because where we are is if we don't, we're gonna make a choice between giving you the tools that you need, giving you the platforms that you need, the weapons that you need, the training that you need, or keeping up with this high growth. The other choice that we will also have to make is there'll be a lot fewer opium because one way to deal with personnel cost is to reduce the number of people. I don't think either one of those is a good idea. C&O was down here, he talks about benefit, pay, things like that, is quality of life, but you've also got the quality of work and the quality of work for what you do as sailors and as Marines and the ability to get you what you need, the ability to get you while keeping faith with you but to get you what you need. And the changes aren't big, the changes are pretty modest, but if we don't do it, the choices get worse and worse. The longer time goes. When I finish, which is gonna be pretty soon here and I know y'all are happy about that, standing out there in the sun, I'll take some questions. I think we got mics here. And then anybody wants to take a picture, shake hands, let your mother know that you're actually at work, I'll be happy to stay as long as you want to and do that. The only thing I would ask is if you're gonna have somebody else take the picture, don't make it on the job training. Teach them how to use your camera or your phone before you get to me, okay? I'm gonna end the same way I started. Thank you for the work that you do. The level of skill, the level of expertise at Kings Bay, the level that I saw today, the level that I get briefed on all the time is extraordinary. Nobody, nobody else in the world can match what we do. Nobody can touch you in terms of skill level, in terms of education level, in terms of how well trained and how good you are. Thank you for doing that. America depends on you. Whether they know it or not, whether they appreciate it or not, this country is in your debt. So Kings Bay, thank you. From the Navy, Semper Fortis, forever courageous, from the Marines, Semper Fidelis, forever faithful. Thank you all very much. So who's got a question? Anybody? I convinced everybody. Way back there. You gotta come up here. First question, gotta come up here. It's okay, really, come on. There was a hand way back there. First question gets a coin, but only the first question. Thanks for the coin, sir. I'm E.M. One-Spiker, USS West Virginia. I was wondering if the developments in Ukraine and Russia are gonna impact the overall mission of the Navy and here in Kings Bay with the strategic missile deterrence? President, yesterday said that we are not gonna have a military response to things in the Ukraine. But one of the things that events like that or other events around the World Show is just the need for that presence. I mean, we were there. We were, we've got ordinarily and do today. Have a ship in the Black Sea. We've got Marines, Marine Black Sea Rotational Force. The presence that we have, no matter what happens, no matter where it happens, gives our leadership options. They get, the president makes that decision, but we give him options. In terms of the strategic deterrence, that, you know, it's a, y'all have an interesting job because success is when nothing happens. And I hope you're very successful. But it shows everything, shows and has history as shown, the need for that constant deterrence, that need for that constant presence. Of whatever, of whatever kind. The last thing I'll say is, you know, I went on with Virginia today. We are now designing and doing the engineering work to do the Ohio class replacement. So the subs that come behind to do the strategic thing. We're on schedule. We're on budget. And we're gonna start building the first of those ships to go on patrol in the late 2020s. But we gotta start building them in the early 2020s to do that. And Kings Bay is ready to deal with those replacements today, even though we're beginning that process to get there. The last thing I'll say is that because of the cost of the Ohio class replacement, we're gonna have to have a conversation in this country, particularly in Congress about how we pay for it. Because if Navy pays the entire bill, it'll take between a third and a half of all our shipbuilding budget. And so all our other programs will just be, it'll be catastrophic to the rest of the programs, including other submarines. So including our attack submarines. So this is a national asset. This is a national thing. We ought to have a debate about how the country pays for it. So thank you. Thank you, sir. Who's next? Well, the first line is always gonna be chaplains. It's gonna be the other professionals that you talked about. There may be some specialized instances on counselors that mainly for bases, there may be some instances that they'll go aboard ship, but our chaplains are stretched pretty thin right now. And particularly when we're at sea. And giving the great chaplain corps the other professionals that we have out there to talk about things like suicide prevention, like family readiness, sexual assault, the things that have the potential to undo so much good work that has been done. I don't think we can do too much of that, but I wanna thank you for you and the other chaplains out around there because your ranks are stretched very thin. And just as a very personal aside, I ran for governor in 1987. I was a guy that just graduated from college who drove me around for a year. He is now a Navy chaplain who deploys with the Marines. And so I've got a little bit of a personal life line into that community, so thank you. Sir, MT1 Starkey from TTF. With the budget cuts and stuff, why is the Navy still paying senior and listed to perform as a detailer when they're no longer the sailor's advocate when that job could be performed by an E3 or E4? Oh, I'll give you a very honest answer, I don't know. But, I mean, just thinking through it, you want some senior and listed to talk to people coming in? You also want some of those E3s, E4s, do it also. But you gotta have some senior folks there just to explain what life is like throughout a Navy experience to be the first mentor that people have. And as I said, I can't give you a technical answer to the question, but that off the top of my head is the best I can do. Anybody else? There you go. EM2 Briggs from the USS West Virginia. Requests to know what is the future of the Navy's retirement program and whether or not these are going to be retroactive? Well, there's a group looking at retirement that Congress set up and they're gonna report back pretty soon. One of the things that has been that everybody, whether in uniform or civilian leadership has said since the beginning of this process is that whatever system you join under ought to be the system you retire under. So if any changes are made, as I said, this group hadn't reported back, but the clear expectation is that if there were changes, it would apply to people who joined after those changes were made. Thank you, sir. All right. One more. Y'all don't wanna go home, do you? Late on a Friday afternoon, is that? Yeah. Good afternoon, sir. AB2 Brown, attached to Sub-Ace. My question is, is there a possibility that we are still looking at homesteading the service members in certain regions, such as Southwest, Southeast, or the other ones? Nope. No, I mean, sometimes because of your rating, sometimes because of some specialty that you have, you may end up in one fleet concentration area for a long period of time, but the notion of homesteading is probably, no, it's just not there right now. Thanks, sir. All right. Kings Bay, thank y'all again. Thank you for the things you do for America, and thank you for the skill, the way you do it. As I said, I'm gonna come right down here. Y'all, when your dismiss, break ranks, come on up, anybody wants to take a picture? Shake hands, I'm happy to do it. Just make sure you know how to use your shipmates' phone before you get there. All right, thanks, Kings Bay. Shipmates, let's give Secretary Maynus another round of applause for what we would like to do, sir, is to present you just a couple of small tokens in appreciation of your leadership and thanking you for coming to visit us today. It's a first here, a model of USS Georgia. And I'm very proud to present it to you, sir, from the entire team here in Kings Bay. Thank you so much. That is awesome, thank you. I've got one other thing for you, sir. Just a commemorative token here presented to the Honorable Ray Maynus, Secretary of the Navy, from Team Kings Bay. Thank you, sir. Thank you so much. Appreciate it. And at this time, Secretary Maynus will be available right up here in front, or at the torpedo display for anyone that would like to take photos or otherwise greet the Secretary. Dismiss.


Have a seat just to make sure that we can hear you.  I'm gonna leave mine on but Before you begin and so we can test out the microphone Could you give us your full name spelling each of your names? Yep I think I'm gonna be pull up just a little closer to the microphone perfect. Thank you It's just for sure Thank you very much your honor miss Weiler the the key I'm going to give you is that when you speak you should be able to hear your own voice over the speakers All right, so just lean close enough to the microphone to be able to do that. All right Are you originally from Minnesota? Where where did you grow up? No? Thank you, thank you, and you can even pretend you're yelling at me if you want to or make your make your voice hit the back of the room All right, okay, so you said that you grew up in Arizona. Is that right? Yeah, what part of Arizona? The whole state yep, okay When did you make your way over to Minnesota? Were you still in high school? What's the last grade you completed through high school? Okay, can you tell the jury the different cities or towns that you've lived in since coming to Minnesota? I guess Minneapolis Minneapolis nice to red wing And St. Paul Do you recognize what's shown in exhibit one? Yeah Do you see the speedway where you used to work? Yeah Yep all right, so Do you have something that looks like this in frame? Both styles. All right What I like you to do is just take this and see if you can draw a circle around the speedway where it used to work You recall something that caught your attention outside of the speedway Yeah You tell the jury what caught your attention The uh police like in that area it's always police When did you first notice or where did you first notice the police and then what did you see happen after mr Floyd was in handcuffs? What's the next thing you do? Are you a little nervous today? Yeah. Well, that's okay. Just take your time And uh, we have plenty of time to get through this. Okay, think to yourself What you next remember Seeing after you noticed he was in handcuffs I think they um ended up putting him in the um the police car thing Did you do anything with your cell phone as you began to notice? This incident involving mr. Floyd and the police Well, as you guys seen that um on the security thing that I had my phone You right you had your phone. Did you do anything with your phone? Like recording it? Yes. Yeah, okay Uh after you've noticed the police officers putting mr. Floyd in the car I don't I don't remember Okay, I can't think right now Well, um


Okay, we'll jump right in. So in the long list of superlatives you can use to describe Kepler and Gaia, stellar rotation will be one that people in this room will appreciate for a generation. Before Kepler, we knew, I would say broadly, the rotation of like a thousand, a few thousand stars. It was hard work, me sign eyes, and for a handful you could do phase curves with Kepler thanks to the incredible precision and years-long photometry that's available. We can take something that looks like this little animated rotating star and we can transform it into a rotation period. So the thesis work by Amy McClendon, working with Susana Grain to be about cool stars 18, showed us more than order of magnitude increase in the number of stars a rotation period is available. So here we have the rotation period on the vertical versus the implied mass from the kick. Kepler and Gaia about 34,000 stars. And the diagram I'm sure we'll see throughout the week is this sort of manifold that we expect at stars. Perfect. That stars, as they age, should slow down. They should lose angular momentum and sort of march up this manifold. We have handfuls of clusters sort of before and during Kepler that we could use to calibrate this surface. And this is the so-called gyrochronology work. We've heard about it and we'll hear about it again with Gaia later today. So that applies that a star should sort of move vertically in the space. That a star here, say an M-worth of stars that people in this room care about, should march upwards in this diagram. And in principle, we could imply ages for tens of thousands of field stars directly. This is the dream, this is the promise of gyrochronology that many people in this room are working hard to fulfill. Now, there's a lot of interesting structure in this diagram. The sun is here near sort of this conspicuous edge in the diagram, which we're going to talk about. There's, let's see a little better here. There's interesting structure as the M-worths seem to tip up here. There does seem to be an upper limit. There's lots of very rapid rotating stars here. And I will draw your attention to further an interesting sort of like bifurcation here split in the distribution. That's what I'll be talking about today. This was discovered in his initial paper in 2013 just for the M-worths. It said, indeed the distribution was possible that there was a slower and more rapidly rotating chunk here, about evenly split. There's also the very rapid rotators, sometimes called bifurcation here, very rapid rotators and hours to a day period. So there's a bifurcation here somewhere around 10 to 20 days, you can see it again. A little split here in the distribution. The mystery deepened a little bit when they extended the sample to all of the Kepler sources. And that this bimodality you can see in these clumps here of range of the white curve versus a period. You can see it for the M-worths, you can sort of see it for the K-worths, but by the time you get to the G-dwarfs in the F-stars, there's no evidence of a bimodality. This is an interesting mystery. Two possible explanations that represent it. The first is that it represents a variation in the star formation history. This is arguably the simpler explanation. Jaroconology must work, and therefore this bimodality represents a bimodal star formation history. The second, and though it seems a little out of hand, it's not totally unreasonable, is that this bimodality represents a new unknown phase of stellar revolution, some sort of spin down phase or rosmy number that is jumped through quickly. This is not completely unreasonable to expect, especially considering the observation that it was only the K- and M-worths. Maybe they are special, and then the G-dwarfs were traced into star formation history. So how do we tell these two scenarios apart? There's a few ways of poking them that I will discuss today. Do we actually see the bimodality in FMG stars? I'm buried in the lead here a little bit. Is the bimodality everywhere? You need something like Gaia. And can we connect this feature to other age indicators? Okay, so with Gaia DR-1, which some of us haven't quite forgotten, we were able to filter out subgiants that were contaminating this sample. So there was a lot of these subgiants in Amy's original sample that the kick mistakenly identified as dwarfs. And so this period histogram here, before the Gaia filtering and after we see nicely these G-dwarfs actually do show a viral rotation period distribution. And this age, this dip here, corresponds to something like 600 million years depending on what flavor of gyrocnology model you adopt. So that's the first test. The FG, maybe the FGK and M-stars all show consistent bimodal gap. The DR-1, of course we are now gloriously in the Gaia DR-2 era. This is the full catalog of Amy's rotation sample, and it has 34,000 stars with good Gaia distances and parallaxes, of course. And you can see lots of interesting structure here. The subgiants, there's binary stars up here, this is an incredible glorious detail in this diagram. I'll just brutally grab a box of only the clearly single main sequence stars. We'll throw out all these interesting subgiants, we'll throw out all these binaries though I invite you to come talk to me about those. And we'll left with about 16,000 stars that are obviously single-looking main sequence stars. This gives us a more complete view, so in the inner regions here within 300 parsecs, we can see this bimodality again clearly, this is my PowerPoint isochrone. Again, this gap is somewhere around 600 million years. Very clearly stands out, maybe even some other structure. Fine, thank you. Some other structure, clumpiness that seems to be visible, but there might be even more age structure in here. But again, you can see this nice track right here, clearly showing up in the nearby stars. Now Gaia doesn't work, DR2 doesn't work just for the inner 300 parsecs, we can explore this as a function of distance. And we can see that as we march out, okay, within 500 parsecs, you can still kind of squint and see this little gap, maybe in the 500 to 600 parsecs, and then it starts to go away when you get out here towards a kilo parsec. This bimodality seems to disappear. In the correct direction, not just distance from the sun, but instead galactic height, which we know to be an age indicator, this bimodal distribution here, I smoothed it and centered it around a 600 million year old gyrochrome, gyrochrome, isochrome. You can see that within about 100 parsecs of the midplane, the bimodality is very strong, and as you march upwards from the disk, the fraction stars, the young stars, defines as you expect. So this looks like it is consistent with a burst of young star formation here near the midplane. Now there's something unexpected that we saw when we were looking at this diagram, and you can squint very hard, maybe, and see there's sort of a yellow band here in the middle. This is not particularly colorblind from there, I apologize, but you have the single main sequence stars here, the equal mass binary track line right up here, and then there's sort of a gradient in color. So let me blow that up and twist the color scheme a little bit. And indeed we see a period gradient that seems to exist across the main sequence here, one of this portion of the main sequence, and throughout the binaries. And we see this goes in this diagram from blue on the left to red, slow rotating on the right. This is pretty cool. Now this here is the evolution you expect from 10 to the 8, 10 to the 9, and 10 to 10 giga-years, 10 to 10 years for a cavework, for example, that would be a star right about here. And you can see that during the first gig year it kind of marches slowly redward, and then between 10 to the 9 and 10 to 10 years it actually makes quite a large jaunt upwards and left. This is somewhat inconsistent with what we're seeing here. We see presumably young rapidly rotating stars lying bluer and maybe fainter and older slow rotating stars existing here on the top right. This also I'll note goes the opposite direction of what you expect from metalicity, that younger stars should be metal rich and therefore redder, and they should draw younger stars to the right in this diagram instead of see them moving to the left, which is sort of interesting. Perfect. So this has all been observations and games with just a couple of field of view, something over here, but of course we have K2 and now, as we all heard, we have TESS. This is going to let us use gyroponology and rotation to study how localize this bimodality, i.e. this star formation history is, and on what spatial scales can we compare? We have lots of massive star formation history nearby galaxies from HST. Could we play this sort of games of understanding how clumping star formation is your pattern is in our own field of stars? Could we see the effects of spiral-armed passages triggering star formation, if that is indeed how star formation happens? I think there's some really interesting quote here. Just to tantalize you a little bit, we are working on this. This is a figure from an undergrad I've been working with Zoe Bell, who's been working to use the Everest Likers, like many other people, to get rotation periods. You can see Recepti here showing up very nicely, but we do see evidence here. You can squint and see the bimodality showing up in the K2 field as well. So it's not just an artifact of only the Kepler field, which is exciting. Okay, and in my very last minute, a total left turn. For the last six years, we've been studying the gender dynamics of questions and answers in astronomy, most of the double-edged spinning, but also the last two cool star spinning. There's lots I can say about this. We can give a long talk about this. For me, Sarah Schmidt, Stephanie Douglas, a ton of people in this room have helped with the study. The takeaway, men ask twice as many questions as women, especially even when you account for the demographics of the audience. And we've learned that the longer a Q&A period goes, the better the resulting gender ratio is. You sort of approach the attendees' gender ratio instead of some skewed ratio. This brings us to two obvious conclusions. Let junior people, let women and minorities and people of color and disabled people, who don't usually feel like they have the opportunity and encouragement to set up the mic, let them speak, and let them speak first. There's a triggering that happens when people speak first. And then I want to applaud the cool stars organizers for making sure we have long Q&A periods, because when you only have one question, the person who's very confident in the proper role raises their hand and gets that one question all the time. So let me encourage you to step back just a centimeter and help encourage the younger people who are here for the first time for one of their very first meetings, help them make cool stars a wonderful event. And with that, I will conclude that Kepler is awesome, Guy is awesome, and you were awesome. Thank you. I must say that the directions to the chairs said exactly that. Perfect. But we're looking for hands, and that's why I'm going to talk for a while so that I can find, yes, there's one at the back, white dress right there. Hey, Jim, thanks for a great talk. I'm familiar with this bi-modality idea in Elizabeth Newton's work, too, and there it seemed to correlate with H-alpha emission. So you mentioned other age indicators, but I wish you had more time to speak on it. Are you looking for other spectroscopic age indicators to sort out whether that kind of signature is also present in the higher mass stars? Yeah, there's an evidence of, it's like von Preston Gamp and other kinds of bi-modalities and structures that were created by activity previously. I think, and Elizabeth can correct me about this if I'm wrong, I think bi-modality represents the chunk of very rapidly rotating stars here that are quite young, and the sort of field age stars here that are older. So really there's sort of three clumps. There's a very rapidly rotating clump here. There is these two branches of field star population, but that's in sort of several hundred million year range. I believe that's where most H-alpha structures you see come from. These are all H-alpha active, thermospherically active, and then most of these, well, I don't know, when you get out here it's an interesting debate. Most of these are thermospherically active. There's one over there. Thank you for this talk. Identify yourself, please. I'm Svi Mazle from Tel Aviv University, and I was very pleased that you used our catalogue of period when you arrived from Kepler, and I just wanted to ask whether your derivation of the period is using the same auto correlation technique so one can compare the results of the... Yeah, well, what I've been showing is exactly as Amy published it, the auto correlation, her best auto correlation function period. In the follow-up work of Endemic K2 we've been doing a lot of comparisons between loam-scargle and auto correlation, and you can chat with me and we'll think it's about what our plans are for doing it right or at least sort of right. Okay, two more, and then we'll talk. Yes. Okay, my name is Sasi Kekar from Bugsbuck Institute of Solar System Research. I have one and a half questions. So here you say that, originally that this dip implies a dip in star formation. My question would be what would cause the dip in star formation and are there other scenarios of the two you listed here that you can think of? What would cause it in star formation? I don't know. I'm not the right person to answer that. You can imagine exotic things like, again, passive spiral arms that trigger star formation, which would suggest that if you look this way versus that way along the galaxy, the stars here should be slightly younger and these should be slightly older. That's not really supported and what we see in other galaxies that by 600 million years, by a giga-year, things are all mixed up. So I don't know the answer. And I think there are other explanations for this. I think we could talk about variations in winds and breaking. We haven't been able to account for abundances here that can play into this. I don't know. I think I have presented it in a very simple way. One last question. Thanks for the thought. Where are you from? I'm from MPS, in Germany. My question is about what, if we are looking at systematic detecting rotation periods because of the G-key, K and M-thorff, have these problematic with the amplitude of the variability and the irregular modulation of the patterns in the light curves at the sun? Are you aware of that some of the methods are not able to retrieve these rotation periods? We definitely do see some interesting aliases. The autocorrelation function tends to do a little better in terms of not picking out the half period or the double period, but it's not perfect. Thankfully, the bilatality doesn't show up at a horizontal lump, which would be suggested of a Kepler systematic, but K2 has a lot of systematic structure we have to worry about. So I'm much more concerned there about seeing clumps at all masses at fixed periods. And then, as I think you suggested, there's a bias that you're always going to see, you see almost all of these rapid rotators because they're easy to see the spots are large. By the time you get down to solar age, the spots are very small. So we don't necessarily have a complete picture. It's very hard to invert this into a true star formation's history because you're missing 60% of the stars out here or something, right? As a function of age, certainly because they get fainter and harder to measure. Okay, let's say...


All right, thanks for making it back after lunch Right now right floater will talk to us about modernizing relay D and the there is a rocky road to HTTP 2 Please welcome right Thank you. Yeah, okay. It's It's great to see you here, especially after lunch. I think I had a plate with that much meat and fish and all that and No, I have to talk so We see I hope it's worth it so Let me talk about modernizing relay D and a little bit about HTTP 2 and why The focus of this talk shifted a little bit to to other things We did in relay D over the last few months By the way this People rarely know that relay D has its own Puffy mascot version that I once Did myself. It's a plumber the pluffer plumber puffy That you see in the corner so a little history lesson about relay D and In the in the latest context Then I will talk about some new features and enhancements or things that have been requested many times over the years and I finally unslacked and fixed or finished them About HTTP 2 and what is being worked on there and some some other future Stuff and then I hope you You have some questions Afterwards, so we lady got imported in 2006 So it's almost ancient now And it was in interesting piece of code because I Wrote a little demon that was able to do health checks on on different web servers like using ICMP checks or TCP checks or a little HTTP check and then based on the health check results Configure a pf table to so in the back in the days We had these rdrs in pf and the rdrs code only Forward to a round robin table so pf got the request and uses the round robin table To send this to a backend host and this was static So if one of the hosts in this table was down pf did it anyway So the little idea was just to health checks and based on that populate the table and Then about the same time PR Eve showed up and He has Had implemented something very similar this SLBD simple load balancing demon that he did And it was the same idea and it had a few few more things That I didn't have in my implementation and and he liked others. So it's like oh great. Let's import This this load balancing demon I convinced you and We can also get a new developer because we also imported PR Eve in the project basically so Based on that we we we worked on the code made it ready for the tree and this Because I didn't like the name SLBD We named we renamed it to even worse thing to oh My gosh, but what oh it was host state D in the beginning and And this yeah, that was the famous first comment with many to do's so It got in in 2006 in 2007 So in the beginning it was basically just running health checks and then configuring pf4 this as I say redirects or the layer three forward But then I got asked Myself by somebody who who wanted to to run an a load balancer in a big web application and was a platform where you could buy a screams from German soccer games and all that and This magenta company was involved there and they asked oh, can you do load balancing and do your support and SSL termination I said sure and then I looked it up and Implemented it actually and this is how relay D got support for the the the relays a layer seven Part so the difference is one thing I can terminate here TLS connection on the front and talk HTTP to the back end so this requires me to to go on all on the Application layer now right so this is basically SSL team termination in this Particular our product as well as good because it was it was a free low test basically because thousands of people at the same time wanted to Order their soccer game that they could watch online and all that and I Was sitting in the data center with with some people watching the game and at the same time see oh really D crashed Okay, let's fix it run it again and so it was kind of live Testing the good thing is we had relay D redundant. So if one crashed the other one took over and nobody noticed any problem So this is how relay D got the relay part if you're now the syntax it has redirects and relays So the relay parts came a few months later You can do a little bit more so for example instead of just doing the load balancing based off on source IP What pf basically does you can look at at the cookie value? so that you Maintain some persistence. So if a client provides a specific cookie you send it to Does the same back-end or you hash in some other values to make it less obvious for for some attackers basically so This is all part of the layer seven Over the time it got renamed multiple times SLBD in the beginning as I said host state D, which I never really liked it was a compromise Host state D. We thought it is clever until some Americans told me that the first part is not Really a good idea and the last thing I came up with a name Relay D which because it is more than just something that monitors the state of servers it is relaying from a to b and I even like a relay the name More than like proxy because a proxy is usually very application specific and a relay is very generic Relay D can do generic TCP, but it can also do TLS and HTTP so Getting stuff from a to b in the middle. Now, that's what relay D does and a few years later Relay D got an offspring basically when we removed engine X from open BSD and We we needed and Another web server. I said, okay, I can we turn relay D into a web server because it already has Part of the implementation, right? So HTTP is its own piece of software, but it's based on relay D still and it tried to sync code back and forth Okay, long history but meanwhile in 2019 The web has changed a bit, right? So everything is distributed and everything is like Cloud and apps and whatever and people don't even use real web servers anymore You have some Node.js and whatever is fancy right now So We we see okay, we need to clean it a little bit up Technically like like cloud just changed to use lip TLS instead of open SSL That's just the detail, but this enables us to use modern TLS versions basically Or SNI later and But still what relay D Lacks or lacked partially is do some Filtering for the requests blocking based on source IP things like that or Assume forward to a back-end where rather nodes have different ports This is one thing that in the in the in the past you had like a pool of web servers and all of them had their own IP and And but all of them were running on port 80 and now this is not the case anymore If you have some whatever Docker farm and and things like that. Yeah, so this already existed in a way in 2007 that there were some I think ruby Frameworks where you run the services on on some weird ports and some front and forwards it But I always said that's stupid who does it right? So we never supported it Yeah, so it's about small features that we added and then there are available now and then The biggest thing to add is just we're running on an old version of of the HTTP protocol right relay D is still HTTP 1 1 only It works in most of the cases. I mean there's no regression yet, but It can be better. It can be maybe faster and all that with HTTP 2 so In May I decided to to have a hackathon with myself So I went to the craft work Which is kind of a cafe co-working kind of thing and sit on my chair and was hacking there and people were looking at me What is he doing there every day as I had my one big hackathon where I implemented for some of the obvious things and Yeah, some some others So one thing is that was actually a bug and you about this bug It was a little bit hard to fix, but I finally did it. So relay D has So you have the relay statement Usually a listen on and a forward to so one side the other side and then you can attach a protocol to it for well for the in this case the HTTP filters So in the past The problem was So I added this possibility based on a back end or on a URL path to switch the back end Which is very common, right? You have a website and it has static content like slash images or slash static and you want to serve this from the Whatever small static web server and then you have all the dynamic stuff you want to serve from your Node.js go rust whatever wrap application so in order to do this you have to write a Such a filter and then you can switch the the back end It only contains one IP here right now, but it can be a pool of multiple hosts as well so the problem is This didn't really work Because relay D took in a keep alive session like HTTP Keep alive a persistent connection it it selected the back end for the first request and Then This was sticky right then it kept on using the selected back end for all connections of the same of all requests of the same connections now I have it So the work around was really terrible basically enforce the connection to close For each request So like in the old time like one request per TCP connection This work, but it was not a nice way So I fixed this finally to be able to to remove this and now you can Really use use this feature to have multiple server back ends Um in order to use it it only works with tables right now, but as I said you can put one IP address in there and There's there's some special implementation details. So you have one listen on and one or more forward two rules So the first one is always the default That gets selected The other ones are kind of ignored unless you you're selected with a rule But you have to define them here as well You can also add a Health check statement there if not, then their hosts are always online So this is working now also like since May and The from two filter rules and something that I added Actually, the code was all partially there and we lady and I never finished it somehow and I forget about it but now you can you can write these rules and specify source destination matching syntax so that you you give a specific URL only to to Internal hosts and think like that. Yeah, so you can so it's Yeah, I know it looks a little bit like pf, right? It is not pf it People said is it really wise to use the same syntax and I said well it is an application layer Firewall In a way. Yeah, so this is possible now. You can also block Based on IPs or network rangers and things like that and it's evaluated and last match in order of course SNI so If you ever looked at open SSL itself and how SNI is a server name indication is Implemented if you if you if you use the library it is kind of a mess, right? You you have multiple SSL objects and you somehow need to to deal with this with some callbacks and all that and so In Libre SSL we have lip TLS, which is a really nice TLS library That is sits on top of the lip SSL library But it's really easy to use it is it's safe. It's Nicely designed. I wonder myself. Why is it? Such a rare thing not so many people are using lip TLS. Yeah, I think lip TLS is one of the main Advantages of Libre SSL. It's a really nice library. Just just try to use it You can make a TLS server with just a few lines. It's very easy to use and there are some We used it with rust as well and all that because also very easy to integrate lip TLS in other languages And you don't have to deal with all this op-nesses elements So lip TLS made it easier to add SNI support and SNI basically means instead of having one server certificate So server has a default certificate once again, but it can select a different Certificate based on the server name that is sent as part of the TLS header or it's an extension So now you you can By default relay do you don't have to configure your TLS it looks up there the when you do SSL it looks up the TLS in a magic pass yeah, etc SSL then The relay dot key and and things like that. It's documented in the man page But now you can configure multiple months and it was it still uses this pass etc SSL and so on as documented But then with the specified Basename file name so you add multiple ones and then if they match it uses a different key Very easy to configure actually just drop in the new new keys and And add a line for each and then your SNI works And then in the back end your web servers can actually deal with the different host and and all that I wonder who has looked up this IP address yet, but anyway OCSP another thing HTTP got OCSP support like some time ago. It's even mentioned in the Httpd and relay D mastery book Shameless plug from Michael W. Lucas So there he Mentioned and describes how to use OCSP with Httpd. This is called that what back at it And now I finally got this over to relay D with a little different syntax because relay D has this you see the magic pass again So when there's a file dot OCSP Either for name and port or just name then it will use it. So the OCSP is basically I'm not a TLS person per se But it basically means you you ask the West web server and the web server can give you a signed OCSP response from from the CA that says Tells you about the validity of the certificate OCSP stapling and So when you do that it it really you don't see a functional difference But it makes it easier for your clients to look up the validity of the certificate and then fasten things up because you don't have to do an external request to a OCSP server or use a Revocation list or something like this So the route to HTTP to it started really nicely Took this picture from a motorcycle trip this year So yeah, you have some Stones and it's rocky and all that but on the other side you have the beautiful view on this is fear much that are they right? so it looked like that and Okay, it's like okay I did these features on my list and then I started well We do need HTTP to support after sometime like open VST very often is and We get asked about in a new thing. We say no, that's terrible. We don't do this Like a CPI move terrible thing and APM works great, right? So we didn't get a CPI until somebody implemented a CPI and With a lot of work. We have a really nicely working a CPI stack the good thing was a CPI boss We didn't drink the Kool-Aid and we didn't get the reference code. We have like a really nice and Working implementation now the same Herpenbeb with VMD, right? We hyper-visors and VMs are bad but now we have a really nice implementation that Contains VMs much better than QM or does and and so on and with HTTP to we said this is HTTP HTTP to and these What was the previous name excuse? I just lost it skippy. Yeah skip Speedy speedy. Yeah, why do I think skippy as speedy speedy? So speedy was yeah was basically this in the beginning speedy had one major difference how it compressed the headers But you say oh, okay all these additional states and multiplexing that looks very dangerous, right? So I think it it's very easy to make it wrong So we were skeptical about this But I said, okay now it's time actually 2019 as the time since HTTP 3 is is it already standardized or they plan to do it this year I'm not sure if the RC is out yet But so since we get HTTP 3 now HTTP 2 is vintage computing. So let's implement it, right? So I started working on it But but because it needs so many changes in in relay D I started something very nasty instead of sending millions of patches around on an open BST list I started just moving ahead doing stuff and then later when I feel confident enough I I will share it and incrementally not in one block But instead of doing this just silently on my own I said, okay, I shared this github you can follow it But this is like work in progress code, right? So HTTP 2 is multiplex multi-stream protocol. That's a major difference It is using header compression and then it's a small binary protocol So instead of like opening a telnet and typing get HTTP 2 whatever or NC You know you you basically have the same you have the headers But they're all encoded. So it's binary protocol for the headers and for the communication itself One thing that they did all headers are lowercase now And They define some standard headers that the most common ones with these colon syntax So method is now a header not something of the String anymore and authority used to be the host header But you can still add arbitrary headers below that like user agent or your own x organization and all that So I'm not going too much into the HTTP to Detail but one thing for examples relay D was written in a way that it expects request response Or actually it does support pipelining in a way that the client sends multiple requests and then you answer these reclass crests in the same order So it's Possibility and HTTP one It's broken in many cases, but it's still They try to solve some issues with it and that never really work But usually it's still the same request response scheme even if you just make it in chunks But HTTP to like one client connection like a single TCP TLS connection Can I can open multiple streams? And this totally asynchronously so you open a connection for your web server and the web server can push Responses to you basically because there's some picture like the logo of the company you're accessing website that you might need So it's just sending you the logo without you ever asking, right? That's one feature or you you can have this one connection and then start multiple streams Connection Requesting different things like the CSS and the JavaScript and the page all at the same time over one connection And then the server sends the responses in in frames and in packets With their stream ID so you know where they belong what they arrive totally out of order It doesn't matter anymore So you just need one Connection and you can also send your DNS requests over it, right? And so it's all going there So really he was not designed for this so one word This is actually a branch in my my code because it's I Need to merge it in at some point, but it's it's a lot of Mechanical work in the tree. I had to split like one main structure that was used for the relaying request response I had to split it into like a peer like one side and Then multiple streams to it so that like one peer can have multiple streams and So the way so you can dial it directly access some fields like the peer address anymore And then you you look look it up and see so this is a large diff actually And I'm really scared about getting this in but I get an okay banner and then it's it's all fine Okay, I still have time H pack so I talked about header compression. So this whole thing these HTTP header is doing header compression actually in in speedy day, I think they use deflate or something and that was vulnerable to what was a crime attack so and so an HTTP to they use this HPEC algorithm and actually I After looking at it. It's really nice idea Let's just think about it. You have an persistent connection where you send multiple requests so some headers like past slash is Something that repeats all the time. So HPEC allows to reduce it to one byte Just in a static table and an index and you have other common these Colon headers as very common combinations where they just index and then you encode it like this and Then other things that might be your your string like user agent the user agent is usually a very large thing But it never changes So when you have one connection where you do 100 requests, you only need to send us user agent once then HPEC adds it to a dynamic table and Then the next request only sends you an index in this table So instead of the huge string for the second subsequent request in any of the streams in this You only send like the idea 127 or something like this. Yeah, that's a Very simplified Description of how HPEC works. Yeah, so your index well-known headers and you'll learn Other headers dynamically during a connection. So the request size of the headers can be stripped down to just a few bytes So I implemented HPEC It's not an open missy yet because there's no use case for it But the plan is to to use this in relay D and later HTTP D for the HTTP to code So it's it's secret. I looked at existing implementations But usually it's like, yeah, it doesn't really fit and actually it was a nice experience implementing this stuff and I looked at the I mean, it's always with algorithms, right? People learn algorithms are saying you have to do very smart and you have to invent your own algorithms to make them even better But all these tricks are really Scary once you do multiple loops and pointer Mathematics and all that so I said, okay my implementation. I think I want it to be fast, but The goal is not to optimize it as much as possible. The goal is to make the code somehow Readable and obvious So even if I allocate more data or my my whatever my space complexity is a little bit more than it could be The idea is to to have this a really solid piece of software so smart tricks are all always dangerous and I also want to utilize some like like Otto's Maloch tricks for example Like free zero and things like that so So This is the year also online the thing is I spent some days running AFL housing on my code And I have like hundreds of test cases to verify it actually AFL after running for like 10 hours found one really hard to find back and I was able to to fix it so I'm thankful for for that Yeah, so it evolved into a little API it's part of the man page So you see I even wrote a man page and all that so it sits there. Nobody uses it, but It intends to be clean so that we can integrate it later Yeah, but this is basically it we have h-pack we have the streams something that works with it and I can parse maybe the first HTTP to request but then After I submitted this talk actually the last month I Yeah, had some Changes so I couldn't really much focus on on continuing this work So is it way poor there now it is not because there's already some code and we have time I mean really the existence 2006 so we have time until 2026 or whatever to add HTTP to support, but I'm still working on this. Yeah By the way, this is a tunnel in Glarros in in Switzerland as well So I have some initial code. You see these magic HTTP to preface That's what they send over often you negotiated HTTP to with this TLS LPN you can switch to it So it is compatible that you support both protocols and then the first packet is this Weird preface. I don't really know why they're doing this because it's static But it's part of the protocol and then you have you have the basic header some kind of TLV, right? the stream IDs and then each packet so they're Different ones like data, but headers as well the the h-pack compressed headers And then some some other Packets that that you don't have and then HTTP one well world. I like go away. I think that's still from the speedy Google days But also use the scary ones like window update and all that so that reminds me a bit of TCP, right? And that's a whole goal of like Removing control from the OS and moving it to the browser by by doing things like that And ping is a heartbeat for example I'm not sure How dangerous this is but then in just a month ago There was this this list of CVEs, right? You've probably saw on all web servers ten minutes, okay, yeah and Because all HTTP to implementations out there are vulnerable to to denial of service attacks and all that because they didn't get the Multiplexing right to have resource exhaustion attacks and all that all the things we were concerned about back in the days, right? so we were right and they were wrong and So we we can try to make it right now learn from these mistakes and Implemented properly the thing is some of these issues are really related to the protocol itself So you really need to Work around this, but it's possible. So I'm really glad that this happened before we finished the implementation so quick now just quickly is HTTP to over TLS over UDP or DTLS and this is HTTP 3. Yeah TCP is that we need this it is a UDP base protocol that you cannot really firewall anymore But the standard now, so Of course you can still block it Yeah, you have UDP port Is it 443 I think it's something you so you can block the quick port and I'm doing this on my firewall So so they they go to to the HTTP ports But this is basically the same protocol with the main difference that The HPEC algorithm got a little bit changed And this makes sense because HPEC has some state and It expects the individual request to arrive in order What happens on a TCP connection? but now with was quick they can arrive out of order and then it The HPEC might cause head of line blocking where you wait for a specific state to to advance But otherwise, I have to admit once we have HTTP to it might not be too difficult to add quick support But no, we're not doing this Okay, that's about almost the last slide Real AD so let's rewrite it and rust because that's what everyone doing is doing now Actually, I do started to like rust, but yeah, I'm kidding really seriously I did implement MG in rust RMG after they announced Emacs and rust are Emacs a serious project. I made my not so serious project RMG But this is Basically on my roadmap. There's not so much. I know right now. I think HTTP 2 is the most important thing We did get a few other features like new binary health checks I know and some web sockets compatibility So we got some user contributed patches into real AD as well. Yeah, just keep on sending them So for the questions if we still have a minute. Yeah, we do I Would also like to ask you right now. What do you think is needed these days?  That you cannot do with real AD I Will utter the evil words like full rooting domain support So you can run multiple real ADs in multiple different rooting domains. It's still not there Yes, I Have a patch I Have a patch now for about half a year or maybe yeah like this That allows one real AD to to to run in multiple our domains and relay between them and all that And I think I have to upstream this patch Somehow It's not too long Yeah, I have to really this this is a good thing I should remember just Digging out this the patch and to do this which I really wanted as well I actually used it, but then I didn't have time to polish it up and release it I had I was using VMD were multiple VMs were running in different routing domains And then I had a real AD in the front end to to do some like TLS termination per routing domain for that I had exactly that's problem. Yeah, I Didn't really understand how the lady works with HTTP to on the back end So I get it correctly that the age really the support is supposed to support the HTTP to in the front end as in terminating the HTTP to Yeah, but can I also use it to like establish new HTTP connection to downstream? To the upstream that's that's a good question. So he asked what I do with HTTP to support in the back end and Talking to the server Not in the in the front end part, right? Well, the basic idea of HTTP to is more or less if you look at it it it's bi-directional so you open a connection and Of course, you want data from from the server but every side can make push or Request so it's it's not such a big difference Talking from a client to the server then from a server to the client because the handshake is almost the same and then all that so my plan is first on working on the Reladies server side to the outside and Then yeah in the beginning it would have to Most probably that's in my poc that does one request. I'm talking HTTP one to the inside But in the end it doesn't really matter. I mean relay D for example long before we had a of two in pf Supports not six four it can do six IPv6 on the outside before on the inside So just generally it doesn't really matter you can use different protocols and then the relay can translate it But the plan is first accepting HTTP to and doing HTTP one and the inside I wanted to ask about the tooling for debugging HTTP to connections on open BSD What would you use? What would you probably need to implement for instance that you I mean nowadays? You just netcat to a server and you know writes in it in the text But it that's probably isn't really possible anymore So are you thinking about writing some tool that you could like plug between netcat and you and that would like Create this binary Stuff well, maybe even the streams and stuff so you can actually test it with something that is yeah scriptable Well one thing When Google did speedy that did one thing right because they define speedy to be TLS only So there is no plain text version of speedy And then the committees came in and they said all we need HTTP to over a TCP as well And I'm not sure if I think the browser's didn't want to implement it I'm not sure if they do it now or not, but in theory at least that was the design You cannot netcat it because it's TLS always and I think it's a good idea I don't plan to support HTTP to over TCP at this point because well it could but That's not the idea. The idea was that it's always TLS. So then you cannot netcat it So how do I debug it? Well, I don't know printfs and all that you could probably I Mean in I D we had something like this or I think can be The demon could put it into a PCARP format that you can use TCP dump on it, but Yeah Sorry, just a plug if you if you extended netcat with lip TLS because it's so nice and easy and What not then? You could basically establish a TLS connection using netcat and then Then you would have a basis for debugging Maybe or you something you can try to write a pet to modularize. It's better Yeah, yeah, well, maybe it would be interesting thing to do this SSL men in the middle in netcat Okay Other questions. Do we still have a minute one more question or something like this? Yeah one last question Okay, otherwise. Thank you very much panel Yeah But but the the God talked trade the train tunnel is is much longer Yeah, but I live in Switzerland, so I like the tunnel Okay, thank you


Hello everyone, and welcome to this talk about Phenops and observability. As last time I was first talking about this in November here in Brno, and it was also unpopular by the title Phenops, so this time the same, which is cool. I'm first going to reflect a bit about the talk in November that I had about the Phenops, and then we will move on into some more technical details about the talk, about the presentation actually. So this part was already on November, so I will go through it a little bit faster. So the whole engagement started when my friend Andy Thompson and I were engaged in a project with a large company that needed us to sort out their Phenops engagement. Back in the time, I was a DevOps engineer, I didn't have any actual Phenops experience, so Andy and I started exploring what Phenops is, what it means for a company, how we can do it properly, and all that stuff, you know. At the same time, they had the first task for us, and this is the biggest cost they experienced was related to EBS volumes, and everything is happening on AWS, and the overall money the company spent on a monthly level was anywhere between five and seven million US dollars monthly, and big pile of that money went to EBS volumes. So we started exploring what is going on with EBS volumes as engineers, you know. We looked into the patterns of use cases, how teams are using EBS volumes, what they do with them, and all that stuff. But at the same time, as I said, we were exploring the Phenops. What Phenops is, how it is managed, how it is defined, and what should we do as now part of the Phenops team, DevOps slash Phenops team. While working with them, that was really a great experience because we were able to see firsthand how engineers in this large company were struggling with the idea of being into the cloud and not being able to use the cloud in the way they wanted. The promise of the cloud is that you get your resources when you need them, how you need them, you set them the way you want them, and all that stuff. That wasn't happening for this enterprise because the enterprise had strict rules about how to use cloud resources, how to manage them, when to dispose them, and all that stuff. So that was frustrating for engineers. At the same time, we were, as a team, engaged in a lot of talks with the management. So we were able to tell that the management was struggling with similar issues. They were struggling with the ways to control engineers from spending money on the cloud. So it is like a clash. On one side, you've got engineers willing to do stuff in the cloud, experiment, and try out new things. And on the other side, you've got management, which is keen to control the spend, which is natural. So both approaches make sense. While preparing for the whole stuff and exploring EBS and exploring Phenops, obviously Phenops.org is the place to go if you want to learn about Phenops. And this is the beautiful definition standing on the Phenops.org site, which is almost completely not accurate. This is really nice when you are in the management, and you want to explain how Phenops is the next big thing. So a lot of parts of the company working together towards the common goal of making the world a better place, using less money, and everyone is happy, and everyone is collaborating, and all that stuff. But in reality, what we have found that Phenops is after some exploration is this. So Phenops is purely about saving money in the cloud. The whole talk about changing the culture in an enterprise where every engineer would think about the cost did not happen. I at least didn't see it. I was on that project for three years. And the reason why I didn't see it was the miscommunication between the management and the engineering team. And that actually guided my colleague Andy and me into building a system that we hoped that would help facilitate this communication.  On both sides, you've got people who are willing to do something good for the company, but it's just that they're talking different languages. They have different goals, and that was always the problem in this. So going back to engineering, and I will be during this talk, jump a little bit on the engineering side of the things, and then on the management side of the things. So what you see here is a flat, simple model that we used to analyze what is going on with EBS volumes. So we said, okay, we want to rely on AWS giving us the data, so we need pricing from AWS. Then we would like some metrics about EBS, about how EBS is used. If you want to see if EBS volume is unused, or if you're not pushing any data towards EBS, you are not using it maybe, and you want to support these claims with actual facts. So if you can pull metrics, and if you can prove to your engineering teams that there is no byte coming to or from the EBS volume for some amount of time, that should be good enough for them to comply that this EBS volume is actually unused. I mean, physically and actually unused. Also there were all kinds of specific conditions in this enterprise, how you declare that something is used or not, or how something can be used or not. For instance, they had a lot of accounts in AWS which were not claimed as production account. They are not labeled as production account, but they were used as production account. How this happens is another story. The good story about that is that they would push a new service. They would find themselves a client. They wouldn't get the approval from the company to create a production account to support this client, but they had a client. The client wanted to pay. They asked the management. Management said yes, onboard the client. They onboard the client on the development account, and they just continue rolling in new accounts, new clients, and new clients on this dev account. And at the end, you've got a dev account supporting production. And there are all kinds of things you would find when you dig deep into these accounts that were used. We had communication with around 400 different teams running all kinds of projects in this enterprise. So we had to come up with some kind of custom rules that would help us control the whole thing. On the other side, we would collect the data, apply some pricing, AWS pricing on them. We would apply some rules, and we would do some logic around that, and we were able to generate two types of results. And this was very important for us. The one type of result would go into engineering teams, which means that we would meet with the team, and we would present the data. And we would say, this is what we have found about your EBS volume usage. This is the list of EBS volumes. These are the ones that are not used. These are the ones that are not used properly. And these are the ones that are optimized. Them you don't need to touch. And we would support all this with actual data. The teams would get back to us. We would get into all kinds of heated discussions around what is wrong with EBS volumes. You know, these stories like, we can't touch this EBS volume. Why? A colleague of ours created that volume and left the company. And we don't know why this volume exists in the first place. And you would say, yes, there is no communication with that volume. There is no bytes going to or from that volume for three months. So maybe it's safe to delete it, or at least to snapshot it and delete it. But they would be reluctant, because they would say, well, we've got contracts, and we've got SLAs with customers. And we would rather continue paying for that volume, even though we don't use it, than risking some outage of service or something like that. So again, these are real life stories. Not something that we read in books, how we do stuff. One fun fact, the management didn't like this approach. They preferred this. This is the same stuff, but a little bit with colors and different shapes, it looks a little bit more modern. And this resonated well with the management. When we had to present to teams what we do, we would use this model. And engineering teams responded well. This is simple, this is very simple design. But when we went to talk with management, this was the one. Because then we could say, yes, this is something magically that happens, and then we process some data, and then we generate some results and all that stuff. Apart from the first report, that was a recommendation for engineers. We generated a second report that went to the management. And what we did is we defended, well, we presented some recommendations on the company level. These recommendations would be like this. There is an opportunity to save X amount of dollars per month if you would be willing to perform these and these activities. We already conducted talks with the teams, and we already got their estimations on how much time would they need to implement the changes. So that usually would be like open the Terraform file, change something, apply changes, and that's it. But we went through the talk and we collected the data about how much would it take to perform these changes. So now we would prepare a report for the management and we would say, if you want to save this amount of money, it is possible. You need to come up with the resources for engineering teams to spend this amount of time to execute these changes. And now management actually had something to make a decision, some enough data to make a decision. It was palpable, it was not tacit anymore. You could actually anticipate how much time you would need, how much effort you would need, and what kind of money you would save. So again, this was, and this is one of the, I love on these talks to present these guys. This is Niklaus Wirt. Niklaus Wirt is most famous by being the father of the Pascal programming language, but some of his work is much more than that. I personally like this plea for a lean software paper that he wrote in 1995. It's a five pages paper that explains benefits of a simple design. You can find more modern books about this, like David Furley has this book, Modern Software Engineering, which is great talk about how you simplified the design, how you start from a simple thing, how you add stuff into this simple thing and how it becomes more and more useful. The whole talk, the whole approach is don't try to be modern, don't try to be tech for tech, but try to build tech that would be actually useful, that would sort out or deal with some problem. So again, this is the influence of the guy. This is how we build the models. After this initial success of the approach, the company said this is all one of my friends like to say nice and dandy, but can you do it for some other resources? Like we've got DynamoDBs, we've got RDS instances running around, we've got EC2 instances running around, and we have no idea how to save some money on these. And again, my apologies to all the people that are dealing with the actual software that was built to generate metrics or to generate the observability in different definition scopes. This is a different kind of observability that we are doing. We are not getting so deep into how many processes are being run on certain instances and how much memory they use or how much processing power they use. This is a little bit different approach. We wanted to be in the middle between engineering and what can be good for the company. So we said, okay, if we are going to do this for another EC2 or DynamoDB, let's say, we would like to build a framework. We would like to build something that can be reusable, that we can extend, that we can just add new stuff into the play. So we would have an engine and then we would just, well, you can say, add plugins to this engine. So the engine would become aware of how to analyze DynamoDBs and all that stuff. So we started simple. One of the colleagues on the previous talks talked about ECS. And it is the simplest form of running containers on AWS that you can actually orchestrate a little bit. What we wanted to achieve now into technical stuff, what we wanted to achieve here is try to minimize the need for orchestration and increase the choreography. So we wanted a lot of independent services running around, not needed to be orchestrated because we wanted to avoid this. If it comes to orchestration, then we will deal with that. But until we have to, we would like to avoid it. So we said, okay, let's treat this as a microservices architecture. Let's put the analyzing engine for EBS in one container. Then we put another one into a second container, like that one would analyze DynamoDBs. The third one would analyze, I don't know, ADS instances and stuff like that. So we started adding more and more services following the same pattern. Every service would have a different model, but the model would contain same stuff, like get me AWS pricing, get me some metrics that I need to assess whether this information is useful for me or not. And some costume parameters, like the stuff that I talked about, whether some account is production account. So I cannot really apply some of the rules on the production account. Again, bear in mind that metrics and observability in terms of, let's say, how Mitsitip C is explaining observability is different than this. This is like a high level of observability. Let's call it like that. We would rely on AWS to give us data. So what we did is we put these containers in place. Then we needed to somehow start them. We had an idea of running them every day, collecting the data. And once a month we would generate these reports that we would push to teams that are targeted for actual optimization and to the management so they could figure out whether they want to invest resources into optimizing the infrastructure and saving money, eventually. So we said, since we don't have a lot of orchestration, let's do it simple. We created events that are timed. So we said, in the morning we will schedule these containers. They will spin up, do some stuff. It was serverless, it was Fargate enabled, so all that is fine. After that, we figured out that a lot of these containers would actually go and fetch prices from AWS. We would need some secrets to store them somewhere in the AWS. And then somewhere around this point, we started thinking like, okay, how about we make everything idempotent and serverless? That means that if we are going to build it, I can run it many times a day and generate the same results. So I wouldn't mess up with my previous data. That was idempotency. And then serverless, well, events, ECS, Fargate containers and all that stuff still is serverless. So that was cool. Then we proceeded and we said, okay, we want to go through accounts. And we want to analyze what is in these accounts. We need some permissions. We need some account roaming stuff that will go around. And around this point, we started thinking like maybe this orchestration, microservices organization is not the most optimal one. Can we put aside the tool, the piece of code that would go through all these accounts and do this for us and just hand us the token and we continue doing it like that? And that is doable as well. In our case, when once we put it in motion, it wasn't really good use of our time. We figured out that it was, even though it's elegant in the code way of things, in the architectural way of things, it wasn't really elegant in the ways of us having to maintain that. And we always had in mind that at some point we will hand over this work to the team that should contain to maintain this. So we wanted to keep it simple and follow this approach. So we said, no, every container will have its own engine that will roam through accounts so we could separate them even more. So we could avoid the orchestration. That was another reason why we avoided this because this would introduce some piece of orchestration and we couldn't scale them in the same way. So we kept the isolation thing. And then we said, okay, since we wanna maintain this serverless approach, let's write some of the data to the Aurora serverless. Now, still idempotent, the data written in the database is not going to be there longer than a day, because we don't need it longer than a day. So if I delete, if I rerun the task, it would clean the data for the task. It would regenerate the data and the data would be fresh for that run. So this is why we could keep the serverless approach. We needn't keep this data for, I don't know how long. And that was fine. Also, obviously something on AWS, you need to log stuff, you need to, yeah, you need to send, send information. And this is the final stuff. This is where we, at the end of the day, the only piece of orchestration that we had to introduce is once all the tasks finish, we would push the data into the S3 bucket through some reporting mechanism. And from that point on, the data would be picked up by some of the Power BI or Tableau tools. It would be invoked into the Snowflake. It would be reorganized or done something different with it. That point company said, okay, can this be used for something else? And this something else we didn't do because the contract was expiring and they didn't want to go through with the whole idea. But the question was, can we use this approach to make company more advanced on the market? Like how do you do this? Well, you make your company serverless or use more serverless stuff. This company used a lot of EC2 instances, a lot of monoliths running around. So if you wanna be more modern company or more competitive company on the marketplace, you should follow from the tech side of things how we see it. You should approach the way you build software a little bit different. So this is the slide that we prepared for the management actually, when we went on a meeting. And we said, okay, if the management would like to be innovative and competitive, and if the tech has a strategy how to support this. So that means, I as a company wanna be competitive, cool. What the engineering stuff can do? Well, we can build serverless, cool. So we've got two ifs. Then, score teams by their current level of serverless adoption. And then finally, management needs to approve resources to re-engineer and re-architect and rewrite these applications. This is how it goes. And the final stuff is this. When we went on a meeting and we discussed with them and all that stuff, and you know what it turned out? The company was more keen saving money than becoming competitive. Because the company was already dominant on the market. And their idea of being competitive on the market was based on, okay, if we can do it with not a lot of effort and create a bigger gap between us and the competitors, that was cool. But when you introduce them to the idea about how this works in engineering mind, like you tell us what you want. We propose how we can assist you in the journey. We give you the estimate how much time, resources, money, people we need to make this happen. And then you say, yes, we want to make it happen. And then you follow it through. And then you can support this approach using this tool to provide you with a proper analysis on how much teams are serverless, how much teams would be capable of follow through the whole stuff. And this is, for us, for all of you who are my age appropriately, let's say, you remember Hitchhiker's Guide to the Galaxy and the Bubblefish, and it was all that in our heads. Like, you create a tool that is a good tool to communicate between engineers and management. They usually don't know how to communicate. I'm sure all of you have been in situations where you wanted to talk with the management, but it doesn't really go both ways. So us providing this tool felt like providing the Bubblefish. On one side, we were able to explain to the management what can be achieved with certain amount of resources. And on the other side, we were able to provide the engineering teams with what needs to be done, what should be done, and why we need to do this. Unfortunately for us, they didn't go through with the second part of the idea of modernizing the approach and making something better with the whole stuff. But the first part was good enough. So the Bubblefish, in our case, really worked for the first piece. And it was very much thanks to Nicholas for the whole idea about how to keep it simple and how to be. There is one thing I want to mention at the end. It's really tough to build a software that you believe will create the benefit for the company and not introduce some nice and flirty technical things you can brag about because we as engineers like to be proud of what we do. And if you're not using the latest grade, you didn't see Kubernetes in this presentation. You didn't see any, well, not any actually fancy stuff that we did. It was simple, plain engineering that we did that did some good for the company. But I need to say it's not really easy to defend this approach when you go into the purely engineering audience. You would get all kinds of questions like, why didn't you reorganize your microservices in a different way? Why didn't you do this AWS service or that AWS service? And it's really hard to defend when you have to offer an explanation like this. Okay, we knew which team is going to continue doing this after us. We were there for a limited amount of time. We wanted to build an application that will have a longevity, that will be able to continue operating after we are done, and that will be able to be successfully managed by the team that we give it to. So that influenced a lot of decisions that we made during the architectural phase of the stuff and all that. If you're building a product, if you're building a service that you want to commercially offer to someone else, that's a different thing. You're building it for yourself with a different goal in mind. This was made especially for this company with the idea of then being able to take over the product, well, the service from us and continue using it and it saved really a lot of money. Well, it's not that difficult to save money when you have a company that spends like five to seven million dollars per month, but it was fine, it was cool. In the end, I think the last year went through like the company said, if they don't save us as a team, Andy and I, if they don't save more money than they cost, you don't have to pay for those two guys at all. And that was really easy sell on the management panel. But I wanted to say, I wanted to re-emphasize this one more time and it is the end. If the company has a strategy and if the company is able to communicate through the strategy and if the company is able to stick with the strategy, then tech guys will follow. Otherwise, you will have all kinds of clashes between the management side of things and the engineering side of things. And this is it. And once again, thanks for Nicholas too for his appearance. Okay, questions. Yes. Oh, to repeat the question. Okay, so the question was like I started as a DevOps and then I went into the FinOps and then I worked in establishing the communication between engineering side and the management side and was there enough management knowledge in the company to utilize, to use, to ride on this experience? I would say yes. The first piece of the puzzle that we put in place is a nice prove that it is. The company management really went along with the approach like okay, if you give us, if you give, I mean the ABS, the first thing was the key. This is the key that we use to unlock the company. So when we proved that we can give you the estimate and the amount of money that you can save if you follow the estimate, if you facilitate the needs for the estimate and when that happened and when the company had actually the results to look at, that was the moment when the company said like, okay, this can be done. But that was the piece of the work that was following the company's strategy. Like we want to save the money and that's cool. The moment we get to the second part of the puzzle when we said like, okay, you want to innovate or you want to be more competitive, that was the moment when the company had to say no, we are not really into that. Which is cool, which is fine. The bad thing is if you as a company want to propagate this idea, but in reality you don't really want to follow through really. You want to save the money and you want to look good so you would talk these things because they sound good and a lot of people can align with the ideas, but not in reality. So my answer is yes because it was, I think yes because it was properly aligned with the current company strategy and this is why it worked. It was still amazing how these two, I would say gangs in the company didn't communicate well. Everyone wanted their stuff. If you talk with engineers they would moan about not being able to use the cloud freely to create new clusters to do stuff. They would moan if someone would warn them that they need to clean up the stuff after themselves and all that things. On the other side management only wanted to control. You know what they did? They imposed strict policies on how much money you can spend and you know how engineers got to this? They said like okay we will build something that is below that limit and as long as we are below no one will bother us. And then in this threshold they could do all kinds of stuff and not a lot of them were good. So while analyzing stuff we discovered that actually a lot of accounts that were below the budget, the limit, were very not optimized because they didn't care. They cared only on doing the daily stuff and be done with it because they were below the radar. The company didn't even notice these accounts because they were below the limit. Some manager that was leading these teams was good enough, I would say, fighter to go on a meeting with company management, upper management, to fight for a good budget and then after that he would bring the hunt back and then he would say hey I fought for this budget, we have the enough budget, now you go and play. And these teams were really happy but still not optimized. This helped company actually analyze and isolate these accounts as well. So again, back to the question. I think yes because of this. The strategy in place was correct, it was about saving the money, the engineering was able to follow, the model was simple, it was easy to translate, we would invoke new, a lot of that was written in Python, some of that in Node.js, so both very popular languages that was really easy to invoke new engineers into the game and they picked up things pretty fast, small isolated pieces of code, so relatively simple approach. Yes, why we didn't? Accounts management. Cost management, yeah. So that is absolutely good question. The problem with this approach was that they were using this before us and to properly investigate 400 different accounts using AWS cost management, required them to spend a lot of time over and over looking through that. And you know, in that big company, things change on a daily level, not on weekly or monthly level, that is drastic changes up and down in the use cases, in the usage, in the patterns, in all that stuff. They wanted something that would be automating the whole stuff, so cost management is a good approach. It also fails to fit in the, to fill in all the gap about custom stuff. Like I said, they would have production load running on a dev account and you can't just undo that load. You can't just take it out and move it into the product account. So a lot of these goaches, instance, an example, they wanted us to introduce something they called Big Red Button. You know what Big Red Button is? They wanted us to create a UI with a Big Red Button. They would click on the 15th of December and they wanted us to shut down all kinds of resources throughout the company because they would go into the low usage mode until, I don't know, 10th of January. That was also part of the whole activity. But we started with one idea. 10 days later, we would have all kinds of exceptions from that rule because again, large company, large organization, a lot of managers were able to fight their way and get the exception from this rule. So we had to avoid them. And when you go, I mean, a lot of these AWS solutions were really good and you can use it for that as well. But when you go into an enterprise, having so many different tweaks and things, if you are not willing to dig in and build something custom, not really gonna work for them in a way they expected. In the end, we were using the same stuff as cost management. So we would execute API calls towards the AWS pricing API. We would execute cloud watch calls to get the metrics about EBS volume. So it's underneath, it's all the same stuff. It's just that it was not really usable for them on the company level. They tried two or three different FinOps specialized products for that. So products that are fully baked with all kinds of reports and all kinds of things you can think of. The answer would always be like, tag everything properly and we will handle something for you. That tag everything properly does not go well with that kind of a company. It's tag everything properly with two teams, three teams, five teams. That's really cool. But when you think about 400 teams and you have to go in them, we also had a situation when we would go to a team and we say, hey, you need to tag things properly so that the tool would be able to pick stuff up from the account. And they would say to us this, and I quote, like, when you come to do our job every day and when you have to also manage the existing infrastructure and deliver new features in this amount of time, then you come and you say to us, implement this. And they would just refuse. And they knew they were below the budget limit, they were bringing money to the company and they would just say, no, not interested. And that's the reality. It's one thing that when we talk about it, but this is like the real stuff out of time. So thank you for your time. I hope you enjoyed and see you next year in Brno again about something else.