and welcome to Restaurants Hawaiʻi. I am your host, Shera Matsuoka, the executive director of Hawaiʻi Restaurant Association here on Think Tech Hawaiʻi. And today we are having a great conversation and as Hawaiʻi's businesses are still dealing with staffing shortages and the restaurant community is no exception. You know we're all looking for employees and according to research done by the National Restaurant Association, half of the restaurant operators believe that recruiting and retaining employees will be their biggest challenge through the rest of this year and perhaps through next year. So today we're having a great conversation and another resource for our restaurants, the American Job Center and I'd like to bring on Mark, hey Mark, Mark Minard who is the business services coordinator at American Job Center. Hey Mark. Hi, Shera, how are you? Great, do you want to tell them a little bit about yourself? Great, yes, thank you. First of all, thanks for having me on. Really appreciate you folks highlighting the American Job Center. The American Job Center, basically what we do is we try to match up job seekers with employers. That's our main mission here. What I specifically do is I work with the various businesses to try to make that connection. So we offer one of the resources in trying to connect job seekers with the business but also for the business we do other services and we'll get into this a little bit. I think when our conversation is such as job fairs, helping businesses advertise for their open positions and really trying to recruit the best possible candidates. Thank you. Well, you know, my husband and I always have this conversation is where are all the employees? And I just pulled up the Census Bureau statistics of the Hawaii population that has been declining the last few years. So with our population declining, that even gives us a smaller pool of employees that are potential employees for our restaurants. So if our population is declining and it says 0.7% or 10,358 less people from the previous years and we can see a constant decline, Oahu's population was actually the highest in this report while Hawaii Island and Kaua'i Island, their population of residents increased but on Oahu, our population went down. So the American Job Center, their resource and what services they provide is so wonderful. I mean, just for any business, we're not even specific to restaurants today but any business that's looking for employees, the American Job Center is on Oahu, in Hilo, Maui, and Kaua'i. So Mark, let's first talk about the services that you provide businesses. And one of the services I want to touch upon is the HigherNetHawaii.com. Can you share a little bit of how HigherNetHawaii.com works? Sure. So HigherNet Hawaii is a site where we have a database of people who are registered into that website. So anybody who is actually applying for unemployment is required to register into HigherNet Hawaii. They're required to give all their personal information and then also a resume. It also is a website where businesses can also register in the system and then they can go ahead and post for open positions that they have and also what it does, it allows them to actually find appropriate candidates by searching for different job skills and skill sets that they might have. That's great, Mark. And what we were talking about earlier, Mark and I, was like a lot of businesses forget that this is a resource for them where they not only can post but they also can search candidates. And we were discussing how even the American Job Center does hiring events. Mark, you want to share a little bit about how a business could contact you about doing a hiring event? Sure. So in the past, we've done your typical hiring events where we have several businesses come and we've had up to 30 businesses come and we use the, we're located actually at the adult cannery. So if people are familiar, there's adult theaters and then there's the Komokai ballroom. So last year we had our biggest event that we had was during the pandemic and it was one of the only in-person hiring events that we had. So we've done those traditionally. There are other places that do similar job hiring events, job fairs. But what we really want to focus on is on the individual businesses themselves. So recently we've taken the approach of actually working one-on-one with businesses and to have hiring events for that specific business. So we've done, for example, we've done it with Zippies. We've done it with some vendors out in the airport. We are actually going to have a hiring event this coming Friday from 9 o'clock to 1 o'clock with a specific hiring event for Waihata. And so what we want to do is we want to work, really build that relationship, that one-on-one relationship with the business so that we're able to help one is to understand what their need is and what kind of employee that they're looking for. And then with the help of the business, help promote that business, get the word out that, hey, we're having a hiring event. These are the types of individuals that we're looking for. Come on down to the American Job Center, where you can meet the staff there and actually apply for those positions. So that's really the thing that we're focusing on is that one-on-one relationship with the business. Again, so that we have a really good understanding of what their need is. Very good, Mark. And everyone's burning question is going to be, what is the fee to have American Job Center assist them with a job there at their business? Yeah, so the great news is that we're a free service. So anybody who comes to visit us and needs our assistance anyway, there is absolutely no cost to them, no out-of-pocket costs whatsoever. And that's going to be music to the business's ears. We need resources and the reason we're having this conversation today is because, like I said earlier, many businesses may have forgotten that American Job Center is a great resource. And even if there's a person out there who knows a potential employee, a candidate, somebody who's looking for a position, they also provide lots of services for those people. So this would be our businesses and restaurants pool of candidates and how the American Job Center helps them is by all these resources. So, Mark, let's start off at the very top, which is, you know, you all have different services that you provide people who are looking to get back into gainful employment. So we can start off with the, should we start with the youth services? Sure. You know, Cheryl, as you were mentioning, the current situation out there with the job situation is, you know, we have all these people who are unemployed, right? And then we have all these businesses who are looking for positions. So ideally, or technically, that equation should work, right? You have people looking for jobs, you have employers hiring. So that should work. Unfortunately, what we've seen not only here in Hawaii, but also nationwide is a, is, business is really struggling to find employee, right? And we're no different. And, you know, we hear the same story from across all business sectors. You know, we call people and they don't show up for interviews or they might show up for the interview, but they don't show up for the first day. We even had, you know, a business tell us that, you know, we onboarded them.  So, you know, there's many different challenges. And, you know, one thing I think that we, that we do advise businesses on is that, you know, because of the situation that we're in, we really need to look at other employment pools and to see where else can you, can you hunt for a potential employee? And so one of those areas are youth population. So one of the programs that we have here is our youth services program, the work Hawaii youth services, who primarily are helping young people who, for whatever reason, haven't been able to, you know, finish high school and get their diploma. So we offer them an opportunity to get their high school, their GED. But it's not only just the academic component that we're, we focus on. We really want to lay out a career path for the young person. So along with, you know, getting their GED, we do work-based learning workshops. We offer occupational trainings. In the youth services, we actually offer four tracks. One is in construction, one is in customer service, health care, and in office administration technology. And so those young people are able to choose from one of those occupational industries, obtain a certificate, a credential. We're partnered with, with our local community colleges to have instructors come in. And then with a construction component, actually we're partnered with BIA, which is a pre-apprenticeship. So after they go through this training, they're able to come away with a credential, which is something that is, you know, really valuable when you're putting on a resume. And then once they've gone through an occupational training, then we can also offer them a work experience. So we pair them up with a job site where they can practice those soft skills that we teach here in the classroom, and then to actually get some hands-on experience, right, for that first job, for them to, you know, have something to put on a resume, also to have some good references, which is really important, as we all know. And that's also an opportunity for businesses to partner with us. We're always looking for new job sites as work experience sites. And it's also a great way to potentially get a great candidate for a position, because what can happen a lot of times is they'll come on board as a work experience participant. And then at the end of their training, since they're now, you know, they've been with you for a while, and they kind of know the ropes, and you've gotten a chance to see them, see if they're, you know, they're a good fit for you or not. You know, at the end of their training, if you have an open position, it'd be a great way for you to fill in those spots. And we just spoke about earlier is also the skill assessment survey that helps them identify. So you want to cover that a little bit? So on hiring at Hawaii, there is, there's different things on there, on hiring it that you're, that an employer can use and also a potential employee. So when they're registering for unemployment, there is skills assessments that are on hiring at Hawaii, which will kind of test you on, on your abilities or not really your abilities, but your, what are your, what is your skill set? And what is your aptitude, right? So, you know, it'll take you through a battery of tests, and then at the end, it'll kind of give you a job, a hate, you know, based on your answers, you might be a good fit for, and it'll give you a list of different industries and different types of jobs.  And I think that's so valuable to know where your strengths are, because what you may be interested in, and you find out your strength, it could be a whole area that you didn't realize. Then another resource that the American Job Center also has is free lap use of laptops. And Mark, do you want to cover that a little bit? Only because what I heard was when schools, you know, wrapped up in schools done right now, it's their summer break. Many of the students don't have access to laptops, if they already own one in their home. So, you know, for them to even search for a job or even if they wanted to update a resume, by the way, everyone listening, if you do know of a candidate that is looking for a position, you know, we don't require resumes. If you know of somebody, just have them reach out to me at info at hawaiirestaurant.org. And we try to fit the candidate in their area where they live, because many of them even need to use public transportation. So, I'm pretty sure that in every neighborhood, there's a restaurant nearby. So we always try to fit the candidate with that community that they live in. So you want to touch a little bit about the laptop use? Sure. So at the American Job Center, we have a resource center, which is again, it's open to the public, free of charge. Anybody can come in and use them. And we have laptop stations. We have currently at least 13 laptop stations where anybody who if they're, you know, looking to go online and let's say, you know, go on Indeed or one of these other job search engines, you know, they're able to come in and do job search, register for higher net, you know, we have staff here to help them with that. If they're looking to apply for a job, let's say that, you know, a lot of most right applications right now are online. So if a person doesn't have to be young person, it could be anybody, right? A lot of people may not have access to these resources can come in and then use our, you know, our equipment to do that application, fill out those applications. And the other great thing about a resource center is not only do we, you know, offer these laptops and equipment, but, you know, if people are, you know, if they're not too sure about their resume, or they're not too sure about, you know, going on an interview, and they want to maybe brush up on some interviewing skills, maybe it's been a while since they've been on an interview. We have staff here who are fully prepared to help them with that. We have workshops that we do, you know, to help people.  You don't just want to have a generic resume that you put out there, you know, and then for interviewing, right, if you want to go through some interviewing skills, we have staff here that we do a workshop to, you know, kind of go over those, maybe some of those typical questions, you know, how do you answer, you know, tell me a little bit about yourself, which is, sounds like an easy question, but it can be very difficult, right? Sometimes you don't know what to say. So we, you know, with that, the free resources that we have there with the laptops, we also have that, also that one-on-one resource. And let's also share about the access to translators. Mark, you speak which language? I'm fluent in Japanese. Yeah, I'm embarrassed because I'm full Japanese and I don't speak the language, but you do have access to translators.  Yes, we have resources available to us where we can, you know, find people who can help us with either, you know, it can be over the phone. A lot of times it is over the phone translation, but if somebody wants to come in in person, as long as they give us a little bit of lead time, we're able to find people that can come in and help us with translation services. And it's all about partnerships too. We use other resources with our different business partners that might have access to people who speak other languages. I know, Cheryl, that you have a wide variety of people who speak different languages with all the different restaurants and ethnic restaurants, so you're a great partner always. Yes, I do. So every single language you can imagine, including all the South Pacific Islands, we have every language we can translate at the Hawaii Restaurant Association. And then there's another pool of candidates, if they are not aware of the American Job Centers. Hey, Mark, let's talk about individuals that have disabilities. Sure, you know, as I kind of touched upon, or, you know, really going out there and looking at those different, let's say non-traditional employment pools, and what Re-Youth was one that I mentioned, persons with disabilities is another. When the word disability comes up, you know, unfortunately, we're all human, so we have these unconscious biases, right? So when we hear that word, you know, the immediate thing I think a lot of people might think of is, oh, well, you know, what is the accommodation that I might have to make or is it something that I'm even able to make for an individual? But, you know, a lot of these people who are looking for that first job or just an opportunity can be your best candidates because they're really looking for that, just that chance, that opportunity to work, right? So they tend to be the most loyal, the most hard-working. So it's a great talent pool to kind of find somebody specific for your position. You're so true. In one of my restaurants, we had a very loyal person. That person came to work on time every day. It didn't matter when raining, storming, that person showed up, and that person worked in the kitchen and was very, very dependable and reliable. And sometimes, as you mentioned, right, as long as we understand what their needs are, we can all work together because we're all human, as you mentioned. So yes, that's a huge pool that I think too, it would be a great pool to tap into. The other pool, Mark, that we discussed is senior citizens. Yep. So, you know, we have our youth on one end and then we also have our senior citizens on the other. So, you know, we see a lot of individuals who come in who maybe have been retired for a little bit and are looking to maybe start, maybe not so much a full-time second career, but something part-time to kind of, you know, fill in the time. But we also do have some senior citizens who are coming in or looking to maybe start something new, try something new. So, there's different programs that we have where we can provide services. One is called a CSEP program where they place the individual in a work site where they get paid to stipend. And so, again, they're able to report to work. And, you know, for those who are, again, looking to maybe fill in some time, some spare time, it's a way for them to earn some income. You know, the other service that we provide across, you know, not just for seniors, but for everybody is also for trainings, right? As long as they meet our eligibility requirement, there's literally hundreds of different kinds of trainings that we can provide for. Again, with our partnerships with the community colleges and other training providers, you know, anything from meat cutter to, you know, give your CNA to your CDL if you're looking to drive a truck, right? There's literally hundreds of different kinds of trainings that we're able to support. And, you know, we can provide tuition up to $4,000 actually for you to get certified or get credentialed in some of these training. And for me, when I speak to the seniors that are looking to get back into the workforce, if anything, the pandemic taught us, you know, the isolation. It's so lonely and that they need that social interaction. So, you know, I told them at the seniors that I was speaking to, which is actually, you know, my classmates are all retired and they say, you know, even a hostess or a host or even, you know, being social again would be so welcome for them. So, again, that's another employment pool that we need to think about and to tap into. The other pool that we want to discuss also is our homeless population, Mark. Right. So, we do also have people who are experiencing homelessness. So, we also do provide services to help those who are looking to get housing. So, along with that housing piece, though, goes hand in hand is employment because part of the eligibility requirements in order for them to be able to help them with rent and finding a place to live is to be employed. Right. So, it goes hand in hand then in what we do here at the American Job Center. So, you know, those are also a pool of people who are really looking for an opportunity. Right. And, you know, I think that that goes across the board of what we're talking about, you know, the our typical clients that we see that are coming in are really people who are looking for jobs. Right. They're not just people who, and unfortunately, I think a lot of businesses might experience this is that, you know, when people apply for positions, a lot of times they're just kind of going through the motions, right, because that it's a requirement that that unemployment, right, that you look for a job or at least three jobs per week. So, a lot of times, you know, and we get this story a lot from all the businesses that, you know, we get these resumes and inquiries, but we call and call, we don't get a call back. So, unfortunately, I think that that might be the case where people are just kind of going through the motions. But the people who are actually coming in and using our resources and coming to seek assistance with us are actual people who are looking for jobs. So, they're the people who are, you know, they're ready to work, they're wanting to work.  Thank you, Mark. And I'm getting the five minute warning. Is there anything else that you want to close with? Any message out there? Do you want to give them your phone number to your office and email address before we wrap this up? Sure.  I'm sure you know this more than anybody.  So, you know, I really would like to just put the word out there that we're always looking forward for new partnerships. And any way that we can help, again, you know, help a business with finding that right employee or even with other services, you know, there's a lot of other things that we do here at the American Job Center to help businesses with, you know, for example, when we're talking about business that might, but let's say that they need to do a layoff or they're going to be closing, right? We have what we call rapid response service, where we can help those employees try to transition maybe into some of those trainings that we're talking about, or maybe into another job, help them with the hiring process or the unemployment registration process. You know, the job fairs is the other thing that I was talking about. Or even, for example, if businesses are having trouble, let's say with, you know, let's say a small business is having trouble with their accounting, that their practice is very maybe a lot of small businesses, you know, can't afford, you know, to hire somebody to keep the books. You know, there are ways that we can help through that rapid response. It's called the offer version is to maybe help that business, you know, call in a consultant and maybe get them trained in the right, you know, ways to, you know, do their accounting so that their business is stronger, right? So, you know, it's not only the things that we talked about today, but there's definitely other ways that we can help businesses. And there's probably ways that we haven't even talked or even thought about. And it's about having that conversation and building those relationships and finding those new ways to help that I think is really our strong point, is because we have individuals here that are willing to make those connections. And we've, and you've been a great partner to the Hawaii Restaurant Association, Mark. Thank you always for reaching out to me and inviting us to all these opportunities where we can find employees. So we appreciate all of the support that American Job Center and Mark are doing. So in closing everyone again, there's so many resources out there American Job Center is a great resource for all of the restaurants and businesses to find employees as we struggle right now with the lack of job, the lack of employees. And again, if you have any questions, please give me a call or email me at info at HawaiiRestaurant.org. Again, I'm Cheryl Matsuoka, the Executive Director of the Hawaii Restaurant Association, the voice of Hawaii's restaurant and food service industry. Thank you so much for joining us. Thank you so much for watching Think Tech Hawaii. If you like what we do, please like us and click the subscribe button on YouTube and the follow button on Vimeo. You can also follow us on Facebook, Instagram, Twitter, and LinkedIn, and donate to us at thinktechhawaii.com. Mahalo.


Hi everyone, I'm Ed Baker, and I am your host producer on the Addiction Recovery Channel. Welcome to the show. It's my great privilege today to welcome Dr. Kimberly Blake to our show. Thank you for being on the show, Dr. Thanks, Ed. Thanks for having me. Dr. Blake is a physician. She has been in Vermont since 1990. She came to Vermont in 1990 to complete her residency at the University of Vermont Medical Center. She has since enjoyed a very busy practice in OB-GYN. She's currently with affiliates in OB-GYN right here in Burlington. In addition to that, Dr. Blake recently has expanded her medical role in Vermont as a physician with Howard Safe Recovery. Howard Safe Recovery, for those of you who don't know, is a facility that specializes in harm reduction. Dr. Blake has become the physician there who prescribes medications for addiction treatment. I'd like you to just begin, if you would, with a little bit about your role at Howard Safe Recovery, Dr. Okay, sure. About a year ago, I started a halftime position at affiliates in OB-GYN. It's been planned for many years to have a break of being halftime for about a year and a half. And I found myself with a little bit of feeling a little bit at loss with all my free time. So I looked around for different things to do and it came up in the community that they were looking for a provider to provide buprenorphine treatment. The program at Howard Safe Recovery is a rapid access treatment program. So the people who decide on any given day if they decide they want to, instead of using a substance, they want to start treatment instead, it's aimed for people with opiate use disorder that they can start that same day. And I was very impressed. I mean, had a child who had addiction, the struggle of waiting, you just need to get treatment whenever the person feels that they're ready. And so I was really very pleased to partner with Howard Center, who has such a great reputation in our community for their ability to treat people and take good care of people. Yeah. I want to recognize that work and I want to thank you for that work and recognize how important it is to change around the structure of the way we deliver health care to people with opioid use disorder. It's not business as usual. Motivation comes and goes. We have to be there at that moment where a person is motivated.  Your appointment for your first appointment is next Wednesday at 9 a.m. In between that point and next Wednesday at 9 a.m., literally anything could happen. It literally is a matter of life and death. So I want to thank you for your work and I know that that's not easy work. Now, I'd like to begin, I guess, with a little bit of your personal story. I know you have a personal story to tell and I'll just leave it at that if you'd like to go into that. Sure, sure. So my husband and I have two children, our oldest son, Sean. We found out had a substance use disorder when he was in high school. In his senior year, he started having some very erratic behaviors. We weren't sure what the situation was. We brought him to his pediatrician and she did a drug test. Turns out it was marijuana, which we were a little puzzled with. We didn't think that had that degree of mental health change, but it turns out it was the very high dose. Oils that you vape and it had some pretty profound behavioral changes in him. Sean was subsequently diagnosed with bipolar disorder. And we think that his youth precipitated an episode of mania that happened in his senior year in high school. So Sean went into treatment. He went into treatment locally. He did really well. He was able to start college, but then things went downhill again. He went back to treatment at UVMMC at the day one program and again did very well. And when he was not using substances, he was a very capable, he was a very intelligent young man. He was able to do amazing in his life. He was sober for almost a year when he entered the Navy. And so we felt like he had a good, you know, basis in sobriety. He was very involved in 12-step programs. He got through basic training and sub-school training. Really, he was doing very well. But then he started to have a little bit more free time. And during his free time, he was visiting a friend who lived in New York. And again, it just started just with smoking some pot. Didn't seem like anything too serious. But it quickly got out of control. Sean ended up in a psychiatric facility after a suicide attempt. And the Navy questioned their ability to have him stay, given his risk. But they also offered for him to help them find heroin dealers around the base if he would purchase drugs for them and try and find out who the dealers were. And for Sean with the substance use disorder, that went really pretty horribly wrong very quickly. I went down to visit him one time and I noticed that he was showing signs of withdrawal from narcotics. And I, at that point, I was shocked. I had no idea that things had escalated to that degree. So he was discharged from the Navy and we went and got him immediately into treatment. He went into a long-term treatment. Again, he did very well until he had a pretty rough relapse in 2013 or so. He was missing in New York City for several months. We just, we didn't think we'd see him again, to be honest. But he emerged, was treated and diagnosed with a bipolar disorder. Again, went into treatment. This time on some medication-assisted treatment. At that point in time, there was no availability in Vermont for him to go on methadone or continue the methadone he'd been started on in New York. So he went on Vivitrol, which is a different type of medication-assisted treatment. And that worked well for him. He also had a significant alcohol use disorder component to his disease. So the Vivitrol was a good prescription for him. And he flourished and did well again. He started to slip up with alcohol. And then for the next few years, really struggled with the bipolar disorder and alcohol primarily. As a family, we thought the opiate use was a thing of the past. We thought it was something that he had conquered and was never going to use again. Sure. Unfortunately, because of some of his alcohol-related activities, he ended up incarcerated. And it was shortly after release of incarceration that he used an opiate. We don't know if he even knew what he was using, but it turned out to be 100% fentanyl. And he died within instance. And he was brought back with Narcan and CPR. His brain did not recover and he became an organ donor two years ago. I want to thank you. I want to thank you for your courage in expressing that so fully and so completely for our audience. Kim, I'm terribly sorry that you experienced that and that he experienced that in your life. Unfortunately, tragically, this story is not only your story. There are tens of thousands of families in America that are experiencing this today. We know today that substance use disorder, opioid use disorder is a chronic brain disease needs treatment and recovery services over the lifespan. It sounds to me that Sean had not only substance use disorder, but also mental health issues. So he had basically three disorders from what you say, mental health issues, issues with alcoholism, and then issues with opioid. So this is a very complicated, a very complex young man. And I'm certain that you and your family did everything, everything in your power to help him. And it's tragic that you lost him. Do you think, I'd like to begin to talk a little bit about stigma. Do you think, I'd like to read a quote. This is from Facing Addiction in America. Facing Addiction in America is Vivek Murthy's Surgeon General's report of 2016. For the viewing audience, if you want to learn about addiction and you want to learn about treatment and you want to learn about brain disease and medications and recovery, read Facing Addiction in America by Vivek Murthy. This is a quote regarding stigma. Surgeon General Murthy says, Stigma has created an added burden of shame that has made people with substance use disorders less likely to come forward and seek help. It has also made it more challenging to marshal the necessary investments in prevention and treatment. He goes on, we must help everyone see that addiction is not a character flaw. It is a chronic illness that we must approach with the same skill and compassion with which we approach heart disease, diabetes and cancer. Now I would just ask you, did stigma play a role in Sean's illness? I think most definitely. I was explaining to Ed, sometimes I feel like a difficulty within the recovery community in stigmatizing people who are opiate users versus say someone with alcoholism.  I think for Sean, I think that he did have an opiate use disorder that he was ashamed of. I think that he did not seek treatment even though that was available to him. His primary care doctor was Dr. John Brooklyn who told him at any given point he was happy to prescribe Suboxone to it. He never took him up on that offer. He never made that appointment. He had even seen his psychiatric nurse practitioner just a week before he died. He didn't mention anything to her about potentially using opiates. Though looking back we think that he had been using them since he was released from incarceration. Dr. Murthy in his 2016 report Facing Addiction in America quotes a couple of very shocking statistics. One is that at that time there were 23 million people in America with substance use disorder. Of that 23 million people, 10% are receiving medical treatment. That's 90% or not. And a lot of that has to do with stigma. People with substance use disorder feel ashamed that they have substance use disorder. They hear people talking about them in disparaging terms. This is a weakness. This is a choice. This is criminal behavior. We need to begin to speak about substance use disorder as a disease. There's genetic predisposition towards substance use disorder. There's environmental predisposition towards substance use disorder. There's brain adaptations that occur with continued substance use. It's not a choice. It's a disease. The more the American public can accept that fact and begin to respond to people with substance use disorder, the same way as you would respond to a neighbor down the street that all of a sudden is stricken with cancer.  What can we do? Can we bring over some dinners that they can freeze to make their life a little bit easier? Can we mow their lawn? Can we express some sort of compassion? This is the way we need to respond to people with addiction. So they feel comfortable saying, I have addiction. Where do I get help? Now, I think one of the most beautiful things about people is that we have the capacity to turn tragedy into motivation and commitment. And I think you have done that. I would like you to talk now a little bit about your event that you're sponsoring in the end of August.  We had a quiet just event with some friends and close family members at Battery Park and we watched the sunset. And that was something that Sean always loved to do. We had some of his favorite foods. And I heard kind of through the grapevine that there was no longer going to be an overdose awareness event in Manchester anymore, which made me sad and we don't have something really here at the end of August to commemorate people lost to overdose. So I wanted to have an event that commemorated people whose lives have been lost to either a mental health disorder or to a substance use disorder. I also wanted to have people who have successfully navigated a substance use disorder and or mental illness, give their recovery stories. For me as a parent, it wasn't until I heard speakers at AA meetings that we went to with my son when I felt the stigma that I carried changed. When I could hear people's success stories, so people who could go on and have an amazing life in recovery, that really turned things around for me to say that, yes, we need to invest in treatment for people. We need to get people the care that they need and that stigma was a barrier not just for the patient but also for family members. So I put together an event. I invited several of my favorite speakers that I've heard, as well as a very, very dear friend of mine who lost her daughter last year. Madeleine Lindsemeyer, Madeleine's mom, Maureen, who I've known for several years, is going to be present. Sharon Lamb, another close friend of mine who lost her son, has written a book about being a mom of a child with a substance use disorder and she's going to be there. She's going to be signing books. We're going to have some refreshments. At the end, after the speakers, we are going to have a tribute to people who've lost their lives. And that is really meaningful to me that we have it balanced in a way that people understand the magnitude of the loss, the number of people who are affected, as well as to hear the people who have navigated this and turned things around. It's inspiring. Now, what's the date, what's the time and what's the location? Sure. So it's August 24th. It's a Saturday. The speakers are from five to seven and then afterwards we'll have the tribute. There'll be light refreshments. It's at the Oakledge Park in Burlington. Oakledge is one of Sean's favorite places. There's a really, it's beautiful. There's a beach close by. There's a playground. We have rented the lower pavilion. So it's undercover if there's weather issues. Wonderful. Wonderful. Thank you so much. Yeah, I'm looking forward to it. And for the viewing audience within commutable distance, if you can possibly come, please come. I mean, this is a special event. This is a time to join together and face what's happening, both the tragic part of loss and also the very enthusiastic, hopeful part of commitment to change and moving forward. I think it's interesting that the event is at the end of August in commemoration of those who have gone by and also celebrating recovery. And then September, of course, is recovery month. So it kind of, it fits in beautifully. Now, do you want to talk a little bit also? I mean, you're involved in a lot. There's also your activity with mobilized recovery, which I think is incredibly exciting. Would you talk a little bit about that? Sure. So mobilized recovery is an effort, it's a combined effort between The Voices Project and Facebook. It's a Facebook community public health initiative. Several months ago, I filled out an online application through Facebook to attend a training. The training was paid for, the hotel, food, everything was covered. And it was a two-day event. And the purpose of mobilized recovery is to start a national recovery advocacy group. This group, the training, we learn things like better social media skills, how to talk to an audience. We learned about how to approach politicians, how to talk to a politician and find out exactly what they really feel about a subject, because they can sometimes be a little bit hard to read. And also how to build a community and part of the purpose of my event is to get a group of committed people that want to partner with advocacy, that want to impress upon our communities and our politicians the importance of need for treatment, that people are in dire need. And there are certain things in Burlington, particularly housing, which are really a huge struggle for people, especially early in recovery, trying to navigate. So there are lots of things that we can do. And I was beyond thrilled to be one of only 50 applicants. 50 applicants were chosen out of a thousand, and then they opened it up to another 50, so a total of 100 people from across the nation were... Wow, congratulations. Yeah, it was very impressive. And to meet people who are so much further along in their advocacy than I am was really empowering. Ryan Hampton and Garrett Haid, who are the co-founders of this mobilized recovery, are just an incredibly inspiring gentleman. So that is a plan for a recovery advocacy page through Facebook, which is going to start in September for Recovery Month. And we'll be more about that. All right, so we know about the Our Stories Matter and where to go. How about this? If the viewing audience, if there are people out there that are inspired by that and that are drawn to that, what can they do? In order to find out more about that. Sure. So at the event, I'll be collecting people's contact information, and I'm going to get an email list together. And once the Recovery Advocacy page rolls out, I will communicate that information with people. All right. There's also always ongoing information. My husband and I put together a website to commemorate our son. It's ForKindness.com. F-O-R or the number four? F-O-R dash kindness. And I always post updates on that page. That's a website. It's also on Instagram. ForKindness.com. Thank you. Well, I mean, if get your pencils in paper, ForKindness.com, Our Stories Matter and Mobilized Recovery. Exactly. You are certainly one active person, and I want to appreciate that fully. Thanks. For the viewing audience, I think one of the things that becomes clear here is that here in Vermont, we are very fortunate. You mentioned Dr. John Brooklyn. Dr. Brooklyn, for those of you who don't know, is one of the chief architects of a healthcare delivery model, specifically for people with opioid use disorder, that is the most successful probably in the world. And right here in Vermont, we are leading and rolling out that kind of treatment, making it available. I think in Chittenden County, we have no waiting list, which is really something. We have people like Dr. Blake. We have countless, thousands of people in personal recovery from alcoholism and in AA and NA. We have recovery centers that are, I think we have 12 recovery centers in Vermont. We are leading in our efforts to respond to a tragedy. This really is an all-hands-on-deck situation. We need everyone to participate in any way they can. Exactly. And I would like to mention something called positive messaging. During our talk today, we haven't used the term addict. We haven't used the term substance abuse. We haven't used the term drug abuser. These types of terms tend to perpetuate stigma. They elicit a punitive response. Some of the terms that we have used have been we call person-first language, which I think originated in the mental health movement. Person with schizophrenia rather than schizophrenic. That's carried over into the substance use community. We don't say addict any longer. We say person with addiction. We don't say substance abuser any longer. We say substance use or person with substance use disorder. I just want to emphasize to the viewing audience that mindful choice of language is something that everyone can do every day. And believe it or not, when we engage in this, we become softly spoken advocates. We begin to shape the environment by the way we speak. And you are an excellent example of that. Yeah, it was a process for me, but I feel that our language is important and how we speak about a person. We really need to recognize this as a person first. Yeah, we communicate so much in our language. So, I guess in closing, I want to reiterate my great respect for you, my gratitude, and my camaraderie. We're of one heart, so thank you for being on the show. Thank you. And did you mention that you're going to be at Our Stories Matter? No. One of our speakers. Thank you.


Just getting out of Expendables 4, it was hilariously bad. I've never been impressed with any of these films, but I also never left thinking, my God, do they have $2 to work with on this budget? And it looks absolutely fake. Oh! At times you could tell me it was made using the power of a PlayStation 2 and I'd believe you. You know it's bad when Megan Fox is the best actress in the film. This is also easily the worst cast of characters in an Expendables movie. Now, if you're only here for the action, there is a lot of it in the second half. They only had budget for one set piece this time. And Jason Statham is the main focus in this, is character Christmas, who I think was probably shooting four different movies at once.  And some of it looks like it was repurposed footage from Meg 2. I guess to summarize, I would just say, it's not good.


Writing in the Washington Post this week, E.J. Dionne claims that it's the Austrian School of Economics that's responsible for all the gridlock in Washington, D.C. I only wish that Austrian economics had so much influence, and I only wish we could get some more gridlock in Washington, D.C. Dionne claims that it's the popularity of the ideas of Ron Paul, Ludwig von Mises, and F.A. Hayek that's causing all of this gridlock in Washington, D.C. Well, I think it's certainly true that the ideas of those men are popular. I don't think it's quite true to say that they're influential as spending deficits and money printing grows ever larger. Dionne claims that the New Deal got us out of the Great Depression, that World War II got us out of the Great Depression. But in fact, real consumption per capita during World War II was the same as it was during the 1930s. It was the return of peace, prosperity, sound money, and free markets after World War II in a relative sense that led to prosperity. He also claims that Keynesian economics has saved us in the current crisis, that Ben Bernanke and Hank Paulson came to the rescue and got us out of this crisis. Well, we would argue that we're certainly not out of the crisis completely and that Ben Bernanke and Hank Paulson have only increased the supply of money and national debt in the process. He also makes the curious claim that Hayek said in the road to serfdom that one step on the road to welfare redistribution and we're going to end up in Nazi Germany. Hayek, of course, never said that. If we look at what happened from 100 years ago, let's say, to today, what we find is in America 100 years ago where the average American had no contact with the federal government outside of the U.S. post office. Today, look what's happened. We have a national debt of $17 plus trillion. We have a federal government with unfunded liabilities of over $200 trillion. We have our military and bases in over 150 countries around the world and we have a national security agency, the NSA, that is spying on Americans and people around the world.  And we also have 2 million Americans in prison. So Hayek's theory about the road to serfdom certainly brings some truth today. We may not be in Nazi Germany, but we've traveled down the road to serfdom a very long way. And that's the Mises View.


OK, now it's time. So we are going to be presenting one story, essentially how we use several cloud-native foundation projects to build a telemetry pipeline, mostly with a network automation use case in mind. And the key elements are Kubernetes and some operators, including the stream C1. And yeah, the idea is we will be introducing ourselves the context. Then we will describe a bit the project and the platform supporting that project to then cover some details about the solution, as well as some associated problems and issues we had. We will be using a slightly different format for this. And before closing, we will be also sharing some highlights about the impact in our organization. But before that, Fernando. Hi, everyone. I hope you are all having a great conference so far. So I'm Fernando. I'm working at Fastly. I'm basically enabling Kubernetes for Fastly control playing applications. Kubernetes is not a new space to me. I think I started on the early stages, thanks to companies like 20 in Agra and Bitnami. I had a great experience there. And I also had the pleasure to contribute to the Kubernetes open source community with bug fixes, features, and even custom controllers, like a 20 secrets manager. So this is me. Then I hand over to Dani again. And yes, this is Daniel, just an engineer at Fastly. I've been doing lots of DevOps stuff, mostly in the Barcelona area, so also from Spain. And yes, I also love open source. And I try to contribute as much as I can. And also, let's introduce a bit our company, Fastly. Maybe you already know about us. Our mission in the end is to make internet users happy. And how do we try to do this? It's mostly by having a distributed architecture where our customers can execute code as close as possible to the end users and deliver contents, being in mind, performance, and reliability. So yes, we have a relatively large network defined by software. We let our customers to run applications in our Edge. And yeah, the idea is to make our customers as autonomous as possible to let them just to create whatever they need in our platform and benefit from our infrastructure. Some numbers, if you are more used to request per second, 8 billion requests per day means an average of 10 million requests per second. So yeah, the size is relatively high. And that was quite clear probably almost one year ago that we had an outage, a one hour outage, and we were on the news. So that's scary, but also interesting at the same time. And yes, this is more or less how our network looks like today. So we have many points of presence across the globe. We try to put them close to our end users. And the main concept here is that the scale of the network is quite high. And as a consequence, we need more stuff, so things that are not in that data plane in our fleet. We need the control plane as well to orchestrate and to manage all this large network. Actually, in the control plane is where most of the automation magic lives. And this story is about one of these automation services, which is autopilot. Topilot is probably not a very original name. I think I have heard about autopilot three times already in this conference, but this is our autopilot. And in order to introduce the problem that it solves, let's take a look to how our pop networking looks like at high level. The idea is that, yeah, we have devices, network devices, compute devices. And they are connected to several ESNs, other ESNs. So some of them act as transit providers for us. So essentially, when we send traffic, they can route the traffic anywhere on internet. We have other relationships with other ESNs. So we may have direct links, bidding relationships with them. But here, the key message is that in order to deliver traffic from our platform, there are multiple links available, so many routes for a given destination. And in the end, we need to make decisions about what is the best way to forward traffic to our end users and also to the origin servers. And that is what autopilot is trying to solve. It's about bidding in mind performance bidding in mind capacity, how hot our links are. We need to route traffic accordingly. And automation here is not a new thing. If you browse our engineering blog post, you will find previous iterations of this. If you are also working in a large-scale network, in a big tech company, maybe you have your own solution for this. And who is behind this last iteration of autopilot? Essentially, it's the NCO team that it's in the network systems at Fastly. And initially, we had six engineers focused in that initiative. And these engineers were, we had mixed skills. So there were software engineers, network engineers, also SREs like me. And here, each one also of the key messages. In the end, there are many implementation possibilities for something like this. But we need to bid in mind the team supporting, extending this solution. And that was also a key item in our design. So autopilot at very high level. We are not reinventing the wheel. We follow the typical pattern of measuring things. Then computing if we need to push a network change. And finally, also the routing manager, like a wrapper of our routing infrastructure, which in the end also applies the changes to our fleet. Let's actually focus in the yellow box, the telemetry pipeline. And what is that telemetry pipeline doing? It's essentially consolidating many inputs, telemetry data from many sources. We have systems, information, so metrics, mostly from our Prometheus infrastructure that describe the status of the links, what is the capacity of the links we have. Then we have flow data. And that is a super key for this problem. We are sampling the network data we are moving in our fleet in order to know what are the destinations of every packet we are sending across our network. We are sampling, of course. And then we also consider performance data, even at application level. And even we are also proving the different links we have in order to know what's the performance and the status of these links in our network. So how the pipeline actually looks like. This is the portion that cares about the net flow, the s-flow data consumption. So essentially, we have network devices, switches in our case, which are emitting s-flow information with an s-flow agent. We encapsulate that data using a DTLS tunnel. So everything is UDP, and that will be relevant later. Then in the actual pipeline, we process, we have a first stage that is kind of enriching and aggregating some data using PMACT, that it's another open source project, to finally push all the data to a Kafka topic that is acting like a buffer. So finally, we have an API in top of this that is maintaining an in-memory view of the network state. So offers and GRPC API, other components are consuming, including the controller that is the one that implements the magic. In order to enrich that telemetry data, we also need to know the routing state. And we do this via that route manager service. And finally, as I mentioned, we are also collecting, consolidating metrics from our Prometheus infrastructure, essentially some counters and other data directly coming from our switches. So in order to run these services that architecture I described, we need to process data at a certain scale. Just to give you an idea, it's more or less given the sampling we are doing. It's about processing hundreds of thousands of packets per second. We need a runtime to manage these components, these services, to deploy new versions, and so on. And of course, as everyone wants probably, we want to run this in many different locations as close as possible to our points of presence. So we need to put this in several places. And we also want to be a dissolution, wanted to be also cloud agnostic. So we didn't want to couple this to another cloud provider. Given our control plane is very diverse, we wanted to be able to run this also in some dedicated data centers we have for control plane workloads. And yes, surprise, in order to run control plane workloads, when we started this project, we didn't have a proper standard. So for the data plane, yes, everything is super standardized. So every component we put in our fleet is using some frameworks and well-known patterns. But for the control plane, it's a different story. We had a mixture of different infrastructures, different cloud providers. And even in the network control and optimization team, we were already using Kubernetes, but many Kubernetes clusters in Google Cloud, mostly dedicated for individual workloads. And actually, regarding Kafka, Fastly was already using Kafka for other use cases, but we didn't have a proper production ready setup. So there was actually even opposition from the team to incorporate Kafka in the project scope. Even there was the perception that that was complicated, it was complex. We were a small team. So managing something like this could be a challenge. And now I'm handing over to Fernando who will describe a bit more the Kubernetes posture and what's the platform point of view and how the Kubernetes offering and internal offering evolve to match these requirements. So thank you, Dani. Kubernetes at Fastly circa 2020, it pretty much looked like this. You want a cluster, you get a cluster. You want a cluster, you get a cluster. Everyone get a cluster. Teams were pretty autonomous here. Like teams will own like cloud projects or cloud accounts and creates their infrastructure. The problem is that led to a very fragmented, underutilized and expensive infrastructure. Like Dani mentioned, we have several clusters over there, hosting only one service maybe. And since it also came from mostly individual initiative, there was no standard way of creating the clusters, configuring them, or even deploying to them. Some folks will use Terraform, some other folks will use the cloud console directly, others Pulumi. Some people will use them to deploy, others Royamel, other Customize. And there was also a maintenance burden because mostly the engineers deploying on Kubernetes, they didn't have the experience to administrate Kubernetes clusters. So Fastly decided to create a Kubernetes team. Initially, we were four, two folks in New Zealand, one in Canada, and myself in Spain. And the goals were to build a Kubernetes shared platform for the control plane, standardizing infrastructure deployment and application deployment. And the scale eventually cross-region, cross-clouds, and eventually elevate also developers capability to deploy faster and scale. And the latter brought the project name, which is also not very original, Elevation. So let's talk about Elevation a bit. Elevation first version was indeed pretty simple. We started with one cluster per stage and only one region. But we designed everything to scale cross-region. So if we need more region, we will create another cluster. We standardize it on hand via Fluxidy, using a GitOps pattern, because GitOps was not new to Fastly, even when they were not using Kubernetes in the beginning. And we deployed to one single cloud provider. But the stack, the whole stack was designed to be cloud-agnostic, because we were envisioning that some people would request other cloud provider or even bare metal clusters. So to accomplish that, we did things like implement authentication with our identity provider, not tying to the cloud AAM. We were using hardware. We're using hardware as the container registry, not GCR, not TCR, or anything like that. We're using Vault for Secrets Management, because Kubernetes Secrets, and also we're using Ingress and Gnex, and we're using the cloud Ingress. And also, Ser Manager is not the cloud. Google Cloud Ser Manager, AWS Ser Manager, or anything like that, we're using JetStack Ser Manager to issue threads from lesson create and also from our recently released root CA. Our Serability Stack is pretty common. Prometheus, Grafana, Splank, Fluendi. So it's also open source, not tying into anything, not tying into any cloud provider. So as soon as we released the first version, we started having the first customers, and we tried to gather feedback. The most important feedback we got is, hey, we need more regions. We need more cloud providers. We would like to have bare metal clusters, because we are hosting latency-sensitive applications. So we want to be closer to this pop or this fast lead deployment, or this cloud service, right? So we got it. The other thing was reducing the onboarding overhead, because the onboarding at the beginning was a bit convoluted. So folks will register the name space and will register a predefined service account. So then we will have to configure Bolt so the service account can read in a specific path in Bolt. And there was a bit convoluted, like I said. Also, the pilot team, they wanted a proper Kafka support in the company without needing to go and create more VMs, peering with the VPC, or something like that. So they wanted something in cluster and they asked for help. Also, there were other teams that they were completely new to Kubernetes, Helm, and all that stuff. And when they tried to get into our platform, they were like, oh, geez, this is overwhelming. Can you folks build more abstractions to engineers like us, more used to networking than Kubernetes? So also, we provide a service mesh in the first version and we were not providing a lot of visibility into a service mesh, so that was a reasonable ask. So we did, and with that come the second version of elevation, which is more or less how Kubernetes looks at fast lead today. It became pretty quickly, sorry, it became clear pretty quickly that with 14 members, we won't be able to scale properly and create cluster cross region and cross clouds and whatnot, so we got two new members that have been very helpful, one in the US and one in the UK. So this is what we did. Clusters in Google Cloud, in AWS, Barre Metal is spanning across three different continents and regions and so on and so forth. And we have today, I think like 20-inch clusters and I probably were creating three or four more in the coming weeks, so that was what we did. One of the key message that I want to share is that we started relying a lot in operators, the existing operators and operators that we built in the team. And that was a huge win for cell service, making teams more autonomous and automation and stuff like that. This is one of the examples. So we built a custom controller to configure HashiCore Bolt. So we have also a tool called Kiverno, maybe you're familiar with it, which is a policy management tool. So now, when people are registering a name space in elevation, under the hood, Kiverno will create a secret engine, KV2 secret engine that will get mounted automatically in its last name space.  Kiverno will create a bolt role and a bolt policy using our bolt controller. So that solved the whole onboarding issues and we relied in more controllers. Also, we provided some abstractions. We created a library chart and also a default chart that will encompass all the best practices, like making sure that you have both disrupting budgets, that you have both anti-affinities and that you don't have to care about, should I put this annotation for this or that annotation for that? What ingress class? What's your manager issuer? And stuff like that. So with this simple YAML file, people will be able to deploy an HTTP application with TLS and ingress routing. We made some observability improvements. So in elevation, I think this is pretty common pattern. Whenever you deploy your containers into elevation, we provide with a default dashboard, which is called the workload overview, right? So you will get CPU memory and stuff like that. But we were missing a lot of service miss visibility, so we built in a bunch of LinkerD dashboard into our default dashboard. But of course, we enabled the LinkerD UI and we enabled teams to execute on their name space LinkerD tab, LinkerD top, and all that stuff to troubleshoot because it's freaking important for teams like Autopilot, which is a network service, right? Other than that, GitOps is cool, but people were like, okay, so this thing got merged. So now what? Is it deployed or not?  This is just an example. So to make sure that they understand like the thing got deployed correctly. And of course, we talk about using an operator in the cluster. We agreed that the string c seemed like a strong, a strong operator, well-maintained to provide cell service around creating Kafka clusters and teams could be very autonomous on that. So I'm gonna talk about a little bit how it is the user experience in elevation today, what teams like Autopilot are actually experiencing. So we have an initial exploratory phase where we have like ARBAC rule relax, relax it, policies relax it. And these are basically development clusters. We also have a playground project in our container registry in Harvard. So people don't need CI to push an image to it. They just go ahead, develop their thing, iterate, push their images, and then we will allow a hardware playground project in our development cluster. There's no enforcement either on using flags, GitOps or anything like that. They can use the default chart that we provide, their own health chart, Roya ML customized, whatever they need to iterate and get their application running and then being able to move on to the next phase. Obviously we have a bulk cluster per cluster. So they go and put their secret there if they need to. And once they feel comfortable, they go to the next phase which is build and then deploy. So they just push their Docker file and their health chart. If they have a health chart and not using the default chart, then a Jenkins job will kick off and then Jenkins will contact Haskell Vault with the hardware plugin that we built in-house to issue ephemeral hardware robot account tokens. So then Jenkins will sign the images, will package the health chart if any and sign it as well and then push it to Harvard's FastList project which is the only one allowed in production and staging clusters. After that, they can just go ahead and deploy using flags. So they just need to create a pull request against our HEM releases repository and some teams even go very innovative and created their own Jenkins pipelines to automatically generate HEM release from a Docker image tag, updating only the key values that they need. Then flags will go and synchronize and deploy the HEM release to the Kubernetes cluster. So now I hand over to Danny to explain the good about and the ugly things about all of this. Correct. So yes, we are gonna now present some details about the solution in three different buckets and that reminded us this film. Film by the way that I didn't know but it was recorded here in several locations in Spain including Almería that is in the south of, let's say Valencia. So yes, let's first see the first buckets, good things and one is like configuration management for this solution. So this piece of Jamel, hope the guys in the back of the room can read something but this is just an example that comes in the StreamC documentation. It's just how to create an example Kafka cluster to define several properties, including the sizing, how, which some defaults and even some properties of the shoekeeper cluster that it's backing Kafka and even other operators that you can plug to this setup. But we didn't stop there. As Fernando mentioned, we are using Helm charts to push workloads to the elevation to the Kubernetes clusters. So we also created one for the telemetry Kafka and that chart is actually exposing some values. Some of them are sizing matters which are not abstracting the cluster too much but some others and I think the important piece is the autopilot sites value that it's in the bottom. It's the list of posts that are given deployment, given autopilot deployment is expected to support. We have equivalent properties in other autopilot services and the idea in the case of Kafka is that if you define a specific pop you want to support in a cluster, automatically you are gonna get the topics you need for the telemetry data for that pop.  Then some processes that are emitting the data like SFACTD from PMACT can pick the secrets and directly start pushing the data to the topics and the same for the reader in a predictable location they can pick the credentials and that's it. So everything is packed. And here so the first key point in this section is like we don't have a separate tool on our configuration management system for the Kafka cluster. It's just another helm release in as we have for the other applications. And in fact, in the past I have been promoting for instance Pulumi a lot also externally but not having to require a different tool to manage your Kafka cluster is even better than the best of the tool. So more things about the helm chart we have in the middle that is also as I mentioned a good abstraction opportunity and the sizing knobs which are very couple to the Kafka setup could be even removed. Now that we are supporting the solution in many pops we can predict how much, let's say how big the Kafka cluster needs to be given the number of pops we are supporting. So we could even drop these values and in the end just have the list of pops we are gonna support. Regarding portability, these are just a few commands. This is the helm releases repo Fernando was mentioning. So we have several helm releases as Jamel files in a given folder, one folder per cluster. And in the end in order to deploy autopilot in a different cluster with the same configuration is as simple as copying the helm releases. And in fact that is already outdated where autopilot was already running a few weeks ago and we had many different clusters some in AWS, some in GCP. And everything looks more or less the same from a configuration perspective. Regarding pluggability and mostly focused in operational tooling. I think also the stream operator does a great job exposing some operational flows directly as Kubernetes primitives. So in this case if you want to browse which Kafka topics do we have you can just use QCTL and list these objects or even other maintenance operations like I want to trigger a rolling upgrade in my cluster. You just, the stream operator maps this to an annotation in the stateful set for the cluster. So in order to trigger it you just put that annotation and the magic happens. Then pluggability also with the observability systems the stream community is maintaining like some great Grafana dashboards but it's not just the fact that we have Grafana to display all the Prometheus metrics it's the fact that all telemetry services can share a common dashboard where you can easily correlate information from the meter of Kafka with the Kafka metrics themselves and that is super powerful to in the end identify and correlate problems you may have in your setup. And the same applies to the logs we are using Splank we are forwarding everything to Splank and in a single query we can check okay what are the logs in this case of the telemetry component so the one that is consuming the data but also the logs from the Kafka cluster and you can easily correlate events. So summary of this section Kafka in our setup is not a special thing it's just another workload in the cluster and we use the same tooling and the same processes to manage Kafka as well as the other workloads we have in the telemetry pipeline. Then things that are not so great in this setup is the fact that yeah we have an operator an operator is an extra component you need to install and maintain in your cluster compared to some software as a service solutions and actually that brings an interesting topic that is ownership of the operator. Right now the operators in the elevation clusters are maintained by our platform team so Fernando's team and teams like network systems they are just consuming the operators however we feel like this something like this may not scale well so in the moment we have other teams consuming or creating the Kafka clusters just coordinating the teams to perform a Kafka upgrade normally in hand with an operator upgrade can easily become a full-time job. So yeah we bet that at some point if this solution is kind of consumed by other teams we may need to involve other teams to support the solution. And Fernando.   So the first very interesting challenge that we found is that you know autopilot telemetry API receives like a constant UDP flow non-stop it's non-stop from the switches. So we noticed that pod that got restarted and rescheduled into the same node somehow the UDP package started to be black hole. My teammate and friend Danny Kuchinski thankfully discovered a bug in Qproxy which basically you know when you're going from one endpoint to zero endpoint Qproxy will flash the contract entry corresponding to the load balancer external IP and the pod IP, right? But contract, sorry the flow will still hit the node if it's rescheduling the same node. So a new contract entry was created before the IP tables not got applied so the contract entry had the load balancer external IP and not the pod IP. There was very interesting we found that bug they fixed it pretty quickly so that's done. This one is also a good one, very interesting because it was a whole team effort and I appreciate that. So we hit a hard limit in AWS security group rule limit and then we reach out to our team. There was nothing they could do and basically the problem is that the AWS load balancer controller will create an excessive amount of security group rules per load balancer. So it's one imbal rule for the node security group per client traffic per all our source IP which is something that autopilot for instance uses a lot. And then one rule for the health check on each subnet in the VPC. So one of the recommendations where I'll switch to ALB and this is not HTTP so not very helpful or switch to ELB it's deprecated right? So not an option either. And we needed also to preserve the source IP because otherwise we couldn't perform authentication based on IP and stuff like that. So we didn't want to implement something like proxy protocol or anything like that. So we came up with a creative software solution which basically was let's disable the security group rule creation at all and let's just use Kiverno to create a Calico global network policy that will allow our block traffic. So we did that. But you know like in Kubernetes there is an orphan dependent rule right? So a load balancer service cannot own a global network policy because the global network policy is cluster scope and the load balancer is name space scope. So Kiverno couldn't delete the couldn't garbage collect the global network policy if the service, if the load balancer service got deleted. So we implemented our own custom controller, custom service controller based on a notation only to garbage collect the Calico global network policy. This being rolled out in production like three, four weeks ago and it's working fine and we're not expecting more issues in that regard. So handing over to Dan again. Yes, let's quickly review the impact of this setup in our teams. The first box actually represents the first PR we had in the telemetry repo. So and it's actually the bootstrap. So just gopher app is an internal tool to bootstrap a Golang projects. So just creating the structure and the second is represents actually the moment where we enabled the telemetry pipeline in a non-production cluster and it only took like four weeks to get the full solution working in that non-production pop and considering that the team was quite small with also maintaining all the services and doing all the projects that was pretty nice. Also reviewing the history of the repo with the helm chart for the Kafka setup. There were some changes in the last months but they are mostly connected to Kafka upgrades and operator upgrades and not to scale the solution. We essentially were able to just adding additional values to scale the solution to all the pops. And in fact, if you take a look to the helm releases repo where we have the actual workloads we are running in production, you can see that there has been already like 10 engineers contributing to the Kafka configuration that silently means these engineers have been creating Kafka topics, Kafka users even without knowing. So yes, we have like this impression that okay autopilot actually manage itself and given this right now we are working to consolidate other telemetry processing in the same architecture and also the net of teams or network operations they use this type of network data to operate the network every day but there are even other use cases like capacity planning and implementing other solutions on top of this information. Yes, and not only the Kafka like the telemetry pipelines consolidation we are also now migrating other workloads we had in these dedicated GKE clusters to elevation so we can just focus in other challenges we have. So let's now start closing the session. Main points we need like an scalable platform to run a network automation at Fastly and Kubernetes is a great fit for this and in addition our very specific set up with some operators like the stream C1 and also some other configurations is helping us in projects like autopilot and it's a telemetry pipeline. So thanks for being here. I think we have maybe not so much time for questions but maybe we can accept one. I don't know if there is a microphone for this or if you want to ask and I will try to repeat. We use mutual TLS for not exactly, well just it's a DTLS and yes, the perfect. So the question is how the Kafka consumers and the producers how they do authenticate against the Kafka cluster. So okay, so how we distribute the certificates across the different workloads in the cluster. So here stream C is our friend. It's when you declare a Kafka user that it's a specific object. Stream C is a custom resource. Stream C gives to the cluster. Stream C actually there are different like user authentication types. Yes, we are using TLS based authentication and that means that the stream C creates the credentials and stores the credentials as a Kubernetes secrets. So then given the workloads using Kafka live in the same cluster, we can directly pull these credentials from the other workloads. Correct, yes, we are using that shared Kubernetes offering from platform. So we have a dedicated name space for autopilot services. So yes, all these services live in a single name space. Thank you. So thanks.