show today, Ola Ialoja. My name is Marisol Ruiz. I am the co-founder and vice president of the Hispanic Chamber of Commerce. Our president, Barbara DeLuca, a host normally with us today, won't be here, so you're stuck with me. But I'm very excited today. We have a wonderful guest. His name is Dr. Kelly Iaquina. And we have a lot to talk about. We know 30 minutes is not a lot, but we're going to try and get as much information that we can from him as possible. Okay. And he is the, sorry, I keep doing that. I'm so sorry. Dr. Kelly Iaquina is a trustee at large at the Office of Hawaiian Affairs. And he's a president and CEO of the Grassroot Institute as well. Today, we're going to focus a little bit more on the Office of Hawaiian Affairs. So we're going to get right to it. And let me start off first. How can I address you? Would you like me to call you trustee Iaquina or Dr. Iaquina? What would you prefer? Well, Marisol, I'd be delighted if you just called me Kaylee. I'll try. I'll try. But thank you so much for taking time out of your day. We know you're really busy and there's a lot for you to do. But we're going to start off a little bit about if you can tell me a little bit about your background, please. Well, certainly. And Marisol, Ola Ia Loa. And thank you so much for inviting me to be on the program. And to all of the members of the Hispanic Chamber of Commerce and all your fans of this program, just Aloha, it's a delight to be with you. But you asked me a little bit about my background. My name is Kaylee Iaquina, Hawaiian Chinese name. And basically, my ancestors came from China and from Hawaii. I like to put it this way, a very smart Chinese boy came over here to Hawaii. And that was back in the 1800s. And the reason he was smart is he married a beautiful Hawaiian lady. And he became the Aquina clan. And so we are one of many, many people who are carrying the last name Aquina. My background is varied. I've had several careers. The first was in Christian ministry. I worked with young people, especially on the Waianae coast on Oahu. I headed a program called Youth for Christ. And then after that, I went on to complete my PhD in philosophy and ethics and became a teacher of professor here in Hawaii. And also I taught and lectured in Asia. But what I've do most recently is head up a think tank called the grassroots institute that solves our economic problems. And in the public sphere, I ran for office. So I'm in my second term as a trustee at large in the office of Hawaiian affairs, which is a wonderful organization dedicated to helping empower Hawaiians to improve their conditions in everything from economics to their social sphere. Right. I have a question. So you're in your second term. And I understand. So you started your first term 2016. So their four year terms are there. Can you continue to serve? Are there any caps on on how long you can serve as a trustee? Marisol, that's up to the people of Hawaii. Okay. How long they would like me to serve them. But there are no term limits for trustees in the office of Hawaiian affairs. Ah, okay.  Well, that had to do with my deciding to come back and get involved with the public world in Hawaii. I was lecturing in Asia in China at the University of Beijing and observing how the people interact with their government over there. And I realized that while there are a lot of problems outside of the United States and outside of Hawaii, we have our own problems in Hawaii that need to be solved. And so I came back and decided, you know, I'll get involved. I got involved in civic organizations and a think tank, as I mentioned earlier. And then I decided, you know, I'm part Hawaiian and I can help the Hawaiian people by running for office. And so I ran for the office of Hawaiian affairs and was elected in 2016 with about 200,000 votes. And that means that it wasn't just Hawaiians, but it was the broad spectrum of our community that felt that I could accomplish something for the sake of the Hawaiians and for the community. So that's what I've been doing ever since then. Oh, that's fantastic. And then so I'm not Hawaiian. I'm Latino. So how does, I mean, it's obvious that OLA helps and empowers Hawaiians, right? But how does it impact non-Hawaiians? You know, Marisol, that is a great question, Marisol. But before I respond to it, let me say that a trustee, a venerable old trustee many, many years ago turned to me one day at the trustee table and said, you know, Kaylee, most of us Hawaiians are mostly something else. And that's because we are people who love people and who accept people and are open to people. And so many people have come from across the world to the Hawaiian islands that my story of a mix between a Chinese man and a Hawaiian woman is the story of everyone that all races come together and combine. It's just a wonderful thing. In fact, I learned later in life that I am also Hispanic. I didn't know that. Oh, really? That we were the Hawaiian Chinese family. And grandma, who came from Chula Vista in California, would always make patele and postada. And all of these different foods that my Chinese and Hawaiian friends really didn't know about. And then I eventually learned that in our ancestry, we have a smidgen of Puerto Rico. Oh, Puerto Rico. That made my food and my life a little more spicy. But going back to your question, the Office of Hawaiian Affairs obviously is committed to helping Hawaiians. It was established by the 1978 Constitutional Convention in order to better the conditions of Native Hawaiians. And that's everything, housing, to family, to economy, to education, and so forth. And in many ways, sadly, Hawaiians are at the bottom of our demographic standards and lists in terms of being the people with the greatest number of illnesses or the most incarcerated or the most impoverished. And so the Office of Hawaiian Affairs exists in order to change that, in order to improve the conditions of Native Hawaiians. But you asked, what is the impact upon people who are not Hawaiians? And it's a big impact. Well, first of all, whether you're Hawaiian or not, you care for the people of the islands, especially the indigenous people who are here before others. And so I think that it's a wonderful place here where whether you're Hawaiian or not, you can join in with the Office of Hawaiian Affairs to help the conditions of Native Hawaiians. One way in which that practically impacts people is that a lot of our homeless and houseless population is Native Hawaiian. If their needs aren't met, that becomes a burden not only on them, but also upon all other people in our community. And so when the Office of Hawaiian Affairs helps to resolve housing problems and empower people economically to rise up from homelessness and houselessness, they're doing something that not only helps Hawaiians, but helps the rest of the state. It also relieves the state from the burden of having to care for that, because OHA can provide greater resources than just welfare from the government. But there's one other which OHA affects everybody. The Office of Hawaiian Affairs is the 13th largest landholder. Okay. It also is entitled to 20% of the revenues of the public lands of Hawaii. And that is about 1.3 million acres. That's the land on which we have the airport or the university or hospitals or state property and so forth. That is a substantial amount of revenue that goes to the Office of Hawaiian Affairs. And in fact, it's not all the revenue they deserve. We're trying to get it all back, but it still is significant. And what's done with those revenues affects everybody. Okay. Virtually every decision in our state that has to do with land or air or water or people is something that the Office of Hawaiian Affairs weighs in on. And so if you care about everything in Hawaii, the Office of Hawaiian Affairs is one of the ways in which you can be involved in helping to bring about the best solutions and the best management of all the things that affect us. Right. Now I'm going to ask you a little, maybe a little tougher question here. Speaking of revenues and kind of money coming in to OHA, with OHA the past history, there's been some controversy over the proper use of funding by some of the trustees and employees. You were personally involved in exposing and dealing with the situation.  Well, thanks for the opportunity, Marisol. One of the reasons that I did go into public office, and OHA is a public office in terms of the trustees, we run statewide for election. One of the reasons I went into that is because I care a lot about the quality of our government. I care that we have a government that is honest and transparent and accountable and limited to the purposes it was established. That's so important here in the United States, and the Office of Hawaiian Affairs is part of government. But it's not always been the case that it has been operating as best as it could in terms of accountability and transparency. When I was first elected, I remember going to my office and one day a member of the administration, the staff, came to me with an envelope and a check. And the check was for $23,000. And I said, well, what do I do with this? She said, well, you just put it in your bank account, keep track of it. And it's for miscellaneous expenses that you have. And I thought about that. And I said, this is kind of strange because I know that in real estate, you don't commingle funds. You don't take money from your clients and put it in your own personal bank account. And then as I looked at that, I started to learn that other trustees, some of them, not all of them, but some of them had used those funds for all kinds of personal expenses or lavish dinners or other things that were not appropriate. And then that got me thinking that it was important to really review the financial practices of the organization. So I set about promoting an external audit that would be done to take a look at how the finances were being used. It took a little bit of fighting to get that going, but eventually we got that audit and we had another review of the same audit and learned that there had been huge levels of fraud, waste, and abuse. And the purpose was not to try to catch people and get them into trouble. The purpose of that was to correct something so that the funding that goes into the Office of Hawaiian Affairs can actually get to the people. So it can actually be used for housing and jobs and education and healthcare. So I'm very pleased that I had the opportunity to work with my colleagues on such an audit. And as a result of that, many measures have been put into place over the last several years in the Office of Hawaiian Affairs that now make it a place that is very accountable in terms of money. We have very good record keeping. We have very good accountability and transparency in terms of the finances of the Office of Hawaiian Affairs. I'm very proud of that and of my colleagues, our work together in order to change an institution from being something that wasn't accountable into what it needs to be for the people. And that's so important because the bottom line is now that helps more people here in our community to be able to benefit from the funds and go into the organization. Yeah, I think that's really interesting, too, because I mean, correct me if I'm wrong, but I feel to do something like that takes a lot of courage. I mean, the Ohio, when was it established? 78? 78. So we're talking what almost 45 years. And then you come along, I think it takes a lot of courage because that's a long time for maybe certain habits or things have been run a certain way. So to somebody to come in and I don't want to say disrupt, but kind of disrupt and say, hey, we got to do things the right way and shed light on that. I'm sure you had some people that, you know, it must have been challenging for some people, but did you find that really difficult or was it one of those things that you knew it's going to be challenging? But it's what we have to do. It was easy for you to do in that case. Well, let's put it this way. Not everybody was comfortable with the results that were coming to the surface. And it was important that the truth come out and people be held accountable. That's something that is part of Hawaiian culture. We have a word that many people have heard called Pono and process called Pono Pono, which is to make Pono. Pono is a word that means righteous. It's more than just correct. It's being right in a way that is accountable to all parties. And I think that as Hawaiians, it was important for us in an organization like Oha to become Pono. And so this was a win for us culturally. But something I think that we share with all other cultures, people are best served when things are done the right way. And that's something that I just learned in my Ohana growing up, my family. The place where most people learn their values. I'm so grateful to my family and to my church for the opportunities I learned as a little child, that there's a right way and a wrong way to stand up for what is right. And so part of my decision, you asked me about this earlier, why I decided to go into the office of Hawaiian Affairs. Part of my decision was to serve my people, the Hawaiian people, but it was also to do what was right. Because I think that here in our society, we need to have the courage to stand up for what is right. And I'm grateful for the opportunity I've had to do that at the office of Hawaiian Affairs. Yeah, I agree. I think it's easy as an individual to do the right thing, but there is sometimes a lot of resistance, but it takes courage, right? It might be easy, but it does take courage to fight the forces. But how many trustees are at Oha? Can you tell me a little bit about that? Yes, the office of Hawaiian Affairs is governed by a board of nine trustees. Four of them represent islands, or about half of them represent specific islands, the Big Island, Maui, Molokai, and Kauai, while four of them at least are like myself at large, representing the entire state. Got it. Okay. Now, if there's nine of you, because I'm thinking for us in the chamber when we have our committee meetings or board of director meetings, right? We don't have as many, but do the trustees, do they all need a degree in decision making? Is it challenging for you to be one of nine? How does that, what does it take us behind the scenes there and what that looks like? Sure. Well, decisions are made by majority vote. Okay. So you always need to have a majority in order to pass a resolution or an action item, a decision. It's very much like the legislature and the way decisions are made and policies are adopted and money is spent. I have to say that on almost everything, we agree, because most of the work is very obvious in terms of what the right thing to do is. Right. For example, last year we provided over $10 million in grants and in aid to non-profit organizations. This year, it's going to be closer to $20 million. And much of that is going to go to help with the Lahaina relief from the terrible wildfires that took place. I'm very proud of that work that the trustees carry out. And almost unanimously, the funds are vetted and awarded to worthy groups that are helping people across the state. But sometimes there are differences. And so the board is a board of elected political office holders. So there's politics that comes into it. So let me put it this way, it can be very vigorous at times. And there are times, yes, when I have been the one vote standing against eight other votes. And let me clarify, this doesn't mean that they were all wrong. It doesn't mean that I'm the right guy and they were not right. We are differences. We stood up for what we each believe. And from time to time I have found myself being the lone voter for a certain position. I'm glad that we live in the United States of America where that's possible. Where it's possible to be able to voice your opinion, your conviction and rally others around it. So that's what takes place. So in answer to your question, most of the time we're in full agreement at the board trustees. But sometimes we have some vigorous political battles in order to fight for things that we believe in deeply. Right. But yeah, I agree with you. If everyone is of like mind to do the right thing, I could see that that most people would be would be in agreement. But so you've served so far two terms. So let me ask you this. Are there any goals that you've been working on during your terms? Oh, yes. When I went into the Office of Hawaiian Affairs in 2016, I had three goals. And I've been working on them ever since. The Office of Hawaiian Affairs essentially is a massive trust fund. It governs funds that have come to the Hawaiian people and disperses them so that they can be used for worthy purposes that empower Hawaiians and the community at large to some extent as well. Now, this trust has to be protected. So my first goal is to protect the trust and protecting the trust means making sure that we deal with fraud, waste and abuse, get rid of it as much as we possibly can and to ensure that that doesn't happen again in the future. I'm very pleased because I think we've made major strides in that direction. And that was my first goal. My second goal in the Office of Hawaiian Affairs was to grow the trust. And that is not just to leave the assets at the level they're at now, but to increase the size of the assets, everything from stock portfolio to land holdings, not just to hoard money, but rather for the purpose of creating a base of wealth that can actually be used to make a huge difference here in the islands. So protect the trust, grow the trust. And the third area is to use the trust. And I'm very pleased to see that we are doing that in new ways. As I mentioned, last year we gave about 10 million dollars in grants to nonprofits. This year we're almost doubling that. And in the future, I'm excited that I believe we're going to actually be giving more to nonprofits that are on the ground working with people in everything from programs for nurturing children to taking care of the elderly, from the keiki to the kupuna. And so these have been my goals pretty much lined up with these three premises to protect, to grow, and to use. That's fantastic. And for us, like with the Hispanic Chamber, we feel so blessed and fortunate to have connected with the Office of Hawaiian Affairs. I mean, we're actively working. I mentioned a little bit earlier. We had our first Latin business expo and our idea is to kind of promote businesses and education within the minority community. And so for us to partner and with OHA is very exciting because here's another group that's advocating for people, right? Trying to represent and protect natives, right? And we are natives as well. So we have some events coming up and we're working with OHA and hopefully in the future again, do some cultural events with OHA. So we're really excited about that. And for you, as you work towards the advancements of the conditions of Native Hawaiians, are there any concerns that you have regarding residents of our state who are not native Hawaiian? Well, that's a great question, Marisol. Most definitely I am concerned about Native Hawaiians because that's my fiduciary duty as a trustee of the funds that were left to help Hawaiians improve their conditions. So that's where the most of the work goes on in the Office of Hawaiian Affairs. But at the same time, we don't live in isolation. We live in the midst of a community. I love a phrase that is often used at the beginning of public gatherings in Hawaii. A pule kako. A pule kako. Let us pray together. But I also like to say a hana kako. A hana kako. Let us work together. You see, apart from each other, there's nothing we can do. But when we come together, we can do great things. And one of the things that I'm most saddened by is when different ethnic groups and different cultural groups have differences that push them apart rather than bring it together. You know, our differences should define us, not divide us. That's most important. And so I'm very saddened that sometimes in promoting the betterment of Hawaiians or in seeking to resolve some of the wrongs of the past, some people have become somewhat embittered and that puts a distance between them and non-Hawaiians. We have to work together to share the aloha spirit with everyone because what's good for Hawaiians is good for everyone. But it's also true that what's good for everyone is good for Hawaiians. And so what that means is that we need to work together in order to solve the problems here in our islands and in order to make Hawaii a better place for everyone. I love that message. And then we're nearing our end here. Let me ask you a question because I mean, if our viewers or anything like me, they might not entirely know. As far as positions for the trustees, you were saying people vote for the aloha trustees, but can anybody vote? Is it only, I know it's the Office of Hawaiian Affairs, is it, you know, I mean, can anybody vote for aloha trustees? I'm glad you asked that question. Whenever we go to the polls to vote for governor or for senators or for legislators representatives, we also vote for aloha trustees. Everyone vote because the aloha trustee position is a government position. And that means that everyone has a stake in helping to be able to ensure that aloha and its resources are led by people who are able to help and who are committed to the highest standards. I love that. So we're actually nearing our very end and you taught me something today and I hope I don't butcher it, but please correct me. Ehana Kako, let's work together. Ehana Kako, let's work together. The Hispanic Chamber of Commerce is so excited to work with aloha in the future. We're so glad that we had you here today and sharing your knowledge and what you're doing. We'd love to have you on again. I know we focused on aloha today, but if you would be gracious enough to attend another one of our podcast, we'd love for you to talk more about the Grass Roots Institute. We didn't have time to get into that, but I know that's another organization that you're strongly involved in. So if you'd like to come back and focus on that, we'd love to have you. My name is Marisol Ruiz. I am the vice president and founder of the Hispanic Chamber of Commerce. Thank you, Think Tech Hawaii for giving us the space and the platform to promote members of our community. Thank you so much and have a wonderful day. Thank you so much.


Well recently I posted on Instagram about throwing away your cast iron skillet and boy Did I stir up a hornet's nest we got so much Feedback that I wanted to spend an entire episode talking about iron and the good things about iron and the bad things about iron So here's why I came to tell you to throw away your cast iron skillet First of all, I have no skin in the game. I got no dog in the fight I spent a lot of my time in the south where I loved my cast iron skillet use my cast iron skillet This was before I began measuring serum iron in my patients and myself I follow people's serum iron on an every three months basis now Why do I do that? because there is substantial evidence Some of which I've credited in my book the plant paradox more recently a study That I credited in unlocking the keto code that iron ages us iron Rust us and believe me. We do not want to rust Strikingly studies have been done of people who routinely donate iron through blood donation versus people who don't donate blood and Low and behold people who routinely donate blood live about seven years Longer than people who don't donate blood and a more recent study shows that the people who donate the most regularly Actually live longer than people who donate less frequently. So why is that? one more thing Women in general live about seven years longer than men Why is that? because ladies for a good half of your lives once a month you donate blood and Men no longer wrestle saber-tooth tigers. So we don't donate blood on a routine basis And that actually explains one of the interesting longevity disparities between men and women now sadly we Really really really want to absorb iron and make no mistake iron is critical to make hemoglobin The molecule that carries oxygen around our blood make no mistake We have to have iron and because we have to have iron We avidly absorb it through our intestines for one actually really good reason because up until modern times most of us had worms and parasites in our intestines that actually Sucked iron out of us actually caused micro bleeding In fact when I was in medical school in Georgia in the 1970s We would make rounds and pull we all had little tweezers and we would pull hookworms out of the noses of kids because hookworms were endemic in our population in the south Even in the 1970s and we saw a lot of anemic kids and adults with hookworms or other parasites Nowadays we really don't see that very much. So we were designed to absorb iron as much as we could now Because all of these studies show that iron is something that ages us I started looking at my patient with iron overload and a number of patients actually have a gene That makes them really absorb iron and those of you who have Hemosiderosis know that the treatment for you is actually Frequent blood donations to get rid of all that iron that you aggressively absorb And I take care of a lot of patients with hemociderosis. Okay, so that's why we absorb iron so aggressively That's why in my patient population. We ask People with high iron levels first of all to stop using their cast iron skillet if they're using it and one of the things that's surprising is a great number of my patients men and women who have high iron are Using cast iron skillets and I found that if we could get them to give up their cast iron skillet Then recheck their serum iron level three months later It was normal and in fact one of my patients wrote in the comment section on Instagram that she's one of my patients and we tried twice to Have her get her cast iron skillet back and every time she started using her cast iron skillet up went her iron levels Okay, so that's the point of all this now We also have people that aggressively absorb iron from dark green leafy vegetables And if they're not using a cast iron skillet and they still have high iron levels We ask them to get rid of those dark green leafy vegetables like spinach kale Swiss chard and switch to lighter green and that usually does the job If that doesn't do the job we ask them to start drinking tea Black tea or green tea or English breakfast tea orange pico tea. Just not herbal tea The cool thing about teas is that teas really block the absorption of iron in fact during the day in the office I drink four to five cups of tea every day Just to keep my iron levels low and this will apply to Those of you who are listening who have low iron levels and we'll get into that next What do you do to absorb iron besides use a cast iron skillet besides eat dark green leafy vegetables? Take vitamin C Before a meal or before you're going to have dark green leafy vegetables Vitamin C really helps you absorb iron. So those are the tricks Okay, is it ever okay to use a cast iron skillet? well number one if you have heavy menstrual periods and You menstruate frequently and you have low iron. Yeah, it's a great way to get iron into your system And I have nothing wrong with that if you have low iron I'm all in favor of getting your iron up and you're the person who I want to use a cast iron Skillet, I want you to get dark green leafy vegetables in you every day And I want you to take a vitamin C before every meal and those are great tricks to get your iron levels up Occasionally we have to give you iron pills The problem with iron pills for a lot of particularly women is it makes you nauseated or constipates you So if you're going to take an iron pill make sure you're taking some magnesium along with the iron pill and Make sure you take vitamin C Every time you take the iron pill that'll really help you second if you're anemic make sure your doctor is checking your serum iron level and Make sure that your blood cell indices like MCH and MCV are normal That could indicate that it's not an iron problem at all that you're anemic So this is easy to sort out just make sure your doctor is doing all those steps like I do in all my patients secondly, there are a great number of people who have low iron levels that have Nothing to do with being a menstruating female If you're a man and you have low iron levels Guess what? There's a really good reason for it and they're Worth looking into first of all most adult males with a low iron level We need to be looking for either an ulcer or a polyp or even an intestinal cancer And it's easy to discover just recently. I had one of my patients with a new low iron level And sure enough he had a couple of bleeding polyps in his intestines Luckily, they weren't cancerous and low and behold when he had his polyps out the next time we saw him His iron level was back to normal. Yay Now older adults may be anemic But their iron levels are perfectly normal Many older adults suffer from what's called myelodysplasia which is Simplistically a pre-blood cancer and most of the time we don't have to do anything about that So the point of all this is Just because you're anemic doesn't mean you have low iron Same thing if you have too much iron make sure That you are taking the steps that I asked you to do Okay, one more thing that I see primarily in my vegans and vegetarians who are anemic Many of you are women You have a very high phytate diet What the heck is a phytate? phytates are part of the plant defense system to prevent being eaten by you and To prevent their babies their seeds from being eaten by you My interest in the plant defense system centers around lectins Which is are designed to convince you that it'd be a bad idea to eat Certain plants and certain plant babies, but phytates are part of that defense system as well and phytates Block the absorption of minerals and iron just happens to be one of the minerals that phytates block So phytates are actually present in grains. They're heavily concentrated in grains. And so if you're Getting most of your nutrition from grains and beans are loaded with phytates and you have low iron and You're not having heavily menstruing female Please look into lowering the phytate content and you'd be amazed how well that works in my vegetarian and Vegan patients in raising your iron Finally Please please please stay away from calves liver folks in my upcoming book You're gonna find out how dangerous calves liver is for you And I'm sure I'm gonna get a lot of comments about it and trust me. I grew up in Omaha, Nebraska I have nothing against beef Until now and I've written about this extensively and the studies just get Overwhelmingly sad about the dangers of calves liver Here's the solution have all the chicken liver you want Chicken livers are great for you if you have low iron Okay Last but not least just because you're anemic does not mean your iron level is low and Low normal iron levels predict doing very well length wise longevity wise Finally, what about my grandmother? Everybody wrote in who's 92 years old and used your cast iron skillet all over life Great your grandmother may very well have had Other reasons why she was anemic like for instance Parasites most of us up until the current generation had parasites in our intestines and good for her And who knows we probably used cast iron skillets because we did better because the parasites kept sucking iron out of us And just because grandma made it to 92 Using her cast-iron skillet actually exception proves the rule Study after study shows that a low normal iron is beneficial to you long-term More amazing episodes just like this one watch now Aluminum pans unless they are lined with ceramic You should throw them away as well because aluminum in itself is toxic to our brain


It's The Cube. Here is your host, Jeff Crick. Hi, Jeff Crick here with The Cube. We are on the ground in San Mateo, California at the headquarters of Keynote Systems. We're really excited to be joined in this segment by Jennifer Taha, the president and CEO. Jennifer, welcome. Thank you. Thanks for coming. We're really pleased to have you here today. Absolutely. So thanks for having us. Interesting times at Keynote, a company that's been around for a very long time, but recently you came in a little while ago. You went private. Talk about what's changed since you've got here. Fantastic. Well, it is our 20th year at Keynote. This is a business with a long history, serving customers in retail, financial services, telco, travel, media. And we have a long history of providing performance data to help customers understand what their consumers' experience is like. We took the business private in 2013 with Toma Bravo. That's how I came to be the CEO of Keynote. And it's been a phenomenal journey, one where we've really turned our focus and our mind and our innovation towards honing in on helping our customers make that digital experience count, which is increasingly important as our customers see their business models move from traditional brick and mortar to significant online e-commerce practices, and more increasingly to truly mobile or omni-channel experiences. Yeah, I was going to say the importance of the mobile was certainly not the same as 20 years ago. No, every industry we deal with is being disrupted in some way, whether it's just through mobile subscribership and that immediacy, that need to have an amazing experience in just a couple of seconds. The increasing demand of the consumer is really driving a huge impetus and urgency of companies to try and make that experience hunt, whether you're starting on your laptop or you're starting in a store, you move to your iPad at home, you finish it up on your phone later, so that omni-channel experience really matters. And there's a whole ecosystem sitting behind that, whether it's the network, the internet, your connection, how many apps you have open on your phone, your operating system. That quagmire of complexity is really challenging for our customers, right? And these are customers like USA Today and UK News or News UK, sorry, Commonwealth Bank in Australia, here in the US. Think of any retailer you can imagine, they're our customer, right, any travel website. And so when they spend all this time and effort building a great user experience and a great product and a promotion and offer, that audience acquisition, and something doesn't work and you start to see that spinner or that lockup, right? The customer blames that brand, if you're Chanel and you spend 110 years building a brand only to have it fizzle on a little tiny 2x4 screen, right, that's a real challenge. So we're really trying to put performance in the context of business outcomes, like customer engagement, like brand engagement, conversion, subscription, transaction volume. And that's what we've spent the last couple of years building towards. And last week, we launched a major innovation in that area, which we're really excited about. Go ahead, tell us what's the major innovation? Digital performance intelligence. So it gives you the ability to understand how you're competing against other competitors. So for instance, if you're William Sonoma, it's not just good enough to have a good Black Friday. You've got to know how you're doing relative to Creighton Barrel and Macy's and the other players in your space. And because we come at this from outside the firewall with synthetic and real user monitoring, we see it the way your customers see it, right? It's also understanding how third party attributes in your mobile website or your mobile app or your website behave. And if they don't respond well, and for instance, your payment gateway fails right when you're about to buy something right before you leave to go on vacation, you're never coming back to that sale. So every penny that that company spent getting you to their website or their mobile app, getting you to buy, when you didn't hit submit or it failed, by that you're not coming back. 90% of consumers won't. What's so different is 20 years ago, I as the owner of the website controlled the end-to-end experience. Now I'm relying on the mobile carrier that is not mine, the phone that's not mine, and then increasingly in this kind of API economy that we live in today, like you said, all the third party components that I'm leveraging. So how are people dealing with that? Do you help them across the board? I mean, that's really a challenge when I don't control it all. That's what's really exciting about our new product. Information is power. So if you can measure and monitor and over time optimize the performance of those third-party call-outs, those third-party APIs, you can increase the chances that your customer will have a great experience. If you're not measuring them, you're not managing them from an SLA perspective, it's really hard to have a dialogue. Equally we see companies changing dramatically in terms of how mobile apps are built and how the omnichannel environment is managed. So there used to be a distinct IT organization and a very distinct marketing organization. Now we're starting to see this being very fungible and the rise of the chief digital officer who's often technically capable but owns the entire product experience. And sometimes you see even mobile development reporting into the person who has the fastest growing but potentially smallest revenue stream, that of mobile commerce. That changes a lot of dynamics in the organization but makes it even more important that technical people can have a dialogue with business people that makes sense. And if you're talking about HTML5 and responsive and code instead of talking about conversion and time on site and bounce rate and account signups and subscriptions and impressions, people start falling asleep on both sides of that conversation. So part of our digital performance intelligence solution and also just a culture of really helping to drive that amazing customer experience is also about enabling that dialogue between the technical people in an organization and the marketing and sales and line of business people in an organization. Talk a little about the impact of big data, again, that's changed a lot in 20 years where now people are not only relying on the data that they get within their own systems but now there's a huge amount of data that's outside of their systems that they can pull and leverage. How are you seeing that impact which you guys are delivering and your customers delivering better value? People like to talk about big data but it's the small data, it's the information that matters that's important and it's about distilling all those different sources of information into a meaningful context that drives better decision making. So for instance, we track performance on all different parts of a digital transaction or digital journey. Which ones actually make the biggest difference? Where does revenue leak? Where do customers leak? Where are competitors winning or losing? Oh, I spent half my life in the last few years talking about the revenue gap. All this money, I was a CMO before I was a CEO and I deeply understand and have empathy for all the investment and the justification of that investment that goes into creating this brilliant consumer experience so that you drive revenue growth, top line growth and ultimately profitability. What if you're putting all that to work and you're failing in the last mile? And that's the performance and context piece of it. But understanding where you're failing and where your biggest issues are is what really matters. And in the future, thanks to big data and mobility and analytics capability and just the fact that everybody has a processor in their pocket, the democratization of computing and development, et cetera, in the future, I think our customers will be able to not only create a great experience but put high propensity, high value buyers in a VIP swim lane for their online or their mobile experience in real time. If you think about where micro segmentation is going, where our ability to drive performance data in real time is going and bringing the context of those things together, imagine the experience that you could have as a VIP customer in a virtual environment. I also think that the physical world is going to become increasingly more important. As things go virtual, there needs to be a physical closed loop and we see that in wearables, for instance. We see a lot of people think that some of the wearables products coming out into the market are gadgets, but it's just the tip of the iceberg. We don't even know how wearables are going to be used, but as people try and bring those online and they start dealing with sensor data to manage their health and wellness, to find their keys, whatever the case may be, they're going to need to, one, make sure all those things are performant, highly performant, to make sure that consumer can manage all that data and use it in a meaningful way, right? And three, make sure that they're actually creating long-term value because consumers are showing a propensity to quit really fast right now. I mean, really, we could go on forever and ever. I want to shift gears. But the consumerization of IT and the expected behavior of applications based on my experience with Amazon and Yahoo and Google really has changed what enterprise customers now expect with their interaction with those customers. But let's shift gears. Talk a little bit about you. It's great to look in your LinkedIn profile and see you came from Procter and Gamble, which is music to my ears because I used to always say there's just not enough good product management in Silicon Valley. You should just get the pipeline going and just ship all these well-trained right through Cincinnati. They had a little house shock when they got out of here, but that's okay. So talk a little bit about that experience because it's a big company. They've been doing it for a long time. How that impacted you and how you've been able to apply those principles out here in Silicon Valley. You just heard a bell ring and people clapping. That's an old Procter and Gamble trick. That means we sold something. But I learned that at P&G about 100 years ago. No, I feel like I grew up a Procter and Gamble. It is a place that invests in developing not only professional management and professional product leaders, but professional leaders. Right? And you really learn that the only way to be successful in a crowded, noisy market is to really find that ownable territory that sort of uniques two bell rings. The knife. It's the last day of the quarter. I can't even believe I'm spending this time with you. Exactly. Oh, it still matters. We still keep count. Absolutely. We're still very performance oriented around here. But so finding that ownable territory, finding that kind of unique fit with a consumer's need, or more importantly, an unarticulated need, and then envisioning and engaging and bringing people along on the journey to make that happen, that's one of the things I learned at P&G. The first product I sold at P&G was Sunny Delight. I'm a long way from dropping pallets of Sunny Delight into inner city Cleveland grocery stores. That experience are toughening you up though. Hey, scraping bubblegum off the shelves is good for you. Some of the principles that you learn in doing that, like where is the connection between what a customer thinks they need, might not realize they need, and what you have, and how do you bring those things together? Right? And also getting people engaged in that process. When I got here and said, you know this testing and monitoring thing, it's interesting, but we really need to care about is driving that user experience and putting performance in the context of how the line of business thinks about the currency trade that they've got with a customer, whether you're trading data, or their subscription, or you're using their time, or their eyeballs, or ultimately dollars and cents, it is that instant where you're going to succeed or fail. And I learned at P&G that all of the preparation, everything you do to build a brilliant business, a brilliant campaign, a brilliant relationship with the Walmarts and the targets of the world, it comes down to that second. Where the currency changes. Yeah, when they're standing in the aisle and they're deciding sunny delight or a 7-up, I guess, right? Yeah, so I'm ever so grateful for what I learned at P&G and I'm always rooting for them. I'm still a shareholder. I've never sold a dollar of P&G stock since I was an employee there. And I look back on that experience all the time because I find that fundamentals are the way I manage, the way I lead come from Cincinnati, Ohio. Absolutely. I'm going back there for a Reds game for my grandfather's birthday, but that's another story. So I just want to wrap it up, really talk about how you use that in your leadership. I think we were talking before we went on camera about this currency, and one of the few people I've ever really heard really explicitly talk about the currency of time, currency of attention, currency of money, and increasingly the currency of my data in exchange for services. Talk a little bit about what it meant to come here, 20-year-old company, public, not necessarily probably in the best of time since that IPO in 2000, which is kind of tough, and what you've tried to do here from a leadership perspective to bring that energy not only to the customers that you serve, but the employees that you have here in the building. Yeah, that's a great question. In my first year with the business, I saw over 100 customers myself, which meant that I was on a plane a lot. And I listened. I did a lot of listening. And one of the questions I asked those customers is, why have you been buying our product? 20 years, why do you keep coming back? What is it that we do well? I wanted to really understand what the cornerstone of our business was in the customer's eyes so that I could come back to the employee community and talk about what we do well, but importantly where we could go if we focused our efforts. And being private enables you to do that. I haven't wasted a second on quarterly earnings and SEC filings and a lot of the governance-oriented activity that you're responsible for, never mind the scrutiny that comes from the public market. We do have a highly focused owner. And there's a lot of performance management and reporting and visibility associated with how the business is going. So we still drive hard for results. But we have a lot more flexibility to make strategic shifts, to redeploy investment, to think differently about the market. None of that matters if you can't get your employee community and your partner community to come along with you for the ride. And they did look at me like I was crazy for about the first 90 days. Some of them still look at me like I'm crazy. But I think that making an employee person feel like they're trade-off, the time they're not spending with their spouse, their family, their parents, the things that they like to do, their travel, et cetera, making that trade-off hunt in the workplace and making them feel like they're part of something that's bigger and maybe achieving something they didn't think was possible is a really important part of leadership. And being honest, talking transparently when things are not going well, but also celebrating the success when things go well. We bring that bell at every opera, whether it's a $200 dollar deal or a $2 million dollar deal, that bell gets rung. We had to reinforce the screws. That's good, that's a good thing. You can come off the wall. All right, so last question before we wrap.  This is part of a woman in tech feature. And you have a daughter, I think we talked about before offline. What advice, and again, we offer this up to everyone that we have on, would you give to families, young women, not young girls, older girls, who potentially want to pursue a career in tech? You got here via Sunny Delight, that's a great story. Now you're running the show. But what advice would you give for families and young women that want to pursue a career in technology? Build self-confidence and self-belief. I was very lucky I had an immigrant father who raised me to believe that I could be anything, do anything, try anything and succeed in anything as long as I was willing to put in the time, be focused, surround myself by smart people and not be afraid to ask stupid questions. So that confidence and that will to try things to have a go and not be worried that you might not get it right or you might not get into the best middle school or whatever the case may be, I think is really important. The second thing is that you don't have to be technical to be a leader in tech. I have a liberal arts degree from the University of Michigan. What was your major? Organizational behavior and business management, right? And I'm really good at organizing things. But you have to, you have to find, sometimes you have to find the opportunities yourself and you have to have that ability to sort of step out and go it alone because I haven't had a massive peer group of women in my career. I've always worked in industries where I was one of the few women at the table, not at a table full of women. I never felt like that was a problem if anything I felt like it made me stand out. It makes you stand out if you fail. It makes you stand out if you succeed. And I have a good sense of humor. So I would teach your kids not to take themselves too seriously, right? Have a laugh. We don't do stuff around here if it's not fun. We spend a lot of time, so last time I'm looking at our head of communications, we spend a lot of time talking about making it fun, making the trade off worthwhile, right? And I think you need to raise your kids to be fearless about trying different things about surrounding themselves with smart people and about making sure that when they do come into work every day or when they go to school every day that they're enjoying it, that they're doing everything that they can to make it interesting, to make it entertaining for themselves and for the people around them. Great, great tip. Thank you, Jennifer, so much. Jennifer, at the heart of the president and CEO of Keynote Systems for spending a few time on an exciting quarter because the bell just keeps ringing. Absolutely, thank you very much for coming, Jeff, appreciate it. I'm Jeff Rick, you're watching theCUBE. We'll see you next time.


all right we've heard from flat flash let's bring in Michael K and Paul O'Neill now and guys want to get your take on this game flash kind of talked us through it in those those bookends a chance late for the Yankees but you'd would have hoped they had had a leader a better chance had it not been for a lot of pitchers having to be used I don't understand what they did I think it was the case of them saying look how smart we are we're smarter than everybody else you do that with an opener when you don't have a pitcher and they have plenty of pitchers they could have gone with Tanaka they could have gone with half the start the game where he's comfortable they could have gone with so many others that they could have gone with Davey Garcia and said they send Garcia out for one and he gives up the home run and then they take him out so if they thought that the grand plan was to bring in the left hander half and that cash would overreact and then take out all of his left handers he didn't do that so flipping the lineup didn't work I think that was the plan I thought for sure they try to get at least three innings maybe four innings out of a out of Davey Garcia it makes no sense to me this is one of those game changing series changing decisions and because of perhaps ineffectiveness in a role that he's not used to remember he has not pitched in about eight days he's much better when he's on normal rest this was a case of the Yankee saying look how smart we are and it blew up in their face. Paul I want to get to Michael's point Jay Hap obviously gave up a couple of two run home runs he did not get the job done but he's a veteran starting pitcher who they put out in the bullpen he could not go through his routine didn't do a good job against the lefties in the lineup and gave up a couple of home runs tough spot for him. You had no doubt about it I just feel the the same way I'm from from the old school you have a starting pitcher you start him and Jay Hap was not the same pitcher today his two seamer he he became a change-up pitcher it was change-up after change-up I just never got comfortable he got out of one jam but you know what seven runs is too much for this pitching staff to give up to the Tampa Bay Rays they're not a great offensive team you know the the Yankees kind of knocked at the door a few times but you know sometimes you're going to have to win games with four or five runs you ask five


Hello, good afternoon. Does anyone actually get the reference to my title? Like, I see, like, too much of it. Yes, so it's a reference to a really awesome book, Thinking Fast and Slow. I highly recommend it. No relation to this actual, like, talk, though. So yes, my name is Lynn Root. I am a site reliability engineer at Spotify. I also do a lot of open source evangelism internally. And you might know me from Pie Ladies as well. Also, unfortunately, I'm going to take up, like, the whole time. So if you have questions or want to chat, you can come join me for a convenient coffee break right after this. Okay, another quick question. Has anyone read the Site Reliability Engineering book, aka the Google SRE book? I think I see a few hands. All right, well, I highly recommend that book, but the TLDR of, like, every chapter seems to be used distributed tracing. So with the prevalence of microservices, where you may or may not own all the services that a request might flow through, it's certainly imperative to understand where your code fits into the grand scheme of things and how everything operates with each other. So there's three main needs to trace a system, performance debugging, capacity planning, and problem diagnosis, although it can help address many other issues as well. So while this talk will have, like, a slight focus towards performance debugging, these techniques can certainly be applicable to other needs. So I have a bit of a jam-packed day-to-day. I'll start off with an overview of what tracing is and the problems we can try to diagnose with it. And I'll also talk about some general types of tracing we can use and what key things to think about when scaling up to larger distributed systems. And then the inspiration for this talk stemmed from me trying to improve the performance of one of my own team's services, which sort of implies we don't really trace that Spotify. So I'll be running through some questions to ask and approaches to take when diagnosing and fixing your services bottleneck. And finally, I'll wrap up with some tracing solutions for profiling performance. And as I mentioned before, I won't have time for questions, so you can catch me right out there. All right, in the simplest terms, a trace follows the complete workflow from the start of a transaction or a request to its end, including the components that it flows through. So for a very simple web application, it's pretty easy to understand the workflow of a request, but then add some databases, separate the front end from the back end, maybe throw in some caching, have an external API call, all behind a load balancer, then scale up to tens, hundreds, or thousands of times. It gets kind of difficult to put together workflows of requests. So historically, we've been focused on machine-centric metrics, including system-level metrics like CPU, disk space, and memory, as well as app-level metrics like requests per second, response latency, database rights, et cetera. Following and understanding these metrics are quite important, but there's no view into a service's dependencies or its dependence. It's also not possible to get a view of a complete flow of a request, nor develop an understanding of how one's service performs at scale. So a workflow-centric approach allows us to understand relationships of components within an entire system, and then we can follow a request from beginning to end to understand bottlenecks, hone in on the anomalistic paths, and figure out where we need to add more resources. So when looking at a very simplified system where we have a load balancer, front-end, back-end, database, maybe an external dependency to a third-party API, and when we have redundant systems, it gets particularly confusing to follow a request. So how do we debug a program of a rare workflow? How do we know which component of this system is the bottleneck? Which function call is taking the longest? Is there another app on my house causing distortion of machine-centric metrics, or performance metrics, something like the noisy neighbor's problem? So with so many potential paths that a request can take with potential for issues at each and every node and edge, this can be mind-numbingly difficult if we continue to be machine-centric. So end-to-end tracing will allow us to get a bigger picture to address these concerns. And looking at the magnitudes of what we're operating at Spotify, you can see that tracing, if we did it, would help us a lot. So real quickly, there are a few reasons why we trace a system. The one that inspired this talk is performance analysis. This is trying to understand what happens at the 50th or 75th percentile, the steady-state problems. And this will help us identify latencies, resource usages, and other performance issues. We are also able to understand questions like did this particular deploy of the service have an effect on latency of the overall whole system? Tracing can also clue us in on anomalistic request flows, the 99.9 percentile. The issues can still be related to performance or it can help identify problems with correctness like component failures or timeouts. Profiling is very similar to the first, but here we're just interested in particular components or aspects of system. We don't necessarily care about the full workflow here. The fourth one, we can also answer questions of what a particular component depends on and what depends on it, particularly useful for a complex system, complex systems. So when with dependents identified, we can also attribute particularly expensive work like component A, a significant workload with disk rights to component B, which can be helpful when attributing costs to teams and service owners or component owners. Then finally, we're able to create models of our entire systems that allow us to ask what if questions, like what would happen to component A if we did a disaster recovery test on component B? So there are various approaches to tracing. I'll only highlight three of them here. The first is manual, it's also very simplistic, where you are just generating your own trace IDs and adding them to your logs. And there are very simple things that can be added to your web service here, especially ones that do not have dependent or depending components that you don't have access to. You won't get any pretty visualizations or help with centralized collection beyond what you typically have with your logs, but it still can provide insight. So this is a flask example, super simple, using a decorator. Here you can simply add a UUID to each request received as a header, then log at particular points of interest, like at the beginning and end of a request, and then any other in-between components or function calls where you want to propagate headers. And this is exactly what I ended up doing for my service, which may be wished for a better way, hence this talk. I must admit I do a lot of conference-driven development. So if your app is behind Nginx, that you're able to manipulate, you can also turn on its ability to stamp each request with an X request ID header, as you see here with the add header and proxy set header. You can also add a very simple, like you can simply add the request ID to Nginx's logs as well. Next up is black box tracing. This is tracing with no implementation across the components. It tries to infer the workflows and relationships by correlating variables and timing within already defined log messages. So from here, relationship inference is done via statistical or regressional analysis. And this is easiest with centralized logging, and if there's somewhat of a standardized schema to log messages that contain an ID or a timestamp, it's particularly useful if instrumenting an entire system is too cumbersome, or you can't otherwise instrument components that you don't own. And as such, it's quite portable, and there's very little to know overhead, but it does require a lot of data points in order to correctly infer relationships. It also lacks accuracy with the absence of instrumenting components themselves, as well as the ability to attribute causality with asynchronous behavior and concurrency. Another approach to black box tracing can be through network tapping, using S-Flow or NFDEMP or IP table packet data, which I am sure the NSA is quite familiar with themselves. And then the final type of tracing is through metadata propagation. And this approach was made popular by Google's research paper on Dapper. And so components are instrumented at particular trace points to follow causality between functions, components, and systems, or even with common RPC libraries like GRPC, and that will automatically add metadata to each call. So metadata that is tracked includes a trace ID, which represents one single trace or workflow, and a span ID for each and every point in a particular trace, like a request sent from client, a request received by server, server responds, and then the spans start and end time. So this approach works best when the system itself is designed with tracing in mind, but not many people do that, right? So this avoids guesswork with the inferring causal relationships. However, it can add a bit of overhead to response time and throughput. So the use of sampling traces limits the burden here on the system and the data point storage. Sampling anywhere between 0.01% and 10% of requests is often plenty to get an understanding of a system's performance. So when starting to have many microservices and scaling out with many more resources, there are a few points to keep in mind when instrumenting your system, particularly with the metadata propagation approach. So in terms of what to keep in mind, and I'll go into detail about each in a second, we want to know what relationships to track, essentially how to follow a trace and what is considered part of a workflow, how they are tracked, constructing metadata to track causal relationships is particularly difficult. There are a few approaches each with their own fortes and drawbacks. And then how to reduce overhead of tracking. The approach one chooses in sampling is largely defined by what questions you're trying to answer with your tracing. And then there may be a clear answer, but not without its own penalties. And finally, how to visualize. The visualizations needed will also be informed by what you're trying to answer with tracing. All right, so what to track. When looking within a request, we can take two points of view, either the submitter point of view or the trigger point of view. So the submitter point of view follows or just focuses on one complete request and doesn't take into account if part of that request is caused by another request or action. So for instance, the evicting cache here that was actually triggered by request two is still attributed to request one since its data comes from the first request. The trigger point of view focuses on the trigger that initiates the action. We're in the same example, request two evicts cache from request one and therefore the eviction is included in request twos trace. So choosing which to follow depends on the answers that you're trying to find. For instance, it doesn't really matter which approach is chosen for performance profiling, but following trigger causality will help detect anomalies by showing critical paths. All right, how to track are essentially what is needed in your metadata. This essentially boils down to, it's very difficult to reliably track causal relationships within a distributed system. Now the sheer nature of a distributed system implies issues with ordering events and traces that happen across many hosts and there might not be a global synchronous clock available, so care must be taken when deciding what goes into crafting the metadata and that is threaded through an end-to-end trace. So using a random ID like UUID or the X request ID header will identify causal related activity, but then tracing implementations must use some sort of external clock to collect traces. And then in the absence of a global synchronized clock or to avoid issues like clock skew, looking at network send and receive messages can then be used to construct causal relationships because you can't exactly receive a message before it's sent. And a lot of tracing implementations use this very simplistic approach. However, this approach lacks resiliency. There's a potential for data loss from external systems or inability to add trace points to components that's owned by others. Tracing systems can also add a time stamp derived from a local logical clock to the workflow ID where this isn't exactly the local system's time stamp but either a counter or sort of a randomized time stamp that is paired with a trace message. So with this approach, we don't need the tracing system to spend time on the ordering of traces it collects since it's explicit in the clock data, but parallelization and concurrency can complicate understanding these relationships. And then one can also add the previous trace points that have been already executed within the metadata itself to understand all the forks and joins. It also allows immediate availability of the tracing data itself as soon as the workflow ends because there's no need to spend time on collating or establishing the order of causal relationships. But as you can imagine, metadata will only grow in size as it follows the workflow adding to the payload. So it basically boils down to this. If you really care about payload of requests, then a simple unique ID is your go-to, but at the expense of needing to infer relationships. You can then add a time stamp of sorts to help establish explicit causal relationships, but you're still susceptible to potential ordering issues of traces if data is lost. Now you may add the previously executed trace points to avoid data loss and understand the forks and joins of a trace while gaining immediate availability of trace data since causal relationships are already established, but then you suffer in payload size. And then there's also the fact that there are no open-source tracing system that actually implement this last one. So end-to-end tracing will have an effect on runtime and storage overhead no matter what you choose. For instance, if Google were to trace all web searches despite its intelligent tracing implementation, it would impose a 1.5% throughput penalty and add 16% to the response time. I won't go into very much detail, but there are essentially three basic approaches to sampling. First is head-based, which will make a random sampling decision at the start of a workflow and then we'll follow it all the way through to completion. The next one is tail-based, which will make the sampling decision at the end of the workflow, implying some caching going on here. Tail-based sampling needs to be a little bit more intelligent, but it's particularly useful for tracing anomalistic behavior. And finally, unitary sampling, where the sampling decision is made at the trace point itself and therefore prevents the construction of a full workflow. So head-based is the simplest and probably most ideal for a performance profiling, and both head-based and unitary are most often seen in current tracing implementations. I'm not quite sure if there's a tracing system that actually implements tail-based. All right, what visualizations you choose to look at depends upon what you're trying to figure out. So Gantt charts are popular and definitely quite appealing, but it only shows requests from a single traced, and you definitely have seen this type before if you've looked at the network tab of your browser's dev tools. When trying to get a sense of where the system's bottlenecks are, a request flow graph, AKA a directed ASILCA graph, will show workflows as they are executed, and unlike Gantt charts, can aggregate information of multiple requests of the same workflow. Another useful representation is a calling context tree in order to visualize multiple requests of different workflows, and this reveals both valid and invalid paths that a request can take, best for creating a general understanding of system behavior. So what the takeaway here is, there's a few things we need to consider when we trace a system. You should have an understanding of what you want to do, of what questions you're trying to answer with tracing. And certainly there will be other realizations and questions that come from a traced system. For example, with Dapper, Google is able to audit systems for security, asserting that only authorized components are talking to sensitive services, but not without understanding what you're trying to figure out, you may end up approaching your instrumentation incorrectly. The answer to this question will help identify the approach to causality, whether from the trigger point of view or from submitter point of view. Then another important question, how much time do you want to put into instrumenting your system? Or can you even instrument all parts? This will inform the approach that you use to tracing, be it black box or not. If you can instrument like all the things, or at least some of it, it then becomes a question of what data you should propagate through an entire flow.  Do you want to understand all the requests? Then you should be prepared to take the performance penalty on the service itself, and then you can have fun storing all that data. Or is a percentage of the flow is okay? And then if so, then how do we approach sampling? And that's in your answer of what we want to know question. So for understanding performance, head-based sampling is certainly fine. You also need to think about whether or not you want to capture the full workflow of a request or only focus on a subset of a system. And this will also inform your sampling approach, be it unitary or not. And so in terms of performance and understanding where bottlenecks are, you want to try and preserve the trigger causality rather than submitter, as it shows the critical path to that bottleneck. Head-based sampling is fine, as we don't need intelligent sampling. And even with very low sample rates, we can get a good idea of where our problem lies. Since we essentially care about the 50th or 75th percentile. And then finally, a request flow graph here is ideal since we don't care about anomalistic behavior. We want information of the big picture rather than looking into particular individual workflows. And so most often, once you are tracing a system, the problem will reveal itself as will the solution, but not always. So I do have a few questions to ask yourself when figuring out how to improve a service's performance. First one is, are you making multiple requests to the same service? Round-trip network calls are expensive, and perhaps there's a way to set up batch requests or accept batch requests on your end. Perhaps your service doesn't need to be synchronous or unnecessarily blocks.  Is the same data being repeatedly requested but not cached? Or maybe you were caching too much or maybe not the right data? Is the expiration too high or too low? What about your site's assets? Could they be better ordered to improve loading time? Can you minimize the amount of inline scripts or maybe make your scripts async? Are there a lot of distinct domain lookups that add time with the DNS responses? And how about decreasing the number of actual files referenced, or maybe minifying and compress them? There's a bunch of stuff that can be done with the front-end part. And then finally, perhaps you can use chunked encoding when returning large amounts of data. Are you otherwise able to have your servers produce elements of response as they are needed rather than trying to produce all elements as fast as possible? All right, now probably the most interesting part. So about the current tracing systems that are out there. So there is an open standard for distributed tracing, allowing developers to instrument their code without vendor lock-in. And they do this by standardizing the Trace-Span API. One criticism I have of open tracing is that they don't prescribe a way to implement more intelligent sampling other than a simple percentage and setting priority. There's also a lack of standardization for how to track relationships, whether submitter or trigger. It's pretty much all submitter. And it's mainly just a standardization for managing the span itself. But mind you, it's very young specification that's evolving and developing as we speak. There are a few self-hosted popular solutions that do support the open tracing specification. Probably the most widely used is Zipkin from Twitter, which has implementations in Java, Go, JavaScript, Ruby, and Scala. The architecture setup is basically the instrumented app sends data out of band to remote collector that accepts a few different transport mechanisms, including HTTP, Kafka, and Scribe. So with propagating data from a service, all of the current Python libraries only support HTTP. There's no RPC support. And Zipkin does provide a nice Gantt chart or a waterfall chart of individual traces. And you can see, you can view a tree of dependencies, but it's essentially only a tree with no information, like latencies or status codes or anything else. Using PyZipkin on which other libraries are based, you can define a transport mechanism like I did here with HTTP transport, which can be just simply posting a request with the content of the trace. You can otherwise make one for Kafka or Scribe. But then, otherwise, it's just a simple context manager being placed wherever you want to trace. IGAR is another self-hosted system that supports open tracing specification. It comes from Uber. Rather than the application or client library reporting to a remote collector, it reports to a local agent via UDP who then sends out traces to a collector. Unlike Zipkin, which supports Kafka and Elasticsearch and MySQL, IGAR only supports Cassandra for its storage. The UI is very similar to Zipkin with really pretty waterfall graphs and a dependency tree. But again, nothing to help aggregate that performance information we're interested in. Their documentation is also horribly lacking, unfortunately, but they do have a pretty decent tutorial to walk through. Their client library for Python is a bit cringe-worthy. So this is a trimmed example from their docs, just meaning to give the gist here. Basically, you can initialize a tracer that the open source or that the open tracing Python library will use and create a span, a child span with context managers. But their usage at the end of time.sleep for yielding to IO loop. It's a bit of a head-scratcher. Its docs also make mention of supporting monkey-patching libraries like Requests and Redis and Uralid 2. So all I can say is use at your own risk. After I presented this at PyCon a couple months ago, like the day after they created an issue and basically made a comment in their code, reasoning why, but I still don't get why. So there are a couple others I'm not familiar with, including AppDash and LightStep. And there are a few more that don't have Python client libraries yet. And in case you don't want to host your own system, there are a few services out there to help. There's Stackdriver Trace from Google, not to be confused with Stackdriver Logging. So unfortunately, Google has no Python or GRPC client libraries to instrument your app with, but they do have a REST and RPC interface if you feel so inclined. But they do support a Zipkin Traces where you can set up a Google-flavored Zipkin server either on their infrastructure or on yours and have it forward traces to Stackdriver. And they actually make it pretty easy. I was able to spin up a Docker image and start doing traces within a couple minutes. Annoyingly, they have a storage limitation of 30 days, same with their logging. And my last criticism is their UI. They have simple plots of response time over the past few hours and a list of all traces that are automatically provided in the UI, but you have to manually make analysis reports for each time period that you're interested in to get all the fancy distribution graphs that are not automatically generated, unfortunately. And then finally, Amazon also has a tracing service available called Xray. I only set up their demo app, but it looks like they do not explicitly support Python, only know Java and .NET apps. But the Python SDK botto has support for sending traces to local demon, which then forwards to the Xray service. And what's nice about Xray, despite it being proprietary and not open tracing compliant, is you're able to configure sampling rates for different URL routes of your application based on either a fixed request per second or a percentage of requests. However, it's not possible to configure these rules with bottom. Also, or almost redeemable, is their visualizations. So while there's the typical waterfall chart, they also have a request flow graph where you can see average latencies, captured traces per minute and requests broken down by response status. So basically, AWS and Xray seems pretty cool and probably the most useful out of all of these, but it'll take some time instrumenting your app and introduces vendor lock-in. And some honorable mentions that do app performance measurement. I don't have personal experience with these, but Datadog and New Relic might be of interest to some of you. All right, and a quick opinionated wrap-up. Got like a minute here. If you run microservices, you should be tracing them. Otherwise, it's very difficult to understand an entire system's performance, anomalous behavior, resource usage, among other many aspects. However, good luck. Whether you choose a self-hosted solution or a provided service, documentation is all around locking. Granted, very young space, very much growing as open tracing standard is developing. And as I mentioned, language support isn't 100% even if it might not even be there. And there's a lack of configuration for relationship tracking or intelligent sampling and available visualizations. But it is indeed an open spec that can be influenced or you might feel so inclined to implement your own, to which I say good luck. And then finally, all of this and some pretty graphs and stuff is up on my blog post if you're interested. Thank you.


What's up guys Michael here with jaspies, and this is our last filler for the quick hit baseball mixer 4 box break, but this is a 2022 top scrim blaster 1 box break And at the end of this there will be seven spots given away into the main break for this filler No vet commons are gonna ship But yeah, so we've got all the names Thank you to everyone who got a spot made this filler happen got all the teams in the list So just gonna roll a dice one in a one Snake eyes only two times for the randomizer one and two Two times, and we've got Robert down to John present my list here All right two times again for the teams one two Two times there, and we've got the Mets down to the Jays All right, so we got Robert with the Mets John with the Rockies James with the Yankees Jose with the Pirates Jeremy with the twins John with the A's Michael with the D-backs Brian with the Mariners Jeremy with the Dodgers Michael with the Padres John with the Giants Eric with the Rays John with the Royals Jeremy with the Cubs John with the Orioles Simon with the Angels James with the Guardians John with the Astros Tristan with the Red Sox Jeremy with the White Sox Kevin with the Phillies Robert with the Reds John with the Cardinals and Cardinals Nationals and Brewers Jared with the Rangers Brian with the Marlins Craig with the Tigers Robert last spot mojo has the Braves And John has the Blue Jays So just alphabetize this here Print will be good to rip switch that there The official list is in to be quick just a little blaster box here See who gets in a little quick quick hit mixer With Shohei rookie year diamond icon. Yeah, no Beckham in full ship Such a mones refractor Blake Snell backwards Just pink Troll Del Bruhan Randy Gouriel Jr. On the sepia Rendon refractor Zack short hyper refractor. No, that's a prism refractor Tristan Mackenzie Pink cruise ready Robert and Blake Snell sepia All right, so that does it for the pack That's not why you guys are here You guys are here to see who gets in To the main break So we'll take our names again pop them into a new list And we're gonna roll on your dice three into three hard six six times top seven People get into the mixer. So let's do it one two three three four five six and seven Seven times there. Thank you to everyone who got in to this filler made this happen Sorry to the people who unfortunately didn't get in Jeremy just missing it That's spot eight, but let's see who got in John with two spots there James Getting in Robert with his last spot mojo snuck in there Jared John again and Jeremy Who all right, so you just missed there, but you still got in so these seven names here are Getting into the mixer Yeah, but that does it. Oh no Don't tell me I forgot to do it. Okay, we're gonna have to you can check the timestamps here You can check the timestamps sorry You can check the timestamps on the randomizers the dates Seven times there and these were the top seven Yeah, James with the Y You got your spot there. Oh Crap Yeah, I didn't switch it but As you can see 103 21 on that one and 103 40 Here's the date. All right Cool. Anyway That's gonna do it. Sorry about that I I know I forgot to switch it You know sometimes it happens anyway. Thank you guys everyone who got into this filler next up will be the four box quick hit mixer Yeah, and I'll see you guys very shortly. Bye guys


Hi there, this is Tracy Tokahama again. Thank you very much for the opportunity to try to answer these great questions I'm going to do so in an abbreviated format here. If you need a more elaborate explanation, please don't hesitate to write me So the first question we teachers want and need to know more about the neuroscience of teaching and learning However, the teacher training programs available to us do not have much neuroscience. What can we do about that? This person really hit the nail on the head This is a problem around the world really teacher training programs Very few of them have kind of got up to speed here to include more of the brain in Teacher training programs. So what can be done at this stage? You can try to self-educate I would highly recommend that you have a look at these web pages from the Welcome Trust the Ennenberg Foundation the Dana Foundation The Society for Neuroscience They have a lot of wonderful short courses video classes posted on their website as well as links to High quality information in both video as well as article formats You can also I invite you to do this go to my web page and and watch the videos That summarize making classrooms better, which is the integration of neuroscience into a regular classroom practice There's 50 different things that are supported by neuroscience and are proven to make a difference in student learning outcomes here You can also have a look at other freely available Resources such as the mind-brain and education journal which can be downloaded from directly from the International Mind-Brain Education Society webpage and you can also continue to do what you're doing right now participate actively in society conferences Raise your voices and discuss this in universities or teacher training programs that you're familiar with and call attention to the fact that This is a big gap in teacher knowledge right now as I mentioned to you before the OECD countries have a special working group They create publications that are meant to educate and influence policy decisions by government So hopefully this will catch on sooner than later with enough individuals working on this in their own spheres of influence This can eventually become the norm Second question what changes have you personally made in your teaching style as a result of brain science? I've done a lot of thinking about my own personal practice. So thank you for this question I become a lot more conscious of the idea that every moment is really a teaching moment And this has a lot to do with a better understanding of social contagion and how Individuals influence each other's emotional states which in turn influences what level of cognition they can function at Also realize the amazing power of feedback and again This is also attached to the idea of Affect and learning how a person feels about their environment whether they feel safe or unsafe Really makes a big difference in whether or not they're open to new learning moments I also presume that most people do want to learn even those kids in our class that sometimes look like they're putting up a They're resisting this. This is you know, your natural brain state is to learn, right? But many times what might look like resistance is actually just being unaware of How or where to begin their learning venture and I think that's really part of my job is to help students find The importance of each lesson in their own personal lives. I also realize the impact of extrinsic factors on my own teaching It's not always just you know, what what's about me and the student their genetics come into play their prior experiences Their eating habits sleeping habits is a lot of other things that can influence student learning outcomes And so I've kind of learned to depersonalize some of these elements I realize that sometimes it is not just all about me who are about my Exchange with the students, but it can be that other things aren't influencing the outcomes of that students performance in my class. I Believe that everyone can and will learn and that's my job. I have to figure out as a teacher You know, how can I make that happen? How can I identify the starting point of my students? So I have to believe that all students, you know, their brains are plastic They are able to take a new learning sometimes it'll take a little bit longer with some students than others however, and I also have learned and realized this is really kind of fascinates this mix of Neuroscience and psychology, but students live up to the expectations of the teacher and this is high or low Which is fascinating to me So there's a lot of psychology and this but it influences student learning outcomes to a great extent And I think that holding students to high expectations believing that all students can learn but also just you know raising the bar a bit and saying You know what? I know you are capable of more. I'd love to hear your voice more in the class I'd love to see more participation. I'd love to read more examples of what you're talking about The more you encourage those students the more you tell them I believe that you're gonna be able to do this You're gonna see better results. So I think that that's another thing that is very important And it's a huge play in this whole balance between mind and brain What's going on in there when you do that finally? I also believe that it's my job, you know, not just to teach a Subject area. It's not to teach a class on whatever the neuroscience of learning My job is to help the students learn how to think it's not necessarily what to think right because we now know that in Google Just about you know any date fact figure concept theory And that is not what I'm here for the students can find that on their own What they need me for is to sort of challenge them to up their own game and to sort of think about how they actually Think about things and so that they have long-term skill sets that can go way beyond my own personal class Finally, do you think streaming or grouping students into classes by ability is a good practice? Are there any possible negative effects this can produce? This is a great question The short answer is absolutely no I would not agree to grouping students or putting the high students or the low students in separate groups But again, you know the the longer answer is in general in education the true answer is almost always it depends You know, it depends on your goal if your goal is To speed up your content knowledge acquisition and to ease the load on teachers It's great to put students who are at a similar level altogether. That just really makes the process just easier, right? Faster, however, is that all there is to it if your goal is to improve, you know Life-long learning skills cooperative learning or communication then this is really a bad idea The world is not made up of people who are, you know, very homogeneous in nature in general So learning to work with people who have different skill sets is very important In addition to this though, you have to remember that, you know If I were to ask you who learns more when you have mixed ability students in a single class The one who already knows a lot or the one who doesn't know very much If I put those two students together the one who knows a lot and the one who doesn't know very much Who's gonna learn more out of that encounter? Well, they're both gonna learn, right? But actually the one who knew more to begin with actually learns even more about the subject because one of the best ways to learn is to teach So creating exchanges in classes where students do peer teaching is really powerful, especially in groups with mixed ability That's actually something really positive So finally how can technological pedagogical knowledge be combined with knowledge of individual student learning? If a teacher is just getting to know the students or simply put how can teachers use technology to develop knowledge of how students learn best? First of all, let's just remember technology is just a tool. Okay. It's another tool It's all on humans really to develop those Relationships between individuals especially to student teacher relationships, right? How can then technology be leveraged in the best way to facilitate getting to know students better so that you can respond to their Needs better. Well, one thing you might consider doing is employing or taking advantage of the online Disinhibition effect that Seller coined in 2004. This idea is that you may be in a face-to-face context Students might be hesitant to share things But when you are online you feel disinhibited you feel much more free to share information Basically because you feel there's this anonymity about your exchange So posing poignant questions to the group. What is one thing that you're really afraid of or what is one thing that you're particularly proud of? Just asking those generic questions in a discussion structure can oftentimes bring out a lot of new information about an individual student That they might not be more forthcoming with a face-to-face class You can also use traditional tools for example journals or reflection exercises or discussions do it within the virtual platform If you have online journals for example, you can ask students You know once a day or once a week to simply jot down a few ideas based on a key prompt and Using that respond to the students that shows a high degree of your interest in their earning But it also gives them a space for Reflecting on how they are developing in your class and their own frustrations that they might be feeling Which they're probably more likely to do in an online Anonymous journal context that they might be to do so face-to-face with you, right? Similarly with discussion boards You'll find that people if you give a key prompt that's usually Valculated in almost everything that people shares an opinion has values attached to it If you choose the right types of questions You'll be able to stimulate a lot more exchange between students as well as a lot a higher level of sharing and that Permits you a new lens with which to see the students and to get to know them a bit better One thing that's really great in language classes that I find really fun is if you tell the students in their very first class Okay, we're gonna introduce ourselves But you're gonna do this online and you're gonna just tell everybody a bit about yourself But you're gonna tell us three things that are true about you and one thing that's a lie You might get somebody who says, you know, I was I was born in a circus. I'm a triplet. I Love rainy weather and I enjoy sleeping on trains. We're not really sure, right? But what's fun about this is that one of those things is a non-truth whereas the other things are characteristics about the individuals So they're sharing and at the same time you get some insight as to their level of humor And you'll often find that the lie stimulates a lot of good exchange between the students especially online So do take advantage of this Sometimes students are hesitant because of their writing skills to participate in technology in this way to share in this way So you should give them the opportunity to also just record their answers if you think it's better just to have audio there So there's multiple ways that you can leverage technology But the main idea is that these are a lot of things that teachers intuitively do in a face-to-face context But which actually are freer because of the anonymity of an online platform It can permit you a better insight as to what these students value need. What are their goals for the course and so on So thank you once again for the opportunity to participate in your wonderful conference. Hope you guys had a great time Stay in touch. Thanks. Bye