Hey, hey, this is Carlos. I'm the founder and CEO at Product School. Today I'm here with Che Sharma who's the CEO and founder at Epo. Hey, Che. Hey, great to be here, Carlos. Thanks for having me. I'm happy to have you on the show. Now I was taking a look at your background before this interview because I don't get across many data scientists that turn into CEOs. So I'd like to learn a bit more about that part of you before we jump into your current company. Yeah, definitely. So as you said, my background is in data science. Most of that time would say Airbnb, where I was the fourth data scientist quite early on. Stay there for about five years and then worked as an early data scientist at other growth companies. The transition to CEO was honestly out of a desire to build better tools for data teams. So in that regard, it's a pretty natural transition to be the practicing the sort of thing that you're building for. So just for anyone who's been in the data space or adjacent to it, there's been huge secular trends in how people do analytics work with the rise of the modern data stack. And that just implies that we need a different set of tools for the way people work today. And here's the interesting thing that I've seen from my perspective of working in product. Those tools used to be for the data teams and engineers, you have to be somewhat technical to get value out of those tools. And a lot of them are becoming much more visual. And we are seeing marketers, product people, other stakeholders using them. So I just want to learn more about that. What is your angle and how do you see the opportunity to create a tool that is flexible enough for other people so the data team doesn't become a bottleneck? Yeah, completely. The way the trends have gone is that it started off by you just have the basic ability to deal with large volumes of data. The birth of the internet and cloud infrastructure meant large amounts of data. And then you had the basics of distributing compute with Hadoop to be able to do anything at all.  And so you ended up with these full stack data scientists for the only ones who could use it. But just the different parts of the supply chain have just gotten so much easier.  And it's all quite easy. Orchestrating jobs used to be, we literally had to build airflow Airbnb, Novice, DBT and other tools to make it easy. And so you actually now have the ability to build a tool stack. You didn't have to start from the ground up, which I think has led to all these great other tools come in. And I think one of the big things is so many of these capabilities used to exist in-house at places like Airbnb. But now that they're turning into actual companies, they're getting front-end resources and designers and self-serve becomes a lot more possible. Yeah, so let's talk about that. I feel that passion. You've been definitely part of the problem as a data scientist for different companies. So how did you experience that old school way of building data stacks and how did you arrive to the solution that you're building today with Apple? Yeah, completely. So again, my most formative data science career was at Airbnb where we started off and we started off with just MySQL replicas and crontabs, which for anyone who deals with data info, that's not a great place to scale a practice. We eventually adopted Hive, which is a way to run large jobs. And then we built Airflow to manage it. But along the way, the story I always like to tell is the thing about Airbnb in comparison with other companies is it was founded by designers. And so it wasn't founded by people who were naturally data quantitative people. It was founded by people who were much closer to a Steve Jobs-Walt Disney approach of intuition and design. And so as a data team, even while we're spinning up all this infra, we also had to drive impact in a way that would win over the orc. We didn't just come in with a halo that we can win arguments. And so the thing that always struck me was that we did a lot of great things at Airbnb, but the thing that made the data team into what it is today, which is probably one of the most sophisticated orcs on the planet, was the experimentation infrastructure. AB experiments is this fascinating workflow that builds this intimacy with metrics where you don't have to know things about ETL and databases and stuff like that. Instead, you just know this fairly simple paradigm, probably doing it since the fifth grade, dividing people into two groups randomly, having one group do one thing, the other do another, and see who does better and just kind of engaging with science without having to know all the ETL or data. And that ended up being a huge fit for Airbnb. Airbnb was such an entrepreneurial place. So the idea of being able to prove the quality of ideas without political processes, that was a great fit. And we ended up seeing some incredible impact during those years. I would say right around 2014, 2016, there were some huge changes coming from small interventions and large interventions. It was an incredible moment. And I remember those moments because this concept of AB testing was really revolutionary. And there were pioneers like Airbnb that made that tangible for the non-negative user. I remember the first time I was exposed to an AB test, and I was able to just literally tweak a couple of variations, put them in market within minutes, and then start getting insights and then making decisions on that. Now, today, we see in product management, AB testing or experimentation is a huge component. It's a huge skill that is being required in interviews. So from your perspective, how much knowledge in experimentation or AB testing you expect from a PM in order to be successful? Yeah. This is a big point of passion because if you consider the state of tooling for most of history, the assumed knowledge of a PM is actually quite high. There's a lot of ways you can shoot yourself in the foot within the experiment, not tracking enough data, having bias data, having improper setups, using statistics improperly. And these tools never really helped you there. They were happy to let you set up experiments and give you positive negative results, but never quite trusted them. And I think that leads to a lot of situations that you see today, which is that if you want to be a PM running AB experiments, you purchase some third-party tool for hundreds of thousands of dollars and then you still have to hire some data scientists who are actually going to give you trustworthy results. And so that was really a big part of what led to Epo being built, was to say, how can you have something really trustworthy as if you had a data scientist pulling data from the warehouse, giving you the ability to storytell, to root cause, and to socialize, all while being reliable. So the answer is up until Epo, I would say PM would be responsible for knowing data lifecycle and statistics. But with Epo, you basically just need to understand the scientific process. What do you want to learn? What is the best way to learn it? What are you looking for? So what are some of those best practices for PMs to collaborate effectively with data scientists and vice versa? Definitely. So this is something that I felt on both sides of the aisle, both as a data person and as a PM. There's a kind of startup cost of just making sure your data house is in order. So it's worth your org having at least one or two data people just to see, do you trust your metrics? When you say you're counting the number of purchases, is it a good number of the number of purchases? So if you're running an experiment that's meant to increase revenue, increase purchases, is the underlying data actually high quality or not? So I would say that there's a startup cost, which is just, do you actually trust your metrics? Once you have that, then I think their interaction with the PM and the data team becomes, do you have what you need to root cause things? For example, you have a purchase. Do you know like the marketing channel that came in on, or the subscription tier they're on, or the device they're on at that moment? Like the sort of things that might be explanatory for like why a certain conversion happened or did not, or why some effects are happening or not. Like making sure that it's all in place. So basically making sure you're well instrumented and that you trust your underlying data. Like that's something that a data and PM, data person and the PM should invest in early and make sure you're in a good state there.  You have to do it all, friend. And I agree. I've seen this evolution in product tech in general, but specifically with data tools where a lot of companies start with the spreadsheet, literally just putting numbers and creating formulas. And then suddenly they have these elaborate architectures with a bunch of different tools to a point that it's also unclear, right? Like what pool to use, who to ask, and then why did we get here? And there's a lot of tools out there in the market and examples. Some of those companies are public today. That's it. I mean, it's clearly value, but at the same time, there is this confusion. So I'm hoping you can help us identify, like as from a product perspective, what are some of those key layers for them to have a data stack? And how do you see Apple kind of positioning the whole stack? Completely. So lately, as in in the past five years, you've seen a huge consolidation onto a common stack. So the good news is, there's basically a right answer now. If you look at any sort of growth stage company, so post product market fit, and even earlier, you'll see this data warehouse model.  My favorite of the three is Snowflake for a variety of reasons. And then what you do is a lot of different SaaS tools will out of the box write to Snowflake. If not, then you need some sort of tool like an ETL tool, like a segment in five trends, get it into Snowflake. And then once your data team is in place, you can bring on a DBT to actually calculate what you need. So when we say the modern data stack, you're basically talking about Snowflake with DBT on top with some tools to get the data into there. And the benefit of that is that the most trusted metrics are usually pulling from a few different data sources. For example, your event data might not be very reliable, but your engineering transactional tables are actually quite reliable. Maybe you have multiple points of sale, and so you have purchases coming in from different sources. So basically the paradigm is, get everything into one place so that you can make logical choices on how to best model your data. And you're saying that that one place that is like high level enough can be accessible by people who are not just on the data team? It should be made to be. So the sequel ends up being the lingua franca of ending up with a high quality data store. But a lot of companies will get started by just kind of using the raw data and pulling it. The engineers, if they were pulling purchases, they would just do a postgres or something like that. The benefit with Snowflake is that your PM may or may not have access to a postgres table, but Snowflake can be given broad access to for pulling things. Yeah. One trend, and thank you for clarifying, because I appreciate those types of specific answers that we hear so much, so many options. And then I think putting these layers helps inform that decision. One of the trends, everyone's talking about AI and chat GPT and how are you going to change the world. We've been hearing for quite some time already. How are you thinking about really using AI and other technologies to get to insights faster, to not just present the data, but help person make a decision with that data? Yeah. Let's see. If you're trying to say I want to uncover insights to how to best improve my product with AI. I'll say I haven't seen it yet with chat GPT or AI. That level of meta analysis, I've seen more shallow queries being done quite readily, or if you want to author a blog post or something like that. You can use chat GPT to do things like how do I query this type of data and then it'll give you some sort of example, but it's usually not something to completely operate on. The thing with data is that for the more consequential the decision, the more precise and cleaner data has to be. If you are just making some sort of directional call, it's like, okay, plus or minus, whatever, it's fine if it's like 10, 20% off. If you're making calls, I'm like, should I launch this product or decommission the code and undo the last month of work, which is basically the call you're making with an AV experiment, then you kind of want the right answer. You want to make sure you're making the right call. Yeah. It's reminding me of what you said first about, do you trust your data? Yeah. That's a big part. It's a big part. If you want data to be part of the decision process, you have to first trust it. So kind of shifting gears a little bit into your team and then how you grew your company. Can you give me an idea of how big you guys are using any metrics that you feel comfortable with? Yeah. I mean, we're around 20 or so odd people, a bunch of data practitioners from across the industry. Again, my formal career, Airbnb, we have Snowflake, Uber, Stitch Fix, Pinterest, DNA in here. So quite a lot of different experiment capabilities. I would say some places that we tend to really spike on is one, data engineering, because experimentation is one of the more intense data engineering applications out there. That's usually what people find when they start building this in-house. And the other is design, because again, if you want trust, if you want people to make consequential decisions on things, then it has to be really clear what's going on. You need to not only feel empowered yourself to understand what you're saying, but you need to have the tools to storytell to your org. So I think those are two places where I take particular pride in the quality of our team. And Kev, for you personally, how have you evolved as a CEO? You can imagine when you were in a smaller company to today where there's more data people and designers and others helping you realize your vision? Completely. Well, the first thing is just actually making your vision and strategy really concrete. I think a big part of what the CEO needs to do is set the direction of the company and say what are things that we will be doing, what are things we won't be doing, and especially in the next quarter, two or three, like what is the very particular focus that underlies the strategy? Painting a really clear picture of where does the company need to go. You can do this in collaboration with your executives and others. And then the next stage would be making sure you have the right team to execute that strategy, like hiring the appropriate people. Once you have the people, making sure they're in the right positions, they're unblocked, they can operate really fluidly. And then from there, so much of culture stems down for the CEO. So also making sure you're extremely intentional on the culture you set. I come from a technical background as well. And speaking from experience, talking about strategy, culture, recruiting, it's not something that came naturally. So I'm just curious to know how were you able to acquire those kids while also probably letting go of some of your data skills to empower your data team to also grow? Yeah, I mean, I have always found that as a CEO and founder, I learned the most from other founders. You know, I'm very fortunate to have a nice cohort of kind of contemporary founders and also people who are much further ahead, who I can constantly learn from.  And, you know, everyone's got a job to do. Some are kind of directly building the products and we're selling it. You know, as a CEO, you have to take stock of like, as everyone's kind of doing their jobs, does the whole add up to more than some of its parts? And if not, then what's missing? What, you know, is there some process? Is there some direction? Is there something else to add? How is your product team structured? We structure our product team on the life cycle of experiments. So, you know, every experiment, you know, starts from some leadership mandate on like, what are the priorities for the business? You know, is it growth? Is it profitability? Is there something? Is there some other strategic outcome from there that translates into metrics and from their product teams need to come up with ideas for how to drive those metrics? You know, there's sort of a planning and setup stage. From there, you give birth to certain experiments, you set them out in the wild, got to make sure that they run properly, that they execute as you expect them to. There's no issues that arise along the way. And then you make decisions and you evangelize those decisions. So kind of each step of the life cycle is something that we've organized teams around to make sure those moments are always highly trusted, always appropriately collaborative. And, you know, in the end, they lead to EPO champions getting promoted. You know, I always say like our ultimate success criteria is that the teams using EPO are rising up within the organizations. They're always seen as reliable. They always seem to know what's going on. And, you know, the org just really wants to put more power into their hands. I love asking this question to what I call product CEOs or people who come from a building background who understand what it is to get something done. At the same time, we were talking before that company is bigger than product. Product is a huge component. And at the very beginning of the stage of a company, probably it's everything, almost everything. But for you now, where obviously you have a strong product and a strong product team, you also have other things to take care of. Like, how do you structure your team day to day or week to week? Like, where do you put your most of your energy these days? Of my personal energy? Yeah. Yeah. I think a big thing is to say, like, you know, of this, of the different initiatives going on, which are the ones that need further investigation, correction, you know, effort added, you know, perhaps like the org is a little bit unevenly bent towards one side. And you know, you're going to have to make a staffing decision on this place. And so you get your hands dirty in it and start doing the operating just to get a feel for who do you need to actually be in there. Sometimes you get a sense that a certain, you know, piece, whether sales or marketing or product or end or whatever is having trouble working through some blockers. And so you kind of get in there and again, get a feel for how it's going, working with your executive there, of course, in parallel. And then other times, you might have a strategic decision to make. And so there's not some like direct doing to happen, but you want to work with the executive and actually go through this decision together. So, you know, how I decide to personally spend my time, there's a mixture of reasons why I might focus on marketing or sales or products. But the most important thing is that it's done intentionally, right? You start out the week and you look at your calendar and you say, you know, what are my priorities and then does my calendar match my priorities? Yeah. And you mentioned recruiting and having a strong executive team a few times. I want to double click on it because I agree. The only way I feel personally, say, moving on to different areas is by trusting a team. And I know how it is to find these type of people. So how, what's your philosophy around hiding? Yeah, I would say the big thing, especially with an executive team, is you have to trust them. You know, you have to like, really, you know, these are the people essentially taking care of your kid, right? You have to feel very good about kind of putting this set of work in their hands. And so that requires one, doing the proper research and, you know, talking with other people of, you know, hiring these executives, thinking it's your own experience to actually intentionally lay out what sort of background you want. The other is to just psychologically impact yourself as a CEO and just kind of convey to someone like, what do I want to see from you to make sure I trust you?  So when you make the outcome of trust a kind of first order goal, then it sort of becomes like, okay, you need to look within yourself, you need to look within the function, you need to learn whatever, like, what does it take to get there? What sort of handshake agreement can you make for the executive to make sure you get there?  And if you don't hit those outcomes, you know, what else would you want to see?  That's the I'm smiling because that's a part that data can help, but sometimes it's not enough, right? And the interpersonal dynamics and, and really making sure there's a strong commitment and trust. It's, it's critical, at least in my experience as well. And obviously, as someone who comes from a data background who's building a data product, I can imagine you guys drink your own champagne and leverage some of Apple internally. Yeah, yeah, completely. You know, we've set up some experiments here and there. We use our own feature flagging. And I think beyond just literal ABA experiments, we just believe in running experiments, right?  Tell me more about that because I, obviously, I believe in that and I heard from other CEOs how important it is. The counter is if there is not enough psychological safety, what's going to happen if that experiment actually doesn't work, you know, because and so I want to know more about how you are able to create those conditions for people to feel empowered to actually try things. Yeah, completely.  Basically, every decision is reversible or irreversible, right?  Does it actually matter? You know, like, if you change your mind on this decision, is it something you can take back? And if it's the case that you can take back the decision, it's reversible, you know, you can start doing this other process instead of this process, you can do this other, you know, workflow instead of this workflow. You should just try it out, right? Like, there's, if the cost of making the wrong decision isn't that high, you should just try it out. Alternatively, there are other decisions which are kind of closer to one way doors. You know, once you go down this route, it's actually very, very messy to undo. And that's where you actually have to align and treat the decision a little bit more seriously. And you know, you may run, you still may run an experiment just to de-risk it, but it's going to be something that's much more intentional.  But the vast majority of decisions are actually reversible. So what's next for you? What's that vision for Apple?  You know, for all of our companies, we want to make sure that, you know, everyone in those companies is, those companies are able to test out ideas, show impact, you know, be able to show ideas are good without having to win political processes. And so for that, you know, one starting point is to just say, let's make Apple intensely collaborative so that, you know, decisions being small, you can kind of all get together, make those decisions, make those decisions, kind of document it, turn it into something that's knowledge-generating. And then the other part of it is to just increase the scope of the type of decisions that are being made with Apple. You know, there's a lot of things that are not just the eB experiments that, you know, still have that shape that you should be able to try them out and see how it goes. So I don't want to say too much more until we roll it out, but I'll just tease that, you know, the idea that we're trying to change a culture, and that involves, you know, bringing much more people into a collaborative decision-making framework and handling a lot more types of decisions is very much on our radar. Well, I really enjoyed chatting and learning from you. Thanks so much for your time. Absolutely. Thanks for having me, Carlos.


Ethiopian staff working for the United Nations or African Union do not live in space and will be punished for any law-breaking. That's the stern warning from Aris Ababa, following the arrest of several UN employees for a specified offences. Take a listen. United Nations staff will be punished for breaking the law. The Ethiopian government has warned, saying they are not in space. Those comments from Ministry of Foreign Affairs spokesperson Dina Mufti on Thursday, following the arrest of several UN employees for unspecified offences. They're not in space. They're in Ethiopia. They have to respect Ethiopia's law one by one. If they don't respect the law of the land, they will be legally held accountable. Whether they are UN or African Union staff. Ethiopia declared a state of emergency earlier this month after rebellious forces from the northern Tigray region and their allies threatened to march on the capital. Since then, hundreds of Tigrayans have been arrested in Aris Ababa, their families and colleagues have said, along with 16 UN staff members, whose ethnicity has not been revealed. Seven of the UN staff were later released. Police say the arrests are not ethnically motivated. Foreign citizens have also been caught up, with governments of the U.S., Britain and Italy reporting nationals detained. Over the past week, envoys from the African Union and the U.S. have visited Ethiopia to push for a ceasefire in the year-long conflict with the Tigray People's Liberation Front. But at the briefing, Mufti reiterated the government's position. That is, the talks will not be held until the TPLF withdraws, stops attacks and recognises the government's legitimacy. The TPLF said last week it had seized the town of Kamis, 200 miles from Addis. On Thursday, state-affiliated broadcaster Fana reported that the Ethiopian military had repelled attacks near Kamis and pushed back forces near the strategic town of Mele. A TPLF spokesperson did not immediately respond to a request for comment.


A we now turn to topical questions. To ask the Scottish Government for its reaction to the findings of the AMB Union report on Northsead decommissioning costs Cabinet Secretary Keith Brown. As made clear on our programme for government, the Scottish Government recognises that the opportunities that are presented by decommissioning and we are committed to ensuring Scotland's d sinon consecutive project on allate their company position or any job. Yesterday, the Minister for Business, Innovation and Energy, and I hosted a round table of industry representatives who focused on the challenges and opportunities that they have bore by decommissioning and whose further engagement is on-going. Scottish Enterprise is developing a decommissioning action plan that needs to be published by the end of the year. They are working closely with Scottish ports and harbours that can understand their capabilities and to i'w ddod i nhw ymwyngoedd sydd yn gyntafol i ddechrau'ch gwahodraeth gweithio o'r cyfleoedd gyda'r sector cyfnogau. The Government's priority is to encourage industry to maintain the infrastructure that exists already in the North Sea, so that the value of oil and gas reserves can be maximised. The GMB report notes that the current structure of decommissioning tax relief is in their words, severely restricting the potential for new entrants with more agility and a lower cost base to extend the life of many UK CS fields. The Chancellor's autumn Ar fluorescent druffham's statement presents an ideal opportunity to offer the industry the support and clarity it deserves. The UK government must provide support in widening access to recomissioning tax relief to ensure that the full potential of late-life assets can be realised. As called for both by the Scottish Government and the industry." Gillian Martin is mentioned to model the Chancellor Give's Ottoman statement. Tomorrow, his statement will have the opportunity to outline vital support for the North Sea oil and gas industry. running representationism on behalf made to the UK Government on decommission tax relief? The Cabinet Secretary for Finance and the Constitution wrote the Chancellor of the Sh tracker on suppose 13 November. It calls for the autumn statement to improve the access to December 3 for decommission tax relief. The letter also highlighted addressing the fiscal barriers to asset transfers that can also extend the The future of assets and reduce decommissioning costs. In addition, I have made representations to the previous the Treasury about loan guarantees for infrastructure within the North Sea? Julian Martin. Yesterday, the Scottish Government signed the Aberdeen city region deal, which includes investment for the expansion of Aberdeen harbour. Can the cabinet secretary update Parliament on how those plans are progressing and how increased harbour capacity will help the north-east of Scotland to capitalise on the opportunities of decommissioning in the North Sea? As a member says, we did indeed sign the Aberdeen and Aberdeenshire city deal yesterday, which comprised around £250 million of support, equally split between the UK Government and the Scottish Government, and an additional £254 million was also committed by the Scottish Government in relation to capital activities. The Aberdeen harbour expansion has both regional and national significance, and it will strengthen the harbour's key role in supporting the economy in the north-east, in particular the oil and gas sector. Aberdeen harbour board has progressed discussions with the preferred bidder for the implementation of the works. Scottish Enterprise is in the process of engaging with Scottish ports regarding their interests in providing facilities for decommissioning activity. It is also investigating what can be done to make Scottish locations more competitive for those types of projects. A decommissioning action plan is scheduled for publication in December 2016. The cabinet secretary will have seen the report from the GMB this week that makes clear that the Scottish Government and the UK Governments must be taking action to make sure that Scottish workers get those jobs and that all the work does not go to other countries. The report very clearly calls for an urgent investment fund between the UK and Scottish Governments. Can the cabinet secretary please tell me what plans he has for such an investment fund, separate from any city deal arrangements? What we are concentrating on currently is trying to first of all identify, as the GMB report seeks to do, the size of the opportunity that is there. It is quite clear from a factor of 46 to 1 that the massive amount of expenditure and decommissioning will be concentrated on plugging wells and abandoning wells. That is where the high-value jobs, where the high-value work comes. Of course, there is a possibility for taking top-sides and deconstructing those in harbours and ports. We want to understand the level of that opportunity, the level of investment that is required to make sure that we can tap into that opportunity. Although it is urgent that we do that, it is also true to say, as the industry will tell you, that the cost reduction activities that they have been involved in have now pushed out quite substantially their decommissioning plans in many cases. What they are now concentrating on in those cases is maximising the economic recovery from those facilities. We will keep very close to the industry. Paul Wheelhouse and I discussed this yesterday, we will talk further with the trade unions and those ports and harbours that are interested in taking on that work. We will make sure that it is appropriate at the time that those opportunities become available. Does the cabinet secretary support the decommissioning programme put forward by Shell, UK and the Brentfield? Recognising the real growth to be captured in the north-east is through learning and innovation, supported by the oil and gas technology centre, funded by the city deal yesterday, which I visited yesterday, and exporting those skills globally. That is a very good point. The point that the member makes about the Brentfield, just to go back to the point that I have just made in response to Jenny Marra, is that there are more than 100 companies active, many of which are Scottish companies involved in that process in the Brentfield. A huge amount of economic activity is going on in terms of decommissioning right now. The member is also right to say through the oil and gas technology centre that, if we can produce the innovation that is required and provide collaboration between the different actors that are interested in this work, then we can develop such expertise that can be sold around the world to the benefit of companies in the north-east of Scotland. Lewis Macdonald Clearly, technologies in the oil and gas technology centre and elsewhere can play a key role in taking forward decommissioning. However, does the cabinet secretary acknowledge that the time for action is now and that the urgency of the opportunity is that Scotland has to act quickly if it is to gain those opportunities at this stage while companies are making their decommissioning plans going forward? I agree with the member that we have to act now, but I would make the point again, as I did to Jenny Marra, that the industry itself says that its plans have changed quite substantially because of the low price of oil that they have had to get involved very substantially in cost reduction. That has also led to some more marginal fields being pushed further out in terms of the length of time that they will be exploited. We will respond to what the industry is doing. That is the way to do it, to try to identify the opportunity. Of course, there are opportunities that are there, but it is important to recognise that a lot of that is going on right now in many high-value jobs in Scottish companies and UK companies that are being sustained by the decommissioning activity that Scotland and the UK have an extremely well-in. At the same time, we should of course be aware of future opportunities as well. To ask the Scottish Government whether we will consider holding a public inquiry into deaths in childbirth in NHS Ayrshire and Arran. Cabinet Secretary Shona Robison, the death of any mother or baby in childbirth is a tragedy for all involved. Following concerns raised by local parents, I have asked Healthcare Improvement Scotland to undertake a rapid review of the processes that Ayrshire and Arran have in place to review and capture learning from significant adverse events and to look at the internal investigations that the board has undertaken. I have offered to meet with the parents concerned on the conclusion of that process. I should say that we have the lowest ever recorded maternal mortality rate and stillbirth rate in Scotland. Last year, 88 babies were born alive as a result of the processes that we have in place to drive down the stillbirth rate compared with the 2011 figures, but we are certainly not complacent. A programme of work is under way led by the Scottish Government's stillbirth group that I established. Through our Scottish patient safety programme, we should better understand the incidents and causes of stillbirth to continue to drive the rate of stillbirth down further. I thank the cabinet secretary for her response. We welcome yesterday's announcement into an HIS review, and naturally our condolences are with all the families involved in those cases. However, in the tragic case of the Morton family, they wrote to the minister in January of this year. Then again in May, my colleague Brian Whittle followed this up with a letter in June, and finally a one-page cursory response was received from the minister in July. In her response, the only reference to her review is that of a pre-existing Government review into maternity services announced the year before. So, may I ask the cabinet secretary why has it taken FOIs, whistleblowers, a BBC investigation and negative press headlines and, indeed, the lodging of this topical question in Parliament today, before the cabinet secretary finally accepted that this was a serious problem and announced the formal review? Can I say first of all that some of the at least three of the six cases that have been mentioned so far predate the HIS review that was carried out in 2012? Let me just go back for a second to the HIS review in 2012, because there was a detailed review of the adverse event management in the spring of 2012 at the request of the then cabinet secretary. Some improvements have been made. It is important to say that, within Ayrshire and Arran, as well as the rest of Scotland, the deaths of neonatal babies and the rate of stillborns and, indeed, maternal deaths have reduced. That is something that we should acknowledge. However, what I want Healthcare Improvement Scotland to do is to look at some of the concerns raised by the families about the processes and procedures followed or not followed by Ayrshire and Arran. What I want to assure myself of is that what was supposed to happen in each of those cases did happen. If not, we need to know why that was the case and to reinforce with Ayrshire and Arran and, indeed, any other health board that, when there is a significant adverse event, there are very clear processes that his established that should be followed. Indeed, if it comes to a reference of a case to the Crown, there are very clear guidelines there as well. I want to assure myself in all of those cases that those processes were followed. If not, then action will be taken to make sure that the lessons are learned from that. Jamie Greene I reiterate that we welcome the specific review, but we also want to make the point that all potential options for audit and scrutiny should be on the table, including the option of a third-party inquiry. I wonder whether the cabinet secretary will agree to that suggestion. The issue that a lot of people have, Presiding Officer, is that four years ago, a similar HIS review into north NHS Ayrshire and Arran made very specific recommendations on what could be done to improve the reporting of significant adverse events. Previous reports also flag problems around fetal heart monitoring, for example, but staff shortages are preventing midwives from being able to attend those mandatory training sessions and the health board is still failing to carry out serious adverse event reviews. Therefore, in addition to the HIS inquiry that was announced yesterday, what further measures the Government is taking to ensure that any recommendations coming out of an inquiry or review are not just acknowledged but acted upon and if the cabinet secretary will take personal responsibility for overseeing the process. Let me remind the member that, when the first review in the spring of 2012 was carried out, Healthcare Improvement Scotland developed a framework and a programme of reviews. It was to try and ensure that there was a consistency and approach to managing adverse events. That national framework was published in September 2013, and then it was refreshed again in April 2015. In 2015, Healthcare Improvement Scotland also held a series of progress meetings with boards to understand how they were continuing to implement the national framework. There was action that came out of that review in 2012. New procedures were established and, indeed, HIS went back to make sure that those processes were being carried out. What I need to understand about Ayrshire and Arran is that in those cases where families have raised concerns, where those procedures and processes followed, bearing in mind that half of the cases predated that review in 2012. That is what we need to know. In terms of any action after his report to me, we need to wait and see what his say before we judge whether further action is required. However, I would expect all boards—I have asked my officials to make sure that all boards are reminded of the processes that are required to be carried out when there is a significant adverse event, and to remind them of the guidelines for referrals of any cases to the Crown Office. The member also mentioned the issue of staffing. What I can say is that we have an increased number of midwifery staff in the NHS. There is a 4.1 per cent increase in the number of midwives over the past few years, since September 2007 to June 2016. Overall, NHS Scotland meets the Royal College of Midwives-recommended midwife-to-birth ratio. We are also aware that, sometimes, there are challenges in particular areas, but in Ayrshire and Arran, it has recently recruited six additional midwives to the team and are planning further recruitment later in the year. The national review that is coming to me in the next few weeks is also looking at working with the Royal College of Midwives to make sure that the models of care being delivered in our maternity and neonatal units are the best that they can be, and that is also looking at what the workforce tools should be in taking that forward. I am very happy to make that review available to Parliament. I thank the Presiding Officer for his indulgence, and I urge the cabinet secretary to make this inquiry wider and deeper. On a due date in 2009, my wife Patricia, having been sent home and been physically sick, was finally admitted to the southern general maternity unit despite their protests. A consultant, junior doctor and two midwives examined her that day.  She was given morphine and put to bed. Overnight, her baby died and had to be delivered by cesarean. Patricia's liver ruptured and she spent 19 days in intensive care and high dependency. For 20 months, we asked Greater Glasgow and Clyde to explain how they would prevent such a failure of care reoccurring and impacting other lives. They blanked us and, when we were thereby forced to take legal action, had a QC wasting thousands in taxpayers' money to defend the indefensible. It took five years before they eventually conceded this year. What will the cabinet secretary do to ensure that other people in Ayrshire, Glasgow and elsewhere in Scotland are not treated as badly when they experience such a tragedy? To Kenny Gibson, the way in which the case was handled and the experience that Patricia Gibson had was absolutely appalling and, indeed, the whole family had. We would expect all health professionals to treat anyone who has suffered any bereavement with care, dignity and respect. Clearly, that did not happen in that case. We should learn the lessons from that case. If Kenny Gibson wants to follow up with me further around some of the issues arising from that case, I would be happy to do that and to make sure that, through the national review and the work that healthcare Improvement Scotland is undertaking, that we make sure that cases such as that, where possibly can be avoided, are avoided. I would be happy to have a further discussion with Kenny Gibson about that. Thank you very much, Presiding Officer. I start by saying that our thoughts and condolences go to all the families that are affected. Indeed, I thought that Mr Gibson made a very powerful and, indeed, emotive case that will have touched all of us in the chamber today. I welcome the announcement of a review in Ayrsyn Arran, but I do believe that the cabinet secretary should go further. The BBC investigation found that there were more than 25,000 adverse incidents since 2011, including the death of 26 newborns, 79 stillbirths and three mothers, two losing their lives. Those figures are heartbreaking and our thoughts are with every single family who has been affected by the death of a child. The investigation also revealed that 500 incidents were related to staff shortages and more than 100 were due to delays in treatment. It is increasingly clear that there is more and more pressure being put on our front-line NHS staff who are doing an amazing job and with thousands of posts going unfilled in our health service. I have raised the issue of workforce planning with the health secretary before, and she is right to say about the numbers in terms of the NHS workforce, but the reality is that vacancies among our nurses and midwives are increasing in Scotland. I would ask her, in light of the investigation, if she will extend the review to look at staffing more widely across all maternity units in Scotland. I will address the issue that Annasarra raised about adverse events—the 25,000 that have been cited by the BBC. They range from minor adverse events such as slips, trips and bumps, administrative errors and right through to very significant adverse events. It is important to make sure that people listening to this understand that we are not talking about 25,000 significant adverse events. Every adverse event should be investigated and lessons learned from all those 25,000. However, let me focus on the significant adverse events, because we absolutely want to ensure that we, through the patient safety programme, continue to reduce those significant adverse events and avoidable deaths. Let me reiterate what I said earlier. Through the patient safety programme, which has been operating now for a number of years, we have seen the lowest levels of still birth on record. We have seen the lowest neonatal deaths on record, and we have seen maternal deaths decrease as well.  We have seen huge progress being made through that patient safety programme, so it is fair to say that our maternity and neonatal units are safer now than they were four or five years ago due to that programme of patient safety work. We should thank our front-line staff for the work that they have done in achieving that. However, we absolutely must not be complacent and that work will continue. It is overseen by external scrutiny, so there is a group called Embrace, which looks at units across the whole of the UK. Every year, they do a report, and in that report it highlights any unit that is above the average for deaths. That information is out there in the public domain, and those units would then be expected to address why they have a higher than average level of avoidable deaths. That is a very, very important external scrutiny. The duty of candour will also add another layer of external scrutiny to that. On the workforce, as I said in the answer to the previous question about workforce, the overall NHS Scotland meets the Royal College of Midwives recommended midwife-to-birth ratio, and it is important that we make sure, through the national review and the work that we are doing with the Royal College of Midwives, that those workforce tools, which are very important to make sure that we have the right staffing level and the right units, are applied. Where there are vacancies, clearly that is a challenge, but we are working with boards to overcome that. There are more midwives out there. We have had four years of increase in student midwife numbers, and we now have new midwives coming out into training and into post, and that is something that we will continue to support. There is a huge amount of interest in the subject. I am only going to take one more question. There is no more time, I am afraid. Brian Whittle. Thank you, Presiding Officer. I have been working with my constituents, Mr Morgan and Mr Morton, investigating the tragic death of their son, Lucas. In scrutinising action plans produced from a previous root cause analysis review, we were shocked to discover that areas recommended for improvement in 2010 were the same areas that failed them and their son five years later. How can Mr Morgan, Mr Morton, Ms Logan and the public have any faith in any new Government review when the lessons have not been learned nor recommendations implemented from past systems failings, which, after all, is all that Mr Morgan and Morton is looking for? That is precisely why it is very important that I understand whether or not the processes and procedures that Ayrshire and Arran were supposed to have followed were followed, whether that is in the Morton family's case or any other of the cases that have been raised. I need assurance of that. If they have not been, we will take action to ensure that that happens. The 2012 review that was a previous review did establish some very important changes to ensure consistency in applying the significant adverse event processes and, indeed, making sure that lessons were learned and that there should be standardised criteria for, for example, referral to the Crown in those cases that were appropriate. If there are cases that have those processes have not been applied properly, I will absolutely take action to ensure that that is addressed. I have offered to meet with the families once I have his report to discuss the findings with them, and I would extend that invitation to the Morton family. I apologise to the members who couldn't get in there. Question 3, Liam McArthur. To ask the Scottish Government in light of reports that the Chinese company Sinofortone is willing to reopen investment discussions, what steps it will take to ensure that any investment is transparent. The Scottish Government is committed to attracting investment in jobs to Scotland from China and elsewhere, but there are currently no talks scheduled with Sinofortone. We have been clear throughout that the memorandum of understanding with Sinofortone and China railway number 3 engineering was about exploring possible investment and involved no legal, contractual or financial commitments on behalf of the Scottish Government. If projects involving the Scottish Government do arise, full due diligence will be taken forward in the normal way. The Scottish Parliament would scrutinise those projects and, if there were any concerns, projects would not happen. The Scottish Government condemns human rights abuses wherever they take place. We are committed to engaging with the Chinese Government on human rights, an issue critical to China's long-term prosperity and social stability, as part of our overall engagement. Respect for human rights and the rule of law is one of the four guiding principles that be set out up front in the Scottish Government's China strategy, and that underpin all of Scotland's dealings with China. Liam McArthur. Thank you very much. I thank the cabinet secretary for his response, but it was difficult to ascertain from that what he plans to do differently. Indeed, he referred to the fact that any processes in the future carry along in the normal way. Can he tell me that, has he looked again at the protocols that he uses to assure himself on human rights and other issues before he gets the First Minister to put pen to paper? The First Minister said that she would learn lessons. What lessons have been learned, and is there anything different? We would seek to be as transparent as possible. Just in relation to the MOU that was talked about before, it was not published at the time of signing as it related only to exploring investment opportunities, not specific investments or projects. We have since released over 70 pages of material relating to the signing of the MOU, and responded in detail to all 37 written parliamentary questions and nine FOI requests. I repeat that there was not a deal being done, there was a memorandum of understanding, discussions were taking place, and that is a way that we would seek to approach that in the future, to make sure that we are as transparent as possible and to make sure that any projects that were agreed through the Scottish Government were made available to the Parliament so that the Parliament could scrutinise those projects as they progressed. Again, I am interested and grateful to the cabinet secretary for his response, but there was nothing there that suggested the approach next time round would be any different. The First Minister assured this Parliament that she accepted that there are lessons for the Government to learn, and we will reflect on and learn those lessons. Can the cabinet secretary set out for Parliament what those lessons are for future deals? I have made clear in my previous response exactly what our approach should be. We would seek to be as transparent as possible at all times. It is important to realise, as I have said already, that there was not a deal being done here. Those were preliminary discussions, as in the nature of previous Administrations that the member was involved in previously. There is a point at which those discussions are confidential. There is a point at which we can come public. Of course, if it leads to projects, the Parliament has a very legitimate right to make sure that it is able to scrutinise those projects. We would make sure that the only actions that we took were consistent with being as transparent as possible and making as much public as possible during that process. However, it is very important that we have a Government that is willing to go out and look for investment from around the world, including with China. That is exactly what the First Minister and I have done previously. E-mails that were released under the FOI indicate from previous talks between Sinofortone and the Scottish Government that they were interested in energy projects and large-scale affordable housing projects on a scale of 3,000 to 5,000 homes, which would be manufactured in a purpose-built production facility. The company indicated at that time an interest in Falkirk and the Central Scotland area that I represent. Can the cabinet secretary confirm whether that approach to delivering affordable homes is one that he supports? Can he advise the chamber what role, if any, Chinese steel would play in those projects? Is he willing to rule that out at an early stage? Apologies to the member. I was not clear what was being asked to rule out. In relation to the affordable housing element, of course, this was not the Scottish Government building those houses, nor was there a project designed to build those houses. All that was done was a discussion that was being had. As the member has mentioned, one of the local authority areas was also involved in that process. We do not control the affordable housing that other parties want to build. We do not control that. Of course, we have our own programme in relation to affordable housing and the ambition to create 50,000 new homes over the course of this Parliament. That is what we are concentrated in. However, if somebody else wants to come in and invest in housing, of course we will look to see how we can support that. Of course we will be concerned about the nature of that, but we will be between the parties that are contracting in that regard. To repeat, that project was not an agreed project between the parties involved in discussing this. There were no projects in fact, which were being agreed at that point. All that was being agreed were discussions. As the member mentions, some mention was made in the emails that were released about possible areas of interest, but no projects were proposed. It is at that point that projects are being proposed. Of course, we want to have the due diligence that is undertaken and the scrutiny of others to make sure that those who are satisfying the Parliament's need to look at those projects as they were brought forward.


So, while we were all focused on the first 2020 Democratic Party primary debate, we didn't really realize what Nancy Pelosi had done. She just gave Mitch McConnell a gigantic legislative gift. And this is on top of the passes that she keeps giving to Donald Trump, time and again. Because we already know that she refuses to impeach Donald Trump, no matter what you can't convince her. It doesn't matter that there's more than 10 instances of obstruction in the Mueller Report. It doesn't matter that Donald Trump is in violation of the Emoluments Clause or violated campaign finance reforms. She doesn't want to impeach. Okay, so if you don't want to impeach, maybe you can use the influence that you have as Speaker of the House to draw attention to some of these really serious allegations recently that have been lobbed against Donald Trump. So as you all know, Trump was recently accused of rape. So is Nancy Pelosi going to use the influence that she has to draw people's eye towards that to at least have the public hold them accountable? Not really, because here's what she said about that as well. I haven't spent any time on that. She said raising her arms in frustration. I don't know the people you're referencing. I don't know the person making the accusation. I haven't paid that much attention to it. Okay, well, that's kind of weird because you're in a position of power. And this is a very serious accusation. So you think that being from the opposition party, you would draw people's attention to this so that way we know what the president is doing so we can hold him accountable. But nonetheless, she doesn't care. She gives them political victory after political victory. But as I alluded to, she just gave them a legislative victory because she could give them as many political victories as she wants to. And that's still horrible.  However, when you start giving them policy concessions, that's when you're doing real damage and you need to resign.  So what did she do? Well, as Zach Carter of HuffPost explains, as horror stories detailing conditions at the border began piling up this week, McConnell passed a bipartisan bill expanding funding for Trump's immigration authorities by 4.6 billion. Native Democrats in the House, including Representative Alexandria Ocasio-Cortez, wanted to include some basic humanitarian safeguards on that money. But Pelosi, citing resistance from self-described moderate Democrats, decided instead to just pass the Senate bill. No strings attached. The Trump administration's immigration atrocities cannot, of course, be chalked up solely to financial constraints. Its flagship immigration deterrence policy separating children from their parents was not adopted out of budgetary desperation, but under Pelosi's guidance, the official Democratic Party response to the Trump administration's abusing immigrant children is to give more money to the agencies the Trump administration relies on to abuse immigrant children. She just gave Mitch McConnell exactly what he wanted with zero pushback. I mean, and as I read this story, I'm reminded of all the mainstream media news pundits who always praise Nancy Pelosi. Marcos Molitzas of the Daily Coast delivered her thousands of flowers and were always told how effective she is as a leader and how she always gets Donald Trump and gets under his skin. But then you see things like this. She's not an effective leader. She's a feckless leader, as is Chuck Schumer, who was also incredibly weak. But in an interview with Jake Tapper on CNN, AOC explained why she thinks this probably passed. Well, you know, they all want to get out on the fourth and leave. So let's just not even debate this. Let's just give them what they want so we can all be done. The House bill passed resoundingly. There were only four Democrats that defected, including myself from that bill. And the reason I defected is because I needed to send a message to my constituents and my folks back home. And in fact, my constituents asked me to vote no on the bill. But if it ultimately came down to it, if we needed 100 percent every single Democrat to vote on this bill, we could do it with the amendments. But the problem right now, and the question at hand right now, is that Mitch McConnell sent us a bill and we're just putting a big check mark on it instead of even trying to negotiate. I have indicated that I'm willing to stay here. I don't need to go home on vacation. I don't need to go home to July 4th weekend. I will stay here all weekend to make sure that we get this thing done. And instead, what Mitch McConnell is doing is that he's relying on the time pressure of recess to try to ram through a bill that is completely irresponsible to the American people and to those kids on the border. So understand Nancy Pelosi, she could have used the majority that Democrats hold in the House to her advantage to get protections for human beings laid out explicitly. But she didn't do that, presumably because she cared more about getting this all done in time for everyone to take the 4th off. This reminds me of when Chuck Schumer fast tracked a bunch of Donald Trump's judicial appointments last year because he wanted to give Democrats running for reelection the opportunity to campaign.  How much longer will this continue to take place? She just gave Mitch McConnell a gigantic legislative gift and she is refusing to impeach Donald Trump. In other words, what she's doing is she's allowing someone in power to be above the law, not even opening an impeachment inquiry. I don't know how anyone can be okay with this. It makes no sense to me. But here we are. We're in a situation where we have such an ineffective, faceless leader that gives Republicans all that they want. But because we care more about civility and decorum, that's okay. It's disgusting. I hope that the stories that AOC shared today of people having to drink toilet water in these migrant detention facilities, I hope that that makes Nancy Pelosi feel like shit. I hope that that keeps her up at night knowing that she has the power to do something and she could have possibly done something, but she didn't. I hope that that's on her conscience.


Embracing new media and language learning. Developing language skills is a key enabler in the global economy. Even as the rise of the internet, smartphones, and other technology, the world is becoming smaller but as it always has, language can present itself as a barrier. As English is still the de facto global language thanks to its common business, user at time that Tefl providers adapt to the modern world and embrace new media to not only improve existing techniques, but develop new ones and embrace existing online communities? Definition of the new media. What is new media? New media is defined as products and services that provide information or entertainment using computers or the internet, and not by traditional methods such as television and newspapers. As discussed during the course, the key to ensuring student progression is to maintain motivation and interest during lessons, one of the key tools to develop and maintain this is the use of games and communication activities. These are areas that new media thrive in, by its very nature its interactive and current. Analysis and evaluation. While many language schools have adapted to new media through the development of online courses delivered through Skype, online course books, and language games, are they missing a trick? Over the last few years, there has been a rise in the use of smartphone apps for language learning. These take many forms, but they can roughly be divided into two groups, traditional and gamified. Where traditional apps utilize the mainstream teaching approach with vocabulary, grammar, and pronunciation drills and activities with heavy use of realia, such as Rosetta Stone and Babel. While gamified apps take a gamification approach, and try to reduce the drill approach and utilize games where appropriate, and where not appropriate the ad elements of gaming into it via progress mechanics, such as points, leaderboard badges, narrative elements and leveling up. These gamified apps have proved to be very popular with students, as in 2008 nearly 170 million Americans played computer games to achieve multiple learning goals, spending almost 12 billion dollars. And as we can see from research, these are not wasted dollars, as those using gamification platforms performed better overall and in particular in practical assessments. Tools. Other areas of new media that have been thoroughly utilized by TEFL professionals, are the communication-based tools that the internet provides, such as Skype and other video and voice conferencing software, allowing for decentralized learning and learning on demand. While these services have their supporters, many feel this reduces the benefits of having access to a native speaker. These criticisms are aimed not only at technical limitations such as poor connectivity, but on more key elements such as the difficulties in fostering a community and rapport in general, which is discussed during the course can be the difference between a good and bad class. As well as more fundamental issues of not having visual or social cues on when students are struggling, plus it's harder to control a class from being distracted by external interferences. In addition to Skype, the internet has been a key resource for TEFL teachers, and the benefits and limitations have been discussed at length during the modules on this course. While these areas of new media have been embraced and utilized by TEFL providers, there are still areas that could be utilized effectively, while mainly focused on young to young adult learners, they could prove to be very useful. One area that may prove to be useful, is to focus further on games, in particular the use of mainstream games with alternate language settings. In the same way, that films provide access to natural and authentic language in a current and engaging manner, games can provide even greater benefit due to their interactive nature. A student can see and understand the effects of their interactions, this harks back to natural language learning of young children. However it must be noted that not all games are suitable for this, and should contain at least some narrative story, or a mix of text and audio options. There have been numerous studies on the benefits of games in relation to language learning, while mainly focused on specialist computer-rated language learning games, they do highlight the benefits of this approach, with its rule-based approach and natural discovery of language. In addition to this, some games improve further upon this, with the addition of native non-educational speakers to a narrative-based environment with massively multiplayer online role-playing games, as this provides additional natural language in an unpredictable yet controlled manner, the context is controlled even if the language isn't. Furthermore, having a game interface may provide a benefit to some learners, as there is a barrier to embarrassment, and allow students to save face when trying a new language and attempting fluency. This may be particularly beneficial with shy students, or where cultural issues might prevent attempts in traditional scenarios, due to fear of failure. While this approach may have benefits, it may be hard to convince the main stakeholders in TEFL to utilize these tools, as TEFL is a business and utilizing games in this way may not be seen as professional, and parents of young students may not be particularly pleased with their children using games in this manner. Do you want to teach English abroad? Take a TEFL course today. However, we can take elements of the mechanics of these new methods of learning into the classroom. Gamification is not limited to online platforms but can be utilized in classroom-based activities as well. In fact, a key component of TEFL could be considered gamification on a base level, with its high use of games and motivational systems. Perhaps we as TEFL teachers can utilize more elements of this methodology within our classrooms, and utilize narrative and progression mechanics to help drive motivation through some of the more difficult aspects of English learning, as well as drive more creative role-play elements, to maintain motivation and develop entertaining and rewarding language learning for all.


Okay. Well, hey, Martin, thanks so much for joining me on this fireside chat. You know, I'd love maybe a quick intro for all the folks who are joining us in Fluent Con, maybe who you are, what Chronosphere is, just a little bit about yourself. Yeah, for sure. Thanks for having me, Anurag, and really excited to be here today, chatting with you. A little bit about myself. So my name is Martin. I'm currently the CEO and co-founder of Chronosphere. We provide a hosted monitoring solution to companies adopting cloud native. We help these companies monitor their infrastructure, so primarily Kubernetes these days, monitor their applications, which is generally microservices oriented these days, and monitor their business as well in real time. And the core technology of our product really came out of Uber, and a lot of the open source observability projects that came out of Uber. And that's actually where I spent four years of my career before founding Chronosphere. I let a core part of the observability team there where we created projects such as M3, which is a distributed and scalable metric storage engine. It's compatible with Prometheus as long term storage there for metrics. We created Yeager, which is the CNCF graduated distributed tracing project. We actually completed the trifecta and created a logging platform as well internally, but unfortunately, we never open sourced it. So yeah, I spent a lot of time in my career in the observability space, both solving problems for Uber with these direct solutions, but also for the broader community via the open source channels as well. So a little bit about myself there. And again, really excited to be here today and looking forward to our chat. Awesome. And Yeager, alongside Fluentee as a graduated CNCF project.  You know, I think as probably one of the leading folks in that observability space with Yeager and all these other projects. You know, I'm curious, what should companies be thinking about or what should users actually be thinking about when solving for observability? Yeah, it's a great question. And I love the fact that you put users in there, right? Because it is probably what the focus should be is what should the users be thinking about. And even if we think about who the users of observability are, that's been changing fairly rapidly, I'd say, right? So historically, perhaps the users or practitioners of observability and monitoring has been isolated to the SRE department or, you know, perhaps a core infrastructure team. However, if you think about modern development and application developer, they not only have to write and develop their application, they also have to test it, that to deploy it. They have to monitor in production. They have to remediate issues when it goes wrong. So really for us, observability, you know, is more than just a practice. It's probably like a cultural mindset, much like how DevOps is. And it really is something that all developers, you know, we are seeing embrace and adopt this culture and this mindset. We're seeing that more and more. So really the end users of observability is all the developers out there. And if you look at it from that perspective, they're really trying to optimize for one outcome, which is to know when something is wrong and to remediate that issue as quickly as possible. And ideally before end customers find out, right, whether they're external customers or other perhaps engineering teams and whatnot. And I think optimizing for that outcome of remediating particular issues in their applications, really three questions that we're trying to solve and answer for here. The first of which is can I get notified and how quickly can I get notified when something goes wrong? Because if you don't even know when something goes wrong, or your customers find out before you, that's really not a great place to be. And I'd say the second one is triage, right? So once you do get notified, something is wrong, figuring out what is the impact. Is it impacting all of my customers or just a subset of my customers? Is it one cluster or another? If you get working up in the middle of the night, is this something I have to deal with now or can I wait until the morning? So triaging the issue and knowing the impact is a fairly important question that we need to answer. And then the third step or phase or question that we do need to answer is can I root cause this issue? Can I find out the underlying root cause of the issue and really provide a fix here for it? So I think those are probably the three steps or phases I'd say that developers go through in achieving their outcome, which is to remediate the issue as quickly as possible. And for myself and for us, the team here at Chronosphere, that's what we think about when we think about observability and that's what we think and users should focus on. We do hear a lot of definitions out there where it's more, I would say, concentrated around perhaps the data types like metrics, traces and logs and the three pillars per se. Those data types are definitely important for sure. But they are the types of data we need to answer the question and arrive at our outcome. The data types by themselves don't really give you observability or better observability. Just taking the three of them off and say, hey, I have logs. I have metrics. I have traces. Doesn't necessarily mean you have observability or great observability and producing more of each of those data types that also doesn't lead to greater observability. So we do think that an outcome-based approach for the end user, which is the developer, is perhaps a better way to think about observability as a whole as opposed to a data approach there. Yeah, it makes a ton of sense. I think everyone's getting hammered in with these three pillars, logs, metrics, traces, you got a checkbox, all of them. And it's sometimes we just forget about the user in those cases. And you mentioned the three steps or phases. Are there things you'd recommend to users going about to meet those or maybe you can help clarify those pieces a little bit? Yeah, we'd love to double-click on that for sure. So when I mentioned the three phases, they are, I guess, phases or steps in the sense that they are generally sequential. You have to do one before the other. You do have to get notified that something is wrong before you can go and fix the issue for sure. I'll say that the best way to think about these three steps or phases is that it's still, again, optimizing for the outcome, which is remediation. And it's not necessary that you have to go through all three phases to remediate an issue. So if you think about it, if you're mid-deploying your service and you get notified that something is wrong, first course of action is probably to roll back that deploy instantly, right? And that could be your way of remediating the issue. You don't know the root cause there, but you've remedied the issue. You've avoided customer impact, I'd say. And that's really what you're trying to optimize for, right? So it's really about that. And perhaps at the notification phase, you can remediate instantly. So there's no need to go triage and to recall the issue. You can sort of do that after the fact. And I think that's important as well. You don't really want to be doing sort of root cause analysis live during the incident with the pressure of knowing that the business is down or impacted or anything like that. Perhaps, and I'd say also generally issues are introduced when we change a system. When we leave a system alone, generally there's not as many issues. And I would say the highest percentage of causes of issues is when we introduce change to a particular system. So that's when you get notified there that that resolution or step to remediation is fairly quick. The second one is triage.  You haven't actively, you're not actively introducing a change to the system. So you do want to triage the issue and sense of knowing what is impacted, what is the impact to, you know, subsets of my, of my customer base, perhaps all of my customer base, perhaps, you know, you can isolate the issue down to one cluster or one availability zone or one region, right? And I think that helps, you know, how bad the issue is and how much sort of urgency you need to put into resolving the issue. But often we find that at that step, you can also remediate fairly easily without root cause analysis and can imagine, you know, most of our modern architectures are spread across multiple clusters, multiple availability zones, multiple regions. So a quick sort of resolution to that to remediation perhaps is to route your request around those impacted zones or clusters, right? If you know the issue is isolated to cluster A or to zone A, route your request away from that, such that again, you're, you're, you've remedied the issue and your customers are not impacted, you know, yet you have time to really figure things out, not under that time pressure. And then, you know, there are definitely occasionally those issues where, you know, you can't do any of the first two and you really have to get dug in and really dig at what the root cause is in production. Again, this is probably not the preferable one. You know, I'm sure the developers would prefer not to have to debug things live in production under the time pressure of actual impact to the business. But that does need to happen sometimes and when it does, you know, you really got to get dug in there, figure out what the root cause is, roll out a fix and remediate the issue that way. So, you know, they are sort of sequential and dependent on each other. But I think, you know, again, the outcome is remediation as quickly as possible. And from any of the steps, you can really achieve that outcome as quickly as possible. And I think that's the best way to think about those three phases there. Yeah, it helps a lot. And I think as you talk about remediation, is there going a little bit back to kind of the three pillars and the three data types? Or there are certain data types that make themselves or lend themselves to make it easier to remediate, right? Is like metrics or traces, is that going to help you move faster? Or, you know, how do the data types kind of relate back to this? Yeah, it's a great question. I think the short answer is yes. You know, there's obviously a lot of details in that. You know, I think when we think about the data types, and as I mentioned earlier, you know, just by having all three checked off doesn't lead to better observability in any way. And just having more of each data type doesn't really accomplish that or achieve that either. I think if you look at the outcome, which is remediation and the phases, there are particular data types that are better suited to particular phases, right? So if you think about notification, generally when you're trying to be notified or something, that notification is generally done in an aggregate view of all of your data or all of your requests, perhaps, right? So, you know, you're looking at how many requests am I getting per second? How many errors am I getting per second? What's the aggregate latency of those particular requests? So I think, you know, to answer or to help with the notification phase of the problem, metric data is generally a more optimal type of data to go and solve that problem. Doesn't necessarily mean it's the only one and it's the necessary one. But if you think about what you're trying to measure there, it's really an aggregate view of numerical data.  And generally metrics, which is, you know, values over time. And if you think about notification and alerting, you're generally checking those numerical values against a particular threshold. So, you know, I think metrics is perhaps better suited for the notification phase of things. Perhaps triage a little bit as well. If you think about triage, you're really trying to dig at one level deeper into the error count or into latency a little bit. And then perhaps having labels or tags on your metric data to better slice and dice by a particular cluster or an AC or region can help you with triage there for sure. And the actual individual request itself perhaps is not as required for those phases. But as you shift later into the phases and the process a little bit into more steeper triage and root cause analysis, I think that does lend itself to data types like logs and traces a little bit more because logs and traces are, you know, by default a little bit more of the both have a little bit more information there. And I think, you know, as that transitions, there are definitely, I would say, lots and traces perhaps more efficient at solving those particular phases of the problem. So, you know, I think all three types and I'm sure they exist for a particular reason, all three types are definitely useful throughout the whole process for sure. But in my mind, it's a little bit about optimizing for the particular phase. And now this doesn't mean that you need to have all three instrumented, right? And I think this is what is great about, you know, that the announcement earlier today about FluentBit and sort of extracting metric data off of logs because quite often it doesn't mean that you need to go an instrument for all three types of data but converting between one type to another type to optimize for the use case is a great advantage, I guess. And, you know, again, really happy to see FluentBit and FluentD go down that path in enabling those use cases. Yeah, let's talk a little bit about that. You know, I think, you know, we have a couple of sessions at FluentCon that are going to talk a little bit about more on the tracing side with FluentBit as well as just the metric side. We'd love to just get your thoughts about maybe metrics, logs, how these things all stem together and FluentBit's kind of new announcement there. Yeah, for sure. So, you know, I think there will be a session later today from Mike at Neymar Marcus. And if you look at that session, I don't want to ruin his session by any means but he was already, you know, via a custom plugin already generating metrics off of logs, right? So this is already happening even though there is first class support now in FluentBit which I think is great. Users were already doing this out of necessity. And if you look at Mike's use case, it really is to extract metric data off of the log so that he can alert off of this and get faster notifications because, again, metrics is perhaps a better optimal method of that phase than logs is, right? So, you know, I think this is already something that end users are leveraging already that the need is there already. I think what's great about the announcement and correct me if I'm wrong here, Anurag, but I believe the capability of extracting metrics from logs has existed in FluentBit for quite a while but the big announcement today is that it's happening in the metric extraction component. It's happening in Prometheus format and I think that is also really great for the industry as a whole, right? And if you take a step back and look at the monitoring and observability industry has been a huge shift towards open source standards, right? So, you know, for FluentD is the graduated sort of CNC project for logging, same for Prometheus for metrics and I think that the best part of that isn't that there is a solution. It's great that there is a standardized solution but that the standardization around the format of the data there, right? So, if you look at all of the projects in the CNC ecosystem, most of them are, in fact, if not all of them are emitting metric data. If we take metrics as an example, emitting metric data in the Prometheus format already. And I think that is hugely advantageous for the industry as a whole because it means from an end user perspective, from a developer perspective, you're not locked into one storage solution, whether that's a storage solution you're hosting yourself or whether that's like a vended storage solution, right? You can instrument in one way, in one protocol and every solution out there sort of supports it, right? And again, from the metrics perspective, if you look at the back ends, there's a lot of different solutions out there in addition to Prometheus. I mentioned M3 is one that we have been sourced out of Uber, but Cortex and Thannos is there and available as well. And I think the sort of movement to these standards and the power of that movement is also seen by all the vendors that are providing monitoring and metric space solutions as well in the sense that they all have to support Prometheus as a protocol now as well, right? And again, all of this, I think, is great for the end user and the industry at large because you're not locked into one technology as a back end or you're not locked into one vendor as a back end, which I think is great. So it's great to see that, you know, that ability is supported at first class now in Fluent Bit, but also that the exposition format is in the industry standard, which is Prometheus. And that's great to see, I would say. Awesome, awesome. Yeah, I think that's how we were thinking about it from the Fluent D side, the Fluent Bit side. How do we just conform with the standards? And I think, you know, a big upcoming project in this space is definitely open telemetry. And we announced some some earlier stuff today where we're saying, hey, we're going to have some integrations going on with the protocols they're building. We'd love to get maybe your take on the approach of that project, these projects together and just maybe your take on open telemetry. Yeah, we'd love to. I assume most folks watching this call are somewhat familiar with open telemetry, but if not, it is a collection of APIs and SDKs, again with the goal of standardizing the protocols and the clients that is generating all of this observability data. So, overall as a whole as a project, I love it because it's pushing the industry towards more open source standards for sure. I think if you look at open telemetry and the project really started off around having sort of standard client libraries for just retrace data, it expanded over time to include metric data. And I believe then, you know, the natural progression would be to expand it even further over time to include log data as well, right? And if you look down that path and they have, you know, similar, again, APIs and SDKs across all of the major programming languages, which, again, standardizes the instrumentation and the production of the data, which I think is great. If you look at it as a project, you know, it has support for tracing right now. Metric support is being added actively right now. And I think log may be coming soon. And I think that's going to be great for the industry moving forward. I think, you know, perhaps in a year or two, or perhaps even sooner than that, you will see a lot of the applications we are writing instrument in open telemetry from the beginning. And that's great. And in fact, it's not only just the three protocols there, it's a single client for all three types of data, at least two types of data right now, perhaps the third type down the line. I think that's great for new applications moving forward. But I think if you look at it from a practical lens of the things that we need to monitor today, there is so much existing instrumentation that is pretty I would say impractical to go back and re-instrument from a company's perspective for existing custom applications that have written themselves. Or sometimes it's impossible because you're pulling, you know, a dependent library or an upstream project that you're using and you don't really even have control on how those things are instrumented. So I do think that, you know, open telemetry hopefully is the future and that is the standard there. Eventually, I think taking it from a practical looking at it from a practical perspective I think projects are a great here because there is a lot of sort of back good support for existing protocols and existing instrumentation that is this today and it's tackling the problem not from a client perspective but from a processing perspective outside of the application itself.  I think it's one that's going to be required as we sort of handle a transition that's going to be for multiple years. And I think, you know, with this design of FluentBit and FluentD where you're processing outside of the application itself, lends itself to have other advantages as well. So one of the companies we work with, they're called Tecton, when we talked about their use of FluentBit, what they were using it for was to actually augment the stream of log data that was coming out of the application itself with additional metadata around the environment that it's running in. So they were augmenting it with the cluster and the namespace of the Kubernetes cluster that they were running in. And I think that is a hugely powerful thing to be able to do. And I think they were using kind of the exact feature name, I believe it's called the rewrite tag feature or something like that in FluentBit. But I think, you know, that adds a bunch of fairly powerful additional value as well in the sense that, you know, now you can sort of standardize the additional metadata that you add to the streams which is always a hard problem to solve, right? You can imagine if you ask every developer to emit environment or the cluster name, who knows, you know, which way they're going to go do it in a standard format, there's going to be weird camel casing and all sorts of other things in there, right? So I think being able to do it in one centralized location is important. And again, sometimes if you look at it from the end user perspective, sometimes it's actually really hard if you're an application developer when you're writing your application and instrumenting your application. It's actually really hard to even know, hey, which cluster am I going to be running in? How am I going to go get that data? It's actually something that may not be possible from the application developer perspective. So I think this approach from FluentBit of sort of processing all of the existing streams of data coming out adds additional value there and unlocks a bunch of use cases for sure. And as you mentioned, I believe there will be support for all the protocols that open telemetry are going to be crafting and sanitizing as well. So, you know, sort of it's not an either or thing. It's a different pattern. I think both will really help the industry as a whole moving forward. Awesome. And I think both of us would probably agree that observability has changed significantly in the last three years. New projects, new protocols, new standards, you know, as someone who's at the forefront of this. What do you think the next three years hold? Like what is maybe the future in Martin Mao's mind of observability? Yeah, I'd say the future is always hard to predict for sure. But yeah, let's definitely have a crack at it. You know, I think if you look at the future, what I believe is, you know, this trend that I talked about at the beginning where every developer adopts this observability mindset, I think will continue, right? And I think the trend that there will be a huge transfer of both knowledge and skill set from that core SRE team, from the experts in these practices today to all developers everywhere. And I really hope that that transition continues to happen. I think as that transition happens, and this is hopefully as well as that transition happens, there is a focus on the outcome as opposed to the input or the data types as well. So I do see that happening over the next three years. So, you know, having the developers optimize for the outcome, which is remediation as quickly as possible. And I think if you assume that that is the direction things are moving in, I think there are a few implications of that or a few outputs of that. One of which is I think you're going to see a lot more of what we're seeing already where there is conversion between the three data types to optimize for the various phases that, you know, and really the developers are going to be optimizing for the various phases here. So do you think you'll see a lot more of this what we've seen today already of like transferring the data types to solve a particular phase and to remediate as quickly as possible. But not just that. I do also think that, you know, as part of this shift, I think there is also going to be, I think, better context being passed between each of the phases are there as well. So, you know, going from notification to triage to root cause analysis going through the three phases there, I think there'll be a focus and sort of innovation on passing more context throughout each of those. And, you know, one example I can give here is my co-founder Rob Skilinton gave a talk a couple of years ago at CubeCon where we showed how you could jump from a metric data point on a dashboard which you would use for notification and triage straight into the underlying request in the distributed trace system which you would use for root cause analysis, right? So really trying to not begin your search again as you go through the phases but really take the sort of effort you put in from each phase and then sort of use that more effectively in the next phase. So I do think that we'll see, you know, more things there. And actually I think that we'll also see sort of better integration between the tiers as well, right? And by tiers, I mean infrastructure and the application tier, right? So you can imagine and I think you may have alluded to this earlier today that, you know, that there are some plans on de-influent bit to also sort of collect infrastructure stats or infrastructure metrics from the hardware itself in addition to the streams of, let's say, log messages you're getting from applications, right? I think having that in one place perhaps unlocks a bunch of potentials as well in the sense that, you know, you can imagine if you know that there is a particular spike in infrastructure in CPU or disk usage around a particular time and you know which applications are emitting data from that particular instance because you're processing in a single place you can imagine that there could be better correlations there and that could sort of unlock a lot of value as well, right? So, you know, I think all of that will be sort of outputs of hopefully this sort of focus on outcome and driving better outcomes for the end user developers. I do think that there are going to be sort of other implications of this as well and I think one of the implications is that the amount of observability data that's going to be produced is going to continue to grow up into the right and I think that's going to probably grow and outpace the growth of infrastructure overall, right? Because as more developers adopt this mindset, there's going to be more instrumentation and more data produced and it's going to be great because, you know, we needed to get to better remediation but I think the unintended consequence of that is a lot more data being produced and, you know, it's not something that's just limited to, I would say the monitoring of the observability space we see this in large data as well and other industries as well and I would say, you know, that has implications for, you know, the central observability team or the SRE team that is managing and running all of the infrastructure and all of the observability tooling that the rest of the developers use and depend on and there's probably two large implications there. The first of which is I think that as the observability tooling becomes a more of an important tool, I would say in the toolset of developers the reliability of that system is going to become more important and, you know, this is coming from my experience at Uber where we built a hugely powerful metrics backend storage yet, you know, we couldn't prevent a single developer from writing a single line of code that inadvertently emitted high cardinality metrics or inadvertently emitted, you know, you can imagine a lot of log messages and took down the backend and impacted every other developer in the company, right? So I think there's going to be a lot more focus on reliability of those tools just because there's going to be a larger dependence on those tools. And the second of which is, you know, I do think that the monitoring data is as I mentioned earlier is going to outpace and grow at a much faster rate than, you know, our spend or our use of infrastructure and I think at a certain point, the central observability team or the SRE team is going to have to focus on how to, you know, implement best practices for the developers to understand the implications of the instrumentation and sort of optimize this data that's been produced to still, you know, solve the problem and optimize the outcome but perhaps not in a way that's like, hey, just produce as much data as you can and sort of hope for the best. So I think there will be a lot of focus on how to do with that side of the problem as well looking forward. But yeah, that's probably my best guess that we're going to see in the next three years here. Awesome. Yeah, I know I think everyone's going to be a part of it, right? If you're watching this, you are probably at the forefront of observability so really appreciate your answer and your honesty, right? It's maybe an expensive future. So with that, I think yeah, we can go ahead and close up and thank you so much again Martin for your time and your insights and we'll chat again soon. Yeah, thanks for having me today and hopefully next time we chat we can do it in person, fingers crossed.