Hello, everyone, and welcome to Open Observability Talks. I'm your host, Doton Horvitz. And here at Open Observability Talks, we talk about anything DevOps, observability, and open source. So may the open source be with you. I'd like to thank our sponsors, Logs.io, the Cloud Native Observability Platform. Logs.io takes the best-of-breed open source projects, such as Prometheus, Open Search, and Jega, and offers them as a unified observability platform built for scale. For those joining the livestream, stream or on YouTube or Twitch, feel free to share questions and comments on the chat. It definitely makes things more interesting for us here on the fireside chat. And let's move on to today's episode. Last episode I discussed here, the challenges of monitoring Kubernetes operationally, things such as configuration complexity, high churn rate, et cetera. Today, I'd like to talk about the challenge of monitoring your Kubernetes spend. With the current financial climate, cost reduction is the top of mind for everyone. IT is one of the biggest cost centers. And companies realize that they simply don't understand the cost of their Kubernetes workloads or even have observability into basic units of cost. So we'll discuss this and the Phenops discipline for addressing this. And there's also a fascinating open source project, OpenCost, which aims to provide an open standard around that. And for this topic, I invited Matt Ray, who is the senior community manager for the OpenCost project, is also a veteran in the open source and DevOps communities, and also a fellow podcaster. Let me invite Matt to the stream. Hey, Matt, good morning. Good morning. Thanks for having me. Thanks. Glad to be here. And thanks for taking this live stream so early in your time. You're based in Australia. So it's like 6 AM now, right? Yeah, yeah. Yeah, it's early, but, you know, Montreal never sleeps. Yeah, I'm based in Tel Aviv and working a lot with the US, especially West Coast. I know the challenges, but with Australia, it can be even more interesting.  Yes. A small anecdote, by the way. Last week, I delivered talks in Belgium at FOSDEM and the Config Management Camp. And there, I ran into your co-host, Michael Cote. So I hope he's with us today on the live stream. I did invite him personally. So maybe before we go into today's episode, do you want to tell us a bit about the show? Sure, sure. So software-defined talk is a podcast that I formed with two of my friends, Brandon Wichard and Michael Cote. Each of us has kind of a different background in the enterprise software industry. Cote is, he's coming from, he's currently in like kind of marketing, but he's got a background as an industry analyst. He worked for Redmonk and 451 Group. He also worked at Dell and Mergers and Acquisitions. So he's got kind of an interesting industry background. And then Brandon has been a product manager for quite a while, different monitoring platforms. I think he worked at Boundary back in the day. He's worked at OpenID or identity-related startups. And we all work together at BMC, one of the granddaddies of monitoring. And my background is engineering and community and developer relations and kind of down that path. And so the three of us have different kind of viewpoints on how the industry works. And we've been podcasting for about seven or eight years now. And I think we hit episode 400 last week. So we're still going strong and we all live in different sides of the planet. So it's fun to bring that in. Cote lives in Amsterdam. I live in Sydney and Brandon's back in Austin, Texas. That's amazing. This show has been around for, this is the third year, but looking at more veteran shows such as yourselves is definitely a lot to learn there. So great to see that. And hopefully my followers that like podcasts will definitely find interest. I highly recommend that. And let's talk about Phenops. That's a hot topic these days. As I mentioned in the opening and with the potential recession, many organizations these days are looking for what they're spending money on and definitely cloud and infrastructure cost is typically the second highest line item after the salary cost, I think. So it's definitely top of mind for everyone. So before delving into the details, let me let's start with level setting the basics. Can you help us figure out what Phenops is all about? Sure, so Phenops is, well, there's the Phenops org, which is a foundation under the Linux foundation. So they're open source without being code-based. It's a group that came together to kind of talk about the intersection between cloud finance and operations. Everybody is, not everybody, but most people are starting to run a lot of operations in the cloud and it's a different cost model.  And so from the finance side of the house, that's a really different model.  That's kind of where Phenops lives. And so nobody is, it's a new practice.  And I was like, no, you keep moving forward. The jobs of the future you don't know about today. And if you told me when I was a kid that I would be working at an open source, financial operations, monitoring platform, I would just look at you cross-eyed. And so it's a new thing, but they put on their first conference in 2019, they put out a framework, if you will, that kind of explains what they consider, how you think about these things, what you need to consider, what principles you need to adopt, things like bringing observability into your entire stack, bringing in all the stakeholders in. It's not just engineers, it's not just finance. It's lots of different folks in between. And how you track these things, how business, it makes decisions.  What do you need to do?  It's a great framework if you haven't heard about this. And if you haven't started and you're in the cloud, you should start. Yeah, I think, first of all, definitely. And it's very easy just find it in FinOps.org. So it's very easy to find it online and lots of useful resources. It is under the Linux Foundation, but as you said, it's not about code rather than about, let's say, principle and guidelines. And what I found, not specifically just in the FinOps Foundation, but in general in applying FinOps. We, for example, at Logs.io where I work, we have a designated FinOps team. And when I try to explain to other people or to newcomers or others what it is about, I emphasize always that it's about maybe first and foremost about communication and about culture. That's for me, the essence about maybe breaking silos, obviously providing visibility and observability into the cost units, but also creating this ongoing conversation about cloud costs, loop these costs into business decisions and getting people to talk together, bringing business and finance side of the house to talk with engineering or with product and the normal circumstances, these organizations don't communicate so smoothly together. They also think differently, engineering with the agile, with the quick, with the fast, with the break fast, fail fast and finance with the very long formal processes of both both sides of the house need to adjust to make this happen. So that's for me. And maybe one more point that I found useful is the core principles, like the six core principles that I think are very good for those who are just starting their house. Like the collaboration is obviously one of them and ownership is another very important thing that engineering can't just say, okay, we are about building the software, someone else to take care of the cost and the infrastructure elements and things like that. Now it's an integral part, accountability, ownership built into the, baked into the organization. Obviously the observability side of things that comes with it, you can't take ownership if you don't have very clear reporting and the dashboarding and ways to see where things stand. So all of these I think principles are very useful, especially for those who are new to that. And even before getting into Kubernetes and things like that, whoever uses SaaS and uses cloud, definitely highly, highly right now. Yeah, and they've got a great O'Reilly book called, you know, Cloud Fan Ops. And second edition just came out like a week ago, two weeks ago. So if you haven't bought that book yet, you know, definitely check it out. It's by a bunch of the different authors or the authors, most of them work for the Fan Ops work at this point. And so, you know, I highly recommend that just to anybody who's getting into the space.  And we mentioned several stakeholders involved. I mentioned like business and finance and engineering program. I'm curious from your perspective, who are the core stakeholders that you see involved in these processes? Well, so, you know, to clarify, like my day job is I work for a Kubernetes cost management platform. So we are on the more technical end of the spectrum generally. You know, a lot of my coworkers have come from some of the larger, more established cloud financial platforms.  I mean, you know, as soon as Amazon started letting people do S3 and EC2, you know, bills started showing up and people started having conversations with finance.  Everyone's trying to work that out. Most of the people I talked to before I, you know, changed roles within KubeCast to the open cost side of the house is I was working with larger, our larger customers, you know, customer spending million dollars a month on AWS, you know, that sort of stuff. And mostly, I was seeing folks from the engineering side, you know, they had been told that they were spending too much and they needed to get a handle on what was going on or they were a large enterprise where lots of different teams were consuming cloud resources and they needed to sort out, you know, charge back or showback.  And so the nice version of it is showback where you show who's responsible for, you know, hey, you know, team A is 60% of the usage and team B is 40% and chargeback is when you actually have a established budget. It came from the days of enterprise software where you had all this, you know, compute internally and people had to share. And now that, you know, you have to share the bill for an external course, it's chargeback.  Engineering generally doesn't pay close attention to their bills there.  Nobody thinks that way. And so initially, just like the FedOps model talks about, you know, crawl, walk, run, we like to just bring in some observability, get people looking at what they're doing and, you know, you don't have to, you know, you don't have to have chargeback. You don't even have to, you know, it's showback.  And in some cases, we even call it shameback.  You know, probably not. But, you know, it is costing $1.30 an hour. So, you know, it's just getting people comfortable with the idea that everything you're doing costs money somewhere. And so, you know, we like to bring that cost monitoring into the conversation. And how do you, beyond the tooling, I know that you're from the tooling side of the house, but looking at it from the end user perspective or the driver, the agent within the organization trying to drive this awareness, how do you create awareness amongst engineers to cost and to spend? Usually someone outside of engineering has noticed the bill, right?  Is this gonna continue every month? You know, is this, what's this growth look like? And so, you know, somebody who's responsible for that bill goes over to engineering management usually.  They started at the management layer and say, you know, this is, you know, can you justify this? They're not saying stop it, you know, because clearly the business is there to deliver some purpose, you know, IT is not just running servers because they like blinking lights. They're there to, you know, deliver value.  Can we see what's going on?   And if it's not, we'll come back and revisit this, but it's usually coming from the business side. They've got concerns, you know, and if you're in a small startup, everybody's on the same team at the beginning, right?   And maybe it's just fine, but I can tell you, when I look at customer's bills, it's not fine. They're, usually there's a lot of waste, what we call it, we call it idle, right? Because you've, with the cloud, there's kind of two models of consumption. There's usage-based. So if you're doing S3, you know, well, S3 is not a great example, if you're doing like Lambda, right? You pay by the usage. You pay, every time you make a call, you pay some fraction of a cent.  And that's great. But the other, the more common model is, but based on what you've allocated. You know, I say, I need to run 15 EC2 instances, and they're going to run 24 seven for three weeks. Well, I don't get to say, well, you know, they weren't really busy for part of that time. So I'm not gonna pay the full price. Like Amazon doesn't care. You have those machines allocated to you, you will pay the full price. And so if you're using them 100%, that's great for you. If you're using them 5%, that's great for Amazon. Cause they're gonna resell that capacity to someone else. But, you know, you're paying either way. And so what we would look for is that unused cost. We wanna optimize, we wanna optimize for usage. You know, if your usage is spiky, well, you know, sometimes you have to, you have to give yourself some headroom. If your usage is flat and you're paying, you know, a lot for hardly anything. Well, you can optimize that. And that's just one form of optimization. There's lots. Yeah, obviously. And you mentioned Amazon in general, the cloud providers, they do provide, first of all, you get the bill and you have some breakdown in the bill and they have their own cost tools such as, I don't know, AWS cost explorer, I think such as that. So what can you do with these and where do they fall short in your perspective? Right, right. For most people, the Amazon bill, there's two bills. There's the one page bill, which nobody really likes that bill.  And there's no breakdown. And then the other bill, it's called the Cost and Usage Report. And that is the very, very, very fine grained JSON report of everything that you do in EC2 that costs money. And Amazon drops it in an S3 bucket for you and you can consume it with the tool of your choice. And all of the different billing tools that read your bill, they're gonna look at this and this is where it has your discounts. You might get savings plans or reserved instances. You might have something, discounted pricing that you've negotiated with Amazon. Well, I keep saying Amazon, but Azure, GCP, all the cloud providers do this. They have very, very fine detailed complicated bills that need to be processed. And those aren't even readable. And so that's kind of the two ends of the spectrum for you as consumer. And then Amazon provides tools like the Cost Explorer. And it says like, look, your bill was, we'll just say a million dollars. And as you drill into it, you can say, well, half of it was EC2. And then you drill into that. You can continue to go down and see like per instance, how much you were spending, which days, what you were paying for, you've got some VPCs, some storage, all those things. And you can see all the details, but it's not machine readable. It's not something that you can easily integrate on your side into your visualization tool of choice, your visualization tool of choice, whether it's Grafano or Crystal Reports or Excel, you're gonna wanna consume that and put it into your financial engine or your monitoring engine. Both of them are end points that people care about. And so that's, and they don't know anything about Kubernetes, right? And you probably don't. You don't want Amazon saying, hey, business team A did this and business team B did that. You're like, stay out of our business. So they don't know anything about what's happening inside the nodes. So they're like, when you get your EKS, your Amazon Kubernetes bill, it's EC2 with a management fee. You don't know anything more than that for which namespace was doing what, which deployments were costing you. That's completely opaque to you as an AWS customer. From your bill aspect. And that's what we do. So before getting into the tooling, and just I want to clarify, so FinOps started around the cloud costs in general. And you gave some good examples. Some people don't even understand what they have like reserved capacity, how well they utilize it or when they do on demand and how the on demand there performs and the trade-offs things such as that. Lots to do there as well. But since we started touching about Kubernetes and this is the topic of today, how is Kubernetes spend different from the cloud spend we've done so far? So I mean, Kubernetes spend for most customers just part of their bill. Most shops are not 100% Kubernetes. They're gonna have some traditional workloads that are running on cloud instances. You got some Windows machines running in the cloud. That's generally not Kubernetes. You've got some S3, some databases, beanstalk, whatever it might be. That's not the Kubernetes side of the house. And so for us, we saw that most workloads were headed this direction and not most, but a significant portion of the market and the tooling did not really serve them. And the first edition of the Cloud FinOps book didn't really cover containers and Kubernetes really at all. Because I mean, there's a brief chapter on it, but it's brief. So we kind of saw this opportunity and that's where KubeCosta and later OpenCosta, the open source component came from is we wanted to make sure that everyone could kind of give visibility into what's happening there. So what's kind of different about it is it's just not a box that had been opened by most of the cloud tools at this point. Yeah, but I do think, at least when I try to analyze the way that I did that in this organization, previous organizations that I worked in, there are some different characteristics that I do see like the difficulty to track when you're looking at the cloud costs and the shared resources, allocating the spend, you gave some, maybe you alluded to that before, allocate spend to cost per customer, per team, per different environments, things like that or tracking the cost efficiency of your Kubernetes work cloud allocations over time across different aggregations. Can you say what you've been seeing with your customer? Well, Kubernetes changes the game. For a lot of shops, moving to the cloud was with lift and shift, right? They're like, hey, now we don't have to have a data center. And they just move their workloads, they're fairly static, not very dynamic workloads into the cloud, they clean things up and for some of them, they didn't get the savings they really expected in the cloud because they didn't really change their operations, they're just now in somebody else's data center. But if you've made the transition to more of a cloud native model, which is essentially what Kubernetes is where resources are allocated on demand. You've got some applications that are going to run for maybe a certain amount of time or they're going to run wherever it's cheapest. And they don't have a lot of, that they are potentially ephemeral, they don't have a lot of state that can be killed and rerun and moved around. When you start to get into that use case, billing becomes more complicated, but also you can save a lot more money. Kubernetes allows you to potentially condense the amount of compute you're using. So now instead of having one application per instance, you can say, well, I've got a cluster running there and let's deploy a hundred applications to it and the cluster can be resized up or down and those instances will remove, will move to different compute nodes as necessary to run. And just like virtualization saved, potentially saved a lot of money by reducing the bare metal count or condensing it into more powerful servers that cost less because you had fewer of them, Kubernetes allows us that option too. And from the billing side of things, that can be a nightmare because you've got your application today, it's running on node one and Kubernetes decided that it wanted to move it over to node two, to node three, to node four.  That application is just running all over the cloud, all over what you're paying for. But when you get that bill, it just says 15 compute nodes and you have no idea like, well, what did team A do? What did team B do? What was, how much of that 15, how much of that 15 machines was any of the particular team? And so that changing characteristics, it makes things a little more exciting, but potentially there's savings because you can look at it and say, well, we're paying for 15 nodes, but looking at the computer, the idle across the cluster, well, we've never passed the 20% usage. Maybe we don't need 15 nodes in our cluster, maybe we can get by with 10. And that's an opportunity for savings. Or we look at that and say, hey, our Kubernetes cluster has been up for three months. We've been running 15 nodes, we could reduce the size of it or we could call up Amazon and negotiate reserved instances. We're gonna go ahead and pay for 10 nodes a month at a 50% discount for the next year.  And so, and your workloads, they might be really dynamic, but the base infrastructure they're running on is statically priced. And so that's comforting on the financial side of things. When things become more predictable, even if on top of it, on top of your Kubernetes, you're deploying 30 times a day, that's fine as long as the EC2 nodes just stay there and get charged the same amount. Yeah, that's from the infrastructure, but still when trying to create this accountability that we talked about before and need to attribute this attribution model per team or per customer or per environment, then the fact that the deployments and namespaces and so on are not really isolated and they actually share the underlying resources that you use, mention nodes, we can also mention the persistent volumes, the load balancers and so on, then the ability to make this attribution and then make the forecasting, the capacity planning and also negotiating based on where you expect the business to grow in the different teams and different product lines and so on becomes maybe a bit more challenging, let's put it this way. For sure, for sure. Especially a bigger enterprise. And remember that some of this is justified from business perspective, like if it's a, I don't know, a spike in incoming requests for Rihanna's Super Bowl halftime performance this week and you're in the business of the media, then it's expected, the spike is tightly related to what you deliver to your customers, the value and hopefully you know how to monetize or this is the top line KPIs, it's fine. So it's different when the cost spikes because of a legitimate business need that arises, unlike cost spikes that are just bad utilization or someone just left the EC2 testing or machine learning model running or something. We've been malicious instances, right? When you see a cost spike in a namespace and you're like, why is, what's going on over there? And we've definitely helped customers find unsecured instances or applications that have been, they start mining Bitcoin and you're like, why did this cost you $100 over the weekend when it was a dev instance that wasn't supposed to be doing anything? And that happens too. It's weird to think of the finance as a intrusion detection, but there you go. Yeah, improved more than once. That's definitely one of the common stories that people discovered Bitcoin mining after employing simple phoenops monitoring. So let's talk, I think we talked about all these vendors and one of the challenges is also that they speak slightly different language. It's thought there is a way that you can also compare your AWS to Azure, to GCP or to other bills because there is no sort of common way of communicating which for me was the way for me to explain to others about the maybe the most basic value proposition of open costs. So let's move on to open costs. This is the new open source project, new kid on the block in the phoenops in the cloud native space for sure. That recently, by the way, been accepted to the CNCF sandbox. So a big congratulations to you and the team there. So can you tell us a bit about what open cost is about? Sure. So Kubecast started, I guess about four years ago now and the two founders had come from Google. They did some other work before starting Kubecast but they were already well familiar with the open source space and started Kubecast as the Kubecast cost model as an open source project and then added additional value on top of it. And but the intention was like, let's get this out there as the standard for Kubernetes cost monitoring. Let's, they started it and COVID happened, kind of disrupted a lot of plans. But in June of last year, 2022, we announced that the CNCF had elevated the Kubecast cost model to a sandbox project and they renamed it open cost. One of the things you're not allowed to do in CNCF land is have the company with the same name as the project. You got to give up your trademarks and stuff, which is good. And so in addition to the code base, they've been working with other vendors, other individuals, other end users on writing a specification for what it means to monitor Kubernetes for cost. How you identify different types of usage, whether it's idle or allocated. And so there's both a specification, the open cost specification V1 that talks about allocation monitoring. And that's the first pass of open cost is what it does is it goes and looks at the cloud API. So it says AWS, you have four EC2 instances running and how much do they cost per hour? And the API just says, list price is this and it takes that and then it compares that to your Kubernetes usage. It says, well, we've got five namespaces and we've got this many instances. These are our workloads or pods or containers and it lets you slice and dice that. So you break down those EC2 instances by all of those Kubernetes primitives and that's essentially what you get with open cost today. There's a UI to let you explore this. The data is stored in Prometheus. So if you have a Prometheus compatible tool, we could put it in a different back end if you want. The folks over at Grafana are storing it in the mirror. People use Thanos, Cortex, Victoria metrics, you name it there. Somebody's put it in a different Prometheus compatible back end. But open cost is, it's a CNCF project. So it's a patchy license. The goal for us with open cost is just make it the, ubiquitous default monitoring stack for cost. So as soon as you spin up a cluster and a Kubernetes cluster in any public cloud, you just throw an open cost on it to keep an eye on it and then put it on the dashboard of your choice. And so that's what we're doing with open cost. And who is the, today essentially it supports both on-prem environments and also a cloud managed Kubernetes. Can you mention who is currently? Yeah, yeah. So because this was the engine of Kube cost, it already came working out of the box with AWS, GCP and Azure. There is support for on-prem pricing. So you can upload or provide through a config map a default pricing. So you can say, Hey, in my data center, we charge a dollar for, we charge a dollar per hour per core and we charge $2 per hour for RAM. Some nice simple pricing. And that's what I did in my home instances. It's like, I don't have fractional sense. I just wanna see nice round numbers for my home usage. But it lets you set that pricing. There's support for more fine-grained pricing. You can charge on GPUs, because you may be consuming GPUs for AI or whatever. And so that's in there too. So that's what it shipped with back in June. And then the open source community has started adding other sorts of platforms. So we've got some patches in for Scaleway, a European provider. We've got some Aliyun, Alibaba's public cloud. That's in there. There've been conversations with some other providers. We've got a document for how to get started. Got very friendly Slack channel over in the CNCF Slack. So come ask questions. I'm happy to help you add your public cloud. And it's not that hard, because we're not digesting the bill. So I mentioned the cost and usage report as this multi-gigabyte JSON file. Every vendor does it differently. Those files come out, they don't come out in real time. So that's one of the weird things about cloud billing is you have your on-demand cost.  And you're like, okay. And then maybe 48 hours later, Amazon says, well, you did have some discount savings. You had a couple of credits. You had a reserved instances. It was actually only seven cents an hour. And that might change how you look at things. And so open cost, that is a lot of complexity. Going and doing that reconciliation, parsing that bill on-demand, finding the data in there. Open cost doesn't do that yet. And that's a, I can tell you from the Kube cost engineering side of the house, that's a lot of work. Most people don't, you know, from the open cost side of the things, most people don't seem to miss that. You know, we're really just looking at like, how much does this generally cost to me? And you know, the actual numbers are less important than the direction. Don't have to make sure that our listeners understand. So essentially it's not the actual bill that factors in all the credits and the discounting thing. It's sort of a relatively static mapping. It's the on-demand price. Whether from the managed cloud provider or if you map it yourself for your own friend. Yeah, yeah. So math on top of that, right? Right, right. It doesn't, it does not parse the final bill. It's going with the list price. And you know, the list price, for people like me, I pay list price.  For, you know, so a lot of small and medium businesses are not in any sort of negotiation. But even on places like Google with their continued usage discounts, you know, that would show up later. But that shows up like days or even weeks after you've spent the money. So you know, they're kind of retrofitting your bill afterwards. OpaCost doesn't do that. OpaCost is really looking at... You have to look at it by the way. It's a large engineering effort that at this point we're still, we're still looking at other cost sources. So, you know, right now OpaCost provides what you're, what's allocated for your Kubernetes. So the instances, the storage and networking, you know, we're giving you that cost just, you know, based off of your deployed Kubernetes cluster. Right now we're actually headed towards the direction of out of cluster costs. So you've got a remote database. You know, the database is a service that you're integrating with. You know, Kubernetes doesn't actually know anything about that. But if you, you know, want to incorporate that bill, if you want to see that with OpaCost, that's what we're headed towards, is bringing in external asset costs.  Well, that's where we're headed, which is actually different from what Kubernetes does. So we're kind of diverging because open source is usually about people scratching their itches. And the itch that most of our users have is I have other costs that I want to bring in.  It's just not source. Yeah, so KubeCost is free for single clusters and it gives you 14 days of storage. So you can deploy all the KubeCost you want. And, you know, it's free until you want to do, like, federated views and, you know, more storage and stuff like that. But maybe someday they will open source their bill processing engine, but that is a enterprise beast of its own. And, you know, on a different podcast, I heard an interview with the product manager for Amazon's billing engine. And he said that he's pretty confident that the Amazon billing engine is the largest non-government billing, non-government software project in the world. I mean, think about that.  And every run is running their workloads on AWS and they're generating, you know, hundreds of thousands, if not millions of metrics per second. Or no, no, no, no, per customer.   And so, you know, bills are very, very, very complicated. Yeah, I can tell you, you know, my company is on the larger end of the medium size, let's say, and from what I see, the small enterprises definitely that all these packaging and all these crediting math can definitely make, move the needle and change entirely the bottom line. And you have your dedicated person, whether you're an ISV or a medium size or enterprise, but so I'm just wondering, but you're saying that the pain is less from the community at least, the demand comes more on the PaaS side of things, like platform pieces that they incorporate for data or others. Are there any other items on the roadmap for an open cost project? I mean, definitely, you know, one of our goals with OpenCost is to move out of the CNCF sandbox. So, you know, it's been open source for just about six months now.  Last week, we announced Grafana Labs is now a contributor. You know, they are deployed on thousands of clusters. I mean, I don't know if that was their number, but I've heard the number of, you know, thousands of Kubernetes clusters thrown around, some of them very dynamic. And so they just deploy OpenCost under everything, just as, you know, a monitoring agent to get visibility into all the costs. And they're pumping it into Moomir, which is, you know, their storage backend.  And, you know, so they've shown up, started contributing, making OpenCost more efficient. And part of the roadmap is, you know, what sort of stuff is appealing to those sorts of shops. They've got, they've brought some things. We've had, you know, folks from other cloud vendors shown up. They want to have, you know, better support for their clouds. You know, that's always on the roadmap. We, you know, OpenCost is slowly diverging from KubeCost.  And so part of the roadmap will be things like, you know, taking, you know, taking these open source contributions, getting our own release cadence. And as we have more external contributors, more, you know, different documentation, we can move up the CNCF ladder, you know, graduate out of being a sandbox to an incubating project. And part of that is just having more external folks involved. And, you know, it's still early days, but we're having, you know, a good, healthy share of contributions from outside. Open telemetry is something that has popped up. There's some open issues, features for that. People would like to see OpenCost implement that. And that's the sort of thing that, you know, OpenCost can do faster than KubeCost, because we're a much more, you know, we're a much smaller, more efficient project. We just, you know, we need more community folks to get involved. That's amazing. And then glad to hear that you have some more people outside of KubeCost. So it's KubeCost people, but now also Grafana Lab people, any other major figures that are involved in terms of entities that got into the project? Yeah, yeah. So when we launched OpenCost, you know, I mentioned it wasn't just, it wasn't just KubeCost, you know, open source and something. We had a, the specification. We had folks from Adobe, Armory, AWS, DT, IQ, Google, you know, New Relic, SUSE, Pixie, Red Hat. You know, so those folks were all involved with the specification.  And so some of those shops will be taking the specification and releasing their own implementations. And so, you know, part of that is, eventually we'll need to form like an acceptance criteria framework, you know, that looks at your implementation, ensures that you are implementing the API. You know, so it's got all of those, you know, scaffolding needs to happen. You have some of it, some of it's in place, some of it's in progress and, you know, different. And now, and today, like as we're working on the external allocation costs, you know, their different contributors are getting involved, which is great to see. And it's already, you mentioned that there's a young project, but still beyond KubeCost it is implemented in others, I think in each AIS and others, right? So let's. Yeah, yeah.   But I'm hoping to get more names to publish. We've got a couple of partners who have, you know, added OpenCost APIs to their products. So I think Vantage is one of them. I'm just going blanks right now, but you know, look to see more OpenCost compatible.  Well, they, you know, they'll put the OpenCost APIs on there so you can pull your financial data out of them. Whether it's, you know, actually provided by OpenCost, the project, who knows.  Exactly. And I think you mentioned OpenTelemetry, it's a great role model to follow. And if you can also converge and do something between the projects, it's definitely going to be a force multiplier. Before we wrap up the main part of this show, can you share, you mentioned briefly before some of them, but how can people join the community conversation, learn more and get involved? Right. So, you know, the primary, the CNCF runs a Slack. So if, if you're a member of the CNCF Slack, join us over on the OpenCost channel. OpenCost.io is the website. GitHub.com slash OpenCost is where the project, the website, the helm chart, those are currently there. We have a calendar where every, every two weeks, every fortnight, we have a working group, which we kind of gather to talk about, you know, we have an agenda like, you know, what we're working on, what, you know, we need help with, you know, what people would like to see. Some people show up and they just, you know, want to talk about their issue. And some people say, you know, hey, let's, let's start this external asset working group. So we're going to have a group kicking off a new specification. And a, an example project, those are, those are about to kick off. So if you'd like to see external asset costs get added to OpenCost, you know, join up, I think, I think we're going to implement as three as the example. But once we're done, we'll have a specification and documentation and, you know, and a working example. So you can add whatever it might be that you'd like to attract in your cost monitoring and, you know, and then tie it back to your Kubernetes usage. That's, that's what we're doing over in OpenCost. Amazing. And just a note for the listeners, if you're not, even if you're not a member, an official member of CNC, if you don't need to pay anything, the Slack channel for CNCF is open for everyone. You can just open your user there. And once there, you have all the channels in the world for all the projects, one of which is OpenCost, has OpenCost and you'll be there in the conversation, but don't think that you need to be some sort of formal member or something open for everyone. And you should also join the FinOps organization. So FinOps org, they have their own Slack. There's not an OpenCost channel, but there is, there are, there's a lot of channels for different clouds, different tool sets. They're working groups for things like OpenBilling. There's a Kubernetes and containers working group that obviously we're active in. And so OpenCost is, is the only FinOps certified project and CNCF project. So we're the, we are the intersection of those two worlds. And so, you know, definitely join either or both Slack's and I'll see you there. Yeah, sibling organizations under the Linux foundation. Yes. Great, that was fascinating. And with that, I'd like to wrap up this part and with the few minutes that we have left to cover some interesting bits and some breaking news. And very happy for you to stay, stick around with me for these parts. You probably have some interesting insights, especially with your perspective and familiarity. The first one I wanted to share actually is something that I have been working on recently. It's I call it the metrics essentials trilogy. It's three articles that are meant to cover some, a lot of the common topics that I keep on encountering with users, with community members, with customers and others. So it's, I called it the phantom metrics, the expensive metrics and the unreadable metrics. Phantom metrics is why your monitoring dashboard may be lying to you a bit about the basics of how it works and what to expect of it and where the monitoring will not show you exactly where in real time happens and to take it with a relevant perspective. The expensive metrics is maybe somewhat tied to what we talked about today about why your monitoring data and build may get out of hand and associated costs with it. Like cardinality problem and other that. And the last one is about maybe some guide to effective dashboard design for DevOps type monitoring. So you're more than welcome to check it out. It's on medium and very happy to, some of that by the way ties back to topics that we've covered on this show, like with Ben Siegelman about cost of monitoring and others. So definitely if you follow this show, it will resonate, but I think it's a good summary and do share some feedback. Glad to make it a starting point for a broader discussion. The next one is a CNCF blog about what's new in Prometheus ecosystem. Things such as the agent mode, native histograms, newly added service discovery mechanisms, the prom lens project has been contributed to Prometheus and much more. So it offers a good run down list and I highly recommend you check it out. Also I highly recommend checking out the episode we had here on the show recently with Julian Pivoto, which gives much more in depth, but this blog on the CNCF blog is definitely worthwhile checking out. Another thing we mentioned here before about open telemetry. So I saw on the CNCF blog a very interesting post about migrating from open tracing to open telemetry. For those who are not familiar, open tracing is a deprecated standard, the API specification that existed before and was then merged into open telemetry together with open sensors and some other pieces. So many older implementations that used to run on open tracing now needed to migrate to open telemetry. This was a very good walkthrough on what you need to, in order to do that in a pragmatic way. Matt, have you had a chance to do some sort of a migration such as that or play around with that? Not yet, but I already had that tab open. So I'm definitely going to read that article. Yeah, it's a good one. It used to have a shim that made it very easy, but then again shielded from the real migration work that needs to be done. So I definitely advise not to take the shim path. I think even the shim is already sunsetted and not supported anymore. It's important to understand it's been, it diverged, it expanded far beyond open tracing. I mean, open telemetry project, API specifications are definitely worthwhile doing a deep migration path rather than the shallow one. So it's a good coverage on that. And also some CNCF project updates in addition to the good news around the open-cost project. We had some good news end of year about some projects like I think the most prominent ones were Argo and Flux that have graduated from the CNCF incubation. So they're now in the graduated state. And some others next month on the episode here I will have Chris, CRA as most of them know them, the CTO of the CNCF will be here with me on the show and we'll definitely be talking about the changes, the project landscape and some of his predictions. So do join us on next month's episode. I promise it'd be interesting. He's an interesting guy. And Matt, anything else that you found interesting this week or in the past 10 months? Well, I just wanted to point out open cost. We're going to be at the Southern California Linux Expo next month. So if you are in the Southern California area, definitely show up at the conference. It's North America's largest open source community conference. Scale 20X, right? Yes, Scale 20X. I'll be giving a talk. And the open cost will be sharing a booth with KubeCast. And so hopefully I'll have some stickers there if you show up. And of course, I look forward to seeing everyone at KubeConnyU if you can make that. Yeah, I'm definitely going to be there. So I look forward to meeting you finally in person now. Yeah, it's a long haul from save, but I'll be there. Yeah, sorry. Yeah. So do check it out. And by the way, there are lots of co-located events on the first day. It's been reshuffled on the, the CNCF did some bit of an organization there because it got a bit out of hand with all the colos. So now it's consolidated. For example, all the Prometheus Day and the Hotel Day and others now are consolidated to one open observability day. Not to do with open observability talks here at the show, definitely touching upon the same, the same topics. So it makes it easier just one day of all the colos together. So highly recommended also to check out the relevant colos. And look forward to seeing you, Matt, and all the others. They're highly recommended. That's great. So thank you very much, Matt. How can people reach out to you after the show? Yeah. So if, if, if you're not in Slack, if you want to get a hold of me on the email, I'm Matt Ray at kubicost.com. You know, I'm on LinkedIn and Macedon. I kind of stopped using Twitter, but I'm Matt Ray and, and those places. So I'm usually pretty easy to find Matt Ray on GitHub. So I look forward to catching up with folks. Amazing. So thank you very much, Matt, for joining me on this early time there, Australia time. It was a fascinating talk. And thank you, of course, all the listeners who joined us today on this episode, all the episode as always are made available on all the favorite podcast apps or on YouTube. So do check them out. And if you are listening to this show, to this show, to this episode on, on the, on demand, then do know that we stream the episodes live on Twitch and YouTube. So just find all the details on openobservability.io or follow us on Twitter at openobserv for updates on the next live streams and to share your comments, suggestions, news bits or anything else. And if you have something specific that you want to talk about on the show that you think that your subject matter expert on these relevant topics do feel free to submit a top proposal on openobservability.io. I'm Dutan Horvitz. Thank you very much for listening and see you on next month's episode.


So I'm Lakeham everyone. I'm here with brother Conry from Florida originally, but now is it Oakland? Yeah, originally from Florida. I've been out in the Bay area for about a year and a half now, so living in Oakland now. Yes, we're just gonna have a nice little chat. It's gonna be tough because I do know the brother. I have to pretend like I don't a little bit so that in Charlotte you all can as well and benefit from his story as I have. So, yeah, my Shalala is Ramadan. Kind of start from the end a little bit, but checking how's your Ramadan going so far? So so far Ramadan is going pretty good. It's my second year. Last year being the first first time I experienced it, I'm shortly after taking my Shalala. So this time around I just feel like you know it's a have some new goals. A better understanding of Islam overall and just really looking to make the most out of this Ramadan that I can like everybody else. It's like a growth in every step like growth in development. I say it's like a marathon not a sprint. Yeah. So it's just another lap in Shalala. One of many to come in Shalala. So you mentioned it's being like your second Ramadan.  We got you interested and your whole sort of spiritual journey that led you to your Shalala that in the parking light house. Oh, man. All right. So yeah, so okay, yeah. So I was born and raised in the Christian household. You know, my grandfather was a preacher and so I grew up, you know, always having a faith in God and understanding that it was a higher power that, you know, there was a creator something besides myself and, you know, growing up as you hear different stories or the way things were related to me personally, I've always had some questions about that. I was always very curious and as to, you know, deeper questions about the Creator, you know, what was the purpose of life, what that meant for myself, what that meant to my culture and as far as the way I looked in the world, how people display that and for me in a lot of instances, I saw a mismatch as far as what people kind of proclaimed to follow or profess and was really looking at their actions and I didn't really grow up knowing a lot of Muslims or being around a big Muslim community, especially in Florida. But with that being said, now that that kind of gained a deeper understanding and gotten in the community a little bit more, it may not be that there wasn't a community around, maybe my eyes just wasn't open to it, which I feel like translates to a lot of a bigger part of here in America as well, you know, sometimes it's easier to not see something if you're not really looking for it and so essentially fast-forward to, I would say, about college, you know, I'm just kicking it with my friends just like any other regular college students, you know, one of my friends being an atheist, not not really observing a higher power in his life. I had another friend named Ross who was Muslim and I was, you know, Christian but somewhere in the middle at the time and so we often had like many discussions, we would sit down, we'd kick it, you know, we might, you know, play the video game, play some 2k and and joke around with each other and also there was different things that were occurring around around this time, I would say about like maybe 2011, 2012 and interestingly enough, I believe the Trayvon Martin situation had just occurred around that time, which was, you know, social injustice and so we were all having a conversation about how we felt about that and how we were gonna process like our feelings and you know, how did we kind of, how could we kind of comfort ourselves and then continue to, you know, go to the same school and just put it in the context that situation happened about 30 minutes away from where I went to college at University of Central Florida and I noticed that through our conversations, man, I remember my friend who, for instance, who he didn't really observe a high power, I just remember him really being very angry in a sense and not really, you know, the way he talked about the situation and talked about coping with it, I kind of, I knew, I felt like, you know, that may not be the healthiest way, that may not be the healthiest route and even myself, you know, I realized at one point I, as much as I professed to like believe in God and to know that everything was okay, I just, at that time, I didn't, I didn't really feel that, I kind of was like, you know, it's something that, you know, I don't really know how to process my feelings and I remember my, my buddy Ross, he gave some really good insight and commentary and everything that he kind of said, he mentioned, you know, kind of came back to his faith and he often brought up Islam, but not in a way that was very like pushing it on you. It was in a way that was very like, hey, you know, hey, this is what we believe in and this is what, this is how we take this and, you know, the answer to racism and certain problems and differences is just the very fact, you know, it comes from the ayah and I don't want to misquote it, but it says something about how we were all made, you know, we were all different, we made you different so that you can get to know one another and for me, that's one of the first things that stuck out to me because at that time, I had a lot of like confusion about how I felt, you know, like, I wasn't to the point to where I felt like, you know, racist or anything like that, but at one point I felt like, well, you know, I don't know if I see, if I see, you know, a brother like yourself, I don't know if, you know, walking up and down the street at that time was like, you know, I don't really know what that brother might think about me, you know what I'm saying or I don't know what to think about that brother and Ross had conversations that surrounded around, you know, posing thoughts like, you know, well, why would you think that? What is it that's got you thinking that way? You know what I mean? And I felt like, you know, previous to that, I did do a little bit of research on like Malcolm X and his story.  What was it that, you know, brought peace to his life? And in a way, I read his autobiography and towards the end, the thing that grabbed my attention the most that made me want to delve a little bit deeper in Islam was the fact that he talked about the first time he went to Hajj and how he, you know, that was the first time in his life he, he genuinely made really good friends and could pray next to a, you know, a Caucasian brother or a Indian brother or a, you know, brother from Pakistan and all these different places. And he talked about it experience in detail. And for me, as a young man reading that book, that kind of stuck to me. It was like, hmm, so Islam is that common factor, you know, and then fast forward, should I just fast forward a bit? Because it, so I say, should I go with the flow, as you feel it, we can double back if we need to. Yeah. So a lot of, so there was a couple of different moments, you know, I felt like leading up to me, taking my Shahada that as I look back on it had to do with, like they say, you know, Allah guides whom he wills and no man can misguide whom Allah guides.  And so for me, you know, I looked at that and I was kind of like, I looked at the cashier and I'm like, I see the cashier looking at this guy, you know what I mean?  And my cousin's like, it's a piece of candy is whatever, you know, so he kind of walks out the store.   And before I could even apologize to this guy, I looked at him and I'm kind of like, you know, I gave him some extra money and I'm like, here, take this.  Like, I would never judge you based off your skin color or anything like that.  And then I promise you, man, I'm not making this up at all. I like to read. I'm an avid reader. And at that moment of me and him talking, I looked down and amongst like some candy and next, there was a newspaper stand. And on that newspaper stand, it was a Quran. And I asked him, I said, I said, Hey, man, how much is this?  And so I was like, okay, cool. You know, took the book. I can honestly say funny enough. I didn't really, I didn't really, I opened it. I didn't really read it. I had it at my house probably for like a couple months during which that time I was getting ready to end up moving to California. And after that happened, right as I was moving out, I realized it was a monster around the corner from my house the whole time I was there. You know, and so I found that very interesting. You know, fast forward, I moved to California. Dude, you know, in the middle of the pandemic, just kind of reset my career at the time I was working in the airline industry. I had gotten opportunities to travel and see some things and go to some museums. And one thing I noticed everywhere that I went during this time, during my travels, these Smithsonian museums in DC and different places was that, you know, I would see people who to me, quote unquote, looked Muslim, whatever that look is, you know, now as far as the, you know, jab or Kofi, different things like that. But now that I know better, I know that once again, Islam and Muslims, like we don't have, there is no stereotypical look, you know, it's more so about, you know, being humble and how you dress, you know, and interesting enough, man, I was fascinated by like the exhibits, fascinated by some of the art and things that I saw. And I just remember thinking like not knowing what to make of it, but that it did feel kind of natural to me.  So fast forward, I moved to California. I'm here by myself. I have like a friend or two in the area.  I was living the dunya life in the sense, you know, just hanging out, you know, went to party or two stuff like that.  And I, and when I started to think about, and I think everybody in the pandemic kind of went through this, where I found myself kind of in a lonely place at one point, thinking like it's got to be more to life than this. And, you know, like, these are all temporary pleasures, you know, that we seek out.  But if, you know, if you go to the clubs and in a sense, I'm not, you know, telling anybody or distracting anybody from partying, I'm just saying that these are the realities of it.  And in reality, it's just an illusion.   And my dad has always been a person who, while he's not Muslim, was very rooted and centered on like having good morals and just being a good person overall. And he oftentimes introduced it to a lot of Muslims or his friends that were of the faith. And interesting enough, you know, when I went back to visit him, one of his best friends, he pulled me to the side.    How's your, how's your spirit? And I say, you know, my spirit is good, man. I'm good. You know, I kind of already knew he was Muslim. So I'm like, I'm like, Oh, okay, here we go. He probably about to hit me with a little speech or something.  And so I said to myself, I said, I joke. And I was like, yeah, man, I read Malcolm X.  And so, you know, I jokingly said that, but funny enough, he looked at me and not in the non in a non judgmental way. And he was like, you know what?   And honestly, I was supposed to fly back to Oakland, uh, like the next day. And honestly, when I got in that plane, man, I kept thinking about that Quran that I received. I kept thinking about some of those experiences that I had previously in my life and kind of where I was.  And, uh, and so when I got back, you know, I realized that there was a mosque right down the street from where I was living. So yet again, you know, feeling like this was another sign for me to, you know, take a deep, take a deeper look or at least just stop by, you know, so, um, to back up a little bit, I had been to a mosque, maybe once in my life before that, my same friend, Rossi, who I was speaking about before in college, he, uh, he took me to a mosque in Ramadan and kind of, we just, you know, I didn't fast that day, but it was such an amazing feeling to get there and sit down and eat with different people and just kind of like, you know, go through, went through the prayer, didn't really know much about what I was doing. But once again, it's one of those things kind of felt natural, you know.  And so at this time, plenty enough, it was, uh, you know, we're social distancing. So at that time, some mosques were open. Some mosques have been closed alternating, you know, between outdoor spaces. And this particular mosque, they were, they were at the park. So when I stopped by at like 12 o'clock, 1pm, I was just like, yo, like, where are the Muslims at, you know? And, uh, so I didn't let that, like, deter me or discourage me or anything, but I just thought it was odd. And so I left, right? I later that night, I'm walking past the same path, man, headed to the store just to get some snacks, bro. And, uh, I didn't know where I realized, you know, there was a light on in the mosque. And so I looked at it, I kind of walked up, and I almost turned around to walk away. And I saw two brothers come in there.  And I'm like, not knowing any, not knowing every one of me, how you let's lie. All right, cool.  And so I went, you know, they asked me that I do you know, and it was a very, it wasn't a very like, it wasn't a bad way, but it was very firm way. Like, did you do so that I'm just like, whoa, like, did I, you know? So I'm like, not knowing like all the details and stuff. I'm like, you know what? I know I'm supposed to maybe wash my hands, you know, be clean to come in this place. And, uh, and so I did, I went in the bathroom, I came out really quick. We went through a prayer, I believe it was moderate time, actually. And so we go through, you know, we go through the prayer. And funny enough, you know, I'm just in the back trying to keep up with the movements and everything.  And so I kind of laughed and I was like, something like that.  I bear witness that there's no deity worthy of worship itself for our law.  And if, and it's inviting you to ask questions, which in my previous, you know, religious experience, we didn't really, you weren't invited to ask questions like that. It was more of you're not supposed to be asking this or like, you're just supposed to accept this. And that's it, you know, or like, you're Christian or you're, you're this and these people, they just don't believe that.  And I'm just like, hmm, but that doesn't really leave room for much like tolerance. That doesn't really leave room for much openness of discussion, you know. And yeah, man, so I would say that from that point, I did a little bit more, I just kept, I read the Quran a little bit. I did some research. And honestly, once I started thinking about just the the oneness of God, and what that truly meant, and how, you know, we must stand firm on these things. And that if you say that, you know, you believe in one God, you believe in the most high, then it can really only be one, you know what I mean? Because if you say it's two or three, then what made that, and then what made that, what made that, you know, and so finding out and understanding that Muslims, that we see our Creator in a way that we try our best to describe, but even we know we truly can't name it per se, you know, Allah, the most high, like for me, I lost some animal to Allah. Like for me, that that resonated with me very deeply, man. And so I said, whatever this is, whatever this is about, I want to learn more about it. I want to know, you know, all the people that I knew, or I've seen examples of who are Muslims, wow, may not be perfect, but they all had a certain way to move. They all had a certain look about them, a certain level of peace that I felt that I've seen people operate with.  Yeah, and ended up getting a whole lot more. So, you know, thankful, thankful for that. And so that next week, I ended up taking my shahada. I had found out like, yeah, during Juma at that time that they were actually praying outside of the park. And so I went to that area and I was in one of the first things I can tell you, I was I was super amazed by how many people were there. You know, because from coming from that point of not seeing anybody at the mosque to like, you know, like, whoa, there's a whole community of Muslims out here was amazing to me and was and it was very cool and seeing all different like, you know, colors and shapes and sizes and not races, because you know, it's only one race, the human race. And once again, it's just, you know, it was amazing, man, to identify with that and to see, you know, to see kids and other people just running around and just being themselves. Yeah. So essentially, that's what made me, yeah, that's what that's what brought me to this long. Yeah. The first step in the door that day in the park, you know, with my house and I believe it was Amon Daoud was there. Yeah, Amon Daoud was there. It's probably the very best situation. That's a good, a very good brother. And yeah, we're worked out very well. Yeah, man, it's a lot, a lot to unpack kind of there. I think, kind of what you were saying at the end is just sort of there's a full circle about the, the oneness, the Taohi, right? From your sort of initially the beginning of your thought process, at least as you tried to hear was coming out of a trademark marriage situation and I'll say the situation. It killed this kid and the country was like divided between whether or not killing them was good. Exactly. Killing them was bad. Exactly. And that division was not entirely racial, but it was racialized, that's for sure. And, you know, to taking a step back to just the very like the logical proof of lai lai hei wa lau, just that there can be no God but one God. That's the definition like infinity equals infinity, right? Like the oneness, as you said, like made from the, I also, I don't want to go away, you know, made you from different tribes so that you may know each other.  You know, we all kind of come from one soul and that's, we come to the dunya to experience this and to be separate, to know each other, but that, that oneness is kind of what we come back to. Yeah. Yeah. And also, I was going to say to offer like another piece that really inspired me or kind of changed my thinking versus traditionally is the concept of, I believe it's being born and fitra, right? Being born and fitra.  We, you know, we all kind of have like this good and bad nature and that the mission in our life is to fight to be good over our natural evil selves. And in a way like, I didn't feel that, you know, like, and so when I heard like the explanation of like a concept of like fitra, you know, that we're all born like knowing the most high. And initially, like we already professed like, to him, you know, to our creator, you know, before we were born, to me, that made a lot more sense to me, it made a lot more sense that once we get out in this dunya or this outside world, that, you know, all these influences will come at us.  And as long as you make toba, as long as you turn around and repent, that, you know, that, that a lot of most, a lot of what's all it will like forgive you, you know, will give you that grace, the most gracious to ever merciful.  And if you mess up again, you can make that turn again, you know. So that's what I love about Islam, man. And also, me and my brothers. That's a well, it's super that you brought that up, because as you was talking so many different times, I was thinking that, that picture, like you, as you said, like going to the museum and like something there, like, it's like, you already knew it. You just didn't know that you knew it.  And I've heard from a lot of other Muslims, like you were, you already muslim, you just didn't know it yet. It was like that shahada was that final, like, sort of acknowledgement, really, of what you know. And that's a, you know, some people say, it's just on the same revert, instead of convert. It's linguistic. So we all know that ultimately that we all come from the same source and we all return to the same source. And that it's there with us. And especially for the world, as you said, boy, if people could have a greater sense of the oneness and not, I'm not trying to be critical here. It's just thinking about it is, yeah, we don't have to be bad people. We're born to be bad people. Like people have good instinct. They want to treat each other right. You want to treat even like animals, right? And all that. So that's in you. You can nurture it. And if you messed up, it's okay. That's all amazing. Amazing stuff. Thank you for sharing. Just to pivot a little bit, I suppose. The kind of question always comes up for Converse and you mentioned a little bit about your father. Different people have had different experiences as far as like family dealing with the shahada. How has it been for you with your family? In my situation, I personally feel like, you know, I've heard a lot of other experiences as well.    And for me, I don't look at it as changing my whole way of life. I look at it as finding, fine-tuning my life, you know. I feel like some of the answers, some of the answers to the questions I had have been answered. When it came to my father, my father, because I feel like, because he knew Muslims prior, I mean, he knew them to be good people and knew the opposite of what I've been shown on TV and, you know, what the mass media tries to portray. He had a very comfortable feeling about it, you know. I feel like he's one of those critical thinkers and people who question everything.   Did you just watch a movie? Or did you just, you know, and for me, I feel like we didn't have any major battles. My mom, in a sense, right? Wow.   But it was very, even if I tried to bring up, right, because I was excited when I first took Masjada, you know, so sometimes we want to jump out and share that joy and kind of like, you know, send a little iota verse out and, you know, say that. My mom's like, what does it mean?   And I'm like, well, that's not, you know, that's that's far from the truth. That's far from what this is about.  And that I feel like, you know, helped me, me knowing more, me doing my own research and being kind of convicted and standing firm. After a while, people people really came to accept it. So, you know, I found it funny how my dad came out here out west to see me. We're hanging out. And this is around, like, lunchtime. He's like, hey, man, don't you got a prayer? Isn't that your time?  It's like, yeah, you know, you are right. Time is coming up. So, you know, I'm thankful for that. And even back in my family history, I was surprised to find out that my father, his stepfather, who he, you know, he calls his real father and everything like that, he was Muslim. And at one point in time, when my family was very just on track on that side of the family, as far as very successful and just, you know, conscious and business wise, like he mentioned the fact that because of the way my granddad conducted itself, the way that he moved, the way that he did business with people, the way that he was just a very genuine person and always looking out that that was due to his faith and that was due to him just practicing and being who he was, who he came to be, which was Muslim, which was someone who submits to the will of the most hot.  Like, no, that's not it, you know, or, or, or you're going to start dressing this way or dressing that way.   And for me, that's, that's just amazing, man, because, you know, even in my previous experience, I could go, I've been to 20, 30, 40 different churches and I've never experienced what I've walked in the building. And we've said one thing and prayed on one accord and, you know, fasted together as a whole, all in one. And so for me, Ramadan was definitely a very eye-opening experience and kind of like a deep dive into it. Yeah. That sounds like a good pivot, I suppose, because that's Ramadan now and yes, we're kind of getting into it. Yeah, I guess if you forget my memory, it's Ramadan bringing a little fuzzy. Your show was like right before Ramadan, right? Yeah, it was like, I promise, what's funny enough, right before this Ramadan, I was kind of thinking about it and I realized that I took it probably two weeks before Ramadan started. So I had heard about Ramadan a little bit before that. You know, I've heard about the concept of fasting, but I just didn't know what it, you know, I didn't know what it deeply entailed. I didn't connect, you know, the prayer aspect to it. And during that first Ramadan, I got the chance to do that. You know, it was also a time where the mosques and things like that were kind of, you know, closed down and open with COVID and everything. But for me, because I didn't have anything to compare it to previously, it didn't really seem off to me. I just thought like, hey, I guess I think this is how Ramadan is for everybody. Everybody's kind of probably, you know, doing their own thing with the individual families and maybe meet up every now and then, but to find out in here and then later on really actually see the sense of community, you know, the actual, like, you know, no, no time in my previous life that I sit with complete strangers and just sit down and share food off the same plate or, you know, or, you know, or break fast at the same exact time. And so something that was very comforting to me, comforting to me, even though, you know, during certain periods felt kind of lonely was the fact that or the thought that, you know, there are three million, four million, I don't know the number off top of my head, people around the world that are actually conducting themselves in the same way and going through similar and same emotions. So it's like, you're not alone. You know, not alone. Three million probably in this country. Yeah. Oh, wow. Probably like a billion some in the world. Yeah. Obviously, I'm not a numbers guy. No, it's not. Well, yeah, I guess, you know, that's just like straight, straight into the deep end in the first couple of weeks. And then also, yeah, a lot of people struggle, born muslim people and convert a wild COVID was isolating. Yeah. And yeah, I guess you don't, you don't know what you miss if you hadn't seen it before, but like the community of stars starting to open a little bit now and all that. But yeah, so you inshallah, no other new muslims have to go through a COVID lockdown, Ramadan again, but kind of what was just a little bit like that first Ramadan, what would like the struggles with it if you could like go back now and talk to yourself then and just like any pieces of advice or what would like the key takeaways or whatever else is like, how did it hit you that first time? Oh yeah. The first time I would say it felt like right, because I took Masha'a the two weeks before it felt like, you know, I thought I thought I was like, okay, I'm gonna, I'm gonna do this. I'm gonna go straight into some class and we're gonna go, you know, I'm gonna learn everything from A to Z, all of it, all of it right before Ramadan.  So I would reiterate that back to myself and really, really just tell my like, because I would hear it and it was like kind of going one ear and out the other. I would think like, oh yeah, they're just saying that, but they're just saying that to be nice or they're just saying that to, you know, but the reality like, no, man, it is, it's a lot, you know, it's a, because I really, really wanted to grasp the full concept of Islam, like especially with, you know, converting or reverting, like, I wanted to be sure. It's not that I wanted to be sure, but I wanted to, I wanted to be sure and I wanted to be firm, you know, I wanted to be able to just like, you know, our sisters and our brothers and people all across the world who are sometimes under oppression, like they face, you know, to stand firm on your, on your faith and to stand firm on what you believe in, that's commendable, you know, that's great.  And I would also jokingly say, stick to your support. Actually make sure you play in and prep for that because my first man, my first Ramadan, like there were a couple mornings, I, you know, I woke up a little bit close to close to five time, Fajr and yeah, man, missed that support. I later found out and understood, you know, that there's a blessing behind that. There's a blessing behind, you know, getting up and prepping yourself and eating and, you know, and sitting and being done well before the prayer. And so yeah, man, I would, that would be the advice I had to myself is just to continue to take your time and actually seek out the reasons why you're doing certain actions versus just doing the actions. Yeah, that's great. We appreciate you sharing. I've that's probably half a hundred other things that's going to come to my head, but man, that's a really beautiful story. It's a little strange, like, you know, telling it to the camera or whatever, like I heard, you know, and seen some of it already, but the install always be second of many more. Yeah, many more, man. I don't think I'm ever going to, man, I don't, I don't plan to change or to, you know, I feel like this is it, bro. Like, you know, like a lot of people say, you know, the Quran is perfect and the Dean is perfect. And what it is, is that I feel like it's a perfect system, just like with anything in a sense, maybe imperfect people, you know, which is not something to be ashamed of, which is not something to gloat over. Like there are a lot more great and good people that I've met on this journey to Islam and in this life. And, you know, on this journey, this Dean that I feel like can help keep me afloat. And I've been surprised to find other people and, you know, brothers and sisters I've met, you know, that, you know, maybe have been born Muslim or had this Dean for a while, who said they felt rejuvenated over been like, Hey, like, nah, you inspire me, you know, so I find that cool because it's weird.  Yeah, we keep we keep each other accountable, man. And that's one of the best things I love about this Dean, man. And, you know, just want to keep going, man. And hopefully, one day share some light to other family and friends. And, and just so they can have the same sense of peace that I have, because I feel like no matter what I go through now, or no matter the feelings that, you know, we may have like, there's an answer.  You can do that five times a day and even more, you know, that's just the established times, you know. And I believe it was a treat I went to. And I heard someone say, you know, look at, look at making it's a lot like, you know, you got a meeting, you know, when you go to work, you're not late, right? You know, when you go to work, you make sure you're there on time, right? Like, why not show it for the person or for the one, for the one that created you, you know, for the one that gave you life. And that just stuck, that stuck with me, you know, even if it's five, 10 minutes a day, like we can take that time, you know.  And the brother who gave you the Quran, like, but without telling you, hey, you want a Quran, it was just having it there. And it was just being kind and forgiving and merciful, which, which your cousin, it was, right?  Without him, necessarily even trying to push it. Exactly. Living it. Exactly. And I, and that's what I would say. That's what I would say stuck to me the most, man. Like, and I think it's very important that because I've kind of observed some things since being Muslim, I've seen that and sometimes in circles and spaces, we can kind of, you know, even as Muslims, maybe close ourselves off or like, you know, you just assume that that person may not want to hear what you have to say or be open or, but you never know, man.   You know, at the time was just like, yo, you know, this and keep in mind, we didn't speak properly for like, maybe two years.   You know what I mean? And I was just like, that was an amazing feeling to experience. I also say that just funny enough, once again, just adding to it's that piece of, like you said, a little dollar here and there, I had a, you know, in college, I joined a fraternity and different little club organization schools. And one of my, one of my frat brothers was Muslim. And the thing is, you know, I didn't realize at the time his name was Mo, short for Mohammed, you know, peace and blessings be upon him.  But when it came down to certain social events, when it came down to alcohol and things like that, Mo wasn't a part of that, you know, like, but not even in a way that was just like, Oh, all of you guys are, this is not that like, no, he was a very like, he just led by example, you know, when I went to Mo's house to hang out, you know, his house was a certain way, he had a certain peace to it. It was, it was clean. It was like, Hey, if you can take your shoes off of the door, please, you know, and just realizing that, you know, his, his wife, his kids, you know, in a way, they operated was just very, very cool, very beautiful, very, I thought to myself, if I one day maybe want to start a family or a growth and this is kind of whatever system he's following or whatever kind of has him structured, that's, that's kind of what I want. You know what I mean? And I'm still kind of figuring that out. What funny enough, but yeah, man, I think that once again, I think that man, the world needs it. So when it comes to dollar, man, if you have the chance, if you have somebody that you know, who may be open to hear, all it takes is just putting that, just putting it out there and just taking it off from there. Never know. It might be 10 years, whatever down the line, just plant a little seed.  I keep my phone. I don't know about the clock. I feel like we probably called a raffle. We lose people, but I really appreciate you coming out, sharing your story with us. And I know it's not easy to be so open with the world, but like you said, it's a lot of different tribes and backgrounds and everything. So when we know each other, you never know who might find this, who might find it beneficial. So yeah, it may a lot reward you for all the benefit that the Uma and everyone else may receive from viewing this. And thank you. Thank you again. And thank you. You're my brother. You know what I mean? Like we just to get to get those reminder texts every now and then to get checked up on to me, it means a lot. You know what I mean? To get his opportunity to speak and to, you know, I've seen other videos and things like this as well that, you know, I looked at, I really looked into and to me, you know, the work that's being put out there, you know, humbly is very, I feel like it's great. You know, it's attractive. Like you said, you never know who's going to touch man. And so, you know, I'm glad to, for all who are watching this, you know, I appreciate you all and I look forward to making us, making us a lot with you in the future. InshaAllah. InshaAllah. Yeah. And vice versa brother.  You might be new to it, but true to it. So yeah, I mean, so good, smiley character and adopted that you represent so well. So thanks.


Hello, my name is Rosa Maria Costi-Cisneros and I am collaborating with the Independent Theatre Hungary. I am here sitting down with Michaela Dragan, who will be talking about Romassine, the age of the witch, which is the work that she was involved in, and we will learn a little bit more about the production and the work from your perspective, but before that I'd like to know a little bit about you as an artist and some of the work you do in your practice. Can you share some of your work? Sure, hello. I am Michaela Dragan, as Rosa said. I am an artist from Romania, based on Bucharest, but I also work in Berlin and different other places. I define myself as a multidisciplinary artist and I am one of the actresses from Romassine, the age of the witch, and I am also the author of the play. I wrote this theatre play during my residency at Royal Court in London, and it says a lot about the politics of Brexit, and also about the new age of the right wing in Europe, and it talks a lot about immigrants, it talks a lot about witches, techno witches, and the future, and it talks a lot about my art practice on Roma futurism, and concept and art, concept and aesthetic that I tried to develop in 2018, when as a Roma artist I felt this pressure of always talking about our oppressive history as Roma people or about the discriminatory present, and I just figured out that there is no mental space and no projection regarding our future as Roma people. We don't project ourselves in the future because we wear this transgenerational trauma of the slavery of the Holocaust, we have no moments where we can think about as in the future or as scientists or Roma people as inventors. So basically, Roma in the age of the witch is just to go further with these principles of Roma futurism, where the Roma culture and history intersects with technology and witchcraft, and why witchcraft? Because I think it's a feminist political act to define as a witch, knowing the history of the witches in Europe, and I think under this umbrella of the witch, many feminist energy people can be included because they are seen as outsiders. And so you were an actress within the work, but also one of the authors? The one author regarding this theater play. You're the only author? Yes. Okay. And so can you talk about that process when why you wanted to perform the work and not necessarily see someone else perform the work? Yes, this is a very good question. This is a very good question because I promise to myself, you know, working afterwards in the actor's crew that I will never do this again. Like I just, I just want to be outside, you know, to see my writing, my theater play, how it's built. I don't want to be in interior because I don't have the exterior outside experience that I founded that more exciting. And so that kind of internal, external lens or perspective. How did you get some perspective on the work? Did you have colleagues from the from the theater company support you friends and family? How did you navigate that? Because I think this is true for many people that are actors and writers that are somehow very close to the work. So how do they step back? How was that for you? So first of all, it was like a very concrete issue that we couldn't find another Romani actress to perform in the theater play because this specifically this theater show, we realize it in another city of Bucharest and most of the actors of Roma actors, they are based in Bucharest and many of them, they have engagements here in Bucharest so they can't go for a long period of rehearsals in another city. So this was one of the problem. And the other thing it was like I wrote the Kali character. It's so much me. And I was so self-cynical building this character that it was funny for me to perform it. But at the same time, I really regretted that I wasn't outside and I can't see the show because the show is so beautiful. It's the scenography, the stage design is virtual and there are there are different 3D projections. It's really the stage design is a whole show by itself. So and also like being an interior like sometimes is you are subjective because you write the play and sometimes you know the actors or the director they get different impression and you have to negotiate some things. But it was a really collaborative thing what came out as a show and I was very happy to work with Tina Turnheim from Berlin. We invited her in Romania to direct the theater show. She also works on futurism a lot as an art practice. So I communicated with her so incredibly. So she supported me a lot. Also my colleagues from the from Juvenile Pen and also the people from the State Theater in Sfantu Giorgia another city here where we staged the the play. So yeah there was a lot of people supporting me and I'm so grateful and I really want to say that even if I wrote the play like I got so much influence from Afrofuturism first of all creating creating Roma futurism and I I read so many stuff that other people suggested and recommended me and also the people from the residency from the railcourts from London they advise me a lot in the process of writing the play and I want to say this because I'm always pro-collective and I think that you know I'm really against this idea of promoting only one individual because I think the credit has to go to all the people who are involved in a theater show and from my experience from my collective we get inspiration a lot from each other and we have so many discussions and we write collectively and we come up with that sometimes the roles are flexible you know the also like the director the theater director you know sometimes gets you know a lot of ideas from us from the actors and I think this is this is how the theater should look like you know there should not be power relationship between actors and the director sure sure and it's I also wonder you know if that's a reflection of you know some Roma families because I know that that's you know this kind of working together and that everyone has a strength and we can kind of put that together and and that we're stronger together exactly and and it's also not to be afraid of the differences that we have but rather that's kind of like our superpower so no I completely agree and I love this idea of thinking about the future because as you said earlier oftentimes that there's a lot of kind of looking back at the historical kind of context or maybe making a claim and saying like we're not this and there's so much more to us so this is really lovely to think about this kind of futurism and and technology and if we've learned anything in the last year is that technology is a big part of so many of our working lives but also our creative practices so if you could maybe talk a little bit about something that stands out for you of that process of working with technology and the script you kind of talked a little bit about the sonography of the work but can you is there something that stands out that you'd like to share with us so yes I have to say that in the last years my art practice was focused on developing this concept of Roma futurism and I really hope that this can become a movement and other Roma artists can they can take further you know the ideas and create their own perspective on the Roma futurism so for me it's very important to create arts that is focused on the future I think is a very radical idea you know to to say and to share this idea that Roma people prevail in the future you know despite all the racism that we confront despite the deportations despite the crimes because very recently we heard about the the killing of a Roma man Stanislav Thomas from Czech Republic in the same way as the police killed George Floyd in U.S. in the same way they police killed them in Czech Republic so these things are happening you know so I think like saying like we will prevail in the future we will exist in the future you know can be a very subversive and radical idea yes we were talking about Roma futurism and that's a radical idea of you know staking a claim in that Roma will prevail and you also mentioned the the police brutality and the killing of Thomas so it feels quite important at a time when there is a lot happening that is true for many communities I love this this kind of conviction that you have with that you know the Roma community has a lot to offer and will also kind of pave its own direction is that true in the work do you does the work kind of reflect this this point yes totally like Roma futurism reflects on our past reflects on our history reflects on the racism that we confront and comes with alternatives so many times in Roma futurism I use fantasy or or this character the cyber witches the techno witches who wants to change the oppressive history of Roma people because we have to ask ourselves we know without racism without oppression where the Roma communities will have been now you know how how the things will have been working for us you know having the same privileges as white people as Roma people and uh this is this is why I like so much working on futurism because it's an escape you know and let us to to imagine a thing to work on fantasy to work on fictional things to to promote the Roma subjectivity the way in which we want to write or rewrite our history to say our history from our perspective um taking into consideration that our history was brought you know by known Roma people so and also I think Roma futurism gives a lot of hopes you know of it's is this hope that in the future we can get rid of all this oppression and how the world will look like when we have the power to you know to to have movement to have global movements to to to have people that represent us to control to control technology and uh the story the story of Roma scene actually announces the era Roma scene actually comes from Anthropocene and announces a new era where the Roma techno witches have the power to control the politics of Europe and it talks it talks about a lot about the who has the power in this world who has the power to to live legacy and history in this world and how will be like when the immigrants the vulnerable person they their dreams can become reality how the world will look like will be also world will be also a place that is threatened by the climate crisis by the racism also on or will be you know the world that we all dream about it and I talk specifically about the climate crisis because this is also one of the topics of the play like the techno witches they uh in the show they put this this problem on the on the table you know is we live now in the era of the climate crisis and our world will look so different in 20 years so we also took this responsibility to to imagine feasible politics against the climate change um I don't know I it's it's a it's a very complex theater play is is my latest theater play I was so privileged that I had advisors working it was the first time and I had advisors working on this theater play so I'm I really I really hope that people will get a lot of messages from it that's lovely that's lovely and um thank you so much for for also being quite bold and brave and thinking about um you know the the current topics but also the future um because it's very I think quite challenging sometimes to to be one of the first and to be visionary and to discuss some of these these bigger topics that affect us all and you know again aren't easy to to navigate and you're doing that through work through technology through the theater um and it sounds like also that you're reflecting on that um and your voice in place within this larger broader narrative um yes I'm I I haven't seen the show live but I would love to see this this piece of work particularly because you know um there's there's something with technology that you know it changes the relationship of of the body to body and so I'm quite curious what that might look like live um so thank you so much thank you Mihaila to you to all the advisors to everyone involved in creating the work um and yes thank you thank you too and please share share with me your thoughts after you watch the play yes yes and your criticism too because I'm very open to criticism yes well you know I see criticism is actually a way to grow um and it's however how it must be you know depending on how it's delivered but I do see it as as a way to grow and and welcome it so yes no problem thank you bye bye bye thank you