Today, we're going to talk about Cortex. Specifically, we're going to talk about what happens since the beginning of this year, and then what do we have in plan for Cortex. So here's the agenda. Actually, let me introduce myself first. So my name's Alvin. I'm one of the software development manager at AWS. I joined as a Cortex mentor about a year ago. I started to look into the query path a lot. So I have a little bit of technical knowledge there. But nowadays, I mainly do releases, doing some show work, like upgrading go-round time to 1.19.2 and all that, or circuitry patching and all that. And here with me is Alan, who is the actual brain of Cortex, and I'll let him introduce himself. Hey, my name is Alan. I also work for AWS on all these teams. Been working with Cortex for the last two years. Working in scalability, availability, became maintainer in the last year. Yeah, trying to make Cortex a better place. Cool, thanks, Alan. So Alan does most of the work. I essentially do nothing, and just tell Alan to do work. And here with me is the FedRex. Of course, he's not here with me physically, but he's very unfortunate I couldn't make it. So FedRex is actually a very long-time Cortex user. He deployed Cortex clusters for Adobe, and he runs, I think, the number he told me it's hundreds of clusters. And if you were ever in the CNSafe Cortex channel, then you ask a question, then Cortex is the person who will answer your question. So he is really active in the channel, answering any questions regarding configuration, optimization, and all that. Let's but not the least, we also have maintainer from Germany, and then he said it's a Niklas. I didn't want to go into linking and searching why it feels like stalking, so I just grabbed his GitHub profile picture, and that's why his GitHub profile picture looks like. And he's the maintainer for the HelmChart, and he's very active if you feel that, hey, HelmChart's missing some configuration added, added, and then Niklas will always be there to merge the PR. Cool, so to this agenda. So I would imagine not a lot of people know what Cortex is and how does it work, so we will do a quick introduction to talk about Cortex, and then we'll do a little bit of the architectural dive deep for Cortex. Then I want to introduce three exciting features that's coming to Cortex in the next release, which will be 1.14. And I'll give one operational tip, so if you're running Cortex right now, then you should do this when you get back to work. It's very useful. We run a lot of Cortex cluster back in AWS, and that's one of the tips that save us a lot of memories. Then we'll just look back to see what happened for the Cortex release 1.13, then we'll go through the voice of community or we call it the roadmap, then we will talk about the call to actions, we need help, Cortex need help, and then I would love more contribution to Cortex. Then we'll do our favorite part, which is the Q&A. All right, so what is Cortex? Cortex is a horizontally scalable, highly available, multi-tenant, long-term storage for Permethias. So what does that mean? When we think about Permethias, initially when it was designed, it's designed to be installed on a single machine, scrap a cluster of metrics, and then it's stored onto local drive. The problem is your local drive cannot be as big as you would like to, so you usually have a very fairly short-term retention period. So Cortex trying to solve that problem. By in order to be able to store a lot of metrics, millions, billions of metrics, you need scale. So essentially what Cortex is doing is to take bits and pieces of Cortex and then make it into microservices. So you have a microservice occurring, a microservice or ingestion, a microservice for the moving data to the long-term storage, a microservice for reading data from the long-term storage. So like saying Cortex is a long-term storage is a little bit misleading because it's not just storage. It is actually a system that allow querying. And Cortex is a CNCF project that's incubating and Apache 2.0 license, so you can do anything with it. Bunch of contributors, about 250 of them, lots of watches, 5K star, and so far we have about 5,000 commits. And then Alan here is making the commits going up every single day. So it is actually a very active project. This is a very high-level view, a bird's-eye view, at a very high level of Cortex. So essentially what it is, a typical use case of Cortex. You have a bunch of Prometheus, right? So you have one premise for cluster A, one premise for cluster B, so on and so forth. What you can do is you can configure Prometheus to do a remote write into Cortex. All of them can just do a remote write to Cortex. If you want a differential between clusters you can do add a label during remote write to differential different clusters. They're just saying to Cortex, then you attach your dashboard tooling like a Govanna or whatever dashboard that you're feel like in, then you get a global view of your matrix, right? You don't have to go to like Prometheus A to look at cluster A, Prometheus B to look at cluster B's matrix. So because remote write, it is the protocol that it's fairly stable from Prometheus. So you don't have to just use Prometheus. You can also use the tools like an open telemetry to send metrics to Cortex.  And you can send a lot of metrics to Cortex. You will be able to handle it. So next I wanna dive a little bit deeper into that Cortex icon to see how it actually work under the hood. Oh God, this looks complicated, Alan. I think I need help for this one. Well, I will try my best here. Yeah, as you can see, like the community is saying that Cortex can scale a lot. It can send a bunch of metrics as high available but it's complex to set up. So this is what uses to be Cortex in the past year. So we are trying to, we are deprecating some dead code and some deprecated storage. So we're moving even from the code base and don't worry, this is not what Cortex looks like anymore. It looks more like this. This is a typical Cortex deployment that you can find. You can see like in the yellow there, that's the write path. So remote write come from from ethos. And in green there, that's the read path. So we still have some components. I will try to explain what are each one of those components and hopefully to make more sense at the end of the disk. I will start with the write path. So what happens when Prometheus send a write request to Cortex, right? The first component that's rich, that is called distributor and what is the distributor? Distributor is basically a gateway, but that will forward the request to the writing gestures. In gestures are the storage nodes, distributor will just forward the request for them. But why we need the distributor? So distributor, it will do sharding and replicator data and do like pertinent sharding and replicator data. And optionally, you can set up things like shuffle sharding that will improve your tenant isolation or zone awareness replication. So distributor is the guy that will make sure that he's sending one copy of our data to each AZ. It also does things like rate limiting and AJ data. So if you have a Prometheus server that is sending is deployed in a AJ mode, distributor is the guy that will receive one sample and throw on the floor the second one. After the distributor, we have the ingestors. And ingestors are basically a multi-tenant TSDB. So remember, Alvin said that what Cortex does is take Prometheus and put in different microservice. Prometheus is basically a TSDB and a query engine. The ingestors is the house of the TSDB. So the first time that I receive a sample to one tenant, ingestors receive a sample for one tenant, it will create the TSDB block, the TSDB instance. We'll keep appending that. And the after is configured, but typically after two hours, you'll send those TSDB blocks to the block storage. And the block storage can be Google Cloud storage, S3, any block storage that you want. We have support for Azure as well. But now, we can see that as I was sending data, so the data that were on this was replicated. I sent to one replica for HSE. And now I have all these data, duplicated data on S3. So there's where the compactor comes in. The compactor will get all those blocks. It will compact and compress those blocks and make sure that this data is in the optimal way to be queried. It also does things like housekeeping. So if you configure your potential period for one year, compactor is the guy that will start to delete blocks that's older than one year and things like that. Again, all those components can be shuffle-sharded and deployed with zone awareness. So you have easy, tolerant, and tenant isolation. This is basically the right path. But then you have to query your data, right? So the read path, the first component is called query front-end, which you do. This is a similar thing that distributor does but for the query. It will shuffle-shard and make sure that query is for a given tenant or spread across this. But it does more than that, right? Like it does QoS, for instance. So it makes sure that one tenant is not starving other tenants or do cache of cache. So imagine that you have your dashboard there that's refreshing every minute. And instead of recomputing or re-executing the whole query, the query front-end will just fetch the delta from the last refresh to the refresh right now. And do things like vertical and horizontal-sharding, query-sharding that I think Alvin will talk a little bit more about that. But it's basically trying to split one query in multiple smaller queries so I can run that in parallel in multiple queries. After the query front-end, we have the query-er. The query-er is the house for the promo-ql engine now. So we run the Prometheus promo-ql engine in that component and basically receives the query request, fetching data from ingestors for recent data or from historical data, merge all of them, evaluate the query, return the result back to the query front-end and to the customer. Do things like rate limiting as well, like to prevent out-of-members and things like that. Now we have the store-gator. What is the store-gator? Well, it's the gateway for the store. Basically what this guy is doing is it's keeping an up-to-date view of the block storage. So every time that I receive a new block or I compact a new block, store-gator discovered that, advertised that to the query-er so this block starts to be query-able. And also download parts of the index, the block index to make sure that we can have a faster time series look up when you are running queries. So basically this is like a normal Cortex deployment. Officially, you can also run rulers and alert manager and those components are basically a multi-tenant version of the Prometheus ruler and alert manager. Again, zone awareness, again, shuffle sharded. Rulers will basically evaluate, record and alert in rules. We will send the alerts to the alert manager. Alert manager, you, the dub group and send the alert for the right explanation like slack or page dirty, you name it. Basically this is what Cortex is right now. Those are the components. Hopefully it makes more sense after that. And now it's back to Alvin. Yeah, definitely, I think it makes more sense than the diagram we showed at the beginning. All right, cool. So this is the list of the company they are currently using Cortex, a running Cortex cluster. Cool, so now I want to introduce the three features that I was talking about. And the first feature is the open telemetry bridge for tracing. If you are the operator of a Cortex cluster, you will like this feature. So this feature essentially allows you to send traces to different destination. In the graph here, we have the example of sending it to AWS X-Ray. So the story behind this feature is because one day I was just writing the normal status report that manager should write every day, Alan came into my office and he said, Alvin, Alvin. I think there's a bottleneck between query and query from end. I don't know how to do it. I told Alan, there's the anchor, Cortex supports the anchor. So the person open trace, just, no, set up the anchor. Then Alan said, no, I don't know. I don't know how to set up the anchor. I said, well, spend some time, set it up. And Alan threw up his hand in the air and said, hey, you're not helping. Then he went out of my office. So I get back to my work. Few hours later, Alan come in with this screenshot, exactly. I said, hey, Alvin, look what I got it working. I got it working with X-Ray. Now I see there's a bottleneck between the query from end and the query. It's because the queue is overloaded. There's a queue between them and it's overloaded. Oh, cool. This is awesome. How did you do it? And I said, oh, yeah. I integrated with the open telemetry. And I said, okay, cool. Awesome. Alan asked me, should we open source this? I said, of course, why not? So this is hours of work from Alan. It is awesome. If you're ever run cluster, you have a problem. Use this feature. It will help you troubleshoot a lot of issue. Alan here actually use the trace multiple times. So I define multiple issue and do optimization. Even for the query vertical sharding, Alan used that to analyze the hay. It is actually boosting performance. And then, which I'll talk about a little bit later. So yeah, so with the open telemetry support, you can send to multiple destination. That's the major selling points, like to Jagger, to Zipkin, to Kafka, and to AWS X-Ray. That's a lot more. You just have to configure the exporter. And the reason why we say the open telemetry is good is because open telemetry is a CNCF project. And it is the protocol. It's a specification. So you don't have to worry about vendor locking. All right, cool. The next feature I want to talk about is a partitioning compactor. So I haven't finalized a name for this feature. So I promise I'll work with the creator of this feature to come up with a better name. But for now, what's the partitioning compactor? Permitius has a limitation. Each block, so Permitius GSDB is essentially bunch of block, right? And each block has a limitation of 64 gigabytes of index size. And the reason is because they have a reference that it's only able to address up to 64 gigabytes. Sure, we can fix that problem, but that problem is a little bit hard to fix and might take a little bit long time. Imagine switching from 32-bit Windows to 64-bit. It will take a while. We don't want to wait. We can wait, but we don't want to wait. So what we're doing here is that, hey, currently in Cortex, if you try to merge two blocks, whose index size is close to 64 gigabytes, you merge them together, and then the result is 100 gigabytes. Then Cortex will choke. What you have to do is you have to upload this no-compact marker to the source block of the blue one and they tell you, hey, just don't compact this. That's a workaround that's not fixed, because if you look at the size over there, 63 plus 63, it's bigger than 100, right? But then if you go through a compaction process, you guys should do the symbol table, index the duplication, which will reduce the index size by quite a bit. So you save about like 26 gigabytes over there. If you still go through with the compaction process, right, 26 gigabytes is a lot. You can save a lot of HD videos with 26 gigabytes. So what we're doing is then, hey, the new compact will say, okay, I'll partition the matrix in such a smart way so that I will still end up with two blocks, but each of them will have a smaller index such that it doesn't hit the limit of the permissive. So we'll figure out like, hey, how much partition we need? We might end up partition maybe four blocks to three or two to two, like in this example. So with this, another possibility to begin to show up, right? You can actually do compaction in parallel because your resource is two blocks. Each compaction can be run in each individual compactor versus before with the blue boxes, you need to do that in one box, right? So the result of the new partitioning compactor is that we actually, it's observing about 50% compaction time reduction in our lab results for a single tenant with 200 million times series, right? We're still doing testing. This is still work in progress. It's almost done. The implementation is there. We are just doing the testing and finalization and then we'll merge the PR. Yeah, so with this feature, you don't have the issue for the bigger than six or four gigabytes issue and then your block will be optimized. And another side effect of the design is that the algorithm is able to figure out who's the source block of a distinction block, right? So you don't have to download all the blocks from S3 into compactor when you do compaction. There's a potential speed up of the compaction process as well. Okay. The last feature I wanna go into is query sharding. This is a very cool feature in my opinion. And before I go into that, I wanna shout out to the Athernos community over there for making this possible because we actually use the Athernos call to do some query analyzer and spit out the outputs and say, hey, this query is shardable, not shardable. And then we saw, hey, we have Quotex user, why not we bring it to Quotex as well, right? Collaboration is always beautiful. So what's vertical sharding? What does vertical mean? When we talk about horizontal sharding, just imagine you have query, you wanna query from day one to day two. So you have two-day query. Horizontal sharding means that you shard by timing interval. So you can actually split the query into day one to just beginning of day two and beginning of day two to end of day two. And then you can split them, round them concurrently and then get the result. They all works. But what if today your query is actually instant query? Where you wanna know? The instant query means that I wanna know the data right now. There's no timing interval. Then how do you shard? You cannot do horizontal sharding because there's no timing interval to shard. That's where vertical query sharding comes in. So I'll do a little bit of deep dive into the vertical query sharding just because I think it's a very cool concept. And then it's the first step to a more optimized, bronchial engine. So this feature is already available right now. You can use it. And I forgot to mention that the open telemetry support is actually in the mainline branch. So if you are the type of person who don't mind using the mainline branch, please do, you can start using the feature. And then cortex thing, we try to make the mainline stable. So because internally at AWS, we actually use the mainline as well. So we test the stuff before we push to mainline. So yeah, so the vertical query sharding is available in mainline branch. And then the speed improvement can be up to, it can be a 30% plus and simple flag to enable the dosato to flag and the documentation. So I just wanna touch a little bit on the documentation. It is under the V1 guarantee just because it is an experimental feature. So it's not in the configuration list. So just be aware that all you can use the slide. So let's do a little bit deep dive under how the vertical sharding works. So consider this matrix. You don't have to steer too high, it's fine. It's fairly simple. You have matrix to calculate how many users you have per region. So I'm using the North America and Europe as example. And then when you collect the matrix, usually you have multiple permissive instance for scalability or redundancy. You wouldn't just have one, right? So that's why you have part one, part two, part three. Redundancy of three is always beautiful and cost effective. And then now imagine I wanna run this query. Don't worry about starting this query. This is like, hey, I wanna know the user per region, right? So, and I wanna get the result. Hey, I have 100 users in North America and I have 95 in Europe. Without vertical sharding, what's done is that the whole query is sent to one query and the query will fetch all the data and just merge them, send it back to query from end, back to user. But if you kind of think about it, hey, I can actually do the query or the aggregation for North America and for Europe differently and color-coded with, I guess it's blue and purple. Yeah, blue and purple, I'll stick with that. So, this is how, without query sharding, how it works. Query from end, query go to the store and the store has its color coded just to show you that, hey, the store doesn't store the same region in the same store. They are just, you know, interleaving everywhere. Query go to the store, do everything, aggregate table, send it back to query from end. Custom is happy. Whoa, when it's a little bit slower, they will not be happy, but, you know. This is what happened to query sharding. It is a little bit more complicated, but what's happening right now is that the query from end will actually do a splitting. Notice that, hey, query one, please, do the European aggregation. Query two, do the North America aggregation. And the query one actually talked to the store to say, hey, please just give me the European one. Don't give me the North American data. What this allows is it reduce the network traffic, right? So right now, store one, store two, does not have to return all the data like before. Overall, the data, the data send overwatch doesn't send, each part will have less data to receive. So query one will do the aggregation for the Europe and query two will do the aggregation for North America. And right now, query from end will have the job to merge those together, but the merging is simple. You simply merge two table. Like you have two roles, merge and two roles, easy peasy. So it's not too much of an overhead for the query from end. And query from end is designed for this kind of stuff. Yeah, and you return to customer, customer again is happy. I couldn't find a happier emoji, so that's the one I have. All right, cool. So that's a feature. And we added this to streaming support belief in 1.13, so it's already available, right? So if you wanna take away anything from this talk, take this away. Enable streaming between query and jester. It will save you a lot of memory issues. So we're actually adding the query in jester made out of streaming. I think it's recently, or is it already released? It's not released. It's not released. Okay, we'll make sure it's released soon. Cause a little happened to Cortex. Cortex has a lot of limits, right? And then we're finding different limits as we go and we try to combine aggregate limits to come. So why this safe memory? It's because if you were to get all the response from the ingester in a query here, it's actually big, right? You can query like the gigs of data and those have to be cached in memory before you start processing. It's gonna increase your memory footprint. You'll see a huge spike of your memory footprint and then you will not be able to be garbage collected as long as you're processing the query. With the streaming, you process it, you dump it, you process it, dump it. For some query, you can process it and dump it, right? So those process data can be garbage collected. So you will see a smaller spike, but it will still be spike. So yeah, so streams are beautiful. Please enable the streaming. It will save you a lot of headache. All right, so looking back. We released a 1.13. 1.13, one of the major feature of 1.13 was parallel compaction. And that was the first set to speed up for compactor. And it's actually complementary to the partition compactor. So partition compactor actually make the parallelization a little bit better. But there will be complementary to making compaction faster. And we also forked the repository from Govana Lab that's Cortex Tool to Cortex, because it just makes sense. We wanna continue to support Cortex Tool as the Cortex maintainers. There's no reason to leave it there, leave it there and say, hey, we don't support it anymore. It's related to Cortex. We want to support it. And I wanna shout out to Ireland and Fredrish and even Nikolas for stepping up to become a maintainer of Cortex when needed the most. Cortex went through, went through some rough time this year, but then it's not back in good hands. Cool. So this is the long-time customer, it's Fredrish, because I wrote this slide before I got his permission to use his name, but he gave me the permission so I can use his name. So Fredrish, he run a lot of cluster at Adobe for his internal customers. And it's fast, right? The ingestion time is less than one second, but Fredrish has been very conservative. In AWS, our ingestion time usually is less than 10 milliseconds. And also, 99.9% of the request is less than 1.6 seconds. 1.6 seconds might seem big, but then if you query a long range of query that's fairly long, that it's actually not too bad of an output of time. And usually I think Fredrish told me that his query usually spent across one to two months. If I remember correctly, I have to double verify. And Cortex will usually take the backward compatibility very seriously. So when Fredrish tried to upgrade from 0.61, which is a very ancient version, to 1.13, it was easy, right? He got some configuration that was removed, but then just removed those configuration from your YAML, then you're good to go. And then he upgrade to 1.13.0 and then was able to support 150 million times series instead of 32. And the BlackHound like was actually just a compaction. The parallel compaction actually enabled a lot of times series. 150 million might, it's not the maximum. Cortex can handle a lot bigger. But there's a few factors we have to consider which can be a whole other talk. All right, so we also have the voice of the community which are the feature that customer users of how is that one. First one, what first of them is the out of samples, of all the older samples for backfilling data. This is already available merged into Premiere CS, I think not long ago. So we will enable that in Cortex soon. Sensitive for down-samples. Down-sampling, there's some discussion about, hey, what exactly problem we're trying to solve for down-sampling? Is it because faster querying or saving the storage? We don't know yet. So that one's I still been discussed. Pretender encryption. It is more for like the people who cares about the, hey, sometimes I might accidentally, I might have like a PII data, personal identifiable information, PII data in my matrix. Then I would like to encrypt it on the rest. Even though I sent to the cloud, I sent to a cluster, I wanna encrypt it at rest. And of course you don't wanna decide one key for all the tendons, they will be, you know, what if like your external customer or your internal customer are two different teams? They would like to be encrypted separately by different different keys, right? Which goes in hand with the delete time series. Quite a feature is important is, imagine that you suspect that, hey, one of my classes compromised. And you go ahead and rebook the access to a key, which resulting in, hey, your matrix are not available anymore for that specific tender. So once you identify, oh, actually, I only have like the two time series. They are affected. There has like a maybe critical number for some reason by some silly mistakes. Then you can say, okay, I'm gonna delete those two time series and then re-enable the key. Then you're back to business. So that's one of the use case for using the time series and pretending encryption. There's other use cases as well. There's a lot of reason driving behind these two asks. And there's a lot more in our backlog. So please do tell us what you want. Join the Slack channel, talk to us, check with Ellen and I, federal, who are all there, who are all friendly people. You say hi to me, I'll say hi back. Oh, I was an emoji wave back to you. Oh, you should use beer, right? I like beer. And it goes through a backlog to upvote the issue that you want with a thumbs up. Or you can use the smiley face, anything. Then to let us know that, hey, this is important to you. If we don't understand what is important to you, we'll maybe hopefully have a conversation on the GitHub or Slack. So if you want to contribute, please just go find out this issue tag with a good first-time issue. Those are something like fixing a typo, fixing this, fixing small things. And then if those are something you're interested to do, please go ahead. All right, and call to actions. Cortex currently have three maintainers. And then I would like more, because there's so much more I want to do, but then we're spreading a little thin here. So if more people can contribute, more people can become maintainers, it would be nice. And help maintain HMTRA, maintain the Cortex.io website and the documentation. So I'm the person who's trying to maintain the Cortex.io now, but I don't have any artistic sense, right? I need someone who really know the web type, web type in order to help me to pre-defy it, organize the information a little bit better and all that. Same thing for documentation. Cortex documentation is not bad right now, but it can have a lot of room for improvement. So if you find any typo, if you find any information can be reorganized on the website, it will be awesome. For example, I wanted to update the architecture diagram on the Cortex website that, oh, I showed previously. And the last one is the automatic benchmarking framework, which is available in the Premier CS repo, but they're not Cortex and I really wanted it because it will be so nice if a peer comes in, I do a slash benchmark, then you can automatically see the performance difference or that, so you don't have to do yourself. Eventually I might get to that, but then if you cannot wait, please tell me that. Oh, I forgot to mention, please engage with what CERNOS community is trying to do. They are trying to create a Procure Engine that's scalable, right? So please let's join force with them. Let's create a more scalable Procure Engine that supports sharding, beta sharding and all that, right? It's going to the arena of like how do you optimize like Procure as in the SQL in the old days. What is this? So thank you, that's everything. Sorry, I'm a little bit over time, but I guess we have some time for questions. I can maybe kick it off. I noticed that the architecture of the sort of query and storage looks very similar to Druid. Was there any exchange of ideas there or have you guys looked at Druid at all? To Druid, the database. I guess not me, because I have no idea what Druid is also. You might check it out. It's a very interesting, similar structure, yeah. Cool, cool, yeah, that's what I was checking out. With the object storage on the back end, yeah, anyway. Any other questions? I have maybe a weird question. Sure. So in the new architecture diagram, you have many components. I wanted to know which one were stateless and I could run on say on the spot instance versus some that are stateful that I don't want to lose. This one, right? I'm imagining the Compactor, for example, if it's interrupted, not a big deal, can restart. Yeah. Stuff like that. So. Weird question. So some, so distribute or query front end queries, they are always stateless. Some components we can choose to run stateful or not. Like, ingestors should be stateful, we should not never lose your data. But components like storage gateway, you can choose. Mainly because like as we are downloading parts of the blocks, it makes sense to be stateful because open restart, you don't have to download everything again. Compactor, usually we run stateful, but just because we want a more disk. It doesn't need to, but like as I wanna put a PVC there so I can put more disk, if I have a huge tenant that I have to compact. But if you are noted, has the disk enough for the size of your tenants, it can be stateless. Okay. That was my question. Thank you. Thank you. Hi, thanks for the talk. I have a question regarding diagram. Why a ruler talks to the querier and not to query front ends? It's a good question. We are thinking and changing this. And the reason because we want to, right now this is not 100% correct. What happens is like the rulers run a query embedded. So it's the same code, but running the same process. Logically it's like that, but right now it runs. The ruler doesn't really talk to the query, talk directly to ingestors and install gators. But we wanna change that. We need to do lots of benchmarks to see if makes sense, especially because our rulers are running instant queries like getting recent data mostly. But we want to change that so we can get some of the optimizations that we do in the query front end. Okay, thank you. And one more question. How distributor does the duplication? So usually you have like a console or a TCD. So it gets one of the pairs and say like, this is the main one, this is the one that I'll drop the data. Okay, and those prometeuses, they need to talk to the same distributor all the time or you can have multiple distributors and they will. Everything like that. That first arrow, there is a load balance. Okay, thank you. Very good observation that Buck will be from and PR will come by the way. No, we actually start working on data already. We disappear to enable that, but it's just a first step. Feel free to chime in in that PR or the issue. Hey, so quick question. You mentioned, is it a feature in progress to a sample, or sorry, to accept out of order sample ingestion and how are you looking to approach that? It's a great idea. Very helpful. So you wanna take that, Heather? How? Yeah, so I think the how is mostly prometeuses, right? Yeah, so like prometeuses, TSTB already have support for it. Basically it's creating a new head chunk to accept the out of samples. There is some overhead there. In our case, it's basically making it available for the customers to enable it. Yeah, just before we enable, they wanna be very careful about not to have users set up their own tools. So when I do our own benchmarking, wanna see, hey, wanna give them a set of recommendations, if you enable this, maybe you don't wanna backfill like one year ago, simple, they might cause some issue like that. Cool, so I guess that's everything. We run out of time. Thank you so much for joining. Hopefully you enjoyed the talk. Thank you.


Hello. It's a talk of Serge Guelton. He's developer of Python and the talk is surviving in an open source niche, the Python case. Thanks. So we are back to a more intimate area, human scale, not like the one before. So I'm going to spend the next 50 minutes speaking about a project I developed. It's nothing as a scale of Python 3 or whatever. There may be less users of this project than the number of people in this room, but that may change in the future. Anyway, I think that the idea I want to share are beneficial even if you don't use the software. So a few words about me. So I'm basically a research engineer in a security firm where I do compilation stuff. I'm also a research associate researcher in a French university, a developer of the project I'm going to present to you and an LLVM committer. So what is Python? Python is a compiler, a static compiler, not just in time compilation, basic compiler just like you would use GCC for CRC++. You can use Python for Python code, but not for any Python code. It's for scientific kernels, only for the scientific word. And the input is not plain Python. Well, it is, but it's a subset, a strict subset. So there is no extension or whatever. Every code that compiles with Python is also a valid Python code, but not in the other way. So basically you write your scientific kernels. You had a few comments, so as there are comments that can be ignored or whatever and the Python code still runs, but the Python compiler understands these comments and use them to generate native code. The comments are not meant to be intrusive. There is only a few comments for function declaration, nothing at the variable declaration level or no extension or whatever, just state which function you want to export and this module will be compiled as a native module and only this function will be available in this module for this particular signature. So here I have a Rosenbrock function, I don't know what is it, but I can write it in umpy and I can export it stating that it accepts an array of integer or an array of double precision floating points. So why is it an ish? It's an ish because scientific computing is an ish. Well, there is a quite big community in scientific computing and Python, mostly due to the NumPy, SciPy, Jupyter, and Matpotlib stack and when the Python performance are not enough there is also a wide range of choice to optimize your code, ranging from Siphons that will be presented in the next talk to NumEx, Pro, NumBar, and a few others. And there is also a lot of things happening that are trying to happen in the world of compilation and Python. There are some very long-term projects like PyPy, Siphon, or Jiffon and some recent projects like NumBar or Hope and a lot of dead projects which is an int about the difficulty of the task. Copperhead to generate GPU code, Piston from Dropbox, and Eden Swole all, Parakeet, and all those projects are now dead but they tried some ideas to compile some kind of Python code. The ideas in the air when you want to compile Python codes say, okay, well, it's easy. I just add types to every function declaration so I don't have any lazy binding or dynamic dispatch. Everything will be static and then I translate that to C++ so choose your poison, it doesn't matter as long as it's a statically compiled language. And I make a lot of assumptions about the imported module. I don't care about how Python actually imports module. I use import just like I would use just include or things like that and I stick to a subset of the language which is easy to compile. And it's relatively easy to make that work. But you can also do some more advanced stuff. So that's just translation. Translation is not compilation. Translation is what cat does or said does. Compilation transforms the code. So you can take all the compiler knowledge that exists since the 50s and put it into compiling Python. There are some specificities in the language that make it an interesting target. You can use just-in-time compilation instead of ahead-of-time compilation and you can try to support a larger subset of the language, support generators, support class, meta class or what you want. And if you're really, really bold, you can try to be compatible with whatever Python code exists. Only byte code but any byte code supporting the import mechanism, lazy binding on this kind of stuff, that's what PyPy is doing. And you can also try to be compatible or to optimize native extensions. Optimizing native extension is a big deal. It's very important, especially in the scientific world because people tend to use Python as a glue and to write candles in native code and then they want to optimize cross-native codes. Python is somewhere in the advanced area. It's definitely not in the expert area and most of the basic stuff has been done. So it's an open source software. BSD started six years ago. You can find it on PyPy, on the cheese shop, on Konda or on GitHub, depending on the way you want to use it. There are some Python dependencies, networks to do some graph compiler algorithm, NumPy for all the scientific stuff, PLI, which is Lexiac in Python, for the sub-language we use in the export line, and guest, I'm going to speak about that. Victor told us that moving from Python 2 to Python 3 is a matter of syntax. But when your input is Python, moving from Python 2 to Python 3 means changing the input of your program, like changing the SQL schema or things like that. So it was actually a very difficult thing for us to move from Python 2 to Python 3. It was not just adding parenthesis around prints. It was changing the input, and when you meet the range built-in as an input, it creates a list or it creates a generator. So it's different stuff. You need to compile them differently. So that was a difficult step, and we introduced a thin layer to abstract the Python ST in that, so that's generic ST. And you also need, and that used to be a very difficult step, you need a C++ compiler, a C++11 compiler, and back in the day, it was not the case. It's actually still difficult to find a compiler because that handles every C++11 constructs. Even on Linux, with Clang or GCC, there are still some steps that are not well handled, and when you start to switch to Windows, then things start to be very difficult to support. But it's getting better because Ubuntu now shapes a decent version of GCC, so most people have a decent compiler that's no longer a source of issues on GitHub. The community is quite small, so the bus factor is of one, me, but I do receive a few contributions, sometimes from students of mine, so it's not really a choice, but they do enjoy the contribution, and sometimes from foreigners, so that's something I really appreciate because I'm not trying to grab a lot of attention, more than having fun technically and provide things that can be interesting for people as users, but sometimes users do submit a pull request, sometimes it's just to fix a typo in the readme, and sometimes they do implement stuff. Behind this compiler, there are three ideas that make it quite different from other Python compilers. The first idea is that there is no mixed-mod where Python code calling, or C code calling the Python C API lives within pure C code and everything works together. That's a siphon approach, and our approach is either you can compile to pure native code and there is no call to the Python C API apart from the frontier, or you can't compile. So that may look like a very harsh choice, but there are a lot of nice things. First, as there is no longer any Python C API call, you can release a gil. So you can make multi-freaded calls to Python generating modules and release a gil, and that works. So that's a nice point. The other point is that as you generate Python free code, you could generate a native module that can be imported from Julia or from Rust. That's not implemented, but that's something that would work. You could prototype your code in Python and then import it from Julia or Rust. That's something I want to try this year. The other point is typing. Writing a good type inference algorithm is difficult, so either you have to read a lot of bibliography, which is in itself difficult, or you have to try to reinvent the wheel, and then realize that there is a lot of literature, but that's not for nothing. Or you say, OK, duck typing, it's really similar to template C++ template instantiation. In fact, C++ templates use static polymorphism, and duck typing provides dynamic polymorphism, but in a static world, there is a match between the two. So basically, we generate C++, and it happens to be correctly typed. We don't try to infer the type of... We just generate metaprograms, C++ metaprograms, and we instantiate them for several types, and native code gets generated. That sounds magic, but it actually works. So the typing code in Python is not brain-dunmaging, and that's cool for me, because that's not my expertise area. The next idea is we don't generate low-level code. If there is no loop in the original code, we don't try to generate loop in the final code. We basically rewrote the Python built-ins, part of NumPy, random-module iter tools in C++, in a full-template and generic way, and we generate code to this high-level C++ library. The good thing is that if you want to increase the library support, you just have to write a bunch of decent C++ code for some definition of decent, but still, it's feasible. Another good thing is that as we keep the source-to-source translation at a high level, if you put... We had this feature. If you put OpenMP annotations on your Python code, then they can be translated to C++, and they still have the correct meaning. The sematic is respected, and we actually support the full OpenMP... Most of OpenMP free language from Python, and you have your multi-fledged code that runs on multiple cores. So, that sounds like a lot of stuff, but there's not that much code. So, some commits, obviously. Pythonic is a C++ layer, so it's less than half a thousand... No, 40,000, 50,000 lines of C++ code, and that's most of the code. There's a lot of tests, too, because that's a hobby. So, I don't want to spend time debugging because I have better things to do. So, if I have a lot of tests, then I can just write my code, launch the test suite, and it takes on GitHub three hours. Travis, three hours to run all the test suits, but then I can just go cooking, shopping, or whatever, and then I go back, and, okay, it works, and I can move to the other tasks. So, that's not my work. It's a hobby. So, having a lot of tests is a good way to have a hobby that can still be useful. And the Python code is actually very... Well, there is less than 20,000 lines of Python code to write this compiler. So, maybe because part of the job is moved to the C++ stuff, but also because Python is a high-level language, so writing compiler in Python does not prove to be that difficult. So, how does it work? You write your Python code as you would do. You had these Python export lines. You learn the syntax of this single line. Then you call Python. It itself generates a C++ code that can be combined with any descent C++ compiler, and it generates a native library that can be imported just like a regular module. That's part of the Python module import mechanism. But, wait, it's not a translator. That looks like a translator from Python to C++ to native code, but that's not, because recall, C++ is just a convenient backend. It's a very convenient backend, but it's just a backend. Python is a compiler. You can view it as a source-to-source toolbox. You take your Python code, move to the abstract syntax tree, refine the syntax tree and optimize it for scientific computing and then dump the result, either as Python code, which makes debugging easier, or as C++ code. So, there are three important pieces in Python. Analyzes, which try to gather information about the syntax tree. Transformation and optimization that both transforms the code, either to make it easier to analyze or to generate more optimized code. So, that's just a bunch of keywords about the kind of compiler analysis we do. Use the chains, computing the effects of a function on memory, on argument, on global memory. When you call a random function, there is a side effect on a state. So, your function is not pure. So, there are things that you can do with your function, things that you can do. You need to do a regular expression on the AST to replace an expression by a simpler one, a constant expression, which is basically equivalent to the constex keyword in C++, but down without any keywords at Python level. Laziness analysis, computing that least comprehension could be transformed into a generator expression, which avoids generating the world list. Computing whether an expression is pure, whether it has side effect or not. If it doesn't have side effects, then we can move it around. If it has no side effects and only constant parameters, then we can fold it at compile time. So, you can write Fibonacci of 20, and Python will realize that Fibonacci is a pure function and just compute that at compile time and replace the result by the variation of this function. You can compute range for some values, stating that this variable is going to be between 0 and 20, and then maybe perform some optimization based on that. This kind of optimization we do is a generalization of loop unrolling, but for any iterator. So, you can unroll a loop on a set, on a list, on a tuple, and that would work, constant folding, but interprocedurally, thanks to the analysis I put, remove some modular operation, which proved to be a very costly operation at the assembly level when it's done on induction variable, that's something people tend to do when doing image processing. Forward substitution to avoid temporaries, instruction combined to make some patterns appear. Simplify the code based on the range analysis, remove that code, all this kind of stuff can be done at the Python level. Then you say, okay, it is basically replaymenting GCC, but at the Python level, what's the use? But GCC does not understand the semantic of a Python code. It doesn't know that NumPy.1 has no side effects. So, doing this kind of stuff at the Python level, it's just because once you are at the, say, LLVM level, at the byte code level, you don't have this information anymore, so it's the right step. There is several layers of abstraction, and at the Python level layer, you can do some optimization, and I don't do register allocation at that level because that's not my job. But wait, there are other compilers, and Numba, for example, expects your code to look like that. If it looks like that, there is loop, variable declaration, and then you can compile that to efficient code. But you can also write a Fortran code that looks like that. We are in the 21th century. You can expect to write higher level code that still performs the same operation. Actually, this code can be rewritten in NumPy using that. And that's higher level. There is some temporaries that are generated because of the code to SAM or because of the array expression at the end. But that's easier to maintain. Scientific people tend to write this kind of code, and then they say, okay, I want performance, and so I remember I used to write C, and they go back to this implementation. Even in Fortran, you can write this kind of code. So there is no reason why you should stick to that. So Python tries to compile this kind of code, which proves to be more difficult in a way, but that's why you need a compiler. One of the most challenging kernel I had to handle was this one. It was on Stack Overflow, and the guy just pushed his code and said, okay, this is slow. How can I make it faster? And basically, the answer was, okay, that's high level code. Write it at a lower level. Make all the loops explicit. There is still plenty of loops. And then you can call Siphon on Numba, and it will get faster. And that's very pragmatic. I'm not stating that Siphon or Numba are bad compilers or whatever. They are very pragmatic, and they work, which is something very useful. But still, we can dream a bit, and when it's not your work, you are allowed to dream. So Python tries to optimize this kind of code, and we can reach performance similar to Numba or Siphon on this code, while not writing all the loops. So that's why we try to do, but I'm relatively alone. There are a few power users, so some people did use Python to compile codes that went into a small robot in the Baltic Sea. So there is Python code that runs under the sea, which is very cool. There is a firm in France, in Grenoble, that used Python for their daily task. Some academic works have been published using Python as an engine, not just as a subject of interest. And Martin did introduce some way to use Python in PX. And there is, also there is not a lot of developers, there is a lot of bug reporters, which is cool because when your work is useful to someone, you're happy and you're more motivated to spend a few more hours in during the night to improve your stuff. And some users' suspects are very nice. And as it's a small community, you tend to learn people to exchange not about only code but about any subjects. And I do appreciate that part of the open source life. But the road is very long. Supporting NumPy is a tremendous task. We're not supporting the World NumPy API, even if we try to improve on that. Moving from supporting only Python 2 to supporting Python 2 and Python 3 was very difficult. Supporting OS 6 was okay. Supporting Windows is getting okay since next week, not last week, but it's still difficult and it's only for Python 3 because Visual Studio for Python 2 is stuck to Visual Studio 8, 2008 and it does not support C++ 11 at all. And so where would I find the motivation to go on that way because it's not an easy way? So my opinion, when it's not your job, you have to find an interest. So either it's for fame, but that's not exactly the case. Or because it's interesting as technical challenge. And one thing that deserves Python in a way but also I find fun is you have to be good in optimization, low-level optimization, understand assembly when you try to debug vector SSE instruction. You learn a lot about the Python language because you're manipulating the AST. So you know stuff that exists in the syntax and then you say, oh, I didn't know about that. And then you dive into the language and you write a lot of modern C++ or C++ that tries to be modern. And there's a lot of stuff to learn that. So just for me, it's interesting to write this code because I learn a lot. But it's also interesting because I tend to meet other people and the scientific community in Python is very friendly. The first time I went to SciPy, it was really I used to go to academic conference and it's not saying nothing that people in academic conference are less friendly than people in SciPy. It's just not the same kind of people. So I really love speaking with these kind of people, sharing ideas and then you learn tricks and optimization and benchmarking and you just grow in knowledge and friendship with people that you would never have met otherwise. Even if your project is in a niche, there are still interesting people to meet and that's very cool. The kind of thing you discover is, okay, Jupyter exists five years ago. I didn't know about notebooks and then I saw this conference and I tried, okay, SciPy does it so we should be able to have this Jupyter magic and now there is a Python magic. You write your code, you call Python with your compiler flag, just the same as GCC flags and it generates native modules imported into the kernel and you can go on. You learn about capsule. A Python capsule is an opaque object around a native function or a native data and it just provides a minimal interface for embedding a string, an embedding pointer and it is used to pass data from native word to Python word back to the native word. Python can generate functions like here's a full function. We'll accept a pointer, a pointer to a matrix and it generates not a Python function that can be called at the Python level, but a capsule that embeds the native C++ functions hidden generated by Python and then you can pass this to SciPy as an optimization routine and it works so there is no Python glue anywhere and SciPy native code calls Python generated native code without any overhead and I didn't know about that but Martin told me about that and he said, oh, that would be cool and because we generate pure native code that's easy to do in Python and implementing that took me two days because of the original design and well, that's just things I didn't know about in Python. You also discover that there is a standard to represent floating point number, normalized but NumPy doesn't care about it which means that when you do complex numbers operation and the imaginary part is not a number and you apply this by infinity, what happens? Who cares? I don't but the standard does and NumPy people don't care either and I discovered that because my code, my native code was running slower than NumPy code and I couldn't understand, I looked at the code okay, that's complex duplication and then I looked at the binary from NumPy because most of NumPy is written in C and say, okay, that's not the same complex operation and then you discover things because Python can generate vectorized code not vectorized like vectorized array operation but vectorized like using AVX or SSEs that are available in modern processor I developed some more skills on that mostly based on Boost, SCMD but also you know, learn to debug that so that's just technical skills that happens to be funny but wait, I also have a family I'm the proud father of two lovely girls and I want to spend time with them so there's two options either I teach them Python and I try that but that was not a success or I don't spend that much time on my laptop and spend time with my children so how do you find a balance between your regular work which has nothing to do with Python or optimization your family, your health if you want to do sport or whatever and open source either you sacrifice an element which is a possibility but from my side I try to make sure that when I do open source there's a benefit also for other items when you gather more technical knowledge then you're better at work and for instance tomorrow I will present something for them related to my work but as a side effect they pay for my travel to come today so that's cool you can try to meet friends or family I have sisters that live in Brussels I'm going to meet her tonight that's cool you can try to raise money because when your wife is playing piano if you develop she's not very happy but if you develop and you get money then she's happy so if you find a way to find your work then it's more legitimate to spend your time on your laptop you may be an idealist and things that sharing knowledge is a good thing and then just speaking in front of people or teaching scientific Python to researchers in France is something you enjoy to do and then you can do that so I'm trying to mix everything so Python is just a stone maybe it's a keystone of that but it enables a lot of things and to my mind it's a pet project so it's not work so at one time I was working with one of my former students and we wanted to be very good engineers so there was a very harsh review that was ongoing for every pull request and we tried to write the best code ever and after six months motivation was gone because when you wait three weeks and refactor your code and in the end it gets in but it's one month later it's not fun at all you do that at work and you pay for that but there is a balance between the previous thing and I wanted to be correct but not too much correct I don't care if I'm not supporting this subfeature because maybe someone will notice that there is an issue and raise an issue and then I will redevelop that way it's okay to do that because it's your free time and we switched from this very harsh review to still review but lighter review and now it's healthier and I don't have a Twitter account or trying to advertise a lot just doing my pace doing it at my pace and some people tend to use it even if I'm not advertising that much so as I'm not making a living on that it's okay and I'm probably happier like that happier but I still want more I have a funding to have Python being a piece of sci-fi or a piece of sage which is a good thing for the project because it's bigger kernels so it meets the limits of Python and I also get to meet a new community and the sci-fi community was very welcoming and I really appreciated that but they were also they also had a lot of requirements they say okay why not Python probably not but why not but first you have to support OSX, Windows and Linux and then what is the size of the binary you generate because we have requirements on that okay C++ no the size is going to be huge and are there a lot of contributors okay no that's not the case but till now the binaries generated by Python are very slim because there is when you don't focus on a field and then people tell you please optimize that there is a lot of low hanging fruit so I have a blog post later on you could click on that and that explains how I made my binaries 10 times to 20 times smaller just by using the compiler the right way using C++ the right way Windows support as long as you stick to Python 3 it's not that difficult project maturity is going to be difficult but just to showcase or to express my feelings six years ago I was hired at a company and they wanted to start compiling Python projects but then I was leaving okay you're leaving well instead of doing that on our own you can do that as an open source software and we will pay you for that because we already have a grant and the job will be done and that's okay that was a very good idea very nice from them and then there is a European grant from OpenDreamkit for improving Python improving it's used to I regularly give teaching lessons about numeric Python in the group Calcule from France and I appreciate their help and I try to be active in the French community on Linux FR they are very friendly and well I just enjoy writing it helps me make my mind more clear and also some people tend to enjoy reading it so it's okay and sometimes very strange things but very happened I received this in that mail that was two years ago and that was okay hi you don't know me I read your PhD thesis when someone told you I read your PhD thesis that's strange very strange but they investigated my work and they say the thing you've been doing in Python it's nice I think you have ideas that can be helpful to us and then we work together for one year that's cool I mean that was completely unexpected but just saying thank you was nice but going that way was very nice and last December there was someone from Google Brain that just sent me an email saying okay we are using GAST your Python 2, Python 3 layer and it works, it just works we are happy, thank you and wow that's great that's the best thing you can expect so that's my little story use Python if you like to contribute to open source because it's fun and if you have any questions we have 10 minutes or so hello Stras okay very nice talk by the way I really enjoy it I would like to know since you are using NumPy do you rely on BLAST or LaPack implementation or is it pure C++ implementation no we are not that mad for all the dot operation we are falling back to the BLAST what we can do that NumPy can't do is the BLAST API is not only about matrix multiply so we match patterns and we say okay this pattern is implemented in BLAST so we generate the right call so and we just we can use BLAST or open BLAST as a backend I never tried with MKL but I assume it should work that's my next question thank you thanks for the great talk I was wondering how does Python compare to NumPy and Python in terms of performance so the question was what about performance I didn't show any benchmarks here because I want to make friends basically it depends so first NumPy is a jit so there is a jit compilation time it's cached but it depends on the usage so I will speak about Python, Siphon, NumPy and Python there are a lot of kernels that Python supports that are not supported as is by NumBar or Siphon because you have to expand the loop and in that case we generally match the NumBar performance Siphon is generally a goal so we try to be as fast as Siphon generated code while keeping a higher level input language or input and without any annotations sometimes NumPy is faster sometimes Siphon is faster sometimes Python is faster one place where we shine is when you have for instance it was at the beginning I can go back to that this function the Rosenbrock function we are especially fast on that because we know how to generate AVX instruction for this kernel and Siphon relies on GCC or CLANG to generate these instructions but it's too late because of aliasing so it's not as efficient as manually generated patterns because it's from a compiler point of view it's a difficult task it's difficult now but still difficult to generate good vectorized code I am no but not for sure that NumBar has a decorator that updates Ufx but I'm pretty sure that they don't match Python performance for that kind of kernels then on some situation for instance for the grayscott for this kernel Siphon is still slightly faster than Python but there is a lot of work to match the lower level interface with that interface NumBar supports classes Siphon does support classes Python does not because in Fortran there is Trun like Fortran and there is no class in Fortran so that's legitimate and it's not a big deal but some kernels I can't compare to them because we don't support the same input basically that's the idea we all go the same way the single thing Python has that the other don't have is native vector instruction support and in some situation compiler optimization I didn't speak about that too much but I have a blog post on that the modulo operation is not optimized by GCC or clang and if it's not optimized by clang it's not optimized by LLVM in fact and so neither Siphon nor NumBar took advantage of the optimization we developed in Python so in that specific case we are faster because we have Python specific optimization thank you I have a question about dependencies in generated C++ code what does GCC or clang compiler depend on you mentioned some algebra library BLAS I suppose standard library is there something else you need to have to be able to compile currently we depend on the standard library for all the random stuff for instance for vector implementation we did not implement that we depend on BLAS for the linear algebra and that's all and we depend on boost.cmd for an abstraction layer for vector instruction it's shipped with Python so you don't need to install it if you're concerned about dependency when I install my code then you just need modern C++ infrastructure and that's okay plus the BLAS but you have them because you have the NumPy installed probably not 100% sure and that's it I'd like to ask about do you use SIMD instructions inside C++ code? do you use intrinsics, compiler intrinsics? not intrinsics because of boost.cmd which provides target independent abstraction layer you manipulate vector data and it just generates the right intrinsics depending on current platform not for old platforms but x86 is okay and ARM for example it's probably okay but not latest instructions I'm not 100% sure thank you very interesting I have a question about array operations and NumPy so one of the performance killers when it comes to array operations is auto creation of temporaries where you will just write arithmetics on arrays so if I understand correctly the question is when you have that kind of expression that's a big array expression the last assignment in NumPy when you do that there is a temporary array that is created for each node in the expression and that's a double source of throw down the first source is from memory locality perspective it's not good because you're allocating new memory and writing to it so the data are not in the cache they are created to hold the new value and it's also bad from a loop pressure point of view because there is a loop for that operation and a loop for this one and a loop for this one so there is a lot of loops and not good memory locality and the usual way to catch this pattern in C++ it's to use expression templates so that basically you delay the duration of the expression until it's assignment and that's what we do so we have expression templates and we also but they are I think this kind of expression is relatively easy the rosin one would be more tricky what? the rosin example would be more tricky the rosin was relatively okay this one is difficult and capital U and capital V are views on other here lower case U and V are views on uppercase U and V so you have an expression that creates a view and then you use this view even just here and you update the contents of capital U through the view of smaller case U and doing that expression templates is tricky but it's also a big way to learn that you have move semantics also for member functions in C++ there's a lot of things to learn when you want to optimize this I'm not saying that the beautiful code I ever wrote but it turns out to work which is already a good thing but that's a difficult part thanks see you again and have a nice weekend


Internal Revenue Service IRS tax news IRS and Treasury issue final regulations on e-file for businesses Final regulations. I don't buy it with the government and regulations. Nothing's ever final. They love regulate You give them half a chance. They're gonna increase the regulations.  Remember the covet thing where they're like, don't worry Just 15 days to slow the spread. That's all we need And then it's like five seconds later, you know, five seconds later, they come back and they're like, okay We just need to like social distance for like 60 days and that's it And then they come back again and you're like the goal the goal of these people and they're like last one I swear you can trust us. All we need is for all small businesses to just shut down You know, that's all we need. We just need to shut down the businesses And then they and then they come back again and they're like, this is it. This is the final regulation I'm for serious this time. I'm for serious. You can trust us We just need to mask the children. You know, that's it.   You can see it in my eyes, but that's what I was born to do It's like my I was here to rule by by by government mandate here So in any case, we learned this one from tick tock and one of our top department of energy officials who always has these great ideas And this one everyone just needs to like take your dog's muzzle Modify it so it fits on your face And then only use the restroom when escorted by government approved potty experts Who will yank your choke chain if you miss the can or if there's like too much splash or something because you didn't aim You know for the side of the bowl In fact to avoid forcing government approved potty experts from strangling you by choke chain It's recommended you just sit down What no, that's not another rule. The sitting down thing isn't another rule It's just like an addendum to the last rule. The last one was the last rule This is just like a little modification. Anyways, why are you even whining about it? It's for your own safety for crying out loud. It's like what did do you want to be strangled to death by choke chain? It's a final regular talk about a slippery slope letting the government add regulations is way more Hasidus than those gateway drugs the dare program used to warn us about in high school I'll tell you what? Yeah, we we need some kind of school intervention program warning about government overreach or something like that Wait a second I I thought that used to be like history class Or like when they used to teach about the u.s constitution and stuff But wait a second The government would never lie about an emergency Just look at honest joe biden as he struts outside in ukraine Holy crap take cover mr. President those were air raid sirens. Wow.  President doesn't even check the skies for bombs or anything Like if the bombs did come they'd probably just bounce off his chest like he was superman All of your diseases are in perfect balance. Here's the door to your body You see and these are oversized novelty germ. Here's what happens when they all try to get through the door at once So what you're saying is I am indestructible. Oh, no, no in in fact even a slight breeze Unless unless possibly he's just lying about the state of the emergency Possibly staging the entire attack siren warning thing I mean didn't the administration admit they told russia that president biden was visiting ukraine So they shouldn't bomb it And the sirens went off right as he was walking outside with a perfect camera angle Like i'm i'm starting to think the only danger he was in What was from having a heart attack from the staged and timed siren going off thing My ear canals are very sensitive Look at this Huh?  Oh good. Thanks I are 2023 dash 31 february 21st 2023 washington the department of the treasury and the internal revenue service today issued final regulations Amending the rules for filing returns and other documents electronically e-file These regulations will require certain filers to e-file beginning in 2024 You got the td 9972. There's a link to that here affects filers of partnership returns corporate income tax return Unrelated business income tax returns withholding tax returns certain information returns restriction statements disclosure statements notifications actual reports and certain excise tax returns lot going on here for those those that will be affected The final regulations reflect changes made by the taxpayer first act the tfa To increase e-filing without undue hardship on taxpayers. So obviously they're trying to get the e-filing Up in this is an attempt to help with that effort So specifically the final regulations reduce the 250 return threshold enacted in prior regulations to generally require electronic filing by filers Of 10 or more returns in a calendar year The final regulations also create several new regulations to require e-filing of certain returns and other documents Not previously required to be e-filed Require filers to aggregate almost all information return types covered by the regulations To determine whether a filer meets the 10 return threshold and is required to e-file their information returns Earlier regulations applied to the 250 return threshold separately to each type of information return covered by the regulations eliminate the e-filing exception for income tax returns of corporations that report total assets under 10 million dollars at the end of their taxable year and require partnerships with more than 100 partners to e-file information returns And they require partnerships required to file at least 10 returns of any type during the calendar year To e-file their partnership return. So to help with this process the irs created a new free online portal There's a link to that here last year to help businesses file form 10 99 series information returns electronically So obviously those are the ones you got to send out to like contractors and whatnot So electronifying those types of returns No one has the information return intake system the iris of the iris the iris is watching us So we want to make we got to we got to make sure that we're in the iris Sites here. Let's take it upset. So this free electronic filing service is secure accurate and requires no special software Though available to any business of any size iris May be especially helpful to any small business that currently sends their 10 99 forms on paper to the irs So if you're sending paper returns, then you might look into the iris system So in recent years there has been tremendous growth in the availability of e-file services and the use of e-file across the tax filing spectrum in 2021 about 82 percent of all corporate income tax returns were Were e-filed and almost 90 percent of partnership tax returns were e-filed Further reducing the volume of paper returns filed Freeze up staff and resources to further enhance services for taxpayers and improves overall efficiencies while reducing postage printing shipping and storage and their associated costs and burdens So the irs receives nearly four billion information returns per year and expects to receive nearly five billion by 2028 and 2019 the irs still received nearly 40 million paper information returns Even though approximately 99 percent of all information returns for that year were e-filed So the final regulations generally provide hardship waivers for filers that would experience hardship in complying with the e-filing requirements and administration Exmititutive exemptions from the e-filing requirements to promote efficiency and efficient tax administration So there's links to those rules here and there'll be a link to this in the description


Today, we're going to discuss a pretty intense rocking progression warm-up exercise, a ground-based abdominal mobility promoting exercise. This is the rock to squat, but it's a rock to a single leg squat or a pistol squat. Without further ado, let's try to get it on the first try. Okay, so we're rocking and we catch and I fell. Try again, switch sides, just like that. So you got to make sure that that leg that's down, you got it. You got to make sure you're actively pushing there. I can't just relax because then I'll fall, okay. I've got to make sure I, this is not passive. I can't hang out here for very long and it's hard to talk here. If I do this, then I can kind of jam into my ankle, but that's not giving me the clearance that I need for my other leg and it's not promoting stability. It's not promoting the stabilization that I need to squat deep. Squatting deep is not about getting down there. It's about holding a good position when you're down there. If you've watched any and all of these videos, that is the biggest lesson that I have for you. You need to be able to maintain your position under duress, okay. So that duress could be speed. It could be lower mobility, squat depth, it could be force, it could be people yelling at you, I don't care what it is, I want you to get better at all of them, okay. So if you want to progress from your rock to squat, the rock to single leg squat is a great next step.