Okay, so this is actually the CHIP Ultra Cap. It's short for our reflowable Ultra Capacitor. The world's first practical reflowable EDLC or Ultra Capacitor. It's designed for SMD applications and it survives a ROHAS compliant JDEC standard reflow profile. We actually designed this product initially for solid state hard drive power loss protection applications. The idea is that today in a solid state hard drive about a quarter of the real estate on the board is occupied by usually tantalum capacitors that are needed for onboard energy storage to support the drive on a power loss event to give it time to move data from RAM to flash. Our device actually can replace the tantalum capacitors, the same tantalum capacitors just with two of these devices, so a much smaller footprint and a comparable price point for the solution. So we think that this is actually a no-brainer for SSD manufacturers because it frees up that real estate on the board, which is actually some of the most valuable real estate on the planet as it turns out. There's a lot of energy in this device. It's a 400 millifarad Ultra Capacitor. It's 2.3 millimeters thick, so it's specifically designed for SSD applications. It's also very low ESR, so relatively high power and it's designed for 85 degrees Celsius and 2.1 volts in very long lifetime. Our business model for this is actually a licensing model and that's because projected volumes based on our customer feedback have been in the tens to hundreds of millions of units per year. We want to make sure that we get this technology to market quickly so we are working with major capacitor manufacturers to license the technology. There are about six or seven in the world and we're working with about five or six of those major capacitor manufacturers today doing testing and evaluation and moving toward licensing deals. We expect first production in 2019 from some of our licensees for this chip Ultra Cap. I should also say the chip Ultra Cap is enabled by, I guess you are right, so I should also say that the chip Ultra Cap is enabled by our materials innovations including the electrode which is a binder-less composite electrode so it has no polymer binders that would otherwise melt or disintegrate in a solder reflow process which is a very high temperature process and also on our electrolyte system which is a very stable electrolyte and very low vapor pressure for both good temperature stability in the reflow process but also in the end use operation which can be relatively high temperature but for very long periods of time and that electrolyte is also very innovative because not only does it provide good temperature and life performance but it also provides good electrical performance in terms of capacitance and ESR. So is this the first one that's reflowable? It's the first what we call the first practical reflowable Ultra Capacitor so this has been attempted in the past with some sort of serious limitations in some cases they have been extremely high ESR Ultra Capacitors, orders of magnitude higher than this device which essentially limits the applications severely to just a few sort of niche applications. In some cases the reflowability is very limited after sort of one reflow cycle the performance degrades substantially and then after a second reflow cycle again it degrades the applications and normally the standard in the industry is that you have to survive at least three reflow cycles our device actually sees little to no variation in performance after even six reflow cycles and the ESR is in the hundreds of milliohms range or even less than that which makes it the world's first practical no excuses no excuses Ultra Capacitor. And this one you say is double A does that mean I could just put it in my microphone or what is it going to be for where would people put this? Alright so this is actually a high temperature wound cell this again is our first ever product so we developed this for 150 degrees C operation and today it's used in oil and gas down hold drilling applications and also industrial applications for instance in oil and gas when they are below the surface they see temperatures that can be 100 to 150 degrees Celsius while they are drilling just from the earths from the sub-training environment and they need this as a pulse power buffer or a backup rechargeable energy storage technology for powering the down hold tool so to speak which is an instrumentation and telemetry system that they use to steer the drill bit during the drilling operation and industrial applications they are used in things like high temperature sensors or temperature loggers and things like that where they need a power source that operates beyond say 85 degrees Celsius where most other energy storage devices fail. Is it allowed to do with backup power? When you talk about the SSD you don't want it to stop working it should just work forever? Yeah that's correct so this is mostly for a backup power application so basically when there is a power loss event this will allow basically to deliver the energy that is necessary to move data from the hard drives to the RAM memory so it's a cash to flash memory and well also one other thing that I like to add is that this can be reflowed but when we say that means that we can do this for like five or six times in a row and we don't see any sort of variation in the SR and also in the capacitance which is a very important point so every time that we reflow it there is no such a variation which means that it's very compatible with the process. Are you working with the automotive market or the airplanes and stuff like that? So currently we are mostly actually working with let's say capacitor manufacturers and also with the electronics industry that's the target for this application. We think that in the future this part could also be applicable in automotive industry as you know we're also working on a version of this that can work up to 125 cells.


Hi everyone, thanks for coming to open source summit specifically for coming to this talk analytics at 1,000 qps and beyond I'm Gian Rellino, and I'll be talking about how we do this with Apache Druid So, okay. Who am I? I'm Gian Rellino. I am a committer, and I'm the PMC chair at Apache Druid And I'm also co-founder at imply a company that is all built around Apache Druid Well, we'll talk about today first a brief intro to Apache Druid Then I'll talk about the three levels of high load analytics We're going to talk about level zero one two, and then we're talking about how to try this at home So first off a brief intro to Apache Druid. What is it all about? We like to think of Druid as an engine a rocket ship. It's a database. It goes really fast But what Druid really is is it's a big calculator. It's a big fancy calculator Scales to thousands of servers does analytical queries really really quickly Druid's purpose What it was originally created for was to power analytical applications I've got a screenshot here of an example. This is imply pivot a analytical application that we build at imply And the thing that makes analytical applications What they are is things must queries must respond very quickly. There's gonna be interactivity between the user and their data lots of pointy and clicking lots of Very rapid fire interactions think applications like Google Analytics So some people use Druid with imply pivot the app that we make many people make their own apps Apache Druid speaks sequel and you can build your own app on top of that Druid's use at a lot of different places So there's a here's a bunch of articles that were written in the last few years about Druid use Druid enables analytics at Airbnb Netflix uses Druid for real-time insights ensuring a high-quality experience Salesforce uses Druid for insights as well and Mopub a Twitter company Uses Druid to query terabytes of data in seconds Some stats here in the top right of the slide Of what's happening in real Druid clusters. So real Druid clusters are doing a hundred plus billion rows a day of ingestion Retaining over a trillion rows over a year of data Hundreds of servers in a cluster the largest cluster is actually over a thousand servers And all this was sub-second to a few seconds of query latency and a mix of streaming and batch ingestion So Druid is a database Druid's a very scalable database a very fast database And for these kinds of applications It is super super important to be able to do large numbers of queries and at the same time Maintain that high query speed and consistent query performance And so I want to talk now a bit about how we do that and what Techniques we take in the software itself injure itself and also what techniques people that deploy Druid do On the deployment and configuration side in order to achieve what we're trying to achieve here so For this talk, I want to talk about three different levels of how this can be done and each of these levels Is kind of going out from the hardware into configuration. So the first level zero. That's about CPO and IO Really here focusing of the software itself in the hardware Level one talking about scaling. So that's the next level up talking about how you take How you take the software and how you scale it out to a cluster in a reasonable way And then level two handling heterogeneous workloads level two is once things are scaled out Now you may have many different kinds of workloads running on the same data and how do you handle that? So we'll talk about each of these levels in turn each one has some techniques. I want to highlight The first one level zero the most innermost level CPU and IO So the goal here is to minimize the CPU and IO required to execute a query This is all about local efficiency These techniques apply equally to single servers or large clusters the ideas if you had one server You want to do as much as you can on that one server be as efficient as possible And if you have a thousand servers the way to get good performance at that thousand server scale is to make sure each individual server Is running as efficiently as possible? There's many techniques to achieve this and I'll discuss some of them but first off why does this matter well it matters because Efficiency at this level at the level of CPU and IO at the hardware level Maximizes the number of queries that can run per unit of time for a given CPU and IO budget And the idea here is that less resource use per query means we can do more queries Good way to think about this is sort of like a bin packing problem Imagine you have a hundred CPUs and each query takes ten seconds of CPU time If you make it the most efficient use of these CPUs possible you can run up to ten queries per second because you know ten times ten is a hundred and So that means that if you want to scale beyond ten queries per second You're either going to need to add more CPUs or you're going to need to reduce the amount of CPU time each query takes That's what that efficiency is about So I want to talk a bit about how we do that The first thing I want to talk about is the software the most fundamental aspect of Druid is that it's a column oriented and segmented database What I mean by that is every table is split up into segments So if you have a table of a billion rows that might be split up into 200 segments for example Each segment tends to have a few million rows in it and within each of those segments We store data in a columnar fashion with indexes So what you're seeing here in front of you is a schematic representation of one segment of data So this segment has eight rows in it You can tell because the data section of each column here time artist city price and count The data section of each column has eight elements in it Real segment would have millions, but we can't show them on one slide So an important thing here The first thing to see here is that each of the columns is stored separately. This is important because Real queries tend not to use every column in a data set They tend to use maybe two or three four or five columns out of a hundred row data set for example And that means that that storing the data separately column by column is very helpful because it minimizes the amount of IO we have to do It means we do not need to read any of the columns off disk or even from memory to the CPU Meaning we're saving on IO and saving on RAM to CPU bandwidth Let's walk through an example of how we would do a query using this columnar data So here in the top left of the slide, we have an example SQL query. We might run this data set. This is a Ticket sales data set. So let's say we want to see Total price of tickets sold grouped by city Where the artist equals a particular artist just in in this example So that's select city sum of price From sales this table where artist equals Justin grouped by city And I'm gonna Just focus on what happens on this individual segment. Of course in real table or in real Query we'd be querying many segments and then merging the results of those different segments But I'm just going to look at what happens at one segment for for this example So the first thing that we're going to do is we're going to resolve the artist filter So artist equals Justin and we're actually not going to look at the data section of the artist column for this We are going to look only at the dictionary and the index The first thing we're going to do is we're going to look at the dictionary to see What the dictionary code is the id for Justin in druid We use a dictionary to assign an id to every value of a string column like this So Justin has been assigned id zero, which we will find out from the dictionary Then we'll go to the index And we will see that id zero corresponds to this Rows zero one and two the index has the same number of elements as the dictionary It's a one-to-one there the first in this entry is for the first dictionary entry and so on So once we've done this we now know that rows zero one and two are the ones that match this filter And we did this without having to actually read the data out of the artist column using these indexes Next we're going to go to the city and price columns And we're going to walk the data sections together So we are going to first look at And we're only going to look at those first three rows We're going to first look at city two price 1800 then city one price 2912 then city two and price 1953 We then group those together And then finally use the dictionary of the city column to replace those dictionary codes one two With the names of the cities la and sf finally giving this result that we have on the bottom of the screen La 2912 and sf 37 53 And you can see that the advantage here of storing things in this way and doing the query in this fashion Is that we never had to read anything out of the columns that weren't being used So count and time were not read at all There is no need to copy data from disk to memory or from memory to the cpu We also did not need to use the data section of the artist column very valuable We did not need to scan the index even of the artist column because we were able to use the dictionary And then do a random access into the index to get the bitmap in question Then in city and price we did not need to read any rows that didn't match the filter So we only read those first three rows So this is really being very economical about What data we access in order to resolve a query And therefore minimizing the cpu and io required to do a query There's many many other techniques that we use like this, but this is an example of some of the most basic techniques Next I want to talk about locality. This actually builds on the prior techniques It builds on the column orientation to make things even better. Let's say that we are doing a lot of filters on artist This is something that's a really common pattern in analytical applications It's common to have a specific column that you are almost always filtering on it doesn't always happen But it is very common and when it does happen we can take advantage of it And the way we take advantage of it is by sorting data based on that column So let's say we're going to sort the data based on the artist column What's going to happen here is that instead of all the artists being all jumbled up inside each segment We're going to see that all the zeros are together all the ones are together and all the twos are together of the three artists And every other column is sorted the same way as the artist column So we're actually resorting the entire data set based on the artist When we do this if we filter on artist You know, let's say that we filter on artist number one We with the with the unsorted data on the left We see that we actually have to read every block The data set is split up into four blocks here for this column We have to read every block because there's a one in every block With the sorted data on the right We only have to read two blocks because the first and the last block did not have any ones in them This is really helpful. Um, this further minimizes the amount of IO we need and further minimizes the amount of cpu We need four queries that involve filtering on artist So here's an example Using query profiling a really powerful technique where we see exactly where cpu time is going An example using query profiling to show why locality is so powerful So I mentioned blocks in the previous slide that that example with four blocks in druid We store data in blocks in the segments. So each column is split up into blocks of a certain number of values And we read one block at a time and each block is individually lz4 compressed so what we're seeing here is that This is a query profile where most of the time is being spent in this function lz4 decompress safe This is the lz4 decompressor and this means that we're actually spending most of our time Reading and decompressing blocks this actually counts both the time to read the block and the time to decompress it And this is in this example The data set that we're running the query on we're doing a high concurrency query workload Generally filtering on a specific column. This is an iot data set. So we're we're filtering on the particular time series But we're not sorting by the time series And what that does is it means that all the the data for an individual time series is all jumbled up And so we're spending all this time reading blocks On the next slide, what I've done is We've rerun the test And in this version of the test we have sorted the data by the time series by the name of the time series And what this does is now instead of spending 90 of the profile on lz4 decompression You can see we're spending a much smaller percentage and you can't see it in the profile But we did measure this to be eight times lower cpu time Which is really powerful and this shows why it's so powerful and so important To ensure that your data is sorted in the right way to achieve the right locality Because this 8x difference in cpu time 8x difference in io as well Is an extremely powerful thing Another important technique in level zero is deferring computations So the key insight here is the data size generally decreases in the later stages of the query So imagine you're doing a query that does a filter followed by a group buy That filter is going to throw away some rows and that group buy is going to generally reduce the size of the data set The idea here is you want to move computations as late as possible So here's an example Let's say we're doing this query. So we're grouping by the lower case of a user So take the user convert this to lowercase And then we're counting the number of records for that user from a traffic table And then we're ordering by the count descending So this is the top users by number of hits in a traffic table So in this case, we're grouping by lower of user and selecting lower of user So we're grouping out an expression which requires computing the expression during the irrigation step This example took six seconds to run Now one transformation that that you can do Is instead of grouping by the lowercase of the user, you can just group by the user This defers the computation till after grouping It's not done automatically because results are not the same if the function might map two inputs the same output So let's say your users you could have two users that Are different in case, but have the same lowercase representation In that case These two queries return different results, but let's say you know that won't happen Let's say you know that users are unique in a case insensitive way And you just want to lowercase them for display In this case you can group by the user and then select the lowercase version for display But use the the column directly for grouping In this case the query runs eight times faster It runs eight times faster because that expression is being deferred Until later in the query I don't have a query profile of this query But if you were to look at one You would see that the version that groups on lower of user and selects lower of user The one on the earlier slide you would see it spends a lot of time computing that lowercase function This is one reason why looking at query profiles is so powerful It tells you exactly where cpu time is being spent and helps you reduce that cpu time So next let's move on to level one scaling talking about how we scale out these individual servers into a cluster And issues we run into along the way So with scaling we have two goals The first is we want scaling to be lightweight We want performance of distributed queries to be similar to local queries single on a single server The idea is that when we're going from one server to two, so let's say Maybe you're splitting up a single server. You're splitting up a 64 core machine into a 32 core machine When we go from one server to two, uh, we don't performance be very different The second is we want to scale linearly So we want 200 servers to handle two times as much load as 100 servers if they're all the same kind of server So what are the impediments to that? One of the main things that can go wrong with Lightweight scaling that can make scaling much heavier than it needs to be is having a lot of data movement So the performance degradation from one to two servers can be high You could actually have heavy weight scaling if data movement is high So there's many techniques to reduce data movement between servers during a query. I'll talk about a few Uh, one is sketches. So sketches. This means uh ways of doing approximation using much less data transfer than the exact solution So examples are hyperlog log and data sketches These sketches are useful for count distinct medians quantiles approximate joins that kind of thing Another is another kind of approximation. There's a little bit of a theme here Approximate top ends are useful for top ends style queries, especially on columns with billions plus of unique values Another technique that's not an approximation Is to distribute joinable tables to all servers in advance of a join. So this speeds up a join One reason that joins are famously resource intensive and distributed systems is because of all the data movement that is required But if we can distribute those joinable tables ahead of time Then there's actually no data movement required to do a join. Uh, which makes them Just as inexpensive as doing a join on a single server This makes the broadcast part of a broadcast join free and is a very powerful technique Next I want to talk about uh impediments to linear scaling and the biggest one there is the number of servers involved in each query Now I want to talk about why this matters it matters because uh Of the variance in performance between servers, which can just be down to random chance So imagine a situation where each query on each server has a point one percent random chance of running slower than normal So with one server, that's no big deal Most queries are very fast and the overall throughput is very good only point one percent of queries are running slower than normal But with 50 servers Each individual query has a five percent chance of experiencing at least one slow server because it's got a point one percent random chance per server And that really uh, it adds up the chance that no server is slow the chance that they're all going to be fast. Um Is not 99.9 percent. It's only 95 percent And then with 500 servers it gets even worse. There's a 40 percent chance Uh, that at least one server is going to be slow The overall throughput there is significantly reduced because 40 percent of queries are getting hit by a slow server So what can we do about this? Uh, recall that in level zero We took advantage of the fact that many Use cases involve filtering on certain columns Uh, these are things like time user ID, etc In level zero, we use that to sort the data by those columns. We're going to filter on thereby improving locality Here in level one, we're going to use it for partitioning. We're going to partition the data set with those columns Now the idea is that when we partition data set with those columns, let's say we have a thousand segments in a data source Each if we partition by time then each day is only going to appear In a relatively small number of segments each day of data will not appear in all thousand segments It might appear in you know, let's say five or six segments Or if we partition by user ID same thing for user ID What happens here is each query uses a subset of segments instead of all segments Which minimizes the number of servers that need to be involved because the number of segments is low Another approach you can do here is speculative requests The idea here is that you send the same request to two servers Uh, in whichever one returns first you use that one and then cancel the other one The idea is that if you have one slow server and one fast server And you don't know which one is which you send it to both and you take the fast one This increases total load in the cluster because you're essentially doubling the number of requests being made But it does reduce variance Uh, we actually have not implemented this technique in druid although it is a technique that is used in other distributed systems Okay now on to level two heterogeneous workloads So in level zero we talked about minimizing the amount of cpu and io required for an individual query So we can pack more queries into the same unit of time Uh in level one we talked about scaling from a single server to a whole cluster without introducing new bottlenecks And now level two we'll talk about when you have a whole cluster How do you run different workloads on that cluster without them interfering with each other? So the the question here is what do you do if your workload is not uniform? Maybe you're mixing high concurrency app driven work with low concurrency But computationally expensive reports at exports and ad hoc queries The problem here is how do you avoid affecting application driven work? Which in most cases that app driven work is the work you really want to be very high performance So there's a couple of techniques we can use. Um, yeah, it's all about resource management There's a couple kinds of resources that we have I like to think of resources in terms of releasable resources and non-releasable resources So a releasable resource is a resource that can be acquired and given up repeatedly while a query is running An example here is time on cpu So a query can take some cpu time and then it can give up the cpu And then it can grab the cpu again and then give it up again. It can do this repeatedly In druid Any resources that are releasable are shared according to a query priority. So every query has a priority It can be assigned by the user. It can be assigned automatically By based on administrator policy there's various ways of assigning it, but each query does have a query priority and higher priority queries get these shareable releasable resources Before lower priority queries and they get to preempt lower priority queries Now with non-releasable resources the situation is trickier The example here or an analogy here is is a car on a freeway The space on a freeway is non releasable A car will take up space continuously until it leaves the freeway It can't pop up in the air and then pop back down. It must take up that space Once it enters and it can only give up that space once it leaves So the idea here Is to reserve some resources for higher priority work Going back to that freeway analogy Some freeways have a carpool lane or a high occupancy vehicle lane that is reserved for vehicles With one or two passengers in them in addition to the driver The idea is that we want to encourage those kinds of vehicles They're higher priority And so we reserve some space for them To ensure that they have better service We can apply the same idea to databases. So in druid resources are reserved using query lanes This is a Evoking that that carpool lane concept Similar to the carpool lane on the freeway The idea here is that there's certain kinds of resources like Certain kinds of buffers certain kinds of space and thread pools certain kinds of Scratch space on disk that are not releasable and must be reserved when the query starts and must be Retained by the query while it runs And those kinds of resources we have some set aside reserved for high priority queries So they cannot be starved by low-party queries In druid, we can also use server tiers for the strongest isolation Server tiers the idea here is that you actually have a separate pool of servers dedicated to handling certain kinds of queries Some people have a dedicated high-party Pool of servers that all their high-party queries go to Or some consider that their default set of servers and they have a dedicated Low-party set of queries or low-party set of servers. That's the low-party queries go to Queries go to These kinds of techniques Guarantee resource availability for high-party queries, but they can reduce overall efficiency. And so whenever possible, it's best It's it's better for more kinds of resources to be shareable And so we always endeavor to make as many kinds of resources shareable as possible in druid Okay, finally, you can try this at home. Of course, this is Apache con druid is open source software Stay in touch with us. You can check out the project website at druid.apache.org You can follow druid on twitter at druid.io is our handle and also join the community We're very active on slack github and the apache mailing list Go to druid.apache.org slash community Thank you, and I hope to see you there. Bye. Bye


All right, so welcome to the August 5th, 2021, TSC meeting. Looks like most of the folks here are people have been here before so you know all about our antitrust policy. That is one thing that we have to take into account during these meetings and then the second thing that we have to take into account is the code of conduct which is linked in the agenda. Of course, all are welcome to join the meeting. The first announcement is an announcement that you've seen many times over. But if you do have anything for the developer newsletter goes out on Friday, so feel free to add anything that you think is relevant for your project your lab your working group your SIG anything that might be of interest to the hyperledger developers and community. So, any other announcements that people have. And I will point out that Tracy originated the use the raise your hand button thing. So, if you want to say something, please use the raise your hand feature. All right, thanks right. I guess there is no other announcements so we'll move on to just the quarterly reports I see Sarah is here from Eroha I know that not everybody has had a chance to look at this yet. But there was one thing that Sarah brought up, or the Eroha community brought up around the assume CRS is the common repository structure is that correct Sarah. Hi, yes, so we got a little bit confused or we got, we got everything about the renaming master to main we're almost done with this apparently we just need to ask to remain the last two repos we don't have access to them we don't have permissions. But with a common repository structure I remember that someone from the hyperledger team, push something some sort of a tool about this some time ago. What was that tool and if that wasn't this exact thing. Because I think it was that ripple liner thingy. I don't remember the name correctly unfortunately I'm sorry. And this doesn't mean that we, we can just manually get and add those additional files in there. How does that work this it's a little bit confusing for me as a probably not that technical person and the team was not able to help me with this and with this tool. So if there is a way to explain to me in a very simple terms. What should we do and how it is connected to that repo tool. That will be awesome because I got a little bit confused with this. No problem. So what what we had discovered is that the repo link tool didn't work for everybody. So what we ended up doing was creating this common repository structure. I think it's, if you click on that it actually takes you to the link. I think it's in the TSE GitHub repo, which is just guidelines for actually son of wiki. We need to change where that link goes to, because we've created the, the new, the new guidelines on the TSE GitHub repository that basically tells you what the files are that are required to exist in the, in the, in the TSE. So the license, the code of conduct in the security have certain required content that has to be in there and then there's some other files that are required that are listed below and then some optional files that are kind of recommended or should. So if you go through this link, Sarah, it should tell you all of the sorts of files that you would need to have in your repository to be considered compliant with the common repository structure. So we no longer using the tool. Okay, awesome because that one was confusing. Can you please send me the link to the chatters. Okay, see. Okay, great awesome. Thank you so much. I'll look into it and yeah doing this manually might be the easiest way of doing it. Thank you. You're welcome. And then secondly, rise the repositories that they don't have permissions to is that something that you can help. I'm doing it right now. Okay, thank you. All right, any other questions and for for Sarah for the folks who've had a chance to read this me comments that people might have. If not, we can, you know, people still haven't had a chance we can leave this on next week as well for any additional comments. Sure, I'll join next week to if there are any questions or if there are any questions we can also like answer them in comments or anything I'm okay with any, any option here so yeah perfect. Thank you so much for joining. So we also have in the list of project reports that are due a report from Indian areas. So I think you were actually a week ahead of time which is great and we appreciate that. But if anybody has any contacts with the India areas team, let them know that their report is due so that we can take a look at those next week. I think there's also another one that's upcoming for next week as well so take a look at the project update calendar and see if there's anything there that you might be able to help with. Any questions on quarterly reports before we move to the discussion section to quiet morning this morning. No questions. Okay, well hopefully we'll have some good discussion in the discussion section. So the first thing is the incubation entry considerations. So I had a go at why had a listen to last week's TSE call and it sounded like what we wanted to do is kind of break up the discussion that we had into the considerations and the best practices. And then a stab at kind of putting that together at the top of the document, kind of taking what we already had and separating out into kind of considerations and best practices. I thought this would be a way that we could see if we could finalize this incubation entry considerations and so if there's any sort of comments or things that we're missing from this, this is probably the time to bring it up. Thanks a lot for putting this together. This is really nice. I think in the comments on the previous version. I had asked about having a section on community. What do you think about that. I do remember those conversations heart. Did we have anything specific in the community section that we wanted to add. Well, I guess what I posited last time. But we want people to be able to gauge whether their project is likely to be accepted. And one of the things that I've observed is that the TSC is more likely to accept a project that has maintainers or contributors with a lot of experience in the open source community. And particularly those with hyper ledger experience, then a project whose contributors are totally new to open source. And I thought maybe we would want to sort of put this somewhere. In this document, just to let people know that the TSC generally looks more favorably on projects with people with open source experience. And if not, you may want to talk to someone with more open source experience. Just something along those lines. Whether you know this document is totally binding or not. You know, that that sort of matters less to me. Compared to, you know, can people read this document and figure out if their project is going to be is likely to be accepted or not. Okay. I have an idea. Heart, I am in the process of typing something in so that we can take a look at and see if that is what you think is appropriate so the TSC is more likely to accept projects that have contributors familiar with open source practices. And that's what I've typed in. You mentioned, and even more so people who maybe have hyper ledger experience. Is that something that we want to add to this. My concern with that heart is that it may seem a bit insular. And so I'm a bit hesitant to add that clause to this but happy to hear thoughts on that. So my thought on that was that it does. I agree that it does appear to be insular. But that has been the reality of the voting process. Okay. Did you. So I was about to say the insular concern was my big concern with that even though that may be reality. If we put it in words and put it out there it's almost like putting up a big sign. No new people encouraged. And, you know, we could need to find a way to phrase it to encourage people to recognize that's what goes on at the same time still encourage them to join. So I put in some points about how hyper ledger labs is a great place to grow that experience, rather than saying if you're not in the club don't try because I mean it's it's no matter how we phrase it that's how some people are going to view it if we put any soft phrasing on that. So that's that's one risk with acknowledging that reality and writing. I like the cause and Dano I added it right if you wouldn't mind reloading and just scrolling down to the community section, which I added right be right after the maintainers. This is what I've added hyper ledger labs is a great place to grow this experience using the words that you. You mentioned there Dano. That's on on the way this is reading. I know your hand is up and I think you have a comment. Sure. Yes, thank you. So I like this. Can we also encourage people to to reach out. So, so what I, I agree with the, we don't want to appear as insular. What I was hoping is we could say that, you know, look, if you haven't been involved with hyper ledger, you know, please try to reach out with with people that are involved. So, I don't know if we could add something like that and say that, you know, hey, if you're not as familiar, you know, please, please reach out to people. I don't know how we want to phrase that. But I think something like that is good. You know, if you look at the Apache process. They have the whole champion role that is designed to help people that aren't as familiar. You know, get involved and I don't know that we need to formalize that, but maybe we could encourage people to reach out. Yeah, I know in the best practices section, the first bullet is a bit about that reach out. I don't know if that covers specifically what you're looking for heart or if we need to be stronger in the wording there. I don't know why if you wouldn't mind scrolling down to the best practices. Thank you. Before you respond as you read that heart Daniel you have a comment. Yeah, I wonder if we should rather than saying hyper ledger labs is a great place to start if we add another thing, you know, joining existing projects, not joining my joining participation in existing projects or starting new projects and hyper ledger labs is great way to develop this. So, you know, make it accessible to the existing projects as well. Participating in existing. I have added that Dano as a comment heart. I'm sorry, I think Grace is next. Thanks. First of all, thanks for seeing this looks really great and I really like those best practices down there. So, going back to the section above right even it's going up. Sorry. Thank you. So I think that when we're talking about the, you know, I would almost want to say around the community it's a nice to have like or having prior experience it is not a requirement or something like that. Because, as you said, insure, but it is a nice to have. And it's saying, or maybe saying with open source practices, regardless of hyper ledger I don't know exactly what I'm trying to get to but just trying to open that a little bit more because that's because yeah, it is true hearts crack that is how the voting practices typically go, but it's, I don't know, there are ways to just go around it and I want to just make sure we're emphasizing it. Yeah, grace I, I think that's an interesting point I know and a lot of the other in the other sections I tried to use the word should versus must. Right, because I don't think these are 100% finding, as you mentioned, right. And, you know, these are, these are considered things that we consider, and then they wouldn't necessarily stop us from accepting a project but they definitely are things that people should think about as they, as they think about bringing in a proposal these are the sorts of things that we've had, obviously, back and forth discussions on whenever we see project proposals so happy to make that a little less strong, if you will, and happy to take suggestions on how to do that heart. Yeah, I think Grace's suggestion sounds great. You know, we do want to emphasize that, you know, this is what has historically happened but you know it shouldn't stop people we don't want to. We don't want to raise a bar in people's minds about proposing projects. Do we think in the overall considerations kind of where we're introducing the considerations that where that's where we should be saying you know these are not, you know, hard and fast rules. These are things that are, are things that people consider as we look at proposals but wouldn't necessarily stop projects from, from being accepted. Yes, so I think that's exactly what I had in mind when, when, when we started this document. You know, it's not a, it's not a rubric, it's sort of a guide. And, you know, if you sort of the, the idea is that if you meet all of the considerations then you're probably very likely to be accepted. If you meet few of the considerations, you're not very likely to be accepted and if you're somewhere in the middle well, you know, it's, it's hard to say right. But I think that's a great idea to add the point that these are not all necessarily hard and fast things. They're just guidelines that should allow people to gauge whether their project is likely to be accepted or not. And I think you've worded that better than I did. But I agree with your intent. All right. So when it brings up the reload, see if I captured this correctly. Bobby. Hi, I would just like to see a comment maybe directing people to the learning materials working group if you're new to hyper ledger because every two weeks at our call we teach people about the community. Okay, I think that's a good best practice to add Bobby. All right. And Bobby is the learnings material. Sorry. I know I got it wrong. Is the, how do we say this the materials development working group is, is it doesn't meet every other week and that's what you go through or does it. Yes, yes, we start with introducing people to the community, the working groups, the projects, how to get information on all that stuff. And we go from there, depending on who's on the call. Okay. Sorry, I'm typing and not looking at whose hands are up at the moment. And the wiki does support, you know, multiple people editing. So if someone, you know, also want to edit. No hands up. Did anybody have a thought on the introduction to the considerations about the hard and fast rules. I loved what I wrote. Bobby, I don't know if this captures that third bullet point under best practices captures what you have intended. If not, feel free to wordsmith that as you will. That's perfect. Okay. All right, other thoughts on what we have here are we feeling pretty good about this at the moment. I think both of us in chat. Should we ask people to reach out on the TSC mailing list in the community architects mailing list. I think either is, or I think both is fine or the TSC list is fine. Are we. Okay, so I specifically removed TSC Arun from the first bullet point and said existing chat channels mailing list or direct communication with members of the community. This may be my my desire to have people reach out to more than just the TSC. If they're, if their project might overlap with an existing project or their project might, you know, add to something that already exists out there. I think the, the first place I'd love them to start right is with that project or with that lab and not directly with the TSC and so I think that's why I was a bit suspect there and removing that from the which mailing list or which chat channel. Yeah, thanks for bringing up that good point I guess I agree what you're trying to I mean what you're intending to say over there. However, I want some place where it is must in addition to these optional places. So by making it too wide or to scope, but this we never know where a particular discussion happens, which is fine to discuss outside but it's some like we should somebody should know where to look for some new proposals coming up instead of monitoring all these channels. Okay. Are we talking about so in the proposal process we do have the. I think we have a call out to send to the TSC mailing list so are we talking about when the proposal is being formalized or prior to it being formalized. When it is being formalized. So I think that's already in the process, and I tried not to duplicate what was already documented in the process in this document. At the top I do link to the proposal process so hopefully that would tell people that they need to do that if that's not in our proposal process and I agree Arun we probably do need to fix that. So it's the, I'll just read out what is written on the HIV process. Over there it's written as the seed of new project has to be weighted in a public forum like TSC mailing list. It does not mandate TSC mailing list it says like the TSC mailing list. So we probably should then get the proposal process updated to reflect that when you are ready to formalize the proposal to send it to the TSC mailing list because I agree right that's the right place to to formalize the request. And I know, well, I think that's another I've said that I'm thinking about the fact that before we used to do everything via Google Docs, which did proposals via Google Docs and with Firefly we just started using the GitHub repo which obviously all of us should be following the hyperledger hips repo to see pull requests that are coming in. So I think obviously the TSC mailing list is a good place to point out that you have a pull request, but I do think that the hips hyperledger hips repo is the formal place for these now. But we should definitely update the process because I don't think we've updated the process to reflect that. Okay, we will take that as an action item. Let's put together something for that. Let me take a note that I just volunteered to do something. All right, any other comments on this document then. All right, so if there's no further comments, I think we can then move on to our next agenda topic. All right, so our next agenda topic is kind of in preparation for the next TSC election. So I had captured here in the agenda all of our past decisions. I know Rai has a proposal that he would like to share with us. So I'm going to hand this the floor over to Rai for him to talk about that and then we can discuss. Thank you, Tracy. So this is a very rough sketch of what I would propose the hyperledger TSC transform into. And that is something closer to what LFPH does and LFN, which is the central governing board, you know, the TSC, the TAC is appointed in this way where each premier member gets a seat. I suppose that each top level graduated project gets a seat. The working groups in the six each get a seat. And that the how those seats are filled is delegated to the people or those groups so those can be through several elections or however those groups want to report who sits in that chair. And then the chair and co-chair would be elected as before. This would result in, you know, a movie a sea change, but it would be somewhat back to the way it was originally. The original TSC was appointed. And I'm not married to any of this. The goal is to have a deterministic process for getting a TSC seated with more representation from the projects. This also gives an incentive for projects to to reach graduated status. And I'm this, I put it to you. Please send arrows in my direction. Yeah, that's people. Any ideas about what this looks like any reactions, negative, positive. Arun. Hey, I want to first understand what this fell by ways ideas ago meant like what came to your mind why you wanted like that. And apart from that, I guess I have many other questions on this proposal, including, for example, what are we considering top level graduated projects as are they at the DLT level and are we not considering other projects because given that hyperledger is going to change the charter definition, there will be not just the DLT projects but other than that, we are going to consider other projects, but even at present we do have other projects as well. So, yes, I mean, I'll go one by one, sorry. I'll answer your first question. The charter says the charter for hyperledger says that the goal the mission is to create and enterprise grade open source DLT and code base. That never happened. That that was discarded. That idea of having one single DLT was discarded, not on quite day two, but maybe day four or five of hyperledger and the charter has not been updated to reflect that. And that's why I say it fell by the wayside years ago, because we, we are not trying to create a single DLT. Troy. Hold on before before before Troy. The second question was around the top level graduated projects, getting one seat, and that would be any graduated project not just DLT graduated projects but any, any graduated project that we have so right now we have six of them. And that would obviously add additional seats as we graduate new projects. So, hopefully that ruined that answers both of your questions. Troy, you're next. I guess a few thoughts. One concern would be, I think this could end up turning in one vote that we have now into many votes. The top level graduated projects might be a simpler problem but having all working groups and all SIGs all get together and decide seems like you're going to end up with a vote again. And, you know, I'm not sure on the motivation for the premier member point. That seems more like at the governing level than at a technical steering committee. So I'm not sure what that point is is actually in this list right now. Thank you. So I'll answer your your last question first. I was using LFPH and other projects as a model. So that's why it's there. And if you look at our list of premier members. Most of them already are represented on the current TSC. And then your, your first question was. Sorry. It was around the process of how do you get SIGs working groups and to kind of figure out who that one person is that they're going to submit for the SIG or the working group chair. Right. Right. So I wanted to make sure that SIGs and working groups continue to have. I wanted to have SIGs and working groups have a more formal voice. So this was my idea. You're saying it will turn into another election. I'm sure that it will. But that would be an election that would be for the SIGs and working groups. You know, we can point them at oppa vote and say vote how you will, we can enable voting in the groups.io. I just, that's not something that I want to run. Sorry, I wasn't sure if I should raise my hand again. Well, I guess the point here is because you're going across groups, you're just going to end up with another vote. So I'm not sure, you know, I had very mixed feelings about this because on the technical side I do think it's nice to elect and I'm kind of leery of appointments of, you know, premier members sending whoever they feel like. And I think when you go across groups like working groups and SIGs get one total instead of one each. I think you just got back into a vote again. I'm not sure that this will accomplish the goals, what I'm saying. And I'm leery about appointments. All right. But while I saw a hand, I don't know if that was a raise hand. The TSC means the technical steering committee, it should focus on the technology side. So I'm really concerned with giving, sadly giving each primary member a seat. Because they already got a seat in the governing board, right. I would prefer if we give more seats to those projects or those members who contribute to the most code. That should be, that would be better from a TSC side. Thanks. Anybody else have any thoughts. Thanks. Yeah, as far as the technical stuff to, as far as that goes. The TSC really do a lot of technical governance. I mean, I know that was the intent in the beginning, but it seems more to me like it's the TSC has sort of become a some kind of community management entity. Like when was the last time the TSC made a technical decision. So Hart, the comment there is about. What is, is in kind of direct conflict with what Val Wow said, is that what I'm getting? I want to make sure I understand. No, I mean, I, I wouldn't say it's in direct conflict. Wrong words. No, no, no, it's, yeah, it's totally fine. It's just that, you know, we don't appear to be making a lot of technical decisions. That's all. Okay. So the premier members. Yeah, that there's, I think just because a premier member doesn't mean they're going to point people who don't know anything about technical stuff to the committee. I think they're aware of what the committee's purposes are and who they would appoint. So I don't feel the same apprehension about letting tech premier members appoint one of the seats. And the odds are they're going to pick a very technical person in a project they care about who's involved in the project. And maybe we could write that in to the encouragement if this goes through. But I think that was the first thing in second, I'm concerned if we use metrics like whoever's committed the most lines. If we make it an external metric like that that is easily gameable. I'll just commit all the Ethereum reference tests and boom, my line commits are just amazing. So my concern of making it a technical measured number. We just move electioneering to a different area. Maybe. I have a lot of concerns with the way this is structured. For the identity projects we have a lot of contributors but I don't think we have very many premier members though a lot of them remember premier members have become involved over time. And, you know, I don't know what this will do the representation of the technical community itself. The elections, although they're imperfect at least they reflect kind of the general commit level of the code. And, you know, I know we've had a lot of debate and discussion around how you could gain the voting system. But for the most part, my understanding was most of that was academic, meaning we actually haven't seen electioneering or, you know, a lot of shenanigans occur at the at the ground ground level. So, you know, I have a lot of concerns about whether this will affect the ability of the technical steering committee to remain neutral, and to also kind of act as a counterbalance balance to the governing board which already does reflect a lot of the things are a lot of the qualities that are listed in this new proposal. I know you still have your hand up or. Okay, thank you. Other comments, anybody has. All right, I'm going to add my two cents here. So, I, I'm torn. I think there's some good stuff in here I think there's some things that will have to figure out if we go forward with this. One of the things I think we have to figure out is. I don't think for number three we can leave it up to them. I think there has to be at least guidelines or something that we put together for how that vote would work and what that might look like. I think that the other thing that is potentially a concern is I know currently the get contributors that we do not only can includes the projects but also includes the labs as people who are eligible voters as well as eligible people to to bring on to the TSC. I don't see that piece of the community included in here. Things that I do like I like the fact that we are attempting to get more of the projects involved. I don't. Well, I know we do not have people on the TSC today who are part of all of our graduated projects, let alone our incubated projects. I like the fact that we're trying to get more people from all of the projects across the board included in this. I also like the fact that we're trying to get the SIGs involved. My initial thought and that was well SIGs aren't part of the TSC but I do think that going back to our breaking down the silos conversation that with the conversation with the SIG chairs is something that is quite interesting and intriguing. So I like trying to bring more of the community into these conversations and having representation. But yeah, a couple of a couple of concerns, a couple of things that I like about it. Thanks. And as I said, I'm not married to any of this. I mean, I could see you're right. Not having labs on here was an oversight.  Thank you. Daniela. Yeah, I just want to make a comment because our no is not here and he's usually the representative for the TSC on the board that this was not prompted by any board discussion in regards to the seats by the premier members. So I just want to make that very clear to everybody. Thanks Daniel. You're welcome. Heart. Yeah. So I guess one comment is I know that the TSC election is one of the like worst possible things to run. And it causes a lot of a lot of, shall we say pain and suffering for the LF staff. What I'm worried about and what I think, I believe it was Troy who brought this up is that, you know, right now you're stuck wanting running one election. How do you know you're not going to get stuck also running the like working group, SIGs and labs elections as well right, or even the, the top level graduated projects right how are they going to vote on who gets that seat right, you know, presumably it's going to have to be some voting process. And, and, you know, what if each, each project decides to do something else, you know, it could get really messy. So I'm a little worried that this might actually end up that the way the proposal is stated right now might end up being more work for the LF staff, rather than less. Thanks heart. Dana. One of the questions brought up earlier is what does the TSC do what is the technical and I think one, what observation I think that needs to be made is that the governing board is not a public meeting. While they may release notes we don't know who's with the back and forth is like what the tone of the meeting is versus the TSC where I mean recording this and we're putting us on Wikipedia for the basically the world to see. There might be one subconscious thing that we're driving towards is maybe there's a desire for more open governance, and maybe having the TSC be open like this could fill that people's need for the field that there's open governance on this. So that's, that's one thought that crossed my mind that I thought was worth surfacing. No reelection item on it yet. Thanks channel, Nathan. I'm confident that this would cause a lot more work for the Linux foundation staff, even if we as maintainers ran the elections the controversy that it's created every time we've had the election would spill over to the Linux foundation staff to manage and it would probably have already boiled over by the time it gets to the staff, which would make it a lot more tense situation it would make. It would make it a lot more difficult to manage. And to Dano's point if we're really trying to push towards more open governance, we should open the governance board. Rather than try to change the nature of the TSC to where we lose the TSC, and I would also push on all of us if we're really not happy with the administrative nature of our agenda, we need to put better things on the agenda. And it says we can't do more technical architecture work that we can't work harder towards unifying the platform to be an open source distributed ledger framework like the paragraph above says, I think the truth of it is we're probably doing the best things that we can do and these governance points that is rather tedious is one of the best ways we help support the maintainers and take a substantial load off their plates so that they can make good code progress. So, you know, in terms of how we're serving the community. I think a lot of the things that we're dissatisfied with are the things that we have to do in order to take the burden off of others. So, you know, overall I'm not very satisfied with this proposal, though I do like the idea of pushing the SIGs and pushing even the labs to be more represented on the TSC. And I do like the idea of having more seats available on the TSC in general because I think it will help the number of voices that we get to hear and I think it will help balance some of the election problems we've had where some projects have a lot more votes than others. But there's, because there's simply a lot bigger projects. All right, thanks Nathan. Part. Hey, yes, thanks. So, first of all, to address both Dano's and Nathan's comments, I really agree with Dano that the, the TSC is sort of the open governance arm of the governing board. My opinion is that if the governing board is the CEO, the TSC has sort of turned into the COO rather than the CTO. And, you know, I would be all for encouraging more open governance from the governing board. And I think a lot of the decisions that potentially probably should have been made by the governing board. A lot of these decisions have been made by the TSC.  Like, for instance, when Bessie joined while I certainly supported that, I thought that probably should have been like a governing board decision rather than a TSC decision. You know, things like that, like changing the charter, why is the TSC editing the charter not the charter not the governing board. Well, I would talk a lot more about this if this call or not being recorded but that's another story. With respect to Nathan, yes, for the entire history of the TSC we have tried to push back into doing more technical things and we seem to have continually failed. So if you have any advice or suggestions on how we can be better about that and find good meaningful technical work to do. That would be amazing. All right, thanks Hart. And thanks for my morning laugh. Other other comments that people have we've got some folks on the call who haven't had a chance to speak up yet. Feel free to now is your chance. I can't believe that Gary is silent on an election item. You have the floor. I'm going to say enough, just because you addressed me I was just going to say, I mean, I think enough's been said, I don't have any, anything, any genius comments today. This is David I'll agree with some of the other earlier comments that it seems like this would turn from one problem into many smaller problems, they may not be, you know, they might we might seem they might seem like like they're smaller but they might actually be more work for the staff team. Right, thanks Dave. Other comments people have. Alright so I'm right I guess I'm going to leave the feedback with you. You know, I am not quite sure where we want to take this it sounds like there's pieces of this that people like in pieces that a good number of pieces that people don't like. I think we're probably not in a position to say everybody's 100% happy with this. Sure thing. My expectation was not to come out of this meeting with a decision or everybody 100% happy with things. I want to have a discussion about how we could make the TSC. It's a page title right. It's a proposal for reforming the hyper literature TSC. So that's, that's why I want to get out of it. This discussion. So, thank you all. Alright, thanks for that. Yeah, if there's other comments, you know, feel free to to have those in the TC chat or with right directly if there's something that you think would be worthwhile to to move forward. Grace. One question. You're probably trying to wrap. So, thanks. I agree. Just want to say that with the spirit of inclusion and adding more projects, also some drawbacks just echoing everyone else's comments. But does it make sense for the TSC to be voting on how the TSC is going to change the government board decision like heart was kind of suggesting what what's is this more a board decision. Do we know it's probably a good question. That is the sign that we went through with the expansion. And the way that happened last time was the TSC asked the governing board to make a change and the governing board made that change. So we can make suggestions to the governing board and they can decide what the right steps are. It definitely is part of the charter so it would have to be something that the governing board would approve. Or or disprove disapprove. What's the right word. Decline maybe heart your next. Yeah, grace. I totally agree with you. This is something that ideally the governing board would be doing. But this kind of thing has been sort of continue over the history of hyper ledger this thing has sort of been continually pushed down to the TSC. So you get the TSC doing a lot of things that the governing board, you would think would do, but doesn't end up doing. Thanks heart. Any other comments, we still have a couple minutes before we at the top of the hour so if there's anything else hurt I'm going to assume your hand is just still raised. I didn't want to say anything else I'm sorry I forgot. That's fine. All right, so if there's no further comments. I think we can close the meeting. Thank you all for attending and thank you for the lively discussion and the input. Thanks Tracy.


Hello, everyone, and welcome to today's PyTest talk. I'm very happy to have with me Adam and Satish, and they're going to talk about how to convert PyTest into NF tests.  Yes, that's it. Still a few more minutes. Hello, right. So, yeah, we thought it would be useful to give a little demonstration on how to convert an existing PyTest into an NF test. You know, so as you update your modules, you can migrate them to a new testing frame that we started to shift over to. Maybe some extra hints and tips that you might like. And just brief over here how to do it. So, yeah, PyTest workflow has been really good, to be honest. It's been able to test this massive model repo that again, of course, modules and some of the pipelines as well. But it's indirect. It's using PyTest to call a sub process. And then we have to write that sort of next low pipeline. And then we have to test the files that are coming out of this. So we're always a little bit out of the loop, you know, always waiting to see what files are created and testing those files. NF test offers quite a lot of better advantages because of its integration. So, for example, for PyTest workflow, we have to write this custom main NF workflow. And that uses, that basically wraps up the tool you want to test. And then it runs that workflow as the sort of default workflow. Whereas in NF test, we can have these sort of when and then statements so we can really find control what exactly what you're doing within each test. And like I said, in PyTest, you can only really test files that are created and just look at the hash that's created. Whereas NF test supports this snapshot, which is really useful sort of automatic data structures that just, you know, check the state of the file and then check that it hasn't changed. We can also because it's built in or extension of next low itself, we can actually do loads of different assertions. So we've only really stretched the surface here where we does all file contains and names like that. But there is additional plugins for things like faster files and we can start to really dig into the granular nature of it. You know, we can test these variants appear in the pipeline and stuff like that. And because it's extending next load and groovy, we could start to write our own code to extend this with plugins and extensions.  So firstly, let's have a quick overview of a PyTest versus of NF test. And so, yeah, at the top, we've got this tests main NF and that is a workflow. Like I said, it kind of wraps up the module or the process or the workflow that you are testing with PyTest. We've got a configuration file. We have to add this just to export the files. Most of the time, that's all it does. But sometimes it adds some extra settings, you know, like an argument to a command or something or additional setting that you might be interested in. And then we have finally, we've got the test.yaml file itself. So this is the actual file that PyTest looks for. And it contains a shell script that it's going to run. It contains some tags that are continuous integration looks for that says what the test is testing. And then it also contains the like assertion statements. We've basically come down to is this file created? Does the file have an MD5 or does it contain these strings? When we go to NF test, the first thing we've done is we actually moved where the tests are. So they used to be created in a slash test slash modules NF call fast P. This is a fast P example. And now we've moving them into modules NF call slash fast P flash tests. And that just makes everything self contained and adds a little bit of ease of use when you're writing them. Because you're not having to do is massive relative imports where you go around the entire tree. You'll see that when I demonstrate it. The actual test file is a main NF test and NF test is some really nice tool into generating a lot of this for you. So it does most of this for you. And this will take the role of that test YAML file from PyTest workflow. The snap file I've just mentioned is the equivalent of the MD5 sums and the contents and checking that the output, the output of the the process or workflow is the same. This can be quite complicated. You can actually represent, you know, things like the coming out of the process without actually writing a file or, you know, or values of the meta map and stuff like that, which isn't available in PyTest workflow. We've got a configuration file. This is actually optional. This could this generally you just copy it straight from the PyTest workflow if you're if you need it. But it's actually not that needed because you don't need that file writing configuration. Finally, we've got a tags YAML. This is a lot simpler. This is just a lot simpler than the equivalent in PyTest workflow. It's just a single set of tags to match the folder. And basically what happens is our continuous integration picks up that tags YAML and checks if any files have changed in that module. And that means that for NFCore modules, when we update it, we don't have to test 500, 600 modules at a time. We could just test the one or two that changed. All right, time to do a little live demo. Okay, so here I have a callista and this callista index. So it's a really simple test. We have one workflow here. This is the tests met modules NFCore callista index main NF file. We have a single workflow test callista index and all that does takes one input file and puts it into the callista index process. Here's that relative input I was talking about. We have to go back six layers and then back all the way through by putting it in a test folder. You'll see it's a lot simpler. Okay, so if you want to generate a new test, we can actually generate it using the simple command NF test generate process because we're testing process, but that could be a workflow. NF test generate process. And then we point up the main NF file that represents callista index. All right. So just looking there, it's created me a file here and I can open it. I'm just going to move it to the side here so we can do the side by side. We have here we've got the main NF test. We can see it's kind of got names, script, process. It's all fairly self-explanatory and the documentation can help lead through any of these bits if you maybe not quite following what they do. But the important bit is here we've got this when statement that says what the test is going to be and then then which is the assertions at the end. Now to lift over our high test workflow, what we need to do is essentially take its input and put it into the callista index process. NF test will do the process for us. So we just need to specify the input. So we can literally copy that file and we can drop it here. I'm just going to tidy it up a bit and get rid of this because it's just an documentation. And that's basically the start of our workflow. Now there's a couple of more administrative things we need to do. So firstly, we've started to do a relative import. So what we do is just go back one folder and import the main NF file. This just means that it's all just nice and self-contained. The other thing is we start to put it in a test folder.  So I'll just have to do it now. So I'll just create tests, main NF test, drop it in now, move. So this now means that it's in the test subfolder of callista index. Tags. So we should add the tags. I've just copied them from another screen to make it easy. But basically we have four tags for every tool, or three or four tags for every tool. And they just tell the continuous integration what this test actually is. So it says this is a module. It's an NF core module. This is what part of the callista suite. This is specifically callista index. And sometimes that should be slash. So we have now four tags and they'll be picked up by the continuous integration. We've got one more thing we do. So by default, NF test adds this assert statements one by one. It will go through each of those and test them. So the test is top one first and then test the second one. We've started to move over to using this other thing they provide called assert all, which basically does all of them at once. And if any of them fail, it will raise an error. This just helps capture any errors that you've got. There are use cases for using a certain zone, but in general, this is what we've had to do. So I think it's cropped off the edge there. Assert or set process to assess a certain snapshot matches. And this is the snapshot is what we're going to be creating to match our file. Okay, so now we've got, we actually don't need to prime to get rid of that one because there's no parameters to set. All right, so now we can test it. So I'm just doing NF test, test. I'm going to get the modules, but you could use a tag to do that just as well. I'm using Docker profile because I've got it ready on my machine. And then we've got a clean snapshot, but there's a couple of snapshot options for refreshing the snapshot or creating a new one. Here we can see the snapshot has been created. This is adjacent. It's kind of not human readable exactly, but you can certainly check it. One thing I've been starting to do is actually you can go and you can get the hash. So this is the old PyTest workflow file. And we see the MB5 of the output it's created. And you can see it's two we can actually find that in here. And we can see that the hash matches. So it says that the test hasn't changed anything. We can pretty much do a direct translation from PyTest workflow to test. And that's pretty much it. So now we just need to add our tools. And we've added the started building. Oh, I forgot the tags. So we need to add one more thing, which is the tags file. So tags.yaml. And this is really before we build NF Core Tooling, which we hope to do pretty soon. So we'll do a lot of this for you. But yeah, we're going to add a tag. And a tag is basically the name of the module, which matches this tag here. And then it is the path to the module with two globs to say any file within that folder is changed. So we can now that and we're ready to go. And if you can always just run that test again, make sure it works. And then okay, green tick. So I'm going to switch over to Satish now. He's going to show you how to do a daisy change module. All right. So I'm going to share my screen. Okay. You are able to see my desktop, right? My VS code. Great. Okay. So what we have, what Adam has shown us is an example of what we can call as a simple module that is a process on its own. It doesn't require any other processes for its execution. Callisto has another tool within its suit. It's called Callisto Quant. And this is the pie test workflow of Callisto Quant. You can see it actually needs two processes, the Callisto index for which we just created the test. And then the Callisto Quant process itself. So if you look at this test file, it has basically actually two tests, one for single end and one for paired end. And if you see any one of the workflow itself within the workflow first, it's running the Callisto Quant process. And then the Callisto, sorry, first running the Callisto index process, and then running the Callisto Quant process, which basically takes one of the inputs from the Callisto index process. So this is in pie test. So how do we do this in NF test? NF test provides a setup method. So I'll just quickly get started with porting it over. So first I'm going to generate the NF test file for Callisto Quant using the generate process command from NF test. So this kind of like gives you a boiler parade test module. I have already created my test folder within the Callisto Quant directory. So I'm just going to first move my test file into the test folder. And once you have that there, so we'll start with the changes right away. First is replace the absolute paths with relative paths. We're going to add the modules, the tags. So as Adam was telling about how we're going to add different tags. So the tag modules and tag modules NF core, we are adding them by default to all NF core modules, then come in the name of the suit of tools. And then we can add a specific tag for this particular process itself, which is Callisto Quant. All right. So I'm done with my adding of tags, removing the params block. We don't need, because we don't need to set any params here. And just cleaning this up. First, let's create a test for a single end. So I'm going to change, you can add a name to a test. So you can provide this as a name for the test. So this process for this test, we will just call it single end. And now going back to the pie test, you'll see that the Callisto Quant process actually takes four inputs. So it takes input of the data itself, and then it takes index and two more additional inputs. So what we'll, and the way inputs are provided in NF test is through positional inputs. So what we'll do is we'll create additional input files and give them their positional names. Right. But this is for the Callisto Quant process. So the when block and then block. But remember, we first need to run Callisto index process before the Callisto Quant process. So the way that is done in NF test is by using the setup method where you can specify to run a particular process before the primary when block itself. So the syntax for this is within the setup block, you can specify which, what is the name of the process that needs to be done. So in this case, Callisto index, and you open this run block itself and now provide the path to the Callisto index process. So which will be one folder up. So index main dot NF. And within and beneath that, we provide the process block to run this particular Callisto index process. So as if you remember from Adam's demo, Callisto index just takes one input. So it's pretty straightforward. And if you see it's taking it in this particular PyTest workflow, you'll see this is the faster file of SARS-CoV-2 genome that is being indexed. So you will need to provide that in the input. So we'll quickly, I'm just going to get this from my previous one. Right. All right. So what we have here is basically the setup method that is running the Callisto index process. This is the path to the index process. And within the process block, you're providing the input that is needed to run the Callisto index process. Once we have that, now going back to the PyTest workflow. So looking the first input for Callisto quant is the input block here. So I'm just going to copy this and provide that as the first positional input. The second input is Callisto index out. And it's basically taking the index of the Callisto index process. So to provide that here. But if you quickly check, there's a mistake there. It's actually called index not IDX. Now the index process, the index output channel, if you look at the output, it's actually a tuple containing the meta and the index. But for providing the input to the quant process, we actually just need to provide only the index and not the meta. So we need to map this process again. So we'll provide index meta index and just map out, take out the meta and just have within only the index within the process. And these will be provided blank, as you can see, as compared to the PyTest workflow. So, okay, so we have all our inputs now. And as Adam was suggesting, we are wrapping all our assertions within the asset all block. So that would be asset. There we go. Right. So we're getting there almost. So first, we have our setup block here that runs the Callisto index process. And then in the primary when block, we have the inputs that are needed for the Callisto quant process.  But if you look at the Callisto quant process, it also has an extra config. So the Callisto index process did not need any additional config. But the Callisto quant process needs these external parameters to be set. So what we will do is we'll just copy this next one config and put it in the test folder right next to the main.nf.test. So I'll just paste it into this folder. And once I have it within the test folder, you can specify the config that needs to be applied at the top of the test file. So in this case, I'm just providing it nextflow.config. Right. So it's got its nextflow.config as well. So we are pretty much there. So if we run this process now, you can always refer to a test or you can run a test by its tag. So in this case, you can use the dash dash tag, Callisto quant, and run it using profile Docker. And you can see it prints out the test name, single end, and notice this hash that is right next to the test name. It's quite similar to nextflow where each test is run within its own directory. And this particular tests are run within a .nf test folder that is automatically created in the parent directory. And you can check this directory for the work and the output of a particular test. So in this case, you can see we created one snapshot called single end. So now let's quickly go over and verify the snapshot. So that is just basically run the test again. And this is going to fail because we have already tested this. And this is probably going to fail because the output has some inconsistent files that generate inconsistent snapshots. So what do we mean by that? To understand that, let's have a quick look at the Callisto quant process itself. So if you look at the Callisto quant process, you'll see that it is emitting five different channels. So here, as I said, my test has failed because the snapshots don't match. And if you see it is these particular nf test now gives you a very visual way of diffing the snapshot. So you can see exactly which this is the previous snapshot to the left and to the right is the one that's being generated right now. And you can see there are some files that don't have matching MD5s. And this is a very probable case that you'll probably see if a file contains either timestamps or paths running for that is particularly about that particular run. So it'll be generating a different snapshot. So in those cases, the snapshots won't match. So just looking at the process itself, you'll see that the quant process is emitting five different channels, abundance, HD5, run info, log and versions. So in these, the abundance HD5 and the run info particularly the files that don't generate a consistent snapshot because they change upon every run. So in these cases, it's a decision to be made of whether to what to include within the snapshot. In this case, because the abundance file is primarily the truth file, the main output of the quant process, we can check only for this process and include only that in the snapshot. So in most cases, that should work. In cases when you cannot generate a consistent snapshot of your main truth output file, there are other ways of checking for contents of the file, particular contents within a file and not the entire file so that you can generate a consistent snapshot. So what we're going to do now is change our process. We'll specify exactly instead of capturing all the output channels with process.out. We'll actually now specify the individual output channels to capture. So what we'll do is we'll just take, we'll create some more assertion lines and we'll specify only process.out.abundance and within this match.match is the method that is actually comparing a previous snapshot to the current snapshot. And just like we can have named keywords for emit, you can also provide named identifiers for items within your snapshot. So you can name that abundance because we know that the abundance HD5 and running for do not generate consistent snapshots will not include those in the snapshot. Next, we'll include the log, give that a name and then the versions, process.out.versions and match that to the versions. Right. So now because the snapshot has already been generated, we'll use the update snapshot command to re-record any new snapshots that are, that have changed within the test file. So you can see that it gives you a warning that every snapshot that fails during this test is re-recorded. But I also want to point out another thing is when you update the snapshot, the previous entries within the snapshot are not deleted. So this is what you can see is shown in the summary here where it says it has created three new entries within the snapshot, abundance, log and versions for what we have specified as the named identifiers. But there's one obsolete as well because it's no longer present within the snapshot now. So NFTs provides another command to clean any obsolete snapshots. So the command here is called clean snapshot. And what this does is it'll remove any obsolete entries within the snapshot that are no longer presented than the test file. So if you, let's take a look at the snapshot file itself. So here you can see entries from the previous and the current updated version of the test that we have. But now that we have run it with clean snapshot, it's now only has the entries for abundance, log and versions. So that's what you see within the snapshot. Great. So now we have created a test for single end. So, but if you notice that the Callista Quant process also had the actual pie test module also has an additional test for paid-in data. So if we are to create another test, what you can do is you can have multiple tests within the same test file. So you can copy and paste this within the same test file and provide different inputs. But looking at the tests for single end and paid-in, both require the same setup of Callisto index. So when a setup, what we have done here is we have provided setup within that particular test scope. But if you wanted to provide a common setup, a global setup method for multiple tests within your file, you can move the setup block from within any particular test scope and put it above beyond the any of the test scopes. So that makes it global and it will make available the output of that particular process to any particular test, any number of tests that you might have within your file. So now that I have moved the setup block outside of the single end test scope, I'm going to copy the the test block for single end, change, call this paid-in and provide the input. Oh, actually, I think I've given paid-in data for the single end. So I'll actually change that here, my bad. So if it's single end, the input needs to be just one fast queue. There we go. So now I have the global setup method that will provide the Callisto index for all the different tests that are within the particular test suite. We have a test for single end providing the input here. And then we have a test for paid-in providing the actual inputs. But notice it might be just one test file, but the snapshot that is created, and there's only one snapshot for each test file. So if we have the same identifiers for your entries within the snapshot, they might collide. So what you need to do is provide some more identification or delineation between exactly the test entries within the snapshot. So for this is single end. So I'm going to change, add this identifier of single end to these identifiers for the snapshot and for the paid-in, we'll add paid-in identifiers. And there we go. All right. So that should be ready. But since I've changed the inputs, I'm going to update the snapshot. The single end has passed and it's running paid-in now. But because we have changed the names of the identifiers for within the snapshot, we now have some obsolete entries within the snapshot. So you can run the clean snapshot command to remove those obsolete entries within the snapshot and make sure you have only the entries that are in the current test file. So just as a recap, you can have a setup method that's global and have multiple tests within any single test file. If you're having multiple tests, make sure the snapshot identifiers are unique enough so that they don't clash within the snapshot file. And since we have, and now I have run these both for collector quant, one advantage of providing these multiple tags, now I can run all the tests for Calisto including index by just providing the tag Calisto. So when I do that, it's going to find all the test files that contain the tag Calisto. And in this case, first you'll see it'll find the Calisto index process. Hopefully there we go. And it's running first the Calisto index process. And then it'll find the entries for Calisto quant process and verify, run the test and verify if the snapshots match. So in this case, you can see it's first running single and again. And voila. So we have now with one command, all the tests for Calisto including index and quant as a final step, add the tags that Adam just mentioned about. These offer the GitHub continuous integration to pick these tests up. And once you have, you can just comment your changes. So it will basically be the test folder containing everything that you have added. So finally, within my test folder, I have the main.nf.test. I have the snapshot for the test. I have a next-float config because it's required only for this quant process and then I have the tags talking about it. So that's pretty much it. So that was an example of running simple modules, creating, converting PyTest modules for a simple module and also a chain module using the setup method. So we have started this migration during the last hackathon. So we have some more examples with nf.test such as Aplicate that has examples of both simple and unchained, FastQC and Highset2. These are just examples. There are a lot more modules have been updated with nf.test. So if you go into any modules repo and if there's a test folder, that means it has nf.test within it. If it doesn't, please consider creating a test for those. So with this, we actually would like to invite everyone who has previously contributed a DSL2 module to nf.core modules. Currently check the module repo that you have submitted. If it already has nf.test, verify if the assertions there are actually verifying the truth of the module itself. If not, consider creating one. And for this, we have added recently a step by step of creating all the steps that Adam and I have just shown you for simple, unchained and chained modules. This is currently on the nf.core.downs, two modules page. So you can find step by step instructions there. So with that, thank you so much. Reaches out on nf.core, nf.test channel. We'll be back with another bite size with nf.test for workflows and pipeline level. Thank you so much. Thank you very much. This was very informative and I'm sure it will have a lot of people Do we have any questions from the audience? I've dropped the pull request in the chat. That is actually the code we were doing. So you should, all those changes today, you can go and look at them if you want. Awesome. It seems that you answered all the questions. Actually, my question, you answered at the end. Sorry. I've dropped the documentation link there as well so everyone can answer it. Perfect. Then I would like to thank you both again. I would like to thank the audience for listening in and the John Zuckerberg initiative for funding our bite size talks. And I hope to see you all next week. Bye bye. Bye.


heavy metal toxicity and unintended consequence of a gluten-free diet. Hi, this is Dr. Ruscio and let's discuss this issue because a recent study did in fact show that those on a gluten-free diet had significantly higher levels of lead, mercury and arsenic in their either blood or urine. So is this something that could be detrimental to your health if you're eating gluten free or partially gluten-free? Well, the good news is in my opinion the answer to that question is no. Now while those who diet gluten-free were shown to have higher levels that were significantly higher than controls who did not eat gluten-free, the the change from one group to the other, both of those were still well within the normal and acceptable ranges for levels of mercury lead or arsenic and cadmium in in this study in the sample ranges. So according to what the normal range for the test values are, let's say this is the range, let's say here's someone who eats gluten, here's someone that eats gluten-free. It was a significant difference between one group to the other mathematically, statistically, but in terms of does this have any real world impact or relevance? Is it something that should make you rethink your diet? In my opinion, absolutely not. It's certainly an interesting finding but I want to take a moment to address this study because this information spun the wrong way, may make you rethink a gluten-free diet, may make you think that you have to do heavy metal testing and treatment just because you're on a gluten-free diet, which of course would be quite a bit of expense and therapy to go through if it's not even really truly needed. So yes, again this study did show significantly higher levels of mercury lead, arsenic, and either the blood or urine for those who were gluten-free compared to those who ate gluten. The difference wasn't really clinically meaningful, so it's just something really important to be aware of. Now where does this come from? It may be that some of the grains that are used in place of wheat, rye, and barley that are gluten-free, like rice for example, may have higher levels of some of these contaminants. Might you mitigate that risk if you eat organic? Possibly. There's no documentation that really answers that question but certainly I think there's some plausibility to that. But again, you know, just because something is statistically significant doesn't mean it's clinically meaningful or meaningful in the real world. So if you hear about this issue, just take a moment, take a deep breath and realize that this in my opinion is nothing that is cause for alarm. And if you're someone that does better on a gluten-free diet, you can continue to eat a gluten-free diet without any worry about any potential overload of heavy metal toxicity. That's it. This is Dr. Ruscio. I hope this information helps you get healthy and get back to your life. Thanks.


Today's March 3rd, my name is Timmy Ngo and you're watching this week's CalTV Entertainment Weekly Wrap-Up. This week in Entertainment News, Moonlight won best picture at the Oscars, making it the first ever LGBTQ film to do such.  Natalie Portman and her current husband welcomed their second child together, a baby girl named Amalia. In theaters this week is Logan, which is Hugh Jackman's final film as the Wolverine, The Shack, and Before I Fall. In new music, Lord released her single, Greenlight, which is so good I almost cried the first time I listened to it. Coldplay released Hypnotize, and Mike Bull made it release a song with Migos, YG, and 21 Savage called Gucci On My. In albums, Ed Sheeran, our favorite mathematician, released his album Divide. That was your entertainment in a minute.