Well, good morning afternoon or evening everybody. Thank you for joining us for today's webinar. Today's webinar is on the ARDC's thematic research data commons on careful fairness. I'm introducing the session today. My name is Catherine Barker. I'm the Community Development Manager for Research Data Australia. Now this is a position funded by the ARDC to support and grow the RDA community in Australia. So it's my pleasure to introduce this session for the ARDC today. It is part of the Research Data Alliance's 10th anniversary event series, which you can see on my background here, which is comprised of a series of themed monthly events. August is disciplinary data. So we have Jenny Fuser to kick things off in a moment to tell us about the thematic research data commons on Hass indigenous data. I will need to leave this webinar early, unfortunately. I believe my colleague will put some links in the chat about RDA, how to join, how to find groups and find out more about the activities and events. And if you have any questions about the Research Data Alliance, please do get in touch on my email address, which will be posted. So at this point, I'd like to hand over to my colleague Jenny to kick things off for the webinar today. Thanks, Jenny. Thanks very much, Catherine. Hi, everyone. I'm just going to share my screen to get us started. So hopefully you can all see that now. I have no idea, but hopefully that's working for you. So good afternoon, everyone. I am Jenny Fuser. I'm the director of the Hassan Indigenous Research Data Commons at the ARDC. Before we begin, before we begin, I'd like to start the session by acknowledging and celebrating the first Australians on whose traditional lands we meet around Australia and indeed indigenous people around the world. I pay my respects to their elders past, present and emerging. I'm on Ghana country in South Australia, and I acknowledge that their sovereignty was never ceded. I'd also like to acknowledge the vibrant history and deep legacy of Aboriginal and Torres Strait Islander people's research and teaching, one of the oldest knowledge systems in the world. Okay, so a little bit about the Australian Research Data Commons. The ARDC is funded by the Commonwealth Government's National Collaborative Research Infrastructure Strategy, or NCRIS. We partner with the research community and industry to build leading-edge digital research infrastructure to provide Australian researchers with competitive advantage through data. Our mission is to accelerate research and innovation by driving excellence in the creation, analysis and retention of high-quality data assets. So how have we done this? So for the past several years, we've operated a range of programs and services over four portfolios. In particular, our platforms and software and data and services portfolios focused on co-investment programs supported by the underpinning storage and compute services and engagement activities in the people and policy portfolio. Under this strategy, we ran a series of open calls for projects. These open calls were oversubscribed and had a success rate of approximately 30%. So we realised we were simply unable to meet the demand for bespoke digital research infrastructure solutions. But what this process did give us was unique insight into the digital research infrastructure needs of the Australian research community. And in turn, this has influenced our priorities for the future and underpins our new strategic direction. Oh, I missed one. There we go. So the ARDC is now focusing on delivering digital research infrastructure through a suite of thematic research data commons. The thematic ARDC model is focusing efforts in domain areas that align with national priorities, researcher demand and concentration and potential for research translation. So this new strategy to deliver fewer, larger, coordinated activities will enable us to meet the needs of the greatest number of researchers through a small number of strategic priority areas. And a helpful analogy used by our CEO is that we're moving from the bespoke couture of the catwalk to the more accessible fashion of the high street. And in keeping with that theme, we can consider the thematic research data commons as a fabric of national research infrastructure capabilities selected strategically rather than competitively and co-designed with the research community. So the fabric has both nationally focused platform capabilities that strengthen and support the broader system. They're the horizontals here in this diagram and a deep focus on identified national challenges and opportunities shown here as the verticals to provide balance for a national system of digital research infrastructure. As a hub of expertise, the ARDC is positioned to drive best practice in the creation, analysis and retention of high quality data assets and to share these expertise across multiple research domains. But what is a research data commons? So research data commons brings together people, people are critical skills, data and related resources such as storage, compute, software and models to enable researchers to conduct world-class data intensive research. Current suite of research data commons are people for health and medical research planet for earth and environmental sciences research. And our focus today is the humanities, arts, social sciences and indigenous research data commons. Importantly, these commons are not standalone entities. Rather, we expect the learnings developed in one commons to be shared and applied across others. This will not only reduce duplication and effort, but offer unique solutions to varied challenges. So the Haas and Indigenous research data commons began in 2020 as a program of four projects shown here on the slide. These projects serve as the basis for which we are building a Haas and Indigenous research data commons over the next five years. And today you'll also be hearing from Robert McClellan from the language data commons of Australia projects, which is one of our foundation projects. Our approach to the Haas and Indigenous RDC differs slightly to our approach to people and planet in that it was born from a series of scoping studies undertaken by the Department of Education, who then identified four investment-ready activities which were subsequently funded through the October 2020 research infrastructure investment plan. So while we've held plenty of consultations, workshops and roundtables to gauge researcher requirements, we were already committed to these pre-designated projects. However, the consultations that we have held have been really valuable exercises and have enabled us to identify capability gaps for Haas and Indigenous research, which in turn has guided our planning from June 2023 onwards. So everything that we do is underpinned by the FAIR data principles. And the FAIR principles for research data management were published in 2016 and relate to making digital assets findable, accessible, interoperable and reusable. We support the FAIR principles and we ask for data to be as open as possible because accessible research inputs and outputs drive research impact, quality, translation and innovation, which is important for research and for society more broadly. Having said that, we recognise that translating the FAIR principles into practice will vary for each discipline. Nowhere is this more important than when considering Indigenous knowledges and data governance. In the context of the rise of the FAIR principles in the community of international data infrastructure builders, the Global Indigenous Data Alliance formulated the FAIR principles to address a clear deficit in FAIR regarding Indigenous people's rights and interests. So while in the context of FAIR it is suggested that data be as open as possible but as closed as necessary, the FAIR principles extend beyond this considering collective benefit, authority to control, responsibility and ethics. And this has been useful for raising awareness in data infrastructure building communities as a starting point, but we're aware that it is just that, a starting point. In building the house and Indigenous research data commons, we want to strive to better reflect Indigenous people's rights and interests. This in turn will also inform our people and planet research data commons to better address Indigenous people's rights and interests. So if you would like, we're aiming to be carefully FAIR. And I'd like to give you a few fairly basic examples of how we can apply care to being FAIR. So first of all, findable. So to make findable, apply care to findable, we want to ensure that data is labelled and described using terms that are inclusive and considerate of diverse communities. So for example, when describing datasets, using terminology that respects cultural sensitivities and avoids terms that could be offensive to any particular group. Make sure that data is findable, not only by using standardised metadata and keywords, but also by considering the preferences and language of different communities. So for example, providing metadata descriptions and keywords in multiple languages to cater to a wider audience and ensure inclusivity. So accessibility, making data available and accessible to both humans and machines. To do this, you can ensure that data is accessible to individuals with varying levels of digital literacy and technological access. This is a big problem in Australia with many Indigenous communities having very, very poor digital inclusion levels. So to as an example, you can achieve this by offering multiple modes of access, such as downloadable files, web-based interfaces and even physical copies where necessary. We also need to recognise that individuals may have different language preferences and English literacy levels and provide data in formats that are understandable and usable for them. So to achieve this, you can translate data documentation, metadata and user instructions into languages commonly spoken within the community to ensure inclusivity and understanding and work with community champions to support this. For interoperability and to ensure that data can be combined, integrated and used across different systems, it's important to take into account cultural, linguistic and local context considerations when establishing data standards and metadata schemas to ensure they are relevant and meaningful to different communities. For instance, if data standards include geographical information, make sure they are adaptable to various location naming conventions and local place names. We can recognise and incorporate Indigenous knowledge and traditional practices into interoperability standards to ensure that different world views are considered. So for example, developing data schemas that can accommodate both scientific classifications and traditional ecological knowledge allows for holistic and inclusive data integration and reusable, so making data well described, allowing others to understand and reuse it. We can provide guidance on how to ethically and respectfully use the data, especially when it involves sensitive cultural or social information and address potential biases in the data and provide context for interpreting it accurately. For example, including a section in the data documentation that discusses potential biases in data collection and provides suggestions on how to interpret and analyse the data in a culturally sensitive manner. And we should provide clear guidelines on how to cite and attribute data in a culturally sensitive and respectful manner. So for example, including guidelines on acknowledging Indigenous communities or other cultural groups whose knowledge or data is being used in research while respecting their intellectual property rights. So remember that the care principles are meant to enhance the equity and inclusion while the fair principles primarily focus on enhancing the technical aspects of data sharing and reuse. By integrating these principles, you can create data sharing practices that are both technically robust and respectful of diverse communities. And as Levi Murray from the Indigenous Data Network who lead our improving Indigenous research capability program of work says, generally if it's good for mob, it's good for everyone. So now I'd like to hand over to Robert McClellan from the Language Data Commons of Australia to provide some insight into care and fair from an Indigenous perspective. So let's move over to you Rob. If I can figure out how to stop sharing my screen, there we go. Welcome everyone and hello. My name is Robert McClellan. I'm an industry fellow with the University of Queensland and I am also the program manager for the Language Data Commons of Australia. So thank you, Jenny, for inviting me to be here today. And I just want to say how exciting it is to be able to speak language, particularly in a language of vitalisation context. My community, I'm sitting here in Bundaberg on my own country here today, but in my community alone, we had that common misconception that language, if you only went back five or six years, we all had that common misconception that language was lost, language was somewhat say extinct, but it really shows the impact that can be made among communities when given access to data and data that is usable and to be equipped with the skills and the tools that we need as community to be empowered to start to make that change. So and bring back those cultural revitalisation practices. So I always think it's very important to try and embed language now wherever we are and be using it every day. So I think that's quite fitting in kicking off today's conversations and I'll probably echo a few themes that Jenny has already kicked off in shaping today's conversation. I'd also like to give a bit of a shout out to, I feel pretty technically savvy in that I've got on one of my screens, I've been able to view the attendees. I want to give a warm welcome to my colleagues, both at the University Queensland and the extended institutions from which you are joining and welcome you here for this conversation because I know that many of you are also part of the good work that is happening in this space. So thank you all for being here. So I'm going to tell you briefly just a little bit about the language data commons and I think that quote that I've shared with you up on the screen is probably fitting to kick it off. So the language data commons is many things to many people in particular, we identify a few different user groups. So looking at data users in terms of terminology is important and getting the right terminology. So I want to stress that but we look at data users in terms of people who are wanting or groups of people wanting to use their data. We're looking at the terms around data stewards or data custodians in terms of those who are responsible for that material. And then we're also looking at communities and what I found very quickly into the piece of joining Eldaka is there were many different interpretations of community when I was talking about community meaning very specifically in the space that I've come from being mob and indigenous community while there's also other language communities on the continent so-called Australia and also two research communities and what that extends into further. So the terminology has been a very important thing for us to focus on moving forward and I guess I won't go through unless we have time at the very end I won't go through some of the very specific aspects of the data commons because as I've said it's many things to many people but when we're looking at language material for Eldaka it's about finding securing collecting and organizing when it comes to the tools that we want to use and the skills that we also need to work with that language material well it's also about analyzing repurposing and training. So I guess our overall goal is seeing that communities as I've listed in front of you but also to mob are able to find their language material and use it now that's actually a very big task it's easier said than done and that's the journey that our team is embarking on but quite respectfully yet eager as well on that journey so very exciting bit for us but I guess we have learned a lot of things along the way and I'm hoping to share some of those things with you today so that communities can access their data and do what my community has done and even better in terms of language reuse and starting to socialize that again in in a in a cultural context I'll lastly say on this note that we are concerned with nationally significant language data and that in itself extends to a few different things such as you know um Old English Corpus um Oslang Sign Language uh Pacific Languages um also to Indigenous languages as well um and I will echo what Jenny has said and um our colleague Levi has also said in terms of the curb cut effect which I'd urge you all to to google and have a have a look um more around that that ideology around what what is good for a significantly marginalised group is good for everyone and so I certainly share with what Levi said what is good for mob is good for everyone which is why we find it extremely important that we send to First Nations perspectives at the forefront of everything that we're doing here with Eldaka so I'll just share with you um um just this this resource comes from GIDA which is the Global Indigenous Data Alliance so I'm just going to speak briefly to that as they define data sovereignty and they define data governance in the mainstream sense um in terms of you know Indigenous data sovereignty well that terminology has been out there in the literature since the 1970s um but there's been a resurgence of this language particularly in 2007 with the United Nations declarations on the rights of Indigenous peoples um and it's that one's about reaffirming um you know Indigenous peoples rights to self determination as political entities and honours the principles of Indigenous control over Indigenous data so there's a bit of a theme that has really come out with the work that GIDA has done um I guess the other thing I wanted to share with you around this was that the Research Data Alliance in 2017 acknowledged that there there are in fact um tensions between protecting Indigenous rights uh when and and the interests in data and supporting uh open data which Jenny has gone through with the fair um the fair principles so they then had other academic collaborations that came after that particularly notably in 2018 they had a Botswana conference and the participants there concluded that the existing frameworks that were around um were all oriented towards three things that was data people and purpose um and while they both I guess Indigenous and mainstream principles were identified um around I guess the data centric principles being the ones named in fair well they also noted that the Indigenous frameworks that existed emphasised the people and the purpose um oriented principles so that's where the care principles were then born I just want to maybe offer this to you to reflect on data sovereignty in the West in the the mainstream sense looking at it as the concept that information which has been converted and stored in binary digital form is subject most importantly subject to the laws of the country in which it is located well I'd like to maybe put it to you to reflect on data sovereignty and put it into this concept here where we look at um well it's quite easy to conceptualise data sovereignty in fact when we look at the map of Europe you would not go and ask people in France to comment on and design policy make policy decisions for those in Italy it would be completely inappropriate and that's not lost on anyone we we completely understand that and we can very easily see how data sovereignty would fit into this European concept as you see it fits into the continent of so-called Australia likewise here's the IAZS map which shows 250 different 250 plus different languages and potentially over 700 dialects of that language so I put it to you that the same must be said about indigenous communities here um acknowledging that they are heterogeneous um in nature not at all following the myth of homogeneity that we were once um taught about falsely in our country's history so acknowledging the diversity of those groups and how data sovereignty would work in the the context of this continent here this is the other thing I want to share the authors that talked about um particularly led by Russo Carroll um and many others the authors believed that um harnessing data for governance and acting in the governance of data while there was a relationship between the two so you've got governance of data and then on the other side you've got which well governance of data purely um occupied with those those decision-making processes and ways in which we are treating and managing data but also too on the flip side they have found it important to stress the data for governance which is a push towards this nation rebuilding ideology which is quite strong in countries like Canada the Americas um New Zealand namely all countries who are predicated on treaties with their First Nations people we are the only Commonwealth country here in Australia who does not have any treaties or you know any existing treaties with our First Nations people so it is a very it is quite in that sense a different context I think the language that we have been using for a long time for mob in this community around nation building um and that being a Western terminology itself we're talking about self-determination um not to say that nation building isn't what we should be going towards yes if we learn from those other countries it is it is a way that those um Indigenous communities can assert themselves in quite a strong point um I'll acknowledge to the work of Miriam Jorgensen I've got a book here from the University of Arizona who speaks very much in depth on rebuilding Native Nations and that's the name of the book I'd urge you to have a little look of that too if you are a governance nerd like me but I enjoy it it might be a bit heavy reading for some but it's great um the reason the the reason the authors involved in the Gida work um position this was because they believe that it shifts Indigenous peoples from invisibility within data ecosystems and it shifts them to more vibrant contributors to data policies practices ethics and innovation and I I think um this is where this conversation is going is how communities are asserting themselves in this space so um I'll share with you we have um I guess Jenny talked on the fair I'll talk a little bit more on the care look it's no secret historically early explorers linguists anthropologists and other researchers well they've all developed a reputation for collecting material and data in ways that are widely considered unethical in a modern day context so by Indigenous communities there are increasing levels of mistrust within large institutions both both Indigenous and non-Indigenous to handle data in a conscionable and a culturally responsible manner so Indigenous Indigenous research communities across the globe face these challenges and that was made very clear at the recent collections for data um collaboration which was held in Vancouver where we developed the the uh the Vancouver statement on managing data um so therefore we must aim to disconcert these existing collections methods uh the ones that do not support the security and the longevity of Indigenous datasets and we must move then to or one reframe institutional relationships with communities one that we're very cognisant of um as Eldaka being part of a large colonial institution um that is the reality of which we're working so that does impact the work and how we engage with communities um very aware of that and then two supporting the necessary reforms to enable communities voices to be included and to embed First Nations perspectives within the data lifecycle and decision making processes pertaining to those specific collections again very aware of that um it's sort of been said I'll just briefly touch on it that the authors of the the care um the architects of the care principles believe that that current movement of FED did not fully engage with Indigenous peoples rights and interests um and also to including you know creating value from Indigenous data um and grounding it in world views I wanted to stress that because I'm going to talk about that soon and realizing opportunities within the knowledge economy um so that was that was work led by Carol Airtel in 2020 and also to one of our um very own academics at ANU here Dr Ray Lovett was part of that process too so important to um acknowledge that but what they stressed in this was that it's not one or the other in fact they are complementary of each other you know the goal is for data stewards and other users of Indigenous data to implement care and fair principles in tandem um which we know and so these are more so ways about how we're sort of balancing them both the other thing that they stressed was that it must be done in full you can't just pick the C or the R it's got to be the C A R E you've got to do them all in their entirety okay the point about this slide is just saying that care did not come from nowhere there has been bodies of work that have been done different frameworks different analysis that have been um completely have been workshopped that the care principles have been pulled from um and I said to you before about the orientation the orientation of of the data that being data uh you know principles that were aligned with data people and purpose while this is this again I'm just expressing it through the table that is presented in one of the Carol uh Russo etel papers so if you look across the top um you will see the various different papers which were on the first slide that I showed you um you will start to underneath that you'll start to see some of the key themes that had come from that and you'll also see how they have been grouped into the ones that were associated with data the ones that were associated with purpose and the ones connected with people um I urge you to take a little screenshot of that and reflect on that because I'm going to keep on charging on forward got a lot of slides um okay so this was an interesting one uh in terms of the these are a better description of of the care principles um I don't want to go through them with you specifically because I'm just mindful of the time um I just want to say that the principles do have many applications from an LDACA standpoint I guess we are very much uh interested and concerned with like access authentication data management those kind of concepts but this has has been extremely applicable to everything that we do so I do want to share a few things that we're currently doing at the moment in terms of how we represent the careful fairness as described by today's session um look collective benefit was in is is extremely important um this one's talking about you know supporting indigenous groups community use and the reuse of data the use of data for policy decisions and the evaluation of services creation and use of that data and how that is reflecting community values the authority to control uh I wanted to talk about here again this was around indigenous peoples must have access to data that support indigenous governance and self-determination which is a critical thing in these uh Australian context um this is about mob using data to assert their rights in an everyday changing well in an ever changing modern society I suppose but it's also about the rights uh for us to engage economically and as it's determined by those 250 plus heterogeneous groups we do have the right to do that and we need the data to support it so we need to be consciously working um with that in mind um I want to share uh when we're talking about responsibility there is a responsibility also to nurture respectful relationships with indigenous people from whom the data originate um back in the day a lot of our um historical material lists um there are whole thesis uh well this you know thesis have been developed and doctorates have been conferred for non-indigenous people which contain mostly indigenous knowledges and those people who hold the knowledge have been listed as informants obviously we don't do that anymore but that point about nurturing those respective uh for relationships is particularly important and what I want to acknowledge on that front is western research time frames while they don't always align with the necessary time that is required to establish those relationships with communities and what that means is indigenous researchers who are faced with those challenges well this puts immense pressure on those mob who are working in this space to actually succeed in that western research environment so I want to also acknowledge those um indigenous researchers who are currently or have been in the past currently are or potentially will be working in that space in terms of what we are facing to succeed in a western research environment I think that's important to stress um I also want to acknowledge to oh well that's with the authors um also spoke about the aspects of the relationship including investing in the capacity development um and increasing community data capabilities and embedding data within indigenous languages and cultures well this is something that we're very much aware of we know that lack fella can't be what lack fella can't see and when we are faced with those challenges and succeeding with integrity in an ever-increasing conflicted world um even beyond the research space it can be quite a difficult job for indigenous people working in that space um but we can't truly represent those perspectives without putting indigenous people in those positions and working with them and those communities to increase their capabilities and see that they are starting to reflect the representational leadership needs that um we all need collectively um and also to just mindful uh on the other side of the coin we're mindful of overburdening mob but also to expecting them to volunteer the amount of times we set up um steering committees and just want to place mob on there because um it's beneficial or it ticks the boxes well this process if we engage with them in an inauthentic way well that process is considered black cladding that's what we call it and sometimes we we see organizations doing black cladding to satisfy funding objectives but is certainly not appropriate when we consider though the need for respectful relationships um as articulated through the care principles um again and ethics is talking about you know indigenous peoples um putting them in that position so that they are able to minimize harm maximize profit promote justice and allow for future use but in fact um it's that representation and participation um of indigenous people you know for them to actually discern and then assess those factors so that's the important part there we talked about the authors talked about worldview so I want to share this this uh that was that was within the literature but this particular framework is something that we have used with Eldaka um and it comes from Sarah and Williams so when it comes to indigenous governance the western governance lens often fails to accept appreciate that indigenous peoples of Australia managed to govern their day to day lives effectively for millennia while exhibiting the complete array of human intellect and also respecting the right of their neighbors across the continent to manage their societies with their own specific style of indigenous governance based on their law and law and custom Sarah and Williams argued that the western governance lens does not adequately understand the full impact that colonization has had on the indigenous human physical and sacred worlds they also articulated the critical need for trauma informed policy responses to be developed noting that western governance processes fail to acknowledge this um um I shared this one because this this was this was an indigenous worldview particularly balancing the three worlds uh in the notion the greater notion of balance and harmony so um seeing that when one was valued over the other and we fell out of balance and we fell into a state of disharmony what were the things that we would employ from a cultural governance perspective to bring about balance and harmony again for that longer sustainability this is coming from the longest you know longest living continue our longest continuous living cultures on our planet in terms of sustainability and and and that kind of worldview so I again I'm I'm sort of rushing you all through this because I am watching the clock but um that is an important thing to balance too happy to talk further about that if we get the chance but certainly something that was at the forefront of LDAC as planning which has then in turn influenced a lot of things that we've done moving forward um we had a reciprocal governance model which we um which we shared from the start you know around this these high level governance methods but what this was really speaking to was around that responsibility and ethics part of the care principles this has been guiding our team from the early days within the process and these three things are in reciprocity the point about me positioning purpose in the center was really built on the work of Simon Sinek the start with the why again I'd urge you to go and look up that as well um truly understanding why it is that we're doing what we do and noting that that why is going to be different for those different um user groups data custodians and communities that we're working with with LDAC a um a very large part of our engagement was uh well it was about a part of the project was about the engagement and the design um I'm actually not going to talk too much about this one because I'll push us over time but again this was a standard process looking at trying to shift us from an informed consult low level of engagement um trying to position us in a in a place where there was higher organizational impact yet understanding the higher level of complexity um but how did we do this in a way that wasn't so gammon you know we're looking at a I guess a bit more of a matrix and how we're putting people up into um into that higher module I do want to share this little quote I know Angie Abdullah who's a travel way um Aboriginal woman talked talked about design when she talked about that participatory um that participatory design concept saying that um she said we have to get beyond participation it immediately implies you are not leading and that's my biggest problem it has to be Indigenous led to ensure that Indigenous knowledge is respected the whole way um Indigenous services have to be driven by Indigenous people so she said as Indigenous communities have put it elsewhere we don't want to be consulted we want to be at the table so wherever possible how are we working in that top quadrant when it comes to Indigenous communities and representing those um those values so this was a way that we tried to really transcend those those um you know those standard levels and reframe it into a slightly more involved model um I want to share this you know we have our our DACA project steering committee which comprises of mostly the CIs or the Chief Investigators of the project um it is a cross institutional relation relational governance model where it is quite complex in that there are many there are many universities who are involved in the outcomes of our project who have decision making uh authorities with from a from a governance standpoint so it is quite quite a um interesting way um but how are we doing that Blackfella representation stuff how are we representing those various communities um those CIs are charged with the responsibility to make those decisions but how can we be assured that they are making the right decisions we are now coming into a a time in our if we take it from a social point of view we as mainstream Australians are going to make a decision soon in a referendum that impacts Indigenous communities you owe it to those Indigenous communities to be able to make an informed decision likewise these are CIs sitting on the steering committee owe it to those communities to be making informed decisions so we need to see that we have a governance structure that does the best it can be with the resources and the constraints placed upon us to represent those views we are establishing a communities board and we are establishing a scientific board as well where we can start to represent those views and be assured that we're making those informed informed decisions um which the management group then take direction from so that's one of those things um I want to share this with you this one directly involves communities um we if we are designing uh infrastructure that should benefit those communities well we need to back that by data we need to have the data to that um informs us and guides out the way that we work so we are doing um we are doing a call out for a quite a quite a comprehensive research project in terms of working out how communities are working with language data and what they're doing so I'm actually very excited about this one and um do see one of my a few my colleagues online who are working heavily in this space too so we're very excited about that um as to not assume that we know what's best and even as black fellas working in this space we ourselves cannot report to know what's good for communities we need to see that we have got that hard data there to support our decisions so just making that very important um and lastly too I just want to share um a little bit of an example that you'll see the care principles are broken into further sections but um I just wanted to acknowledge because I guess what what I what I was left with thinking about care was this is very easy to apply to new data but what was still unclear to me and I'm I'm wanting to have that wanting to be informed is what does this mean for how do we do this retrospectively or for historical data that's the that is a challenge that exists that um we're still sort of juggling with um as I say it's easy to do it for new data but I do want to give you one example and I won't do it justice but I do see um my colleagues online from the uq fryer library and I did want to just briefly share um a success story that I just thought was fantastic they have done this initiative called evaluating and respecting Aboriginal um and Torres Strait Islander knowledges and perspectives in uq special and research collections and what they've done is they've been able to go through and better describe um within the metadata structure which is supporting that accessibility they've been able to better describe some of the material and the collections that they have got in their collections but one of the things that I thought was pretty deadly was as I said before that indigenous knowledge is how that was represented and shifting you know black fellas from being just listed as informants shifting them to a position of co-authorship which is really how we would do that with new data um in this day and age that is the way we'd go about um creating those works but I just wanted to give them a shout out too because um I thought that was pretty fantastic and and I think they do have that available online for you to read through a little bit more thoroughly but with that I will probably conclude um so thank you for that thanks Robert um I think that we will take any questions if there are any there's nothing in the Q&A box at the moment Jenny well if anybody's got a burning question now's your chance to put it in the Q&A box I'll give people a few seconds then I have some that I put in the oven earlier someone okay question from John Brown do you have a link to the Vancouver statement that you mentioned yes I do and we do have it through the ARDC one um I'm yeah I'll find that and put it in the Q&A unless you've got it at hand Robert oh I can grab it I'm on there now okay okay so thanks for that John we'll make sure we get that to you uh any other questions that people would like to ask where I dive in with one of my own um okay Brittany Carter oh any thoughts on the use of traditional knowledge and biocultural labels and notices from local contexts um we we recently Robert jump in if you want to but recently we held a data licensing access and authentication for Indigenous data round table up in Brisbane it was our project partners and some invited guests we had quite an in-depth overview given of the TK and BC labels and notices I think it's important to mention the notices especially from a white perspective and we are going to be looking at carrying a pilot on applying the notices with a few collecting institutions and then hopefully creating relationships with some communities to work on the labels as well so obviously with the as you may know the the notices are really there for application by the collecting or the custodian of the material and the labels are to be applied by the communities that the material relates to so we do have a pilot in in process okay so John and I just wanted to just briefly just say on that front we're also looking from LDAC as point of view we're looking at piloting some work just specifically to Queensland around that to better understand it and see how its application would work in our context so we we really wanted to give it a good hard crack before we we made any further comments on it but we're committed to that okay so if I can just take that a little bit further Robert you mentioned the problem of applying care retrospectively is it going to be less difficult less fraught to apply TK and BC retrospectively or is it the the same cluster of issues well I I won't speak too much on it because we're bringing in a lead who knows far more and did a PhD on that specific thing so but as Jenny sort of alluded to in my understanding is when we're applying the notices that's quite a different process so I associate applying the notices to that retrospective material and then that I understand that to be the way of opening the process up to community to get them in to work with that material so in that regard applying care principles to retrospective data there might be a there might be a solution that lies within the TK okay notices yeah all right that's great Jenny we've got one minute to go I think we should probably do a wrap up sure um well I'd like to thank you Andrew for fielding the questions for as much as they were and thanks to Robert for his insights and thank you to everybody who came along and attended I hope you found it useful please feel free to reach out to myself or Robert if you can't find Robert's email address reach out to me and I can direct traffic his way I'm very happy to continue the conversation offline okay and how will we make the recording available Jenny slash Asha um Joe will make our wonderful comms team will make the recording available and if you've attended we'll send the link out to you afterwards but we'll also be sending it out to our mailing address you can sign up to the has an indigenous email if you would like to on the ARDC website excellent thank you very much thanks everyone all right thanks everyone


Excuse me, it's a little awkward now because so many people have left, but we've heard about a lot of amazing opportunities at the meeting, and you've also heard that the funds are not infinite. So I guess a question to ask is among the various things you've heard, what are the dependencies? And so are there some of these that where if certain things don't happen, then the rest of it can't go forward anyway? And so in terms of thinking about priorities, which are the things, you know, we don't want to throw, the problem with asking for priorities is there are all those ones that are lower. So let's talk about the higher priorities. Which are the things that need to go forward in order to make progress in this field? And without which, the other things that we would like to see happen are not likely to happen. Is that a reasonable question? If a hard one? Nobody's going to want to answer this, I know. I'm going to take a stab at this. I think I heard some consensus on the value of continuing to build the catalogue. There are examples Manolis Kellis using from the roadmap data talked about the connection between Alzheimer's and the immune system, which came out of, you know, unbiased production of data from many different tissues. And if you had focused on Alzheimer's in the brain, you would have missed the immune connection. There's a very, very stimulating, you know, talk. And so I think there's an argument that says, I don't know who said it earlier, that you really don't know what you're missing until you see it. And to some extent, that catalogue is going to be dependent on the practical realities of what cells you can get. And so groups need to be able to get access to the material if they're going to make a proposal. So somewhat, it will be directed by, you know, what's out there in the community already. So by its very nature, the project has to connect to the community for samples. I think that's the way it's been in the past. It'll probably be the way it will be in the future. But the systems have been evolving. There are all sorts of organoids now. My colleagues at Salker have kidney organoids, brain organoids, et cetera. And so I think there are other ways to get to it. But I think getting at this catalogue is going to be very valuable. Not in a way that you can couple it with function. I think they're not either or. I think if you pick the right system, and, of course, this is a ground up, actually, right? Because investors are going to propose different systems if the project goes forward. And if your system couples with an assay that's high throughput, so be it. We want to get to that encyclopedia. I think all the PIs think that that's an important goal. But it has to be coupled to the continued development of the catalogue. Well, in fact, if you're going to have a path for community generated cell types to enter a pipeline once the cells and the technology are available, there has to be that infrastructure, I think, of some production act, some level of production activities. So let me just jump in back. We have a project, Mike Snyder and I, this California Institute for Generative Medicine has a stem cell genome center. And that project is to enable genomics in the stem cell community. And some of the funds from the center flow to those investigators to prepare the cells. That was on a competitive basis that those groups were chosen and say, hey, here's an interesting biological system will help you do the genomics. And you've got to have all the cells. You've got to have hundreds of different cardiomyocytes or whatever. It's not necessarily a cell catalogue, but focused more around the disease. So I think that's a reasonable thing to do. The groups that have the ability, either they're brought into the consortium directly as part of a proposal as we did being rented with Jamie Thompson, or as John has a lot of collaborators getting samples within his community in Seattle, that it'll have to be an integral part of the project either way. Yeah, and we've, you know, as many of you know from the sequencing program, we're trying to invent ways to engage as much of the community as possible in developing those samples opportunities. I'll go over there. The other one I wanted to comment that I wanted, or the other priority that I wanted to mention, which we haven't. Well, I guess we have talked about, but not explicitly as a priority, was the collection and vetting of data through the DAC and the DCC. It seems like this cannot go forward without a robust effort for those activities. Is there a fairly good agreement on that? No. I just want to stress out one point. If indeed the goal is to move forward in having more cells added to the list, which I fully agree with, and this is something others have said, and I'll just voice my, I'll put my opinion in the same direction. It's not that they need to be fully characterized, but characterized on a subset of specific features that need to be defined so that we know exactly which ones we should prioritize, and at the same time keep investing in in-depth characterization of a small subset of cells, such as what we have right now for the FG2, the K562, and the GM12878 cells. I think you need to stretch the grid on both ends, not just on one end. Yeah, I guess I wanted to echo the idea that sort of encode, the Tier 1 lines are incredibly valuable for, they're kind of like model organisms for cells, right? We choose a few model organisms because they're useful and you explore them, and encode is choosing a couple cell lines to go really, really deep on, and at least as a methods developer, that's incredibly powerful to have that mountain of data to synergize with when developing potentially new things. And I would also argue that just as far as just understanding cell biology, it almost doesn't matter what kind of cell you choose, in some sense, to understand these broad sort of organizational principles, but going into those Tier 1 lines and trying to get at functional validation or magnitude of effect for these regulatory elements would be very, very powerful, I would argue, and add another dimension, a validation dimension to these cells that are sort of model organisms of cells. And I think that that, I think, fills a lot of the peoples of what encode should be in terms of sort of breadth-first understanding of a small number of things and also depth-first, but much less dense, as was just recently said. Thanks. Well, any comments on that? I will set it, but I'll just echo it. I think there was consensus around it, maybe there wasn't, but validation as a theme, right? Meaning that as a more significant component of what's going on, even if it's directed at validation of things that were predicted by encode 1 through 3, right? I strongly agree that there was a consensus around the notion of validation. What I struggle to understand is what people actually mean when they say validation. And I ask that actually multiple times, because I know what I mean for very specific purposes that I happen to care about. I actually don't even want to say it, I don't think it's relevant to it. But I'd love to know what it is that you mean in the back of people's minds when they say the word validation, and it's usually coupled to the S word function. So it's kind of... And I don't think it should be... The notion of it, I don't understand what it is. And I don't think it should be overly specified, meaning it's a sort of thing that one could look at a bunch of... Sure, yeah. No, the spectrum of what it means for someone that something has been validated. So one of the discussions that we've been having with our council is it's really awkward, I can't see a V, then I can't see my people. That's all right. It is about the balance between top-down and bottom-up, investigator-initiated and consortium and so forth. So are some of these various things that we've talked about including perhaps validation, the kinds of grants that could come in, if we would say we're going to set aside some portion of the encode budget, whatever that is, whatever the total is, and whatever the set aside is, that we anticipate that there will be really interesting applications that will come in that won't be part of a cooperative agreement, but they'll be working on encode-type problems such as these validation problems that would use encode data sets in an interesting system and look at validation. And could these be reviewed effectively at a study section like GCAT? I'm really getting down into the weeds now, but these are the kinds of things that we're going to have to start grappling with starting at 4.30 this afternoon, right? I mean seriously. Or is this the sort of thing... So now you're asking that we describe in a program announcement or something what we mean by validation versus letting people come in with ideas about what they mean by validation and there may be different ones. And so we could do that and we could either have our review office set up a review of people who are really thinking about this or the reviewers are really thinking about this or let it come in through CSR. So just, yeah, come. I think if you maybe made it one level higher and didn't call it validation and for example called it a program in in-depth functional studies of encode elements, I think that's something that could catalyze people, you know, generally because you don't want to... I think the word validation... I was using validation as an example of the sort of thing that might come in under this. But that in-depth functional characterization and, you know, to include... It essentially becomes de facto validation. But that, I think that would organize... Rather than just wait for the stuff to come in, that would organize a response because a lot of people are interested in it. So, I mean, Paul, okay. So I think that what you suggested sort of initially that instead of having a small group of people decide what validation means or what assays are the best to sort of put out, you know, an RFA, put out something and say we want ideas. Give us your best ideas and have a grant review panel look at the ideas because, I mean, we've talked about this over and over again in the past day or so that we all agree that validation and characterization of these things is essential but there's been no consensus on what's the best way of doing that. That's one big hole, as you've said, that we all are struggling with and we all may have our own ideas but I don't even know if we have formalized projects yet. But I think that's something where that's the sort of the best of science when you let little people come up with their own ideas, choose the best of those, give them money and then see what comes from that. So I like the idea very much of having that be an investigator-initiated approach. And talking to people, various people in the community, they were excited about functional tests and validation but they didn't want, they really adamant about it not being the same production centers. They got to really bring in some new blood or open it up, let everybody compete and get the best ideas, as Karen said. I would put in two comments. The first is, I wholeheartedly agree, I think there are two scales to it. There is functional validation by the expert for a small number of things at exceptional depth of follow-up on the mechanism, which has huge value. And there is functional validation that is done more on the bread scale, still in the biological system, still with an assay that needs to be done by the investigator and there has to be a balance in the portfolio between these because there is value in each. And whether they're both done through the same mechanism, I don't know. And the other, because you posed it very specifically about the study section, there should be a special emphasis panel. I do not think that you would get the necessary expertise in the room in the standing study section, definitely not in GCAT, which is a great study section, but that's not its area of expertise. Other comments. Are there other questions that other NIH staff want to ask? Yeah, John, go ahead. I want to make one comment regarding the sort of RFA versus GCAT and I will disclose I'm a member of the GCAT study section that the problem with RFAs is that there's a timing issue, right? In some way, you just flush out what's ready and you don't wait for things to develop. And I think the advantage of a program announcement, maybe with rolling receipt or whatever, and I think a study section like GCAT, as long as it was prepared and new, I don't know, I haven't tracked the number, but my impression is at least 20% of the members, if not more, are recruited ad hoc to deal with the specific grants that come in. It could be, that number could even be much higher. So I don't think it's impossible, but they're, you know, I think the rolling aspect is the most important, is the fact not to capture one point at a time. Yeah. Okay, anything else? If not, I'll turn it over to Elise. Thank you. So I just want to thank you all for being here. I have just some brief slides. Really just thanking everyone who has helped organize this meeting, the organizing committee, and the, and co-consortium, and do we have all that coming up? Just this one. So we really couldn't do this with the organizing committee. It really was huge in organizing the meeting, as well as leading the discussions and synthesizing this has been really incredibly helpful, as well as the Capital Consulting Corporation. So the encode consortium obviously needs to be thanked again, as well as the encode external consultants panel. And a big thanks to our colleagues in the NHGRI Communications and Public Liaison Branch, specifically Lvaro Encinas and Chiara Palmer, who are in the back who have been here with us for the last two days and have made it possible to have the webcast and are doing this with a very rapid turnaround. It's already available. Last night's already available. So thank you very much for your hard work on that. And then finally my NHGRI colleagues.  And we really appreciate that everyone really did participate. We tried to keep this a small meeting so everyone felt comfortable to speak up and think it really worked. Thank you so much. Appreciate your help.


Let's get started. So our talk is called tales from on call fun with operating at CD at scale. My name is Geeta and I will be joined over video by Chao Chen. Both of us work at Amazon web services. So let me introduce a little bit about us. We work for EKS. EKS is a managed Kubernetes service. What that means is EKS manages the control plane for you. All aspects of it. Performance, scalability and availability. So all the components here in the blue rectangle are owned and managed by EKS. Customers don't have access to those. They typically just manage their workloads and sometimes the worker nodes. So within the control plane, the team I am from is the HCD team, which focuses on operations and contributions to HCD. Now let's hear from Chao about our HCD environment. Thank you Geeta. Good afternoon everyone. My name is Chao, a software engineer at Amazon. I've been actively working in the operation of HCD, the distributed key value store that is used by Kubernetes as its primary data store. In my virtual talk today, I'll be sharing some of my experiences and insights of operating HCD in Kubernetes clusters. HCD at EKS. Each EKS HCD cluster is a three node cluster. Evenly distributed across three availability zones in a region. Availability zones are isolated data center located with specific regions in which public cloud services originate and operate. Second, EKS HCD uses static IP to advertise HCD client. What endpoint it should connect to and for peer communication. We also use static volumes to store wall and DB files. Static here means every HCD node where we use the same IP and volume after the previous node is terminated. It guarantees data durability even if HCD Chrome is lost or permanently. It also indicates the HCD membership is static. There's no membership reconfiguration. It simplifies the operation. EKS HCD supports version downgrade from 3.5 to 3.4. It's an EKS private patch and there is a reference published in upstream. EKS HCD 3.4, 3.5 and 3.3 does not have storage and API layer schema incompatible issues. For example, the difference between the 3.3 and 3.4 wrapped internal protocol buffer schema change is a list checkpoint request. If in a 3.4 liter the experimental list checkpoint feature is enabled, once it starts to replicate the entry to the 3.3 follower, the follower as its server cannot understand the entry schema and we just panic during deserialization. Of course, EKS HCD runs as a system D diamond service. Unlike runs HCD in a container, we run HCD as a system D service because it's a simpler approach with less overhead, involved like network isolation and container orchestration. Fifth, EKS HCD operator agent runs in the same box as HCD. In EKS, we run HCD operator agent in the same box as HCD to manage provisioning, health checking, taking periodic backup and storing persistent storage, automatic defrag and monitoring, etc. No space alarm self service or every touched upon later by Kita. That's the introduction of HCD at EKS. So now that you guys know who we are, let's get to the agenda. So today we want to talk about five operational issues that we see while operating HCD. The first one is where the storage quota provided by HCD is not enough for the workload. Second one is the revision divergence issue where the nodes don't agree with each other. HCD can run out of memory. That's the third one. We sometimes see timeouts mostly related to the maintenance workflows of HCD. Chow will talk about that. And sometimes we see requests that are too large than the limit HCD runs with. Let's get started. So the first one is the database size quota exceeded. Before I get to the production issue, let's recap a few concepts. So this is a simplified view of HCD layers. We have raft for consensus and then we have the backend storage, which is BoltDB or the DB file, which is a big memory mapped file. The concept of quota applies to the backend. This is a toy example where we show key A, which is shown in pink, and key B, which is shown in green, and those two keys have filled up the file. A1, A2, A3, they represent updates to that key. And A del represents the key getting deleted. Same for B1. B2, B3, they're updates to that key. So things to note here is that when the file fills up, there are multiple revisions per key occupying space in that file. The other thing to notice is that it's copy on the right semantics. So every update to the key takes up a new page in the file. Deletes work the same way. So a deletion will also add to the file. When the file hits the quota, an alarm is raised, and that needs to be explicitly disarmed. Today we run with 8GB quota, which is the maximum supported limit upstream. When the alarm hits, the cluster becomes read-only. Any modify operation, put operation cannot get in. The next concept is that of compaction. So compaction cleans up the old history. So in our toy example here, if we compacted everything that was in that file, only B4 will stay. Everything else will get cleaned up and the files will have free pages or holes, if you will. Under normal circumstances, when there is no alarm, these holes are usable by HCD. But when the alarm has already triggered, the put request will be declined even if there are holes in the file. In EKS environment, compaction is run by API server every five minutes. The next concept is that of defragmentation. Chau will talk about this more when we visit the timeouts. But defragmentation basically removes all the holes and packs up the live data into a brand new file. So again, in our toy example, B4 will sit by itself in a brand new file. At this point, the size of the file will drop. However, like I said, the alarm will not clear by itself unless we call a specific disarm API. All right. So in production, multiple times we see workloads exceeding our 8 GB quota many times unintentionally. When this limit is reached, cluster becomes read only and your on call gets paged. The interesting thing is that the compaction as run by Kubernetes API server stops working when this alarm is raised. This is because the compaction workflow from Kubernetes API server needs to do a put request before calling the compact API. And since that put request cannot get in, it never calls the compact API. The operator wakes up and then we typically increase the quota. And then it is a coordinated activity. We request the customer to do the deletion of objects before returning the quota back to 8 GB. So why do we see this issue? There are three main factors, object size, object count, and then multiple revisions due to fast updates. So this is an example from production where we see this last key here, the admission reports, there are 2 million objects. So even if the object size is just few KBs, this will rack up GBs of space quickly. We have a blog article out about this, which you can find online. The second example, object size. This is when the workload typically has a big key or a big binary blob in part spec, such as SSH keys typically. Because it's part of the part spec, it gets replicated. And the part spec becomes big, like 500K, 800K, those are the bigger ones we have seen. While this is supported, this is not optimal. And there is an easy way to optimize this by referencing the big blob instead of embedding it in the part spec. So we talk about that in the blog. And the last one is these repeated updates. We often get questions like I have just 1000 pods, how come they're taking up a GB of space. So remember that when the quota hits, there are multiple revisions per key. So if something goes through fast updates, it's taking up much more than the size of that object. Consider a 800K part spec, and it goes through 10 updates. It's now taking up 8 MB per pod. So 1000 of those is going to consume the 8 GB. Typically, we see this together with the large object size. Audit logs can help identify what's changing fast. And we have our CloudWatch query example in our blog. Other providers can also have similar query or on-prem also. You can find the query for audit logs. Here is an example of a fast changing objects. Here, the scheduler is updating a pod repeatedly just to record the fact that it's not possible to schedule that pod on any node. So these are unintentional side effects, but they do eat up the quota. This can be monitored for. So for monitoring this proactively, API server has a metric for this. So there could be a monitor on that. Detecting it reactively, it has already happened. Then there is this specific log message that you can look for, which will tell you that database quota is exceeded. Mitigation. So today, as I said, when the DB size approaches the quota, it pages one of our team members who are on call, and they wake up and they do this coordination to get the cluster back into operational mode. We have some work in progress to automate some of the workflows we do manually today. In long run, we would like to work with the community to enable a self-service experience for this, such as a file system.  Maybe in case of HCD, you wait a little bit, but then your cluster comes back to rewrite. We have thought of increasing the quota. We used to run with 4GB. Now we run with 8GB. Increasing it any further will need more testing and validation for performance. We are at the limit supported by upstream. All right, so that's the first issue. Now let's hear from Chao about the revision divergence. Thanks, Kita. Let's go to the next topic, HCD revision divergence in a cluster. Here is the animation of the revision progress in HCD. We can tell we are watching the key foo to get notification when the key value pair is changed or updated or deleted. You can see whenever we update a new version of the full key value, the revision is incrementing and as long as we delete that revision, it's also increasing. HCD uses this global, monotonically growing revision number to keep track of changes to the data stored in key value store. When multiple nodes in the HCD cluster are making changes to the data simultaneously, it's possible for the revision numbers to slightly diverge and converge to the same revision eventually. However, diverging revisions could also mean that two or more nodes in the HCD cluster have made conflicting changes to the data. For example, one nodes successfully complete the modification or the other nodes discard or partially commit the modification. In EKS HCD, this will cause sustained growing revision divergence across HCD nodes. Let's take a look at the real-world example. Recap each HCD cluster in this ring node cluster. The orange line represents the maximum of the revision across three nodes and the green line represents the minimum of the revision across three nodes. The blue line represents the gap or the divergence of the above two lines. So you can clearly see the divergence grows starting from 10 o'clock and then indefinitely growing. So we observe these failure symptoms and configure in our alarm systems to alert on-call if the divergence grows for an hour. It helps our team proactively mitigates the data inconsistency problems before customer notice the impact. We also open a HCD feature request to demonstrate an example to set up the alert roll currently incorrectly in Hermitius and Grafana. The link is shared below and it's on my to-do list to complete it. You may wonder how come the revision divergence grows. So this is the coast snippet from API server updates the objects. So it's using a HCD transaction API inside that transaction. If the HCD servers stored a proprietary key modification it is the same as the requested API servers kept revision number. Then it will result into a put request. Otherwise it resulting in a get request. So as you can see if initially the prepared keys modified revision is different with this core pattern from client the revision divergence will grows impacted to Kubernetes. The core Kubernetes components will fail to acquire release and stop functioning. For example Kube scheduler and Kube controller manager. And also many other leadership-based controller will fail to acquire release and stop functioning as well. In the other words deployment scaling and post-scheduling will fail. So the following screenshot is an error message from the Kube controller manager and Kube scheduler. Next mitigation. The mitigation can be simple once we detected the failure mode. We just remove the revision lagging behind member and join a fresh new member to be in sync with other members. So it's an exception to use static volume where basically replace the DB. So this is followed by the upstream operation guidance of the encryption. Next root cost. The root cost can be classified as two categories. One is ACD and VCC partial commits and followed by ACD process panics or like CQL the process. It's well summarized in the upstream data inconsistency summary and the maintainers did a tremendous great job to preventing this in current 3.5 release. So if you're interested we should you can take a look and contribute in that robustness test framework. The second is a little bit unknown about a both DP or it's called ACD backend corruption. So about the revision divergence root cause he mentioned there was a talk by a maintainer Marik before me if you missed it please check it out he goes he does a deep dive on how they found the inconsistencies and now have a robustness framework for testing. All right the third issue ACD can run out of memory under overload. This issue cascades because if one node goes um the same workload that brought that one down goes to the next one brings that one down as well and now we have quantum loss. Typically our case studies suggest that the workload that causes this are large unpaginated range requests typically the same one get parts repeatedly. Why do the why do we see this issue so these are typically the unpaginated list requests like I said it's basically spiky workload it spikes too much too fast. Every new request is a new allocation in hcd. hcd will never return any cached answers and the mechanism to free up that memory the garbage collection is asynchronous. So there is a window where hcd can run into memory pressure and can um. So we have two mitigations for this the first one is a change on the api server side this change when it sees an unpaginated list request it paginates it and it before sending to hcd. So hcd client and server will always see a paginated request. The pagination limit is configurable the link to the cap is on the slide even if we do this pagination we could land up with a page that the request can still be spiky and can still cause the issue depending on workload. So we have a second layer of defense on the server side this is a server side throttler which is implemented as a grpc interceptor. So it's watching every incoming and outgoing request. If it is not a range request it doesn't do anything to it it just simply goes through. If it is a range request then the throttler will check if the box is under memory pressure by just consulting the resident set size of the process as a percentage of the total box memory. If the memory is under pressure then the request will be admitted after delay it will throttle it. So this graph shows the test where you can see the throttling in action the blue line here is the memory pressure when it goes past 65 percent which is the threshold it's running with the throttler which is the orange line that activates and it starts throttling the request this is the same test except it's trying to show that the throttler and the memory pressure feeds into our scalability system so in this test the orange line is still the throttler metric but the green line is the total system memory. So this box got scaled up from 4gb to 16gb because of the throttler signal and the memory pressure. Someone might want to use the same changes we are using. You may not need it in case you have total predictable workload you can always provide for the peak workload use a bigger box but if you are interested in trying out these changes please do reach out to us right that brings us to the fourth issue let's hear from Chao about the timeouts. Thanks Gita let's go to the first topic timeout and causing 5xx one of the top contributors of Kubernetes API server 5xx is ac online defrag stop the word. So as you can see from the diagram when API server trying to get an object using key1 if the request hits to a acd server that is defragging then request with hands until timeout what is acd defrag? The following statement where we quoted from acd document website the key value store is effectively immutable its operation do not update the structure in place but instead always generate a new updated structure or faster versions of keys are still accessible and watchable after modification to prevent the data store from going indefinitely over time the store may be compacted to discard the oldest versions of data. So you can see from the db file there are multiple revisions for a key value pair or just a single version a single version of key value pair like k2 so after we compact revisions 7 you can see any key value pair that is deleted it will be compacted or any previous version of key value pair that is superseded with a new version that will also be compacted and leaves free pages in the in the db file. Compacting the key space key three drops or information about keys superseded prior to a given key space revision the space used by these keys can then becomes available for additional rise to the key space so at this point after compaction the db file will not shrink but the free pages can be reused later like if you insert a new key value pair then the free page can be reused after compacting the key space the back end database may exhibit like internal fragmentation any internal fragmentation is space that is free to use by the back end but still consumes storage space compacting old revisions internally fragments sd by leaving gaps in the database back end database fragmented space is available for use by sd but unavailable to the host of our system deleting application data does not reclaim the space on disk so that's when the diffrag uh come come into the play after diffrag db file size shrink shrinks so what's the mitigation e-case adopted is reduce the frequency of online diffrag to run once every 48 hours for each member and to reduce the latency of diffrag we advise customers to keep the number of keys in sd small and increase the disk support on demand if there are a lot of objects or key values under mitigation it could be a diffrag offline if the what diffrag offline means is take down the sd server a diffrag the db file shrinks the file size and remove all of the unused pages and then bring back the sd server online so you could mitigate this if the availability risk is accepted um some notes is remember to diffrag one member at a time uh usual words uh acd upstream has some great proposal to graceful uh diffrag acd um well either from client or from server first is from server to make diffrag concurrent internally and not blocking uh other uh other uh from client if it's a multi-node cluster uh the sd server can notify the client i'm going to diffrag please fill over to other endpoints that is not diffragging brings us to the last issue which is request size too large so acd has a limit on the request size the default is 1.5 mb that's what we run with but sometimes we see workloads which want to push more data through a request uh why do we see this uh typically it's a workload issue so here's an example where a customer was using endpoint objects and they were updating the service and it got bigger than 1.5 mb it turned out to be a known upstream issue and the solution was to use endpoint slices instead of using endpoints so mitigation typically we prefer to stay with the upstream limit for this issue and we have helped customers uh change the workloads to match best practices to make this issue go away all right um so to summarize uh we talked about five operational issues that we see first one was the database size exceeding uh second one was revision divergence we talked about the panic due to out of memory we talked about timeouts due to defrag uh and we talked about the oversized request uh that's it from us if you want to share your experiences operating at cd please reach out to us we would love to learn uh if you want to contribute to please check out the contributing guide at city maintainers are here so please check out the booth uh and feel free to reach out to us on slack any questions uh thank you uh maybe you can use the microphone over there uh how much are github's controllers causing all these issues sorry can you please repeat how much are github's controllers causing all these issues because they can hit a lot on etcd right so um can't put a number on it but we do see list requests uh from argo time to time any other questions all right thank you have a have a great conference thank you


The On The Run 2 tour showed no sign of the power couple beyond saying Jay-Z slowing down. Described as one for the history books, the car as latest tour performance at the London stadium was one not to be missed. But if you were at home jealous following every OTR2 hashtag on socials wishing you were screaming 99 problems but you not witnessing it was actually one, then don't worry, your boy ties here with what went down. Being in London, the couple also took a moment to dedicate forever young to those who were tragically affected by a Grenfell. Jay touching his little green heart on his lapel was one of the many symbolic images which was why this was one for the history books. Beyonce flanked by her husband and backed by a mostly female all-black band and dance troupe was how she might word it flawless. Naturally, one of the biggest talks about moments was the surprise announcement of the couple's new LP, Everything Is Love. And while we're sure you've already been listening on repeat using another title-free trial, we're gonna need to talk about all the secret nods to the couple's relationship that are hidden in those lyrics. We all know the past few years have been rocky, there was Lemonade where Beyonce talked about Jay's cheating and his 444 which served as an apology.    Ultimately it looks like B has found her forgiveness and wants us all to too. Ariana Grande has a song on her new album called Pete. But after Ari released the track list for Sweetner, some fans thought it was too soon for a Pete interlude with one tweeting, Pete girl we know you love him but are you dumb? But her reply will confirm everything you've ever dreamed about love. Ari said nah and then for real, the truth is, tea emoji, I've been the f***ing through it and life's too short to be cryptic and s*** about something as beautiful as this love I'm in. So Pete it is. Yes, adopt me, Sweetner is about to go on pre-sale. Hands up if you're excited for an interlude with Pete. There's also Zane Drums, Elle Malikos just dissed the self-indulgent Met Gala despite once going with his girlfriend Gigi in 2016. But now says he only went to that one to take the piss. I did go but I didn't go there to be like, yo, take me serious. I was taking the piss. Oh, okay. I went there as my favorite Mortal Kombat character, Jax. But despite dragging one of the celeb calendar's biggest nights all the way down its very own iconic steps, Zane did at least backtrack as far as his back on girlfriend is concerned saying that the only redeeming part of this year's ball was Gigi's stained glass Versace dress, which stole the night. And while everyone's trying to start drums about Selena Gomez being hurt by all that Justin Bieber and Hailey Baldwin PDA-ing saying stuff like, Selena is doing her best to not pay attention to what Justin is up to but it's impossible. It's painful for Selena to see Justin with Hailey yawn. Selgo is actually living her best life on vacay in Italy and even snapped a cute fan pic with her wizards of Waverly Place co-star David Henry. Hey, Henry. Chloe Kardashian said she will speak about everything on the new season of Keeping Up with the Kardashians. And when I say everything, I mean all of those Tristan Thompson cheating rumors too. If you have absolutely no idea what I'm on about, allow me to explain.  And yes, we do actually know her name but we don't need to get into all of that. It was also seen in another video frolicking in a nightclub with two other women seemingly also doing the kissy kissy touchy feely thing. This all came out days before Chloe was due to give birth. Since then, it's been speculated that they are at home with baby tree playing happy families. But we actually want to know all the goss and after a long wait, our girl Chloe says she's going to spill all even if Tristan isn't happy about it. According to a new report by Us Weekly, the new season is one not to be missed as we find out what really happened between the star and her bae. Well, I don't know about you but I'll be waiting for all the team. There's plenty of drama going on inside the villa but unfortunately there's also some going on outside the New Love Island arrival Zara McDermott. It's been reported that since joining the show, Zara has become the victim of revenge porn at the hands of an ex-boyfriend with intimate images and videos allegedly being shared of her across WhatsApp. While it is unknown if Zara is aware of what's going on, an insider tells the Mirror that the McDermott family are shocked and disappointed that anyone would breach her trust in this way. Revenge porn is when someone shares sexually explicit images or videos of another person without their consent, often with the aim of causing them distress. Unfortunately, once someone has sexually explicit images or videos of you, it's hard to control how they use them. For more info, go to www.victimsupport.org.uk


What's up everybody jazzy here from jazby's case breaks.com We're giving away 15 opulence basketball spots in this one pack filler of contenders fat pack basketball pack break number four So there you go 30 total spots everybody gets a random team the NBA will quickly distribute the hits out of the fat pack No member no veteran common ship, then at the end of the break We'll take the top 15 customer names in a randomizer or take all 30 customer names in randomizer and the top 15 Happy you guys will get a spot each in the break itself Would you be kind of next in a separate video? So let's roll the dice and do the break five and a three eight times. Good luck one two three four five And what was it eight six seven? Matt down to roberts eight times one two three four five six seven boom Aaron as DJ Calla would say All right, just a quick glance guys get a quick little filler So Danny with the Grizzlies Debra with the Lakers Pelican to a robber eat robber So you're gonna stick with what you got All right guys, so the spectra football fillers are up First one's already down to two left the next one's at nine. We can easily break that tonight guys.  I have no idea, man We do do a lot of guaranteed hit breaks on Instagram We do them like hit packs or hit drafts But they're like mystery boxes. I mean, that's just random whenever Nick just decides to whoop up something. It's never planned So maybe tomorrow maybe sometime this week All right guys, but if you're looking for like guaranteed hit breaks, we do a lot of those on Instagram All right, so let's go back We'll randomize the customer names again and the top 15 get a spot each in opulence roll it It's a five and a one six times. Good luck One two three four five and six the final time Oh Jeremy Robert Robert Danny Justin Danny Matt Nick Robert Danny Michael Shawn Matt Robert and Robert Congratulations. You are in Top 15 get in the break coming up next is opulence basketball


Welcome back to the FNA and today I want to expand on the how to shoot reference playlist with two surefire tips that will improve your reference game. I don't think I was able to incorporate more buzzwords or keywords in that intro, but today I do want to talk about two things that have helped me tremendously when I shoot reference and it was because I wanted to animate a shot at home and I was just not getting it right. It did not work at all, so I tricked myself into a specific performance and that's what I want to talk about today.  I do animation analysis clips, I do acting analysis clips, I do rig reviews, product reviews, it is the pitch at the beginning. I do a bunch of stuff on this channel so check it out, look around if you like it, subscribe and if you don't that's also okay maybe I'll convince you later but that's that for the pitch let's get to the reference. So what happened? I had to animate the shot for myself where I was running into into the frame and I had to look for something and one of the common mistakes in terms of not just reference but in general when a student animates, not just a student I do this myself. So it's probably more like a general thing is that you need to give your character enough time to process things. By that I mean when your character is in the scene and it's going to be a pantomime, it's going to be ellipsic as well. You have to make sure the thought process is there. You can't rush the character through the scene, through the objective, through whatever they have to go through. So let's say the character is seeing something, which for example I want to show later, you have to give the character time to process that information, react to that and then make choices based on whatever is going on that what they're seeing and what they're reacting to. That goes back to characters actually listening. So I have when you have ellipsing so that if two characters, you can just have one character is wait, wait, wait, and then let's go and deliver. They should be listening to the other character and reacting to scoopy physically to scoopy again, whatever ways, but you want to make sure that your reactions and that you're acting in general should be truthful and as if it was for the first time. And that's the key thing. So as an animator, you will do things over and over. You will analyze things frame by frame and you're going to get very familiar either with the action, with the ellipsing, you're going to loop that audio over and over and over. So it's tricky to get a performance that is fresh, that is new, that is something where you are reacting to something for the first time. So I was trying to run into the scene and look for something and I caught myself going basically through the motion. It was just kind of looking in, looking around and I'm blending in that example. It was just kind of walking into the shot and just looking around and that was it. It just didn't work. It was just, it was bad, it was just bad all the time. So what I did was I, so silly, but it has really helped me, took a piece of paper and I wrote something down that I had to read. So when I got into the scene and looked at the piece of paper, I was reading this and then I moved on to something else. So basically I was looking for something and I have to look into different areas of the room that I was running into. But if you just kind of pretend you will kind of look and look and look and it's not that pause where you actually do look, process this side yes or no and then go somewhere else. So I was tricking myself by writing something down on the piece of paper. So something short, obviously your reaction and your reading will be short. Write something down longer and then you're going to read this for a longer period of time and then you're going to move over.  You're not rushing through the motion. Now if you don't want to do that, you can also just take an object that you're trying to find and tell someone to hide it somewhere in the room. Obviously if you have to come in and you have to act this way, they got to hide it somewhere here and up back there. But if you come in and you don't know where that object is, you're going to actually spend the time looking, reacting, seeing it and then moving on to your next acting point. But at least it's more honest because you are trying to see it, you're trying to find it and it's not this kind of rehearse or over rehearse where you kind of just kind of move around. It doesn't have that eye lock and darting and processing and moving over to something else. So whatever you're doing, if you're realizing that this is just kind of stale and I'm doing this over and over and over, find ways to trick yourself. For me it was writing something down on a piece of paper, putting it somewhere and so it forces me to take the time to read and then to move over or hiding something in the room where I don't know where it is. Now your shot might not be physical where you have to look around, it could be something else. But it's that idea of maybe I will ask someone to do something or I will do something that would trick myself into taking enough time for the character to reprocess or do whatever. Because again you don't want your character to rush through things because they don't look alive. They don't have that amount of time it takes to see information, process it and then react. And I think that plus the combination of other things that I talked about like you don't want to pretend, it's a big thing in reference where you don't want to pretend to lift something light, you don't want to pretend to shout but then you're not. It's like when you have lip-sync and someone's yelling or you're acting like this and you don't have that energy of yelling that reference is not going to work and that's why you're shooting reference because you want to study it to see how your body moves and so on. So I think if you have a list of those type of things plus tricking yourself I think it's really going to improve the way you shoot reference, the way you act in front of the camera and all the information you get from that and all that combined you can put that into your shot and make it so much better.  So it's something fresh and born out of the moment if possible. Which is tricky if I animate this because we're not really the in-the-moment type people because we are frame by frame analyzing how long does it take to close the eye for a blink. So it's kind of this push and pull between you want to be spontaneous and do something quick and new and fresh and original while laboring over it frame by frame over and over and over. But that might just be me. I like to just shoot reference where it's something more in terms of something original in terms of a choice, a move or just a certain gesture or something that's character-based that works within the scene, the context and just who that character is.  I prefer to do all that later on. So when I shoot the reference, it's the basic construct of an original performance. That sounds super arrogant. I know it's not always, I'm not that good. But that's the goal, that's the hope that it will have that construct in terms of originality. And then later on I go and I push the poses, push the silhouette and all that stuff. Unless it's really cold forward and it's something very specific that I want to shoot. Obviously the way you shoot reference will vary depending on what you need and then and your style and so on and so on. I'm curious what you think. Let me know in the comments. Maybe you're doing the same thing or maybe you did that, post something else on top of that. But tricking myself is the way to go forward. Now I don't want to trick you into my workshop but if you feel like what I said is helpful, if you think that's cool, I want to try that and you want me to help you and make your shots even more awesome. I have workshop, it's the page at the end that you can sign up at anytime. Link in the description with all the information. And that is that for today's clip. Speaking of day to day time, thank you for watching this clip until the very end. I appreciate your patience as always that you do so and that's that. Thank you. And if that was helpful, maybe you want to subscribe and don't miss any of my uploads. And if not, that's also okay. Maybe we'll keep watching at a later time. But that's it from me. Thank you for watching and I'll see you in another FNA.