みなさんこんにちは。私の名前はヤスノリゴトです。私とランサンは、エネルギーのデブロクションのフォアフロントについてお話しします。ここはアジェンダーです。まず、エネルギーのデブロクションについてお話しします。エネルギーのデブロクションの基礎はエネルギーのデブロクションについてお話しします。そして、ファイルシステムタックの問題は、ダックスはダイレクトアクセスモードです。次に、ランサンは、エネルギーのデブロクションについてお話しします。ファイルシステムタックについてお話しします。エネルギーのデブロクションについてお話しします。では、結局についてお話しします。私の自分を紹介します。2002年から、レナックスとレナックスのオーエッセスについてお話ししました。レナックスとレナックスのオーエッセスのオーエッセスについてお話しします。今、フジツレナックスのデブロクションのリーダーを開催しました。数年間、NVDMについてお話ししました。NVDMの多くのインファンスメントをしています。例えば、フォルトロケーションやフォルトプレデクションを紹介します。レナックスの基本のNVDMを始めましょう。ここは、NVDMのキャラクターリスティックです。NVDMのパーシステントメモリーのデバイスはディーラムの中に入れています。NVDMについてお話ししましょう。でも、このデータのパーシステンシーはパーシステンシーの中に入れていません。でも、NVDMのディーラムのアクセスレーテンシーは、NVDMEのアクセスレーテンシーではなく、ディーラムのフュージャーのアメリーではなく、ディーラムの使用方法は、メモリーデータベース、ハイアラッキーカールストレス、ディステビュータストレス、 etc。そのように、最も有名なプロダクトは、データセンターパーシステントメモリーのモジュルです。NVDMのインパクトは、その普通のアイオレアがNVDMのリダンダントになります。まず、ページキャスがNVDMのリダンダントになります。スローアイオストレスが、NVDMは、まず、ページキャスではなく、次に、シンクシステムコールは、リダンダントになります。シンクシステントメモリーのプロダクトは、CPUフラスのインパクトは、それから、パスタンシーが足りない。オディッションでは、システムコールは、リダンダントになります。アプリケーションが、NVDMのリダンダントに立ち、リダンダントのリダンダントになります。その時、システムコールは、多分、カネルモードとユーザーモードに関するために、新しなインターフェイスを、NVDMに行われている。しかし、NVDMは、普通のソフトウェアでは、多くのソフトウェアが、その記憶は、丸く見えません。意味してください。あれは、NVDMのソフトウェアを使用し、まず、スタッフのパワーを準備する必要があります。GPU cacheは still volatileです。スタッフのパワーを突然にすると、そのデータは not be stored.次に、NVDMのデータストラクチャーのコンパチビリティを必要です。NVDMのデータストラクチャーはないのです。如果、スタクチャーが変わるとソフトアップデートはデータストラクチャーのためにオーディションで、コラスピングデータを担当する必要があります。データが壊れた場合、ソフトアップデートを担当する必要があります。最後に、データエリアを担当する必要があります。ソフトアップデートはフリーエリアではないのですがデータを利用する必要があります。オーディションで、カーネルマスターサインでオーソリティチェックでコンフリクションを担当する必要があります。ファイルシステムは使用しています。ファイルシステムは前回のコンフリクションでフォーマットコンパチビリティのファイルシステム、データコレクション、ジャナリングのコーピオンライト、レジョン・マネジメント、オーソリティチェック、 etc。カーネルマスターサインで使用しています。しかし、ファイルシステムは使用しています。と言うと、ファイルシステムはもっと重いです。そしてシープユキャッシュフラッシュは足りないのでファイルシステムに新たなアクセスを担当する必要があります。前回のコンフリクションでファイルシステムにアプリケーションを担当する必要があります。アプリケーションはIo interface like SSD or HDDです。アプリケーションはこのモードを使用できません。次はファイルシステムダックス。ページキャッシュはファイルシステムダックスを使用しています。アプリケーションはアクセスエヴィリムエリアを担当する必要があります。でも、ファイルシステムを担当する必要があります。次はIo XT4のアプリケーションを解決する必要があります。次はデバイスダックス。アプリケーションはアクセスエヴィリムエリアを担当する必要があります。デイスラッシュデブスラッダックスはアクセスエヴィリムエリアを担当する必要があります。では、リード・ライトを使用できません。デバイスを売り、新しいアプリケーションでアクセスエヴィリムエリアを詳細に、PMDKが協力されています。ファイルシステムダックスとデバイスダックスがアクセスエヴィリムエリアを知り、スラッシュデブスラッシュ第一のスライドアクセスエヴィリムエリアをFileSystemDux-dev-pmm is created and DeviceDux-dev-dux is created.NUDControlCommand can create this device when it creates namespace.Here is the example of FileSystemDux.NUDControlCreateNamespace command creates-dev-pmm0.You can make FileSystem on-dev-pmm-s or-dev-pmm.Please note DeviceDux is character device.Since you cannot use read or write for DeviceDux, you cannot use daily command for backup.As I said, PMDK is a set of libraries and tools for FileSystemDux and DeviceDux.It includes many libraries and language bindings and tools.It is for not only Linux, but also you can use PMDK on Windows.I would like to introduce some of them.The first is Live-pmm.It is a low-layer library.It calls a map to use NVIDIM and calls suitable CPU cache flash instruction and etc.Live-pmm object is a high-layer library which supports transaction of the object on Dux.It is for general use case and it's highly recommended library in PMDK,but users need to understand how to use its transaction.As I said, DeviceDux is character device.Then you cannot use daily for backup.So,Dux.io command is provided instead of it.I'd like to introduce two new libraries of PMDK.The first is Live-pmm.2.It is a new low-layer library.It introduces new concept, granularity.pmm.2 granularity page is for traditional SSD or HDD.pmm.2 granularity cache line is for persistent family.It is the case for process needs flash cache to make persistency.The third is interesting.pmm.2 granularity byte is for persistent family too,but it is for the case for platform support CPU cache persistency.It introduces new functions to get unsafe shutdown status and bad block.This library uses library of nd control command internally to get this information.Unfortunately,it's interface is different from old live.pmm.The second new library is liverpmm.It's new library for rdmm.The first library for rdmm is liverpmm.It's experimental status due to no user.Liverpmm is created with user's requirement.There's a difference between old liverpmm provides and customer's expectation.Here's a quote of its liverpmm presentation.Currently,official release of main library of this library is planned at2020.I need to say file system does is still experimental status.It is very expected interface.The management way of nvdm is almost same with traditional file system.Operator can use traditional command to manage nvdm area.If application call nmap for a file on file system docs,then it can access to nvdm directory.In contrast,device docs requires full management by tools of PMDK.Otherwise,a software need to possess whole of the namespace.However,the experimental messages is shown when the file system is mounted with docs option.Because there are some issues in kernel layer.So,experimental status has been for few years.So,in this talk,I would like to talk why it's experimental yet and what is the obstacles.Okay,let's start issues of file system docs.What is solved and what is current issues.In summary,there are two big reasons.First is that file system docs combines stress and memory characteristics.This causes corner case issues of file system docs.The second is a more additional feature was required,but it is or it was difficult to make it.The first issue is configure docs on and off for each iNode,like directory or file.Second is coexistence with copy and write file system.So,let me introduce corner case issues.Most important problem is updating metadata of the files.In file system docs,we expected that application can make persistent data with only cpcache flows as I said.However,this also means there is no chance to update metadata by kernel or file system.As a result,update time of the file may not be correct.If an application use write some data to file on the file system docs,and a user remove some blocks on the file by truncate system call,cunl cannot negotiate it.Data of the file may be lost.If data transferred by dma or rdma to the page which is allocated as file system docs,similar problem may occur.Here is the current status of this issue.For general write access,it was solved by introducing new map sync flags of mmap.When it is specified,page fault is occurred when write access,then kernel update metadata at the time.PMDK already specifies these flags.For rdma or dma data access,for kernel or driver layer,it's solved.It can wait truncate until finishing rdma.However,user process layer like infinite band or video card,it is not solved.Truncate cannot wait the completion of transfer because it may too long time.Fortunately,there is a walkaround.It's on demand paging.In OTP,usually driver or hardware does not map the pages of dma or rdma area for application.And it maps the pages when application access them.Then kernel or driver can coordinate metadata at this time.Melanox newer card has such future.Next issue is DAX on and off for each inode.Here is the expected use cases.The first is need more fine grain settings.Users may want to change the DAX mode depending on each file.The second is change DAX attribute by application.Configuration is always painful for administrator.Then,if application can detect and change it,it will be helpful for them.The third is performance tuning.Since the right latency of nvdm is a bit slower than run,user may want to use page cast by DAX off settings.Finally,it is good for workaround when file system DAX has a bug.So,what is the problem?If file system changes DAX mode attribute,file system need to change methods of file system between DAX and normal file.But they may be executed yet.Data of page cast must be moved silently when the DAX attribute becomes off.This problem was very difficult. Fortunately,it was solved.And how it was solved here?The DAX attribute is changed only when its inode cast is not loaded on memory.File system can load suitable methods for each attribute when it reloads inode to memory.So,page cast of the file are also dropped.User can use this feature with a new mount option,a mount-dux-inode.The DAX attribute is changed by this command,like,this is an xsx example.xsxio-change-atm plus x is DAX on,and minus x is for DAX off.Please note the following.All of applications which use the target file must close it to change the DAX attribute.File system will postpone changing the DAX attribute until dropping inode cast and page cast of the files.Currently,admin administrator may need to operate drop cast to achieve it yet,eco3vmdrop caches,because inode caches may remain due to race condition.However,this operation affects all of page caches and slabs of the system.So,we are trying to solve this problem.We made patches to evict inode and de-entry cast as soon as possible when the DAX attribute is changed.It said I don't cache flag and de-cache don't cache for it.Then,sync inode before eviting inode.The final issue is co-existing with copion-write file system like reflink dedupe of xsfs or butterfs.In this feature,if there is the same data block on different files,file system can merge it as same block.So far,if only file system managed such block,it was enough.But in file system DAX,it becomes not enough.The first problem is that memory management layer also needs to understand merged block.Reverse mapping for plural files are necessary.A merged block has only one abstract page,but there is no way to how to save offset of plural files.Very difficult.Second is need-enhanced IOMAP and file system DAX for copion-write.IOMAP is newer interface instead of block IOLayer.In other words,structured buffer head.It can lock and submit IO for plural pages at once.File system DAX depends on IOMAP feature,but it does not support copion-write.Though xsfs already use IOMAP,but IFES also needs to use it.Ransom will talk how to solve them.Ok,it's my turn to talk about how to solve the issues of file system DAX.I'll do it in two parts.First is how to support reflink slash dedupe for file system DAX.Second is how to improve the nvdm-based reverse mapping.Please allow me to introduce myself.My name is Ranshiyam.I'm a software engineer of Fuji's Nanda.I used to work in in-baptic system.Currently,I'm focusing on Linux file system and persistent memory.Here is the background of the issues we need to solve out.Currently,FSDAX is still in experimental statutes.It is because that reflink and FSDAX cannot work together.Here we try to use them together and see what will happen.Firstly,create a new xsfs file system with reflink feature enabled.And try to mount it with DAX option.Then,the error message will be shown like this.After little investigation,we found that the dmessage told us the reason.DAX on XFS is experimental.It cannot be used together with reflink.So,what are reflink and FSDAX?Why they cannot be worked together?I will explain in the next pages.What is reflink?It is a file system feature that files can share their extents for same data block.For example,copy file A to file B.The normal copy will take some time to duplicate data extents.Each file will have its own data extents.As a result,the larger file we copy,the longer time it will cost.But what will happen with the reflink copy?We execute copy command with a reflink because always flag.It will finished immediately.Because it won't duplicate any data extents for file B.Just map its data range to file A.So,it won't cost more time or occupy more disk space.Even,file is very large.As we key,reflink has two advantages.Fast copy and save storage.Since these two files are sharing data extents,what happens if we want to write some data to the file B?In order to prevent both files from being modified,we need a copy-on-write mechanism here.It copies the shared data extents before new data is written.This process is shown in the figure below.When we are going to write data on file B,the system will allocate new extents and copy data from theextent1 to the new extents.Then,write use data into it.At last,remap the new extents to file B.So,this isreflink.And what is FSDAX?It also called filesystemDAX.FSDAX is a mode of an open-dim namespace.In this mode,page cache will be removed from the IO path.It also allows mmap to establish direct mappingsto persistent memory media.So,why reflink and FSDAX cannot work together?We have invetiscated this problem in depth.We found two main issues.Firstly,we need to enhance IO map and FSDAXfor copy-on-write mechanism.The IO map interface needs to be extendedto support copy-on-write.The implementation of copy-on-write should be addedto the new FSDAX.Filesystem should also adapt to the new IO map interfaceand support coexisting of the two features.Secondly,memory management layer also needsto understand merged block.To achieve this,we need to improvethe current MDM based reserve mapping.I will explain how we solve these issuesin the next few moments.Let's start with the first issue.We need to enhance IO mapand FSDAX for copy-on-write.Let's take a look why we need to enhance it.Look at the flowchart on the right.The normal right will firstly load datafrom disk to page cache.Then write data on the page cacheand finally flush the page cache to the disk.This process implies a copy-on-write mechanism.But in FSDAX mode,as is mentioned before,page cache is removed from IO path.Data will be written without a copy.As a result,the data block in whichthe data is written will be incomplete.So,to solve this issue,what must be implemented?As we know,XFS uses IO map model.It implements IO map begin and IO map end.The FSDAX implements actual interface.Reflink is implanted by XFS.New extent will be allocatedin IO map begin.But it also needs to store source extentfor copy-on-write.In actual interface,the wholecopy-on-write mechanism needs to be added.copy source data from source extentto new extent allocatedin IO map beginand write user data in it.Finally,the new extent should be remappedto the file.Otherwise,it seems nothing is changedwhen we read the file to checkwhether the data have been written.In summary,we need to enhancethe current framework in these three levels.IoMap,FSDAX,and XFS.Let's start from IO map.IoMap model provides a structurenamed IO map to storethe distance where the data to be written.IoMap begin allocates the new extentand fill its properties to theIoMap structure.But it is not enoughfor copy-on-write mechanism.The copy-on-write also needsthe source extent info,such as the start block number,which means where to copy from,the length,which meanshow long to be copied,and maybe some others.So,we introduced a new typeof IO map to distinguishcopy-on-writewith normal write,and another structureIoMap called SourceMapto remember and passthe source extent infothrough the whole IoMapprogress.The code div is shown as below.After creatingthe new source map,what we need to do isto fill its members.The structureIoMapis always filledat the end of IoMapbegin.The structuresource map is only filledwhen it is share extent.If it contains real data,then store the extentinfo into the source mapand set the typeto be IoMap copy-on-write.In this way,the copy-on-writeis able to executein the writePASSand MAPPASS.Let's see the writePASS.First of all,the DAX driver providesan interface calledDirectAccessto translate the offsetinblock deviceto physical memory addressin persistent memory.With this feature,read or write datawill be easily replacedwith memoryCopy function.So in the writePASSwhen the IoMapCopy-on-write is setwhich indicatesits needCopy-on-write executed,we can use this interfaceto obtain theaddress of source extentfrom the source map.And thenmemoryCopy the source datato distant extent.After that,memoryCopythe user datato the distant extent.In the MAPPASSwe will get a virtualaddress by mmapping a file.Writingat the virtual address will causePageFort.FSDAX supports PTFortand PMDFort.ThePageFort handler usesIoMapModel as well.So,SourceMap for MAPPASSis similar withAlignate for writePASS.Just translatesourced address by direct accessfrom the source mapand memoryCopy source datato distant extent.The final stepis a bit different.Justassociate the page to theVMA.VAPPASSwill be done in userspace.Duplicateis another important mechanismfor reflink.It allows the existing filesto share the same data extentin order to save storage.It requiresa compare function.For normal mode,the generic dedupe functioncompare data in page cache.Butit is not suitable to FSDAXmode because of no page cache.So,we introduced a DEX compare functionfor FSDAX.Again,we use direct accessto get address of two dataextents and thenuse memoryCompare functionto compare whetherthey are same.If same,it means thatthe two files can share the data extent.However,we should pay attentionto check if the two filesare both enable FSDAXor neither.Files with differentDAX flags cannotbe dedupted.At last,we also need to remapthe new extent to the fileor clean upif error occurs.Till now,we areable to make reflinkand FSDAX work togetherin the right pathand a map path.However,it just looksfunctional on the surface.In depth,there is another issueneed to be fixed.We usually thinkof nvdim in FSDAXmode as a block device.So,we can seefiles share same data extentbecause of reflink.But,sinceit is nvdim,we also need to thinkof it as a memory device.In another word,files are sharing the samememory pages.So,the memory managementlayer also needs to understandthe shared block.As a memory device,nvdim may failin hardware level.That means the page is brokenand cannot beaccessed anymore.The system triggers memory failureto handle this.When memory failure occurs,the system will trackall processes associatingwith the broken pageand then send signalto kill those processes.The track from memory pageto a file is usuallycalled reverse mapping.In this case,we callit nvdim-basedreverse mapping.The current nvdim-basedreverse mapping can only supportone page to one file mapping.However,forreflink,because filesare sharing the same page,we need to improve itto support one pageto multiply files mapping.To solve thisproblem,I have thoughtof many ways.Here isthe approach to solve this.Each idea is based on the resultsof the previous one.In idea 1,I createda dexarmap arbitraryto store more file mappingswhich caused huge overhead.I reduced the overheadin idea 2by introducingstorage lostinterface.Furthermore,in idea 3,I removed the useless overheadcaused by storage lostand addinterface to support morethe sdx mode.Let's start from idea 1.To support oneto multiply mapping,thefirst thought is to store morefile mappings in one page.The old implementationuses pages mappingto store file mappingand pages indexto store its offset.It can only store onefile mapping.So,I introduceda dexarmap arbitraryto store morefile mappings.To save memory usage,weassociate file mappingand offset to pages mappingand pages indexat the first time.It's same as the oldimplementation.Because the page is sharedwith many files,theexarmap arbitrarywill be created at the second timeassociating.And insert file mappingand offsetas the trees noteat the second timeand later.The arbitrary's rootis stored in the unused memberpagespreventat the second time associating.It is todistinguishif a page is associatedonly once or more times.If it is twiceor more times,the page index is usedas reference count.After associating,it is ready to reversetrack in Memoraphilia.As is shown in the figureon the right,the Memoraphilia getsbroken page locked first.Then iterateFifthMapingsimmap to findVMAs of processes.Collect a list ofprocesses need to be killed.Thenunmap those page mappingsand kill processes.Finally,unlock the pageand exit.With theDexarmap arbitrary,we only need toadd an iterationof it outsidethe process tracking.This double iterationmake one tomultiplyreverse mapping possible.However,in some cases,each page may containoneDexarmap arbitrarywhich may cause huge overhead.It should not be underestimated.Also,this per-page tracking methodonly works for filebecause the associationonly supports file.Metdata cannot be handledif the Memoraphilia hits on it.To reduce the huge overheadwhile still being ableto build reverse mapping,we then look intothe file system feature.XFS has a featurecalledRmapBtreewhich recordsthe owners of each dataextent.The owner could beFile,Metdata,and other file system data.So,we introduceda file system interfacecalledStorageLostto return an owner listinstead oftheDexarmap arbitraryisto reduce the huge overhead.The interface is implementedby file system.Firstly,currytheRmapBtreefor ownersof the broken blockone by one.Secondly,returna list of ownersfor the memory failure use.The association could be very simple.Just store the five systemssuperblock in pages mappingas the specific colorwhenwe execute tracking.And storereference counts in pageszone device data.It is not usedin Memoraphilia case before.The track processis similar with previous.The difference iscurryStorageLostto create an owner's list.This list is usedfor tracking frompage to file mappings.Then,iterateeachfile mappingsimmap to track processes.The rest partis same asprevious.In this way,the huge overheadhas been reduced.However,itstill leaves a little overhead.We needextra memory spacefor the owner's list.It's better to remove the overheadcompletely.Instead of creating the owner's list,we handle every ownerduring the curing.We still call storageLostto curing ownersof the broken block.Since the file system curiesowners one by one,we can call MemoraphiliaHandler to handle each owner when it is found.Then,as usual,track owners'immap to findVMAs of processes.As a result,theextra memory spaceis not necessary and removed.The one to multiplyReverseMappingis implemented.Furthermore,there is room for improvement.Currently,Memoraphilia can only supportFSDAX mode.It is necessary to make ita common thought handler forNVDM.In order to support more thanFSDAX mode,wedive into driver level.We introduceda driver interface called Memoraphiliafor each kind ofNVDM device.The PMEM device isimplemented by callingStorageLost.The DAX device is going to befinished in the future.The one tomultiply ReverseMappinghas been implemented so far.There isno useless overheadwith Memoraphilia interface,it is compatiblefor all NVDM modes.With StorageLost interface,it is compatiblefor all five systems.Butcurrently,there are some remainingdifficulties.The first one is thatthe PMEM driver cannotobtain five systems fromLVM orOtherMAPT device.We can useGetSuper functionto get five systems whichis created directly onthe device.But it is not suitablefor MAPT device.Sothe other method needs to be found.The second one is thatStorageLost requiresfive systems has a featureRMAPT treeto track owners from block.For example,the realtime device of XFSdoesn't support it.So we cannot trackfor realtime devicefor now.And so does otherfive systems.Conclusion.We talkedabout the following.Basis of NVDM forrelax and issues ofFileSystem DAX.SupportRifling and Deliv for FSDAXand fix NVDMVest Reversed Mapping.Communityhas made many enhancementfor NVDM on Linux.And we have worked for NVDMto remove experimentalstatus of file system DAX.We expect itwill be achieved soon.Thank you very much for listening.


the Brightline Transformation Compass, a holistic tool to empower change makers inside any organization, keeping them focused on successful transformation. First, we define our North Star, a simple yet powerful purpose that every employee and stakeholder can get behind. Whatever happens on the way, the North Star shines constant on changing. Deep dive into the Brightline Transformation Compass.


including myself and if you haven't yet if you could yep there you go Craig you're being better than me and faster than me and I'm loving Joseph the okd4ga it's the most hopeful shirt I've seen all day so thank you for that and we'll figure out getting a panda logo on that t-shirt and when we go GA I'll at least get some stickers printed on the OSTAN's team we'll get them auntie up and print us some new stickers stickers so welcome everybody and welcome back from vacation Christian the map behind your head is that where you went on vacation actually yeah that's I didn't go too far on vacation actually so I live right here in Hamburg and I went up up to the coast of the North Sea for a week yeah awesome yeah my furthest trip so far has been from a ferry ride over the Vancouver and back so I'm totally jealous that you got a real vacation in well-played so I'm gonna share my screen and we're gonna I put the Craig put the list in there so you can add your name to that and we're gonna drive through today and see what we're doing I'll share my whole screen here just why not I think I have a clean thing yeah there's still some stuff so first why don't we kick it off with the okd4 update from Vadim and Christian where are we at all right I think I'll leave this one to Vadim as I wasn't here last week okay so yesterday we released better five I don't think it has any wait it has all the fixes merged to run a single master install you would need a beefing machine for that like 16 gigabytes of RAM and you still need a bootstrap node but other than that I give it a try on a few platforms and it seems to work so I would appreciate some tests you won't be able to upgrade and you won't be able to make host modifications using MCO but other than that it's fully functional more changes in better five there's been there's been just mostly a collection of all fixed merged and I think my connection is dropped and you can't hear me anymore right no we can hear you you're okay you're so good I just seen the screen pros and I was not sure was it my laptop or is it good so for other news we're working on rebasing okd on 4.5 codebase meaning you would get a vSphere API and more changes from which we're cooking in 4.5 and we're hoping we would be able to use upstream MCO thanks to work from Christian and we would use Fedora 32 as a basis meaning new kernel and more shiny stuff from Fedora project that's pretty much it I have created a milestone to track this data because we're not sure if we would be able to release it next Monday and I created a milestone to track the official GA release so that the t-shirt would be true there is still quite a quite a few large things tackle on the way but I'm pretty sure in a few weeks we would be able to have at least the approach that and would be able to finally go gee that's all I've got okay so I I saw a few things in the OpenShift dash dev a few people had been testing I believe but I don't think we've gotten any feedback yet so that I think is is there anyone this call that's that's tried deploying the beta the beta 5 silence silence is golden some days it's pretty mean you mean the okd beta 5 4.5 yeah thank you the okd beta 5 is what I think what we're officially calling it beta 5 no one's done that yet so yeah so there'll be some work over the weekend on your home labs and everything else everywhere else so give it a workout so hopefully we will get that done Joseph has helped a hugely getting the okd.io landing page up to snuff apparently there's a few more broken links or links that your question know all about did you I didn't see any PR requests on that I think you were hesitating about where to point them looking in the chat and this is a couple of dead links or at least very old links yes probably so if you point them out to me I'll either remove them or find a more relevant one and hopefully we'll get there he's also been working on updating the logo for so it has a an actual four on the camera on the side so stay tuned for a new okd panda image to go out with four hopefully that's good so cool let's see what else CRC char his Charo on the call we him any update on the single node installer has anyone else besides Charo tried it out don't see Charo this time so okay snout all right and I'm gonna stop sharing for a minute because my screen is going a little wonky there was some talk of using GPUs and okd for is that an update that we should get today Vadim or is that for a future topic that's something for the future I should contact Zonko again because of their schedules we probably lost him okay so the only other thing that I have is we we moved yes yesterday what we were supposed to do an okd working group AMA session on and we've moved it to I think it's June 1st next Monday as part of the briefing series that I do at I guess it's 1600 UTC or 1700 UTC just the hour before this and I'll send a note out if people want to join that just to try and drive some more interest in an okd for and answer any questions that are out there there have been a lot of questions that I've been noticing and people participating in the Kubernetes OpenShift dev channel from a number of different spaces so there seems to be a quite a few people who are testing and working with okd for but not participating in the working group so I'd like to to get some of those folks enrolled and engaged in the community conversations here so I want to open it up now if people have other things that they want to talk about today that's what I had on my my list of things that I should make sure I check off today awful quiet here let's see what's going on glades on here can you hear me yep you're fine yep cool cool I just wanted to share that for the past weekend I've been testing okd or on vSphere trying to use IPI not your PI and I haven't been successful the Fedora machine that I got from the beta 4.4 it's not booting up I get some some weight error on the kernel what during the initialization and do some investigation and realize that there are a few bug zealots opened for for I think it's a for CoroS actually CoroS issues not Fedora issues per se so I just read in the chat that IPI might come off in 4.5 is that specific places should look for nightlies in order to test IPI we don't have nightlies based on a 4.5 yet but we could create rather I could share a testing build I'm using before we sort out the branches and the the commits we want to use because we would have to configure additional CI configuration we would need some more things on the infrared to make those work and have it tested for vSphere because that's apparently a huge platform so we should test it better than we used to do assembling official nightlies would take some time but we could share something we have right now and hopefully that will work if it won't and we would find a problem if it or CoroS we would have to wait for official nightly so that we could have something to share with our engineers and that would be something more complex than just some random things I built over the weekend but yes giving it a preview for the nightly definitely possible yes Joseph is asking does it make sense to list the PRs that are necessary for GA so we can count them down so I'd say that there's still two things here so we want to get everything merged into master but for the 4.5 release timeframe we will still be building off of a branch so I think the more important ones are getting that into master but on the other hand that doesn't necessarily block GA we just really want to get the certainty that it's eventually going to merge but before we call it a GA so it's yeah it's only well actually Vadim made a good great list of things we still have to get done before GA on the okd repository like a tracker bug I think that's probably the best place to track this yeah there's going to be individual PRs not all of them are open yet so yeah can you share the link to where that is everybody made that now or maybe just share your screen and share the link to at the same time so we can see that I'm always there's so many places too many places yeah one second I'll just send the link so it's essentially two issues on the okd repository this one's for MCO and the second one is for the installer this one is the installer and what we also want to get done before GA is replacing all the ironic images with RDO yeah with RDO courses so and then we'll probably have to have builds of these containers in an open-shift CI as well in Pro RDO is the well the upstream for the reted open stack so it's an open stack distribution and that's what we use to run everything on the that's sort of the base for the open shift open stack platform this may be a short meeting because you guys aren't as talkative as you usually all are RDO for ironic bare metal yes so the only other thing besides these getting to GA which is sort of the end all and be all of everything in my world at least and I'll add these four to here is the Kubecon North America is has their call for papers out and I would love to see a talk submitted by maybe some of us here to talk about building an opinionated Kubernetes distro there are lots of opinionated things out there so I'm wondering if anyone's game to do that submission it'll probably more than likely be a virtual event but it'll still be be an interesting one so if you're game for that let me know and we can collaborate on it and get it in in in a timely fashion our Christian and Vadim searching for a new project if OKD is released oh no you're not searching there will be no we'll stick around I think yeah at least I will yeah there will be I don't think we'll ever let Vadim or Christian off the hook on this one just because it hits GA it's still going to keep in sync with OCP's release cycles and the new features and functions so this this project is going to be ongoing on no Vadim for CEO I unfortunately Paul got the role but you know there's always a possibility that Paul Cormier will retire and we can you know put Vadim up for CEO you don't want me in the CEO in my world is a cluster it's the operator like and I stick to that yes there you go all right so anyone else got stuff that we we need to to get done today I know this this is shockingly quick for an OKD meeting I can't believe it must be is Neil not on you know I don't want to cut you off at that at the but if anyone wants to to work on the a draft of a call for papers for a coupon just reach out to me and I will help craft it and hopefully get it get it accepted and we can go on and we might actually I've been trying to get a white paper done sort of categorizing Kubernetes distributions with a couple of other folks so I'm looking to see about getting some more visibility for OKD out there in the world once it's GA so anyone else and there is always that nebulous fedora magazine article that never seems to get rid written either with our all of our spare time so that could be another task coming forward so with that I can't believe you guys are this quiet today I can answer I can answer a question Joseph brought up earlier just to add more conversation here so just if you'd asked about if the auto scaler would work with these fear once we got everything together so just as a as a blanket of statement in OpenShift the cluster auto scaler uses our version of cluster API which we call machine API so anything that the machine API operator is able to control so vSphere AWS any of those providers we have the cluster auto scaler actually speaks to the machine API so the auto scaler you know it can do anything that the machine API providers can do so any provider that is available through the machine API the auto scaler should be able to scale it also in also in version 4.4 yes yeah okay okay we didn't try that out okay didn't know that thank you yeah if you look at the way we deploy the cluster auto scaler we're using in for this should be in 4.4 we use a provider called OpenShift Machine API that's the name for the provider we give and that you know so the auto scaler is just creating like machine sets and machine deployments it's modifying those so it's not actually speaking directly to the provider the machine API is speaking to the provider so any any provider implementation we have that would work through IPI or through UPI I suppose any provider implementation that machine API can talk to cluster auto scaler should work with those providers you know there might be there might be some minor feature gaps but the general behavior of scale up scale down you should be able to do okay we'll try that out maybe tomorrow good to hear. Gleason here again do we need to do anything special to get the auto scaler working or is it set up as part of the installation?  You get a link to the documentation on that and is that something that we should create a short video on or something? Yeah let me yeah I can grab the upstream documentation and link to it and we have we have an FAQ and of some of the stuff let me let me take a minute here to grab a couple docs. You can do that that would be great I have one more thing because there's always one more thing and just when you think you're gonna get off early I did a talk the other day with Albin Crickey from Kinfolk IO and the folks who have flat car along with a whole lot of inspector gadget and other kind of tool tooling and things and I was curious if anyone had looked at flat car at all in terms of okd4 and what what this if that was a possibility of something to collaborate with them on has anyone even heard of flat car okay? Flat car is basically a continuation of container Linux. Literally rebuilds container Linux. It has ignition. Did ignition 3 or is it still 2? I'm not entirely sure if they have ignition 3 then we should be ready to go. There will be an interesting experiment in fact we should have some kind of a board for these kind of experiments we want to do with okd4. I think I would love to play with starting with GPUs and replacing the fedora grower with something different but there is no place for us to coordinate on those efforts. We could add a column on the community board and and they weren't they weren't thinking about it it was just something came up in a conversation so and I don't I haven't heard of anybody who wants it so it's yeah looks like it's spec 3 but I'm just I'd be curious so maybe on the community board we have a column of areas for experimentation and just to keep track of that in case that so we can add something there so thanks that that was great and thank you for those do we need to do a little demo on on doing this auto scaling stuff it sounds like it's something that some people are asking a lot of questions about I just I can certainly do a demo next week on it I mean I I probably would just use OCP because that's what I'm right now but like it works the same on it. Let me just see if I can we don't have a meeting next week I'm just looking at my calendar next week maybe even just do a little briefing on it and give me two seconds to get there something to add to the conversations that a lot of the traffic we see in the Opyshift dev channel or the Kubernetes is lags that most of the people that are trying OKT they went through the the user provided infrastructure side so if if you would be able to use in your demo you know examples from user provided that would be very helpful for the community. Right I'll take some time to see if I can get OKD up and running if I can use vSphere I will but I'd you know I don't have a vSphere account right now so the main things available to me are like AWS and GCP are what I work on most of the time so if I can just do like an OKD installed the AWS that would probably be the easiest for me but the mechanics of what I'm going to show they won't it won't matter whether it's running on vSphere or AWS or GCP because none of those details will bubble into the things that I'm gonna I'm just gonna be using the resources that you know the operators look for basically. So how about because I have I have a slot on Wednesday the 3rd of June at my 9 a.m. you I think you're your east coast aren't you noon I actually think I because this is a reoccurring thing people asking this question there if you're asking in the working group I'm sure there's people out there asking as well so it might be let's let's aim for that the 3rd of June and we'll do it seems it seems like a good good topic and is that like is that like an open shift commas or something I I'm on I'm online now four days a week I found someone to do one of the days Tuesdays for me doing briefings from on all all kinds of topics by commons members and different working groups and Monday's the plan is to always do an AMA with one of the upstream projects so you guys are the guinea pig for next Monday at 9 a.m. and I've invited that EAM and Christian and I can I'll post it on the the working group session too and we can post it if we can on open shift dev and it's just you know I'm gonna get that EAM and Christian to give a quick update on what OKD is where we're at with the release cycle right now and point to all the right resources and then we'll see you know we may be the same group of people all talking to each other again who knows who will come but I think it would be pretty good and I don't know maybe we do depending on how long your auto-scaler demo is maybe we do that I was thinking was gonna try and get who did the single cluster demo to that was was that Joseph or that was Charles so from Old Dominion to see if they would join so maybe doing a couple of little demos I don't know how long it takes you to do an auto-scaler demo and a single cluster node thing but just I have I can go as long as I want on Monday we could go two hours if we need to and maybe that's what we do on Monday is have the AMA do a quick here here you do auto-scaling and here you know here's how you do it on single cluster just to have things to talk about too if that sounds reasonable for folks and then I can clip it all into the little videos that I like for my YouTube keeping and so people don't have to watch the whole two hours and listen to me blather on in between when you guys go silent on me it sounds good to me like yeah send me send me some info I would probably want like 10 to 15 minutes for mine because it just getting the auto-scaler working is one thing but then to show it scaling out and scaling it back in can take it could take four or five minutes because you need to wait for them yeah but yeah like I would probably just do that I would show people you know we'll we'll turn the auto-scaler on we'll make a workload and force it to scale up we'll watch it scale up then we'll delete the work node and we'll watch it scale down you know that should be enough and then we could do like Q&A or what people yeah how's that sound what is that could you delete the VM and vSphere in an event where people ask yeah yeah I think it's a different format for for doing meetings it's not bound to a an agenda like this one yeah and we yeah last time we had this free flow discussion and we thought we do need a forum for that as well and the AMA I think is very it's very well suited to do that but well here we should probably more focus on the agenda and you know get get things really done yeah and the world's biggest dog just walked behind you unbelievable what is it iris wolfhound or something he's a great day in Canada we have oh go Pogo or the Loch Ness monster just went behind like what's that yeah totally so I just wanted to respond a little bit to some of the things that Joseph is saying here so you say like you want to see machine deletion and then the auto-scaler kind of fill in actually that doesn't even take the auto-scaler the machine API will do that for you so like as it stands currently if you do not scale down your machine set but you were to delete a machine not the node please don't delete nodes that that disrupts things but if you delete the machine object then the then the machine API will actually take care of removing that and then creating a new one for you so like the auto-scaler to exercise the auto-scaler we oftentimes look more at creating workload in the cluster because what the auto-scaler wants to look for it doesn't look to see if the size of those groups have changed what it looks to see is are there underutilized nodes meaning nodes that could be reaped from the cluster and are there unscheduled pods and if there are unscheduled pods and there's availability it will increase the size of the cluster otherwise it won't but just doing things like deleting machines and watching it fill back in that's actually the machine API that's doing that for you not the auto-scaler okay maybe can use a very small VMs so it would see an effect maybe faster with single single parts if we deploy and delete right right so what I will grow what I'll probably what I do in these situations is like I'll probably turn the auto-scaler down so it only looks for like a very short time to respond and then yeah we make workloads that basically have such large memory requests that we can only fit one on a node and then it's really easy to tell the auto-scaler like okay I want to rep you know make 10 replicas of this workload it should make 10 machines to fill it out so what we'll do something like that probably okay okay great idea all right so I will add you to the the call on Monday so you'll get an invite and I will post a note on the mailing list with the details for that funnily enough it's exactly the same number that you've dialed in for blue jeans here today so you don't have to worry too much about that but that's that's how this whole thing works is there anything else people would like to cover off today yes because my idea of the countdown page because I don't I think we should not underestimate the importance of this GA thing I would say because yeah the time frame was so long people are waiting a lot of that I think we should yeah we should we could do something to yeah make it more interesting to yeah to gain more awareness that there is something cool coming in the pipeline and yeah don't know over PR counter is the right thing here or if we find something something better but I think it's worse yeah if I can jump in here go ahead they can jump in here real quick so I would really like to set a date once we're able to do that so I think we're getting very close to to actually being able to do that but right now there's a few more things that need to happen internally the planning and then we can really commit to a date I think we're getting close but I would like to push that conversation out at least till next meeting and then see where we were at at that time and yeah yeah but I hope you understand I agree we should we should definitely make that a marketing yeah we should market that we yeah yeah definitely because because other projects celebrate every every single change log entry yeah remember Diana remembers a Twitter talk yesterday yeah that that's what's top of mind for me so I think we have a little bit of time so maybe between getting a logo and a launch and we can make this celebrate this in a couple of ways it probably go back to we can do a blog post about it on OpenShift.com I'm happy to do that and write that up and I'm get the word out that way I don't think you're gonna see red hat do any big press release or announcement about it it's gonna have to be totally community driven because it does and compete with OCP and all of the other off product offerings so we kind of play under the radar from corporate marketing so what can happen I can make that happen and then we'll just I'll use my platform on OpenShift Commons briefings we can host an OpenShift Commons briefing announcing the GA and we can start the cadence of OKD working group AMA session so we did an OKD for AMA next week so do we have it it sounds to me like this is three to four weeks off if I'm a betting person or is it sooner than that Christian I would say four to six weeks part of eight weeks less than eight more than a month so I have time to write some articles and I will just bake it in there and we can Joseph and I have time to do that I think the countdown is a good thing but it's also and let me not misspeak it's it's 4.5 OCP that we're waiting for to drop is that correct or is it 4.6 so yeah we'll be merging all the changes in the 4.6 development timeframe which is open now and then for 4.5 we will have a separate branch so we will be basing off of the 4.5 release branch but then add a few additional commits on top of that to enable everything to work with IgnitionSpec3 and Fedora CoroS from the 4.6 release on it'll be built off of the same branches of the release 4.x branch just like OCP that was our our goal for GA and now we sort of to not have to wait for that will be releasing GA probably off of the F cost 4.5 branch but we want to be sure that all the changes will then land in 4.6 so we don't regress in any way from 4.5 to 4.6 because once we go GA we can't just you know there is this promise of stability then so we really have to be sure that all the changes will land in that timeframe but we can say this is GA. Okay wow well that was good and I'm glad I recorded that because it was complicated and yeah no tape packs on GA and yeah try and say that in a tweet okay and so I'm gonna try all of this is complex and difficult and a little bit annoying at times but yeah I know 140 yeah so that's I think that's the other thing the other thing that I was thinking about too is that maybe I should try and see if I could find a Twitter if okd.io was a Twitter handle that I could steal to start tweeting out some stuff too but let me see if I can get that done so with all of that anything else we should be there's nothing else in the chat here that I see that I missed. Is the sample operator also blocking thing or because I think there is some progress going on there? It would block the upgrades but that's a bug once we have one of the biggest hurdles to GA is official enhancement merge once we get all the org to agree that yes we want to okd4 yes we want to the way we have outlined and we're working on we will be able to follow box for the samples operator until then it's like semi-official and they are not like bound by this. I just lost your voice Vadim I don't know what that last one is so once we merge the enhancement that would be a bug and they will fix it in 4-6 time frame we will be able to provide a pull request. Is there a PR for this in enhancement somewhere to follow the progress? Yes chat doesn't work for me but I think Christian would they be able to find it because he filed that enhancement. I just put it in the in the chat. Yeah I think it's not too far off we just need buy-in from from those teams that are actually concerned with maintaining what we propose and outline there yeah I think it'll go through and we'll we'll have that very soon to merge and yeah then we can really look at implementing everything I mean for most the things we have done the code already samples operators certainly the thing we have to look at fixing actually and changing up a few things but we need you know before we put in any effort there we just need to get to clear the way to agree on a way to do this. I would very appreciate to hear in some and not today but in a later meeting of how the relationship between OCP and OCD is because I always thought OCD is something to get used to OCP and if you have enough traffic on your OCD cluster some you can switch over to OCP because this is a set what we plan to do and I think lots of yeah that's definitely the thing we want to enable and so it's a sibling distribution we've been calling it because it's it's the same code base but then it has a different base operating system and with the with the base operating system there is a upstream downstream relationship but it is not in the actual cluster code so the cluster code is the same which is why yeah it's a sibling distribution and then in the future of course you'll be able to upgrade from OCD to OCP with the commands you've been using to upgrade from one beta to the next and just switch out the base image and you know become a customer that way definitely yeah there is a company which does exactly that every single day they pick the latest nightly and try to upgrade to OCP all those upgrades have failed so far they're muted again you're still muted so you left us with all your back all of the upgrades have failed so far yeah because of the ignition version and compatibility but once they would pass I will definitely notify the workgroup meeting because we see those upgrades failing and passing on our dashboard internally using telemetry that's another incredibly interesting experiment to play with and we would have to confirm with legal and support organization that your cluster would be supported after you do that because there is a chance that it won't be yeah I'm sure that there will be limitations on on the you know on the workloads we can migrate but if it's you know if there's sort of a supported product or operator on the OCP side and you do that beforehand on OKD you'll be able to transition that workload into OCP I mean that I think legal is not gonna be against that I mean I can't really speak to that but it just makes sense for Red Hat you know as a company so I think that's in the pipeline and we want to want to enable that so people can try out stuff on OKD and then you know migrate that cluster to become an OCP cluster sure that's a great business model it's absolutely fantastic if it works all right then I'm multitasking as everyone can probably tell and trying to update a road to GA column there in the community page with the list of the open issues and links to them so if I did if I missed something please add it there Christian and Vadim I think there's one more but I was in the thread here today so anything else we should capture today if not hang me on the Kubernetes Slack if you're interested in collaborating on the submission for KubeCon North America and we can write something up and let me know I'll reach out to Cheryl and see how long his talk on the single cluster install would be for Monday and if he's available to do that too and Mike all I will add you to the BlueJeans invite so you have an official one in your inbox all right I'll let you all go and you will get all of 15 minutes back in your day so quick run test beta 5 and we'll see you mostly in the Kubernetes Slack there I think that's where we're all living these days so take care thanks to everyone bye


The King of the Ring is an attempt to bring Marines together, have a little fun day, have a little unit cohesion. I was expecting to win the grappling portion of it. I was not very confident in the night fighting every time, but it was fun, it was a good time. It goes back to train, you know, like to fight. If I train and keep fighting and I'm able to do this here again in Garrison, then when it comes time to do this in battle, then I already know what my body's already used to it. I already have that muscle memory going on. I know somewhat what to expect and how to overcome it. I feel like Marines, a lot of time, forget what we are all about. We get into this corporate mindset with computers and offices and stuff, and we forget that we are in a military institution and we are warriors at the end of it, so we should always be ready to respond to whatever happens to the worst situation, right? So if you know a Marine is good to go and he would be able to react and defend himself and do what he's got to do in the worst possible situation, at the end of the day that's how it is.


Okay, thank you. Well, I'm also going to talk about early RT initiation, but in a different context in South Sudan. I'm going to present the implementation of our Test and Deed program. Well, South Sudan, just for an introduction to the context, South Sudan got independent in 2011, and since December 2013, a civil war has been going on with an important ethnic component. The region most affected by the conflict is the northern eastern part of the country, but in 2015, classes spread to Western Equatorial Region, which is in the southern part of the country affecting locations like Yambio, Mundri and Esso. This region had been a relatively peaceful area until last year. MSF, Operational Centre Barcelona, is present in Upper Nile region, where they have projects taking care of internally displaced people, and also in Western Equatorial in Yambio, where we were supporting the district hospital until last year, and this is where we started our Test and Deed program. Regarding HIV situation, South Sudan is a country with a low prevalence of HIV, 2.7%, but in some regions, this prevalence is higher, especially in Western Equatorial, which has the highest prevalence in the country, almost 7%. We find very centralized HIV-canceling and testing services, as well as access to antiretroviral therapy, which is only available at district hospital level, so we can already say that reaching 1990-90 in this context is a big challenge. That's why we came with this idea of project, to try to bring HIV-canceling and testing and art closer to the communities, also implementing an early ART initiation, because we want to assess the feasibility and acceptability of this intervention in this context. So the location we choose for our project is Western Equatorial former state, including rural areas of Yambio County, including three payams. These payams are the second lowest administrative division in South Sudan, with a total population estimated at almost 50,000 people. To develop this project, we carry out different activities. We basically organize mobile clinics around all these rural areas in Yambio County to offer counseling and testing to the population reaching our clinics. And when we find a positive patient, we offer the possibility to start art that same day. Most of the patients that have been initiated, that was same day initiation, but it's true that we have a few patients that decided to come back later some weeks or even months. And then we arrange with our patients to organize mobile clinics, not necessarily in the location they were tested, but still in the community, to do the follow-up and the drug refill. We have an important component of health education and mobilization with community health workers to make the population aware of our services and to spread health education messages. Also, since we are in a stable context, we have a contingency plan. This contingency plan consists on different activities that are based on different levels of security. The levels of security come with the, for all the projects in the country, regarding different criteria or observations. At level one of security would be like the normal situation. We already give our patients a phone number they can call in case they run out of drugs or they have any problem. And already in our pharmacy we keep our contingency stock. Level two of security, we might find already some areas with restriction of movements. So we would move with our community health workers to refill the drugs of these patients. And also we give an extra month to our patients to keep us a security buffer. Level three, there is already high tension. So there can be an open armed conflict. We cannot run our normal activities because of security, but we still need to assure the treatment of our patients. So we would organize distribution of runaway bags containing three months of ARVs, also through our community health workers. And at level four and level five, the security situation is very dangerous and an evacuation might be planned. So at this time we just, we can just try to donate our drugs to the district hospital and had over files and register. Today I'm going to present our, the data from our program for the first six months. We just started in July last year. So from July to December 2015, I'm going to present some data regarding acceptability of the services based on the update on the HIV counseling and testing and the ART initiation. Some data about our patient outcomes, how many patients were remaining care at the end of the period. Also we are collecting data from the lab, CD4 count and viral load at baseline and follow up after ART initiation. Today I will just show the CD4 count at baseline. And our study was approved by the MSF and the South Sudan Ministry of Health Ethics review boards. Well during these first six months we tested 5,262 people with a positivity rate of 3.4 percent and we initiated 146 patients, 82 percent of these positive patients. Out of them, 79 had a CD4 count less than 500 cells. At the end of the period in December we still had in care 140 patients. We had registered three laws to follow up, three transfer to other clinics and no deaths. Unfortunately we also had the chance to implement our contingency plan. In September 2015 the conflict started in the southern part of Yambio where a fight between local militia and the army from the government started and this area was declared a no-go area because of security. We had already initiated 21 patients and we kept them on follow up with the help of our community health worker. We kept 19 of them. Yes. Then in December 2015 there was intense gun shooting in Yambio during a few days. We stopped our activities and we passed to level three of security and we needed to organize distribution of runaway bags thinking of how possible evacuation. We prioritized the patients need and refill at that time so we distributed runaway bags to 25 of 29 patients with appointment that week which is the 86 percent. Again we restarted our activities. We came back to level two of security but again at the end of the month again classes in two locations to the west of Yambio interrupted our activities and we needed to follow up some patients. We organized distribution to 51 or 56 patients need and refill again with our community health workers. Just as conclusions we find that there is a high level of acceptance of this intervention in the community regarding the HIV-canceling and testing and artinization. We could say that with this strategy we're helping to bridge the gap of the first two 90s. It's true that we need a contingency plan in this context and we're quite happy with the outcomes of the implementation but security situation is still a big limitation because there is not only we cannot reach those locations but also there is the displacement, there is displacement, it can be more difficult to find our patients etc. We had a quite good number of patients remaining at the end of the period but it's a short period and we still need longer evaluation including data about biological suppression to come up with some guides some ideas that could help for further MSF interventions focused on conflict settings with weak or non-assisted HIV services. I just want to thank to all the people that make a test and treat possible especially to the to the staff on the field and the communities of Yambira.