Good evening everyone, welcome to this Journal Club meeting. Today's topic will be taken by Dr. Amai Kulkarni. He is a staff radiologist at Joravinsky Hospital and Cancer Centre at McMaster University in Canada. So as you know, in a field like radiology, there are constant updates and technological advancements and newer reporting systems and data analysis and that kind of thing. And with our busy schedules as doctors, it is very difficult to keep up with this kind of technological advancement. But because for this, we have the Journal Club meeting, we have it every month. Today's topic will be combining the strengths of radiologists and AI for breast cancer screening. The publication is on the RSNE website. I've posted the link in the chat box for if anyone wants to go through it once during the lecture. So if I can hand over to you. Yeah, sure. Thank you so much for the introduction Abhishek. First of all, thank you for Indian radiologists to give me this opportunity to talk about this important topic here. So as Abhishek mentioned right now, Journal Clubs is a really good opportunity for brainstorming some of the ideas going over some of the published literature and then trying to make an informed decision about what you feel about that paper. As we all know, artificial intelligence is the big buzzword, no matter where you go in whatever field of medicine or even out of medicine. It has just become a really, really important aspect of our everyday lives, including use of chat GPT. So I'm going to jump into my presentation. So just Abhishek, would you just let me know if you can see the screen share. Perfect. So as we all know, the talk is essentially to talk about this journal, but I thought that I will take this opportunity to ask also briefly discuss about what is artificial intelligence how we are what are the basic things that we need to know when we're reading any AI related paper or any or getting involved in any area that talks with our colleagues. So we will basically go over that then I'm going to use something called as a claim checklist and I think Abhishek has provided a link for that claim checklist in the chat. The claim checklist is nothing but a checklist which which you can use to just go through when you're reading any AI paper to make an informed decision about what is the quality of that AI research. So you can go through that the journal article itself is from the Lancet. It's an open source article so you can download both the articles. Let's start with some questions so if you don't mind. Can you please pull up the polling questions if you can but if you don't have polling I don't think we have too many participants live on this portal right now. So you can even use chat for answering it's up to you. So we can go over the first question so you can, you can go through the polling questions there so do you read any AI related AI related papers that you can. Give me a second it's just causing something on my screen. Okay so question number one is do you read any artificial intelligence papers as of now, so you can just say yes or no, or not frequently, I've given you three choices. So I can see that some people are answering the first question the second question is you can already see, can you evaluate your people with the tool as you already know we are going to discuss with that. And then the third question will be, what is more important when it comes to evaluating any of these AI papers do you want to know more about data sources, or description the model is more important, or detailed training process how they train the model is more important. So maybe I'll give you a few more seconds and then we'll move on. Alright, so it doesn't matter so we can we can go ahead, but we are going to come back to these questions at the end again, just to briefly discuss of what we learned. So thanks, thanks for getting the poll out. Okay, so we can hide the poll. Alright, so let's go into the topic so disclosures. I personally do not have any relevant financial disclosures. This general presentation, but I have all have listed the disclosures from the authors who presented or published this work. So what is the problem why do we need all these AI related papers or AI softwares, especially when it comes to the world of breast cancer screening. As we all know that there are, like in any screening program, there are true positives, which are going to be the exams that are going to call back and they're going to turn out to be pressed cancers. Then they're going they're going to be false positives. These are the ones which are when you call something back on a screening mammogram, and it does not turn out to be anything. Then true negatives are that you read a mammogram as a virus one or two, and after 24 months or 36 months they came back for the repeat screens and there was nothing that developed on the mammogram so which means that you did not miss the cancer. Then what are false negatives so false negatives are the mammograms which are read as pirates one and two when you're doing a screening study, but they essentially have a small canner cancer which could be missed. And with any screening program, this is always a risk that it might happen that you have some of these false negatives, you might have a bunch of false positives, and false positives are nothing but a majority of the call back rates which is usually between 5 to 12% depending on whether it was a baseline screen what was the category of density of breasts. These are essentially are somewhere around four per thousand for most of the screening programs. So we want to see if we can develop something which can detect more cancers amongst these false negative group. So for example, if you read a mammogram, gave it as buyers one or two, you said that the study is negative, but there was actually a small cancer that was missed. So we want tools to capture those things and this especially becomes more important in category C or D density breast where detecting breast cancers can be a bit of a challenge. Then we also want to develop tools which will help us minimize the positive so which means that we call something back, but it does not end up being anything so that this is a quite anxiety provoking for some of the women who undergo breast cancer screening that the moment they get a call from the screening center. Oh, by the way, our radiologist wants you to come back to our center, they have this flight of thoughts like is it a cancer, how big is it will I survive for five years 10 years so it can be anxiety provoking for quite a few patients so we want to develop something which can also minimize some of these false positives. These are the current options that we have. So as we all know some of the mammography softwares that come that come with something called as a cat, which is nothing but computer aided detection. So, all the previous generation cats they would detect an abnormality on any of the screens, it will put squares and down kind of markers for either calcifications or a mass or asymmetry. There's another version that we have, which most of the industry leaders have been calling as AI cat there are multiple different names for that. It does not only detect the abnormality but it also tries to give you a score by some classification to really like an x or z score which will say it is 58% chance that it is valid or something like that. Some of the newer AI cat techniques can also pick up a slice of tomosynthesis stack that this is the slice where the abnormalities present. So we are quite far in using these cats cat tools as of now. The other options that we have is something called as peer review. So you read a bunch of screens and you put some of those screens as for double read for one of your colleagues. Now this type of this type of double reading or peer review is widely used in many, many parts of Europe. So even at our place we do tend to do a little bit of peer review, but it's not completely randomized we usually like to limit it to things like for example a single view asymmetry which we don't feel confident to call back, we put those cases for double reading and then we make a consensus or conference opinion. We can either design conferences around to do those kinds of things or design a workflow where a random portion of all the screens go for double read. Or the third option is you could potentially explore the role of artificial intelligence as your peer reviewer. So we are going to discuss about that in this paper. But before we do that, what is artificial intelligence many people think that just a big blocks, big black box something goes in something comes out, and then we have no idea what it is so for that we need to go back into history So the person who changed it for us is this individual. He designed a AI model involving convolution neural networks with multiple layers of the networks built within it. And it was called as Alex net and it won a large scale visual recognition challenge in 2012 and that is when everything started to revolutionize in the world of radiology for us. Does it always work. I mean, we never know. I mean, if we all remember the launch of this iPhone in 2012. So Craig was trying to, you know, show how the face detection works on the iPhone and it failed on live stage.  It has come a long way, but there are still some challenges. For example, if an algorithm is asked to identify how many of these photos are dogs, it is going to have a hard time because there are some dogs, and there are some chocolate chip muffins in all these pictures, and some of them actually look like a dog. So the algorithms need to perform so well that it does a job of identifying things into a given in a given stack of images, as good as or slightly better than radiologists. If we want to really use AI as a leveraging tool to minimize some of the burden of our work, or to help us aid in detecting some things which we might otherwise miss. So the biggest challenge, as I said in screening mammogram are category C and category D density press. So this is a screening mammogram with MLO views and CC views, and there was a locally advanced cancer on this mammogram which was initially missed at the time of screening exam. The patient came back with a palpable concern and there was a massive lesion in the right breast. Then we employed some AI tools. For example, this paper here, I've quoted that at the bottom of the screen. They tried to detect some of those abnormalities on these screening mammograms and then all those red dots that you are seeing on these mammograms are nothing but called as heat maps. So the software developers as well as people who know how to write codes and extract data out of these algorithms, they can generate these things called as heat maps. So they are just trying to understand how this algorithm is working to detect abnormalities.  Whereas in the other case, the suture markers which were placed as well as a small metallic marker from previous biopsy, the algorithm was trying to detect all of those things as abnormal. So you need to really understand how it works to get a complete sense of how you can, how you can embrace this technology. So how can we come up with some solutions? So there are different methods for representative classification tasks. So what is representative classification tasks? You are going to throw a bunch of images at this algorithm and the algorithm is going to give you a binary outcome, whether the, for example, the screening mammogram is abnormal or normal, or for example, if you add a stack of CT images and say whether there is appendicitis or no appendicitis, just to give an example. So these kinds of classification tasks can be done by some of these models. So I know that this diagram is busy, but I have quoted the reference there down below if you want to read more about how these classification tasks are designed. Most of the radiologists including myself, we are involved in the space of AI research when it comes to this portion of artificial intelligence. This is nothing but we are trying to use some predefined engineering features and then trying to see how we can enhance those features or use those features to detect certain abnormalities or quantify certain abnormalities in images. For example, you will have a retrospective data of say 25,000 receptive lung carcinomas, you will go in and do segmentation of each and every one of those CT images, and then try to extract texture features out of it, either using some types of some types of software programs using different computer language options available out there such as Python. And then you see what texture features are having a recurring theme that these features are causing abnormalities.  So this step is somewhere most of the radiologists get involved in doing research. And if you look at this portion here, which is called as B. So this is nothing but deep learning. This is the one where you can use what we discussed about Alex net or are those deep convolutional neural networks. So this can be supervised or unsupervised or semi supervised supervised will be that a radiologist or an expert will keep be keeping an eye on how the algorithm is detecting findings. So it will be refined over multiple multiple cycles by using different types of data sets, and then you get an outcome as yes or no. And there are all these multiple layers of complexity which are in between, which are doing all the work for you so for example, you give it a chest CT and then it will try to find out where is the cancer. So this is just a basic of how some of these algorithms are designed and how they work. So as we discussed, there are these two kind of types of representative classification tasks. This is a really good paper if you want to know more about deep learning and I feel that this is really important before we dive into our today's journal club article because we need to know what these algorithms are actually doing when we when we are asking you to detect abnormal screen or abnormal screen. So there are different types of image classification tasks. Let's start with, for example, this is an actual CT of the abdomen through the liver. This is portal venous phase you can see the blood in the, in the portal venous circulation and oral contrast was administered. You can definitely see that the patient has liver lesions, and then there's also normal liver normal stomach and the normal screen portion of the kidney bone soft issues, etc, etc. So one task that you can do is of something called as object detection so we talked about that when we discussed about cat right. So this is nothing but trying to detect abnormalities and trying to say okay this is metastasis this is aota this is stomach and this is spleen by putting a different color codes and trying to give you a better kind of an idea. So the CAD models that we used before the AI CAD for mammography screening they would give a circles for asymmetry and mass and squares for calcification so that was a task of object detection. Then the next task is going to be semantic segmentation so what is that so it tries to segment the area of suspected liver metastasis on its own. And then it color codes that for in this example as yellow and rest of the stuff is blue so it is just trying to pick up the area with metastasis so it is a specific task we're asking it to identify any specific type of abnormalities on the image. Then the next one is instance segmentation so not only it detects that these are metastasis but it also segments individual metastasis separately so for example 1234 it has segmented those separately. So each instance of metastasis is being segmented on this size. So this is all good right I mean we are really happy that we have algorithms but then how good are we doing. So this was a really famous trial. I have given a link down below, but this was a really famous trial where they tried to see if they can train pigeons to detect abnormalities of mammogram, as well as pathology slides, and they saw that the accuracy for calcified as well as non calcified findings it increased from 50%. To 85% after training the pigeons. So do we really need AI. Now I'm going to leave at this point and then jump on to our discussion today. So, if anybody wants to access this publication you can just pull up your phone and scan that QR code and you will get the complete paper. So the title of the paper is combining the strengths of radiologists and AI for breast cancer screening so see how we are trying to leverage AI to improve our outcomes in this paper. And the most important thing to know here is that this is a retrospective analysis. So before we move on. If anybody has any questions at any point just note them down or type them in the chat or we can discuss at the end of the session. I think there are also some participants on YouTube live they can type their questions on YouTube live as well. So now, let's go to the publication, and then try to see what was the purpose of the city. So they, the main purpose is to integrate AI into screening pathways so right now as you know that most of the places just incorporate a radiologist who looks at a bunch of screening studies and then decides whether they are normal or abnormal abnormal experience get called back. So some places as we discussed earlier today they employ some form of double reading either as randomly assigning some cases to other reader or using conference or consensus rounds to discuss whether they should call back certain findings or not. The goal of this study was to create a two part system and they're going to go that and go into that later. And then they wanted to assess what is the effect on sensitivity specificity when a stand alone AI system was used, or it was used in the setting of a decision referral type of approach so we'll discuss about that later. The, for the purpose of this study, one of the authors is a developer of this particular software which has been used all the codes and the algorithm of the software is is shared on the, on the main publication website in multiple different appendices we can read about that if you have more questions. Now what is the problem and what is the need for designing this kind of study. There are a lot of studies which talk about how AI has been comparable to radiologists in detection and classification of abnormalities. Then, usually what happens that the data is insufficient when it comes to the accuracy of these examinations of these algorithms in the setting of screening population, because remember a large portion of screening population is going to have normal exams, and there are 24 out of 1000 or somewhere on the abnormal mammograms with a cancer. So the large portion is going to be a normal population whereas most of the research studies look at and reach data where there are more cancers. So this is an abnormality or this is the problem which needs to be addressed in the screening setting. Then, most programs still use mammography as an agent so it is not technically alleviating any burden of radiologists to do the task of double reading or anything else. The results of increased AI in screening has always shown that there is some reduction in sensitivity of the screening program itself. People have used AI in triaging followed by incorporating recommendation for supplemental screening. However, the problem in this type of study is that they did not consider the inter reader missed cancers remember I talked to you about false negatives so they did not consider that just give me a second I'm actually doing this presentation from our hospital and we have motion detector sensors so the light just went out I just started just give me a second. So these are the problems and these are the problems that we are trying to address in this paper. So should be, will it be probably, you know, nice to imagine that something like this is going to happen. So this was a nice cartoon which I bought online so there's a computer which is doing the task of, you know, your double reading or detecting a diagnosis. And then the radiologist is doing something else as it is shown here just trying to relax which is never the case but. But then what happens is that the computer is saying that I'm being sued for missed diagnosis. So what do I do now so there are quite a few ethical concerns and issues that still need to be addressed before we completely embrace AI as a technology for helping our day to day practice. So the solution proposed by this group is that why, why don't we try to see if we can combine radiologist and add together and see if it can be one of the alternatives to consensus conference or double reading so at least the task of the second double reader can be minimized so that they can provide care to some other patients or do some other work at the same time. So how did they achieve that so they the current pathway which they have is you do a mammogram, then reader one reads the mammogram reader to read the mammogram and then there's a consensus conference decide whether it is a virus one or two or by zero. The second option is where they have a standalone pathway where reader one will read all the mammogram AI will read all the mammogram on its own and then there will be a consensus conference. And the third and the most complex pathway that you can see here is what they call it as decision referral pathway in which the mammograms will be read predominantly by an artificial intelligence algorithm. And what happens is that all the mammograms that the AI software feels confident about their triage as normal. All the mammograms where there is the it lacks, it lacks confidence, or then it is confident for a cancer, all of them go to a pathway of decision referral. And this is when the radiology steps in, and then tries to figure out how to deal with those cases so this is in a setting where it is used as a decision referral pathway. This is what paper also explores about. Now where did they get data from. So all of the mammograms, which are part of this study are from German national screening program between January 1, 2007 to December 31, 2020. So there were eight screening sites, and they obtained. necessary regulations clearance as for you regulations, and then they divided the data into an internal test set and external test set now this is really important because you want your algorithm to assess multiple different types of images at different locations or different settings, as that is going to be helpful for them to understand if the algorithm is working fine in the real world. So then they created once data set as a test set, and the second one as a screening set. So I'm just going to turn off my video here because the light is just turning off all the time, but I'll turn it off back later on. So now you have two data sets which are going to be used for assessing this algorithm, and they gave a split of how many cancers and how many normal mammograms they have. So this is the data pre processing step, which is again really important before you test your data on any AI study. So I don't, I will not go much into the details of all these things here but essentially the bottom line is that if you look at the data here this is the internal testing set, internal testing set, all the six centers. So all of this data was mixed, and then it was subdivided into three sets called as training set, validation set, internal test set, and then you have the last one where there were two centers which were completely separate, those are called as external test set. Now this brings me to a really important, really important concept of something called as what level of the study is data disjoint. What does it mean? So any patient belonging to this group was never a part of this group, and vice versa. Now why is that important? So when you train your AI algorithm using this training set and validation set and then test it on the internal test set, and then you send all of these images to the external set to this algorithm again, it is going to have a bit of a concern for bias, because if some of these patients are also part of this set, then it already knows those patients. So it is going to detect those cases perfectly well. So this is like you're trying to prevent AI from seeing anything from this data set, and this is called as the data is disjoint at the patient level. So each and every patient belonging to two separate groups was not mixed at all. So this is an important concept and just shows you robustness of that study. So results. So this is the important thing to remember. So what did they find? So there are a bunch of AROC curves here, but essentially what you see is this small, really, really small square here that is the most important one. So the red dot is for a radiologist, the purple dot is for an AI stand alone system in this case, and then the blue curve is nothing but your cancer detection, the mean or the average cancer detection. So the radiologist is actually work is actually doing a really good job of detecting cancers when it is used as a standalone technique when the AI is not being used for any work done by the radiologist it is doing something on its own. So the radiologist did a really good job in this case. Now this is the second one where again the red dot is the radiologist. And then again the second one is the AI standalone so if you look at this one this was tested on the external test set. So even then radiologist was doing a good job than a standalone AI. So it just shows you that radiologists are still going to be really beneficial when it comes to delivering cancer care. AI is going to be more useful in situations where you need that extra help or you need something else that can help you go through your day go through your volumes are going to add more value to the work that you're already doing. Now, this is an important thing to understand so they published their results, also in different types of densities. So for example, category a category B category C category D so as you expect the category D density breasts, they're quite a few amongst the entire population, most of them are category B and category C. So even in category C and D density breast, the algorithm is actually doing a good job. And then this is just a split of all the different by rats by rats categories three, four, five and five, like malignant with invasive component, because this is how we get our data for all the cancers at the end. So this just shows you what were the different types of abnormalities that were detected by the algorithm as opposed to an individual radiologist. And just to give you an, just a color schematic diagram so radiologists are going to be in this like pink kind of shade green is decision referral model and purple is standalone so you will see that the, the referral based decision referral model has actually done well when compared to the individual readers like radiologist alone or alone. So for instance, for example, for example, if you look at this inside you disease, where the standalone model has done a little bit of a good job than individual radiologist. Then if you look at this intramamory lymph nodes. This is where the decision referral pathway has done a good job then both of those. And obviously, in the setting of globally symmetry radiologist have superseded all the other AI models so charts and diagrams are usually the most important to go through any of you in any paper essentially. So this brings me to the end of our first part of the presentation we went over the purpose of the method in which how they did the entire study, the results what they found after doing the research. Now, the next most important thing that I want to discuss in the second half of this presentation is claim check, checklist, but before we go there doesn't even have any any questions that we want to go in this first part. No, it doesn't look like we have any questions so far. Okay, sounds good. Okay, so I'll just keep on going to the next session then. So claim checklist so what is claim claim stands for checklist for artificial intelligence in medical imaging this was released by the radiology artificial intelligence journal.  So if you have minutes to bring up the checklist on your phones or devices that you're using, then I'm going to bring up a word document of the checklist that I want to show you all. All right, so do you see the word document there. Okay. All right, so this is the claim checklist so essentially what you're supposed to do. So let me give you a bit of a historical background about this checklist and how we, how I became aware of this one. So we started something called as an AI general in our program for all our residents and fellows. And then it was essentially led by one of my colleagues, who is an expert in the field of research as well as AI. And then we decided that we want to create some kind of a structure when we are doing the general clubs. And then the most important thing that anybody should remember is that there is going to be some industry support when it comes to publication as well as providing grants for these AI projects. And you have to be really careful about the quality of the AI research being done before you really understand the results and before you try to decide okay this is what I want to do at my practice setting. So that's why this checklist was developed like you have to go through each one of those questions separately. There is an article available online as well for the claims checklist which gives you idea of what each one of those question numbers means like for example, what should you look for and study objectives what should you look for a scientific clinical background. So it's a really self explanatory process and it actually gets pretty, pretty easy once you do a few of those. The fourth most thing that you have to know is that once you start from patent abstract the important question is that did they identify any AI methodology so for those who have brought up the paper on their on their laptops on their devices, you can see that in the in the abstract, they have the authors have made no reference to what was the AI methodology was it deep neural networks convolutional neural networks, did they have some kind of a software. So that is a big issue, because not all the articles are going to be full text or free full text. So you would not even realize if you want to buy or pay money to get access to this article or not. So you will have that in the abstract. Then, the next step is structured summary of study design methods and results it was provided by the authors here, including how the population was divided into internal test set and external test set, how it was tested on the model and what were the results that they got out of that. So that is a really good indication that they provided in the title and abstract. Then we go to the introduction part, you need to know for any paper, not just AI but for any paper you need to know what was the scientific and clinical background including the intended use of this of this software as well as clinical goals. So in this case, they made it pretty clear that they want to use AI for the task of a double reading or consensus reading in the setting of either using as a standalone reader or using it as a decision referral approach. So it was very, very clear. So that is really nicely explained, and they laid out all the objectives and hypothesis so you are pretty confident about that. Study design, so this was a retrospective design, you can even see that in the title of the paper. So it is important to share that information with the readers because otherwise you cannot make an informed decision of applicability of that of the research. Study goals such as model creation, exploratory study, physical study, non-inferior trial. So this one was exploratory study because they want to see if they can apply AI as a double reader. So this is an exploratory study. Data sources, they have mentioned it extensively well in the study that what data source was used, where they got all their mammograms from, what was the nature of those studies, why were they done. So it was quite useful. So eligibility criteria, everything was fine in the study except for one issue, which was they did not describe how were the patients selected. It says that there were some 1.1 million mammograms which are in flow in the study between the time period from from 2007 to 2020, but they did not say whether they were all consecutive mammograms, or were they randomly selected. That is again an important aspect because if they were not clear in the way, in describing the way they selected the patients, there is a potential risk of introducing bias. So we had some bias by selecting a few instances of cases where it would, it would tip the finding in the in one direction than the other. So it is really important to understand that how the data was selected. Pre-processing steps, they include that very well. We already went over that. Then data subsets, they decided and explained it very well how the data subsets were created. The data elements were defined really well. We exactly know what they studied and how the patients were assigned to different categories. Now de-identification methods. So this was fully anonymized mammography studies and patient data was not available for most of the AI models. And then it was only used in the, in the testing and model validation step when they tried to create the model, but when it became, when it became a testing sets for internal test set and external test set. All the data was anonymized and stripped off of any patient data pathologies or anything like that. So there was no way for the algorithm to know whether this mammogram was abnormal or normal. So essentially what you can imagine is that they try to simulate a screening environment by in the retrospective data. So the other problem was that how missing data were handled. So this was not clarified in the paper. So you have a little bit of an, you know, a risk of bias. For example, they said that normal mammograms were considered to be studies which were normal and then 24 consecutive months they were monitored. And then they saw if there was any recurrence of any, any new cancer which developed in the next 24 months, but they did not say what did they do for patients who did not follow for the next 24 years.  So this is a clinical, this is basically a concerning finding because they did not see how they handled the missing data. Then they did not say if, if the patients who had a biopsy at one side but then they moved to another site for their surgery, how was that missing data handle, because sometimes there could be up staging down staging for example trying to figure if you have just the diagnosis of DCIS on biopsy, which is non-invasive, but at lumpectomy it can get upgraded to invasive carcinoma depending on how the biopsy was done. So if the surgery was not done at your site and the patient did not undergo any surgery, then what happened to them? Was there any, any evidence that we missed the invasive carcinoma in those cases? So these are really important steps where you want to discuss about how the missing data was handled and they did not describe that in some cases. Now the ground truth when it came to, when it came to designing the algorithm was histopathology. So they even had normal histopathology diagnosis for patients who went into the testing of the algorithm. And remember this is the test set and validation set, which is a step before giving that algorithm an exposure to the internal test set. So this is just development of the algorithm. So at that time it was histopathology, which was the ground truth, and then that was as a reference standard. So the next one is that number 15 rational for choosing the reference standard. I'm not going to explain that at all because they did not talk about the alternatives. They talked about their own algorithm and how it is working as a second reader. So this is really important. So any, most of the, most of the research paper will have something where they will say that simultaneously three models were developed with different types of inputs. And those were tested randomly at whatever data, and then the robust model was selected for further steps. So in this case, it was not described, all they said was that there was one model, and the weights of the model was changed. So it could be one of the subset of different steps or different models but essentially they need to have more than one model, which can be used for testing and then one of those can be selected, whichever performs the best. Then the source of ground truth annotations qualifications and preparation of the annotator. So this was all done by radiologists. So they looked at all the buyers for a five buyers three or any buyers one or two lesions which for biopsy, and they annotated them using small octagons or hexagons and then that data was spread into the algorithm, and that is how the algorithm was created. So measurements of inter-reader and inter-reader variability. This is an important step. And then what methods were used to mitigate variability and or result discrepancy. So in this case, they had, they basically had two different anonymized readers, and they were trying to independently sample or segment the lesions or find abnormalities. And the whole data set was re-sampled with replacement. So that is really important step and this is how they made sure that they accounted for inter-reader variability. And this is what is going to address one of the issues of inter-reader missed cancers. Remember, one of the points we discussed in the problem slide that some of the previous research did not consider this inter-reader missed cancers. And that is what they were trying to mitigate in this study. So it definitely adds more value to the previous ones. Intended sample size, how it was determined. So essentially, they did not give any indication of how the sample was determined. They just said that it was a retrospective data set from 2007 to 2020. So they did not describe how they arrived at this time frame. But he would assume that probably it was considered sometime, but it would be nice if they would have given it in the publication. So how was it assigned? So we already discussed that they compiled six centers together and two centers together. Among the six centers, they randomly divided that population into three different proportions. Each one was assigned a test training set, validation set, and internal test set. And the external test set was totally different. So that is really well done for this paper. We already discussed about point number 21. In this case, the data was disjoint at the patient level. Really important step for any AI algorithm. Because you don't want the algorithm to know what already exists in your validation set. Otherwise, it is going to be biased. You will know that patient, it will know that image. So it is going to give you falsely assured results. So in this paper, they have given all the details of the models in the paper as well as their separate appendix, which you can open and really look at what software libraries were used, what framework was used, what packages were used if there were any. And then number 24 initialization of model parameters. So this one is something called as transfer learning. So you, you basically teach the algorithm with the help of radiologists and then you transfer that knowledge to understand how it can work in the real world. And then number 25 details of training approach. So it is also providing the appendix really, really important if you're reading any paper where they don't provide these details, never trust that paper unless you can email the author asking for these steps. So can you please give me the details of training approach, if not provided then you have to guess that something is wrong and this is not a good quality paper method of selecting the final model again as I said there were no multiple models used. So this is not really mentioned for this one assembling technique. It's one of the techniques which some of the AI models use. This is not applicable for this study. The metrics of model performance was judged by the outcome of sensitivity specificity when it was used as a standalone compared with the radiologist or as a decision referral model when used in conjunction with the radiologist. The physical majors are mentioned, and they have gone through the statistic really well. So they have used area of receiver operative curves, then robustness of the analysis and sensitivity analysis. It is quite good validation or testing on external data set, it is really important but some of the researchers really struggle here. It is really difficult to get external data. So most of the times researchers end up having their own data into two different models, or two different groups and the second group is called as external data but that is not really the case. What is the real world problem when you don't have external validation or external data. So for example, you will see that there are a lot of AI algorithms which are available commercially now. And it's a separate discussion, I don't want to go into that right now, but when you or your practices buying any new AI product, then you have to use another type of checklist, it is called as Echlear Checklist, even just like our Echlares. So it's called as Echlear Checklist and you want to really make sure that how the algorithm was tested and how validation was obtained. And this is that step. So what does it mean. So I'll give you an example. Here is, let's let's imagine that there is an AI program which is designed for detecting abnormal mammograms in a subset of population in say North America. That subset of population may have slightly different proportions of category E category a category V category C and category D density press. So for example in our population if we look back, there were around 55. I think mammograms or something like that on one of the charts there. So the proportion of category D density breast is low but if you deploy this algorithm in an environment where the proportion of category C and category D density breast is really high amongst your spinning population, then your algorithm may not work exactly how you want it to work and how to overcome this. So imagine that if I design an algorithm, what I would do, I would reach out to different parts in this world and try to see if there's any possibility of collaboration. And then you do this thing by either asking that collaboration site to send their data to you to test on your algorithm, or you export your algorithm to that site and ask them to test it on their data set. So this, why is this a problem, it is because of the review ethics board restrictions. So external validation is always a problem, and you need to go through multiple steps to get it done.  So always remember the importance of this validation step. Now results. So we already looked at the results and they gave a flow of participants or cases using diagram to indicate how inclusion exclusion was done. Then demographic and clinical characteristics were described in each partition performance metrics on all data partitions was was pretty nicely done. They tried to compare performance of different types of models as a standalone or as a decision reference model. In the estimates of diagnostic accuracy and their precision, they gave a confidence interval, it was mentioned, then failure analysis of incorrectly classified cases this was not mentioned. So what does this mean. So for example, if the algorithm classifies a mammogram as by rights for or by rights I mean by at zero, and then calls it back but you know that this mammogram is actually benign or negative and the reader already has called is called as a by rights one or two. Then how was this discrepancy handed this is something which is which was not mentioned. Then they did also did not mention what happens to the false negatives called by the algorithm, because when it was using in a decision referral system, AI was classifying mammograms as completely normal or classifying them as not completely normal or completely abnormal. But amongst the first group where it classified them as completely normal. What about the false negatives in that group. They did not say how that data was handed. So it is an important step and important consideration. In this discussion they spoke about study limitations including potential bias statistical uncertainty and generalizability, they completely are aware that this is a retrospective study where they tried to replicate a real world experience of reading the spinning mammograms, but the proportion of normal versus abnormal mammograms the proportion of different density breast the proportion of different types of abnormalities such as architectural distortion mass calcification scope of asymmetry global asymmetry, the proportion of those things is going to change in the real world. So there is always a limitation for that when you actually employ them in the real real world and you actually saw that on the results that the radiologists were the ones who are calling most of the patients with global asymmetry back and the algorithm was not actually doing a good job in doing that. The limitations of practice including intended use and clinical use they have provided a clear idea how it can be used. The most important in the other section is that did they give any registration number and name of the registry for the algorithm, which has been tested. This is really important and makes things a little bit more concise in saying that by the way this is the registration number of our model you can go and know more about this thing and make sure that you feel confident that whatever we designed was actually really good quality stuff which is our which is recognized by some of the partners in the industry, then the full study protocol they have actually released that. And then the source of funding is also mentioned so this is just a reference of this claim checklist so there are 42 points which we went over for this paper. And as we see that most of the things were really nicely done, except for a few exceptions. The study is quite really good and robust. So I'm going to stop sharing this portion now, and then we can open for any questions and answers if anybody has any questions or any comments. Again no questions the chat box but I do have a couple. How much better would an AI software be in relation to CAD software. Yeah so that's a really good question so for AI and AI CAD so CAD is doing one classification class right it is just doing detection. It's just showing that this is a normal this is a normal this is a normal there are multiple studies which are presented where they looked at the variability between a reporting radiologist and a CAD software, and somehow it has never shown any added advantage to the screening program. So what is the added advantage to the screening program. Say you detect four out of 1000 mammograms as breast cancer as you're expected to. But what about the others which undergo enormous study and they have missed cancers, and any modality which is going to detect those missed cancers is going to have something called as an incremental cancer detection rate.  So remember that tomosynthesis may add a few couple more cases, once in a while, but it doesn't add a lot of numbers in terms of incremental cancer detection rate. So when we are deploying any of the AI or CAD tools, it is really important to see how many more cancers you're actually detecting when compared to the radiologist. And that is where some of these softwares have really not shown a lot of significant difference. And even in the AI study or AI software that we use for this paper, you saw that the radiologist was doing a good job at detecting cases or cancers, when the AI will just use as a standalone reader. So that's why some of those softwares can be a bit of a challenge when you really want to rely on that software. It can be a good double reader, but it is not a good standalone screening tool. Whereas what the, what the authors of this paper described was that they wanted to use that AI software in a decision reference system, where it is going to help you detect probably more number of cancers, and then probably have an increased confidence in saying that normal mammograms are normal mammograms. So that is reducing your false positives. So those are the two metrics which are really important when you use any AI software in clinical practice. Now we have been using AI CAD in our, in our particular, in our clinics, it has actually worked pretty well. And as I said in Thomas synthesis settings, it actually detects a slice where the abnormalities present. So it's definitely a good tool as a second reader. That is what I would say but there are quite a few papers out there which looked at comparison of CAD with the radiologist and see how it fares, but it is definitely not detecting more cancers than the radiologist. Right. Okay, and just a couple more. So how far do you think, so I think of course Canada would be a lot further in terms of how they use the software and you know where they use the software but how far do you think we are from a democratization or something like this. So that's a really good question. I think the biggest challenge is funding when it comes to developing any softwares. And unless there's an industry support, it just becomes really hard. So if you just look back and look around you, you will see that there are so many oncologists are so many people from into medicine they're on like multiple different clinical trials and you know trying to test out some new drug and all this kind of thing because there is a lot of industry funding for those kinds of research. Now, the issue with research in the ideology is that there is not a lot of strong industry support when it comes to developing the algorithms. And that's why most of the studies are like, for example, my hospital, one of my colleagues does a lot of these research so he's like an isolated example who does a lot of research in AI for example he's developing an algorithm with support of a research grant to detect HCCs in liver, like all the arterial hyper enhancing observations and he has even gotten you know external validation and those kinds of things so it isn't bits and pieces, but the strong industry support is somehow lacking. The biggest challenge with radiology AI related research is that you need multiple multiple number of images large datasets for the algorithm to be really robust at detecting abnormality because you need, as I said, like, you know, amongst 1000 mammograms only four are probably going to have so you have to generalize that to such a level that you have equal proportions of normal mammogram and a very small proportion of cancers in them to see how those algorithms do and getting that data is a challenge. So that's why it is still a little bit far away from generalizability, but it is entering into mainstream. For example, some of the newer CT scanners, they have anyone MR scanners they have some kind of AI algorithms, which are based on gas, which are nothing but general real adversarial neural networks. And whenever the patient has like motion artifact for example on CT chess, the algorithm tries to correct for itself. So the algorithm saves you a bunch of time or repeat exam or repeat radiation dose. So we are already seeing some applications of AI in our day to day radiology work but when it comes to the task of detecting managing radiation, I think we are still a little bit far away from getting applications into clinical practice on a day to day basis. Right. Okay, and just one last one before I let you go. This is not entirely relevant to your article, but just in terms of AI in general. So how different do you think our job would be say 15 years 10 or 15 years down the line. So in the sense that as young radiologists do we need to be practicing a little differently or concentrating a little more in a broader skill set. Yeah, that's a that's a really good question we get that type of discussion all the time like we have you know medical students who come for shadowing us in our department. And everybody has concerned that you know what is going to happen to radiology in the next 10 years because this is what I want to do and then it is going to take me five six seven years depending on what path I choose. With a field be different or any different than what it is right now. So the only thing which I would say is that probably a radiologist who knows AI is going to be an added value down the line. We are still far far away from AI replacing radiologist jobs because that is how people usually portray that as a threat. So even now the model that they trained for this paper, you saw that the radiologist were doing a good job. And then there are just some other things which you never know how the AI is being trained on. And it just takes you probably a lot of time to realize that there was probably something which was missed or something incorrect was done so I would say that from a future perspective it is probably nice to know AI. And that is why such type of general club activities can be a really good start point where you get these discussions in the group of your colleagues, maybe if you're in an academic setting, ask your residents to present some general clubs find some journal articles, use a checklist to self train yourself about some of the concepts like what is meant by validation like for example if I'm, if I'm, say if I'm a chair of a department and I want to buy a new CT scanner, and the industry representative say hey by the way, we have this new scanner.  But as a chair of the department, I need to make that informed decision, whether I'm paying two lakh rupees for something and I need to know how that AI software was trained. So even as an end user, I think it is really important to embrace AI or have some knowledge of AI. Maybe we will have different kinds of reading rooms in the future where the patient will be registered. So at the front desk, you will have chat GPT run through the entire clinical note. And then you find okay this patient has a history, and that history points towards an increased risk of cold rectal malignancy. So maybe when I'm reading the CTF dominant illness, even if it was done for rule out accurate appendicitis, I'm running the entire colon probably more carefully because I know that this patient is at an increased risk of cold rectal cancer, which was not disclosed in any of the notes before or not provided in the So it can become a really good adjunct to our practice. And even as patients, I think there is some increased applications down the line. For example, if they want to know if they're say in your clinic, and you give them a diagnosis of Byron three.   What is locally advanced cancer. So that sends them into a spiral hole and they just get really anxious about it. But if you could develop some software available in clinical areas they could just type in there. And something like a large length model like chat GPT just gives them what they're supposed to know based on their experiences. It is going to help them as well. So I think I think there's definitely good application if used as adjunct to download and and I am, I will not be surprised in the next years if we actually start using some of these things in day to day practice. Thank you. Thank you so much. So thank you for all your efforts. I think we'll see you again next month for the MRI teaching course. Thank you for inviting me for this article. Thanks to the entire team of Indian radiologist. Thank you.


Everybody back here to the Siegel talks at the Monteney Siegel Theto Center, the Graduate Center CUNY, the City University of New York in Manhattan. And while slowly spring is approaching and while slowly around the world and some countries, the life is taking first steps to get back to normal. We in New York are still under lockdown. It's still the epicenter here and the city is closed. We can't get out. I know in France it was a D-Day they called it yesterday where you can go out for the first time. I think with masks stores are opening. You can go and buy clothes, but you can't touch the clothes. You can try them on and only one or two people in the store. So it gives us a little hint of what's happening. We heard from South Africa yesterday. We heard last week from Romania, from complicated situations in Hungary and others. So now today we are turning to the Ukraine. We have with us a significant and important playwright, Natalia Boryshbid, who has been at the Siegel, whose work has been shown at the Royal Court and many, many other places. She was also a good friend of the late Elise Dongsson, who was a good friend of the Siegel Center, who we miss very much, who passed away unexpectedly. And with us also we have Alice Terekov, a translator who is joining us from Brussels to do that good friend of Natalia. So thank you both for coming. It is a time where we need to listen to voices of artists, where we need to hear really, really hear what they are saying. Perhaps these existences we all have now, where there's a fixed reality outside and uncertainty in science, where we spend time with ourselves, is a time perhaps we can listen a little bit more. That questions were very basis of our existence, also of our work and what we do. Ukraine is very much in the international news for many years now, but when it comes to corona and everything, it's also what we hear, messages are very confusing. So what is going on in Ukraine, Natalia, can you tell us a little bit? Now the situation, sorry, the situation in Ukraine is not stable for the last six years, because the war started in 1940, and it is still going on. So Natasha would like to comment first on the general situation, which is for the last six years is quite unstable and complex, because in 2014 the war that was provoked by Russian Federation has started and it has been going on ever since. We have changed the political situation very much, last year we won the election of the president in exchange for the Ukrainian position, and therefore today the situation has deteriorated, and this is exactly why in the world, so little is said about Ukraine, because the situation has started to unfold back. So the political situation has changed about a year ago when the president and new president was elected, who is kind of less pro-Ukrainian, and in a way the general direction has been kind of turning back. That's why Ukraine is kind of less talked about or less urgent in the news. So coronavirus has sort of landed in Ukraine on a bizarre soil, on a not very kind of, yeah basically in a way dealing with it, it has been some kind of a relief. What do you mean by a pause? A pause between war, some kind of active actions, some kind of struggle, and I will say it later. When she meant relief, she was more referring to certain pause, pause in fighting, pause in the war activities, pause in some kind of activity that was so unsettled before. So she was actively spreading the rumor, the fact that it is very difficult to check, that Putin was going to make some kind of action before the elections, which was supposed to take place on April 22 in Russia. He was planning a massive nuclear attack on Ukraine. And that's why for Ukrainians in some sense it is a temporary salvation, so that everything becomes a pause. But prior to elections in Russia on the 22nd of April, Putin had in mind a very big attack, potentially a nuclear attack or just a massive attack on Ukraine. All this corona stuff has kind of paused the elections and also paused those plans, whether they were rumors or not. So for six years you have war in your own country? So corona virus is for you, as you say, perhaps even more kind of a break or a relief, where everything is pausing, when things come to an halt that perhaps could get worse. That is, yes, in some sense corona is a little less scary, it turned out, well, probably, than war. But it doesn't mean, of course, that we don't get sick, people don't die. It doesn't mean that thousands and thousands of people, millions, probably remained unemployed. It's all, of course, sad. But the general mood is that we have a break. So in a way, yes, to what you've said, and it's kind of, the virus scenario is kind of a deviation, a better option in a way. It doesn't mean it's a good option. It doesn't mean that people are not suffering, people are not dying, and thousands or maybe millions by now people are made redundant and have no work and no means to earn. So it's not a good scenario, but in a way it's kind of a better one compared to what it could have potentially been. So all war activities in the occupied territories that has all stopped? Well, at least they are not on pause, something is happening all the time, people are dying there, but there is no deterioration of the situation, there is no, as it were, of the offensive. So in a way it's not completely paused, there are things happening there all the time. But at least there is no attack, it doesn't advance, it doesn't get worse. What things happen there at the moment? And what is happening there at all? What is happening there now? This is a vial of the current conflict on both sides. The conditional border is located in the military unit, which, from time to time, makes exits and such attacks. People are dying. So basically, the situation is kind of a war situation, but it's soft in a way. And there is military forces on both sides of a so-called border, and they kind of cross the border into each other's territories on a regular basis and kind of make some activities. Exchange fire? Yes, there are cases, yes. But it's more of an everyday soft version. The government in Ukraine also is not decisive when it came to corona. Can you tell us a little bit about your government, and if you trust what they do? How does the government react? Yes, there is a feeling that they are not decisive enough, and what is the attitude towards the government and what it does. Do they trust him? It's always different, there is no single opinion. There are a lot of dissatisfaction with the fact that the quarantine method is so long, that it is so big that there are so many unlimited restrictions, there are prohibitions, everything else. But my feeling is that, in fact, it's all ... I don't really like our government, but it seems to me that they did a lot of things right. That is, they overestimated themselves. They closed a few things, of course. And thanks to this, there were not so many terrible consequences in the country as in other countries. So, basically, there is no, of course, shared opinion that everybody shares. There is not one opinion. But basically, there are lots of people saying unhappy about all the restrictions and all the things closed and you can't go for a walk and this and that, and it's been going on for too long, so they're not happy about that. But in a way, even though Natasha is saying that she's not an admirer of the government in general at the moment, though she believes that they've acted wisely in a way that they acted in a strict way, they kind of overdone it maybe a bit, maybe, but all the restrictions and this overdoing it as a reaction probably led to the situation getting as bad as in some places in the world. There is still opinions, you know, rumors going on and people believing that there is no actual virus. It doesn't actually exist and it's all some conspiracy special services in America included or somewhere else. But this also exists as a fallout for how long have you been under lockdown in Kiev? For two months. For two months. So how is the situation for theater artists in the Ukraine at the moment? Well, of course, all the theaters are closed. Just a few days ago, the rehearsals were allowed. State theaters, the artists continue to get salaries and in general, it's not scary for them, but independent theaters, of course, are dying, literally dying because they can't pay the rent, they are actors, well, in general, it's a disaster. So, I mean, theaters are closed. And a few days ago, there was some kind of allowance in terms of rehearsing. So rehearsals are possible now under certain conditions. In state theaters, the staff is actors and all artists are getting paid regularly. But of course, all the independent scene is in big trouble because independent theaters can't pay rent, they're not earning anything, so it's basically a disaster in many other places in the world. Not everyone can watch theater from the screen, that is, not everyone accepts it. And of course, lots of theaters were sharing those performances they've got on the video, but not everybody can connect to a performance on a laptop screen. So, but they were doing it. Under what conditions are rehearsals allowed at the moment? So basically, Natasha is saying, because I am not in rehearsals, I don't have a personal experience, she is not in rehearsals. So, she basically doubts that the mechanism is somehow written down in detail anywhere at all. So, I mean, she supposes that the actors and everybody should be wearing masks and, you know, have antiseptics and I don't know, whatever. Social distancing and all that, but she doesn't have any experience and also doesn't know whether they follow or not, or where are those instructions, maybe they've been sent. How does it feel for Natalia to be a playwright in a country where there was a war going on in its own territory for over five, six years, and now being at home for two months, not being able to go out, how does it feel? What are your feelings like a playwright who lives in a country where there was a war and sits at home for two months? What are your personal feelings? Well, for a playwright, it's probably a situation of grace, in the sense that there is something to write about, and of course, a wound that doesn't heal, it contributes to the search for a new piece. But a strange feeling, yes, with this two-month break, as if you were running somewhere all the time, running, running, running, and you couldn't even stop and think about what was going on, but suddenly you were stopped, and somewhere else they were running somewhere, and suddenly, all the time, even some difficult things, I thought about it tomorrow, but also the war, I thought about it tomorrow, and this tomorrow came during quarantine. Let's translate a little bit. Sorry, I didn't want to interrupt you. So basically for a playwright, in a way, it's a brilliant situation with no distractions and everything, and also the wound that is not healing, the inside wound, the trauma that is not healing itself, is kind of a constant source of things to write about, to create new plays and new writing, but you write with these two months in space. The feeling is that I've been running, we've all been running, running, running, and now we've been stopped, but the legs are still running, and there were tough subjects, i.e. war, etc., that you were kind of postponing in terms of meditating on or thinking about, and it's like, okay, I'm going to think about it tomorrow, but now is tomorrow. Now is the time, there's no excuse to postpone. So basically during these peaceful two months, the biggest realizations or meditations on the past few years have been happening, and in this pause you look outside and there's beautiful spring, beautiful weather, and everything is kind of still and not moving, and this is kind of the beauty of the end of the world sort of situation. So what you say in the meditations, what goes through your mind when you say, now you are thinking about all you are supposed to think about, what do you think about? When you say that there's nothing to postpone and you need to think about serious things, what thoughts do you have? Probably, well, about the future, first of all, what will be for me and my family, and what will be generally for humanity. Probably about myself and my family, of course, think more about my country. So probably it's mostly about the future, what's it going to be like after this, and my future as well as my family's future, as well as the future of humanity, but mostly my future and the future of my family probably, yes, and my country. Also, there is this new fear that is developing that after the pandemic, I don't want to, I probably won't say that for the first time now, but if you go through it, it's like before some storm. I hope I'm wrong. Also, there is this new fear that is developing that after this quarantine is over and it will be over at some point. Something horrific is going to follow. Something is going to start that is being postponed now. I never probably verbalized this before this moment, but hopefully I'm wrong in this expectation of some horror following this peaceful and lack of action and quietness. I really hope that I am mistaken in my expectations. So you hinted that your fear that Russia will launch a military strike and use this preparation, this pause for preparation. You talked about an open wound that normally is there for writing. What is that open wound for you? What does it mean? I've never thought that I'm going to live through wartime and that war is going to be a subject of my creative writing, that this is what I'm going to be writing about. So that ongoing war is something that is so overpowering that it is hard for you to even imagine different subjects? Of course there are other subjects and when it's been going on for a while, life goes on and people get distracted and people forget a little bit and it's talked less about. So there are other subjects that pop up, of course. So basically there is a certain use. You kind of get used to the situation of the war going on and in a way it informs your writing in some way, but a bit less dramatic, less tragic. But these two months of a pause, they sort of bring back the fears and bring back the sharpness of realisation of what's going on. So basically the war is not restricted by its location in Ukraine. If you look further into the world, there are a lot of conflicts, there are a lot of troubles of all sorts. And in a way you kind of lose ground from underneath your feet more and more. So all of these things they're not just about these worries and these fears, they're not just located in Ukraine and Ukrainian people. It's basically what occupies the minds of all thinking people at the moment. When did you become, when did you think about becoming a writer? And do you think in the face of war and violence, do you think what is the place of a theater writer, of a theater artist or theater itself? Do you mean when she was even planning to become a writer? Yeah. When did you think about becoming a writer? And if you remember that feeling of a place of a writer in the world and a place of a theater writer, a theater artist, a person who writes for a theater now, who is his place? Well, it's between those people who started, who are now, of course, a great prophet, two different people. But I, to be honest, I was brought into this sphere in life, into the theater, and I went into this process without thinking at all. I didn't even remember anything, to be honest, what I was thinking about at the theater. So basically there is such a huge distance between that person and this person right now. I was kind of led by life and I was led into this sphere of life, of theater, and in a way I was quite irresponsible and just kind of unthoughtful when I walked onto this territory. I remember that I quickly understood enough that it was a good means for me to understand some of the issues that are tormenting me, my problems, my injuries, and I just didn't know that such a volume, well, from the age, there was just a volume of these problems, injuries, experiences, compared to when I was 20 years old. Again, there was no war, there were not many things that exist now. Well, understanding is still there. So basically at some point I realized that a theater play, a script, is a very good way for me personally to deal with my own issues, questions, problematic and traumatic experiences, that it's kind of my very efficient tool. And with time and with time going on and with age as well, these things acquired volume. So compared to what I was when I was 20 and with those issues and problematics, I mean, there was no war at that time. Now there is, it's just everything acquired volume. And the understanding and realization, the understanding is grown. Well, I guess I started to take more of myself. If I used to be egocentric, then now I'm probably, since, well, I guess I'm also self-explanatory, and since I got used to working with documentary materials, with material stories, then all the last years it's learning, communication with a huge number of people, with a heavy weight. I probably would say that I now take on more onto myself and kind of take on bigger subjects into my writing. If before I would probably say I was more egocentric and was kind of dealing with my own things more than anything else. But, you know, there is a limit to that. There is a limit. And for the last few years, I've been working a lot with people in terms of documentary theater. And so it's now more and more studying and researching lives of people that are full of hardships and of all sorts, full of troubles. So it's kind of broader. What is the role of theater artists in the Ukraine, in society? What is the status of theater in the Ukraine? Well, until recently, theater in Ukraine was quite conservative, not modern. The process, probably, is the same war, as strange as it is, the theatrical situation is much better. So there were a lot of young artists. So basically up to a certain point in the past theater in general, in Ukraine was quite conventional, quite conservative, sort of old fashioned and opposite to contemporary sort of thing. And in a way, the air that was brought in by the war and the change was kind of a good one. Lots of artists, lots of young artists with new things appeared.  What do you think is the theater important in Ukraine? What is your hope that the theater will be doing? What is the name of the theater? What is your hope? What should the theater be? What is your hope for the role of theater in the future? And what should they do? What are their responsibilities? Well, their responsibility is to be in the present day and reflect, to analyze everything that is happening in the modern world, to ask the most difficult questions to the public, to the audience. So probably the responsibility of theater artists at the moment is to stay in the now, in the current moment, have the courage to do so and to have the courage to keep on reflecting on what is going on, to keep on asking uncomfortable questions, asking themselves, asking society, you know, the audiences. Well, as any art, theater should also expand its knowledge and not entertain, or repeat some old story. As any art theater is there to give a way of expending consciousness or a way of thinking as opposed to entertaining or providing a distraction. So it's a way of expanding the thinking and the consciousness. Is there anything changing in your mind for those two months that you have time to think? Do you think your work in theater will change? For those two months, how did your consciousness and feelings change? How do you think it will change what you did before? I was in different states for these two months. From the first state it was quite hysterical, it seemed that now I have time to do everything and I will write a few pieces. Instead of that, I was doing other things. During these two months I have been in different states of myself. I experienced different stages. The first one was a kind of hysterical one and very agitated. Now everything is paused. I'm going to keep up with everything that I've been postponing. I'm going to write a lot. I'm going to finish everything that has been unfinished. Here is the time. It was a very fun time to spend with friends who came to Taycum, who were also in quarantine. We had such a narrow circle. It all resembled the characters of Chekhov who had absolutely nothing to do. They carried themselves on different fields. Then there was also a stage of a fun stage of having fun with friends who were not seeing anybody who were in lockdown but would secretly somehow make their way and we would spend time together. It was very comparable to those characters in Chekhov's plays that had nothing really to do or to work on. They were spending time at each other's country houses and doing nothing much. Last week there was a moment of enlightenment and tension I would say. And it's been recently this other stage that is more of some kind of enlightenment or some kind of things got clearer and more thoughtful and more kind of grounded and thoughtful. That's what I think. To be honest, I think people don't really change after quarantine. I don't see any big changes in their personality. People forget everything and they go through fear. It seems to me that the experience of working and interacting with personal contact is the real discovery of that time. And when I'm in this more thoughtful situation, I don't see any bigger and calmer state of processing things. This is what I've come to so far. Which is basically that people won't really change after this lockdown is over. People are very forgetful about what's happening that will completely transform what follows after. People are forgetful and experiences and worries and fears pass and they are memories. They become memories. But this type of communication is so intense recently. Everybody will share that and everybody will remember that. In general, the idea to pause at least a few years to relax nature is a good idea. The main thing is that it is not easy to think that time will come in time. Generally, this idea of a pause for everyone is quite a positive idea. Maybe we need to think of something that would be a regular thing every now and then. We organize this huge pause for the world to have a little rest from us and our activities. Ideally it wouldn't be organized by the illness but more organized by us, more initiated by people and organized as a reoccurring, as a regular thing. How do you organize your day and do you have time to write? Do you write? Yes, I do. I live in Kyiv and I live in Kyiv and now I live in Kyiv with my family and I have more responsibilities on the one hand. On the other hand, I don't need to run or go anywhere. I just need to have the opportunity to write and work in a very nice office with a massage. I'm normally based in Kyiv but right now I'm away from the city in my country house with all my family. I now have more like domestic responsibilities and things to do and organize but at the same time I have less things to run to go to drive to. I don't have to drive anywhere. I'm just based here, I have all my meetings and I kind of have more time and a very nice view onto the garden and yeah, there is space to work. The only difference is that my family is also at home. If they used to go to school for work now they are at home and need a lot of attention. I think that this is a situation that doesn't complicate everything. The only complication for me as a writer is that compared to normal life there is a difference because all my family is constantly around and goes nowhere and needs my attention much more not to say all the time that everybody is experiencing the same so for a writer it's a bit of a difficulty to deal with. So you are in a way in confinement but you are missing the solitude you need to write. Yes, very much. So do you write for two or three hours or what? I don't know how to organize my work that's why I write only when I need to give up something I wait for some text from me. Now I write first of all I wait for the song and I dance every day near the computer to continue. So basically I'm quite bad with organizing my own work schedule or discipline and I mainly write when there is a deadline and somebody is really waiting for me to send something and so currently there is a Royal Court Theatre in London that is waiting for me to come up with a script and I do regularly dance around my laptop to produce something and to keep on my work on that. There is a problem that the song was also written by Alice Thompson but the time has changed for the past two years from the moment we came up with the project that for me it is not so relevant it is a big problem. The big difficulty here is that this idea was born when Alice Thompson was still alive two years ago and the situation has changed so much that the urgency of the subject or the whole idea is a bit less relevant for me now so that's a challenge in terms of writing that. Does writing give you something in these times to hold on or would you rather not write at all and experience the moment and wait till it's over? What you are writing gives you some kind of a soil or would you rather not write and just relive it and then start working again? Yes I have a feeling that I would rather relive everything and then reflect a little and then start working. There is indeed some kind of a feeling inside that this kind of a feeling lost and kind of not having a ground to say yes that I do want I wish I could sort of just live through it and process it and digest and then go back to active writing But when I don't do anything I have a feeling that I am wasting time and I can't afford to waste time but when I'm not doing anything following this desire and this logic I feel like I'm losing precious time so I start doing something and this is the vicious circle that never allows me to stop working and process things so every point is triggering itself One of the messages we heard from many writers is that it's okay not to write it's okay to experience it and to make sense out of later not to always be busy and to see also what is happening so in progress do you read a lot then or how do you keep your mind busy or is your mind also taking a pause I read a little I listen to music that my husband plays in general I like it I really like to be in nature and spend time on walks in the garden and I spend a lot of time digging gardens, planting some plants so that I don't get distracted and we watch a lot of movies I do not read that much at the moment I listen to music that my husband plays which the majority of it I like what I am really enjoying at the moment is nature and having this contact with nature so I love walks I love being in the garden I love doing something in the vegetable garden planting something and really fascinated by this it's really bizarre because I am so inexperienced in this in my past I have never had time to do any of these things in the garden and with plants so the way life slows down we see plants grow we see things we normally don't pay so much attention to and this is one of the things that we take walks work in nature cook take care of us and our loved ones and our family do you think that theatre and performance will help to process this moment do you think that theatre is a medium that at the moment works in your country for the society to work through social problems, political problems, personal problems or is it television shows or movie theatres how is the mood in the Ukraine in the theatre what is the activity of the theatre how do you think it will help or now actively helps the society through all these processes to work through and support and now and the future friend do you mean now or in the future after, yeah in the future, yeah will it help to survive it all I hope as a part of this process I will participate in this and support some initiatives I think people will miss work and I hope that something will change I really believe so that theatre will be active and useful in processing whatever we are experiencing right now and me as a tiny little part of this huge mechanism I will support initiatives that would be coming up I'm sure and I believe that people will be artists will be so hungry after all this time when they were deprived of the opportunity of being together and creating together that they will be really up for collaborating and working and making things better but on the other hand we were forced to finance it and the government didn't pay much for the theatre so we can't get any grants so we can't imagine what I said but at the same time I remember all the financial cuts because the government has cut a lot of finances so there are not that many grants we can apply for and in a way everything that I have just said I'm not quite sure what are the practicalities of all these things that we would be so up for doing after Olimpia Dukak is a great actress from America who ran a theatre of her own she said the reasons why she got into theatre have nothing to do with the reason why she is still doing theatre and perhaps that's good and bad on both sides are you finding a new reason, will you find a new reason to be in theatre, will you be reinforced in your choice to create theatre will you think that perhaps in that war times you are in that theatre is not able to really help shape, bring change I would probably understand I see less and less reason for myself to work in theatre I have been working in films lately I have been inspired in Ukraine because the situation with the theatre is not simple our theatre world is very small and I don't know where to go next, what direction to develop that's why I don't do a lot of things for myself I see less and less reasoning for myself personally in terms of development in theatre unfortunately in Ukraine I am more and more engaged in working with films scripts and I am more and more inspired by that field generally the situation with theatre is quite complex for many reasons here I don't really see clearly a future for myself in there I wouldn't bet that I have lots in the future we really thank you so much for taking the time to give us an update on the atmosphere from the Ukraine that for 5-6 years lives through war where corona time is a happy break with some kind of relief for most of the world it gives us an idea of what you all go through really thank you for sharing and giving us a hint how it must be for you to create and to work and others thank you for translating and one can feel how serious the situation is I hope that your prophecy is not realising for the Ukraine or for all of us many artists comment that the climate change will be the big thing this might be just a little early rehearsal a gentle one and that what mankind is facing as problems will be more severe and come back and that we will have to change and we will find ways to do that together to participate as Peter Seller said and to be engaged and also to prepare for the time afterwards tomorrow we have a guest from Palestine and Pidan Zaidan will talk to us about the situation there we will hear from Brazil we will talk about the situation also in a country that is facing so many problems and on top of it Edouard Elvismouma and Ermin Yolo from Cameroon will give us an update from how they experience this moment we all do experience it very very differently and today we got a slice of a reality from Ukraine, from Kiev and from her country house so really thank you for sharing and serious about this situation we all in, we all share and we do care about you and everybody who was here in the program, the work you do for the theater and I do hope that it will be a more optimistic time also coming especially for the Ukraine and that their solutions will be found and perhaps this pause might help you said it came to a stop I don't know what lies in front of us thank you all for listening again to our listeners I know how much is out there, how much you all do and how much we have taken up with our daily lives where in a way as Natalia said we are never alone but we also not in the company we are normally are it's a strange situation with many beautiful moments but also complicated ones for everybody complicated, complicated ways of living for many people on the on earth and I think theater has always been on the side of life and has been at the forefront anticipates perhaps also possible solutions and so thank you for giving an insight thanks to Hallround for hosting us out of Emerson College in Boston and the Segal team, Andy Jacqui and Sanyang and I hope to see you again we'll be back to the Segal Theatre and the World Court Commission will work out well and there will be opening nights in Kiev with the work of yours and the community will get together so thank you all and stay safe and stay tuned into us and I hope to hear and see you all again bye bye