So, I am an automation architect and working with InfoStretch for the next eight years. So, basically InfoStretch is in like Ahmedabad, Pune and we are also expanding in Bandor as well. So, in these talks I will cover, so we developed some QE products which mainly based on like we do the automation. So, we have created some products around automation and we have also a product like predictive QA and some AI based tool. So, basically in these five minute talks I will cover, so one of our products is called QMetry Automation Studio. So, before I, so let me just give you, overview of this product through one video file. Quality tools will help you achieve that goal. QMetry Automation Studio, QAS. Yes, I mean this is not so easy. Approach to the software development life cycle for a faster time to market. Today you need to be more agile and work with various DevOps constructs. Today you need to run. Doing this is not so easy, especially when you are working with complex enterprise applications. They are integrations with a need to release features quickly. And that is why we have designed a tool specifically for you to tackle these challenges. So, now you can focus better on testing and finding bugs and our quality tools will help you achieve that goal. The QMetry Automation Studio, QAS is designed to manage any automation effort seamlessly. Whether it's UI, mobile, web or web services automation, QAS supports all these with common standards and guidelines for execution. The Automation Studio offers support for open source technologies such as Selenium and Appium. The tool supports an advanced automation strategy to address significant market needs. It is designed for the manual teams making the transition to Scripless Automation and it supports automation experts with coded Scripps. This tool is designed to support complex mobile conditioning scenarios like barcode scanning, touch ID, location spoofing and more. It offers an extensive authoring support with BDD, keyword and code-driven approaches to offer an approved automation experience. Today you need to support a sophisticated device world with a multitude of test cases, scenarios and Scripps for these variations. The Test Automation Studio aims to solve this problem with its built-in objective spy built for iOS and Android to test and analyze any application with a single click. The fundamentals of test authoring are changing to accommodate natural language for ease. This enables the product and business analyst teams to pitch in and author test cases and manage their projects easily. The Qmetry Test Automation Studio has made BDD and TDD as the de facto standard with its content assist and step viewer to enable better communication across teams. The OmniChannel Digital Enterprise demands a lot and QAS is the Swiss Army knife you need to ensure scalability and comprehensive coverage for your application. The studio brings out-of-the-box scalability containers and leverages the cloud to provide the right CI integration with mobile conditioning libraries and many other modules. No automation effort is complete without the right coverage and analytics. QAS offers a detailed dashboard that showcases all the results of your automation initiatives filtered by various types of attributes. The Automation Studio is designed for DevOps and the same scripts can be used by developers, automation engineers and operation leads enabling seamless DevOps outcomes. Simplify your automation efforts with the Qmetry Automation Studio. Quickly learn and start your new automation project. Talk with us now. Yeah, so basically this QS, so now you've got the idea, right? So basically it's an automation tool. Yeah, so Qmetry Automation Studio is basically an automation tool, right? So where like, you can create a uniform script that can be worked on the different platforms like a web, mobile and the cloud base also, right? So let me just quickly go through with the product itself. So I have this, the product installed in my machine.  So as I said, it's a uniform way, right? So all your resources like object, locators, your script, your implementation, right? It does support different techniques to author your test script and all, right? If you see in my video, right? So we do support like BDD, keyword-driven, test-driven, right? So based on your choice, you can write your script in any like BDD and all, right? So let's say, let me just open quickly one BDD scenario. So we have like an in-built capability for test data manager also, right? So every test needs valid test data, right? So that test data, so this studio can consume in any other format like JSON, XML or TXT. You can put your test data in the external source. So this BDD scenario, so here you can see, right? So for this particular searching, flat with the data, right? So this data I can create in the under the resources. And the similar way, like it's very easy to create a new automation script. So we are offering like some in-built steps, right? So it's very easy, like very quickly you can create your test. So if you see here, like you can use the different checkpoints for assessment, verification. So it also does support like web service. So you just need to define your web service request in form of tabular and it will automatically generate the test case. And it's very easy to create a test for mobile web as well as API, okay? And so it's also like in-built automation framework also, okay? So when I say automation framework, right? So there are like a different concept, like we have implemented in this, like we can create a component. So component means like, so let's say in my application there are some components which is having like a common characteristic, right? So I just need to create one element for that component and that I can reuse in the different pages also, right? So it will reduce lots of your time in terms of implementing your automation, right? So let's say if I open this calendar, right? So it's a most common example. So let's say in every application, the calendar component can be used in the different pages and all, right? So in normal traditional automation, we generally write the locators for each and every elements and all. So by using this framework, like it's very easy, you just need to extend, it's a Java-based framework. So you can just quickly extend with the web component and describe the component characteristic by using some annotation, right? And then I can use this component in any application or any page, okay? So for regarding, so it's leveraging like Selenium and Appian. So we can use all the Selenium and Appian capability through this. So as well as like we have created the mobile library also that can help you to solve the complex mobile scenarios as well, like bar code and if I want to automate the scenario, like deposit a check, right? So it has like an in-built library also. So by using this common BDD steps, we can perform such automation also. Okay, yeah. Thanks, everyone. Okay, great. Thank you everyone for joining. My name is Misha Ivinnik. To actually clarify one thing, I live in Canada, so the topic and the talk that I'll present will be probably a bit different for you guys. So the stress-driven development. Basically here I'll try to tackle the problem of stress we have at work as developers and how I'm personally trying to tackle it and continue my work in a creative and innovative way. So as I mentioned, there is a problem right now, stress that we have at work. And that stress basically arises from the fact that nowadays office becomes our life, especially in the states where they try to make an office as appealing as possible, where they serve you lunches, they build those fancy rooms where you can play games all day. So they really try to attract you and basically stay in the office. So as a result, you don't really separate life and work as much. And it's very difficult for us as engineers because hopefully we enjoy what we're doing. So you say, oh, as a developer, I work extra, I love what I do, so there is no problem. I would actually say there is still that problem of continuously working for your employer. There's also stress that may arise from the fact that you might be a consultant and the consultants usually work 24-7. Or you maintain legacy systems that also add some stress because you can be innovative when you work in legacy systems as much as you would like to. And so the way I try to tackle that stress is by something that people really try these days, especially in North America. It's some sort of meditation. Or there is an app that's called Headspace, which I found a nice introduction to just the breathing techniques that help you start your day in the morning, 50 minutes sitting in silence and really get yourself together before you go to work. It really helps you to kind of clean yourself before you go into that daily routine. It's daily routine that usually adds the stress to your work. So another thing I've tried before is the float therapy. It's also a great thing to try because what it does to you is basically it's the noise and the sense isolation capsule. So you don't hear anything, you don't see anything and you lay in the salt water for like 90 minutes. The whole thing nowadays is that we have so much noise pollution, light pollution and this kind of isolation for a short period of time really helps you to remove the stress. And the reason why I even talk about it is as soon as you get rid of that daily stress, you can actually be more creative, innovative and productive. Studies have shown that 40 hours a week or even like 100 hours a week that some people try to do, it's not going to help you to deliver a better product. Yes, you might deliver a product faster, but again, the hidden cost of any software isn't maintenance. Hence, when you're delivering a software fast with a stress in mind and just pushing yourself as hard as you can to have something on the market, you're probably not thinking as much or as well as you would about the architecture and how to actually write this software in an easy to maintain way. So if you actually get rid of the stress, you will be able to write that software much better.  The good thing I've noticed myself because I have a lead position is to lead by example. So really try yourself, take days off. That's a big problem I've noticed especially in America where a thing called unlimited PTO, paid time off, is a thing where they don't have a set number of days for you to take a vacation. You choose when to take a vacation. As a result, people often don't take any days off at all and if the game adds to the stress, you have to take time off. Speaking at the conferences and traveling is also a great way to get rid of the stress. That's what I found for me very useful. Read the book. It's also a good way to, especially a non-technical book, it helps you to kind of broaden your horizons. And to finish my talk, what I like to usually say at the end is yes, life is short. You can do very little things to prolong your life but it's up to you to choose how wide your life would be. So really do all those things I've mentioned. Find your own ways to work with the stress and get rid of it and maybe share your tips with me. I would be very interested to hear from you guys. Thank you. Hi everyone. Hope you're all having a wonderful day. So I'm Megha. I work with S&P Global. I'm a test engineer there. And I want to take this opportunity to discuss a Selenium Grid cloud automation solution we built to overcome a problem that basically entailed having to run, as you can see, around 1,200, 1,300 test cases every time there was a major build. And these were all UI test cases to be run on browsers and the underlying, the tools that we're using, Selenium, Java, our own test automation framework that we built in-house. So earlier, a year back, we were running all these cases on an on-prem Selenium Grid, which basically consisted of powerful Windows machine acting as the hub and to which were connected a variety of Windows VMs, few Linux VMs. And we were basically sending all our 1,300 test cases to this hub to be sent over to all these connected nodes. Now, as you can imagine, as test automation engineers, we already have our code base to maintain. On top of that, you add the infrastructure that also needs maintenance, upgrades, hardware upgrades, all these changes from time to time. So as a solution to that, as a workaround to that, and there was also the problem of limited hardware. Having to run all these test cases, we were limited in the sense that we had 20, 30 VMs to run these on. And once the load actually increases, the CPU and memory use its spikes, your tests slow down. There's nothing wrong with the application, it's slow down. So the next step we took was to dockerize our framework, run the test cases in Docker images, customized Docker images. So that helped us to quite an extent. There was less maintenance headache because Docker, right? And every time the load went up, Docker was zippy, it was fast. It's any day lighter than having to run on a VM. So that was that. But even then we were heavily limited in the sense that the infrastructure on prem was still not able to facilitate, was not able to handle all that load. So we decided to host our grid in cloud. And for that, we did a lot of POC. We started from the very basic that, you know, we are going to spin up EC2 instances in AWS cloud. So we began from there. We started doing our POCs. Then we thought about ECS. Then Fargate was launched around the same time. We started thinking about, you know, hosting on Fargate. So I'm going to talk very quickly about that. But before that, I just want to launch our containers on cloud. So as you can see right now, it's a 503. So we have a Jenkins job, which basically upgrades the Fargate services, which are basically three in number, one service for the hub, one service for Firefox, having, as you can see, 15 containers, and one service for Chrome, having five. Because most of our test cases are, you know, meant for, are supposed to be running on Firefox. So I'm hoping in less than two minutes, the grid should be up. Yeah, so I was talking about the limitations we were facing with on prem, you know, hardware. Every time that number of test cases would increase, or, you know, we have to run the same test cases multiple times, the infrastructure would completely give up. There was cash getting created. There was a lot of maintenance headache. So we wanted to get rid of all that. So we thought of using Selenium. We thought of using AWS Fargate. Now Fargate is AWS solution for running your Docker containers. Some of you who may have worked on ECS, may be wondering how is Fargate any different from ECS?  You have to spin them up. You have to maintain them if there are patches required, anything maintenance related required. The effort still goes there. Fargate is one step better in the sense that there is no maintenance.  You don't have to look after them. Fargate will do the job on the fly for you. It will create those instances. You just have to, you know, be very particular about what configuration you want. If it's taking time, it's a good sign. While it loads, I'll also touch upon another aspect of Fargate that you are charged for the amount of time that you're running your test cases, right? And you're charged by the second. So if your test cases, right now, our test cases take anywhere between 50 to 60 minutes. With this, we have cut down the time by one-fourth right now. So as long as we need the instances, we keep them alive. Once we are done with them, we shut down the services, and, like, in a day, we're using it for barely 15 to 30 minutes, and we're charged only for that. It could be because, you know, of the internet. And the last thing, it won't start right now. I don't suppose it would. But the last thing is there's also auto-scaling involved. So suppose the request comes in for, you know, running the same batch of cases simultaneously on different environments, you know, that load has doubled. So there's auto-scaling logic also involved, which will add your predefined number of containers to the grid, and that will happen when the CPU and memory usage spikes. It starts touching 80%. So AWS will automatically detect that usage at the containers so that your tests are not skipping. They don't die down due to lack of containers. And basically, they're not in queue for a long time. Sorry, guys, I lugged out today. There's no internet. I mean, the internet is not on my side. So if you're interested in the solution or you're looking for something similar, just let me know. We can talk about your offline. Thank you. Hi, I'm Shridhar. I'm working as a QA in ThoughtWorks Technologies. Okay, so today, the whole day, we have spoken about mobile in one of the tracks. So I'll just quickly show you how to inspect an element on one of the mobile apps. Has anyone used it before? Has anyone tried this? Inspecting an element on mobile apps? Android.ios? Yeah. So yeah, I'll quickly start in the interest of time. So I'm using a tool called apm desktop. So after this, I'll just quickly share an experience. We had a specific problem and how did we overcome that? I'll just quickly share it after this. So I'm just starting the server. You can see it, right? Okay. So first I'll show how to do an iOS one. So these are the basic desired capabilities which we have. So what's a desired capability? So we know apm is a client server thing, right? So client server application. So the capability is the one which says, so for any client server application, we need a session. So the desired capabilities is the one which the client says to a server like, I want this type of session with the server, right? So that's the, in basic terms, that's the desired capability. So in here, I have added a few basic ones. There's the app path. So the app path is again for simulators. So it should be a zip file or a PPP file. If it's an iOS device, then it should be IPFI, okay? So platform name, automation name and device name. So version and also UDID. So I need a UDID of a simulator because I don't have any real devices connected. So I'll say just instruments, instruments, iPhone devices. This will give me the list of simulators available on my box. I will just select one. So UDID is equal to this one. So this will short me an app which is in that path, the zip which I have saved there. So this works, the APM desktop works with both iOS and Android. Yeah, so this started, booted me a simulator and it's launching the app. Okay, so this is the app and it will launch me the inspector here. Yeah, so this launched me the inspector. So in here, I can just select on this like similarly how we inspect the web elements. Similarly, we have to do it here, right? So if I see here, I can see the label as user name, name as user name, value as admin, like all these things I can get. And I can just use the APM like find by iOS, find by Android, find by, and I can just select this element and I can do the actions on it. Right, makes sense? So this is how we do it for iOS. So for Android, we can do it using the same tool, using APM desktop. But let's explore something different. So Android SDK provides something that you can do by automator viewer. So this one ships along with the Android SDK. So we can use that to inspect the elements on an Android application. So I will just go to... So this used to be there in Android home, but with the latest version, they have moved it to bin tools. Yeah, so this is the Android automator viewer. So if I just click on this, so I have an Android simulator open here. So in Android, there is no difference between a simulator or a real device. It's just the ADB command which works for both. And this one will get me the current snapshot. So if I have to select something, if it's some app which is open, then I can just get the... So if this is the app, so it gives me the package name, content description. So content description is nothing but the Android... Fine, I will just quickly explain that later. So this gives me all these details. So we can inspect the element using this tool and we can just perform the actions on it. Okay? Yeah, so this is what I wanted to show on the APM desktop. So one more thing is, currently I'm working with a client which has web application and also Android and iOS native applications. Yeah, one minute. I'll finish. So yeah, I'll quickly show you. So in the web application and the native application, the flows are almost the same, except some customized like my account kind of applications, my account kind of pages. So what we did was like using APM we can find, like you can see here, say like there's a single mobile element which I can find it for web, also for Android and also for iOS. So using the APM field decorators and also multiple strategies are available to find the allocators here. So like we did this and also say there is something to say, I would say like we have to read an alert, we have to read an alert. So the way we read an alert in iOS is very different from Android and also in the web. So how we did that was like we added an interface which provides the runtime polymorphism. So if it's an Android driver, then do it in a different way, like use a different application, like use a different element for it. If it's iOS then use a different thing. So like this we solved that problem. Yeah, thanks for this opportunity. Tested firsthand and some of it that I haven't tested firsthand. So the first item I'd like to mention is Aspire. So following on from Dimitri's talk on stress-driven development, Aspire is a wearable device that monitors your breathing. So you can monitor your stress and whatnot. I actually have two of these devices. I would normally bring them around to demo but I didn't pack it today as I came here. It's an app. You can see your breathing in real-time as well. This device connects over Bluetooth with your phone. You can either wear it on your waist pocket or you can wear it on your bra strap. I'm looking around. I don't think there's a large amount of the audience wearing bra straps here. I don't know. You could all be wearing bras underneath. I'm not judging. No judgment here. Hashtag. It can give you notifications when you're tense. I don't know if this might cause more tension for some people. But I found it useful. There's been one or two situations where I'd been in a doctor's theater surgery and I had this little device vibrate against my skin to tell me I was stressed. Unfortunately, I was in a situation where I couldn't get out of that stressful relationship. I imagine if I wore it here in Bangalore, it'd be buzzing every time I walked into traffic. Probably. It can give you stats over the day.  I think nearly every wearable these days can monitor your step count and how long you've been seating as well. So you can monitor more other elements of your health if you like. Why did I stop using it? I found when I wore it on my bra, it would actually rub against the skin and cause discolouring. So that wasn't all that great. I moved as well and misplaced the charger. I ended up getting a replacement. It also wasn't working very well with Android Oreo at the time because it was Oreo had just been released. So it's one of those things that I've used once or twice, at least for a month or two. I really enjoyed using it, but it's one of those things you buy once and use for a bit and then get out of practice. I would like to practice more mindfulness techniques, so I might recharge my spire and start doing it again at some point. But that might be one combat to the stress-driven development. Monitor how much you get stressed during the day. Another device is the Android Wear. When I was a contractor at Google, they gave me an Android Wear device to help with testing. So I wouldn't have gone out of my way to buy it myself, but they gave me one to test with. I didn't like it. It was annoying. There's a reason why I hardly used it. Has anyone used a wearable watch before? Hands up. Keep your hand up if you're still using it. Yeah, that's actually not too bad. That's better than I expected. I'd like to experiment more with wearable app development at some point, but I wasn't super impressed with that. Something slightly related is not really wearables, but I was experimenting with using the Nike Runner to monitor my running. It uses a lot of similar technology with what you'd use wearables for. There are watches and whatnot for this. The reason why I stopped using the Nike Run app was I broke my ankle about seven months ago. You don't really do much running with a broken ankle. But I would like to get back into it. I liked a lot of the data it showed. It was all free. They had training programs you could sign up for. They had this every Sunday that had a 5K challenge where they would tell you how many people were doing a 5K challenge. And it would kind of motivate you to just go out and run for 5K. And it was nice to see improvements over time, gamifying some of those stats. PeriFit is for women's health. It is a device that you wear on the inside. So, guys, you can't really test this one out. All that is loose. It's used for... It is used for pelvic floor strengthening. So, you wear it and you squeeze the muscles and you see this butterfly go up and down. And you're supposed to collect little flowers. Sorry. Is that... Am I at five? Okay. If you're an all-male dev team, I think it might be a little challenging to test this in person. But if you do overcome those challenges, let me know. But thank you. The problem we were dealing with was innovation. And function, innovation is equals to... So, I want to write this because this is something... is the key for what we've figured it out. Is function, innovation is equals to the organization that people you have into architecture. The architecture that you put them into. Keep our mechanisms into culture. Right? So, the biggest pain point here was nobody wanted to write this. No, not a single engineer wanted to write this. But there are a set of engineers who really wanted to write this. And those are all engineers who had worked in open source. And we saw people who actually contributed in open source. They really understand the value of tests and like craftsmanship for writing code. So, then we started asking everybody to contribute in open source. And suddenly everybody started realizing, okay, probably, yes, it's good to write tests. Right? Mechanisms. When you give people processes, simple processes, they understand, when they do it multiple times, they understand the value of them. So, we had no processes. So, we started putting a lot of processes, like from test development, retrospections. All of that kind of led to an answer that we need to stop wasting time like releasing stuff. We just write small code, right? And we just test so many things across mental platforms. That... And then we moved to an architecture which was bought. Essentially, each part has a product manager, a designer, a set of engineers, and they own a problem autonomously.  And they can take n number of months to solve that. But the problem is your component are not like this. Your components are shared. So, how do you solve that? So, that's it. Looking at this and this, we kind of created a team called cd slash ci. And we looked at our architecture as any time we need to solve a problem, we need to create a different pod which productizes a problem. Like for example, we productize this infra at point of time. We now have productized continuous-driven integration. We've productized writing tests. We've built in-house frameworks to make tests very easily. We've built frameworks so that everybody can manage their components and still make staging available for everybody. But it was not simple. I think if you guys want to bring that change, focus on culture, if you kind of make people, make engineers see and contribute in open source, that would be a big leverage point. So, I hope this kind of helps.


Welcome once again, welcome everyone. And please, if you don't speak, mute yourself. And otherwise, I would like to start with a very quick introduction of Bettina Rill, who's our speaker today. Bettina has been working as a patient advocate in the melanoma field for several years now. And before that, she was a medical doctor and a scientific researcher. She also holds a PhD in medicine research. And Bettina has been doing a lot of research and work around the theoretical framework of patient involvement in scientific work and biomedical research. And she also does a lot of very practical stuff with the melanoma community and beyond. And without any further ado, I would suggest that you take it away, Bettina. I start the slides. There you go. And you just let me know whenever you want me to advance the slides. Thank you, perfect. I hope everyone can hear me well now. And I would say we just get started. So what I would like to share with you is about the next hour or so is why I believe that thinking like a scientist is something that is of enormous value to us as patient advocates. And in the next 45 minutes, if you could jump to the next slide, please, I would like to give a brief introduction where we should care about science in the first place. Why is it worth our time? Why thinking like a scientist gives us the power to support our friends with evidence and apply that relevant to our field? As one part of one field of science, because science is obviously a very, very broad arena that is particularly relevant to us is medical science, of course, because we are all in cancer. So we will for sure spend a lot of time today on the medical science. And I will go into detail about how one can actually get started in reading original scientific literature or some medical scientific literature. There will be a very, very, very snapshot, like a very brief introduction to the Esmall magnitude of clinical benefits scale, because I think that is a useful tool for us as advocates. At the time, we're not allowed to really go into depth, but the link will be on there. So I just would like to give you an impression of what it is about. And then, of course, we're going to sum up the next slide, please. If you could move to the next slide, please. Okay. So why should we care? Why should we care about science? Now, there are two aspects of why we should care. One is about individual patients. We all, many of us are patients themselves, or we do help single patients to cope better with their disease, to get better care. So if you really truly want to help people, it is often that it is the latest scientific developments that make the biggest difference for patients, especially when they're in difficult situations. And this is usually when they come to us, when they're out of treatment options, when they tried everything that was out there, or, however, on the opposite side, they've just been diagnosed with a disease and don't know where to go. So understanding what the latest scientific understanding is allows us to give good help and provide a context that helps patients make better decisions because they are able to discuss with their medical team. We should support evidence, not eminence-based science. This is something I actually had the time to be able to add another slide, but I already had too many slides. But this is something that I have actually borrowed from the trial workshop that I teach every summer, where also, junior oncologists are told to place their decisions based on evidence and not on the eminence of other clinicians. True patient empowerment is, in my mind, impossible without certain understanding of the scientific area we are moving in. If we really want to have impact and impact the decisions around us, we have to first understand what is going on. And obviously, it is a protection from quackery and form hopes. And someone in my network said knowledge is protection, and that is something that really resonates with me. So it is important to help a single person. Next slide, please. Next slide. Okay, super. No, no, that's as Advocates. This is perfect, thank you. So now we're here as Advocates. You are in a special course, you're already a part of a particular training program. So we're not just patients, we're Advocates. So why does it matter particularly for us as Advocates? Now, if we want to push for progress in science, and this is the ultimately where our cues will come from, hopefully, one day, it is the medicine that will bring us there. We can only influence what we understand and what we know. The science, how we currently conduct science affects us. Anyone who has ever been on a clinical trial, we know that it is the trial design that defines your experience on the clinical trial. And so most, yeah, I know I've seen from the names that some of you know me already. So you know that clinical trial design is a topic I'm passionate about. So the way a scientist conducts it, so the methodology affects us as patients. We cannot change them if we don't understand what science is about and by which rules it is conducted. It is impossible to provide accurate understanding or accurate health care information to patients without understanding what is going on. And if we are able to support our advocacy plans with evidence, the chances that they will be successful is way harder, a way higher, because it is harder to discard evidence-based claims than if it was just an opinion. So, and apart from that, I believe that as advocates, we have to question ourselves in what we're doing and we have to ensure that what we are doing is really truly in patients' best interest. It is not just our personal experience, because as a single person, we can never, ever be representative. We are just one person. We have a certain view of the world. So we have to kind of, like I think of it as an insurance for ourselves and back insurance, that what we do is truly for the benefit of the community. Next slide, please. So to sum up, scientific thinking is the basis for evidence-based advocacy, and that is why I believe that we have to care. Now, next slide, please. The good news is that you do not have to have a scientific degree or wear a white coat as Ben Golderka in this Guardian article said and he will share the slides afterwards so you have the link to the article and it's definitely a really worthy one. Science is not about authority. It is about the method, and that basically means that it is a matter of being willing and able to learn about the method and it's not about getting a scientific degree in order to be able to follow. Next slide, please. Next slide, please. Perfect. So something that for us is important and that I think is a fundamental kind of way of thinking about people, about knowledge and about learning. This is something I'm very passionate about. Knowledge is infinite and no one needs permission to learn. So it is not about what should patients know. In my mind, it should be what do patients want to know and what is sufficient and what is too much. It's up to the learner to decide. Of course, we as advocates might have an opinion that we know from experience and unidiagnosed patients will fare better if they know this much.  And I think as advocates, it should be our ambition to provide and encourage people because these people will go and talk to their medical teams and they're surrounding in a very different way. They will be able to argue at a very different level and they will be able to enforce their positions to a way higher degree than if we just give them the single leaflet that was spetted by someone else who has never been in their shoes. Something that we have to really, really read, however we be aware of, and I like this Albert Einstein quote from there, everything should be made as simple as possible but not simpler. There is a certain level up to which we have to bring our people because certain things cannot be simplified without losing accuracy. So it is about finding that in that interface to make it as simple as possible. And that is actually extremely hard. Simple is hard. Putting complicated context, complicated content in a simple accessible way is an art. So there's a real challenge for us because our audience tends to be diverse, they come from different backgrounds but at the same time, our audience is highly motivated. Next slide please. I believe that it is what makes us different from many other learning audiences, it's the motivation and the effort that people are willing to put in. In my network is people who are willing to spend their weekends over extended periods of time to learn and we have had clinicians coming to our events. We always have educational sessions and the feedback we always always getting is that they're amazed how well educated the audience is and how evidently they learn. So I think that is something that makes us unique and something we should be proud of and encourage. And this is something that I actually send this slide and other Albert Einstein is something that I send to our faculty. It's like, if you can't explain it simply you don't understand it well enough and that's the challenge I put them under in order to deliver speeches, talks that are meaningful to the audience. This is actually that it's not on the slide but something for you to remember the moment you go into organizing events and trying to share and spread scientific knowledge it will be up to you to moderate the discussion and to breathe your faculty and that is a true challenge because people are usually especially medics and not you can be bossed around by anyone and they usually do worse presentations you can imagine. So a classic example is you give them a 15 minute slot to talk until they send you 45 slides. You know from the beginning that it will never work. I count 1.5 to two minutes per slide which means 15 minutes is 10 slides.  Please put all the relevant stuff in the first 11 slide. So this is something to consider it is about knowledge but it is also how we make this knowledge accessible and share it and it will not be possible without putting considerable effort into the educational aspect of knowledge. And to add to the bad news next slide please. It is possible but it does require effort and it is a skill. So without effort without input it will not be possible to sail on the forefront of science. So it is possible, you do need a degree but it does require effort and time to put into it. So without I would like to thank you for being here in the first place because this is a pretty technical and I call it slightly nerdy topic. So I really appreciate your interest at this late time and so close to Christmas where I'm sure we would all be or would like to bake cookies or something similar help. Next slide please. So I would like to now start with some principles of scientific thinking that I think are absolutely crucial and fundamental. For me, this is the essence of modern science and I would like to give you a bit of context why I'm because I believe that this is really, really important. Modern science is based on the principle of falsification, which I call the art of nitpicking. The fundamental, and of course this is kind of a bit tongue in cheek, but I would like you to understand because it is crucial to understand how modern science operates next slide please. It's, oh my God, on the right it says like I blown it up too much. The underlying principle of everything we're doing now and the underlying principle of falsification is the realization that we only ever see a part of the total truth. And you might be aware of this Arabian fairy tale about the wise men who are arguing about who knows the absolute truth. And there's a wise blind man who makes them actually meet an elephant. And usually the original story is that the elephant is in a tent and every one of the other people gets to touch a part of the elephant. So someone as you can see on the right, someone touches like a leg, someone the tail, someone the nose, and everyone comes back and has a totally different story to tell. And that's a bit summarized on the left. That some, like I think the one who touched the leg like compared it to a column, there's someone who touched the tail, compared it to a tree or like a branch or something like this. And in the end, while everyone was right, no one had the full picture, so no one came back and said, I have touched an elephant. So this is a very, very powerful, in my mind, powerful image and something that we should be aware of. Our understanding of the truth is only ever incomplete and partial and we have to treat it as such. Next slide. In the Middle Ages, when there were the first movements towards scientific approaches, what people did were they tried to verify their claims. So verifying means you look for evidence that supports your beliefs, while falsification is just the opposite. You are trying for reasons that that's basically true that you are not wrong. This might seem very similar, but it is the difference between right but not being wrong. The verification part of the story is that people will only consider evidence that supports them and will ignore evidence that basically shows them that they are wrong. We can also call that cherry picking. Falsification, on the other hand, will go looking for evidence to prove that one is wrong. At the point where we no longer find evidence that we are wrong, we assume that we are kind of right for now because tomorrow there might be something coming along that shows us that we were not right or not entirely right or that we got aspects wrong. In a way, it's a kind of insurance for ourselves that we are not overstating what we believe. So when you now, and that's the next bit we will go into, is how to read actually an introduction, how to read scientific literature. It is an approach and it sounds quite negative, but I think it's a healthy critical thinking. It basically means that we are looking for reasons not to believe the authors of anything we're reading. If we cannot find anything, we will go, yeah, that looks kind of all right for now. And that is the basis of modern science, of modern scientific thinking. This is also the difference between a typical quackery or fake like pseudo scientific claim for be it green tea or any type of snake oil. If you go through there, these people usually just, if they cite evidence at all, and usually that is like not really true either, they will only cite support in claims. Never things that disprove them. And that's the difference today between true science and pseudo science. Now, I always say science, next slide please. Science however is a huge term and it bridges a vast majority of different areas. And then two big differences in there. And I really like this quote that not everything that matters can be measured. Not everything that we measure actually matters. And out in a way is the difference between quantitative and qualitative research. Next slide please. Quantitative research, as I said, is anything that can be measured where you can attach numbers to it? Well, qualitative research is non-numbers. If you want it would be things like surveys, interviews, focus groups where you just cry your results and the result is rather a text and not a number. Next slide please. If you start, if I start to think, I started thinking about which fields of research actually purchase as patient advocates. And it's a huge number and I'm sure this list is not complete. If you start very early on, fundamental biological research, basic science is relevant to us. Maybe not the most relevant, but it is relevant to us. The most relevant probably for us is clinical medical research. So these are clinical studies and the publications about clinical studies. However, also epidemiological research. So anything that is about population-based research. How like for example, the frequency of cancer in certain populations, general health issues in populations. This is epidemiology. Have systems research. How well do our health systems perform? What are waiting times for our patients? How long does it take to receive a diagnosis? Equally affects us. Have economic research. I'm sure that everyone of us are now is aware of the economic implications and the challenges, especially in oncology. So we are affected more and more with health economic research outcomes. Sociological research. Anything that comes in the field of behavior, anything that goes more into like how people perceive things is if you want to support a sociological research with its own methodology. Political research is yet another one which all falls under like the big umbrella. And these like sociological and political research, they do a lot of quantitative research. They don't do so much like qualitative or they have a mixture of it. And sometimes what one tries to do is try to quantify aspects that are actually at the base through qualitative. Now, if we want to continue now to the next slide, I would just like to give you a brief introduction on how we started in my own network. So as Tamasha really said at the introduction, I found it on a number of patients in Europe and because I'm a medic by training and then work years as a scientist in the fundamental biological research lab. Science is close to my heart and I've done a lot about it. And as a PhD student, you get tortured at a sense. So how to properly read scientific papers here. That was a bit of motivation or well a bit of background that helped me to design this workshop on how to read scientific papers because this was a demand that came up on our forums all the time. People were asking all the time, how do I get started? How do I do this? Where do I find this paper? So that we decided to run an entire workshop and this was like two and a half days on a single study on a single publication. And we took obviously something that was relevant to us in melanoma and really took it apart. Normally as a PhD student, you will have one hour a week where you just have some paper. But of course what we did is we took it into parts and then provided the background for people to understand what each of the figures meant and to understand the statistics behind them. And it turned out to be a really useful exercise for our network and has helped enormously to raise the level of the discussion on our forum. So this is something that I would highly recommend. And I'm really happy to find that you are under the link below there. You'll find the program of the workshop. So feel free to copy. It's not, you know, it's better for everyone. Now, next slide please. Why would we actually care to review original scientific medical literature? Why go through the pain? Why don't we just rely on the guardian or anyone else to write it up in a more digestible form? For me, there are three aspects of it that are relevant. The first one is a time aspect. Scientific publications are the first thing that appear before anyone shares it, translates it, it goes all over the place. So this also means that this is English. Scientific publications and medicine, the important ones today are in English, whether we like it or not. Also, whether we like it or not, not all physicians are up to date on the latest progress in our specific disease. And that matters especially when it is a rare condition or it is a condition where patients simply don't have time. So if it is a time sensitive issue, you cannot hope or you should not hope for your physicians to be able to speak, you're way better off to read it yourself. The next point is correctness. You cannot rely on public media like the Guardian reporting correctly on medical findings. Often what happens is, and Ben Goldberg has worked on that and published on that, is what happens is that scientific publications often end with a statement about the future, like a forward-looking statement as it is called in any investment context. So it's like the vaguest, most speculative, biggest hope at the end of any paper. It goes something like that and then we're gonna save humanity. And this tends to be unfortunately what the general media picks up and turns into the headline of the report about the paper. So the most speculative and weakest part of a scientific publication turns out to be the headline of a general media report and that of course is an issue as such. In addition, by relaying our lay versions, these often often make crucial details. It does make a huge difference whether a result was found in a person or whether it was a fish or a cell in a dish. You would be surprised what lay summaries often omit. So by reading the original publication, you can protect yourself from that. And the latest one is empowerment and great power comes with great responsibility. Knowing what is going on is part of it and now knowledge is protection, but knowledge is also power. So in order to be truly empowered, we have to understand the science and empowerment in the sense of having an educated communication with your medical treatment team. Only by understanding what is truly going on, you'll be able to negotiate as equals. So it is for me definitely worth investing the time to get there. Now, we will go now in the next part a bit through the practical aspect about how to read actually these papers. Because first before you can read them, you have to find them. Now, where does one find scientific papers? Scientific medical papers.  The link is on there. PubMed is a publicly available database. It's based on Medline, which is a repository, but what we all use is the service. You've got a layer, you've got the search field and actually PubMed has produced extensive how-to guides on their own website. I put the link up there. Please have a look. Better you know how to search the more you're going to find out. But this is the reliable source to go looking for scientific papers. However, next slide please. Even PubMed, which will not give you ads about dubious products or snake oil, which Google will still return for you. Still, even PubMed is only as good as you know what to look for. So in order to be successful in your search, you should look for a specific about three points. Whom you're looking for, what you're looking for and where you're looking for. And I'm going to go through each of them now in turns. Next slide please. So the first one is whom are you looking for? If you look for scientific publications, you will find the authors of that publication listed straight under the title. I should have potentially a screenshot here. Medical research has a code how articles, like how authorship works for articles. How it usually works is that the one who did most of the work is the first author. The last author is usually the one who runs the group and who got in the funding for it all. In different stages of your career in clinical research or medical research, you need different things. If you do your PhD, you need first-author publication. If you want to show that you're an established researcher and able to run a lab, you'll need last-author publication. So you get a feeling for where people are in their career and how important they are by looking at the authorship position on a paper. Now, which name do you want to put in PubMed? The first one I would recommend is you look for the established authorities in your research. If you don't know who those are, go to the conference program that is most relevant to your cancer. So you can go to ASCO, ESMO, ECHO, or I think for you, for in myeloma, ASH is probably even more important for myeloma SMR would be an interesting one. Go through the program. Don't just take one, take two conferences, three conferences, and you would see like conferences from different years and you would see that the names start repeating themselves. Usually the groups are not that large. You maybe end up with 10, 15 names if at all. Take one of them, start looking. If you want to do a quality check on your treating oncologist with that in department as well, then you know where they end up, they're in the hacking order. It doesn't mean that the ones who don't publish are bad, it just means that they're not, they don't have the influence because it is in the end by a publication that people are influenced on what is considered the best standard of care. Now, you have found that person and this is the person you wanna put into your PubMed search. Now, still, lots of these people publish lots of different things. Next slide, please. So you wanna look for what and something very specific. What you wanna look for are the big unanswered questions and also going through the conference programs, as I said in the earlier one, we'll help you identify those if you don't know that already because usually we don't start reading medical papers because it's so much fun. It is because we need answers to a very concrete, very pressing situation. So for us, and I know I'm gonna give you an example, it would be resistance after immune therapy. Patients who do not respond to immune therapy become desperate and the next options are not so clear. So this would be an area where we are really, looking for answers. Also too, if you still think they find it hard to define, it is very, very, it's worth going to medical meetings and see whether they don't have bookmarks or articles about heart meetings because they usually pick up the highlights of the conference which would always be a reflection of the big unanswered questions and the most relevant questions. Now, for a standard stream, this is like for the big new unanswered questions. If you are like in a scenario where there are already a standard treatment, guidelines are really good source of information and you can go to the societies of the disease, like ASCO, like ASCO or like hematological societies that will have published guidelines. Those would be a useful place to look up once the treatment has been established. In order to make it into a guideline, however, there needs to be enough evidence of a certain treatment. So this does not help you if you have a reason or like what I call the edge program. So the really challenging bits usually don't have enough evidence that it may be a guideline. Has someone put the, there's a background noise? Is that okay? Tamash? Yeah, I can hear the background noise but I don't know where it's coming from, so. How it's gone? Yeah, I don't know what it was, I'm sorry. Just once again, please, everybody mute yourselves if you're not speaking. Everybody but Bettina should mute themselves. Perhaps the two people are not muted. So. It's better now. Yeah, much better. Thank you for that. And now the next thing is what I call where next slide please. PubMed is already a really great repository. It's not as bad as Google but even in PubMed you find endless papers and it is neither recommended nor possible nor even advisable to read all of them. It's a typical beginner's mistake for PhD students and I can tell you I read a lot of rubbish when I started. If you don't wanna go through the pain of doing that I think it is a smart approach to start with the highly regarded established journals and in medicine that would be the New England Journal of Medicine and the Lancet. Basic research you wanna find in nature or science and then there will be usually respected the electricity specific journals like blood for hematological diseases. We have some Malanoma journals that where we would go and what is constantly highly regarded. If you limit now you put in the person who believe is relevant in your field you have found the topics you care about then start with the highest ranking publications and start reading them. What the help stand is once you have found a paper there will be references to other publications and then it is wise to follow those references and basically build a tree of knowledge if you want. Now I said highly regarded and how do you kind of know what is highly regarded or not. Next slide please. There's a way how we can measure everything and even remembering that I said not everything we measure matters and vice versa there actually is a factor that is supposed to measure the quality of a scientific journal and this is called the impact factor. Now it is a combined factor between how like it's supposed to measure or capture the quality of a journal and it is based on how often how often articles published in this journal get cited by others and how long that happens. There's something that is just hot for like a month or two and then disappears is not as important as something that will be cited for 20 years. Now there's a huge ongoing debate and there's a link I made you can read up on it. There's a huge ongoing debate whether impact factors are the right thing or the wrong thing and you always hear this kind of oh this is not important we shouldn't be measuring this but then the next thing is that today scientific careers are made on impact factor publications. So for professorship in Germany one nature paper will be sufficient. If you wanna go to Harvard or other places you need more than one nature paper. So while there are discussions around it the thing is so the impact factor is not without yeah without issues it is still a good rule of thumb to find things like journals that are trustworthy and discard ones that are less trustworthy. So just for I think something like Lancet or New England are now in the order of 50. So if you have a journal that has a 1.5 you know that it is probably not worth your time. So that is more if you wanna read in addition to New England, Lancet, Nature Science and your Journal of Interest. So if you are really not sure you've read something it sounds too good to be true and it's in like a low impact journal it is wise to be cautious. It's also I think a good exercise to generally look at publications in publications in low impact journals and then compare them with high impact journals probably publications and then see what the typists look like because you will find that oddly enough it is in the lower impact journals that they will promise you to save the world while high impact implications often have very modest rather boring sounding titles because they're way more specific, way more well they're concrete in what they claim and that is why they're high impact in the end. So if it sounds too good to be true with insurances it most likely isn't true either. So a word of warning to that. Next slide please. Now once you found this article and you now know that you want to read this unfortunately what happens now is that often these articles are not just free for download so you just don't click on the link and you have the full thing. Unfortunately many of our high ranking journals as they pay world so you pay for access to the journal. That might not look fair to you but unfortunately this is still the way it is. Now there are always ways around any access issue also to papers. We listed some the open access button is a bit of a pain but you can like if you look up the link it will explain how it works it is a way to get your hands on a paper. I actually prefer I think it's easier if you look up the first author usually the affiliation of the author is listed there together with an email address. Drop them an e-mail say you read you would love to read their article because and ask them to send you a PDF that usually works. They tend to be flattered about the interest you usually get the article very very fast. Another way would be research gauge you can register your organization under research paper free and then you can upload your own publications that you will hopefully produce in your network and also ask authors about a copy of their paper. Authors can upload copies of their paper for free and anyone else can then download them for free. These even the paywall journals give authors the right to share an earlier version of the published article for free so that you will always get your hands on. Usually the only difference between that and the final published article is that the layout is not perfect yet and it doesn't look as pretty but the text is the same. Some journals offer free access for patients like patient access. Read up on there and then you can only have to sign up and then you get the papers for free. It's also worth to have a good connection to your oncologist and just send them an email. I do a lot of that. They often have access to other institutions where the university pays for access and they get the PDFs and can send them to you which tends to be the easiest way to get access. At the same time it's also a nice hint that they know what you're reading and might read it themselves so that could have a double function.  Now we go about how to actually read it and read it that you get most out of it because I assume we all read well enough. It's not about just reading it. It is about dragging out out of this entire information what truly, truly, truly matters. Now the most important point of any scientific paper is the results section. The results in a medical paper are usually presented in figures. So you should focus your attention on the figures and the results section before doing anything else. Because if you start with the conclusions a typical beginner's mistake is to read the abstract whether that you have to do in order to know what this is about, the introduction and then the summary and the discussion. That's the point where you fall for the author's narrative. And not everything that authors claim is also correct because they obviously have a vested interest to look as good as possible. So if you don't want to fall for that you have to make up your own mind first before reading what they think and then you will see whether you agree or not. And you might well find that you missed important bits and you got it wrong but then in itself it has been a valuable exercise. And often it actually happens that we do not entirely agree with the authors and that's where we have these valuable points in progress. So it's important not to be blinded to the beliefs of others. Now, next slide please. Scientific papers are not random pieces of literature. A scientific paper has what I think of an anatomy, a backbone and it will always be structured the same way. There's slight variations between journals but as a base you can say that the paper has an abstract which is the short bit of short paragraph summarizing everything that's in the paper. An introduction that gives you the background and the motivation for the study. Materials and methods that describe is like how the results were generated basically the cooking recipe. The results section which I already said was the most important point that's the meat of the paper. Then there's a discussion, what the results mean according to the authors and a conclusion. Materials and methods are the section that can move around. Some papers have both materials and methods to the very end because it does break the reading flow. If you're just like after reading the introduction I'm really excited about this material like these results or this story and then the first thing you start reading is and then we use five microliters of this or that that is not very much. So materials and methods are often now the last final section. But this is usually what differs. Some papers also have like a discussion conclusion as in one section but the base is always the same. Next slide please. Now we will just one word of warning. Results are important and especially in medicines these are statistical results. There's always this quip among scientists. Don't trust statistics you happen to fake yourself. In order to really truly understand whether or not you can trust statistical results in any given paper you have to understand how they were generated. So while materials and methods might not be found to read there are actually a crucial part of any scientific publication because only if the methods that were used in order to generate the data you were looking at the results can be valid. A wrong methodology will mess up the results and then you will not be able to trust the results. So the devil is in the detail materials and methods are dry reading material but there are important. And I always joke that having a statistician as a friend is a very, very useful. It's worth a lot. Having someone who can help you go through the statistics and explain them is really, really helpful especially for people like us. And we have now started having regular statistics workshops at our conference for example. Now my personal recipe would be the next slide please. This is how I tend to read the paper. I start with the extra what is this about. Then I go through the figures to see what the data actually really looks like and can I really understand from the figures what this is supposed to be all about? And if I don't, that's already not a good sign because usually people put a lot of effort into the figures and the figures will tell you what the bones or what the meat is on those. Then I go through the results and see whether there's something I missed in the figures and whether I have already actually seen everything. If anything in there is unclear, I will go to the methods and the materials and see how it was generated and whether this can explain to me why things looked the way they looked. If there's something I wasn't clear or that I still wonder about, I will then go back and read the introduction. Sometimes I jump back in the middle because you might look at the figure and go, why did they look at this? And then the introduction will hopefully provide you with a reason why they did that. Then only I will read the discussion and the conclusion and then come to the conclusion whether I agree or I do not agree. And then, as I said earlier on, I was sigh at the last sentence and I will worry that this will be the one that will end up at the next general media or mass media publication and which patients will then fall for. Now, next slide please. One quick word and we don't have the time to really go into detail about this is the ESMO MCBS on Magnitude of Clinical Benefit Scale. I put it here because I believe that it is actually a very useful tool to help systematically about scientific publications and this is a very specific and important trial. Funding the mayor where there's a lot of material on the ESMO website about why this scale was constructed and I will briefly give you an introduction what it is and how it was and why I think it should be already detailed. The Magnitude of Clinical Benefit Scale is a way to evaluate findings from clinical trials. We have the background on again. Did someone unmute themselves? No, actually not. I don't understand what this might be, but it's gone again. Okay, nice. So what this scale is is basically it looks only at very hard data endpoints. So every clinical trial that runs has an endpoint, a primary endpoint and a secondary and there can be several secondary endpoints. The primary endpoint is basically the reason why we're running this clinical trial in the first place and a hard endpoint would be overall survival because that you can measure without any doubt something like progression-free survival which is already a bit trickier and severe toxicity, which is great for toxicity for anyone who's familiar with the toxicity grade scale. What this scale does is it puts it into a systematic system and then ends up with a grading that tells you whether this is a very good drug, a movement drug or rubbish drug. You do not have to agree with a tool but I believe the thing is that any tool is only as good as it is appropriately applied. You will only use a hammer for certain things but not others and you should think of this scale in the same way. What it does help, it gives you a framework how to look at a paper and unfortunately I don't have the sheet here and they have made kind of a helper sheet where they can, that helps you to go through the paper and it's very specifically asked you, what is the primary endpoint? What is the difference between the new treatment and the old treatment? And you do this very, very systematically. If you then look at two different treatments, you end up with two recommendations and that then allows you to say this is better than that and that is the full point of the exercise. And while it is not easy to get started in it, I believe it is worth the time reading up on it because we're definitely gonna do this at our next annual conference. We're gonna go through the melanoma papers that are relevant to us and try to understand whether we, well first to understand what they have done and then see whether we agree or not. A word of warning because you're in a hematological disease, this scale has been designed for solitumers, so not for hematological tumours. But it is now, EHA is now currently adapting it for hematological diseases and they have done first pilots and it also seems to perform quite well there. I know that people from us are involved in there. I think Jan is involved in there. So please check out what they're doing and then use the appropriate one for your disease area. So that's another reason why I don't wanna waste too much time on this now, but please look out for it. It helps to go systematically through different studies and it gives you in addition to that, an argument if you find that something that is ranked very highly on such a scale and for example, it's not available in your country for reimbursement. This is the perfect bit of borrowed authority that can help you to support your claims in terms of evidence-based advocacy. Now, I would like to, we're coming to slowly coming to an end. I would like to summarize that a bit and have a few words on how to get started because I'm aware that that's the daunting or the most daunting bit. Next slide please. You just have to start. It's about start reading the original publications in your specific field of disease. Don't be too worried if the first time is our heart, it does get easier with practice and usually when I started like in the research, I started in youth field, I would spend like a week or two do nothing but reading. Of course I was lucky that I had access to all these publications, but I would start with one point and then would go through the references, mark them, then look all of them up, reading them and continue that until I had read a few kilos of paper. I personally quite like and I must admit that I like them in hard copies so I still do print out my favorites, at least the relevant ones and I mark the bits that are important and I put notes on them and I keep them. I have a folder for the really important things. I have a folder like I'm resistant to targeted therapy because that's something I was interested in and I keep them and I go back to them. If you just throw them away or you just read them online without annotation, you will have to read the same thing over and over again. If you go back to see your own notes and you have a structure in place that will really, really save your time and your own. It does help to find like-minded group of people for discussions. More I see more, so it is helpful to discuss the paper and see what other people believe. Form a journal club. That's actually an established concept for anyone in science. As I said earlier, we had one every week. It was an hour every week where you get together to discuss one piece of like one paper. Everyone has to prepare for it and then you have a round of discussion around this paper. So it might be an idea to start a journal club in your advocacy group. You also want to think about finding the filing system very, very early on because we're totally guilty of it ourselves. I had a professional filing system at university and we didn't have that. We just saved them in a folder. It's already a shared folder between several ones of us so anyone who wants access can have access, but we haven't saved the papers clearly so they're very difficult to find back. So which means you have to open lots of different papers and it's a pain. So it's worth to think about some filing system either investing in the library program and these are like databases for literature. Some of them are quite nice. We're currently investigating a few ones. I have something I'm happy to share that but it's definitely something you want to consider and ideally something that you can also share in your community that everyone else can benefit and profit from the knowledge you have recorded. As a general rule, I believe it is very, very important to curate your patient forums. Patient forums should not just be a group like a forum for support where people send each other hugs and hold thumbs or hold their fingers when scanning time is coming up. We have started establishing on our forums a very, very strict rule that any claim with regard to health, especially when it's about new drugs needs to be supported by evidence. That doesn't mean that we say, you know it has to be the clinical study but we want people to put where they got it from. If it is not coming from a source that we identified as reliable as specified earlier on but let's say it is the daily mirror we use that as an example or as an opportunity to explain why this is not reliable health information. I would warn against just kicking these things off a forum because that means that people will just leave your group and will go somewhere else where you will have no influence whatsoever. So you have to be kind but it is good to have clear rules on a forum and we have seen that these rules are now spreading from our central forum to other groups that copy it and people then come back because they know that this is a trusted like this is a place where you get trustworthy information. And I think that's the most, the single most important thing you can do to create a better scientific understanding in your community and keep people safe from false information and sort of science. Start with a pinning your advocacy claims with data, however simple, however basic get it from somewhere else, borrow it but start thinking about any claim in terms of what do I need in terms of evidence so that people cannot refute what I want. And then I think last but not least I would like to warn you that in any type that also research, there's the good, the bad and the ugly. So it is about understanding what good research looks like. What is good in medical research is not good in psychological research or political research but there are measures of what good clinical research looks like and what a good qualitative research should look like. For example, just to give you a concrete example if you have surveys, surveys have to look a certain way in order to have questions that are not leading because if you ask in question that basically pushes people already in the direction of a certain answer you have a bias here, a survey on what you're getting out is usually a confirmation of what you wanted to hear but it's not valid science. So it is important to spend some time thinking about quality control. Next slide please, two useful tools things that we are using. On the left side, this is a Google form. On the right side, this is serving monkey towards on that from our own experience and that's not representative and you might have very, very other experiences. We've used Google forms, they're free. They integrate very well with any Google drive and if you have shared Google drives with your input in your organization this is like seamless integration. They're quite pretty. You can choose the photo, you can choose the background. You can have them integrated into the email so you send them out and everyone has this in their email so it's nearly impossible to miss. However, I have limited data analysis and we have been using that. As you can see now, I use a concrete example. This is just a workshop. We just ran like three, two, three weeks ago in through the bed and people had to choose between projects and we found that actually it was not able to give us correct numbers for the people who had chosen each given workshop. So I would be careful with that. On the right side, this is our survey monkey. We call them monkey surveys actually and we use them for nearly anything because it turned out that this really helps to streamline around conferences. So we use this for anything like applications for conferences, registrations for conferences but also researchers. So we're using them actually quite a lot. However, as you can already see it's not quite as pretty as the Google form. You have also have to pay for relevant features. For example, in order to be able to export the data. So get the data out from the survey and what we use a lot is export as extra cheats. You have to pay for it. However, it has very powerful functions. You can easily, for example, set up anyone who has like a reminder image or everyone who hasn't filled in the survey and the analysis are really good and useful. And I forgot to add this. You can get a charity discount which is really, really interesting. So before you sign up and pay make sure that you have found out about the discount and applied it. Now, next slide please. I would like to summarize that we as advocates have to be aware that argumentation based on evidence is extremely powerful. And that for me is the single most important reason for us to get engaged. Scientific thinking is not got given. It's not bestowed upon us. It is learned and requires training and training makes perfect. One note, the MCBS is a useful tool for a systematic way of thinking and judgment. It's not the end of all of our worries but it's a good starting point to get into judging publications. With that last slide, I would like to thank you for your attention at this late time of the evening. And then now we have about half an hour left for questions and I'm happy to take questions. Thank you, Bettina. Thank you very much. I don't see any questions here in the list but now is the time for you to raise your hands or to type your questions into the chat box on the bottom of your control panel. And while you do that, I would like to ask a couple of questions to you, Bettina, if you don't mind. Some of them came up during your presentation and some others are ones that I have been parking for you for a while. And one of those is that you set about the paywall problem so that it's quite difficult to access papers which are behind the paywall or in any protected system. I wonder if you have ever tried working with any of the open science initiatives like what the frontiers publishing house is doing now that they give free access to all papers published in their journals. Do you think that it makes sense for us patient advocates to approach these new initiatives that are about open science? Listen, I have my daughter sitting in front of the door and it's really cold here, it's minus five. I have to let her in, I'll be back in two minutes.  I'd like to have questions, I'd be back. Yeah. While we are waiting for Bettina to return, please go ahead and add your questions or just raise your hands if there's anything that you would like to ask from either Bettina or myself or also we have Anna Vajejo from ILMO Patients Europe. So if you would have any questions to her, then don't hold yourselves back and just ask those questions and go ahead. But then I will have some other points or some other questions that I would want to bring up with Bettina when she's back. Hello everyone, I'm back. Sorry for that. Hello, no problem. So you have a question? Are they open science questions? Yeah. Okay, so they're two sides to this story. We've been generally advocating in VINBO Collect, they're on Twitter about open access and supportive that. At the same time, the reason we got into it is to help our patients to argue with their medical team to get the treatment they need. For that, I go where the knowledge is and we don't have the time to fix the world before we can give our people the answer they need. So it's a bit about having a pragmatic approach to go and find this access to stuff that you need today. Why at the same time taking a more long-term approach? For example, in research projects, we have now stipulated that if we engage in a research project, the publications afterwards have to be openly accessible. So that's something to do as a, if you as a patient are engaged in a research collaboration, stipulate that the results have to be open access. That works. So now my other question is that how do you handle at least in your own communities or community, how do you handle this problem that scientific questions or inputs from patient representatives or patient communities are often looked down as something less valuable or less meaningful than what is done by so-called proper researchers? Have you ever been confronted with this issue? I mean, not really. I mean, just like they're different level states. So first I'm personally reviewing research protocols for others. So they send it to me because yeah, well, they know by now what they get back. The next thing is that especially in the Netherlands, foundations now need the support of a patient organization in order to get funding. And we have now started using that as a tool of basically gaining influence. If we don't agree with the project, no support, that's it. And we've gotten pretty like, I mean, at the beginning, it happened to me as well that people would send you like, could you support our research project and then send you a pre-written letter of support, but not the project. It's like, you must be kidding me, right? I mean, I must support a project I've never written. I've never read it. I have also worked as reviewer and that was anonymously. And then I got actually a thank you for the constructive remarks. I'm not sure I would have gotten them hit the no go home course. But you know, I think there's a lot of about us. I think we shouldn't spend so much time worrying about it. I think we should work on our quality and then just go with it and be like cutthroat. So if you have no one prevents you to publish comments on papers on the websites, for example, make use of it.  But once you have reputation to your name, I think that's totally fine. So I do this now with Tini. It takes time, unfortunately, and you have to keep yourself to do it. But I think that's the way to go. Some people are actually looking for patient reviewers. Yeah, I mean, building reputation is something. I mean, this is what you've been speaking about in terms of systematic and scientific thinking and structuring your work is actually, this is also the way how you build your reputation for any future work. At least that's my understanding. But tell me one more thing. What about unpublished negative results? Because we get a lot of questions about, okay, so there was this trial and we know that it failed, but the results are never published. How can we get access to, because this can also be quite relevant for future understanding of whatever should not be done, actually, in clinical trials. Have you found any way around this? So what's your view on that point? So I actually have two points on this. So first one, something that people are not aware is that to publish negative results costs as much effort as publishing positive results. People have this naive idea that they have result and then it's kind of published. And that's unfortunately simply not true. Writing up a publication is huge effort in addition to having produced the results. So they're having the results sitting in front of you is even not a third of the publication. It's a huge effort to write a publication. Publishing negative stuff takes you as much time as publishing positive stuff, but publishing negative results will never go into high impact journals unless you flop the huge study. It will never go high. So if you have a pile of positive results and a pile of negative results and you know that your next 10 year review is underway, what are you gonna do?  So that's a problem we have to think about when we just argue we wanna see everything published. Point two being in my opinion, if we conduct research on humans, the publication should not be optional. A publication of any research conducted in humans should be published end of story, whatever the result that should become like a compulsory like just coming from the fact that people have been exposed to research and potentially harmed. And the least we can do is to publish it. So these are the two points I would like to share. So in this case, we don't have to insist on high impact journals, but we have to insist on getting the data published somehow. Yeah, and of course we can whine about how Lented in New England is not accepting negative results. There's enough about out there, but I think that's a bit pragmatic, right? I mean, I wanna keep our people safe and we can find progress as fast as possible. If we fix society and humanity and understand at the same time, fine, but let's be realistic. It's not how the system works because the incentives for people to keep their jobs are different. Yes, no, but I agree. I mean, and I'm happy to hear that you also represent this point that no matter what the results are, as long as it impacts the lives of people, it should be published. And then we can of course argue about where and how it should be published, but it should be published. Now, also the other thing is, I wonder how you structure this time wise because often we find that at least in my field where I work in HIV and AIDS and also to some extent in hepatitis, people simply get exhausted and energy depletes and then they don't join the journal club anymore and then they don't want to talk about it. Also, there's oftentimes policy matters take over. So we just suddenly find ourselves in the middle of yet another access and policy discussion and science is somehow pushed into the background. How can you keep this scientific engagement and interest on the right level? No, I won't read until the essay said science is like, I mean, science is just an overarching area, right? I mean, we have contact with different areas of science and my background is obviously medical and biological. So that's where I'm at home. At the same time, we should run policy based on evidence. We shouldn't run policy because it kind of feels right today because our feelings might change tomorrow. And even when we support policy, it should be evidence-based and there are actually no think tanks out working on evidence-based policy. So I think that is for us to say really actually, I think it's like for our own network. This is a very important new area to get into. Health systems research, for example, is very, very powerful. If you can measure how long it takes to get a patient from diagnosis to treatment and you can correlate that with outcomes, that is very, very hard to refuse. If you just go there and say, oh, this is all taking too long, how likely are they going to listen to you? So I think this is like, we should look at it as an ally for us, a powerful tool for us. This is at least how I see it. And this is also science. A different type of denavid is that it runs differently, but it is the science and it still operates according to classification principles. It's not the basic understanding is the same. It's just a different area, a different methodology. Now, actually, of course, you also know when most of us on this webinar know about you, Patty, and this is one of the efforts of one of the initiatives that try to make sure that there's patient education and the higher, more professional level of patient involvement. Are you aware of any other similar initiatives that could be that you recommend that you could perhaps look at in terms of education or educating ourselves? Well, I actually don't think that there's anything of the scale that your party was. I think that's a unique and one-off and maybe there are country-specific networks. I met someone from, I don't even know how to pronounce it properly, I posted it, like the Irish, like an Irish person. Yeah, not the E, but the I. So the Irish one, not the European one. And they seem to have a really quite impressive program in place also about educating their community. But I think the thing about science is you can't teach it. It's not like, it's very, our interests are very disease-specific, very focused on our own problems, our own disease, our own, even within our disease, they're different issues. And it requires a lot of personal skills. So you cannot just say, you can't put people in a room and talk at them for a day and then they will know how to do science. That's not how it works. So I think we have to find a different format, like encouraging these journal clubs and let it become the standard of, our standard of interaction, our standard of argumentation. Because what I often find is that people think they become advocates and they go like, I haven't studied this.    So I think we have to kind of, I think you see it as a culture change. We can help people and we can establish certain kind of, you know, like forums, like journal clubs or formats and share formats that work and keep doing that. And I think that will increase awareness to do too much higher degree. And I mean, as you know that, it's like in our, in my own network, my ambition is like, we want top education for our people and we want them like technical education at a level that really moves the needle. I alluded to that earlier on. I think that's above all a matter of how we educate about scientific education. How do we make science accessible to people who haven't studied biology or medicine? So I think this is more how I think about it right now. Okay, so one last round, if anyone has any questions that you would like to ask Bettina or anyone else from the MPE team, then it's your turn now. And I will count back from three, like three, two, one. And if nobody else has any questions anymore, then I would like to thank you very much, especially you Bettina for the time and the effort. And it was fascinating as always. And please remember that the webinar has been recorded and will be uploaded to the MPE website. And the MPE YouTube channel. So if you go to the Melanoma Patient Europe website, then you can watch the webinar within a couple of days. It's going to be there in an edited version. And thank you very much for your attention and see you real soon. And Merry Christmas, Happy New Year everyone. Thank you. Thank you. Bye bye. Thank you everyone. Goodbye. Bye. Bye.