Okay, everything is good. Good morning everyone. I'm Adrian Solomona. I'm the deputy director of the Southeast European Research Center and I would like to welcome you to the first parallel session of today's feed for RRI online conference. It's a session on paving the way to institutional change via quadruple helix RRI embedment methodology. Just to give you an overview of what we will go through this hour, we will have a very brief introduction done by me on the feed for RRI co-creation methodology followed by actual showcases of applied RRI experiments. A bit of discussion on some comparative observation of the experiments that we had in feed for RRI and then at the end of the session we will open the floor for questions and answers from everyone. So if possible try to pin down all the questions, all the idea, all the discussions that you may have throughout all these presentations that will be done by various people as you will see now. Now because we talk about the experiment and institutional change and all these things I would like to just briefly provide an overview on what do we mean by experiment in feed for RRI and specifically for this feed for RRI experiment as you can see here on the slide is the actual action of engaging quadruple helix stakeholders in an observation and co-creation in an ongoing research towards identifying ways that this quadruple helix co-creation would enable RRI and open science embedment in the ongoing research and of course in the organizations that take part in this project. So basically this is in a nutshell the idea of the experiment that we had in feed for RRI and the goals basically of these experiments as you may have seen if you took part in the previous days by just as a reminder was primarily to foster the RRI and open science adoptions by institutions to build up skills that will facilitate the better implementation of RRI and open science and ultimately of course the end goal is to actually trigger and foster institutional change based on RRI and open science and all this was done by following a structured approach a unified methodology let's call it that you can see here on the slide and all the four experiments that have been organized in feed for RRI followed this approach in order to make sure that we can have comparable results and results that can actually lead to practical and policy implementations. So this specific methodology started with an appraisal stage in the sense that each experiment organizer was asked to map its internal and external stakeholders quadruple helix stakeholders for the external part and to identify for example their interests motivations expected outcomes from adopting RRI and open science and of course to select the most important RRI pillars to focus on as each experiment was usually selecting two or three RRI pillars from the pipe six that are available together with the open science concept in order to ensure that the experiments will have an applied and more tailored let's say approach rather than adopting all the actual pillars but as you will see in the comparative analysis side overall all the RRI pillars have been adopted throughout these experiments. Afterwards a design stage followed when together with the internal and external stakeholders the experiment organizers jointly developed the objectives or reassessed the objectives of the ongoing or the new research projects that they were doing the experimentation upon in order to facilitate this quadruple helix collaboration. Afterwards the implementation stage took place meaning the actual performing the actual research related to the research project or the initiative that you will see slowly in the case studies that we have displayed in the sense that they re-managed and they reassessed the way that research is being done in order to do it jointly with the internal and external stakeholders that were identified and of course afterwards after the implementation phase a follow-up phase was done in order to wrap up the lessons learned what can be done next what what measures can be taken in order to institutionalize the best practices. Throughout all these experimentation of course in order to facilitate learning across the experiments a mutual learning stage was done through which all the partner and all the experiment organizers were actually exchanging best practices and at the end we utilized specialized key performance indicators in order to assess the success and of course the effectiveness of each experiment and some of these indicators that you can see briefly here the quantitative examples for example the representation of the entire quadruple helix ecosystem interest in RRI of each internal and external actor at the beginning and at the end of the experiments in order to see if their interest was raised similarly for the awareness and the perceived usefulness of RRI. To understand it actually these experiments produced a positive impact in the desire of these actors to adopt RRI and open science and more quantitative ones such as the actual numbers of stakeholders involved pillars the quadruple helix consensus solutions and so on and so forth that you will see more in an applied manner from the three exemplification of the experiment. And of course we had also qualitative indicators in the sense that we try to observe in these efforts to implement RRI and open science in ongoing or new research projects what policy change recommendations appeared what organizational change recommendations are required in order to properly foster RRI and open science. If we receive any collaboration proposal from outside from interested parties yes yes we really enjoyed it we want to be involved in more RRI related initiatives and of course especially we paid a very big attention to proposals received from outside society when it comes to collaboration with researchers. And of course we looked into how this best practice modification of RRI and open science adoption has been done at the entire quadruple helix ecosystem. So in a nutshell this is the approach that we took in order to facilitate the organization of these experiments and now without any further due I would like to invite the first experiment organizer in order to in order to share with us and I'm sharing again the order of the presentations in order to share with us up to 10 minutes presentation along their experience and their outcomes from their experience. Rakel I will stop sharing now if possible please to share your screen. Yes hello everyone I will now start to share my screen you see yes yes okay I will talk about experiment one yes I will start by telling about who we are what our goals in the fit for RRI experiment were our main challenges lessons learned good practices and plans to further embed RRI in our work. First of all who are we we are a private company in Portugal big company with a lot of accredited laboratories and working around the world this to give a better idea of who we are. So the original goals of our experience were to anchor the experiment in a co-experience model to an ongoing project in the energy field which is a field in which we are very active in research and development of innovation and to also test previous outputs of the project on governance settings and sectoral viability with respect to RRI. The project that we anchored the experiment at the beginning was this project Moebius which was about modeling optimization of energy efficiency in buildings for urban sustainability but along the way we decided to have a new goal which was to build our own ISQ RRI model and what was and how why well I will tell you about how it was well the main challenges right at the start were that when fit for our project began RRI was a concept that was completely unknown to ISQ so we are a big company we are very active in the research and development of innovation very active in several R&D projects European projects in several different areas we are a house of engineering but we never had heard about RRI before so of course this was a big challenge from the start so along with it came the learning step of course because for us to implement an experiment on RRI we had to know what RRI was so the second main challenge that I can point is the learning step which was a very very heavy step but very very interesting so to understand what lies beneath the concept of RRI and open science the third main challenge for us was to get all ISQ researchers on board we are a big company with about 1400 employees but from them from this number only we are only about 50 researchers but we wanted to get everyone on board so this was again our this is also one of the main challenges for us so the first thing we did in our experiment was to start involving the people so after our first initial diagnosis then with some key people in the R&D activity in ISQ we started promoting these these events internal events for us to start reflecting together all the team of researchers reflecting together about what we are doing what is the the impact of the research that we develop so not mentioning right at the first the term RRI and then at some point of the events we would then summarize what the concept is about in 10 minutes and that was one of the challenges as well and then after presenting the concept of RRI we would start brainstorming and talking and reflecting together about what we are doing and how it was of course we organized after this first joint reflection sessions we started a training a training program let's say our training actions and for that we counted on some experts that were also involved in the projects for instance as University of Mino they are experts in open science so we promoted several training actions about what is this of the components what is this of science education what is this open open science open access so several events were promoted then and again we tried to involve all researchers and that's a big challenge meanwhile during the project duration a new ISQ R&D policy department was created because we had these units individual units of research that were spread around the company working together with the operating areas but then we decided that it would be better so the company decided that it would be better to put every researcher together so the research department was created during the experiment and one of the the advantages we had was that well the new director of the department used to be one of the people involved in the Moebius project so the one from the beginning of the project so he was more or less aware and after participating in all in all the sessions that we organized he was quite aware of what we were doing and what this concept was about and so he was very keen of embedding RRI in the new department division vision and mission and this was of course an advantage for the success of our experience what did we learn with this very briefly so we learned that the joint session with joint reflection sessions were key to get everyone get everyone involved these these are really in interest important because if we want to implement an institutional change of course everyone needs to be on board especially everyone when we are talking about with responsible research and innovation everyone that deals with research and innovation so we we we think that this was key to promote this joint reflection sessions with everyone then another lesson learned is that researchers during these sessions researches were faced with issues and ideas that are not usually on the table in their professional lives and work routines and again I want to stress that we are a private company so yes everyone in other companies or in other type of organizations like universities and I know the majority of the audience is coming from the academic part of the quadruple ux but yes people don't talk that much to each other we we are always so busy in our business in our activities that we made most of the times or sometimes well in our case a lot of the times people didn't talk to each other so there was no space for this reflection and discussions and what are we doing where are we going this kind of philosophical discussions and this was an opportunity for people to to talk and exchange ideas about what they were doing inside the same company and inside the same activity of the company so yes and when faced with some of the questions that we launched for discussion of course the researchers were faced with some issues that they don't think about in their normal day yes the tailor-made training sessions constituted an important awareness raising action and being a company and having all the resources that yesterday we talked a lot we talked a lot about from the foster portal and from the fit for our iPod together well with all the resources being a company we felt the need to tailor our sessions and tailor our contents in most of the cases because we found that some some of the resources that are available are very tailored for an academic or an university context sorry yes going and another lesson learned is that RRI is a flexible model not one size fits all and this is very very important if we are talking about implementing RRI in a company at least because for our for our right to be implemented in a company it has to be to have proven benefits well good practices and I know that my time is almost over good practices so this the selection of the components that we would give priority to was done based on the concerns that were the concern of the researchers so the concerns that we heard that the researchers had while promoting these joint reflexion sessions also yes that yeah thermal tailor-made training sessions on the selected components to accommodate the organization's need this is very important and also to plant the seed because yes we really think this is about planting the seed there is no immediate change that we can see in mindsets and everything so this is this has to be a long-term work yes and plans for us to embed RRI in our work so we are building our own RRI implementation roadmap we want to engage more with the society because given the turn of the experiment we ended up not having so much time to do everything we wanted so yeah the society part of the quadruple elix we didn't get there that much yes to set and measure IQ RRI model implementation indicators maybe also aligned with the more more he indicators continue this RRI training make it in couple sorry for each new researcher and to spread the words thank you thank you thank you very much Raquel for this excellent presentation I believe it was very spot-on and it really shows the substance that the fit for RRI experiments have achieved and this is a true impact case study so congratulations for this now since you have to leave I opened the floor for questions now I just I'm breaking the rules for just for you and I will just a five-minute window for questions I have already received in private a message with the question that I will ask now but in case anyone else is can think of any questions just after Raquel answers this first question please feel free to type it in the chat or to ask now just the first question and then we go to Giovanni now the first question since you are of course your business of course like okay I know you don't like to call yourself industry but you are business see any clear link between RRI and return on investment or how does it support the organization on the long term let's say with any return on investment strategy well looking at from a business perspective of course the anticipation part or dimension of RRI and the meeting the expectations of society of course this is if we look at from a business perspective this is working for the market so of course I think I see in my perspective this this is perfectly aligned with the doing what the business wants to do so yeah I don't I see I see it as an advantage of course it takes a lot of work in the time to to put this mindset because of course we want to meet the expectations of society and not harm society and not harm our world right and sometimes the expectations of society are good products that the market wants are not exactly good for the society itself but yeah I think it's perfectly possible to align both excellent super thank you now I think Giovanni wanted to say something I'm not sure is this in the case Giovanni, yeah, please unmute thank you very much for a very interesting presentation I would like you if possible to articulate spell out a little more what can count as a proven advantage of RRI for a company so it's on the same line of Adrian's previous question so what are the things that are perceived as a meaningful advantage for a company I would say that it's something that is good for business putting it in a simple in simple words of course and what is good for business it's good for business that researchers feel that they are doing something meaningful for instance and of course this goes in line with RRI if we think of it in all the depthness of each component and the whole the whole dimension of the concept I think and also as I said to Adrian as I told to Adrian seeing seeing that the innovation that we work for in fact are expected from my society so by the market and this of course is good for business and yeah I have been working with some other organizations because I was I really I'm a converted person you know yesterday we were talking about the converted ones I'm a converted one of course so I have been spreading the word in other companies from into the engineering field and also in other other universities and sometimes companies see it as a marketing advantage but I think this is the first the first view before getting in depth about on what RRI is about and what are the dimensions of RRI I think I don't know if I answered your question okay thank you very much now there's a question from Gordon please go ahead my name is Gordon Doulton I'm the coordinator of the Ring and Grip projects the Ring project is a global project in RRI and we had a work package that is dedicated to the competitive advantage of adopting RRI and there was a global survey conducted with interviews and the deliverable is now available under work package five under the Ring project and we're both the financial the branding and the marketing value of adopting RRI mostly to industry is being explored and second the grip project then is like people RRI it's going to implement five case studies in the Ring and Marathon where we're creating action plans for five case studies but we're we like to think that we're going you know sort of all the way where we're creating we're implementing change management and a very large number of interventions so of course training is one intervention but successful changing of strategies and change management's permanent strategies then is how we're going to measure our benchmark on how successful we are so I was interested to hear that you didn't succeed in engaging with quadruple helix and I'm wondering how successful did you get top management buy-in or institutional change well thank you for your question and for your hints I got very interested in your hints so it was very useful thank you I will check this specially Ring project available from work package five it will be useful for me and for the work I mean I'm doing with other other companies and the other organizations that are starting to work with RRI your second question as to your second question well how can I say it there's a momentum I think sometimes for change to happen there's a momentum and there was a good momentum during the Fit for RRI project for our change for our institutional change which was the momentum of creating a new department and aggregating all research activity that was that being done in the company for years of course so that momentum was a good momentum that facilitated to get the top level management top management level support but of course there's the momentum and I have to admit that with this COVID situation I believe that it got a little bit suspended and I hope that it's something that has to do with this complex situation that we are living I haven't been in the office since the 11th of March although some colleagues are but I haven't and I'm for the strategy that we planned and that we designed it's not exactly the same to be together with everyone and and of course we are dealing with exceptional situations so I hope that once everything goes back to normal we can continue and still count on top level top management level support I believe we will but I'm not sure okay thank you very much Raquel this was very very informative now just to be cautious with the time I would like to proceed so thank you very much any further question of course we can take by email in the chat and we can facilitate the answer later on I would like to invite Andrea Ricci. Hi good morning to everybody we start sharing my screen okay it should be in presentation mood so I try to be quick to save some time for the questions. Here it is what we did in Sapienza for our co-operation experiment. First of all our challenge we aimed at establishing and which were our haze the idea was it implementing through an ongoing Sapienza project named Sabirin Co a responsible governance framework capable of strengthening the university capacity for generating social impact and value and of course we want to convey some specific message to this experiment among the sustainability because our main assumption that we are trying keeping nowadays also in our policies at Sapienza is that responsibility should be seen as a proper and effective governance dimension to be considered in the management organization together with the most traditional one that are mainly efficiency, efficacy and cost effectiveness. So more concretely which were our objectives first of all as we are a very complex organization like 100 thousands more than 100,000 students and 4,000 researchers and so forth to test in a pilot scale a multi-actor co-creating the responsible governance in a complex environment in order to like see in a small context limits and opportunities for a big organization at Sapienza. To involve different stakeholders also based on a quadruple ethics approach in an open debate about responsible research and open science focusing also on sustainability circular economy and biomaterials with the attempt of shaping a common agenda to engage families and children in order to support them in experience through hands-on experiments the role of sustainability in their daily lives. As a result we select some pillars to run the experiments as you can easily imagine the main pillar of Bata Rai was governance because we were working in implementing a governance framework but we also consider two other pillars that are those of science education and public engagement. So mixing the idea of having like an internal governance framework even if supported by stakeholders together with a completely external dimension of science education public engagement towards our community. Well let's start seeing how we worked for the governance perspective which was the most complex one for sure. First of all we put together about 15 internal and external stakeholders so represent them from the industry world from policymaking from civil society together with our researcher and we ask them to work in a co-creation according to a co-creation approach to design the elements of a desired at least desired RRI based research center and we use the theory of change by current wise so we ask them to work according a logic framework with inputs activities and as I don't know who was there yesterday for the keynotes trying to differentiate as cannot say in the complicated relationship between outputs so the immediate results of erection then outcomes had impacted and really measured the real change you can have in your organization. So the results were encouraging even if not absolutely clear. By the way we were able to like come out with some outcomes that were for sure that the setup of a responsible governance in research funding and performing organization needs that scientific matters are informed by RRI for example touching teams such as research integrity, gender balance, open science and open access. In addition it was important that all the stakeholders were sure that responsible governance in research organization is by definition open to society and to external stakeholders in a logic of continuous feedback. Another important thing was the need of establishment of a common language platform and procedures because well they perceived the between for example the public and the private world for the private world but also between hard science and soft times some differences in languages because some sometimes we also dress it as somehow implicit dimension of RRI especially in the teams of art sciences. And again the most important things which is sometimes which is somehow linked to the previous one is that to implement real institutional change there was the need to bridge profit and profit research and innovation actions because that usually characterize some matters and in addition responsible governance should be based on a shared set of values that should have been reflected and we're working on that on guidelines with proper indicators and in the stressing the social value of research and innovation. So based on these feedbacks some months later in 2019 we tried to formalize our governance perspective in order to make more feasible a concrete implementation of our responsible governance and again we had to choose our work to work because the were the same people that we had in the previous meeting so internal and external stakeholder and in this case we decide to work through the balance scorecard so trying to define main objectives and then some objectives indicators induction with related targets aimed at focusing on the implementation of our responsible governance according to form in dimension the financial dimension which is also I would say resources dimension so also in terms of human capital or infrastructure that you have then the user's perspective so the external perspective but also then focusing on the internal processes so how we can change our internal processes and then the perspective of growth and innovation how can we improve our work and how can we innovate through responsibility the formal results was this balance scorecard with a general objective which is implement our right based research center and the sub-objective for each of the four dimensions cited before and this was further implemented we will not go in detail on each of these tables then you will see them but you can of course have access to the slide through tables based on indicators target induction for each dimension just to see an example here in terms of like financial perspective we say realize new investment on responsible and sustainable action and this was like linked with three indicators with a proper target and with a set of related actions as well so I will go quickly on this then now we can move to the education and engagement perspective that took place in May 2019 in order to cope with the idea of working on sense and vision and public engagement within this research center named Sapienko we organized two hands-on workshops for kids and their family and these workshops were linked with other public initial engagement initiatives already in place in Sapienza that deals with the opening of our like University Museum during the month of May the two workshops were dedicated to sustainability and more in detail to bioplastics preparation with the ingredients so for example organic waste or coconut flour and to the creation of super heroes but with 3D printers using organic materials what we wanted to achieve also apart from giving the idea that sustainability it's in our daily life was the idea that science education and public engagement represents a responsible action itself that show our national aspects are connected to all the right pillars and you have like some pictures that's the bioplastics and the kids preparing bioplastics and these are our SAPI heroes and the kids working on this as far as it concerns our quadruplex approach we can say that cooperation amongst the holders was really favored by the experiments venue because Sapienko was funded to be an interdisciplinary with the after-hub devoted to procreation and prospectivization by the way there is a common difficulty that is grounded in the academia in establishing a quadruplex approach because somehow the university system at least in our national context is still reluctant to some forms of prospectivization when there is not a proper national strategy or an incident structure to foster and favor it institutional change while we use a teaser which is not exactly the one that we are talking about so our arrival in science but we use termination which is somehow a trending topic now in university and that means the capability of the academy of implementing trusted mutual and socially fruitful science and societal relation and in this way by using like the anchor of third mission we were able to start a process laid the groundwork for some degree of institutional change but the real outcome that we are trying to achieve is make researchers aware that responsibility and openness are not administrative issues to talk about but they represent another value in their daily research work. Well for doing so this some policy support is required at least internally and we started working on this creating a working group both with research and administrative staff in order to work together in the definition of indicators for the third mission and we also launched a successful internal portfolio funding for implementing science and societal relation and well we had some slowing due to COVID-19 but we have like more than 20 projects approved and we are like going to see how they will work in this. Next goal, keep working on responsible governance keeping this collaborative approach as well implement engagement and science and educational activities within Sapergrinco and even within the entire university on a regular base trying to involve not only civil society but also other political actors and trying to connect these experiments with long-term initiatives. In this case we are working creating a connection what we did with series project which was the one funded under the European University's call Erasmus Call where we're establishing open labs for cooperation. That's it. Thank you. Thank you very much. This was very very good. As I wrote down in the chat we will keep the Q&A session at the end just to better manage the time therefore if write down on you know digital your questions just not forget them and I would like to invite Nancy if possible to share her experiment. Hello, good morning. Can you hear me? Yeah. I'm going to start setting my monitor now. Does this work alright? Personally I cannot see but I'm not sure if... It works Nancy. Okay. So I am here to present the use case from the Open University and what we did at the Open University is that we took a little bit of different direction so we did not plan a use case study which was related to responsible research and innovation and open science and the application of it but our topic was a little bit different. It talked about text and data mining of big scholarly data but we tried to do the whole process in an RRI way and this was the first time for us and we learned a lot of new information with the guidance of the project partners and we also have drawn very many important and useful conclusions that we plan to apply in any future project that we are going to run. So in reality the problem that we were trying to solve is the fact that right now there are a lot of outputs out there that students and researchers are able to access via the university subscriptions but this access is only for people's eyes to read and it's not for machine to machine communication and the latter point is the one that prohibits text and data mining. So we wanted to see how we could text and data mine content and it's not only open access because right now there is a large corpus of open access content available on the internet but at the same time we wanted to take advantage of a recent UK change in the copyright law which enables people to perform text and data mining for non-commercial purposes so we wanted to see how this could be made possible via this research study. We gave our research project a name and we called the CDUTDM because we got inspired from edurom and I suppose that the vast majority of you know that edurom is just an internet connection where someone who is affiliated with the university no matter where they go whether their affiliation is in the UK but they visit another university in Italy, Spain or some other place they can get access to the internet so that way we envisioned that edutdm would work for a machine to machine access for content that it's not only open access but via different work it would be made available to those who want to perform text and data mining. This was a very difficult task to begin with and we did not know the outcome of these results and therefore what we wanted to do is that we wanted to create a group to discuss the idea around it and develop a conceptual solution where this working group that we created would agree upon. So we wanted to have a lot of stakeholders from various places understanding the main concept of this idea and then seeing how this idea could be made possible. The outcome of this is a white paper the link of which is included in this slide and one of the problems that we met and you will also see that of the white paper is that we did not cover some ideas that were difficult to solve and discuss during this short period that this case study was about to like for the timeline of this study therefore we did not intentionally cover the ownership the development and the management of the tool. We just wanted to see in general as an idea what kind of attention this is going to receive how people are going to see it whether they're going to have a positive or a negative response and we wanted just to bring more attention to the matter rather than discuss important things such as ownership and development management but those are questions that we could not solve at the moment and require further discussions. So to get a little bit more in detail we created a working group which had members from the vast from wide variety of stakeholders and examples are the public systems for example experts on text and data mining, policy makers, other organizations that were in position to create recommendations and spread good practices and the industry. Of course we understand that even though we wanted to include the many stakeholders in the group we left others outside. The reason that we did that it's not because we did not foresee that their contribution was needed but again this was a limitation in the study because we wanted to do this we had to do this within a time frame and we had to be reasonable and pragmatic as to what we could end up having in the end. Therefore for example we did not involve the plain users or we did not involve any researchers from a UK academic institution and we don't involve any researchers from a non-UK academic institution so that we know approximately what is this that may be common or not or uncommon between their responses and way of thinking and needs but nonetheless we tried to be to have a wide variety of experts for example in the publisher systems to see whether there are changes between those publishers who offer closed access articles only or hybrid publishers and open access publishers for example. So what we did with this working group is that we had six sessions and then we had created a mailing list where we exchanged also a series of emails. So the white paper was the outcome of these six sessions and the email exchanging. Everyone had to agree on everything that we concluded and we wrote down at the white paper in the end but generally speaking we focused most on the technical challenges and as I said earlier not so much on the organizational challenges so for example what it was the conclusion of our investigation was with regards to the technical challenges but there is a lack of the common standards or adoption of standards to make this idea happen and that publishers use a variety of formats and of course there are issues with regards to the systems interoperability used between the publishers platforms and or university platforms and all the tools that are being used by text and data miners and what is this that for example organizations that promote openness and relate to recommendations and best practices would like to promote and then with regards to the organizational challenges even though we did not touch on them and we did not dig deeper deeper into them we identified that there were issues such as ownership incentives especially for publishers who would have to change and amend things and the cost of course and then who is going to what kind of agreements are going to be with regards to standards formats how is this going tool is this going to be delivered and what are the risks and who in reality is going to get benefited from that and what about those who don't see the actual clear benefit right at that time in front of them and then to proceed a little bit further with regards to the technical section we created a survey to establish the position of the stakeholders where we asked them a variety of questions that related more to their needs and what kind of TDM queries they may be running or how do they see themselves using this service and then we created the requirement analysis where we created stories for users and stories for publishers so we took some hypotheses and we said that as a researcher I would like to be able to do the x thing from a publisher system for example and to collect full text or to collect the data set or to get this type of information and with this information I would like to be able to do an analysis on the x topic and then while looking at the users we also looked also at the publisher's point of view and then we also asked the publishers what is this that they would like to see from this tool and then what they came back to us saying is that they would like to be able to evaluate the usage of this tool and they would like to know who had access to the tool and what kind of information this person got from the tool so it was more about knowing their audience and their customers and how the information is being used so that they can monitor it and they can use it for their future needs as a publishing house and then we developed a conceptual design which is presented here and in reality to make it very simple this graph over here it has a user who does a call on the API of the UTDM tool and then it requests some content from a publisher and when then what the tool is doing is that it goes to a variety of publisher and it tries to bring to the user back the results that the user is asking and then hidden behind this user action there are some extra actions such as the authentication of the user the authorization the aggregation of the content and then some other optional services and then we also defined the services that are required for to implement the UTDM tool and what we wanted to have is to create a core service of the UTDM tool that it's going to have all the things that we discussed earlier such as the validation the URL redirection and the aggregation but also some extras which would going to be beneficial for the user the publisher but also the medium like the university that would sit in the middle and this would be for example the casting so that it would not produce a lot of traffic to the universities and publishers load or the searching of the aggregated content or how these usage could be monitored either by the publisher or by the institution that sits in the middle which is the university and then we created also a table where we tried to identify the other most important organizations or publishers that provide the content for text and data mining and we tried to make it in a nice visual way give the benefits of the UTDM tool and while we were doing this research in an RRI way we in the beginning we were very much frustrated and we were not quite sure whether the things we are doing we are doing them in an RRI way and we are doing in a way that it's more inclusive and we had a lot of challenges but we also learned a lot of things there so one of the challenges that we identified is that researchers must have a very good understanding of the stakeholders that are related to the research question and it took us a couple of meetings of brainstorming to find out who is going to be affected by this research proposal and who would be interested from this in this research proposal and sometimes you just realize that it takes a lot of thinking to look around in a spherical view and try to see all the stakeholders that are being related and then we also wanted to see that funding for following the RRI process is always a challenge because right now this project has stopped because we cannot receive a funding and doing it in an RRI way means that it takes more time than the one that we could do if we were not doing it in a RRI way and then we also found out that we involving all the stakeholders from the beginning of the process it was very very interesting but then we also realized that at some point we because the discussions varied from the first discussion to the last one that at some points when the discussions were more technical we would lose the interest of those stakeholders who were not as technical or their role were not so technical and then we were when we were talking more about the organizational aspect of it or like about best practices then perhaps we would lose other stakeholders who either did not have the interest or did not have the capacity to participate into the conversation but nonetheless we were very happy that we did this based on the quadruple helix actors even though we did not include some of their main components and stakeholders and we wanted to we are very happy that we did it in the first for the first time via project because we did not had the pressure of doing this like in real life actually participating in the in the RRI component in real life so now in the future we would be able to perform in a RRI way a project because we have learned all this knowledge from this fit for RRI project which was whose aim was to teach us how to do this in a RRI way and there were some surprises for example in the beginning we thought that we should not involve too many publishers and then it turned out that some publishers found out about what we were doing from a word of mouth so they contacted us and they asked to be involved into these conversations so this was very surprising for us and then there were also others who were interested about the experiment and about the aims of the experiment for example someone from within the open university found out about what we are doing and they were they contacted us again to find out more information about this so those are very good things on promoting the project but also on the impact that this may have perspective in impact that a project may have on others who may not be very much familiar with what we are doing and at the same time as a researcher we did not think that they would be interested in what we are doing so this is everything for me and i hope that i did not take a lot of your opinion. Thank you very much Nancy this was really really excellent very detailed and thank you also you did part of my job because you summed up some of the key points that i wanted to say and thank you very much for that we can proceed directly to the questions also do you know with the time constraints i already received some questions but in the meantime please feel free to raise hands you know write a note or do something let us know if you want to ask questions i'll just proceed to some of the questions that i already received in private messages from you from the audience there's a question actually for for all experiments was there any clear mindset change towards RRI in terms of adopting RRI inside the organization any anyone who wants to answer from the experiment yeah well clear mindset i would say that is exaggerated for sure there's a new sensitivity for the topic and it is entering in our overall agenda of the institution so we are start thinking about responsibility and openness we approved policies on these so there's improving attention now we really need a general cultural jump probably this happened in some just some part of the organization but not in the entire organization thank you very much Nancy in your case as i said earlier our project was not so much to change people's mindsets within the organization within the institution but we also realized while we were having the discussions with the variety of stakeholders that concepts that were presented from one point of view let's say the publicers were not concepts that were thought from another point of view let's say the text and data miners or those who promote best practices and especially with regards to openness and using open tools so in our group this has happened but not at an institutional change yeah and it's actually very interesting to notice this and it's a very good case study and it was actually on purpose that we had in the consortium and as part of the experiment the open university because it is a completely decentralized organization as compared to the traditional higher education institutions in Europe and this showed actually how RRI can be embedded in these two alternative types of institutions thank you very much Raquel if you're online mindset change well as i said before i think it's more like we planted the seed and yes i think in in time things will start at least the city is there and people are starting to think about it so i think it's not clear but it takes time to see the change yes thank you very much and the second question that i received as a private message from the audience relates to involvement of society did it work or why it didn't work Andrea you can start again if you want okay oh well yeah it worked i mean that's probably even though i say that we are a little bit reluctant to cross virtualization i understand that there is well a mutual need of cooperating with civil society and also the society is asking for reference point to to cooperate with academia so i think that this is like as Raquel said in terms of mindset it's an interesting seed that we've planted and we're working on that thank you very much uh well in the case of finance of course it's a different situation so it did not apply that much ken well in our case as i said also during our presentation that was the part that was a little bit skipped because of our activity we work a lot of public authorities and we work a lot with the industry which are our main clients and partners and we but at least we know that yes engaging with the public public engagement it's something that we really need to improve and there are plans for that already so yes i think it's a matter of time thank you very much are there any other questions that someone wants to ask live besides the written ones i just want to add something maybe yes but at least in our case it's it's noticeable that researchers are quite resistant to involving the public in the research so yes that's also something that we need to work on with the researchers it's themselves excellent okay if there are no further questions i would like to thank you very much for joining specifically this parallel session it was a very engaging and actually applied discussion and hopefully you got the answers that you were looking for everyone that attended and for us it was a very good learning experience as always we can close the session now and as a reminder in 12 minutes the last session of today's uh strand of it for our i will start please connect using the appropriate link it it is going to be fully interactive and it is actually going to capitalize on everything that happened until now during this so thank you very much bye bye thank you bye


Well, gloves, that's one of the primary forms of safety, first line of defense really. We have rubber gloves that protect us. That's really the first thing, once we get in the bucket, we have on so we can touch anything that's primary. You know, we're on the ground, we take them out of the bag, we test them, do visual inspection, just to make sure while we're doing the job, we didn't get no pin hole, no cuts, anything in the glove or we'll do the air test or you can put water in there and just fold it down and get some air in there or something water and just look listen see there's air coming out of the glove or water coming out gloves are pretty much our first line of defense right there.


How many of you already use my pie to to check your production code? Wow, that's quite a lot For those who aren't this might be a little bit steep in the beginning But I hope I won't scare you away from my pie My name is Vita. I'm a software engineer and now a co-founder of of a company called one plane We started five years ago and what we do is we trade stocks mostly in Europe and We do that automatically and semi automatically we are based in Prague and Everything we do on the back end is in Python 3.7 at this point and We also happen to use a lot of async I owe and I like to think we were very early to start using type annotations and and my pie so Static typing is still Quite a new thing to the Python ecosystem and the community We're still learning how to use it and the tooling is Still being actively developed and for those reasons it is sometimes a bit difficult to maybe not get started with static typing but to actually cover complex code bases with static types But this by these challenges I Believe it is really worth it because When it is done properly it can help you avoid a lot of mistakes and in the box before you even run your program Before you even run your unit tests I'm going to talk and two Chapters the first one is the the high level approach you might want to take when you when you have a big code base and you want to cover it with my pie and Then we'll talk about a few examples of code that is not the usual hello world function and how you might go on about typing that and In the end I'll remind you that it really is worth it even though it will look a bit complicated sometimes so I Mentioned before that we started using static typing quite early at a point where we already had a couple hundred thousands lines of code and My pipe was very early back then and it it was crashing on the code I don't mean spitting out Typing errors, but actually crashing so we had to start gradually and only cover our code step-by-step and a big lesson we learned Unfortunately not in the beginning but over time was that the default my pie configuration is quite lenient and If you don't make it slightly stricter than the default is you might Learn a few bad habits that will come and bite you later And you will still have to fix your code and their annotations. So I would recommend you Whatever the code is that you're going to run my pie on I Would recommend you to have full coverage meaning There are no functions which have no type annotations or partial annotations So these are the config options you you might use for that Second these are optional, but you might want to consider them These restrict some forms of dynamic typing in your code Some of these options are difficult to enable but if you can do it or if you're starting with a new code base I would definitely use them and and this You really want to do with my pie and and static typing It's sometimes easy to fall into a trap where you think you know what you're doing and you or you think you know what my pie is doing and It might not immediately tell you that your understanding is not quite correct So enabling some warnings will help you with that Since covering an existing large code base is a huge amount of work. You want to go step by step so you begin by opting in meaning you run my pie only on the modules you've already covered and You might even start with a single module with a very small step and then you then you just keep adding on and And you want to defend your progress by adding this check into your CI Pipeline so it runs before your test to do and then of course you never want to make that list smaller You only ever want to expand it What worked well for us was doing an internal hackathon where a couple of developers stayed at the office overnight and work hard to to increase our coverage and We're still not completely there, so we might have to do a few more sleepovers when you go the opt-in route you need to deal with imports because You are covering maybe just a few modules, but those modules might be importing other code which You maybe are not ready to check in the beginning So this is how you tell my pie to not complain too much about other modules A word of warning that follow imports Directive has another option called skip the documentation warns you not to use that we did and It was a terrible idea. Don't do it At some point when you're opt-in list is sufficiently long You might you definitely want to switch to opt out meaning you run my pie on everything by default except for some modules that you exclude in your config and of course, there might be dozens of these Ignored modules in the beginning when your exclude list is huge But then of course over time you work to make that list smaller and smaller until it disappears The benefit of getting too opt out is that any new code you add to your any new modules You add to your project will be checked by default covering unit tests is Tricky matter Despite me recommending strict strict configuration for my pie I will backtrack on that for tests and Just make my pie a little bit more lenient for reasons explained in this code sample By the way, I'm going to put the slides up online So you don't have to date photos of everything There will be a lot of configuration and code when you use mocks and monkey patching in your tests, which you often do There is no way to explain that to my pie as of yet. It is a very complicated problem So you just need to ignore those places where where you monkey patch But despite these challenges, I would urge you not to ignore all your test files completely Because even when you partially cover them with my pie You will get some benefits because my pie will be able to check that your tests are using your your tested code as Intended meaning the annotations are being respected if you build your own Python packages You should know that even when they Do you have type annotations in their code and my pie passes on them? If you use that package somewhere else my pie will not follow those annotations by default So you need to tell it. It's very simple. You just need a marker file added to the package and that's it But unless you do that You don't benefit from from annotations in packages When you use third-party packages, which might not have type hints There's a few options. You have you might write stubs. This is something we don't really do so I won't go into detail You might want to ignore all third-party packages or Better you ignore again very explicitly ignore just those that Don't have annotations So now that you know Generally how to approach a code base We can talk about a few examples of what you might find in your code base the first example a Very useful frequently used tool generics and type variables Who here has heard of these or maybe even use them? Wow, that's that's really good. I think this is one of the most useful and needed features Let's take the example of a weighted average, which is a very simple formula simple computation where you add up values and you average them using weights and Critically we will want to implement this average as Incrementally updatable meaning you can keep on adding values to the average and getting back the result so you might start by writing a very simple class for example that starts with some internal pre-computed values and You will be able to add a weighted value to the class up to the to the average and Then you just add a simple method that that can calculate the average at any time You will notice that we are using floats as the data type in there So we are explicitly saying that we can only calculate averages for floats but Imagine you not only want to do that, but you might also want to use decimals Which is a an arbitrary precision data type in Python so of course as written as annotated that class will work for floats as expected and Of course, it will not work with decimals because you said your values were going to be floats By the way, the reveal type function is extremely useful That is provided by my pie. So it's undefined at runtime But for debugging what my pie thinks your variables are it is a very useful function. So if you want to allow floats or decimals a Good good way to do that is to parametrize your your weighted average so you make it a so-called generic class and you say it is parametrized by this Type variable which we called algebra type in this example and we restricted that type variable to either be a float or a decimal then your original class will be very similar to the previous version, but You will suddenly have a small trouble with the number zero By the way, this code block contains a very small lie Maybe some of you can see it, but it's not very important at this point. It works and Then in the rest of your class instead of saying float you will be saying algebra type So you've parametrized the type Now when you want to use that class When you instantiate it you need to add the value for that type parameter When you create the instance like this, so the first few lines are a weighted average of floats and You can see that it also returns a float and of course the second part is a weighted average of decimals What is nice is once you create a weighted average of a certain type you cannot change your mind and start mixing the types that is desired There is an even possibly nicer and cleaner solution Which is to say Actually what I need to do with my numbers is to add them multiply and divide them So I don't care if those are floats or decimals or something else. Ideally, I would just say there are real numbers So in theory that sounds great, but in practice the abstract number types in Python don't really Work that well or aren't that useful yet? And this is a good example of Of typing in Python being quite pragmatic it isn't and Ideal world it is a pragmatic world and you need to be pragmatic to you So your type annotations often won't be perfect. They won't perfectly describe what you had in mind, but they will approximate it and another very important examples example is Understanding the difference between nominal and structural typing So these are fancy sounding words, but it's nothing complicated Nominal typing you already know that's that deals with class inheritance. So Animal examples seem to be popular in computer science for some reason. So I went with one and There's a base class of an animal and then we have a duck that apart from whatever behavior animal has can also quack and then Suppose we want to make a function that accepts something that can quack and make it quack So as annotated here, this works because it's it's very trivial You're just telling my pie that your function needs a duck because only ducks can quack and and that passes However, imagine you wanted to have another animal that can quack and you want that function to work for that animal too So we could create a Penguin which possibly makes sounds close to quacking But it would be wrong to inherit that from a duck that would be very wrong So you just inherit from animal, but then of course your make it quack function doesn't work because it was told to expect ducks so Nominal typing means you use You use classes and class hierarchy when specifying the types you need in contrast to that We can be talking about structural typing where you where you describe your types in terms of the capabilities they have so here you're creating a thing called protocol in Other languages you might have heard a term an interface or maybe a trait and This this is actual code those three dots are valid python syntax in In case you didn't know and this really just tells my pie that there is an interface or protocol called can quack which exposes a public method called quack and When we declare this and change our functions slightly so it now accepts something anything that can quack then this will work for both animals and The interesting thing is that we didn't have to inherit from that protocol that protocol is a class, but that is more of a syntactic convenience and Now any object that will have a quack method will meet the requirements of this function so this is very useful for For duck typing pun intended Another example we encountered is when you want to Somewhat define your own type without creating an entirely new type a Very simple example of that is when you have a function called place order maybe that accepts a price and quantity of some goods that you're buying maybe and Doesn't matter what it does with that maybe it will save it in a database or whatever and Wouldn't be nice if we could somehow differentiate between a price decimal and a quantity decimal They really are just decimals, but there is a very clear semantic difference between prices and quantities If we could do that It would make our code more readable because when you read those annotations You will clearly see this is of type price and this is of type desk of type quantity and It would also make it hard to mix them up so you wouldn't be able to accidentally pass a quantity in place of price I Come from a company that trades under financial markets and confusing Prices and quantities or buy and sell is not a mistake you want to make so the first option that you might think of is to alias a Type so you say there is There is something called price and it really is equal to decimal and then you can use that price as a constructor That that works But unfortunately that doesn't create a new type This is just a convenience for you. So you don't have to type so much It does make it a code easier to read when you suddenly start writing price instead of decimal But as for type safety you get absolutely no benefit another option that exists in the typing module is a function called new type which kind of aliases an existing type, but It is a true alias in the sense that my pine now and Understands that it is a price and not a decimal so What what this does is you can still create decimals, but then you need to wrap them in your price type and from that point onwards my pine knows it's a price not a decimal and If we defined a function that takes a price you will see you can't pass a Bear decimal to it and you can't even pass a quantity to it even though it really is another decimal so this is what we wanted and We're now able to differentiate between the types This all works to a point once you start modifying the values You're back to the original type because really it is just a decimal under the hood and once you start making Operations meaning calling methods on that type It will return back the original type. So there is a limitation you should be aware of the only perfectly correct solution of defining a new type is To actually define a new class and implement your type and the behavior you want so That is nice and clean But of course you will pay a runtime price because your price Implementation will probably not be faster than the than the decimal type. That's already in Python implemented in C So that way lie a lot of interesting dilemmas about preparing static typing purity or preferring Pragmatic runtime performance and simplicity Now in Python you can do a lot of metaprogramming a lot of Lot of magical tricks and my pipe cannot always understand them a Good example of that is the data classes module. That's new in Python 3.7 Or if you're familiar with Django, then Django models Are an example of metaprogramming that my pie wouldn't be able to figure out by itself So you actually are able to write plugins for my pie that help it understand Magical code This is even newer than my pie itself There isn't much in the way of documentation yet, and there's just a few working plugins out there We also had to write a plugin and If you need to go that way too, then you might find our plugin useful because it's got twice as many comments as it has code and So this is a bit hard at this moment you probably won't need it But if you do I'm sure this will all get easier over time the final example I'd like to share is overloading function signatures that means properly typing the case where Your function might take different sets of types of parameters and return different types of results based on the parameters a Simple example is is Having something that's indexable Like a list is but let's let's say it's it's your own type so here we could have a series of Numbers and we want to be able to index individual numbers, but also slices and We want my pie to understand that when we use a single index a single value is returned And when we use a slice then a sequence of values is returned So we begin by creating a generic class. This should be familiar to you by now like I said, it's a very common tool and What we need to do next is to explain the the two versions of the square brackets operator Which is called done to get item in Python. So these two They look like method definitions. They actually are But they don't have anybody again those three dots are exactly what you want to put in there and you annotate them with an with an overload decorator coming from the typing module and All they do is they explain to my pie What are the possible ways of of calling those methods and Then you just add the actual implementation which in this case is very trivial and That real implementation has to be typed so that it includes all the versions of the signatures that you mentioned previously So this is a bit wordy But it actually makes a lot of sense once you get used to it. The syntax is easy to understand. I had a lot more In store for this talk, but I had to cut a lot so to fit into the time limit So I only have seven takeaways for you One is to try and make your life harder by using stricter configuration than the default is second is to go bit by bit don't take Too much on in the beginning and go module by module and get to opt out when you can Definitely learn to work with generics and type variables because those are your friends You will be meeting them quite a lot learn to use protocols because they are very much in line with the Dynamic spirit of Python so you don't have to create Classes for everything just for static typing Be aware of new type because it can add more semantic to your types Not everything has to be a decimal or an integer. You can call it user ID or price or quantity Writing plugins is hard But it is so important for my pipe to spread and become more popular that I'm sure it will get easier eventually and the last example overloading Looks like boilerplate, but it's not very that complicated and it is useful. So Who here thinks that typing is complicated after this talk? I certainly do but There are good reasons for that one is that we have to learn and understand new concepts as developers and that is great because they They force us to think about our code more and in ways we perhaps didn't before and another reason is that the tooling is still quite young and It's developing Very fast very actively, but there still is a lot of issues to cover and if I made just one final sentence that is once this becomes more popular and More prevalent once we learn how to use this then all our code will be Much less error prone and development will be more fun. I can already see that in smaller projects Thank you One short short question if somebody is close to microphone Thank you for your talk. I have a question. So beyond primitive type annotations do you think that Optimizing for a human which Python says readability counts and that kind of PEP 20 things And optimizing for machine that my pie in this case is a zero-sum game or it's kind of something else That is a very good question zero-sum game At this point we are making concessions to my pie. Definitely. We are adding code and Structures to the code that we maybe would not do otherwise But I don't think the gap is too big and if it is done correctly then You might be making the code easier to read for humans as well when you alias your your complicated type annotations and Use them cautiously Then I don't think we lose all that much very little and we gain a lot All right, and it's very hot topic, but we are running out of time. So thanks again Vita for this nice talk