Okay, great. Perhaps we'll start here. Thanks everyone for joining us today at this meeting of the Australian Sensitive Data Interest Group. This is an interest group that's co-facilitated by ARDC, the Australian Data Archive, PHRN and MDAP. So let's move on. I'd like to begin by acknowledging and celebrating the first Australians on whose traditional lands we're meeting today. As I was saying, I'm in Canberra, so that's the Ngunnawal and Nambri people, and I'd like to pay my respects to their elders past, present and emerging. As I was saying, this is a very cold land at the moment, but it's beautiful land to be on. The sky is blue, and the light is bright and clear, and it's a great privilege to be joining you from this land. So as you may have noticed, the meeting is currently being recorded so that people who can't join us today can find it on our YouTube channel. So for that reason, if you don't want to be captured by the recording, I would recommend that you turn off your camera and microphone, and that way you won't show up in it at all. I would ask that everyone here turn off their microphone anyway during the presentation, just so that we don't get any audio interruptions. But if you don't mind accidentally showing up in the recording occasionally, then it would be great to leave your camera on, and that way our presenters can see a couple of friendly faces. So a couple of bits of admin before we get started. For those of you who aren't already on it, we have a mailing list for this interest group. That mailing list is where we advertise meetings that are coming up, we share news, but also you're able to post to the mailing list. So you can ask questions. We had a bit of discussion a little while ago about the way that data relating to children is classified in different universities, which was quite interesting, and I would love for us to have more discussions like that. So if you want to sign up to the mailing list, I will post a link in the chat once I've stopped sharing my screen so that you can go there and sign up. I will also be posting a link to our collaborative notes document. So that document will be open to edit during this meeting. We put in there a space to take notes to record, and I encourage everyone to take notes. We record useful links that have been posted in the chat, things like that will go in there. We also have the recordings and slides for all of our previous meetings, and the recording and slides for this meeting will be there as well as soon as they're available. So a useful document. So that's enough for me. I would like to introduce our speakers today from the ARDC supported Monash SIRP project. So I'm really looking forward to hearing a bit more about secure platforms for sensitive data. I, in particular, I'm quite interested to hear about the range of different kinds of projects that can benefit from using that kind of platform and a little bit more about what is actually, I think, I understand in theory, but understanding a little more about what in practice is involved in getting a project up on one of these platforms. So I'm really grateful to Vivca, Matt and Juliana for presenting to us today, and I will stop sharing my screen so that they can present to you. Thank you, Nicola, and good afternoon, everyone, and thank you for joining us today.  We are here to talk about the scalable governance, control and management of fair sensitive research data project, or more known as the e-research platform, secure e-research platform, SIRP. Matt, if you move to the next, thank you. A quick agenda of the main items we're going to take you through. Thanks, Matt. So we're going to take you through the needs for the project, the project, and the lessons that we learned throughout. So SIRP project is co-founded by the ARDC and is a national collaboration across multiple institutions. We deliver a secure, trusted, and scalable environment for data governance, control, and management services for data historians. But we also provide secured remote data analysis environment for researchers. The primary objective of the project is to lower the barriers to making sensitive data fair, findable, accessible, and interoperable, and reusable. We do this by emerging technologies, processes, and controls to build a trust between data custodians, researchers, and collaborators, of course. We aim to deploy and run trusted prevent technology called the SIRP software to stock, and it's managed nationally by consistent service. We will achieve this by deploying and multi-tenancy arrangements to enable research collaborators across jurisdictional boundaries. There was a slightly change of a scope last year to expand to a national platform, and that's Keypoint, which was developed to service the University of Queensland. We will talk a little bit more about Keypoint in a minute. Our second aim is to employ exemplar projects, research exemplar projects on the SIRP service, and integrate the SIRP across the specific research applications. This will help us to demonstrate the capabilities and the benefits using the platform. And our term aim is to establish three communities of practice, data custodians, users, and infrastructure COP, and we want these communities to enable training, knowledge sharing, development, and the dissemination of best practices and principles related to the use of the SIRP service. Matt, thank you. The need for the project, for the service. Data custodians are responsible for managing sensitive data. They require a robust and secure environment to ensure the proper data governance and control. And we've identified several drivers behind these needs for data custodians. So there is a lack of control. So data custodians must have mechanisms in place to remain in control over their data and prevent unauthorized releases or loss. Also ensuring that data is used only for authorized purposes. This is a significant concern for data custodians. We also identified the need to enforce strict access control to prevent inappropriate use of sharing and the sensitive data. There are also several factors that influence the need to improve the sensitive data collaboration. As the data protection regulations become more strict, data custodians and researchers need to adapt to ensure compliance and privacy protection. There's also that concerns for organizational reputational risk. No-dates organizations really recognize the potential of reputational damage if sensitive data is mishandled or misused. So leading the reinforcement of secure collaborations and practices. So this is very important. With the advancement of data-driven projects, there is a growing demand for data to be fair. So collaboration between data custodians research is crucial for achieving these goals. Thank you, Matt. By understanding the drivers behind these needs of both the data custodians and the researchers, we can work towards bridging the gap, which is crucial for advancing knowledge while also ensuring data security and compliance. To meet these needs, Monash established a research collaboration with Swansea University in the UK to adapt and deploy an instance of the SERV service on the ARDC's Nectar Reach Cloud. SERV has the potential to be scaled nationally to enable Australian institutions to manage their own data while also supporting effective cross-institutional collaborations. Thank you, Matt. We offer a user-friendly interface with a single easy-to-understand and navigate user experience where users can efficiently manage their projects without any hassle. SERV allows users to manage and provide data quality information right at the point of their upload. Users can also easily access the quality of their datasets, enabling them to trust and utilize their with confidence. We offer a full suite of analytical tools, allowing users to work with the products they're more familiar with, either a statistical software programming languages, visualization of tools. Our platform will support Sims as integration for a small analytical workflow. And we know and understand that data comes in various forms and formats, so our platform also accommodates the structure and unstructured data, providing flexibility to store and analyze diverse datasets. SERV provides secure and remote access, enabling users to work from anywhere in the world, regardless of location, which is very important these days as well. Thank you, Matt. So I briefly mentioned that Keypoint is an outcome of the project that was again to attend the needs of the University of Queensland and is a data analysis platform that was developed by QCIV. It has been co-designed to meet the diverse requirement of the researchers in the UQ, providing them with all the necessary infrastructure, software system and analytical tools to conduct prior for data analysis, addressing all the research questions. Keypoint is still in version one, and it was released end of last month, but it's still testing and just to ensure the optimal performance and reliability, so it will be ready very soon. Thank you. I think that's all for myself. Yes, thank you, Matt. I will now hand it over to Vivek from UNSW, partners of the project, and Vivek will take you over their onboarder examplar projects and challenges they've faced. And that's it. Thank you. Thank you. Thanks, Juliana. So yes, I'm Vivek Akhats. I'm the research manager at the Centre for Healthy Brain Aging at UNSW. And as way of background, our centre collects data from a number of longitudinal cohort studies of older adults. And we have a long history of sharing this data with... Sorry, I got distracted. We have a history of sharing this data both with people within our team, but also with researchers from around the world. And one of the things that CHIBA does as well is that we lead a consortia called COSMIC, which has data partners or partners on all six populated continents. And the sharing of that data is quite complex. So based on all this, the previous data manager at CHIBA, Kristen Kang, established a partnership with Dementious Platform UK, which uses the Swansea SERP for their data sharing. And DPUK has been extremely generous in sharing all their know-how and templates with us, which has been very helpful in establishing DPAU. At this point, Dementious Platform Australia is servicing only studies as part of COSMIC because that's what we've got NIH funding for. And it's been also very, very helpful that we've got the ARDC funding and as part of the SERP project at Monash. It's taken a little while to get the tenancy agreement in place. That's definitely one of the learnings and challenges. Getting legal sign-off on agreements is a time-consuming process. But in any case, we've gotten there. So the platform, well, it aims to enable lots of dementia research, which of course is a big global health challenge leading course of death in Australia. And of course, it's going to increase as our population ages because age is the biggest risk factor for dementia. And I guess we know from our own experience with our own data set how important it is to have a secure way of exchanging data. I feel like when you send your data to a researcher somewhere, you can never be quite certain that they're going to do the right things with it. Will they store it appropriately? Will they not share it with other people, etc. And the other beautiful thing about SERP for us is the multiple institutions that one can contribute data sets and also the multiple institution and researchers who can access them from anywhere around the globe. Matt, could you advance the slide, please? Okay. So DPAU is a data platform to enable scientists to discover and access data from what we call contributing research studies, which are the data custodians, and in that way to allow secondary data analysis to aid in the prevention diagnosis and treatment of dementia and age-related diseases. So maybe just talk a little bit about the types of data that people collect. So there are questionnaire-style data about medical history, physical and mental health status, medication use, family history, lifestyle, prior trauma. Some of this data is definitely quite sensitive and we don't want people to be re-identified. We do at least a CHIBA, but many other studies of this kind do medical exams that do neuropsychological exams to examine people's cognition, memory and thinking, IQ, mental health assessments, again highly sensitive. We collect imaging data, so MRI scans and PET scans of brains, and these scans may include identifying facial features. There are ways to remove those features, but then you're no longer dealing with the raw data and you may be removing some opportunities for analysis in the future. We collect genomics data, including whole genome sequencing, and I'm sure this audience knows that these days that actually can be quite identifying, you know, with the advent of 23andMe and Ancestry.com and, you know, crimes being solved by people's DNA sequence because they can identify close relatives, etc. It's potentially, de-identified data is actually in this way potentially quite re-identifiable. We also collect digital data, such as recording of voice, and there's a move within dementia research to collect more digital data, such as geospatial data on a participant's residence and their movement in the local area, and we also know that this sort of data now with the use of AI and other technologies can be potentially re-identifiable. So these types of data are sensitive and may allow a person to be re-identified now or in the future, and I think this is where CERP really comes into its own because we have good control over where the data sits and the potential export of any data or identifying information, and it just really means that all the data custodians can sleep a little bit better at night. Next slide please, Matt. So the DPAAU platform consists not just off the data warehouse on the CERP. If we have a public face and information website and on that website you can see a study directory with some sort of big picture metadata of the studies that contributing to DPAAU. We have a matrix that shows users the studies and the sample sizes, but also what types of data each study has collected. Do they have a medical exam? Do they have MRI scans? Do they have genetics data? So it's a big picture, but it can give you an idea of, oh, there may be four or five studies in the matrix that actually could help you address your research question. And then we have a data explorer, which is behind a login, but anyone with an educational email address can easily get a login to this, and the explorer provides access to de-identified but individual-level data for a subset of about 30 variables. So things like age, smoking, whether the person has a dementia diagnosis or a particular prominent genotype that leaves a high predisposition to dementia called ApoE. And then we also have an online data application form. So we actually provide some level of governance or assist the contributing research studies with the governance. So any study that deposits data with us will have to sign a data deposit agreement, and any project that results from a successful data application will have to sign a data access agreement. Then the contributing research study data are stored on Monash SERP, and then it's provided in a separate folder to each approved DPAU project in accordance with what the applicant have ticked on their data application form. The DPAU projects can also import additional data onto the SERP. So say we don't have all the cheaper studies on SERP at the moment. Someone might want to combine something that is on SERP with a data set they already hold elsewhere. They can import that so they can do a mega-analysis and actually do the analysis across the data sets. Any results or graphs or anything people want to use in papers, manuscripts, the export of those results is monitored by DPAU staff. So you have some oversight over that people are not just exporting the whole data set or anything inappropriate like that. Next slide please Matt. So this slide I want to just try and demonstrate how DPAU adheres both to FAIR and the five safes. So FAIR, Findable, Accessible, Interoperable and Reusable are all in salmon orangey coloured text below and the five safes is in teal. So five safes relating to project people, data settings and output. So in terms of making the data findable we have the data directory and matrix which is public-facing and then we have the data explorer and it just shows on the left you can when you go into the data explorer you can select what type of data you are interested in. So I'm interested in alcohol. I only want to know how many people that may have data on alcohol whether they never used it currently used it or used it in the past. I don't want anyone who only has missing data on alcohol and I could then do a similar thing for smoking or genotypes. They are only interested in people who have the Aperfor risk genotype. I can then filter my data on these types of variables and also at a study level I can say I'm only interested in studies where they actually collected MRI data and then you'll get some high level output of how many people may fit your criteria and what sort of sample size could you potentially garner by collating data across multiple studies. You can also filter on say I only interested in studies that are conducted in Southeast Asia or in low income countries. There's quite a few different things both at a study level and an individual participant level that you can filter your data on. So the next panel across with the theme domain family and variable that refers to a data ontology that the collaborators at DPUK have been developing called the CSERV data ontology and this helps to make the data interoperable. If you want to combine data across multiple datasets you will find that people code sex in many different ways and actually if you want to then combine it and do the analysis it's tedious. So the CSERV has a standardized way of categorizing data into themes and then you go down into it. So theme could be medical exam data, it could be medical history data and then domain you're kind of breaking down in further categories until you get to the variable and the naming of the variable has a structured way of doing it. So this of course helps make the data interoperable but it also helps us to keep the data safe while allowing the data explorer to actually do its work right. If we standardize the data across for the small number of variables across studies it allows people to actually do a little bit of exploration before they submit their data request to see that their analysis is going to be feasible without revealing to them the individual data points. So the next panel I guess is around the application form and the governance that the website and DPAU supports so it makes the data accessible and reusable but in a safe way in terms of keeping the project safe people have to apply they actually have to describe what they're going to do and it also means you can do some vetting of the people who are going to access the data and only with the access to the SERP you're ensuring that only the people who are approved to access the data actually are accessing it. Of course sharing of your password for the SERP is a big no no and that's insured by multi-factor authentication and then we get to the SERP itself it makes the data accessible and reusable by anyone with the right credentials across the globe. It keeps the data safe you don't have to worry about someone doing you know reverse engineering of the data to identify participants for whatever weird reason people might have to do that it's a very safe setting and of course with the monitoring of the output the output is safe we have an idea of what people are outputting from the SERP and we can deny output requests if we don't feel that that's appropriate so the SERP provides the data storage multi-factor authentication for access and monitored export of results. Next slide please so some of the challenges and lessons oh it needs a lot of money and it needs a lot of time and it's slow so we've our teams received NIH funding to establish the DPAU for sharing a Cosmic Member Studies data which has set covered six continents and Cosmic has 58 member studies and we approached them all from 2022 onwards and the pie chart on your right just shows the engagement we've had so a small percentage starting at 12 o'clock have declined a number of studies have said they're unable to engage so most of the reasons for that is that the data custodians have limited resources to curate the data even to the basic level that we require and to you know dig out their metadata and the data sets and actually provide them in an interpretable way to us we are offering to assist them with application of the the CSERP ontology but just even to kind of get everything together and have the energy to a momentum to actually get the data delivered to DPAU is a barrier for some studies we have some a handful of studies where the the principal investigator is reviewing the process and making up their mind about whether they can engage with it quite a number of studies are sitting with the I guess contracts and legal officers of their institution awaiting review and as I said this is quite time consuming it can take months before it gets pushed through and then we have a small handful of studies that are signed up and we've also offered because we realize it can be a barrier that some studies will only provide their metadata so at this point in time they won't be providing the data to sit on the CERP but they are providing metadata and in some instances also sufficient data to populate the explorer and at least in that way it makes their data findable and it may be that these studies have already in place a different application process and they don't want to complicated by adding the DPAU on top of that and so people can find the data they can apply directly to the data custodian and then if they're approved and they want to combine it with other data on the CERP they can import it and I guess one of the things that we are really grateful for is that the ARDC funding really provided the necessary focus to establish a relationship with Monash CERP and to facilitate the local learning here at GBA we don't I'm definitely not a techie person I think everybody in the project knows that now we don't really have the resources within our team to establish anything like this by ourselves and I think our relationship with the Monash has actually made the CERP onboarding the smallest challenge for DPAU so super grateful for that. Next slide please Matt. So at this point in time we've focused on what I call flat file data but there is also a need at some point to expand that to onboard imaging and genetics data and I know our colleagues at DPUK have already managed this in some way and we are hopeful that once we get more grant funding that we will have the resources to expand that to DPAU as well. The other challenge I guess is that we know that harmonisation methods need more effort and thought to optimise even like if you do a neuropsychological assessment people don't always use the same instruments they may have different ways of measuring what we call processing speed or executive function or different types of cognition and so there's different ways of harmonising that and this definitely needs more effort. There are great enthusiasm for federated analyses and of course DPAU complements very closely. Dementia's platform UK who have on boarded 51 studies and also the Alzheimer's disease data initiative. They have a slightly different setup called the Alzheimer's workbench but they are also applying the CERP ontology to the 48 studies that they have on boarded. So the ontology actually opens really up the door to do federated analyses in a more efficient way. We are talking both to ADDI and to DPUK about developing interoperability across platforms architectures to allow this federated analyses across datasets located in multiple regions. Our DPAU manager is currently involved in a federated analysis with the people at DPUK where they've got the variables and data dictionary and then they're writing the script in UK and Brewery is running the script on our data here but even just doing that is actually quite complicated and time-consuming and that's it for me. I'm going to hand back I think to Matt. Thank you Rebecca and Juliana thank you and good afternoon everyone. I am Madyshek, Senior Project Officer at Monash University and manage many of the deployments to Monash CERP. I will talk to some of the interesting technical challenges we have encountered for this project. Firstly, XNAT. XNAT is fundamental to one of our exemplar projects and is an open source application used to receive and store medical images from hospitals. A review and testing was completed to ensure the application can be controlled in a sufficient manner and can be accessed by approved users only. Once this was completed a script was developed and made available to projects to allow the download of images from XNAT Direct to Strip Storage. Our risks were identified and in consultation with project stakeholders, governance restrictions were put in place. For example, when it is configured for a project to be able to access XNAT from CERP, it does mean that all members who have access to both the CERP project and the XNAT project can be able to have the ability to make the downloads. So as part of a risk mitigation for this, access to the XNAT repository was restricted only to a very limited number of users, approved users, and the XNAT Utils application used to download this to CERP was, the permissions were only allowed for this for a very specific and small window while the data custodians could set up the project. We also had issues using the most recent version of the XNAT Utils tools. We did have to revert to an older 2020 version of the application to get this to work due to a bug with the software that had been introduced in the most recent version. Issues like this add to the ongoing maintenance of projects required to ensure the services maintained to a satisfactory level. Also, Active Directory at its core, CERP integrates with the Monash University Active Directory for provisioning of projects, managing of the memberships and their corresponding roles and commissions. CERP requires full delegated access to the Active Directory systems to manage these aspects. And as you can imagine, that's we aren't able to do that fully due to some restrictions with Monash University and allowing access to the Active Directory. As a result, we have developed a custom integration tool for Monash CERP in consultation with Swansea. This doesn't allow us to communicate with the Active Directory but not have direct control. The restrictions this currently puts on us is that all users must have a Monash ID created to utilise the platform. This means every user with external institutions must have a Monash email address to access and also matching multi-factor authentication. This is still a work in progress. We are always looking to improve this. We always look at new opportunities and new technologies that we can do to help with this situation. For hardware, this has been a real challenge, especially since the COVID pandemic has amplified the situation. This has led to increased lead times and the scarcity of some components, especially with the current of GPU-capable hardware. Costs are also on the rise in this area and need to be considered when forward planning. We've mitigated this by carefully reviewing our usage and ensuring we're submitting procurements only for what is required and carefully managing this on a regular basis. We're doing quarterly procurement missions to ensure we have enough but we're only really ordering what we need when we need it. For software and various project requirements, from a general perspective, each project has unique requirements in their own way. For software, projects require specific software usually. For example, we have received increased requests for specific geolocation or mapping software that requires integration with online mapping tools and specific package repositories. The effort to complete due diligence on this software to ensure it meets our security standards and to configure the software to download maps in a safe and secure manner requires extensive effort. For resourcing purposes, it is important to assess each project on its merits and understand the software requirements and data being used. This ensures the appropriate hardware and software is available for each project. Some other lessons learned and for Becky to touch on some of these as well. So project onboarding, it's very important to get early buy-in from project. It was very helpful to have stakeholders immediately to consult with and to understand the requirements and be able to build the platform to those requirements without that buy-in and constant consultation. That is very difficult. As Rebecca mentioned as well, legal agreements not established early did create some significant delays for the project. So do encourage these to be started as soon as possible, even drafting before projects if possible so that you have an understanding of what you're dealing with and there's always usually delays with just dealing with legal teams. Also this relates exactly to ethical approvals as well as Rebecca mentioned onboarding can be easy but actually waiting for ethical protocols or other agreements can take a long time. We did have very successful collaboration efforts with our communities of practice where the community collaborated to create training guides, minimum requirements for trusted research environments and a shared knowledge base which is being actively used by the platforms in the community. Compute power for one project was not sufficient and we did have to find another solution. This was due to hardly limitations on what was available for us to purchase. We did require a very high-powered machine that we could not get in a satisfactory time. So we did have to shift one project to a different infrastructure to allow use of a high capacity computing. Also listed some operational needs and agreements that are important to have for this kind of projects. These slides will be shared after agreement so don't worry about writing these down or anything. I will make these available afterwards. Finally here's an example of some of the interesting work we're helping to facilitate. The RISE project is evaluating the impact of health interventions within specific at-risk communities. A high volume of unstructured data is being passed through the platform. The second project praise uses AI and machine learning to predict fracture outcomes for Victorian trauma patients. This project requires high GPU compute capacity and high volume data transfers of clinical images. ASMID is establishing a unique imaging repository comprising 3D body images. This involves the integration of XNAT that was mentioned earlier with SERP and Keypoint with the ability to establish the use of demoscopic mapping software available on the platforms. SERP and Keypoint is also supporting the Atlas project which routinely acquires electronic medical records related to sexual disease transmission. This is using data linkage technology to facilitate the merging of additional national data collections. Thank you for your time today. I'll stop sharing now. Fantastic, thanks all. It's interesting to see the range that it's not just the medical because I know that this kind of platform has emerged from medical research but or been highly used by medical research but to see it starting to be utilized by other kinds of project is really interesting. I'll open the floor to any questions we might have either if people want to put them in the chat or just throw a hand up either way. Yeah, Kristen? I'll get the ball rolling. Matt, something that I saw on that last slide that caught my attention because I don't know much about it and I feel like it's something I'm going to have to learn about doing a peer, that privacy impact assessment. Was that specifically, was that the SERP itself? Was that the kind of project level? I don't know much about peers. I've started to hear them in different discussions. I know that ONDC data place has had one or is doing one currently. Could you give us a bit of information about that? Absolutely. Your question and thank you Kristen for kicking us off. Appreciate it. You can always rely on you. The the privacy impact assessment is fundamentally going to need to be done on a project by project basis. One thing we've learned from this is that as you're aware, you just said you're not not aware of how it works and what needs to be done and we found that's pretty common. So what we've tried to do is collect the information of what is required. So there is some foreknowledge of what's ahead and what is required. So yes we did do so how it works here at Monash is we have as part of the OJC, which is the legal office at Monash University. We have a sub office called the Data Protection and Privacy Office. So they have a template they follow for a private impact assessment and we answer the questions with them and they provide us a recommendation. From this project we have developed a template of the questions that can be asked. So an understanding of what is required is available. But usually the advice from legal is that this is done on a project by project basis, not on a platform level. Obviously a lot of the questions are going to be the same on a project basis because it's related to platform specific governance etc. So it'd be very similar questions and answers. Excellent, thank you. Oh and just so we had a question before about whether we'll be able to share the slides and as Matt said we'll make those available. I'll both email them around and link them from the collaborative doc. So thanks to our presenters for sharing those.  Yeah I was interested on the harmonisation question with the dimension platforms. Because as I mean as you say this is kind of the how do you facilitate the interoperability between systems. It sounds like that was a significant part of the result. Your non-response or your declined response from your possible partners was we don't have the time to do the harmonisation fundamentally. And I say we're kind of working on a couple of projects around this as well with ADA and the various projects we're in. What is this? I mean essentially I'm guessing this is a manual process you've kind of got to go through. You just got to map, produce and then could you describe a little bit about that as to what the resource intensity is. Because this is a often this is where you know the heavy lifting comes from. Yeah so just on those projects they don't actually have to do any harmonisation themselves necessarily. They just have to deliver the data but I think if people don't share it regularly it's not necessarily going to have a good data dictionary or anything like that and you know if you've got to make it available you've got to make it usable as well. The ontology, I know that the DPUK group are working with people to apply some type of machine learning to at least try to predict what a variable might be and then there would be some human oversight to say well yes that gender is sex or you know some of those easy ones hopefully you would pick up and they're up to about 80% accuracy with that so that definitely is a help. So that's kind of just to get the variables labelled appropriately but then you know there's the second level harmonisation. Okay are males always coded one, females two or whatever and I know the actually the ADDI group have developed a tool to help with that but my colleague who's much more into coding etc says it's easier just to do it yourself but if it's a non-expert like me I might actually choose to use the online tool if it's a student and I guess the idea for DPU is as people work with the dataset and these recoding of data is done that we try and at least within the cosmic consortia to kind of get it consistent across the datasets that are being used and then okay harmonisation when people don't collect data in the same way I mean sex is relatively easy you could go by the chromosomes or what people identify as but imaging data if you use a different scanner for your MRI scans which you know people by nature are going to do they're not all in the same city in the same institution one of the things cosmic likes to look at is differences in ethnicities but well if the Asian studies done in Hong Kong and the Caucasian studies done in Sydney well is it the ethnicity or is it the scanner that's causing the difference in how people's brain volumes might be different etc so there's lots of complexity around that and I think we've recently been looking actually at some imaging projects with some people from Swansea you know I think sometimes people kind of just put the head in the sand and say we're just going to go ahead and do the analysis but I don't know that there's enough rigor in making sure that the measures are comparable across studies that's probably a particularly challenging one other times people kind of just use set scores etc and kind of centre the means and and try and make the datasets comparable in that way but again you may be washing out differences between groups by statistically trying to make them comparable. Thanks Vivca and I think we've got a couple more quick questions one is is there a technical architecture document for the DPAU site that could be shared? No. So I know that the data explorer Rory has been building that Nashini and she is looking to publish that and make the code etc for that available and you know I think she's found okay she built it and then she went back to actually document it so that other people could use and understand her script and that in itself is challenging so she's she's working on that the website etc we utilized sort of the template that DPUK had but we rebuilt it at a point so there are probably procedures and manuals but no there's not a neat little package for the bow on it I can say here you go. Thanks and we did have oh and our next question has already been answered which was could you elaborate on the reasons or requirements that drove the development of the new key point platform for QCIF and Matt has responded we could not realistically deploy deploy to QCIF in the time and cost required and that's for the timelines of the project it just wasn't feasible to get that done with legal agreements and whatnot it was going to that really needed to start a lot earlier for that to be realistic just take a lot of time which was really underestimated. Well I had a question sorry Nicola I can just interrupt that I kind of wanted to make a point on that also on the cost right it's nice to want to data share but if you don't have the resources to do it it's really hard okay Chiba is kind of lucky we've got a lot of philanthropic funding and we spent some of that on our data sharing efforts but you know you need people and you need to pay for the infrastructure you know having your digital data as you know from we researched last year right it costs money to have it sitting somewhere and you know having a tendency on SERP doesn't come for nothing so I don't know we kind of if we want to do more data sharing we kind of have to build in those costs somehow in when we seek our funding. That's an excellent point so and I was going to ask it's almost a related question I think because I mentioned Matt that people who access SERP need to have Monash credentials and that's not a I mean I guess I don't know about the process for acquiring that for at Monash but is that still a fairly manual process is there a you know a time delay are there limits on the number of people who will be who can who can do that I yeah I have questions yeah no there's no real limits for one there's a lot there so it is it is more steps so we have a relationship with the identity team at the university they understand their intentions for why we're using these accounts they're a very stripped-down account so they don't have access to all Monash systems it just gives you access to an email and gives you an entry into the active directory so we can have a listing for them basically we then like to link that to SERP so it's a technical hurdle we've had to put into our operations we have a like said a relationship with the identity team so when we do on board our external partners we are able to quickly collect that information for user information submit that to the identity team we usually get them back within a couple of hours honestly but the agreement is within a couple of days they'll have them ready for us but typically it's within 24 hours and usually you know very very quickly so we can still turn them around very quickly it doesn't affect us except for the fact it's a couple of additional steps in the workflow and that you know it is an inconvenience for external users in the sense that they'll need an additional MFA method from their typical account there will be a second account they'll have to manage as well to connect to SERP that could mean basically you know if they're if they're using they'll need Octa or Google Authenticator so if you've got one of those apps already it's usually pretty simple but if not you will need to install it most people these days do have MFA on their phones so that isn't so much the hurdle but the fact that you need to rely on a second account to access it is and something we're constantly looking for new ways to improve there's actually been some new news just in the last few weeks that we need to investigate and we're waiting for some updates from our e-solutions team to understand the ramifications so that's fantastic in terms of that that relationship that you have with the identity team I think I could see that being an incredibly important thing in terms of managing yeah how that works yeah we just have five minutes and we do have a little bit of closing up admin so I think we might just take this last question do you store data on SERP how does retention and disposal work for that data yeah so we SERP is really an application layer it's a management layer to manage the access to virtual machines manage file in request file out request and users and attached to that is our secure storage that we manage here at Monash so that secure storage is is accessible only in SERP within a secure private network it's a dedicated hardware and infrastructure within our existing data center for for retention we basically set review periods on a 12 month basis to check in with projects make sure nothing's changed and that they're happy to keep storing the data for disposal first there's a backup so whenever the data change there is a most recent backup stored so for disposal we have an off-boarding process where we need to gather some key information so first off you know is it to be disposed of completely or to be archived and then depending on those answers it will branch off the workflow into how we'll act on that but we do have an archival process whether it be five years seven years which are mostly the standard or indefinite or we have a disposable method where we destroy the data is how we label it excellent so the intention is not that the SERP secure storage is the long-term long-term home for the data but rather that if it needs to be retained long-term it is moved into archive correct yeah sorry that's one thing I miss saying there is that SERP is not a storage location it's an analysis platform so we shift data in and out as required some data can be there for you know a couple years or more that's fine if it's being actively used but we do know use it as an active storage we have longer-term storage infrastructure we'll shift it to great thank you very much well I'd like to thank you again for for the presentation and for everyone for the discussion afterwards that was really informative for me definitely as this is not my area of expertise but it's an area of great interest I would before we go I will just throw to my co-chair Kristen who would like to share an event that may be of interest to you all thank you yes before we go it's cross promotion time so the program that I manage is called the Hassan program I know that a number of people on this call are involved with the program in different ways it is ARDCs co-design program currently with the clinical trials community but going to be expanded in the next few months and then over the next few years to address health research beyond clinical trials the program's been running for three years the we're launching the platform that's coming out that's one of the major outputs out of that program so it's a data discovery platform so if you're a researcher looking for clinical trials data in Australia there is it now a national platform that you'll be able to search for it and also a platform that you can then request access to the data in one spot rather than going through to a whole lot of different people in their different email addresses so in a few weeks time on July 18th we're going to be doing our formal showcase and launch for Sander and the platform Nicola already put the event bright link in chat and I've just done it as well so yeah encourage you all to come along and yeah thank you fantastic well that sounds sounds like an event that could be pretty relevant to most of the people here so then all I have to say again is thank you to everyone for for attending today we will send you the link to the recording and the slides and I look forward to seeing you at the next meeting thank you thank you everyone


This is Melissa Armo with a stock swish, and I'm here reviewing the spy Just wanted to go over this this trade is actually working here Although the first one was up and then fail got stopped or retook it We'll see if this goes anywhere today 130 if it's gonna go anywhere I did say in the email when I send it out to the room. It's gonna be into the close So let's just take a look what happened here first in the morning Going back So here's what we did this morning right out of the gate It went up mark it made a new high failed didn't stop out though for a while. This looks great gave it the stop of 289 80 and actually let's see exactly what time that hit There it was 1030 and 32 but it held okay and then It's set up here on the 15 minute So what I think is best is if you do get stopped and want to do retake You've got to get some kind of confirmation or you set up on a higher time frame So this did the target is not a big target here for this But pretty much going to see if we can close strong today back near the high Which actually will turn around the day from the loss in the first stop to positive So we'll see if that actually occurs when I'm waiting. I'm watching. I'm seeing where this is going And I'm watching it very carefully So good luck everyone that did the first trade of the second trade if you're in the second trade If you didn't do the first trade you could have done nothing today I really wasn't I really like it actually besides the market and I didn't think the market was up too much Previous high just to look at it this morning again brand new on time high in the spy was two ninety forty one Close to forty two and we're heading to two ninety one. I mean we're heading there. We could get there today Really have to watch around two o'clock Time period here this afternoon to see if we're gonna the market has enough strength to push it through So good luck everyone if you did it and we'll see where it goes into the close


In this study, we developed a deep learning-based method to generate accurate maize krapland maps at a resolution of 30 meters in the one, seasoned planting areas of China. The results were highly consistent with the data recorded by statistical yebooks, indicating that the produced maps are reliable for research on food and energy security. This article was authored by Qi Xin, Licheng Zhong, Yunchu, and others.


Okay, welcome everyone to this week's edition of Encompass Live. I am not your usual host, Christa Burns is on vacation this week. My name is Evelyn Nipsacont and until recently I was the cataloging librarian at the Nebraska Library Commission. I am now the head of cataloging at the University of Nebraska-Lincoln-Schmidt Law Library, but I have had this Encompass Live scheduled so I decided to go ahead and continue to do it. For those of you not familiar, Encompass Live is the Nebraska Library Commission's weekly online event covering a variety of special library activities and topics. It is free and open to anyone to watch. The sessions occur every Wednesday at 10 am central time and they are recorded so they can be watched for free later if you miss a live session. They include a mixture of presentations, interviews, book reviews, anything library related really and ours online presented by Library Commission staff and my guest speakers. Today I guess I used to be Library Commission staff but I am a guest speaker now but I am your host and your speaker today. I just wanted to say before we jump in that if you have questions as we go along please feel free to ask them, type them into the question box or if you have a microphone you can let me know in the question box that you want to be unmuted and I can unmute your microphone and you can ask that way. All right, so let's go ahead and get started. Our topic today is metadata manipulations using mark edit and open refine to enhance technical services workflow. Once upon a time metadata manipulations meant something similar to what you're looking at in this picture. Working with cards in the catalog and that is no longer the case with electronic records and records coming in from a variety of sources using non mark record metadata and so today I want to talk about a couple of pieces of software mark edit and open refine they're both free and I wanted to talk about ways in which you can use them to make your life easier when it comes to working with mark catalog records and other kinds of metadata you may encounter in your library. The first one I'm going to talk about is mark edit and it is a freely available program that can be downloaded to your computer from the website at markedit.reecet.net. It is created by a wonderful librarian named Terry Reese and he does this all on his own time. This is not his day job. He works at I believe Ohio State University but he maintains it. He does updates. He is working on a version right now to work on a Mac so it's just the labor of love for him and he's really awesome. I'm going to tell you as much as I can about mark edit in half of this presentation. There's a lot more that can be covered. I've given a three hour class on it. If you're at Nebraska and the technical services round table of our Nebraska library association is getting Terry himself out to do a pre-conference at our state library association and that is a whole day thing so there's obviously a lot more than I can cover in one hour presentation when I'm also trying to talk about another piece of software to give you the highlights but there are a lot of tutorials available. If you go to this mark edit website this is where you download the software but there's also a tutorials link on the side and that leads you to a lot of YouTube videos so it's a really great resource when it comes to the software itself and when it comes to the help available you can email the creator Terry Reese and he responds quickly. There's a mark edit listserv that you can get on so there are lots of resources for help with this software. These are the features that I'm going to briefly demonstrate using the mark edit software. You can batch edit records so if you have a bunch of marked records that you need to add the same field to all of them you don't have to go and touch them all individually you can just use mark edit to add the same thing to or delete the same thing from all of them. You can extract fields from records to create a spreadsheet or do something else with your data. They have a function called mark compare where you can compare two different mark records mark join where you can join a bunch of smaller batches or even individual records into a larger batch or the opposite of that mark split where you can split a large batch into smaller groups or even down into individual records. You can merge records with similar data to create a new record. You can actually use mark edit as just a cataloging software to create records from scratch if you don't have access to any other software that will do that for you. A couple of relatively recently developed features are RDA Helper which is used to convert records created under the old cataloging rules AACR2 and converts them to something, I guess a hybrid record would probably be the technical term for it but it has RDA resource description and access, the new cataloging rules, compatible fields and it does it all to a big batch of records automatically. And then there's also mark next which is the newest addition to mark edit and it is a suite of tools that sort of help you play around with link data concepts and using your mark data in whatever the brave new world of postmark is going to be with big frame coming from the Library of Congress and things like that. So those are the features I'm going to talk about. I'm going to actually jump out to where I have the software installed. You can download it on your computer or they have a variety of options. Actually let me go to the market page and show you the download options. Let's see here. Mark edit start screens. If you go to downloads over here on the left hand side there's the various options that you can download. Once you've got it on your computer it will let you allow it to install a desktop icon or whatever you want to get to it. Let's see and I have it here. And what you'll see is any of the functions that you use the most often you can customize the start screen so that you can use whatever works best for you. The first thing I was going to talk about is batch editing of records. So if you have a bunch of records that come in and you need to do the same thing to every single record you can do that. The thing about mark edit is that it takes a mark file and then converts it into its own format so you can work with it and manipulate it and then you have to convert it back to round trip it out back in the mark again. So the first thing to do is to go to mark tools and we want to make sure that the mark breaker function is selected from that functions menu and then you have to find whatever file it is you're going to work with. I'm using an example a batch of records for documents from the federal government that we would get monthly files like this coming in when I worked at the library commission and then we would decide whether we wanted to import them into our records. So you just browse through your computer to wherever the file is that you want and then you have to specify where you want it to go. This is the dot MRK is the extension for the mark edit file and then you can execute and it will convert the records. It tells you down at the bottom how many were processed and then you click edit records to get to the screen where you can do just that. And so there were nine records in this batch and these, they're for print, let's see, therefore print items but they also are available online and so let's say that I wanted to add a field to all of these records saying that also available in a PDF online and so I don't want to go through and add them all individually so at the tools menu you can choose add and delete field. This information goes in a 530 field so you simply type in the field number and then whatever text you want it to say. So I'll just say also available as a PDF online and then you click add field you'll get a confirmation from a pop-up telling you how many fields were added and then if you go and look it's probably not the easiest to see but they are indeed added to the field. There's a 530 field here and it's the same for all the records. You can also delete fields. These have a 994 field that was automatically added but we don't necessarily need to use that for anything so I'm going to go back to that tools menu go to add delete field we want to delete 994 and as we don't care what the data is you don't have to put anything in the second box if there was, if there were the same field that had two different types of data you couldn't specify the data then it would only delete the ones with that data but I'm just going to delete the field and again it tells me nine fields were deleted so that means it was done from every record. You can see nine records isn't that much to work with you probably could go in and edit all these individually but when you think about it sometimes you get packages from vendors and there are thousands of records it's really a time saver to be able to do things like this. The next thing I'm going to talk about was extracting fields from records and this is a function that I used to, when we got these same government documents records I as the cataloger would download the records from the government's website and then I wanted to pass it along to all the GovDoc staff to be able to evaluate whether we wanted to put these into our records or not let's see if there were some questions come in and actually two people asked the same question so this is good the five other fields have indicators or sub fields yes that was an oversight of my part if you do want to put indicators and sub fields in there you do need to type those into the field data box you put the 530 into your field and then the field data box you do need to put in your indicators and sub fields and that was an oversight of my part excellent question I was talking about extracting fields as part of our workflow I would extract information from the mark records into an Excel spreadsheet and pass it along to the GovDoc staff and then have to stare at the mark records and make sense of them so I would for this one you need to go to the tools menu and you can customize this as to which ones you want to actually be on your home screen but you can also find all these functions in the tools menu and so you want to export and then export in this case I wanted to export a tab delimited records that could be used as a spreadsheet and so you choose export tab delimited records from the sidebar if you haven't gotten there already and then you need to set your file paths again similar to what you did when we were editing the records and so I'm going to choose that same mark records file and then you need to specify a place to save it and you notice it's going to be a text file a tab delimited file rather than a mark file and then you select the field delimiter I want it to be tab that's what's going to tell it to go on to the next field basically and then you get to choose the fields that you want to export I recommend selecting normalize field data that takes out all the mark subfield codes things like that so I would do let's see you can either select them from this list or if you wanted the title you put in the 245 field and if you only want a particular subfield you do the subfield in there and then add the field and it goes up there so whatever else you want if you want the call number you can do the 086 and add the field or let's see here if you wanted a 110 for the corporate name so it's probably a corporate name rather than a personal name but it might not be in a 110 it might be in a 710 so I'm going to do that you might not actually want to add a subfield you might want a whole field in the case of a 710 because it might be human services and then a smaller division division of public health or something like that so you might want to see the whole thing if this is a process that you do often you can save these under settings click save settings and then you'll be able to save it as a text file so you can load that again the next time you want to do the same thing over again if it's just a one-time thing you can just click export if you are coming back to something that you've saved the settings before you don't have to go through adding these again you can just do settings and then load settings and it'll give you a chance to reuse the settings that you would save before you're demonstrating this as a one-shot field so I will just click export and it tells you where the file has been saved and so at this point I go and open up excel and I would open the file that I just saved and you'll probably need to switch it so that it shows text files and it'll give you some options for how you want it to display you do want it to be delimited so that the computer knows the tab character is separated into a new column you want the tab to be the delimiter because that's what we saved it as and when you click finish you've got your various field it looks like some of these had a 110 but we've got the column numbers and we've got the 245 so you can just customize it and then you can do whatever you want with this spreadsheet if you need to set it on to somebody else it's just a way of getting a subset instead of a whole mark record and I'm not going to bother saving this because we don't need it until later let's see now we'll look at mark compare and this is a case in which you take two different mark records that could be in your records for the same item and they you want to see where the differences are maybe you've got a record from two different vendors and you want to compare the pros and cons of each of them and again this is something you could put on the home screen you can also find it under the tools menu and the examples I'm using are a record for a print version of a book and the record for a ebook version of a book and then you have to specify where you're going to save the file it's going to be an HTML file mark compare execute then you need to go and find where your file is and it just kind of highlights the differences between the two records so the greens things highlighted in green are the things that are unique to the print record things highlighted in pink are the ones that are unique to the ebook record so again I don't necessarily know if you use it so often in your day to day work but I think it could be useful when you're evaluating records save two different sets of vendors and you want to see which one might be superior for your uses things like that so that's the mark compare function the next thing is mark join and this is just what it sounds like you take either small batches of records or individual records and you show mark where to find them you have to start by selecting the destination that you're going to put them to so I'm just going to call it mark join and then files to join I'm just going to pick some random files it's pretty simple to go through that you can open up the file and instead of having two different files with ten and nine you'll have a file with nineteen records altogether mark split is the exact opposite of that you choose your source file you choose where you want it to go it's going to kind of put it all together in a folder and then you can either tell it the number of files to split it into so if you have a file with five thousand records and you want it to be into five files you can do that if you don't care about that and you just want to tell it you want to separate things into individual records you can tell it to do one record per file so you can either tell it how many files you want at the end or how many records you want in each file each of those can be useful in a separate in particular situations and so you process it I don't know why you would want to use a control field in the file name I have never done that this says nine files have been generated you saved it to my documents and so all these M split with numbers those are the various files so now you have nine individual files rather than the one with nine files all together you can also merge records so let's say if you had I don't know records from a vendor that had pretty decent cataloging but they didn't have OCLC numbers and then you had a batch where they did have OCLC numbers but they had been put in cataloging and publication and they weren't very fleshed out and you wanted to merge them together so they had OCLC numbers but they also had good quality records you can merge them again it's just as simple as figuring out which ones you want to put together and then you can either merge into source so if you don't want to merge them all at the end you can just merge the second file into the first one or you can specify an entirely new file if you don't want to overwrite what it is that you had and so it will give you a source and I'll just call it merge results so if you only want to merge particular fields you can choose that if you want to merge them all together you can do that too and so then you do that and then your file is located wherever you specify in your source file so a lot of it is just navigating to the right spot on the computer telling it what to do and then letting it work its magic and again I could go into much more detail on all these things but we have a lot to fit in in hours of under time but please do stop me if you have any questions I said you can also create records from scratch there is a mark editor function which is either if you customize it on your home screen you can do that or it is also under the file menu and you just get a blank slate you can do file new and then it gives you some options for starting with so a book, a serial, visual material, etc etc so if you choose book then it pops into the standard fields that you might use for a book and so then you can just type them in and create your records so if you don't have access to a bibliographic utility or for some reason your local system doesn't allow you or you're not in a situation where you can use your local system to create records you're away from your library and you really need the record Marked it is a good very basic software for creating records from scratch as well as doing all these wonderful things to edit them then there is also the RDA helper function as you may or may not know cataloging is going through kind of a transitional period between the Anglo-American cataloging rules AACR2 and resource description access the new rules if you use the RDA helper function you can take a batch of AACR2 records and add things to make them RDA compatible I would say they're not necessarily truly RDA records because you haven't gone back and looked at the item and made sure that you're transcribing what you see but they will be hybrid records that will work in a catalog with RDA records I have known some people who have used this function to fully convert their catalog to RDA compatible hybrid records and then you can choose which ones you want to add these extra 340 340 X fields may not necessarily be important depending on what kind of material you're working with but you can add them in most people will probably want to add the 336, 337, and 338 delete your GMD your general material designation evaluate the 264 for publication information so once you've chosen it it can also expand the abbreviation so it will if your records as p-period for pages and you can have it spell out pages I have a question coming in, I miss how you can convert the files back to mark after you use the mark breaker oh yes skipping back to mark breaker for a second if you are in mark editor we have already converted a file, I'll just open that one we were working with before when we were adding and deleting fields once you've done all the changes you need to do that can be either batch changes to add a field to everything or if you want to actually go in and touch each record individually compile file into mark is one of the last options and then we'll just let you name it as whatever and then that's how you save it and then you can import that file back into your local system that is a good question can we use the aggregate utility on authority records too you know that is a really good question I'm not, I have never asked that one that is definitely something that I would say contact Terry Reese about he's really really good his contact information is on the website and he's very very good about responding let me see if I can figure out to me it looks like it's mostly referring to bibliographic records so if I was pressed I would say no but definitely talk to Terry and maybe something he's working on who knows okay so I think that's all the questions if not please go ahead and if I can order a question by accident please go ahead and resend it I'm going to go ahead and talk about the RDA helper function let's see I have a file of a ZR2 items and it says RDA process has completed let me see if I can find that file I called it RDA items and so I guess I didn't show it to you beforehand but it has the 336, 337, 338 it's added a sub-bill E with RDA I know at least for OCLC's practice it's they don't even prefer to put the sub-bill E up before the sub-bill earlier in the string but Marketed just put it at the end oh I know someone excellent answering the question that came in earlier Marketed is market agnostic so it will work on all market formats authority, community information, etc so that is a good answer to that question thank you so that is the RDA helper function RDA Marked next was the next thing I was going to talk about and it is the Swedish rules I'm not going to go into great detail about it because I haven't exported a whole lot myself and we only have so much time to get everything in the thing that I have found most interesting is the link identifiers and you can use it to add let's see here I'm going to use one of my government documents files and then you specify a saved file all linked and what it can do is if you're familiar with the ideas behind link data one of the movements is to instead of just putting strings of information in our authorized headings is to also have a URI a Uniform Resource Identifier for a person's name or subject heading, things like that and so what this is doing is it's linking fields to the Library of Congress's link data where they have concepts and people represented as Uniform Resource Identifiers and so if you would check that box and then hit process and it takes a while it's going down under 2, 3, 4 and I'll go find that file actually I will it's a mark file so I'm going to break it into the market files so you can see it more easily and what it has done is it's added a Subfield 0 in fields like a 100 field for a personal name and it has a link to the URI for this person's name Jennifer D. Schmidt so this is kind of a baby step towards link data I don't know if a lot of people are doing this widespread but you'll notice it also has it for the 650 fields, the subject headings down here it has a link to the link data version of the Library of Congress's subject heading so these subfield 0s are at least a concept that's out there and so they are being added in by some people I think it's very widespread but a lot of people think this is kind of a baby step towards link data you know in the environments we're in we still have to use the strings where we actually type in somebody's name in an authorized form but we can also use these identifiers that will someday be more computer readable in order to make link data work with our library data so I think that's kind of cool like I said I haven't explored a lot of the other part next tools in detail but there's information out there on the tutorials that Terry Reese has on his website and so definitely if you're interested in that that's something that I would recommend exploring I will at this point also say that any of the links that I'm sharing they will be collected into the Library of Commissions delicious website and when you get the email but the recording of this is available there will also be links to all these so you don't have to worry about writing down URLs as I said so does anybody have any questions about mark edit before I go on to open refine and if you are playing around with these things after the fact and you have questions definitely feel free to email me I will be happy to do my best to answer them I don't see any coming in at the moment although you are certainly free to still go ahead and type some in if you happen to think of them I'm going to go on now and talk about open refine you may have also heard this called google refine it is a tool that was originally used or made by Google and now they call it open refine available software that is available at openrefine.org you download it onto your computer and it's running on your computer but it also uses your web browser in order to allow you to actually interact with the interface if you are interested in this I will say right now as you can see there is a book called using open refine that is a really good introduction to this software there is also documentation available on the website I've also mentioned a website called freeyourmetadata.org which uses open refine a lot to sort of show you how to mess around with metadata and do things with it that we are not currently doing in our library necessarily so keep that in mind as well that is also going to be in the delicious link after the fact freeyourmetadata.org and so here are the features of open refine that I'm going to show you separating multiple values in the same field analyzing the distribution of values in a particular field and cleaning up inconsistencies and I'll talk a little bit about this also has link data applications so you do go to open refine.org and download it and then when you work with it close this tab it is in a spreadsheet like this I have already gone and uploaded a file to work with a tab separated file it did take a while so I'm not going to show you that stuff but I had a test set of metadata actually I downloaded from the freeyourmetadata.org website so if you necessarily have a project that you are working on you can get some test metadata from there this is from I believe the powerhouse museum so it is some of their collection data and so the first thing I said that we can do with this is separating multiple values within one field depending on where your metadata came from you might have like a string of say subject headings separated by a semicolon or some other kind of character and they are all switched together in one field I know that my experience with nonmarked metadata was working with delvin core fields in content dm for the Nebraska memories project that we did at the Nebraska library commission and so we would string all the subject headings together in one field separated by semicolons but there might be situations in which you want each value to be in its own field so we will do that but first I have to create the project this is the screen you get after you import your data I'm just going to call it encompass live test because we are just working with this form this encompass live presentation and then hit create project I might have to start over I try to be proactive and do this ahead of time but it really is undermining my best efforts so now you can see the whole step I had a file of the powerhouse museum data and it is uploading it into Google refine you'll notice that some of the screens say Google refine even though they pretty much brand the software as open refine these days and it says almost done it's working on it you can see the the little bar is almost full it seems to always get stuck at this last point but it's working on it and while we're waiting for this I'll just talk a little bit more about open refine I feel like it's particularly powerful for dealing with large amounts of data where you wouldn't necessarily be able to go through and touch them all and find inconsistencies you can analyze it to see did everybody use a geographic subject heading the same way or did some people just put Chicago Illinois or something like that and you can analyze thousands and thousands of records very very quickly let me see how we're doing here it's still going okay you know I tried to bypass this and I'm you know best play plans but it's working on it because while we're waiting for it I will just talk through some of the things that we would be doing what exactly I mean by each of these things separating multiple values in the same field it's just what I was explaining let's see oh what kind of I have a question coming in good what kind of files can be uploaded with open refine let me answer that for you I know they have a list I think it's pretty flexible there's a lot of things let me go to their website and at openrefine.org I'm going to go check out their documentation just to make sure I'm not missing anything you know I think they can do XML they can do I think CSV files come separated values tab separated values FAQ that seems like a good place to start yeah I'm sorry I'm drawing one here off a definitive list of everything you can do but I do know that it's a it's a pretty extensive listing an Excel spreadsheet a CSV file like I said a CSV file then this is of course not working let me try this again I'll start over I was going to start with scratch opening up Google refine over refine make sure that we're not encountering any glitches here and it starts up with this little start screen and then it opens up in your browser let's see I have a question coming in about what software do you recommend to those who are new to mark files in general I guess that would depend on what I mean I'm going to need some more clarification from you do you have a local ILS system that lets you work with marked files to respond to that and so I'm going to try again uploading this file currently working in internship with a corporation environment where you may or may not have a mark based catalog you know if you're just working with you know actually creating your records I mean outside of just your local system probably mark edit would be the most basic software I can think of to work with mark files I guess you can create them from scratch in there you can edit them and I believe you can also export them in files other than mark if your corporation's software does not take mark up there are things that you can do to you can probably export it as something else and then see if it's something that can be used by your system I'm sorry I feel like I'm not giving you a great answer but it's hard to know without knowing your local system and yes there are a lot of tutorials on YouTube for mark edit so I think that will hopefully help you do what you need to do with with your system ok so I'm going to let that run I'm kind of wishing I had screen shots to show you everything now but I'm going to go ahead and talk about the things that I was going to tell you about open refine when it says analyzing the distribution of values in a particular field like I said you can tell it to look at a particular field and they call it faceting and clustering and so you can do the facets and it will go through if you have say a place of publication field and it will go here and tell you anything that looks similar you can group it together you can do what they call it clustering so it will bring together things like Chicago, Illinois where Illinois is capital I where Illinois is I, capital I, lowercase l, lowercase l period where Illinois is in parentheses where it's not it will sort of let you evaluate anything that might be similar and then you can say you can choose ones you can say we want to do Chicago, comma, capital I, capital L and it will fix all of those like that and make them all look the same so you can do that for simply analyzing and seeing where the differences are you can clean up inconsistencies whether it's things that were typos or whether it's things that you know policies have just changed over time and so not everybody is doing things the same way I really hope it's going to work so that's almost done I have someone with their hand raised in the audience I don't know is that a request to talk with your microphone if you would like me to unmute you please go ahead and type that in the question box and I can certainly let you speak that way so anyway I'm beginning to feel less and less hopeful that Joe from Refine will actually let us see what it is that I'm talking about I have a question coming in asking if the webinar will be archived and yes it definitely will you will get an email with the link to the recording Christa our host is out this week on vacation so usually it's up within 24 hours but it might be next week but you will definitely get an email with the recording we don't get to see this up about heart edit so anyway I'm not entirely certain I'm going to get to show you okay so here is the important screen and you can call it whatever you want and then click create project and this one takes much shorter obviously we have 11 seconds remaining so this creating the project link actually works much faster I'm so glad this actually worked out we have a question coming in asking if open refine can find inconsistencies with upper and lower case yes I believe it can we can look for some of those right there but yes it's a very very powerful tool and it can definitely help you find almost any kind of inconsistency in your metadata so I was talking about how you might have you want to separate values in fields they have a category field here in this powerhouse museum metadata which is a lot of subject headings also some more descriptive type of stuff and so they are separated by vertical bars which is an interesting character that I've never seen used for this before but if you want to edit this you can go to category edit cell each column has a little arrow next to the column name and that is where you get all your options to edit things you choose edit cell and you want to split multi-values that will tell you or let you put in what separator it is and so we have the vertical pipe and you hit ok and it works on it for a little bit all much faster than the initial upload I promise and now it has split things into multiple columns so this space script and models and space technology used to all be in the same column and now it has split them out separately so that's something kind of neat for something you can do now I'm getting a comment coming in saying that the vertical pipe is an option inmarked for extracting records so that's cool and that's possibly where these came from so once you've got things separated out and you want to kind of see what you're dealing with you can facet it and a text facet when we're dealing with something that's mostly text and then you watch over here on the left hand side you'll get a list of anything that is used in this field and then there's little gray numbers after it tell you how many times each of those things are used so we have a lot of numerical things here this is the type of day that I'm not totally familiar with so I'm going to skip down to the text things and those will probably be more meaningful so we have four items where AC generators are used it's two items where acid jars are used so you can it's kind of cool tool just to see we have 69 things that were considered to be advertisements it's a snapshot of your your collection data basically and so if you want to do what I was talking about before and kind of see where some of the inconsistencies might show up you can hit the cluster button and this is where it shows you and so yes to answer the question that came in before it does show you differences in uppercase and lowercase letters so things that are close but otherwise you know they're a little bit different so let's say our policy is to capitalize the first word and then not capitalize the other ones unless they're proper nouns so it shows you these things you say yes we want to merge it and then this new cell value is where you get to choose what they are all going to look like so if you put it in there with a lowercase second word then it will take all these ones where both of them are capitalized and it will make it look like that and so you can do the same thing for this one tell it if you want to merge and we want to have the lowercase second second word same thing for this one we want to merge and you measure the instruments et cetera et cetera and then once you've selected everything that you want to change you hit merge selected and recluster and then so you'll see those are kind of taken out of the equation you've still got all these that are similar but not identical and so you can go through and make more changes and so it appears that most of the inconsistencies in this file are exactly what the question was talking about they are differences in capitalization but it could also be you know it can find things like what I was talking about before where it has Chicago versus Chicago Illinois things like that so it's really really good at finding things that are close but not quite and allowing you to standardize things across a lot of data so you can so you want to merge that one and then if you hit merge selected and close then it takes you out of that option so those are the main things that I've been aware of using this for and I've not had chances to play around with this as much as I would like I would love to get into more projects where I'm able to use open refine but those are covering most of the things I talked about separating multiple values analyzing the distribution and cleaning up inconsistencies I will just briefly talk about the linked data applications because this is definitely something that I have not used it for but similar to how mark edit had the mark next functions open refine also allows you to encode your data in a way that can be used for linked data and link it to other sources let's see and just for the record you can't get your data out of here by exporting it in a variety of sources and these extensions built in it has freebase which is a store of linked data data we could be a type data that's out there so you can choose to link it to other systems that are out there and then make yours available and it will use identifiers and things like that that are already available in a linked data world let's see here we have a couple questions coming in can you remove specific data for instance 505 fields that are just a list of the alphabet yeah I can see why you wouldn't want that in your data specifically with open refine I assume that you could what I would say you could do let's see I think that if you you go through and sort your data and you you'll get a list here and so if you see the one that you want like the ABCDFG etc you can choose edit and I think you can just delete it all and say apply and it will just get rid of anything that has that data in it you could also do that in mark edit where we were talking about earlier when I did the delete field option I didn't specify a particular type of data in a field I just told it to delete also in 994 fields but you could tell it to delete a 505 field and then specify in your field text exactly what it is that you want to remove so I believe either of these pieces of software could accomplish that for you another question coming in does it handle diacritics correctly that one I do not know off the top of my head I don't work with a lot of items that have diacritics in their text so I'm going to have to unfortunately defer that for you I can certainly research that and email you the answer if you are interested but you will probably also search the documentation on the open refine site and I'm sure they can answer your question on that we have another question coming in does mark edit or open refine have a skull checker tool or a way to find the spellings for mark edit yes I'm pretty sure it does I believe I have used that open refine I'm not so sure I will say I have never personally used it with the powers that it has the fact that it can identify things that are not capitalized consistently I am confident that you would be able to find a way to do to look for the spellings when it does the cluster thing we also saw capitalization differences just based on the data that we have here but certainly I would say if you were looking at a particular field and you asked it to cluster similar I believe would bring together both correctly spelled and incorrectly spelled versions and then you could change the correctly spelled ones to correctly spelled ones I hope that answers your question does anybody else have any questions coming in and like I said I feel like I've just totally scratched the surface on both of these especially open refine I know it can do way more things than I just told you either of these could easily be a full day presentation on their own but hopefully this has given you some information so that you can jump in and explore these things on your own I would definitely be glad to answer any other questions you have either now if you think of them or you can definitely email me later I have links available now under my own personal delicious account they will also be on the library commission's website when the recording is ready so wait a second to see if there's any other questions coming in if not then I thank you for attending and please join the library for more encompass lives in the future thanks bye


Well, I guess when inflation is 9.2 percent, crime waves are spreading across the country, and your president is a barely sentient, broken-down, chunky cheese animatronic, you don't have much you can do, but point to something in deflect that's completely irrelevant. In this case, the Democrat media is attempting to make an issue out of the fact Josh Hawley raised his fist in solidarity with protesters, suggesting he was doing so with rioters. Even though this picture was taken hours before the first clashes with riot police. The media and the Democrats are all having a big laugh over this video that was presented at the sham show trial yesterday. Take a look. Senator Josh Hawley also had to flee. Earlier that afternoon, before the joint session started, he walked across the east front of the Capitol. As you can see in this photo, he raised his fist in solidarity with the protesters, already amassing at the security gates. We spoke with a Capitol police officer who was out there at the time. She told us that Senator Hawley's jester riled up the crowd, and it bothered her greatly because he was doing it in a safe space, protected by the officers and the barriers. Later that day, Senator Hawley fled after those protesters he helped to rile up storm the Capitol. I'm sorry. I'm going to tear apart their entire premise, but first check out this free coin offer from Noble Gold. Folks, we are to the point where gas is $7 a gallon in some places. A 10-year-old Honda Civic could set you back $20K, and rent is over $2,000 a month. Welcome to 9.2% inflation. You might have heard about gold IRAs, but don't know much about them. That's why Noble Gold has a team of experts at the other end of the phone. They'll put you straight on what you can and can't do to get yourself to financial safety again. And if you're quick, they're giving away an incredible 1-tenth ounce American Eagle Gold Proof Coin with every qualifying IRA or 401K rollover. You can't go wrong with Noble Gold. Call the team now at<PHONE_NUMBER> to find out more, or visit noblegoldinvestments.com. That's noblegoldinvestments.com. You can also use the link in the description or pinned comment. Make sure to tell him drone tech sent you. All this scandalous video shows is Josh Hawley jogging through the halls as he was most likely evacuated by the Capitol Police. They're also running this quote from one of the Democrat Capitol officers who is one of the more cringy characters on this show who proved the hearing seriousness by proclaiming quote. I'm sorry, how exactly does pumping your fist at peaceful protesters diminish one's character? Here once again we see this leftist trope that they're trying to normalize that the only legitimate protest is one pushing their agenda. All others are inherently bad. Says Michael Fanone who has since retired from law enforcement and is now on CNN as a contributor. Seriously? Come on! So this guy is a Democrat, hates Republicans, hates Josh Hawley, is a CNN contributor and we're supposed to take this all seriously? We are a nation with many pivotal historic moments that shook the world, but few come close to the word spoken by one very serious American hero. As Michael Fanone recounted his harrowing experience at the January 6th insurrection, he channeled the Founding Fathers themselves while describing an elected representative he was supposed to be protecting, saying, Josh Hawley is a b**ch. He ran like a b**ch. These words will reverberate through history. So, so super serious, it's just weird how motivated the media is to express their outrage that Josh Hawley would show his solidarity with these protesters, which I'm sure is violating like 300 laws, but then to also mock him for evacuating the Capitol like they were supposed to do? So push it! All of it! Alright folks, that's it for that one. I hope you enjoyed it and found it informative. If so, please hit that like button, share it, and leave a comment to let us all know what you think. Thanks a lot.