So our first speaker is Salim Rukos from IBM Research AI. Salim is an ACL fellow and an IBM fellow. He leads the mastering language group at IBM. His group has pioneered many classical NLP methods like statistical parsing to machine translation and even coming up with the blue metric, which has won the Test of Time Award at the NACL conference last year. And yeah, I mean, I can go on and on about Salim, but I would just let him speak about the overview of IBM's effort in question answering and summary parsing. OK, thank you, Avi. I'd like to first thank the organizers for organizing the workshop and doing the best thing possible to invite me. Thank you. So I'd like this to be interactive.  So please ask questions if you would like to. So we have an initiative at IBM to push natural language processing research. We call it mastering language. And it covers many topics. This is a subset of the topics. You can't list them all, but these are important from a commercial point of view in the shorter term. So identifying user intents for chatbots, labeling legal documents with obligation, control. So these are identifying important things in a legal document. Extracting entities and utterances to improve the dialogue. If you say I want to get a hamburger with lettuce and tomato, getting the components correctly. Extracting mentions, coreference, and relations for text mining from long documents. That's typically for analysis of for media monitoring called center analytics. It's under the rubric of information extraction. Answering questions from simple FAQs, facto it's to richer multi-hop questions for both enterprise corpora and knowledge bases. Integrating knowledge with semantic parsing and reasoning. This is sort of a longer term research for us, trying to see if we can use knowledge and reasoning into improving NLP systems. We are trying to do that in the context of question answering. And obviously, there are many other areas. I'm not listing here. And equally important is these techniques have to work and scale across languages. The other thing that I want to mention is the NLP conferences are growing geometrically or exponentially is the same thing, but exponentially sounds better. Probably marketing point of view, so I have to use that word. Even though it always bothers me. And the other comment is the field is very dynamic now. It's very active. Lots of people are working on it. Even people who never did NLP are doing NLP because of the deep learning, made things a lot more straightforward. So things are happening at an incredible pace that sometimes I hear some people say, oh, no. Not another result. You can't keep up. It's very hard. But I think it's an exciting time. And the other thing I want to make a comment on, you have to keep running. I mean, that's always been the case. It's just a bit harder these days. All right, I'm going to talk about three topics. How much time do I have? 45, right? Starting at? I started at 9.05. Oh. Oh. OK. Just for me to manage my time. So three topics. One is semantic parsing. I'm going to talk about abstract meaning representation. I know the other speakers are not here. And this afternoon we're going to hear more about semantic representations. So we'll have a discussion when he presents. Then on question answering and reading comprehension. And finally, I want to close with a brief overview of a leader board we're setting up for reading comprehension for the domain of IT support, which is an enterprise domain. We need to, from an IBM perspective, solve problems for enterprises. And they have unique challenges, I would say. All right, abstract meaning representation. Who's familiar with abstract meaning representation? Can you raise hands? OK, very good. So the idea is it captures some of the meaning of a sentence. So all these sentences here, the boy wants the girl to believe him. The boy wants to be believed by the girl. They all have the same AMR representation, abstract meaning representation. So that's the precision to which it captures things. And it can be presented as a graph. And I'm going to show you the graph in the next picture here for different sentence. The boy wanted to visit New York City. And what I want to highlight here is that the boy is the arc zero or subject of want and visit. So you have the same entity having two arcs coming into it. And so that captures implicit things or coreference sometimes, as you will see in some other example sentences. So this is the first point I made, multiple parents of a node. Also, there could be extra nodes in the graph. There are named entities, certain types that we inject into the graph. They're not explicitly presented in the sentence. They're implicit. So for example, the fact that there is a type city, which has the name as opposed to nominal. And then the specific New York City is the string. So there'll be extra nodes in the graph that don't have explicit word markers. And finally, there will be extra words in the sentence like the and to in this example. And there are more of these. Typically, the function words are never in the graph. They're really in the arcs implicitly. OK. So we have been working on this. And we've developed a parser a couple of years ago called Stack LSTM AMR parser. I'll describe the Stack LSTM in a moment. It's a transition-based parser, a sequence of actions. And it's essentially a sequence-to-sequence model from the words to the actions. And the input is plain text. The output is the AMR graph, which you can compute from the sequence of actions. So I want to take a moment to talk about the Stack LSTM. This may be small. But if you look at the picture, push A means that the head of the stack is A. So we push something on the stack. Then push B, then B. Then push C, then C. Then pop. Now the head is B. It goes back. And now you push, you push on a new path. So this allows you to share histories based on your stack operations. And that turns out to be a useful thing from a modeling point of view. I'm not going to get into the details. There are at least two posters on the subject, one here. And I saw the other one. I cannot see it from here. The angle is too sharp, where we talk about much more details about the parser. But fundamentally, there is the stack on your left, where we look at that. And there is an LSTM that creates a vector to represent the top of the stack. There is the buffer, which is the input sentence that you have pushed in, what remains of the input sentence. And then the sequence of actions that you have taken so far. So these three LSTMs will create a vector representation that's used to compute the probability of the next action. And there is a list of actions here. But if you are interested, you can talk to the posters, poster presenters. All right. Now the key limitation, I would say, or the key assumption in this parser is that there is a notion of alignment between words in the sentence and nodes in the graph. So when is AMR unknown? You cannot read it, I know. So this is how you capture the question, AMR unknown. Then link and square is a name. And completed is the fundamental predicate at the top. So there are words in the sentence that will align to some of the nodes in the graph. And there's another example below. I'm not going to spend time on that. And the way it works is given a graph and a sentence, you need to get an algorithm to give you node word alignments. And once you have the node word alignments, then you can create the sequence of actions that will create that graph using those alignments. And now the problem becomes, given the gold actions, gold, quote unquote, because there could be errors, right? The alignments may not be correct. But we call it gold. So the gold actions with sentence are the sequence-to-sequence model that's the parser. So obviously the quality of the word alignment will have big impact on the performance of the parser because the data will be, you missed everything. No, I'm teasing you, I'm sorry. So I'm talking about AMR as you might expect. I figured you might like that. OK, so given the sequence of gold actions and the sentence, we can build our model. So there are many things we worked on and the poster will talk about some, but I want to highlight that node alignments, improving the quality of the alignments had a big effect on performance. And using reinforcement learning also had a big effect on performance. And we did also the use of attention to have soft alignment to help with deciding how to predict the action. And we also switched to BERT, which obviously is a stronger representation of the words. And what you see here is the performance of our baseline system from the original parser. Then as we added improved alignments that gave 1.3 points, then as we added attention, get another 1.5 points, then added BERT, which gives 3.3 points, then added the reinforcement learning gives us 75.5, which at the time was state of the art because we were comparing to the first paper, but then at that same conference, somebody else had an improved system. So we're not quite state of the art, but that's OK. That's the nature of the beast. I have been looking more carefully at the results in detail here. So going forward, I expect we will focus more on the SRL performance because it's pretty low. So there are components you can measure. Instead of measuring the overall performance only, you can measure some subcomponents. And I would say the core reference and the SRL are relatively weak. So we have a demo of this system. So I'm going to show you the output of the system on some example sentences just so that we see what happens. The boy wants the girl to believe him. We get something like want. So the boy wants believe, girl believe boy. Arc0 is girl, right? So that's good. OK, reasonable. The boy wants to be believed by the girl. Now the parser is unpredictable. It does a good job, actually. It gets it. It's boy wants believe, girl believe boy. That's good. Then the third sentence was, the boy has a desire to be believed by the girl. Unfortunately, here it makes a desire as the predicate as opposed to want. So clearly, the MR does not collapse sentences correctly enough. So we need to add a layer to merge these things when we have strong evidence for that. Then the boy is desirous of the girl believing him. Oh my god. And it's really bad. It misses the him and the boy being core reference. So there is a boy who wants to establish that the girl believes in he. But it's not the same he. Not so good. This one, actually, not so good. The boy's desire for the girl to believe him, pretty good, except for the desire issue, the desire and want. By the way, I think we have a long way to go to make these parsers robust. You can give them a little bit of different sentences and they make a mistake. So work to be done still on this space. Now we want to apply this for enterprise domains. We started on IT support. And as you might expect, IT support is very different kind of language than the AMR-3 Bank. Even though, I have to say, unfortunately, the AMR-3 Bank uses not so good English data, as some of us know, which is unfortunate. It's a historical mistake. I'm not going to name the organization that funded all this research. But nevertheless, it's general purpose news. And whereas here, it's more technical. Event reader stops reading events. State change to value too high. It's different languages, really.  There is a lot of imperatives in the tech domain. People say, do this, do that. So the subject is implicit, ARC0U. We also added about a dozen concepts like you should FTP the attached save file. FTP is to be added. If unable to telnet, complete the TCP IP configuration. So there's about a dozen of those. It's not so bad, actually, from a predicate point of view. We added 17 types for things like software, product, file, hardware, computer, operating system. And we had to deal with the distinction between product names, acronyms, and abbreviations. You just need to know what these things mean so that to help you do things. So I'm showing here the IAA, Interidentated Agreement on the various data sets. So there is original AMR corpus on news and web from the LDC. And the IAA there is about 81%, pretty high. And the system that we have is at 75. So it's reasonably close. Maybe we'll get a few more points, but you're not going to beat the IAA, I don't think, by definition, because the test set is that accurate. We also tried to do at IBM some AMR annotation. So we did squat questions, squat answers. We didn't do a huge effort. They just wanted to see how it works. So our annotators were in the 70s. Then we did this on IBM support where we wanted to do a more serious effort. And still, they were still in the 70s. And you can see that the system performance is much worse on the IT because the domain, the language is different. So that is a robustness issue. So not only is it lower, 68, 65 versus 75, because of the quality of the three bank or AMR bank, there is also a domain issue. So this issue of domain adaptation is a focus item for us. So we did a couple of experiments on domain adaptation where we showed the performance with the baseline system. If you add 5,500 IT support AMR sentences, you really improve the performance from 55 to 64 or 65 with the enforcement learning. And we're doing now with Roberta just to see what the performance are. So I think we'll end up probably 66, 67, which is closer to the 70. So now we're investing into doing a second pass on improving the quality of the AMR three bank or graph bank. So to see if the annotators can be more knowledgeable of both the domain and the AMR. And I think they've become much better, actually. So I expect we'll have much better numbers next time we measure it. OK, that's topic one. Topic two, question answering. So you have a question. You have a corpus. You find typically you stage it like this. You do document search. Then you do sentence or passage retrieval from the top documents because you can't afford to do these calculations on all possible paragraphs or sentences. And once you find a set of passages or paragraphs, you then do reading comprehension or you find the answer span for the answer to the question. We have built systems. So we have a framework to plug and play components. And this would be presented at MNLP. But basically it allows you to plug components and to enable various groups at IBM to plug components and see if they can do things. So for example, there is a query reformulation module that goes into the search IR. Then the sentence and candidate selections are another component. Then there is entailment system to do additional re-ranking after the answer selection. And then potentially, as we have more systems, some additional answer-ranking model. So that top branch, I will show you the output of the demo of that system. Also in parallel, we are trying to see if we can use MR parsing, then converting to logic, and then use logical inferencing to answer some questions. OK. So to show the system in the context of a IT support DB2 use case, I'm sure it's small, but I'll read the question. When are secondary log files for circular logging freed? And the top answer from the reading comprehension is that blue delineated line. When the database is deactivated, that's the top answer with a score of 0.37. And the next answer with a score of 0.28 is better because it adds the conjunction. So it's when the database is deactivated or when the space that they are using is required by something. I cannot read it. For the actual log file. Thank you. Because I'm at an angle. That's hard. OK. All right. This is a very tough query. By the way, this is a subset of queries that do very poorly on search engines and things of that type. So just to give you an example where reading comprehension can add value. OK. Everybody's familiar with squat. Can you raise your hand? OK. All right. So there's a question. There is an answer. And they have three possible answers. And you measure the overlap between the system output, which is here, Santa Clara, California, and the three possible answers. And you get maximum F score of overlap. Then they realize that these systems are very brittle. I don't know if you're familiar, but if you play with some of these systems, if you change a little bit, things fall apart. So to improve the research agenda, they added questions that don't have answers. And the questions were on purpose computed to be misleading. So they are very close to the paragraph. The problem with squat is that both version 1.1 and 2.0 both have this bias of the turkers see the paragraph, and then they compute the answer, the question. And that's a fundamental, not a good thing, because there is a high overlap between the lexical choices in the question and the paragraph. But nevertheless, this leaderboard was, I would say, was seminal in my mind in terms of encouraging a research agenda and get everybody going on question answering. So it's really a fantastic accomplishment. So in the context of the unanswerable question, Victoria has about 60 full-time teachers, and the question was, how many full-time genitors does Victoria have? And the top system lose 20 points. You know, when you run on squat 2.0, if you build a squat to 1.1 system, run it on 2.0, you will not do very well. And the progress is shown here. So on the left is squat 1.1. This is time and the exact match metric. And the red curve is the best performance at that point. So that's the state of the art. And then the others are all the attempts. So it was a very active leaderboard, a lot of submissions. And the state of the art curve is moving fairly quickly. This is a span of a year and a half, roughly, even though it says two years here, because there's a couple more additional submissions. But most of the progress happened in a year and a half. And on squat 2.0, progress was even faster, I would say, because very quickly people developed techniques for robustness. So these leaderboards have been very, very effective to speed up progress, I would say. So Google recently released the Google Natural Questions Challenge. This one is very appealing, because these are questions from real users. So there is not that observation bias of seeing the answer. And it has 300,000 training questions. So it's a huge, large set. And the specific way they set it up is you are given one page. And then you have to decide what's the answer if there is an answer. And there is the concept of a short answer, which typically is three-four words, and a long answer, which is like the paragraph containing it. But not all questions have both, because some answers come from tables. There's no long answer. And there is some binary questions. Do birds fly or do humans fly? And there are questions that don't have answers also. So this is the leaderboard. So we have built a system for this. And we did a number of techniques. And there is a poster on this somewhere. And I would say the ensemble. And what I'm showing here are some of the results for one version of the system. Things change. A month later, everything is different. But at least a month ago, we were getting about two points on the short answers and 2.7 points on the long answers. That's what this parenthesis gives you. Data augmentation was also helpful at that point. Subsampling of the negative examples was also helpful. And we tried different pre-trained model, fine tuning, starting from squad, from squad 2.0. We have different techniques to do pre-training. So a variety of ways of getting initial models. And they typically helped, but they hurt on the long answer. I have to say, we have not done work on the long answer for real. We just have a convenient way of doing it, which may or may not be. I mean, we know it's not a good thing. So we need to improve it. But at this point, our focus has been on the short answer. And then attention on attention. And I believe Lynn will talk. Is Lynn here? Hey, nice to meet you. I have worked for Lynn for four months. Haven't met him before. So he will talk about it. And there is, as I mentioned earlier, a paper at E-MNLP, which talks about this system, which is a demo system, in that case. OK. So the leaderboard on natural questions is the same idea. So there is the ones that are in the square boxes, I guess. These are the state of the art at that point. And then they stay there until somebody improves on them. So you can see the short for the lots of submissions that are a little bit less than the state of the art. So this is the blue spot compared to the red dot. So our system, actually, at this point, is still at the top of the leaderboard for short answers. For the last six weeks, I guess, two months now, roughly. It's not as active as I thought it would be, but we'll see if that will change when the next conference deadline comes up. So I think then we'll see potentially some activity. OK. So you built a system, but now you go to an enterprise that have different kinds of questions, tech support, legal. So domain adaptation is the issue. So to try to learn a little bit about domain adaptation, we took MS-Marco, which is another reading comprehension task. And we extracted from it subdomains. And the domains were biomedical, computing, film, finance, law, and music. And this is how much data we extracted for each domain. So the biomedical domain is really large and unrealistic, 20,000 questions. You're not going to get that in an enterprise setting. And the others are more reasonable, even though I believe for enterprise it has to be more than the hundreds as opposed to the thousands. It's hard to read, but I'll try to read an example question for you. So what is the civil forfeiture law? And the answer happens to be the first sentence in that long paragraph, which is civil forfeiture laws represent one of the most serious assaults on private property rights in the nation today. So that's the answer in MS-Marco. So if you try the squad, it was not squad actually. It was trained on squad. I was going to say the squad system. So they tried, the group who did this work tried two different reading comprehension system trained on squad. And if you try those system trained on squad data and run it quickly on MS-Marco, the performance is 52%. If you train from scratch for the biomedical domain, which had a lot of data, 20,000 examples, you get to 66%. So clearly, all the learning on that MS-Marco stuff does not help, sorry, on squad does not help. But if you do fine-tuning, which is fine-tune on the, so you benefit from the squad system, but fine-tune it, you get 72%, and that's the best combination of variations on the same theme. So clearly fine-tuning is an important element of many domain adaptation solutions. We also looked at how much data do you need to do fine-tuning. And on the biomedical domain, we did 100 sentences, 200, 1600, and then 100% is all the data. So it's 20,000 for biomedical, and it's 2.6k for computing. And you can see that with the simple fine-tuning, nothing more sophisticated. I think more sophisticated approaches are needed. You are getting half of the way, let's say, with 1,600 questions on biomedical, compared to 20,000, right? So it's a significant reduction. And on computing, you're getting there by, it's a very unstable number. So I don't know if I trust any of these numbers, because they fluctuate, you know, 74%. But roughly speaking, 400, 800 questions seem to be a reasonable target here. The initial system is pretty good already, actually. So it's like one of those lucky things, I guess. So we are now trying to invest into developing methods that are robust, that will work for a variety of domains, and that hopefully can use less data, because I think most customers are in the hundreds as opposed to the thousands, in terms of question-answer pairs. OK, I'm going very fast. Yeah, OK. So finally, the last topic I want to talk about is we are creating a leaderboard for IT support to push domain adaptation research, actually. So it's very hard to find high-quality data, you know, question-answer data. And as I said, we expect hundreds of training samples in enterprise domain. So we need to figure out good techniques for domain adaptation. So we decided to explore finding answers to technical support questions. That's a significant place where there's a lot of use cases for companies. And the data set that we're collecting would allow us to do both IR and machine-reading research, even though most of the discussion today is on machine-reading comprehension. The big difference is in tech support, the questions can be complex. So sometimes I talk about the length of the question and the length of the answer. So squad and natural questions, they have relatively short questions, 15 words or less. And the answers can be short or can be long. This domain, you will see, has long questions. So the length of text is longer, so that requires a bit more modeling than we typically do. And this domain answers can be in tables sometimes. So we have about 840,000 technical documents. So it's called tech notes. These are like FAQ types of problem and solutions for a variety of IBM products. And we have both the HTML web page and we'll also provide the detect text formatted. So if somebody wants to do it from the HTML, they can. But we also make it easy so you can use the detect text also. So let's look at an example. So questions come with a title. In this case, Net Cool Impact 710, state change value being used by the omnibus event reader, long words, is too high. And then there is the body of the question. The value being used is a date and time in the future and as such is preventing the event reader from capturing the current event. So it's quite good description here in the body, more meaningful. And the answer is that region that is in red. And it says the simplest solution is to manually reset the event reader state change value via the GUI, stop the event reader, open it for edit, click the clear state button, exit the editor, and restart the event reader. I think the most important thing here is that there's a lot of words in the question relative to this answer. Let's look at another example. So here, the title is very brief, unable to uninstall Data Studio. And then there is a body which is quite long. We use Data Studio 311 with DB2 on Windows while trying to new version of Studio. We are able to install it, but unable to uninstall the existing 311, getting a JVM error. How can we delete it? And the answer is that little red snippet again. Please try to uninstall all products, including install manager. Then reinstall IM and Data Studio for 1.1.2. I picked shorter answers because I can read them if it's long or boring, but I will show you some statistics here. So the questions are, the median is about 30 words. And there are some long questions. There's a long tail. Some questions are quite long, right? 85 words or more. The answers are about 45 long. So 50% longer as a median. But again, it spans. It's almost uniform. So the way we obtain these questions and these answers is remind to support websites at IBM. And from those, as you know, forums and things like this, somebody asks a question and somebody, many people answer. And one of the answers is voted as being right by other people. So we focused on the ones that we've been voted as right as a source of data to justify the answer. Then we also required that post to include a link to a Tech Note page so that we can find the document. And then we asked our annotators to find the answer in the Tech Note. So we are more confident that these answers are correct because they've been voted by other people as being correct. Because remember, these are very technical questions and average humans don't know how to answer them. So that's the way we managed to get a vetted question answer pair. And these numbers might change, but we probably will end up with giving on the leaderboard 400 questions for training adaptation, giving 200 question answer pairs for development, and we'll keep the 500 questions for testing blind. This way we can have a meaningful. Yeah, go ahead. So people we have been using for a number of years for doing linguistic annotation. Like for example, they're familiar with information extraction annotation. So they are used to language annotation, not technical, per se. So they're not technically? No, no. It's very hard. Actually, we use a couple of people who are SIS administrators to answer the questions. Even that, that's hard for them because there is a variety of topics here. It requires a lot of expertise to answer these questions because different products have different kinds of knowledge needed to really answer. So they did not do a whole lot better than our scheme of using the outside world to give us the answers. And then, OK, and I think I did this. So as I was saying, I think we hope to have this over the next couple of months. We expect to have a setup where we will give a question and end tech notes, end documents. I think 50 will be the number, but that might change. We'll see. And there is a good chance, or I believe it's 50%, that one of the tech notes contains the answer. But there is also 50% that don't have answers. And the goal is to find the answer, basically, the shortest span of tests that has the answer. And we will use, for the reading comprehension, which is that setup, the QA metrics. But also, we can add some IR metrics because we could ask for what's the document ID? Because you have to pick which one of the pages is the right page. It's not a great IR competition, but it's just a useful metric to have. And as I said, another month or two maybe, we'll have a leaderboard. And I'm looking for people who are interested. Maybe we can do a preview. As soon as we have the leaderboard running, we can have some people who are interested to try it. And then they can be on the leaderboard early. I'd like to work with you, like you, to work with us if you are interested. I think that's all I have. To give this to you. OK, thank you. So we are going to open to questions. Mr. First Part on the AMR Parsons, what did you talk about there? Yeah, I mean, basically, really it's a review of our AMR parser in terms of how the flavor of how it works. And were you there when I talked about the IT support domain for AMR parsing? Yes, yes. So that's really the one area where I feel we need a little bit of help. I guess a more serious question would be, so you're using this squad kind of systems, which are end to end. And you're also developing parsers for AMR. I suppose you use that for question and answer as well. So what do you see as the disadvantage of each approach? And why do you need to do both? Right. So AMR parsing is a way to compress the information in the natural language into a reduced representation across many instances. So that could be a better representation than the word sequence that we used today in our reading comprehension. How to use AMR graphs into a reading comprehension system is a subject of active research of some people in this room. I'm looking at them. So when they get positive results, they will talk about it. So that's really one approach for the use of AMR graph. Another approach that we are also going to explore is AMR is a bridge to more logic-based representation of the meaning. And now you could use reasoning to help you answer some questions based on the AMR. So there are sort of two purposes for the work on AMR. Or really, I don't per se say it has to be AMR. It's like whatever we can get our hands on. And AMR is clearly the practical point for us at this point. So since you are here about AMR, there is one thing that we noticed that I didn't present it when I was presenting. There is the trick of using ARC off to simplify things. And unfortunately, that's not normalized in the AMR bank, which means that if you produce the other version, which is ARC zero structure, you will get penalized in the metric. So if you normalize, you can improve your scores by one point. But we don't do that, per se. But we would like to propose it to become part of the standard at some point. So it's a small thing. You had that table where you showed that for domain adaptation, where you showed that the performance improves with fine tuning as you add more question answers. How did you select those question answers? Oh, this one was just random sampling, nothing special. Is there a way to employ some other strategies? Yeah, it's cheating, right? Because the reality of it is this is after the fact that you already have a big corpus so you can pick the right question answer pair.  So you want the random sample, I think, from the point of view of understanding how much is needed. Unless you can come up with a very clever way of how what's the right way to add a question by a human given the previous questions, which is not practical at this point. I would say this work here is preliminary. I am not convinced of the numbers yet. So this is just to just get a feel where we are. We still have time for some questions. I guess I'm more generally curious about what it takes to get the number one position on the leader board for, say, natural questions as you did. So how do you decide when a particular idea you have is worthwhile or is not going to work? Do you do extensive hyperparameter searches over every configuration? Or is it just the case that you had one idea and it magically worked and got the first position? No, no. Realistically, you have a deficit. So you typically can predict sort of how much. I mean, it's not guaranteed, because sometimes we think we have something that works, doesn't pan out. But it's definitely more of the latter, which is you have many ideas. You try them.  So it's really, you have to try many things on the deficit until you decide it's worth now submitting a system. So I think we submitted maybe two times or three times. Three times. Three times. So we started in April time frame. So from April to August, five months. So every month and a half, roughly, we were submitting a system. We've done enough work in six weeks that there is movement. Now, no guarantees whether we will keep moving or not. But I would say, generally speaking, things tend to move. It's not a huge effort if you are interested. Thanks for a great talk. So I'm going to ask a speculative question about using AMR. So I haven't done any work with AMR, although I'm aware of it. So you did mention that this sort of trying to use AMR for question answering, it's still sort of this work, ongoing work, whatnot. But I have a different use case where we want to take specs written of software of some kind and then try to transform them into some formal models. So that is stated in some logical formulas. So one way I can imagine doing this is to do it taking something like taking a dependency parse or a constituency parse and then transform using that as the intermediate structure to transform. Because the data sets again that we have for this are really small. So I would like to take advantage of any intermediate structure that I can learn reliably well and transform from that to this form. Could you comment on whether if you were to do something like this, would you trust AMR parsing to be accurate enough to do this versus going to, let's say, starting from something like a statistical parse? I mean, of course, we can try empirically. Yeah, I mean, unfortunately, the answer is always the empirical thing. But the question is, where do you invest? I would say the domain adaptation challenge is really fundamental, even though we have condensate dependent vectors to help us improve these things. So because of that, I think I would be careful about AMR yet to be doing a great job at parsing, even though I assume these documents are fairly simple in terms of the sentence structure. So it may do very well, because if the sentences are short and simple, then the AMR cannot go wrong. It's like it's going to find the answer. So dependency parsing, I mean, AMR is really solving the things you need to do with the dependency parse at the end of the day, because you have to deal with coref. We have to deal with who did what to whom. So the distinction between AMR and dependency parsing may not be as dramatic as one might think. I don't know, he is more an expert on semantics, so maybe Bert can answer this question again during your talk or now if you want.


Good morning and welcome to our workshop on Longitudinal Data. My name is Berthe Lichtwald and my colleague James Rayner and I will be running this workshop for you today. The roadmap for today looks as follows. We will first have some Mentimeter time. You will then hear an introduction to the UK Data Service, but just a short version to give you some background information. We will then focus on what Longitudinal Data are available via the UK Data Service. And the next session will then focus on where would you find the data? How would you go about accessing the data? And most importantly, where do you find resources to get started and help if you get stuck? And in the end, we have a hands-on practical. You saw it in the course description. We will then have a practical bit and you are very welcome to do these exercises. We will give comments and hints and guidance on that. Now, before we start with the presentation, Mentimeter time. And I'm handing over to James. Yeah, so we're just going to start with some Mentimeter, just a little online quiz. I run the questions now, which should appear at the top of the screen. So if just join at menti.com. I use this code 69912950. You'll be able to answer these questions we've got. So the first question we've got today is, what do you associate with longitudinal data? So when you think of longitudinal data, what comes to mind? So we've got a few answers already. So we've got a few. It's good to see quite a lot of answers appearing. Yeah, repeated measures, panels data, repeated cohort study. Some answers in there, not sure what longitudinal data is. That's fine. Good to see you here today to get more of an understanding about following data over long periods of time. Long-term studies. I'm serious, difficult to get hold of. Yes, it can be. Yeah, great. So it's quite a good mix of understand levels in there and be able to agree to most of the responses we received and still going up. So between our responses, which is quite good. Yeah, thank you to all those who answered during that question. Moving on to the second question we have. What research topic are you interested in? So I want to know what area of research background you come from? Why are you looking to do longitudinal data? What area of longitudinal data do you want to research? Maybe looking at sort of health or labor economics. So really we've got quite a few. We've got health is quite a big one there. Household economics, disabilities, education, inequality, ethnicity. No, that's really good. I thank you all for answering really quickly as well. It's probably one of the better home sort of responses that have gone into me to that we've had. It just shows that longitudinal data. There's obviously loads of different uses. You can use it for loads of different research topics that can be used. Long-tune data can probably be used to research a majority of these things. Yeah, thank you for those responses. Let's move on to our third and final question then. So it's just a bit of a multiple choice question. So we've got three questions, some research questions here. We want to know which of these do you think cannot be explored using longitudinal data and it's maybe a bit subjective. So the first question is are people's chosen career path influenced by their parents' occupations? Can that be explored with longitudinal data? Can one individual's economic zones affect their mental health or is CBT an effective treatment for depression? So which one of these cannot be explored? Out of the three, which one would you say probably you might not be able to do as well using longitudinal data? Yeah, so we've got a good mix of answers there. Most of you answered the third one, so is CBT an effective treatment for depression? So I think, yeah, this is what we would say probably couldn't be able to explore as well with more data compared to the other two where you've got sort of people's chosen career path influenced by their parents' occupations. That's something you could look at. If you're looking at a cohort study, you would obviously have data on parents' occupations as well as sort of future what the cohort members would study or have as their occupation. And then the second one, how does an individual's economic circumstances affect their health? That's something you could look at over time in a longitudinal study in different health conditions and economic circumstances. Whereas the third one is CBT an effective treatment for depression. You may need to look at that in some sort of different study, maybe a clinical trial or something, explore that in a different way rather than sort of a longitudinal study. So that's generally what people are looking for there. So that takes us to the end of the little minty-meter question. Thank you all for your participation. That was good responses from that. So now we're just going to continue with the main presentation again. Yeah, thank you very much. And I'll hand you back to Björn to now. Okay, so we will now start with the introduction to the UK data service. That was very informative for your answer and I think we can use that as we go along for whatever comes and we will refer to that. Thank you very much for all your responses. So what is the UK data service? The UK data service is a comprehensive resource funded by the ESRC. It's a single point of access to a wide range of secondary social science data and how wide range that is you will see in a moment. We do not only provide data but also support, training and guidance. And we have provided here the URL for our website and this is a bit of our website which you can see there. And you see on the left-hand side in the very corner find data, a very important thing to click on. However, before we continue, just a reminder, all the slides and the recording of this event will be available afterwards. When you go to our webpage, training and events and this was upcoming events so far but it will be previous events when we have uploaded the slides and the recording. This is where you can find all the materials so you don't need to scribble everything down. You can access it afterwards. Now, who is the UK data service for? It is for academic researchers and students, for government analysts, for charities and foundations, for business consultants, for independent research centers and for think tanks. And where do the data sources come from? We're mainly from central government official agencies that's one source like the Office for National Statistics, for example. We have also international statistical time series data but also individual academics data. So what that is is, if as a researcher you have a research grant, let's say from the USRC, for example, part of the conditions is that you have to deposit the data three months after you finished your project with us and make it available for secondary research. So we have those data sets as well. We have data from market research agencies. We hold public records from historical sources and we have access to international data via links with other data archives worldwide. The types of data collections that we hold are first survey microdata, so cross-sectional data, final longitudinal data, international microdata. Second, aggregate statistics, international microdata. Third, census data. And here we hold accurate data, flow data, microdata and boundary data. And fourth, we have qualitative and mixed methods data. And what I mean by that is interview transcripts, diaries, anthropological field notes, answers to open-ended surveys, audio-visual recordings and images. These are quite the wide range of data types. A different way of saying what we hold is looking at these different types of data again, but in a different way. So UK service, longitudinal data, international data, qualitative data, census business administrative data and also controlled international microdata. And on the right-hand side there's a little bit more that all contains. I would actually like to highlight the controlled international microdata as well. So we hold in our safe room secure access data from the IRB in Germany, from CSD France and from Leases from Germany, which can be accessed. This is possible via our membership in the International Data Access Network. This is a project of six research data centers from France, Germany's Netherlands and the United Kingdom. On the aim here is to facilitate research use of controlled access data between these RDCs. So just to give you an idea of the scope, the UK data service holds almost nine in the half thousand studies. The majority are safeguarded data seven thousand six hundred ninety. And what that means is that after an initial process of registration you can access the data and you can actually download the data to your desktop and you can analyze the data. Then we have open data and we have controlled data or controlled access data and here two different types. So the main type is remotely accessible secure lab data and then we have a very few studies that you can only access in our safe home. James will talk about access conditions a bit more later on but just to give you a first idea of what sort of access level the data are coming with. So we have open safeguarded and controlled data. Open, I don't need to explain, safeguarded as I said you need to basically do an online registration process and you need to also agree to the end user license but it can all be done online and the controlled data so secure lab data access data they require a longer application process of approximately three months and you need to go through an application process that entails also a safe researcher training bit and you need to also pass a test for that before you're allowed to access the data. Now we are coming to the actual topic of today's workshop and that's longitudinal data. So longitudinal surveys involve repeated surveys of the same individuals at different points in time and you have said that actually in the beginning in the Mentimeter exercise I have seen that you also answered that it's repeated surveys so you know roughly what it is but very important and it actually gives you fantastic opportunities in terms of then causality analysis and we will come to that later on as well. So these longitudinal data that we hold large sample data they are nationally representative and also like in every longitudinal study there is an issue of attrition as you can imagine but the teams are usually doing quite a fantastic job of keeping numbers up by adding regularly new respondents and also by keeping the old respondents happy and contact them regularly so not to lose anyone. It's possible that sometimes in some sweeps respondents might not be able to answer and then come back and answer in the next sweep again. So longitudinal data allows a researcher to analyze change at an individual level and also to look into causation and I've seen that again in the Mentimeter exercise I was very happy to see that yes it is also about causation and it's very important and longitudinal data allow especially for analysis of that. So you can study for example the effect of different life events on for example health or the outcome of certain illnesses on other aspects of life for example education or employment et cetera et cetera and although the data are a little bit more complex to manage you will have seen that once you downloaded the data there is a lot of material out there on our website but also on other websites for example CLOSER and I will refer to that later which will actually help you very practically to manage and analyze the data and to get started so don't be afraid of the more complex nature of the data to give you an idea of the sort of data sets that we hold and make available when it comes to longitudinal data here is a list of some selected studies and I would like to start highlighting the cohort studies that we hold they are all from the Center for Longitudinal Studies in London at the UCL and these are the 1958 National Child Development Study MCDS 1970 British Cohort Study BCS 70 and the Millennium Cohort Study MCS and then when we jump down quite a bit to next steps formally the Longitudinal Study of Young People in England they are now also housed by the CLS they fit into that as well as you can see there is quite a gap between the 1970s British Cohort Study and the Millennium Cohort Study of almost 30 years and the next step study with 1989-90 date fits quite nicely in between those two and bridges the gap if you wish so we will look into the nature of these British Cohort Studies in a minute first I would like to then also highlight that we have the British Household Panel Survey and its successor understanding Society or UK Household Longitudinal Study it's two names for the same thing it doesn't matter which one you use it's the same study and this is the largest you will see I have some details regarding that on one of the next slides it's a fantastic resource we also have English Longitudinal Study of Aging I also saw that you are interested in the topic of aging and that definitely will hold a whole wealth of information in these sweeps that we have and make available growing up in Scotland as it says particularly focuses on kids growing up in Scotland very rich resource again and then the Cohort and Longitudinal Studies enhancement resources closer the British Birth Cohort Studies basically look into the impact of childhood conditions on education like later life health and economic circumstances and basically aim to understand children and families in the UK so just a few details regarding the three first mentioned Cohort Studies so they have quite a large sample size of 17,000 for the NCDS in the 1970 BCS and 19,000 for the MCS I have listed here the different ages at which the respondents were surveyed and you can see that they obviously aim to survey the respondents at the same age so it is actually comparable and we have inter Cohort comparability and they all look at a wide range of social, economic, health, medical and psychological issues so the National Child Development Study has become a unique resource for investigating social mobility and the impact of childhood conditions on later life and there are the main surveys and the main sweeps but there are also some with specific for key so for example there are certain ones focusing on partnership history employment history, social participation identity, response and death, parent migration, behavioural studies and essays so that already highlights we do not hold only quantitative data for those studies but also qualitative data so this is quite a unique resource combining both and we will elaborate on that later. The British Cohort Study then is very similar in design and coverage to the NCDS obviously because the whole point of that is comparability and some of the recent sweeps use common questions which is fantastic because then you have basically your excellent harmonisation and you don't need to afterwards invest so much time and effort to make the data comparable so very important opportunities for them and again some of the data collections have a special focus on for example education employment history and others. In the Millenium Cohort Study the special focus was for example on mothers who received fertility treatment and on physical activity and there was also a teacher survey. I would like to highlight regarding the NCS that they have actually looked into making it more appealing to those they are aiming to survey so you can imagine that a 14 year old is not very happy to sit down with paper and pen and complete a questionnaire so what can you do to engage your respondents and the the CLS has come up with a fantastic idea so they use a diary app and also an activity monitor so NCS was the first large scale population study in the world to incorporate objective measurement of physical activity and for that they have actually used for this called an accelerometer and they also used self-reported time use for the same period of time and basically they have developed an online diary and a bespoke smartphone to collect the data and you can imagine this is much more appealing to the generation of 14 year olds and also to actually go with time and make it relevant and to also have an objective measure that can be compared with what was answered in the app for example very interesting very successful I just wanted to highlight that so this is a snippet of the website of the CLS so please have a look yourself and see some more details but basically here also for cohort studies that I've mentioned before you find a lot of details there I would like to also highlight the COVID-19 service at the CLS and that is because it's very very impressive that when you think about the first lockdown having started in April 2020 that already by May 2020 there were surveys underway to basically research the impact of COVID on people's lives and so included as a four cohort studies but also the MRC National Survey of Health and Development and in March 2021 there were even blood samples taken from some of the participants to be analyzed for COVID-19 antibodies so again we had even biomedical measures for this particular survey there's also a closer block on that how the UK's longitudinal studies are helping society navigate the COVID-19 pandemic so that was very impressive how quickly the response came from longitudinal service and also from others and I will highlight that as we go along so three waves have been completed for that COVID-19 survey and the wave one done in May 2020 had already 18,000 participants so that's the beauty of it so if you wish CLS could already use the infrastructure the survey infrastructure that was in place to now quickly and very timely get insights into the lives of study participants including their physical and mental health and well-being family and relationships education work finances during the first national lockdown and actually all these findings of the longitudinal studies have actually informed policy and decision making so the second wave had the aim to capture how participants lives had changed from wave one until the late summer early autumn of 2020 and the topics of wave one were basically mirrored but with additional questions about health care financial transfers, life events and children's schooling some in autumn term because they were disrupted quite a lot as well as schools closed for long periods of time and then the third wave comprised questions included in the first and second waves but then again new additions including questions about the vaccination program and long COVID for example we have a particular webpage on our website listing all the COVID-19 data at the UK data service and here I would like to I have just screenshoted three but the last one again I would like to highlight when you look at that the coverage from 23rd of April onwards this is fantastic how quickly that came about and how quickly basically understanding society reacted to what was going on just very impressive now finally some CLS studies outlook so what's new I would like to mention three new studies the children's of the 2020 study which is a nationally representative first cohort study of babies in England which have been and this study has been commissioned by the department for education and the design is that the aim is to answer important scientific and policy questions regarding family early education and child care determinants of early school success and targeted our babies born in September to November 2021 so we're looking at 8500 families and five waves the second one is the early life cohort feasibility study which will be finalized in June 2024 and there's a website where you can actually follow the outcome of that and the third one is a COVID social mobility and opportunity study course which is a national cohort study of more than 12,000 young people from across England and here very importantly these were pupils in the 11 in the academic year 2020 2021 and this is a design to examine the short medium and long term impact of the COVID-19 pandemic on the educational inequality and social mobility of that particular cohort and the first data are already available COVID social mobility and opportunity study wave one, why is the UK data service here I would like to draw your attention to the content so obviously this study is there to cover the disruption to schooling during the pandemic but then again especially how that has affected young people's educational attainment and well-being in the long and in the short run and following from that obviously the career outcomes so wave one focused on experiences of the pandemic financial impacts, disruption to schooling, access to home learning attitudes to education, mental health and well-being and a GCSE assessment in 2021 wave two covered the activities since wave one university applications labour market and apprenticeship experiences vocational training and again core topics that are also covered in wave one such as mental health and well-being and very interesting on the left hand side of the slide you see the linkage so the study was designed with linkage opportunities in mind to later on enable linkage to administrative data from the national pupil database the longitudinal educational outcomes data set as well as other sources such as participation in the national tutoring program DWP HM Revenue and Customs universities and colleagues admission service higher education statistics agency and the student loans company so you can see in what direction that goes and that's really amazing that so many resources have already been in mind when designing that study so that actually there will be linkage opportunities is a really fantastic opportunity now coming to understanding society or the UK household longitudinal study I had said that's the biggest one you can see why I have said that it's a study of the socio-economic circumstances and attitudes of 100,000 individuals in 40,000 British households this is just incredible and it replaces and incorporates BHPS which run from 1991 to 2009 and you can see here the scope so in BHPS you were looking at 5500 households now in the understanding society survey you are looking at 40,000 households now so the understanding society study replaces and incorporates BHPS it retains the longevity while adding to the standard size and the scope of the study so there are new components for example an innovation panel greater detail on ethnic minority groups but also qualitative and biomedical data collections in principle the coverage includes employment earnings parenting, childcare family networks, benefit payments political party identification, household finances environmental behaviours and then also obviously consent is thought for linking to administrative data for example regarding health and education just to highlight one health assessment there was a nurse health assessment in wave 2 and 3 and 20,000 adults gave biomedical measures including blood pressure, weight height, waist, body fat, grip strength lung function and blood samples and this is all in dataset and 7251 and I would like to also draw your attention to the webinar that we have available on our website which is called an introduction to survey data on biomarkers very often when social scientists start looking into biomarker available biomarkers there is not much knowledge what can be used for what so also the data depositor is an institute for social economic research has provided fantastic information on their website and here you can see the URLs and also some snippets of what's available there on these web pages to give an overview on the biomarkers available in the society and their applications which is very helpful and also there is more information out there and small videos so this is actually very helpful and has enabled research that wasn't available for social scientists before because there was maybe a bit of a lack of background knowledge on that so data linkage usually we are looking into linkages with geographical identifiers and administrative data linkages for example when it comes to education health economic circumstances and transport and organizational linkages what is really good news is that we already offer harmonized data sets so we have for example 6614 which contains understanding society data waves 1 to 13 2009 to 222 and harmonized BHPS waves all in one data set so that spares a researcher quite a lot of work to harmonize the data themselves it's a very good opportunity and then we have this secure access version of that as well so all the latest waves are in there updated files from previous waves and also all 18 waves of BHPS obviously now understanding society as we saw on the list of our covid studies has a covid 19 study as well and that is a regular survey of households in the UK and basically this is specifically to enable research on the socioeconomic health consequences of the pandemic and you can have a look at that if you are interested in that so another study I would like to mention you also said you are interested in the topic of aging we have the English Longitude Study of Aging in our collection and here we have data from a representative sample of the population aged 50 plus in England on a range of indicators regarding well-being, economic circumstances, social participation and health so 10 waves have been collected and ELSA is a short version for that is a longitude survey of aging and quality of life among older people that explores the dynamic relationships between health and functioning social networks and participation and economic position as people plan for move into progress beyond retirement why is that interesting and relevant because one in three people in England are now over 50 which means it's really important to understand what life is like for England's aging population and also last but not least ELSA helps the government plan health care services and pension systems to best meet the needs of the growing population so summarizing the main objectives of ELSA are to construct waves of accessible and well documented panel data to provide the data in a convenient and timely fashion to the scientific and policy research community and describe health trajectories disability and healthy life expectancy in a representative sample of the English population aged 50 plus to examine the relationship between economic position and health the determinants of economic position in older age retirement and post retirement labor market activity and also understand the relationships between social support household structure and the transfer of assets also ELSA has a qualitative element so there are transcripts of short essays which are a result of respondent answers to the following question thinking back over your life with its wide variety of enjoyable as well as difficult experiences please write about three aspects of your life that have been especially important to you and how they affected you and that we have as 505 transcripts in the data collection the topics that are covered in ELSA COVID-19 study again a particular COVID study very quickly reacted again in 2020 looking into mental health financial security employment financial situation volunteering physical health and health behaviors social connection isolation and technological inclusion very important during that time in compensions and retirement so fantastic source as well there is an international platform out there which is called gateway to global aging data and I'm mentioning that because ELSA is actually feeding into that so the aim of this platform is to harmonize aging data from different countries and ELSA is also in there we have another webinar on this particular issue I provided the URL here underneath the small box closer is an interdisciplinary partnership of leading social and biomedical longitudinal population studies the UK data service and the British library and the work areas that are covered by closer are data discoverability policy and dialogue training and capacity building and data innovations there are different work packages harmonizing longitudinal data so just to mention some work package one for example focuses on harmonizing height weight and EMI in five longitudinal cohort studies and another one just to pick another example here work package 20 harmonizing mental health measures then at age 10 11 in selected British cohort studies so again there's lots of harmonizing work already done for the researchers to make it easier to access the data and utilize the richness of the data and finally I would like to introduce growing up in Scotland data which follows several cohorts and these are the cohorts 2023 245 and 1011 of the Scottish children from the early years to their childhood and beyond so look at health neighbourhood development education friendships and again the parents are asked for consent to linkages to research data the aim of the study is to provide information to support policymaking but it's also actually a broader resource that can be drawn on by academics, voluntary sector organizations and other interested parties so the fields of work here are cognitive, social emotional and behavioural development physical and mental health childcare, parenting family, community, networks and involvement in offending and risky behavior and there were just recently studies especially looking into exactly the last point involvement in offending and risky behavior at the transition phase from at the age of 11 the transition phase from primary to secondary school with very interesting findings now this diagram is just here to illustrate quickly the sources of information that feed into GUS so you have health and school records linked to it you have the main interview and then you have also biomedical measures which you can see here in different colours impact a very important and increasingly important topic when it comes to not just longitudinal data but data altogether so what sort of impact can be observed and I would just like to highlight that using the example of GUS growing up in Scotland so regarding policy for example GUS contributed to the development of Scottish Government policies and strategies it has been referenced by politicians in parliamentary debates it has been used by various organizations giving evidence to Scottish Parliament committees it has been used by local authorities to inform strategic development and it has been used by voluntary sector organizations to inform policy development and in practical terms it is used in discussions with nursery staff and parents but also when it comes to informing targeting of support funding applications or used as a national benchmark against which to compare local outcomes and also as evidence when it comes to improvements so you can basically track it so very very relevant and I don't have the time unfortunately to go more into that it's a very interesting field but just some more highlights or headlines for example understanding society data under PIN's child vaccine guidance so he is a chief medical officer referred to the research on school closures and children's difficulties and guidance to the health secretary during the pandemic or the think tank used understanding society to assess universal credit in the pandemic etc etc so invaluable resource and the last one before I'm handing back to my colleague James there was a very alarming finding in the millennium based on the millennium cohort study that one fifth of the 11 year olds were not able to read well and that basically 1.5 million British children at the age of 11 would be unable to read well and this is a great concern because obviously at the transition phase from primary to secondary school that means they cannot utilize what is on offer for them in terms of education and later in terms of employment if that basic skill is not there but last but also the poor reading could cost the UK 32 billion in growth by 2025 and several campaigns actually followed on from that and tried to improve reading abilities before it's basically too late and it was very very successful and one of those was read on get on campaign and there were others but we don't have time now to focus on that but just to mention it as one of the examples back to James thank you Beata I'm just going to touch on a couple more examples of how impactful pieces of research are used in longitudinal data the first one I want to talk about was this piece of research using the millennium cohort study this research looked at adolescent mental health and they found that one in four 14 year old girls and one in 10 14 year old boys suffered had experienced at least one mental health condition and they found this was actually increasing compared with data 10 years previously since obviously like an alarming sort of statistic they also found that the absence of mental health difficulty difficulty didn't equate to good mental well being and they made that sort of comparison that way one of the things one of the reasons why this piece of research was so impactful is that they had this infographic on the right which sort of highlighted different factors risk factors and protective factors associated with either mental health or well being and they grouped them into these four different categories and the colours you see their individual characteristics family relationships, home life, socio-economic circumstances and the wider school and they put so this infographic was used through different government bodies for education, public health England using it in sort of establishing their policies and focusing on sort of improving mental well being and sort of reducing that risk of mental health among adolescents because as this data showed this was on the increase so that's just one example and another one touching on again reading so this was using the 1970 British cohort study so ages 10 and 16 they asked cohort members the frequency at which they read for pleasure as well as them giving spelling tests vocabulary and sort of a mass arithmetic test they found that those cohort members that did often read for pleasure did better in all those things compared to those who didn't they're just sort of illustrates that reading for pleasure is good for general cognitive development in areas not just not just for reading but outside and one of the advantages of the longitudinal study is that actually asked the same question and did these same tests when these cohort members were 40 years old and they found that those who said they read for pleasure when they were younger actually did better in those tests again so that just really highlighted that the importance that reading for pleasure had on participants' cognitive development and their ability in sort of areas just outside of reading so this was quite an impactful piece of research had international impact so in libraries across the world I think in New Zealand, Australia they had sort of these campaigns to promote reading for pleasure due to these sort of positive effects that it achieved as a result of this research so this can be found on this web page at the bottom there if you want to go and read more about that now we're going to touch at how to actually find longitudinal data on our website so how do you go about finding the data sets you want to use so there's sort of three main ways you can do this I'm just going to explain all three of them then I'm going to show you a live demo how to do that on our website so the first way is through the data catalog on our home page you'll see the data catalog there so we'll show you how to find data sets using that not just longitudinal data but any kind of data you want to find we also have a find data so if you didn't want to just search data catalog but you wanted to maybe explore a bit more different topics we do have this find data option as well on our website and finally we have a variable question bank so if you had specific variables in mind or certain questions you can search for these specifically it will give you results of any data sets that have these variables in them so I'm just going to show you how to do that so hopefully you should all be able to see my screen on the web page just the UK data service home page so from here you'll see search our data catalog so if you just click on search there it will just bring up the data catalog it's very easy to use it just shows you at the moment if you haven't typed anything it will just display all of the studies we have so as I mentioned earlier we have 9,399 studies and you can also look at the data series larger series of data sets rather specific data sets for example understand society as well so it displays those and of course you can filter by different topics so if you just want to look at health then reduce the number of results there data type if you want to look at cohort and longitudinal studies again that filter is selected down there now we're looking at the cohort longitudinal studies that focus on the topic of health and you can filter further by access type the country in which the data is a different access type and of course the year as well and you can search the specific terms to sort of filter the results again a bit further if you wanted to search a bit deeper that way so that's the data catalog very simple to use and there's lots of different filters for that so the second option we talked about was finding data so find data you see at the top left of the home page there it's really easy to see find data this first box will take you to the data catalog again and the second box here is what we want to look at browse and access key data so if you select browse and access data it just gives you a way to access data or look at different data sets by different themes so you can browse by different themes maybe education ethnicity housing you can browse by data type as well so if you just want to look at UK surveys there on here business micro data and also we have teaching data sets covered as well as well as more general ways to search but on here if you go to browse by data type and click on longitudinal studies it's just a way to take you to the data catalog and you'll have the cohort longitudinal studies filter pre-selected and as you can see we've got a number of different longitudinal studies here to search through and the final way we talked about was the variable in question bank so on the data catalog underneath the search bar you'll see this option here says variable in question bank if you click on that we'll take to a new sort of search function type display where you can search for specific variables so any type of variable you're interested in so if it was a variation you can just search for that if you want to do variable in highest qualification and it will display all the types of all the variables that has relate to highest qualification and the data set it can be found in so obviously we've got a lot of results here these are basically 10,000 different types of variables across all the studies this isn't just longitudinal studies these are all different types of studies but of course you can filter these down a bit more if you wanted to so that's just a third way to actually if you had a particular variable or variables in mind or questions in mind you can use the variable in question bank as well moving on then to data access so we mentioned earlier about different data access types so who can access the data so essentially the data from the UK data service can be accessed by all registered users but however it does depend on what type of data you are accessing so access conditions can vary depending on the user type so wherever you are based at UK higher education institution or not maybe a commercial user so it also depends on the usage of the project or project type project characteristics whether that be commercial or non-commercial and also as well the specific data sets will have certain data access conditions associated with them which we'll touch on in a moment so the majority of the data from the UK data service can access online to data and obviously the metadata surrounding the data sets if you work for a student at a higher education institution a majority of the data that EUL data will be available to access for you and you can just register and download and have access to that data you can get the data in a range of different formats usually SSS or STATA but you do also have the tab format as well databases and spreadsheets are available as well as word process documents PDF documents there's obviously loads of metadata as well you'd often get the original questionnaires data dictionaries, user guides all those will be available to you along with the data set itself so the step by step how to access the data the first thing you would need to do is register with the UK data service so if you're a student at a university you'd register through your UK institutions absolutely your university account email set up a UK DA user name and password you'd then agree to the end user license select the data you want browse our catalogue using the methods we showed before once you find the data that you like you can click on it usually the access data or download and order you then have to sort of associate the data or put the data with a project you just set up a product online it's just an easy thing you'd need to summarise it can be a short 30 word description how you're using the data what you're using the data for what your research project is essentially then you can download the data if it's an URL data set or if there's any special conditions you can sort of place an order and there'll be certain further conditions or application steps you need to take but you'll be guided through that process when you request access to the data so the steps are outlined on this page so we've got a help and our help pages it's on help you'll see another page that takes you to how to download and order data so essentially it's just instructions on how to download and order data so if you step by step it's really easy to follow and the link's down there so once you do get a copy of the slides you can follow the link through that so that takes you through how to order the data essentially so as mentioned before we have different access conditions we have open license the end user license so this is when you register and agree to the end user license before we can access data some data sets have special conditions special license data which is sort of a tier slightly more restricted access to EURL there'll be certain permissions associated with that and then we have the controlled secure data access which is accessed through the secure lab so this is how they fit into the three different access levels open, safeguarded and controlled there we did touch on this earlier as well so the end user license so this is what a majority of the data sets we have so in order to gain access to the data you need to agree to the end user license essentially there's quite a lot turned in there but the main things to consider is to agree to is that you wouldn't share the data with anyone so once you agree to the end user license and download the data only you can access that data so if you had a colleague for example you couldn't share the data with them unless they've registered and agreed to the license themselves so that's really important to consider generally you can't use the data for commercial purposes if you are using the URL data for any commercial purpose you would need to get the appropriate permissions in place for that from the data owner you wouldn't use the data in any way to identify anyone as individuals households or organizations and you must use the correct data citation in the work in any publications you do use in the data we'll touch on that in the hands on practical and how to sort of use and find where to find the data citation and that's just other terms as well that are important to consider and of this point about here about don't disclose your login details to anyone else yeah that's pretty common sense school pretty straightforward don't share your login details with any colleagues you're working with your login it has to be 100% only you can use that but there's a link on our website to the URL and you'll be you'll be guided to to read that once you go request access to an URL data also I have controlled data so secure lab data can only be accessed via the secure lab so these cannot be downloaded they're accessed remotely so you would essentially remote in through a VPN into a remote virtual desktop secure environment where it's the only place you can access the data or initially if you have safe from data can you access what it's safe from so there's obviously more access requirements for these data sets the first being that you'd need to become an approved researcher and complete face-to-face safe researcher training you need to agree to the service user agreement and breaches penalties policy as well and so obviously the applications for these do take a bit longer obviously because there's a lot more steps that must the application must go through you need to have the applications are screened by the EK data archive and you'd obviously you would often need to be associated with an institution having ownership of the data and access is only granted to researchers who can can justify having access to the data if your research is doing public good and you have a good purpose for accessing data so these are things that are considered in your secure access application access is via online so I mentioned this earlier so you can access via the virtual private network technology and any research that you produce any publications need to go through output process and they're subject to statistical disclosure control so that's essentially anything that comes out of the secure lab outside of the trusted research environment had to go through our output checks covered once you undertake the safe research training the process on that is covered in that training and then I think we hand you back to Beata to cover exploring data online Thank you so we have an online data browsing and analysis system called Nesta which allows users to search for located browse and analyze and download a wide variety of statistical data within a web browser however I would always advise that you please download the data properly from the data catalog and analyze the data on your computer but Nesta is very very useful once you have logged in to browse the data and get a first field for it before you even download the data so I would actually recommend going there but a word of warning is needed here, not all of our data are available in Nesta so you find a selection there but not all of them so don't be misled thinking this is all what we have in our collection we don't we have many more, it's just a small sample of the data available in Nesta but what is in there can be browsed very easily and I will illustrate that in a second basically as I said already before you need to register and that can be done very quickly I saw in the chat there was also a question how quickly can you register for safeguarded data it's an online application so basically you submit your details and that should be very quickly then possible to download the data, add it to your basket if you wish, although basket is misleading because you don't pay for it, it's free so and then you can actually download the data and analyse it you just need to provide, I mean obviously agree to the end user licence read it in detail but also James has explained quite nicely what exactly you need to agree to what you can and cannot to provide some basic information, bits of information about yourself obviously your institutional affiliation etc then add the data set that you wish to analyse to your project that you have described in more than 30 words downloaded and analysed so very quickly once you are in Nesta and here I just wanted to provide where you need to go so you see on the very top of the box the URL that you need to go to in order to find all the information regarding online analysis tools and amongst others we have Nesta and that is usually quite popular by our users for a very quick browse especially for undergraduates excuse me now this is an example of what you can do in Nesta, this is a simple cost circulation I have here looked into parenting skills so how does one feel about their own parenting skills and I have cross-tabulated that by sex and one of the interesting outcomes is that males in that sample feel they are more able and they feel they are a very good parent not as many say that when it comes to the women in that MCS survey the parenting and another one is if you look at the highest qualification how that has developed over time and you can see where there are steep increases and where it goes slightly down but basically a huge upward trend things like this you can very quickly see in Nesta however I would then still recommend download the data and do it properly on your screen but a very first good feel for the data you get using Nesta back to you James for support and resources great thank you so we have a range of different support and resources made on our website so we have video tutorials, webinars we have different data skills modules resources for students case studies, guides, themes we have advice on managing sharing data we have teaching data sets and teaching data sets resources as well as help desk and individual so it's going to take you through some of these things now so most of our training and skills aspects on our website support resources available through the Learning Hub so just from the homepage if you go to Learning Hub at the top they'll take you through different sort of parts of our learning hub depending on what you want to look into so we have aspects on if you're completely new to using data we'll take you through the basics on using data on our website as well we have different data skills modules so depending on what type of analysis you want to do what type of data you want to look at we have different data skills modules we have this section for students and as well a different data type so it goes into the different types of data where it's survey data international qualitative and there's also one on longitudinal data as well so this is a longitudinal data skills module it gives you an introduction to work with longitudinal data I think it takes about two hours to start to finish but you can sort of save your progress and come back to it later so it takes you through what longitudinal data is how to find it, there's little quizzes throughout to check your understanding the examples are in SPSS but you can do the examples in whatever statistical software you'd like to use and it uses the understanding society data set the examples are from that and you can do get a small certificate on completion as well so if you want to have a nice introduction to the work with longitudinal data I do recommend this longitudinal data skills module and would you have the link to it on the slide there as well the case studies on the home page we do have a second on impact one part of this is the case studies so if you want to see case studies of how research using the UK data service data sets has actually impacted policy or impacted aspects in the real world you can look at the case studies web page initially when you go on it it does show you different case studies by different topics and themes so we have business and economy, environment sustainability and then health and well-being but if you did want to browse case studies yourself and look into it further they do have a browse all case studies at the top as well so when you click on that browsing case studies it will actually show you all the different case studies we have so these case studies are sort of produced throughout the year and they just really highlight a really key piece of research using UK data service data that has sort of impacted areas outside of academia whether that's public policy or areas such as that so they're very very interesting and a lot of work has been put into these so please do have a look and read through the case studies so it's not just longitudinal data so that's all data across the UK that are held by the UK data service so as I mentioned before we do have teaching resources so as part of our learning hub there's an aspect on teaching of the data set teaching with real data so from here we'll tell you different types of access conditions and use for using the data set for teaching we'll take you through different teaching data sets and different teaching resources depending on the type of data and some guidelines to consider such as data management and things like that so if you want to teach using the data then please do look at this aspect of the learning hub because that's really good resource and again if you want to search for teaching data sets longer to north specific teaching data sets if you go on our data catalogue teaching data set and then you filter the results by the cohort longitudinal studies we'll actually give you some examples of longitudinal studies teaching data set versions of longitudinal studies so for example we've got the understand society COVID-19 study teaching data set there we also have the help page so the top right you would have help so if there's anything you're unsure about this is the best page to go because it just outlines the different aspects of using the service so you may have issues with registration hopefully you'll find your answer in there advice for new users how to get an access data we have different ways that shows you how to access data in that section and also if you're a school lab user there's FAQs and questions and answers in the secure lab section so if you ever need help for anything go on UKdateservice.ac.uk slash help this is the best place to go initially and hopefully you'll find an answer to your query on this page we do have a specific page for longitudinal data in the help in the help web pages so this is just going to help and then you'll find a section for longitudinal data and it will take you to this page on longitudinal data and studies and it will just outline the main longitudinal data sets give you a brief overview of them and it will take you through or give you the link to actually go and access that data so if you want a nice overview of longitudinal data I do recommend this help page as well so that's just UKdateservice help and search through the data types and then you'll find the longitudinal data studies the last resource there I think that comes to the end of the initial slides we would like to now proceed with the hands-on session we just have 15 minutes left so that's not a lot of time but I think it might be actually quite useful to to basically apply what you have learned so far okay whoever is leaving now thank you very much and goodbye for the others please have a look at our first task slide we would like you to actually use the data catalog to filter all coherent longitudinal studies and then there are a couple of questions and James will guide you through that yeah so using the data catalog so what we showed earlier on our website which I want you to find answers to these questions here so the first one is how many results for cohort and longitudinal studies are there being displayed so if you go on the data catalog and apply that filter for cohort and longitudinal studies how many longitudinal studies do you get what number of results it's showing essentially second one how many cohort and longitudinal studies cover the topic of society and culture so if you filtered these results a bit further how many longitudinal studies come under this topic of society and culture third question is how many of these data sets now come under the safeguarded access conditions and also cover the United Kingdom so if you apply those two filters how many results are you left with and then finally if you do all those questions correctly what's the first page what's the first result that comes up so what's the study name and study number for this page and then a final question if you have time or you want to at the end can you use this catalog to find any longitudinal data set that will be useful for your research so we have that question in Northern Ireland so you can use it to search for any Northern Ireland data sets for example so to give you a couple minutes to complete this task so we'll go through the answers don't have to type the answers in any way it's just to really get use to using the data catalog so we can run through the answers on the following slide and then you can just sort of just check with them to see if you've got the correct ones okay just thinking of the time as we've got a few more little tasks to do and we've only got about 10 minutes left I'll show you the answers here so don't worry if you haven't got them all I will show you how to do this just quickly as well so the first one how many results for current longitudinal studies so if you filtered that it should have been 1,147 and then how many under society and culture that's 297 so looking at safeguarded and colours of the United Kingdom so that's 94 and the top result for this page would have been the under sand society special licence with the census and 711 LSOA identifies in them so that's study number 7248 so just sharing how to do that using the catalogue so if we just search in the catalogue and then applied that filter and for a data type use this with a 1,147 results so all the longitudinal studies we have in our collection and then we did the topic of society and culture so filtering that bit more we've got 297 results and then if we filtered by the access of safeguarded as well as looking at the country of the United Kingdom gives us with the 94 and then the top result there was study number 7248 so that's just a couple little quiz questions just have a go at using the data catalogue and different ways you can search for data so hopefully when you get time later on today you can use the catalogue and find some data sets that would be useful for your research and your studies so I hope that is there and then we're going to push you back to be able to cover the next few tasks for the practical yeah so the next task will be to find the data citation for study number 6614 and there's a small hint in brackets that you can find it under details the top details I think you will have found hopefully this citation and James because you are anywhere sharing your screen could you quickly please show them where to find the data citation for 6614 yeah so the first thing you would need to do is where we know already 6614 could find it yeah so from this page it's under the details sections there are notes on this page and it's just right there on the first page when you go to documentations it's basically a little bit up and then you see the tab here documentation and then we will find the data citation file in here that's a bit tricky because it's so many files actually meanwhile it's not a very good it's called data citation file the yeah and basically here you can see University of Essex Institute for Social Government Research and the citation for this data so it's a second paragraph you basically just copy and paste that bit and that is exactly what you will need for your reference section so exactly thank you very much for highlighting so this is all and it's a matter of copying and pasting the data citation and you have your reference ready okay so that was bad task and I think we have some more but we also have just five minutes so let's see how much is possible so basically that's an online analysis task and you are very welcome because as we said we have the slides available after this event on our website and you could actually do this task at your own leisure if you want and you will then also actually see whether you got it right that is actually quite a nice little exercise so basically what you would need to do is here to login or to register and if you need to register then obviously that will take up some minutes of your time and you might not be able to proceed now but anyway I would actually encourage you to register if you haven't done so yet with the UK data service in any case and then you would go to Nesta and you would click on teaching data sets and then you select the BHPS the particular one here the waves 1 to 11 and go to health behavior health status over last 12 months and then the question is what percentage of respondents reported excellent, good, fair, poor and very poor health that is the question and actually we will not reveal it yet because maybe you can actually use your last 4 minutes to get to the answer and if so please can you then type your answer into the chat no one so far but I mean it's a few steps so I understand that might take a while let's give it until 28 past 11 then we need to rub up I'm afraid but please you can also finish that afterwards and then compare whether your results match what the outcome should be half more minutes okay so this is what you would find actually this is how your your screen should look like when you go to the result and please feel free to compare that at your own leisure afterwards our contact details you can basically find us via these channels and otherwise as we said before if you have any particular questions go to the health desk link and email us we're looking forward to hearing from you and we hope this was very helpful enjoy the rest of your day and thank you very much