Welcome everyone to this presentation on content analysis. This is one very important area as far as media research is concerned. So I'm delighted to present on this particular topic. Over the next 45 minutes, I'll try and explain on qualitative content analysis, quantitative content analysis. I will also show you one example of how a good quantitative content analysis is done and the other dos and don'ts and some best practices of content analysis. So before we start, we must understand that content analysis is one of the types of desk research because here we are not interacting with human subjects or we do not interact with human beings in that kind of a sense as far as survey research or those kind of things are concerned. So along with systematic review historical research content analysis is also a very simple way of doing a desk research but it must not lull us into thinking that this is something extremely easy or it does not require any systematic effort and all. So today's presentation or the reason why we are having a presentation on content analysis is to provide all these important information about how these things are done. Now there are two important traditions in content analysis. One is the linguistic tradition which treats text as an object of analysis itself and we are looking at all the linguistic factors and the other is the sociological or the psychological tradition. We're very interested in the human experience that is manifested in the text or probably which is latent in the text. So it's the letter that we are more interested in although the first one is also something that we do in content analysis. Content analysis is not just merely counting of words or classifying large amount of text into an efficient number of categories. It is much more than that and we will see that these categories and themes are important but beyond that they can either represent some kind of explicit communication or inferred communication. At times this communication is manifest. It is visible from the text itself and at other times it is just inferred communication is something that is latent and we have to find out about the meaning between the lines and the goal of content analysis is to provide the knowledge and understanding of the phenomenon in that study. It's just like every other kind of research where we want to have a deeper and a more systematic understanding of whatever we are studying. True content analysis as the name itself suggests our idea is to distill the words into fewer categories or fewer content related categories because when we classify similar categories they share some kind of a similar meaning and we can draw some conclusions out of it but initially content analysis and even now is not regarded as a very rigorous technique by many people. The quantitative researchers think that it's a simplistic technique and it does not lend itself to detailed statistical analysis but we will see that it does lend itself to detailed statistical analysis and others they consider that it is not qualitative enough because we are only talking about words which are manifest or which are visible in the content and before we begin we must also make it clear that content could mean anything from a feature film to a television show to interviews to a podcast to print reports and maybe even open-ended answers that we get from a survey or whatever. So the text that we consider in content analysis is quite widespread. There are many kinds of texts that we'll be studying here so content analysis we will realize it does not proceed in a linear fashion most of the times although we will talk of the steps for second, third, fourth and fifth but it is less standardized and less formulaic because there is no one simple guideline for data analysis because analysis. Each inquiry is distinctive. Each content analysis is probably different from the other and it results and depends on the skills, the insights and the style of the investigator as well. So one way one caveat that we should put right at the beginning is that there is no one right way of doing it. There are many different ways of doing it so it's not something that is very linear or we can just put a formula about but although I said I will talk about all those steps that we generally do when we go for a content analysis. So we generally begin with a qualitative content analysis and it focuses on the content or the contextual meaning of the text. So we are interested about the contextual meaning of the text or what are the abstract themes that could be present in the text and we will be looking out for those abstract themes also and as I just said it could be verbal, it could be print, it could be electronic, the text that we are studying or the content that we are studying or it could be narrative responses or it could be open-ended survey questions, interviews or even focus group discussions and we want to look out for themes or even concept maps. In a later slide I will also discuss about what are concept maps. So it could be observations or it could be articles or books or manuals and we are trying to look for categories and themes in this kind of a text. So content analysis as I said it can be broken down into these five different steps. Every time the research question or the hypothesis is important. So our analysis or the data that we are looking for or the categories or the themes that we are looking for is dependent on the initial research question or the hypothesis that we have set out and then we select a sample. So a sample does not mean that always we have to select samples because when I talk about James Bond movies for example in one of the examples that I'm going to talk about there we will see that we are watching all the films that it's not about sample. So we will go for the sampling technique when there is so much of data that sampling would do the purpose and then we look for coding or we go for coding, we look for categories and we go for coding, we develop a code book or we develop a code sheet kind of thing and I will show you examples of code books and code sheets also. So we have a research question, we go for a sample, at times we might go for census as we say going for all the available data and then we define or then we look for categories or themes for coding and after we look for coding we train the coders, it's humanly coded, it can also be done by computer programs and we'll talk about that as well. So coders are trained, they code the content and then the reliability of the coding is checked. We will talk about reliability and intercoder reliability also today and finally we have to analyze and interpret the data. So all those I said there is no sample, there is no formula like we are going about it but these are the five steps of content analysis that we can talk about both for qualitative content analysis and quantitative content analysis. Qualitative content analysis is basically about social research, we'll talk about examples in a moment's time and quantitative is generally media research. So we should be knowing about both these forms. If I have to put it again this is available on the net so just the same thing in different diagrams. So we have research questions, it could be based on existing theories, we'll talk about deductive and inductive approaches in a moment's time and then we go for a hypothesis formulation. So it can either be a research question or hypothesis or we can have at times a combination then we go for sampling and unitizing. Unitizing means what are the units that we're going to study, we'll talk about that. We develop a coding scheme, we collect the data, then we do the coding and finally the reliability test and findings and conclusions. So it's just the same thing shown in a diagrammatic format, we'll talk about each of these things. So before we start the preparation, so as I said the first line is about the research question or the hypothesis. Is there an interesting problem that can be elicited? Is there some theoretical basis for that kind of a problem? And then we go for the other things as I said about the sampling, about data collection analysis and how you collect and analyze that. But the basic thing or the crux is about looking out for the interesting problems that needs elucidation. So before I go and talk about the actual coding process and the content analysis process, it's important to understand these two different meanings. So a meaning unit could be a sentence or a paragraph or in a newspaper report it could be just one graph as this, it's smaller than a paragraph or it could be the entire news report or it could be headlines. It depends on your research questions and what you're looking for and the kind of inferences that you're willing to draw out of it. So the first thing, the unit that you take is known as the meaning unit. And from the meaning unit, we condense it further. As you could say this is just about, this is from a medical journal. So the meaning unit is about there is a curious feeling in the head in some ways, empty in some way. We condense it just to take the important points of it. Here we say curious feeling of emptiness in the head. So rather than the exact transcription, we are condensing the meaning unit. It can be done as I said manually, it can be done through computers also. And then we finally coded through a few words, it just gives us a sense of what is happening. So we must have an understanding of what is a meaning unit, what is a condensed meaning unit, and then we'll go for codes and categories and themes as we go along. If I have to just show it to you in a diagram, so this is what it looks like. We start right at the bottom. If you see this is what a meaning unit looks like. This is the meaning unit. As I said, it could be a sentence, it could be a paragraph, it could be an entire article or a headline or it could be a certain interval of audiovisual data or it could be photographs or any kind of content. So this is at the lower level of abstraction. Then from the meaning unit, we derive the condensed meaning units as I just showed you. From there we go to three different processes. The first is to just look out for codes and these codes can be based on existing theories or we might be deriving new codes as we go along. I will talk about these approaches when I talk about these things in details. From codes we develop categories and from categories we develop themes and from themes we might be looking for an overarching theme. So if I'm doing qualitative content analysis, then I'm first of all coding all those units with some words etc and then I'm looking for some similarity in the codes and then those similar codes I'm putting them into categories and a lot of those categories I'm putting into themes. So as we go up from a meaning unit to the overarching theme, we are going higher in the level of abstraction. So there we are as we go higher up, we are more talking about the latent meaning of the text or the interpreted meaning of the text. So right from the meaning unit to the overarching theme, this is what we have to do in a qualitative content analysis process. So the purpose of creating categories is to provide us with a means of describing the phenomenon. So that with each category I can describe what is happening and our idea is to increase our understanding of the phenomenon and each of these categories is named using content characteristic words and at times we might even group them into subcategories and then we go as I said in the earlier diagram, we go higher up in the abstraction process. So that is one very simple way of going about the qualitative content analysis process. We have to as I said this is a iterative process, it's a reflective process so it's not that the first code you develop or the first scheme you develop is right and as we will realize it's also because when we're talking of qualitative research there's also a level of subjectivity there. So we go through those processes again and again, the written material is read through again and as many headlines or as many labels are written down to describe the aspects of the content. So this is known as an open coding process. So the list of categories then we group them into higher headings as I said code to categories, categories to themes, themes to overarching themes. So the idea is to reduce the number of categories because when you are analyzing or when you're trying to draw inferences or when you're trying to understand some process it's important to collapse those huge number of categories into similar and dissimilar ones and the as I said the important part is to read and reread the text while keeping the aim in focus and what is the aim? The aim is about the research questions or the hypothesis we have and that's where we have to embrace our intuitions also as I said because very often we will face problems about coding because two texts will appear very similar and there are too many categories we can put them into. I will talk about those things. So that's where your intuition works and that's where if you have an understanding about how this goes it's much easier. So we have to talk about what the text is talking about and what stands out in the text, how did we react when we just saw the text for the first time? So I mean it's also about as we go along, as we go along coding we also look for these cases, what are the places where certain things are working and what are the places where certain things are not working? So it's important that we have those things in mind as we carry along. So code is more of a label, a name that exactly describes what this particular condensed meaning unit is about. So when I showed you that particular content I was talking about what that meaning unit is about or what that condensed meaning is about. So that can be any number of words. It's what comes into mind as we said it can be about existing codes or existing code books that are there or we can be doing it inductively. We could be looking out for whatever we are trying to analyze. It could be say for example we could be looking into political or business or sports and that would be a very crude code to describe but we will see a very sophisticated kind of codes as we go along. So the category is found by grouping together these codes which are related to each other. Whenever we find out codes which are related to each other we'll try and classify them into categories. So codes are organized into a category when they are describing different aspects, similarities or differences of the text content that belong together. And if there are an abundance of codes that means there are so many codes that we can't get them into categories then we generally go for subcategories. We go for certain smaller categories and then we group them into larger categories. So this categorization process is as I said an iterative process and it's also a reflexive process. So category basically answers the questions about who, what, when, where. So categories are an expression of manifest content something which is visible. We've just spoken about manifest and latent. We'll talk about that as well but categories is something that is directly visible or something which is visible to us as we go along. So one of the researchers had a very nice way of describing categories is that categories are internally homogeneous and externally heterogeneous. So if you look at it they appear different but there is an internal association between them. So our idea is to look for those kind of associations in that qualitative process and the theme is basically looking for the latent content. So if I'm having two or more categories I can be seeing some kind of a theme on those categories. As I said we'll talk about those examples as well. So themes are expressing data more at an abstract level or at an interpretive level. So themes basically answers questions like why, how and in what way or by what means. So this is not about the manifest content but then underlying a meaning of the content. So that is what is described as a theme and theme is basically a means to connect with the reader both at an intellectual and an emotional level. So when we're deciding or when we describe themes then we're basically using poetic and metaphoric languages. In codes we will be using just straightforward languages to describe what is happening there. Then the categories would be at another higher level and finally the themes. But as I was telling you in the beginning there are quite a few problems which we must be careful about as we go about doing these seemingly simple things. First of all a lot of the materials that we come across in fact content analysis as I said right at the beginning is about desk research. So a lot of the material does not seem to answer the research question. So it doesn't appear, I mean we are looking for as I said we are looking for those themes or those categories or codes to answer the research question and they might not be visible right at the beginning. And that's why we said we go for that iterative process. And then the same thing can be leveled in several different ways. So how to code that and especially when we have students doing it right at the beginning or people are starting off with content analysis then this is a big challenge. Because the same unit can be leveled in several different ways. So how do we go about it? And the codes seem to be so if I code so as we go on to the next level of including codes into category. So certain codes that fit more than one category. So what kind of categories do we put those codes into? As I said we are going to you know going at these two, three, four levels. So that is one area that we have to be careful about. So one way to go about it is to first sort those codes into narrower subcategories and then reviewing for possibilities of further aggregation into categories and so on. So this is what we have to be careful about. And as I have repeated this before as well we have to identify and keep on you know reflecting on the initial analysis and adjusting it as we go along. Are you satisfied with these meaning units? Do I need to further decrease or increase the meaning units? Because you know if the meaning units are too big then there will be a lot of they can be coded into many different ways. If it is too small then it will be very difficult for us to do the analysis itself. So after starting also we have to you know look for whether these meaning units are they good enough or the condensed meaning units are they good enough or the codes do they fit with each other or they fit into a particular category or not. So we have to keep on adjusting. So as I said the unit of analysis is extremely important whether you look for a for a sentence or a portion of pages or paragraphs or if we are talking about you know interviews and the number of participants or it could be about time. So what is the unit that we are deciding and that's a very very important decision we make before we go along with our content analysis process. So again this is another diagram about the same thing about the coding system and the four processes. The first is of decontextualization when we are reading the text of just to identify what are the meaning units. When we are talking about the first part the first part is just to look for the meaning units. Where do I go for the meaning units and then in stage two again I go to the recontextualization because here we have to include the content and exclude all the fluff as we see or as the authors here say excluding the draws here. And finally looking for categorization. So where to look for categorization. I will just talk again about that and then we also combine and look for those themes or we could be looking for some kind of a concept there as well. So it could be a manifest analysis which is there on the surface. So we go for the coding system. It could be inductive or deductive. Then we compare with the original data and then we you know use the words to find out and underline meaning in the text. We will just now talk about the sample and that is again a very important thing. Do we study the entire text or do we study parts of the text or do we go for some kind of a sampling as we do in a survey research process. So when the document is too large to be analyzed in its entirety then we basically go for the sampling process and the sampling process can be you know when when I'm talking about newspapers it could be a constructed weak sampling process. It could be random or it could be stratified. We will just talk about those things as we go on to our as we carry on with the presentation. So the sample could be random. We could be looking at random units because as we know the random sampling has its advantages so often we go for that. It could be stratification. We could be looking at certain geographical areas or we could be looking at certain regions or we could be looking at certain categories of population or whatever. It could be intervals. For example, I'm studying if I'm looking for a content analysis of commercials on television then I could be looking for at you know maybe the third advertisement every prime time or whatever. So we might be looking for a smaller number of units to work on. We also go for something which is known as a constructed weak sampling. So if you have to sample newspapers for example then we go for a constructed week in a manner say for example if you start on a Monday then you take the eighth day so that would be a Tuesday another eighth day that would be a Wednesday the next eighth day. So this assumes that when we take the different parts of the day then we get a proper sample of a newspaper. Say for example if I start from say for example on 1st January then I go to the 9th January and then the next one and the next one. So that would give me a constructed weak sampling. There are different ways of doing this constructed weak sampling I've just told you of one way. I could be going for clusters or multi-stage multi-stage would be first of all looking for which of the newspapers. So say for example if I have five newspapers I might first of all go for a random process of identifying which newspapers to sample and then again from those newspapers I could be going for a constructed weak sampling. So it could be two stages of sampling there but important that at times sampling is necessary at times we'll use the entire data that is available. We have spoken about the latent and the manifest content and we'll keep on talking about that because as we go along the abstraction process the manifest data is important the latent data is important. It's not just manifest it's what is behind the or it's what what is available as to an interpreter or what you can interpret as being there behind the scenes kind of a thing or between the lines kind of a process. It's important to know about these inductive or deductive approaches. In deductive approaches the theory or a similar experiment has been done or a similar analysis has been done I know of the categories and in my content I'm looking for those categories. I'm trying to see whether those categories are present in the content that I'm doing there as well and in an inductive process there is no theory so I'm looking for codes and categories as I do my coding. So it's kind of an open coding where depending on my research question and depending on how I see the text I am looking for these categories by myself. So we have to know about these deductive and inductive coding processes. So in inductive content analysis we are creating this open coding and we are creating the labels or our headings in the text while reading it. So it can be done manually if you are reading a newspaper for example you could be writing beside in the margins or whatever. If you're doing it in computers you could be writing down what these nodes or what these codes are as you go along by putting them as samples. I'll talk about some of these software won't be able to demonstrate it here but I'll talk about the software used for these coding processes. So inductive content analysis is also very popular one because especially when we're dealing with data or especially when we're dealing with text which was not available or which is a new kind of a text there this inductive process of content analysis is very helpful. In deductive content analysis we are just testing categories or concepts or models or hypotheses. So we have a matrix already and we are looking at our data and trying to fit in whether you know those kind of things are available in my text that I'm analyzing or not. So deductive approach is also a very important approach especially when we are talking about proving or disproving existing concepts or categories or models or hypotheses. And one way of generating categories especially for the inductive process is could be from your initial research questions and also emergent research questions. As I go on with analysis a lot of these things will come with the analysis so we might start off with initial research questions but as I do the analysis these research questions will be adapted or we'll have more to think about or it could be the categories could be about substantive policy and theoretical issues so it could be about some of the theories that we'll talk about or it could be about things that are that are more important now. For example if I'm doing a content analysis of COVID-19 reporting now there'll be a lot about vaccines and you know about government policies and these kind of things. So these are areas which tell me that these are the categories that I can think of because as you can understand this imagination, intuition and previous knowledge is important for us to generate categories otherwise the entire content would be would be not explainable scientifically if I cannot generate important categories which are which are seen from the content or which are manifest in the content itself. So deciding on those codes or deciding on those categories is a very very important thing. So I'll just explain this entire thing once again through this diagram. So first of all we go for the preparation phase where we are talking about when we talk about the sampling and the selecting of the unit of analysis and making sense of the whole data. So this is the preparatory phase and as I've just suggested we go for either the inductive approach where there are no existing codes. We are coding our content as if we were doing it for the first time and which is what open coding is about or it could be the deductive approach where there are existing theories and we are just trying to see whether those can be applied to the text that we have available. So open coding we then develop those coding sheets. I will talk about those coding sheets. I will show you one or two examples and then we group them and then we categorize them and then we go for the abstraction. So it's basically the same thing. It's about the codes and then looking for the similar codes and putting them into categories and getting the categories into themes. So important to remember the codes, the categories and themes and themes is where there is an abstraction. And finally you could be looking at a model or a conceptual system or a conceptual map or you are looking at describing the entire process through some abstract theoretical construct or something which can describe the entire text that has been analyzed. And when you're doing a deductive approach then we have these structured matrices and then we are coding according to the categories which are there which a researcher has done earlier. We are just trying to look for correspondence or comparison or those kind of things. And there is another way where we start off with theories but then we look for our own codes there. So it's like an inductive after we start a deductive process. So if I have to describe this, this was by L.O.N. Kinga in this J.A.N. in 2008. So I have adapted it from their particular work. As we were talking about when we are coding or when we invite people to code our content or when we are even using a computer program or a software to code the content, the first and the most important thing is to develop a code book. And the code book is done so that the categories exhaustive that whatever could be there in the content, the explanation is there in the code book. It explains that if the content is like this then it could be coded like this. I will just show you one or two pages of a code book. So in code book the instructions are very clear. It could be a number also that if you see a business news then you code it as 101. If you see a business news which is negative then you code it as 102. We will talk about those things. So a code book is the most important thing in a content analysis. If you are doing a content analysis and if you do not have a code book in the appendix or if you do not have one to show, then there will be questions on your research itself. People will not regard your... People will not regard your content to be scientific if you do not have a proper code book to show. So it may draw on existing ones and much of it is very easily available. If you just google code book you'll get lots and lots of code book about research that has already been done. And at times there are also existing content analysis dictionaries and in this age of natural language processing and all these software these dictionaries are very easily available. Just like the sentiment analysis test we do in social media that if it is this particular word then it could belong to that kind of a category. So these dictionaries specify a range of concepts and the words or phrases that can be described as these concepts. So for example just any word, for example hospitalization if there's a word, then what could be the concept that it could be talking about? So it could be a medical concept or it could be an emergency concept or those kinds of concepts. So these dictionaries are available both manually and for computer programs or you could be doing it right from scratch and about your content you know that these are the categories that could be there. So you drop an exhaustive and mutually exclusive list exhaustive in the sense that whatever could be adopted is there mutually exclusive the same code must not fall into another category or there should not be two different codes for similar content. So after we decide on the code book we go for a pilot study and this is important just like in server research and in questionnaires and in other kind of research it's important to have a randomly selected sample of the data. Go for that pilot study using the code book that we have just developed or the coding scheme that we have just developed because there could be problems that can be corrected at this stage before the actual study proceeds we need to correct that certain codes might not be working or certain codes might be falling you know toward three different phrases as we can understand that we are starting off with an abstraction process so it has to be done as a pilot work before we do the actual content analysis. And one another thing that I want to talk about right here because as I said you know we're trying to put in a lot of things into this 45 minute presentation. One is about reliability so if I'm having seven or eight or 10 or 15 or more than one person you know coding the same content we must have agreement between those two people and as we said we have a document from which they are seeing and coding so there must be a lot of agreement in what they're doing. If the agreement or if two coders do not or if you know there are more than one coders who are coding the same news story for example and then they're coding it differently then there is disagreement between them and if there is disagreement then the intercoder reliability is small so we are looking for a lot of agreement a lot of people put a benchmark to that so if it is less than 80 percent agreement then we are not going to take the content analysis so we have to develop the code book and we have to do a pilot study and then from a sample material we are looking for you know whether people are doing it in a similar way so if there is one code everybody should be putting the same content into that kind of code and there are measures like the Cohen's Kappa and Scott's Pi which measure this reliability that whoever is doing it will provide the same kind of a coding so it's important otherwise it will be very unscientific. This is one of the concept maps that I was talking about so in qualitative content analysis it could be about positive leadership and what are the different themes that get out of it so at the end say for example if you could be talking about interviewing a lot of people through open-ended questions and this is a concept map that you get as a result of the qualitative content analysis so with this I end the qualitative content analysis part and I go on to a more interesting part and here we are going to talk about 007 James Bond unfortunately I can't play the theme music here but you can imagine the theme music being played in the background so this is I'm just using some parts of this particular article present you know in a recent journal and this is about you know written by Kimberly Nunezdoff is one of the gurus of content analysis as we know so this is about content analysis of women's portrayals in James Bond films so here we will be doing the same coding process but we'll also be applying statistical techniques here to gather some insight into the thing being or the text being studied so there are lots of research questions then I'm just showing you just to show you how a content analysis is done so the first research question is have the representations of physical characteristics of females portrayed in Bond films changed over time so I'm sure that a lot of our students and many of us when we do content analysis we think that content analysis is just you know analyzing the content but we have to understand the systematic way in which it has to be done has the amount and level of violence against female characters portrayed in Bond films changed over time so level of violence against female characters there are a lot of other research questions I've just I've just put out only three is a female character's end of film mortality predicted by her physical characteristics role prominence sexual activity and aggressive predispositions so content analysis does these two things at the same time it describes the text that we are studying and it also provides some kind of predictions that given these conditions this will happen so for any research to be scientific if it can make some predictions then it's it's a wonderful research so these researchers they have answered this fourth research question as well and here the unit of data collection is each female character who appears in the film for at least five seconds either she appears or somebody speaks about her for at least five seconds in two or more scenes if the character is there in just one scene then they are not I mean taking that kind of a data so the unit of collection is the female character who appears in more than one scene if the character appears in one scene that is not the unit that we are seeing so these people they developed a 13 page code book and each of the female character was coded on those attributes it could be demographic physical role characteristics and a lot of these things I have that code book if somebody needs that you might write to me I can share it with you the team of coders was eight and they call their PhD students as graduate students so these coders were eight PhD students and they were thoroughly trained in this 13 page code book and what is the code book they are saying that if it is this then you have to code it like that and adjustments in the coding scheme as I said were done after the pilot project and they did not go for sample they went for all these 20 films so this is from the journal itself starting from Dr. Noh from Russia with love to to die another you know in 2002 so they went for a census of these 20 films and then it was as we said they did a reliability test on never say never again so this reliability test is not done on all the things here it is done on a separate kind of a film where they are trying to see whether all these eight coders they are coding the same kind of a thing and they found out that one coder they failed to identify two female characters the other seven coders they said okay that female character is codable because she was there for more than five seconds and in more than once in one coder failed to do that so they found the reality reliability was 97 percent I'm just trying to tell you that this is how reliability is done or this is how we have to be extremely careful about the coders or the whether the coders are coding the same kind of a thing and they also this is also in the journal they're looking for that multiple coder kappa coins kappa only the good bad variable had a lower value but since the agreement was also very low I'm not going to get into the details but just to show you that this is how the quantitative content analysis is done so they could be talking about the race about the identity about the accent about the hair color hair length physical appearance so this was about lots and lots of attributes of those persons and as you can understand it's a very very detailed kind of a test and this is some of the results of course I cannot talk about the entire thing but just to show you that this is how first of all they're talking about the descriptive part later on they'll be talking about the research question so a total of 195 female characters were coded in the 20 James Bond films so on an average as you can understand there are about 10 female characters the characters in each James Bond film the role of the female characters was coded into 52.3 of the cases as minor in 52.3 of the cases they are just minor in only 17.4 their role is major so that was one of the characteristics the characters age range from 16 to 17 70 years so again there was a scheme for deciding how to decide the age because they're not going to come and say that I'm this and this is my age or whatever this is what you have to find out the majority of the female characters were Caucasian there were whites about three out of four they were Caucasian and 2.6 percent were American Indian and only 8.2 percent were black women characters in James Bond movies again you know a very you know simple description about how many of them were alive only 10.3 percent were alive and with Bond 69.2 percent were presumed alive we don't know how they were 5.6 were presumed dead so this gives us an idea about you know what happens to female characters in James Bond movies and it makes a lot of sense you know to talk about all these movies in in in such details and then you know this is one of the discussions they show that framed with the social cognitive theory and existing literature and female images which suggests continued exposure to stereotypical media images this study provides further evidence on and so forth so we are talking about theories we are talking about existing literature we are talking about what this study has found out and the importance they're saying lies in the empirically verifiable demonstration so a lot of these things we assume that it's there in the Bond movies but we are providing an empirical evidence for those kind of things and this is just from the code sheet just to show you what a code sheet looks like so this could be you know the film ids so they were all those films they were having the ids and the coder you know all of those coders they were having the idea or whatever then you go and talk about the role and you know then you describe what that role is not described they are there in the code book I'm just showing you the code sheet based on the code book you know there is that code form and you have to fill up these things nowadays this can be done on computers as well but still you know you have these kind of forms so it could be about hair color hairstyle glasses body science physical appearance you know strong physical harm and and etc etc so these are so many categories and and quite exhaustive categories on which people are coding and they're ensuring that whoever codes using that code sheet they will or that code book they will code it similarly this is again from you know another study or by iron day it is from 1993 is written a book on qualitative research it's it's a quite a useful book so so if you're talking about say for example again you know some kind of a code book so just a small example that what are the categories of humor so when do you describe it's incongruity or when do you describe it's catharsis or when do you code it as values or when do you code it as victims so as you can understand it's it's not a very simple process it is something for which you have to have a deep understanding of the text itself this again is from a code book on the manifesto people are studying the manifestos of political parties in in america for example and this is what they're saying that it could be foreign special relationship positive and they're describing what it means it could be anti imperialism and as you can see these are the codes they're providing it could be when do we call it as anti imperialism when there are negative references to so on and so forth so the code book has to be extremely as we said exhaustive and mutually exclusive so you have to have a code book to start the research process so this is again a seven page code book and it is available on the internet if anyone is interested so whether when they talk about the thing about economy in the manifestos then you know whether they're talking about free enterprise whether they're talking about incentives whether they're talking about market regulation so as you can understand that these categories are not simple they just don't drop in from everywhere they're there they're based on some good important studies based on the research hypothesis based on the research questions and the hypothesis but if you are starting on you can do a deductive research because these code books are available you can just tailor it according to your needs and then you can start doing say for example an analysis of these kind of documents so so as I said the idea to show these code books is to show you that this is such an exhaustive and such a very rigorous way of describing content otherwise it wouldn't be scientific it wouldn't be academic and why would people regard our our work as citable if it is not something that is reliable so very important to describe these kind of things I'm just trying to show you three screenshots of computer programs which are are very important or which which I can I have worked on nvvo nvvo 12 is a very good software they provide a one month trial version so if someone of you if some of you is working on content analysis and you want to do that coding etc on on computer on your desktop just by downloading the pds etc then this is a very easy way of doing it if you don't like nvvo then the atlas ti that is also a very very good software it's all these things have trial versions as well so atlas ti 9 is a very good good software to work on content analysis both qualitative and quantitative I found this one also very useful this is called ddos so if you if you don't like the other two you can go for ddos so these are the three recommendations from me if you want to do it using a computer software so just to sum up all the things that we've spoken so far in the next two slides I'll just go and talk about what we have discussed so far once again so first of all you must have a theory and rationale about the content to be examined and why are you examining that content what do you want to get out of it or what what what what is what is the inference that you are trying to draw out of that so what are the research questions you have or what are the hypotheses that you have and that's important to start off and then we have to have a conceptualization what are the variables that we are going to study and how do you define them are you are we going for the dictionary type definitions or we you know are we providing our own definitions because as a researcher we are the boss so there are many ways of defining a given construct how do I want to define it and then you know looking for going for this operationalization because looking for for internal validity and all those kind of things the coding can be done human coding so using the code book and the coding form and all and it can be done using the computer edit text analysis so I've just spoken about three different software and there are a lot of linguistic inquiry and word count liwc these are customized dictionaries or you can create dictionaries also which can give you you know frequency list that you know which are the words which are occurring more frequently and which category would you want to code them into so on and so forth so as I said the possibilities are endless this is just one one process that I have described in this particular presentation and as I spoke about there is the importance of sampling so whether you go for the census as the authors did in that James Bond study by taking all the 20 films or do you go for some kind of a sampling process and that training and pilot reliability is very important because you train the coders and then you look for the intercoder reliability and that would help us to revise the code book and the coding form as we go along and then they say that we should have at least two coders to establish intercoder reliability because if you're doing it yourself and if somebody else does it differently then that would be not not a good research and we could also be applying dictionaries as I said it could be there's a dictionary called KW IC keywords in context so that is one way and then finally the final intercoder reliability and then finally talking about the tabulation and reporting and when we're reporting these things it can be about using you know very sophisticated statistical processes so on and so forth so with this I end the presentation


Welcome. This is Melissa Armo with the stocks. What should I be viewing the market here today? So it's gonna be an interesting way for the market. And I say that for two reasons. One, we have a big number coming out Wednesday. It's a CPI number. The market's gonna react to that. Again, I don't know if it's gonna react positively or negatively, nobody knows because no one knows what the number is gonna be. That's point one. But point two is that actually we're in an area here where we widened the range in the market. We were stuck in the smaller range. Now we widen the range here that a lot of people think the market's just gonna move straight up. That's not necessarily the case. But what if we do? So we're in another area of resistance in the market that the market's gonna really show its hand in that reaction to the data this week. We got one more day, 24 hours. We didn't go anywhere today. They'll look at that. Actually sold off today after we rallied right at that level. Let's actually just look exactly 41762. That is pretty darn exact, peoples. Almost to the penny. So we're probably not gonna go much of anywhere tomorrow. It's all gonna be a Wednesday morning. Wednesday, Wednesday, Wednesday. So get up early, get ready to trade. I'm not saying I'm gonna short or go long Wednesday either way depending on what we do. I think we're gonna have a reaction one day Wednesday where the market's gonna show its hand even though everybody thinks this is it. We made the low for the year in June. We're gonna rally forever. I'm not buying it. I'm not buying it. I wouldn't be surprised at all for lower but also consequently would not be surprised if we are higher because this market has been very tricky, very tricky. And we've been doing selective things. So how do you make money in the market right now when it's choppy? Do selective stocks? Like today we did TSN. It was a nice trade. So look for selective things. If you wanna know how I make the picks and quite frankly how I read the market when it gaps or gaps up, market gaps down. How do I read it when I play it? You would sign up for the Golden Gap course which is late August, August 27th and 28th last class for the summer and only a few more classes this year. Crazy how far this year has just flown by how fast I should say. And it's almost gonna be 2023 before you know it. So if you're interested email me at melissathestockswish.com. Have a great day everybody and be careful this week.


day two of MCH 2022. As you probably all know, over the last few years, all over the world, electronically signed artifacts, let's call it that, became commonplace, even in emerging digital countries like Germany. But if you combine these with dynamic contact, this usually leads to disaster. So instead of screaming into the void, please welcome on stage Kirils to talk about all e-signatures in the world are broken. Thank you. Thank you, everybody. So the last presentation I did before COVID, I was going to do before COVID was actually called Travel for Hackers, how to travel the world easily.  So this is my first presentation abroad since COVID. Hope I'm not too rusty. Anyway, this is me. If you can see the screen, I sure can't. I work in a couple of places, but most importantly, I am a hacker, even though I do manage my own company on a day-to-day basis, I still like to hack. Many of the things I'm going to talk about are available on my presentation site, Kirils.org, if you want to take a look at them later on. Some of the research that I've done in the past and that I'm currently involved in besides this presentation here. In 2017, when Google published their research on Shattered, which is they practically demonstrated the collisions of one function, I took the opportunity to show to the Latvian government that it's a thing. And those of you who took a look at the exploit that Google created, even though they did not initially publish the tool they used to do that, it was really easy to repurpose it and basically create any two different PDF documents with the same SAH1 sum. A different project that we're working on right now is Verbal Security. It's a cool project. You're probably going to hear about it in a year or two when we're done with it, so we're trying to make sure that the devices that people are safe. Anyway, this talk here, really important thing I want to communicate here.  I would guess many of you are technical. I'm hoping that we do have some management people, if not in the room. Yeah? Ah, there we go. Then online as well. And with this talk, I want to bridge the gap between the technical people and the management people because, as you will see, that is what I think is what we need to actually fix the problem. And the problem isn't new on a technical level. It's been discussed on different mailing lists as long as 10 years ago, but we decided to revisit the problem because of COVID, right? And during COVID, as Harold Angel told you, we did have people using more and more digital technologies, people, companies, government agencies that did not use them before. And that kind of brought up this problem. So for the management people online and in the room, how do signatures work? That's, can you actually see that? You can, oh yeah, you have smaller screens. That's good. Basically, we take data, we hash it using, for example, SAH1, or maybe a more secure version of it. And then we sign it using cryptographic function, using, we use public and private key to sign it and then verify it, right? And the certificate is what proves that you are you. Government could, for example, give out the certificate to every person and then they could sign it. So what we are actually signing is the hash, not the actual document, right? But what's more important for this presentation and this vulnerability is we are hashing the bits. We are hashing the bits in the file. We are not actually signing what we see on the screen. And that's basically the whole concept here. Even though we may see one thing, the same bits might later turn out to be something else. Now, there are many different schemes that can be used for signing. In European Union, some of the most popular ones, the fact that we have PDF with embedded signatures, I hate it, but it's there. We have ASICE, which is a container. In Latvia, we also have EDOC, which is basically the same, the overall structure is the same. Now, since this is a container or a compound file, we can actually take a look inside it and it's really easy how it works. But one more thing with compound files is that you can put anything in it. And it's not a hacker thing. It's not that you can hack the government, you can hack the system, and you can sign, let's say, a JPEG or you can sign .mp3. It's intentional. European Union strongly holds that for ASICE containers, citizens should be able to sign any data. And that opens a huge Pandora's box, in my opinion. For example, I could sign, let's say an ISO file, that an ISO file can be interpreted in many different ways, as you know. I could even sign an ELF file, a program that every time you launch it, it displays a different content, different document. And it is perfectly legal, just because in the real life, this is much more complicated than just having a document. Now, that's where dynamic content comes in. Dynamic content are files, not signed, non-signed files where the representation changes due to some factors without bits changing. For example, we could use dynamic OLA objects in Word documents. Dynamic OLA objects are these things where you can put in a formula, or you can put in a different dynamic perspective into the object, and it basically changes every time you open the file. It can even change when the file is open, like this file right here. That's a GIF, obviously, right? We all remember GIFs from the, or GIFs, so if you're in that part of the debate, we all remember those from 20, 30 years ago, of course. But what you can do is, obviously, you can make the first frame, stay there for 20 seconds, for 10 minutes, for maybe even longer. I'm not sure what the max is there for that one. So it can change after a while. We also have static object pre-renders going back to office documents. Now, most office standards, including DocEx by Microsoft, supports or rather demands that when you use any new features that you embed in the document, the program you use to make the file with also creates a preview, which is basically a static graphic file for that object. Now, when you open such a document with modern software, the object loads. If you open it with old software that doesn't support it, you see the preview. Obviously, LibreOffice, Microsoft Office, they generate that automatically, but if you go inside the file, you can change it manually, and then you get different things for different viewers. Now, remote content is super interesting. Remote content means just as in HTML pages, you can load the remote image, for example, or remote iframe in your document. So you open it up, and it loads from the internet. And not only you can actually get dynamic content that way, you can also, of course, track document being open, and you can also decide what is going to display post sending the document, right? Instead of thinking ahead, you just have this link, and then in a year or two, you just change the content. So those are all real attacks, right? And these are for classic document formats. For other formats like GIF or ELF, that I mentioned, you can go even further. I'm gonna show you how it looks. And once again, it all works because we signed the bit, not what we see on the screen. So I have a couple of things here. First of all, the GIF file, of course. It's 613 bytes, so it's less than half a kilobyte, just a GIF, right? For this one here, I made it to change in 20 seconds. Of course, you could have text on it, and so on. Now, let's take a look at the dynamic content. Here's the dockets, right? If I opened up, it's not signed. The file is inside. If I opened up, it opens on my other screen. But basically, what we have here is a sentence. And we didn't spend the time to actually make it look believable, right? But basically, what happens here is it says, John will pay Peter this amount of money, right? John will pay Peter. If we close the document right here, and we try opening it again, we once again have John will pay Peter. And if we open it again, we have Peter will pay John, right? So this document could be signed, and we actually signed it and sent it to the government to just approve the point. And then it's hard to know what happens afterwards, right? How it's happening is quite easy. I extracted the dockets here. As you know, dockets are also compound files, which means you can just rename them to zip and open them up. And this is what's inside, right? So if we take a look here, we have this embeddings, right? And it's basically just an Excel file in this case, which has this great formula right here, which randomly chooses one of the things. Let me show you. Oops. Well, okay. Yeah, so randomly choose one of the cells and it works. That's how we get the simplest forms of dynamic content. Now, let's look at the actual signatures, right? What happens with that? Once again, since we're signing bits, it doesn't really matter. But if we open it up, it looks quite spectacular. So here we have the pre-signed document. I signed it with dynamic content. And this is the most popular standalone app that we use back in Latvia. I have signed it here. If I press validate, it's fully valid, right? And if I try to extract the dockets file, it's the same dockets file. Just for the ability, I'm gonna do that right now, of course, for you. So we have an empty folder here. Gonna not gonna work that way. Gonna export it into this folder here. Okay. And there it is. And once again, this is the same file, right? It says Peter owes some money to John. And if you open it once again, it will say John owes some money to Peter. Right. The other thing was the static preview. With the static preview, we opted to create a demo that will actually allow us to differentiate the kind of app, in a way, that's being used without it connecting to the internet or any analyzing servers. And the second file here. If you open it here with the official tool once again, what we're gonna see is, we're on screen again, what we're gonna see is it has a preview function. And if you open it up, it actually says right there, can't see that. It actually says right there, in bold on second lines, this document is opened with this specific program. And we're gonna say that always, every time you open it. But if we extract it, we will see that it's opened in normal way. And this is because of the preview function that is trapped previously, right? So this is the preview once again. And if we export it, and this is the other file, and we open it normally, it says its document is opened normally, right? So that's the problem, right? But technically, it's not that interesting. I mean, it was super easy to think of the exploit. It was quite easy to create the files. And it's not like it's a vulnerability. At least the government doesn't think it is. Now, of course, this also works for other signature schemes, like for example, a GPG or LibreOffice built-in signatures. Now, the question is, does it really matter? Is it really a problem? And this is the management, the part of my presentation where management people will fill it home. So for management people, what I've learned over the years is, it's not always the technical vulnerability. Is it there or is it not there? The mic is not working too well. Is it there or is it not there? The rather, what's important is, what happens if we exploit it? Can it be exploited and can we accept the risk? Now, does this really matter here? Well, the tempers that I demonstrated are apparent. So it tends to argue that if me and you signed an agreement like that and then I try to screw you over, you could go to court and an expert in court could tell this document was tampered in this way. It's not valid, even though we signed it. And problem solved, right? Because we do have fraud prosecution and that's what the government is betting on right now. So they will not fix it. I tried talking to different people. I contacted local Latvian cert, we got the government involved, the department's responsible for digital signature safety. We even got European partners involved. Most of the European partners didn't want anything to do with it. They wouldn't touch it with a stick. That's like, let's not talk about that. It's not a problem, right? See previous slide. Court will figure it out. So we talked and talked and talked and it didn't work. No one cared. In Latvia, we did do a coordinated release of information to general public, to developers about this risk, but it didn't really take off. So because it's not a problem, right? But now, what if we could make a document dynamic? But at the same time, if someone were to look at it in a court of law, they could not necessarily prove that it's been prepared specifically that way. Now, what I'm thinking here, of course, is remote content. And emails, we do use email. I have remote images turned off. 99% of email users don't. HTML, we have remote images loading from websites. So it's not inconceivable to convince the court that that's how your document department created the document. We just have this remote image there. It doesn't mean it's fraud, right? We use that for emails. It makes sense. So there is this potential problem with that. The other part is because this problem cannot be solved technologically and it cannot be solved politically. One party cannot solve it. We are thinking about always when we find vulnerability or when we just write an exploit for an old vulnerability like in this research here, we try to think, okay, we need to propose a solution, but we couldn't find any technological solution. And politically, they can't do anything about it neither. So we need to work together to be able to fix this. One proposal, the main proposal that we had, the one I personally like the best, is okay, if your opinion wants to allow the citizens to sign MP3 files or .exe files or malware samples, it's okay. We can leave that, but we need to create a new scheme specifically for signing documents because what people sign is what they have in front of them. We shouldn't be signing bits. We should be signing what we see on the screen and that is why we cannot fix it just by one party working, right? If we just change the technological part, then EU and some people are mad that they cannot sign MP3s. If we just change the political part, the technical vulnerability is still there. So that's the main problem there and that is my proposal how to deal with that thing over there. And I'm ready for Q&A here. If anyone has anything to suggest, I know that at the conference, we usually don't appreciate comments instead of questions, but since the tent here is only about one half full, I would love to hear from you. If you've been working with digital signatures or if you've been working on that before, that'd be really great because I have, I mean, I tried, we published this exactly a year ago, last July and not fixed. Thank you. Thank you, Kirils. Just a second, thanks for queuing up. Do we have any questions from the internets? No questions from the internets. So please, the front mic, please. Have you found any ways to explore this with PDF and PDF-A specifically? The newest PDF formats also have dynamic content in them. Yes. Our work-round solution that we propose to our partners and customers is that we use PDF version one with the only static JPEG in it. So that's how we do it right now. But modern PDFs can have dynamic content as well, yes. Do you know if PDF-A already is the restricted format that you would need? That is the safe format, I believe. We didn't find anything in there. The back mic, please. Please talk very closely to the microphone. Thank you. Yeah, it's not a question, it's only a comment. You mentioned politicians and such documents which are digitally signed. But in reality, nobody knows what is a digital signed document. Nobody use it. For example, a couple of politicians in Austria, Germany and Spain answered to an email coming from Ukraine. It was Vladimir Klitschko and it was only a video screen. And nobody knows or only some detected. That's not a question, it's only a comment. Thank you so much for the comment, yes. Maybe I am ahead of my time here. Maybe I'm talking about a problem which is a bit more complex than the actual problems that the users are facing. And I'm going to open up the signature application again. Actually, for the customers, some of the trainings we do, we include a slide on how to verify e-signature using the official app. And it's still a problem. So if I open one of the, it doesn't matter if it's dynamic, if it's temperature or not, if I open one of the documents. What we do is we open it up and we show it to them and we ask them, okay, so this is a screen. Is this document valid? Is the signature only valid? Most of you probably have never used the app. What do you think? Is the signature valid? How many of you think this is a valid signature that will, you don't trust me, do you? How many of you think it's an invalid signature? Okay, some more people. How many of you don't know? Oh, good. You're really smart. So the thing is, this is the official tool. It's the newest version of the official tool. We've had it for 10 years, I think. And you open up the document, it will not say if it's valid or not. You have to press the validate button. And only then, the blue thingy on the left bottom changes into green thingy. That's how the app works. They never, they aren't updating at NOI. So yes, thank you for the comment. And we do have lots of these problems with these signatures. And the real question, I guess, is how do we make it so that it's easy to use, but at the same time that people can trust it, right? Because it's easy to make a phone app that just displays a green check mark.  But what if I make a similar phone app that displays the same check mark? So trying to communicate the bits in the computer memory to a human user, which is not IT savvy, is always going to be a challenge, I believe. All right, do we have any more questions from the internet? No, no questions from the internet. But someone lined up to the microphone in the front. Thank you. Question, could you actually replace a document which is signed? So if I got a signed document, could you replace a document behind it? Thank you for the question. So the attack I'm talking about here talks about prepared documents. Two kinds of those, those were a court, in my opinion, could tell that it's been specifically maliciously prepared, and those were a court couldn't conclude that conclusively. There is attack in 2017 by Google called Shattered. But even there, for a limited amount of computing resources, you would need to maliciously prepare the document for it to be completely replaced. And they work with hash sums over there, meaning that we have two completely different pages which have the same hash. Now, for vulnerable hash sums, it is also possible, of course, to replace it post factum, right? If the hash is broken, and we can generate for a known input, for an output hash, we can generate arbitrary input. We can, of course, replace it with anything all the time, right? So in that sense, these documents, let's say, I'm just guessing, in 50 years, in 100 years, they will be worthless because people will be able to use these old hash algorithms and generate collisions, I believe. And one more. So you say we wanna sign what we see, actually. So would you say that a signed bitmap is okay, that this should be valid? Yes, that's exactly, that's exactly the direction we're going in, right? If we had the political tools, if I can call it that way, to create a new format, and it is political question, right? Then we would probably go for bitmap. That's the easiest, right? There are some strange things in there as well. I don't believe you can have transparency in bitmap, right? But there are different bitmap formats that you can use. Yes. Yes, please, Mike, one. If you want to reduce the whole problem to just having static images to show that you sign, you still have the problem that you need to trust the program that displays the image, and the hardware that the image gets displayed on. So all of that, I mean, it's reduced to only experts can really work with signed data because they have to understand the whole system. And you can't really trust just consumers to understand the subtleties of a system. I think that's the core of the problem. Right, so it makes you think if with electronic signatures, we kinda skipped the part that we are now debating for electronic voting, for internet voting, right? Everyone except Estonia thinks that internet voting, at the moment thinks that internet voting is not the brightest of ideas, and we aren't using it. Now for e-signatures, what the audience, what you are telling me today, I feel like maybe we just slept and didn't notice that that's happening because all that is true, of course. Right, one's more to the signal, Angel. No questions from the internet, unfortunately. Okay, that's fine. And any other questions from the audience? We still got a few minutes left, so feel free to bother him with anything. So it's broken, but how do we fix it? That's the question, right? Because I'm gonna repeat once again what I said in the presentation. It is not that big of a problem in the eyes of the management, and at the same time, it is kinda hard to fix. Front mic. Yes, so if you just allow static content to be signed, so let's say just text, and make sure you use a secure algorithm because if the algorithm is broken, everything's broken, wouldn't that solve the problem? And of course, make sure the chain of trust is complete, so save CPU, save hardware, save everything. Yeah, that would solve the problem out of the presentation, right? But people want and they actually need images or representations when signing some contracts. You can't just have basic text, right? And if you have just static images that cannot change, do not allow dynamic content. Yeah, yeah. From technical perspective, that's great. The problem is, right EU is a slow beast. Yeah, thanks. Any evidence of real world tampering has been found? Yes, I'm not sure if I can say more than that, but we got as a feedback from the people we contacted, we got info that there have been a limited amount of tampering. I think I can name the numbers. So for a country of approximately two million people, I believe it's about two cases per five years. And did it get solved in court? Police solved it. So yes, fraud protection. Final thing I'd like to add, please, if you have any contacts in the government or the formal non-hacker world, talk to them about this. We need more allies outside of Latvia and maybe we can fix it together. Thanks. All right, and with that, thanks very much, Kirillos.