In this video, we're going to mention 20 of the most important research papers that made language models as powerful as they are today. We're going to distill this by covering research since the 1940s, so let's get to it. First off, language models can predict the next word in a given sentence given its previous words. And these models don't actually process words, but they process numbers. And so we need a way to represent language with numbers so that these models can understand them. And eventually, they'll be able to perform intelligent tasks. And this is the basis of language processing. With that, the 20 papers that we're going to go through will represent language with numbers in the most effective way as possible. So let's get to it. We start off with the paper, Mathematical Theory of Communication. This paper was written in 1948 by Claude Shannon, who is regarded as the father of information theory. So essentially with this paper, since we want to represent language with numbers, we do so by representing sentence with n-gram vectors. This is done by first breaking down the sentence into its corresponding n-grams and determining which n-grams are present in the representation. This was one of the earliest ways that computers understood sentences. The second paper is the one titled Neural Probabilistic Language Models. So an issue with the n-gram vectors approach is that they were sparse vectors, so they're too large for computers to process. This paper introduced the idea of neural networks to solve the problem of language modeling. And during this training, the model learns also how to represent words with a dense, fixed-size vector, which may be of size like 64, 128, or 256 dimensions. This is a lot smaller in vector size than the n-gram vectors. The next paper here is NLP almost from scratch. So until now, we would need to learn vectors by training neural networks on language modeling or some other task. But if we change the problem, then we would need to train these word vectors from scratch. So instead of training a neural network on a specific problem, why don't we train it on multiple problems with a unifying architecture? And so different problems could share the same word vector representation. And this is somewhat a philosophy that we even follow today. Now in this research, the core architecture uses convolution to learn vectors. This was inspired by previous research on time delay neural networks introduced in 1989 for phoneme detection. So overall, the NLP almost from scratch paper introduces the idea of learning shared representations of words. Next up, we have the efficient estimation of word representations in vector space. Now, this is the paper that introduced word to VEC. So keep in mind that the goal, even since 1940s, paper was to represent text that is like words and sentences with some vector that preserves meaning and is tractable for computers to understand. The previous architectures in NLP almost from scratch does learn word representations that are reasonably high quality, but it needs a lot of data and a complex architecture to do so. Word to VEC learns word vectors, but with a much simpler architecture. Word to VEC itself is a framework of two architectures. One is C bow and the other is skip gram. We train this network by passing in and also passing out one hot vector of words, but this center layer is going to have the internal dense word representation that will eventually be learned. And so once it's trained, we're going to have a dictionary of every word followed by its vector representation. Now a similar paper is glove, which is global vectors for word representation. This is similar to word to VEC, but the vectors are formed instead of just using local context information. We use global co occurrence of words in order to form the final dense word representations. Another paper that is very similar also is enriching word vectors with subword information. This is the paper that introduced fast text, which is also similar to word to VEC, but it's very useful in morphologically rich languages like Finnish Arabic and some Indian languages. Morphologically rich languages have words that can change their forms slightly just based on gender or preposition. If you want more information on word to VEC, glove and fast text, I have a full video as well. Next is a convolution neural network for modeling sentences. So we've spoken about words and representing words as vectors, but we also want to represent sometimes sentences as fixed length vectors such that their meaning is preserved. So this DCNN does so using a convolution and some max K pooling operations. And this K really depends on the size of the input. And hence, this is a dynamic network. And this way longer sequences of words can be represented pretty well. For more information on dynamic convolution neural networks and also time delay neural networks, you can check out my companion video here. Now the next paper I want to introduce is learning internal representations by error propagation. So this is the paper that was written in 1985 and introduced recurrent neural networks. So recurrent neural networks like dynamic convolution neural networks can process sentences. They get representations of sentences and can even solve tasks involving these sentences. This dissertation is laced with however a ton of math. And so I would recommend actually reading chapter 10 of the deep learning book, because it actually gives a good intuition about recurrent neural networks from scratch and also has very nice visuals to accompany it. Next up is long short term memory. The main issue with traditional recurrent neural networks is that the gradients would either explode or vanish. So the LSTM architecture was introduced to combat this issue for longer inputs. Now the outline of this paper is kind of clear, but there is a lot of math laced into it as well. And so for this case, I would also recommend a popular blog post on this topic by Kola. And I think they explained LSTM's very good with some stunning visuals. For additional information, I have a full companion video on this topic too. And yes, I make a lot of videos. Now I also wanted to call to attention a PhD thesis by Ilya Sofskever. The premise here is the original RNNs and LSTMs that were introduced in the 80s and 90s didn't quite gain popularity until past the 2010s. A big reason for this was that they were very tricky to train. And this thesis showed how they can be stably trained using some techniques on parameter initialization, optimization, and regularization. And this is why recurrent neural networks for NLP became super popular around like the 2013 to 2016 time. Next up, we have deep contextualized word representations. So let's pivot back to word representations like using word2vec. Now the goal is to understand really high quality embeddings for words. Now the main issue with word2vec is that it really doesn't care about context. The same vector is generated no matter where the word is in the sentence. But this isn't good because the word meanings can change, but the vector doesn't. To solve this, ELMO considers context. So every word vector that is generated is a function of all words in the sentence. And it does so by having a bidirectional LSTM architecture. So LSTM networks help them train on long sequences. And the bidirectional component allows ELMO to understand words that come before it as well as words that come after it. The result is a deep contextualized word representation. Hence the paper. And we can fine tune this on any other NLP problem. Now the next paper I want to talk about is a very famous one called attention is all you need. Another issue with LSTMs and RNNs are that they are slow to train. And this is because data needs to be passed into them sequentially. The architecture consists of an encoder and a decoder that makes use of attention for processing long term dependencies kind of like LSTMs. Now this transform architecture was kind of the turning point from the recurrent base architectures to more attention based methods that set the stage for modern language models. For a brief overview on this topic, I have a video and also to code this out completely from scratch, I have a playlist of videos. More fun, more homework. Next up, we have BERT, the pre training of deep bidirectional transformers for language understanding. So BERT essentially is a stack of transformer encoders. The main issue with transformers is that we need to train the architecture from scratch every single time that we need to train on a new task, which is also language related. So language modeling, question answering, machine translation, each of them, we need a lot of examples of each. And this may not be super easy to get your hands on. So to deal with this, BERT split its training phase into two parts. One is a pre training phase where we train on language modeling and next sentence prediction. And then we fine tune it on whatever NLP task that we want to do. And the idea is that with fine tuning, you don't need as much data. Next up, we have improving language understanding by generative pre training. This is the paper that introduced GPT. So GPT is a stack of transformer decoders that also leverage transfer learning in order to learn language tasks. It dives into the same pre training and fine tuning phase that we also saw with BERT. Next up is the paper language models are unsupervised multi taskers. This is the paper that introduced GPT too. So the older versions of GPT and BERT still required quite a bit of data for fine tuning. Overfitting is also easy. And honestly, the fine tuning process is not exactly how humans would learn. We don't need like hundreds of thousands of fine tuned examples just to understand how to translate from one language to another or how to complete the sentence. We might need just like one or a few examples. And this is where meta learning, such as like zero shot learning, one shot learning and few shot learning come into play. Now, while GPT two wasn't super successful, this paper called the language models are few shot learners that introduced GPT three was pretty successful. So GPT three is essentially a scaled up version of GPT two, which has a suite of architecture such that its largest architecture is 175 billion parameters. Now, for more information on GPT one, two and three and its progression, I suggest you check out this video. Next up is sentence BERT sentence embeddings using Siamese BERT networks. This is the paper that delves into a new kind of architecture called sentence transformers. So BERT and GPT produce high quality word embeddings. But how do you get sentence embeddings from this? If you simply take like the average of the word embeddings that you get and just use that average vector as a sentence embedding, well, that's not very high quality. Instead, we would want to train a Siamese network of BERT and a pooling layer on a few tasks. So one of them is natural language inference where we check if one sentence either entails contradicts or is neutral with respect to another sentence. And the other task is sentence text similarity, where we have two sentences and want to see how similar they are to each other. Once these networks are trained, we can then just use that BERT plus pooling layer in order to input the words of a sentence. The pooling layer will take some form of weighted averaging. And then it's that vector that is going to be much higher quality and representing a sentence. The next recommendation here is the chat GPT blog. Chat GPT is probably one of the most well known items on this list, and it is a fine tune chatbot that makes use of reinforcement learning with human feedback. The original GPT architecture is fine tuned to answer questions. We then build a rewards model that says, given a question and an answer, give a score that says how acceptable this is. We then use reinforcement learning to fine tune the GPT model. And this is done to ensure that chat GPT is safe, factual and non-toxic. And I have a playlist of videos explaining this infographic too. And next up here, we have llama. So this is the main paper for llama to llama to is to GPT three as llama to chat is to chat GPT. So llama itself is a pre trained language model. It is open source architecture that is trained on publicly available data. Compared to GPT, it has a smaller architecture, but is trained on much more data. And because of its small architecture, inference becomes faster. And for more information on llama, I have another full companion video. That's all for today. I hope these papers helped you navigate the idea of what papers are important in the field of language modeling and natural language processing in general. If you like the video, please do give it a like subscribe and I will see you very soon in another one. Bye bye.


So we do financial services for startups. And so we have an application which does this financial analysis. We plug into accounting software like 0, and we get insights out of it. I will show you a bit how it works and what was the issue which brought us to want to use web workers. So I might be in a page where we want to do financial analysis of a company. I will pick a organization. So maybe a group of company which has two organizations. We have two loads. We analyze steps. And it will come through. And we'll see some insights for the company. We can change company. We do charts. So same, we get the data from the organization. And for one organization, we'll find analysis of income and expenses, profitability, cash, et cetera. And you can see that, yes, we have three analysis coming. I start taking a bit of time. But we have a second organization which we'll take again sometime. We have different periods. Sometimes other customers want to see a few months. Sometimes they want to see their full history. So I have a full three-year history of a business where we can very quickly analyze. We had only expenses. And until we built a product, then revenue started growing with profitability coming up. The point is to do financial analysis, we want to analyze many things. And so far, we have three on this page. And we started having some slowness with some customers. So we thought about a way to make it faster. And that's where Worker came. So we said we can get the data, calculate all this in the browser, and do many, many calculations. So to do that, we tested our Worker's works. And so I have a page. And the first solution would be to do plenty of calculation in the USUR JavaScript. So I would click, and we see what's happening. The browser is completely locked, which is why Worker's were invented. Because here we have maybe 10 dozen calculations. So these are random, but you get the point. So there are eight things going on, which takes around 10 seconds. In the meantime, the browser was completely locked. Well, if I do the same with Worker, I can select, and it's all going on. So we can imagine the speed and the experience we can provide if we do that for all the analysis we want to do. My base is clear with the rationale. And seeing that is very exciting to me, because I know that we'll be able to do all this super cool calculation very quickly. So to do it again, once it's loading, and it's all being calculated, I can really interact. My user interface is not blocked. And the calculations are going in the background. So if we want to see, here we go, and it works. So another beauty of Worker's is that they are super easy to implement in the font. And I want you to do it. Yes. So to launch a Web Worker, we just create a new Worker. And then we don't pass a function. We pass a URL of a file. Actually, there are two ways of launching Worker's. Either through a URL or through a blob to an aggregation. We went for the URL solution. So you call a Worker. You now have your Worker in your code. And you communicate with message, because it's so asynchronous. So you post a message. You can post an object. And then you listen to the Worker itself to receive a message what's happening and get your response. On the Worker side, you would listen to messages. And once you listen, you can do whatever you want. In our case, when you can see, we call it. So we have many calculations going on and many functions. We didn't want to create one individual file for every potential Worker. Because basically, we have a module, a library, with all the potential calculations. So we want to have always the same. And we call the proper function. So we use in the data that we send itself which function we want to call. And the listeners, so we have a Worker Manager, which will listen to that, execute, and communicate directly with that. So that's to show that it's super easy to implement. A few things which I find interesting also, and which you might like if you want to implement Worker. When, as you're passing a URL, you want to make sure you have data. You want to make sure you have your files available. So here, too. So when we minify and prepare our deployment script, we make sure the workers are separate. As we have a revision, we have a script which makes sure to have this version available for servers. But except that, it works. And another fun fact is that I'm sure when you do calculations, sometimes you love your browser because you do infinite loops. At least I'm guilty of that. I do that. And if you do that in normal, yeah, there is at least someone else. If you do that in your JavaScript, it kills your tab.  When you do an infinite loop in a Worker, you can still close the tab. So I find it quite cool. And I think I will stop here for a few minutes of presentation. Is there any question at this stage? Anyone now wants to use workers? Yeah, I convinced. I increased by 50% the number of people who want to do. Yeah, question. How is the support right now for developers? It's, I would say, all the modern browsers have it. We can check. I think it's pretty good. And in the back, the Worker is basically the exact same JavaScript. So the only thing that you need is the ability to create a Worker, to send a message, and receive a message. And so we see. Yeah, Chrome says that it's there since version 4, so for quite a long time. One more question? Yeah. I'm lucky our customers are using modern browsers, so they don't have counter issues so far. And again, looking at the table, I'm not too scared. And it's a bit to be up. So in a way, if there is, it's not like a consumer website where you need to cater to your customers. In this case, I can say we support this one. And you have to move if you want it. And they are quite up here with the service, so that's fine. Yeah, cool. Thank you very much.


Welcome, everyone. Thank you for joining us. Just as we begin, you know, to introduce myself, I'm Stephen Burke Krueger, head of the data strategy team at Parsons TKO. Parsons TKO, we are the engagement architects as an organization we pride ourselves in our ability to think through the experiences mission driven organizations need to create for their audiences to help accomplish their goals. A lot of our approach to this is what we call engagement architecture. So engagement architecture is the Parsons TKO philosophy and the methodology that helps everyone stitch together their strategy, the people within and externally, the processes that they use, and the platforms that they use to create those experiences for those audiences and drive meaningful lasting engagement. Of course, through all of this, there is the layer of data, the area that I specialize in and that my team specializes in. Data is a sort of a lifeblood for a lot of these these components of engagement architecture that helps us stitch it together helps stitch together the experiences. And that data can be used to build and to optimize new kinds of experiences for our audiences. A lot of those richer experiences end up taking the form of campaigns campaigns are the way in which we organize these experiences for our audiences. And to really thinking through all of these elements of engagement architecture, I introduced this not just because it's how we introduce ourselves, but because a lot of these concepts are going to be particularly important today as we talk about the role of data in creating new and creative and impactful kinds of campaigns. So the topic of the day impactful campaigns the marriage of data and creativity. What we're going to talk through is the role that data plays and help, helping to shape not just the experiences of the campaigns itself, but really to shape the way we as organizations as communications as we are professionals, engage with the process of creating that it's changing the way we work in with the goal of creating more impactful experiences for our audiences. So just to talk through the agenda a little bit how we're going to be spending our time. First, I want to talk a little bit big picture principles and goals of data, the ways in which we should think about data and campaigns. Let's get a bit into some of the the nitty gritty details examples of how we can work with data in the campaign process. Well wrap up with a couple of stories, you know, examples and sort of illustrations of what it looks like what things can be accomplished once you've successfully integrated data into your campaign planning process and campaign execution process. And I am optimistic. It's, I've got a bunch of slides but I'm optimistic we can go fast enough to have some time for q amp a at the end. So please don't be shy, use the chat box, ask the questions as we go. We will do our best to catch up with those at the end of the presentation, recognizing that we might be cutting it a bit close. I also just wanted to offer to everyone here that I will hold office hours immediately after this webinar. So if anyone has time in the next hour. And you want to hang back after class, I would love to continue the conversation and talk a bit more personally with anyone who can stay about their own circumstances and and their own questions. So that's the big picture there. Now I want to start off thinking about campaigns, thinking about data and using sort of proverbial lessons that we can bring into this process. The first thing I will say is that the best proverbs are either true or relevant. So it goes for proverbs. So it goes for a lot of the lessons I'm going to share today. We are all coming from very different circumstances. There is no one size fits all solution to using data in campaigns. So hear me out as we go through this, but really think about your own context think about your own organization, your own ability to shape and build campaigns, your own data resources that you already have access to. They're going to be lessons that need to be modified to be used well in your organization. But I will also say that my advice is as relevant as you make it. So these ideas are going to be instructive, but the actual success of data and campaigns comes from behavior change. So we're going to talk about that we're going to talk about some of the things that I'd like to see people in the nonprofit sector doing. And I really encourage you all as we go through to think about your organizations think about what you can change about the way you run campaigns. What comes up is storytelling storytelling is a topic that comes up a lot in communications comes up a lot in campaign planning and and comes a lot up a lot in the context of data. And I think it's important to reframe the way people often think about and talk about storytelling storytelling is not about you telling your story. It's about finding your place in your audience's story. storytelling is an empathetic activity. We need to understand the context of the people who we are talking to. And this is going to be true in content creation. It's, you know, well, well studied and well understood role of content creation. It's going to be true in how we think about the stories that we want to put into our campaigns, but it's especially true with data, and it remains true with data. A lot of people think data data itself is, you know, it stands alone it is its own solution data works when it can be applied to people in your organization to help those people make better decisions. Understanding your audience's context is important for storytelling, and that remains true for your internal audiences. How do the people on your team. Plan campaigns and execute campaigns, and how can you use data to help shape their mindset and help shape the decisions they make in that process. That's what's going to end up with a new type of campaign. What I'm going to talk about is data and analytics analytics is more than practicing a technical skill analytics is a management philosophy analytics and data strategy as as we more commonly refer to it analytics is is a way of using data to again shape those decisions in your organization. It's a way of using data and it's convening power to help bring your team together to help shape and spark conversations that lead to new strategies that lead to new tactics in a campaign that lead to new ideas about the way we use our platforms, new ideas about the way we create content and string together those experiences. Campaigns are how we organize content. The campaigns are also how we create unified audience experiences. And that that is a cognitive flip that I think a lot of organizations struggle to make. A lot of organizations see their campaigns from the perspective of the campaign producers. We are in our organizations we are in our word docs in our Google Docs and we are hammering out content day after day. We experience campaigns from the producer side. And that's often how we end up seeing them. You know, being flipping about it people have a blog and therefore campaigns are about creating new blogs that they can push out to people. And that's a little bit divorced from what our audiences actually need. So it can be a trap to look at what our audiences can do and focus on that to the exclusion of what's missing and what our audiences need and what we can do for audiences to help bring their ideas together and help curate that experience for their own personal context. And so thinking about campaigns in this way and really pushing the campaign and the experience of the campaign outside of the organization. It's not something that you produce. It's something that people outside have and you need to tap into. That's really bringing together those ideas of storytelling, putting yourselves into their context, and then also using data to understand that context, to make sure that we are creating something unique and relevant for each of those people. So how do people experience campaigns. I think, you know, the, in the ideal world we often imagine content consumption and our audiences how do they engage with us.  They are there with the intent and the purpose and the goal of consuming our content and they are going to enjoy it, and they're going to sit there and really read it and understand it. But in reality, your audience's experience of your content and your campaigns probably looks a bit more like this, they are on their way somewhere else they are doing something else they're on their way home. They're in this crowded context their life is packed, and, and they have probably just a few moments and they're probably not, you know, really curled up and consuming your campaign, probably just flipping through notifications on their phone. And then you really need to understand the complexity the nuance, the different contexts that your audiences are coming from to make sure that you are being succinct enough compelling enough and changing your relevance to that context, because you might have some people like this, but you're also going to have a lot of people like this. And how can you tell the difference and how can you make sure that you're giving the right experience to the right person at the right time. That's a big set of the principles that I want to make sure we focus on. Now putting it fairly bluntly. I'm going to ask you to change your behaviors. I'm going to ask you to change the way your organization plans campaigns and runs campaigns and does things. In particular, some of the things that I hope you will reflect on as we go through the rest of the webinar today is experience planning, how do you conduct experience planning in your organization, including how do you actually see it, how do you, how do you design it what does that experience look like your campaign calendar, really thinking through what is the schedule of a campaign. What are campaigns what do you build campaigns around how much of your content just goes out as an isolated blip without really being connected to the rest of an experience, and how can we either pull that back and loop it into something that's a broader experience, or how can we how can we really put together a an experience for a new piece of content that you've you've just put put out into the world. What a lot of organizations are going to need to focus on is documentation. How do you keep track of everything that you have in terms of content campaign experiences, but also data, how do we understand what data we have and how it can be used and how it is relevant. There's going to be changes in how you use your tools as well. What is the actual experience of using your tool to configure the campaign. What is the experience of using your tool to conduct segmentation and targeting. And do you have the right time invested in that you have the right talent on your team to, to use those capabilities. And often, yes, I think a lot of tools go underutilized, but sometimes you will find you're going to need to introduce something else in order to make the rest of your tools work. So, keeping an open mind about those changes and the way we work with our platforms. I think last and certainly not least is the way that you work with your peers, and how you spend your time. People are the engines of nonprofit organizations, how we spend our time in these organizations determines the outputs determines the outcomes determines the experiences that we create. And in a lot of cases, the more elevated and impactful campaigns are going to require coordination across different people across different departments. There may be historical lines and divisions between some of these. And I'm really going to encourage people to break those down. How do we make sure that we are bringing everyone who can bring expertise and everyone who can do something to change the audience experience together to help find those opportunities and make those changes. So we're going to start getting into some of the more practical and tactical parts of this story. But I do want to offer we've, we've put together a worksheet to go with the rest of this webinar. If, if you would like to I think you will find the link in the chat momentarily. And this is a it's a bingo card so this is a form that will follow along with the rest of the presentation, and it will give us a sense of what things you might learn, and it will give you a place to capture your reflections as we go through the rest of the topic. So the link has just gone into the chat there. So if you open up the chat you should see the link. It will guide you to follow along. It's not required, but hopefully makes it a bit easier to take notes as we go forward. All right. So what's next. How should we think about campaign data. We're going to talk through a few of the different lenses that we use to think about and talk about data. And again today is about campaigns and so, you know, ultimately of these different approaches, we're going to focus on campaign process. But I wanted to just talk through engagement architecture as a lens and data operations as a lens to to think about and understand your data, because having that understanding will help you figure out how to use it in a campaign context. First and foremost engagement architecture. We talked about it at the beginning of the presentation, but it is a way of understanding all the elements of your organization that contribute to your experiences your audience experiences your campaign experiences. And, and they're also where your data lives, you know your data lives in your platforms, your website has its own data internally. A lot of the experience of the website gets captured in your analytics tool like Google analytics. You have an email platform that almost certainly holds and harvests a lot of what your audiences are doing with you and with your content. And there may not sink cleanly with a separate CRM, where you also keep and maintain history with your audiences. There's social media, there's your donation platform, there's your event platform, and so on and so forth, there's association management platforms there's all kinds of platforms in your organization, and all of them hold little bits of those experiences. Speaking about campaign data, it's useful to make sure we understand this whole context and where you can find those little bits of data. But then we also need to figure out how that connects to the rest of your engagement architecture, how can we understand your data in the context of your strategies and your tactics in the context of a campaign strategy, and really finding the connection between the context of a particular campaign. Let's say for example, you know, it's a thought leadership campaign and you want to demonstrate your expertise to would be donors or would be partners, and making sure that we understand which metrics connect to that, which metrics, like scroll depth, or time on page are the right ones for the type of content that you are producing for the goal of thought leadership. And then of course there's also the stakeholders who are the people in your organization who are responsible for these platforms responsible for the experiences they support, and who are the people who are able to help get those data out, and also get the lessons out of that data, and bring those lessons bring those ideas, and then actually turn that data back around and put it back into your engagement architecture, and a new and useful way, making sure that we have the right data. We can find it, we understand where it can be used can understand how it can be moved back and forth and and leveraged in support of campaigns. The next lens that we use to talk about data, most often is data operations. How do we understand the state, the health, the maturity of our data platform. How do we understand the capabilities that data can support, which outreach capabilities, which analysis capabilities, and who leads which data practices again to that point about stakeholders in your engagement architecture, you have different people in your organization, who are more or less expert in different parts of your data practice. And there are also people who need to be consumers of those data practices, who are the people who need to receive the right lessons. And then how do we make sure that we are collecting the data that all of those practices and processes need to really understanding those data practices and the business processes that are related to your data. And the sort of four stages that we use to talk about data operations is the strategy definition, both what are we trying to accomplish as an organization, what's the mission, what's the goal of the campaign. But then also what is the strategy for the data itself. For each data point that you collect what is its purpose, what is its role. These are important things to think about from being successful with data, but also from things like compliance considerations, making sure that you are using data responsibly that you are collecting it ethically that you are in compliance with the varying and multiplying laws around the world. So really having that strategy pinned down is is crucial tracking component is probably what we most often think about being able to collect data that is useful and relevant. Sometimes it's not even a question of collecting a new type of data but just making sure that you have the right context on that data. You can gather a whole bunch of information about page views and those can be useful, but only if you know which piece of content received each page view, or which program in your organization received each page view, making sure that that tracking is working correctly for your purposes and your campaign purposes is is another important piece. Reporting is different than tracking. When you track data, it often ends up living in a database or research tool that doesn't mean that it's in the right format for people to use. And so making sure that we think about what it looks like to turn the data into insights is a is a separate activity. And then lastly the adoption and optimization. And that's that piece that I talked about before how we spend our time and what we actually do with the data. Do we commit to being consumers of our own insights and and using data to both change the way we think and meet and work together, and then actually turn that data back around and plug it back into campaigns and in the form of things like segmentation in the form of things like personalization. So those are are operationally some of the things to think about. What I'm going to talk about for probably about the next 20 minutes is the campaign process itself. So here we are getting to the meat of the matter. In the campaign process, we have these four, sorry, six, rather, six major stages of campaign operations where data can play a an important part. Planning, targeting, optimizing, evaluating, demonstrating and recording. So through all of this campaign process we are looking for how data can lead to better outcomes. How can we use data to change what the campaign accomplishes. What are the right formats at different points, how deep or shallow should we go and different stages of campaign process. And then when should you use the data. There are tie there's a timeline in campaigns of when the data is relevant and useful and when it's too late. And it's not worth digging up anymore because the opportunity to use it has passed. So those are going to be some important considerations here. So let's dive into some of these ideas. Another proverb. How do you create a good user experience with a good campaign management experience. Again, the experiences your audience will have flow from the people in your organization and the work that they do. So if you want to make sure that you have a good external experience, make sure that the people who you have creating those experiences of everything they need are well supported are given the opportunity to be creative are given the opportunity to discover new ways of creating engaging experiences. So that's a lot of what we're focusing on are these these internal parts of what you do to create the space because again there's not one campaign model. And it's going to vary tremendously organization to organization. So the, the, the most general advice I can give is know thyself and really think through people who are creating these experiences and what they need at each step of the process. So again, those six steps. There's really kind of split up in terms of what you do with data across these six steps, and when they are going to help your organization. The first three planning, targeting and optimizing are the ones that you can use for any one particular campaign. Those are ways for in the moment campaign that's launching next week or next month. These are steps you can do to make sure that campaign works and it's successful. The other three evaluating demonstrating and recording. Those are investments in the future. Every campaign gives you the opportunity to use to do these things to grow your organization's campaign capabilities. But they are for the future. And that's why they're often dismissed they're often discarded, but they are tremendous opportunity. They are they give you the lasting impact. They don't give you lasting impact. They help this campaign, and you're going to have to do it again for the next campaign. But those other three those last three four five and six, those are investments in your team. Those are investments in your future. They are tremendously valuable, but too easily discarded. So that's one thing I want to really encourage is that everyone make time, not just for the best of these but for all six of these in order to truly leverage this data to improve your campaign capability. So talking through one at a time. First comes planning. And this is where a lot of the user experiences out there are going to to see their own, you know, exposure to the process. And really, you know, thinking through the create the creative opportunities. What are the ways in which we can come up with something new or different. This is it's the blank canvas it's the stage in the campaign where anything is possible. And you are not yet constrained by the realities of what you have and what you can do. This is the goal setting process. This process, some things that are really valuable are having the right content at your fingertips, understanding what you already have so you don't necessarily always need to recreate it. Again, I made the reference before to people have a blog and therefore they write blog posts, often forgetting that they might already have a piece of content that's just as good as the new thing they're going to write in your campaign consumers have not read every single one of your blog posts. And so, making sure that you, you have your content and your history as an asset that you can reference that you can use, you can bring back into enriching experience, so that you can curate an experience for them. The new library is tremendously useful. How many of us have, you know, had the conversation. Oh yeah yeah we've got something like that and you go rushing to the q drive, or you go rushing into your folders and Google Drive, and you're trying to remember what you need to type in the search box to find that perfect thing. Your content well structured well organized is so valuable to the creative process and the planning process of a campaign.  But sometimes it's as simple as let's just look at the top performing things in Google Analytics. Again, what does performing mean. And again, are there subsets of that, do you want to be able to filter your content, you know, in the performance of your content by a certain topic, for example, and do you have your data structured in a way. So building that sort of content research experience one way or another is going to be a way to help you plan faster and include more value in your campaigns. The next one I want to harp on is audience personas. A lot of us have them. They are a very tried and true and long standing tradition and user experience and campaign design. A lot of audience personas are rooted in the theoretical process of user experience design. So think about your audience personas and there's a picture of a person named Mia, and you know about her pets. That is very helpful. It's a helpful mental tool, but it is often so divorced from your data and the reality of your engagement architecture that it's actually hard to use in the actual campaign planning process. I really encourage you to rethink and modernize the approach to audience person personas to be more rooted in data. How does this persona, how does an influencer, how does a donor represent itself in our platforms in our data platform by platform really think it through these experiences? What are the data flags that they will throw off that help us recognize them? The more data relevant your personas are, the easier they will be to use to design segments to conduct a targeting. And that's a really valuable way to make the planning process more actionable. Lastly here, there is audience research based on search and social media. This is a big one for me. These data, in particular, social media as well, are woefully underutilized. These are both platforms and environments where the audiences we are trying to reach are telling us with their own fingers, you know, typing into boxes on various websites, what they are looking for, what they want, what they're interested in. There are reams and reams of this data hiding in different dashboards, things like Google search console and Twitter is a sort of the most famous one in the social listening space because of how open it is. It's just sitting there waiting for us to look at it and draw conclusions. So there's a lot that you can do an audience research using these data sets. Those are things that I would really encourage everyone to invest more in. So once you know what your campaign is going to accomplish, you need to go through the actual steps of configuring your platforms to execute this targeting. How do we make sure that we are sending the right content to the right person. And I think a little bit, a little bit at odds with the planning process, which is by its nature, usually pretty collaborative. It often looks like this, you know, people gathered around talking about possibilities. The targeting becomes very isolated very quickly, because you usually have one person who is responsible for each platform and they go in there. And they click the buttons and do the configuration to make sure that your ads or your email segments are getting set up, you know, in a way that gets the right content to the right people. This is something I want to shake up a little bit. I know you are going to have your platform experts, but I want to make sure that this targeting process is a little bit more circular is a little bit more back and forth. And so we need to help one another understand the unique targeting capabilities of each channel.  There's no end to how specific you can get with the targeting and some of these platforms. Others like email you can target as well as you have data to target, but the capabilities, the ability to do that targeting are fairly unrestricted, even on a lot of the platforms that nonprofits have access to. You can understand what's possible and can teach one another in our organizations what's possible so that we can then have these conversations about okay well I know this and my platform so if I teach yours about the experience that we had with our audience, then you can you know sub segment and send a different message to that audience that'll help them be more successful. Understanding the capabilities and then the data that we have available. That's where something like a data catalog really becomes important. We ourselves are as platform experts often know what data is we have and what is good data what's healthy data, what's enough data, but we need to be able to communicate that to one another and figure out the possibilities but also the responsible uses of that data. And lastly, being realistic, we need to be able to understand the limitations of our targeting. You know, we can't do everything for everyone and we can't make sure that exactly the right people. And so being tolerant of some of those edges of you know how how specific we can be an understanding that imperfection is okay. There's a lot of room for error and these experiences. Again, people aren't going to be sitting there, you know, going through every single, you know, message that you put out to figure out if everything was specifically meant for them. There's there's a lot of tolerance on the part of our audiences. And so it's okay to be ambitious with the recognition that you might miss something that the edges that making space for that ambition is is one thing, especially that leaders can do to help make their teams feel confident with experimentation and targeting. Next up is optimizing. This is what we do. Once the campaign has begun. And, and again thinking about the experience of this optimization. Once the campaign is running, you don't usually have time to sit around and, you know, be thoughtful and do research. It often looks a bit more like this you are on your way home from work and some things happened new data has just come in, and we got to make a snap decision right there on the street corner, because the team needs time to maneuver in the middle of a campaign. And some things we can do to help prepare ourselves for this part, this optimizing part is giving yourself a clear sense at the beginning of the process of what are the alternative paths for a campaign. What is the other piece of content you can post. What are the messages that you can change the schedule on some push something out sooner or swap the order of it. And then what are the triggers. What do you need to see to know that it's time to change course, and having that playbook for your campaign right at the beginning will let you be nimble. That is what makes data powerful is creating opportunities for your tactics to change based on the data. Without those options, you don't have options in your strategy, then there's no point to the data, because you're not going to do anything different. So, so this is a really important pieces is creating those pathways, building a reliable reporting pipeline and or process is is another thing that makes it possible, because you can have the options but you need the ability to, to get the data, you know, to know whether you've tripped the trigger to change course. I, I say pipeline and or process pipeline makes it sound fancy that's you know something like a dashboard things are automatically calculated. You can check really quick. A lot of your tools have this built into them already, but processes just as important. You just need to know who on my team is going to go check who's who's going to tell us who who is watching at the different campaign, when we need to know is a time to change course. So having that schedule, you know put into your campaign calendar. These are the moments when we need to check. These are the moments when we need to decide. This is how much time we need to make those decisions. And that's, that's going to be a really powerful part of this in operationalizing data in the middle of a campaign. And recognizing these constraints. Let's also be very clear about how thorough data is in this context and the optimizing context, you're not going to have time to write the big report. You don't have to wait until the next quarterly comes out for months from now to know whether or not the campaign that's happening this week is going well enough or doing what you expected. So these communications about data are going to be informal. The campaign usage is a great data and analytics report for the middle of a campaign, just to give people enough to know, okay, I think we should do something different and to give a space for those conversations to happen. So, at this point, the campaign has ended. We've talked through the opportunities to drive the campaign itself. Next, we want to get into evaluating evaluating the campaign is a thing that you do internally, and it is for your people, you use your data to make your people and your process better. Your people first because the people are the mechanism for creativity in most organizations and in all organizations just for now, at least. And having the opportunity for your team to come together and to meet and think about what just happened in the campaign is the best time to extract the lessons. Never again will your team know as much about how a particular campaign went then right around the end of that campaign. So we need to make sure that we create that space, put the retrospective on the calendar, as soon as you know the campaign is coming. Don't let it get run over because the longer you wait, the less you're going to know and the less you're going to be able to keep in the organization about that experience of creating this experience. This is also a great opportunity to democratize hypotheses. What are the questions that you want to ask of your data.  It's the creativity and questions that bring to the data that leads to interesting insights. And your team has different experiences they have different expertise, they saw different parts of the campaign unfold. So making sure that you get all those questions is how you turn your data into the stories that will help people change the way they think about the experiences they want to create next time. And that last point there looking for the small data, the surprises good and bad in the experience. One of the projects that we're doing right now is focused on the user experience of, of, you know, very particular parts of a website and having things like session recordings where you can go in and you can see what a person's mouse does.  What does it actually look like to go through this experience. And you need the space to look at that together, you need the space to understand what, what the, the experience of your campaigns are so that you can change them in the future demonstrating the impact of your campaigns is if there's one thing that usually happens after a campaign, it's this, especially in the context where the campaign was supported by leadership or an outside donor or a foundation. This process of demonstrating is, it is very much an art form. And there's a few, if you go look up the definition of analytics analytics is the discovery and communication of meaningful patterns in data demonstrating impact is all about that communication piece. This is where we are using data to tell a story about how well we did or didn't do. And you need to understand the context of the people you are trying to tell about with your data. It's going to happen in a boardroom and you need something that's really thorough, or is it happening in a bar and you are just, you know, talking to the people you need to convince or impress about the performance of your campaign, in which case you just need bullet points in your head and you need the talking points, you need the stories that this is a very narrative driven process. This is the most part if it's useful and effective, it's going to be narrative driven, because again, you are convincing people and storytelling is the right format for that. So there's a lot of curation that goes into demonstrating, you need to make sure you're pulling together the right pieces of data, the right outcomes of your campaign to create the story of what you've accomplished, and then probably take another crack at it because you probably know too much about what happened. And so how can you really make sure that you're telling the right story about what you accomplished, about what you accomplished and about the opportunities that it created for the next campaign, the next step in what your organization is trying to accomplish. As you do this, as you think about the data, as you look at the data and try to turn it into a performance report, remember that the numbers represent human experience. How can you talk about the data in the context of what people went through as they experienced the campaign. It sounds simple, but it really makes such a big difference to really helping people see it. Okay, this is what it looked like. This is what you put our audiences through. And that really helps bring it home and helps people be thoughtful and creative about what they want to do next time. And then lastly here is recording. Recording the data from your campaign, but also recording the lessons from your campaign are going to make them available and useful again in the future. I cannot emphasize enough how important this piece is. This is how your organization builds its memory. Your organization is able to develop long term relationships with people systemically you as individuals if you're doing one on one outreach if you're in government relations or donor relations. You're very good at building your one on one relationships with people, but your audiences aren't always going to be talking to you. Very often they don't think about the organization as just you, they expect the organization to also have a relationship with them. Recording data from campaigns is the way that you do that. So you need to be clear about what you will do with your data. I think, you know, again to that compliance point before we are, we are making a building a relationship and there's there's a contract that we have with our audiences about why we need to know all those things about them and making sure that you understand the future use cases is going to help you fulfill that that contract fulfill those promises build that relationship, and it's going to help you figure out what data you need to be collecting. At the end of each campaign. There's the point where that data can either float away into obscurity, or you can harness it. And so that can mean migrating your campaign data into a system of record, if that's not happening automatically already. Of course, sometimes it's it's putting that data into a system of action. You know how can we, how can we move an experience from offline events into our email system so that we can make sure we're getting the right materials out the right follow up and having those those tools in place is going to be very important. And putting those lessons in your institutional memory is the next, and the last piece of this is making sure that the data, but then also the lessons the things that came out of the evaluating process. How can we actually make sure that these stories are available to us again when we're trying to do the next one. How many times do we have to recreate the wheel and come up with the same experience again campaign after campaign. And that's a really big piece of making sure that those stories the bond and the organization. So those were the six steps of the campaign process and examples of how we can use data in those steps. I'm going to talk fairly quickly now through a couple couple little vignettes sort of examples what it what it kind of looks like. And how we can think about and visualize the impact the effect of using data and campaigns. So these are just a few stories but I hope they they help illustrate that for you. Again, we just finished talking about the recording process and how it supports long term relationships. Long term relationships are the foundation of some of the most impactful interactions that we have with our audiences are campaigns, sometimes internally try to build people up to that point. And the campaign itself is just one step and a much longer story data is our memory as an organization data is what your organization remembers about the people who engage with you. How people are supposed to grow as individuals through the campaign is a big question that we have to ask ourselves in the planning process to make sure that we're giving the experience to facilitate that growth. And then it's also really helpful to know whether or not you were successful. It's important to know who actually made it through those steps, so that they had the, the, the, the change in thought leadership change and thinking the change in the way your audience interacts or or views the world, depending on what you're trying to accomplish. And to make sure that your audience is, is successfully completing those actions, and then also being rewarded, or at least recognized for for the actions that they take. And a lot of times we're asking our, we're asking our audiences to do things we're asking them to change their behaviors. These are big commitments for sometime asking for. And so they're going to want to know that that you recognize that. And that data is, is again, likely at first at least scattered throughout your engagement architecture. So making sure that we are retaining that, and then using it again down the road. And this is a bit of what that looks like, you know you can think through the first touch that you have with someone.  And the more that happens the more often that they, you know, participate in a conversation online or or consuming content, can you use that to start to understand who is this person, what are they interested in. What are their preferred content formats and building that understanding it's it's not the individual data points it's recognizing the trends in those data points, and then saving those trends as something that you know about this contact this person. And then as they start to become more active they're not just passively consuming but they're actually doing things are actually helping to deliver on the mission in some form or fashion. They're donating their signing petitions, they're, you know, helping to register and volunteer, you know, these are things that we need to make sure we recognize and understand. And that's what gives us gives us as individually sometimes but more importantly as an organization, the ability to to see communities of people, these are our power users. And this is a community that we can serve quickly, we don't need to send out an all staff email and say, Hey, everyone, tell me who are the best people for us to invite to the gala. We know as an organization we can go straight to it, we can build that gala experience, the many step the invitations, the subsequent follow up messages, we can build that outreach as an organization well ahead of time with with a lot less friction in the and the communication process. And again, this is something we've alluded to, but just thinking through all those many steps, the different types of things that people do throughout the fact that reading content early in a relationship can tell you a lot about what is compelling. You know, if you know, based on your organizational taxonomy. This person is always reading about green roofs. That's what really matters to them. When it comes to asking them to take action when it comes to asking them to contribute their own expertise. You know this is a person we can talk to this is a person that we can deploy as an influencer. It's a person who will carry this idea forward for us and we can we can depend on them to help accomplish the organizational mission. So a lot of that was about the, the small data, you know what is the very personal things that we can do with individuals to build relationships. So these these principles also apply to the mass market. You know outreach that we do use your audiences words to choose the words that you use to reach your audiences. It sounds simple, but it's something that we often overlook. We know so much about our entire audience based on things like search data and social listening. We can go and see how they talk, we can go and see how these communities form around ideas, and then we can use that to design our own outreach. And so, helping to figure out where do we align or misalign with the community that we're trying to reach can change your editorial strategy that can change the way or the types of content or the frequency of content or recognize that you already have the right content, you've already figured the words that they use but but how do you know how do you know which of your content is most closely aligned with with what your audience is doing. So, so thinking through these things and getting ahead of the need, making sure that you have this knowledge internally, so that you can use it in your in your outreach is extremely powerful and you know the power here is akin to what what organizations often spend a lot of money and a lot of time on doing full market research studies, going out and conducting interviews and then doing surveys. These are resource intensive processes that will often tell you probably more, or at least different things, but a similar amount of value to what you already have just sitting in databases. And this is an extremely powerful use of data that I really want people to explore. And what does it look like afterwards. Once you've gotten those insights, you can bring that back, and you can apply those lessons that you learn from one type of analysis, often to a lot of different outreach disciplines. The same audience research process can be used, not just to improve SEO using search data to improve SEO is powerful and, you know, often underutilized and can get you a lot, but you can bring those same lessons to an advertising campaign, and often multiply your outreach and multiply your reach and your impact by as much budget as you have to throw at it. It's a really effective way to improve targeting across different channels, including social media, changing the way you talk on social media, helping find new people on social media. So one analysis can be used in many parts of the ecosystem. So what should I do now. I think there are five things based on this that I would encourage people to consider whether you have done. And if you haven't, please do think about doing it. Build the data inventory, figure out what you actually have, what can you use, what can you deploy on your campaigns. This will mean a conversation talk around the building what do we have really, really making sure you understand the landscape of data in your engagement sector. Audience analysis, whether that's changing your personas, or whether that's conducting an analysis of your search your social data, or doing a market research study. Let's refine and rethink what we know about our audiences, as we reframe our strategies around data. We need to really think through the experiences that we're trying to create and how they will change. Think through how you will use data to change your campaigns, change those experiences. And all of these things will take a commitment of time and attention. You know, show me on your calendar where you're going to do this. You know, show me on your agenda, how are you changing the agenda of your existing editorial meeting to start incorporating data. How are you changing the agenda of your campaign campaign planning meetings to make space for this.  Think about what you can do differently. Think about the current current content creation processes, the current behaviors that you have, and whether data can reaffirm or refute what you are doing. Yeah, really, really looking for opportunities to change what we do. And last but not least, I do want to put out a plug a call to action for for our own services. A lot of what we described here is what Parsons TKO does. I think, you know, thinking about that first step the data inventories. Other classes of project that we do our data appraisals. So if you think your organization needs help figuring out what they have and what they can accomplish with the data that they already have. Please reach out and let us know these are things that we can help with. And with that, I have, I've gotten to the end and left nary three minutes for questions. And once again we are going to have. Well there's a survey coming out in the chat, and we're also going to have office hours immediately after this, we will put a link that's going to be in a separate zoom room, but I will head over there in a couple minutes. But in the two minutes we have left. I will thank you all and then see if we had any questions come in that I might be able to answer very quickly. Yes, so we had a question from Susan Patterson. That was our first question of a specific example of how to data eyes your audience personas. Excellent. Yes, I'm data icing I like I like that phrasing. So think about your audience personas. One of the things the closest I usually see audience personas get is keywords what are some keywords this person will use. But that keyword term is a little vague. You know what is that keyword on social media. What is that keyword in Google search. What is that keyword in internal site search if they're getting lost. When people talk the way people use language will change based on the experience.  Is there a way we can recognize this person here. If this person fits behavior XYZ if this person has consumed and read and downloaded 10 PDFs, they are in our academic wonk category, you know, we are really strictly defining audiences by data that we have available to us. Thank you and I think we have time for one more. What suggestions do you have for small organizations without a designated content creator or data person what's the best thing to prioritize. Everyone should have a little time for data data tells us what we're doing data tells us how we're working I know that I have been that person. You know, wearing, you know, I, I learned about all this back in the days when it wasn't digital it was new media. And, and in data is the only thing that is available to somebody like that to help them understand which of their choices, which of their activities are effective. Find time on your calendar, put a block on your calendar to take a walk and breathe the air and relax and put another block on your calendar to even just sit and think about how do I know if what I'm doing is the right thing. And what data can I look at you use a lot of platforms if you're in a small organization you probably touch more platforms than somebody at a large organization does large organizations have one person per platform, you touch them all. So those have little data platforms little data dashboards, just spend time with them and really think through what questions you can ask of each of your platforms. That process alone will teach you so much even before you look at any other data. That I know we're at time. So thank you all very much, not us I think is just our Lisa thank you as but the survey in the chat. Please fill out the survey, and we're going to put a link to office hours in there momentarily, and anyone who'd like to stay and keep talking. I'll see you there.


Hello everyone! My name is Irina and I'm from Ukraine. I decided to take out a tough certificate because I'd like to teach abroad. I took 120 hour online course online because it's really comfortable. You can combine work and studying. This course consists of 20 units and it is as user-friendly as possible. I had six months to complete the tests and believe me it's more than enough. I think this course is really informative and useful especially videos and grammar explanations which I find very clear and helpful. So I'd recommend this course especially to beginning teachers. Thank you very much! Have a nice day and good luck!