Welcome everybody, and good morning, good afternoon, and good evening, and so today we have another great seminar speaker, Anaya Sen from CMU to share with us a very exciting paper, the editor and the algorithm, value of data and the externalities in online news. So we will be using the first 40 minutes for the presentation. If you have thoughts, suggestions, please feel free to use the chat window to share this. And if you have some clarified questions, please feel free to raise your hand. And we will then use the last 20 minutes for Q&A.  Thank you, Feng. Thank you. Thank you for having me here. This is a joint work with your closer who's at LMU Munich. I think he's joined us on Zoom here as well. Christian Boykert, who's at ETH Zurich and in Kethulika, and I'm going to talk about the editor and the algorithm. So let me start with the big picture. You know, we all know that news plays an important role in a vibrant democracy, provides information, especially during the pandemic, we're seeing how potential misinformation, even from mainstream outlets can have drastic consequences. So for credible news outlets to exist, you know, it is super important just as a pillar of democracy. We also know as a digitization crowd, as a platform crowd, that digital transformation over the past couple of decades has severely affected news industry's financial health. And this is, you know, not only in the US, it's across the board, barring a couple of countries. More recently, though, it's been hypothesized that new technologies such as artificial intelligence, the use of artificial intelligence, and machine learning in particular within the newsroom can potentially shore up for revenues. Now, the idea is that you have a large amount of data on people's preferences. You can have algorithms which can mine these sort of data sets and identify certain references which traditional human editors might not be able to do. And I, you know, the paper is about human editors and algorithms mainly because human editors are the status quo, especially in mainstream news outlets. So the broad question that we're looking at is whether human editorial decisions can be aided by algorithmic recommendation. And we're going to try and address this question in a simple news curation setting, but it's a large-scale field experiment that we run with one of the biggest German news outlets. This issue about human editors and algorithms is, you know, is a debate which has been happening in the news industry for a few years now. For example, Facebook, you know, Facebook as a news curator first had human editors, then it fired its team of human editors, then it had algorithms choosing trending news, then they abandoned trending news, and now they've hired a bunch of humans to curate news stories. Similarly, Apple, you know, this became big news in 2018. Apple hired about 40 mainstream journalists from, you know, from the news industry in the US, and they said that, you know, these humans are going to curate the news for all their users. These are news, you know, these are sort of news aggregators in some sense, but mainstream news outlets have been have been really reluctant to sort of take this approach. It became big news last year, like exactly a year ago with the New York Times saying that for the first time they are going to allow personalization on their website. So there'll be an algorithm, which, you know, which will take into account what you sort of read before and try and recommend new things.  So more generally, when we talk about human editorial decisions and algorithms within the news industry, you know, it's not completely clear where one might outperform the other because it's, you know, it's curation inherently subjective. There's a creative production process. You have to try and understand what is quote unquote important or, you know, or what might get clicked, what might drive up your revenues. And you have to be really quick because timing is critical as, you know, in this paper by Julia Kajie, who shows that, you know, most news stories online on some of the biggest news stories get copied within the first three to five minutes. News events are happening all the time, consumers preferences might be changing. And hence, it might be the case that this is a place where human judgment might have a reasonably big role to play. So we try and dig deeper into this question to try and understand how algorithmic performance might be affected relative to human judgment and how does different types of data affect this, affect this performance because any conversation about how algorithms do or how machine learning does in general cannot, you know, cannot be divorced from how much data goes into feeding the algorithm. So we're going to look at two dimensions of data, personal versus social. So individual level versus aggregate. So, you know, let's say new strengths versus what you have clicked on. And also the stock versus the flow of the data. So you need to continuously update your, update your model or the fact that a person has clicked on enough to accumulate a stock of, you know, a stock of data. How do they interact? I frame this in terms of the news industry, but, you know, this is more broadly related to the debate about, you know, what the value of data is, what should, what should policy look like.  There are a couple of studies on the value of data, but you know, they point to, they point to mixed results. In general, you know, I think there's a consensus growing that data might not, you know, or certain dimensions of data might not be as important as they seem, but, you know, the debate is still going on. One of the issues that has come up in these sort of papers is sort of summarized by the sentence in this Amazon paper by Pat Byry and co-authors is that the effect that they identify due to access to longer histories of data can also be driven by continuous improvements in forecasting technology. And what we will hopefully be able to do in this, in this field experiment is actually identify potentially the causal impact of data histories.  And this is, again, important because there's this issue about filter bubbles, algorithmic reinforcement, especially in, you know, in polarized conditions. And that's the, that's the final question that we ask. How does algorithmic curation affect consumption diversity in this context? So if I do run out of time, just in case, you know, as an overview of the results, on average, we find that personalization in particular, so not, not social recommendations, personalizations increase clicks by 3.7% relative to the human editor, but diminishing returns kicking quite quickly. Can human curation ever perform better? Is there a role for human judgment? We do find certain instances, especially when there is limited individual data. So when the recommendations are based on aggregate news trends, we find that the editor is able to identify the average preferences better. Moreover, a human can do better is when there's lack of continuous updating of the algorithm. This is mitigated a little by the existing stock of preferences. But we find, at least in this context, that you need to keep updating your algorithm. How does personalization affect consumption patterns? We find that treated users do reduce their consumption diversity. So they, they sort of do narrow their consumption. And this is, you know, primarily sort of driven because personalization is playing a big role in increased click rates. And we find that this effect gets reinforced over time. And, and I'll, and I'll try and show you a picture about how, how things progress. If you come on to the website 10 times 50 times 100 times. So that's the broad overview of the paper. Is that I don't know if there are any questions at this stage, or I should just dive into some of the details. Okay, so it seems that we're all good. So setting is that we partner up with one of the largest German news outlets. This, this new outlet gets about 20 million unique visitors a month, that's about 100 million clicks a month. So it's quite a bit. They care about clicks based advertising revenue. So, you know, it's not, you know, editorial judgment would potentially take into account, into account, you know, how interesting or how many clicks an article might get. The homepage of the outlet is curated completely by human editors, or content is produced by humans. And a user who arrived at the homepage sees the same content across the board. So it's not, it's not personalized based on location or anything else. So what the experiment does is that each time a user comes to the landing page, he's randomly assigned to a treatment or a control group. If the user is in the control group, then she's going to see all the articles on the homepage as curated by the human editor. So this is the status code. If the user is in the treatment condition, then it will see one slot on the homepage, which is customized based on her reading preferences. And the only thing that we could negotiate with this news outlet, along with the data science team, was slot number four on the page. But remember that, you know, this is a large news outlet with, again, over 100 million clicks a month. So slot number four also gives us, you know, a large amount of traffic to deal with to precisely estimate our parameters. The experiment ran for five months. So it's a reasonably long period of time, which actually relative to other papers allows us to trace out some of the dynamics because often these experiments last for a week, two weeks, three weeks at the most. Just to make sure that we're all on the same page in terms of the experimental setting. So again, as I said, it's a simple curation exercise. What the control condition would be is just the human editor choosing all the articles on the homepage. And they are on average 80 articles on the homepage at any particular point in time. What the algorithm is going to do is that if a user comes on to the website and let's say the user has some prior history, the algorithm identifies what potential article this particular user might want to click on and it might identify, let's say, article six. In the standard status quo setting, the algorithm might identify that article six is more likely to be clicked on than, say, article four. And so it's going to bump up article six and push everything down. So the ranking ordinarily remains the same, but it's just one switch that happens. So again, it's a simple setting, all articles written by humans. In some sense, the initial curation is done by humans as well, but then we're trying to see whether this can be augmented by algorithms and how that varies based on the data that they have. I've been talking a lot about the algorithm. Ananya, could I just ask a quick question there? Yes. Which is, did it ever happen that the algorithm could put onto the homepage an article which would not otherwise have been on the homepage at all? No. So the algorithm has quite a broad choice set at any point in time, because it was going from one to 80. And we can actually see what rank it was pulled up from and it never goes beyond 81 or 82, which is generally the maximum number of articles. I have a question, too, please. In your introduction, you made very broad statement about what personalization would do in general, but your experiment is on a very specific form of personalization. Will you try to explain to us where you think what's general and not generally the lessons you are getting? Right. So I think it's, with this type of a question, there are different dimensions of where it might be generalizable or not. I was getting to one of them, which is the algorithm used. So it's the algorithm and how the algorithm used interacts with the data.  I can have a general conversation about how I see the literature on the value of data and the study. But I think this is going to be one piece of the puzzle. We're not necessarily saying that this is how let's say Google or Facebook would necessarily value their data. But as of now, we don't have that many estimates. And so this is just trying to take a first step. And maybe towards the end, I'm going to talk about the contribution of the paper and how it fits in with different strands of the literature. And I also link it to the computer science literature. And maybe then we can have a broader conversation. But again, as I was saying that, I've been talking about the algorithm quite a bit. And this is one area where there are many different types of algorithms. The data science team, in this particular case, used a highly cited algorithm, which was developed by Google engineers. So that was developed in 2010. That was used as the baseline. And then they put some bells and whistles on it, which we are not completely aware of. We know that we know of the 2010 paper. It's been cited heavily. And the model is reasonably flexible in that it tries to predict an individual's preference for different content categories. But these content categories are extremely fine-grained. And on any given day, there are over 200 content categories that the model chooses from. As I said before, the algorithm works such that it uses a combination of social data, which is collaborative filtering and personal data, which is information filtering. This is computer science recommended system talk. Basically, when there is limited data on an individual, then the algorithm is going to overwhelmingly use social data, that is, other people's current reading behavior. As the algorithm gets more and more information on that particular individual, personal data starts dominating in terms of what recommendations are made by the algorithm. The aim of the machine learning model is to maximize clicks by the user who visits at any given point in time. Of course, we're comparing the human to the algorithm. Here we have a well-specified objective function for the algorithm. For the editors, it's of course less clear. It's hard to get at it. What we do know from the literature and from this particular setting is that the editors definitely do care about clicks and trying to get as many clicks as possible to get a sense of what the editors do. The line in blue is the share of all articles by topic that are produced overall. The red lines are the share of articles from that particular topic or from that particular category or that particular broad category, which make their way onto the homepage. You can see that there is a reasonable amount of variation in terms of what goes on to the homepage and the green shows the overall share of page views. What we can show in this setting is at least correlationally, there's high levels of correlation between clicks and what makes it onto the front page. Of course, that's not causal, but that's again half a step in trying to get a sense of whether editors care about clicks at all or not. Let me move on to the empirical strategy and please feel free to stop me and ask any sort of question that you might have. The baseline specification is we're trying to explain clicks of user i in session s. A session is defined as a 30-minute interval. It's defined as an interval when an individual logs on and if there's 30 minutes of inactivity, then that session ends. The unit of observation is the user session. We look at clicks of user i in session s and whether the individual was treated in session s. Due to this randomization, what we can do is we can use within a user variation with a user accounting for user fixed effects as well as a time dimension, a time domain to capture general trends in the new cycle. This is just a randomization check to say that on average, the randomization happened okay. This is based on individuals who we saw both during the experiment as well as before the experiment. These are pre-experiment characteristics and we find that in terms of the number of days that they were active, you know, clicks during the day, total clicks, clicks from Germany or outside, they seem to be balanced on average. So, you know, the first sort of sanity check. Okay, so in terms of the baseline, three columns here, in the first column accounting for time fixed effects and individual fixed effects, what we find is that clicks on slots for increase by 3.75 percent when the user is in the treatment group relative to when in the control, relative to the control group. But of course, in the way that I motivated the question, you know, the average effects are less interesting than the heterogeneity that it might capture based on the amount of data that it has. And so we try and see how the treatment effect varies based on the number of observations or the number of times that individual has logged on to the website. And what we can see is that if the individual had no prior browsing history, and hence the algorithm was recommending something based on, based on social data or news or current news trends, we find that the editor seems to be able to predict those preferences much better. But as the algorithm keeps on getting more and more information on the individual, the individuals start clicking more when in the treatment group. So the sort of case that we're building is for better, a better match in terms of taste, which is identified by the algorithm. And when we look at, you know, we see the same pattern when we look at the total number of clicks, you know, when there's limited data, then when you're, and you're in the treatment group, the treatment doesn't perform as well. But as I get more and more data on you as the algorithm, then the algorithm starts performing better relative to the human. Of course, this is also super parametric, you know, we can, what we do next is try and trace this out as much as possible. And what you can see here is that again, you know, in the first few visits, which would be, you know, let's say eight to 10, in the first eight to 10 clicks, the human seems to be beating the algorithm, then the algorithm, you know, is learning, and it's improving over time. But and statistically, these points are different just because sample size, but economically, it's not that much of a difference, or at least, you know, depending on your prior, it's not super high. But what one can say is that, you know, initially, the human, the human seems to be doing better, then the algorithm catches up, diminishing returns sort of set in, but even for someone who's been who's come onto the website about a hundred times. And this is, you know, at the 90th percentile, the algorithm outperforms the editor by about 15% here, which is, you know, which is, you know, depending on the estimates reasonably high. So this is, you know, trying to show that, yes, personalization works, social data doesn't work potentially as well relative to the human personalization works, but diminishing returns set in, you know, reasonably quickly. There are some gains, but, you know, maybe not as large as one would expect. But we want to dig into this sort of personalization story a bit more. Basically, our idea is that an individual who is coming back on provides more and more data to the algorithm. Sorry, Ananya, one question. Will you provide us data on whether or not people come back to the site more often or not when there has been personalization? Because another measure of how well the thing works is whether or not people find the site more useful and whether or not they've got more visits. So I mean, maybe one way of interpreting this would be to, like, total number of clicks is probably correlated with the number of return visits, but we haven't looked at that. But I don't think that it's highly correlated with the total number of clicks. Well, that's not obvious.   So something to do with this discussion or whether you've got addictive addiction, you know, to websites and so on. Right. So yeah, I think we can run that. If my memory serves well, we did run that at some point, but I don't quite remember. The only slight issue would be that the algorithm was explicitly trained to maximize clicks of the user. So, but we can see if there are spillovers onto that other correlated dimension for sure. And that would potentially be a more longer term outcome as well. Okay. So again, thinking about clicks and preferences, preference heterogeneity amongst the users, we create a base on users who we also see in the pre-treatment period. We use a measure of sort of how far the user is in reading behavior relative to the others using a pre-treatment cosine distance measure. And for this, if you look at columns one and two, what you find is that on average, the further away you are from the average guy, the worse the treatment effect is. But as I get more and more information on you as the algorithm, then I can cater to your preferences more. And hence, if you have sort of misfeeding behavior, I need more and more data to try and cater to your preferences. Similarly, we look at rank effects and match quality. Again, the idea behind the algorithm is that you potentially click on slot number four because it's picked by the algorithm. And this is an article that you might not have seen if the algorithm had not pulled it up from down below. Of course, we can only see what the algorithm does in the treatment group. So this, the variation that we see here is only based on when the users are in the treatment group. And what you can see is that higher the ranks, so further down the page the article is, the less likely the article is going to be clicked. But as I get more and more information on you, then I'm more likely to recommend an article which is a better fit with your taste, and hence as a higher probability of being clicked. So this is sort of one part of the people where we're trying to pin down how the algorithm works relative to the human editor in terms of social data versus personal data. The other dimension is that of the time dimension, the stock versus the flow. The news is of course, a fast paced sort of industry, you potentially need a lot more, a continuous flow of data to update the algorithm to keep up with what the users might want. And till now I haven't said too much about the time dimension here. So there's like, people come back, there are 100 visits, 200 visits, etc. To credit the time dimension, we use another source of variation, which is what we call the new year bump. What had happened was that when the algorithm was being launched in December of 2017, what the data science team unfortunately did was that they hard coded the year to 2017. So when it became first of January 2018, the model was not updated to fetch personalized data. And they didn't realize that there was this glitch till the 6th of January, because I think they were basically away on holiday and they didn't really check what was happening. So what we're going to use is this sort of data bug as a natural experiment within our experiment, keeping in mind that see on the first of January, when users are treated, the data is still sort of fresh from the 31st of December. But data gets more and more updated as you go along that particular week before the bug is detected. And here what we find is that, again, focusing on only January, only December, January, and February, is that on average, again, the treatment effect is positive. But during those particular days of the new year bug, the treatment didn't do as well. And it did increasingly worse as you went along. So the new year day trend or the bug day trend is one for January 1st, two for January 2nd, and so on. And you can see that it kept doing worse and worse. So this sort of gives you an idea that while you can cater to preferences, you need this continuous flow of data, even if you're this large sort of news out. But one question is, okay, fine, continuous flow of data seems to matter within this setting, but can the stock of existing data actually have like a mitigating effect on this sort of lack of continuous flow? That is, if as an algorithm, I've seen you quite a bit before, maybe I need to update my model less because I have a fair sense of your reading preferences, at least within that short period of time. So what we do is that we, again, use this new year bug period and interact it with the prior stock of visits. So this is on the 31st of December and prior to that. And then again, and this is solely the new year bug period. So this is those six or seven days. On average, the treatment effect was negative. But this is, you know, this is sort of mitigated by an existing stock of data that the algorithm has access to. Right. So again, you know, continuous updates matter, but an existing stock of preferences can actually mitigate that to some extent. So that's the second type of data that we've that we've looked at in this, in this setting, which is continuous flow of data and the stock existing stock of preferences. So this sort of rounds up the first part of the paper, which is trying to trying to pin down the value of data. And now in the minute few minutes, I'll just quickly go through what happens to consumption news, news consumption patterns. And what we, what we do is that we can we construct an HHI index for consumption shares of, of a particular user across different topics. And we try and see what happens. What happens to consumption diversity when you're in a treatment group relative to when you're under control. And what you find is that since most of the effect in our case is coming from personalization, when you're in the treatment group, your consumption diversity doesn't decline. So a higher user, a higher level of HHI means a reduction in consumption diversity. And this spills over a little bit to other slots, but not too much.  What happens, what happens when, you know, when I, when I, when I as the algorithm see you over and over again, if you can see that the consumption diversity keeps on reducing over time. And this is sort of in line with our, our returns to, returns to returns to data picture where the algorithm consistently beats, you know, beats the human at, let's say, you know, four to six percent, like, you know, after the initial burnout rate. And so if you keep clicking, you know, if you keep getting served by the algorithm, and, you know, you have a higher probability of clicking on the treatment, which is personalized to your taste, you do see these sort of dynamics taken. I have other results related to, you know, other user characteristics, which might explain why an individual interacts more with the treatment relative to control. And, you know, this is more descriptive. But again, I can talk about it. But I think I'm sort of running out of time. So I also want to sort of go back to how I started out, you know, a general sense of where we are in terms of the value of data. So as I mentioned before, you know, we contribute to this literature on scale effects in data. For example, this paper by Leslie Chu and Catherine Tucker, they use, they use a natural experiment in Yahoo search to find that actually, if you reduce user history or access to user history from like six months to three months, it doesn't change too much in terms of the search precision. By any at all, they find some diminishing returns sort of in line with, in line with our setting. Schaefer et al, they find some, they find some effects in terms of the value of data. And, you know, I think there's a new version out, which I'm not sure what exactly the mechanism that they highlight. But again, it's like, you know, here, the contribution is that we were able to sort of pin down and bring these together within a field experiment. In particular, if you think of the data curve, maybe it can reconcile some of these conflicting results. That is, when, you know, when you have limited data, then additional data for the algorithm or search engine might play a big role, but diminishing returns might kick in reasonably quickly. Of course, this is one context, but the idea might actually generalize. Moreover, this paper is related to the computer science literature, which has evaluated the effects of recommender systems. And this, the computer science literature has the same sort of issue, like with this scale effects of data literature, which is that they look at offline evaluation. So they have a data set, and then they have a training sample, then they have the holdout, and then they try and see how the algorithm does in the holdout sample. But the computer science literature is more and more like embracing these real-time randomized experiments. And what they do find is that these offline evaluations are often at odds with randomized experiments. That is, you know, the results often slip for some reason or the other. So they are also trending in that direction. And in the computer science literature, you know, they don't focus really on the economic value. You know, they're looking at different features and their measures are slightly different. We can actually try and transform these click-based measures into overall revenue measures for the news outlet. And finally, in terms of the result on how algorithmic recommendations reduce consumption diversity, what this is related to, you know, a bunch of papers by Dokyuli and Kartik Hosanagar. But what we do here is that we have 20 weeks instead of, you know, the standard week or two weeks. We can look at dynamics. And actually, our result is almost the opposite because they look at collaborative filtering, which is social data and find that individual consumption diversity actually increases. And what we show is that actually personalization is the key which leads to reduction in consumption by the end of the century. So those are the three broad strands of literature. And that's all I have, you know, trying to outline the value of different types of data within, you know, this field experiment. We find that it's personalization, which plays a big role, at least not in our case, social recommendations don't play as a big role. Continuous flow of data is required, but it's mitigated by an existing stock of preferences. And finally, algorithmic recommendations, due to personalization, are reducing consumption diversity, and this increases over time. And that's all I have. Thank you so much. Thank you so much, Ananya. So now it's the Q&A period. So feel free to unmute yourself and ask questions. Paul, go ahead. Yes, thank you, Ananya. I enjoyed that very much. Can you, I'd like to push you a bit on the meaning of your measures of consumption diversity, because I could imagine two very different stories. Think of consumption diversity in a different dimension, say, in the kind of food you eat. Now, you might look at somebody who eats Indian food tonight, and Mexican food tomorrow night, and Italian food on Thursday, and Turkish food on Friday, as somebody who really enjoys diversity. Or you could look at them as somebody who's desperately flailing around to try and find out what they like. And if it's the former, then recommendation systems ought to be able to figure out that they like diversity and adapt their recommendations to offer them a reasonable variety in the recommendations. If it's the latter, then the whole role of the recommendation system is to find out that they really like Turkish food, get them to that equilibrium, and keep them there. Now, it seems to me your measures are not really able to distinguish between those two stories. You don't know whether the reduction in diversity you see as a result of the recommendations is the fact that basically the algorithm is better at guessing the type of topics you want to look at. And maybe there are other dimensions in which the actual treatment of those topics is quite diverse. All whether what's going on is that the recommendations are somehow rather narrowing the choice available to the consumer, perhaps because of sort of behavioral reasons, they're lazy, they just do what they're recommended, in a way that in some sense betrays their genuine underlying preferences for a diversity in the forms of their consumption. And I wonder if you could think about how you might be able to use your data to distinguish between those two stories? From the outside again, this is one algorithm, the particular algorithm is sort of trained to identify your preferences and try and cater to that and personal data kicks in as soon as it sees enough number of visits. Again, I haven't bought too much about trying to distinguish these, but maybe I'm thinking whether we can use our measures of cosine distance in terms of heterogeneity in user preferences and try and see how that interacts, how the trends in consumption diversity might be different for people who are relatively mainstream going in relative to those who are reasonably diverse starting out. So people who we observe before the experiment as well as during the experiment, maybe that's one way of getting at it. I'm just hypothesizing right now, but maybe that's something that we could look at. Since we're looking at that in the value of data section anyway, we can definitely operationalize it here and that might be it. Any other thoughts, questions? Yeah, sorry, maybe that I missed something when you were speaking, but did you speak about whether or not the total number of clicks change and what we have a substitution with other clicks? So in general, we find the same sort of trend with total number of clicks that is when there's limited data, then overall clicks go down. But as you go on with better algorithmic recommendations, the total number of clicks within a particular session increases. So what we do find is a little bit of business stealing from other slots and the overall the impact of overall click on average is maybe slightly negative, but not less than half a percent. So it almost seems to be a pure business stealing fact. We don't really dwell on that too much because again, because of the specificity of the experiment, if someone is to adopt algorithms, you'll potentially want to maybe do it for the whole website. But yeah, we do see some business stealing based on our estimates. Did you try to use the fact that for some people, the switch was between the fifth and the fourth slot and for others, from the 20th to the fourth slot and whether or not this made a difference in the consequences. I mean, it seems that for some people, what your algorithm did was basically change the order very later. Some, you know, the new story of such in fifth place became in fourth place. Right. For others, it was the new story which was in the 60th place. Does it make any difference? Yeah, so we have, if you can still see my screen, right? Yes, yeah. So if you look at this one, so this is looking at the rank effect. And this is again, you know, this is sort of correlational because we see variation only within treatment. So on average, as you go lower down the page, you're less likely to click in general. But as I get more information on you, and then the algorithm can pull it up from further down, then you're more likely to click. And this is for, you know, including slots. This is starting at slot five onwards here. And this is starting at sort of slot 14.  And you can see that, you know, the effect becomes stronger. So in some sense, by looking at what happens at the beginning, you would have a measure of how different is the taste of individual from the taste of the editors. If you looked at the what happens at the beginning for the first visits, you would, in some sense, you can measure whether or not the individuals have a very different taste than the editor and what is the effect, you know, what does this make for the how people react to the ranking and so to the right, right. So maybe what you're leading to is sort of merging like this, our measure of cosine distance, which is basically using information on on people who we observe before the experiment and how far they were in terms of, you know, their diversity of consumption. And then maybe interacting that with the rank effects. Yeah, I'm not entirely sure. Because basically what you're trying to say is that the rank effects would also be affected by how diverse you were initially versus not, right. Yeah, and how different you were from the editors in terms of Yeah, from the editors. Right. So this is the the measure of cosine distance is trying to capture how far the user is relative to relative to other users. And you're saying that the benchmark should be the editors. We could do that actually, we could do that because we have we have data on what the control group looked like day after day, hour after hour. So we can construct that measure. Yeah, and then we can interact it with these direct effects. I have a question. I was curious whether the algorithm is, you know, said it's maximizing clicks. Is it doing that in a myopic way or is it is there some role for exploration that is, you know, maximizing clicks, not just today for this user, but in present discounted sense. Yeah, so it's from from what we know of the algorithm and based on the paper that they that that they use as the baseline algorithm. You try and maximize clicks for when the user logs on. So I think it's reasonably myopic in that sense. Because if it had a sort of a longer term, I had an exploration aspect to then, you know, some of the implications would be different in terms of, you know, people getting stuck in built a bubble and so on. True, true. So I agree.  And but more and more in this literature, people are trying to see how algorithms algorithms or recommended systems can be devised such that there is room for exploration. There are some papers coming out with that. But yeah, in this case, it's maximizing clicks myopically. That's, yeah, right. And this is sort of also related to, I think, Jack's question about, you know, do people come back more often, which is a potentially longer term measure, but yeah. Yeah. And you should have mentioned like, personal data information filtering and then social data collaborative filtering as in two aspects, depending on whether, you know, how much history they use ahead. Is there any way to sort of understand how the two interact? Like, I would imagine there's sort of an interaction effect that, you know, the more the more social data that actually helps you learn faster in terms of the personal data. Yeah. So potentially, if we if you look at this sort of picture, you know, it's, it's generally, you know, generally diminishing returns kicking in. And, you know, this initially there's only social data. And then there's, you know, personal data being fed into the model. What we move from the algorithm is that as, as more and more personal data comes, comes within the algorithms reach, then it gets a higher weight in the recommendations. Can we completely separate the two? Not, not necessarily. And that's why I focus a bit more on, you know, when there is no, when there is no personal data, that's like the cleanest that, that's the cleanest situation for, you know, identifying how this, how social data is doing relative to the human editor.  Can I, you know, help us identify the role of personal versus social? There might be some interaction in this middle, in this middle, in the middle here, which we can't separate out. So that's why even in the paper, we try and, you know, we try and make this reasonably clear. Thank you. So speaking of a diminishing return, like you mentioned that the past studies also kind of discover the diminishing return in their data sets. Like, have we seen any papers? Like maybe this is also for the audience. Have we seen any papers with a non diminishing return data effect, like in the sense that I mean, it might be a constant return or increasingly return. I'm trying to figure out like, like, were we ever seen like a scenario with like, yeah, a different situation. Yeah. Yeah. So, and this is, you know, like, related to the broader conversation about the value of data. So, you know, Catherine Tucker and Leslie choose paper, they don't find any effect. They just use a natural experiment. So they don't, they find none effects. The binding paper finds some level of diminishing returns. And here we have, you know, reasonably clear diminishing returns. I think, I think part of the issue is the in the general conversation is that, you know, what if Facebook uses data from a variety of sources.  Then there is some sense that increasing returns might take in at some point. But I have not, you know, it's hard, you know, like, it's hard to get access to that sort of data to actually understand. But I think that's the biggest why when we think about, you know, regulating, regulating, but I haven't seen too many papers which try and which are able to highlight that. Thank you very much. So, any more questions? Not a question, but I would just comment on your question, which is, you know, there are some examples where you, some applications where you will get increased, you would expect to get increased returns, like autonomous vehicles, right? Like, you're going to get almost nothing in terms of value from the data until you get to a very high level. And then once you reach some threshold where now it's actually useful, then you get a massive increase in return. So by the nature of the problem that they're trying to solve, you know, you think about the returns to data, it's like, you need an enormous amount of data to get anywhere. But at some point, it'll, you know, be a huge, it'll be a huge increase. And then maybe after that, it's diminishing again. So I would think, I would think an S shape curve could be quite natural for quite a lot of different applications. But do you know of any, any paper which is even, because I like, in my, in how I see the literature, I don't see any papers. We don't have the data on autonomous yet, right? Because we haven't reached that point, but this is more like from the, from the theory of it, or at least from the people working on it, rather than an empirical paper. Why don't we stop right now, because it's been an hour and I would like to stop the recording. Just to say thank you very much for everybody who's participated and because we are French, who are giving you August off. So, but we are looking forward to restarting the seminars on in September. Julian and Andre have made a very great program. Thank you to both of them, actually, for the great project. And thank you, Ananya. And you know, we're keeping the session open. So if you want to insult Ananya when it's not recorded, you can do it as soon as I close the recording. Bye-bye. Thank you so much. Thank you so much for having me.


Hi everyone, Joe for jaspyscasewrakes.com coming at you with a five-box break of 2020-2021 SPX hockey One spot gets you three random teams and it's four packs per box and one art one card per pack So yeah four premium cards per box. It says right on the box This is break six. So it's the second half of that inner case that we popped open So here are all the boxes right here very big. Thanks to this group right here so let's copy those names and Actually, it's ten names that are part of the the promo here the original ten. So this will be for SPX Hockey and I think that fills up the promo. So in a separate video will be the So I got these names already Right and I got the Cup hockey names already The SPX hockey names So that fills the promo, thanks everybody All right, so we'll do that promo randomizer after this video, but for the break Let's triple it up and then ducks and Jets are commots old. They'll be 30 on each list So let's roll it. Let's randomize it two and a five seven times for names and teams one two three four five six and Seventh and final time William down to Daniel two and a five seven times for the teams one two three four five six and Seventh and final time Columbus down to Philadelphia All right, William with the blue jacket Steve with the Bruins Steve right with the Predators Steve Birch with the Islanders Greg with the Wild Steve Birch with the Ranger Steve right with the Panthers, Dan with the Oilers and Flames Daniel with the sabers Greg with the star Steve with the Devils Dan with the Canucks Jonathan with the Lightning Daniel With the Aves William with the knights Matt with the Cap Steve with the Red Wings Matt with the duck's jets combo Steve Birch with the Maple Leafs Matt with the coyotes Jonathan with the Montreal Canadiens Steve right with the Kings William with the penguin, Steve Wright with the sharks, Jonathan with the Carolina Hurricane, Steve Wright with the Senators, Greg with the blues, Steve Wright with the Blackhawks, and Daniel Katz with the Fly Flyers Fly. Alright, so let's sort by column B and we're going to pause the video and give it a little bit of time for trades and when we come back we'll see if there's any trades and then the brakes stick around. Alright, welcome back folks. Here on a Tuesday, thanks everybody for getting into the action. I appreciate it. Thanks everyone. And here is the 2021 collect two to three autographs and or memorabilia cards per box. There's only four cards per box as well. There you go Rex. Yeah, you got it. You got a room for Donvin Mitchell to keep going. Alright, for the hockey, first one is Lucas Carlson for the Blackhawks. Two color jersey going to Steve Wright and the Shytown Blackhawks. We have Keandre Miller shadow box card for the Rangers. That's gonna go to Steve Birch and we have Joe Pavelski for the stars to $2.99. That'll be for Greg and Dallas. And your final hit behind these three filler cards is Alexander Barkoff auto for Florida. Panthers, that'll be for Steve Wright. Next box, my knife bit me. Come on knife. Two. There's no blink. Oh, two cards there. Three and four. We've got Matisse Kivlankens. Kiv Lennox. 65 out of 99. That's for Columbus. That's gonna be for William Rosenthal. Nice jersey there, William. We got two extra, we got a bonus card here. We've got the Tyler Benson shadow box autograph this time. It's a cool looking card right there. That's for Edmonton and that'll be for Dan Hudson. There you go, Dan. And then we've got Brendan Gallagher for the Canadians to $2.99. That'll be for Jonathan. Jonathan, that's Hollywood with the base card here. And we have Mika Zbenyad extravagant materials for the Rangers. That'll be for Steve Birch. And then we have Alex Belzeal obsidian rookies. That's a sharp looking card there to 349. Also for the Canadians, also for Jonathan. Three boxes to go. We got Calvin Thurkoff, a blue jacket. That'll be for William. We got an obsidian jersey, Nicholas Bodine to 399. That's for Chicago. Steve Wright. And then a shadow box autograph, Josh Norris for the Ottawa Senators. Another sharp looking card. That's for Steve Wright. And whoa, that's a cool looking card there. It's one of those obsidian rookie cards again, but with some purple foil on there. Kind of makes it hard to see, but I think in natural lighting it's gonna look really sharp. That's Maxim Lutinov. 4 out of 99. Steve Wright with the San Jose Sharks. My favorite comic actor? I don't know. Probably, I don't watch a lot of comedies.  And not like in like a comedy, you know what I mean? There's Andrei Shevnikov, Carolina Hurricanes, Jonathan Arnaldi. And we got Nikolai Kenizov, 167 out of 299. I like Skeleton Twins actually was pretty good. That's a little bit of a dark movie, but I thought it was great. Nikolai goes to Steve Wright and the San Jose Sharks. I like Steve Kugen, right? British guy. I think he's pretty funny. There's a Liam Fowdy with the bluejack. It's Chris Farley. There's William Roosevelt. Chris Farley, I would have liked to see what he would have done as he, once he kind of grew out of that sort of slapsticky kind of phase. I think it's a shame that he passed away so early because I feel like he would have been a really interesting comedian actor, right? As he got older and evolved. I think he's, I think he is super funny, but I just wasn't into all the slapstick stuff as much, but you can tell that he was like super sharp, you know what I mean? So that's one of those, what could have been? What could have been if Greg Odin's knees didn't betray him, you know? What could have been if Jabari Parker didn't have ACL surgeries on both knees? There's country Miller, Relic for the Rangers, Steve Burch. Yeah, the trip movies are fantastic. I won't bury another Batman, Will. I won't bury another Batman.  She was only 16. I'm not gonna bury another Batman. I'm not gonna do it, my soy. I'm not gonna bury another Batman. Yeah, yeah, the Alan Partridge stuff, Will Driggers, if you like Alan Partridge stuff, you're gonna love, there's a, there's an audiobook that he did, where he writes as Alan Partridge, I think. It's eye-partridge, I think. Pretty hilarious, if you're into that sort of thing. Well, I like David Chappelle, but Rex's question was, Rex's question was comedic actor, so I guess I'm thinking more, thinking more actor, I don't consider David Chappelle an actor. He has done some acting. All right, like Chappelle show, obviously, but all right, there's Nick Schmaltz to $2.99. Yeah, Will Ferrell is pretty good. $23.99, David Pasternak, Pride of a Nation card for Boston, that's gonna be for Steve Burch. Yeah, the trip movies are pretty, yeah, Kogan, Rob going back with the accents. That's like, it's pretty much like hanging out with me and my friends, Will. Just, we're just that clever and sharp, but with worse accents. There's Alexander Alexaev for the caps. Matt DeLeo. Yeah, I think, I think Rob Bryden probably is bigger in the UK than he is here, but yeah, he does a lot of, a lot of those clever panel quiz shows, probably hosted by like Seaton Fry and stuff. There's Zach Hyman, Super Scripps, Super Scripps, and that goes to Steve Burch and Toronto Maple Leaf. That's XPX hockey in the books. Thanks for watching. I'm Joe for jaskiescasebrakes.com and I'll see you next time for the next break. Bye-bye.


So yeah, I got recommended this for a comment, I saw one comment and it was like, react to SF9 Now or Never, which is a great song by the way, if you guys have not listened to the song, listen to it, what the hell man. Someone commented down below, can you please react to SF9 Now or Never horror version? And I'm like, what? They've already released the horror version and then I searched it up and then it's with bingo music and whenever there's bingo music, it's always just so freaking funny with SF9.  I don't know, I feel like they're gonna get spooked quite easily and since it's SF9, they're massive dorks, so it's gonna be fun. But yeah, I'm gonna let this get straight into this. I'm freaking excited, I hope you guys are too and let's do this. Is this later on in the video? Oh, SF9. Juddus. Oh, look at them. Now they're gonna be doing it for real. Oh, what's this? Who is that? Yeah, wait, who is that? Yes, I understand. I understand. Java Update available. Oh, freaking love that man. Oh, what's gonna happen? I'm not scared because it's the other guys screaming. I understand. What's that? Oh, I would have shaped myself if that happened just then. Innocent. Oh, what's scared? He's angry, he's so scared.  Okay, it's like insomniac. That young man's screaming out when it was Chani. Oh, sorry. Dude, I saw it coming. Freaking dorks, they are such massive dorks. Funny. Oh, I don't know why. I just love it when I see people get scared.  Yay. That, yeah, man. Oh, that was good. That was good. I need to react to more stuff about dingo music and music. No, I just need to react to more dingo music videos because they're so good. They're legit just so good. I need to do those more often. If you guys got any dingo music themes that you guys want me to react to, then comment down below and I'll freaking react to it, man. But yeah, guys, I think this was gonna be all for this video. I hope you guys did enjoy it. If you guys did, then definitely show some support by leaving a like and also comment down below how much you freaking love SF9 and how much this made you laugh, or did you also get scared? Comment that down below and also subscribe if you guys are new to the channel because there will be more SF9 reactions, more dingo music reactions, and yeah, open that up. I hope you guys have a great day and just keep on smiling.


According to the Lambda Cold-Dark Matter Big Bang Theory, the universe is around 13.8 billion years old. First-generation stars would have contained hydrogen and a little helium and virtually nothing else, because that is all that existed at the time they formed. Very few of these Methuselah first-generation stars are still around. What we see around us today are second-generation stars, including our own Sun. In this segment of the How Old Is It? video book, we'll examine what we know about how old these stars are.


Hi, I'm Zor. Welcome to Unizor Education. We continue course of advanced mathematics for teenagers. The course is presented on unizor.com. And this lecture is part of the theory of probabilities topics. And we will talk about normal distribution, normal random variables, and especially about certain evaluation of the values of normal random variables. Using standard deviation. It's called sigma. So that's why it's sigma limits. All right. Well, let's just think about it. Why do we need the theory of probabilities just as a subject? Why do we have to research anything related to this? Well, obviously the purpose is certain level of prediction of the values of variables which we cannot say exactly that the value will be such and such. So there is always certain random process which we are talking about and random variable is something which might take this value or another value. And our purpose using the theory of probabilities is to predict that certain values will occur more often than others, something like this. So this prediction actually is the main goal of the theory of probabilities. Well, of course, this is some kind of a utilitarian approach. Any mathematical subject has its own value and purpose without any practical applications. But in this particular case, theory of probabilities actually was developed from these practical necessities to predict the values of certain random variables. Now, you also know that in many cases, random variables behaved like so-called normal random variables, distributed along certain bell curve where the middle of this bell curve is the expectation, the mean value, and the probability of the random variable to be from to is measured by this area under this bell curve. So this is a non-formal explanation of what normal variable is all about. Of course, there is a function which actually describes this particular curve and that precisely defines the normal distribution. But we are not talking about this. We are talking about intuitive understanding of what the normal random variable is. That's how it is distributed. And the area under the entire curve is equal to 1, obviously, because that's the probability of a normal random variable to take any value, and obviously it's 1, but if you would like to evaluate what's the probability of our normal variable to be in between points A and B, then we just have to calculate the area under the bell curve from this point to this point. All right, so it's very important to be able to evaluate the values of normal random variable, well, actually to evaluate, to predict the values with certain probabilities. And that's what this lecture is all about. Now, normal variables are basically defined by two parameters by the middle point, which is called mean, or expectation, and the steepness of this curve, because it can be this way or it can be this way. Now, in both cases, the area under the curve would be equal to 1 because that's the total probability, but in the case of this second curve, the values are concentrated more around the mean. In the first curve, which I drawn, the values were more spread around. So basically the steepness how this curve actually arises is characterized by standard deviation, as we know, sigma. So there are two parameters here. Now, so we have considered this particular normal variable with expectation mean value mu and standard deviation sigma. Now we would like to evaluate the probability of this normal variable to be within certain boundaries from A to B. Now, it's very convenient to actually look at the symmetry of this picture and consider that A has a position mu minus certain distance and B has a position mu plus certain distance. So these are equal to each other. So the whole bell curve is symmetrical and we usually understand that the values close to the mean value will be more often occurring. So that's why traditionally we just have the interval around the mean value, the symmetrical interval with the mean value to be in the middle of it as an evaluation segment. And we are interested in the probability of our random variable to be within this evaluation segment between mu minus d and mu plus d. So now it looks like our probability depends on two parameters, mu and d. Well, actually it doesn't depend on mu because if you shift this to the left or to the right, the probability to be within this distance d from the mu will be exactly the same. So our probability of our random variable c to be within interval of mu minus d to mu plus d actually is a function of only two parameters. It does not depend on mu, it depends obviously on d and it depends on the steepness of this curve, sigma, right? So let me simplify it even more. Let's just measure this distance d from the midpoint in terms of sigma, in terms of standard deviation. Then it will be basically a function only of one parameter of sigma. Now, what does it mean in terms of standard deviation? Well, there are actually traditionally, there are only three different segments considered when d is equal to standard deviation c, sigma, sorry, when d is equal to two sigma and when d is equal to three sigma. So these are traditionally, I mean obviously we can measure d in any kind of terms and obviously we can calculate it using certain calculus. But I'm just saying that traditionally it's probably sufficient to characterize the whole distribution with only the value of standard deviation sigma. Now considering that mu we know. So we know that for a particular random variable we know what its distribution is. We know its average, its expectation value mu and we know its standard deviation sigma. So for this particular variable we are evaluating what's the probability of this random variable to have the value within the interval mu minus sigma mu plus sigma or minus and plus two sigma or minus and plus three sigma. So these are traditional intervals which we are considering. And obviously if for some reason we know the distribution of our random variable which means we know mu and c and these numbers, these probabilities have already been calculated because it doesn't really depend anymore on any d. We just calculated for three different concrete cases. Then we can evaluate the value of the random variable, the values which random variable takes. We can evaluate with certain probabilities within these three different intervals. And let me just make you a concrete example. Let's say we already have this thing, this is mu, this is sigma, two sigma and three sigma. So it's mu minus sigma, mu minus two sigma and mu minus three sigma. And same thing to the right. One, two, three, mu plus sigma, mu plus two sigma and mu plus three sigma. So this is area from mu minus sigma to mu plus sigma. And with this particular bell curve, with this particular mu and sigma, this value can be calculated. And the value actually is a known number. So let me just give you an example. So the probability of our normal random variable to be within the interval from mu minus sigma to mu plus sigma equals to 0.6827. Now, what if I would like to say something about the values of the random variable, how they fall on this line? Well, they can fall here, they can fall here, they can fall here. Well, obviously this is area where it's more concentrated, but in theory there is a certain non-zero probability that our value will be somewhere there as well, right? So what I'm saying is that if the whole area underneath of this bell curve is equal to one, the area underneath this curve from mu minus sigma to mu plus sigma is equal to this, which means that the probability of our value, our value of the random variable to fall within these boundaries, to fall within this interval is this. Well, what does it mean? It means that statistically if we will perform our experiment, which results actually in the normal variable with these distributions mu and sigma, if we will repeat it again and again and again, then relative frequency of the value to fall inside of this interval is equal to this. So approximately 68 out of 100 cases we will fall within this area and in other cases outside. Does it mean that we can predict the value with a certainty? Well, certainty just completely out of the question, but there is some approximation. Well, we can say that 68 out of 100 approximately experiments will result in the value here. Is it a good prediction? Well, 68 out of 100, not really a good prediction. I mean, it's not really like a significant majority of the cases, but let's consider the interval which is slightly wider than this. Let's consider mu minus 2 sigma and plus 2 sigma. This is an interval. So we are adding these areas as well. Now, to be within this interval is much easier, right? I mean, it's wider, so whenever the random variable takes value, it's definitely more probable that it will be within this wider interval than within this one. Now, how much more probable? Here is the number. 9-5-45. Well, that's a significant growth, you see. So these areas bring a lot. So if it was something like 68% of the cases, now it's more than 95% of the cases when our random variable will fall within this wider interval. So now that makes our prediction significantly more precise. So if we are saying that with the probability of 95% or slightly more than 95%, our random variable will take value from this to this. This statement has much more validity. At the same time, we are widening the interval. So what's the value of our prediction? I mean, if we will predict that our value will be from minus infinity to plus infinity, the probability of this will be 1, but it doesn't really give us any information. So what I would like to say is that we are giving a less precise evaluation. We are giving a wider interval, but we are having this evaluation with a greater probability. I mean, whoever is asking us should really choose what he prefers to have a narrower, more precise evaluation of the values of the random variable with lower probability and wider evaluation with the higher probability. Well, it actually depends on the person and on a concrete task which is in front of him because if sigma is really very, very low, because if values are already concentrated within this very narrow interval around its mean, around mu, then probably we can expand the interval to 2 sigma and say that, well, the probability is very high that it will be within this interval and the precision of this interval is sufficient for a concrete task which the person performs. Well, and now let me just widen it even more. We add these areas. So we are doing from minus 3 sigma to plus 3 sigma. Mu minus 3 sigma to mu plus 3 sigma. And probability is 99.73. Wow, this is almost like 100%, right? Almost like 1, which means that these little tails are really insignificant. Not that it's not possible to get the value somewhere in this area, it is still possible with the probability of 0.0027. However, it's a very low probability, which means statistically, if we will perform our experiment again and again and again, I can say that almost everything will be within this wider interval from minus 3 sigma to plus 3 sigma around the mean value. So our prediction will be less precise, but with more probability. That's always a trade-off. So any practical problem has this kind of a trade-off. If we want to precisely evaluate the result of the experiment, then the probability of our prediction might not be very high, which means it can or maybe it will, maybe it will not. I mean the probability of 0.6 whatever, it's not very high. But if we are willing to expand the interval within which we expect our random variable to fall, then the probability of this obviously is increasing. So in many practical cases, this probability of 2 sigma is basically taken as a standard, so to speak. So whenever somebody is saying that, okay, this particular machine, let's say, which is manufacturing this particular part for something, and this part has certain dimensions which this machine is supposed to basically manufacture with. Well, dimensions might not be exactly like we have set this machine to do. It may be a little bit more, a little bit less, but we can evaluate this more or less. It depends on certain circumstances. Maybe it depends on different voltage in the electrical supply. Maybe it depends on some kind of shaking the table somewhere where this particular machine is installed. I mean there are different random factors which really affect the precision and the measurements of the part which we are manufacturing. So the people who are making the specification for this machine, they can say that in 95% of the cases the dimensions of this particular part will be from 2. And if this from 2 is not very large, then we can say, okay, I like this specification. The 2 sigma is pretty good enough. And if there are certain details which are outside of this range, which means their dimension is really wrong, well, we can throw them out, but considering this probability, I can say that I will throw out no more than 5% of my production as just no good or I will just sell it cheaper or something like this. But the quality ones which I define as this particular interval, the quality will be here. Now some other manufacturer can come in and say, you know what, the normal distribution of the dimensions of my detail, of my part which I am manufacturing, they also adhere to a normal distribution, but with a significantly smaller sigma, significantly smaller standard deviation. So the graph of my parts would be this. So more concentration will be around mean. The mean is supposed to be the gold standard, this particular part is supposed to be, right? So with a greater sigma, we can actually say that with this one, we can have this interval which is sufficient for us because the second machine is smaller than the sigma for the first machine. So we can get a wider interval. So the second machine with a probability of 99% gives me the good result, the part which has the good dimension. So something which is supposed to be put aside as a bad quality product would be in only less than 1%. So if I have to choose among these two machines, I would choose the machine which gives me my good dimensions with the higher probability, right? So that's basically how we are using these formulas just to evaluate the range and the probability of the random variable to fall within this range. If you like it, then we are buying this particular machine. If we don't, we are looking for something else. All right, so these are so-called sigma limits, one sigma limit, two sigma limit, and three sigma limits. So for normal random variables, these are very, very important. So we have to know the necessary validity of our prediction and use the sigma to basically achieve this particular level of our prediction. If we require that our prediction should be very, very precise, which means we have to really have a higher probability of this prediction to be true, then we have to really choose the three sigma interval.  So that's it for today. I do suggest you to read the notes for this lecture on Unisor.com. That's accompanying every lecture. And basically that's it. Thanks very much and good luck.


Hello, I am Thomas Blažowskas from Konus University of Technology, Software Engineering Department. In this video, I am going to present the software process models. For the effective software development, we need to use a proven and successful way of creating a software. It means we need a description of the software process. That description is called a software process model. In this video, we will present the integral parts of the software process model descriptions, such as activities, roles, artifacts, and the overall idea of the software process model. In order to develop a software system, a structured set of activities is required. There are many different processes which involve many activities, but these activities are used in all the processes. It is specification of the requirements or defining what system should do, design, establishing the organization of a system, construction or the implementation, implementing the system, validation, checking that it does what the customer wants, and evolution, maintaining the software until it is discontinued. Of course, there are many other processes such as project management, configuration management, quality management, process improvement, and many others. All the software process activities are performed by the persons who play a specific role in the software development process. Depending on a process, one person might play several roles. In this example, we pictured the roles of the scrum process. Alice is the product owner. She wants her product to be the right one, therefore she will work a lot on the specification of the requirements. The scrum master will supervise the teamwork. He will ensure that the activities of the software process are performed correctly. Carolus is a software architect, therefore he will participate a lot in the design activities. And Thomas is a programmer. He loves coding, implementing the software features, so he will mostly engage in software construction activities. On the other hand, Thomas needs to be aware of the different activities so that he can construct the right product, therefore he participates more or less in other activities as well. The software process artifacts are the work products, which are both the input necessary for some activities to start and output generated by the activity. In this example, we have three factories that represent appropriate activities. For the design factory to produce some work products, it is necessary to provide the requirements. After it is done, the design factory generates the design documents which are provided to the construction factory. In the construction factory, the actual software is being created. This factory produces a lot of work and the code is typically the most important work product. The code is provided to the validation factory which checks the code and creates the bug report document. At this point, we know that the software includes the activities, roles, artifacts. Still, these things do not describe the dynamics or the behavior of the software process model. The important task is to determine and describe the order in which activities are performed, which activities need to be implemented first so that the other activities could be started and that defines the path in which software is created. Beside the order of the activities, we need to determine the flow of these activities. Until now, we show examples that implement the linear flow. Each new activity starts after the previous activity is completed. This flow depicts the notorious waterfall model, which sometimes is not the most efficient model. This flow depicts an iterative style. The development of a product is split into multiple iterations. Each iteration implements part of the requirements. And this is an agile approach. The activities are performed in an incremental iterative way. The product is split into several parts or the increments. Each increment is developed in an iterative way. After increment is completed, it is deployed so that the customer could try it. At this point, we know that a software process includes activities, roles, artifacts. We also know that it is necessary to describe the order and the flow of the activities. The important question is how to put everything together, how to structure all those elements. We need a description that describes a path that is needed to be done from start until the software is discontinued. That description might include texts, diagrams, and it is called a model. In this course, we will investigate several historical and modern software process models in detail. Actually, there are many software process models that are separated into generic and specialized models. The most common generating software process models are the traditional models. These models are not so common nowadays, but many modern models include the activities from these historical models. We will investigate the most down-waterfall model. The iterative incremental models. These are the models that represent an iterative way of creating a software, or an incremental way, or both, like we explained in the last slide. We will investigate prototyping spiral, agile, and unified process models. The specialized models provide a particular effective approach to solve specific tasks. We will present shortly component-based development, aspect-oriented development, reuse-oriented development, and the formal methods. The software process model represents a proven way for the effective development. The software process model is a description that describes a path of what is needed to be done from the start until the software is discontinued. That description might include texts, diagrams, and other materials. The description usually includes activities, roles, artifacts, and may involve various conditions and constraints. The order and the flow of the process activities are extremely important. It is the main difference between the software process models. Thank you for your attention.