And this is Jero Merju, a PhD student at St. Paul Berkman Fellow, a core member of cooperation group here, and here to present a piece of work that's been going on for a while and coming up with interesting things. It really, in many senses, is the first major connection between experimental economics and extensive field observation, as well as some really interesting application of experimental tools to the design of peer production communities. And it's just a thrill to see you here and looking forward to hearing your presentation. Thank you, Yochai, for those very kind of introductory words. This is indeed a paper about pro-social motivations to contribute to Wikipedia, but let me first try and give you a little bit of a sense of the broader research agenda or motivation that's behind it. So usually when I present this research to a crowd of economists, what I ask them to do is to think about all those computer-based products that you see here appearing on the screen which are certainly of high value to many people which are produced by highly successful firms that compete in the labor market trying to attract the best engineers. And what I tell them is that I like to talk about the somewhat recent phenomenon that has taken off in the last decade or so, a little bit more, whereby people gather over the internet voluntarily self-assigned work without any kind of price signal or market coordination, without any kind of pre-design formal design rule or leadership and successfully coordinate towards the provision of functional, free equivalents of those firm-based products, sometimes successfully driving those for-profit counterparts out of their original markets. And so this emerging production model that following Yochai's insight, I will call here pure production, the success of this emerging model is difficult to understand, at least to an economist, through the assumptions of standard economic theory about individual preferences, namely perfect rationality and self-interest. And so this paper is really about trying to understand how we can start to unpack the success and the functioning of that model by appealing to economic theories that rely on non-standard economic preferences. And we want to start doing that, focusing on Wikipedia for two very good reasons. That is, on Wikipedia, Wikipedia is kind of a particularly clean study site for pure production because you do not have or not very much, not very many, economic incentives involved. So for instance, you have no extrinsic incentives to contribute to Wikipedia. It's difficult to get paid to contribute to Wikipedia and sustain your contributions. And at the same time, there are no signaling value that you can derive from your contributions on the labor market. So it's very rare, although I suspect it's going to be increasingly the case in the future, that Wikipedia contributors put their contributions on their resume. And so with that respect, I kind of like that quote by Kizor, who is a Wikipedia administrator, who tells us the problem with Wikipedia is that only works in practice, in theory, it can never work. And what I really like you to do here is to think about the decision to contribute to Wikipedia as a public goods dilemma. So a public goods dilemma is a situation that economists have been studying for quite a while now, and it's any kind of situation which you can decide to take an action that has a private cost to yourself and also a private benefit. As it turns out, the private benefit that you can derive from taking this action is lower than the private cost. On the other hand, the benefit to society of incurring taking that action is higher than your private cost. So here is the tension, right? If you were perfectly self-interested and rational, you would never take such an action because all you care about is the ratio of your personal cost to your personal benefit, right? However, taking that action may be socially efficient. So if you think about the decision to format and put some idea or knowledge that you have in shape to put it in there in Wikipedia, that's costly to you. What do you benefit from it? Well, you already know that information, right? So you can presume that the cost is higher than your benefit for yourself. However, this has huge benefits to society in general. So this is the tension. So economists have been studying in theoretical models the kind of pro-social preferences that could push people to contribute to those kind of global public goods. And those generally appeal to pro-social preferences, that meaning that you do not only care about your own payoffs in that game, but you will put into your own utility function the utility of other people in the game. You will take into consideration the actions of other people into your own utility function and that will push you towards cooperation and overcoming that public goods dilemma. So there has been three classes of models that economic theory has been putting forward in order to rationalize people often observe willingness to cooperate in public goods-like environment and that traces back to the work of Elinor Ostrom. The first kind of motive that has been put forward historically by the theory is based on altruism, sheer altruism. The second class of motive that we have is based on reciprocity, meaning that you will be willing to respond in kind to kind actions of others. If others contribute to the public good, you will be willing to contribute. So this is an example of negative reciprocity, talk about positive reciprocity in that respect. And kind of the last kind of motive, pro-social motive that's been put forward by economic theory is based on social image, whereby by incurring a personal cost to contribute to the public good, you want to signal some quality about yourself to other people and by being able to do so, you will derive a certain utility. So this is very unlike the two other motives, the reciprocity and the altruism one, which do not rely on other people watching what you're doing to be at work. So social image is really about when you allow other people to watch, how do you respond? So what we're going to do in this paper? So we have those theories around and those theories have been extensively tested in physical university labs, but what we're going to do here is something different. We're going to elicit the social preferences of a representative sample of Wikipedia contributors with an online experiment that we will couple with observational data, and then we will use those preferences to try and predict subjects' field contributions to Wikipedia. What's the value added to our knowledge or economic theory in general? Well, this is the first comprehensive test that we have of the relative role of each theoretical class of social motive for incentivizing contributions to real-world public goods, and this is one public good that really matters to the world. So we're going to try and run some kind of a horse race between those explanations, altruism versus reciprocity versus social image, using this experimental data and seeing whether they predict or not the number of contributions that people make to Wikipedia. So let me talk about the design of the experiment a little bit. I'm going to rely on very standard experiment that has been used for a very long time in experimental economics to try and tease out those preferences. So I'm going to try and measure all of those three classes of preferences each time giving you two alternative measures so that I can check for the consistency of the results that we get. So let us start with the reciprocity motive. What I'm going to do is that I'm going to run a very simple public goods game. That's the translation in experimental economics with monetary payoffs of the public good situation that I explained to you before.  Each one of you is endowed with $10. Each one of you has a decision to make. How many dollars you want to invest in a common project, and how many dollars you want to keep for yourself? Each dollar that you decide to invest in the common project yields a private benefit to you of .4. You invest one, you get .4 back. That's obviously not efficient at the individual level, but it also yields a payoff of .4 for the other three members of the group. That is for each dollar that you invest, the group as a whole gets 1.6. So here you have the tension between your own private self-interest and the interest in the group in general. What we're going to do here is that we're going to allow you to condition your contributions on the contribution of the other group members. If the other group members give zero, how much do you want to give? If they give one, two, three, up to 10. We're going to take the proportion of your endowment that you conditionally contribute to that public good as a measure of your reciprocity motive. How much you're prone to conditional cooperation. This is going to be our first measure of reciprocity. A second measure of reciprocity that we're going to consider is a very standard one, two is based on the trust game that's been used extensively. Here what I'm going to do is that I'm going to divide the sample of Wikipedia contributors who are going to participate in this online experiment in trustors and trustees. The trustees are the guys in orange here on the screen, and those are the guys I'm going to be interested in here. Both players have an endowment of $10. The trustor in green here has the opportunity to transfer whatever amount he wants to the trustee and keep the rest for himself. Whatever amount he transfers out of his $10 is multiplied by three. And then the trustee in orange has a private decision to make. How much of the amount that he receives he wants to send back to that trustor. Knowing that he has absolutely no obligation to do so and if he were perfectly rational and centered or interested he would never return anything in the first place.  That guy has been trusting me he's been kind to me you feel an impulse to respond in kind. That's what I'm going to try to get at here. By the way, the decision to send here by the green guy, the trustor, is considered generally a measure of general trust towards anonymous strangers and we're going to rely on that maybe a little bit later on if I ever get there. But for the time being the trustee's behavior in terms of how much he wants to send back is what's interesting to us. You say these games have been extensively used today have they been correlated to actual, so if the last people played this game we have patterns of their distribution in these games and then we observed them in other kinds of settings and we're like the person who gave away was really trusting there also was really trusting we saw them in the grocery store doing this thing in the room. So that's the whole purpose of this paper. We almost have no evidence that the behavior in those games that have been used extensively to test economic theories in the lab how people behave how do those map actually to behavior outside of the lab. So we have a few papers about shrimp catchers and things like that where they cooperate in common pool resources and things like that but Wikipedia is much more interesting, right? So yeah and this is also the first comprehensive test we have all those motives here in the same frame and we can study how they interact with one another so this is the kind of value added here. No, no, no. The rules are fixed. Fixed rules. So, so far so good for the reciprocity motives. We have two measures one based on the public goods game the other one based on the trust game. What about altruism? Well, the workhorse for studying altruism in the experimental economics literature is the dictator game. So we're going to run a standard version of the dictator game. Again, remember here all the interactions in those games are one shot totally anonymous and the Wikipedians are not playing with one another. They're playing with our internet users at large and they notice. So what we're trying to get at here are deep preferences in an anonymous setting. So dictator game. This is a very simple game in which we're going to divide the population of participants into a dictator and a receiver. Dictator is endowed with $10. The receiver is not endowed with anything. The dictator has a choice to transfer whatever amount he wants to the receiver and that's the end of the game. So basically the receiver has no say in the interaction here. He has no decision to make. He has no opportunity to influence the decision of the dictator so that the amount that the dictator is willing to transfer that game can only be interpreted as sheer altruism towards an anonymous stranger. So I'm going to take the proportion of your endowments that you're willing to transfer to that random stranger as a measure of your baseline altruism. However, because we worry that people may be incentivized to contribute to Wikipedia not out of a sense of general altruism towards the public but out of a sense of altruism towards their fellow in-group of Wikipedia contributors. In this particular setting we're going to run a modified version of the game where we will tell explicitly our Wikipedians now you're being paid with another wiki type of contributor how much you want to give. This is going to be a measure of directed altruism in a sense and this is going to be our second measure of altruism that we're going to use here. So far so good, reciprocity, altruism two experimental measures each time. What about social image? Well, social image is something that's kind of difficult to measure experimentally even more so in a decontextualized context that economists like to use in experiments. So what we're going to do here is that we're going to rely on the wealth of observational data that's available from Wikipedia to try and get indicators of wherever people are concerned with their social image within the Wikipedia community or not and assess the impact that that has on the number of contributions that they make. So the first thing that we're going to use are Wikipedia personal user pages. So let me just ask first how many people in that room have a Wikipedia account. That's so cool. I've been presenting that paper in many, many economics conferences and each and every time I ask that question I have one guy to me waving half of his hands at the end of the room. So it's kind of nice. So I won't have to explain as much. Basically each and every... when you create a Wikipedia account for yourself the system asks you for a username that you have to choose and that automatically creates two pages that hold your name. One of them is your Wikipedia user page. This Wikipedia user page is blank by definition but you can post on it information about yourself. You can present yourself to the community, explain who you are and stuff like that. The important thing about it is that it's not crucial to the functioning of Wikipedia. You can perfectly be a very engaged Wikipedia contributor and have a totally blank personal user page and still be able to perfectly contribute in an efficient way. And so what we're going to do here is... here is an example of a user page. That contributor, just as an example, tell us I was born in Windsor, Ontario, Canada, blah, blah, blah, blah, blah, lots of information. I strongly believe that sports are to be participated in and not watched. Here is the kind of things that you can say in your Wikipedia user page. So a very simple indicator. What we're going to do is we're going to measure the size of that user page in bytes and consider as social signalers, so to speak, the subjects who have a user page whose size in bytes is higher than the median in the sample. Those who have a user page whose size is lower is lower than the median in the sample would be considered as relatively less concerned with their social image with the community. That's going to be one measure. A second measure, as always, trying to check for consistency, is going to rely on some awesome data that was collected by my fellow fellows, Michael Hill and Aaron Shaw, with whom I was fortunate to interact and work for the past couple of years. And this is going to make use of barn stars data. So what are barn stars? Barn stars are a social rewarding practice that's mainly restricted to highly engaged contributors that anybody can do. Basically, if you come across some awesome contribution that another contributor has made to the Wikipedia project, you may want to leave him an image of a star that goes along with some personalized text acknowledging the contribution. Here is an example. Here is a Wikipedia who thanks another one for being one of those awesome Wikipedians who produces great content in a collegiate manner, helping out all over and great dispute resolution. Those kind of awards, you won't pass them on the Wikipedia user page of the contributor, the one that he controls and he uses to present himself to the community. You're going to post it on the second page that's automatically created when you create your Wikipedia account, your Wikipedia talk page. Unlike the user page, this page is crucial to the functioning of the community. This is where people leave you messages, try to coordinate work, ask you for references, and maybe hopefully, if you do a very good job and have 100,000 of contributions, we'll post you one of those awards. So now, once you receive one of those on your talk page, after some time, the award is likely to disappear in the flow of conversation. It's going to be archived in old conversations. It's nice that we receive one of those. I'm perfectly speaking, as it turns out, and this is the brilliant idea of Mako Hill and Aaron Shaw, approximately half of the Wikipedia decide to circumvent this by manually moving those awards from their talk page to their user page, which they control, so that it will be displayed for everybody to see forever. What I'm going to do here is conditional on receiving a born star. And this is very important because this is going to constraint my sample of Wikipedia to highly engaged contributors.  Conditional and receiving one, I'm going to consider you a social signaler if you moved at least one of those awards manually to your user page to be displayed to everybody. So that's going to be the second measure that we're going to use. We're going to consider a social signaler by this measure as he has organized a little award section in his user page where he lists all of his barn stars. Right? So two measures, right? Size of the user page, break the sample into relatively big, relatively not big, barn stars, conditional on receiving one, whether you decide to advertise them on your user page or not. That's going to be the measure of social image. Practical challenge that we faced while designing this experiment. Usually experiments are designed to be run in the lab, in a physical lab where people come in a room, in a university. This is an online experiment. So obviously people don't have time online. You need to guarantee a proper understanding of the decision problems and we had some thoughts going into that just to give you a little bit of sense of that. We basically designed flash animations, the one that you saw previously, that illustrate the basic gist of each game real quickly so that people can get a very quick sense of what's involved in the game without having to read through all those tedious instructions over and over again. That's one way we did it. And people also had, for instance, the opportunity to use earning calculators before actually making their decision. So you could try all the possibilities and the scenarios you were interested in before actually making your decision and the confidence that you understood what's involved here, you make your decision. This is how we went about doing this experiment. Subject pool. This is very important. How did we go about recruiting those Wikipedia contributors? In all of the tables that I'm going to show you in the subsequent analysis what I'm going to try to explain is the number of contributions that subjects have made to the Wikipedia project or number of edits that they've made, right? So in all of the tables that you're going to see, this is always the thing that I'm going to try to explain. We only recruit from Wikipedia registered users in order to be able to track their full contribution records just to give you a little bit of a sense of the dynamics of contributions to Wikipedia. Like many other features of the Internet if you think about contributions to open source software think about participation in online message boards and Wikipedia too, the number of contributions that people make is highly skewed. A vast majority of people never participate. A small majority of people make an astonishing amount of contributions. I'm digging into that a little bit. As of 2011, about 200,000 people registered to Wikipedia account each month. Certainly a non-negligible influx of new contributors each month that potentially come in. However, the reflection of what I was telling you before is that of those individuals, only 2% make 10 contributions or more within that first month and only 10% of those 2% make one contribution within the following year, right? As a result, as of 2007, most of the contributions still come from a very small number of contributors.  There is high heterogeneity in the contribution records, right? The vast majority of them made only a few hundred contributions about 5,000 editors made more than 10,000 contributions and you have 200 editors that have contribution records ranging from 100,000 to a million contributions. So this is really what we would like to capture in our sample of participants in our experiment and try to explain as a feature of that space. In order to do that we're going to recruit or Wikipedia contributors from the three following experimental groups. The cohort of you Wikipedia contributors all the guys who registered an account within 30 days prior to the experiment those guys have typically contribution records ranging from zero for most of them to already 300 contributions which is already quite a lot. Second the group of engaged Wikipedia contributors those are the guys who already reached the threshold of 300 contributions and are still active in the project. And then kind of a separate group that I want to consider as a class of itself is the group of Wikipedia administrators. Those are contributors that come from the group of engaged contributors. However they opted in selected in a very competitive peer review process at the end of which the community of editors granted them because they were considered trustworthy with special oversight rights over the encyclopedia. Those are the guys who perform a policing role within the community. Those are the guys who are in charge of active users.  So I'm going to consider those as separate in the analysis and I'm going to talk about regular contributors being group one and two and Wikipedia admins are being group group three, right? Okay so this is the subject so how do we go about recruiting those guys? So basically what we used is the Wikipedia banner system. So we're in December, normally the fundraiser of Wikipedia has been launched. Those are this is the system a banner system that you see popping up at the top of every Wikipedia page each year asking you to donate money featuring the face of Jimmy Wells real big, right? Every year. Basically we used the exact same mechanism to contact or Wikipedia contributors we team up and partner with the Wikimedia foundation to code that banner so that each and every time a Wikipedia contributor logs in Wikipedia, the system looks for his metrics, determines if he's eligible to participate in the study that is if he belongs to one of those three experimental groups if he does, displays the banner to him at the top of every Wikipedia page that he visits right? Until he disables it or clicks on it if he clicks on it, he's automatically redirected to the experimental economics platform that we own is able to perform all of the games, have his earnings calculated and he's paid in real money through an automated PayPal transfer in the end, right? This is how the process worked the experiment was kind of a success practically speaking we had 850 subjects completing a 25 minutes experiment in 8 hours so this is our sample just to give you a little bit of a sense of how representative that sample is from the underlying population of Wikipedia contributors here is the distribution of the number of edits that those Wikipedia contributors made compared with the whole underlying population of eligible contributors. You can see especially for Wikipedia administrators and engage contributors that the distribution match pretty closely. As far as new contributors are concerned, the ones that we get on average are much more active. You see we don't have that mass here at zero. Okay, another way to look at the problem of the representative of our sample is to simply compare the demographic characteristics that we collected about those contributors against the demographic characteristics that was collected in the 2011 Wikimedia Editor Survey. This was the first survey whose purpose was explicitly to get as precise a snapshot as possible of the demographic of the Wikipedia contributors and you can see here that the numbers match up pretty closely, right? Some features are very well known. Female participation is really low. This is a research question of interest in its own rights in my opinion. People are usually older than what some would assume initially and also much more educated, right? You have 30% of the sample that has a master's or a PhD degree. So, here comes the regression tables. As I said, always the things that we are going to explain here is the number of Wikipedia contributions that people make, right? And this is estimating here the impact of the demographic variables on the number of contributions that you make. So, crash course in econometrics for those who don't know about it. Those coefficient here basically tell you if you move from one unit in this variable what's the predicted impact in terms of percentage change in the number of contributions that you make, right? The more stars you have next to the number, the more precisely estimated is the impact. If you don't have any stars it basically means you cannot have any confidence in the effect. The effect is basically zero. The more stars you have, the more significant it is. For instance, here moving from being a male to being a female is precisely estimated as associated with a 36% decrease in the number of contributions that you make controlling for all those other things. That's how you're going to interpret those tables. This is all public information. So, the demographic variables you collected in the survey the demographic information you collected in the survey. At the end when people play all the games, we ask them about those things, right? It's their self-reporting of their age, gender things like that. That's right. And then the number of contributions they make is public information. So, you can just extract that. Okay, so some demographic characteristics actually predict the number of contributions that people will make. However, if you have some interesting underlying heterogeneity here if you cut that sample, the whole sample here is the whole sample of regular contributors, non-admins, right? Now, cut that sample into according to the median number of Wikipedia contributions that they made which is here in the sample 1,900 contributions already. It's a pretty high number. Run the exact same analysis on both sub-groups the below median and the above median group. The above median group are the so-called super contributors, right? What you basically see here is that the demographic variables significantly predict the number of contributions of the below median group from being a non-contributor to an engaged contributor. Once you're an engaged contributor well, it's very difficult to predict how far you're going to go even with basic demographic variables. Among super contributors, very difficult to predict what happens in the space. You can see here the figure for admins too. So, right, this is already interesting in itself but what I'm going to do here back to our interest about pro-social motivations to contribute to Wikipedia controlling for those things that is holding those things equal, we're going to see what's the predictive power of or measures of altruism, reciprocity, social image if any on the number of contributions that people make to Wikipedia. Here is the basic result for altruism.  Basically, no stars which means no predictive power at all. By those measures altruism does not play a role in number of Wikipedia contributions that you make by those measures. The picture is somewhat different if you look at reciprocity. Both measures, public goods and trust yield consistent results in the whole sample of regular contributors I'm not talking about admins here an increase in reciprocity and this is always in both games this tells you moving from no reciprocity in the game to full reciprocity in the game what's the impact in terms of percentage for the 8.8% here, right? Those are precisely estimated and are positive. Again, break the sample into according to the median number of Wikipedia contributions just like I did with demographic variables rerun the analysis and what you get is that the coefficient rises by 50% and is highly statistically significant in the below median group however, is not significant at all in the above median group. What that essentially tells you is that a preference for reciprocity conditional cooperation you participate or participate you edit my article I'm going to come back and edit your article we're going to collaborate on this thing that preference can lead you from being a non-contributor to an engaged contributor once you're an engaged contributor that preference doesn't predict how far you're going to go if you're a supercontributor that's the basic message so far, altruism by the measures that we have do not seem to predict anything reciprocity does what about social image let me give you some descriptive statistics about both variables the one based on user pages the size of the user page the one based on whether you moved your band stars basically all Wikipedia contributors have a potentially blank Wikipedia user page so I have everybody in the sample here the sample is basically broken into those who have relatively big user pages those who have relatively small user pages as I said band stars different 81% of regular contributors who received band stars are in the above median group the so-called supercontributors so when I use this variable derived from the band star this will mainly tell me about what happens in that group of supercontributors so let's do the exact same thing controlling for demographic variables insert both variables in the model see whether they have some predictive power as it turns out based on user page it does, here being a social signaler is associated with a 130% rise in the predicted number of contributions you make again, the coefficient rises by 50% and is highly statistically significant in the below median group but this time unlike reciprocity by both measures, user page and band stars social image continues to push towards higher level of contributions even within the group of supercontributors the above median group right okay so again, trying to sum up trying to keep everybody with me altruism does not seem to play a role by our measures reciprocity does conditional cooperation up to a certain point so the question of interest here now is what's the interaction between both motivational drivers, right reciprocity and social image do they play for the same people, for different people so what we're going to do to try and answer that question is that we're going to re-estimate the impact of reciprocity here but for both subgroup of social signalers and non-social signalers separately here is the table so if you focus here whole sample and below median that's reciprocity for social signalers reciprocity for non-social signalers reciprocity for social signalers based on the trust measure again non-social signalers based on the trust measure what you see here consistently is that the effect is statistically significant always and only in the group of non-social signalers that is reciprocity incentivizes contributions only for those who are not concerned about their social image in the community what that suggests is that both motivational drivers are at play in the population but in different subsets of the population if you're motivated by your social image you will not be motivated by reciprocity and the reverse holds too again you can break that down in the above median group in which nothing happens as we saw reciprocity doesn't have a big role to play to predict the trajectory of those who are already engaged in Wikipedia so far so good that's already a great deal of information let me try now and move out of the case of regular contributors and focus on the case of Wikipedia administrators as I said previously this is kind of a special group because those are engaged contributors who opted in a very competitive and costly peer review process in order to be granted with special oversight rights over the encyclopedia so those are the guys who perform the policing role in the community and they are in charge of dealing with disruptive users and managing, maintaining, curating the encyclopedia and so on and so forth and have some special rights to do that let's run the exact same analysis altruism versus reciprocity versus social image on that group of Wikipedia administrators result for altruism same thing, no predictive power at all by Iver measures right here are the results for reciprocity and so this is kind of surprising right this basically tells you that within the group of Wikipedia administrators Wikipedia participation is negatively associated with the particular taste for reciprocity and conditional cooperation so that seems surprising at first but to me suggests some kind of a thick skin hypothesis right so those are the guys who exhibit the most extreme contribution records and they are in charge of dealing with disruptive users and so I'm going to try and dig into that a little bit later on suggest some hints about what could explain this that's for reciprocity what about social image well by the user page measure we get a significant impact but not by the barn stores one so we get some evidence that social image continues to drive participation within the group of Wikipedia administrators but it's less strong than for regular contributors okay so before I try and wrap up and open the floor for a question which I'm sure you have many let me try and dig a little bit into that thick skin hypothesis for the Wikipedia admins and what I'm going to do here is that I'm going to exploit the data that I have left and not used so far because it wasn't relevant to the theoretical discussion I'm having in terms of economic theory about trusting behavior in the trust game so you remember the trust game right you have a trustor and a trustee both of them are endowed with ten dollars the trustor decides how much he wants to send to the trustee the amount is multiplied by three then the trustee has a decision to make how much does he wants to send back for the time being that measure of how much he wants to send back was a measure of reciprocity now we're going to be interested in the amount that the trustor decides to send in the first place this can be interpreted as a measure of trust towards anonymous strangers right the guy that plays as the trustee has no incentives at all, no obligation to return anything to you so if you decide to pass on some fraction of her endowment to that random stranger which you will never meet again well that means that you're some kind of a trusting type towards strangers so I'm going to use this trust measure and hypothesize that within the group of Wikipedia administrators those who decide to perform relatively more their policing role will be relatively less trusting of strangers before I do that, first step let me try and correlate that trust measures with the number of contributions that people make within the group of regular Wikipedia contributors not admins this is the result that you get basically no effect which was to be expected right why would you think that trust is the number of contributions you make to Wikipedia however if you run that analysis among the group of Wikipedia administrators now you see a negative correlation between the extent of their Wikipedia participation and their level of their generalized trust as measured by this experiment try and precise that a little bit we collect the number of users that each of those Wikipedia administrators blocked the number of pages that they decided to protect the number of pages that they decide to delete from the encyclopedia all of those correlate negatively with the level of their general trust in the experiment significantly so in two out of three measures last piece of evidence that we have here is we ask those Wikipedia admins what's the fraction of their time that they spend on Wikipedia performing this administrative role so what's the fraction of your time that you spend on Wikipedia that you spend performing those admin edits we only have 27 observations here but still we're able to identify a negative correlation with general trust the more they say and self report that they spend time doing admin edits the less they are trusting of general strangers so now this could be interpreted in two ways I think that we have the data here to tease those explanations apart and I want to make that very clear you could interpret that as the simple fact that people are diversely motivated in the world they self select into different contribution patterns and we need those administrators to protect the encyclopedia from potential free riders or disruptive users those guys are performing a very valuable role and we're simply able to explain through those experimental measures how those people self select what's the ecology of contributors to Wikipedia another way to go about interpreting these results is that basically those Wikipedia administrators are able to use their power in a way that can trigger some frustration in the community and that's correlated with the lower level of trust that we have so this is kind of the negative interpretation and I have no way to disentangle those two with the experimental data that I have so I look forward to hearing if you have ideas about that try and tease those apart but this is where I stand at the moment and this is still a work in progress okay so trying to wrap up real fast just to go back to the broad picture here this is really the first comprehensive test of the relative role of those free theoretical classes of pro-social preferences that has been put forward by economic theory to try and explain why would people be willing to contribute in those kind of public goods dilemma situations altruism versus reciprocity versus social image what we do is that we combine experimental and observational data from Wikipedia which is a real world public good in which extrinsic incentives, traditional trajectory incentives play no role in shaping individual behavior conclusion for all regular contributors what we learned so far so the non-admin guys reciprocity and social image but not altruism by the measures that we have appear as deep underlying social motives that predict the trajectory of Wikipedia users from being a non-contributor to being an engaged Wikipedia contributor in this process reciprocity and social image seem to be substituted rather than complement which means that both are at play but in different subsets of the population of contributors third, a taste for reciprocity does not continue to predict the trajectory of those Wikipedia users who become super contributors the so-called above median group that I told you about while a taste for social image does this is what we learned for regular contributors what about Wikipedia administrators? well, there is some evidence that a taste for social image continues to motivate their participation however, reciprocity preferences are consistently negatively associated with the extent of their participation in Wikipedia within this group which suggests some kind of a fixed-kin hypothesis and we try to test and get at that hypothesis a little bit more directly by leveraging the data that we have on trust towards strangers or trust experiments and see that it correlates negatively with the amount of policing edits or policing activities that Wikipedia administrators perform within Wikipedia that's basically the message that I wanted to convey to you today I look forward to talking some more hearing your comments, as I said this is still work in progress so if you have alternative explanations things you think I should check data that I should collect and so on and so forth very much looking forward to hearing about it this is collective work that I also did with Yochai who made it possible for me to do all those things I wouldn't be here today if he wasn't there so, yeah, thank you So as a possible alternative explanation for why we're supprosity we'll be less of a motivator for the administrators I wonder if you've considered that when you're in the administrator role you are going above and beyond what the typical community member is doing so you are not being reciprocated by the bulk of the people benefiting from your work, is that ruled out by the data? Is that something you thought about? I think it's not ruled out by the data and I think it's a perfectly reasonable interpretation of why we do not see an effect of reciprocity in this above-major group I think that seems perfectly reasonable you can think that, you know a reciprocity mechanism can lead you only so far but once you've already reached 2,000 contributions how fervor can it lead you, you know? So I think that's a pretty reasonable interpretation although I'm sure there are others Yeah Any... Your hand waits for... Just a curiosity question Before you do the analysis did you have any predictions about what you would find about what the relative roles of the three different motivations might be for the populations or was it really, let's see what we have when we do the analysis So that was pretty much the spirit The thing is that those theories have been rationalizing the fact that people often want to cooperate in the field and that we observe that in many real-world situations and we really wanted to run some kind of a horse race between those theories trying to see how well they map onto actual field behavior in a context that we care about here, Wikipedia, right? with the broad picture about pure production and things like that So we have no prior as to which one would work best although we have in the lab that pointed at the fact that social image and reciprocity are more effective at sustaining cooperation that altruism which doesn't necessarily mean that altruism doesn't have a role, you know? So, yeah I was in the overflow seating so forgive me if it's I couldn't quite hear it so maybe it's repeating what you asked but my question is about the interpretation of the results the language you've been using describes these as underlying motives that predict the results and levels of contribution but is it also possible that it's kind of the causation that's working the other way that participating more in Wikipedia is making people more concerned about social image or reciprocity up to a certain level that's a perfectly reasonable interpretation and when I talk about prediction here I'm not like ruling out the fact that those are only correlations so this could go the other way around but I talk about prediction here because what I'm really interested in is looking at whether those experimental measures that have been used extensively in the lab we have tons and tons of papers that use those experiments to try and tease out alternative explanations for social preferences whether we can take the result of those experiments to the field and whether they have some predictive power of what people actually do outside of the lab in that sense you can say that those experimental measures predict what people do in the real world so to speak then the question of whether preferences evolve as a result of your participation of whether you have a fixed preference parameter with which you were born predicts what's going to happen in your life afterwards I can't say and it's probably a little bit of both the goal here is really to map the experimental literature in the lab with what people actually do in the field in that sense you can talk about predictions I believe so I think one of your most fascinating findings is that it actually leads to less trustworthiness to strangers and you know how you were saying that you're not sure if there's a more positive and negative explanation for why that is where she looked at why articles are deleted for non-western events or topics and what she finds is that the notability criteria is that for people who are not close to the event they have a different kind of criteria for the notability criteria and so I'm wondering if you could one way to look at this another way to ask the question is do the post-social motivations change depending on one's embeddedness so you can operationalize embeddedness by geographic co-presidents or the cultural connection to the event but essentially she's finding that outsiders were competing against local knowledge and they were saying well your local knowledge is not valid and so we think it should be deleted and it's creating these tensions that you have these administrators are really in a very empowered position making these kind of decisions well thank you thank you very much for pointing that to me I would be definitely interested in having the reference of the paper you're talking about so I don't know about it so I'll be happy to have a look and see whether that can help me yeah thanks Thank you fast and slow it feels to me that there is an inherent bias of those administrators that they're not seeing of 100% of contributions to the 0.10% of irritating jerks and so that becomes a large piece of their mindset and I would say at Citikar the people who are doing the fleet work are the people that users are mostly jerks because they're out there fiddling with people who do jerky things they don't see you don't see they're not seeing it what percentage of action it is they're focused on that bad behavior for me it feels very natural that if I spend all of my time looking at bad behaviors I bet you would find that in police departments that we would find that in correctional counselors teachers who are who yell at students who do bad things and if you're just focused on the bad subset that's what you think well your point about police departments and stuff like that there are suddenly experiments that I'd like to run there again what's the real percentage of disruptive users that you have out there I showed you the numbers the number of people who are actually willing to cooperate and sustain their contributions in that space is quantitatively large relatively really low you have a lot of disruptive users out there so I'm not sure what kind of interpretation we can push based on this data that was the meaning of my message it could very well be that you know you get a lot of disruptive users out there and you need people who are in charge and able to deal with them and this is a good thing if those guys were on the round the whole project may collapse and Wikipedia is as a matter of fact a very successful project however newbie frustration this whole don't bite the newbie kind of stuff this can also align with that kind of interpretation which is the one that I see you pushing a little bit more again I'm not sure which way it goes and I don't have the data to answer that question there's a similar finding about administrators or managers of projects they didn't interact with a lot of people and they didn't have many contributions compared to other active users so I was wondering how maybe that's a question how the organization was organized and how the labor was divided so it seems like a heritage that as a manager or administrator you don't need to interact with a lot of people but you interact with level down people a quite small number of contributors and my question is you talk about trust correlated with administrators of work the game is about I give you money but money is based on the trust of your capacity of your skills or actually your ability of utilizing money to hire other people you know I don't know to buy labor or to do other things I'm not sure how the trust what's the base of the trust what's the base of the trust so as I understand it as I understand it you're asking me whether by using an experiment which involves money I can really get at the trust preference that does not necessarily involve money is that the question you were asking I didn't mean to ask you a different question but I was trying to interpret why administrators would have this kind of relationship with trust so could I be actually the trust for all the information and for other factors that we haven't thought about that's just a thought that's a deep question I'm not sure how to respond to that we'll think about it we can talk about it afterwards if you like I'm holding on I'm probably not even holding on by the skin of my teeth to the argument I'm not a stack up so this is very naive so if I wanted to take if I'm a journalist I'm going to take the wrong conclusion from your report the conclusion I'm going to draw and I know it's the wrong one is oh Wikipedia administrators are not motivated by altruism that's the wrong conclusion regular contributors either right like my head is everybody seems to be focusing on the Wikipedia administrators are not motivated by altruism what that actually means is there's no correlation between the sort of crazy standard test for altruism which tests altruism in a highly artificial environment and so the conclusion I might take from this especially if I had self-reported data I would have asked the administrators why are you doing this and they were reporting well it's making the world better and do you trust people oh yeah I trust a lot I don't trust the bastards who are screwing up with the Wikipedia but I love the other administrators and the other contributors and all the new users we're a very trusting environment it's just one set of bastards is the right conclusion from this to well the test very deep and difficult human motivation such as altruism and second of all is there a method maybe through self-report to see whether in fact to get back to Justin's very first question is there a real correlation are the self-reports would the administrators say no we're not doing this for altruism I sort of guess otherwise your first point what I understand that you're saying is well you know you're running that weird dictator game altruism experiment trying to make an argument that this captures something about altruism and you say well based on this measure I don't see a correlation of number of contribution therefore altruism does not play a role however you could take the question the other way around and say well if it doesn't predict field behavior it's because the measure is bad and this is a point well taken and I think this could perfectly be the case that's why I said throughout the presentation buy those measures but really the approach that I'm trying to take here is a cumulative scientific knowledge approach so those are measures that have been used to study altruism the experimental literature for 30 years right so in a sense in trying to build the castle of scientific knowledge you want to see how much of that tells you about what happens in the real world in a sense it's still relevant to use those as measures of altruism and talk to the to the crowd of economists and see what we can learn from that but yeah presumably we may come up with better measures of altruism that could explain what happens in that space and so you know if you frame it to an economic theory audience you would frame it as I framed it then your comment is perfectly well taken so in Wikipedia I'm just not capturing it with those measures regarding the second point self-report versus experiments I would make the you know traditional answers that economists especially experimental economists make to those kind of comments you know economists in general are very dubious of self-reports economists are not so interested about what you tell me what you do but what about you actually do when I see you doing it and so this is a well the fact that they spend their time and that by product of that they make the world better doesn't mean that their deep inner motivation is to make the world better and you could tell me all sorts of things about your motivations to do many things in life that you just want me to hear which is why I worry a little bit about self-reports if I ask you maybe do you work to make the world better of course you're going to tell me that you work to make the world better right so in a sense that's the question that doesn't mean that self-reports aren't useful that means that you can try and look at what people actually do in very simple frames like this and if that predicts what people do in context that means that you have some identified at least in my mind something deep that's going on here if people's reciprocity behavior in such a simple game can predict the number of contributions you make to Wikipedia I would have an argument that reciprocity has a role to play if it doesn't work that doesn't mean that altruism does not have a role to play you know so you had a question which is altruism is used often to refer to what in economics would be described more generally as pro-social motivations that is to say things other than what's in it for me once you're actually trying to study at a much finer grain in order to design interventions you need to be much more precise in the subsets of motivations that go into this packet that in a background piece you might call altruism generally as compared to selfishness altruism in this case is the subset of things that is entirely unmotivated unconditional on the conditions and I think it's just here and that's a very relatively that's a subset of the population most of the experiments it's about 20% of people who have that but there's a much broader set of pro-social motivations and so part of it is just translation between how we use altruism generally and what we use it for and so I would add to that on top of the kind of what economists call the demand effect if I ask you why you do something you're going to be willing to give me the answer that I want to hear which is one problem with that the other problem with that is that you may not be aware yourself of why you're doing things which is why simply looking at your behavior in very simple scenarios like this can help us understand and unpack what your motivations are but where does power and control fit into that I mean I would say that administrators some of them are controlled things I actually know one small one but I couldn't stick to that but aren't you being something very important though? so I will not endorse that statement officially even unofficially by the way what about in terms of the theory those are the theory that we have around for quite a long period of time control is not a pro-social motivation as I defined it right? I was mainly interested in the motives that could push you to contribute in those kind of dilemmas that are social innate that means you take into account the utility of others in calculating your own and those are the things that I'm trying to tease apart here but I also think that people contribute to Wikipedia because it's fun and that has nothing to do with social motivations but this is not something that has been theorized explicitly so I want to start with the theories that we have and that have been around for a long time and trying to tease them out this is awesome here are two things that I tripped up on in May I sort of relate to some of these issues about what our measures mean and what makes me as reasonably likely is that you use the number of words that someone writes on a user page as a proxy for their sort of social desirability in a context where the activity that people are doing is writing words on a page and so it seems like in lots of contexts you could say people could be writing user pages for lots of reasons one of those deep underlying things because people like to write in any context and the particularly confounding part here is you're basically measuring people's predilection to writing in one setting based on their predilection to writing in another setting but attributing their motivations in the first setting to social utility as opposed to just enjoying writing or something like that it's particularly confounded by the fact that they're doing the same kind of thing in both places so I don't know if that's of interest or comment my second piece is I couldn't quite get why you were saying that reciprocity predicts people moving from non-contributors to median contributors in the sense that it seems like it moves them from minimal contributors to median contributors it seems like the flip from non-contributor to contributor if you want to measure that sort of like important kind of zero to one moment you somehow have to get the people who are like literally not contributors if I've signed up for an account page I'm sort of already even if I've never made a contribution I'm already sort of a non-contributor I mean it seems like it's not quite evidence of flipping from zero to one it's evidence of going from 0.0001 to 0.5 which is important we do have registered users in the sample who made zero contributions right in that sense you're really identifying econometrically speaking what makes you move from zero to something you can explain some of that something but up to a certain point it doesn't explain anymore among registered non-contributors then another way to go about doing that is you could simply study the decision to contribute versus not which is another way to go but yeah I see this as being a matter of terminology or do you disagree there is something deeper or the econometric model that I put I guess like so one of the things you'd want to do the econometric model is then start saying okay so how do we start applying this like so how can we I guess I don't know what you want to do you could just be fascinated by it to understand human psychology what economists do is say okay one of the levers that we can start yanking on we're running out of contributors to get more contributors to get more people to stay so from that point of view saying that reciprocity motivates non-contributors to become contributors to register non-contributors I mean even the notion of a right there really are a lot of registered non-contributors like I make a Wikipedia page and then literally do nothing as I said during the presentation I showed some numbers you have 200,000 people who register Wikipedia account each month only 10% of those then make some contributions make 10 or more you said 10 or more but 2% of those 10% make 10 or more contributions within the following year so you know the people who create the public good value of Wikipedia are really those users who sustain their contributions over time so if you're willing to think about policy interventions you can think about increasing the degree of communication and connectedness between editors you know there have been some interventions in Wikipedia around those lines like the thank button easily the welcome Wikipedia group which welcome new users and things like that those are all things that align with that reciprocity kind of motive yeah so that would be I could put up half of your question maybe we can talk about it maybe one last question I really love this, thank you I know that there's a lot of talk in technology platform design of creating designs that are influenced by or even like take into account like experiments of various kinds and it seems to me like studies like this or the stuff coming out of the Facebook data science team work really well for well established platforms with large numbers of users what is your sense of the scope of possibility for the kind of methods that you apply is it only applicable to like the top 100 most traffic websites in the world or is this the kind of thing that people who are creating, evolving new platforms can be thinking about as well so in terms of methodology this is the kind of thing all the platforms of pure production of the world this is how I think we would build real strong cumulative knowledge because this is obviously one particular context, one particular platform which is really successful already established and so on and so forth so we really need more experiments of that kind and we're working on one follow up with Yochai with over a thousand open source software contributors at Sourceforge which allows to study explicitly how those pro-social motivations interact with monetary incentives because about half of the contributors open source software is a well known fact are being paid to contribute so how do those motivations actually play out with one another is the next step but there are many other platforms to which we should apply this kind of methodology to try and have some some deep conclusions about it, scientifically speaking as far as the policy side is concerned I would like to believe that you can generalize some of this stuff to many other settings again empirically speaking you know I'm trained as an economist I don't have the data to answer that question I don't know well thank you very much everybody for being here


Okay, let me put this turn on. Okay, so thank you to be here. We are here to talk about the 10 years of Wispencil, the past, the present of the future of Wispencil. Before that, a little presentation about me and the Wispencil team. My name is Daniel Martin Atlara and my three passions are the training. I have my own animation school in Barcelona teaching 3D animation. I make consulting mostly for Wispencil productions and make online videos. And I like a lot to make my own animation shots like alike. Okay, and the hero, that was the first open movie made in Wispencil. And I think it's the first open movie made outside of the Blender Studio. It was made in Barcelona. Okay, and I love design animation tools. I'm not a programmer, I work with programmers, but I love thinking about the tools. One of them is Wispencil. Okay, and another one is the Blender Climation. I made some tests in the past years, but in the next year I want to work in this project more. Okay, it's just to introduce inside Blender all the stop motion workflow inside Blender. It's the same thing with Wispencil, but for stop motion. Okay, what about the team? The Christian Siltin, of course, the one and only Antonio Vázquez is the main developer of Wispencil. Okay, this is a true gift for the project. Without him, the Christian Siltin project wouldn't be the same, for sure. Okay, and in the artist's part, there is the great Matias Mendiola, okay? For Ramitina, and not so great, but handsome myself. Okay, and of course, a lot of contributors. It's really nice to see that a lot of developers want to improve the tool working in his studio or in his home. Okay, it's really nice to see all these developers contributing in Wispencil project. Okay, so, Wispencil. Okay, five years ago, in the Blender conference, I made a talk about the presentation, the hero project. Okay, the hero project, and that time is in the production stage. Okay, but one of the goals of the hero of a movie is to put the Wispencil in a production level, okay? But for us, it's not the main goal. The main goal is, the real goal, is to say hello and welcome to the new kind of Blender user. Okay, let me put a short clip, play. But for me, the key is that introduce to Blender a new kind of user that truly to the animators to make, to the production. For me, it's just, the point is, it's the key, right? Okay, I really agree with this handsome guy, okay? It's like a metaverse thing inside the conference, okay? But this is the point, okay? This is the point. It's not about create two animation tools. It's to introduce a new kind of user inside Blender. This is important, because with another way of thinking, okay? And introduce all these kind of workflows inside Blender in the Blender Wave. It's an important thing, okay? It's not just about the tool, it's about the users, okay? So, five years after this talk, what is the state of the adoption of Gris Pencil in the industry and in the artists, okay? Before that, let me put a reel, okay? This is a selection, it's a three minutes reel with a selection of Gris Pencil works, okay? Oh, the sound, oh, the sound, because this is epic music here. I need sound, I compose this music for one, for one week, I need the sound, okay? Let me, sorry, the sound of the, the sound of the video. Just two weeks composing this music, you know?  This is the whole point of the presentation, okay? With the music, this, I can put my mobile, okay? I have mobile with this fault, okay, maybe it's enough for you, for the first. Yeah, sure, but for me, the key is that introduced to blended a new kind of user that truly to the animators to make, to the production. For me, it's just, okay, sorry, sorry, what? No, no, no, it's composed by me, okay? But let me try this, let me try this. Okay, I have this, I have this here, out, the music is, I think, this. There's the microphone here, this sound, where's my mic? Here, okay, so let's try, come on, one, two, no. Again, at rest, now, it's really rewarding to see all this kind of works, okay? And this works is the last three years or something like that, okay? So it's so rewarding for us to see all this kind of quality animation made in Respencil. So right now, there's two feature films made with Respencil. The first one is I Lost My Body was made some years ago and with Oscar nomination, okay? And was made with the Respencil 1.0, sorry for that, okay? Because this was an early version, okay? And the last one is Onicon Wars, okay? This is a really great movie, so intense, twisted story, okay?  And there's four movies in the news for years, okay? Spy Studios from Klaus are introduced with Respencil inside his pipeline. There is another movie, this Mariposa's Negras, and another two films, with no detail by the moment, okay? It's not secret, it's just, it's not public yet, okay? And if you know another production, maybe in France, okay? With Respencil, please let me know, okay? And of course, we are, we discover every month, a new studio, a new artist using Respencil for storyboarding and animatics in studios around the world, okay? To make, making shows for Netflix or Warner and so on. It's so interesting that a lot of people using Respencil for storyboarding, okay? In fact, in the next version of the Blender, we introduce a new add-on, story pencil add-on, to make this workflow for storyboarding and animatics. And you, and if you are interested in training for storyboarding, Mattias Mendiola is your guide, is a guide that makes training for Respencil and storyboarding, okay? So, looking back again, three years ago, this was the, this was the, the roadmap for Respencil. Respencil in VR, DRAW Management improved the drawing experience versus mode, storyboarding tools, and so on, okay? And actually, right now, we have all this done, okay? And what about the EV integration will be coming soon with EV Next? That means that, that Respencil combine in a better way with EV, so you can drop shadows with, with Respencil objects, use note materials, or for example, depot fill, all this kind of thing. This is the first implementation, okay? But this is the way. Respencil will be a real object inside Blender, okay? And what about the VR thing? For me, the Respencil in VR is a personal project, okay? Because I think, no, I know that VR for some tasks are more easy. It's more effective, more creative for some, for some tasks. And when you join, when you work in together with Respencil, I think it's a win-win, okay? I made this, this is half sound, okay? So, I made this prototype, this proof of concept, okay? A few years ago with a developer to show how it works with Respencil with VR, okay? But it's just a prototype, okay? And the idea is for the, the idea is great. We are getting a new prototype, a quick and easy prototype, and we are looking for a developer, okay? To make this thing possible. Because right now, Blender now have a better support for VR right now. And we have Respencil, the only thing that we need is that when you press the control, start to draw, okay? It's this, that gap, okay? And we are looking for a developer to create this quick and easy. It's not a final, it's not a final thing, it's just a quick prototype. And we have a budget for that, okay? So, thank you to, to NextLab. This is our organization format, that put the money to create this prototype in two, three months, okay? If you are interested, please send me a line, okay? Because this is the, the next step, okay? During this time, was more, more, more develop, more development, like official Respencil add-on by Somoil, stroking, interpolation, new modifiers, video support, film improvement, line art, and so on, okay? And the next Respencil developments will be the Respencil integrated in EV, polishing the story, the story pencil add-on, asset manager, okay, for Respencil, okay, specific asset manager for Respencil and make this dirty prototype, okay, of Respencil in VR. So, let me show you a quick video with sound, okay? This is the interpretation tools, okay?  But you can create interpolation or new modifiers like an envelope, okay, to create new styles, okay? Or asset manager, for Respencil, I think this will be huge for production, okay? Have all these assets, for example, for cutout animation, okay, film improvements, okay? Or, for example, some performance improvements, okay? Okay, but the real point of this presentation, the real feature of Respencil is that, okay? Is the Respencil plus geometry nodes, okay? It's, yeah, curl your enthusiasm, okay? Because this is a huge project, this is a huge project. It's not like, okay, combine all two things. This need a complete rewrite of Respencil internally to make compatible with the new core system, rewrite all the modifiers, it's a huge, huge task.  And the, okay, and future Respencil 3.0, 3.3.0, okay? And you don't spend much time in Respencil 2.0, okay? Because you are waiting for this project, okay? Like, but you don't have a real roadmap, you don't have the time, you don't have the resources and the support, okay, to make this happen, okay? It's a tricky moment in the Respencil development, okay? And we are working to see that this is possible, okay? So, fingers crossed, okay? So, but imagine, imagine that this will happen, okay? For example, this is a couple of geometry notes, okay? But imagine you can create, oh, sorry. You can create like these custom brushes inside Respencil, okay, fully procedural. For example, you draw and you have some fireworks, okay? Or you paint with a red brass as you see the drop leaking, okay, things, okay? Inside Blender with a lot, this is, this has a lot of potential inside Respencil. You can create your custom brushes and share with the community and join all the benefits of the Respencil and the geometry notes, okay? Or, for example, this one, imagine, okay? Imagine this, for example, like with a lovely Ghibli style, okay, this river, okay, with the flowers, the grass, the water, okay, in a fully Respencil, okay, thing? You can create this asset for the production that I think is totally, is the future for Respencil, because for tourism, one is to have a modern architecture inside, okay, and make compatible with geometry notes, okay? So basically, that's it, okay? This is the future of Respencil and again, cross it, okay? So, I hope, I hope, I want to believe, okay? And basically, that's it. Remember, okay, in blender.org-slash-respencil is the official page to Respencil. Here you have a lot of tutorials, more like a near 100 blend files ready to the lot and learn by example quality, quality blend files, okay? Tutorials and so rooms is the main page for Respencil, okay? And don't forget to use Respencil, hashtag in your social media, because it's more easy for us, okay, find all the studios and the artists that are really using the Respencil, okay? So, that's it, bye.