Good morning everybody, my name is Mike Hanley, I'm the head of digital communications here at the World Economic Forum and I'm really, really excited to introduce Molly Crockett, an associate professor at the University of Oxford in Psychology and Phillip Tetlock from the University of Orson and the author of the book Super Forecasters and we're here to talk about forecasting failure because we're at the beginning of 2017, last year I think was a watershed year for polling and forecasting and seeing into the future. I'm going to first ask Phillip to kind of give us an overview of some of the insights that came out of what's seen as a great forecasting failure, the victory in the US election of President Donald Trump. What did the forecasters get wrong? A useful place to start is by dissecting the concept of forecasting failure. Most people when they make forecasts, even when they say seemingly decisive sounding things like Trump doesn't have a chance or there's no way Brexit's going to pass, don't typically mean there's a probability of zero or a probability of one. They typically mean that it's extremely probable or it's extremely improbable. In forecasting you're only wrong, decisively wrong, logically decisively wrong when you say probability of 1.0 and it fails to happen or a probability of zero and it does happen. Now I think there were some people at Davos in January 2016 according to Bloomberg who came pretty close to saying things like that. So they should presumably take somewhat of a credibility hit, at least on that dimension of forecasting. One of the very best poll aggregators in the US is of course Nate Silver and his 538 site. A couple of days before the election he was putting a probability of about 70% on Hillary Clinton being the next president of the United States. Some other poll aggregators like Sam Wang at Princeton were up around 99%. They had some pretty good statistics. The aggregators had some pretty good reasons for believing that Hillary would indeed win. Hillary did indeed win the popular vote by a significant margin. She lost the electoral college by a significant margin as well. Now if I were to say to you, Nate Silver is putting a 70% probability on Hillary victory. He was one of the most accurate of the poll aggregators. There were a few forecasters who put probabilities of a Trump victory close to 50%. But there were very few. And the question is how wrong was Nate, should we count that as a forecasting failure? When Nate Silver said 70%, a few days before the election, was he wrong? Now that turns out to be a very difficult question to answer. Nate Silver has been making hundreds, if not thousands of political forecasts over the last several years. And we know that Nate Silver is fairly well calibrated. His system, his methodology is fairly well calibrated. We also know that prediction markets are fairly well calibrated. And what does that mean? It means when they say, when you look at all the times they say something is 70% likely, those things happen about 70% of the time. That means you're welcome. That means you're welcome. That means 30% of the time those things don't happen. In my work with the US intelligence community, in which we study lots of forecasters, the very best forecasters, we call them super forecasters, are wrong a lot. We live in a world where there's a lot of irreducible uncertainty. And the very best forecasting systems are the systems that are virtually perfectly calibrated, like when they say 70%, things happen 70% of the time, when they say 90%, they happen 90% of the time. Systems that are perfectly well calibrated are going to look, there are going to be conspicuous cases in which they look wrong. And if you throw out a forecaster or forecasting system, every time it's on the wrong side of maybe, you're never going to have a well calibrated forecasting system because it is in the nature of the political world that there is irreducible uncertainty. Now, I watched Nate Silver trying to explain to one of these comedy show hosts after the election why the 70% probability might not have been wrong, and he was mercilessly ridiculed. People don't get it. People don't think that way. But that is the current state of the art. If Nate Silver wasn't wrong, were there other forecasters who you can say were wrong? Well, I think the interesting question is how much of a hit should a forecaster or forecasting system take? How much of a credibility hit should it take? When it says there's a 99% likelihood of something happening, it doesn't happen. Obviously, there's somewhat of a hit they should take. How much of a hit is going to hinge on how extensive a forecasting track record they have? If there's an extremely well calibrated system, the hit's going to be much smaller. If that's the only forecast you've ever seen them make, you probably aren't likely to ever believe them again. So if I'm looking at the forecast and the forecast says 70% and I go to sleep feeling quite comfortable, is that my emotional state that's interpreting that number? You really shouldn't go to sleep feeling quite comfortable.  Of course not. I mean, 30% chance is a non negligible probability of a Trump victory. A lot of commentators said that the shock and awe that was the result of both the Brexit referendum and Trump's victory was a result of the emotional, well, first of all, the driving force of the victories in those cases and the shock of the losers was down to emotional responses. Would you agree with that? The commentation over the top of the polls? I think the election was so close. The polls were so close. And the possibility of correlated measurement error causing a cascade in the battleground states like Pennsylvania, Wisconsin, Wisconsin and Michigan, that both sides thought there was a non negligible probability of losing. Trump apparently told his family the night of the election prepare for a rough night. Paul Ryan apparently told Chuck Schumer, you know, look forward to working with you as majority leader of the Senate. I think both sides had some expectations. I think the Trump side was probably the Republicans were more likely to expect to be losing because the data were were in that direction. I think the Democrats were more bitterly disappointed because they quite rationally had a higher probability of winning. So I think things unfolded in pretty much the way psychologists might expect. That's right. Molly, of course, you're an expert in moral decision making. And a lot of the commentary that followed both of those votes was around the emotional drivers of the decision makers of the electorate.  I think what was very clear to me in the aftermath of both Brexit and Trump is just how powerful the motive to express one's moral views to assert one's identity is for behavior. And in many cases outweighs what we might call economic self interest. I think a lot of predictions do focus on economic self interest because it's easier to measure and it's much more challenging to try and quantify and model moral emotions, moral outrage. But as is becoming very clear, these emotions are incredibly powerful motivators, particularly in cases where where people may feel that their voices are not being heard and may use their vote as as an expression of those emotions. And what are some of the ways that psychologists are trying to help economists perhaps or forecasters integrate some of these ideas into their work? We're doing a lot in economics and psychology to try and better get a grip on how to quantify and and and build models around these emotions. And one way that that we're doing this is is is taking a view that decisions don't take place in a vacuum and they don't take place in isolation. They take place within the context of social relationships. And when people are making decisions, they're thinking not just about their own preferences, but the preferences of those around them, their friends, their family, and how their own decisions are going to reflect on their values and their embeddedness in these social relationships. And there's a lot of research going on in this area that I'm hopeful that that will become better at predicting these things in the future. And how what are some of the ways that those ideas are getting integrated into the science of forecasting or polling? Well, I mean, I'm not quite sure. But but one thing that seems to me like perhaps good direction is to recognize that particularly around issues that are contentious or controversial, we should consider the nature of the social relationship between a polar or a journalist and the person who's being asked to express their view or make a prediction. Because people care very much about their social image and their responses are going to reflect not just their true preferences, but also their concern for how they look in front of the journalist or whoever's asking them the question. There was a lot of talk about the hidden Trump voter, the hidden lever who wouldn't declare their allegiance because they would signal to the journalist or to the poll star a particular type of political orientation. Right. Was there some evidence of the hidden Trump voter, Philip? A little bit. The political science polling literature that was something called the Bradley effect and had to really pertain to a former candidate, African American candidate for Governor California and the under the overestimation of his polling support when he lost. And there have been a lot of studies of the Bradley effect attempting to quantify how big a bias that is. It proves to be a pretty elusive and small effect, I think it's fair to say. And I think the Trump effect, I think there probably was a small fact and that would have been quite sufficient to produce the outcome we're talking about today because a small systematic bias in the polls in the critical battleground states could easily have produced the quite decisive electoral vote victory that Trump had despite the fact he had a popular vote defeat. So Malia, can you think about some ways that pollsters might ask different questions that that might get to the source of the truth better? Yeah and this is this is an issue that we deal with a lot in the study of moral decision making because when we when we do research in this area clearly just asking people how good of a person are you? How moral are you? Is not going to necessarily yield an accurate response because as I mentioned earlier people care about their image and they're they're motivated to project perhaps a better view of themselves than then maybe the case. So one potentially fruitful avenue would be instead of asking people what their own preferences are to ask them their perception of the views of those around them in their community. This lets people off the hook off the hook because Billy or vote for Brian, I would never vote for Brian. Exactly it lets people off the hook they don't have to commit to expressing a view that may be controversial but they can share their knowledge of what others around them believe. This will get at preferences in a couple of ways. One we know that people are more likely to believe in outcomes that they want to see happen. They're optimistic so looking at what people's beliefs are can give clues into what their own preferences are. We also know that people project their own beliefs onto others so if I believe X to be true I'm more likely to believe that others will also believe X to be true. So I think maybe by asking people not just what their own views are but what their perceptions of those around them are could help us build better models of collective decision making. Philippa, how is forecasting changing and will 2016 what will the impact of 2016 be on the science of forecasting? I think there's a slow movement toward greater accountability and greater transparency in forecasting. I think people of growing elites are growing even elites are growing somewhat weary of vague verbiage forecasting in which nobody can really figure out what somebody is saying. If I say there's a distinct possibility Putin's next move is going to be in Estonia. That's a very safe sort of prediction for me to make because if Putin moves into Estonia I can say I told you distinct possibility and if he doesn't move into Estonia I can just shrug and say I just said it was possible. I think there's a growing awareness that the vague verbiage in which we express most expectations about political events today makes it virtually impossible to assess who is closer to or further away from being accurate. I think that's one of the reasons why and that's a very comfortable position to be in. It keeps you politically safe but I think there is a growing pressure inside many organizations and finance intelligence analysis and elsewhere to create systems that allow you to keep track records and it's only if you have a track record that you can make meaningful claims about forecasting failure. I think because a well calibrated system that says 70% is going to be around 30% of the time if you focus on those individual cases you're going to get a really skewed misleading picture of forecasting. So you need to create systems like forecasting tournaments from monitoring accuracy over the long term. I think that the World Economic Forum has a collaborative project with an organization I'm affiliated with, Good Judgment Project, in which they're launching a forecasting tournament. I think the Arab Strategy Forum which I think has some connections to WEF is also doing that. I know the U.S. Intelligence Community is doing versions of this. So a number of organizations are moving in this direction. I think it's a more evidence-based approach to forecasting and I think the world will be better off having greater transparency in the process. I think right now we're sort of groping in the dark and the events of 2016 sort of underscore that. Indeed. We're coming back to this emotional question. There's a lot of talk, you mentioned the elites and then there's the opposite word to that, these days seems to be the anger of those who are not the elites. There was a lot of talk about the anger of the electorates last year. As a psychologist, Molly, what is it that makes people angry and makes populations angry and how can leaders and decision makers factor those emotions into their decision-making and actions? One of the biggest drivers of anger is inequality and the unfortunate reality is that there's tremendous inequality in the world today. I read yesterday that something like the top eight richest people in the world own more than the bottom 50% report recently out of Oxfam and not only is there a tremendous amount of inequality in the world today, but that inequality is so visible because everyone in the U.S. has access to the Internet, Instagram, social media. The massive scale of inequality is more evident than ever before. And decades of research in psychology have shown that when people are confronted with inequality, when they're on the losing end of a bad deal, they will often behave destructively and they'll do that even if the destruction hurts them as well. So a lot of people would rather burn it all down. They would rather have nothing themselves if it means they can just eliminate the inequality and sort of level the playing field so that everyone is on a lower level. Right. So there's inequality, other kind of other other things that make people feeling disrespected, feeling that their views are not are not taken into account and feeling that those in power are contemptuous towards them, feeling ridiculed. And I think that there there has been a lot of that in politics. Can you are there are there ways that forecasters can take into account this emotional aspect? Are there are there is there research going into kind of measuring populations, moods and helping decision makers move things forward because of that? I was just thinking about Molly's answer. And I'm wondering whether inequality is the driver of the events of Brexit and Trump? Or is it the culture wars, immigration, things of that sort? And if I had to bet on what the biggest driver was, it would be more culture and immigration and symbolic identity issues than economic inequality per se. I mean, keep in mind that Trump won the macro economic models of presidential elections, we're viewing it as close to a toss up. It wasn't clear. But the economy has relative to what the economy was in 2009. I mean, the Democrats had a pretty good track record of bringing the economy back. The unemployment was pretty low in the United States. It's a 5th range. Now the percentage of the workforce that's actually potential workforce that's working is not this is far from an all time high. But there's there's the definite feeling that that that those in power are reaping the greater greater proportion of rewards. And that there's that is an underlying theme. And as you mentioned, the culture wars immigration, certainly some of some of President Trump's electoral themes were around that. Certainly the Brexit vote seem to seem to turn on that kind of on that kind of emotion. I see these I see these issues as really intertwined. I think that that the culture wars are using inequality as a tool to stoke outrage. And the issue of immigration also relates, I think, quite strongly to inequality. One of one of the the more revealing accounts that I read after the election was a young woman who had voted for Trump. And she she was very angry because she's working three jobs and barely getting by barely able to feed her kids and and and she sees immigrants coming in and receiving government assistance that she's not qualified for because she's just over the over the threshold. And again, people are looking around them, they're comparing themselves to others, not just two elites, but those in their immediate vicinity and status. And yeah, and these are these are really primal emotions. They they tap into very ancient circuits in our brain. There's there's evidence that concerns about inequality are are are present to some extent, perhaps in our primate ancestors and and and I just see I see the the the anger around immigration and the issues in the culture wars as being intimately related from inequality related to inequality, not not the same, but but I think that they're they're all wrapped up together. Certainly. Here in Davos, of course, we gather together leaders from all walks of life and there have been branded the elites and been accused of living in a bubble, partly because of the emotional surprise or response to the events of last year. There one commentator has been widely quoted as saying that the Davos consensus is always very important. So I guess my question to you is it to the both of you is how can, you know, groups of people groups of decision makers be more aware of or make themselves more aware of the emotions and the the forces that are driving. If the forecasting, if their emotional responses to forecasting, everybody sees what they want to see in the forecast. How can they avoid those, avoid those, the frames in which they view things. If a forecaster gives me a forecast, how can I apply a an objective screen to it rather than a subjective screen? Well, I would say something to reassure people at Davos that Ken Rogoff, I was joking, obviously, but the notion that Davos, my man, is that Sam Huntington phrase, the Davos man is always wrong, that's actually not true. It's very, very difficult to do worse than chance. Whether the Davos man is more accurate in the dart throwing chimpanzee is another question, but always wrong, that's not true. I say that the best predictor of tomorrow's weather is today's weather. That actually, experts are extremely hard pressed to beat a simple extrapolation algorithm that predicts continuation of more of the same. That's true. The dart throwing chimpanzee experts can beat a little bit, but predicting a simple extrapolation algorithm is extremely difficult. Which is where you get the surprise around Brexit and Trump because it's not more of the same. Predicting change is the great challenge, whether it's central bankers or intelligence providers, but it's not the same thing that most systems fail. Of course, that's what we're most interested in predicting is change. Hence the disappointment in forecasting. I might just ask you for a last comment. They're only on how decision makers can be more aware, how they can make themselves more aware, whether it's the employees in their company or family, how can they make themselves more aware of the emotional state of those constituencies? That's a great question. I think I would just finish by pointing out that we know from research in neuroscience that being in a heightened emotional state, being highly aroused, being stressed, directly impacts the brain systems involved in predicting outcomes and in making decisions, and in many cases, kind of throws a wrench into things. Increases uncertainty, increases volatility and noise in the system. So I guess to the extent that people can cultivate a sense of calm, a sense of maybe caring and socially aware calm, the better decisions that they'll make. Philip, do you see the reputation of pollsters and pollasters reviving over the course of the next year or two? I think that the future, I will make a bold forecast about forecasting. I think that the trend will be toward increasing transparency and rigor in monitoring forecasting. And that'll be done through a combination of forecasting tournaments and prediction markets. And once we have a plan, pollsters will be less likely to misinterpret particular outcomes like Brexit or Trump. Very good. If there are no pressing questions, are there any pressing questions from the floor? Isn't the biggest mistake we're making is that we're unwilling to admit that some things are just not predictable. In other words, we're taking Trump and we're taking Brexit because it's one event and you just have to guess yes or no. But other things like will a war occur, will peace process succeed and so on. Aren't there too many factors? It's really impossible to predict and it's kind of pathetic that we're even thinking that it is possible to predict. Well, that now seem to have taken a pretty strong position on the futility of forecasting. I would argue that you can't live with it, you can't live without it. All forms of policy planning assume forecasting. Anyone who has a policy preference on anything is making an implicit forecast, an implicit conditional forecast. So you're not going to get away from forecasting. The question is how explicitly are you going to do it and are you going to try to get as much juice out of the system as you can. 10 years out virtually nobody does appreciably better than chance. Predicting things within a narrower time frame, well-specified outcomes, you can achieve increments in forecasting accuracy. They're not huge, but they're palpable and a 10 or 20% increment in accuracy matters a lot. Within a three month to 18 month range which is I think the sweet spot zone for improving forecasting. A quick one. I read last week a French newspaper was going to do away with opinion polls and run out to the French election. Do you think this is a good idea? Very French. Every set. I think we're close. Thank you very much.


The Harry Denman Evangelism Award recognizes clergy, youth, and lay persons for outstanding service in evangelism. Those eligible for this prestigious award practice the Great Commission and faithfully carry out the mission of making disciples of Jesus Christ for the transformation of the world. The award is named for the late Dr. Harry Denman Distinguished Evangelist who Billy Graham called My Mentor in Evangelism. The award was made possible by the Foundation for Evangelism which was founded in 1949 by Dr. Denman who felt it was the business of every Christian to be an evangelist. The Foundation's mission is to promote, encourage, and provide resources for responsible evangelism enabling the United Methodist Church to bring people into a personal relationship with Jesus Christ and help them grow as His disciples. Multiple nominations were submitted. The winner of the Youth Award was selected by the Conference Council of Youth Ministries and the winner of the Lay Award was selected by the Conference Board of Lady. The Denman Award for Adult Lady goes to Billie Jean Baker. Ms. Baker is a member of Oakland United Methodist Church in Dallas and participates in ministries at Union Coffeehouse as well. Despite hardships in her own life, Ms. Baker's passion is sharing the love of Christ with other folks living on the margins and advocating for them when their voices are not heard. Ms. Baker once lived unsheltered on the property of Oakland but after joining the church, she quickly became an evangelist and brought many unsheltered neighbors to church with her. One cold night, she asked if she and her unsheltered friends could sleep in the church building because temperatures were going to dip below freezing. That idea launched a commitment on the part of Oakland UMC to become a place of refuge during inclement weather. As a result, the church has welcomed a long list of unsheltered neighbors into worship as well as membership, many of whom made professions of faith and were baptized. Ms. Baker also has become very involved in Union's ministries on the campus of Oakland. During the coronavirus pandemic, she's been participating in Union's weekly worship gatherings. But she does so outside, out of doors, connected to the service by a speaker so that neighbors may listen with her. Ms. Baker is always an advocate for others, especially persons of color and people who identify on the LGBTQ spectrum. Her advocacy for unsheltered neighbors and her joy as an evangelist has, without question, saved lives and mobilized two congregations to be better followers of Jesus Christ. It is our great joy to celebrate Billie Jean Baker as she receives this award. I help people because it's the right thing to do. I've had a lot of help over the years myself and there's a lot of people that are just like me that don't know how to ask for help. So if I go and offer it to them, then they're more comfortable and they're able to get the help they need. And the reason why I do it is because I know it's the next right thing to do. And that's what God tells us all to do is to do the next right thing and that makes him happy. So in the end, it'll make me happy. Billie Jean finds ways to take care of people's spiritual and physical needs. And that's what it means to be a Methodist. I came to the church here and I was sleeping on the property and I thought well if I'm gonna sleep on the church property, I like the church service anyways. I ought to attend church. And for some reason the rest of the homeless community wanted to follow my lead and follow me through the doors. It was a Sunday morning in late December that Billie Jean came up to me between worship services and said pastor, I want to talk to you about maybe the possibility that I could come inside tonight. You know, I kept blankets and clean clothes and stuff. So I stayed warm but there was a lot of homeless out here on the streets that were freezing to death overnight because they didn't acquire the things that I had. So I was singing in the choir here and then going to church and the night before I had had on the Saturday night, I had three of the homeless see me walking around and tell me well I'm a member of this big old church. Why don't I go and see if they open the doors and let them sleep inside. So the next morning at eight o'clock that morning when they open the church doors, I went in and I grabbed Reverend Rachel's arm and I said come I need to talk to you. She said no I need to go. I said no no this is important it's life and death. One thing led to another and by the time we had our next service we announced that we would open for our very first shelter on that evening and we're able to rally folks to come and volunteer in that next service. We had food and everything and the church doors were open for the first time and people slept right there in the sanctuary on the floor and on the pews and it's gone on since. That moment was groundbreaking for us here at Oak One. It's at the path for us to welcome people inside, not just on freezing nights and not just for shelter but also for a place of community and to experience what it is to be loved and so I'm so grateful for the witness of Billie Jean not just in paving the way for us to open shelter but also for building connections and relationships with so many people who found themselves in very similar circumstances to her. I believe that's my calling to to help the homeless folks and other people hear the word of God and you know just just be able to I stand up for those that can't stand up for themselves because I have a loud voice and I have pink hair so people are going to pay attention to me. I didn't know we would end up with like a campus of union at the Valencia O Apartments but in many ways that's what's happened because she brings our worship there. Since we have the pandemic and most of your churches are not open and stuff I have a bluetooth speaker and union does their church service live on Facebook and I hook it up to my bluetooth speaker and I set it outside. There's three little Latino children they can't go to church because of the pandemic right now so I play the church service for the union on Sundays and Tuesday nights and they'll come outside and they like the misfit band and they like to hear God's word. One of the little girls even brings her Bible all the time and then I even have a lot of my other neighbors that'll come out on their porches and stuff because it's not a bluetooth speaker and they can hear it just so they can hear the word of God. Billie Jean has continued to be exactly that kind of evangelistic spirit that you pray that your congregants will be. I'm really grateful for the ways that Billie Jean is in ministry the way that she's made union and Oakland better churches and more faithful to the calling that Jesus puts upon our lives.


From around the globe, it's theCUBE with digital coverage of AWS re-invent 2020. Sponsored by Intel and AWS. Oh, great to have you with us here on theCUBE as we continue our coverage of AWS re-invent 2020, doing it virtually, of course, out of a necessity, as I'm sure all of you can appreciate. We're joined now by Alex Sanchez, who is the head of CrossGDC Networks at Fujitsu and Fujitsu provider of global IT services and solutions. And so their footprint, again, it is around the world. Alex, thanks for joining us here on theCUBE. We appreciate your time. And I'd like to hear a little bit more about your role. First off, before we jump in and tell us a little bit about Fujitsu for those who might not be familiar with all that you do. Thank you very much, Sean. I really appreciate it. Well, first, let me start by providing some background of Fujitsu. We're a global IT digital transformation company offering a full range of technology products, solutions, and services. We exist to keep our customers business running and we strive to give the best possible experience across every customer touch point. My role as head of CrossGDC Networks makes me in charge of centerizing technology networks across our global delivery centers. And for the past couple of years, I have been working on the standardization of our context-centered platform across all of our global delivery centers. Yeah, you mentioned global delivery centers. So let's jump into that. First off, what are they? How have you structured your business in that respect and ultimately what kind of service or solution are they providing to your customers? Absolutely. So our global delivery centers are interconnected integrated global teams. We deliver a broad portfolio of standardized services, which includes cybersecurity, workplace, and much more. We're based out of eight different key countries. We serve customers in over 100 different countries and we provide support in over 40 different languages. We enable, those GDCs enable us to consistently and resilient provide services to our customers 24 seven through 165 days of the year. The service that we offer as 52 global delivery teams are constructed from fully standardized components. It allows us to be configured to meet our customer needs and deliver flawless global consistency services. You know, you just, you were just talking about multiple languages, right? You've got to deal with countries, environments, continents, businesses with different needs of all, you know, all over the map, if you might say that. How do you balance that? Or how do you approach that when you do have so many customers in a wide variety of venues with a wide variety of needs?  Well, yes. As I mentioned, we strive to evolve our contact centers so that it meets that global need, that global expansion, and we adapt to our customer's needs. We have our GDCs with teams that are engaged and enabled so that we can provide customers with the best customer experience. We like to help our customers reimagine their employee experience. You know, you mentioned, you're talking about the contact centers and I know that you're going through this major transformation right now in terms of how they're operating. Before we get into that and jump a little bit deeper into what you've already touched on, what was the problem before?  Yes. We're always trying to solve something, make something better, put a little finer point on that in terms of what you were doing before. You know, where was the hitch? Well, Fujitsu's Global Delivery Organization tries to build trust at every opportunity. We aim to deepen our customer relationships by adding a value of mix of rock solid delivery innovation and collaboration. However, some of our previous systems, the net always offer us the functionality and flexibility that we needed to provide a diverse range of services to our customers and what they required. So that is the basis of our challenges and what we were striving to overcome. So you've turned AWS, again, Amazon Connect. I know that you've got widely deployed. What was it that attracted you to that in terms of finding the value in it and then what kind of efficiencies and what kinds of improvement in your operations is Connect providing you? Well, being able to think about the art of the possible, adding value to our customers, introducing next generation features, our road with AWS Connected started as a two-month proof of concept with over 150 different agents initially supported out of one of those global delivery centers, providing support and services to one of the regions. So we started as a way to innovate and provide next generation functionality. You know, proof of concept periods are always interesting, aren't they? Because you think you're gonna find out something and you might, but then you sometimes find out something else, right? That you're like, okay, well, there's another application here, there's another service here, there's another layer here. What was it in that period of time for you then as far as your takeaways that convinced you that, you know, this is right, this is good. We need this and so we're gonna jump in. Absolutely. So I would say that one of those things is that we made mark improvements in our customer experience. We were able to rapidly onboard new agents and provide automated features such as call recording, sentiment analysis, integrated callback features. We were able to help our customers faster while simultaneously improving the service quality. You know, COVID has been certainly wreaking havoc in every facet of life, right? No question personally, professionally, multiple industries. So how about the impact in your world? First off, just from COVID-19, how you've had to assess what your clients' needs are, what your needs are and first off, how have you balanced that here in the past year? Yes, well, Fujitsu was able to move 95% of our contact service agents to a remote work environment equipped with the tools that they needed to provide services while remaining safe and productive. Our contact center agents and operations was not able to persist but actually thrive during the COVID-19 pandemic and provide the much needed support that our customers were expecting and providing from us. Yeah, how fast was it, you know, I guess it required, how quickly did you have to respond? Because, you know, I mean, this certainly has caught a lot of people by surprise back in early March and into April. And I assume that Fujitsu is no different, right? All of a sudden you have a pandemic on your hands and you've got to move nimbly and quickly. So just talk about that if you would, that quick transformation that you had to make in terms of responding to the environmental crisis. Absolutely. So with AWS Connect, we were able to automate and simplify the complex contact center flows that we had previously. A product of this is Fujitsu's ability to now make ad hoc changes in seconds while avoiding multiple vendors to actually get those implemented. One example of this is that Fujitsu helped one of our customers move from 4,500 queues to less than 400 by actually doing call tagging attributes instead of just creating independent flows for each one of those countries. And this mainly because of the needs from the operation to be able to quickly create reports based on countries and languages. You know, I know you were involved and might still be, I'm not sure, beta testing with some of the new AWS Connect features that were announced recently, you know, here during re-event. What's got you going there? You know, what's caught your attention and what are you excited about seeing that go into practice on a wider basis within your networks? Well, John, I would have to say that introduction of AWS tasks has greatly helped us improve our agent productivity. We were able to see improvements of around 30% and we expect to refine our customer experience even further by adding additional AWS innovations. Yeah, you mentioned further. There's always a next step, right, isn't there, Alex? I mean, there's always, it's as good as you are now. You can't afford to sit still. I mean, that's the competitive nature of your landscape. So where do you see yourself in terms of rollouts in the future or if there's an area that you think this is the next challenge for us in the short term, what would that be? Well, that is a very good question. Fujitsu provides contact center services to around 300 diverse customers with agents speaking dozens of different languages and we are continually looking to improve those services and experience for our customers as well as our employees. We believe that if our employees are happy and safe and they have the tools that they need to do their work, that would result in a much more improved service to our customers. As such, Fujitsu is also investing heavily in digital transformation. Some of those elements would include a location agnostic delivery. This would actually allow us to create virtual teams with some employees working from Fujitsu offices, while some will continue working from home. This approach will offer a significantly and greater flexibility for our employees as well as an improved efficiency of our services. The ability to introduce self-service and automation by introducing virtual assistants, robotics, voice recognition, speech-to-text conversion, sentiment analysis. It will help us reduce the time it takes for agents or staff in repetitive tasks, allowing them to focus on the more important improvement, adding value to our customers, being able to add tasks such as technology upgrades, knowledge and data management, data analytics, business recommendations from our customers. This would then tie it into what we're doing with improved worker planning as situation changes and definitely COVID has been one example of that. Fujitsu needs to respond rapidly to ensure that we continue to provide support to all of our customers. Our Rota planning system provides insights, recommendations to help us deal with those changes as well as offering a level of flexibility to our employees to align with their personal needs. And finally, and tying this up with those innovations that we're looking into, being able to take those into employee engagement, we're introducing a proof of concept with gamification on some of our contact center desks to provide employees with a rewarding environment that offers an increased fun while also doing the work, reinforcing behaviors and enhancing customer satisfaction. Well, there's certainly a new order, a new world in terms of how we have to operate in a business environment. And I think you hit a key word there, it's flexibility. Ultimately, giving your employees the flexibility to still do their jobs in a very productive environment and a safe environment is critical on a sense like Fujitsu is committed to doing that. So congratulations on that. And thank you for the time today. We really appreciate it here on theCUBE. Thank you very much Sean and thank you for the opportunity.


So over the past few days we have seen the great restoration hit the game, where players who were affected by the inventory white bug have had their gear restored with what's basically the best loot in the game. We also have had players Cluster's group B and C who also got amazing high level loot. You even had some players like myself who were not affected in any way whatsoever receiving 20 Legendos with max stats. Yet I didn't many others didn't suffer due to any bug. What's happened over the past few days you'd think would make people happy but from what I've seen it's actually done the complete opposite. And yeah for sure you can use that old saying of oh yeah people just won't ever be happy but to be honest what people can fly who have done actually here with this restoration in my opinion is taken away from the game. How's it going guys my name is DPJ and if you enjoyed the video leaving a like really helps out and if you like what you see and want to see more outriders on a daily basis be sure to subscribe. So thousands of players received what PCF are calling guardrailed loot. For the most part I kind of understand why they did this as in terms of actually restoring every single player that lost the loot and restoring that loot to be exact would be a difficult challenge and to be honest people who have lost entire characters I will understand and support them receiving compensation for their troubles. But after having a night to sleep on what's happened I actually feel the way PCF have handled this will in fact impact the game in a negative way more so than positive. Right now let's forget about group B and C players and talk about group A players. These group A players lost entire characters meaning they couldn't even log back into SEG characters for a month. These players received their characters back with the addition of guardrailed items at the same time. What this has done is compensate players who have been put off playing the game due to having character wipes with the best gear in the game which has more or less left them with no reason to play the game as they already have what is for the most part the best gear in this game and what it offers. Why continue playing and farming for things you already have? This is a weird situation to be in and I don't envy PCF with their decision making here as it's obviously something they probably thought hard about and in reality what could they have done? What could they have done better? So yeah group A players have received their characters back and at the same time have received the best gear for SEG characters. Now after doing a ton of research into players affected by this I've seen somewhat 15% of players who had this character wipe bug not actually continue playing the game yet the other 85% simply made new classes and continued playing. I myself have many friends who went about it this way. They lost a character and just created a new one. Here's the crazy things these players who continued playing after losing their characters are for the most part are at a point now where their character was restored with the best gear in the game. This has done nothing but take away their need to play on those newer created characters due to being able to just transfer over that gear. As once you get used to these games mechanics the farming aspect is that only real end game there is. Give players what they are currently grinding for means the grind for them is pointless. And while guys this is only those group A players I'm talking about. What about group B and C players who they can't determine apart? Within that time period of the bug happening it seems as though even if you dismantled a legendary because it was bad sold it whatever you will be compensated with that perfect real version of it. I play the game on a daily I never had an issue of anything being wiped for sure I dismantled legendaries I don't think I ever sold any yet I got 20 top-level legendaries for seemingly no reason at all. My powers I play with every day are chasing that top tier loot they never had a problem with this book yet were rewarded loot they were currently farming for. My one pal got 6 top level death shields he's been farming for this weapon night and day for the past few weeks all of a sudden he logs in and sees six of them in his inventory what's left for him to do now. And this guard rolled loot was rewarded to thousands of players who were not affected in any way like myself most of these people feel now that they don't have a reason to continue playing.  I also know there were thousands of people out there that played this game more than me and they didn't get anything. How must these players be feeling knowing most of the people had the best rolled loot in a game handed out to them. I mean I ain't no game developer but I know for sure one's mindset playing this kind of game I've been playing loot shooters and grindy games for many many years now it's a race amongst players it's a challenge amongst players people like to show off their stuff where in terms of a player's deep down envy what they have and what they've earned and it pushes them forward to farm harder and longer. Once that best loot in the game becomes as common as muck even if you don't have said loot it tears you from wanting to play and farm for it because at the end of the day you're farming for the next month to get that top level loot won't really mean a thing and when that's all your game offers in terms of endgame it seriously does affect the game. While it seems people can feel done here who's panicked and just given most people the stuff they are chasing and need. I get the gist that they just want players to be happy and a game in a state where it's playable. I don't get the feeling from them that they care if people continue to play. I get the feeling that they want a game bugged through in an estate where they can just leave it and probably move on as weird as that sounds. I may be wrong here but the choices of late don't seem to be those of a company where they want to keep players happy with longevity because what they have done here is taken that away. I mean it's a weird one as I know people will be sitting there like oh you're just looking for things they can play about. This one hundred percent isn't true. I will continue playing. I still have loot to find and I still enjoy the game. But how many people out there don't have anything to play for? Don't have loot to grind for? Have had loot handed to them so there's no real point now in continuing playing. So this great restoration although I know they mean good and when I make players happy again I get the feeling there's nothing but a great mistake. And all that is short time will probably have the opposite effect to the one they want if their mentality or this restoration wasn't for them to have the game in a state where they can leave it good and move on. All I know is like I said now I've had time to think about this and what they have done. People will enjoy that top level loot for a short time. Then when that realisation of there's nothing else to play four kicks in they will simply move on. But hey guys this is just how I see things. I would want nothing more than right now to be proved wrong because I love this game and wanted to have that longevity and it be that game we all wanted it to be. A game which goes on for a long time keeping many gamers happy like myself who thought it would fill that void missing from other games in this genre. So please be my guest prove me wrong. And on that note the end of the video has arrived. If you enjoyed it leaving a like really helps out. If you're new around here and want to see more outriders on a daily basis be sure to subscribe and if you never want to miss a video I upload can turn notifications on by hitting that bell button. But guys thanks as always for stopping by and hopefully I will see you on that next one.


Or you can continue charging your devices. That's fine, too. Welcome. I am happy y'all are here. My name is Anna. I'm going to be talking about how we, and that is many more people than just we, have scaled the community that's behind the DBT open source project from a few hundred folks to now over 30,000. And this is a slightly polished version. I am the director of community at DBT Labs. And I also run the data function because to us they're kind of closely linked. And a little about me and kind of why I'm up here. Communities are my universe.  I like to tell people the story that I met my husband at a Ruby meetup. So communities are very special to me. Most of my friends. I've met through some community or other. And I think it's just an incredible way to improve social mobility. Communities offer opportunities that you won't be able to get anywhere else. And so I tend to apply that lens to everything I do. And I'm really grateful to be able to apply that lens to a community as large as the one behind DBT. I've been a member of communities, studied communities and scaled communities for about 15 years now, I want to say. All the way from like open source to data and everything in between. I went to grad school because I wanted to learn how open source communities worked and help them work better. So if you're ever interested about that, I can talk to you in the hallway track. But today I'm responsible for one of the largest and most active communities in the data analytics space. And that's the DBT community. So what we're going to talk about today is I'm going to give you a little bit of a background about what DBT is because I think for most of the folks attending, this isn't a project you might have come across. But I promise you it's really big for the data analytics folks. I'm going to tell you a little bit about the growth trajectory that we were on so that you have some context about what scale means for us. And I'm going to give you a little bit of a talk about how I think about communities. I think they're three different kinds. And spoiler, they're not actually mutually exclusive. And then related to each one of those, we're going to talk about how we've scaled awareness of communities, knowledge sharing in the community, contributions, and even culture for 30,000 folks. So this is DBT. This is DBT core. It's the open source heart of the DBT world. And it's basically a DevOps framework for data analysts. So analytics code is still code, no matter what language you write it in, SQL is code. So what DBT does is it helps provide a framework for analysts to work like modern software engineers. So think Rails, but for data people, right? About 9,000 companies use the open source project in some form. There are hundreds of forks, hundreds of community contributions by now. The project is about six years old. It was, the first commit was March 2016. And we just released 1.0 last December. So we have a high bar for 1.0. And this is the growth of our community along this time. So the community has existed pretty much for as long as the open source project has because as soon as the open source project kind of landed out in the world, it transformed the way that folks on data teams work much in the same way that Rails did in 2008 when it first appeared, right? It really helped people develop more quickly and in a much more standardized way. And so DBT does the same thing. And people got really excited about that because it's a really, really new thing for folks in the analytics space. And so you can see that right around 2020 is... AV Guy told me not to turn my head. You can see that right around 2020 is when stuff started to really take off. And that's because we started to build a company around the project. So before it was just an open source project that was supported on a semi-official basis by a consulting company called Fish Town Analytics. And then we rebranded, we became DBT Labs, and we invested really heavily in developing the open source, the open core part of DBT. So we have an entire core team with an engineering manager, and we have a product manager who's responsible for this area. And so we have kind of a really interesting dynamic in terms of the development that we do and contributions. And our growth is still doubling. So we've doubled in the last year from more than doubled, from about 14,000 to about 30,000 now in terms of humans in the community, which is really cool. Looks like a literal hockey stick. So audience participation time. Any guesses what those little bumps are on the graph? I don't know if you can see them. Yes? Good guess, but no. One of those is, but it's also tied to a different event. Anything else? No? So we've actually been running a conference for the past two years. And it's at the end of the year. We announced our 1.0 release at the user conference, and the bump in community growth is a result of people gathering and talking to each other about this community. And every time we gather folks together, we see kind of like a step change increase in the rate of growth of the community, which is really cool. So before we get into like the meat of all of this, if there's only one thing that you take away from this talk, is really that you're never done building your community. It's a living, breathing thing. It's kind of like when you adopt a puppy. You kind of have to be with it throughout its entire life. So whether you have 50 people who are just really getting started using the thing that you've built, or if it's 5,000, you're always thinking about how to make their experience the best one. And so it's really not about you. Okay, so I lied, it's two things. You're never done, and it's not about you. And so you're not thinking about like how do I build a community at the scale that we're talking about. You are more tending a garden and you are encouraging and nudging things to work in a particular direction. You have very little control over the process at the scale that we're at. And I like to think about communities as aligned in three buckets because I always like to ask why. What is the problem you're trying to solve? What is the problem you're trying to solve with your community? And those three buckets, for me, are brand contribution and community practice. So when I say brand, I'm not talking about like company brand. I'm talking about identity, right? So the way that Obuntu is an identity, or in this case, I think folks in here would appreciate this, you know, the arch community has a very strong identity so much so that it's spawned a meme. People are really proud of the fact that they use Arch Linux, right? And so it's like, oh, hi, by the way, I use Arch Linux as a meme. And you literally don't need to know anything else about that community. And it's funny, but it's also a really great awareness tool. So that's brand. You've got contributions, which is kind of what people typically think about when they think about an open source community. It's like people contributing code, right? Maybe documentation, but, you know, depending on the types of humans and the type of problem you're solving with your open source project, contribution can mean a bunch of other different things. So for example, DBT solves a problem for data analysts. They're not usually folks who are writing Python every day in a production grade environment. I expect less contributions in the form of code, but we get a ton of documentation contributions because we make it really easy for folks to do that and to help us. And that's especially important because DBT depends, it sits on top of a warehouse layer and there's many different kinds of warehouses that DBT works with and we can't keep up with documenting all of those relationships. And so we depend on the community to document how things work a lot of the time.  And then finally, a community of practice. And this is where you go into the space of forums, things like discourse, things like Slack, and a community of practice is really folks teaching each other how to do a thing. So if you take the example of Rails again that we've been talking about earlier, community of contributors would be folks who are collaborating on code and documentation and things like that. But the community of practice around Rails would be people teaching each other how to set up Rails for the first time. How to integrate it into their workflow, how to change the way that their teams work in order to adapt to this new framework. And so DBT has a similar model. And the thing that I've realized about communities is that it's really not any one of these things. It's kind of a portfolio. There's a proportion of each one of these things that's present in a community in different proportions. And so for us, it's more like 40% brand, 40% community of practice, and 20% community of contributors, mostly because we have an entire team that has a roadmap and contributes code and because we don't expect our users to have the technical background to be able to contribute in the way that some projects might. So roughly, to me, this maps to three different areas of leverage that you have as you're growing your community. So as your community grows beyond a few hundred or beyond tens or hundreds of people, the way that you think about awareness changes, and that's really related to brand identity, the way that you think about encouraging contributions changes and the way that you think about knowledge transfer and encouraging people to help each other, that changes a lot too. And so let's talk about awareness first. This is really a question of do people know who you are? Do people know what you represent? How are you different? If in a small community you're thinking about, like, I just want to get people to use this thing, right? When you're first getting started, you're mostly thinking about, like, hey, come try this thing for the first time. When you get to thousands of people, the types of things that... the different ways that you get the word out about your community, about the open-source project that you are supporting is going to be different. You can't go out and talk to thousands of thousands of people every month as a maintainer, as a founder. So what do you do? So I have a handy meme. There's going to be lots of memes in this presentation. You know, so the regular or maybe pea-sized brain version of this is just, you know, making your project work, right? That's the easy part. I say this is a non-engineer. I'm sure it's not actually that easy. And then, you know, it's about getting people to use the thing. That's like, you know, tens, hundreds scale. And then at the thousands scale, it's about getting people who are using the thing to tell other people to use the thing, right? And so if you have a community of a few hundred folks, it probably took you a lot of time to get to that point. You were probably spending most of your time reaching out and talking to folks. And so the next unlock for you is to scale yourself and get other people to talk about this thing for you. And how we do this, like there's many different ways to do this, but how we do this is we invest heavily in the people in the community. So for example, we coach people in how to write well. We coach people how to speak. We spend a lot of time helping folks refine talks and submissions to our conference to all sorts of other things. Why? It's really good for folks' careers, right? That's number one most important thing, like do things that help other people and then helping yourself. The second is that if there's really compelling writing or speaking that is by the community about their own work with DBT, that says so much more about the project and the community than if it was a bunch of advocates going out and like, here's a DBT demo, right? It carries much more weight. And there's a galaxy brain version of this. So if your motivation here is to get people to talk to other people about the thing, the galaxy brain version is people who aren't using the thing telling other people about the thing, right? And so this is when you get so good at telling the story of like what your project is, what niche you feel, what makes you special, that folks are starting to talk to one another about it, even if they're not users yet, is it possible? Heck yes, right? There are lots of examples out here on the floor. I don't use streaming data, but I hear folks who do use streaming data that materializes popular. That comes up a lot in the DBT community, for example. For folks here, I don't use Ubuntu or Linux Mint, but it's probably the best first Linux distro for someone to try desktop Linux, right? Like almost everyone will recommend Ubuntu as the first thing to try, or at least they did in my time, I don't know if that's still true now, but they don't necessarily have to be able to use it themselves. They just know that it's good for beginners, right? And that is awareness. That is like galaxy brain level awareness, right? And then the other thing is the way that we think about swag changes at the scale that we're at. So, you know, everyone makes stickers, you've got bottles galore, how many bottles and bags and stickers and t-shirts have you gotten today, right? It's kind of ridiculous. You have an entire closet of tech t-shirts. So, that's no longer fun and interesting, but the way that we think about swag is a little bit more of an identity statement. So, think about that distillation of the brand of your community, like what do you stand for, and design your swag around that. So, maybe the platform that you have is one centered around pride. Maybe it's the environment. So, I'll give you an example. For probably the women in the room, who here owns a pair of Rothes? Uh-huh, okay, okay. Okay, keep your hands up. So, of the folks who have a pair of Rothes, how many of you have talked to someone else you didn't know because of your pair of Rothes, or they were wearing them, or they were wearing it? Uh-huh, yeah, yeah, exactly. So, why is that? What's the big deal about Rothes? They are recycled plastic, yes. So, Rothes have a really compelling narrative. They're shoes, okay?  They're shoes, all right? They are really comfortable women's shoes. And this is, like, comfortable women's shoes are like pockets. That's already a selling point in and of itself, right? So, they're comfortable, number one. Number two, they're washable. And number three, they're good for the planet. And so, that's it, like, that's the narrative. And it is so compelling and powerful for folks that the brand just, like, spreads like wildfire. I think someone had, no? Yeah, we've got a moderator with a mic so that folks on the... Yeah, feel free to jump in. Like, we're a small crowd. Coming up the elevator, not two minutes before coming in here, we had a very lengthy discussion about pockets for women's clothing, yes. So, I'm still waiting for the brand that is going to build a platform around pockets. So, if you're looking for a business opportunity, there you go. But, like, that's what swag is to me at the level of, like, a really large community is, like, you're making a statement. And so, in this case, like, Rothy's isn't swag, it's the product. But the product itself is making a statement, right? It's making a statement about the environment. It's making a statement about, like, women's comfort and reusability and things like that. So, that's really cool. So, I encourage folks to think about that. The fact that she didn't use the word shoes at all... Uh-huh, yeah. Yeah, yeah. So, exactly. Really good point in the front here that I'll repeat is the fact that no one mentioned the word shoes in, like, the first several minutes of the conversation is just a testament to, like, how powerful that identity is. So, that's brand and awareness. Okay, let's talk about knowledge sharing. This is related to building a community of practice. And the reason it's second on my list is because for DBT, those are, like, the two big things, right? Brand and awareness and community of practice. So, a community of practice, just to remind us, is when folks are getting together in a community to teach each other how to do a thing. They are learning something new for the first time, or they are refining their skills. And so, in the case of the DBT community, what that looks like is we have folks who are data team leaders who get together and talk about how org structure changes when you adopt self-engineering best practices. Like, what does that mean for job titles? What does that mean for folks career progression? How do you evaluate folks? Like, all of those things. How do you advocate for these changes in your organization, right? How do you... What are best practices for working with data in this new space? Like, those are the kind of conversations that the community enables. And so, that is what knowledge sharing means to us. The problem is, how do you do this in a group of 30,000 people? Right? It's sort of like standing up in a room and yelling when everyone's gathered for a keynote at a conference like this one, right? It's a lot of people. So, this is where it starts to get really interesting. Who has heard of Dunbar's number here? Yeah, okay. At the risk of putting more steps on our moderators, Apple Watch maybe can have someone tell us what Dunbar's number is. It's basically like the amount of personal relationships that someone can maintain. And if new people come in, somebody else drops out. So, it's basically the average. It seems kind of high, actually. 150, yeah. Yeah, it's a lot. It's a lot of people. Just maintaining relationships like 10 is hard. Yeah, so if you think about it in the context of an organization or a company that you're a part of, 150 is when it starts to become harder to remember everyone's names and what they do. Even if you interact with people all the time, by the time you get to 300, you're like, I know we've met, but... And so, Dunbar's number gets really important. So, if Dunbar's number is 150 and you have 30,000 people in your community, what do you do, right? This is where we do something called micro-communities. So, we start thinking about different pockets of people who are much smaller, who have something in common, ideally within that number. So, for example, we have a community of DBT users in San Francisco, in San Diego, in Austin, in New York, in what have you. We have a community of folks who are data leaders and they have that thing in common with them. We have a community of folks who are working in a particular industry, that folks who are solving problems of the healthcare in finance with DBT, right? Like, in the public sector. And so, it's about kind of breaking things down into different segments and starting to think about what are the unique things that connect people beyond the fact that they're all a part of this community already. And a primary way we do that is when someone posts an introduction, say hi in the, like, community Slack, like, great. You said that you work for, you know, so-and-so company. Maybe it's in healthcare. There's an entire channel here. You said you're from the Bay Area. Cool, great, there's a meetup happening. And kind of helping people find those connections to someone within kind of this, like, smaller subset of the community is a really great way of bringing people in and, like, keeping them engaged. And it encourages people to share more freely because it's a lot easier to talk to a group of 150 people than it is to a group of 300, 30,000. Make sense? And then the other problem you start to run into is platforms. Some platforms just don't scale to the size of, like, 30,000 people. And so, I talk about this as minding the backscroll, and I'll give you a very specific example.  Extremely valuable when you have a few hundred people. At 30,000 people, it's nuts. So this is what it looks like for us on a given day. These two posts are in the beginner channel. They are seven minutes apart. There have been about six that say morning already, and it's not even 9.30, right? And so imagine if you're the person who is helping other people in the community. You're really motivated. You're like, I want to help beginners get started. If you're not living in this channel 24-7, you're going to miss questions, and people are going to feel like they're just, like, shouting out into the void. So, like, sad face, right? It's, the backscroll problem is like a really significant one, and Slack no longer has the affordance of supporting that particular community use case at a certain scale. So what do you do instead? Here's what we're doing, and I can't say that this is the best way, but this is what we're trying right now. This is kind of like Reddit and Stack Overflow, but fewer trolls is the way that I would put it. So GitHub discussions is what we've landed on, and there's a few things that I love about this that Slack doesn't do. The first is it's easier to search things. People ask the same questions over and over again at a certain point. You just want to give, you know, you start sending out links. This way people find the most common thing themselves. You can also pretty easily help people navigate to, like, what is actually the thing that they're trying to do? Do you need help? Are you trying to figure out how to do something for the first time? And the really cool thing is we're going to be able to take this and be able to put it together with our documentation website, okay? And so then you have kind of this entire portal where all of your content lives. We can pull out the discussions that are answered, that are, like, gold standard, that are the most upvoted, we can curate, we can do all sorts of things, and it creates, like, a lot of that flexibility. So this has been a really big unlock for us and, like, a really big part of our strategy to keep up with the growing community. Yeah. And many, like, happy face instead of crying face. I hopefully will report back as we continue to implement this, but this is the solution that we've adopted so far. And it, so far it works really well. It's really easy for folks to navigate and has pretty good SEO, which is really important. Yeah. I just have a quick question. So as the community is growing and, like, all these questions are coming in, do you create, like, strategy more to get, like, your community members to answer the questions, or do you guys actually, like, hire more internal staff, like, over? The goal is for this to be a flywheel that the community supports itself. Like, that's the goal of knowledge sharing at the scale of 30,000 people. Like, you can keep up with all of the questions. Someone was telling me in, like, a related company, they have, like, 8,000 questions a day on Stack Overflow. That's insane. Only the community can answer that, right? So be on a certain scale. You really do need to lean on folks, and the thing that you focus on is making it as easy as possible for them to help you. And so that's what the next part of the talk is going to be. Like, how do you scale contributions? Yes. So are you still using Slack? We are. Yeah. So those micro communities still live on Slack. Yeah? Because Dumbbar's number. But for questions, for things that we think are longer-living, we're moving them to discussions. And it allows us to do cool things. Like, someone starts a discussion, and we don't really know, is it a question about getting started with the project? Is it a bug, or is it a feature? You kind of don't know until several comments in.  Or like, this is a gap in how we document the thing that we're doing, right? There's an assumption missing here. And so what we then do with that discussion is we transfer it to an issue in the right repo. And we link a pull request when it's been done. And everyone on that thread gets notified that, like, okay, cool, we now have like a blog post about this, or this is like part of the next release or something. And so that creates like a really nice, easy transparency loop without having to do a lot of additional overhead. So let's talk about scaling contributions. And this is us getting close to the end here already. So we just talked about how it's important to help people help you at scale. Like, you really can't... In a small community, maybe you have the opportunity to like handhold people when they're first getting started with contributions. You actually want to. You want to get to know the one person who's like, can I open a pull request against your project? The first few times that happens, you're like, oh my God, that's so cool. The hundredth time that happens, you're like, okay, I need resources. I need material. So of course you need contributor guidelines. You need good documentation. But there's a little bit more to it than that when you reach a certain scale. Like those are kind of like table stakes, right? So at this stage, when you are getting popular enough that you're kind of like batting contributions away, you want to start thinking about like, do you actually want contributions? That's not a given for every open source project. And the reason I say that is because you probably have a roadmap. You probably have an idea of what you're building over the course of like the next six to 12 months. So you do want contributions, but really specific kinds.  You probably want to make sure that people are collaborating with you. So you're more likely to prioritize one repeat contributor over like 10 drive by PRs.  Which isn't to say that the bug closing is not important, but like your priorities tend to change. And again, it depends on who is supporting your open source project, right? Do you have an engineering team behind that that is funded? Is it purely volunteers like those dynamics? Those dynamics different? I'd be interested to hear how some folks solve this problem. But for us, because we have an engineering team and we have a roadmap, we have a product manager, we want to be really clear with the kinds of contributions that we're looking for and the kinds of things that we think are really valuable for folks to spend time on. And so here's how we do that. Again, GitHub discussions. But this time it's on our core repo, not our documentation repo. You can see, and I don't know if this is big enough, but there's a few things going on here. We have, the top discussion here is an upcoming feature. So we're kind of signaling to the community, like this is what we're working on, this is how we plan to approach the problem, just like we would expect someone in the community to do before they open up a pull request to like, you know, jump in and talk about it. So we apply the same standard to ourselves. And then you can also see in this example, the number one most requested feature from the community is above the fold, like it's visible to everyone because it's like so highly upvoted. It's actually the last one on this list, column level lineage. It's a really hard problem to solve as it turns out. And so we get questions about it all the time, but we know we're not going to be able to get to it for a while. And it's not something that you can just hack in an afternoon. So we have a discussion about like, why is this a hard problem? And we make it visible. And we point folks to it as opposed to every once in a while you have an AMA or every once in a while you're in Slack and someone's like, when are you doing column level lineage? And we're like, here's the discussion. Right? And like if you want to stay up to date, subscribe to the discussion. If you want to contribute, if you want to work with us on making this a reality, let's talk. So that kind of signaling is super important. And when you give people affordances like this, when you make it easy to discover information, what you will find is that the community will start jumping in and addressing those questions for you. Right? So now anytime Pipeline comes up in the context of DBT, people are like, oh, there's a thread. Just go and look at it. Right? Anytime column level lineage comes up, not only DBT Labs folks, but the entire community is like, there's a thread. Like just go to this thread and everything is there. And it is really, really powerful. So the thing about getting people to help you is you need to make it really, really easy for them to do that. Right? Have things be really well organized, well curated, help people navigate stuff. So like really think about platforms and affordances. I can't say that GitHub discussions is the best thing for everyone. Like we've thought a lot about why this is the best thing for us. Almost everyone who works with DBT already has a GitHub account. Like you need version control in order to make use of the tool. So this is really easy for us to do. Like there's a very low barrier to entry compared to, say, Stack Overflow. You have to sign up for an account. Right? But that might not be true for your community. All right.  Right? I mentioned for folks at the beginning, and if you missed it, I like to tell people that I met my significant other at a Ruby Meetup. And Ruby Meetups are kind of famous for their warm and welcoming culture, the Ruby community in general. But how do you scale that to like, again, tens of thousands of people? DBT as a community is also really famous for that. It's famous as a really welcoming place, as a really inclusive place, and that's really important because analytics is a field that has representation from so many different backgrounds and so many different kinds of folks' experiences and for a lot of folks, software engineering, best practices, things like Git, working in the command line, those are things they're encountering for the first time. It's really important that we create a safe environment. It's really important that we create an inclusive environment that we're talking about and making sure that things are, that folks feel welcome, seen, safe. And so when you're a small community, like, yes, you need a code of conduct and enforcement policy no matter how small or how large a community you have, yes, you need to remind folks about your rules all the time. In a small community, it's easy to remind folks about rules and be prescriptive, but in a large community, you can't be everywhere. You can't moderate everything all at once. And so it becomes really important to have the community step in and moderate for you, right? Just like it is with content. And it's equally important to focus on what people actually do. So I'll give you an example of what I mean by that. Again, this might be too small, but this is our introductions channel in Slack. And for folks who are a little closer to the front, I don't know if I can zoom in, do you observe anything about the way that people write their introductions? How would you write your own introduction knowing nothing else about this community? I'm going to try and zoom in. Yeah, exactly. Name title company. Yeah, there's like a formula here that's really easy to observe. It's like, hi, I'm so-and-so. I'm an insert title here, and I work at Bloch Company, and maybe I live in this area. And that's it. And everyone has kind of the same pattern. And you will see that when people reply, they will also use kind of a similar pattern to be like, hi, welcome. We're so glad you're here. And the reason that this works so well, we didn't tell people to do this, but the formula exists because people see other people do it. It's really easy to observe certain behaviors like that, but the same is true of behaviors that you don't want. And so if you are not that good at enforcing certain rules and certain behaviors, it can really quickly turn into a situation where you have bad examples that people are recreating because they see other people do it. So it's really important to keep in mind what people see other people do. It can be sometimes more visible than the rules that you have. No matter how many times you put the rules in front of people, it's really easy to lose your way. So that's why you have to scale moderation. You have to empower folks to tell you. You have to make your rules and expectations as easy as possible to follow so that other people in the community, just like they know where the Python thread is and just like they know where the column level lineage conversation is, they know that like, hey, we use threads here. Please don't double post. Please, we have a slack bot that says, don't say guys. Just little things like that are easy for folks to repeat and they help a ton in keeping this like a really easy to navigate and a safe and inclusive space. So that's actually it. That's all I got for you. But thank you for... Oh, thank you. Thanks for the great audience participation. I appreciate it. I'm around hallway track. Folks have questions. I know we're just about up at time. I don't know if we have time for one question. Do we? Okay. Oh, nice. Yes. Please. Thank you. This was really great. Let me just ask my question because we're short on time, but thank you for your presentation. Going back to the community of practice and brand and stuff, for I don't know a long time, but I'm just curious if you have any other thoughts or things on how you identify those? Do you see any band diagrams in them? Are they part of the whole... Is there something that they are inclusive of? All three of those. Yeah, it's a good question. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. I don't know. Next question. There's definitely a band diagram. I mentioned at the beginning that for us it's like 40, 40, 20, or like 40, 20, 40 in terms of just like how much time people spend in those buckets. So it's probably all of those things, but the weight each one will have might be different depending on what you're trying to do. And so one of the really cool things about the DBT community is that the practice of what we call analytics engineering. Analytics engineering is the act of applying the software engineering best practice as the analytics workflows. It's synonymous and interchangeable with DBT the project. And that's really cool, but that's where branding and awareness becomes really important. And so what the DBT brand stands for is really closely aligned with what the community stands for. So we have to think about that really carefully. Great question. Thank you. Thanks. Thanks for coming. Thanks for hanging out.