Bonjour, I'm Catherine Lefebvre, the ONAP TSE chairs. As the industry moves to the native cloud, the purpose of today's panel is to share how we can bring together the best of the two words, ONAP and CloudNative. Six members of our ONAP CNF task force will be presenting our ONAP added value proposition. I'm pleased to welcome Shrinni Adepali from Intel, Sibo Perala from Nokia, Ala Goldner from Amdox, Reni Heibi from Samsung, Fred Olivera from Verizon, and Seishi Kumar from Awe. Over the past years, most of the carriers kept investing in their private clouds. They are now considering the migration of some of their network-related workloads to third-party clouds. With the technology evolving 5G AI, there is a need to host applications, data and devices differently. The management of this hybrid environment, private and public, raise complex challenges, like latency and bandwidth requirement, location and privacy concern, including security, resiliency perspective and more. In order to address these constraints, the CloudNative Computing Foundation has defined a reference architecture, in which we believe ONAP has a role to play. Considering this new multi-cloud and containerized environment, Shrinni, what will be the cornerstone of our CloudNative implementation? Okay, thank you, Catherine. Yeah, it is the best, in my view, described by giving two examples. So, what do you need service orchestration for? So, let's take the 5G use case. Many operators have a large number of edge locations at various places, like you have tens of thousands of edges at the cell tower or near to the cell tower and maybe thousands of central office data centers, maybe hundreds of local data centers, few regional data centers, and of course, people like to use even public clouds, even for some of the control plane or the analytics applications. And even with respect to those data centers, if you take 5G RAN use case, for example, it's becoming desecrated, like you have 5G RAN components such as DU, which may need to be deployed across multiple cell tower edges and CUUP, CUCP may need to be deployed across multiple central office data centers. And even if you take 5G core, some control plane functions like AMF, SMF, and user plane function may need to be deployed to the local data centers, as you see in the picture. And on top of it, the telcos want to go beyond just deploying network functions, but also their customer applications for new business opportunities. All these things, for example, need to happen in one single click, right? One should be able to deploy these things in one click, not only deploying, but also data operations. Let's say there is a new image upgrade, right, on the DU. So one should be able to upgrade the tens of thousands of DUs in, again, with one click. So the problem statement really is there are a large number of sites and the computing is basically distributed across multiple sites. And then you would have customer applications and you could have multiple customers, hence multi-tenance is very important. And you also need one should be able to deploy not only CNFs, which is, of course, becoming very popular for 5G, but there would be legacy applications with the VNFs. Also need to be supported. In addition to CNFs, VNFs, one should be able to deploy, as we discussed just now, able to deploy normal applications, enterprise applications, in the same Kubernetes clusters, for example. So again, we are saying that proven is becoming a choice of workload orchestrator for new 5G deployments. And hence, any service orchestrator expected to support Kubernetes-based sites in addition to whatever sites they have such as OpenStack. So as we see it here, this multi-edge cloud computing scale is similar, even higher to hyperscale automation. So hence, we believe telcos, even enterprises, managed service providers need at-scale orchestration automation solutions. Hence, service orchestration is very important. Thank you. Thank you, Srinu. So, Timo, ONAAP is successfully established as the de facto industry standard for NFV as the automation, providing a comprehensive platform for real-time policy-driven orchestration and automation of physical and virtual network function. What will be its role in the context of CNF? Thanks, Katherine, for the question. I believe everybody in this panel has heard comments like, what is the role of ONAAP with CNFs and is there really a need for ONAAP in the era of cloud names? I believe we all have come across those questions. So I think while it's true, ONAAP may not look exactly the same as for CNFs as it is for the VNFs, virtual network functions. We do strongly believe there is a definite role for ONAAPs still. So first of all, it is highly likely that the VNFs continue to exist still for some time, some time to come and they need to be managed and orchestrated. Second of all, PNFs or physical network functions, for sure they will continue to exist in telco networks. And it goes without saying that management and orchestration of those PNFs continues to exist. So for example, radio network equipment, equipment in mobile networks. So considering all that, we anticipate that ONAAP needs to be capable of managing orchestration and automation in a hybrid environment with all kinds of workloads, be they CNFs, VNFs or PNFs. In addition to that, then ONAAP keeps an end-to-end service view of the system. And this is where we see ONAAP bringing undeniable value. So orchestrating the services end-to-end. And those services may be composed of the different kinds of workloads that I just mentioned. So this will be the great value that ONAAP will bring with it also in the era of cloud natives. So with that, back to you, Catherine. Thank you, Timo. So we understand from you the added value of ONAAP with a service orchestrator. Alaa, what are the other values that ONAAP can bring for cloud native? So first of all, the most important thing that ONAAP is bringing to the cloud natives is that end in general to the whole network functions world, which includes physical, virtual, and cloud native function as that it looks on the whole service thing, not just the specific network functions and the types handled by their specific management system. But in general, the whole service, the service SLAs to maintain and the service characteristics to maintain, to orchestrate the service, to view the service from the inventory and to design the service. And I think this is the key difference and the key principle of the network management system that we are supporting in ONAAP. So going to some specific, engaging into some specific details of what exactly the thing that I just said means it of course uses microservices architecture as use cloud native principles by also supporting multi-cloud and the availability to run over any types of cloud. It manage network service and application lifecycle management across multiple wins within that multi-game multi-cloud whether it is OpenStat, Kubernetes, Azure, any vendor specific wins, whatever needs to be supported is supported even if there are several working for SM services under different domains. This is something that ONAAP support. Now, speaking of design, ONAAP design tool support multiple descriptors, Helm, Tosca, HIT, whatever is needed is supported by ONAAP. Some came along with ONAAP creation and establishment in Linux Foundation Networking. The other were added as we continued is our walk to enhance ONAAP. ONAAP OOF, which is the optimization for tool, for placement and homing, for choosing the right location to place workloads. Extremely important for 5G for many use cases to support low latency application. You need some core network functions for system to put close to the user. So this is what that tool does and this is something that cloud native by itself looking into the cloud native word only do not provide here. The overview is for the service and how service should be running. This CEO of course collects telemetry from remote site, analyzes them, generates any control loop action whether it is scale, view on the service level. And this is also what is supported in ONAAP. Of course, support of standard models and APIs by Etsy, by TEMAP, by MAP, by 3GPP. You know, some examples may include 5G network slicing support where all the development for end-to-end network slicing is based on 3GPP, Etsy, NFV, and actually Etsy's are the same specifications. Additionally, ONAAP of course enables the total configuration of network function via RESTful API, is Netcom Kubernetes Service, whatever is needed. And again, there is a variety which is not just limited to one method. And in order to view in the real time, you know, the whole service and how its resources function right now, the AI, ONAAP AI is the central repository, not just for specific cluster, but the central repository of the deep site, network element inventory, and network service status. So I guess again to summarize and to highlight what I said, ONAAP really is the service level end-to-end network management platform solution going across different domains, different whims, when I say domain, trend, transport core, everything different, transport along different whims, different cloud, public, private, specific types of cloud, and combining all those into the single end-to-end type of service. Thank you, Alain. Rani, how does ONAAP fit in the CNF landscape? Thanks, Catherine. So the shift towards the cloud-native architecture in telecom is a coordinated effort between several communities and organizations. There are four pillars to this transition and ONAAP has an important role to play here. While this landscape may seem confusing at first glance, it is actually a well-orchestrated endeavor. Each community provides its deliverables and there is little to no overlap. The first step in building modern network functions is defining the architecture for both the cloud infrastructure and the network function itself. Several groups within the Linx Foundation Networking and the CNCF are working on defining the requirements for such architecture, as well as SDOs like Etsy and the GSMA. ONAAP has an updated set of network function requirement to cover mixed workloads of PNFs, VNFs and CNFs. The requirements ensure that the network function can be a part of an end-to-end service managed by ONAAP. ONAAP requirements complement those provided by CNTT, CNCF, TUG, and ensure smooth orchestration experience. In the implementation phase, ONAAP provides end-to-end orchestration for cloud-native network functions. It enables building network services that are cloud-native and may also include virtual or even physical components. To make sure everything plays nicely together, tools for testing and validating the cloud-native services are being developed. The ONAAP community is working on the evolution of its VTP, the VNF testing platform, to CNTP to support cloud-native network functions. CNTP may be used as part of the batching program and is closely aligned with the LFN Compliance Validation and Certification or CVC program. The result is going to be a coherent set of requirements and tools for implementation, making it easier for both operators and vendors to smoothly transition to cloud-native architecture. Back to you, Catherine. Thank you, honey. Fred, can you tell us a little bit more about how ONAAP implementing Etsy specification? Thank you, Catherine. This is Fred Oliveira. I'm a fellow at Verizon and are working with ONAAP to enable some alignment with the Etsy specifications to allow orchestration and automation using some of the Etsy methodology. In order to leverage this, we're using several specifications from Etsy, including the ones listed on the screen here, Sol4 for VNF and PF packages, Sol7 for NS package, and various interface specifications, Sol1, Sol3, and Sol5. In the way we've integrated these things is we have our SDC component, which onboards packages and we've enabled Sol4 onboarding and design of these VNFs into network services for Etsy-based network services. That can then be leveraged by ONAAP to deploy services and work with an external or internal NFEO to deploy network services and the VNFs associated with that. We've also built in several adapters to enable communication with external VNF managers. Sol3 adapter is a way to interface with external VNF managers and the Sol5 adapter will allow connection to the Sol5 enabled NFEO. And then we also have some Sol2 connection to the EM environment for to have ONAAP act as an EM. Etsy is pursuing a definition of the container-based network functions and we're working with them to enable these specifications in ONAAP. These are currently in stage two specifications, I-511 in particular, to describe the containers and operations. And we intend to leverage these capabilities to deploy onboard and design and deploy container-based network functions into using ONAAP into an Etsy-based environment. And thank you, Catherine. Thank you, Fred. And finally, Sishu, can you give us an overview of where we are with our ONAAP Cloud Native Journey? Thanks for the question, Catherine. Long story short, the primary motive of the, I mean, the primary motive of this entire objective is to have this in picture as a, as we show in picture that we need to have the complete orchestration and the management of all the resources that includes the VNFs, the VNFs and the CNFs to be done by the ONAAP that being a centrally positive thing. So the primary motive is to have this possible as a stepping stone towards the Cloud Native Journey. So what we have done so far is that we have taken the initiative of having the Cloud Native, the CNF support in Casablanca of ONAAP and they actually work started as a TOC, the proof of concept from guys from Intel and other driven mainly from Intel guys, but also that we have joined the main task force here, where in the case, VNFM was introduced into multi-cloud. The intention there was to actually have the existing APIs around the VNFs and VNFs to be supported and also to leverage the CNF orchestration and a management with respect to that. So this close, as you've seen, the diagram are depicting different tasks which are involved in that. It's pretty huge task complex. The initial diagram may actually overwhelm someone who doesn't have the different string of ONAAP, but as you can see, there are multiple issues which have been solved in this as we tend to move ahead. So from there, what we have is this thing, which is what is current work, which is actually happening. So in a nutshell, the previous work actually had a Helm chart which was defining the complete CNFs model and also package in the heat package. So that was what we were trying to solve in current work. So the current work is more like give a Helm chart a first-class region ship during the day zero and day one operations. We are talking about the onboarding, the design, as well as instantiation work with here. So new resource step is being introduced during this season time. That new resource will be known to the both onboarding, as well as instantiation work of it. That way we will keep a Helm chart or Helm-based orchestration of the CNFs or the management of CNFs possible and we can actually take it to the other steps of it. So this is a one time, I mean this flow which we try to have here is actually apart from the various things, we also have delegation mode here with respect to the other modes. Like for example, we are also taking up the previous, as Rani rightly said, we are actually talking about ETSI-based K8s orchestration also for the external VNFM, which is a brownfield scenario. So such things are also being considered. The other key enhancements around this are around the Helm enrichment through profiling where user profile will be given and that will be considered for the orchestration perspective and all. So this is in a nutshell, I can say the major work which is currently being considered and what is there for future? The future is actually what we will require to have the complete end-to-end orchestration perspective of this where in the day-to-operations like what Allah was saying sometime before the scaling, the healing perspective should also be included. We also intend to have the integration which Rani just spoke about and also Fred coming from different SDOs as open source systems like the XG Vela, the CNTT, the CVC. Then on the ZEO side, we have the 3GPP, we have GSMA, of course, ETSI. So these are all things which have been considered from the perspective of holistic involvement of ONAP and orchestration of this and management of this with respect to other entities which are working around CNF. So the other main intention here is also to take into consideration the intent-driven orchestration which is also being worked as we try to do right now but the integration with that with the main flow is also something which we want to consider in future. So with that, Catherine, back to you. Thank you, Seshu. I also want to thank all the speakers and the ONAP CNF task force for this panel. I hope that the audience now understands that ONAP acts as a central network service orchestrator addressing lifecycle management of network function-based services. ONAP already deployed and managed VNF and PNF workloads across multiple sites and is now evolving to manage future services that will include additional CNF capabilities and CNF component. Thank you so much to everybody and then we open the floor for any additional question. Looks like there is a question. The question is what is the difference between PNF's VNF versus CNF in terms of data operations? I'll give my view from the VNF, CNF perspective, from the deployment perspective, I would imagine there won't be much difference from the data operations perspective. Many times CNFs are deployed. The intention of CNFs in general is that the configuration need to be up, configuration get applied to the VNF even when they get restarted by Copenhagen itself. You normally with VNFs in traditional environment and the VNF gets restarted or VNF gets restarted, somebody has to push the configuration again, maybe via netconf or maybe sometimes things get stored locally and they get basically reconfigured and it comes back up. But in the Copenhagen environment, having shared storages should not be assumed. In those cases, the traditional method in CNF is to get a whole lot of configuration via custom resources in Copenhagen, which gets stored in HCD. So whether CNF gets scaled out or whether CNF gets restarted or moved to some other node, things would suddenly work fine. So yeah, that's one difference I can think of, like in CNFs, people normally export configuration via corporate custom resources, whereas in VNFs and VNFs, typically it could be netconf, typically. So yeah, that's one difference I can think of. Thank you. I think we have also another question from the Q&A with speaker. Will ONAP support different Kubernetes operator framework like Kudo or the Red Hat operator framework to support more complex management of CNF when then what N can support? Yeah, I can take that. So Kubernetes operators, nothing but a CRB kind of a controller framework. So the purpose of operators typically is to simplify the deployment of a given application or network function, especially when the deployment involves multiple things to be done together, people tend to go with operators. But even the operator at the end of the day is initiated using some custom resource, the Kubernetes resource, right? That resource itself need to be kind of deployed from central place. So that's where ONAP comes into picture. So whether it is the operator, whether it is a set up discrete COVID resources that need to be deployed in a workflow fashion, that is a job of ONAP and 30th orchestrator. Yeah, so the answer is that, yeah, any 30th orchestrator like ONAP shall support purely Helm chart based deployments, also deployment using Kubernetes operators with some custom resources. Thank you. I think we have also another question on the chat room. Do you plan to include XC-MAC, multi-access edge orchestrator functionality to ONAP? So I think we lost threat, but maybe Seshu with the orchestrator, can you take this question? Maybe I can take it up, I mean, thank you, Shini. Yeah, to me, in my mind, at least, yeah, I mean, we are not thinking about XC-MAC, but based on my understanding, the ONAP with the Kubernetes plugin becomes a MAC orchestrator. I believe that it has enough functionality to call it itself as a MACO, MAC orchestrator, but there may be some additional things that we may need to do that needs to be planned. But most of the, I mean, I would imagine that 90% of the MAC orchestrator functionality would become part of ONAP, eventually. The only thing, one thing which we're addressing at this time is, we don't have it yet, but we are actively talking about is called traffic steering, right? That's one of the functionality which MAC requires, that is if a cloud application is replicated in edges and the traffic is going through the edge, we want the traffic to be, even though traffic is just tied to the cloud application instance, if the copy is locally available in that edge, we really want that traffic to be steered through that local application. So that aspect of it, we are seriously considering at this time. Thank you. Thank you, Shini. I think we have also another question. I don't know if you can still answer. Today the RAN, today the RAN is, ecosystem is containing PFS at VNF, like DU, CU. Any plan to introduce CNF for RAN in the future? So I will give a quick response. ONAP and ORAN SC are tightly coupled. If you look at the service management orchestrator, you will see that the ORAN SC rely a lot on the ONAP architecture. Mainly the most component of ONAP are part of the service management orchestrator. And therefore there is a close relationship with the ORAN SC open source community to ensure that we have a defined whole map that you are building together. So currently in previous release, we have integrated adapter like A1, but also O1, we are working on O2. And I guess when the ORAN SC community will start to tackle this, we will continue the relationship and the partnership. So it's a quick answer. And we would like to invite you to the Slack channels if you want to have more details, because I'm not sure we have still a lot to left for today. Any other words from the panelists before we go? No. So thank you so much for all of you for attending our session and hope to speak to you soon. Bye-bye. Thank you.


Okay, hello, we survived some technical difficulties already, so that's good for resilience. So she just explained that I was going to talk about making better mistakes tomorrow, and I wanted to start with maybe you're wondering who I am, but I think she sort of gave some of that away, but I'll go through it just really quickly anyway. So who am I? So I'm originally from here, California, this San Diego, this is literally what it looks like every single day, but then I ended up living here now, and that's literally what it looks like sometimes. And I know this doesn't talk about making better mistakes and failures, but I don't consider this to be one of them actually, even though you all might. So what was it that brought me to Sweden? And that would be Spotify. So for those of you who don't know, Spotify is a streaming music service, so whether you're on your desktop, in your web browser, on your phone or your tablet, you can log into the app and stream a song, one of over 30 million songs. So Spotify, we have over 40 million people on paid subscription plans, that's as of last month. We have over 100 million active users, which means that once a month they log into Spotify and play a song, so that's 100 million people as of June 2016. We paid over five billion dollars, US dollars, to rights holders since we started, and there are over two billion playlists on Spotify, and we're available in 60 countries, including Singapore. So what do I do in Spotify? So I started as a software engineer, so I was working on data infrastructure, and it was my first time working with Agile, and I really liked it. I liked the approach of breaking things down into quite nice pieces that were much easier to attack and digest and get feedback completely. And I like the Agile processes. I really like retrospectives and the ability to constantly be looking at and questioning the way that we're working and how we're doing it better. So when we lost our Agile coach and switched to another team, I didn't want my squad, which is what we call things in Spotify, to lose that Agile influence. So I took on more and more of that role within my squad, so I was doing development work, but I was also retrospectives and facilitating stand-ups and such. And over time, I started to take on more of that work, and a little bit more and a little bit more, and I ended up liking it so much more that I switched to being an Agile coach. So I was working in infrastructure with security teams and a site reliability engineering team, and then I switched them to payments. And now I'm an Agile coach and also an interim mobile chaplain, which means that I'm an engineering manager for mobile engineers and payments. And I think what this underscores is that if we didn't have this culture at Spotify where it was okay to be a total beginner, right, like my background was a back in engineering and now I'm a mobile engineering chaplain. So if I didn't feel safe looking like a total beginner asking questions that some of you might consider really dumb questions, I think a lot of the stuff that makes me feel so great would fail. So I think this really points to that. So we've all witnessed the downfall of companies that were once great. I don't have to name names. I'm sure a bunch come to mind when you think about it. And this is often due to companies not keeping up with changing market conditions, whether it's changing demographics or failing to capitalize on a new technology. These companies that were once great, these are great types that can never be taken down as some of them aren't even in existence today. And a common theme there is the failure to innovate in a variety of dimensions, whatever it is that it always comes back to, not keeping up with the trends, not taking risks and not looking as the market moves. So I think a lot of, I don't know if I've ever seen this before, but this is one of my favorite graphics. So when we're on the outside looking into these companies which are great successes, they're black boxes to us in a lot of ways. We only really see the highlights real. We see, we hear about where they started and we see all the successes they had along the way. And we don't see the failures or sometimes when they've had to swerve left and then come back to the right and something they thought was living in a workout didn't work out. So we think success, at least on the outside, what people think it looks like is just a straight line for me to be, we know where we're going from day one and we persevere and we get there and we're in reality, we're learning along the way and not always going directly towards the goals of the speed. So we need to make sure that we leave enough work over it, not only in terms of time to develop things, because if anyone who is an engineering here goes, how long do you think something's going to take versus when you actually start digging into the components of how it's working today, ends up taking a lot more time than the things. And also like leaving a wiggle room, not just in terms of time but in terms of the creative process. So being open to new ideas, new influences, being really perceptive to learning new things as they pop up. So part of this comes back to like the tolerance for failure. And does anyone recognize this? Does anyone know what it is? Yeah, exactly. So Dr. Alexander Fleming was doing research on how to define the antibiotic. And he went away on vacation for a few weeks and when he came back, he noticed that this is what he showed, like there was some foreign mold growing in it. And instead of saying, oh, now my experiment that I've been waiting weeks for, the results for has been contaminated. And he took a more inquisitive approach and he said, okay, what is this stuff actually? What's going on here? And when he started taking a closer look, he noticed that the bacteria, even though it was growing in this region of it, anything near the penicillin or this mystery mold, whatever it was, the bacteria was actually dying. And so by having an open mind and taking an inquisitive approach and instead of just throwing out his research just because it wasn't going exactly according to what he had planned, he was able to discover that this penicillin actually could kill the bacteria. And as a result, it was the world's antibiotic, only one of the foremost innovations in medicine of the 20th century. So he says, when I woke up just after dawn on September 28, 1928, I certainly didn't plan to revolutionize on medicine by discovering the world's first antibiotic, the bacteria, but I guess that was exactly what I did. So in many ways, this underscores this mentality of trying to have a learning culture of being adaptive to failure and not thinking, okay, just because I'm not where I wanted to be starting out, what am I learning along the way and how can that help me get faster to a point of success. So again, one of the foremost medical innovations in the 20th century saved lots and lots of lives. So this mindset, how can we make it core to the fabric of our world's working culture? It's really easy for me to stand up here and say, oh, this is really important, we should all appreciate it. It's a very easy thing to do, and practicing can be much harder. So I call it the learning culture. When I talked about reversing, I just talked originally, I talked a lot about failure, oh, it's okay to fail, it's thought by it, but I think phrasing it more as a learning culture is much more accurate, and it's not just about wanting to fail, it's about wanting to learn everything even from our successes. So it's always about being observant and trying to pick up the things we can along the way. So of organizations that have a high impact learning culture, they're 46 percent more likely to be first to market, 34 percent have better response to customer needs, and 37 percent greater employee productivity. So maybe sign us a little bit ironic is the ones that actually are failing more or are leaving more time and don't know exactly where they're going, even though in scenario A, if you know exactly where you're going or where you think you're supposed to be, you might get there quicker, but it's the ones who take time and allow learning process to blossom that end up prospering more. So while the old mentality might have been how can we avoid making mistakes? Mistakes are ultimately inevitable, and they're really fundamental to putting forth the best product that we can, trying to think about in terms of how can we make better mistakes, so not just in terms of product things, but how can we look faster? Even if we are learning and moving towards the right direction, how can we learn even faster? Are we innovating enough? Are we growing people also? Are we giving people the opportunity to grow from an engineer into maybe a product owner or an engineering manager? Because that's also part of it, it's not just about the product, it's about the people also. So at Spotify we call it a think it, build it, ship it, tweak it mindset. So basically what this means is, so we're thinking of a pizza, let's say, so we go, we talk to a lot of stakeholders, we try and, we talk to customers, we talk to other people internally and try and understand, okay, what is it that actually should be part of this product? And once we've gathered all those requirements and have talked about it and challenged it, then we build something. And then we ship it and from there we tweak it. So we're gathering data all the time and then based on how users are interacting with the future, we might, you know, see, okay, people are really like, in this aspect, how do we replicate this or, oh, this really isn't working, like, let's come back and rethink it. And so this is a very different approach from saying we have the perfect idea and this is it and this is what we're going to do, so spending months and months building out the perfect product. It's much more about saying, okay, let's try this, what can we learn from it? What can we, how can we make it work? Let's gather data and see if our intentions are right. So with this mindset for rating the culture, it makes it so that, you know, people aren't afraid to be wrong because even from the get-go, it's not about having the perfect thing and it's not about being right, it's about how can we as a team tackle this problem and bring forth what will be the best for the user in the test form. This also isn't the hit-go, I don't know, for that term, the highest paid person's opinion. So this is also saying that, oh, just because this person's the director of product, that whatever they say goes, it's also, you know, they also have to, to be able to pull the data from things, the products that they push forward and if those aren't working, it doesn't matter if you're the CEO or an engineer, we can go by the data. So an example of that, this is our old Discover feature and it basically gives you suggestions. So based on your listening history, sometimes it will show you songs that you've had listened to in a while that maybe you kind of missed listening to and what your friends listened to also. Yeah, so this, like the algorithms that were powering this, right, like they were fundamentally very strong but the problem is that this feature was hidden so deeply into the product that A, it was hard to find and also it was very much like a lean approach.  So that's another factor kind of against it. So even though the people who did interact with it really love the recommendations, there were other factors that just, that made it not a perfect fit. Does anyone know what Discover Weekly is? Does anyone listen to their Discover Weekly? One person, okay, two people, great. So Discover Weekly is basically, we've taken the Discover algorithms that we had and using the think it, build it, ship it, tweak it mentality, we've enabled to turn it into something that people love or at least people don't love it. But what it is is that we've taken the same algorithms and instead of having the user first try and find the page that the feature was on and have to interact with it, every Monday, so once a week, they get 30 songs that are based on their listening history and songs that people they'll like and then obviously like as we see what they listen to, that they'll grow and change with that. And this has been one of our recent like hit features and I personally really like it. So it's taking it away from being like someone really having to lean in and interact to the product to kind of like to the more sit back experience where it comes to you and it's a it's on a weekly cadence so you know to expect it etc. But how did we get there? So this is actually, this is started in 2014 but I took the screen tour from 2015. So we did something called Year of Music and so for everyone, for every Spotify user they can go onto this website and at the end of the year and they can see okay what were their top ten artists and what were their top genres and what were their favorite songs that they listened to etc. So it was really cool to at the end of the day take a look back and see what it was you were listening to. We also decided to give them a little something extra, something to help play it forward. So this was built on top of the Discover, the algorithm that we're powering the Discover feature and it basically was kind of a test of this playlist concept and people loved to play it forward. The engagement with it was really great so we realized that okay maybe we're right maybe the the actual song suggestions that we're making are fundamentally good but it's the way that we are presenting with people that that's flawed and so by taking this by testing things out a little bit and taking this Discover feature and a playlist concept we were able to turn it into Discover Weekly which I can highly recommend checking out if you're a Spotify user and so again think it build it should be a tweak it mentality and if you want to know more about the Discover Weekly in particular I know these are I don't expect that you'll memorize these links but that would be pretty impressive if you could. I'll have these slides up online also but you could go to that spotify.com or search on the slide channel for Discover Weekly if you want to know more about the the product side of things. As an extra bonus we have something similar to Discover Weekly which is called Release Raiders so basically using your interests and the new songs that are being released to curate the playlist for you of like all the new songs that are coming out which I really like so yeah that's a learning culture and practice but how do we again how do we actually get there right what are some of the tactics that we use. So one is retrospective. This is an agile conference so I imagine everyone in the room is at least familiar with the retrospective and something I always really like is this process of having to be the cadence of looking back and saying okay how can we make this better and how can we improve what we're doing. There are lots of great resources online for retrospective so I won't go into too much here but that is definitely one of the main tools we use to get into that like learning culture mindset. Another big thing these are postmortems. So postmortems are more was anyone here running postmortems? So these are more for like system failure so this is an example of when the Hadoop research manager was slow and what we do is we notice that there's either a failure or a problem and we have an incident ticket is created and after that we'll have a postmortem which is sort of like a retrospective of focusing specifically on the timeline of okay what went wrong and how can we improve it and we walk away with some specific remediations. We try and focus on okay what are just a few actionable ones you can think they're definitely so yeah we try and focus on on getting just a few remediations down but this is another way that we try to keep the learning culture alive and well. So some of the goals of postmortems are obviously figuring out what happened, meeting things with stakeholders, how we can learn from it but but nowhere in here will you see okay we're having a speedy to figure out who to blame this is it's not a blame thing at all it's about people no matter how it happens about people coming together to figure out how to make sure this doesn't happen again because it's okay for mistakes to happen it's okay for us to fail in things but we just we just want to make sure we don't have the same failure twice so the goal of the postmortem is to make sure we never have mistakes but to make sure that we have a different mistake next time for the next postmortem. But there are many ways to learn from failure after the fact and I think while these two while postmortems and retrospectives are two great ways to get into the habit of doing this and to establish a cadence around it how can we actually make sure we get to the point where people feel safe being in a meeting with the head of their engineering department and telling them no this like I don't agree with with the way we did this or this was wrong or or like feeling empowered to say those kind of things so now I have my guide and build a learning culture in three systems and maybe a few harnesses but and so I've boiled it down basically to filtering reinforcing and adjusting and reiterating so those are sort of the main themes of how do you actually get to this point where where you have people learning from things people aren't afraid to make mistakes and they aren't afraid to innovate and try something new so I'll walk through these and give some concrete examples of things that we do and we'll start first with filtering so the first thing we do is culture interview who here does culture interviews with their companies one of some people and who here doesn't like raising their hands okay so culture interview is all about getting the right people in the building and the right people for solidified it would be some are very different from the right people for another company so it's a lot about knowing okay what are our core values here and not just the ones that we say are core but the ones that we are like living every single day um some of our engineering principles are what are our people principles of cultural beliefs are what you can become matters more than what you are so for us it's not always about hiring someone who might have 20 years of experience as a sediment it's it's more often than not we really hire someone who really is always wants to continuously improve is is not afraid to fail and evades the team players some of those things are pretty important to us and maybe they don't have all the experience but we take them in and we say okay we want to work with you and we want to grow you into becoming like the person that you can be so it's it's and especially in like a highly competitive hiring market stage or marketplace again like in tech it can be hard to find people with all the experience that that you want so we definitely take a more like growing people approach but we we take people on once we have have assured that they kind of have the same cultural like center of mind that we have about not being afraid to fail about that you know they don't we're really big on like no ego so we don't you know we're not always looking for singers stars we're looking for Korean people obviously but we want to build rock star teams that's kind of the priority first so so that's something that we filter for another thing whoever learns the fastest will win so it's about like a lot about like being able to learn from failure and we try to make sure that who come in as I said before people who are okay with this so a lot of if you can get this part down like getting the right people in the building then a lot of the rest of stuff especially when it comes to management is much easier so the first step in order to build a learning culture is to filter for people who sort of have some of their beliefs or who who aren't afraid of this so step two feedback and I'm I call something Spotify bingo if I hear it all the time it works so it makes like kind of my own inside joke with something that you'll hear a lot of that Spotify has given them that feedback so I'm sure you probably have it at your own company too like things that you hear people say a lot so something that so let's say even if it's like oh you're doing a really great job you tell that to maybe a man that they'll say okay but have you given them that feedback they're not they're kind of training us not to funnel the feedback into one person but to always be telling people when we came in did a great job and when we think something could have been better so I think that's one thing that helps with keeping a daily practice of like making sure that people are giving feedback because it's one thing to get the right people in the building but no one's ever going to be going to be perfectly in line with like what the cultural principles at your company are so it's a lot a lot about like reinforcing a behavior and also adjusting maybe when someone you know you notice that they're really staying in their safe zone they're not really taking taking risks you know that's another that's an opportunity to try and adjust them you get them to take on a bigger a bigger risk another process we have is loose which is a more formal feedback so you have achievements on one axis and behavior on the other and something some things that we try and think about like some of the some of the questions that come to mind when we're giving this this 6 to 12 months feedback is like do you see evidence that they have ambition to continuously grow and develop both as a person and as a professional so trying to get people who are giving feedback to people to think about is this person challenging themselves are they continuously improving both their technical skills but also as people you know maybe I don't know maybe they are QA and QA and they are trying to push them into more of a product role to try and understand that perspective just to get another thing about seeking opportunities to grow and do they undertake challenges that they have never taken before etc etc so trying to get people when they're giving feedback to think about some of these things so that we can all kind of stay you know be singing the same tune so to speak so this is again about taking taking an opportunity every six to twelve months to say okay where do you see that this person is doing a great job in terms of learning from from mistakes and taking risks and like making sure that people reinforce that behavior and also getting getting the people around to think about okay but where is the person doing an even better job so step one get the get the right people in the building step two make sure you're helping them grow along the along those lines and the next one is modeling behavior and this goes for both like and near all the way up to like your cto and your CEO so most human behaviors learn operationally through modeling so Alba Pandoran he's a he's a researcher and he did an experiment called the Bogo doll experiment and when he did was and took some young boys and girls and he and he had them watch these these people go into a room and some of the people they would be neutral and they would just you know kind of look at this all bit of her some people would go in and they would be really friendly and helpful very very like non aggressive behavior and some people would go in and hit it and shoot things at it and all the other things so like exhibiting really aggressive behavior and then after that would happen was they sent the kids into the room to play with the doll and what they noticed is that the ones who had seen really aggressive behavior were more likely to repeat that aggressive behavior themselves when they were playing with the doll and the ones who had seen kind of non aggressive behavior when they went into the room were more likely to not exhibit aggressive behavior so they were more on the non aggressive side so what what the undersource is that we really do learn a lot like the behavior around us from what other people are doing so if you want a culture in which it's okay to failure it's okay to fail if you don't want like something to be like a hero called for built around he goes and it's important that everyone's living up to that whether you know no matter where you are in the organization so this is an example of something happening bottom up so this is the fail wall that that you can find in the office and what it is is that people in a squad they said okay like let's celebrate failure let's not be afraid of it so whenever they would stand up and they were you know if something didn't go right they wouldn't put it up here I think one that I saw once was I failed because I thought failure was a bad thing that was stuck up to this wall so this is an example of just like a daily type thing that a team was doing to make it make it an ax river was okay to mess up or to look back which can be scary when you're new to a company and you're trying to figure out how it works so so this yeah again something coming from the squad level up and it also comes the other way down right it's a reinforcement repetition so Oscar school or CTO sent out emails but also a lot of posts about how we should be celebrating failures and how you know even noticing okay not not only pointing out the wins that we've had but also saying okay these things didn't go our way but it's really great that that we're taking these risks that we're trying what are all the things we can learn from it so again more of a top down type approach to be modeling this behavior and this one's my favorite so a madness it's your first day in the job first month and you've just very very publicly broken spiral settings here needs to accompany you don't really know how how things work you kind of got a little bit of a sense you made right a little bit but you end up taking down all the servers in London what happens next oh you might first mess up so and if I redeem I might be a little bit scared then other people chime in burn him burn the witch correct welcome likely the most mild first message that's thought of anyone in this channel I think I wrote dns for Spotify dot com insurance says I broke Spotify dot com and what's interesting here is these aren't only engineers nrh Nick he is actually the tribe leaders because he's the head of the engineering department and he's kind of the first one to respond and it's something that's totally organic so he's responding to a new engineer obviously like a little bit joking at first but then saying oh that's that's nothing what about the time when I did this so as the new engineer you know it feels like okay maybe it's not so bad maybe it's not so scary maybe he's a he's an engineer manager and he's also chiming in sure when he was an engineer so you see that it's this is very organic thing across the entire level of the company and then more people chime in David who was working products welcome after this is when Spotify adds you to the monthly payroll and the conversation goes on and on no one breaks DNS like I do I might agree but I bet no one here overwrote the entire ash dot Spotify dot net phones long to records except me and again now it's kind of like a competition back and forth fireballing on all authority the slaves started competition and bring talk to care about his epic outage and so it's kind of moved away from even just what happened with thema to be sort of you know fun competition about who had actually made the worst mistake and fredin says I liked it when I was slightly sloppy with cash headers on s3 items which made clients hammer cloud front so much that mfade who's the chief architect kept running down pretty much yelling stop the prices because amazon called the surprisingly many digit dollar number for a future bill nobody deducted it for my salary which I was happy about yes start sweating so as you can see it's not just it's not to talk to anything but it's also a walk to walk and you see that because we've because we've filtered because we have culture interviews we know what's important to us we're able to get people with this sort of mindset in the building they're able to reinforce and adjust it's always called call this conversation of armor feedback trying to make sure people understand that they should be taking risks both in their personal development and technically and then also as you see modeling behavior and reiterating okay like failure is okay we want to learn things we want to take risks any questions what would happen is what would happen if you would repeat the same mistake three or four times that's definitely happened also and and if we still we still take the same same approach of trying to attack the problem not not the person usually so if it's possible to make the same mistake many times it may get something like in our in our systems or processes that are going wrong but yeah we would definitely try and dig deeper into that but it doesn't just you know it's not like oh you can make one mistake and everything is finding on the way to you but if you make a second mistake no it's not like that i was going to ask you as far as earlier you said you know trying to fail do you mean that's what makes your area do you mean trying to succeed learning from failure or are you really being trying to fail no yeah i mean trying to succeed in learning from failure um but i i would say that there are examples of like trying to push people out of their comfort zone right like uh in personal development i mean we're what we do have like some sort of like failover test such that we do but uh so that would be trying to fail but uh we you know maybe we see someone who's who's working in their role and you know they can go to their stretch zone or they can go to their like super stretch zone sometimes it's worth trying to give people opportunities that are way out of their comfort zone so that they can say okay maybe that was too much for me but then they end up sort of an unstretched zone so it's not that you want to set someone up to failure but to fail but but it's it's not like an outcome so what was your biggest failure what was my biggest failure um i i had a lot of failures so how are learnings and failures shared with other types of skills and one thing we do is if we have like a company-wide uh project so or maybe just a really big project so for example when we launched on the play station we had like a series of project perspectives after that so we would do retrospectives with a lot of the people and teams who were involved and then we would bevel that up to something bigger and something bigger so we do have some process like some sort of process in place for trying to learn bigger failures um and i would say like the guild is another good place where this type of learning happens so uh in the agile guild for example we have a lunch and learn on mondays and people can come and present something or maybe they can just talk about it so uh i would actually say the guild is a tool for sharing some of those learnings we also have like people send emails or their blog posts or sometimes we have internal conferences so maybe we'll have a mobile on conference or uh uh back in our conference and that's when a lot of sharing goes on also uh and i think with especially the agile guild because you have coaches in it and they're working with a lot of teams uh there's a lot of sharing going in there do you find the cultural filter to be easier in certain countries compared to others or is or is it generally the same um in in terms of Spotify specific yes all right yeah um so we are engineering offices are the US and Sweden so we have Stockholm Gothenburg uh Boston and New York and San Francisco um i i would say it hasn't i i wouldn't say it's been so hard um i think one thing that helps is that we're a Swedish company and i think that a lot of these concepts about collaboration and not not being super hierarchical and kind of like everyone uh being an important contributor i would say that's something that at least to me feels very Swedish so i think being a Swedish company and having that like kind of fundamental culture from the bottom up has definitely been a big influence has been super helpful uh but i but i also see people who are hired in the US share some of these values um and if anything having their ghost thicker than in Sweden so i'm not i'm not really sure that i've seen it but but yeah no i'm not doing the US and Sweden anyway but maybe somewhere else but we don't have engineering offices there so i haven't seen any last questions i guess my failure for now is that i can't answer your questions so i'll come back to something okay thank you so much and um yeah my slides are are online at my at my website it's just firstnamelastname.com and if you have questions they'll be around today and tomorrow so just um ask thank you