410,578. Does anybody know what that number represents? That is the number of tech employees that have been laid off in the last two years. Now, the media has a narrative of what building a company is like. And it typically starts with, I read something on Twitter. I had an idea and had a dream about it. I talked to a few folks. And had a billion dollar exit. But it turns out building a company is a bit more complicated than that. And that's especially true if you've been building over the course of the last three years. And it's supposed to be that way. And it's supposed to be that way because if the skies were always blue and the waters were always calm, you'd panic at the first sign of adversity. And you'd never reach your full potential. So my name is Nolan Church and I'm here to talk to you about layoffs. I was at Google for a few years working on leadership recruiting and confidential projects. I joined DoorDash as the 50th employee and the head of talent. I was at CARTA as the chief people officer through a thousand employees. And I'm now the co-founder and CEO of Continuum. Continuum is a talent marketplace for executives and we're series A. I ran layoffs at CARTA and I've helped more than a dozen founders navigate layoffs over the course of the last two years. And I've certainly learned that running layoffs the wrong way can mean negative consequences for a company. But there's a narrative that exists that if you run a layoff that your company is about to die. But I'm here to tell you that narrative is false. And in fact, if you run layoffs the right way not only can your company survive a downturn or a challenging moment employee productivity and happiness can increase. So over the course of the next 20 minutes I'm gonna tell you three stories. One about layoffs gone wrong the CEO was publicly humiliated and the company is barely alive. And two about layoffs going well to where the company healed and now they're growing faster than ever.  But regardless if you run it well or you run it poorly as my VP of HR at CARTA said people will remember the day for the rest of their lives. So it's 4.30 PM we've all had a long day. Humor me everybody please stand up on your feet. Stay on your feet if you were involved in a layoff or were laid off in the last two years.  Remain standing if you thought the layoff treated impacted employees with respect and with dignity. Keep standing if after the layoff employees move faster and were more productive. Take a look around. That's because it's incredibly challenging to run layoffs the right way. And we're gonna learn about that in a minute. Please sit down. All right let's get into the first story. And it starts on February 24th, 2020 at my house in Salt Lake City. I was heading to San Francisco for a couple of executive meetings at CARTA. And I was just about to leave. I'm kissing my wife. She was six months pregnant with our baby girl and she grabs me by the arm. She goes, Nolan, I'm worried about this coronavirus thing coming out of China. Please be careful. And that was surprising because my wife is hard. She does not scare easy. But it turned out she was right to be worried because by the end of that day the Dow had dropped 1,000 points. And on Tuesday the CDC came out and they expected significant disruption to the American economy.  Also on Wednesday a few CARTA employees decided to work from home and more were asking their managers to work from home for the rest of the week. So in an abundance of caution we were one of the first tech companies to close our offices that Friday, February 28th. We thought people would be back in the office in a couple of weeks, but it turned out people didn't come back for more than a year. So fast forward, it's March 6th, Friday, March 6th. I'm at home, I'm trying to figure out my Zoom background. I'm like trying to get my home office set up. You guys remember that BBC interview where the family ran in on that guy? That was me. It wasn't me, but it might as well have been me. So again, I'm at CARTA and CARTA is a bellwether for the startup ecosystem. And the reason why is because we run cap tables and the majority of startups run their cap table through CARTA. And we had started to see our new business slow down. And so Henry, CARTA's founder, had sent me a Slack. And he said that we needed to look at layoffs. And at this point, I thought Henry was overreacting. I thought he was being emotional. I thought he was moving too fast, but it turned out Henry was right and I was wrong. That day we got my people leadership team together and the entire executive team together and we began holding daily meetings. We were gonna run the first layoff of my career. 10 days later, it became clear that COVID was gonna impact everything. This is actually a photo for my camera roll. People were having a really hard time and my neighbors were trying to encourage each other. Three days after that, I went to the grocery store and the entire store was sold out of paper towels and toilet paper. But like any good entrepreneur, I used tech to solve a crisis and I bought up a day. It's still my favorite purchase of the last five years. All right, so before we go forward, I wanna talk about why almost half a million people in tech have been laid off over the course of the last two years. And that story starts in 2010. So in 2010, post great financial crisis, the Fed had lowered their rates to almost zero. Now they began hiking again in 2016, but they only got to about 2.5% before cutting in order to stimulate for COVID. Also during this time, headcounts at tech companies started to explode. If you look early in the decade, headcounts growing fast, but if you look later in the decade, it almost looks like it's growing exponentially. So this is meta and if you look at 2018 to 2022, they went from about 35,000 to almost 85,000 employees. A similar story at Salesforce. From 2018 to 2022, they went from about 30,000 to about 70,000 employees. And similarly at Google, they went from about 100,000 to almost 200,000 employees. So headcounts at these tech companies are exploding. And so are valuations. In 2015, there were only about 250 companies worth a billion dollars. By 2018, about 250 companies per year became valued at a billion dollars. Then in 2021, 787 companies became worth a billion dollars. So headcounts and valuations are exploding. And if you're a tech company, it's kind of like you're at the club.  Drinks are flowing and all you can think about is being in the moment. Biggie Smalls once said, mow money, mow problems. But I think if Biggie was watching these tech companies get drunk on hiring, I think he would have actually said, mow headcount, mow problems. And the reason why is because the chart below, which reflects Brooke's law. And Brooke's law states that for every person you add to a project, it actually moves slower, not faster. And that leads to my biggest learning being an HR for the last 13 years. Headcount is a vanity metric. It is not a sign of success. In fact, the only sign of success is turning your business into a money printer. Such that for every dollar of investment you put in, you get two, three, or four dollars of revenue out. And so that's the story of what happened in tech up until the middle of 2021, when inflation started to spike. Now, the Fed was hoping that it would be transitory, but it turned out to be persistent. And by November of 21, we had the fastest rate of inflation growth in more than three decades. And the Fed said we might have to raise rates a couple of times in order to respond. And boy, they were not kidding. Since March of last year, the Fed has increased its federal funds rate 11 times and 525 basis points. And when interest rates go that high, investors care less about growth and more about profitability. So remember those tech companies that were at the club having the time of their life? It's now 7 a.m., their Mali has worn off and they have to deal with the reality of a new day. And that new day is still the reality we're in today, where we're seeing more layoffs than at any point in time during the pandemic. And here's the thing that nobody wants to hear. It's going to continue into next year and potentially beyond. Okay, a lot of layoffs over the course of the last two years. Do any stick out to you? Two stick out to me. On March 5th of 2020, Airbnb ran a layoff and that makes sense. They're in the travel industry and during COVID their business had essentially stopped. They laid off 2,000 people. But what surprised me was how the media responded. They called it a case study and how leaders can run layoffs. One reporter called Brian, Airbnb CEO, empathetic and transparent right after he had fired 2,000 people. Now on the other hand, there's Vishal Garg. Does anybody remember by a show of hands Vishal? Vishal was the CEO of better.com and they also ran a layoff in December of 2021. Except this time, the media crushed Vishal. They called him cold and that it was a tragic example of leadership. So what happened here? Why was Brian at Airbnb heralded and why did Vishal get crushed? The media wants you to believe it went so poorly at better.com because they ran their layoff on Zoom. But that's false. If you live in a post-pandemic world and you have to do a layoff, of course you have to do it on Zoom. Now I'm not gonna make you watch the entire video because it's quite painful but I will break down what happened. The first thing is that Vishal was clearly unprepared. It looked like he was winging it or going way off script. And because that's true, his communication came off. We actually have a term for it in tech.  He led off by making the event about himself. He then later referred to impacted employees as unlucky and when he had time to process and think about it, he then took to the social media app called Blind where he accused impacted employees of stealing from the company and from its customers. Overall, the better.com layoff did not treat impacted employees with respect or with dignity. Many of the impacted employees said they had trouble accessing their severance, their health benefits and couldn't get in touch with betters HR team. I can't imagine what these people were going through right as the tech recession was taking place right before the holidays. Now, taking a step back, what makes the better.com layoffs even more egregious is that they happened 18 months after Airbnb gave the playbook for everybody to follow. Now, people forget this but CARTA actually ran their layoff before Airbnb and Airbnb followed a lot of their strategies and tactics from what we did at CARTA. And now I'm gonna tell you that story. But before I do, I wanna tell you a bit about Henry Ward. He's CARTA's founder and CEO and I'm pretty sure he has a crystal ball because he has an incredible knack for being right during company defining moments.  He doesn't believe in playbooks.  So I get the slack from Henry and every day I'm meeting with the executive team and my people leadership team. Henry gave us a target to cut 16% of the company and we got to work. And there's a typical playbook that most companies follow when they run a layoff. And it starts with getting smart. And the way that you get smart is you update all of your financial models assuming the worst case projections and you also talk to advisors and leaders at other companies who have run layoffs before. You wanna take from them what worked well and leave what didn't. From there, you're gonna wanna cut all of your non-headcount expenses but you're gonna find that it won't get you too far because typically people expenses account for 70 to 80% of a company's OPEX.  It's important at this stage to loop in employment legal and to design a diversity impact analysis. I like to bring in employment legal at this time to dot eyes, cross T's and to limit company liability. About a day or two before the event you're gonna wanna prepare and train your managers. Now it's important that you move quickly at this stage because if you don't rumors will get out. On the day of the event, the CEO will send a company-wide email or call in all hands. And as we learned from Vishal, it's critical that the CEO is prepared and practices. You're gonna impact the ex-employees, the impacted employees will be exited and their severance will be paid. And then usually companies give it a few days to let the team process and then reset them on the go forward goals. And so that's the general playbook for layoffs and that's what we ran at CARDA. But we also followed two principles from Henry that most companies don't. The first is to be humane and treat impacted employees with respect and dignity. And the second is to cut too deep. That's right, I said too deep and I'll explain more what I mean in a minute. So the first step in being humane is designing as generous of a severance package as possible. We gave all employees three months of pay, extended their healthcare until the end of the year given that we were in a global pandemic and also extended their post-terpanation exercise window so employees had time to exercise their shares. We also decided to leave Slack on so impacted employees could say goodbye to their friends and colleagues. Now, most companies don't do this because they're scared that an impacted employee is going to go on to a public Slack channel and talk negatively about the company. We didn't see that at all. We thought it was critical that every impacted employee have a direct conversation with their manager. This was an emotional and logistical challenge given that some managers had 10 impacted employees on their teams and had to be in these conversations all day long. And finally, we deployed our recruiting team and our managers to help these folks update their resumes and find new jobs. Now, as I mentioned, our other principle from Henry was to cut too deep. And the reason why is this slide from Sequoia Capital. It shows two companies that are going through a hardship. Company A, which is the yellow line, decides to wait to run layoffs. And then when they do make cuts, they don't go deep enough and have to run multiple layoff events.  Company B moves quickly and cuts deeply. They're able to extend their runway and to survive the hardship. So Henry's meeting every day with our managers and as I mentioned, we had a target of 16% of the company and some of these managers thought that we were going too deep. And to that, Henry said, we can always hire people back. And if that sounds crazy, you're not alone. I thought it was too. But it turned out to be a very helpful framework to managers who thought we were going too deep. And in fact, we actually did hire a number of people back over the coming weeks and more into the summer, all of which received severance in their jobs back. Henry also said, we want people who want to be here. And so similarly to Brian Armstrong at Coinbase and Tony Shade Zappos, we decided to allow all employees the opportunity to take the severance that we were giving to the folks that were being impacted by the layoff. And again, I thought this was crazy, but again, I was wrong and Henry was right. By giving everybody the opportunity to opt out of CARTA, those that stayed were intentionally opting in to another tour of duty. So the day before the event, we're training our managers and our VP of HR said something I'll never forget. She said, people will remember this day for the rest of their lives. And it's powerful because it's true. So that night I go home and I'm dealing with the most anxiety I've ever experienced. I was worried about our business continuity.  I was worried about our team and our managers. Could they handle the emotional toll? I was worried about the company and our ability to move forward. Would we ever grow again? I didn't sleep at all that night. The next day we called the all hands and Henry informed the company. We ended up laying off 161 people. Now, anybody who runs layoffs should read Henry's entire blog post because it's a case study in internal comms. But I wanna read one line to you because it's an example of Henry taking full ownership as CEO. He said, it's important that all of you know I personally reviewed every list and every person. If you're one of those affected, it's because I decided it. Your manager did not. It was a brutal day. My team was deeply impacted. We ended up laying off almost half of the team about 30 people. I was in these conversations all day and into the evening. And in my last conversation of the night, I was talking to a recruiter who had just lost her job. And she said, Nolan, I'm sure today was brutal for you too. Thank you for treating me with dignity and respect. Okay, so what were our core learnings? Let me first start with what I wish we could take back and it's one thing. About two weeks before the event, we held in all hands and we told the company we were considering doing layoffs. Now, our goal was to be transparent, but unfortunately, we just scared everybody. It wasn't productive and I wouldn't do it again. On the other hand, we got a lot right. And a lot of what we got right was counterintuitive. The first is that cutting too deep is necessary. And the reason why is because of that Sequoia chart. If you don't go deep, you won't be able to survive. The second is by giving everybody the opportunity to opt in, by giving everybody the opportunity to take the severance, those that opted in wanted to be at CARTA. It helped the team get productive and then get working again. We also tucked in a few of our top performers with salary and equity increases. This was only on a select case and we factored all of these increases into our go forward models. It ensured that critical members of the team didn't leave. I learned that the company was more resilient than I expected, but I think a big reason why is we treated all employees with respect and dignity. And that respect was reciprocated not only by them, but also by the people that stayed. So as you can see, running layoffs is a complex process. But after the layoffs at both CARTA and Airbnb, both companies began growing again and both companies healed. To run a layoff the right way, it takes bold action. It takes preparedness. It takes CEOs who own everything. But regardless of how the layoff goes, people will remember the day for the rest of their lives. Thank you.


A recent study examined the psychological effects of these measures on people's mental health, finding that they can lead to feelings of isolation, anxiety, and even depression. Misinformation and conspiracies have been rife, further exacerbating the situation. This article was authored by Janie Leung, Mariana Schultz, Vivian Chiou, and others.


This is stock number two one T one zero one We are here at Summit Automotive in Fond du Lac, Wisconsin. You're new and used RAM and heavy-duty truck headquarters today We are checking out this brand new 2021 RAM 2,500 crew cab short box This truck has the tradesman level two Package on it. It also has the 6.4 liter V8 Hemi motor And it also has the chrome appearance group Patriot blue metallic is the color we shoot all of our videos in 1080p 60 frames per second So if you have HD capabilities on your computer Tablet or smartphone device turn them on right now because it is definitely your best way to check out the quality the looks and styling and options of the vehicle and Color before seeing it in person And if you'd like to check out all the photos on this truck in the upper right hand part of your screen is a link to our website Click that and check us out there Patriot blue metallic one of my favorite colors on the Rams Glad they brought it back this one with the chrome appearance group gives you the 18 inch steel wheels With the chrome covers and it has firestone Transforce AT LT 275 70 R 18 tires. They are definitely an all-terrain tread pattern on there And if you look at that Patriot blue and if I get my Flashlight out here, and you can really see that metal flake in the paint And if you have your HD capabilities on you can really get a good idea of it So when it's really bright out it really shines Get the chromed front bumpers this one has the Technology towing group or the towing technology group which gives you the 360 camera gives you the cargo cam Also gives you the trailer tow cam Just a really neat system, and we'll dive into that in just a little bit Do you get the clearance lights up there? And we'll take a look at the original window sticker here feel free to Pause this as I go through it. I'm not going to go through everything But just some of the main points on this truck So you can see it is a 20 21 Ram 2500 tradesman crew cab 6.4 liter Hemi MDS engine comes with the eight speed automatic transmission everything on your left. There is your standard options And then your optional equipment is the Patriot blue pro code a hundred bucks towing technology group is 25 25 So that gives you the premium black power tow mirrors The center stop lamp with the cargo view camera the mirror mounted auxiliary reverse lamps the gloss black grill with the chrome accents the surround view camera system the blind spot and Tague trailer and cross path detection And the trailer reverse guidance so lots of stuff that comes with that package This one also has the heavy-duty snowplow prep group for 245 Which gives you the transfer skid plates in the 2220 amp alternator Fifth wheel and goose neck towing prep group is 530 chrome appearance group is 995 tradesman level 2 group is 1145 which gives you the 40 20 40 split bench seating the floor covering Carpet front and rear floor mats and power sliding rear window also gives you serious XM for six months It has the electronic shift on the fly transfer case, which is 295 Anti-spin rear differentials 445 instrument panel mounted auxiliary switches is 145 clearance lights 95 bucks LED bed lighting, which is a hundred sixty five rear wheelhouse liners 195 and those off-road tires are 295 comes with the you connect a point for system with Android Auto and Google I'm sorry Yeah, Android Auto And Apple CarPlay Toe hooks are a hundred bucks dual Alternators rated at 380 amps 295 and the trailer brake controller is 295 total MSRP is 51 155 if you want to get your summit price in the upper right-hand party screen is a link to our website To get all the rebates and incentives on this particular truck three or 36 bumper-to-bumper warranty five-year 60 powertrain and they give it a B rating on Environmental performance and five-star side crash ratings three-star rollover not rated for the rest We'll get to the interior in just a little bit. I'm gonna finish the walk around on the outside here I always like to point out that these 2,500s on the Rams come with the coil suspension back here gives you a lot smoother ride when you're unloaded Notice you get the LED tail lamps there, and that's all part of that They don't really say it, but it's part of that cargo lamp system System as the fifth wheel and gooseneck prep package back here you get the seven pin four pin wiring Trailer wiring and then you have the LED bed lighting you can turn those off and on by pressing that button Which they added that in 2018 is an option But pretty nice to be able to turn those on whenever you need them Tailgate is really light You get the chrome ram logo four by four logo so that's all part of the Chrome appearance group as well, and then you get the towing package back here receiver hitch four pin and seven pin wiring So the towing technology group. This is really a big part of it You get the blind spot the LED side lights you get these cameras right here, which when they're It's part of the 360 camera system But also when you're looking down the side of the truck is it kind of gives you a view of your trailer and stuff when you're backing up These mirrors also fold up like so and you do get directional signals and running lights in there inside the tradesmen package gives you the dark charcoal and black cloth interior Like the design on there Factory formats auto headlamps power windows power locks power mirrors now these mirrors also power fold in which is a really nice feature To have on a tow mirror like that Especially on the tradesmen package and then you can control that blind spot mirror You got to press this button, and then you have to press which one you want to control And then if you watch that mirror You can control that To get you know your trailer set up perfectly power windows power locks power mirrors and as we hop inside here We'll turn the ignition on so you can see the instrument cluster and the radio and everything inside here do you get a digital speedometer this one only has 13 miles on it and and Steering wheel you get bluetooth and information center controls on the left cruise controls and your gear selector on the right remember this does have the eight speed automatic transmission in it and Push button start and there is that eight speed Push button four-wheel drive as well and a neutral setting the 8.4 for C radio AM FM and Sirius XM radio capabilities and You can do Your controls here, and this is really the main part of this truck is the backup cam So take note on how high definition that backup camera is really nice backup camera, and then the same thing with the cargo cam It's really nice and clear and then these side views as you're Hauling a trailer or backing it up or a boat or camper you can actually focus in on certain sides So that when you're backing it up you can get the most View possible, so that's pretty cool right there and Then it also has the 360 surround view cam if you look right there You can see actually all of our individual tiles on the floor here in our picture studio, and it's a pretty cool system so Really like that, and then like I said you can do Android Auto or Apple CarPlay on this radio Where you can project your cell phone to the screen? Why is that important? So if you have a navigation system on your cell phone such as ways or Google Maps You can project it to the screen and use it through the screen, and then it's just like you have nav on the radio So even though this radio does not come with nav You can project it there and use it just like you would From your phone, but then it's displayed on the screen We actually did a demo on Android Auto if you want to check that out It's in the upper right hand part of the screen. So click that and check us out up there Here your climate controls volume controls tune controls Factory brake controller you get six upfitter switches tow-haul mode your stability control Keyless entry right there and Then down here you get two USBs two USB C's and an Augs jack Note that you do get that carpeting on the floor there instead of the rubber floors Get a nice storage bin and then a glove box there and then I think Probably the biggest thing that the Tradesmen level 2 gives you is that power sliding your window There's the button for that and then you do get map lights up there as well Let's take a quick look at the back seats and then We'll do one more final walk around with all the lights on So you do get the latch child safety system back here for any child car seats You may want to put back here, and then you do get that power sliding your window with the built-in rear defrost Get storage underneath there comes with a receiver hitch reducer And you get the factory floor mats back here lots of room Side curtain airbags as well and child safety locks on the back doors And as promised, I'll do one more final walk around here with all the lights on You can see those LED tail lamps Really a good-looking truck for a tradesman package You can see those LED side lights on the mirrors here as well, and they are actually super bright But like I said really a good-looking truck with the tradesman chrome appearance group Patriot blue metallic Has the clearance lights LED tail lamps Really a nice truck nicely equipped And I'd personally like to thank you for checking out the video today and hopefully from this HD video You will have gotten a really good look at this 2021 Ram 2500 crew cab short box tradesman level 2 in Patriot blue metallic and Thanks for checking out the video if you want to see more pictures of this truck or one of our other 450 new and used cars trucks SUVs minivans Wranglers half tons three quarter tons one tons you name it We have it go to our website right there www.summitauto.com full pictures and descriptions of every single vehicle from two locations All at summit auto.com and if you like to check out more HD videos You can go to youtube.com slash summit auto remember to like subscribe and share on this video and all the videos that you see there In fact in a second you'll see a link to subscribe to our YouTube channel in the upper left a link to more Ram 2500 truck videos like this one in the upper right a link to this truck on our website in the lower left and a link to our newest video Upload on YouTube in the lower right. So click those check us out and we're super excited to be offering the brand new 2021 Ram 2500 crew cab short box tradesman level 2 in Patriot blue metallic. Thanks again for checking out the video


Mae'r bwysig iawn i'r Cymru yw Toc Miroshik, sydd wedi cael ei wneud o'r Sloveniau. Mae'r Cymru yw'r Sloveniau Llywodraeth Cymru yw Llywodraeth Cymru. Mae'r Cymru wedi cael ei wneud o'r Llywodraeth Cymru yw Llywodraeth Cymru yw Llywodraeth Cymru yw Llywodraeth Cymru yw 27. Mae'r Cymru wedi cael ei wneud o'r Cymru yw Llywodraeth Cymru yw'r Ffarranol Cymru, a'r Ffarranol Cymru yn ymgyrchol. Mae'r Cymru yn enwyl i ymwyl yn Llywodraeth Cymru, Llywodraeth Cymru a'r Cyfrifyddiad Cymru, yn ym 2017. Mae'r cyfrif varyddol yn ymddi, mae'n ymddych chi'n Llywodraeth Llywodraeth Cymru, ac yn ymddi Llywodraeth Cymru, a'r ddweud yma ar y slyfynion, ddweud yma ar gyfer yr EY 27. A'r ddweud o'n gweld ymlaen i'r prysgaf, sy'n mynd i'n rhoi'r gweithio ar y dyfodol yma ar gyfer yr EY 27. A'r ddweud o'r gweithio ar y dyfodol yma ar gyfer yr EY 27, rwy'n mynd i'n meddwl ymdweud yn gofynio'r ddweud o'r ddweud Mae'r unig i'r unig i unig i'r unig i yng Nghymru yn rhoi'r blynyddiolion ar gyfer cyfwilio. Felly, sydd y brygwch yng Nghymru â'r gwahanol, mae'r gael ei bod yn cael eu ddweud. Mae'r ddeiligio, oherwydd i'r unig i'r unig i'r 27 ym gwrdd yng Nghymru. Mae'r Dau Cymdeithasol, sydd wedi'i bwysig o'r drwng ar y boel. Mae'n ei ddweud i'r unig i'r unig i'r 27 ym gwrdd yng nghymru. We've had the State of the Union message from President Juncker. We've had President Macron with his ideas. We've had President Tusk with his leaders agenda. All of these are putting out the challenges that the EU 27 is going to have in dealing with the future as it will be after the British leave, but also the future as we want to shape it for EU citizens. It's very interesting to see that you have opened, the Slovenian government has opened a process of debate and dialogue with the citizens on the future of Europe and are holding public discussions because next Wednesday we're going to launch our Prime Minister with Tishirk is going to launch a similar citizens dialogue here and we look forward to hearing about that. So may I give you the floor, State Secretary, and we look forward to hearing the Slovenian perspective on the future. I have to say it's always good to be back to Ireland and to Dublin. We have a lot of similarities, meaning in the character of people I can say we both come from smaller countries, but we are full of joy, I would say, despite the difficult times here. Again, thank you for inviting me here and I will start by assessing the situation which we are in right now or we passed through in the previous years. That situation determines also the current state of play in the European Union itself and the debate on the future of the European Union. So I wouldn't agree with those who said the European Union is constantly in crisis, everything is terrible, it will fell apart, it is a disaster and so on. I think that for Slovenia the European Union did a lot of good things, really a lot of good things. Predominantly if I just made to say we had, we eased the very difficult historic dialogue with all our neighbours. You are aware how difficult the history was in that part of the world and since our membership in the European Union, this dialogue with our neighbours, it became really a sincere one. European Union I would say is not in a crisis European Union now is growing and still do not forget European Union is the largest world economy in the world. This is the positive part but it was influenced in the previous years of course by a hard and harsh economic and financial crisis which did a lot of dividing lines in the European Union itself. We were faced with the growth of the populism probably because the middle class in the European countries was mostly affected by the globalization. We were faced with the growth of populism based on the security situation around the neighbourhood of the European Union, I mean here migration and the management of the migration. We were faced with security challenges not just in the neighbourhood but also inside the European Union, I mean here a new phenomena of terrorism. We had elections in two important countries, France and Germany and we can see in other countries despite the more populist leaders didn't win the election but we see especially in the central Europe also in Germany movement of the electoral body more to the rights and more to the sceptical part of the European Union. And of course we were faced with the most evident phenomena which will tackle the future of the European Union and in fact it triggered the debate on the future of the European Union. And this is, you know it very well, the decision of the UK to abandon membership of the European Union and to leave the European Union, I mean here Brexit. From all these events and the situations basically today we have two division or dividing line in the European Union so it's a sort of cross road in the European Union. The dividing line goes from north to south or between north and south where the countries of north are advocating more austerity, more financial soundness while on the other side the southern or Mediterranean countries are more inclined towards mutualisation or solidarity especially in the financial and economic terms and there is a new division which is based more on various solidarity and burden sharing, this is between eastern and western countries. Some of them in the west in the papers we can read that because of the computation of the central European countries they think that they are talking or they are writing also about the failed integration into European Union which I would not agree entirely or I wouldn't agree at all. But obviously we have a sort of mistrust between eastern and western countries and this is a problem that it will have to be addressed in the debate of the future of the European Union. The consequence of the events in the European Union I just mentioned was a meeting in Bratislava, you mentioned it and the declaration of Bratislava which produced the thoughts about more efficient European Union, more inclined European Union towards people and especially to the young people. The confirmation of Bratislava declaration was at least on the declaratory level within 60th anniversary of European Union in Rome by Rome declaration and of course here we are in the debate of the future of the European Union which already started and it will be intensified probably from January next year. After the composition of the new German government and probably on the anniversary of the so called Elysée Treaty between France and Germany but I will say later much more about this. We are in the face of the debate of the future of the European Union and this debate will be closely connected and also determined by debate on the future European budget meaning the future European multi annual financial framework. This financial framework, this European budget will be severely different I'm afraid from the previous ones not just because one of the largest member countries will at least they announced will go out of the European Union and financial contributions but the priorities of the European Union will be completely new and the priorities will be if we go in substance probably the migration, the completion of the European Monetary Union, the social Europe or social below for European Union, the security and the defence and of course possibly also the democratic reshaping or strengthening of the European Union. So based on this obviously new priorities for the European Union we were faced with several ideas how to shape and how to streamline the debate on future of Europe. We are already faced with on the general affairs council that I attend and still there are many different views how to proceed with the debate. For example, for the beginning we have five scenarios produced by the European Commission and you are all aware of them then we have six scenarios produced by the president of the European Commission and this six scenario is basically a call for to continue in a united manner to continue with united Europe stronger and more democratic Europe in the form of 27 countries. There are some ideas that will have to be discussed in a very serious manner. For example, a call for a qualified majority voting in the field of taxation in the internal market and foreign policy. There are a call for democratic conventions picked out from other speech by French president. So there are plenty of ideas how to make this European Union more efficient, more inclined towards the citizens, more oriented towards unity of eastern and western countries and this is basically a call for to continue as I already said in a manner of united Europe of 27 member countries. On the other side we had a speech of the French president again ideas on a vast majority of the areas in the European Union but basically this speech was oriented towards ability and willing meaning those who are able and willing can go with enforced cooperation on particular areas which means basically this idea is devoted more towards the differentiated integration or some of them they will call it Europe of more speech. In the core of this idea of course is to strengthen the Europe area as a core which will be the most advanced and in more details it is a call for Germany to revamp or to renew Franco-German motor for the further integration. This is the basic idea of the French president and to preserve the unity or the inclusiveness of the 27 member countries the president of the European Council produced the so-called leaders agenda to streamline the debate in the format of 27 on different areas how to renew future European ideas. For example on the European Monetary Union we have still a basic division line and different views how to continue between north and south I already mentioned it between solidarity and austerity or mutualisation. There are different ideas how to reshape the eurozone finance minister parliament or eurozone commission's role and budget role of the eurozone whether it be separate or inside the European Union budget or frame. How to revamp the European Monetary stability mechanism into some ideas into European Monetary Fund and here the basic dividing line is still between Germany and France. Of course President Macron made a very interesting idea and he called to create a united market German Franco market which will be basically a new market he signed a single market if I could understood correctly and as far as I know some ideas are already progressing meaning the harmonisation of the fiscal social and economic policies. This will be much more difficult but on the fiscal part the talks are progressing towards the harmonisation at least on the corporate tax which will be a major step forward but still the observation how to develop how to strengthen eurozone basically are still in a very distant position. The area which is going very much with a higher speed further is the defence. You know that the European defence agency is already there and we are talking now about PESCO political and security cooperation which will be triggered this month and Slovenia of course will be part of it. But this is the area probably that it has a potential to go particularly because of Brexit strongly, much more strongly forward. The debate that will determine the future, the unity of the future European Union will be much more much dependent on debate on migration. This here we have as I mentioned quite visible dividing line between eastern and western countries between solidarity and burden sharing. The relocation schemes together with the revision of the asylum code will be at stake and here maybe in Ireland you do not feel it because you are not part of Schengen agreement but we can feel it severely. The problem with Slovenia is that countries are reintroducing border control on the internal Schengen borders and this is quite a serious question for the future of the European Union because it might prevent free flow of people and both obstacles also to free flow of services and also capital between the member countries and the burden of the debate on the future of the European Union as well but nevertheless probably we will have to revise the common asylum system of the European Union or to create a suitable migration policy of the European Union together with more focused view on our neighbourhood meaning especially in Mediterranean eastern part and southern part of the Mediterranean. Again the debate will be certainly very important about more efficient European Union groups but together with the groups also more and more on social pillar of the European Union because social pillar was kind of neglected in previous years at least we feel like this. The last one is the democratic transition or the democratic strengthening of the European Union. There is a common idea of president of the commission and French president to have so called democratic conventions in the European Union also in light to prepare let's say between the Breckins European parties with a real European programme for the next. European elections and here we have the idea of transnational lists which are supported by bigger countries but we have to have a very close look and very close debate or about this transnational list what they do mean because if the system of let's say democratic European Union will go to system one man one vote we at least Slovenians who lived in one multinational society already this means former Yugoslavia which was not democratic and this is different from the European Union but still the democratic system one man one vote in the multinational society can be a very very dangerous one. Of course all these priorities at the end of the day will lead to the question shall we change or should we change the treaties or not and as far as I understood everybody in European Union would like to escape the treaty changes there because to change the treaty many countries if not all need reference and these days with reference you never know where you can go. So the understanding is that probably the reference the treaty changes are not in the focus line but still French president as far as I understood he opened the floor also if needed for the treaty changes in the in the in the conversations with the rest of the membership of the European Union. Basic question of this is how to proceed not just in manner of 27 or differentiated integration but how we will achieve harmonisation in the European Union or convergence in the European Union without emphasis on the cohesion because convergence probably is not possible without the cohesion. And if we will have a diminishing site of the future European budget on the cohesion site then the question will be how the unity of the European Union will proceed. That's why I said that the debate of the future budget will influence also the debate on future of the European Union and that's why for Slovenia the emphasis is on cohesion if you would like to preserve the unity of the European Union of 27 is very very important. And do not forget on the one of the policies that was neglected in previous year unfortunately I would say and this is the for the enlargement policy which Slovenia is a strong advocate of. I think that European Union should continue and should go on much with much stronger engagement that it was till nowadays in the western Balkans which is the area still very very fragile from economic but also security points points of view at the end of the day. The question will be what we would like to achieve with this European Union also on a global level how strong political and economic player we would like to be the European Union on a global level. I think here particularly there are different views inside the European Union on the future free trade agreements whether should European Union be a leader in the free trade agreements or not in the world particularly after the let's say a retreat of the United States between the brackets from the global trade leadership. For all that kind of things for the future debate political elites will need public support. Without public support and public support here I mean the citizens of the European Union countries without the public support there won't be any changes in the European Union. That's why it's important and French president correctly correctly observed that we have to engage in a democratic way the public opinion across the European Union and in European Union member countries that that much more focused and sincere dialogue is needed with the citizens of European Union. We all know and I won't I will not repeat that probably this decision making process and I won't blame just brass it's also the national governments it's on European methods it's too distant from the from the our citizens. So that's why we already launched one and a half year ago public debate on several aspects of the European Union policies future policies and future of the European Union. We complete five round tables on five priority areas future priority areas of European Union. We are one of the rare countries that confirms all the positions that are taken on the level of the European Union before they are taken in the parliament in the discussion with the representatives of people. And we further plan now much more focused debate on the future of the European Union on different areas that I mentioned before with interested public experts universities but predominantly with with with young people. If I may conclude where the Slovenia stands regarding all that that elements that I presented mean we are in the beginning of the debate but still Slovenia would like to have European Union of 27 we would like to preserve and to continue in a United Union. We would like the European Union will be focused on security methods meaning internal and external security but on the other side also on economic security and the security that is Slovenia is very important. The social part of the of the security so social security is for Slovenia quite an important important element. Not just growth but also social issues. We would like to continue with with further integration of the European Union and basically after the Brexit the idea of French president Macron is to deepen the integration for beginning with the Franco German harmonies integration in that mentioned in the in the in the creation of a Franco German market which will be the basis for the for an example for the for the rest of the membership of European Union. In case we won't have that scenario that we will proceed along the ideas of the differentiated integration for multi-speed Europe then Slovenia thinks that we should this should be based on a principles that it remains rooted in the frame of the EU institution so EU institutional framework that it sets a clear path for further integration of the EU 27 that it does not lead to fragmentation of the European policies and the key and that this integrated structured cooperation or different differentiated integration remains open for all the member countries of the European Union. But at the end and I will conclude with this Slovenia switch in that if this scenario will go on that we will be part of the most advanced most integrated or the core part of the European Union.


Hello and welcome everyone to 2021 Linux Foundation open source summit happening in Seattle. Good morning, good afternoon, good evening depending on from where you are watching and and thanks for joining us to learn. How to build an AI marketplace on top of communities myself is an image. I'm the CD for Watson data and open source.  Hello, my name is Christian, I work with an image. We both work for the center for open source and I and I have been working on several projects together. And this latest project, the machine and exchange is what we're going to present today. Okay, thanks. Yeah, so let's get into the business as Christian mentioned, right? We work for a group in IBM core code, so we'll probably get started with that. So the topic of our session today, machine learning exchange. And if you're sharing this name for the first time today is when we are actually, you know, bringing it up for the first time in a public domain in a conference, right? So let's just go right through it. So we talked about code, right? Which stands for Center for open source data and technologies, you know, it's a group in IBM focused as the name suggests very much on open source data. And it includes many technologies, including some very popular open source projects like Spark, TensorFlow, PyTorch, you know, Q-Flow, and, you know, we work in these ecosystems and we're responsible for, you know, ensuring that what we need as part of, you know, making these projects work in Watson also, you know, what we need in terms of enhancing these projects as part of the community request which come in. Now, the beautiful picture you'll see there is of IBM Silicon Valley lab, you know, it's part of Silicon Valley, though it's much away from the hustle and bustle of Silicon Valley nestled in between green mountains. We do have our own cricket field as well as you can see. So and a lot of wood hiking trails nearby. It's just in the middle of nature. So if you want to live in Silicon Valley and work in Silicon Valley and not be bothered about by all the hustle, bustle and traffic, this is a place to be. So I think let's get started with the topic. And if you are familiar and, you know, as we are looking in the machine learning in AI, we're essentially at a very fundamental level, what we are doing is, you know, we are using data to build models which are then automating decisions. Right. And this is throughout the AI life cycle, right? We use data, then middle models, which help automate decisions. Now, this is a high level view of the AI life cycle. But if you look from the pillars of AI life cycle, data sets and models totally stand out. Right. And when you zoom into this, you see that, hey, these steps are probably not as simple as it appears at the very first outset. You know, each of these different horizontal streams will be data preparation, for example, which includes data cleansing, data ingestion, going all the way to transformation, feature engineering, doing data splitting. Right. That in itself is a huge field in itself, right? There are multiple products and multiple companies just specializing in that particular space. And then moves and comes the machine learning in AI space, which is all around, you know, how to create that initial model. Then, you know, launch distributed training, how to do hyperparameters, optimization, how to do neural architecture search, then validating your model, that whole model piece, where you are essentially, you know, just in the machine learning and AI model creation phase and running distributed training, finding the right hyperparameters, that in itself again, you know, this is a field where a lot of the products and a lot of companies specialize in that area. And last but not the least is essentially the area where, you know, you have models deployed in production, but models are like applications are living and breathing entities, right? You deploy an application, you send in the same input six months later, as long as you haven't changed the application version, you're guaranteed to get the same output. Not so much in the case of models, right? You have the same application, same input, but six months down the line. Even if you haven't changed the model, right, it is going to give you vastly different output because models are living and breathing entities, right? And hence, you know, this process is very iterative, you need to be doing it again and again, you need to be rolling out Canada versions. Sometimes your data set is changing, sometimes there is anomaly, so you need these automation pipelines, right? So as we talk about the three pillars, right? So we saw and definitely understood that data set and models are the pillars, but pipelines have become very, very important when we are looking at, you know, automating the whole data and machine learning life cycle.  So data sets, models and pipelines. Now, how do we speed up this AI life cycle, right? Now, with a number of steps, as we just discussed, needed to be performed in data and AI life cycle currently, right? The process itself remains bifurcated amongst various teams, right? We have parts of the team which, as you saw, are doing, you know, data part of the life cycle, parts of the team which are creating models, parts of the team which are doing feature engineering, creating set of new features. And because of these bifurcation and long horizontal life cycle processes, what is happening is, you know, there is a lot of duplication and redundancy. A lot of times, you know, you have similar set of features being created. A lot of times you have similar versions of the data sets being created. The models being created are very similar pipelines, more so the specific pipeline tasks. There is no reuse. There is no sharing. And everyone is working in their sidewall. Now, so it's becoming very clear that, you know, there is a stronger need for a central data model and pipelines catalog, right? Which we can share and use across organizational boundaries, and not only organizational boundaries, also across different parts of the data and AI life cycle, right? So the teams who are doing and creating features in the data life cycle are actually be able to share some of those things when they are in the machine learning part of the life cycle. Similarly, you know, all these different models and data sets, engineer data sets, they are being shared. So now that we need a central data models and pipelines catalog, right? And the other part is, you also want a strong governance and traceability and lineage, right? And why is that important, right? When you go on the internet and if you search, you will find all kinds of data sets. You'll probably find, you know, thousands of models if you Google for it. What is missing is audit check, a quality check, having a proper licensing mechanism. Can you take this data set and can you use it right off the box without worrying about licenses? How all these sources being identified is the proper lineage track as part of the data set metadata, right? You want that all part of, you know, this central catalog which can tell you that, hey, if you are picking up something from here, make sure you are sure that, you know, it has the proper license. It has the proper traceability and lineage. You can find, you know, how the data set has traveled or the model has traveled. So that is also, you know, very central piece of this catalog which we talk about. So with that, we are announcing the machine learning exchange, right? So as you saw, as I mentioned, right? If you're hearing this name for the first time, yes, that's correct because this is the first time we are actually announcing it. We are joining hands with Linux Foundation AI and data. We are announcing and moving this project in open source. And not only we are moving this project in open source, we are actually moving in open governance. That means, you know, the license, the trademark, everything is in a central place and being owned by Linux Foundation AI and data and we will be working with the larger community and the partners. They are in the next foundation AI and data to jointly evolve this together, right? Because we firmly believe that to create this ecosystem of data sets, models, pipelines, you know, we need to do it in a totally neutral way where the foundation has the ownership rights. There is a neutral license and we work as a community member in terms of enhancing this. So machine learning exchange is the project jointly, we are announcing with Linux Foundation AI and data. And this is how it looks like, right? So I talked about pipelines, data sets, models, definitely as we are aware in the data science stream. Notebooks is the language, right? How you actually write data science? I mean, I think we don't need to debate it anymore. Almost everyone is using notebooks to create a lot of the initial models, right? The data science code, right? And that's also, you know, available as part of the machine learning exchange and you can go and look at all these different architect artifacts. Let's go through them one by one. So and before we dive a bit around the architecture, right? So and some of the capabilities. So what does machine learning exchange actually provide, right? So there is a read-only version, which is the hosted version. You can actually hit a website, go there and that's a read-only version where you can look at the assets and browse them, right? But there is also a version which you can actually, you know, pull from GitHub, deploy it at your own end, right? Which allows you to upload, register, execute as well, right? So you can upload your own assets and register them and then you can launch them, right? Including AI pipelines, models, data sets and notebooks, right? It generates, when you register, for example, a model, it generates automated sample pipeline in behind the scenes, right? For example, to deploy your model on top of a community cluster or care serving, which is an embedded engine, right? So pipeline engine here in this case is powered by Qflow Pipelines, which is a very popular open source project for machine learning and data pipelines. And we are using Qflow Pipelines on Tectron under the covers to actually power this. Then the serving, when you are actually deploying your models, that's powered by another very popular open source project called care serving, right? And we'll talk a bit about care serving and pipelines in the detail later, right? And then there are other projects which found the basics of it, right? But at the very high level, there is a UX, there is an API server, and then, you know, backing all these assets, metadata is a relational DB and the assets actually. And it's the metadata is being shared between the object store and the relational DB. So let's talk about, you know, how pipelines work. Essentially, you know, this is the pipelines tab, as you can see, you can go and search for pipelines, you can select a pipeline. And then, you know, after you have a look, and this is the pipeline you want to run, then you can actually launch it, right? And in some cases, you don't like a pipeline you see, then you can actually go ahead and register it in the executable version of it, right? If you're standing this platform at your own there. By launching the pipeline, you can select the parameters and input your own values. And then the pipeline is launched behind the scenes. It's essentially using the Q-flow pipelines on Tecton Engine, and giving you logs being streamed in real time, giving you metadata, giving you visualizations, and, you know, creating a lineage, right? As your pipeline is running, you can have a lineage view of all the artifacts being produced as part of that. Essentially, pipelines are made of pipeline components, right? So we also provide a way for you to actually register your own components so that you can share and reuse across folks, right? So if there is a component to, in this case, what we have is a very simple one, like, you know, it's just echoes.  Some of the tasks which are done again and again, which need to be plugged in into multiple pipelines. This is the best place to come in, plug them and test it out, right, without having the need to create a pipeline. You know, you can just register your component, launch it, test it out, that it works, and then, you know, you can reuse it as part of the different pipelines. Models.  So there are a lot of pre-registered models which come as part of machine learning exchange, around object detection, around texture sentiment classifier, etc. So you can pick and select from them, or you can register your own models, right? Now, once you launch the model, you can look at the model description, and we also generate, because the model metadata YAML based on that generates some automated code so as to be able to deploy your model. You can deploy your model on Kubernetes cluster, or, you know, the embedded KF serving engine, which comes as part of the machine learning exchange, right? So we can select a model, and then just launch it, right, give it a name and launch it and deploy it, and it will essentially, you know, deploy it on the underlying embedded machine learning exchange models on the platform, right, which is powerful. Now to deploy it again, it will use the shoe floor pipelines engine, right, which is essentially taking your model bits, downloading it, and then pushing it on the, either the Kubernetes directly or the embedded KF serving engine, which comes with machine learning exchange. Now, datasets is one of the most important pillars of this lifecycle, right, as we talked about, so you can essentially go, browse the datasets, look at the details, look at the metadata. In this case, we are looking at JFK weather dataset, which is essentially, you know, focused around all the different climate characteristics, temperatures, coordinates, humidity, et cetera around JFK airport. You can select this dataset, you can launch it. Now, what does launching mean in the context of a dataset? Essentially, the default functionality out of the boxes, it's going to download this dataset onto your Kubernetes or OpenShift cluster, wherever you are running machine learning exchange. You can download it and create a PVC persistent volume claim, which can then be used by rest of the parts of the lifecycle, right. So for example, if you need to launch distributed training later on, right, the dataset is now downloaded and made available on part of your on your Kubernetes and OpenShift cluster, it creates a PVC for that for a persistent volume. And as we go through this, like Christian is going to come in and show some of these assets as well. And similarly, notebooks, right? So we essentially, you have the similar mechanism, you can launch a notebook, it will execute it as a batch process. Behind the scenes, we use a project called Ellyra and Ellyra's notebook company to actually take your notebook, treat it as a single batch process and launch it using the Q4 pipelines under the covers, right. So that's our notebooks work and you can also see it later, you know, as we go through some of these steps. Okay, and now, you know, this is the part which is showing you, you know, how you can launch a notebook. By launching a notebook, as you saw in our dataset, I talked about, you know, the dataset which was downloaded on a PVC here, you can launch that notebook which holds that dataset from the PVC and then runs analysis on top of it, right. So this is the notebook which is, you know, pulling the dataset, the JFK dataset from the persistent volume claim and running analysis on top of it and you can see the logs being streamed in real time. Okay, let's talk about a bit about the catalog, the actual content, right. So so far we showed you the framework, how does it work, what are the capabilities, and this one, you know, goes a bit into the catalog and the content behind it. So as I mentioned, right, there are some default samples which come pre-populated, you know, models like around object detection, image caption generator, resolution enhancer, weather forecaster or you know, datasets like code net, finance proposition bank, datasets, JFK dataset, then there are pipelines and pipeline components, right, which we have created, which are around, for example, Trusted AI umbrella, which allow you to detect fairness on your models or do adversarial robustness check on your models, right. And then also, you know, there are pre-integrated sample notebooks which come as part of it. Now, obviously you're not limited to these, but you're running your own instance, you can come and register your own assets and use it with machine learning exchange. And, you know, calling point here that though it comes with some pre-populated assets, the goal here is for you to bring your own assets and register as part of it. Now, let's talk about datasets, right, so as we talked, right, it is one of the most important pillars of, out of the three we talked like datasets, models and pipelines, right. Now, if we see the actual machine learning revolution is fueled by data, right, if you look at something like ImageNet, right, in 2009 there were 3 million images with approximately 5,000 classes, just three years later in 2012, that number has increased to among this 14 million images with 22,000 classes, right. And that essentially fueled what we call, you know, a revolution around image classification, image reduction and all these models coming out, right, where you see, you know, it actually started surpassing human level capability on some of these narrow tasks. So, if you see the graph here, for example, you know, at some point in 2015, right, we started actually surpassing human level intelligence in terms of ImageNet classification errors. So, the point here is like, you know, the revolution and fueling that dataset with large quantity actually helped to get to that level of accuracy and these sophisticated models being produced, right. Now, even if you look at the progress in general on AI, even when we talk from the IBM's perspective, like 2011 we had the JAPARDY, right, where essentially it was more structured data in terms of, you know, the question and answering or in case of JAPARDY like answers and questions. But by 2019, we had project debater which were actually able to work on unstructured data, be able to debate on logical rational arguments with the debaters, professional debaters, right. So, to get that kind of advancement, we need, you know, that huge corpus of data. So, as I said, right, we have seen the power of this AI which was being applied to human language, right, even if you look at it from the perspective of the speech performance, right, and you look at the first DNN back in 2012. And now when you look at, at somewhere in between 2016 and 17, it actually surpassed human capabilities. All the voice and conversation back and forth, which for example we provide in a product called Watson assistant or document understanding all these things, you know, starting to surpass human level capabilities with the explosion of data, right, and then starting creating a huge market for them. Similarly, code is the language of machine, right, AI will help us also to master code, right. So what we need essentially is, you know, AI and machine learning models, right, which can bring the same revolution, what happened on ImageNet, what happened with speech, back to code, right. So essentially AI for code needs its own ImageNet for breakthroughs, right, we need code language translation, we need to find if codes are duplicate, we need to find, you know, they're similar, we need to find areas where code can be improved, performance improvement, memory improvement, or, you know, the holy grail where we just want to give description and one machine to generate code. Now how do you get to all these, you know, sophisticated models, you need that huge corpus of data set. So with that, you know, IBM back in May announced Project CodeNet, which is a very high quality code data set for algorithmic innovation, right, it has around 14 million code samples with around half a billion lines of code across, 4,000 code problems, and 55 programming languages. So if you see, there are, you know, this is this huge innovation and breakthrough which was done as part of this. And that actually gives you a lot of these capabilities, which now you can create a lot of this sophisticated models where, you know, on these diverse classes of problems. So, coordinate is actually the largest, you know, open source data set available for AI for code, right, and, you know, it works across polyglot languages, multiple languages, as you can see, as mentioned here. And by virtue of that, as I was talking about, you know, the content which we have in machine learning exchange is very high quality. So, you know, coordinate is an example, coordinate is available through machine learning exchange, you can essentially download it and use it, and you know, later on we are going to see a quick demo, essentially involving coordinate as well. You know, you can see the description, you can look at the metadata and go through some of the lineage and traceability and make sure the license is correct, all that information is available to you in this single place. So there are associated notebooks and models, right, so there are associated notebooks and models which we are providing around coordinate and we will be expanding this ecosystem. And this was essentially to highlight the fact that, you know, the content which we are actually putting in machine learning exchange is of high quality, right, and we are going to be working hard at it to make sure that remains of that. So the integrated technologies and I wouldn't spend too much time on these. This is data sheen, which essentially is used behind the scenes to actually download the data set and create a PVC. It's a project built on the Kubernetes custom resource architecture and part of LFA.  But if you need more for sophisticated deployment technology, this is, this comes pre-built and pre-integrated with machine learning exchange. Pipelines, right, which is at the heart like everything we are doing for any asset in machine learning exchange is being executed through pipelines. And that engine is powered by Qflow pipelines on Tecton. I will spare some of the architectural details here, right, but as you can see, this is a Qubinative pipeline engine built using the Kubernetes custom resource architecture. And if you're aware of the Qflow pipeline ecosystem, it's very popular with data scientists as well because it provides the Python DSL to program using Python. And this is what we use heavily within machine learning exchange to power all the actions which are happening with any single asset, right? Whenever we are deploying a model or we are downloading a data set to create a PVC or we are launching a notebook, everything is going to be triggered under the covers using this pipeline. Since we are on the topic of pipelines, we have an enterprise product called Watson Studio Pipelines, which essentially is built on that engine where you can essentially get a lot of the pre-built notebooks, which can run much sophisticated Watson capabilities like Watson AutoAI, data refinery flows to handle your ETL needs. You can actually run web service and online deployments, batch deployments, and it also gives you a very solid drag and drop interface to drag and drop different components and create your model, different pipelines. So definitely check it out if you're interested in pipelines and the pipeline's ecosystem. Okay, with that, I will pass it on to Krishan to take you through some of the capabilities of machine learning exchange. Over to you, Krishan. Thank you, Animesh. That was great. Let me share my screen. Can you see my screen, Animesh? Yeah, again. Fantastic. So what we see here, that is the user interface of the machine learning exchange. Like you've seen in the animated gifts that Animesh showed in this presentation. And you can see on the left hand side, it's our navigation where you can navigate through our individual asset types. As you have been told, we have data sets, we have models, pipelines, components, and notebooks. For the demo part, I will start with the models, because the models are really at the heart of anything in machine learning. And we have the benefit of a sister project here at the IBM Center for open source data and EI. And that project is called the model asset exchange. And we're lucky enough to feature those models here in MLX. And I will show one of them, these models. They have been containerized. So they have been pre-trained. And they can be served in principle. If you click on a model in MLX, you'll see a description that tells the user what this model is about and what the framework was that it was trained on and what the license is. And you can navigate to the website where there's more information. In this case, the model asset exchange website. Each of our assets are based on the ML metadata. So when you upload a new asset type, that metadata will tell us what to present in the catalog and also what can be done with the particular asset. So for this model, you can see this model can be served. It can be served either on Kubernetes natively or with KServe. And I will always also show what the images that the Max team has built for this model. If you're interested, we also generate some pure pipeline code. That's the code that will be used to launch the pipeline that in this case will serve that model. So let me go to the launch dialogue. It will ask me a few questions. In this case, I want to serve this model on Kubernetes. I can give it a run name, but also stick with the default. And now you will see beautiful pipelines graph of user interface where you can see the execution graph of that pipeline. This particular pipeline starts with a configuration point where the model configuration will be generated. And once the part is done, the model will be served.  The pipeline will take about a minute or two to complete. And once it's complete, we will see that model has been deployed. And in the input output parameters, you can see that this model has been served on on the local or an IP address from the cluster, which is accessible. And you can go to that model. And the model will have a straggler generated UI that allows to get some metadata on the model and also to influence using for inferencing on pictures. So this particular model is an image caption generator. So let's try it out on an actual image. On image and I took the liberty of getting your profile picture. And I'm going to see what this model will do. We can exit with this model. It will tell you it's a Hollywood movie star Hollywood movie star almost it says it's a man in suit and tie and smile. Okay, at least it finds me smiling. So that's that's good. All right. So this was one example of what you can do with models that have been pre trained and containerized. As the second part of our demo, we will show one of our pipelines and one of a very, very important aspect of machine learning, of course, is that, you know, not only that your models are accurate, but also that they are fair. So we have this trusted pipeline. And that makes use of two other related projects that we've been helping out on. I've been research has put out 2 projects called the adversarial robustness 2 box. And I found this 360 and both of these capabilities are part of this pipeline. On the right side here, you can see the pipeline that was compiled into the format that is more for machine read readability. Not for you would be the best in this case, but we also have the details for this pipeline and we are able to launch this. In this case, we have preset most of the parameters and you can run with it before parameters. And once you could submit that the 3 steps of the pipeline will be executed and it will start with training. So this was my previous run. It shows the model has been trained or think a sample data set of 20,000 faces of all kinds of ethnicities. Both all kinds of genders and age groups. And after that model has been trained. So this pipeline performed a bonus check. And a adversarial robustness evaluation, and it will also show the metrics that got generated with this in this evaluation step. Now here you can see that. The model accuracy on the test data is about 86% pretty good on the samples that we were feeding to that model. It's not so great because we only did, I think, five epochs of training iterations. So that particular model would be would not be very robust against adversarial. I think the metrics there are like highlighting very clearly that, you know, the, the, if you're generating adversarial inputs and and sending it there. The model accuracy drops heavily right so it's it's definitely vulnerable to adversarial attacks. And that would necessitate that model gets, you know, retrained and make more of it. And I think the adversarial robustness toolbox has tools for that as well. Now on the family side. We see a similar output similar metrics. And this is a gender classification model. And the classification accuracy here is about 86%. And one interesting metric here is the disparate impact, which is 91.91. And if it's, you know, I think between 0.8 and 1.2, that means it's, it's not biased, particularly either way, which means that this is, you know, a fair gender classification model. There's no by which was raised or each in this case, if I understand this way. Yes. And if you use this toolkit, like, you know, there are, in addition to this, there are over 70 plus metrics on which you can, you know, evaluate the fairness of your models and find out whether they are biased or not. And if it does found to be biased both with adversarial robustness toolbox as well as fairness check if you if they are found to be vulnerable to adversarial attacks or if they are found to be biased. So all these toolkits actually provide mitigation algorithms right so you can now implement. So there are 10 plus algorithms for example in your fairness to to mitigate bias. Yeah. Thanks. And I think, as last part of our demo, we can showcase our project code net. And notebooks that we associated with this. So, from the project code net, which is, which is extremely large as I mentioned, we have two subsets, one subset for a mass language model and one for a language classifier. Now that language classifier basically, you know, classifies a code snippet. And the output is web or not that code snippet is written in Python Java CC shop. And we will see here some, some more description on the program data set similar to what I'm going to show you this presentation earlier and links to all of those data sets. It will show the license the size of the data set. And as with all of our assets. We have a YAML file, a YAML file describes the assets so that MLX knows what to do with it, what the display. And when you launch the data set, we will use data show under the cars. And we will download the data set and mounted to your persistent volume. And that can be used later in other asset types. For example, the notebook they will show in just a moment. So you want to fix submit. There should be a two step pipeline. So one, one of the pipeline steps takes our metadata that we use in MLX and converts it to the metadata we need for data show. And then once that data set metadata has been generated, data show will be used to monitor. So earlier I ran this. And after the metadata was generated the persistent volume was created. And you can see that you can see have an identifier. You can either copy that and use that, or I can go to the related asset of the data set. The notebook that we have for this one is also called a language classification notebook. They're similar to all of our assets of our asset types. And this notebook can be previewed here with our, with the Jupiter notebook viewer that we integrated. And this is a read on view, but you can see all of the code cells in the notebook.  And at the end, the notebook should take a bunch of code snippets and then run through the classification with that particular model and launch this model. Plug in that data set we see that we created earlier mounted to a local folder and submit this will start our pipeline. And that's once that's once that pipeline one is picked up and there will be a part created on the Kubernetes cluster for that and once that associated image has been pulled and the run has been kicked off there should be locks streaming in here. That will take a while and because that does take a while. I have the output of that notebook that I generated just before this demo that notebook gets regenerate. The pipeline will generate an HTML output. So once the notebook has been run, all of the cells and the output of the cells and be seen in that notebook. And then at the very end of the notebook. Typically, you can see some of the metrics. You can see the training and delegation accuracy. And for this test, for this test, there were 100 code samples with 10 samples for each of these languages here and so on. And you can see that test here was fairly successful. All of these for each of the language all 10 samples were classified correctly. And the notebook that went into this training. However, only had actually a lower classification so that's interesting. So if you go into that notebook, very bottom. And before we ran this, you could see that the code samples that were chosen to previously only had nine correct see samples and nine correctly identified to the shop samples. This was my demo on image. I hand the screen back to you. Thanks, thanks a lot. Thanks for walking us through the capabilities of machine learning exchange tradition so hope you know you all enjoy it. Learning about machine learning exchange and the capabilities it has to offer. Definitely, you know, reach out to us if you have more questions, if you want to try it out later as a lead only version which is hosted at ml dash exchange. Org, and if you want to, you know, give it a try on GitHub, get up.com slash machine learning exchange, right? That's that's the GitHub organization where you can actually go. But if you have to remember one link ml dash exchange or just hit it out and it will take you everywhere else. I believe you have a slide for that. You can share your last slide. Okay, let's bring it up. Cool. Here it is. Right. So these are the links you need to remember like, you know, where to go. Remember, it's a read only version. So a lot of the capabilities you are seeing as part of questions demonstration. They're not available in that read only version you can definitely download the assets browse the assets go through the description but if you need to do more. Go to this GitHub right and actually get your own version running right so yes you can look at the read only version here on ML exchange dot or and go through the different assets but better still like go to this machine exchange.  You know, to get it running on your laptop or if you have a Kubernetes cluster that's best because you know that gives you the full load capabilities and also allows you to register and use your own assets right so strongly encourage that. There is one more thing which I do want to highlight right before we go away there are some. Awesome talks happening from IBM code a team right throughout the open source summit North America right talks, which go deeper into, for example, the cure flow pipelines on tech on project. Or you know how to build a feature store using feast and care serving right how do you actually defend against adversarial model attacks using cure flow and adversarial robustness toolbox which we just talked about right. And so please go ahead and join our other talks and give things a try and provide us any feedback and we'll be glad to see you on as a contributor as a user of machine learning exchange. Thank you.