Well to many school teachers the end of June signifies more than just the completion of a term's work. It often ushers in romance and marriage. To Armist Brooks, who teaches English at Madison High School, it also has a special significance. It certainly has. To me, the end of June is significant because it ushers in July. Oh, it's not that Mr. Boynton hasn't asked me to marry him dozens of times. It's just he hasn't asked me once. I've tried to forget about it, but last Saturday morning at breakfast, my landlady reopened the case. Connie, I've been thinking, if you're going to be a June bride, we better step on it. At last a proposal. I'm serious, dear. You've got to bring home to Mr. Boynton. The fact that this is the month of romance. The time when all living creatures reach out to one another. If I reach out any further, I'll look like a giraffe. June is a wonderful month, Connie. My sister Angela and my brother Victor were married in June, you know. It was a double wedding. Angela and Victor married a half-brother and a half-sister. Where were they married? In the little half-church around the corner? The boy's father was the girl's father as well, but his mother was her stepmother. Of course, her father was the boy's stepfather, but he had passed on by the time they were married so he couldn't give the bride away. What a pity. So the bride's step-uncle gave her away. He was the boy's mother's half-brother, of course. What else could he be? It was a beautiful ceremony, Connie, and within five years, each couple had three little girls. I bet there were six of the nicest half-stepnieces you ever met. Pass the cream, please, Mrs. Davis. Here you are, dear. Now, you be a good kitty, Minerva. You've had your breakfast. She's so restless lately. Must be because she knows it's June. How does she know it's... Oh, I keep forgetting. There's a calendar in the kitchen. You know something, Connie. I think Minerva's engaged. Really? Anyone I know? I don't think so. He's a big Persian from the next block. Wouldn't she be happier with a cat? Forgive me, Minerva. Meow. That's funny. She keeps looking toward the kitchen door. I wonder why. Meow. Meow. Oh, must be your friend from the next block. Now, Minerva, I wouldn't go out there if I were you. I would. If you'd excuse me, Mrs. Davis, I'd better start crimping a bit at nine o'clock already, and Mr. Boynton's coming over at 12. Oh, he can't be this early. I'll see who it is. All right, Connie. I'll take Minerva out in the backyard for a while and get some sun. Fine, Mrs. Davis. Well, it's Walter Denton. Greetings, Miss Brooks. And how is Madison's Ferris teacher this balmy June morning? Fairly balmy, thanks. Come in. Are we just having some coffee? Would you like something to eat, or have you had your breakfast, as if that made any difference? You're right on all counts, Miss Brooks. If there's one thing I can always do, usually it's eat. I've noticed that usually. Just sit down, Walter. I'll boil you a dozen eggs. No, no, no, thanks, Miss Brooks. I've got other things besides food on my mind today. Oh? It's Harriet Conklin. We've decided upon a June wedding. You and Harriet? That's right, Miss Brooks. That's why I'm too excited to think about food. So I'll just have some rolls and this cheese and marmalade and maybe a little honey with my Milton cookies. Well, I'd be eating when you're nervous. Now, what is all this about June weddings? You and Harriet are just kids, Walter. Oh, we're not planning to get married this June, but we're thinking of setting the date for some future June. You see, I've still got two more years of high school, and then I want to go to college.  Definitely not, Walter. Well, that's what Harriet thinks, too. Speaking as one who is intimately acquainted with my scholastic achievements, Miss Brooks, what year do you think we can shoot at? 1981. Oh, there you go, kidding me again. Now, I think 61 should do it very nicely. First, there's some slight opposition to our impending nuptials from a quarter very close to my intended. Namely? Your father. Mr. Coughlin can't stand the sight of me. What's so funny about that? Oh, I'm not laughing at that.  Ridiculous? Well, sure.  You're not giving yourself any of the best of it, Walter. Why, in six years, the entire picture could change. It can? Of course. Mr. Coughlin may be wearing a tuxedo.  It's a shame that Mr. Boynton's so... well, so non-aggressive. We have a non-aggression pact. But he's improving all the time. We've got a red hot date set for this afternoon. No kidding. Where's he taking you? To a matinee square dance at the Armory. A square dance, huh? Boy, you sure got a head start for it. What's that? Well, if you'll pardon my frankness, Miss Brooks. The kids at school, although they're all very fond of Mr. Boynton, I have a pet expression about him. What is it? They say he's got corners on his head. You mean the kids think Mr. Boynton is a square? He ain't round. Please, Walter, he's not round. You can say that again. I just did. Now, what Mr. Boynton needs is some of the old oil. Old oil? Yeah. Like Mr. LeBlanc, the French teacher has. Gosh, every time a woman comes near him, he is a walking gusher. Now, Walter, that's no way to talk about a member of the faculty. Mr. LeBlanc isn't oily at all. He's just polished. Polished and sleek and smooth and oily. He's a walking gusher. I mean, Mr. LeBlanc is a very fine French teacher. You want me to answer the door, Miss Brooks? No, Walter, you finish eating. You do finish some time, don't you? Coming! Bonjour, Miss Brooks. I hope I do not intrude. Well, speak of the devil. Pardon? I said speak of the devil, Mr. LeBlanc. Oh, but why? That is, why should an angel speak of the devil? Would you flow into the living room, please? Sit down, won't you? Thank you. I would like to come right to the point, Miss Brooks. Why, because we do not have time to beat any bushes. Fine, then let's not beat any. Miss Brooks, I would like for you to do me a little favor this afternoon. What kind of favor, Mr. LeBlanc? I would like you to marry me. Qu'est-ce que faire? That means, huh? I, I said I would like you to marry me this afternoon. Oh, of course it wouldn't be you I am really marrying with, but Giorgette. Giorgette? I will explain, Miss Brooks. Although I have long been an American citizen, I spent last summer in Paris. There, I met another school teacher. Giorgette? No, Jacques. Now, that's the point. Jacques, Walter. He, it was, who introduced me to this lovely girl and soon we go on, or how you say it, stay delayed. Goodly for you. For six months we are inseparable. You and Giorgette? No, the three of us. First, we are all just good friends, you know, but after I return to America, Giorgette falls madly in love with me. And not a minute too soon. Oh, now, you do not understand, Miss Brooks. I had almost forgotten Giorgette. Then, a few weeks ago, she asked me that Jacques has gone to the Riviera. She is very melancholy, and she wants to come to America immediately. Now, how can one come immediately without red tapes? Only one way. Marry an American citizen by proxy. This sounds like the French version of life can be beautiful. Mr. Le Blanche, if I understand you correctly, and I don't see why I should, you want me to marry you in Giorgette's name, is that right? Exactly, Miss Brooks. But it must be extremely confidential. I wouldn't want anyone at Madison to know about it. Please, Miss Brooks, say you'll do it. Well, I don't know, Mr. Le Blanche. I'd like to think it over. Well, there is no time to think it over. Giorgette has cabled me that she is anxious to become a June bride. Speaking for myself, I can only say grand chose une épouse de juin. What does that mean? June bride, big deal. You know how women are when they are getting married. No, but it might be fun finding out. Mr. Le Blanche, I'll do it. Wonderful, Miss Brooks. I'm so excited that the prospect of being a proxy June bride that I completely forgot about Walter Denton. By the time I returned to the dinette, he'd left by the back door after quietly eating two legs off the breakfast table. I was just about to join Mrs. Davis in the yard when the doorbell rang. Oh, hello, Mr. Boyman. Come in. No, thanks, Miss Brooks. I'd rather wait out here in the air for you.  To go? Sure. Do-si-do, form a line, come on in. The dancing's fine. Oh, that. Well, uh, do-si-do, you'll have to wait. Can't tell you why. I'm postponing our date. Postponing it? It's extremely confidential, Mr. Boynton. You see, it isn't just me. I mean, well, I'm doing something today that... You know, I've never seen you quite so flustered, Miss Brooks. Gosh, you're as nervous as a June bride. It should happen to both of us.  Then maybe later on I can explain it to you. All right, Miss Brooks, but try not to be too late.  You know, the square dance goes back a good way in our history. Originally of English origin, the hill folks in our southern states adapted it to their own rhythms and tempos. They changed its form and reorganized its pattern. Then they nursed it, rehearsed it, and gave out the news that the south land gave birth to the blues. It's a very interesting topic, Mr. Boynton, but I really must ask you to excuse me now. Oh, very well, Miss Brooks. You'll call me later on? Yes, I will. Oh, before you go, would you mind bending down a bit? Bending down? Yes, this way. Thanks. Those kids are crazy. He hasn't got any corners on his head. What? Oh, nothing, Mr. Boynton. I'll see you later. Oh, goodbye, Mr. Boynton. Excuse me, Connie, but I couldn't help over hearing your conversation just now. Oh, that's all right, Mrs. Davis. Nothing personal. Nothing personal? Do you always go square dancing with another man on your wedding day? Huh? Oh, Connie, it isn't that I don't want to see you happily married. What hurts is that you didn't take me into your confidence. But obviously I have. No, no, I had to hear it secondhand. Walter Denton told me before he left. He overheard you and Mr. LeBlanc through the living room door. Just what did he hear, Mrs. Davis? He heard all about the marriage license and the justice of the peace. Right after he told me about it, he left. Oh, great. That kid talks almost as fast as he eats. Well, Mr. LeBlanc may not know it, but he's just become the co-owner of the best-known secret in town. Mary, I'm glad I caught you out in the porch. I've got a... Quiet, Walter. Daddy's trying to take a nap on the hammock. Come on over here. Oh, okay, Harry. Now, guess what? What? Miss Brooks is getting married. No! Why, I never thought that Mr. Boynton had it in him. He hasn't. She's not marrying Mr. Boynton. She's marrying Mr. LeBlanc. The French teacher? Well, but he hasn't even... they haven't even gone together. Yeah, they'll be going together after today. They're getting married about one o'clock this afternoon. How do you know all this, Walter? There's nothing wrong with my ears, is there? That's when I heard him say he'd be back with the justice of the peace and the license. Oh, but this is awful. What about Mr. Boynton? I know. She's just doing this because it's June and he didn't propose to her again. It's a spiked marriage. That's what it is. I guess you're right, Harriet. We can't let her go through with it. But what do we do? We're just pupils of hers. Oh, you're right. This requires adult advice. We've got to wake Daddy. Okay. Well, go ahead. Wake him. Oh, no, no. Not me, Harriet. I've seen our esteemed principal when somebody wakes him up. You'd better do it. He wouldn't swing on a girl. All right. Daddy? Daddy, get up. You've got to get up, Daddy. What? What's the trouble? Oh, I'm glad you woke me, Harriet. I was having a nightmare. There I was in a school room teaching again. Listen, Daddy. And that's not all. The class consisted of 40 boys, all with one face. Walter Denton's face. It was terrible. Daddy, look who's here. What's the matter? It's just a dream, Denton. I'm bound to wake up pretty soon. You've got to help us, Daddy. Miss Brooks is getting married. Now I know I'm dreaming. Harriet, did you say Miss Brooks is getting married? That's just what I said. Hmm. I didn't think Mr. Boynton had it in him. Well, at the risk of sounding repetitious, he hasn't. She's marrying Mr. LeBlanc. But it's just despite marriage, Daddy, and you've got to stop it. Me? What have I got to do with all this? But don't you see, Daddy, if she marries the wrong man, it'll not only ruin her life, but also ruin her career as one of Madison's best teachers. Hmm. Well, experienced teachers are becoming harder to get right now. What do you suggest, my dear? I suggest we contact Mr. Boynton. Thank you, dear. We'll contact him.  I was talking to my daughter. Walter's right. We should get in touch with Mr. Boynton. But, meanwhile, we'll have to plan some delaying actions. Well, this reminds me of my army days. As the next major, I shouldn't have any trouble in deploying my forces with the utmost efficiency in this campaign. Harriet, you get my shoes? Yes, sir. And what should I do, Major? You? About this? What? What? Brooks agreed to act as a proxy bride so that Monsieur Leblanc could marry his French sweetheart. She overlooked the fact that Walter Denton was in the kitchen at the time. Since then, Madison's version of Paul Revere has erroneously taken the news to every Hamilton town that Miss Brooks intends to spend the rest of her life cooking crepes sousettes. Later that day, Mrs. Davis was helping her prepare for the wedding. Oh, that was a very cute idea you had, Mrs. Davis, for me to wear a white dress and your old bridal veil. Mr. Leblanc should get a big kick out of it. You do look lovely, Connie. But I wish it were Mr. Boynton you were marrying and not as a proxy. Oh, I'll answer it, Connie. Well, it's Mr. Leblanc. How do you do, Mrs. Davis? This is Judge Stevens. How do you do, Mrs. Davis? I'm glad to know you, Judge. Come in, won't you? Thank you. Miss Brooks is right in the living room here. Just follow me. Ah, there you are, Miss Brooks. This is the justice who will marry us, Judge Stevens, Miss Brooks. Glad to know you, my dear. I must say you make a most ravishing bride. Well, I hope to, someday. Someday? But aren't you being married today? Of course we are, of course. Judge, would you mind opening the door, please? It's a bit warm in here. Very well. I did not get a chance to explain everything to the justice, Miss Brooks. I have many papers to sign, you know, to be able to show her setting her where she was being married. But now it's all set. Oh, good. Now then, let's proceed, shall we? There are several services to choose from, folks. Do you want a lengthy one or a brief one? Oh, the brief one, by all means. I've got a date to go to a square dance as soon as it's over. A square dance? Isn't that a little unusual, Mr. Leblanc? Oh, Mr. Leblanc isn't going with me. No, I have a date to go bowling. No, you don't, but you will in a minute. You see, Judge, Miss Brooks is just a proxy for one Georgette Duval. I have the papers right here that say when this ceremony is over, I am married with Georgette. Well, this is certainly one of the most unusual ceremonies I've ever performed. Me too, but we'd better start performing if Georgette's going to be married this month. Personally, I always say, grand chose une épouse de joie. What does that mean? Tune bride, big deal. Yes, now let's see. We'll need two witnesses. I'll be a witness. And I'll be the other one. Oh, but you are the bride. You cannot witness your own wedding. Who can? Where do you see me catch the bridal bouquet I throw? This is all highly irregular, but in view of the circumstances, I guess we can waive the other witnesses. Now let's take up our positions in front of this window and begin. Ah, there we are. Dearly beloved, we are gathered here to join this man and this woman in the farce of matrimony. And so if anyone here knows of any reason why these two should not be united in matrimony, let him speak now or forever. Hey! Who goes there? Why, it's Mr. Conklin and Harriet. You can't do this thing, Miss Brooks. Oh, please Harriet, it's practically done. Now if you and Mr. Conklin will just sit down. Sit down? It will just take a few minutes more, Mr. Conklin. You stay out of this, Leblanc.  Of course not. She's just doing me a favor. A favor? Big-hearted Brooks, they call me. Perhaps I had better explain. You see, Mr. Conklin, Miss Brooks is just a proxy for a girl in Paris. We can come to this country only when we are man and wife. You mean Miss Brooks isn't really getting married at all? Don't make it sound so final, Harriet, but that's the idea. May I proceed now? With your permission, Mr. Conklin. Oh, of course. Go right ahead. Wait till I get my hands on that blabbermouth dinner. Let's see now. Where was I? Oh, yes. If anyone here knows of any reason why these two should not be united in matrimony, let him speak now or forever. Hold the presses! Stop this wedding at once! Well, here comes blabbermouth now. You've made a terrible mistake, Walden. I've made a mistake. Miss Brooks is the one who's making the mistake. But fortunately, I'm here in time to rectify it. They come here, Denden. Well, Mr. Conklin, have I or have I not successfully completed my mission?  Mr. Boynton, you better come in now. I wish someone would tell me what this is all... Miss Brooks, Mr. Le Blanche, Mrs. Davis. Oh, now, do not get the wrong impression, Mr. Boynton. This wedding is merely a formality. That ought to put his mind at ease. This is terribly sudden, isn't it, Miss Brooks? It's just a proxy wedding, Mr. Boynton. Miss Brooks is substituting for a girl in Paris. A girl in Paris? I wish I could finish this ceremony. Yes, please do. If anyone else here knows of any reason why these two should not be united in matrimony, let him speak now or forever. Excuse me, but I've got a cablegram here for Mr. Le Blanche. I am Paul Le Blanche. Let me see the cablegram. It's from Georgette. It says, please cancel plans for proxy wedding. Jacques Poirot and I were married this morning. I would like to have waited a while, but Jacques insisted that we be married this month. Sorry, Georgette. A p.s. June Bride. Big deal. Well, I guess there won't be anything for me to do around here. Just a minute, Judge. I don't think all these preparations should go to waste. Why, Mr. Boynton, what are you driving at? You'll see. Miss Brooks, please stand over here by my side. Yes, Mr. Boynton. Walter and Harriet, you get over there. Okay, Mr. Boynton. Mrs. Davis, Mr. Conkin, on our left. Thank you. And now, Mr. Boynton? Well, now we can have our own square dance. Will she go and form a line? Come on in, the dancing's fine. Oh, look at me instead of being wed. I'm dancing with the fellows. Got corners on his head. Mr. Bride was produced and directed by Larry Burns, written by Joe Krillin and Al Lewis in the music of Wilbur Hatch. Mr. Conkin was played by Gail Gordon.


Hi, folks. Good evening. My name is Manasi Bhattak, and I'm the founder and CEO of Verda. We are an ML infrastructure company providing a platform for the delivery operations and management of ML models. And today I'm going to be focusing on a topic that has come up time and time again with our clients and partners, which is when you're serving ML models, doesn't make sense to adopt a serverless infrastructure. So with that, let's get started. A bit of background on myself as well as Verda. So I found a Verda based on my PhD work at MIT. This was on a open source model management and versioning system called ModelDB. Just as Git is the de facto version control system for source code, we found out that models didn't have an equivalent way to version or manage them. So we built this open source system that is now maintained by Verda. And we have expanded that significantly to provide an end-to-end ML ops platform, just like DevOps and the tooling around DevOps has enabled software teams to ship code more frequently and more reliably. The Verda ML ops platform helps ML teams ship models more frequently and in a reliable fashion. So the reason this talk and the work that we're presenting in this talk today came about was that we help a lot of clients and ML teams deploy their ML models. And serverless is a very appealing paradigm in this case because it provides scaling ability. It also has a promise of reducing costs because you don't keep resources around that you don't need. And so what we decided was that we would run a set of benchmarks comparing ML inference in a serverless setting compared to a non-serverless setting and so provide the sort of best advice to our clients. So today I'm excited to share some of that work with you all. And without further ado, let me get started. What I'll be covering in the remainder of my talk is what a serverless, why is it interesting, unique considerations for ML serving that come into play. Then we'll go to the benchmark and results there. And then I'll wrap up with some key takeaways and how you can determine whether serverless is appropriate for you. So serverless, what is it and why is it interesting. If you think about the different ways in which you can run software applications today, you'll find that there's largely four categories. One is you can start with a bare metal server, manage the hardware and then run applications on top of it. The next one up the chain is virtual machines. These abstract away the hardware and you can have multiple guest machines running on your host machine. Further up the stack we have containers that help us abstract away the operating system entirely and we're left with only the application and its dependencies. At the very top we have serverless where we're only thinking about the function that we want to execute. We're not really thinking about how to run it or how to scale it. That's the value add that is brought in by a serverless platform. As you might imagine as we go from bare metal to serverless we're going up the abstraction chain, which means it's easier to use and yet it has less control and less flexibility as a result. Today I'm going to be focusing on serverless and containers and comparing these two as ways to run ML workloads. Serverless can be a confusing term for many people and so let me throw out some definitions first. This is an abstract one that I really like by Martin Fowler. It says serverless is at its most simple and outsourcing solution. What that means is with serverless you don't need a provision or manage the servers. Also your application doesn't have a long running server component. You don't have a loop that's sitting around waiting for application to get requests. Instead it is much more an event driven paradigm where every time a serverless function is invoked the underlying platform is going to spin up one or multiple copies of the serverless function and then that is going to be used to service requests. As a developer you're only writing the coder business logic. You don't need to figure out how to run or deploy it. The platform is going to take care of that. In addition the platform is going to take care of scaling up and down of resources. If you have a certain burst because your app is very very popular the serverless system is going to scale up to support that. In contrast if it's a holiday and no one's using the app that you've built for workplace productivity then it's going to scale to zero until there are requests. Finally serverless has many different flavors. What I'm talking about in this talk is function as a service. Some of the popular serverless systems are Lambda's, GCP, Cloud Run, Cloud Functions, etc. That's what serverless is. Let's quickly look at why serverless is interesting. Some of the pros and cons when you are choosing to adopt a serverless setting. First of all serverless is easy to use. Developers focus on business logic, don't need to worry about how an application is going to run. Second it's able to scale. The more requests you get and as your workload varies more copies of the serverless function are created automatically. Similarly it's going to scale to zero when your applications is not receiving requests. Overall this means that the maintenance overhead of infrastructure is very low. You don't need to manage the nodes, perform upgrades, tune resource requirements and so on. As a result overall whether we're talking about total cost of ownership or in certain cases depending on the workload even the infrastructure cost can be lower for a serverless system compared to a non-serverless system. So some popular applications that are good fits for serverless include asynchronous message processing, IoT workloads, stream processing and so on. So those are cases where serverless is good fit but that doesn't mean the serverless is a good fit everywhere. For instance serverless applications are stateless. So if you have an application that does require a state to be present in it then serverless may not be a good fit here. Second there are implementation restrictions. So whether you're talking about lambdas or a cloud run there are limits on how long a function can run, the resources it can ask for, concurrency and so on. In addition you can choose or control the hardware. So if you're looking for a particular processor or accelerator the serverless platform might not support it. However a Kubernetes based platform is likely to support that. And then finally because the serverless applications scale to zero when they're not in use whenever the first request comes in and there are no instances running there can be a large latency which can result from cold start. You're just spinning up the first instance of this particular function you're downloading dependencies and that can take a while. So given this background when does serverless make sense usually? Well it makes sense when the application can be made stateless. This usually not a deal breaker. You just store the state in a database, in a file store, blob store and so on. The second one is that resource requirements are modest. So they fit into what the serverless platform expects as typical resource requirements. The third one is that performance requirements or SLAs are not super stringent because as mentioned earlier cold start is a real issue. And so if there are type performance requirements here around latency in particular then serverless needs to be used with care. And then finally serverless makes sense when query workloads are not steady. If you have a very steady workload that uses your servers pretty effectively then serverless doesn't make as much sense because it's not going to save you resources and therefore costs as well. And then finally serverless really helps with reducing infrastructure burden. And so if your team really has a lot of burden around managing infrastructure serverless would make sense. So with that we can look at what ML surveying entails and how serverless plays with that. So you need considerations for ML surveying. First of all ML surveying means making predictions against a trained model. So there are different stages of the ML lifecycle. Your first training, your first cleaning the data, you're preparing the data, then you're training the model and then you're making predictions against the trained model. So we're talking about that third piece of the ML lifecycle. And the first consideration that's unique for ML is that models can be large. And what we mean by that is a model can have millions and millions of parameters. As a result the serialized version of the model can be hundreds of gigabytes. So Distilbert one of the smaller NLP models is actually 256 MB. If you compare that to your vanilla Python function that you might run, that's fairly large. And we'll see how we can run into problems when we're running a Distilbert model in a serverless setting. We run into similar challenges for ML libraries. ML libraries can be varied. They can also have a lot of dependencies. And as a result, loading up these libraries can hit some of the walls that are imposed by resource restrictions for serverless platforms. Then finally ML models may need to be served via GPUs or TPUs. They're more efficient that way and serverless systems currently don't enable that. So we now have a better sense of where serverless makes sense and how ML serving workloads might compare to that particular situation. So next let's look at the benchmark. Our goal with this benchmark was to identify when it makes sense to use serverless for ML serving. And we evaluated that question on a variety of systems. Today I'm going to be focusing on three systems. These are managed services. And we picked these because we found that these systems were most mature among the offerings out there. So first one is lambdas. This is the state of the art in serverless. The next one is running serverless on Kubernetes. This is using the Knative project and we chose to use the Google Cloud Run platform in order to perform the benchmark. Finally for a container-based platform, we use the Verda system. And what that entails is that Verda packages and runs models as containers and does the scaling automatically similar to a serverless setting. So let's quickly look at how each one of these systems works because that'll help inform the analysis will perform shortly. So for AWS Lambda, this is the managed serverless platform. You write code for the Lambda function. You upload the package code to S3. Also upload any other dependencies. And then every time that the Lambda is triggered via an event or an HTTP request, the platform is going to spin up a Lambda instance. It's going to execute it. It's going to scale it up and down as required. And when the request is done and there are no other requests, it will scale to zero. So that's generally how a Lambda works on AWS. If we look at Google Cloud Run, it's pretty similar except now we're talking about containers as opposed to just functions written in a particular language. So in this case, what the end user has to do is they upload a Docker container to the container registry. And they tell Google Cloud Run about the container, how it should be invoked. And then every time that there is an event or HTTP requests, it's going to trigger a Cloud Run execution. And as before, the platform manages all resources, scaling and endpoints. Finally with Verda, as I mentioned before, the platform runs models as containers on Kubernetes. And how it works is that the model is uploaded to the Verda platform. You can optionally specify resource and hardware requirements that you are looking to have met. And then you deploy the model on Verda. Notice that the step was missing in the serverless workflows. And that's because with serverless, you can spin up serverless instances on the fly. However, in this case, we deploy at least one, we will always have at least one replica running for the model. And so that reduces our cold start costs. However, it does mean that if there are no requests, we still have the model running at all times. And then similar to serverless platforms, Verda manages just the endpoints and scaling of models. All right. The last part of the benchmark spec is metrics. We measure warm start latency, cold start latency, auto scaling latency, and also usability concerns. Like can you actually use serverless for this workload or are there hard barriers which make it impossible to use serverless? Finally, for the workloads, we tested these systems on a variety of models, including NLP, computer vision, traditional ML models. Today, I'll be focusing on just a couple of NLP models because the trends are pretty similar across the board. And we have different workloads that vary QPS or queries per second. All right. So before I go to the results, I do want to point out a few caveats. The first one is that serverless for Kubernetes is still evolving. And so the numbers that I'm presenting here are based on the capabilities and software available today. If you rerun the numbers in two months, they might be different. And so what we have done is that this benchmark is a living benchmark. And you can find the full set of results at that URL and also learn about how you can run the benchmark yourself. Second one is we're using managed services in this benchmark. And as with all managed services, there are knobs that cannot be seen or controlled by the end user. And there are likely optimizations performed under the hood, which means that it's a bit challenging to perform a apples to apples comparison. The best that we can do and what we've done in this benchmark is to use best practices recommended by the cloud providers and to use off-the-shelf settings for the managed services. So let's first get started with results around usability, because if you're not able to even use the serverless platform for a particular model, then the quantitative numbers don't make sense. So the biggest hurdle that we found in using serverless for ML has to do with the restrictions that are put on resources. So here I'm listing out the restrictions that are put on Lambda and Cloud Run with respect to memory disk and CPU. And what this means is that if you require memory greater than 3 gigabytes, then lambdas are not an option for you. You cannot use the serverless platform. Similarly, if you need more than four virtual CPUs, it's a no-go on Google Cloud Run. You're just not going to be able to do that. Now with ML, as we discussed before, ML models and ML libraries tend to be rather large. And so it happens more often than not that your ML model doesn't fit in the constraints imposed by the serverless platform. To give you two examples, the first one has to do with distilled BERT. And to run distilled BERT, what you have to do is install Torch and to install the transformer library from Huggingface. In this case, both of them together are greater than 500 megabytes. And for Lambda, this falls outside of the realm of what's doable. And so what we had to do was to surgically remove pieces of library and code and get creative with model loading so that we could actually use distilled BERT on a Lambda. So if your model is larger than the expected resource constraint, you need to get very creative and significant wrangling is involved there. The second one is another example that we have seen with multiple clients. There's an embedding model for an entity and then there's a nearest neighbor lookup model that's going to find entities that are most similar. And because this particular model includes an index that can be pretty large, the model ends up being greater than 20 gigabytes. That's way outside of the realm of any of the serverless platforms. And so you cannot use a serverless platform in order to serve this particular model. So last one I want to highlight here is configuration options. ML libraries and low level linage libraries have optimizations that can be tuned via environment variables. However, in a serverless setting, these environment variables or the hardware that determines these environment variables are not exposed to the user. And so it's hard to set these environment variables correctly. All right, so the set of results that we looked at so far have to do with usability. Can you actually use serverless for this workload? Next, we'll look at some numbers that assume that you've gotten to running your model on serverless. How effective or efficient is it at that? So first thing we'll look at is warm start prediction latency. Warm start is there's already a serverless application running and there's no need to download the model or spin up a instance from scratch. We see here that AWS Lambda and Google Cloud Run are pretty comparable across the P50, P95 and P99 latencies. Interestingly, the Verda platform is actually faster by 2x. This can be attributed to more control over the environment. So this can be processors that we use or specific environment variable settings. Since Lambda and Cloud Run are a closed source, it's hard to accurately identify what might be the reason why Verda is running 2x faster. But it likely has to do with the environment that we're running in. And here I'm noting the configurations that we use for all three systems. We have tried to keep all the configurations comparable so that we're doing a apples to apples comparison. The next result has to do with cold start prediction latency. So this is a case where no serverless functions are running already. And so this is the time required to spin up a new instance of the serverless function and then service a request. So in this case, we find that for Verda, we always have at least one replica running. And so it's extremely fast to make the first prediction. On the other hand, in the Lambda case and Cloud Run case, it takes several seconds in order to spin up the first instance of the serverless application. And so we find that the latency isn't the tenth of seconds as opposed to less than a second for Verda. So the previous results were around time to first prediction. The next set of results are for scaling latency. So one of the biggest advantages of serverless is its ability to scale based on the workload. And so here we're comparing the time required to reach steady state. Which means in this case, we were testing with 100 QPS, one worker per query. We're saying that we know we've reached steady state when we start receiving 100 responses per second. And this is successful responses. Here we see that Google Cloud Run is quite fast, 33 seconds. Lambdas are twice as slow. Verda is significantly slower. And this is to be expected because we're now only relying on the Kubernetes auto scaling. There are no optimizations being performed in order to always have a warm replica is ready to go. And so on. So the optimizations that a serverless system might implement in order to perform auto scaling are not present in the container based scaling based on vanilla Kubernetes. All right. The final results here is what happens when we vary the model size. So here we're comparing the still birth against birth. Birth is twice as large as the still birth. And we find that this does impact the latency. We see that the P 95 is twice as much. And the time to first request, however, is not that much longer. Particularly for AWS for Cloud Run and for Verda, we do see some degradation in the time to service the first request. So that's a quick overview of some of the key results in comparing serverless versus non serverless for ML survey. Before I wrap up with a key set of takeaways. I do have a note on cost because cost can be one of the key reasons why teams decided decide to go the serverless route. And this has to do with two things. One is if we discuss only pure infrastructure costs, depending on the workload and the amount of resources necessary, the pure infer cost may be comparable or even higher in a serverless setting than in a non serverless setting. So if you have steady state in your query workload and your queries have good utilization of the servers, it might be cheaper to just use a non serverless solution. However, one place where serverless does shine is TCO or total cost of ownership. This is a combination of infrastructure costs, maintenance costs and development costs. So if TCO is extremely important to you, then serverless would make sense. Otherwise, it requires a more nuanced view on what are the infrastructure costs that we're actually signing up for. Alright, so with that, let me quickly summarize what we learned based on our benchmarking results. And hopefully this is helpful for people to make their own decisions on whether serverless is a good fit for your ML serving use case. So first of all, serverless solutions have hard limits on resources. So if your model doesn't fit within these resources, then serverless might not be a good fit for you. Second, the ability to configure hardware can lead to better performance and this is easier in non serverless systems. So this is the case where the Verda system was 2x faster and it had to do with the environment that we were running in. Next, scaling with the serverless platforms is much faster than the vanilla auto scaling that we could do in Kubernetes even with custom metrics. There's special optimizations that are performed by these platforms for auto scaling that aren't provided out of the box if you're running containers on Kubernetes. And then finally, the query pattern and workload impacts cost. And so in order to assess whether serverless is cheaper than non serverless for prediction requests to ML models, we need to look closely at the query patterns and workloads. So those were the key points that I wanted to convey regarding the benchmark. Please check out Verda.ai serverless inference benchmark for the whole set of results and also to learn how you can run the benchmark yourself. So with that, thanks very much for your attention. I would love to continue the conversation around the benchmark. Please feel free to reach out and monocle at Verda or data serial with questions and I'd be happy to feel them there. Thank you.


Welcome, everyone. Once again, let's talk about urban planning. And today we'll look at citizen co-production, where citizens are involved in shaping their cities. And while this is often seen as positive, my guests today, Leica Aruga and co-authors, argue that simply involving citizens does not guarantee better outcomes. So in this conversation with Leica, we explore two examples in Norway to highlight the challenges that are posed by limited participation, limited knowledge, and sometimes limited transparency. Leica, welcome to our episode. Thank you for having me. So Leica, citizen co-production is becoming increasingly popular, especially in sustainable city initiatives. So I believe this is why you decided to conduct the research on this topic. Right. So co-production in urban planning context is often talked about as a necessity. This is because the challenges faced by our societies cannot be addressed unless there are collaborations across different sectors and different actors. We approached co-production as a process that generates not only knowledge, new knowledge, but also understandings of what sustainability means in different places. And we focused on the role of citizens because situated understandings of sustainability can't be obtained without their involvement. Let's follow up on that. I read your article and you mentioned that the current discussion on co-production often overlooks power imbalances between citizens and institutions. So was this devoid in the research that you wanted to fill? Exactly. Studies from before have pointed out already that the role of power and politics in co-production hasn't been addressed enough. And most studies focus on why co-production is needed or how it could be carried out. So that's why we were very much interested in the transformative potential of co-production. That is whether co-production could alter problematic power dynamics that reproduce inequalities and unsustainable practices. So we focused on understanding how inclusion and exclusion take place during the design and implementation of two co-production practices. And this made us question how the citizens as a subject was also being co-produced in the process. Well, promising. Let us know the findings then. So we looked specifically at two cases of localizing sustainability efforts. They took the form of democratic innovations specifically as citizen panels that was implemented by the Trondheim municipality in Norway. We chose the cases because of their specific commitments to making an inclusive and deliberative participatory space. We interviewed those who were engaged in the panels and we learned that the way citizens with diverse backgrounds were gathered and they discussed how to make their own neighborhoods and city more sustainable. But as we looked closer at how the citizens were brought into the panels, also how their roles were perceived during deliberation in relation to knowledge experts and professional facilitators, we found contradictions in the way citizens were expected to contribute on one side as individuals with no prior knowledge or group interests, but on the other side also being expected to represent different social categories. So the depiction of the citizen as a ghost subject that we use in our article underscores that the ideal citizen as imagined in many co-production processes simply does not and cannot exist. Let's follow up on that. So I'm particularly interested now about policy implications and implications for individual choice potentially. Well, our study pointed to one possibility of explaining why co-production practices often fall short of what their promise to achieve. So it is important to unpack what co-production does and to reflect on whose knowledge, values and also worldviews are actually making a difference. And without doing so, we also risk reproducing problematic discourses around sustainability, which becomes legitimized through cosmetic engagement of citizens. And future research on citizen co-production should focus on what more scrutiny, for example, or more geographies. What do you think it's ahead of us? Well, our analysis was made with an intersectional way of thinking that dynamically conceptualizes power relations, identities and social belongings. So this really complicates the question of who participates because a citizen subject is constantly being shaped and enacted throughout the co-production process. But also in relation to their social belongings outside the spaces of intervention. So we would like to see more research that make use of intersectional perspectives that can critically unpack co-production practices in terms of who participates, but also the political implication of their participation or lack of their participation. Well, some tips for future research. Leica, let us know your... I'm very interested to know about your personal reflection, so getting critical about your study. What are your thoughts? Well, our intention was really not to evaluate the citizen panels, but... And we want to acknowledge that the municipality actually placed a lot of effort and resources in these additional spaces to engage citizens. And when you speak to panel members, they were quite positive about their experience that they were able to meet and discuss with people from diverse backgrounds about the sustainability challenges in their own areas. But still, when we asked the panel members what were the outcomes of these initiatives, they were quite... They were not so sure about it. And partly, this could be because to know what came out of the panels also required their continued political engagement. So our takeaway is also that without having these collective spaces for engagement over time, it really makes it difficult to challenge any underlying premises of smart sustainability discourses that could be problematic. Perfect. This has been a very straight-to-the-point episode, the way we like it. But I would like to ask you, if there was a punchline for this episode, one or two sentences that would wrap it up, what would it be? So intersectional approaches can challenge taken-for-granted assumptions we make in co-production and participatory processes in general. And it's really about keeping the attention directed at complex and challenging power dynamics that are not always giving the attention they need. So our hope is that we could inspire more research to come that builds on intersectionality. Perfect. We haven't got the perfect citizen, but the perfect episode. Thank you, Leica, for this episode. Thank you so much. Thank you very much.