Okay, so I think that the room is very full, so I'm really glad to be here. Hi everyone, I'm Christophe Fontaine, and I shouldn't be there, actually, because I'm here because Amiens, Portier, and Veda were not able to attend this conference, so they asked me to present that topic here, but I'm happy to be there and to see all those familiar faces. So what we will be talking about is BGP today. How do we enable multi-cluster connectivity in an efficient manner? But what are we talking about? Because are we talking about deployments? Does it work? And on the efficiency, we'll see the different performance metrics. So it will be a very small performance metrics, but keep it up with me. And I will try to make it quick, so we'll have plenty of time for Q&A. So about Spine and Leaf deployment challenge, you already know that. There is nothing new, the Spine and Leaf deployments are somehow common now. We do have some challenges on the controller side because we have to replace all the layer two mechanism with the layer three capability mechanism to deploy the controller nodes on three different sites and on the compute side, well, somewhat somehow that's easy to do as well. The actual challenge is about the top of our configuration. How do we integrate all of that? We could go with uncivil networking, so that's great, but sometimes it's not enough. Where the fun begins is about the multi-cluster story. How do we integrate that? When we have one tenant, one customer, which deploys a set of virtual machines across multiple data centers or across multiple regions, how do we interconnect that? But basically, even that issue already solved. Because with Neutron ML2, we can bring third-party SDN vendors and that's featureful. We already have a solution to interconnect multiple clusters with BGP and honestly, it just works. You also have, thanks to those third-party SDN vendors, the ability to configure the physical switches in addition to the virtual switch. But sometimes the cost means that we have to do something upstream. And what can we do? The solution is called oven BGP agents. We don't have two drivers, one for each use case basically. The first one for BGP as a simple BGP speaker and the other one, so and that's what we will mainly focus today on is the EVPN driver. So how do we enable multi-cluster connectivity through BGP? And I will not go over and over that presentation because some colleagues already did that last year in Berlin. They already presented the war architecture and how they integrated that into OpenStack. Unfortunately, they weren't there today, but. On the EVPN, compared to the BGP driver, the EVPN also brings the compatibility with the actual OpenStack API so that on the tenant side, you do have the ability to integrate and to interconnect multiple tenants. And the workflow is fairly simple because you have an OpenStack CLI, OpenStack API in order to configure the proper database and the database will be updated within Neutron and you will have all the hooks within the EVPN driver. And voila, you do have those interconnection. So honestly, if it works, don't fix it. But honestly, does it work for everyone? Because if we look at that schema specifically, what do we have? We have the database, so directly from the virtual machine to the integration bridge on one node, your packet is teleported directly to the network node then again within the integration bridge and then you have, what do we have? The kernel routing. What if on those nodes, on those obvious bridges, we have obvious DBDK and we'll go very simple with obvious DBDK? We still have the kernel routing. So this means that we have to print all the packets from the user space down to the kernel space and this action will actually slow down the complete database, unfortunately. So is that efficient enough? We're saying that the kernel is actually slowing down the whole database. Don't get me wrong, the kernel stack is awesome. It has a lot of features on the ability to terminate the TCP connections. It's doing a marvelous job, really. So we could say that it's really fantastic for 99% of the IT workloads. But sometimes we just want to push the cursor a bit further and to go one way, one step above. How do we remove that bottleneck for VMs which are exposed through BGP when they use obvious DBDK, Azure VE, obvious other offload, those acceleration technologies? So what do we need? So we have a DBDK app within a virtual machine with all the tunings that we have. So huge pages, CPU dedicated calls, and so on and so forth. We do have over and with obvious DBDK, once again with huge pages and dedicated calls, and then we have a kernel router. So the answer is simple. We need a DBDK-based virtual router, but how can we do that? If it was as simple as that, we would already have done that. And we have multiple options. And OBS stands for Open Virtual Switch. So that's a layer three, and that's not a virtual router as is. It still has some routing capability because when you have a journey tunnel, what do you do? You are actually injecting that packet into that tunnel. And if you will look at the open floor rules, well, the packet is directly routed by OBS. So how does it do that? Basically, OBS already has some routing capability specifically for tunnels. That route is already configured thanks to a net link because it listens to the routing table. So we have a copy of the kernel routing table directly within Open Virtual Switch. And that's great because it works for edges with OBS ML2 before and OVN ML2 today. So with OBS with VXLan tunnels and OBS with Genive Tunnels today. But it was meant in definitely only for tunnels and for tunneling. So does it support VRF? Does it support IPv4 and IPv6 stacks in order to answer to ICMP message or neighbor discovery messages? As of today, no. So this is really only a layer three switch. So this means that we would have to implement all those different features to make it and to transform it as a router. And because it's a switch and not a router, we also have additional issues. So what if you have an empty mismatch? You cannot fragment the packet. Okay, it's not any more an issue with IPv6, but we still have a lot of IPv4 workloads today. So this means that even on the upstream community, this would be extremely difficult to integrate, not impossible, but to transform a switch into a router that would require a lot of effort. And we need that feature now. So what can we do different? We already have a router within the logical router in the oven. So why can't we use it? Because basically we need to inject the proper routing flow directly open V-switch. And oven is meant for that. So let's try to reuse it. I talk about ARP messages and ICMP V6. It's already handled because we do have that oven controller running locally. So all those packets can be directly injected to an oven controller and it will do its magic to either generate or answer those ARP and the neighbor discovery messages. So on that front, we're covered. But basically in order to have that local open flow rule, what do we need to have? We need to have that routes declared into oven Northbound Database directly within the logical router. But how do we access to that Northbound Database through the Neutron API? That means that for every chassis as gateway nodes, for every network of nodes, we would need to have a kind of a feedback look directly from oven BGP agents, so directly FR, which would inject the routes into the Neutron API. And then a Neutron API would do its magic to forward the API and call the proper oven ML2 plugin so that we'll have the route into Northbound Database that would be transformed into the proper logical flow in the Southbound Database. And finally, transform with oven controller as an open flow rule within that integration bridge. But that's somehow difficult because with one node, that's great. But what if you have 1,000 nodes? Because with the BGP driver in oven BGP agents, you have one FR instance running on, and so one oven BGP container running on all compute nodes. This is why it's going to be difficult to do that because we cannot scale it. Let's put it that. So the third and final option was, okay, we do have one oven cluster. What if we bring another one? That other one would be extremely small because it would be running locally on each network node. But the concept would be exactly the same. Oven BGP agents would just forward the routes locally to the other oven cluster, well, single node oven, and that would push the proper routes directly to another bridge. So that was the approach that was taken by Lewis. And yeah, so what are the numbers? Because when we're talking about numbers and performances, we need to think globally because on the control plane side, it's not a control plane issue anymore because we do have an end-to-end DPDK enabled database. So how did we measure that? It was a very simple setup and only one deployment in that specific case with one provider network so that we have one leg directly connected to DPDK-based application. And on the external network, we do have a floating IP held on the BRBGP. So everything's great. And on the traffic generator side, we decided to use TRX because that's what we use on a daily basis, at least in Red Hat, in order to generate and simulate the proper traffic. So on the virtual machine, we actually have open V-switch, or VSDBDK as well, because we could have bring a third-party DPDK-based router but we already have that L3-switch, which is obviously DPDK, so let's use it as well. And everything is configured, I would say, in a standard NAV configuration. So with huge pages, dedicated CPUs, we do have the proper isolation with Tundi and the CPU partitioning profile, both on the supervisor and within the guest. And the only thing that we modified is that we either deploy with the second oven cluster or without it, so either with the kernel data pass or with the DPDK data pass. The kind of tests and traffic that we inject is fairly similar to the IFT 2544, so that's 64 byte frame size, so that's really small, but this is a standard kind of test. On the configuration, we also have multiple routes with static routes and so on and so forth, but honestly, nothing fancy. What does that mean? If it's not fancy, it just means that we are just back to the existing performance number that we have of kernel versus OVSDBDK, because we have a dual port in Telnick with 10 gigabit interfaces and within OVS, within the virtual machine, we do have some specifically crafted open-flow routes really to simulate a Rusa. So, nothing fancy. As I said, on the kernel routing side, we are at roughly 100,000 packet per second, so it's not low, it's not high, that's basically what we have. And with OVSDBK, we were able to reach a bit more formed million packet per second, so that's a huge, huge difference. But, unfortunately, that's only the base case, because doing that means that we are not using any kind of NAT or any kind of fancy feature. This is really great for the multi cluster connectivity when you are just routing the packets from one leg, from one network to another one. What if we want to use the initial deployment, which is basically with a BGP driver? Does this mean that we don't have a floating IP? If we have a floating IP, we have NAT. And if we have that, it really means that we have contract. And, unfortunately, as we enable contract, the preference drops. Roughly, still on the same characteristics, we can go from four million packet per seconds down to 1.2, 1.3 million packet per seconds. Still, it's much better than what we could have on the candle space, because that's really a zero loss test. Also, what can we do? And I talk about OVSDC flower offloads, but what about those smartnicks? We still have two physical interfaces. So, on the compute side, nothing changes. We have either OVSDBK or OVS other offloads. And we are able, in the L2 mode, we're able to reach line rates with a ConnectX5, ConnectX6, with 10 to 25 gig interfaces. Honestly, that's completely flawless. We do have the SRVP performances. That's great. As I said, we don't want to have that grouting capability to be a bottleneck. So, let's try to put exactly the same setup on the network nodes. So, with OVSDC flower offloads. So, OVSDR offloads. And that's where it hurts. Why? That's because the physical to physical flow cannot be offloaded. That's an existing limitation on those ConnectXnicks. And we cannot do anything. So, on the TC flower story, that's great if you are only using one driver, which is the standard BGP driver. So, when you have the virtual machine and the external network directly connected on the single node. So, the virtual function to the physical function. So, the VM to the physical interface, you can offload that flow and you are able to be line rates. As soon as you move to a network node, so physical to physical, just like any other appliance, you cannot do that today. Of course, we're trying to improve that, but that's how it is. But the real question is, do we need TC flower offloads in that case? Because we have the other option, which is OVSDPDK. And with OVSDPDK, on the physical to physical forwarding, that's where the DPDK really shines. And we can be completely line rates flawlessly. In fact, with only one core, we're able to forward 29 million packets per second. And that's a network node. What does that mean? That means that we have the complete node dedicated for forwarding. So, we don't have two cores, we may have 20 cores to forward that. So, with 100 gig interfaces, routing is easy. And thankfully, even if we do have OVSDPDK and the contract, the user space contract issue, the latest improvements in the upstream, well, basically means that we can actually be line rate as well. And we can push much, much more data. Which means that we can still deploy OVSDP flower offload for your workloads on the compute node, but for the network nodes, as of today, we could go directly with OVSDPDK in a single deployment. So, what can we do better? BGP as an interior protocol. Is that really so great? That's an open question. Because we have something else which is called OSPF, which was basically meant for that. I see some weird looks, but that could be another option as well. And with open BGP agents, we are basically using FR. So, switching from one mode to another one could be an option. It's really depends on you, whether you tell us, okay, that's really interesting to have OSPF as an EGP and to keep BGP for, of course, the multi cluster connectivity, but maybe that's an option. We know that some hardware vendors charge much more as soon as you want to enable BGP compared to OSPF, which is enabled and available by default. So, is that something that would interest you? That's an open question. As I said, the physical to physical flow of loads, so-called the half pinning, so you enter into one interface, you may go out to the same interface, we are actually working on that. In fact, on the DPDK user space for the RT flow of load, we know that the CNX, the CNX driver is actually working on that. So, that could be great as well. Also, all those elements are only BGP speakers who are only announcing either a single IP, so it's 32 slash 128 or eventually a network. But, do you see any use case that you will want to learn routes directly on the compute nodes? Because today, the upstream router is used as a route reflector, that's great, and that upstream router is also used as default gateway, so we don't need today to learn routes directly on the compute node. Do you think that it's something that could be valuable for you as well? So, basically, the open question is what do you need? And I kept my promise that we have 10 minutes, complete 10 minutes for Q&A. So, the mic is open. If anyone wants to ask some questions, and if you're too shy to do that, yes, please. Excuse me? The, no, no. Today, that's a new VPN type five that is used for the multi-cloud interconnect for the BGP VPN, EVP and VX LAN. I don't think so, no. And, is there any specific use case that you would prefer type two instead of type five? Yes, there is another, I'll be separate. Okay, yes? In terms of scalability, how many, for basically, like, zero location in geography, or what type of, how many stacks we're talking about here? Oh, actually, it's about the interconnect, connectivity, so there is no limit in the sense that it's a controlled plane protocol. So, if it's about only reaching the other cluster, well, there is no limit in that sense. You will, of course, have the latency issue between two data centers, so of course, we cannot do anything about that, but on the scale size, basically no issue. Well, on my team, we really focus on the data plane preferences and how we can enable that feature for any of these use cases and not on the scalability side, as of today. Yes? Yes, when we said that we're line rate with the RFC-2544, that really means that we want to be able to push as many packets as the wire is capable of. So, for a 10 gigabit interface, that's roughly, 14, yes, 14 million packets per second. So, of course, for 100 gigs, that's 140 million packets per second. So, from a VGP peer perspective, actually, are you peering the four switches or are there's always... In that case, if I go back to the general architecture, yeah, we are only peering with the leaf switch, which are root reflectors. Yes? Yes? Can we go back to the subject of this VTK VTAR router? I don't know if I get it properly. You said that this is a separate of the N-stack running on the network node. So, yes, indeed. In that specific deployment, we will have two oven controllers running on the same network nodes, one dedicated for OpenStack and the other one dedicated, well, one dedicated for OpenStack with the integration bridge in the BRINs and another one dedicated with another bridge, which is BRBGP. So, indeed, we'll have two independent oven clusters, but that second cluster is really limited to that unique node. So, we will not create any kind of tunnels with the other network nodes. That's really local. So, don't we have a total interface between this two... Yes, we do have a patch port in between the BRINs and so it's not the actual WGP that you will find in the computer node that's a virtual one, but, well, that's a simplified one, but you will have indeed an intermediate bridge between the two, but everything is connected through patch ports, which means that, at the end, even if you have multiple bridges interconnected with patch ports, at the end, you will have only one open flow or one data pass flow. So, whatever the number of bridges that you have, you always end with a unique data pass flow. And that's why we can leverage a single open V-switch with DPDK. There is assumption that there is specific interface to doing this interconnection between the sides, right? That's a patch port. Yes. That's what my second question is. Only the OB and VGP and the SAC should be set for the OB and VGP. Absolutely. Yes. Yes, we always have a unique master for each element. So, on the open SAC side, we only have Neutron, which configures open NOSBAN database, and for the second open agent and second open cluster, it's only open VGP agent which is actually pushing routes to the NOSBAN database as well. Yes. It's like a query problem, FRR. Yes. What's the implication of FRR? Well, FRR is here and in order to peer with the actual router, and we are using our configure FRR in order to advertise the different networks or individual IPs.  Yes. Exactly. Exactly. FRR is actually run within the context of the open VGP agent container. So, that's within that. So, it's a separate... I don't know if it's a separate container within a single pod. Yes, I think it's that. So, if you don't have any other questions, anywhere, I will be hanging around here. So, yes? I just wanted to ask, what's the progress and what's the status of the project right now? Yes, you're a good question. So, as you can see today, the open VGP agent code has been pushed from the private, well, on GitHub repository directly to the OpenSec foundation. So, that's now under the initial number radar, and which means that the second open controller, we still have one or two patches that are still hanging around, but they should be merged. If not yet, it will be merged in the upcoming weeks. So, everything should be available upstream, if not already. Yes. So, the next question and the tricky question is about the installation phase and how do we deploy that? But that's completely outside of Open VGP agent project. Okay, so. Thank you very much.


So moving on to the closing I would just invite all the people who helped with the conference the Committee if the committee is here. I would request all the committee to come on stage the people who did the selection program selections Simon Dave I see Dave Anand is Anand around I'm not gonna give the mic to Simon We also had Marcus who's not here So thank you guys for helping putting this program together Can I invite all the speakers on stage next please all the speakers? Please don't be shy I believe few might have left, but I hope we catch the majority here Short people please stand in front of me if they want to Now's the time to give feedback In the form of vegetables and potatoes and eggs What do you guys like tomatoes or eggs Tomatoes So does anyone have anything that they want to give feedback to the speakers in general anything Thank you guys Thanks you speakers No We we forgot to forget we forgot to thank one person by the way, and that's Nourish who is done Without him. I don't think we would have been able to come to Bangalore. So thank you very much So are you guys ready for the next Selenium conference? You just need to take Simon to the side and discuss that with him I Guess we are looking at a different country for the next Selenium conference Handwaves user data and people willing to volunteer right so We probably will try and continue this community because we don't want this Fantastic community do kind of fade out if you don't have a conference So we would try and do a conference so we'll keep in touch and we try and organize something If not Selenium, maybe we will open it up a little bit to do other kinds of tools as well And kind of make it a more broader testing community But I would love to see the momentum continue and you know this community build on from here All right All right, thank you guys and hope you have a wonderful travel back home


So, I thought this shit, okay? You thought you'd laugh and do the dance, but you didn't do anything. So, first of all, I wasn't here for an episode, but I was here, but I could have, I could have clearly understood it when I watched it, that y'all were smelling this land, you, that you see, that was a splendid land, how the streets were wonderful, cool, this place was like a mischievous shit, and what did we do with this amazing apartment? It did not register.  You guys, you like this? Woo! Well, they always know the latest songs on a unit that they did, and they changed it to a crack-mokers song. Yeah, they always used it.  Oh, my God, baby, look. Hocus-pocus, hocus-pocus. Hocus-pocus, hocus-pocus, hocus-pocus. Oh!  It won't be me. Because the titties were wet, hey, we out in public. It's not the needies that just stayed wet here, it's the only thing that's been on our business. I'm like, I'm like, I'm like, I'm like, tell them this. Well, what, Clay? What? Tell them, Clay? I'm very like, I'm very close to them and it makes a thousand people.   It's out of the office. It's out of the space.  No, that's your kids.  They're in the space, it's out of the space, and this ad-division is just out of the air. That means this is it.  Whatever this guy not being. I'm just telling you.  But only if you are, only if you are for your own purposes that you can design the UPS form. Yeah, for the pain of constancy. That's right, though. Let them see the quirmum, man. We're in the quirmum right now. It's quirmum really big, We wouldn't have it any other way, yeah we wouldn't, we couldn't, but it's cold right here, check it out, it's cold.   He was in the Navy dude, he was in the Navy. Hey what's up man, mental health is a real issue which is not often discussed in our community. Post-traumatic stress, depression, all types of things could possibly be going on. You might want some insight that you can't get from your family and friends. I would like to introduce you to the professionals over at TalkSpace.com. TalkSpace.com is a virtual therapist that you can vent to about work, family, friends, relationships. You may just want to talk to somebody who's licensed in the medical field who can help guide you into some of these, you know, with some of these issues that you may or may not be having. So I would like to invite you to talk to the professionals at www.talkspace.com. Use my code south that is www.talkspace.com forward slash south. Using TalkSpace you can connect with your therapist online, video chat, text, email. www.talkspace.com forward slash south. Use my code so you can receive $30 off your first month subscription. Yeah. It's a movie. Ain't nothing new under this time, my brother. Nothing new. From what I heard, that ain't even the same moon that was up there when we was little boys. But I've been following B.O.B. on Instagram. This never got me crushed by anything. You see this here we got on the table here?  Oh, Motel! Got my big watch. Oh, good. You know, who folded these so nicely?  He has great, very folding, you know. Yeah, broke, put it in teal. That's just fresh. That's just so fresh. Yeah. Yeah. We need the double-eyed eyes with that. Come on, bro. The double-eyed eyes with those mucus. Come on, bro. I know what the word is. We double-eyed eyes with those mucus. Let's do your auntie favorite next. Double-eyed eyes. But this one right here is told us you got to see those. That's just like trees, bro. Yeah, that's a real bird. Come on, man. Come on. Look at that. That's just cool. Yeah. We got to move this. Make sure that's the fourth 85. That's just awesome. We just flexed it. You know what I mean? All white, man. All white is doing one. Come on, man. That's just a plug. Oh, my God. I only wanted to put these up here. Yeah. They got it. Yeah. They're all weird. They got it. They do, but they don't. You know what I mean? They don't think they do. They don't think they do. Right. Yeah. I like these. I like these because they won't.  But they will. Like, I don't even know what these kids do, but it's great. Who would ever fuck? Come on, man. You know who don't. That's your brother, right? Oh, no. What's that mean?  That's your brother.  You need to help him. He didn't do it. He heard the album. All white. That's his brother. Check it. Check it.  And then people go crazy. Right.  No. I didn't know it. I grabbed the information. I hold on to it. That's what he's known for. He's the information rapper. No. Understand? It just works.  It's right. That's right. That's right. No. I'll make a face on the button. Just kidding. It just wasn't a bad no. You're talking about beef well? Like I told you. To this nigga put you up on the protesta, okay? I knew you. Yeah, I knew you. I knew I would. Yeah, you did it. But you do that. Is it your problem, Caltopsy? Anyway, we need to be having it. Yeah, they say we dial it off early and stick it up on the protesta. In fact. Yes, the best thing is going to go with graduates who love them, man. I'm on grass. Everybody's eating. Black man need to know to import them. You need to know the importance of zinc. Zinc? Yeah, bro. It's zinc. It's zinc. I'm a genius, man. Vitamin A is the most important thing. Vitamin A? Come on, bro. The vitamin Z? Woo. Come on, man. Everybody love vitamin Z. Hold on. What's vitamin Z? That's why I don't even get the color. Yeah. Come on, bro. Yeah, I'll stop there and see the origin there. Come on, bro. The origin is how origins promote youth. Come on, bro. You understand what I'm saying? When you drink the orange juice, right? And you're going to spit it. Don't be straightening in the end. That's because you got the youth and the pumping. The youth and stuff. Can't the pump free, other way than to get vitamin Z orange juice? I'm pointing too much at you. Come on, man. You ain't downloaded? Come on, bro.  You ain't got that. You ain't got the last 아. No, because your grandma was drinking bright in the early years. The dubstands have got a dubborn. See, then I get like a real monster. He wasn't a nice person. That's why he was really cursing as you listen to him. He cursed the whole town. He was in a theater. What? He was in a theater. I can't read it all. Go ahead. God was done. Good tomorrow for me, God. He was the SD. Oh. He was in a Navy. He was in a Navy. You don't know, see, none of his Navy buddies. It was a tragedy. He was the only one that survived rounds. You were on the train. Oh, it's a tragedy. No point. No point. Out of your mind. He always was. And nobody could understand what he was saying. But this department now pays attention to it. It's a few episodes where Donald Duck did so mad. He pushed the hat down. Rolled his duck feathers up. He rolled it. I got a man on under these duck feathers from being in the Navy. Yeah. He got the same arms and talked about his family man. Yeah. Come on, man. Come on, man. You got to watch cartoon. Come on.  Ain't nobody on the ship. But Donald Duck and Mickey Mouse. Gookies don't even put a candy ball in the can. He goofed it. He fucks up everything. But one thing that remains the same is there's still a lot of goofies out there. You never thought about this? You never thought about this, though. Who was a dog? Who had a pet dog? Who was a dog? Come on, man.  We're all in the book. That's still in the book. That's still in the book. That's still in the book. He or the dog would have had an event compare to this thing or a dog on all fours, same body structure, same face with a collar. Public thing? I don't even know where they came from. Where they came from? I don't even know what you learned in around those ones. Do you know for you, we're going to be on your way.  The next series is going to be between these two men. I don't even know what I'm going to tell you. I don't even know what I'm going to tell you. This is the end of the year. This is the season, bro. The fault is the sun in the moon. We're lining up. That's how I'm going to try it. It's the Pacific time.  It's going to work and it's happening right now. Yeah. It needs that music. Yeah. That's what they're doing. He's the body. The new man. He's the body. We wouldn't have it any other way. Yeah, we wouldn't. We couldn't. D. D. D. D D. D. D. They took away four times. They took away four times. They gave a bunch of honor to the British. Why you looking at my name? He was dying there for a minute. They know him. Yeah, but that's why I said alumnus. Fuckin' we grew older than that. You got a lot of people know him from down there. The niggas who alumnus know him? So guess what they make of him?   The niggas that was a car made by an alumnus. That's a skill. That's a skill. The waitresses go alumnus to encroach the hospital. You know who dry alumnus? The breakfast manager. That means who dry in the first two. So everybody who ever had alumnus and lasted this and people that still have alumnus, they know they need to work. You know who dry alumnus? It's not because we don't know the role no more. The only more funny, the only more funny right around standing room is right now. I'm running for stairs. See, you want to be the stairs. Bang into the front of the plane. That used to be the jump out boy car too, didn't it? I was in the escort. But what was the step in when they had a step in? The jump out stairs. It was the photo. That just looked like a break in the room. You know what that is? The bubble. You know what that is? The bubble. Like you knew all the bubbles in your hood. Like you know what I'm saying?  The bridge let's you know. The bridge let's you know too.  The police come into the hood and look at the drugs. They wouldn't never do that. If you wouldn't have brought them over there. You brought them over there. You brought them over there. You brought them over there. You know what Karim?  But the police, actually some police officers, oh let me stop saying police, one of these police officers contacted somebody that I know is the direct message on Instagram. I listened to the 85th that's for the end of the control clock every day at work. I was like, bro, I did not know that we had to let the officers to take that. No, this is what we're doing for. We just need to touch what, read what each one says. Just by thinking about making you don't need to, maybe he will let somebody go because he lay there. You know what I found? Somebody just let him make us cry. Yeah. He forgot what it was. He was going to give him a ticket but he didn't forget how far he was going to go. Don't say that name. I was trying to keep that name. But there was an officer who came out of town. I was out of town and he said who's your friend? Oh, they were at first. They were at first. I was one of them. I was one of them. I was one of them. I was one of them. But they didn't discriminate from others.  That's the thing I was able to tell We were on this half of the plan. Thank you, thank you. No, I'm just saying, we're on this half of the plan. We're still opening an argument. Yeah, okay, so, I thought we should. Okay. We saw you, man, you do the dancing, you do the dancing, we're looking at you. So, first of all, I wasn't here for an episode, but I was here. But I could have, I could have clearly understood when I watched you, that y'all were telling me that you and DC, y'all were just how the kids were being gostoved. What play, what was it?  Yeah, apparently, cause we, It looked like he's closed in all directions, I thought, I want to hear it because he was too busy doing the math, just with all that, what a fucking, you know what, I don't know if that's, even in his beard, I saw beard doing it. You've got to, he's the same. You can tell him that, you know? I didn't know he was white. And he was all thought in the, did he say he was white? Or is he not this, who would, I don't know. So, I don't want that to be like that. I'm just gonna be good on him. We didn't have to know him, but he doesn't. So I doubted that, I doubted that.  And then, I'm looking like, oh no, I saw him. Thank you. I saw him. But then y'all perceived it, you see, don't you think what you do is words, feelings, what you got in it? Because I remember incorrectly. You want to go back to something. Whatever.   No, I'm not.    I'm gonna face the, I'm gonna face the man. I'm gonna face the, I'm gonna face the man. I'm gonna start face the man, man. You think I can't pull you up on this, let's move the screen. You can sit in a chair, pop a table. I like this, I like this. I remember you were discussing freedom. We will do this. We will take limit. Okay, so my thing is this, my mother's from here, so the step holds you.    You don't, oh, man, sorry. No, first of all, here's how I'm feeling.  Stop giving white people a chance because they get a deal with it. Because they know it's tough. I'm looking at you too, I'm looking at you too.  You're only gonna do that if you want to say that. No, you don't listen to the coach, you're alone. First of all, first of all, man, look, look, look, so the first thing that they got me looking at, that song was like, I don't know about that one, but I was like, I don't know if it was good but it was in white, but I'm gonna tell you the truth.  You don't need to be on the top of it, you know what I mean?   Oh, I mean, I'm cool with the black team because I know how to dance. What is everybody doing? Who didn't do the jumping timelapse? Oh, he did it. A lot of this came over here and stole all of Marv and gave music.  It was a boy named me and stuff, and I don't know what he talked about in there. I don't know what he said. I don't know it. I thought Pip Boor was a man. I don't know what he said. I thought he thought it was him. I don't know what he said.  Give me some time. Nah, man, you don't listen to the coach.     You know what I'm sayin'? In the same way, you'll kill these motherfuckers. I don't need y'all intentions of going racist. Come on, man. And I'd rather like to dance, dance, dance. Okay, first of all, that's it. Come on, that's it.  I don't know that DJ's man.  Y'all still gonna run with his y'all might of post? Nah, no. That ain't you. Yeah, no. They do. They do it and no DJs. But the way you talk it, it's not. That's one of them DJs in New Orleans.  You know what I'm sayin'? So I don't know, man.  That's the right thing to do. I don't know. I'm just saying. I'll give you what I'm gonna say. I don't want no white friends that don't ask white friends. I don't want no white friends that don't ask white friends. I don't want no white friends that don't ask white friends. Paul Walton. No, Paul Walton. Look, it's the same girl, right? Paul Walton. Hey, I didn't hear what you say, nigga. We want to power every week. I don't even say it, nigga.  You know, it's nothing to do with motherfuckers who's dealing with it.  Ain't nobody... Yeah, that's what you say. Ain't no... It ain't nothing to do with fuckers. Anywhere you are. How do you make it? You just... No. No, no, no. The people we rock with, the real... Straight up, I've been around people. I was around this white stripper one time. She was saying, nigga, maybe it was time about their hate. You can't be saying that.  Take it easy.   You know what I'm saying?  You know what I'm saying? I'm not making it up. What do you mean? You don't want to check me in. I'm not making it up. I'm not making it up. What does the boss want to say to him about saying it, nigga? No, no, he said it from the video I saw and he called him, and maybe he said cracker. And he called me a cracker, so I had to say it. But he told him to call him cracker.    I don't need more people at this point, you think everybody like you are? You gonna find out?   Yeah. We got to see them re-touch motherfucker. Yeah. They're great. We got a doorbell on them. Yeah. See what's up. See what's up. What's up? What's up? What are you doing? They might be my home girl, I think. Yep, that's her. Yep, that's her. No, that's what I don't do for this girl. That's this girl I don't like. What else, bro? This is a great moment, man. I don't know. I don't know. What's your reason about that? It's on your chair. No, it's not. No, it's good. No, bro.  Welcome back to the 85 crowd, show forgiveness to the mayor, Carl Zealot Miller. Got a dance, they put up a week ahead of clap. Come on, man, with that, we're back to my class starting in post. Y'all know that we got a word called post-production? How are we doing? Welcome back to the 85 crowd, show forgiveness to the mayor, Carl Zealot Miller. That's the next thing, I got my keys, I got my home drug. Stop it! Didn't you hear from the baby guy hitting this motherfucker? My nigga, now, bring it.  What's the matter with you? Hey, you know what I got this brother?   Rambo is this motherfucker with us. Yeah, man, I got a lot of words to not say. You just did that for us, but we got to ask the Lord. That was good. Yeah, but hey, bruh. Still the Prismacree over here, bruh. I don't need to talk about it. What? No! No, bruh. You're not the Prismacree over, bruh. Let the, hey, yo, let them see the Krumas, bruh. We're in the Krumas, bruh. Krumas in the B, bruh. You heard that? Krumas in the B. Krumas in the Krumas. Krumas in the B. You see the Krumas in the B? Krumas in the B. Who that? Boop out. Okay, who that down on you? Them your people. Yeah, I bet. Y'all pull up with us. You know, get in here. We just ate a fine stock of it right now. You know what I'm saying? Telling these people about Roach, Cliffs, and Tears. We back, man. We back. We good. How y'all feeling? I'm feeling good. How y'all feeling? Yeah. I'm worried. Bro, you were beat. You jumped off in this office.  No, no, that's what that was about. We did that thing. We did that. We did that. We got accepted. But there's no real answer. There's no real answer. There's no real answer. There's no real answer. We did that. Hey, I was there. I was leaving. And Clayton.    Did you? I made that. Oh, wow.  Why, you gonna put it in? Why you gonna put it in? I'm not gonna take it.  Is that till when it's saying that there might be tragic cases from there?  You're leading to it. That's called leading. No, no. It's not me, it's not me, it's not me. I mean, you're my partner, but I ain't know you the type of thing. It's time to move fast and keep real. So we're saying we don't pull our dips back. Dips don't get pulled out out of here, man.  And you just choose to stop what you do. That's not what we do.  I'm not... You just reminded me that we must, we must absolutely must this episode, show much love to all the women out there who appreciate dips, dips pitches, dips, dips, ice cubes. But only if you have a... Only if you have a boy who wants to shoot. Who do you design the UPS for? Yeah, for the parents. Consistently. Consistently. Yeah. Ladies who appreciate the dip that they ask for. You can't even see the diamond no more. Y'all can't see it.  I've never... I didn't know that you was... Don't, don't reference that. Don't, don't... Well, can I clean you something? You gotta... You gotta let go of what you did to that. Like, what did I do? I don't even think... So you don't even start. You start and you don't know where to do it. That's how you do it. Come to a place. You do a test and you say, kick what? Kick what? Oh, wow. And then if you just leave it, then you just leave it. And then you just hide it where my fingers are done by nature. You know what I'm saying? You get it, man? I don't know. See? Don't do that. Don't do that.   I'm not. I'm not. I'm not. I'm not. We're from the fucking... I'm not.  I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not. I'm not.      I don't know, man How do you know that? I don't want to check my vote then when I see it That's me and the nigga's here up there Oh, he's dead John's Creek? Is that his level? No Yeah Oh, no Ain't that where he at?     Because that take a whole lot to go through to let her see it.   She need to know the real me.      Yeah. Don't you see nothing wrong with that, nigga? Just don't make a vibe to it. Oh, okay. Like, who is to do this crap? You don't even know who's been caught on the phone. You got too much feeling. Put him up. Who are you? Who is he? I don't know. He's a little kid. He's going to examine them. This nigga love them all. All of them. You can't be mad. You're right. I was saying, Titty beats that 80-mage kid.  This Titty right there. That's like a Titty that went to college. Because the Titty is a wet place. We haven't put it. That's beyond our best behavior. Because that's been a hard hit, Titty. Titty did everything because this Titty in place, Titty taped it. Titty took a little piece of the shirt. And then Titty came out. Right. Titty on the shirt. All that.  They just do something to my head of work. I find Titty. Legs just be trying to stink and hide Titty. Uh-uh. I don't want any Titty in it. Trying to wear that button on the shirt. This Titty is under them buttons. And then they just have the Titties down. Come on, bro. Titty's looking like two Titties. That's like the 20 Titties of Titty. Come on, man. Titty is that Titty. I don't even think about it. I think that happened in my shirt a lesson because now that I'm gone, I know that we were set up to fail, bro. You said 8th grade, but I don't know what 8th grade was. I don't want any Titties. You weren't supposed to get them Titties about 10th, 11th grade. That's why we came back from 7th grade. We came back from 7th grade. The girls were fine. We were still little boys. We were still out there. They came back grown. Getting their hair done.  They ain't got tears in the field growing. They're ready to work real big. They just told me, I don't want no more Titties. Come on. Bro, puberty was very confusing, man. I had no help. I was out here speaking. They said I'd buy myself. You said nobody tell you when you're 13, yeah. This was happening.  Hey! Now I know what the deal is. My son, he's got a real textbook.  I'm not wearing with him. What do rules to do? I'm gonna take it like this. I'm gonna take it like this. You can do the whole, but you're gonna fill it back up. One thing, hey, one thing about whole, once you get in there, stay there. What? How do you get in the fuzzy cup? Leave it in the fuzzy cup. Leave it in the fuzzy cup. Let me eat the fuzzy cup. No, you won't. You lose. No, I don't think about it.  That was just telling me, if it's real, then you won't have to make... I wouldn't get to that spot. Ain't nobody ever felt good. It's important, man. He's real, man. He tried to make pussy was not cool for me. I remember. I remember when he was fucking... He was the nicest nigga in the neighborhood. No fake would really push me. I know. It's important. I grew up watching the run-up. It's hard to see. What did I do to you, and I was gonna spray it on you. Let me try.   I wish I could go back to my family, but he told me all the news. But he died before I got here. He's a real... That's cool. He told him you could be a little more on the hospital. He's a real... But when you start, he's gonna be a little more on the hospital. You're gonna be a little more on the hospital. You're gonna be a little more on the hospital. You're gonna be a little more on the hospital. You're gonna be a little more on the hospital. I didn't get a purpose when I started eating fish. I was getting coochies. Coochies. Coochies? What do I do? Coochies are great. Coochies are great. Coochies are great. Coochies are great. Coochies are great. Coochies are great. Coochies are great. If you get a... Yeah, you get a ear lift. The nigga that did the car work. Ear lift, nine o'clock. Making a lotta noise with all the slow-man laid down on the ground. Yeah, that nigga used to get pushy.  And then you're coochies. But now, Lord... I do that, but I come out and I spray the tires, though. You can tell who's been pushed that way. We don't know. I don't do it alone. It needs to be in a big hand now, baby.  I mean, let's go in and have some fun in here. Who's going to give me a barbecue ready and give me one of these to be hot too? None of the fun? No. Yeah, but come here. Come here, man. I'll give you one. Do you need me? No. Come here, man. Give me a quick lemonade. I'm trying to be great. Two of them don't get hot. Let me let them catch me, Peter. I mean, why is it that there's already a little situation and nothing about it?  I got a coconut bar in there, if you come out of that. Yeah, the red, white and yellow coconut bar, they sit right over there by the next door. You be too fast for funky things. We're friends, I'm a friend of yours. We don't even know the story. I'm getting there. Give me two of them lucky charm cereal bars with the milk in the middle. That's the brown milk in the middle. They can live.  Tell them I take the stage names when you're getting there. Tell them I'm instrumental. Tell them to give me two quirkfuls of that match between the liquor and me. They don't want no boo or peanuts. No, bring me two of them. Let me let them have some ice-cream ice cream. Now, I want a big red to bring me that pink lemonade and then give me a local bread out of that and cool them down boo up in the jungle. Oh, give me, give me a real green soda Look, give me a mystery ad here.   Look, that's what I need. I need you to give me two of those coca-cola and some of that CK1 cologne. Get that, that they gotta be glad it's here. The coca-cola with the name on it or just a red dog, just to get up.    Hey, you're over here. You're over here, you're over here. You're over here. You're over here. You're over here. You're over here. You're over here. You're over here. You're over here. You're over here. You're over here. You're over here. You're over here. You're over here. You're over here. You're over here. You're over here. You're over here. You're over here. You're over here. You're over here. You're over here. You're over here. You're over here. You're over here. You're over here.


Thank you. Mr. Lewis, I know your organization also is advocated for these types of networks. Do you wanna add anything to that? Well, I certainly agree with Mr. Chilo. The choice of different types of builds is important for local communities so that they can take the risk that is appropriate to their community's needs. Let me ask you this, Mr. Lewis. You talk about digital redlining in your testimony. How can legislation like the accessible, affordable Internet for All Act, the Lift America Act, and the American Jobs Plan address digital redlining? I think if the bill includes options for the subsidies or that funding to go to local governments or local co-ops, it can certainly also help with redlining. As Mr. Chilo said, those communities usually are better at making sure that everyone is served because they live in those communities or those government officials live in those communities and so they would have incentive to make sure that whatever is built is built to everybody. Ms. Cheney, do you wanna add anything to that? Sorry. No, I think the other witnesses answered it appropriately. Okay. Well, let me go back to Mr. Lewis then. In your testimony, you talked about the need for a permanent broadband benefit and the need for additional majors to increase affordability long-term. Do you think that we can close the digital divide over the long-term without addressing those issues? I'm concerned that we won't. The cost of broadband right now, just from what we see from the numbers of how many options that people have is really driven by monopoly or duopoly prices. I think that doing the work to study the prices and the cost of broadband long-term so that we can really determine the impact of the market and if that cost is actually affordable or if it's simply the cost that's going up and up based on the choices of industries is critically important. We've already seen the last few years of 20% rise in broadband costs according to FCC data that's far ahead of inflation. So these prices needed to be studied.