All right, welcome, everybody. Hopefully, bellies aren't too full. After having some lunch, this is kind of a warm room. So hopefully it doesn't just put you to sleep with a full belly in a warm room. I would say that if you guys start feeling kind of warm in the back, just open some windows. Don't be shy, because we were kind of in a sauna in here earlier. My name is Dustin Black. This is my colleague Marco Karn. It's hard to tell us apart. I know. We are members of kind of a special team within Red Hat's storage and now storage and hyperconverged business unit. We're actually a technical architecture team. And we do a lot of performance analysis of full workload stacks more and more. So the big disclaimer here is that we're going to tell you that this session is designed to tell you about our experiences. So I'm not here to dictate to you the right or wrong ways of doing something. We're trying to give you what we've learned from experiences. So the tools that we're talking about are things that we have used that have been effective for us. And we're going to tell you the stories about how well those things have worked for us, what we did and didn't do right, and kind of the fundamentals you should think of along the way. There are somewhat specific examples in here of tools and the way we use them and the way we test. But hopefully you get out of this as kind of like something a bit more broadly applicable. It's more about the methodologies and how you think through these things. So these are very simple slides, too. We had fun, just we're just going to throw them up here and no templates, so why? So why performance tests? Why does this matter in the first place? Think about what you always start with when you're building a system. And in our case, in particular, a distributed system. That's something to note is that our performance testing was in how an application runs on its server. It's how does a pretty distributed application with lots of different clients run on a distributed storage system. So there are a lot of pieces going on here. So when you start trying to architect an environment, you have hardware specifications, you know what CPU and memory and network performance specs are and to some extent what to expect. But what you never really have is a full idea of how these things are going to operate holistically. You can come in with some educated guesses, but without actually doing the legwork to say what does an application or a particular workload do on this hardware. And again, particularly a distributed hardware set. You're not really going to know what to expect and knowing what to expect can be extremely important in building out an architecture. You may find that I can kind of balance things to one piece of the stack, the CPU or whatever, but if I don't fully understand how it uses everything together, I'm not going to be as effective. And you also have to consider what different workloads you may have.  There's usually a variety of activities that are going to be happening across the system. So you can't just come in with specs that are written on paper. It doesn't solve all of the problem. When we do performance testing, most people come to the question of performance with the idea of what's the maximum that I can get out of something. So this is just a simple example and we've done numerous things kind of like this. But what I wanted to show you here is that across a couple of different hardware configurations, you may do some analysis and say, oh, well, this configuration looks like it may be my best option. And in this case, this happens to be a difference between whether a system has 12 drives, 12 physical hard drives in it versus 24 physical hard drives in it. So you can see, okay, 24 physical hard drives, I can actually push more IO through the system and therefore I get greater performance. And if I just kind of glance at this and say, so that's the better way to go. I get better performance for this workload out of a 24 drive system. Let's get a 24 drive system. The trouble with this is that it doesn't always take into account the value of the dollar that you spend on this. So we often actually take a look at our data in a different way. This is the same data plotted now actually considering instead, what are we getting performance per dollar spent? So how efficient is our dollar going to give us this performance goal that we're aiming for? So in this case, you actually see it kind of inverts things. It inverts the story. If I go back, the differentiators here are 12 disks versus 24 disks, but it's also how we're actually building, in this case, the Gluster volume that was behind it, whether it's a distributed replicated volume or a distributed dispersed volume, or ratio coding on the right side. And not only does the 24 drive system appear to be better, but it also appears that using a distributed replicated on RAID 6 is better. When we go here and actually look at it from a price performance standpoint, you see that we now move to both a 12 drive instead of a 24 drive system and a distributed dispersed volume instead of a replicated volume. So understand what it is you're targeting, what you need out of this. Maybe it is pure performance.  Fine, but understand what you're going for because it's going to change your testing methodologies in your data collection. This is kind of drilling down into that example a bit. So once we say, okay, our 12 drive system is our best building block. But now there are a lot of different ways I can leverage that 12 drive system, different protocols I can access through, different ways that I can arrange the hardware and RAID configurations, different ways that I can arrange a software stack on top of it. So the example here is that these are all the exact same hardware. Nothing has changed about the hardware. The only thing that's changed is how we're leveraging that hardware, how we're actually interacting with it. And you can see very dramatic differences based on your choices. This is one particular workload that I pulled out as an example and what we're looking at is a four gigabyte file throughput. So this is kind of a, for Gluster in this case is kind of a sweet spot for performance. Large file throughput it does quite well with, but under what circumstances. So if you take a look here, our best overall performing for both RAIDs both RAIDs and Creates is again that 12 drive J-Bot distributed dispersed system. So no RAID, no replication, we're using erasure coding and we actually get our overall, and this is price performance again. So our overall best value of performance out of that configuration. And if you look down here just to compare it to the worst configuration of that, in this case I have to use the NFS client and I'm doing a dispersed volume on top of RAID six. And you see that configuration, I get horrible performance by comparison. These are important things to understand in your testing so that you know which direction you may need to send the architecture or it may be an inverted thing. What if the solution comes to you and says no, no, we can't use Gluster's native client, we have to use NFS. Well now my choice becomes different. If I say okay, that's fine, but we're not gonna get as great a performance out of NFS. That's okay, we understand that, just get the best performance you can out of NFS where required to use NFS. Well now my selection becomes different. If I were doing J-Bot distributed dispersed with the Gluster fuse method of accessing the files, you can see that this design is clearly the best choice. But if I look at the same design, which is J-Bot distributed dispersed and I do the same thing with NFS, this is not my best choice. My best choice within NFS is actually to do a distributed replicated volume. So the protocol, if I have forced, if I'm forced to use one protocol or another, it may change my architecture choices as well. So we always like to say start with the workload. Don't go in architecting a system until you fully understand the workload or you're going to make mistakes. Another thing drilling down even a little bit lower into the layers is that there are many things you will find along the way that are the small knobs that you can turn and often the small knobs have small effect, the low level things. It's really the architectural decisions that have the biggest effect on the performance of a distributed environment. But there are certain things at a low level that you may find that you can tweak that may have a huge difference. Some of the big ones we've seen are RAID controllers. You know, in this case, make sure you design your RAID controllers with a battery-backed cache and put it into write-back mode. We're going to get all of the benefits of that caching applied to all of our IO and we see it makes a huge difference. But it's really easy to order a system with a RAID card that doesn't have a battery-backed cache and then you can't leverage that. And it could be a big detriment to you. So understand these things. On the network, jumbo frames make a huge difference. Tuning your VM-30 ratios and background ratios whether you're using Direct or Buffered IO, some of these things, again, might be dictated to you by the application and how you're using it, but some of these may be choices that you can make. Just understand those things as you're going in. So how do we perform this testing? Yeah, the question is, how, what do we use? First of all, it's important to understand that you're not using or not testing a single system. You're testing distributed systems, but you still need to make sure that every component of these systems work as expected. So make sure you check your network, make sure you get the actual expected throughput, make sure your devices like HDDs, STDs live up to the expectations you have to them. Make sure the CPU delivers as expected, although we found out that is pretty tricky. So if you have an idea how to do that, probably let me know. Yeah, do some smoke tests, make sure that works. Other fundamentals we learned over time, make sure you run enough iterations of your tests. A single test is not exactly giving you the correct results and make sure you run them long enough. We've seen interesting caching effects happening once we went from five minute tests to 50 minute tests and to 30 minute tests. Calculate your standard deviation. If one test gives you totally different results, make sure you have covered your bases in that regard by doing the deviation checks. And we learned that the halfway to check one variable at a time, do not change anything beyond that. Because I remember pretty well when we did that, results changed dramatically and we didn't understand why. Collect submetrics of all the affected systems. Not only the systems you're testing, but also the test generators, the load generators. We ran tests where our servers didn't go beyond the load of 30% and we didn't understand why.  So yeah, make sure you monitor that. Find out which is the system resource that gets saturated first. This will be your showstopper basically. That's what it is. In our case, it's typically storage, but that of course heavily depends on the workload you're running. Unless we want it to be storage. Well, that's... It exists. We're building storage system. We want to eke out as many IOs as possible. If our bottleneck turns out to be on the network, that's a problem for us. That shouldn't be happening. Right, but also make sure not to overdo. I'll explain why. Monitor as much as you need, but keep it to the bare minimum so that you don't have to take into account monitoring effects because monitoring doesn't come without a cost.  You will end up with horrible amount of monitoring data and you need to correlate those data in the end. So for example, is it really necessary to do a monitoring within short-lived VMs?  Make sure you capture whatever you need, but not what you don't need. And also spend time on bringing up those metrics in a meaningful way so that you see resources you want to see at one glance and not spread all over the place. So what are the fundamental metrics to monitor? Of course, it's CPU usage, memory saturation, or paging effects, storage device loads. You want to know when your storage device goes bonkers or hits 100%. You want to see, is it network throughput that's limiting my tests? There is one point to all of these. They are local tests only. They have not been correlated overall, the systems that are affecting these tests. And these local metrics are of course pretty difficult to correlate if you have them on each system separately. So spend time on that. Think about how you can do that. And one very important point we learned is make sure you know what you're looking at. Is looking at averages good enough for you? Do you have percentiles? You have to be answered within a certain amount of time so the maximum latency comes into play. Do you need per driver or per load driver metrics? Do you need them in a correlated way? Most of the metrics you can collect can be reported in either way. So make sure you know what you're aiming for. And as I said, monitoring averages can be very deceiving because, and we'll see that later, if you only look at the average, your tests may be pretty fine because you're doing a lot of simultaneous IO ops, for example, but is that good enough for all of them? And yeah, the max within a percentile may be a better way to look at it. So typical tools we use for local metrics and you probably know all of them. That's SAR, that's VM, that's iOS, that's top. These are all fine and good tools, but they have one common problem. They give you the view on the system for a point in time, not how the load develops or monitoring or trending. So you have to find a way to do that. Typically when you only look at one system, you do it manually, but what if you do it for more than one system, Dustin? So systems generate all of this data and when you're using tools like we're on the last screen or a lot of similar tools, you can dig into it on a per host basis, but again, we're testing distributed systems which could have a lot of nodes, a lot of VMs, a lot of clients involved in this and correlation of that data is really important. So definitely look into tools that allow you to aggregate data into a central place. These are a couple of the common things you'll find out there that we have used in the past. We used to largely use CollectD. If you're not familiar with CollectD, it's basically an actual service that's running on each node that's being monitored and it has kind of its own format for generating all of this data. It can store it in a local file, but what's beneficial about it is the demon's ability to actually push that data up to a remote data store. So all of the clients running CollectD, you can tune what they're collecting and what they're pushing up and so there's this constant push from the nodes to a central collection point. Now if I have 20 nodes in my test environment, I have all of that data in one place that I can leverage. Prometheus is newer, Prometheus you'll find, Red Hat is integrating into a lot of products nowadays. It works in a similar way to CollectD except that instead of running a demon to actually go and collect data into its own format that it's pushing up, Prometheus is actually leveraging kernel level information that's already being reported by the nodes. So it doesn't actually have to do that kind of active data collection. It's more of a passive data collection. It's not even really data collection, it's data presentation. So effectively Prometheus takes that local information that the nodes already reporting just via normal system tools and it hosts that information up through a standard presentation method via a port on the node. Now the difference here is that you end up in a pull versus push. So CollectD is using a push mechanism, running a demon, collecting this data, pushing it out to a node that's then aggregating all of that for you. Prometheus on the other hand is just hosting it. It's just advertising it. Hey, I have this information about this node and therefore we have to have something that comes in and actually scrapes that off of the node. So it's definitely a flip in how you think about doing things. I think in the end Prometheus is kind of the way to go. I'll talk a little bit on this slide about some of the limitations you have to consider when you're going with one or the other. When we were using CollectD, largely our receiving in was GraphEye. That's kind of a lie. GraphEye itself is really a kind of the graphing tool. It's the thing that's creating the time series visualizations for us. But it has to work in conjunction with the receiver. So there's sort of this modular plug-in system that acts as a receiver for CollectD. So all that data we're pushing with CollectD goes to the receiver. You also have to separately manage a database system that's going to be holding all of that data. You have to be a little bit cautious that whatever database and collection system you're using doesn't roll up your data too often and lose resolution. This has been a problem for us in the past is we built a system, we were monitoring things, we kind of stepped away from that environment for a little while and we came back to it. And all the time our database had been running and it basically got to a point where a time limit expired and it starts compacting your data. What that means in visualizing your data is instead of one second resolution and being able to really understand what was happening during a workload, I've now rolled up to 10 second resolution. If I wait too long, it rolls up to 30 second resolution or one minute or 10 minute resolution. And so all of your historical data is a lot less valuable. That kind of roll up is probably fine if you're using it to actively monitor systems that are operating. If you're using it to monitor tests that you're running, you probably won't maintain that resolution. So you need an efficient database system that can actually hold the data without rolling it up like that. So keep that in mind if you're going to build one of these. Graphite does a great job of graphing time series data, a lot of flexibility of how you can build these graphs and charts. It's not particularly user friendly for creating dashboards though. I wanna look a lot of this data together and do a lot of comparisons. You can build the charts and graphs but it gets a little bit cumbersome to do that kind of effective analysis, especially if you wanna share it out and collaborate with other people. Prometheus, you can think of it as the Prometheus server side of things is actually doing the work of going and scraping the Prometheus client, right? And again, the Prometheus client's not really a client. They call it a node exporter, passively hosting that information, right? So the server now has to go out and scrape that information from each of the nodes and collect it up. So the active part of that work is now happening on the server side. And we get the same type of time series graph generation. It's almost a bit more cumbersome to use than graphite when it comes to creating those individual charts. But it's largely because it's a lot more flexible. It's a lot more powerful in the query language that it uses to build those charts. And again, directly using the charts that it generates is not very user friendly. So what you really need in the end to visualize everything is a tool like Grafana or Kibana. There were some other ones out there. We've largely used Grafana. And it can basically take either graphite or Prometheus data and then turn that into this really customizable dashboard posted on a web interface. And you can get down to fine tuning all of these charts and plotting them against each other. This is just a small snapshot of what we've done with some of our graphite dashboards or Grafana dashboards. This is like, you can actually see the highlights. This is a series of pre-runs. We talked about running three or more. So imagine we ran three iterations of it and we wanted to understand what was happening across them. And you can see it's a very high resolution just by how small the lines are in that time slice. This is only like a 10 minute time slice. So we're getting very high resolution plotting these things and I can easily look down and make these comparisons. I can drill down into things. I can decide whether I want to plot these things in averages or centiles or whatever flexibility I need. And this is a small snapshot. We have many metrics on one dashboard that we make a pair against each other. I think this is all disk related stuff but we have CPU and memory and network related things that we're recording all at the same time. So very, very useful. As we said, you need to understand the subsystems so that if you just run your workload against it, you don't fully understand where those bottlenecks may be and how to tweak the architecture or configuration unless you have this data. So now we'll get into the tools we actually use for generating workload. And we're gonna start at the low level and kind of work our way up through getting more and more sophisticated in your testing. Hopefully some of you guys are familiar with FIO, pretty standard anymore. I actually have a hand in making sure that the FIO package is in RHEL now. So if you're a RHEL user of RHEL 74, I think, and beyond, you can just yell install FIO and it's there. Very flexible, very versatile. You can run a lot of different workloads. Obviously we use it a lot for our storage related tests if there's a ton that you can do with this. And as with any Unix-type tool that's very flexible, it's also very complicated. There's a long man page with a lot of configuration options and you kind of need to know what it is you're doing. It's one of those things where maybe you go out and find somewhere on the internet somebody else's test the way they run it and hopefully use their config as a starting point because you may not wanna write your own from scratch. But we have throughput latency information that a lot of hits the brand type stuff that it reports is very good. Small file is a really great tool. It's pretty simple to use, it's written in Python, actually by one of our performance engineers at Red Hat in England. This one allows us to do aggregated tests across clients and it is focused on generating small file type workloads against a storage system, which is really important for us when we're testing a NAS-type system. We wanna know how things that are highly transaction-based and there's a lot of metadata type activity we wanna know how that performs and small file reports that in a really nicely consumable way. So we've used this one a lot. And prior to getting into using FIO where we used to use IO zone a lot, this is again, this is like a classic, very, very flexible performance testing tool. Still highly recommended in some ways I like IO zone more than FIO, but as a team we've kind of standardized on FIO. But I've got a lot of historical data here. It's still a very great tool and it does some of these, you can generate some of these types of graphs itself. So if you just need to do kind of visualizing your own information, it's great. So, again, this is just repeating and excuse me, digging in a little more to what we had mentioned about distributed clients. If I have a big distributed system that I'm testing, I need to make sure that I'm generating enough workload to do that. So what we're talking about here is kind of related to how do I do that in relation to these tools I just described. Thankfully they all do it pretty well these days. FIO didn't used to do distributed testing very well, but now it's got a pretty good system for it. But you need to know how each of these tools does that. It's not the same and sometimes the terminology changes between the tools. For instance, what one tool calls stonewalling and other tool also uses the term stonewalling, but it means something different. So you gotta watch out for that a little bit, understand what it's doing. The example here is if I want to know, and this is commonly the question we're asking, if I want to know what the performance of a system is under a maximum load over a period of time, I need to generate that maximum load. If I need to do that with 12 or 24 clients or oftentimes we ramp up the number of workers depending on the workload, I may have 100 workers or more actually pushing data to the system. Well, if I just hit go on that and start recording, what happens? Well, the clients don't all instantaneously start generating IO at the same time. All of these processes have to compete for their client resources to actually get their IO processes up and running. I don't want to start measuring the load that it's generating at the time that I hit go. I want to make sure they're all actually running before I start measuring. Same thing on the tail end. As soon as the first client stops doing workload, I want to stop measuring. So I only want to measure load during the timeframe that all of my client's operations that I've started are doing the work at the same time so that I actually have a good view of what's happening during that timeframe. That's true for our case most of the time. You may actually have different needs but know that these settings are there that they have to be used appropriately or you could end up with measurements that aren't quite accurate. This is kind of just a visualization of how you'll handle that. You usually have some controller nodes. Sometimes a controller node is one of the clients and in a lot of cases there's no reason I can kick off the job from one of my client's systems but I may use a remote node to actually manage all of these things but I need to actually aggregate, I need to push out the workload and divide it up among all of my clients and all of my clients then need to generate the workload up to the system that I'm actually testing. And so that's all very important for getting that just right and honestly you're likely to get it wrong the first time so play around with this a little bit. It gets to be a little bit tricky. And finally as we mentioned that you have all of these tools that are all able to do these distributed tests that they all have different types of user interfaces, little bit different syntax and how you operate with them. We are performing hundreds if not thousands of these test cycles among a project that we're doing and we may be flipping back and forth between using small file or FIO or IO zone or different tools. Make your life easier. Wrap some kind of scripting around this that's going to simplify things and is going to do the tests in the standardized ways that you expect to repeat. If you're interested in getting started on that there are things like P-Binch which are actually wrapper tools that begin to do some of this for you. And that can be really effective. We kind of started rolling our own before P-Binch was really maturing.  But our batch script user interface is consistent. So how I run an FIO job looks the same way that I run a small file job and it looks the same way that I run an IO zone job. So it makes it a lot more user friendly and portable. These are rough around the edges. They were designed for our internal use. So you're not going to find that they're perfect but take a look at them. They could be a good starting point for you. And we have, those are kind of like workload generators. They're pretty small. They generate this kind of IO, right? Do four gigabyte throughput for X amount of time or whatever and see how that is. But you can see that's very synthetic. So what we want to do is keep moving up the stack of maturity. And the next thing to do is to do something like SysBinch. This is a tool we've used a bit. You'll commonly find that this is used out in the wild for database testing but it's actually a pretty flexible tool and can test a lot of different things.  And you have some flexibility in the kinds of things that it tests. You can combine these things together. So you get a lot of good information out of SysBinch. SpecSFS really takes us up. And I'm talking about SpecSFS but the spec committee has a lot of great tests. Downside to these is you have to actually purchase them or be a member of the spec committee to have access to these. But they are really, really well-designed workload tests that really simulate very realistic scenarios. In order to use this data, especially if you're going to publish any results of this, you have to have actually approved results summaries by the spec committee under most conditions. So a lot of restrictions, if you're using it internally, it's a lot more flexible. But if you're ever gonna publish data, this can be a little bit cumbersome even though the way that it generates workloads is actually very, very nice. We've largely used the streaming video capture which they call video data acquisition which does simulations of like high definition streaming video of CCTV scenarios and also runs like an interference read workload. And you look for how many camera streams can your storage solution support before you hit a latency threshold. Really nice test, but again, a little bit cumbersome to use in the end. I'll pass on to Marco for taking this next step up. Yeah, exactly. Now that we laid out the foundation for all of our tests, we want to take it to the next step, meaning we're not taking a single resource. We want to test, we want to test a combination of all of them. Typical that's network CPU storage, memory, think of any combination of them. These tests do require more than a simple script to run. Most of the time they consist of a web server application server database server. And of course the client systems are also there and you need to take care of them. So we went for a very popular testing stack called dvdstore2, which as the name already suggests, simulates a dvdstore. The dvdstore2 version only consists of a database of any flavor you can use. MySQL, Postgres, dvd2, I think anything, yeah, mostly anything works. And the clients are Windows systems, but we managed to get them up and running in containers on the client side, so that works as well. Customers typically log into that dvdstore browse, check out some dvds, pay for them. And the configurables of these tests are typically the number of customers. This is a number of threats within the client system. The database size, the larger the database, the more dvds are available in the store and the more transactions you get. But as you can clearly see, this is mostly network and storage intensive, so it doesn't burn too much CPU cycles. So a graph we get out of this looks like this. So what we've done here is we've tested a product that's called RIV, basically that's hyperconverged storage and virtualization solution, meaning we run ref on Gluster on just three nodes. And we wanted to know how many VMs of a certain type can we spin up on that system. So we started working out should the database sort of look like, how large should the database be? How many clients can we run versus that database? And we did some baselining running just with one client and that was the number of OPMs we can get from a single client. And then we scale up to, as you can see, a hundred and more stacks. A stack is a client and the database. And what we found out rather rapidly was that we are running into one system constraint hitting in probably about here, somewhere around here. From there on, we saw that the number of total OPMs, meaning the aggregated number of OPMs, basically leveled out at around 7,000, no, 750. 7,500, 750 is the OPM per thread. But what this graph also shows is that right from the beginning, the number of OPM per individual thread or per individual stack, dramatically goes down all the time. So if you want to look at it from that single stack perspective, run one because it doesn't get any better. If you're looking at the overall performance, yeah, you should probably stop at 45, 46, maybe somewhere around there. But you can always push it out further, doesn't get worse. So it's pretty clear that at this point, one of the resources was saturated. Turns out it was storage. We didn't see any pressure on CPU or memory or network. Yeah, it turns out this curve too, matches very closely a lot of storage tests we've done in the past. This is a full stack test, the full application, but we clearly hit a bottleneck and plateaued out. We don't see another cliff at any given point, but this curve very well matches what we usually see when we're trying to push more and more IO workload with a larger number of clients through a system. We eventually hit a maximum storage that we can do, and if we ask it to do more, it will try. I can keep scaling out, but I just plateau out the total that I can get out of the system. So yeah, adding more VMs doesn't get you anything more, but it also doesn't break the system. Right, so we were looking for something that uses the resources in a more balanced way and makes use of more resources than just storage in network IO. So we came up with DVD Store 3, which is similar to DVD Store 2, but implements a web server in between the client and the database. So more like a real-world scenario web shop, typical web shop. It adds some features like reviews and premium membership, but that doesn't mean it's that much different. And as I said, it stresses CPU memory and IO more equally than DVD Store 2 does. So with that test, we wanted to see how the latencies work. So again, this shows the average OPM versus the latency per OPM. And if you look at this, blue bar, these are the OPMs, they grow to a certain point. This is not on the slides anymore. And the red bar is the average latency and this is what I mentioned before. If you want to stay within the average latency, it doesn't grow that much. I mean, it's going from around or it's not on there. Doesn't matter. Right here, it's not that much different to with a single stack. But if you need to stay within a certain maximum latency, then you see that from 12 stacks onwards, the maximum latency goes through the roof. So if you need to have, let's say 99% of your requests answered within the average latency, that's fine. If you need to have them within the max latency, well, it doesn't even align, you should probably stick to five or less because latency from there on, the maximum latency you can get, doesn't mean every request gets that much of latency, but there are requests. So think of a trading system. You want to be your requests answered within a certain time. You don't care about average latency, you care about the max latency. And this is what is important to look at, depending on your workload again. If we made the mistake of plotting only the blue and red lines, then we would believe that we could scale out to a larger maximum number of operations because our average latency wasn't changing much. So what we learned by seeing the max latency is that clearly we shouldn't have, we really hit a latency problem before we hit a throughput problem. And we need to account for that. Yeah.  You want to rerun those tests in half a year's time, in a year's time. Make sure you use whatever you like, be it Ansible, be it Puppet, be it BashScript, make it repeatable. We had to create hundreds, maybe thousands of VMs over time. You don't want to do that manually. You want to script that. You want to make sure all of your containers get the same configuration every time. You want to make sure that the caches get cleaned up on the hypervisors, maybe in the VMs. So make sure you have a consistent and reproducible way to do that as well. And I guess that brings us to the end. Questions? We have a 10 minute warning, so we have 10 minutes for the questions. Come on. Sure. I can have one. So when you do your benchmarking, and you have the results, and you have the numbers, and you see that the numbers are maybe not matching what you expect, but you're also seeking for the kernel bugs, like you know that it should not behave that, so you try different channels and see that if the kernel A was better than the kernel B, do you do something with that? That's something we do not do, but we are in very close contact with engineering, and we have peer reviews with them. We have peer reviews basically every other day with them while we did that. During that project, we also did throughput tests, and those throughput tests were like, oh man, this is so far off from anything we would like to see, and we threw it back at engineering, and it turns out that we were doing something wrong, and there were bugs, and yeah, in the end, it aligned pretty well, but we do that, yes. I'm gonna go back to another slide, too, to answer that. This one. We kind of just touched this real quickly, but this is just an example. We're running through a lot of different test scenarios, and we were sort of under pressure to complete our tests under a certain amount of time. We really, really want to be as scientific as possible, so each yellow box indicates a change that was made to the environment, right? So this was our starting point. It doesn't really matter what they are, but this was a set environment, and we tested that, and we got certain results. Then we said, okay, let's see what happens if we change this variable, and we got results. So let's see what happens if we change that variable, we get results. Well, here we kind of got under pressure to accelerate things, and you look at what we did, we changed like five things in one line. Guess what happened? We got pretty different results. What caused those results? And we had to go back and kind of unwind all of this to figure out what had changed, because it just raised more questions than it was worth, so sure, we kind of had a shortcut to better performance, but now we need to know why we were getting better performance so that we could use that information. So to the end of kind of like AB type testing, at a higher level in the stack, yes, we as a team are constantly doing that. If it gets to a lower level in the kernel type stuff, that's usually outside, because we're more of like a post product team. If it gets down to that level, it's more of a reference back to, you know, engineering and performance engineering. Yes, there's another question. Yeah. So do you also conduct a kind of continuous benchmarking to catch progressions in software or kind of like a set of benchmarks we run every couple of days or whenever something changes? No, that's not within the scope of these project nature things. We get a certain task, we fulfill it hopefully with the expected results, but then we are off to the next project. Yeah, we have a broad set of themes doing performance within Red Hat though, so we actually have a performance engineering team whose job it is to figure out at the low level what needs to change in the product before it's released to make sure that we're getting the best performance results. We also have QE teams that run their regression type of stuff that you're talking about. A change happens, we run that, see if we get different results. So there's a lot of that work going on, but it's a little outside our scope. Yeah, well, I have the same questions as just one edition. So if you do this set of tests, do you, for example, possibly do engineering or some kind of maintenance team who then, for example, can do regression testing with the same set? Like, let's say you have the architecture done, your project is done, but then they do some software of this, like new kernel stuff. Yeah, that happens a lot. What is your setup to them and they use it? Actually, we share a set up or a number of setups with engineering directly in Bangalore, for example. And they do further regression. They do. Yeah, depending on the scope of the test, resources may get directly passed around, including the lab as it exists physically. Now QV, go use this lab and do some work. Sometimes those things are scoped for a shorter period of time and they learn what they need from it and maybe start implementing that in their CICD workload for continuing to test these things. But yeah, there's a ton of sharing that's happening, which is great for us. I mean, again, the pressure is often on us to produce these results as quickly as possible because it's tied to some software product release or some document, some collateral we want to generate of architecture document or something like that. So when we hit some of those rabbit holes that could take you really deep into some strange areas of troubleshooting, it's really important for us to be able to sort of pass those things on because we just don't have the time to do them ourselves. So knowing that we have other resources to help us out. And I always encourage everybody to know in your own organization, whoever those other resources they can help you.  There was another question over there. Yeah, so you actually just do benchmarking. You don't do optimizations and you don't change the optimization algorithms or anything else. No. Just do benchmarking, just do fixed variables like these. There's no underlying science. No, there's definitely optimization we're doing, we're working at a higher layer in the stack. So the optimizations we are doing is more about tuning the kinds of things that you as a sysadmin or an operator of the environment would change. So again, everything that we do is post product. The software is already out there released. It's in front of the customer. We want to make sure that we give you the best guidance for how to run a workload under those conditions. So we're not down at the low level tweaking kernel parameters most of the time. Sometimes we do some tunables if it makes sense, but we are doing that. We're just doing it higher layers in the stack. We're not touching code if that is a question. Yeah. Well, yeah, I mean, there is lots of kinds of optimization algorithms. So benchmarking only on fixed parameters like these, they only give a set of possible results. Yep, right. And you can go way, way deeper. Yeah, but. And you can make a science out of it. That's fine. Yeah, absolutely. Any more questions? That's it. Thank you all. Thanks, everybody.


the founder of Code Pink, and welcome to another episode of China Is Not Our Enemy. For those of you who are new, this is a campaign at Code Pink where we wanna help educate those of us that are very undereducated about China. While the US government is funded with millions to miseducate and distort information on China using lies to get agreement to their aggression on China. Inflaming Asian hate internally and abroad, all while using hard-earned taxpayer funds to aggress on China to lie to his people with ships in the South China Sea, selling weapons to Taiwan against the one China agreement signed by Nixon, violating human rights in the islands around China and destroying pristine ecosystems. Not to mention that war is the greatest contributor to climate change. So yes, spreading hate and othering at a time with all the problems of the world, we need cooperation with China, not war. And that is why we bring to you episodes of China Is Not Our Enemy. We are watching a rise in the use of media to spread the State Department talking points and we're shocked to see them come out of Trevor Noah and so irresponsibly. Even ridicule him himself and adding in a no at Asian hate hashtag means he knew fill well what he was doing, spreading Asian hate. The characters drawn in his piece were racist and serve war, not peace. So we have invited two voices from Africa to discuss the issues that Trevor raises and that we see now spreading across media sources. So before we move to our guests, I wanna introduce you to two of our new team members at Code Pink and turn you over to their capable hands. Lauren and Allison, Lauren's done much of the work to bring this webinar together and all the forms that have invited you to be here today and it's gonna take care of supporting you on the Zoom and YouTube chat. And Allison Lau, our newest team member is facilitating the conversation between Mika and Kampali. And of course behind the scenes is Emily Dorell, our amazing social media mistress and guru. And so she'll be there to make sure this stays live. And then also this is a huge group of people, volunteers across the world that work on this project, China's Not Our Enemy. And we have an amazing also coalition. And so today's webinar is brought to you by members of our coalition to end US aggression on China and instead spend those funds on the needs of the people. And we thank them for all they do for peace and for joining us today. It is just world educational, Massachusetts peace action, pivot to peace, the Chow Collective and the People's Forum. So take it away, Lauren and Allison and thank you all so much for joining us from across the world and the US and for Mika and Kampali for being with us today. Onward to peace.  Thanks to everyone for tuning in today and welcome. I'd also like to thank Lauren for all your hard work in organizing today's webinar. And thank you to all our co-sponsors as Jodi just went over. So I'd also like to remind you all about our related petition entitled Tell Trevor Noah, Sinophobia is No Laughing Matter, which addresses the Daily Show segment that we're talking about on China and Africa. Their portrayal of what is happening in Africa at the moment reinforced the debunked myth of debt trap diplomacy, China's relations in Africa, spreading already growing anti-China misinformation. So sign the petition to tell Trevor Noah that China is not our enemy, stop spreading dangerous misinformation that will only increase US aggression on China and add to growing anti-China sentiments both at home and abroad. Thank you to those who have already signed. You will find the link in the chat to sign the petition or you can also go to codepink.org slash Trevor Noah. We will be delivering the petition to the Daily Show next Monday on the 24th of January. So get your signatures on. And next I would like to welcome our wonderful panelists for today who are here to respond to the Daily Show segment and share with us some information about what China is doing in Africa and what that looks like on the ground. So first I'd like to welcome Mikayla Mika Nando-Ursko. I hope I pronounced that correctly. Mika is an educator and researcher working at the Tri-Continental Institute for Social Research and a member of the Organizing Committee at No Cold War. She is also a part of the Media and Research Collective Dong Sheng News. You can find her on Twitter at Mikayla Nando and you can find Tri-Continental at tricontinental.org or at tri underscore continental on Twitter. You can find No Cold War at nocoldwar.org or at nocoldwar on Twitter. And Dong Sheng News can be found at dongshengnews.org. Our second lovely panelist is Kambali Musevuli. Kambali, a native of the Democratic Republic of Congo and one of the leading political and cultural Congolese voices is a human rights activist and analyst with the Center for Research on the Congo Kinshasa. So Mika Kambali, welcome to Code Pink. Thank you for taking the time to meet with us today and share your expertise. So I guess the first question to you is have you watched the 13 minute clip of Trevor Noah's The Daily Show segment on China and Africa. What are your initial thoughts, reactions? Mika, why don't you start off first? Sure. Can you hear me all right? Yeah. Thank you so much, Lauren. Thank you, Jody and for everyone who made this webinar happen. I'm really glad to be here just a little bit about me in relation to Trevor Noah's and this whole conversation is, I've been based in South Africa for the last 10, 15 years. And so I've followed his work a lot and his transition. So it was interesting watching it and as somebody who's been working more and more in trying to understand the China-Africa relationship through Tri-Continental, through the Dongsheng Media Collective is two things I think stood out to me that are more symptomatic of Western domination in terms of ideological hegemony and two, I think a part of our needs to deepen our understanding and broaden our understanding of what exactly is colonialism and how it operates in a neocolonial form, not only on the African continent but in many parts of the world. So there are two things that I would just want to touch on is one, what actually is colonialism? That's one of the questions that I felt, you leave that video having a misunderstanding about because in the classical sense, colonialism is the foreign domination through land ownership, through military domination of other countries primarily to exploit their resources. And a common misunderstanding today is that just because we've got flag independence in African nations is that there isn't the kind of same Western powers.  But in fact, and we, I think we demonstrated it really well in a study we published in July last year called defending our sovereignty where Tri-Continental basically looks at Western foreign militarization on the continent. And two things were clear. One that I think is fairly well known but is often, you know, underspoken about is the fact that the United States has 29 foreign military facilities now on the African continent, largely in Western and, you know, Central Africa and East Africa as well. And France has 10 or not 10 has bases in 10 countries which I think they have two in one of them. But even though that's well known and I think Code Pink has been part of that kind of work of unearthing what the US's foreign military presence looks like and the fact that it is the military budget is equal to the next 11 countries after it, you know, it has a mammoth budget is the fact that the US military apparatus in the African continent is not as they purport for diplomatic or security issues but at its core actually reinforces its kind of neocolonial interest on the continent. And that in fact, it does of course try to push back against China and China's growing trade relationships and diplomatic relationship on the continent. But if you even look at some of the US military documentation like in the Dick Cheney era, there are literally documents that say one is called assessing and strengthening the US supply chain capacity in a certain region. It's very clear that they're there to defend economic interests. It's not a coincidence as well that like France, one of their biggest military presences is in Niger very close to where they get their biggest source of uranium that basically is generating one in three French light bulbs today, right? And this is in a country in Niger where a very high percentage, I think it's close to 80% of people don't have tenable access to electricity. So I think the one thing is that I found something worth understanding about the conversation is that Trevor Noah is part of a larger misconception and misinformation that stems from a historically dominant Western ideology that also tends to hide the fact that it continues to operate in the same kinds of ways. And before I let Kambali jump in, the one part that struck me as well is I think the second thing he says is that he says with the stroke of a pen, you can sign away your sovereignty, something along those lines. And I found it striking because another thing that is not talked about a lot, but in recent years with the Pandora papers, with the Panama papers and things like that, we're starting to better have a grasp of the fact that it is the financial systems that exist today that are extremely predatory. And if we wanna talk about signing away our sovereignty, it's actually predominantly Western aid or financial loans and investments, as well as things like illicit financial flows where it's basically condemned for not un-condemned theft, where for Africa, for example, I think in the last 10 years, they averaged it out that 50 billion US dollars leaves the continent in various illicit financial formats, which includes like trade mispricing where certain things aren't priced correctly, where it's a balance of payments, where on the books, they aren't balancing out. And it's a form of sanctioned theft that's facilitated by various people in different levels of in the African continent, the like elite who benefit from this, as well as the kind of bigger multinationals who have this like arm of global financial consultants, you know, all these, what do they call your McKenzie's and all of these people who are able to push things around for them. But the reason I'm raising this is because how if we're gonna talk about things like the debt crisis, or even why that Africans go to China, we need to understand the context in which we are living in a debt crisis. The fact that if 50 billion per year is leaving just in what is known, because I'm pretty sure it's much higher than that, how are we supposed to deal with things like Africa's infrastructure debt? Where annually, if we wanna like reach our infrastructural needs, we need 100 billion. So already 50% of what could be that budget is leaving the continent. And I think that's an aspect of the conversation that is lacking the nuance of the context, the fact that there's a bigger financial system that has a bigger proportion in terms of the way it acts in a predatory fashion. Definitely the context is so important when understanding an issue. So Kamala, do you think this was an accurate portrayal of what's happening? Do you think that it's an accurate picture that he painted of neocolonialism in Africa? What are your thoughts and your reactions when you were listening to this segment? Absolutely. I mean, technique shared literally everything at the context in which we have to look at what China is doing on African continent. Speaking about the video of Trevor Noah, the first example he gave in the video got me already start watching it. I mean, I did finish the video, but the first example of speaking about the Ugandan airport, knowing that he has resources to do actually research to know that this was a whole story, surprise me that he actually put the story on his TV. So to just look at a few facts, right? He spoke about the Chinese are taking confiscating the Ntebe airport. The one of the, it's the main airport in Uganda. And the fact is because of Uganda's failure to pay the service agreement for the loan they took and the loan is of $207 million. Here's the fact that we can find that $207 million was given to Uganda through the ExSim Bank of China, the export airport Bank of China. The work started in 2016, in May of 2016 to expand the airport, the Ntebe airport. The loan is a 2% rate, an interest rate, right? That's something actually to know. I wonder who will give Uganda $207 million for 2% interest rate. And as the news came out, it's a journalist named Yasin Mugera who wrote an article on the Daily Monitor. On the 27th of November, the Ugandan government came out officially denying that this allegation was true. No, there is nothing of the nature that they have lost control on China is going to confiscate. So in the video of Trevor Noah, he does not speak about the Ugandan government, right? He starts by saying that the Chinese government on November 28th responded on the Twitter page, but a day before the Ugandan government already. So the information is widely available to verify the information about this loan that one, Uganda has not defaulted on the loan. Two, there is no confiscation of the airport. Three, the closures of the loan gives Uganda seven years grace period for payment, which means if you look for 2016 to now, literally at the end of 2022, that's when they can have issues with making payment if you make payment late. So they literally have a seven year grace period. So when you see that, like I'm able to pull out the fact about this loan and what is actually done between the two countries, China, I mean, the bank, China and Uganda, I'm wondering how come Trevor Noah was not able to find the same information that he actually put in December on national TV in the United States for some formation is not being held accountable for it. That's the one thing that's very disturbing. And as Mika stated, the entire Chinese hysteria that's happening is often pushed by Western media and for those who may have watched last month, there was an interview of the former president of Côte d'Ivoire, Laurent Babo by Pan-Africa Television. And in that interview, he was asked about China in Africa and his response was teller. And he said that Africans do not have a problem with China. There may be other nations who have concerns with China because they're losing their privileged position or trade on the African continent. What Africans do need right now is partners who are able to build infrastructure for them. It can be from East to West and the Africans will have to choose what is best for them to develop their country. So we are clear, at least on the African continent and I can speak on behalf of the Center for Research on the Congo that China's work on the African continent is a misconstrued. It's not clearly explained what is actually happening but when you look at actually what is happening as I mentioned, $207 million or 2% interest rate to expand your airport, I think it's a great deal and we can look at further clauses to find out what can be improved on that but what Trevor Noah put in his video was definitely an equivocation and that it should not be taken as fact. He has access to this personal information that I've shared but he chose not to present it and he chose to skew the information to join the group of those who are pushing the entire Chinese sentiment around the world. Definitely. So sort of speaking to that idea, Trevor Noah himself is South African but the Daily Show is an American TV show. I think you've both already sort of touched on this but do you think in this episode, the perspective of positionality is he's coming from? Is it more of the American TV idea or him as a South African individual and how did you interpret that? Kambale, why don't you speak to it first and then I'll hand it over to Mika. It will be hard to say that Trevor Noah is speaking for one or the other, right? Definitely he's not speaking on behalf of South Africans. I do believe as he also of his shows, he's pushing the dominant narrative of the American elite toward China and using media as a medium to push that information. It's very clear right now that the United States is losing ground on trade on the African continent. One, they do not have the capital to support African countries in development and they've never had that interest of developing the African continent. I mean, if you think about the decades that they've had of access on the African continent, what is the US trajectory on the African continent? Speaking from the DRC, I will say that when we had our first democratic elected Prime Minister, Patrice Lumumba say that he wanted Congo's resources to benefit the Congolese people and he had a progressive agenda for the development of the Congo. Within weeks he was deposed, within months he was assassinated with the support of the CIA, which is a US government entity. So we can talk about the numerous schools across the African continent, the coup against Guami crewmen, the support of African dictators, and so on. But what they've done on the African continent, they've more, the United States is more engaged in supporting oppressive regimes that provide unfettered access to African's resources. But now the game is different. Young Africans are looking at the reality of the country. We are a very young continent, right? Over 70% of the population on African continent is under the age of 35. This is a workforce that really understand that our land has the resources that's needed. Many of us believe that we must continue the agenda that the fathers and mothers of independence put forward for the development of the African continent. Guami crewmen said it very well. We need to control our land. We need to control our resources. We need to exploit those resources for the benefit of the masses, which means building the roads, the hospitals, all the infrastructure that we need for our country to develop. While we look at China's engagement on the African continent, it follows the lines in that, right? There is extraction of mineral resources, but that exploitation of the resources goes into what? The profits around that exploitation goes into building roads, hospitals, like the basic needs of the African continent that Western nation will not be able to provide. So Trevor Noah's agenda falls in line of the US agenda to counter China on the African continent because they are losing the trade access that they have on African continent. And China is providing much better deals that Western nations could provide to the African leaders. Sure. And that's something that was sort of left out of that narrative as well. So Mika, as somebody in South Africa, what are your thoughts on this and anything to add to what Kumballi has said about the positionality of his piece? Sure. So I mean, I think of course, anyone reading it with an eye for the details, if you just look at the production, it's already clear that as Kumballi said, I mean, I'm not surprised at the sources he cites because he is using, you know, CBSN, he's using BBC, he's using CNN, he's using the kind of classic center to write new sources that continue to, rather than illuminate, continue to create, you know, a blurred vision of the situation, which just feeds the US's cold war aggressions to be frank. But I mean, I even thought it was interesting he or his producers cleverly used a Indian media house as one of the second clips, which I thought was their way of showing, we're not just citing white people, we're also citing people from the global South, but that media house, WION, I think it's World, World is One News, something like that, is a historically neoliberal, historically, you know, problematic news station that doesn't serve the interests of the working class or the Indian people who have faced a really tough time, especially now with the pandemic. But I think in terms of like the perspective, again, I don't, as Kamala said, he doesn't represent necessarily how South Africans see, but there is a way, and I think the way that the South African media landscape is designed is it is very easy to bandwagon and to oversimplify because it's in the interests of even the South African elites to keep us a little bit uncertain of what the facts of it are. But I will say that I was actually very glad to see that a lot of people on Twitter who are experts on kind of China-Africa relationships, you know, there's Deborah Broutijam, who is a well-established data analyst who's looked at, you know, has talked about debunking the debt trap narrative, who like has collected it at John Hopkins, which, you know, is an elite university in the U.S., all basically said they were disappointed by the narrative that was shared in Trivedo's piece and saw it as defamatory and a hit piece. But I think also an aspect about it that I think is very frustrating as somebody who his own, you know, thing to stardom is that he's a child of apartheid.  And so I found it disturbing that when he said, say what you want about, you know, European colonizers, at least they were upfront. And I don't like it for two reasons is that again, the thing of it was because as I tried to explain earlier, the same kind of new colonial relationships continue to grow in fact and deepen in many places and particularly on many places in the African continent. And number two is the thing about being upfront because I think what really has been a problem for a lot of young people in Africa is the over dominant narrative that it's just simply the corruption and the incapacity of our leaders that reproduces the inequalities in society. And so to say, you know, A, there's no longer any kind of new colonial relationships reproduced by the Western powers and B, that they were upfront, that's not the case. And I think it also kind of gives a little bit of a patronizing thing to also Africans in the past as if we just kind of like gave it up, you know, they came with this, you know, they came and even the jokes he makes, they hired it as an A, B and B. I understand jokes, comedy are also objective but there's more nuance to it. And I think it undermines African agency to just say they hired it as an A, B and B or whatever it is. So I think that does undermine his fellow Africans. As you've already mentioned and Jodi mentioned, this is a moment in which to be sensitive around how we talk about China in relation to the Chinese diaspora that continues to experience extremely brutal forms of violence and abuse is a bit saddening. It's saddening as a person who comes from a context of knowing what racism is and how racism destroys families and destroys lives and destroys communities. So that was also an aspect that I think was more disappointing than anything. For sure, it's definitely not a great time to be Asian-American at the moment. And I think speaking to that idea of, you know, sort of infantilizing African leaders and, you know, taking away their agency, the segment sort of made it seem as though these Chinese loans are too good of a deal that they have to accept them, making it seem sort of like there's no other options, no one else is lending them money. What are your thoughts on this take? And, you know, if nations don't take loans from China, what are the alternatives? So Mika, I'll let you speak to that question first. Sure. Well, I think there are two things that I think we should note around this debt trap narrative is one, it is straight out of the U.S. State Department because it was, I'm blanking on his name, who was before Blinken? It was under Trump. Pompeo? Yes, it was Pompeo. It was Pompeo. And thank you, Jeffrey, in the chat. It was Pompeo who coined the term or popularized it, the Chinese, you know, debt trap. And so as Kambale said, it is disappointing that there was no effort to look into the multiple available sources who give extensive detailing of how this is a myth. And aside from the fact that we can talk about how it is a myth, if you look at the stats and everything, there are two elements that are raised that Kambale has already raised in part is that if we do even accept that, okay, there are issues, there are countries like Angola who have a large share of their external debt is going to China. I think it's like 40% is owed to China. They still have a totally different economic and ideological character. One is the fact that, and they talk about it in China as like slow investment where China has a longer trajectory ahead of it. So there's not the sense of trying to poach people immediately or poach people at all. It's like, this is a part of a longer term investment. And in fact, we've seen this through the fact that they've provided debt relief during the pandemic, some of the highest amounts of debt relief in terms of the percentage to the debt unlike private Western lenders as well as IMF and such, they did so but reluctantly and at a smaller scale, but that they've also forgiven certain debts. I mean, Zambia saw one of the biggest debt forgiveness or cancellations with, I think it was 200 million. So the approach to loan, loaning money to Africa from China is totally different because it's seen as an investment, it's over a longer period. There are no conditions in terms of how you politically operate as a country, which essentially was what the IMF and the World Bank did in the 70s and 80s across the world was you can only do this if you fundamentally like change your economy in a way that favors our private interests and our economic interests. So that's one, but two, I think even if, and this goes back to my first point about context is take the case of Angola that I mentioned, they do have a relatively high external debt, generally speaking, but what isn't spoken about is the conditions in which they go to China and they look for these kinds of agreements. And one of them, and I mentioned it before is around this question of like embezzlement and financial fraud reproduced by not only the Western multinationals but also by local elites. And the one example I want to give is the former president's daughter Isabel De Santos of Angola. I think it was in January 2020, the Luanda leaks came out and the International Consortium of Investigative Journalists found that more than 400 companies were not only hers but they included at least like 94 recognized tax havens where the profits of these companies were going to. And one of those companies is one of the state oil or energy. I think it's called Sonangan or something like that. But basically she was implicated in massive amounts of tax evasion and embezzlement. And this isn't a country where by the end of 2018 Angola had around 103 billion going in capital flights which again is theft.  It sounds really nice. Like these little money birds just are floating off but it's a form of theft through predatory financial practices. So understanding then that you're losing a hundred billion in a year just to that aspect in a country where more than 70% of the population are living in a rural area below the poverty line then you understand the context in which you start to look for alternatives and you start to look for loans. So I think that's an aspect that in Trevor Noah's discussion in any other discussion needs to be at the forefront aside from the fact that it's purely propagandistic and trying to further a no-call to agenda and a demonization of China aside from those obvious facts. Yeah, so I guess Kamala like in response what is China doing? How is it different than these other predatory loans and what does it actually look like on the African continent when with this Chinese investment? So I will use the example of China's contract the Chinese contract with the DRC which has been ongoing for almost close to now 20 years. In 2005 and six a few Congolese bureaucrats went to China, struck a deal, a nine billion dollar deal for development in DRC. The plan of that deal was $3 billion was a joint project between the DRC and China to exploit cobalt and copper and the proceeds, the profits being generated by that project will pay for the remaining $6 billion part of the deal was to build the infrastructure that we needed. So it was a mineral to infrastructure swap. So at the end of the project, normally Congo will be dead free, right? So when that took place around 2007 and eight there was pressure from the United States and there were literally blackmailing the Congolese government where at the time Congo owed about 14 billion dollars of debt to the World Bank. So they used the Brenton Wood institution, they used the State Department to tell the Congolese government that you need to renegotiate the Chinese contract or we are not going to forgive that debt. And that was ongoing and there is even a very interesting article in 2000 if I'm not mistaken, it was in 2008 on the Financial Times where you have the Chinese ambassador World for World said in the interview that Western nations are blackmailing the Congolese government around this deal. So I found when I read the article I was first and I said, what? How in a financial news magazine this high level official, this diplomat can make the statement. If you a year after that, many people may remember that Hillary Clinton visited as a secretary of state visited Africa and she was in the DRC. Once she went to DRC, she was all over the news. The main reason why she was all over the news is because there was a college student in DRC who asked her a question that was mistranslated which she responded very bad. I mean, you can find the video on YouTube. The question the student asked was we see the interference of the American government with the Chinese contract. And we see you here in Kinshasa. What do you think about this US interference? It was translated as what does your husband think about this Chinese, this interference with the Chinese contract? And she responded that are you asking me or are you asking my husband? And if you're asking my husband, I am the secretary of state so I should not respond to that. So in the US media, that's all that was discussed that there was this young Congolese who asked a question and the secretary of state did not respond, this was a journalist. I mean, they did not focus on the essence of the question that a young Congolese is asking the secretary of state of the United States about the interference of the United States in the Chinese contract. What happened after she left, the month she left, I think it was July 2009, as soon as she left, the $3 billion guarantee for the loan repayment was removed from the contract. Which meant that now in 2009, when Congo was literally debt free, it automatically became a country with debt owing China $6 billion. No one is speaking about that. That actually the United States made the DRC become a country in debt of $6 billion. A year ago during COVID, in January, Councilman, the foreign minister one shows up in Kinshasa. Just as is done for Zambia and other African countries, forgive a few debts, start discussion around what China can do and the support that is already, China is doing a lot of infrastructure projects in the DRC and also there are some exploitation of our resources. No one is doing the scale of the work that they're doing in the Congo. So as soon as he left the DRC in January, two weeks later, guess who shows up in Kinshasa? U.S. Generals show up in Kinshasa, Africa, the U.S. African command. These Generals come to Kinshasa and as he was seen after their visit, literally to counter China's diplomatic visit to DRC. Right after in March, the Congolese government once again says, we have to renegotiate the Chinese contract. For someone who's only hearing the story now, right? You don't have the context of what happened in 2006. You don't know what Hillary Clinton did in 2009 that this contract has been renegotiated while we are more in debt. Why we should ask why right now, right? We have to again renegotiate. What were the clause that was now renegotiated in 2009 that need to be renegotiated now? And what is actually the fight that's happening in the DRC? The fight is simple. Congo has cobalt, has large deposit or lithium. These resources are essential to modern day technology, your electric cars and so on. And it's a prime mineral right now because of the development of the world and moving away from fossil fuel to renewable energy and so on. And Congo produce over 50% of the world's cobalts. We are the number one producer in the world. China has one mine, China molybdenum as a mine that exports cobalt. How did China get this mine? You'll be surprised. There was no debt diplomacy for this mine. There was no influence on local elite. The United States through one of its companies sold it to the Chinese. I think people don't understand what I should say. China has one of the largest cobalt reserve in DRC. They did not buy it from the Congolese people. They bought it from an American company called Freeport McMorrin and a Canadian mining company called London who controlled the mine at 80% and Congo was getting 20%. As Freeport had financial issues for its operation in Latin America, they needed cash so they sold the mine to the Chinese. So when London and Freeport sold the mine to the Chinese in DRC, what will happen? China will have 80% control of the mine and DRC will get 20% control as it was the initial clause. So how come in 2021 now it's presented that Chinese are exporting Congolese with a control of 80% of the mine where no one said anything when Freeport McMorrin had London control? And those things happened over and over. We spoke earlier about the NTA Bay. Like whenever you start getting into the details you start finding out what is actually the issue. And I will end by saying Africans, we know what we need and we are willing to work with anyone who will develop our continent and help us as a fair partner. And this is actually what took Mika and I on a trip this past month, not actually, it's about almost two months now. In November, we went to Senegal in Dakar where there was the FOCAC, the Forum on China Africa Corporation that was taking place in Dakar, Senegal. Started, it was the 29th and the third year for November brought in almost 54 ministers, foreign affairs ministers and governments to the table to discuss China's engagement in Africa and cooperation. While we were there, we were clear, I mean, that one first the article that we are discussing that Trevor Noah put in his video, that article came up on November 25th, four days before the China Africa Conference. So every time there is a FOCAC taking place in the world there is a hook story that's published that gets pressed and people don't get the context that this is actually propaganda. But in the meeting we'll unfold it. What did we see? And I hope Mika also can add on more to some of the, what we saw there. We saw that China's commitment to Africa is long-term. We saw that China has a plan. China pledged to give Africa one billion, those are vaccine, 600 million of that is a donation that 400 million dollars will be locally produced and much more planned. But what is clear to us at the meeting is that we as African, we have work to do and the work that we have to do is not China's work. We have to deal with our compradors because some of our compradors that we saw at the meeting were not actually in the meeting negotiated. They were in the meeting enjoying having a flight to Dakar or engaging in not providing a counteroffer of what is good for our people. But how did we end up with leaders who don't represent us? I mentioned earlier, Patrice Lumumba, the Democratic elected prime minister of the Congo was assassinated by the CIA, which has made the Congo for over 60 years unable to choose their leaders and every election that we have, there is US interference. So on one hand, we have leaders who do not represent us. They are part of the elites, right? Many of those leaders are forced into power by the United States to some of their interests and they are the ones making decisions for us. So China is engaged to say, I'm ready to work with you. As we as young Africans, we must work to make sure that these compradors don't continue to affect our lives. But for more we saw, I believe that what China is providing to the African continent through the China-African cooperation that we attended, is something that Africa needs. But we must have leaders at the table that reflect the aspiration of the African people. Wow, what a great response. So I think sort of following from that, sort of how do we then sort of distinguish and respond to valid criticisms of China's activities on the continent? And, you know, in other sort of areas in the world versus anti-China rhetoric. And how does this sort of tie into media portrayal of what colonialism is, what neocolonialism is, such as in Trevor Noah's segment? Mika, do you have any sort of response to that? Sure. I think the first thing to ask that isn't always apparent to people in very comfortable positions like Trevor Noah is what is actually at stake in this moment, right? At stake are the lives of Asian people and people of Asian descent who've been feeling the discrimination that we mentioned earlier. At stake is a clear, increasing buildup to what could be a new Cold War that will clearly most negatively impact the people of the global South, as well as the people, the working people, the ordinary people in the U.S. and even in Europe. And if we ask ourselves what is at stake, we then need to begin to ask ourselves not only are we debunking it as Kambale is saying, like actually showing what the relationship is, but also I think there's a level of like optimism that I think we need to look into some of these things. As he mentioned, we were at this forum on China Africa Conference. And I think what struck me the most was in President Xi Jinping, the president of China. In his opening speech, he said, he quoted one of the most like famous and important leaders of, you know, African national liberation history, Leopold Sengor, who was the first leader of Senegal in terms of the post colonial era. He quoted him saying, let us answer present at the rebirth of the world. And oh, actually that was his closing sentiment. Let us answer present at the rebirth of the world. And as Kambale said, what we find with Africans is we aren't present saying we are present, we are stakeholders, we wanna push our agenda when it comes to any kind of international agreements. There is a very parasitic elite group who have access to these conversations and who are steering them for their personal benefit. But China time and time again has shown not only historically where, you know, Kambale mentioned the coup that killed Patrice Lumumba. When the coup was happening against Kwame Nkrumah, who was the first leader of Ghana, he was on a diplomatic visit in Vietnam. And he actually got the news from Premier Joe Enlai, who was one of the leaders during the time, got the news of the CIA or, you know, Western backed coup that his country was going to fall to a Western agenda that had the face of, you know, African elites. And from those early times, China had shown a commitment that is historically and ideologically totally a departure from the historical and contemporary one of the West and the US of solidarity, of building, you know, infrastructure in the late 60s, early 70s, the Tanzania-Zambia railway line, which would connect Zambia, a landlocked country to the coast in order to export at a time in which there was no, from the 500 years of Portuguese colonialism, there was barely a school in Guinea-Bissau and other territories. There's barely any public resources and infrastructure for African people. And China, not only in its historical experience, but contemporarily has allowed us that opportunity has been with us, producing that kind of the conditions in which we ourselves could self-develop. And so even though it might seem small, it's not insignificant that China has funded the building of, you know, big hospitals. They've began producing the CDC of Africa. The Disease Control Center is going to be fully financed by China. It's not insignificant that they even built things like stadiums where we can host a lot of our own cultural events that are our own exports. A sport is a big thing on the African continent. And another thing that's not insignificant, which again might seem symbolic, but shows a level of dignity and respect that historically and contemporarily we do not get from the West or from the US is the fact that diplomacy has been really high up on China's agenda. The first diplomatic visit made every single year for the last 32 years has been China's like foreign minister coming to Africa. He just completed his first one where he went to Kenya, Eritrea and Comoros, which I'm sure for many on the call, if they can locate where Comoros islands are, I'll be very impressed, but it's a tiny archipelago on the East Coast just facing Mozambique's Northern part. And it's a population of 850,000 or so, just like under a million people. But they chose to prioritize that in their visit. This is, if you think about its size, what is that island that's next to New York, Long Island? Long Island is bigger than that place. Long Island has I think seven or eight million in population, but you don't see people like treating it with respect. The fact that they went to the Comoros islands, I think shows a level of respect for the small and big players. And even if it's just symbolic, it is significant that high level people go there and have these conversations in respectful ways that recognize our conditions, but also our potential to want to self improve. And lastly, I'll just end off with the fact that at the end of the day, if you look at China's own development in its own country, that is something Africans are interested in learning about. How do you escape the legacy of Western imperialism, of feudalism?  And a lot of those things have found material root in Africa, such as the fact that some of the most popular phones right now, particularly in East Africa are techno phones, which are much cheaper than American and Western brands. They also have a lot of features that take on the reality and the needs of Africans, where there's some phones that you can put in three SIM cards because Africans always are trying to play the different networks. You don't always have good signals. So you have to have three SIM cards. In Ethiopia, they have keypads that you can use Amaric, the language of that, or the dominant language of that region. The fact that they have cameras that also show your skin color a bit better. They are taking into consideration also what people's needs are. And of course, there is a commercial interest behind that. China wants to develop. Africans also want to develop too, but there's still a more humanistic overall approach that at least provides space in which we as Africans can be owners of our own destiny and protagonists. But what we have to struggle with, for those of us who are part of social movements, trade unions and stuff, is to be leaders in our different nations and to be leaders who will address China in forums like Focac, because ultimately that's where we find the big problem is we aren't present at the rebirth of the world. Thank you so much for that analysis. I guess to continue on that, to Kamale, sort of what is the context of cooperation between Africa and China, such as Focac? And what happens, for instance, when nations default or have trouble repaying, or with the situation of COVID and how that complicates the situation, how is that cooperation? What does that look like? How does it play out? At least that contract that I've seen so far written by China, actually so one, this past month I was just in DRC, I saw one contract that was written as a memorandum of understanding. Those clauses exist. What many people that have seen negotiate contract with China, they're trying not to put the framework of the World Trade Organization because something can be challenging for African nations to deal with that, but they are always close for arbitration. Even when we're talking about the Ntebe Airport, the 207 million dollar deal, they are close for arbitration about how it will unfold, where it will be. Do remember Uganda had, when the contract was being signed, they had some concern with where the arbitration would take place. But those things are actually done at the signing agreement table. So for me, it's more question of the level of engagement that Africans have to be with China. It has to be from, it's already happening from respect, understanding the needs of our people and making sure that whatever is being implemented from the country itself is actually also what's happening on the ground. COVID has affected many of the project that China had in store, but I always remind people about COVID what was really inspiring to me, even with the delay of the project. We have a huge Chinese population on African continent and many of them working in China's project and others are doing investments or doing trade and so on. One thing that I really appreciated about the Chinese is when COVID started, Chinese people stayed with us. Now I'm in Ghana, I'm speaking with you from Ghana and I can tell you that the United States Embassy had shot of plane for Americans to fly out of Ghana during COVID. Chinese did not get on shot of plane to fly back to China, right? They stayed with us during COVID, their projects were delayed and they lived with us and they've been on the continent for a long time. They supported liberation movements and through our continuous engagement, now I hope we will get to know more, like one of the things that we hope could happen is, how can African people learn more about China's experience? One of the Chinese experience that's very important to us is the fact that China has eradicated extreme poverty. That's one of the biggest human milestone in the history of mankind that took place in the past, no, two years, right? That we as Africans, we want to study because we want to make sure that no, we have, the people have access to food, they have access to water, have access to development so that whenever we are engaging with them, we are learning from them and we are learning more, say, about their history. At least that's what I was saying. I will end by saying that today is January 18th, it's a day after the commemoration of the assassination of Patrisse Lumumba. Patrisse Lumumba was assassinated in January 17th, 1961 by the CIA, bringing that up because I read an article today about how he was celebrated in the ERC. The Congolese prime minister went to a memorial place to put flowers and then in the end of the article, he stated that China is building a memorial for Patrisse Lumumba in ERC and that whenever Patrisse Lumumba remains, which has been found in Belgium, we return to ERC, that memorial is where it will be. That just give you a sense of even how China is connected to the reality of Africans because I don't see the United States building a memorial for Patrisse Lumumba, right? China is building a memorial in ERC we Congolese at the time. I think that's the way for much more engagement with Chinese to understand the history, to understand the experience and to learn from the experience for us to be able to transform. Wow, such great analysis on that. I think sort of to wrap up now, even though this is a very complex issue as we've sort of seen throughout this webinar. So where do you see the African continent and African nations being present in the future and what part does China play in all that in a sentence or two? I know it's very complex, but where do you see the future then? Either one of you. I can start that way, Mika you have that's what. I keep quoting Lumumba because I think that he's someone who has to be studying more. I saw someone in the chat roads that people should read White Malice. I said book that was just released in 2021 by Susan Williams, very important book. But Lumumba in his last letter to his wife says something about the future of Africa, right? He believed that Africans and Congolese will continue to fight until we are free. And then he said something that I always remind people, Africa will write its own history. And it's not going to be the history that's written in Paris, London, Washington and Western countries. It's gonna be a history of glory from North to South where Africans are coming together to transform the continent. And he said that we are not alone, Africa, Asia and free and liberated people from all over the world will always be at the side of the Congolese. That's what I believe the future of Africa is. China Africa's cooperation shows that, right? China Africa cooperation shows that Chinese have answered the call where Africans called for help. They call for support. They say that we need an ally to help us in development. And they've said that we are present. And China is present with us. We as Africans, we have to fight to make sure that those who are speaking on our behalf reflect our aspirations. And I know that the future of Africa is bright. If we keep that engagement with China, where we're learning from them and that we are doing our part to make sure that the entrepreneurs are not controlling our countries, the future of Africa is very bright as Patrice Moon was telling. I love it. I love it. Mika, do you have any closing comments? Any responses to that particular about the future? One would be just echoing of two things. One would be echoing the sentiment shared by Kambale is yes, ultimately there is a serious level of work that has to happen amongst African people to organize ourselves in a position in which we can be part of these conversations at a state level. Because when we were in Senegal, the day after the conference concluded was one of the biggest general strikes of taxi drivers who there was going to be a price hike in the registration fees and things like that. And you just come from a forum where China's sharing so much and he mentioned the 60 billion but it was the whole very classic Chinese 10 points to everything. They were out of the 10 points they were gonna be 10 projects of connectivity, 10 digital projects, 10 agricultural projects, all of that is being proposed. But we didn't see and when part of the conference it was a business section there was a business summit within the conference. Whilst we saw young Chinese people whilst we saw a proposal being made, we did not see young African people were represented. We did not see even as a gesture you could do it even very liberally as a gesture the young entrepreneurs who populate the continent, Africa has one of the youngest populations who are largely unemployed and you didn't see them represented in this discussion. And so there is a lot of work on our part to bring ourselves into that conversation because ultimately, although China has different actors which is important to deconstruct they're not all one homogenous glob which is the racist way that the US targets and creates misinformation. But we have to fight to be part of that conversation on a state level ultimately if we wanna see some of these material changes that have been happening actually benefit African people. And lastly, I think I would almost have a call to action for those of you who are peace activists part of the peace movement anti-militarization is that in March, I think there's going to be the African land forces summit in Columbus where basically I think Africans are gonna African leaders or leaders of the military forces are going to the US supposedly we'll see what happens with COVID to basically look at different military gadgets discuss strategies, all those things but basically to amp up military cooperation between African people or African leaders and US leaders. And so my plea would be like how can we find ways in which we can denounce these kinds of initiatives and unmask what they really mean unmask that they do not serve the African people unmask that they serve the interests of multinationals who want to profit not only from our labor and our land but from our killing. So I would just make a plea that when we find even Africans coming to your part of the world to participate in militarization and its amplification that any of you who are involved can voice your protestation of these kinds of initiatives because that would help the African people. Well, some really great comments and some really great analysis and thank you so much to you both for sharing your expertise, sharing your insights and just illuminating what's going on on the continent. I just want to again, thank our co-sponsors as well as our panelists for this amazing and insightful discussion that we've had today and thank you to everyone for tuning in. Thank you both to Mika and Kambali. These are our co-sponsors on the screen and I hope everyone has learned something and enjoys the rest of their day. So thank you everyone for tuning in. Thank you very much. Thank you, thank you, thank you, thank you for hosting us. Thank you so much. Your comments were so great. Your analysis was so amazing. I'm sure you saw in the chat how much everyone was learning from both of you. I did not realize there were lots of comments. Yes, very good. I was trying to follow, but I only picked up a few. Yeah, they were alive. I was just wondering. I think so.