Hi, my name is Joe Gordon. We'll talk about how Diablo vs. Havana, how OpenSack has matured. So a bit about me. I'm a full-time developer on OpenSack for HP. I actually don't do anything internally for HP, or not too much. So I work all upstream. I'm a Nova Core. I've been involved in Essex. I've been trying to work on making OpenSack scale and actually work so people could actually use it. So a brief history, hopefully you guys know most of this now, history of the integrated release, so what OpenSack has been from Diablo to Havana. So Diablo was way back when Swift and Nova did everything else. Nova had Volumes Network Identity. Keystone was there, but it wasn't integrated at the time. And we had Glance. Next came Essex. We added Keystone and Horizon, and we got rid of Identity and Nova, because it didn't really belong there. Folsom came along, and we added Quantum and Cinder. So we got rid of Nova Network and Nova Volume, although Nova Network is still there. And Quantum is still trying to work on getting rid of Nova Network right now. Grizzly, we didn't do too much more. We just kept working on what we had. And this is what it looks like at this point. So it's nice and big and complex, and it's only getting bigger and more complex. So this is the Grizzly, and there's no, I don't unfortunately have this log yet for Havana. But Havana added two more services, Solometer and Heat. So now we have, I believe it's nine projects running, over 32 services to this big complex system, and it's getting bigger. So you look back, we're trying to compare Diablo versus Havana, what's changed. A bunch of these actually didn't exist in Diablo, so we're gonna mostly ignore them for this presentation. So Horizon, although it existed, it wasn't integrated, Neutron wasn't there, Solometer and Heat weren't there, so we're gonna ignore them for most of this presentation. And looking back, so we get back to that same original list that we had before. So this is what we're gonna be focusing on for Diablo versus Havana to get a good comparison. So OpenSec is grown as you saw, it's grown huge numbers, lots of new services. It's grown out, we have lots of new services, Heat, Solometer, we have Trove coming out in Icehouse, we have Ironic, Marconi and Savannah in Incubation, and hopefully those will come out in a release or two. So we're growing the project out in many ways, doing more things, trying to, as a project, as you can see, it's getting involved in more components, but it's also been growing up. So each project is actually getting more richer, and part of that has been splitting out Nova into smaller projects where they could actually focus on important pieces. So for example, we split out Cinder, so Cinder when it was a Nova wasn't that much, it had one backend. It didn't really do too much, it was sort of a second class to this, and now it's a first class citizen doing great things, there's a new API, lots of future rich aspects to it. We've also been adding extensions, so API has been growing, so Nova has been growing their API in huge amounts. We have a core API, we have extensions. So this is a list of all the extensions in Diablo. It's not too many here. This is the list of all the extensions we added since Diablo. So it's a lot more, I think it's about 60 or so here. So we're really growing out the APIs in really big ways. So one way this is changing is we're getting bigger, we're growing out, and we're also growing up. And that didn't scale well. But we're gonna ignore that. So once we normalize for that growth and the project scope and what we're doing, what else have we been doing in the past for releases? So part of the inspiration for this talk is that OpenSec infrastructure uses Diablo and a Havana-based cloud interchangeably. These Rackspace and HP Cloud is on Diablo, I believe, and Rackspace is on Trunk, which is essentially Havana. So they use them interchangeably. So they both mostly work, right? So what have we been doing besides that? If they both work, they're not using new features because they have to have the same feature parity in both of them. So what else have we been doing all this time? And I wanted to know the answer to this, so I thought I would give this presentation and find out myself. So it turns out we've done a lot. There's sort of some high, big categories of this. There's some new drivers, lots and lots and lots of new drivers. As you usually don't actually see these drivers, they're all in the back end. Lots and lots of testing and fixing bugs, performance and scalability, that's the big one that all everybody cares about. And also a lot of forward-facing changes to plan for the future and make things easier in the future that you don't necessarily see until the future is done, but they involve many cycles and many man-hours to fix. And also we've changed the process a lot. We've grown from a small project to 1,600 developers, I think, since the committers since the beginning and 1.7 million lines of code. And so we've grown a lot. We had to change a lot of the process to adjust for that and we're still working on that. So more drivers, this is a pretty straightforward, easy thing to look at. We just have new drivers, Nova. There's two examples from Nova and Cinder. Nova, we have Bare Metal, Docker, Hyper-V, Power-VM. They all came in different releases. Cinder has, this is only a small sample of Cinder. None of these exist in a Diablo. I think it had LVM and NFS, I think, and maybe that was it. Maybe one more when it was in Nova, back in Diablo. Now we have all these other great vendor plugins. So we have a lot more drivers. As a user, I don't actually see that and so something on a cloud doesn't know about new drivers because that's just how you run the cloud in the back end. So better testing, fewer bugs, that's sort of the goal. Make sure everything works. Don't have bugs for users to see. So there's some big categories of this. Unit testing, integration testing and fixing all the bugs we keep finding in all these and then some. So it actually turns out in unit testing we're actually no better than we were before. So I got these numbers by comparing the percentage of lines of code for unit tests to the lines of code of actual project code.  So the code has increased in all dimensions. So we've more or less maintained the level of the number of lines of code percentage-wise to actual working code. And retrospect, this actually makes a lot of sense because we had good coverage from the beginning. We have about 80 or 90% in Nova, I think somewhere in the mid-80s. So you have pretty good code coverage and then whenever we add a new feature we always make sure there's more tests for it. But we don't actually spend a lot of time just adding unit tests for the sake of unit testing. We try to make sure they come in in the beginning of each feature and each bug fix. So this has been true sort of across the board. There haven't been really too many big changes in unit tests. This actually caught me by off guard. I figured we're growing, we're having more testing. Thought we were really gonna do much better on this. Turns out unit tests are great, but that's not enough. So Tempest is the open-sack integration test suite. This is the big one we've really grown a lot in leaps and bounds. So back in Diablo it's this little small project called open-sack integration tests. It was written by Soren, I believe. I don't think it's, he's somewhere in the summit somewhere. And had 107 tests more or less. They didn't really do too much but they had the really basics and they caught a lot of bugs and they were great. But it was only 107 tests covering mostly Nova. Flavors, floating IPs, image metadata, images, key pairs, volumes, which at the time was in Nova. So very basic things. And there's only 2.6,000 lines of code. It was actually the first release of Tempest was in Diablo. And we wouldn't actually use it to gate on that point.  So looking at what we have today, it's actually grown by a order of magnitude. It's 10 times bigger in most definitions. 1,200 job, 1,200 tests, 24,000 lines of code. So we've really actually grown this one out in many big ways. And actually it's still too small. This probably should grow by another order of magnitude the next year or so, if all goes well. It's 1,100 API tests. So API is important. We like to test it, make sure it doesn't break. Want to make sure that we don't actually break the APIs, that's really important for our users. So we have lots of tests for them. So that covers all the main projects now. Compute, Nova, Image, Glance, Identity, Keystone, Network, Neutron, Nova Network, Object Storage, Swift, Orchestration, Heat, and Volume Cinder. So we cover all the big ones, 102 files. We have some CLI tests. We want to make sure our CLI doesn't break also. That's mainly you run a CLI command. You want to make sure it doesn't stack trace on you. We've had a problem to this in the past when we added these tests in. And we're having some new kinds of tests coming in that aren't actually listed here. We have scenario testing. So you want to test it a big complex scenario. You spin up an instance. You attach a volume, do something to the volume, detach it, delete the instance, that kind of thing, a big story. We're also working on performance and stress testing. So you want to make sure our cloud works and it works in all metrics, stress, and other. This actually all runs in parallel. That was a big push in Havana. And we did a great job of it. And it takes only 20 to 30 minutes to run in gate. So that you push your code up for review. 20, 30 minutes you get results back saying it passed or not. Not too bad of overhead for a developer to actually to have for getting such great testing. So just to break down the API test more to sort of see what we have. So NOVA is a huge project. It's one of the oldest ones. It was one of the first projects in NOVA and OpenStack, NOVA, and Swift. So we have a lot, a lot, a lot of tests for it. 785 API tests. Keystone's been an old project and also very big. There's a lot of new tests in here. There's a V3 API that came out that has full coverage in Tempest, I believe. And we have a lot of other ones for all other projects. Imaging, network, object storage, and volume all have tests. And we're gonna start adding more things for things like heat and others as we grow out. So API testing is a really big part of making sure everything works. Because the APIs are all the same and we haven't really, we've read some APIs, but we still support all the APIs in Diablo. The question I sort of had was, could we run Trunk Tempest, Havana Tempest against Diablo? And so the example is this case is DevStack, which is our development environment. It turns out you sort of can. So it took me a few days to get this up and running. So the basic principle is we have new APIs and old APIs that have been in Diablo. So let's try the new Tempest against the old APIs to see how well they actually worked. So Diablo has been end of life about at least a year ago at this point. So it actually isn't supported. There's no branch, it's just a tag upstream. So it took a bit of tweaking to get it actually working. And the results we actually ignore a lot of these, a lot of the results because many features, all those extensions in Oven, all the new APIs and projects that weren't there are tested and we could ignore those results. So I actually got 622 out of those 1100 or so tests to run. I got 154 of those to pass and 110 are compute. So actually it's more tests than we had in all of Tempest back then. So just some results of that. So that was, I thought a little surprising. Actually it worked. I didn't think it was gonna work at all. It didn't work as well as I thought. Part of that is some of the results I found in Diablo we actually had really poor parameter validation. Passing bad parameter, it would just return a 500 break wouldn't tell you. Really bad user experience. And we've done a lot of work in fixing those. Some good examples of invalid key name, pass a key name and something would fail quietly in the background and you would sort of curse to yourself and not what's going on. Now I actually tell you, hey you can't do that. Examples of your key name is too long. It'll say you can't do that instead of just truncating it and not telling you. Another example is the min and max count for multiple creates. So you say I wanna create between this amount of instances and this amount of instances and it used to accept negative values which doesn't make any sense. So we actually tell you that hey that's silly don't do something that doesn't make sense and we actually have a better user experience. There's some bad things I found actually which is we actually changed the APIs and we broke things and we're sorry for that and we're not gonna do it again, we promise. Well, we'll try. We change a lot of the negative cases so you do something and it's gonna fail on you. So it used to be a 400 error saying this is invalid, I can't do it. Now it'll just return a 200 and say it'll be quiet about it. I didn't actually dig down on how this changed but that was one of the big things we noticed. The temp has changed for that along with the projects and that's something you have to be more careful about as you go forward and make sure the APIs stay actually stable. There's also really poor pagination in Diablo. This is really important for a really big cloud. You have 5,000 instances or 10,000 volumes or something. You wanna make sure you don't have to do, you don't wanna get all the results every time you wanna use pagination there and it didn't actually work in Diablo and we didn't test it, it didn't work which is no surprise there. We did a much better job in testing and actually having it work in Havana. Another big one is you actually only tested in Jason and Diablo, we have two types of APIs, Jason and XML, there's supposed to be the exact same thing once Jason won XML. As developers we generally like Jason but we understand the value of XML so you wanna support both. We did not test any of that in Diablo. We actually tested all now. All the temp has tested in Havana will run both XML versions and Jason versions. With the same unit test, so it's actually really great and we're trying to make sure they actually look the same from the user point of view. There's a lot of false negatives here in the results because the API is good but there's a lot of corner cases that aren't really clearly classified. But a good example is you have metadata in images and we change how we use that slightly and that will break things. So even though the APIs have stayed stable we actually have adjusted a few corner cases that make things a little hard to make your code really, your test really flexible from Diablo to Havana. And so there's actually been some problems there and that's a problem we're trying to work on. It's a really hard one for us and we're slowly trying to fix that and make sure that your code actually is, the APIs stay the same, that's important for us and we're gonna keep working on that. I think in conclusion Diablo was pretty bad and I think it's safe to say that now. I think we knew that at the time but it was, we're always making progress just better than before. We did a bad job of a lot of things, bad job of testing it. This wasn't actually running against gate. APIs were sort of okay and we haven't done a great job historically of making sure we don't break APIs and we're more aware of this now and we're really putting a lot of effort into testing and making sure that things actually stay the same. So critical bugs is another big part. We don't like critical bugs, they're bad, right? Turns out we have more of them now. I didn't see this one coming. So in Nova we've actually gone up, we've stabilized I think is good but there's overall there's really no trend. There's just some big reasons for this actually where critical bugs, a few things is that as the project has grown it's actually been harder to track all the bugs and classify them in the same way. The number of amount of code has actually grown too so what's not really shown here is the amount of lines of code and features has grown in massive ways and so the critical bugs not exploding it actually sort of means that we've been keeping up in the level of quality with the scale and not doing any better which is actually pretty good. So that means we're not getting better, we're not reducing the number of critical bugs, we're keeping it sort of stable if not slightly higher, slightly lower while expanding the scopes of the project. So that's pretty good. Some big categories of these bugs are those non-gated features, you add a feature in, it doesn't work, you didn't add a tempest test since we're not gating on it so when you write a code before it merges it you're on all of tempest on it to make sure it doesn't break anything. So if it's not in the integration testing it may break without us knowing and so those are hard ones to catch that may be critical of the important feature. We didn't test it, we didn't know it broke and so it breaks and we have to fix it. Another one is dependency if you actually float the dependencies upstream this has been a very long and painful battle for everybody and not everybody agrees with the floating dependencies. The big one is an upstream project who wanna create packages, let people downstream create packages. We don't wanna say that you have to use this dependency that's four years old because that's all we tested. So we actually float a lot of dependencies saying this is open-ended and a new version comes out and it breaks everything. We accept that in the gate because that sneaks in because it's a dependency change and we don't actually check those. And that's actually on purpose and so we can actually fix them right away. So those happen every few months and everybody freaks out and they're fixing a few hours and everything's fine but those are always critical. They block everybody's work but they're fixed very quickly and then we can move on from that. So that's another big category these critical bugs is. Gates down, that's critical. Nobody can do any work, no patches emerge. That's critical, we gotta fix that. Another hard one to catch in gate is race conditions. So if something fails 10% of the time it's gonna get in probably. So we're trying to do a better job of actually not letting those in and identifying those but race conditions could be really big and really hard to detect in gating. So it passes all your jobs you ran against it to make sure it works but it fails 10% of the time and you only ran nine tests so you miss it. Performance is another hard one. We run really small open stack clouds to test. It's DevStack, it's all in one. It's not what anybody should be deploying in scale. It doesn't work at scale I think. So we really don't do a good job of testing performance. This is another thing you're actually working on. So another big category of critical bugs is performance. Somebody comes along, tries it out. Somebody running a continually deployed cloud actually tests it out and says, hey, this doesn't work. This has happened quite a few times. Somebody comes along, somebody racks based or whoever is saying, this isn't working guys. Please fix it and they'll tell us right away that we put the code in last week. They ran it the week after. We fix it pretty quickly but it's a critical bug. That means all hands on deck. Do everything and kind of fix it. So performance and scalability. This has actually been a hard one for me to test. Performance is a little easier but scalability is sort of hard for to run on a few VMs somewhere. So there's some big things underneath that we've done to really make things better. So Nova Scheduler is still not great and but it was much worse in Diablo. It actually didn't support active active. We can only run one scheduler. We still have a race condition but we fixed it or worked around it. We have the race condition of two schedulers aren't aware of each other. So you have two of them running. One schedule something in one place, other one does the same thing. They don't know about each other. The compute node is over subscribed and that's bad. So we have a way around that. We fix that problem so they can have as many schedulers as you want which makes a lot of sense when you want to support that. There's some other small things we've done so far and we have a lot more work to go forward. We use an RPC fan out or we used to use RPC fan out before. At one point in Havana it was there but it wasn't doing anything. So there was one piece of the schedulers using it but nothing else was. This is all this extra traffic going through and the schedules had to process each one one by one. And Python's not really that fast. So we try to reduce the amount of extra overhead we have that's not necessary. So we got rid of that and that fixed a lot of things. I think there's an example from Bluehost they said where the fan out, the schedules are working so hard just to keep up with the RPC broadcast they couldn't do anything else. So it was consuming all the CPU and nothing was happening. So we got rid of that. A really big one we've worked on over the past few releases is the database. We added indices, unique constraints, smarter queries, don't look for everything all the time if you don't need to and fewer queries, don't keep doing the same thing over and over and over again. Try to pass the data around when necessary.  And we've worked really hard on that. There's been a lot of people putting effort into that and you've come a long way. There's a few other examples. One example is a service group. So by default OpenStack services respond right to the database every 10 seconds saying hey I'm alive don't forget about me. And we need to know who's alive and who's not so we need that. But writing to the database every 10 seconds isn't great. An example is you have 10,000 nodes that's 1,000 database calls every second which for a silly overhead that seems like a lot of for just a small thing it seems like a lot of overhead for a small thing. So there's a few things we've done with an adjustable interval. So maybe you don't care about 10 second intervals we need to get five minute intervals. So that reduces the frequency. Or it could use things like memcache or zookeeper. Cause this data is mostly ephemeral anyway. You don't need long term records about how alive somebody was or how long ago they started listening. So you can use something like zookeeper and memcache to be is a better way to scale this. We don't test these out actually by default. So they hopefully work but they may not. That's probably something we need to test going forward. But this has been a big help for making things sort of scale a little easier. Some other examples is the workflow for moving around images or large pieces of data. Turns out that's slow to do inherently no matter how fat your pipe is it's gonna take too long. We're all impatient and nothing's ever fast enough. There's been a lot of work across all the projects to do this. Glance and Oven Center have all worked on how to move large data around more efficiently. Turns out we were pretty bad about in Diablo. Diablo is a very early release. It was pretty rough in many ways and we didn't really think really too closely about making sure we're doing things efficiently. Want to move data around as least as possible as a short answer. You don't want to copy around to four locations and mount it in four places to do one thing. Moving around once at most and leave it there. So we've done a lot better at that. I think we've seen some real results with that. Another big one is some, and this is in Keystone PKI tokens. So that back when in Diablo you had to talk to every time you had to validate a user, you had a round trip to Keystone before the user could talk to the project. So talking to Nova or Glance had to talk to Keystone round trip each time and Keystone had to validate you every time. That's a whole extra round trip. You shouldn't really have to do it. So this has been some great work recently to actually fix that. So now you actually have the token. You keep the token around long term and you pass the token and it's using PKI so you don't actually have to talk to Keystone every time. So Keystone needs to be this big choke point for the whole project in some ways. And now they've sort of removed themselves on that choke point that could do authentication and all that wonderful things without actually having to have a lot of traffic on a Keystone, which has been really great. So here's our cool thing we have. I work in HP so we actually have a public cloud that's actually done all this.  So he's actually run a public cloud on Diablo and Trunk. So he knows all about this too much. Yeah, too much, thanks Joe. So yeah, our current public cloud that's available is running a version of Diablo. We took the decision to default to fork essentially Diablo back in late 2011. And as you can see from some of the stats on the board, we made 1,477 separate patches to Diablo to some 40,000 lines of code. Joe's actually already talked about a lot of the things that we added in the previous couple of slides. We had to do a lot of work to get volumes working properly for us. Security group, the performance around security group rules was very troublesome so we had to do a lot of work around that. We had to do a lot of work to get windows, windows instances working and to get rescue working properly.  The, we found a bunch of security vulnerabilities around file injection, around maybe denial of service attacks, that kind of thing. Quota handling, validation at the API layer.  We also spent hundreds of hours analyzing the database, putting indexes in. There were no indexes shipped in the Diablo database so we put all those in. In Havana, it all just works. We get all of this stuff for free. We've got, what, a thousand lines of patches which we are in the process of upstreaming will maybe have five lines of patches permanently applied to our Havana code base. Now we're operating our Havana or actually Ice House based cloud in private beta at the moment. Looking to roll that out soon. So overall our experiences are like night and day between Diablo and Havana. Thank you. So you can see it was pretty bad at one point. So some other big changes made us at least forward facing internal changes that you guys hopefully don't know about too much because we don't really want the users to have to care about it too much. We split out projects. You guys probably know about that one.  It's hard to have one project with a thousand developers on it or one repo. It's really hard to do so we split them out, makes it a little easier. People can focus on what they know. So Cinder's a great example. Cinder was a second class citizen when it's Nova Volumes and now it's a great project. It's a lot of extra great features that were never there and Nova Volumes never would have existed. So you have people focusing on the individual components. Makes it a little easier to manage. We have APIs between them so we could change the internals of each project without affecting the other project, without affecting related open stack projects too much. We have this project called Oslo. So Oslo is a way to, we have something like nine projects right now that are integrated. That's a lot of code that's duplicated. A lot of things do very similar things. So we have an Oslo project which is to pull out all the common code and put it in one place so you don't have to have cut and paste everything everywhere, which never works. We have the place that we're still working on Oslo to always work in progress, where we haven't actually started using Oslo. We have nine copies of the same thing, slightly different and slightly different versions of working states and it's really painful to deal with so Oslo's been a great help for us. Getting rid of code, making it all look the same, less work for everybody, it's been really wonderful. Another thing we're always working on is this the road that's alive and continuous upgrades. So we don't want you guys to have to take your cloud down every time we have a new release. Another one is people want to actually always deploy open stack. They want to be continuously deploying it all the time. This part because upgrades are hard, so that simple answer is upgrade more, which makes no sense. The idea though is you upgrade smaller amounts and it's a little easier. You spread the burden out over a longer period of time. You don't have this one, you know, half million line patch you're applying to everything. You have lots of small things that you can understand a little better. So we want to support both of these and this has been a long work in progress. Companies are doing parts of these today, but I would think everybody says this is not easy today and we're going to try to make it better. We want to support this upstream stock, so it just works for you guys. And we're a lot to work on this. There's lots of sessions on it today. And this has been a big thing for the past two or three releases and it probably will be for the next few. Iterate on the APIs, another big one, this guy's thing, this guy's UC, this is something you see. The APIs were never really that great to start with. We're not always. We learn a lot from actually people having, people using them and us using them. So we have a lot of lessons to learn and we try to fix them, but we can't change the APIs because you guys are using them and we don't want to break it. So we had Iterate on the APIs, have new virgins coming out. Nova has a V3 API, Glantz had a V2. Maybe they're V3 now. Cinder had a V2 recently. Keystone had a V3 recently. So we're all really iterating on these APIs to make them better, add more features and make more sense, simplify things, and generally make the experience much better. So processes, this has been a really big one for us. Turns out we're the biggest open source Python project in the world that I know of. If anybody knows anything else bigger, I would love to hear. Define that number of users, lines of code, you have it, I think we're bigger. That's really scary, turns out. Python's not really what you think of. Let's write a large, massive thing to deploy a data center with 1,000 developers in Python. That sounds a terrible idea. It doesn't have some really great things you want, static analysis, all these other really cool things. You can pilot code to see what happens, get compile time checking. We don't really have any of that, so we have to work around that. We have people actually deploying trunk all the time, which is a terrible idea we would think, but we want to do it. So there's actually a lot of work we have to do to make sure that actually happens. So there's a lot of big processes that have evolved, especially from Diablo. Diablo, we had nothing. We had no gating. We had, I think under 100 developers. Everything was done by hand and we've really evolved the process to this really nice automated system that allows us to keep growing. And we're always working on it. So before we get into the process, some big principles we have from driving tenants. Never break trunk. There's a few reasons for this. People want to deploy off it, so always make it work for anybody pulling trunk. It should always work. It may not today, but this is a goal. And developers are never blocked. When you're working in a patch, new feature, you take trunk, work on your code and push it back up. But if the code's broken upstream, then you can't do anything. So we don't actually break all these 1,000 developers.  So we try to make that not happen. We actually try not to let any breaking patches in. As I mentioned before, we do a pretty good job, but we're always looking to do better. Transparency, we're open source. We're also an open ecosystem and you try to be very open about things. Anybody can see everything. We have to try to have no-back in conversations. We have a few big ways of conversation. Mailing list, IRC, the review process, all that is out of the open. Try to make it very easy for everybody to see what's going on. Automate everything. It's hard to do things something twice the right way, at least for me. So we automate it, makes it easier. Why have a person do something that a computer could do? Now, when your project is this big, a good example is some of the reviews. So we've taken, running the unit test, does the unit test work or not? You don't want to humans do that every time. Somebody's gonna forget. So why don't you have the computer do it for you? So that's where we have the gating. So our Garrett and our infrasystem will actually run all the tests for you.  It makes it really great. Write a piece of code. You hope it works. You push it upstream. You find out if it works or not. Makes life really easy.  Everything can be done for you. We also try to automate the review process as much as possible, which is hard to do. Make sure all the unit tests, so we make sure everything works. We also send some basic style guides. We actually enforce automatically because somebody's saying you missed a space somewhere is not what you want to do as a human. That seems like a big waste of time. The biggest limiting factor right now in open-sec development is the human being. So that the reviewer and the developer, should try to make things as efficient as possible for the reviewer and for the developer. Gallitarian. So anybody here could commit a patch if they want. Anybody could review anything, the whole process is out of the open. So you have a cool thing you want to do. You push it upstream, we'll review it. If you want to review somebody else's code and say that's a great idea or I think there's a bug there, you could review it. There's no, it's an open process. Anybody could do it. We want to keep it that way. We have no benevolent dictator for life and that's by design and we try to keep it that way. So we're very open at Gallitarian model. Be strict, so reduce burden on the reviewers. So one of the ways to make the scale is you actually push, be strict, somewhat arbitrarily sometimes, you usually not. But by being strict you can actually move the process from the reviewer to decide, is that right or is that right? So you say, we're gonna pick one or the other, that looks right and automate that as much as possible. As I said, the reviewer is a big limiting factor and this is actually something that a lot of talk about at this summit is how to make the process scale even further when we don't have enough reviewer. So part of the big answer obviously is more people reviewing the code but there's a lot of other things we could do. So back in Diablo, what did the process look like? There actually was no automated gate. It was all just sort of wide in the open. In the release, the actual, the month of the release in September of 2011, there's 87 developers on it. It's not bad. At that point you could actually all talk to each other. I think everybody probably knew everybody else on the project by name or by IRC handle or something like that. There's no gate, no get, no garret, garret or review process. We're using a bizarre and launch pad for everything. There's a very different world. People are fixing silly things. There was no automated anything. It didn't really scale well. So we've done a lot of work on making this process a lot more efficient. So where we are today, we have a lot more developers. This is the last release. Over the lifetime of the project it's been I think 1600 developers or so. So that's a huge number of people working on this. It's a little insane in some ways. So in the last, in one month long we had 346 developers. So we've really had to scale the development process up. The big part of that was breaking the projects down, automating the workflow here. We have this big integrated gate. So before we had no gate. So gate makes sure nothing breaks, integrated across nine separate projects and 30 odd services in those nine projects. We have one merge pipeline. The reason is because something in Nova may break something in Cinder or vice versa or Swift or Glance. So a patch, you have to make sure that none of the patches in the pipeline before you will actually break your codes. You can't actually merge things independently. We have one integration test that tests everything together and we test everything in the pipeline so we make sure we never sneak in a breaking bug. And this happens every so often when we have something that sneaks past the gate. We get ourselves in a strained wedge state. We can't undo it because the gate is broken. The fix won't break the gate. And we have ways around that. But it happens once or twice a release, so almost never. And it's always very quickly fixed. So running this kind of thing is a little crazy when you think about it. So we have nine projects, 30 odd services running in a large system in Python, all over the, cross hundreds or thousands of machines. And so debugging this is really hard. I'm not going to try to debug it. I'm sure it's painful for you guys. It's painful for us. Something doesn't work. Your patch doesn't work. That's pretty easy to figure out. If you have a race condition, something weird, spooky, in the dark, failing on you, you don't know what it is, fails 10% of the time, it's really hard to debug. So we've actually done a lot of work on making this possible and fixing this. There's actually a session upstairs about how to fix this right now. So we classify tempest test as not as passing and failing anymore. So it's not, does this test work or not? How often does it fail? And so this has been a really painful thing because the gate's not perfect. We can sneak past anything that fails a fraction of the time. And that happens a lot with nine projects, 1,000 developers, 30 services running all over many servers. So we actually have a problem. If something fails 0.5% of the time in any kind of call, that's terrible for us. That means we're running hundreds or hundreds of open stack clouds a day. And each runs running thousands of tests that we have now, 1,100. I think it is in tempest right now. So something that fails 0.05% of the time or 0.5% of the time is still really bad. We'll see that several times a day, if not several times a week. So we need to actually ration out the point where we can actually fix those bugs yet. But we're trying to work on making sure we can actually fix all these strange transient failures that we see. And we're getting better at it. Do you bugging this really hard? All the logs look a little differently right now. We're trying to fix that. You have to trace things across separate projects. I may not fail all the time, so reproducing the bug is really hard. Identifying the root cause is really hard. We had a bug recently. Where's a bug in HTTP lib 2, I believe it was, that was the result was we're across talking RPC and REST calls, so HTTP calls from the command line, and RPC, which is AMQP calls. And it was very hard to debug a Tuck. I think about a week of five or six guys working full time on it to get it out. We have very strange bugs these days, and it's really hard to catch them. So we're trying to do a much better job of actually doing that. Part of that is a project I worked called Elastic Recheck. So the background of this is we have, at one point, we put all the logs into Elastic Search, which is a great tool. We classify them. We could search on things. We have these great parameters we could search on. So we have all the logs for. We keep two weeks of logs around in Elastic Search. That means you could actually, if you find a query that fingerprints for a bug, we could actually see how often it occurs. That also means if you have your job fails and gate, something goes wrong in your check, we could actually say, is this a failure I know about or not? So we're actually doing that now. That's in running an open-sack interface today. And it's been really great to actually help people know what's going on with this thing. This broke. It's not your fault. This is a problem in the system. Just letting you know you don't have to figure it out. We'll figure it out separately. Now we actually have a list of all the bugs. Some of the bugs, how often they occur. We can actually really identify what's the critical point to fix and work way down these hard bugs to deal with. And this is probably going to be, I think, the big challenge at Ice House from a QA point of view is how to get rid of these transient bugs. It's a very asynchronous world out there in open-sack. Nine projects, 1.7 million lines of code, in Python no less. And it's really hard to deal with. And I think we're actually breaking ground on this, which is a little scary. We're very scary for me. Because I don't think anybody else is doing this. We're trying to do all kinds of crazy things. Let's talk about Bayesian filters for logs right now to identify anomalies in it. You name it, we're trying. And we need better ideas. This is going to be a really hard problem for us to deal with. And I think it's going to be a big push in Ice House. So however we're going to keep maturing, more testing. We never have enough. There is never enough testing. Testing is never aggressive enough. We're a strong believer. If it's not tested, it's broken. So that means anything you push in, it's not tested. Odds are actually probably broken. So we try to stick by that mantra and make sure that we do a better job. So every API to be tested extensively and then some.  I think it's still a release or two down the road to get it really working well. We believe this is a great feature. All the big clouds have been pestering us for years now, in fact, and have really been taking seriously for a long time. And there's a lot of big steps to do this.  The example is you're going to actually be able to upgrade compute nodes slowly over time and not all at once. So you're actually going to support upgrading in Ice House. Upgrade all of your code, your database, and then upgrade the compute nodes slowly. So you don't have to upgrade all 5,000 compute nodes at once or 1,000 or 100. We can actually let you do that slowly now. And we're working on getting a gate test for that in to make sure it doesn't break and we actually say it actually works. Improving quality while the project grows as a whole doesn't have been a really hard one. So I focus a lot on what was there in Diablo. But as the project grows, we want to maintain the quality while improving the quality, or improve the quality rather, over the whole system as it grows. And that's been really hard for us. Part of that is the QA team and the infra teams aren't that big, and they need to grow. And so if you have developers out there who want to work on it, we would love them. But keeping your project so big, running with all these separate components and the integrated gate and integrated release has been hard for us. And we're doing a great job of it now, I think, but we're always looking to do better. I think any bug after a stable release is too many. And I think we all agree on that. So we're actually trying to make sure this never happens and get better and better at making sure this all works. And scalability. I think the biggest cloud today is Bluehost at 20,000 nodes. This is on Falsum, actually, which nobody thought was possible, at least I didn't. This is one cloud, one AZ, one deployment, no cells. Then I do a bunch of changes to actually get that to work. And we're actually working with them to learn what they did and how to fix it, along with all the other public clouds to really fix it and make things really scalable to be on it, you can even imagine. So this is another big challenge we're working on. That's the big part is that performance, not doing silly things, making sure all the pieces scale out, all that. Overall, it's been a lot, you know, Diablo vs. Advent is a lot changed as you saw. We have a lot more to go, I think, and we've done a great job, I think, of making it something actually deployable. Thank you. Any questions? Thank you.


Good morning and a very warm welcome to the sixth meeting of the Constitution, Europe, Excellency Affairs and Culture Committee in 2023. Our first agenda item is the impact of the digital first agenda and its evidence relation to the impact of the BBC's digital first agenda on Radio Scotland's planned schedule changes to music programmes. We're joined this morning by Professor Tommy Smith OBE, Artistic Director of the Scottish National Jazz Orchestra and Head of Jazz at the Royal Conservatoire of Scotland, Billy McDonald, Director of Piping, the National Piping Centre and Professor Simon MacKeryl, Professor of Media and Music at Glasgow Caledonian University. A warm welcome to you all in a packet to have an opening question in 2016, when the BBC Royal Charter and Framework Agreement was reviewed, if I could quote from Fiona Hyslop on some of the objectives at that time. She stated in the chamber that they expected that the structures were expected to deliver better outcomes for audiences and implement commissioning and editorial practices that will support the growth and sustainability of Scotland's creative industries. I would just like to have your reflection on the plan changes and what impact that will have on the growth and sustainability of your particular areas of interest. I could perhaps invite Professor Smith to open. Hi, thanks for inviting us along this morning. I think that reflecting on all of this since that date, Tread and classical and jazz has definitely gone on a trajectory which is going up and it's incredible time for our nation but I think the appalling decision that the BBC didn't seek our external artistic expertise from the vast community of professionals before they didn't actually announce that they were axing these three genres, it was leaked and I think that that process and accountability to the public was flawed and draconian but I think Simon is probably better to answer that question because he's more of a a guru when it comes to the marketing and that initiative. John, the impact I think on the three niche genres, it's devastating. I think that it probably falls into three areas to me really. The first one is the impact on pipeline of talent so the actual talent pipeline in Scotland so these guys here, obviously back in many moons ago, I was also played on the radio when I was an actual musician but I'm not a professional musician but for our professional musicians the pipeline of talent is now simply going to disappear if the BBC is rescinding on broadcasting live sessions for jazz traditional piping and for classical musicians so that's one area of impact. I cannot see how moving or proxying any of these services down to England would have any useful impact for the development of professional Scottish talent. I think the second area of impact is slightly more difficult to put your finger on but there has been this tacit agreement that the BBC gives is essentially acting as a proxy for the national record or the national archive of live performance traditional music jazz and classical music music in Scotland as opposed to Scottish music but they have acted in that way for more than 70 years in terms of linear radio and now obviously on digital platforms as well and so that record that we've got is now substantially under threat and it's never happened before in the entire history of broadcast public service broadcasting in Scotland because I mean if you even way back to the 50s and the 60s and 70s BBC Radio Scotland were broadcasting all of the contemporary dance bands musicians like classical and to some extent at that stage even then they were broadcasting piping so there's an area of impact there which is that they they're essentially saying we are not going to service this record for the nation so there's a cultural value in that which just disappears overnight and I think that's a really significant public service impact and I think also then the third year which really concerns me as well is about non-professional and community music essentially so you've got the impact on the pipeline for professional music making you've got the impact on the actual living traditions in Scotland but you've also got this impact on kids and particularly I think poorer kids who you know we know there's an unmet demand for music education in Scotland and we know that for many of the kids all around Scotland they essentially the only way in which they can contact the wider community of piping and drumming or jazz or or early development for classical music is via radio and whether that's linear or downloadable or whatever that service about putting them in touch with the rest of their community of practice is about is really under threat here so I see it in those terms in terms of those those three areas particularly thank you and mr mcdonalds Simon's actually put that very well I think and put across a lot of the concerns that we have if I can speak to piping specifically piping is now more popular than it's ever been it's been taught more widely across Scotland and not just Scotland across the world and the pipeline show in particular has been absolutely at the heart of that growth over the last 60 years and just backing up Simon's point of the the benefits of the the career trajectory of every piper I can think of who's playing at the top level has developed through for example on live broadcasts which especially now when creating music commercially is becoming so difficult for the traditional arts the bbc contribution in terms of their live sessions for pipers and any musician really is invaluable for pipers careers and I think backing up the the point about the archive of material as a nation we should be absolutely proud and you know putting that archive out there to the world you've got 60 years of recording of the best pipe music of our national instrument and if there's any chance of that stopping then I think we have to do all we can because you know if you think 100 years ahead in terms of this archive of the best recordings we have of the best musicians did it stop in 2023 did people stop playing this music because who's who's then going to take on that responsibility of noting that tradition and keeping that alive and I think it's worth pointing out it's not just about looking back in terms of tradition the only way we develop within our tradition is to have that that knowledge that we have and the the kind of the root of the tree which we then grow from and I think if you look at piping traditional music jazz now we're absolutely on that it's an amazing country to make music in but we have to make sure we support that and keep it going thank you thank you thank you very much thanks for joining us this morning and for those initial comments I'd like to ask about jazz nights particularly yourself Tommy can you give examples of how that programme has helped to support new emerging artists in Scotland sure for decades there's been a programme ever since I was 15 there's been a programme it's been under different names take the jazz stream be up to hip hop jazz house jazz nights still the same two overslot in the dead zone on a sunday which has limited audience nine o'clock and everybody's in their bed preparing for work except the jazz musicians of course but since 2015 about 60% of the artists that appear on that show are Scottish 40% are English or overseas and of that 60% 34% are established musicians the rest are emerging artists who use it as a stepping stone for their career because the first thing you need to do when you're a young musician and you practice the 10 000 hours to get to point a is to become famous in your own town and then famous in your own country and then you go across the border if that is not there you're not going to be able to do it you can't just jump to London and and and become famous you have to start locally and then nationally then globally there are some people who do not go national there's some people who don't go global but what's important is that the people who stay here the indigenous people who work here and don't go across any any boundaries because they're maybe at this level they need to also have a resource to play their their new albums or promote their their new tours because every artist doesn't just make one record and then they're established they make one at least every year i've made 30 records maybe too many but there's more people like me and they need to have a place for their music to be heard by the public who are eager to hear and not miss any new emerging artists or new collaborations from artists at the respect maybe between genres or tours that are promoted they don't want to miss that and two hours a week is really nothing compared to a lot of european countries who have 24 hour channels or even four or five jazz shows a week we're really at the bottom of europe when it comes to this and to eradicate it completely is is completely nonsense for me because i've toured the world and i've always seen scotland at the bottom of everything when it comes to our genre and we're really been working hard to work at the infrastructure having one jazz school compared to norway six we're starting out we've got a lot of fantastic musicians that are winning awards nationally against great competition from england wheels in in ireland and you know music you don't you don't need money to make good music you need passion you need heart but you do need a door that's there to be able to open up so you can really get on that that stage and um i mean i don't know if that's answered some of your question that's that's great thanks i mean i was going to go on to ask you then about the bbc's response um where they've said well you know we've got jazz programmes on radio two uh radio three uh we can incorporate scottish emerging talent into those programmes i did i did have a chance to go back and look at the last month of output um that's currently there on bbc's sounds the jazz programmes i looked at all the track listings i didn't see ffergus mcredi or georgias acil or i don't think any scottish artists in any of those so i i'm just wondering like is there is something problematic about the formats of those programs on on radio two and radio three that makes it hard to reflect that that kind of ecosystem of scottish talent that you're talking about it's it's london centric there are more musicians down there than up here there's more people down there than up here and the the programmes on radio two you with jamie cullum i mean it's not hardcore jazz jazz has many genres within itself you've got noralees and dixieland and swing and bop and hard bop and modal and fusion and then you've got electric and free and avant garde and it goes on and on and you've got the the amalgamation of of local scottish music in jazz and like ffergus does and a lot of his contemporaries do but radio three you've got jazz record requests which is mostly dead music but you have to be requested so you might have to call up and pretend you're someone else and request your own record that's absurd but maybe the people who have good are good at various accents could probably get away with it anyway so yet you've got uh uh free niss which is more of an improvisational avant garde show which is again another niche thing in there and they rarely play uh scottish artists unlike jazz knights they've played 40 percent over since 2015 of english and welsh and irish and overseas artists they don't kind of do it um you'd have to go back to january where ffergus and george the cecile were represented on those shows and over two weeks say 265 hours of listening you would have had one jazz track a day of say three to four minutes it's not good enough but that that's the situation there and and the the uh the forward motion the bbc are are are presenting to us to say well we can move it move the listeners to those shows or we can put some on the afternoon show the janis foresight show and i had a i had a thought about that and i think if if jazz was say uh a sweet raisin end of the in the afternoon show the janis foresight show will say a ball of porridge with one little sweet raisin in the porridge the listeners who like jazz would have to eat the whole ball of porridge to get that raisin but unfortunately uh if janis foresight was to make the porridge it wouldn't be a sweet raisin it'd be a sour plum because she's on records saying that jazz is an earth cup of tea and but she does like some singers um so it's it's kind of if you're if you're leaving it to a presenter to represent your genre you'd have to get the right presenter who's really open-minded and doesn't have you know that kind of vocal um well everybody's got an opinion and they're entitled to it but yeah absolutely um can i just ask you then and i guess there's a question for for all of you as well but i'll go back to you yourself first tommy i mean it with jazz nights i i stumbled across it um i i'm not usually awake at that dead zone on a sunday night you know um and i'm glad that i did but is there is there more that we can do to promote that linear content um through bc sounds digitally because i think there's a lack of awareness about the three programmes that are that are out for um for cuts at the moment and i think that that's a shame because you know we're we're missing something if we don't know that these programmes exist and i think many of us are on a musical journey as well trying to trying to learn about new genres and and you know trying to wake ourselves up to new talent and it's often you know difficult to find these programmes it's not obvious you don't stumble across them that easily i think i think jazz nights on the other shows they provide a a very valuable educational and and entertaining source of um discovery for existing jazz fans and new jazz fans and in the um in the age of streaming where um if you were to compare it to like visiting the uh the world's biggest all-you-can-eat restaurant um these shows are like having like one of the best chefs come and sit beside you and guide you through what is best to eat um you're not just there going well what do i do there's so much what i'll listen to um so those shows really are there to educate even even on the jazz night show they have an educational aspect to it which is great which Michael does that it's done it for for many years um because jazz is difficult to get into but when you're into it you're into it for life not just for five minutes if you don't have a music musical education as i've done so yeah and just on the consumption point i mean the BBC faces a massive shift in consumption habits right so everybody and they've set this out already publicly with their digital first strategy and it's understandable you know they face a huge challenge from music streaming services which they can't compete with like Spotify and apple music and so on um but i think i think the question here is really that if the argument is about um what consumption is changing that is not an argument about um high quality production you know and the platformisation of these programmes so jazz nights pipeline and classics unwrapped niche musical programmes all of these programmes could be platformised if you like you know they could be and in fact they are on on BBC sounds in some respects but the issue is about consumption i think is a is a sort of tangential argument because if the consumption is changing and we know that 90 percent of adults listen to live radio every week but we also know that you know there's been a huge growth in music streaming so the BBC is in a really difficult position because they are seeing you know Spotify come in apple music come in and basically their lunch on terms of music streaming but none of those commercial services have a public service remit so none of those commercial services at all so there's been a so for instance in 2017 music streaming was eight percent of audience time and in 2022 it's now 20 of audience time but that shift that involves these global music streaming companies doesn't actually touch the question of the public service element at all you know i think it's a it's a tangential kind of argument to me so the argument that the BBC is somehow in competition with these services i think is also problematic you know they have a totally different remit and i i feel like this this argument that you know we need to produce more and more talk and talk content for the digital platformisation of the BBC really just it doesn't speak to this argument about the public service remit of these programmes because i can tell you right now Spotify is not going to make pipeline you know or or jazz nights you know or or indeed classic sun wraps necessarily and so i think that's that that there's that question about well consumption is changing but it doesn't really touch this idea about the public value the cultural value of the production of these programmes you know for niche music things that feeling yeah i totally agree and i think i think it is about that the kind of cultural responsibility of nurturing other national music you know so i think timings yes you could look at when these programmes are on but as you say the the more accessible platforms make it accessible it's about what's on that i think is the most important issue and i think you know a dj type radio podcast you know with mostly chat and a few bits of music is nowhere close coming to live sessions from musicians outside broadcasts that cost money and are not necessarily commercially viable but that's the that's the cultural responsibility i think that the BBC have thank you very much thank you dr all thank you a great deal sympathy with what you're all saying and my point was anticipated by professor smith when he talked about european countries how they do this in our core in norway has is i think you've possibly alluded to or another country i don't know several national radio stations for different types of music you can listen at any time of the day to jazz music or classical music or folk music as well as a news channel and all the rest of that i suppose my question is is it is it realistic for pbc scotland to try and shove all this stuff into essentially one frequency and why why are we why are we not looking to have more frequencies with more diversity of music rather than trying to save a couple of hours a week that's exactly what i wrote to them you can't be cutting this you got to be increasing it because we've got the the capacity at the moment i mean i'm my job is saxophone player that's my job sorry 15 years old and the bbc made me i'm here because of the bbc i was on the radio show when i was a kid and it's their fault that i'm sitting here nobody else's but when it comes to excuse me i lost my thought here it's too early in the morning for me multiple frequencies and all that yeah each each year i audition 75 young people to take six seven places at the royal conservatory of scotland the rest go to england or further afield we have more musicians here that meets the eye and the more musicians who become famous and more inspiring who reside here and who are indigenous here they inspire more and it has a rolling effect um you have to these musicians need somewhere to play to present their music in glasgo that's where the scotland's only first jazz school began and there wasn't really a deep scene there now every night you can go on here jazz because the young musicians have created that scene Monday to Sunday sometimes three times three concerts on a Wednesday night um if if the school was in edinburgh edinburgh would have a scene it would grow be exponential um and because we've got artists who are winning all the awards it's going to inspire and the inspiration is going to create like a trajectory which you can't even imagine now many years from now when we're all dead what's going to be left is it going to be ratings is it going to be how much of the discuss is going to be what that person said no it's going to be art and music and poetry and stories and that's what's going to be left so we need to fight for it because we're fighting for the people who we're not playing now we're fighting for the people who haven't even picked up an instrument does anyone else want to come in on that point so no i think again i mentioned this earlier that i think it's we would welcome more you know more diversity and more places to put that music but it's about that support for the creation of it in the first place and the ability to create that that these programmes are provided just to make a point on that you know on on the linear listening i turning on a radio set and listening to the radio live that is dwindling you know that that is disappearing and i mean the bbc have published that stuff themselves about that potentially channels are getting turned off more and more of your both commercial and broadcast ones but in a sense you know that that's not really in a sense at stake they are saying that's at stake you know for the production of these programmes the problem here is we've got a very high production cost for these programmes because they do live recordings they do live sessions and i think you know the idea is you know if you have a platform if it's bbc sounds or whatever all of these programmes you know they are mostly 70 80 music often say for pipeline there would be 12 sessions a year of live invited people into to perform and i think also for jazz nights you know i don't know how many people are going in a year but the point is that costs a lot of money you need the studio you need the technicians you need to you need to pay people to come and perform and to be able to broadcast it so the actual argument to me about platformisation of the material isn't really the key point the key point is actually about the production rather than the consumption and you can chop it up any which way you want but if you're not producing that kind of original content on bbc scotland then who is you know who's gonna who's gonna do that for for you know for the national cultural heritage and i think also if you think let's say someone does manage to get the finance together to make these recordings where's the outlet then you know because it's these co-programs that support our traditional music and jazz and classical music that put that out there the figures that told me it said about the the amount of scottish players on on radio too in the jazz program that's minute and so that there's multi layers to that so it's the creation of the music but let's say we can get funding to create that music which is very expensive who's going to play it on air on these mainstream programs i disagree with you about the very expensive part i don't think it costs that much money in the scheme of things you know with the bbc budget which is public money it's not itv so you know itv can question their ratings and all that stuff but the bbc have uh you know over 600 million not all of it comes back to scotland as you know but it's not i would i would say it's not hugely expensive on the scheme of things even though uh a soul pipe or a soul jazz musician would find it expensive to do themselves the bbc funds from all the various pots can easily afford this so why did they choose to completely eradicate these three genres without skimming some of the other programs which cost one thousand times more than these little programs that we're talking about i mean i just don't see where the ideology is coming from and actually sorry just just to see it for the diaspora point you know things like pipeline is huge overseas you know and i think at one point was the most downloaded program for bbc radio scotland at one point but of course you can't get those figures from the bbc because they're they won't i mean i've tried fois to the bbc they won't give you those figures so the the diaspora point is very important for you know other aspects of Scottish soft power overseas and also tourism and so on like that you know there's many many people who listen to those pro group pipeline for instance that will come you know come to Glasgow for piping tuition come for a holiday and spend an awful lot of money here the world championships brings in a lot of money piping live and so on like that so there's there is a real economic impact as well i know i mean it oversees colleagues pipers and drummers in the states canada, new zeal and australia, britany across europe they are astounded at the idea that the national broadcaster in scotland is about to essentially cut the feet off of live music sessions and outside broadcasts for the national instrument my other question was the bbc in scotland has a fantastic resource in the form of the bbc scottish symphony orchestra is radio scotland making any significant use of this resource is there is a place being given or should more of a place be given to the output that they have i paid with my own money to hire them for three days and they sounded pretty good to me i think the bbc should use them that that's you know we have to apologise for nicola benedeti not being here but she would be the one to really respond to that question she's busy but on on her behalf absolutely it should be utilised they're they're an orchestra that are contracted they're employees so they have a lot of rights musicians usually don't have rights they're all self-employed they it's a very fragile ecosystem you're as good as you are on your next concert you can't rest on your laurels of a record you've done or a concert that got a good review you have to practice diligently and your career has its ups and downs you need every help you can get you know and especially after the pandemic and then the lack of confidence in the audience it's a real tough tough game for anyone thinking about picking up an instrument if you think about all those parents out there they're going to go hmm i don't think you should focus on the trombone johnny because there's no future in it maybe study engineering or or something that can really help the hard economy instead of this word that we keep on hearing from the government soft economy you know it's it's it's it's there the bbc if they wanted to they could have this it's their decision i know nobody can tell them what to do but it's their decision it's the same with when we wanted the jazz course in scotland uh no one could tell any university you have to do this they had to decide themselves and it took people with the right ideology and the right jobs who weren't just accountants um they made that that decision and it's a great decision but we can have more of everything more schools more shows we're definitely got more musicians because of the inspiration that we've had from all the great musicians before us can i start a point in there about you know this this european context as well you know i have had a look internationally across europe at where things and how things are funded in terms of traditional music particularly and every single country we are an outlier in in this continent by not having any state funded national sound archive there is virtually no other country and the reason for that and is as well known by academics for years decades has been that the bbc has by tacit agreement been the national sound archive and i think what we're at this point now is is saying well that tacit agreement that the bbc is going to record the national audio archive essentially is now being broken in scotland where we know that audiences are less happy than in other parts of the united kingdom with the bbc if that's now then going to be broken then it will open up the question about how do we then how do we do what other countries do if they're not doing it how do we do it like iceland finland spain latvia belgium ireland sweden hungary and to sound extent even wales in the uk all have state funded national sound archives and and some some cases have live radio stations as well so it opens up a big question there about well if they're not doing it then how are we going to do it and you know how are we going to pay for it thank you thank you miss boyack thanks very much convener um listen to the evidence and actually having read submissions it does feel like this decision kind of cuts right across the bbc charter four of the five key principles of bbc charter it feels like it cuts it it's the opposite direction of bbc across uk which is meant to be one of the core principles of operation and when you look through the annual plan again it just does not deliver for those two high level ambitions so do you think this is just not being thought through because if you don't have production on the ground in communities in scotland then it's not about which radio program it's on it's just that has gone so have they just not thought through the detail on this and you know is there any sense that there's an acknowledgement that this doesn't deliver what it's meant to deliver as a public broadcasting company decision was taken yeah well i think that the uh the decision i mean as far as i'm concerned it didn't feel robust um i'm director of the scotland national jazz orchestra and if i want to for example buy a microphone for the orchestra i have to go to my board and ask their permission it is just a good idea but it didn't seem to be transparent that there were minutes or a proper systematic process to make the decision to cut these three to cut these three genres without telling anybody or asking anybody's advice um and when we were there we we went uh with uh Alison Thullis um Donald Shaw and Colin Currie the classical musician we questioned them uh from every angle and we felt that we we that they hadn't thought it through and that we were there to actually give them advice and we weren't there to give them advice on how to go forward because first of all i didn't want to be complicit in their cuts i wanted to know how they got to that decision and it seemed to be a kind of smoke and mirrors past the buck there was no um a real systematic approach and and there weren't any minutes of that meeting either that we were at which was an important meeting nobody was taken out so if that's the way the BBC do things um i don't think it's uh uh a flawless uh against systematic approach it's got it's got to be tighter than that for public accountability it's flawed i agree i mean the the thought of replacing live sessions and outside broadcast with you know essentially a dj program doesn't fit that remit in any way i mean the other thing um BBC very big on digital and BBC sounds which i'm totally up for but you still need the productions to actually put on BBC sounds so it feels like a a lack of thinking it through and um in terms of the impact on the core areas you've talked about education you've talked about accessibility there's these three areas of um Scottish culture pipe classical jazz without this there doesn't feel like there's a pipe stream through in terms of live performances and that that's a critical disconnect so it we're obviously speaking to the BBC next week but i think it's useful to get that reinforcement that without this pipe stream there's a major impact on those three areas and has that really been acknowledged at all no i mean it feels very odd the BBC now when when you think about the process itself for decision making when when we've when we spoke to them um the conversation felt to me like it was about audience metrics audience reach audience metrics and that only answers a small portion of the question about how do you how do you understand the charter you know you cannot understand the charter solely through audience reach downloads you know and metrics for and so on geographical or or sheer numbers metrics there has to be some way in the decision process about understanding how you're meeting the public service remit and and the nations and regions remit you know and i think you know if you look across the rest of Europe we are very unusual in that there are no external experts as far as i can tell in any of these committee decisions at the BBC and so you know we've made a bit of a stink about this and we've had a campaign which which has been great and lots of signatures of people have signed up to those petitions but i think we're at a juncture really where the BBC has to acknowledge the fact that the process itself is totally oblique take it everybody agrees with that i think consultation is the key there and certainly i didn't know of any consultation in advance of the proposed cuts that's really helpful thanks you have to look at the the charter that you were mentioning specific points and i looked at it the other day and i picked out a couple of things which are still if if we do go forward with their plans you're going to miss key points like to show the most creative and highest quality and distinctive output services in many genres that take creative risks you're not taking risks if you put on something that has three or four notes on an afternoon show in the middle of a bowl of porridge it's not going to happen it's for me it's it's i don't know if you've ever toured america and stayed in the harbour johnson's hotel but they it's just the BBC is like the harbour johnson hotel they've got 36 flavours of ice cream but one flavour of food that's why i see it's coming on so all becoming homogeneous and you know they're supposed to reflect and represent the entire diverse community of all the united kingdom nations and regions and they're supposed to support the creative economy across the uk and raise awareness of the different cultures and they're supposed to commission and deliver the output and invest in the creative economies they're supposed to do this at the most creative and highest point that means they've got to use all these amazing musicians that are here they're not going to do that they're going to pick some easy listening record that will appeal to the mass and put it on the afternoon show and if you're lucky in in your your i mean the jazz people are not going to listen to a show where they're only going to sit through two hours in here five minutes and it's not going to happen yep that's really helpful finlayd did you want to come in thank you that's good thank you miss mental thank you convener and thank you very much um it's been incredibly informative and um i have to lay my cards on the table i'm an ex bbc scotland employee um and i have to admit that it was steven duffy that introduced me to jazz um it was the bbc scotland symfony orchestra that introduced me to classical music and um i would say the same for piping um with regards to the world band world piping championships so thank you very much for coming along um my question is um i was going to ask about the archive but professor mcaryll i think you've cleared that very very well um there is something about a continuity of archive uh a knowledge as to what's going on um the ability then to make other programmes from that is is very important um slightly been touched upon um clearly the um the content of the programmes are really important but i'm also interested to know your thoughts about how the the backroom staff the experts in the difficult in the different musical genres um how how you feel that could be a loss to um Scottish culture widely and also ensuring that the children can learn musical instruments in different genres as well i don't know um professor mcaryll do you want to start um well yeah i mean i think you know from where i set the bbc has got well since the 1920s radio broadcasting developed a really admirable expertise in backroom staff and and expertise there if you read some of the research that's been done in 1940s 50s and 60s on music programming in scotland at that time the bbc employed directly musicians we still employ musicians directly obviously via the the symphony um Scottish Symphony Orchestra but the producers or like entertainment producers and so on were extremely knowledgeable and and experts in the musical field and that had a huge influence in the 50s and 60s and even into the 70s on all sorts of different genres such as how they presented like classical how they presented um dance bands for instance Scottish dance bands you know how they presented and actually even piping as well you know and other areas and i think um my sense is that the bbc still has staff um who who are knowledgeable but nowhere near as many staff as obviously used to have back in days when it had a more substantive budget and so on in relation to to programming so i think i think it's this move is really difficult it must be very difficult for those people who are working at the bbc who are committed to these programmes and one of the reasons that we're all here together and that we went to meet them together is because i think it's essential that this is not about classical piping or jazz separately this is all together they're actually done in the same department you know for for niche musical programming in the bbc and there's great expertise in there and as not least as well also the presenters of those programmes are not direct employees but come in and so i think it must be incredibly difficult for them but it's also indicative of how the bbc operates in the in the commercial digital age that actually there aren't that um there isn't that you know critical mass essentially of highly knowledgeable highly skilled music i mean for instance i'm not aware that any of the commissioners are musicians themselves i don't know whether that that that's true or not or or indeed how musical expertise from the community outside the bbc or anything plays into the commissioning process itself because it's it's it's totally um untransparent how that process works thank you for that um just i was doing a bit of reading as well and the operating licence for bbc scotland says that it must provide content and music of particular relevance to scotland and i think there's something really um i'd like to hear your thoughts on the the difference of hearing uh somebody that is of scotland um introducing programmes and what what that adds perhaps to the content if i could mention that in the pipeline specific topic um you know gary west who's the current presenter and um ian mckinnis before him and and going right back to people like jaws macklewham they're fonts of knowledge on this and it's not as easy as just reading a sleeve note of an album and then seeing what that says there's there's decades of research and knowledge that goes into presenting a show like that and i think again if you read a sleeve note of the back of an album that's one thing but people who have that innate knowledge and understanding of the music actually presenting it makes a whole difference to the whole show and that educational side of it you'll you'll get information and nuggets of information about the music that you can't get in any other way and there's not the albums yeah well exactly and albums are so difficult to produce now that it's um it's not an option i think that's one of the key things about specifically for pipeline the idea that they will be able to move to a DJing show pipe you know and we met with the BBC and they said well piping's still going to be there in a saturday night same slot same show it it's it's a it's not you know if you remove the budget for those 12 live recorded sessions a year plus the the outside broadcasts at piping live and the master's competition the glenn phyllisol with piping championships and so on if that is cut back to the extent that it's essentially a disc jockeying show then it isn't performing the same function you know and the other and as finlay says there isn't this catalogue commercial catalogue that you can lean on for that show which is why the programme has always recorded local musicians across scotland from all over scotland and internationally as well when they visit um so that that whole function just just disappears you know and i think that's that's really a key aspect of this you know if we're saying that it's still going to be there well it's actually it's a bit like saying a head teacher in a school saying do you know what we're actually going to take the library away from the school there'll be no books left in the school but it's okay because you'll be able to go online and download five books a year it's something like it's akin to that okay thank you tommy anything to add no i don't no improvisation then i can do that mr camren thank you very much convenient thank you to the panel a lot of this has been covered already but i just specifically wanted to ask firstly about pipeline and piping and you've been very eloquent mr mcdonald about not just the educational purpose behind it but also it's been referenced to the international reach of the programme i'd like to bring kind of local perspective because representing the highlands and islands there's a lot of interest and in fact a lot of concern about it disappearing and i know that i think six thousand more than six thousand signatories have backed the petition in support of it and i'm sorry but very thank you for the correction ten thousand but i think that that just shows and i suppose i just want your observations on on the kind of local importance of programmes such as that yeah absolutely i mean it's if you look at the the performances that are recorded and put out on pipeline for a start you know you've got people like angus mccall from benderlock we're very much local it's not as if it's just a you know recording in glasgo all the time we the bbc have been brilliant to be honest that going out and recording in different communities and recording people from those communities and i think that's again one of the things for just so desperate not to lose is that fact that you'll hear someone playing from benderlock who plays in a slightly different style than someone from glasgo or from the east coast or and that's that's represented fully on all of these recordings and again if you look at the world pipe band championships for example you know bands from all over scotland all over the world competing and being being broadcast so i think it at the moment it does it does look at that it does deal with the local elements of the music and they'll interview you know the the sessions where the pipers are interviewed they talk about their upbringing and music their their journey through music and whether they be from lewis or sky or woven they talk about their their own area and that's when you know when young people hear that that's i did it myself i remember sitting you know on my grand's table on a sunday i used to be with my dad and my whole family listening to pipeline and hearing the music and listen to these people talk about their lives and music and that was the inspiration for me and i think it's so important to capture that moving on to to young musicians because i think this is is really important um i think it's well known that many established musicians credit these standalone shows as integral to their success and i like all the panel to comment on that you've covered a lot you've covered this a bit already that's the first point and the second point is to the bbc have said and i think we'll say next week that basically their their defence to this is that they will cover all this in other programmes um and i think it's important to i mean it's not my view but it's it's what they will say and um it would be useful just to get you but all on record as as to as to that and what your response is to that so if i could start with finlay please i can speak about my own experience i still remember my first-ever pipeline recording you know going into the queen market drive to do that that was a that's a huge moment in any young piper or any young musicians career being asked to go into that but i think it's also that um you know when you're in a group of like-minded people for me it was a pipe band you know that was the talking point did you hear who was on pipeline this week did you hear whatever performance and it creates that feeling of community amongst the young musicians and you know it's something cool that they all have that maybe their other friends don't have that don't play music it's that it's that way in to kind of to make it their own and you know so there's there's lots of different levels of it and you know it was almost like a you'll remember yourself Simon your first being invited to play on that show for the first time was like a huge moment in your in your musical life you know so it's hugely important for for youth development and creating that community of young people having people inside the BBC who are well connected into those musical communities will disappear and therefore how will they be able to know who the really fantastic young saxophonist in forest is or how will they know who the next best piano player from the hebrides is going to be and i'll invite them i cannot see a producer in Manchester or London being well connected into the regions of scotland to know what's happening in communities around all these areas so it's absolutely vital i think i would also want to just say you know in terms of the linguistic and cultural diversity point about croon lua and the gallic the gallic broadcasting elements of this i mean you know one of the reasons why there is a separate gallic language piping programme is because it serves quite a different audience in scotland you know it serves an audience that are first language speakers of gallic and so therefore it's it's different if you're talking for them they're they're talking about the very very rich heritage of piping and traditional music and song across across all the gallic speaking areas of scotland but then to perhaps you know in some cost cutting exercise to then have you know the main programme translated into gallic or whatever you know that doesn't serve the gallic speaking audience at all you know and i think your point about the local regional differences here is absolutely key for any any of these genres if if they don't know who's out there particularly for the kids you know coming through then they won't be able to hear them and those communities will not hear themselves on our national broadcaster. Professor Smith. It's quite ironic hearing the the differences between the politicians and the commissioners at the BBC because it sounds like the politicians are more aware people like Pete Wishart who was a musician and Alison Thewlis who's vice-chair of the app jag in the parliament and even Miss Minto who's got experience through Stephen Duffie in the various other shows you seem to be very clued in at the heart of the the matter but you know if you're if you're a musician in scotland you're going to be lucky now to be on those shows maybe two or three times every five years that first time like my colleagues have said is the most important time that very first time when you're young because it's so inspiring for you to get on that ladder which is precarious very precarious and you have to do so much hard work to get there if that's not there i don't know where the door is and i've heard that the the youth competitions for trad and potentially for jazz is the avenue the BBC are going to focus on for the emergent artists but you're only going to get one winner and where are the where's the music going to be played what show perhaps if if it's elevated enough that one single musician will be played down south because it's going to be in the news but what about the rest of them there are so many musicians of scotland we're lucky to be on the shows two to three times every five years as it is already so the future is bleak say the least thank you for those answers thank you convener mr golden thanks convener i think we've covered extensively the kind of programming side of things on on radio and that's really helpful for when we can ask the BBC i wonder if there's anything else out with that sphere that you would like to ask the BBC in terms of supporting jazz piping and classical music i was quite interested that professor mcernnell mentioned for example musical experience at a commissioner level and i wonder if there's anything whether it be you know on you know tv or or visual digital formats or in terms of structuring of how these programmes are commissioned that you would like to add so we can then ask the BBC about that well i mean for me i think one of the one of the really interesting aspects is what metrics are they using to measure their delivery because it seems to me that the only metric that they seem interested in at present is you know the stuff that's easily be measurable which is downloads which is audience figures and you can get that stuff you know through the joint radio joint audience research Rajar and the BBC media nation these reports all deal with that stuff but the problem with that those metrics is that they don't really deliver on the public service remit they can tell you you know how are you doing in competition with the commercial broadcasters you know how are you doing and of course the things that are always top of the list for the for commercial broadcasters whether it's on podcasting or in other forms of radio consumption are talk so true crime things like that you know drama and so on so this this this question about really niche musical areas is never going to be covered by the commercial broadcaster there's there's it's too expensive and there's not enough audience figures in it so it really is a question about the public good to me so i think what i would like to see from the BBC is first of all much more transparency about how those those decisions are reached to me they should be publishing on an annual basis you know by programme those metrics but also a conversation i think about how are you actually delivering on the public service in the charter remit as well going beyond simply the downloadable figures and the other thing i would say is it's much harder to measure the linear than it is to measure the digital so it's much easier if somebody signs in and downloads a programme or whatever you can capture the digital and the platform based stuff much easier than you can capture somebody listening in their car in the A9 do you see what i mean so i think for me it's about that question of about how are you actually metricising the charter commitments that's really useful would any of the other panellists like to contribute i think just the consultation and the transparency thing as well you know i don't want it to seem as if personally i'm BBC bashing here but if there had been consultation and transparency then we could have helped you know we we there's a whole industry of musicians that want to see this continue and succeed you know so i think the the transparency and consultation process would be very welcomed that's really useful between the three genres there's over 22,322 signatures and if we took all their license fees and we could make these free shows no problem absolutely although they would not be allowed to watch strictly or any of the other London centric programmes but you know the intention of the the license fee it's there to allow the BBC to make shows which have smaller ratings they it's not there to constantly please the majority like itv it's it's a different and it's a tax i mean the ons classifies it as a tax yeah it's there it's fixed it enables them to make these smaller genre programmes for the entire country and and i and i don't really think they've got a way of measuring who exactly is listening it's an archaic process i mean you maybe you should ask them how do you know that miss jones is listening to the jazz show how do they know yes i think we've got some interesting analogies for when the BBC come before us back to you convener thank you so much i think we've covered most of the areas that the committee wanted to this morning and i think i'm back to that kind of Scottish element of it and the in Scottish cultural identity and the soft power that was mentioned earlier was really important but just a final quick question and it was something that professor smith talked about earlier on and i'm taken by example one of his own concerts i was able to attend which was the collaboration with tam team burn it was delivered as a jazz version of peter in the wolf in scots language which kind of to me summarises all of these collaboration issues that we have but how important are are these programmes to that innovation and those kind of projects that are completely unique to scotland i'm glad you mentioned that because you know the scottish national jazz orchestra is funded by creative scotland and so we reached the entire country and we did pier and wolf on scots we've also done it in japanese and we didn't use any scottish funding for that tour because the japanese could afford every every penny we've done it in norwegian and next month we're doing in doric with an actor called joice falconer from river city and it's that's a very niche language i don't know if any of you speak it but it's it's fascinating for example if i was to say in english i wasn't a bad boy in scots it would be wasn't a bad boy and in doric is a wasn't a course lund so it's a very distinctive language but we are going to go up there and represent that and i think that's important for a national body like scottish national jazz orchestra to do we're not just doing it for commercial reasons we're doing it for the good of everybody thank you um i don't know if you want to comment at all well it's been fascinating evidence session thank you so much for your attendance this morning and i think you know we'll have the the bbc um to committee soon to reflect on some of the issues that we've covered today so thank you very much i mean i'm moving to private session