So, in 1943, some years ago, Thomas Wetzen said a really interesting sentence, and if you don't know him, he was the founder of IBM. And what he said back then is, I think there is a world market for maybe five computers. And look at us now. Yeah. And I think that's a really great example of the cyclical nature of technology in general. Why are we talking about the cyclical nature and expectations management in tech? Well, I think everyone might have heard about the thing called AI hype that could be happening right now around the world. And I actually love this Gartner graph because it really highlights that what we are feeling are around the world, that the hype cycle is very strong at the moment. It is very much happening. But as technology professionals, we also know that that is not the only truth or so to say. We know that, OK, soon, likely, it's going to go down, and the slope of entitlement became and then, actually, we will get to the plateau of productivity when the hype disperses. We can actually see the benefits of the technology in the long run and so forth. But mostly, this is just to set the scene, that I think we know that the technology behind these things, innovations, is quite nice, and we can really utilize them. I'm very happy to welcome you to an offer you can't refuse, discovering Chicago film sets with Amalops and Kubernetes. I'm Annie Tavastero. I'm CMO at Vision, which is a Swiss DevOps company. And I'm also a CNCF ambassador, as well as an Azure MVP. I also have done such things like Cloud Cosy Podcast, Cloud Native Live, hosting for a couple of years and so forth. So a lot of fun stuff. I'm very excited to be talking to you here today. Fantastic. Great to have you with me today. So hi. I'm Andi Pollak. I'm the author of Scaling Machine Learning, where I'm diving into different apologies of how you can do distributed compute, starting with Spark, and progressing into TensorFlow and PyTorch distributed approaches. And I'm also a Databricks ambassador, which means I'm working extensively with the open source community, such as Spark, Amalops, Delta Lake, and so on. Perfect. So a bit about our agenda today. We have about 30 minutes, so we're going to be quite quick. So we have introduction, which is happening right now. Then we have some best practices. Then we have a quick demo. Then we have the open source stack that you can use to do all of these things that we're talking about. And then we have some learn more resources in the end. So very much I think we're going to have a fun ride today. To set ourselves up, I think we can kind of agree that Amalops loves Kubernetes as a whole because AI really requires a lot of the things that Kubernetes can offer. So it really requires partability, customizability, performance, consistency, micro-pervices, composability, and security. So Kubernetes offers a great platform, a great orchestration tool to manage Amalops and AI in production, which is also the reason why we are here talking about Kubernetes and Amalops together. But how does that journey look like then from that side? Yes. According to 2019 research, 70% out of all the work that researchers are doing to develop machine learning models never got to production. And that because a lot of them were under, they didn't get enough budget, they didn't have enough support from the teams, and these things are starting to change. According to the recent research from the last year, we are seeing that only 50% of the actual models that are being developed don't get to production, which means 50% actually gets to production, and that shows that there is a market, there is a need, and we finally understand what are the benefits that machine learning can do for us as an industry. And so our questions shifted, not from how do I develop the best model, that's always a big question, but actually how do I enable the machine learning folks, the machine learning researchers, data science, and so on to get their models into production. And what we're seeing now that our stack has expended. We used to have, as developers, as practitioners, we used to have our dev environment, our staging environment, our production environment, and now many companies are adding another environment. That's the experiment environment. That's not the A-B that we're all used to, but it's kind of similar in a way that we need to give researchers a place that they can run their experiment and scale from four cores that they usually start with to thousands and thousands of machine, looking into distributed training as we go. And those environments are extremely hard to develop because challenges like hardware compatibility, the different model, image size, and so on that we're going to dive into them today. Perfect. Then the title and the description did have movies in the description, so why that, how come? So Chicago actually is historically very well known as a film city. So back in the day, actually over 80% of film production or film makers of the movies were produced in the US, so over 80% of them were done in Chicago. So historically very strong movie city, and even nowadays Chicago ranks among the top hubs for moviemaking, so obviously Hollywood is quite big as well, but Chicago is out there with the other hubs as well. Just as an example, there's only one studio, for example, that has over 36 stages, spread over 1.6 million square foot feet of studio around the city. So that is very much kind of like a testament to the moviemaking capabilities of the city, which is why we have selected to talk about movies in this particular cube con. Now, some of the famous movies filmed in Chicago that you might know or you might not know, depending on how deep you are in the topic, Ferris Bueller's Day Off, for example, Transformers 3 and 4, exciting. Then we have Batman Begins, The Dark Knight, Batman vs Superman, The Batman, so a lot of Batman movies, for example, have been actually filmed here, which is actually exactly why, but now that we're going to be running through the six and some extra ones, best practices for MLOps, it's going to be the special edition Batman. So very happy to go through these six and some extra with you today. So we're going to kick off with collaboration. Yeah, I love that addition. There's always a big conversation about DC vs Marvel, but today we're all leaning into the DC space, so that's fine. OK, so the first one is collaboration. And collaboration speaks about systems of people, right? It's how do we find the language to communicate and converse what it is that we need with different parts of the organization? So a lot of the challenges is how do we develop with the customer in mind, knowing that now maybe one of our customers is our researcher. They care mostly about papers. They want to get a high age index. And we need to build a system that is good for them so they can build state-of-the-art machine learning, and later on we can push it to production relatively fast. So we're changing a little bit the way we're thinking about the world and the system that we build. And we also know that now we need to have kind of a shared language, and with the boom of LLMs, all of us kind of learning what everything means, what are tokens, how are they different, why is it bytes and tokens, and data entries, and so on. So this is really the first piece that we need to tackle is the collaboration and finding a shared language within the organization. Exactly. And if sometimes people might say that developers or data scientists might be a bit of a lone wolf, so can definitely Batman be. But as such, as Batman, developers do also have to learn to collaborate, for example, as Batman and the others did in Batman vs Superman, so you do have to kind of learn these skills to slay all the monsters that are attacking you so that you can team up and become the superhero or the MLOP team of your dreams, for sure. But more best practices, version control, and model management. Yeah, just one sentence about Batman that I want everyone to understand. So Batman is the ultimate solopreneur, right? When we think about Batman, he develops his own ships and his own cars, his own things.  So this is something to bear in mind that even our greatest heroes understand. Exactly, yeah. Yes, so the second practice is something that we see very often, we're most familiar with that. We need a place to have a version control, but now we're adding another layer of model management. I need to version my models themselves. I need to manage them. I need to know where are they in my lifecycle of development. Are they only an experiment? Do we need to scale them? What are the hyperparameters? What are the variations of distributed settings that they need? What is the GPUs that I'm running on top of them? Because I might need that for inferencing as well. Inferencing is when I'm deploying my model to production and serving my customers using that. So there's a lot of aspects around version control and model management that we need to add into organizations in order to take it to the next step. Exactly, so for example, tools like MLflow or Kubeflow can really help with model versioning. And version control is something that Batman has had to learn the hard way as well because there's so many versions of it. So you have the Michael Keaton version, you have the Paterson version and so many others. And even there, you really do have to do a lot of version control because all of these different versions have different costumes as well as different mannerisms and so forth. So they need to have distinctive features. The next step, we have automated testing and CI CD. So really, this is super important because it really benefits the code and during the code reliability, efficient deployment and continuous improvement is top notch. So CI CD pipelines automate the integration, testing and deployment processes and reduce the manual intervention and accelerating the delivery of reliable models to production then down the road there. Again, this is really critical as we're building a feedback loop. The CI CD want to be able to be as close to production as much as possible. So again, bringing fresh data and real production use cases going to help us a lot in figuring out if that model actually makes sense. And we'll touch a little bit about that, about feedback loop and how it's kind of like a place where we don't know how to measure our model especially with LLMs now because we don't know if that text is being generated as high quality enough or not essentially. Exactly. And I think another good point that Bathman has made about automated testing or like tool that we know if Batman uses it as well, what we can see offered is part of this maybe. So here and there he says, can I proceed you to take a sandwich sir? So he really helps in the background. He helps Batman with these food needs or with for example, he's for example, testing his equipment, making sure all of these things are happening and it all is happening quite automated and really easily and independently from the needs of Batman, which is a testament to the importance of automated testing for example. Yeah, so then we have containerization as the next best practice. Yes, so for long, long time now, we most of us used to use either VMs or Docker leaning in between the two and their different tools. Some of the challenges that we see here is now if we want to containerize everything together with our model, we're looking at images of about 60 gigabytes. And that could be really, really big, especially if we need to download that and if we need to do auto scaling. So now we're changing things a little, but it might be that during time when my pod is going up, I might start downloading it in buffer mode and there's some different open source tools that I will need to download my model in buffer mode. And there's also a question about the hardware. So VMs are usually faster because we don't have another layer there that we have with Docker. So those usually get better throughput from the network, but then it means that we need to program our image and program everything that we do to fit a specific VM and a specific operation system where it runs, where Docker kind of makes our life much easier. So here we're facing a big challenge, especially with inferencing when we need to scale up horizontally and we need to create more pods on the fly to serve more customers. But how do we containerize that LLM model in order to be able to scale relatively fast? And we have another bullet point about scaling, but I want you to think about it because this is another real bottleneck that we're seeing in the industry. And it seems like only big companies found a solution and some companies are actually leaning into renting their own servers again. So it's kind of like what we've seen 10 years ago about cloud versus on-prem. So it's an interesting discussion that happens now when each company develops different solution based on their budgets and resources. Exactly, and also Batman is an expert in containerization because he has the Batmobile and he can use it to isolate himself from the world around him. So he can even be protected from explosions, lava pits, everything around him. And I hope that your dev environments are not as dangerous as the explosions in the lava pits in Batman's life, but they could be as stressful, but hopefully not as dangerous at least. But there's also monitoring, logging, security and documentation, which is a lot to fit in the one best practice, but we'll try. Yes, we'll definitely try. So who here handled security for containers and scaling up in SQL injection, DOS, D-DOS? Okay, I see a couple of folks, great. So history kind of repeats itself only on a different flavor. So what we're seeing here is things are very similar to SQL injection where we can put any string that we want and essentially we're sending it to our servers and those servers are going to give us the information back. And because it's very similar to prompt injection where we're sending essentially a string to the server, then history kind of repeats itself and it's a good opening for prompt injection, different things like XSS, opening a shell, opening a shell with the actual server so we can do whatever we want. All the traditional network attack, OpenAI just announced that they've seen patterns of D-DOS attack in their network. This is why they've been down since yesterday. And that makes it really, really hard for us when we think about how do we wanna containerize our model and how do we want to create also a sandbox for that already heavy model that we need to serve in order to make sure we are protecting our customer's data and we're protecting ourselves from all of these attacks. And it also correlates to the feedback loop because I wanna know my model is okay. So how do I distinguish between what was an actual prompt injection attack or what was an actual kind of a bad prompt that got into the system because my user is not educated enough and my feedback loop returned an answer that it wasn't supposed to return and maybe I need to retrain my model again. So we're entering this phase where we need to have gateway for our AI that it's a little bit more sophisticated, something that is similar to WAF web application firewall where we have filters of what is going in, what is going out, which add another layer of complexity because of the nature of how these models behave. That again, we can put it in a strategy and target it when we need to but we should be aware of that as well. Yeah, and Batman is also a bit of a security fanatic. I think a lot of you probably know that. So he really does monitor Batman with, for example, all of these screens that he has in his Batcave. So he is really focused on monitoring, as you can see from the amount of screens there and he's really focused on logging it all as well as using that for security. But obviously he might be keeping illegal recordings of all the people of Gotham which is not maybe a best practice that you should be following. So don't do everything that Batman does but something to consider. Yeah. You know, the next Biden administration makes something new. Yeah. Then next up we have scalability. Yes, scalability is a really interesting topic. So if we go back to the slide where we look at the different environments, right? From the experiment all the way to production then we see different places where we need to scale our clusters from experiment. We need to go to distributed trainings apologies where we need to configure all the hardware, configure how we're distributing, configure MPI and a lot of different things depending on what it is that we wanna achieve. And the goal here is really to abstract all these from the researcher. So they won't need to deal with all the ops underneath. They'll only need to deal with how the data looks like and what's the best model to use here. So that's a really big challenge now how to abstract all of these things. Second thing is inferencing. We're actually in production and when we're serving the model are we serving the model in batch or are we serving the model in streaming? If we go into batch maybe we can have some prediction around what is happening with the amount of requests that we get. We have some previous knowledge of what's going on but if we're thinking about streaming if we're building a solution such as serverless or SaaS or anything like that then the predicting how to grow, how to scale my cluster becomes really hard cause it could be that adding just another node another pod could take something like an hour. And so looking at the SLAs understanding what to scale, understanding what to do with streaming is a completely different use case than what we're doing in batching which is a little bit more predictable. And also we need to add layers such as caching that is another big questions. Like how do I do multi-tenancy for a specific customer if I have a customer profile that means my LLM is slightly different. So how do I know how to cut the base LLM? We have the network. How do I cut the base LLM and keep the edges just for the customer? So that's another question that the open source world is trying to answer. And for auto scaling specifically there's a really cool project named KEDA. I know probably most of you are familiar with that. It's a CNCF project that enables us to auto scale based on cues. So instead of looking at CPU cycle or GPU cycles now I'm looking into cues and I can configure it to look at specific things that are relevant for my LLM or for my machine learning that it's not based on specific cycles of a machine. So a lot of open questions and that's based on very specific use cases. Yeah, exactly. But KEDA just relatively reasonably graduated as well so that is very nice for them, congrats. But Batman knows as the last Batman tip of the day Batman knows something about scalability as well at least in here where he's actually using his backbone ability literally to extend his body and legs and arms by becoming a mecha robot which is one way to do scalability but it is scaling his body literally. So that's very nice. But then moving on to best practice rapid fire. So there was a few tips that are kind of more condensed here than the big ones that we had dedicated there. So MLops loves DevOps. So if there's any DevOps or Kubernetes cloud maybe professionals here we're wondering, okay, should I be just starting to focus on MLops? Is it something where I should be exploring more? And that has been maybe a key theme across this KubeCon as well on we need more kind of resources or ideas from the cloud native scene towards MLops to solve a lot of the challenges that Adi has been, for example, mentioning here. So while we do that transition I think we have to keep in mind that all of the DevOps practices that we have learned and loved within our previous work, those all usually apply. Like obviously there's a lot of things that you need to learn in top of with MLops space but don't let's not forget all the good DevOps tips and tricks that we have learned that help our lives. Understanding the bottlenecks is also super important. So really kind of digging deep into, okay where are in that long road from research to production that Adi kind of mentioned in the beginning where are the actual bottlenecks in the organization? Is it for example in the self-service models for data scientists or is it when you're taking actually things in the production and so forth. So there's a lot of spots that you can kind of look out for. Up-leveling your skills. So if you are doing a transition from a certain area to another one, obviously focusing on learning the skills that you need to master to get there better there. So if you are, for example, cloud native KubeCon like a Kubernetes person then might be worth looking into Python, math and basics of data scientists. And vice versa, if you are more from the data scientist side and trying to get into MLops then it might be worthwhile to learn with more about cloud services and cloud native and so forth. Last but definitely this time, definitely not the least it's all in the data for all of these things because if you put that data in you're never gonna get good results. So talking about the data of our presentation today. So we have the Chicago data portal filming permits from the transportation department. That is a bit of a mouthful but essentially this is data open data from the city of Chicago that is about how many filming permits what kind of filming permits does the city give out? The data is there, but we're gonna do a quick run-through of general parts of the data as well but anyone can dive deep on their own time as well. Me as a movie fan, I think I love this data so much it's a lot of fun. And also obviously there's a lot of different things that you can discover from there. So just like a quick tip, it's there. So as you can see the data set here just a few days ago I took screenshots because I didn't wanna trust the internet too much to be fast, that's always a bit tricky. So we have for example, or 14,000 views, downloads and so far it's all the data comes from all the way from 2015 to current. So unfortunately for example you cannot see the filming permits for first viewers day off or so forth but it still has a good amount of data there. It has columns such as application number, application type, work type, applications that is in software so there's a lot of things there. This is how it looks like if you start like opening up the rows so you can see there if you can see, well it's quite big. So you see current milestones, applications start date, application process date, name, comments. And you see here for example that there's a prom doing some filming for their high school prom which is kind of cute in my opinion, that's very nice. And there's a lot of student films being recorded and so forth but there is a lot of exciting ones if you can look through it there's a lot of things you know. Some movie filming and inter-filming and so forth and if we go here I think this one has yes so we have the NFL in zone is doing some filming here so if anyone's a hockey fan you know where to go on that date then. Don't disturb any film sets obviously but you know where they would be. So there's a lot of interesting facts here you get obviously the address, exact address what are they doing, even more utility data what are they actually doing, what kind of camera sets are they using or even the primary addresses of the people who are asking the data and so forth but there's a lot of fun stuff I do you want to talk a bit more about it? Yeah so I always say when you do BI or kind of data analytics you can shake the data as much as you want to confirm that what it is that you want to see. So just bear that in mind when we're doing that but some of the interesting things that we wanted to know is which street are probably going to have film location the most and so we looked at which street requests the most permits which one get accepted so you can see here that Clark Street, Wabash Street, Walker Street and a bunch of other of them get the and the one that gets the least request is Paul Cren but it still gets request so it's more than the one that don't get anything and of course we kind of did some diving in and then we asked ourselves okay so which month should I go to this street to find a film location and we discovered that March 2023 was the most crowded in terms of request applications so definitely if you're a movie fan, if you're a Batman fan, if you know there's a movie coming up and they're now filming you need to go to Wabash Street on March next year and then we can we put it on the map to see how it looks like, what's the distance if we need to run from place to place as well so we can better plan our next visit to the city. All right, should we go for the demo? Yeah. Right, so I talked about all the challenges we had with Kubernetes and one of the biggest challenges that Annie and I had was actually getting GPU pools apparently those are scarce and you need to book them in advance and have a very big company logo attached to them or work for the biggest big clouds. So the second best that we did and again to demonstrate what it is that you can do is we actually took pre-configured LLMs and we fine-tuned them so here in Azure what you can do you can create your own deployment and so I created my own deployment I took GPT 3.5 Turbo and I added it I fine-tuned it with some data that I put here inside so you can see this is the Chicago Q&A Jason L Limes that we added and I call my model Annie because I can. All right, and now that I have my model fine-tuned I didn't need to deal with any GPUs, lucky me. I wanted to start experimenting running some things against that to see what I get back and so I started building instructions sometimes you'll see things like a prompt that it's more of a template size you have you tell the model exactly what it is that it's supposed to do this is how you kind of focus it into the right direction so you have examples, you have the questions that you wanna put and you wanna add a response now this often time works and this is where you can add like hard coded rules to your model in order to make sure it doesn't expose any specific data or it answer to some specific request or even if you wanna change specific terminology that the user puts cause the user might not be extremely familiar with your product terminology and you wanna switch it to something that exists for example within your docs and so on this is possible so we can do all of that but then we decided to just play with the prompt specifically for the case of this demo so here I have some input that I'm getting from a user is there any permits with application start date November 23 and you can see there the way we structured the sentence was specifically to fit the data that we gave the algorithm so applications start date this is a column in my data and it was really important cause otherwise the model didn't respond well so something to bear in mind and we're doing just an HTTPS request to Azure using the OpenAI on Azure so we're sending that request, we're packaging it we're sending it over and then we're getting some response and the response is interesting but it says yes there is a permit when an application start date in November but it also says that the permit with a specific number here 1850015 has an application start date of August 28 so you can see my model hallucinate a little bit although I played really really hard with telling it what to do and this means that we still have a lot of work to do around fine tuning, around prompt engineering around working out with our data to make things work and so my Batman is traveling in Chicago and hopefully we'll find them soon but this is just the beginning we still need to do a lot of work to reach a model that actually works for our needs either if they're internals or they're external needs so the whole process around productionizing, fine tuning adding observability, the feedback loop, the quality assurance is really really critical and by the way for quality assurance there are some known ways to go about that there's the rouge and there's the blue from a hunking face that helps you assure that the actual generate text makes sense and one of them is actually adding a human in the loop so having a dedicated person that looks at the response and say okay that makes sense or it doesn't was really really critical for that. Good, Batman is traveling actually from the venue to the filming location as well so that's useful that's perfect, yes and then let's switch this a bit perfect, so that was the demo quickly there so then open source staff for LLMs that we would kind of recommend you to look into is obviously thinking about things such as secure GPU pools and hardware you need to do get that, open source models obviously so Lama, Dolly, Claude, NBT, Falcon and so forth would be at the top of the list and hunking face for example as you explore all of these LLMs starter pack that was pitched or mentioned by Priyanka in the keynote this week was a really great place to start with and start playing around with highly recommend that. You do need a specific hardware for that so just make sure you're, if you're going for this make sure you have the hardware that is needed. Yeah, so hunking face I also mentioned that already quickly there, really good stuff and there's been a few good demos of hunking face within this cube and cube con as well so I would recommend checking in that as well and Langchain as well as Kubeflow, Airflow and MLflow for good running there as well. So and data obviously is needed as well and make sure that that works really well but then what is actually Kubeflow that I'm going to mention a few times within this presentation already? Probably many of you have heard about it but if you have not so Kubeflow is an open source platform that's designed to make it easier to deploy manage and scale machine learning workloads on Kubernetes and it provides a set of tools, frameworks and best practices to really streamline the development and deployment of those machine learning models. So it is really an end to end ML workflow so it really helps you cover the entire machine learning workflow from data pre-processing to model training to deployment and monitoring and it also helps with scalability and resource management so you can really leverage Kubernetes orchestration capabilities for easy scaling of machine learning workloads and really dynamically allocate resources then and it really helps with reproducibility and collaboration as well and collaboration has also been a topic at hand throughout this presentation also in the Batman tips so that's also why this is really important so it really helps you with capturing metadata, environment details and code versions associated with each experiment as well. When you think about building the control plane it helps you manage a lot of the metrics underneath to get combined with ML flow that gives you another layer of managing your models themselves. Exactly, so for example if your bottleneck is around for example having a self-service model for data scientists in your organization that really helps them boost their productivity to flow is a great resource for that. But here are some learn more resources so obviously the linked slides you can also find them in skit as well but I've added them to github as well. There's the data link there that if you want to check that out it's a lot of fun to play around with that. There's the practical MLOps book if you want to get more hands on like okay this is how you get started in Azure, AWS or so forth on MLOps, highly recommend that or I'll book there. There's Adi's book there that she has written about scaling machine learning and then I did a session in KCD Washington DC more focused on kind of like the overall how does Kubernetes fit into the MLOps world and so forth so if you want to tune in there and then there's also a bit more resources there on Kubeflow for machine learning as well. Now what have we actually gone through here today? It's been a tight 30 minutes. We've learned about collaboration, version control, automated testing, containerization and monitoring logging and so forth and scalability and we've learned that be like Batman was the theme of the beginning of the presentation but remember that don't always be like Batman because we are really living in the real world but thank you so much and I think we have a bit of time for Q and A now we'll also be here obviously after the session so you can come and talk to us afterwards. Thank you. Thank you. Thank you.


All right. Good morning, everyone. Welcome to Docker and Ironic, a match made in heaven. My name is Scott. And with me today are Vlad and Bernard to help present. And we're all folks from New Agh Networks. And I'm sure everybody recognizes the Docker Whale. I'm not sure that everybody recognizes Pixie Boots, the ironic bare metal bear. But that is the other key logo for this presentation. So what we're going to talk about this morning, I will give a very brief introduction to containers since I'm sure anybody who's been to any of the other Docker sessions will already be familiar. Talk a little bit about deployment approaches for containers in production data centers. And then we'll talk about why Ironic helps with that and walk through an example deployment demonstration. So containers versus virtualization. Traditional virtualization, you have a hypervisor or a host OS. Then you've got a hardware abstraction layer. And then you've got the OS within the VM itself. And then you've got all of its libraries. And then you've got the app sitting on top. Containers are a much lighter weight solution because you get rid of the hardware abstraction layer and the additional guest OS. You're running your applications with some custom libraries and binaries on top of the host OS natively. So rather than having multiple layers, you're just dealing with processes. So containers are great for some things. It's a single OS to manage, lower overheads, better hardware utilization. With Docker, the really neat thing is the lifecycle management. And you get very quick launch times because you're not booting an entire OS. You're just moving. You're just launching a process. But all good things in life don't come for free. Containers have issues, too. It's Linux on Linux only.  Because you're sharing the same kernel, even with C groups and all of the other goodness that people are constantly putting into Linux, there's still issues with multi-tenancy around process coexistence, around network, and around storage. So one network approach that you'll see that I call the Russian doll approach is, eh, just throw it in a VM. That's great. It is the easiest way to get started. For some workloads, performance is perfectly acceptable. But the overhead is substantially higher.  So for optimum performance and scale, you just deploy containers on bare metal. You put your hosts on a network in your data center. You build lots of really big homogeneous clusters. And that works great for people like Google, Facebook, Twitter, where you've got one application. It doesn't work so great for people, for instance, in financial services, where you've got high security applications and low security applications. Hospitals are another good example. There are places where you need to secure patient information, and there are other applications where you don't really care. So you've got all of these security considerations where a rogue container on hypervisor one, or on Docker host one, can get potentially to other containers here and get to other containers throughout the cluster. So the easy way to deal with that is separate clusters. High security cluster over here, low security cluster over there. And you just build that out. That loses a bunch of flexibility because if you don't size your containers appropriately, or size your clusters appropriately, then you end up needing to take hosts from one, move them to the other, and re-network things. So avoid it if you can. The other thing, containers don't exist in a vacuum. Containers need to talk to VMs. Containers need to talk to bare metal. You need to have some way to make sure that your containers are only talking to the things they're supposed to talk to. And with the way Docker networking works, that's difficult to do because of all of the layers of NAT and such. So how do we approach this in a simple way without requiring changes to Docker, without requiring changes to many of the things that people are already deploying? I'll hand this over to Bernard and he'll talk through one approach to that. Thank you, Scott. So one of the solutions that we believe we can build is by using Ironic. So for a long time, OpenStack has been mainly focused on virtualization use cases. So typically using Nova Compute together with KVM to start VMs. Since a couple of cycles, Ironic is now in the picture to allow you to boot bare metal servers directly with Glance. So typically, you have a Glance image that will load into your bare metal. So that's basically Ironic with OpenStack. So how does it actually work? Well, Ironic is connected to the power control interface of your bare metals. So this is typically an IOM interface or IPMI, which will allow Ironic to boot, stop, or reboot bare metals and to later on PXE boot an image. So typically, if you use KVM, you use Glance to store your virtual machine images. With Ironic, you will use Glance to store your bare metal images. And in the same way you will do with KVM, you will PXE boot those images directly to the host. The host will boot and get started directly on the bare metal. So the other thing we built specifically with Nuage is to Ironic integrate nicely with hardware gateways. In this case, the Nuage gateway, which will allow on the go Ironic to reconfigure the port from that gateway to allow network connectivity and network provisioning from the bare metal to the border over there in order to be able to reach VMs in your data centers, for example. OK, and so typically one of the problems that Scott highlighted is that security issue. You want to separate clusters, right? So in this case, we have tenant 1 and tenant 2. And what you want is to be able to have tenant 1 completely separated from tenant 2 both on the compute level and on the networking level. So how do you do that? Well, with Ironic you will boot a cluster of separated bare metals. But then you will also integrate and create with Newton two separated network that will be instantiated on the gateway. And so each bare metal will actually have a single link to a port of the gateway. When the bare metal boots, we will instantiate the network policy directly on that gateway, which will allow both separation on the compute level and on the network level on the gateway. The other advantage by using Ironic and bare metal directly is that you can actually use full line speed. So for example, if you use a 10 gig nick on your bare metal, you will be able to use those 10 gig because you don't need an OVS. Since it's pure bare metal, you directly output the frames on the wire. You don't need a laboratory neutral agent, for example, since our gateway is natively capable of doing the network functions such as routing and switching. So with this solution, you can have full line speed, 10 gig or even 40 gig. The other thing you want to be able to do, and this is very important, is to be able to bridge between your bare metal and the VMs in your cloud. So typically, I guess, you're an open stack. You've got another compute, a couple of KVMs, plenty of VMs running in the back of your cloud. What you want to be able to do now is to bridge this with the bare metals that you start. And by using Nuage and Neutron with the plugin we got today, you are able to basically instantiate a network policy on the gateway and to bridge into a single overlay bare metal and VMs. And finally, the thing, since Ironic is a module from OpenStack and it's nicely integrated and so on, what you can do is use heat to have one single API call to create a whole stack, which means on one side bare metal, the other side VM, and then network provisioning in between. So you can really easily create a template, start that template using an API call to heat, and that template will create everything from bare metal to VM to the network, both on the gateway and in the overlay. So I think it's time for a bit of a demo. What you want to do is highlight the use case that Scott presented, which is basically a Mezos cluster launch. So I don't know if you are familiar with Mezos, so let me quickly go over it. What you want to do is you want to have a single tenant Mezos cluster, which is going to run containers. So the way Mezos works is basically you've got Mezos master, which is basically, you can see it as Nova scheduler for containers. And then you got Nova Mezos slave, which is the workhorse, which will start the jobs as containers on the bare metal directly. So how do we want to do this? We want to have the Mezos slave on the bare metal directly, and we want to have the Mezos master as a VM. And of course, you want network connectivity in between by using the new hash gateway and our overlay plugin. And again, in this case, it's a single tenant. And what you want to be able to show later is to add a second tenant and have both nicely separated. So here we go. So we got a whole open stack cluster. I'm going to log in into the controller as tenant one. Here we go. Perfect. So initially, there is nothing, no instances. Exactly. And we pre-provision one network, which is the network you want to use to bridge your VMs and your bare metals. So let's go in CLI. Let's use Ionic to show all the nodes. And in this case, we got 10 bare metals. We got 10 bare metals nodes that you can use to start your image. In this case, it will be a Mezos slave. Again, why?  Each of those bare metals are connected to one specific port on our new hash gateway. And this is the key integration we got with new hash. You can specify which bare metal is connected to which gateway port. So let's start the whole stack with heat. I'm going to orchestration. And I'll launch my stack, which will define my VM side, my bare metal for my Mezos slave, and the network in between. Here we go. So Mezos cluster one. And all of those are the parameters that we need to specify in order to launch that stack. OK, for the bare metal, we used the Mezos bare metal image. We pre-provisioned that image in Glens. For the VM, we used the Mezos master image. Again, it's in Glens. And we launch it. Here we go. OK. It's creating it. So beautiful representation of all the objects interacting together. And it's building it. So it takes a bit of time to boot the bare metal. About five, six minutes. Why? Because if you're a bit familiar with Ironic, you need to go over a nylon or IPMI boot up, get an agent, then reboot it, and get the final image of our PXE. That's for the technical details. So here it is. At this point, it's spawning it. And if I go back in CLI and show all my nodes, what I can see now is that I should get two nodes, which are in weight callback. So that's basically I pushed my image to the bare metal node. And I'm waiting for the node to come back. OK. Here we go. And finally, I skipped about five minutes of video here, but we got both bare metal, as you can see, and one mesos master, and one marathon to manage all your docker, container, and jobs. OK. So again, what's the key value here is the fact that between your bare metal and your VMs, we were able to stretch one single network dynamically to create both the overlay and the gateway provisioning to enable to have those specific two ironic nodes on the network in order to be able to reach those VMs. And the nice thing is ironic objects show up in NOVA. So we have those four NOVA compute objects. Here we go. OK, so let's connect to mesos. At this point, what you can see is this is mesos master. We got two slaves, which are the two bare metals. So this is sounding as a VM. And the two bare metals booted up, and we were able to join the mesos slave, which means at this point, mesos should be able to accept jobs and launch them on those two bare metal slaves. Again, why? Because we bridge the network in between. What we got also is 16 CPUs. This will be important for later. Then we got marathon, which is, you can see I said, orchestration for docker. Integrated with mesos. Here we go. And the next thing I'm going to do is launch a job. Let's say you want to launch a dupe, for example. You would push that job to marathon. Marathon will schedule it on mesos, and mesos will push it to the two slaves on the bare metals. Here we go. OK, so at this point, I scheduled my job on marathon. Marathon is, so that job is made actually from two sub-tasks, you can see it over here. And if I go to mesos, I can see that those two tasks are staging. So this means that it's being pushed to the bare metals, to the slaves. And the slaves are downloading all the executables in order to be able to run it. And you can nicely see it's actually load balanced across those two nodes, so 58 and 59. Here we go. And now it's running. So let's scale it a bit. At this point, it's two sub-tasks. Let's go to, I think, 40. And what we integrated into this demo is an auto-scaling capability. So I want to scale it. Let's say you want to grow your ad-oop cloud to 40. In this case, it's going to push those 40 sub-tasks to mesos. Here we go. And you can see mesos is scheduling them one after the other on those two bare metals. And what you can also see is on the lower left side over here, little by little, those 16 CPUs, the 16 logical CPUs are being used one by one up to the limit. So we're at this point at 12, 13.5. Here we go. 15.5 and 16. Perfect. So at this point, basically mesos is completely overloaded. It cannot start any more job. Marathon is being stopped at 32 out of 40. Each sub-task being one half of a CPU has been defined here. And so the thing we integrated is that auto-scaling capability. Basically, if I refresh my instances, what you can see is automatically we spawn a new bare metal on them dynamically. So at this point, Ironic is creating that new bare metal. We've got now three bare metals, three slaves that can run Docker. And you can see that the first slave just joined the cluster over here. And one by one, the remaining eight jobs are being launched on that third bare metal. And we've got 24 CPUs. That's it. It's scaling up. So again, by using our gateway and Nuage, we're able to dynamically add those ports into that overlay network and bridge it with those mesos and marathon VMs. Here we go. And what I want to show you, which is key for this demo, is the VSD. So this is our policy abstraction, which is actually running the whole cloud, both the gateway and the overlay to the VM. And if I connect, I can see my network that I use to bridge both the bare metals with the virtual machines. And what I see is typically those two objects on the top in green are my virtual machines. And then I got three objects lower down, which are my gateway ports. And you can see, for example, this one is port 114. And this was created by Ironic and Neutron together. In order to be able to, when you start your Ironic, when you boot your Ironic, this port is dynamically put into the same network as the VMs. So it's one single tool to bridge both your network over bare metal and virtual machines. And I think Scott is going to go over the second part of the demo. Thanks, Bernard. So Bernard shared how to build and scale one cluster with Ironic. But the interesting part is the flexibility of being able to build and scale out multiple workloads and multiple clusters with Ironic with separated tenants. So we've got tenant one that we already built out. I'm only showing two of the three posts there, but you got the idea. And then we have the second cluster that we're going to build now. And what you have is traffic passing from the VMs through the V-switch on the hypervisor. And then it's encapsulated in standard VXLAN and can integrate with either a Nuage gateway, which has layer three capabilities, or we support a number of other gateways in layer two mode. So there's quite a bit of flexibility there. So let's switch back to the demo. And we'll log in as tenant two. And this is going to look a little bit familiar if you were paying attention. So no instances. We can go and take a look at the stack. We can launch the stack. And in this case, we're actually launching the same stack. Just a little bit of Benny Hill music to wake people up and to entertain them. We're launching the cluster in the same way. It was, for those of you who were entertained by the music and not paying close attention, it was exactly the same heat template that we were using to launch the second cluster.  We get a couple of slave nodes. Those launch quickly. And you can see that we've got three nodes that are active and two nodes that are in process. So again, you can see everything from Nova, which is great. And if we were showing Neutron, you could see all of the ports allocated to the different networks. And we'll see that in a second. So here we go. Stack is up. And we can go through exactly the same process of completely independently autoscaling the second cluster. The first cluster is happily running. As long as you've got capacity someplace in your data center attached to a gateway, you can use those bare metals to attach to any of your tenants. So there's no need to build out independent clusters and dedicate them to specific tasks. So here we go. We've activated two slaves. And we're going to use them to launch another job. This is a different job. This is the play framework. And we're going to scale this up to 40 instances in exactly the same way. So a little bit of a reprise. You can see the active tasks. The resources get used up until the point where they're exhausted. Now, if you were doing this for real, you would have the threshold set so that you wouldn't completely block launching new jobs on your cluster before you started scaling it out. But in our case, we decided that it was fine to do that. So here we go. We've got our next bare metal. That's happily launching. And it will shortly be connected into the network. Docker booted on it. And within five minutes, you've got additional capacity on your cluster, completely independent, completely separated, and fully flexible. So it really does make launching high-performance Docker clusters really easy and painless. And here we go. We're starting to launch the rest of the job and we're good. So ramping up, we've now used our 20 CPUs. We've still got a little bit of space. So now what I'm going to show is exactly the same thing. I've actually launched a fourth bare metal node here. But you can see that you've got the two networks, 10 at 1 and 10 at 2. You've got full visibility of both. And you can see that even though we've got ports on adjacent switch ports on the same gateway, it could be different gateways, it all works together. So a couple of other points. In summary, we've connected ironic heat and NOVA to build a flexible, configurable bare metal plus a VM-based solution using Neutron. We happen to have the only Layer 3-capable VX LAN gateway that integrates directly with VMs. We're working with other people to share with them what we've got so that they can also add Layer 3 and integrate with us. So that's coming. You'll have choices of other gateway vendors. And we're also working with the gateway team in Neutron to make this a standard API. But the key point here is the consistent networking and policy that Nuage supplies. And we also provide seamless inter-DC capability. So if you have a cluster in one data center and you've run out of capacity, you can bring VX LAN traffic through tunnels over to your other data center and have that seamlessly built into your cluster. You're going to lose a little bit in latency. Probably don't want to do that for long. But it gives you capacity where you wouldn't necessarily have it otherwise. So a couple of other points. I could do two or three other presentations on all the cool stuff we do. But networking policy across bare metal, VMs, and containers, we've demonstrated recently 100,000 containers across 200 hosts launching and converging with traffic passing in under 10 minutes. So you can have your bare metals. You can have your VMs. You can have your containers all working nicely together. And it all behaves well. It's scalable and forcible and is supported under OpenStack. So thank you very much. Anybody have any questions? No? All right. Oh, question. So why are you using a hardware gateway to manage the network? Sorry? So why are you using a hardware gateway to manage the network? Why are we using a hardware gateway to manage the network? Yeah, right. Because the other option of using provider VLANs, or provider networks on VLANs and running VLANs through your network, means that you're limited by the number of VLANs you run to any given cluster. And you don't have the flexibility. So with most modern top-of-rack switches, they can terminate VXLAN. And then you also don't need to bring traffic back to a central routing layer. You can route directly at the endpoints and send traffic straight to the other end. So does the containers on different hosts can also communicate with each other? If the containers are on a common network, then they can communicate with each other based on how you've set things up in Docker. So you have the outer enforcement layer of cluster one is a security zone. Cluster two is a security zone. And then you have all of the Docker security within that. But what you see is people don't believe that Docker security is adequate for many workloads. Because in our chart, I didn't notice that you are using any network manager to make sure that two different containers from two hosts can talk with each other. So I did not see that you are using Flannel or Vivo or whatever else to make sure that. So if you want to deploy other container work, or if you want to deploy a container network abstraction, we would strongly encourage you to deploy the Nuage VRS on your Docker containers. It works great. It scales to 100,000 containers. It's really fast. Flannel and Weave may get there eventually, but what we were focusing on here was the Docker on bare metal and the ironic use case. So like I said, we can't talk about everything in 40 minutes. But excellent question. Anybody else? All right. Thank you very much.


This time on Partners, it's tribal natural resources from bison research that promises to deliver selenium rich meat to local residents relearning the power of prairie plants To an inspiring outreach program that mixes youth and horses Tribal colleges and neighboring land grant universities are working together to meet the challenges of the Great Plains Welcome to Partners in the next half hour will travel the nation and see breakthrough work in research education and Extension that's what CSR EES is all about Helping universities generate valuable knowledge for those who need it and educating our next generation of Americans And now it's time for Partners native language For centuries native Americans have solved problems Debated issues and come to consensus by using the talking circle It allows individuals within a group to speak their mind without interruption for as long as they feel is needed The speaker is assured that his or her viewpoint will be heard in its entirety and listening is key to its success for our mind-bodied spirit in 2001 the cooperative state research education and extension service of USDA Funded a unique collaborative group consisting of four tribal colleges and four land grant universities from the Great Plains Working together they decided to improve tribal natural resources and natural resources education in Nebraska North Dakota and South Dakota The talking circle was used by the group to determine the goals of this ambitious project Students faculty farmers ranchers spiritual leaders elders and natural resource professionals all spoke their minds All these things are healthy those opinions shape the actions The group would take during the next five years on the joint natural resource education project Here is their story Shunk on ha means is great spirit dog When our tribe first saw the horse running across the prairie and never seen this before they of course had very limited vocabulary And so they all they could think of was it had four legs like a dog So the first guy you seen him you know runs back to the tribe the crier or whatever and you know I've seen something amazing. I don't know what it was but it was walk home and walk Kata us is the ultimate So to add that word to the dog it was describing dog He just won't believe this thing is magical And they knew that from the second that they laid their eyes on that animal And I nearly became some of the best horse people in the entire world Today it's in the spirit of these kids. They're born with it. I was born with it Lisa Coulomb is the director of Sinti Galeshka University's ranch Here local youth learn how to ride do ranch chores and take care of the horses Beyond horsemanship Lisa sees this as an opportunity for the kids to connect with their rich cultural heritage and to the land Within each Indian person. It's a spiritual thing. It's born with any once you actually out here and you're on the horse It's a spiritual connection That takes you to the land itself that actually makes you look around and realize you know that the earth grows plants and that The animals need that to survive We have open enrollment between 12 and 15 kids every four weeks And they come to the program every day after school our university vans pick the kids up from the schools deliver them out here to the university's ranch and We really try and instill a lot of respect for the land a lot of respect for mother nature And try and bring those principles back into their daily lives where they're caring for animals. They're being responsible This program isn't for every kid as a ballet or piano is not for every kid But I guarantee you and nine out of ten kids that come out here end up staying through one of our programs One person that completed the program is Savannah cook in addition to having become a competitive barrel racer Savannah now works as a volunteer at the ranch They taught me how to bow race and how to ride the horse right and how not to act when you're around a horse She helps with the kids mainly by just being there. She's really become a good mentor It's just really fun because you know something they don't for once So it makes you feel like all older and stuff. It's fun there's always something to do you're never bored and Doesn't matter whether you're riding the horse or not.  Just little issues within yourself You're insecure about or you feel bad because maybe a family member is away. I Think that it just gives them security within themselves. It's somewhere where they're spiritually free Everybody loves the grooming part of it and I think that's an important connection with the animal and it's a respect Level that's created within you and your horse before you even get on the horse They do any of the painting that we need on the ranch Low maintenance putting boards up here and there hanging a hook picking up trash and so they really look after each other It's shared by everybody everybody owns it when the kids show up and say, you know Whose horse is this one? It's your guys. It's it's everybody's it's the communities and that's the way that's what we want to educate Education is our largest tool here on the reservation Over the last four or five years. It's been a struggle and we are tribal institutions So we do not have a lot of capital to maintain and to keep things going We do our best if it wasn't for the natural resource education project and the extra funding that came in in order to make some These extension education projects possible, you know, we might have fell by the wayside We have a real sustainable program here and we have something that many different kids have participated in Iowa State University provides leadership for the natural resource education project the only USDA initiative for future agriculture and food systems program that combines 1994 tribal colleges and 1862 land grant universities It's a big garden takes a lot of people to work it. We've had 15 different people come and work on it Hank Miller is the natural resources director at Nebraska Indian Community College Hank and others have established this 10 acre garden near Santina Brasca as part of the natural resources education grant The goals of this effort are to produce food for needy community members serve as an outdoor lab for Community college students and reconnect residents to traditional plants The native people of this area have a long tradition of using the land to produce food a Lot of the elders are really worried about losing those skills. What we're trying to do with the community garden right now is Use technology in a good ecological way that works with the old ways and try to reinstill that interest again I think the average wage in Santis about 8,000 a year So I feel that part of my job is trying to help teach people how to raise healthy food But we want to do is we want other people doing what we're doing not necessarily as big but doing that for themselves and What we're trying to do is it create food for the elderly the poor whoever really needs it. It's the Indian way The other thing that's kind of we've had white culture and Indian culture both come together and work on this together and The communication relationships. I think that's even getting better and I like that And I think this could be a real good example about how we could all work together We hope that students or other interested people within the reservation After we get the marketing established that they become excited and want to do it for themselves, too So we're trying to create opportunities as well as increase the food supply Another grant project is the Sinti Galeshka University's native herb garden on the Rosebud reservation The emphasis here is to reconnect the residents with their rich medicinal plant heritage This is the mullin plant and it can be used as an analgesic pain killer And it's widely used for chest complaints such as bronchitis Aaron Blackfeather is a Sinti Galeshka University intern at the garden Aaron and others know the importance of this effort Since many of these prairie plants are over harvested the garden serves to protect the species locally It also acts as a hands-on training ground for students We're growing all types of mints, different types of sage Mullin, fennel, bee balm Because the ones that are growing out in the fields naturally They're being over harvested by people who come in from out of town So yeah, this would be a great project to restore them You can make teas out of these plants and drink it Or else it can be applied to your body, the tea You can grind it up and combine it with other stuff like olive oil That would be like a lotion It's really great to take care of plants and watch them grow And learn of their different uses and their habitats I learn better when something's right in front of me I can make a medicine or create like a salve or something Whereas in class, it's kind of hard for me to absorb all the information when I'm reading it But I'll remember it better when it's hands-on In addition to providing educational opportunities Sioux traditions are also being practiced here at the Rosebud Reservation herb garden We offer the tobacco and it's a sign of respect towards the plant Because we have to give something since we're taking part of the plant Anybody who's interested in coming out here and learning about the plants It's going to find out how they're used and what they can be used for To get the word out to more people, Sinti Galeshka University started Herbs Why Herbs The series ran consecutive weeks in the local newspaper We served Todd County and the Rosebud Reservation We had a lot of people comment that they liked it A lot of the ranchers, the farmers and a lot of native people They also liked it, being a native myself We were nomads and relied on herbs to keep us healthy, feed us And so I thought it was a really great idea People read it, people liked it Another natural resource project at the university is focused on sage A plant that is decreasing rapidly on the Rosebud Reservation Project scientists have collected plant samples And are conducting propagation trials with future plans to reintroduce sage on tribal lands The sage project came about because of the collaboration That was formed through the Natural Resources Education Project Our colleagues at Sinti Galeshka University had needs and ideas And we will have trials here on the reservation We'll also have a set of trials at South Dakota State University And Iowa State University So the NRE I think in this project has been a catalyst Back in Nebraska, Hank Miller also collaborates with other partners Such as the University of Nebraska on prairie and riparian woodland restoration In a hundred years Nebraska has lost over 80% of their native tallgrass prairie Now the group is taking cropland that erodes easily like this cornfield And converting it to profitable prairie Fields are plowed, holes are dug The planting crew goes to work, the plants are watered Eventually fields like this will nurture young saplings and other native plants We want to create opportunity and economic viability So within this prairie restoration we did these woody florals Which are dogwoods and willows and different types And we can harvest those and sell them to the florist market We're doing seed collection on wildflowers We're trying to create our own package and our own marketing of wildflower seed from each reservation The networking and the relationships and the bonds that have been formed have been incredible Incredible personally for me, but also for me to do my job I've got a question I can call anyone of those people involved in that grant And it's there, you know, and tribal colleges, we don't have a huge budget So the help of these 1862s are immense and people are so willing And if we're willing to work together, we can be better at the end than more separate And I think this program that we've had shows that immensely Talking Circle participants identified these four natural resource areas as points of focus for the project For centuries, bison have roamed the vast landscape of the Great Plains Known as Tatanka by the Sioux, tribes here depended on this robust animal For food, clothing, and shelter The plains Indians led a nomadic life following the herds throughout the ever-changing seasons Today, bison are still greatly valued by Native Americans In fact, Sitting Bull College, Sinti Galeshka University, and Fort Berthold Community College Have partnered with North Dakota State University in a unique research project Centered on this great animal of the prairie In this study, we initially wanted to look at what the slimming potential is From the soil and the vegetation We did, came out here and collected the soil on five different bison herd pastures Looked at the soil meditation Selenium is a natural mineral found in prairie grasses It has strong anti-cancer properties The project's goal is to find out which grasses are high in selenium Raise the herds on this mineral-rich feed And then market bison meat as a value-added health product Natural selenium, this high selenium protein complex Can reduce colon cancers by as much as 60% And other cancers 20 to 60% So it's great marketing potential I think if you can create a niche market with it The potential to provide a high-end product that's healthy Not only healthy for the Native Americans, but also healthy for the general public Is a great thing to look at What you find with a natural selenium Whether you seed in a meat product or in a grain product Is it gives you the proper selenium protein complex That gives you the most powerful anoxin or selenium in the diet That gives you the biggest benefit in terms of anti-carcinogen We knew we had high selenium potential in this region along the tribal lands I thought it would be a natural fit for the tribes to look at For some economic development The high selenium product that we created here really comes from the soil It is that parent material that is created from the cretaceous shale Which is a rock that is broken down to create soil We actually brought in the bison herds and selected 12 animals And looked at the selenium content in the hair and in the blood To see are we getting that selenium from the plant to the animal Once the samples were gathered on reservation lands They were transported to the labs at North Dakota State University Kevin and his team ran the samples through a spectrophotometer To determine selenium content By correlating which grasses the animal grazed on Data would reveal the optimum plants for raising bison with selenium rich meat The reason we looked at the hair was there was an initial research That showed there was a great correlation between hair and meat And so we looked at the hair because it gave us the potential to say Yes this tribal herd will have a high smart potential and this one will not What was interesting was out of the five different pastures we've looked at There's a difference among pastures. Some pastures seem to have a high level of selenium Some seem to have very normal levels of selenium Some of the tribes will have potential to create a niche market Others will not because of what they have for initial parent material And our next step is to look at are we getting selenium in the muscle tissue Once the research is complete The tribal colleges and North Dakota State University Will turn their attention to getting the value added product to the consumer We'll go through the next year to see the marketability And what kind of return they can get from having this high niche market selenium bison meat Upon completion of the natural resource education grant in 2006 Student internships will continue between the project's tribal colleges And state land grant universities The blueberries are special because of their link to the past They were used extensively by all the tribes on the northern grade plains They're one of the many plants that there's a great interest in revitalizing Because of their healthful benefits and their cultural applications Harry Hartman is on the science faculty at Fort Berthold Community College Working with North Dakota State University He and community members from the Mandan, Hidatsa and Arikara nations Have been trying to re-establish this plant that at one time Provided important nutrition for local residents The actual impetus for the Juneberries came from the talking circles That were held for the NRE grant They were mentioned repeatedly by elders, by practitioners and by educators Hartman, a graduate of South Dakota State's Prairie PhD program Is now putting his natural resources knowledge to work He and his team are trying to determine the ideal conditions for growing Juneberries At a 500 tree test plot at Fort Berthold And here at this community garden at White Shields The plant is being put to the test Samples are taken to determine which soil provides optimum growth Various types of plastic used to retard weeds are being evaluated Some plants are fed by drip irrigation Others rely on weather's precipitation Three different cultivars are also being tested We purchased one-inch pot seedlings from the only supplier in the northern plains area And we're going to try and correlate all of those factors To find out what's the best conditions for re-establishing Juneberries For the tribal members who'd like to have a plot The hope is that Juneberries will again become an important part of daily life As one Fort Berthold elder expressed during a talking circle session Our home had apple trees, crab apple trees A big strawberry patch and plum trees Now they are all gone But the road back to that kind of life is one that will take patience and determination Especially with Juneberries We're hoping lots more for the future when we actually can harvest Juneberries don't yield for four to six years And when that is happening, we foresee elders and youth Harvesting together, preserving together So this is a long-term plan We're planning on being here for five to ten years minimum I believe that having the four tribal colleges And the other four mainstream colleges and universities As partners within the NRE program has been very beneficial So that's been very important to us It provided us expertise And especially the tribal colleges Because they are so similar to our college Taking a look at what is making them work and what is not working We can really learn from each other So having that partnership was very beneficial From water quality studies at Sitting Bull College To the Youth and Horses program at Sinti Gileshka University From the gardens of Nebraska Indian Community College To bison research at Fort Berthold Community College The Natural Resource Education Grant Has brought together the talents from tribal colleges And neighboring land grant universities And their cooperative efforts will help build a sustainable future On tribal lands of the Great Plains for years to come On the next edition, Partners presents Protecting the Homeland In the wake of Hurricane Katrina America's land grant universities went to work From Eden, the Extension Disaster Education Network Distributing vital information to Mississippi residents To new storm and flood resistant housing solutions in Louisiana Welcome to the house, our showcase of solutions for the Gulf Coast home That's Protecting the Homeland, next time on Partners For more information on tribal natural resources And other Partners episodes, log on to this website