This is research computing and engineering episode one with open MPI with George Basilica and Jeff Squires guys. Welcome to the show Thanks. Thanks Okay, so starting off exactly most people listening to this are familiar with research computing but MPI How would you describe MPI? MPI it stands for the message passing interface and and at its heart. It really is just that you know message passing So you start up a bunch of parallel processes together and MPI is used to affect the inner process communication between them Right, so you you're doing send and receive primitives and and various other types of primitives But at its heart MPIs, you know Send this message from this process to that process and the other process does the the matching receive of it and so on There are some some very handy things like collective operations as well So you can do broadcasts and scatters and gathers and reductions and things like that But but at its heart. It's really about communicating so moving data and moving bytes from one process to another It's rather amusing actually my wife laughs at me. She doesn't know why I have a job She says, you know all you're doing is moving bytes. How hard can that possibly be? but one of the the challenging things about an MPI software implementation is that you really need to do with very very high performance, so you know you want to get the minimum latency and the Maximum bandwidth and and be very efficient in your memory usage and your IO resource usage and things like that So all of these things get factored into a very high quality MPI implementation so that we can deliver, you know Very well performing Middleware stack to the user who just really wants to compute and you know compute their fast 48 transforms Or whatever the problem is that they're trying to solve in parallel Would just want to be the tool in the middle that that just works and works very very well for them Probably a lot of people who are listening who have never written a parallel program before with MPI probably wondering Communicating between systems. This is a solved problem. We have the internet, right? I mean what why would we need to redo this again? But MPI actually has a concept of what you're sending right you say like I'm sending 24 doubles to processor to and it's very simple There's no opening a socket or anything and it's also network agnostic, right? There's there's gonna be multiple types of networks here where a computer can communicate Exactly these are you know I get this question a lot. Why why don't I just use sockets? Why why do I need to use this MPI thing and you highlighted some of these things already? There's no connection management. You don't need to what's the IP address over there? What port is it listening on who knows who cares, you know What if it's not even a TCP based network that you're on what if you're on shared memory or InfiniBand or Quadrix or Cray or something like that you just want to send your data You just want to send it and have the other guy receive it and how it gets there. It's irrelevant You just wanted to get there fast And be able to do you know discreet messages. That's another advantage here So sockets or streams, right? You have to loop over reading until you get the entire set of data and then assign us You know assign structure to it so that you can interpret the message whereas with MPI you send a discreet message I'll send you four doubles in an int and you're gonna receive four doubles in an int You don't have to loop over, you know polling to get all of the data and things like that And not only that is it it's a discreet message, but it's also typed just like we said It's four doubles and an int so you can send a struct you can send actual data structures You know down across MPI and however it gets there. It doesn't really matter You're just sending the data and receiving the data and all the the network magic that has to happen in the middle Just happens automatically for you. That's kind of one of the points of of why MPI exists And we should probably point out quickly too. This is a Distributed memory parallel every one of these processors has their own discreet memory space if your code calls malloc and there's nothing keeping them from Saying which rank is actually calling malloc everybody called malloc and they all have their own little memory space So if I have some data as a CPU and I'm trying to give it to another one I have to explicitly send it to it and they have to actually explicitly receive it also correct That is correct. Excellent point here. It is it's all explicit parallelism so you explicitly send and you explicit really receive and You know depending, you know everything that you do you have n copies of Of your application running right and this is this is kind of one of the difficult things to wrap your head around for people Who are new to parallel programming that when you launch a parallel application? Let's say you launch a 32-way job or a 64 or 128-way job or something like that One of the most common ways to do it and there are other ways to do it But one of the most common ways to do it is that you're really just launching 32 or 64 or 128 copies of the same Executable and so they're all running independently, but yet they know who they are so a common Paradigm is I launch all 64 copies of this executable and the very first thing they do is figure out who am I? Oh, I'm number seven out of 64 And so that I know that my portion of the work is over here You know I go to the index number seven and and that's my assigned work over there and things like that So this is one of the difficult things in wrapping your head around parallel computing is that you have all these independent agents running Simultaneously sometimes they synchronize sometimes they don't and so on so it's it's just a new way of thinking for those who are accustomed to programming and serial Okay, so open MPI under what sort of licensing is it available under can commercial application include it and use it or Also, what type of network types as a support some MPI libraries either support Ethernet or Infini band You have to recompile to use a different network type. Is this an issue with open MPI or not? So you got you got two questions in there and tell you what I'll answer the license part here And I'll defer the other part there to George The license that we use an open MPI is BSD and I'm not a lawyer. This is not legal advice But it is my understanding that you know, that's one of the most permissive licenses that's out there So, you know, there's a lot of people out there who love GPL and that's great and GPL is good for a lot of different things But our goal in the open MPI community is to be as inclusive as possible We want to include everybody we want to include the users and the researchers and the academics and the vendors and you know the whole HBC community and In order to do that we had to get you know, the least frightening license out there and and in our Research and our lawyers told us that BSD is the one you want to go because that will be you know The most inclusive and people can do whatever they want and basically again my understanding is all they have to do is cite our Copyrights, but it encourages people to to join us because they can literally do whatever they want with the source code to include Releasing it under GPL if they wanted to But you know our goal was that anybody can distribute this source code for free There is no source license. It doesn't prohibit somebody from doing value add and reselling open MPI But you know, we wanted to set the license barrier as low as possible To encourage development and participation from all corners of the HPC community So do you know of actual software vendors right now with a commercial Application that's actually shipping with open MPI is a supported distributed memory parallel system So I'm gonna I'm gonna have to give a little bit of a weasel answer there Well actually no for for Sun Open MPI is their MPI Sun is open MPI is cluster tools on The Sun platforms and so you know, they have a whole team of engineers at Sun who work on open MPI and release it as part of their you Know High-performance computing products and so on There are the weasel answer that I'm gonna give is about ISVs who use open MPI there I do know of a couple of them. I don't really want to say their names And this is the weasel part mainly because I don't actually remember if they have released their products yet with open MPI or not And if they haven't released them yet, I certainly don't want to announce before they do But as some of the reasons that open MPI is attractive for ISVs is that you know, honestly, it's free it's pretty production quality it generally just works and You know, it takes a hunk out of a price For what the ISVs resell their their software for they can actually, you know, reduce their price a little bit rather than having to pay someone else for An MPI license. So from that perspective, it can be pretty attractive to the ISVs Okay, so they would probably want multiple network type support So what sorts of network types as open MPI support? Do you have to recompile it to enable a different network type or is it like an ethernet only library? So we support us Name a network and I'm pretty sure we have support or if we don't we have it soon Right now we support about 10 different networks. They are mostly targeted toward high performance Computing networks. So you will find in Finneban, Mirinet and so on and there are some other more Exotic networks that we don't have right now But the support will be there in the in the near future The other interesting thing is that you don't have to recompile everything. So you don't have to have multiple MPI For one per network and so on In open MPI everything is modular everything. It's included inside. It can build in one go So once you have your MPI library You can easily change from one network to another by just adding one parameter at the command In the command line and that's it open MPI will do the magic to switch to the right network If we don't specify open MPI, we try to find what's the best network available Was the one that will give you the most performance and we try to use that one now You support a shared memory type of network, right? Of course, what if I have What if I have to in Finneban connected nodes and each of those nodes has like say four cores And I'm going to run across all eight cores in the system do the because I have to use Ethernet between the two nodes Well, the ranks inside one node still use Ethernet or can they actually use multiple network types at a time? So they can use multiple networks at the time. So inside the node If you don't let's suppose that you don't specify anything by yourself So you allow you let open MPI figure out the magic So internally in the node will use share memory because as far as we know This is the best way to exchange messages between cores. I mean between processes running in the same processor And then for external communication then we figure out what's the best network So it will be from the user perspective It will be completely transparent and you can use as many network as you want in same time So if you have a very cheap solution actually to have a pretty good network at least from bandwidth perspective It's to buy for 50 bucks three network card one gig and put them on your cluster And with open MPI that will be completely transparent. We use all three of them So when you look at the bandwidth that you get instead of getting one gigabyte you will get three. Oh So the MPI library will actually bond for us With no administrator intervention Absolutely, and so to clarify one of the things that George said there is that you know We automatically pick the best network for you that may actually be plural. We pick the best networks looks yeah for you And we also do it on a per peer pair Basis so when we're looking at the best networks to connect We're looking at from process a to process b and that may be a different answer from process a to Process c for example if a and b are on the same node We're gonna use shared memory, but if a and c are on different nodes Oh, well, let's use infiniband because I see an infiniband link here or no I don't have infiniband, but I've got three network or three ethernet one gig cards Well, let's use all of those and stripe across that automatically for you So it is a really nice thing. You don't have to have kernel level bonding We'll do it up at the user level for you and just stripe large messages across it and round robin short messages across them as well How much control does the administrator have over this say a cluster has a an extra management network on it Is there any type of? cluster-wide configuration that can be made to prevent the management network from being used Yes, so For those who knows a little bit about the open imp. I we have a lot of parameters So some of them can be used to restrict The usage of some of the network so in the in the case where you have three Ethernet you can name the one that will will never be used or you can name the one that will always be used So I can say okay ATH zero. It's something that it's restrictive for Let's say management and NSF. So that's it I I let open imp knows that he will never have to use this network and Everything we happen internally so there's a lot of a Administrator level is there also there's also user level tunables that can be adjusted without recompiling Absolutely, so, you know, this was actually one of our founding philosophies in open MPI that state of the art back when We created open MPI was that okay? You want to change one of the internal parameters of MPI now you got to recompile or relink or do something like that? So a guiding philosophy to us from the very beginning of the code base has been you know Instead of using a constant even if it's a compile time constant Make it a runtime constant and make it a parameter that can be changed on the MPI run command line or changed on the you know if you supply a configuration file to open MPI or An environment variable there's a bunch of different ways you can change these parameters and so on but you know every time that we have to make a decision like should I use a An eager protocol or a rendezvous protocol or or you know a million others But eager versus rendezvous is a very easy one to discuss We made it a runtime parameter so that power users and system administrators can actually tweak these things the average user They're not going to care. They're never even going to use these parameters They're just going to MPI run and and the right magic will happen usually But for those are you know power users or administrators who want to set it up for it for the naive users There's a lot of control that you can Exert over how open MPI runs itself internally. How many different parameters are actually modifiable into current That's a really good question So so George and I were preparing for this interview yesterday and we were actually trying to count and I'd say it's upward of 500 Wow. Wow. So then between the 500 different ones and all the possible combination there must be you know Many thousands of different combinations that can just be tweaked at runtime the permutations are totally insane and and and I actually want this is a perfect opportunity to say that We have a lot of plans. So the current release series is 1.3 and I'm gonna say that because 1.3 will be released today or Monday or something very very soon from now We have a lot of plans for the 1.4 series and and one of them we consider is a very critical Usability thing so that the idea of being able to tweak the runtime parameters. It's wonderful a lot of people use it They love it and they're they love the ability that they don't have to recompile and they can just change open MPIs behaviors But it's also quite challenging. I mean upward of 500 parameters. That's that's a lot. How do you even know where to start? So in 1.4 We're gonna introduce a feature where we actually have a kind of a rating system for the parameters from 1 to 5 and 5 is the casual user will want to use these and One will be you know a back-end developer will want to change these and so our goal is to Severely limit the number of parameters that you see when you just look for hey What parameters can I tweak if I'm just a casual user? Well, we'll only show you a dozen or two. These are the most important ones that you'll want to tweak Well, let's say I'm not I'm a little more than a casual user. Okay. Well, here's here's 40 or 50 that you might want to tweak Or I'm a power user. Well, you know what? Here's about a hundred of them or so that you can tweak So you can set what your experience level is and what your interest level is and we'll show you that many Parameters and let you choose and tweak and things like that so we consider that to be a pretty important usability feature coming up in the 1.4 series but For some of the more important Obvious ones that users and or system administrators might want to tweak George. You want to you want to take this one? Oh so It's impossible to give a list and they are they are really Many and they they how you use them it will depend on what exactly you plan to do So there are some for the processor affinity some others from memory affinity So if you really want to have control of on how the processes are created and how they are bind To a specific resource like processor or memory. Well, you can use them I know people who use them, but I can count them all with one hand Now I use them on a regular basis. We have some other features that allow the user to Well or parameters allow the user to to find out if their application is correct So we can dump at the end when the user call MPI finalize we can dump A small status of the MPI library so the user can see if he had some Resource leaking or something like he didn't destroy some communicators or there were some communication In progress like, you know, the non-blocking sense or receive Which is illegal from the MPI perspectives and the user should take care of them. So that's That's another another one We have some that we take care of the Some of the network like the one that require registration of the memory at the MPI Level like MPI leave pinned which will allow us to keep the memory pin. So we don't have to do it every time This basically give us more performance on on these networks And I think I will stop here because otherwise I will go on for for hours Yeah, there's a couple though that might be useful for for system administrator types You might want to set, you know, like don't use the management level You can actually set that at a global level so that all your users will will implicitly use that so, you know Don't use ETH0 because that's my my management network. You can also say, oh, you know Make sure that when the MPI run they're actually using the infini band network or the the quadrics network or whatever network and so on So you can set kind of defaults on a global basis So that you know the users who don't know or care just MPI run and get these values Automatically it can be good to just absolutely set these in and ensure that that's the stuff that they do George mentioned a couple of the debugging parameters as some of those are my personal favorites one of the ones that I like is that We have a parameter that if you call MPI abort or if you trigger an MPI error for some reason We'll actually delay for a little while before we actually kill the the parallel job Which gives you the opportunity to attach a debugger and see what's happening in a live process rather than perhaps a core dump or something like that I think some of these you know debugging level controls are actually very important because Parallel program is just pretty hard and so tools that the MPI layer can give you to do this Particularly interaction with other development tools. I think that's a genuine the useful thing So if a user using a cluster or an administrator wants to see what a current runtime value is set to How do they find that out and how do they actually modify them? so we have a we have a common tool that allow the user to see everything inside either everything or by categories so We have different we call them component in open MPI Some of them that are related to the network some other related to the runtime and so on and You can ask the specific parameters of one of them so the tool is called Umpi info and And if you want to look for the for the parameter You can say minus minus param and then component and And framework or framework and component Or you can say simply all all and that will basically give you the whole list of 500 Parameter that you can change. Yeah, and this is exactly the area where in the 1.4 series We're gonna apply some of the you know what level am I oh I'm a casual user so instead of showing you all 500 You know will only show you a dozen or two So this is there will be some neat improvements in this area and so users can set these values either at the command line with MPI run some options They're executable or they can actually set environment variables Or they can set the set of file which can be either set by the C Admin in in the in the same directory where open MPI was installed or the user can set it directly in her in their home area in the dot open MPI directory So there are some default file that open MPI open MPI will look for every time MPI process is started So there are really many many ways to set this argument Usually, yeah, so usually if a user wants to set something that will be forever On on the cluster he will set it in his home area Okay, okay, not bad So if they wanted to have it there all the time they could make their own little config file and leave all the options in there Okay, so that actually sounds really flexible compared to some systems I've used before because there's been that compile compile So if a site if an administrator providing an MPI library started off with a ethernet cluster and then later on a faculty comes in or something and they add on InfiniBand this is actually really powerful because they can have one MPI library that will take advantage of the best network and then on those different networks depending which one they run on they can modify all These little tweakable parameters on exactly and kind of the power there is they don't need to recompile You know, oh, we you know, we got another round of funding and we added InfiniBand to it You can just re MPI run and you'll see the performance difference You know of depending on your application, of course, but you know if you're an application or bandwidth hungry application You know, you'll definitely see a market improvement You know moving to 10 gigabit ethernet or or InfiniBand or or what have you nice nice now Some users who have ran into problems on the cluster I run before this is a pretty common question And a small scale my application runs fine When I increase the problem size the application hangs Now I tried to explain to them eager rendezvous Could you guys go into a little bit more depth exactly what eager and rendezvous are and Why MPI send while blocking it doesn't necessarily always block mm-hmm so eager and rendezvous with some kind of Terms that we we use in between to scare people Basically for us it make a difference on how you want to send this data, right? So of course we can do something very easy every time you send a message We take whatever was in the message as big as it was it doesn't matter and send it to the remote node and To be honest for for a good user for a powerful user the one who knows exactly what he's doing I think that does the best approach because that kind of user the receipts already posted now We know that not all of not all users do that So then we the MPI library try to be Libraries not only us because most of them do the same try to help them a little bit So instead of sending the whole message we send a little piece With the marker okay with the MPI tag inside and the remote node We do the matching and when this matching is done which means the memory. It's ready to receive the message We send the remaining of the of the message. So this is all the that's all That's the rendezvous protocol now eager. It's everything that is smaller than this rendezvous We send it in just one go. Okay, so that's that's the difference between eager and and rendezvous And this can really trip people up because exactly what you said in the beginning there Brock, you know Maybe when I was first running my application I was just sending small messages and they were just sent eagerly there was no implicit synchronization with the receiver But then they they increased their data size they increased their problem size and and they do MPI send with a larger message and Underneath the covers open MPI switches into a rendezvous protocol and suddenly that send is not going to complete until the receiver Actually doesn't matching receive and so they can see a little bit of unexplained behavior here But in all fairness the MPI spec there there is a standard that we adhere to right the MPI standard It it says nothing about eager rendezvous or anything. These are frankly their their implementation details, right? The MPI standard actually Does not say well, it says that MPI send may or may not block and that's an implementation detail But to to send and expect that there is you know a receive buffer there That is that is implicit and it's in its implementation to find about exactly what's going to happen So we actually have license from the spec to be able to affect this kind of behavior And it's really it's a very difficult game to play, you know Because an MPI implementation has to tread a very fine line between resource consumption and Performance, you know, everybody always wants that super low latency in the super high bandwidth and so on But they also want memory to run their real application They don't want us to consume oodles and oodles of RAM, you know until they're ready to use it, right? So if you send a one megabyte message and we send it eagerly well That means the receiver has to consume one megabyte of RAM Because you have not posted a matching receive yet We have to receive it and store that message somewhere until you do a matching receive and then copy the message into your target buffer But we had to consume, you know all that extra memory In the meantime, so a rendezvous protocol is kind of a resource saving mechanism. That's that's why we do it That's what George was talking about. We try to give you a little help there But the eager is trying to get it get it sent get it off that machine It's safe to reuse that buffer so that the rank that sent can keep computing correct That's correct. We usually do it, you know for small messages eager is usually applied to something small enough that You know if you if you send a 2k message, well, it's fine We'll buffer it on the other side because it's small enough and it doesn't really matter But it does give you the best latency as well So if a user finds themselves stuck in this kind of situation and they want to prove it to themselves that You know, this is what's causing their problem They can actually adjust these eager cutoff points between the rendezvous and eager, right? It's just another one of those tunable parameters. They can just specify in the command line Yes for most of the network this this is the case And so the thing is that that will not necessarily correct their application because if the application deadlocked because Well, they don't follow the MPI Specs like, you know, the one of the biggest mistakes in that user doing MPI is to To have both processes doing MPI send followed by MPI receive So this as long as we are on the eager this will work fine because send it's kind of behave as a non-blocking So you you send a message you give it to the network and then you go on the receive and you will receive what? What's coming from the from the pier now as soon as you go for large messages? This will not happen because the send will become blocking and here we are you will deadlock in your application And of course the first thing that people do is blame the MPI that there is a problem No, it's Standard actually Specifically outlaws this communication pattern. Yeah, it says if you do this the MPI implementation is allowed to deadlock And so don't do that Yeah, so now that now they have proof that uh It's their fault and not some fault elsewhere in the system Fun little trick that you can do to verify that your application is deadlock free Is to change all of your your MPI sends to MPI S sent synchronous? and What that does is it forces a synchronization with the other side that the send will not complete until the receive Has been posted on the other side and if you can change your application to completely use S sends instead of regular sends Then you know it's deadlock free It's not going to be as performant as it was because you're doing all the synchronization But you know the the SN pretty much forces a rendezvous protocol and you know You can use that to check for correctness of your application Arguments you could just quickly do this with either like a preprocessor statement But couldn't we also just set the the runtime parameter for the eager rendezvous limit to zero Well, that's the open MPI trick no Yes, yes So it's a little difference between you know an MPI standard trick versus a specific implementation trick trick I see I See okay, so you're releasing 1.3 very soon There was some big changes in that is there any change major changes from that you'd like to point out Or do you want to run right in and describe what what you want to do in the future? Let's talk about 1.3 because we've To be honest, we've been working on 1.3 for a very very long time But much longer than we intended actually but in in fairness when we started the 1.3 series all the work on it We said at the very beginning that this is going to be a feature driven release It's not going to be a schedule driven release We're going to get all these features and we're going to take as long as we're going to take and Unfortunately that take as long as it's going to take took about a year and a half But that's okay. We're going to get it out in the next the 1.4 series will very definitely be a schedule driven release But in the 1.3 series, we have a lot of very cool stuff actually There's a lot of open fabrics updates in there. So we did a bunch of stuff to optimize memory registration usage We did message coalescing actually a fun little story about message coalescing we message coalescing is the idea that You know you're sending oodles and oodles of short messages and eventually you back up the network so that you know You're you're sending faster than the network can send and so therefore you're you're backing up into software level cues instead of hardware level cues and The the trick that you can do there is you can actually collapse Some of these short messages together into a single message when you see that it's sitting in a software cue Oh, that's exactly the same message. Well, I'll just bump a counter and say oh We'll send two of these instead of one and therefore you can actually save You know some some bytes that are sent across the network when that message actually gets out on the wire That's a neat little optimization It has very little to do with real applications Yeah, when I when I've looked at these things before they were more of a a benchmarking trick It's it's a benchmarking trick and and somebody got some cool papers out of it And so we resisted doing this optimization for a long time Because it frankly it adds complexity down in the you know the deep voodoo You know layers down in the in the bowels of open MPI and we're like well We just don't want to add that extra software engineering to do it But we finally bowed to pressure the sales guys beat up on me and other guys saying you need to have message coalescing so In the 1.3 series we have message coalescing so you can get really nice benchmarks Things out there. We also added support for iWARP some some pretty cool flavors of 10 gigabits Ethernet Nix we added support for XRC, which is Melanox's proprietary protocol for Decreasing some some resource usage. So that's some pretty nice stuff We have full support for MPI 2.1 as well So the MPI forum has gotten back together and and started working on newer versions of the standard and MPI 2.1 was kind of an important milestone that it fixed I don't know somewhere in the order of 50 bugs or so in the spec itself And so we support all of those fixes Couple other pain points that you know, we love open MPI and we think it's great But there certainly are some things that can always stand to be improved one of them that We we definitely got to complaints about from users was what then when an error occurred in an MPI application You you tended to see, you know 500 copies of the same error message You got that same error message from every MPI process in the job and so it just scrolled by you know I could grief. I only need to see that once. Why do I have pages and pages and pages of output? So we actually added a duplicate filter so that when you know all 500 of your ranks die Simultaneously, you'll only see that error message once and a little counter saying oh, and I received it 499 times more as well Some other cool stuff we did we integrated with Val grind There's Val grind the memory checking debugger on on Linux has an API that you can actually integrate into your application Or in our case into the middleware that will do memory checking for you and we can do some neat stuff saying hey You started a a non-blocking send, but then you actually started changing the buffer and that's that's bad Don't do that. Yeah, there's actually been some trouble with that in the past. You especially using Networks like infini band, right? We're trying to use Val grind would give you a bunch of crazy errors Yeah, so in finnaband and open fabrics in general and a couple other types of networks as well They're OS bypass networks, right? So the memory may be coming from places that Val grind is unaware of because the operating system honestly is is Unaware of or at least it comes in a in a non-conventional kind of way So we actually worked with Val grind It's API and the open fabrics drivers and in open MPI and kind of smoothed all those things out and said You know for for areas that Val grind would traditionally say oh, this is bad memory We just use the API to say no no nope. It's actually you know, we know a little better So Val grind this memory is okay. So all those false positives kind of kind of go away So that's actually genuinely useful So that sounds like there's a lot of neat stuff that I'm gonna get to play with once I go one dot three the stable install In the future. What is slated for one dot four? So we continue some of the thing that we did in the one of three Jeff didn't mention them like We have support for for torrent different kind of for torrent and we'll improve what we already have and we'll add more support so That's one. The other one is the the MPI thread multiple So we are kind of thread safe before but nobody had Nobody invested enough effort inside to make sure that everything. It's fine So the one three in the one three we did that so now we know that we are thread safe So from correctness point of view We are where we want it to be but we know that from the performance point of view We lag a little bit behind so that will be one of the major thing that I will be in the in the one dot four MPI thread multiple with the best performance you can get and we will add more support for for torrent We have some other plans to integrate more with tools So to to help the user to debug their application and not only to debug but one to understand exactly where the Performance penalties are coming from and how to make their application better Then of course we have the mca parameters that Jeff was talking about us to show less parameter to the user We we plan to to to have the connectivity map So right now the problem with open MPI is that if we if you don't specify what kind of network You you plan to use and if there are multiple networks there open MPI We do it stuff internally, but the user never know and that's something that we'd like to to address So to give a feature to give a parameter to the user that say you know do whatever you think it's best But dump me the the connection map Yeah, it's actually a surprisingly difficult feature to add and why we haven't added it so far Because it's a distributed decision. I know we said before You know what you use between A and B might be different between B and C and A and C And so we kind of have to gather all of this data and print out a nice little matrix to the user So it's a surprisingly difficult feature to add. That's why it hasn't been added yet Yeah, I may actually want to add the the MPI thread multiple stuff. It's actually kind of handy Going into future looking towards like mini core thread multiple allows better machine of MPI processes with like a threaded style So you could have like one MPI rank per node and it would run in cores worth of threads mm-hmm So this going into the future where we're getting many many cores on a system We may not want to duplicate memory so much with the discrete memory spaces Or we may want to avoid the amount of communication going on and actually allow use a threading library instead And that should make that a lot simpler An important caveat to our thread multiple support, you know, so our point-to-point Operations are our thread safe or we're pretty sure they're thread safe We'll see what happens when people start running real apps on them But a lot of the support Functions are not thread safe yet So that's going to take a lot of time and effort to do but you can do your MPI sends and receives and tests and Waits and things like that and those should all be be thread safe But doing all the support functions those are gonna require some more work from us. Yeah like the attributes and So on yeah, yeah, okay, okay, so this has been really good. I have a couple of questions for you guys um, if you could be dictator for a day of the MPI forum What major changes or any even even a minor change something that bugs you would you want to make to the MPI spec? Hey George, why don't you take this one first? Why me? Well, Georgia what George is alluding to both of us are actually on the MPI forum right so George represents University of Tennessee and and I represent Cisco and and both of us actually have multiple proposals up in Front of the forum already, you know a various different flavors some big things some little things and so on and so We're not dictators for a day But we do have our own little pet peeve projects that that we want to answer, you know and get get in there So for example, you know my I think the biggest proposal that I have in front right now is a better ability to layer MPI underneath different languages So, you know, it's cool that we have C C plus plus and Fortran bindings But what about those guys who want to do Python? And I've actually even heard people who want to do Ruby Matlab, you know various other languages that they just want to use MPI as the underpinnings for I think we as a standard can actually do a better job of being able to support that kind of model George well, so Jeff was partially right because Yes, I have my pet project with It's not what I would do if I was Dictator of the MPI actually I think that I would try to slash out some some of the things Yes, I think that it's the problem with MPI is that it gives so many I mean We have a lot of function and they are all useful for some for some people But when you when you try to learn something new and you look there and you see wow man They are four hundred for different function that a Each one of them do something different. I think it's a little bit scaring for for people and On the other side, I think that if we decrease this the the number of this function We as implementers can focus on what really matter You know be scalable do Give the best performance out of the cluster and then some other people can write libraries on top of it That will give to the final user the the extension that he's he's looking for so It's something that I I Don't know if I would like to enforce, but I think that it's It might be an interesting idea to look at tiny MPI Why not necessarily tiny but keep everything related to communication and so on inside and then move Everything else in extensions to MPI. So take MPI on the biggest loser is what you're saying Nice pop culture reference there Jeff Okay, actually Jeff I'm gonna wind up here in a little bit But there was something you and I had talked about at SC this year That was actually quite interesting to me given that the the next show after this is going to be Joshua Anderson from the hoomdee project Which is a Cuda which free goes you don't know what Cuda is. It's Nvidia's graphics card implementation for actually putting scientific applications on a graphics card and there was some discussion about how can we better write Distributed memory parallel Programs that are utilizing graphics cards, and we actually wanted to we were talking about mixing MPI in with that now What kind of ideas do you have for doing that? Well, there's there's some interesting ideas going on there And the very first thing I want to say is that you know, none of this is going to be the answer but I think that MPI has at least a small role to play in You know the GPU and hardware acceleration of mathematical operations and there's there's at least two different ways to do This and I say this and this is came up at super computing because a gentleman from the the Nvidia booth came up to me and said Hey, you know, I actually have some customers doing MPI programs and doing GPU kinds of things with it I want to be able to MPI send and receive directly from GPU memory because it's it's separate than the main memory And you know we can do some forms of sending receiving But we can't do direct forms where I actually are DMA directly the message to and from the GPU memory Can we figure out how to fix this problem? So that's that's one thing that we need to fix Network stacks, I'm sorry tiny MPI But the other one that so we are you know I was talking to this guy from Nvidia and we said, you know, well, what about MPI has this function called MPI reduce Where MPI will perform a you know a mathematical reduction function for you So, you know every one of your MPI processes has you know 2000 double precision numbers that are part of a vector if I want to do a global sum across that vector This is an intrinsic operation for MPI that'll do the communication and computation all in kind of one step for you And then give you the answer But this kind of operation is is very natural to farm back on to some kind of hardware acceleration Be a GPU or otherwise and so One of the things that we did when we want we walked away from super committing this here saying yeah That's a very intriguing idea We should Investigate being able to have MPI be able to take advantage of hardware acceleration now in addition to that So that's a parallel operation right the the reduction operation a global sum across all my processes, you know our global Product or whatever the operation is there in the in an upcoming version of the MPI standard There's a proposal from a guy by the name of Torsten Hoeffler from the University of Indiana who has a Proposal for a new function that allows you to do a serial reduction So I'm just in one process give me the the global Sum of this this vector right here So I don't have to use the MPI mechanisms for communication But I do want to use the MPI mechanisms for computation and this seems like a very natural way to hide some of the GPU Complexity behind it you've got people who already are familiar with MPI if we can just make MPI reduce be a little bit faster You know because we're farming it out onto accelerator hardware or we can give you this new you know local reduction function that you know not not Not a communication thing but just hide some of the CUDA or open CL Which is another programming language for hardware acceleration kinds of things hide that behind there So they don't have to learn a new API, but still give them the power of that acceleration You know that could be pretty neat and so we just kind of completed phase one of that in the open MPI code base We added the infrastructure to be able to have plug-ins or components for CUDA and open CL and MMX and SSE and things like that So now the next step is you know to actually write some of these plug-ins that you know If you've got a GPU we'll just automatically use it very much in the in the same light that George was talking about before that Open MPI will automatically choose the best network by the same token will say oh you have a GPU Well, I'll just load up that plug-in and anytime you call an MPI reduce all I'll farm it back there Actually that that brings up a good point that there's an opportunity for a lot of research here because those accelerators are hanging off of a bus That you know maybe fast maybe not and we're seeing this sometimes with the GPUs So it actually be kind of nice if you could do that serial reduce and it would mad it would just magically take advantage of You know 128 bit SSE or 64 bit SSE or a GPU if your data is large enough to move across that bus Utilize the massive horsepower and a massive memory bandwidth and move it back and actually be efficient. So it's actually a really Lost over completely the decision about whether you're gonna farm it back to the GPU or not It might be well, you know, we're only gonna do this when you know It's more than a hundred doubles or something like that or the hardware is not even capable of doing double precision I can only do floating point precision. So there's actually a decision you have to make first about am I gonna farm it back On to the hardware or am I just gonna do it on the main CPU and this is kind of the power of the open MPI project So, you know, I'm from industry. I'm from Cisco George is from the University of Tennessee's from academia You know, we got to have this the yin and yang of the project that I went and did the infrastructure part of this project And now I'm kind of handing it off to some of the academics to say alright now you guys You know come up with this come how do we do this? I don't know the right answer We need to experiment we need to research and figure out the best way to do it and the universities and the academics are Much better suited to doing that than than the vendors and that's one of the things that makes the open MPI project work so well There'll be another PhD out there. Thanks to this idea Absolutely, and you know, that's part of the reason that you know I said back at the beginning here that you know the open MPI project We really want to engage everybody, you know, there's a lot of people with smart ideas out there and You know whether you're a grad student with just a cool new idea for an algorithm or something like that We we want to talk to you and we want you to have your stuff be part of open MPI. That's why we're a community That's why we work together. Oh cool that that that's really great. So Where where can the open MPI project be found and how can actually people get involved? So the our central website is open-mpi.org and If you go there probably the best way to get involved is to just start looking at our mailing list join our mailing list Join the conversation ask questions and things like that and you know if you actually want to start contributing There actually is an intellectual property agreement. You have to sign Because we're trying to do this properly, right? We don't want to just take code and and hope that you know We can redistribute it under BSD license there actually is a form that you have to sign and the form actually is the Apache software foundation form We we very definitely benefited from that project's work there We we just changed Apache to open MPI and it basically, you know It gives us the right to redistribute things under under the BSD license again same disclaimer I'm not a lawyer. This is not legal advice, but you know for anyone who contributes code back to open MPI We do require that that Form is signed now. I don't want to say that to be scary, but it is one of those necessary legal things here But we love having more people involved and it usually starts with just a conversation or or somebody saying hey You know, I've got a I've got a wacky idea and you know We start talking and and and fun and interesting things happen from there Okay, cool. Well guys, thanks for taking some time out to talk with me Thank you for being the inaugural show of research computing and engineering and I hope to see you guys around them tend to be at SC every year and see you guys there, too So thanks again for being on thanks for inviting us. Yeah, thanks for inviting us. This was great. We appreciate your time No problem. Thanks a lot


The new year will not only bring some new faces to the Bemidji City Council, but a temporary new location as well. Once the city adjourns business for the holidays, the City Council chambers will be getting a facelift. The main objective for the renovations is to make the chambers ADA accessible. The current plan is to build a ramp and move the council desks forward, but this will likely only be a temporary fix. What we're going to do in the meantime is it's a short-term fix. It's to get our current desk of the council up to code. And so what we'd like to do over the next year is plan, take a look at how we can use that council room better. Is it currently meeting the needs of the community? With the renovations expected to go into the new year, the first few council meetings will have to be moved. For the first time ever, council meetings will be held at the Sandford Center. While this is only a temporary location, the city wants everyone to remain informed. We will be noticing that the meetings ahead of time, the January 7th meeting for sure will be over there. The January 14th meeting, there's a possibility it could be here in the council room, in the conference room, because it's a work session. But we will be web casting, we're going to be attempting to web stream the meetings. The City Hall renovations are expected to cost about $10,000. If you enjoyed this segment of Lakeland News, please consider making a tax-deductible contribution to Lakeland PBS.


Do you hear me? Good so good afternoon everyone. My name is François Tijoux and I will be talking about working the DRM and KMS drivers to drag-and-fly BSD And maybe closing the window About myself I'm a consultant, citizen man I'm a former CCTLD system engineer. I worked for the .fr registry. I have been Xorg and Xeleven user as well as a BSD user for a very long time I introduced free BSD in the .fr registry So some .fr machines are now free BSD machines and I've become a drag-and-fly developer since 2011 Drag-and-fly is a UNIX like operating system. It's of course BSD based It was forked from free BSD a little bit more than 10 years ago now One of the goals of the project is to be scalable Drag-and-fly at many goals at the beginning among them The project leader decided to try to Scale for multiprocessor operations differently than what most of the operating system were doing So drag-and-fly use this per core replicated resources doesn't use complicated algorithms and many operations are naturally a lot less and The goal is to keep caches as hot as possible Drag-and-fly also some very innovative features which can be useful for some workloads such as database stems It has a unique file system which has database features The hammer file system is capable of doing a story retention. Every file operation you do is Retain you can go back in story with changes See snapshots and so on Hammer also does the duplication and real-time replication to Distant file systems in a master-slave way Swap cache is a second-level file cache It uses existing swap infrastructure and is optimized for SSDs A graph is better than many words and You can see here the scalability of the system This was a benchmark I ran on a dual-zone system On the orange tail on the horizontal line, you can see the number of postgres clients And on the vertical axis This is the number of transactions per second you get So I've compared Dragonfly with two competing Linux based operating systems and Dragonfly is a green curve We'll now talk about DRM KMS drivers In detail. We had the first tentative to port the new DRM drivers to Dragonfly BSD in 2010 at that point it Became more or less obvious than if we wanted to keep running graphical applications and X11 in particular We had to implement What is called a KMS KMS means canal mod system and it's a new way to manage graphics and Linux and now other Unix like your operating systems So we had a student try to port code from Linux in 2010 He used a compatibility layer But sadly He ran into trouble very early. So he stopped his work He failed the Google Summer of God project, but try to continue later on his spare time He had some code which was apparently working on his machine But I never could get it to run in mine. So This was a bit of a waste So I started again with free BSD Two years later approximately Dragonfly was based on free BSD. So we still have many common canal APIs It seemed like a good idea at the time So in the summer I started to implement some of the low-level APIs required by the canal By December I had most of the Intel driver part in It didn't work But at least it was compiled And it took many moments to get it working and working reliably We had to implement many low-level subsystems which we never had the need for previously and Particular the i9-15 driver is special for it uses shared memory Memory which is attached to the main CPU and Yet the GPU and CPUs part of the silicon are not coherent Memory pages have to be mapped in some special way to be seen by the GPU and Mapped differently on the CPU side and we also have to manage caches page by page So this is Something very low-level very Hard to to get working reliably and it uses something called the Pat P80 Which is The page attribute table a new feature on modern CPUs So we implemented Pat support in the canal, but first we determined Cache management was the only missing part to get the driver working reliably by completely disabling CPU caches So we had the system booting like it was running on a 50 megat CPU Everything was slow as hell. You could see lines being redrawn, but at least Graphical operations were perfect Text was displayed correctly Windows appeared So everything was running fine, but very slowly After having implemented Pat support I started to work on porting the Radiant driver Which is also some very common hardware seen at MD CPUs. It's the Radiant GPUs are now integrated into AMD CPUs. So it's very important to get that working By October of last year I had the TTM and ready on part of the DRM subsystem mostly put in TTM is a special memory manager used by the ready and driver Seals actually a bit more complicated that what the Intel driver uses It's a very complex subsystem and also how to get working reliably But we finally managed to do it By Jolly of this year So One of the problem we encountered later was free BSD drop the ball More specifically on the Intel driver. There were no Significant dates to the Intel driver after the initial Commits by free BSD people The latest supported at where is two-year-olds by now See as IV bridge GPUs It's also blocking for free BSD people for the canned updates their Radiant driver and generic the arm code without breaking the Intel driver so it's a mess and Things that I switched to Linux for the new Intel upstream in September of last year and My first goal my most important goal was to get as well hardware support working there were Many issues we had to take on and in particular the free BSD code was nothing at all what the Linux drivers looked like Most modern features and maybe from Linux 3.2 3.4 the code is vastly different Free BSD also decided to keep The old DRM one code completely separate from the updated code So they have two DRM repositories The old one the new one I Ran a diff between the twos and most cut was the same. So during to in free BSD is DRM in free BSD plus a few more functions It's a mess so I tried to rebase on Linux and Selected Linux 3.8 something as a target The Linux 3.8 branch is interesting for it is the first Linux branch with working non-working as well as support Most people also based their work on Linux 3.8 the free BSD radiant developer Use Linux 3.8 as basis open BSD false also targeting Linux 3.8 and That was important for me because open BSD is apparently the best As the best BSD KMS implementation so far everything is working their feature complete with Linux and So I get I figured I could follow in their past and Use Linux 3.8 as a base so I first try to remove the most messy parts and so put both DRM directories into the same one I Removed the all DRM one radiant and in the Intel drivers and replace them per by their updated version. I Also had to clean up the HGP driver HGP is still used for many operations Linux people are moving away and don't have to use the HGP driver anymore for with recent Intel GPUs But we still have to do with it for now So basically I ran diff get diff between the Linux and Dragonfly code Try to figure out which parts were important which parts were pointless noise implement the important differences piece by piece so as to try to keep things relatively simple for each commit and not to break everything at once Among the noise problems I had Was the free BSD Intel driver used different file names for example There's one low-level API to handle Output management like screen size auto detection of whatever features the display at the end of the cable has Which is called I to see In Linux and I see in free BSD and Dragonfly. Well the free BSD Intel driver decided to rename files having I to see in their names to files with I see in their names, so Pointless differences and times of thousands of call lines of difference We add functions Present in different fires we add renamed functions for not really any good reason Renaming things from I to see to I I see was not only done for file names, but also for function names It took me a long time to figure out what was going on, but once I learned I had to Move functions arounds to reduce differences with Linux. I basically reduce the differences by half Only by moving functions around so I don't know what he was thinking, but it was a mess The free BSD Intel developer also decided to completely rewrite the driver to change style Whereas Linux used if pointer statements. He used if pointer Not zero or if pointer not know statements He added for example parentheses to all return statements in Linux We had written zero in free BSD. We had written parentheses zero and parentheses There were many things like that. He also did white space Linux used for example 8 10 12 White space characters at the beginning of some lines and the free BSD author Moved everything to four white space characters For no good reason at all as the steel the style was nice. It was clearly easy to the eye, but it introduced Nenomers amount of noise We add Tens of thousands of thousands of line of code differences only caused by this style change So I had to reduce this noise before Moving to the important pass Which I did later There are some difficult spots where I wasn't able to completely Translate code from the free BSD way of doing things to the Linux way of doing things Which is the gem a virtual memory code Jam is a low-level memory manager used by the Intel driver Different mappings depending on if you are in the CPU context or the GPU context or so on to flash caches and well, it's very rarely very low level and Very dangerous if you introduce back there If you make a little mistake you can crash your machine have your machine reboot Instantly and so on There's also the I to see versus I see API which is completely different in free BSD Dragonfly and Linux so I had to do To adapt the driver and work around these differences For the rest I implemented Linux API Which is the same all ideas you first Google student code I David show try to do Which is also what the open BSD falls are doing For that I used some wrapper code which was already presented free BSD This is completely crazy free BSD people used Linux wrappers for some of the drivers in particular in filibund drivers emc netapp panacea's people Spent years to implement Linux wrappers to have Linux drivers work in free BSD almost out of the box and The free BSD Intel developer decided to not use them So I copied this file from free BSD Adapted there to them to Dragonfly I Also used missing parts which I took from open BSD since they also have Linux API wrappers and For the rest we implemented some of them in Dragonfly such as idea Which is used almost everywhere to manage small integers in Linux See there are many reasons why I decided to use Linux wrappers One of them was the work was almost already done by other people in free BSD and open BSD but Graphic drivers in general are very complex They're fast-moving targets, especially when they're in the Linux channel And it makes more sense to change the Dragonfly canals to behave like Linux it's more Easy in the long term it helps to reduce technical depth and It's really I Think it's really the best approach to try to make the Dragonfly canal behave like Linux and Not try to change the graphic drivers All the way to be of like drivers which were developed and Dragonfly in the first place That could be more technically Correct in a way, but we just don't have the manpower to do that Only for the Intel driver Intel has something like 20 different developers working on it full-time and I'm the only one doing it in Dragonfly and then only my spare time So it makes more sense to port the Dragonfly canal to the DRM driver APIs than the opposite so we have many Incluent Linux header files which are Just wrappers to existing Dragonfly BSD APIs or generic BSD APIs They're used by I-915 radium TTM and They really help to reduce the difference compared to Linux I didn't choose them everywhere still in particular I kept the locking directives completely separate so the locking directives are specific to Dragonfly and I wanted to keep things that way to make it more obvious and the canal Subsystems worked differently in free in a free BSD Dragonfly and Linux. So I wanted to Get as well support working The first step was to reduce noise So I reduced the pointless noise caused by style differences function defined different orders and then I moved to Free BSD APIs, which I replaced by Linux once So many parts are no completely identical between the Dragonfly and Linux drivers and This really helped we fixed many bugs just by using the original Linux code and not the code modified by the free BSD developer We had angles almost everywhere The drivers know much much more solid After that, I figured out I had to update Various subsystems Particular the interrupt code was completely different in Linux 3.8 than what was in free BSD Which was based on Linux 3.2 or 3.3. I'm not completely sure I Also add to update the ring buffer codes your brain buffer is a part of memory which is used to store commands for the GPU Which is then processed in a ring fashion Commands go to the end of the buffer and then they start to be processed again at the beginning of the buffer and so on The output management code was also very special by output. I mean VGA outputs all sort of plugs and cables you can use to plug this place to GPUs it's completely different in a has-well in previous GPU generations The outputs were more or less linked to a different chipset and They are no on the CPU die and they also are no completely digital There's still some special handling for VGA support, but If I'm not mistaken, it will be removed soon in some future Intel GPU generation So we add to update the mod The driver mod management and output auto detection code Which was one of the last part I got working and Then we were able to display graphics on as well But they were correct. That was kind of fun actually for you could see window frames be displayed and window contents start to appear little pieces by pieces tell bits by bits and the former small rectangles small rectangular areas and it was caused by graphic specific Table management structures The GPU part of Intel's silicon as Its own memory management units, which are completely different Then the memory management units on the CPU part. So there are different page tables We have to manage the structures like we do for generic virtual memory operation But we have to do it differently and in particular as well as different cache attributes than previous GPU generations So once I got that right We finally had working operations and pixel perfect operations as well That was done in August of this year so quite quite some So so that was the canal and I will now speak a bit about us on for the canal part is Not independent test to be used with traditional user applications and like XORG or libraries and so on So until 2013 we used something called pkgsrc, which is the package management systems Which was created by the netbsd guys It was containing very very old versions of x11 applications and libraries Particularly we had XORG server 1.6 It was too old a version to be able to handle Recent GPUs, so I Had to update it. I tried to do some work in pkgsrc, but for many reasons it didn't happen And I myself lost interest in packaging In 2013 dragonfly switched to a new system called deports It is based on 3dsd ports and automatic adaptive layer Automatic tests and validations Version wise it was much better. We had XORG server 1.12 which was enough to get The most recent CPUs GPUs at the time working And we also had all sort of admitted software to get with it Sadly we once again start to have the problem we had with pkgsrc 3dsd port versions are beginning to lag 3dsd guys took care to update KRO which is Graphic library used by many applications To handle two dimensions null surfaces that you care to update that 3dsd still has an old User land Intel driver in its port system So I had to create a local package and Used a much more recent version It improved performance and fixed many bugs with the old driver We had the small artifacts to black bars black rectangles and so on which is now fixed by the new version But I hope 3dsd will be able to update the port system and have recent libraries and applications So the current state is the Intel driver is mostly synchronized with Linux 3.8.16 Except the lower the low level of your 12 memory code cjm code there DRM ready and the DRM ready and driver is Synchronized with Linux 3.8. It doesn't have the bug fixes from Linux 3.8.13, but it's not really a problem at this point in time and the TTM Memory manager of code used by the ready and driver is based on Linux 3.9 So generic DRM part of the code is bit of a mess we still have to update it to Make it more like what is in Linux 3.8. I Already updated some parts but others are much more older and Troublesome In the future I plan to keep synchronizing the DRM code and in particular the generic DRM code with Linux 3.8 something It's required by features like DRM Prime DRM Prime is the subsystem which handles Hardware using two different GPUs and Only one set of video outputs like if you have a laptop with a Nvidia GPU and an Intel GPU You you need some special handling to be able to display graphics on the video outputs Otherwise your own get a black screen or Whatever My goal is them to start Updating DRM drivers to more recent versions once we will have something more like Linux in the in the generic DRM code And then maybe start porting new drivers and in particular We had many people ask about Nvidia support and for that we will require the new driver a Requirement for porting nouveau was also to get the TTM memory manager working reliably Which was done with ready and so now we're good to go Another problem we have is once we start running Xorg We lose console display The traditional VGA consoles with 80 by 25 Text characters don't work anymore once we start Xorg If I try to go back to the console for example, I will get a black screen or frozen display I could try it in real time, but it's probably not a good idea right now And so we could try to Reset the GPU to be more like a traditional VGA VGA GPU once we quit Xorg or Alternatively implement a graphical TTY layer, which is what the free BSD people are doing Which was also done by open BSD Net BSD already has a graphical console It is used by non-PC architectures in particular Well, there are some patches floating around but nothing serious at this point in time So I wanted also to thank some people a constant in Belossoff Was free BSD developer who ported DRM KMS in general and the Intel driver in particular to free BSD Alexander Kabeff started to port the red end driver to free BSD and The job was completed by Jean-Sébastien Pedro and on the Dragonfly side. I did most of the porting from free BSD I did the general generic DRM part as well as the Intel and red end driver and also the GTM memory manager required by red end and some critical people Man all of that work and work relatively Jonas Hoffman spent months working on bugs on the i9-15 driver It was the one with an IE was able to display pixel-perfect pictures by disabling cache on the system Madeleine added pad support to the channel and fixed critical petrol memory bugs and Joris Jovalinly and Marcus Pfeiffer who are also Dragonfly developers and fixed critical bugs and spent long long weeks trying to find bugs and fix them.  Do you have any questions? Yes? Well, I just I Just discovered BSD was doing some significant work during this conference. So I'm running an intentional moving cursor Okay That's with a radio So Interesting see that all three projects Dragonfly and BSD have decided to take the Linux code as is So Slightly different fixes to the Linux code Should be combined No Yeah, try Yeah, trying to keep the compatible years compatible and Maybe not maybe merge them but trying to see what's going on in other BSD projects and See if bugs are fixed or improvements made. Yes. It's a very good idea in general You Said all three projects are moving to Linux compatibility layer. It's not completely true Open BSD false did that net BSD false are doing it. I just discovered it this morning I'm doing it on Dragonfly, but the free BSD Intel developer Has a different approach apparently is trying to completely re-implement the Intel driver on his own and I don't see how that's workable With BSD projects don't just don't have the manpower to do that Any other question? No, don't be shy. Okay. Well, thank you


It's a good question. Your name here. So, hi everyone! Hello! Hi! So, three days was Asia. How was it? Fantastic! I think it was exhausting, but it was also amazing. I mean, I don't know. I've seen so many, so much positive vibes here. I mean, people coming together, and I just want to ask how many people have you met? Like, everyone, let's think about how many people have you met from different countries, different communities. Yeah, I mean, everyone came together, and I don't know how to put it into words. On food. Exhaust amazing! Amazing! Well, personally, I think during the last three days I talked to about 67 people. I just make that up. So, what we want to do now, so we would like to have another big round of applause for everyone. Let's make it a little bit formal. We would like to thank our sponsors of this year, which are? Fine. Okay. So, let's start, of course, with Red Hat, and we've all seen the Red Hats everywhere, and so that was great, the support here. And I'm amazed at how many different projects Red Hat people are working, and people who I knew weren't at Red Hat before they are at Red Hat now. So, we are as a small company. We wonder how we can also get so many amazing people into our company. All my friends are going to Red Hat. I don't know what to do. But they deserve it, and they support events like this. So, thank you very much to Red Hat for making everything possible, and giving us so much personal support. Harris, thank you very much. And then, of course, we would like to thank Google, who have been an amazing sponsor, not just of the First Asia event, but also of the First Asia organization. And we are running Google Summer of Code, as you know. We are running Google Code, and many people here in this room are actually mentors. I hope in future we also see more of the students here at Google Code at the event. We had Google Summer of Code students of First Asia here, maybe just wave with your hands, actually, to an amazing 3D fashion app, and we hope to see more of these cool projects this year. I would also like to thank App Time for their great support from Japan with the Postgres community. It was just a great experience, because we asked Satoshi, how can we bring in more people from Japan? How can we bring in sponsors for First Asia? So, we are able to run everything, and Satoshi just said, listen, let's not have a long discussion. I will just help you guys. I have a company, we are working with Open Source, and we just want to support. This is great. So, Satoshi really like from one day to the other, he just decided to come in and do this. Thank you very much. Then I would like to thank Treasuredata, Masahiro, thank you very much. Treasuredata is from the US, in Mountain View, but you have a lot of teams all over the world. You have different people that you meet online. So, in a way, you give the image of First Asia, like bringing everyone together, and that's why we are very happy to have you here. Thank you. Thank you again, which is our own company, of course, we sponsor, and I would like also like to thank Mozilla. Where are the Mozilians? Upstairs. You know, they are having such a great time upstairs, they don't want to leave, they still want to talk about Mozilla. Okay, so the Mozilla folks are just fantastic, because they can't stop working, and we hope to see all that, as we are all using their work. So, let's thank Mozilla, and let's get started. Next, it's like the Python Software Foundation, Kusher Derst, Director of the Python Foundation, is standing behind. It's actually a great track, how was the Python track? So, let's give a round of applause for the Python track. MySQL was a sponsor, we were sitting here in front, and it seemed like a lot of people had specific questions, so that seemed to work out well as well. Are the MySQL folks still here? I don't see them, okay, they had to leave already, but thanks to MySQL, I would also like to name all together in one row, Ace was the sponsor, they provide the great web space, NUS Enterprise provided us with the space here, JFD Asia was great support, with Meng, who was also founder of the Hackerspace, Silicon Streets hosted the Python track, Hackerspace is hosting something tonight, right? And EwoksPilot and E27 were media partners. Thank you to them as well. These were the sponsors, we had quite a lot of sponsors, but by the way, for next year and for the organization, we are always looking to more, and now I hand over back to you for the community part. Yes, so, I would like to invite all the First Asia team members to come on stage. Hopefully, we also in the team... Harish is just an every category, right? It's like Roland, Harish, please! ProHick, and, okay, so, shall we have a girlfriend, but couldn't come today, she also wants me to... Okay, so, do I miss anyone? Totally. Okay, please raise your hand if I miss you. The photographer. Sores, where are you? Sores, come up! Quick! Make one before. I think Stepan is making the video. Stepan, where are you? Please come up. Michael. Michael is taking videos, photos somewhere. Michael is trying to... He found Michael. Michael Zhang, please come up. And my sister, who left for me this morning, she was here to help as well. On behalf of the First Asia team, thank you so much, you guys, you guys are awesome, without you, nothing would have been possible here in Singapore. Thank you so, so much. So, if any... I will pass the bye, right? If you want to say something, please go ahead. Speech! Speech! I think we're going to get a bit out of this. I think I'm going to get a bit out of this. We are all running away, bye! Quick, quick, quick, just run, just run! I'm not known for saying much. Why? I'm not known for saying little. I'm not interested. You're known for always running the right roads. Thank you. This is great, and I think we really captured the spirit of open source here in Singapore, and I feel a lot inspired. So, I would like to make a short wrap-up about what went on. Justin, how many people participated in this event? I think it's difficult to get the exact numbers, but what could you see so far? Well, in terms of the total registrations, we actually got about 900 over registrations. I think it's 917. Something like that. 917. How many people actually turned up? That's the question that we want to know, right? So, I would say based on our numbers and everything, it's probably about 600, 700, but we haven't really correlated everything together yet, about that number. So, it's a pretty good number, right? It's about 20%, 30% drop-off, which is pretty good, right? And it's all because of you guys. It's all because of you guys and all the speakers who have created all that great content, fantastic content, right, for FOSS Asia. So, thank you very much. A round of applause for all the speakers. Wow, I would love to say a few words, because I've got a lot of real questions. Yes, actually, if we open the floor for the speaker, Ronan is also one of our speakers. He would like to say something. With Ronan and Harish, we are never sure what category to put in. Where should we put the picture? Because it would just be everywhere. Organizers, speakers, community, sponsors, you guys, both of you, just do so much. I don't know. What do you say about the event? It's a bit bigger than I thought it was going to be. And, well, yeah, I guess the highlight for me, as probably one can guess, was Friday's guest of honour. That was extraordinary. I wasn't sure if I was going to get a yes. And he did. For those who weren't here, Dr. Vela Krishnan, the smart national minister and the environmental minister, turned up in the room. That was amazing. I would perhaps sort of reiterate what he's trying to encourage this community to do, which is start doing useful stuff. Be a bit brave. Step out and do things that maybe aren't obvious. I think he said something like, be more millionaires. Oh, yeah. I think that was it. That was the important thing. To do is, please, become a millionaire. And that, from the standpoint of the sustainability of open source as a means of producing technology, software, hardware, otherwise, that's, we'll be talking about it, but that's actually a really big deal. Anyway, I'm rambling and I'm tired. The other thing, the fact that, despite the slightly chaotic method of organization that we ended up employing, the whole thing went very smoothly. And so I'm really pleased about that. It's gone to things that have been much worse. So this was much enjoyable. I've been looking forward to, after some sleep, doing it all again. Thank you. Wow. It's a coming of age in many ways, right? So, from the previous years of doing the Linux couple, Linux conference, and to see Force Asia, which is more than just a Linux space, I think that's something which is, to me, that's a coming of age. So we have been mainstream. We now need to do the next level. Whatever the next level means. I guess it's getting the million-dollar paycheck for all of us. That's the target. So I think that has to be the next year's hashtag, right? The million-dollar hashtag. But it's remarkable. I'm thrilled to see so many people. Whether it's Red Hat or not Red Hat, this is something I would have done anyway. So I think that's the key thing, because it's really building a community and there's enormous opportunity. And I think those who attended my previous session about building an emergency network, all of the things that we need to do, that needs to be done. Creating systems, especially with IoT stuff, we don't know what is going to fail, so we need to make sure we square that out.  Let's pat on the shoulders, and on the backs and say, well, job well done. Now the next work starts tomorrow. So let's have fun together. Thank you very much for the participation. So during the last three days, we received a lot of feedback because also the participants, a lot of things went wrong. Like, we didn't start on time, we couldn't show up, and there's so many rooms, so basically they didn't know where to go, we changed the room all the time. So a lot, a lot of things that people are not so happy about, but we don't want to hear this. We were sent out of feedback form tonight. If you have any ideas to help us out to improve the event for next year, please feel free to write anything that you want to say, but for now we only have time for one speaker to say something very positive, very nice, before we start to clean up the place. So, do you guys nominate anyone? I have one person in mind. If you guys don't have any ideas I will tell the person, another person to go on stage. Anyone want to say? I'm delighted by you to come here. That's how you feel about the moment. Gosh, this is like... Oh please, you gave a lot of feedback. Don't want to take a test blanket or something. You gave a lot of feedback. This was my first time in Asia, ever, first time in Singapore. It's amazing. I've met so many amazing people here. I really wanted to come here because I've gone to a lot of open-source conferences in North America and started to see the same people all the time and I really wanted to meet new people and hear about new projects and just see the perspective of open-source projects from a totally different culture, a totally different set of issues and things that we're trying to solve. So it's really good for all of us to always be mixing together and get outside your comfort zone, talk to people in projects that you might not use that you probably still learn something from how those projects work. I'm here. I tried to spread my little bit of Git knowledge and hopefully some new little Git user groups pop up all around the world and let me know if I can help put that in any way. Yeah, I'd just like to say that she's amazing. All these people are amazing that made this all happen. Mario, you ran away and I'm still there. So thank you so much for making it easy and possible for so many people from around the world to come together and a lot of good ideas are going to come out of these three days. There's going to be like a wall of GitHub commits that happen and mailing these messages and new features and so that's what pushes the community forward is these kind of events and the face-to-face meetings are really important and I value these a lot. So thank you. Okay, so let's give a shot. Actually, it looks like some of you can't get enough. You just want to stay longer. They want to continue more talks, more workshops, coding, hacking, sitting together and don't worry, it's possible. It's possible. The Hacker Space who's here from the Hacker Space? Hacker Space Fox. Yeah, raise your hand. So the Hacker Space runs a friendly session tonight so you can just go there, hang out, meet more people of the local community or some people of your met already and just continue hacking and of course you can how do we get there? Justin, good question. How about Valentine's Day? Actually, Valentine, yeah. Okay, so you walk towards one door and then you go to Polar Vista which is on the green line and you take it straight all the way to Lavender and then you ask and then you ask just go in the embrace. Okay, so Valentine he is at the Hacker Space almost every single day, well a year, maybe 300 days a year and he still cannot explain to you how to get there. I cannot explain to you.  Hi, I'm Justin one of the co-founders of Hacker Space. So it's actually how to get there you take the MRT like he said go all the way to Lavender the address or take it down while you have wifi still No, no, it's 3, 4 yeah, okay Hacker Space done SG Hacker Space SG, there's no dot in there right and get the Google Maps but those people who don't have the Google Maps is 344 King George building Is it a bee? Yeah, a bee Okay, whatever Anyway, so how do you get there for those people who have no internet how do you get there once you reach Lavender you go out of the station you turn left you go out, you turn left again go out the escalator turn right go straight go straight all the way down right, turn left turn right got it was that the way yes it is I vouched for that we missed one instruction so fun, and going up the stairs when you get to the really large meme tree you've gone too far you better come back the meme tree, yes okay, so there's a meter group and in the meter group you can see everyone who signed up for the session tonight maybe you exchanged telephone numbers so in case you don't know how to get there or you lose your way or you can just go home and sleep okay, so this is great the question now where is it next time, when is it again of hope Singapore Justin is looking forward to more exhausting work here so it's again in Singapore before we changed more often we decided this time we set up everything here nicely and we want to run it again we have a great community here we have great co-organizers so let's continue it for year or like let's decide next year what we do next year so let's do next year in Singapore again it's been an awesome experience and when is it it's 18th to 20th of March so again in the middle of March so easy to remember next year again apart from that Asia is also organizing meetups in different places like Dubai and we are open to announce any projects or something that we have on our community channels please talk to Hon Phuk about that we also have different co-programs please sign up for that there are big chances for you guys okay, so that's the end and do you have something for the end what we can do next I need to clean now if you want to clean I want to get a car about helping clean up so what's it look guys, my dad it's almost there okay, so I give you one minute and yes seriously, yes I give you a minute you have a minute over it's okay then a lot just in time to clean so actually you can just say how well the Mozilla sessions how did it go how do you like the community in Singapore and everywhere around the world who are here Michael, where are you from Switzerland how did you go on the Mozilla trip up there was the coffee good? we really liked the coffee first of all thanks to everyone from your organization the Mozilla trip was going pretty good we really enjoyed having you there I hope you enjoyed it too um could have prepared something is there anything you want to tell them they're awesome yes, he wants to tell you something just as one of the sponsors myself from Mozilla wanted to thank everyone especially the organizers for making a great event yeah, it's been really awesome to see this community been involved in open source in Asia for many years but this is a very special gathering so thank you very much Mozilla, you are late so you didn't have to say so the organizers we can look up on a lot of photos from Asia so let's finish for the day I have a question so a lot of participants have actually asked me all the videos and photos that are being taken how do they get in contact or how is it going to be informed all of the participants have actually asked about all the videos and photos that have been taken here because they would all like to either get some of the video talks that they didn't attend maybe see them online actually my question is for the organizers to suggest a way because I've got a personal youtube account that I most likely will upload a lot of the stuff but in order to get the word out maybe if I get in touch with you guys then you can inform everybody who's got tickets to the event to send them a blast out on where all the slides all of the photos and all of the video material this is all the media for the event maybe you can notify them in one of these times because the video will also take a week or more to produce so many have seen it already actually we are not just using one channel we are using different channels for example there is a first-age affiliate account but you can just use first-age as a hashtag for it usually Instagram also Twitter is a hashtag regarding the videos different people make videos Michael Chang was doing one video we have a youtube channel of first-age maybe we can post it there but you can also post it on the video and python stuff is up already you can say something in a moment but I would also say maybe you can post it on a server that we link through the first-age website of 2015 under 2015.first-age.org and everything will be linked there in a few days Michael do you have some information there? so the videos for the python track as well as the death off track and maybe the videos that you will be doing we are putting it up on ngds.sg ngds.sg is a curated youtube channel right now which has all the little videos not just of first-age videos that are taken in the Singapore community so if you want to find out more about the tech community in Singapore like the ruby community javascript community all the videos are pretty much up there so you can check them out ngds.sg right so by him? yeah I didn't okay yeah so thanks a lot Michael and it's been awesome we are really looking forward to doing videos and I would say one of the last applause for our video guys who took so many and where is Michael? Michael Cannon please come inside you didn't get your applause Michael Michael Cannon Michael Cannon you didn't get your applause from up there and taking lots of photos of everyone so we really appreciate that my pleasure so I think that was enough talk and appalling everyone is awake now it can help now to clean up things so what is there to do? clean the floor we will clean the floor at the end but please check out that we pick up the trash everywhere if possible and there are a lot of stickers on the walls that we put up like to stick up like where are events or what is going on and actually why didn't we do that before we weren't allowed to do that so we have to take off everything as much as possible please go around if you see a sticker or something or some papers on the wall please just take it off and throw it in the rubbish bin and help us to clean the places apart from that before you go Martin had a question who is staying for the next few days still in Singapore so that we know who to connect with maybe we can meet up in the hacker space one thing I'd like to suggest especially for those who are in Singapore for the first time to do a little bit outdoor exploring and explore the Singapore jungle and I don't mean the urban jungle but the real jungle are we going to go into the jungle or are we going to go into the jungle we need to make a decision on that but if people are interested in outdoors I'd like to suggest that we do that do a full bounce for Pulau Ubin and also tonight if anybody is interested in music I'll be exploring some Irish folk at one place and there may be a chance where I get to play but we'll see I don't know what the program is so if you guys want to follow me along just it's on Clark Quay and Molly Malone but I don't know the details yet so we'll just go there in the evening check the hashtag Check Pound Farsi OK, thanks a lot that's a wrap up and just continue thank you


The latest Division II women's soccer coaches poll is out and Bemidji State has made the list for the first time this season. The Beavers come in at 22 on the poll and are ranked third in the central region. This is their sixth appearance in school history, making the top 25 five times last season. The Beavers began their season last week with a 1-0 win over Oklahoma Baptist and begin conference play with their home opener tomorrow night against Minnesota Crookston. If you enjoyed this segment of Lakeland News, please consider making a tax-deductible contribution to Lakeland PBS.