And his talk, a deep dive into the PyMongo MongoDB driver, is going to start now. Thank you. Thank you. So a quick poll so I can know how many slides I can skip here. Who here is new to PyMongo or MongoDB? Okay, we'll do a little bit. Those experts amongst us who have crashed through replica sets in Sharding will just bear with me for a couple of minutes. My name is John Willis. I run the developer advocacy organization for MongoDB in Europe. So my job is to get you to start using MongoDB and to stick at it. Those are the databases. They're no good. MongoDB is going to rule the world. So we've been doing this for about nine years. It's a pretty mature database. And it's designed from its onset to take away the pain of programming. Anybody who's ever built a program with an SQL database knows about the pain of object relational mapping and the horror of taking beautifully written object-oriented or functional code and casting it into SQL statements. So from its onset, MongoDB was designed with a set of native drivers in mind. Drivers are essentially the client APIs that are laid to interact with the database. And it was purpose-built so that you wouldn't even have to know about JavaScript object notation pretty much, which is the coin of the realm for the database. That's what we store in MongoDB. It's JavaScript documents, not Word documents and PDFs, but JavaScript object notation documents. But when you interact with a driver like the PyMongo driver or the Java driver or the Node.js driver, you're using the types and the objects of the language that you program in. So in Python, you're using dicks and tuples and arrays and date times and all the objects you know and love. So although underneath the wire we're all JavaScript and actually a binary version called Beeson, which I'll talk about, you don't need to worry about any of that. We handle all of that complexity for you. And then beneath that, then again, we've got the document model, which is JSON-based. And then underneath that, you can effectively go straight to one of the storage engines. So the default operating mode for somebody downloading MongoDB for the first time is they run up a single node on their laptop. And in that instance, you're running directly with what we call the wired Tiger storage engine. It's the default storage engine in version 3.2 of MongoDB, designed to be highly performant and to run very efficiently on clusters with high memory instances with lots of cores, right? So that's the design center. We also have a legacy data engine called M-Map, which we used up to version 2.6 of MongoDB. If you're running legacy applications built on 2.6 or versions of MongoDB before, you can start off using M-Map before you do the conversion. We also have an in-memory database in Beta at the moment. That'll be generally available in 3.4, which comes out at the end of this year. And we have an encrypted database. So if you're storing credit card information or you're doing e-commerce data or you've got medical data or anything that you want to make sure is secured at multiple levels, we can put it in the encrypted storage engine. But the key thing is these layers are essentially insulated from you. They're operational decisions. The driver doesn't care about them. The driver works independently of all of those layers. So the great thing about MongoDB is you can start running a single node, then move to a replica set or a sharded cluster, change storage engines, move to an encrypted storage engine. You don't need to change a single line of code. And that's the real utility of MongoDB. You separate the concerns of operational deployment from development. And we have a range of security and management frameworks and technologies that wrap around this for our production deployment. I'm not going to talk about any of that today. I can direct you to lots of information on that for anybody that's interested at the end of this talk. And we're doing lots more talks about MongoDB at the MongoDB Europe Conference on November the 15th in London. I do recommend you sign up for that. And I've got a discount code for that for anybody that's interested as well. And so the drivers support just about any programming language you can conceivably use in the modern world. Obviously Python, we're here today, but PHP, C++, C, .NET node, Java, of course, Ruby, and a lot of the frameworks. So we're ubiquitous. And we're no longer in these kind of monocultures where there's just one programming language rules all. A modern, distributed, mature development organization uses the most appropriate language for the most appropriate time and place. So being able to support all of these languages is important. And one of the key tenants of the driver philosophy in MongoDB is that drivers should have effectively the same semantics so that when you move from Java to Python to Ruby, you don't have to radically relearn what you're doing. And so a lot of the semantics we're going to talk about today are essentially cross-driver in their application, although I'm going to talk specifically about PyMongo. Now, underneath the covers, we don't use just standard JSON. If you were putting JSON text into the database, it would be horrendously inefficient. You'd have to reparse it every time. So BISN is essentially a binary encoding of JSON. It's a public spec that's available. And the BISN essentially adds two things that we don't get from JavaScript. Type information so I can tell ahead of time this is an int or a document or an array or a null or a daytime. And length so I can skip objects efficiently so I can run down a collection very efficiently. I can tell if there's this many objects at this size. I can skip to the end. I can seek into it. And every driver uses BISN. So every driver will have a BISN library included inside of its driver code. And BISN is usable whether you use MongoDB or not. So lots of people take the BISN capabilities out and just use them as encoding mechanism for JSON. In fact, there's a company in Germany that is doing exactly this at the moment. And if you want an example of what they're like, well, they're like Google protocol buffers or thrift, or if you're as ancient as I am, abstract syntax notation one, ASN one. So it's essentially a binary encoding of text data. So there are three standard topologies or layouts of servers that you're going to see in MongoDB. And it's important that you understand these for what follows because the way in which we interact with these clusters is the way in which we will understand the driver helps us. So the standard one that we start off with when we download a developer kit from MongoDB and we get a driver from APIs that MongoDB is, I run the driver, I run some application code which links the driver in and I connect to a server. I've done this on my laptop a million times and it just runs a single MongoDB process which points at generally slash data. The data files are there, great for development, not recommended for production or anything other than development, but very simple to get up and running. The next and the most common deployment for MongoDB is as a replica set. And a replica set is the way in which we establish durability concerns over the data. So a replica set essentially allows us to say how important is this data and therefore how many copies do I want to keep and how are they distributed across a particular cluster of systems. So if you look at our MongoDB Atlas service which is our database in the cloud, we can mandate that each replica member is in a separate AWS region. That means we're isolated in terms of network, power supply and geography so that we can guarantee that even if a region goes down a replica set will survive. So replica sets are designed to give us data durability and they're constructed as a primary, this node at the top here which is designed to only accept writes. We can't write to any node in a replica set except the primary. And then a set of secondaries. In this case the canonical replica set is a three-node replica set which gives us full tolerance for any single node failing. But you can have up to 50 nodes in a replica set. And so when we write to the primary, the primary's job and the replica set's host job is to ensure that those writes are replicated to the secondaries. And we can specify a concern called a write concern that tells us how important those writes are to us. So if I'm writing log data, I might set the write concern to be just acknowledged which essentially says I get a round trip to the server and everything's funky dory. If on the other hand I'm writing transaction data for a bank, I need to be sure that every write goes to the server. In this case I would specify write concern a majority and that will ensure that a write is only returned as successful if a majority of the nodes accept the write. And write majority is the way most of our production clusters run. So replica sets fail. And so if the primary fails, now I have a node I can't write to. Well, the replica set's designed to automatically recover from this. And the way it recovers from this is it has an election. So the remaining nodes agree on the most stable, most up-to-date member and they elect that member as the new primary. So what we get there is that secondary now becomes the primary. We now have a stable replica set again and now we can accept writes again. And that typically happens in two to three seconds and can be faster if you've got a faster network and faster clusters. And then recovery is when we add the other node back, it will essentially do a sync with the primary and come back up with a complete copy of the data. We're now back to where we were. Note the primary is moved and the whole idea is that primary can rotate around the replica set at will. We don't really need to care where it is because the replica set will manage handling where the primary is. And part of what the driver allows us to do is to establish the connection to the primary when it moves or when it fails. And we'll talk about the protocols that it uses to do that in the next couple of slides. The last cluster environment is the sharded cluster. So the sharp eye amongst you would have realized that there's a finite limit to how much traffic you can pump through a single node in any compute cluster. So primaries give you a cap and it's essentially a cap on the amount of bandwidth or memory or CPU that that single node can handle. Well, a sharded cluster essentially removes that limit by allowing you to run a unified cluster in which there are multiple replica sets which give you multiple primaries. And in that scenario you run some additional demons called MongoS's which are shard routers. They root queries, writes, and reads, and deletes to the appropriate shard based on a key that you define. We're not going to talk too much more about sharded clusters today, although you should be aware that what we talk about what's happening in the driver, very similar driver code is used in the MongoS to establish and manage connections between the MongoS's and the driver code itself. The key thing to remember is your driver doesn't change. Single node, start off. Move to replica set, don't change. Move to a sharded cluster, don't change. There are some code changes we suggest you make and we're going to talk about those to handle failures that happen at the replica set level.  It's not just a kind of a passive API, a library that you load. More often than not, in libraries like PyMongo and Java, it's a multi-threaded client. So it starts additional threads. So it handles authentication and security. It does the Python to Beeson mapping that I talked about. It handles the error handling and recovery for the event that we see failures. It manages the wire protocol so how we bundle Beeson objects onto a socket and send it to the server and handle those responses. It also has a connection pool so that we can essentially manage a set of connections over a set of servers where we don't have as many connections as we have clients. And finally, it does topology management and that's a very specific choice of words because there's a specification which I'll show you at the end which defines what topology is from a driver's perspective and what topology is essentially. The driver's understanding of what a cluster looks like and so a lot of what we'll talk about for the rest of this talk is how the driver gathers topology, understands it, uses it to allow the client to interact and recovers when the topology changes. So if you look inside the PyMongo code there is a topology object that encapsulates all of this interactivity and so when you start reading the spec, if you ever do get to that point, it's called Server Discovery and Monitoring. There's a link at the end. I would always have the PyMongo code alongside in your favorite editor because it makes it real to look at how this stuff implements it and I've got to say the PyMongo code is some of the easiest code to read I've ever come across. They put a lot of hard work and effort into making that a really manageable, easy-to-understand project. So I'll skip over this. Most of the people here seem to understand MongoDB but this is just a standard set of Python calls that we might make so we call MongoClient, we provide it with a host and a port. This is for a standalone instance so we're connecting to local host.  So what happens when we're actually connecting to a replica set? So when I want to create or collection a replica set I can give it some number of nodes. One node will work. In this instance we're giving it two hosts and we're giving it a name, replica set. So replica set is the default name used by mTools which I used to try out some of this stuff. mTools is a package provided by an MongoDB employee which is just a great way of building clusters very fast and very easily. I do recommend you look at the mTools github repository. So when we ask the client to do this, what happens? So the MongoClient actually returns immediately. So when you call MongoClient, if you called MongoClient in PyMongo 2 it would stall because it would be establishing connections to the server explicitly. Well, in the PyMongo 3 client, which I recommend everybody use if you're not using already, it returns immediately because it's actually spinning up some threads in the background. And those threads are monitoring threads that are designed to establish the topology of the cluster. So we've given it two hosts. So the way that the spec operates is we spin up in PyMongo one monitoring thread per server. Now, we only know about two servers because we only told the MongoClient about two servers and we started host one and host two. So we start monitor thread one and monitor thread two. And what these monitor threads do is they send the call that's called isMaster. Now, isMaster is an anachronistic name for a function that should be called getStatus. It predates clusters and replica sets from when we only had a master-slave deployment model. But effectively, it's getStatus. And so isMaster will be sent to each one of these nodes. And the isMaster response that first returns, in this case, is to monitor thread two. And it says two things. IsMaster is false. That means I'm not a primary. And just to double-sync that in, secondly true tells me it's explicitly a secondary, but it also returns a list of hosts. So remember, when a replica set is constructed, it knows about all the other members. So every member of the replica set is aware of all the other members of the replica set. And it continues to be aware by a heartbeat that this replica set handles.  Now, I ran this earlier on.  But the parts we're interested in for the purpose of this talk are it tells me the list of hosts. Now, in this case, I've run them all on the same box. I don't recommend you do that in production. And isMaster is false. Secondary is true. And the replica set name is replica set. So what we do is we take the current topology, which right now is, I know about two nodes, the seed nodes that were passed into Mongo Client. And I'm going to combine it with new information that I've grasped by running isMaster on the first node that replied. And that isMaster has told us that there's three nodes in this cluster. Host one, host two, and host three. Excuse me. So right now we've got a monitor one that hasn't returned, hasn't got a response from isMaster. A monitor thread two that has got a response. So we now know for sure that topology is telling us we have one real node in this cluster. And as a setting, it would be set as RS secondary. There's a status field inside it. We also know that there's three particular hosts in this cluster. So we're going to spin up a third monitoring thread. So now we have three threads, all monitoring the servers, all who have sent isMasters. We've had one isMaster reply. And remember, these threads are running in parallel, but they run with all of the constraints of the global interpreter lock and the threading issues with PyMongo. So they're not super liquidy split fast, right? So monitor thread is returned. Monitor one and two are still waiting for isMaster. In the meantime, what's your code doing, right? So your code startMongo client, it returned immediately. So if it returns immediately, well, I'm going to do something with the database at some point. And so obviously I might call an insert, right? So I'm going to insert one field. Simple document, A, B, right? And it's the same replica set. So it knows it's an insertion to a replica set. This insert is going to block, right? And it's going to block because I can't write to a replica set unless I have a primary. Why? Because you can only write to primaries in a replica set. That's one of the constraints of the cluster. So the Mongo client will hold that right because it knows it doesn't have the complete state of the replica set at this point in time. It only knows about one secondary. So now, again, isMaster response from host one, right? And isMaster response from host one tells me this is the primary node for this cluster, right? Now there's a server selection protocol. It's a whole separate spec. I'm going to talk about it today that tells us how to pick servers to do particular operations. But the one thing that server selection knows is I can only write to a primary. So now that the primary's up, the client doesn't have to wait. It doesn't need to know about the rest of the cluster because the cluster will manage itself. The client knows if I can get a write to the primary, the primary will either satisfy the write concern and write to a majority of the nodes or it will return an error to me, in which case I can return an error to the client. So I can go right ahead. So this write will succeed to the primary host, right? And in fact, the primary will respond with an OK, will send the response back, and the insert will be satisfied. And in fact, you know, in most uptimes, this whole round trip will only take a matter of a couple of hundred milliseconds. So we have write succeeding, we have a cluster topology that's got three monitor threads running, but we only know about two nodes. But at some point in time later, host three is going to respond, right? And at that point, we will update the topology again. We now have three monitoring nodes with three is master responses and we have a complete view of the topology, right? And this is what we call steady state. So this is the standard operating model for a cluster. And now what will happen is these threads will essentially wake up every 10 seconds and run an IS master, right? And the IS master will say, everything's OK, they'll update the topology and go back to sleep. And they may run in parallel, they may get slightly out of sync. That's all kosher, right? We don't mind about that, because as long as we're running inserts that are succeeding, that will all be fine. Unfortunately, life has a nasty habit of intervening. You know, the sixth saddest word in IT, I've ever heard of, we have not experienced a failure, right? Or I have never experienced a failure. Guarantee, I remember I was one of the first people to use Amazon's S3 service in 2006 with building an online backup system and everybody who ever used it said, this stuff's never gonna go down. And sure enough, for like two and a half years, it didn't go down. And then it went down and took out the whole eastern seaboard of applications when the first region went down. Now Amazon had told all the developers, you need to be multi-region to be fault tolerant, resilient to the level that we want you to be. But everybody was piling their nodes into AWS. The default region was East Virginia and they took out most of the startups in North America.   And then you can never, ever say that again, ever, right? So, life intervenes. So we whacked the primary host, right? Now, the threads only wake up every 10 seconds. So you can whack the primary host and the cluster could recover and in a lightly loaded cluster where there isn't many right activities happening, it might never notice. The topology would recover. The 10 seconds would come by. I would say there's a new primary. The topology would update. Clients would never see it. But of course, in a heavily loaded cluster and why the hell would you be using MongoDB if you didn't have a heavy write load? The whole point of no SQL is to handle like thousands to millions to hundreds of millions of writes per second, right? Well, it's likely that before a monitor thread wakes up, the insert is going to hit the client and the insert is gonna hit the client and now it's unpredictable what will happen in the client. It might send the request but never receive a response. It might be halfway through sending the response. It might get a server close. We don't know, but all those errors will be wrapped up by the driver and sent back to the client as a connection failure exception. So you don't need to worry about sockets or timeouts or all of that crazy networking stuff. We do all that for you. All you need to know is at the point when you were making a write the write primary died in some way that interrupted your write. So what do you need to do in this situation? Well, you retry, right? And I'm gonna talk about how many times you need to retry in a couple of minutes but you're gonna retry. So what happens when you retry? Well, the primary is down and you retry the insert. Now the Mongo client now knows the primary is down. It's no longer a surprise to it. So it's going to hold your write in the same way it held write to start. It's gonna wait for that write. It's gonna put it in a queue and it's gonna essentially put out an all points bulletin. It's gonna wake up all the threads. It's gonna say, hey guys, hey girls, there's no primary in this cluster. That's a big deal. And it's gonna essentially, instead of waiting every 10 seconds, it'll call those threads every half a second until the primary returns. Meanwhile, the insert is waiting, right? So elections, like I said, happen pretty quickly between two to three seconds. So within a couple of seconds of putting out the all points bulletin having all of these threads calling his master, we're gonna elect a new primary. In this case, host two. Now host two, it's probably, if you've followed our guidelines, it's the same topology or it's the same server size and scope and bandwidth and network and memory as host one. So we shouldn't see a drop in performance. We're slightly less fault tolerant because now we're only replicating two nodes rather than three, but we've still got some replication. So now the insert will concede. Remember, the client will go, once I have a primary, I'm good to go. So now I can send the write straight into the server. And then, the third node will recover. We'll send somebody down to the data center to plug that network cable in. We'll replace the power supply. We'll install a new node on a different AWS region. We'll put the fire out in the data center. We'll take the cup of coffee out of the network connector and the system will recover. And now we're back in steady state. Know that in many cases we may not have the same primary and that's fine. The topology is comfortable with the primary moving around just in the same way that the replica set is. Now you can change that. You can have a priority, which essentially tells you, I always want if the node, if host one is available to have it become the primary, what happens there is you're going to get a second election. So the minute the host one comes back, you'll have a new election. And every time you have an election, you're likely to get a retry error from your code or a connect failure. So we don't really want to promote lots of elections. So if you're managing a cluster, what typically we say is, if you're doing maintenance on a cluster, say you were upgrading all of the cluster members to a different version of the operating system, take out each secondary first because secondaries will not interrupt the ability of the driver to write to the cluster and then take out the primary last and take it out at a time when your traffic is quietest. Now, we all know run services that are global in scope or that never turn off, but there's always a window when you've leased amount of traffic and that's the time to take your primary down. And the drivers will handle it, like I said, as long as you don't take the primary down for an hour. So what does this mean for you as programmers? Well, if you want to establish connectivity to a server, if you want to know, I have connectivity to a particular server. Calling Mongo Client is meaningless. Mongo Client will wait for 30 to 40 seconds before throwing an error because it's got those threads and they're waking up every 10 seconds and just saying, gee, I wonder if there's a server available. Getting a client object back doesn't give you any state information about the server. What you need to run is an IsMaster yourself explicitly. So you can run client.admin.command. Admin is a database that's always available, so you don't need to create a new database by accident. And IsMaster will return, and if it returns, then you've got a connection. If it throws an exception, then you know that you still don't have a server that's available. And IsMaster has another couple of properties that are interesting. One, if you've got authenticated servers that require using them in password, IsMaster does not need authentication to run. It's purely, effectively, a server ping, so you can very quickly run it on a server. It's also efficient, and of course, it gives you round-trip information to the driver, which the driver can use to establish what's the best server to use for a range of queries. So don't use Mongo client, use admin. What does it mean for queries? Well, find with recovery. Remember what happens when we have a failure? It's going to fail, and then it's going to queue your query until it re-establishes connectivity with the cluster. So MongoDB is not a panacea. MongoDB will manage MongoDB, and MongoDB will stay up for as long as you want, right? Clusters are enormously stable, but we can't manage your network, your IT team, the rogue individual who walks into your data center and kinks a fiber-channel cable, as it happened in Bank of Ireland a couple of years ago, or, you know, other external influences. Like, the most... I worked in a digital equipment corporation way back in the day. The number one thing that used to take out the data center, and this was a plan with 5,000 people, somebody with a JCB digging up the power cable, happened all the time. So, you know, those kind of outages, your driver is going to eventually time out and tell you things wrong. So if the replica set doesn't return, then it means something else is up. So if you get a second connection failure, you need to percolate that up into your application, because that replica set isn't coming back anytime soon, or any meaningful time frame that makes a difference to you as a developer. So the guidance for drivers is, we're not going to automatically resolve your issues. You must participate in your own salvation, but you only need to retry once. So these retry loops you see in lots of code, which is for I equal 1 to 5 retries, retry, don't do that. One retry is enough. If it doesn't work once, it's never going to work, and it's not going to work. So it's a very important program, so very important that you just do that one retry. What does it mean for inserts? Well, for inserts, we've got to be careful, because an insert might succeed, but the response might not come back. So we might send the BSUN object and the command to insert to the server, the primary gets it, then we lose the network, and now the response doesn't come back. So we want to essentially establish that we don't do a double insert. By explicitly adding the object ID to the document before we do the insert. This is something the client would do anyway, but if there's already a document ID, it doesn't try to override it. Now when we do the insert, either the insert will fail or will succeed and everything goes on as before, or it fails and we retry, and when we retry, if the insert had actually succeeded before, we'll get a duplicate key error. So remember ID is a unique index on it, so you can't add objects with more than one same ID. So the duplicate key error will essentially identify that you've already done the insert. Now you're essentially back good as gold and you can carry on as before. Now updates are trickier, right? So updates change a document in place. So, well, I mean, there are three ways to think of this, and two of them I always think are a little bit odd, but people are odd and we have different ways of programming. The first way is it doesn't matter if I undercount, right? It doesn't matter if I undercount. So if I call update and it doesn't succeed, so what? Carry on. Maybe we're just counting log entries and missing a couple isn't the end of the world, right? Or it doesn't matter if I overcount in which case I can retry. I always retry every increment. But if you can't overcount or undercount, you need to think about creating an item-potent protocol for the database. And that's beyond the scope of the driver to help you with. You need to think about essentially turning updates into a series of writes, and those writes may be duplicated and you can use the client to essentially eliminate duplicates or more often than not, if you're using an event sourcing model and I did a big talk on event sourcing, so maybe you can catch me again and catch that talk. If you're using an event sourcing model, if you turn all your updates into straight-up writes, you can essentially eliminate duplicate writes by using the same kind of ID model to essentially add a unique ID to each write and then look for unique IDs which are clashing. Or if you're using the ID field itself, it will send back a duplicate key error. And then you can accumulate and aggregate those values into the updates on the server-side or aggregation framework. But the whole point is there are some operations that just can't be designed from a driver's perspective to be recoverable. And that's just the nature of a distributed, clustered database system. And it's one of the trade-offs that we make. You know, my SQL and Postgres and Oracle will tell you, we've never had a failure, right? Until you do have a failure. And when you have a failure, it's catastrophic. You can't write to the database at all, right? And what we're trying to do is avoid that catastrophic failure. Two more kind of tuning fields for you. There are two parameters that you can put into the cluster. Connect timeout milliseconds is how long the monitoring thread will wait to establish a connection to a server when it's essentially pinging with its master. So this is something, again, we can't tell you. So if you're a high-frequency trader and you put a cage of servers in the New York Stock Exchange, right next to the New York Stock Exchange, well, connect timeout milliseconds might be two milliseconds. If you can't recover a cluster in two milliseconds, you'll lose billions of dollars of transactions, right? On the other hand, if we ever put, you know, I don't know whether we've got servers running MongoDB in an international space station, but you can be damn sure the latency on a satellite link to that space station is going to be slightly lower bandwidth than between ourselves and a data center and Amazon. In that case, you might want to extend the connect timeout milliseconds from greater than 30 seconds. So the other one is server timeout milliseconds. So connect timeout milliseconds is on the thread. Server timeout milliseconds is on the client. And we used to use the same one, connect timeout milliseconds. Server timeout milliseconds is something we added because you essentially want to understand how long am I willing to wait in that queue to complete this write. And so if you've got a system that you can afford to queue writes up, if you've got back pressure, so imagine your writes are coming in off a queue that's in a Kafka queue and that back pressure can percolate down the Kafka queue even all the way back to the clients, then you can afford to set this timeout to be very high. If on the other hand you've got IoT events coming in that you need to process in real time, you want that to set much lower, in which case you'll get an error back to the client if you get that timeout. And so these again are areas where you can tune your environment to control what happens in your application. So all of this is documented in a great document written by an engineer in the U.S. called Jesse Giroudavis. He did a talk on this at MongoDB a year ago, so I'm kind of channeling his talk. He has a set of links. I don't know whether you can read that up here. I'll put all these decks up on SlideShare afterwards, so you should be able to get it if you go to my account named Joe Drumgul or Jay Drumgul. The full monitoring inspect, server and discovery monitor inspect is on GitHub. Very easy to read and download, and we accept change requests on that because it's something you don't understand, do come and ask us. But the key thing is this has established a consistent model for how drivers interoperate with clusters, both sharded and replica sets, although I only talk about replica sets today, for all drivers. So all drivers in MongoDB now comply with this spec. So now, if you're writing code that more or less works in PyMongo, that same code pattern should work in Ruby and Java in the same way. It may work slightly differently in C because the semantics for single-threaded drivers are different from multi-threaded drivers, and we typically don't build multi-threaded clients for languages that don't have explicit threading baked into the language, otherwise we'd be pulling too many third-party libraries in. And so the cluster will manage itself, but the driver requires your help. So when you're writing MongoDB client code, don't forget that you're in the distributed network system. The seven fallacies of building distributed systems is as old as the hills, but it's still true. And so what we do is try and help you as much as possible, but you must participate in your own salvation. So you must engage and read this spec. You're not going to become an expert from this in my talk, but reading the spec will really help you to understand how topology is managed inside a driver cluster. That's pretty much it from me. Those of you who read the talk summation will have gone, what about inserts and queries and updates? How do they work? Well, when I started to do this talk, I hadn't really got my head inside the driver spec. I used to run presales from MongoDB, but they wouldn't let me write code in presales, so I moved to developer advocacy because now I can write code at least half the time, and I have to talk about writing code the other half. So all the operations in MongoDB on the client side, apart from topology management, which is the most complicated and interesting part, we're on the same model. Validate the parameters, and there's lots of parameter validation. Convert the data into Beeson. Now, Beeson, actually, there's an intermediate format called SON, which is essentially an ordered dictionary. So SON objects have the property of being able to convert themselves to and from Beeson. We take the SON object, and then essentially we look for a socket in the socket pool, and within that socket or that server, we ask for a server that's suitable for this operation. This is the server selection protocol I talked about. Again, there's another big spec also in GitHub, which I can't go into today. So the most simple server selection is I need a primary to do writes, but for reads I can use secondaries. Once I've found a server that fits the selection, I take the Beeson object, I inject it down the server socket. It's just a straightforward TCP IP socket. I wait for a response object, which comes back in Beeson. I unbundle that into a SON object, and then I unwrap that into essentially dits or arrays or whatever the client expects to see. The only subtly with queries is we don't actually return anything. When you do a query, we just return a cursor, which is client-side only, and when we query a cursor, it's only when you ask for a cursor element that the iterator goes and gets a list of the elements, and when we query a cursor, we get 100 elements at a time. So that's the MongoDB client, PyMongo client. I've got time for some questions if people want to ask questions. Otherwise, thank you very much for listening today. Question down the back there. Wait for the mic, otherwise I'll have to repeat it. I'm so lazy. Okay, the first question is that you mentioned that people often use Beeson serializing logic themselves without using MongoClient client, actually. So maybe it would be a good idea to just refactor module for serialization and deserialization outside MongoClient, because if I remember correctly, I wanted to serialization logic. Yeah, just Beeson, and I had to install a whole MongoClient package and other colleagues came to me asking about why do we need Mongo here because I just wanted to use Beeson. Yes, you made a good point, but it's a trade-off. We're here to support MongoDB developers primarily, and for them, it's easier if they just pull one package, which is a totality of what they need. If we were to separate that package out, then it makes their lives more difficult. So it's a trade-off. You have some pain because you just want to use Beeson, but the Pi Mongo developers have a much easier life because it's all bundled together. Yeah, I see. Could you also refer to the slide with logic for retry of insertion to database? So this logic, you insert first, then let's assume it fails.  It is in acceptable block, so the second accept won't catch it if I'm right. My error. I will amend that code. Okay, so thanks. Yeah, Anna. Anna also works for MongoDB. She's allowed to make announcements. I just wanted to say really quickly, if any of you use MongoDB with Scientific Python or NumPy, you should come talk to me because we're developing a new package and I would love to get some feedback. One more question here. Thank you for the conference. I would like to know about the motor driver. What's the plan for porting it to Pi Mongo 3? Say it again. About the motor driver. The motor asking key error driver. Don't know about it. I'd have to ask the driver team. Okay. Come to me afterwards. If you give me the question, I will send an email to the driver team and find out what their plans are. Okay, thank you. Do you know about libraries like Mongo Engine and what people, what do you think about this? I mean, it makes it look like it's a normal SQL database and makes it look like all the same? I have a personal opinion. This is not a MongoDB opinion and all developer opinions are personal opinions, no matter what people say. Why would you use something that makes MongoDB look like an SQL engine? The whole purpose of using MongoDB is to make the things you put in the database very similar to the things you interact with in your programming language. Why would you obfuscate that layer with an SQL layer? I know what you're going to say, I already have SQL code that I want to write to alongside. If you have a compelling reason that you need to write SQL queries or write to a database, then absolutely use a layer. One of the things that we have in the commercial offering is a BI connector. It won't let you write to MongoDB, but you can definitely do SQL queries to MongoDB. We support the idea of querying MongoDB using SQL, and there's a range of... there's a mapping capability that essentially allows you to unwrap arrays and nested objects into new tables so that you get a tabular view of your nested data. But you can't write. You've got to write with a driver. So Mongo Engine may allow you to write, in which case, yeah. If you absolutely have to use SQL, then use the tools you need to use. Mongo Engine, I bet you underneath the covers is using the PyMongo driver. Just to complete about Mongo Engine, it could be useful to validate document before MongoDB 3. So, didn't talk about it today, but we have document validation. Yeah, since MongoDB 3.0, yeah. Yes. Before, there are no validation documents, so Mongo Engine could easily validate your document. Well, so I was at a conference recently, and somebody quoted an engineer from their organization. I think it was Gil. He said, you know, he formed a law called Andres Law, which is we can build it today, or we could wait six months and Amazon will build it for us.  You know, things that you build today, we're probably going to build tomorrow. So if there's a broad community of people who need a capability, we spend a lot of time talking to the community. So I can't make any future-looking statements on what we will or won't do, but surprise, surprise, you know, having some kind of SQL capability against the databases has been something that's been talked about for a long time, and we introduced that in the last release of MongoDB 3.2. We're going to continue to enhance that capability. So we rarely drop features. This is why IsMaster still runs, right? IsMaster was the original status call from the very original version of MongoDB. So one of the good things and one of the testaments to our CTO is he's obsessive about backwards compatibility. So we tend not to break stuff in the past in order to bring stuff forward. So lots of our customers are still running on MongoDB 2.2, shame on them, but they are. One down the back there. Yeah, I have a question about automatically recover of the cluster. You told us that when you have cluster of three nodes and primary fails, then afterwards new primaries are available. And I had a situation not far ago that I have a cluster, a five-nodes cluster, and when three of the nodes failed and one of them was primary, then I ended up with two secondaries who couldn't elect new primary, and this could be fixed only by either forced reconfiguration of the cluster by MongoDB or by waiting to at least one of the failed nodes recovers. Is this normal behavior? That is normal documented behavior. That's the way the cluster is meant to respond. So a cluster will only recover a primary if a majority of the nodes survive a failure and those majority of the nodes are connectable, which is to say they're all in the same side of, say, a network partition. So if you lose a majority of the nodes, the two nodes that are left will come up, but they'll come up in secondary mode only. So the driver will say, well, I can't write, but that select server spec says I can still query. So I can still write queries to the secondary nodes, but any write will fail. And that's what I mean about the catastrophic failure. A catastrophic failure is where I can't elect a primary because I don't have a majority of the nodes. Now, that catastrophic failure can be caused by failure to isolate your nodes or through a data center failure. It doesn't really matter. I mean, one of the things that I say is MongoDB nor any other distributed database is not a panacea, right? People ask for 99.99999 uptime because they don't know what that means, right? And then I say to them, do you know that your power provider for your national grid doesn't have that level of uptime? So what are you going to do when the national grid goes down?  And I say, who's it for? Nobody has a network. The whole grid is down. Yes, you're running, but the rest of the world can't connect to you. So it's just, you know, we're responsible for ourselves. The rest of the world has to worry about the rest of the world. So we'll make sure the cluster recovers. If you set it up so that the nodes are independent of the single points of failure within your domain of control, then you shouldn't have three nodes failing at the same time. If that happens, then that means two things have happened. You've joined them to a single point of failure, like a single node or an Amazon region, or more often than not, you poisoned your own cluster. So you might have destroyed your cluster with a code error. So that happens all the time. Okay, thanks. Okay. No more questions? Go forth and write excellent driver code. Thank you very much.


Hello everyone, this is financial author Ahmed Dunn of Ahmed Dunn.com. On this channel you'll find financial travel and more videos just like the one you're watching right now. If you like what you see, please give it a thumbs up and consider subscribing. Hello everyone, this is financial author Ahmed Dunn of Ahmed Dunn.com. Today I'll have another episode of a credit card video and I'll be presenting your Canada Best Credit Card for 2018. So for this video, I'll talk about a little bit from here and then for the next part of the video, I'll take you to my laptop screen where I'll be showing you some other things you need to know. So whenever you're looking at any best credit card rankings or any best credit card links or any website, keep in mind that these websites are in general about 99% of the times they're affiliated websites and they are actually actively promoting the credit card companies. So for example, if you look at the top five credit cards for the best viewers Canada 2018, the first credit card is the American Express Cobalt credit card. The second credit card is American Express Gold credit card. And the third credit card is RBC WestJet World Elite credit card. And the fourth credit card is SPG American Express credit card. And the fifth credit card is RBC Visa Infinite Avion credit card. So all these credit cards do not have any full travel insurance coverages and they do not provide concierge service, which is a main thing you should be looking at whenever you're choosing a best credit card, except the RBC Visa Avion Infinite credit card. So the American Express credit card, they maybe are good for obtaining points, but they're not a good credit card. They shouldn't be best travel credit cards because they're not giving you lots of valid features. The main important features just as I just mentioned that they do not cover a full travel insurance and then they do not have concierge services. So these are all actually affiliated links, which I'll show you in a moment how to detect if any credit card, any websites, they are promoting these credit cards and they're providing you with affiliated links so they can make money based on their ranking. So the more good things they say about the credit cards, the more people will sign up and the more people these sites will make money. So they are actively promoting these credit card companies and in order to get unbiased and honest opinion, check for those websites which do not provide any affiliated links or which are not affiliated any credit card companies. This website will not provide you any link and will not give you any even link. What it means that they're not collecting money from the credit card companies. And also I have my own website which is ameddan.com and I do credit card reviews, but you will not see any link which will take you to the credit card website to sign up for credit cards. And also on my YouTube channel you will not see any links which provides affiliated links that I can make money of those links. So let me take you to my laptop screen and I'm going to be showing you some very important things which will help you to detect how to detect these websites which are promoting the credit card companies and which are providing affiliated links to make money. And the best credit card rankings actually is to make more money of you. So you have to be careful about what you see. There is no really a best credit card. It all depends on your lifestyle and what you are looking for. However, if you are looking for best credit cards in general, I will just stay away from those cards which do not have full travel insurance coverage and which do not give you concierge services. So let's go to my laptop screen then I will show you some more information. Thank you. These are affiliated rankings and the website has interest to make money by recommending or giving you these rankings. So let's take a look at the reverse Canada website. So the first thing I would like to show you the top 10, the 10th annual travel reverse credit cards for 2018. And if you scroll down, then you will see these credit cards and as I mentioned before, these are the top five credit cards. So now let's click on the link of the first credit card. So if you click on the link and what I'll do, I'll copy the link, so copy link address and then I'll put it on the note to show you how you know that this is an affiliated website and they're making money by giving you the link. So whenever you click the link and sign up for these credit cards and the credit card companies is paying these websites. So I pasted the link and if you take a look at here, so if you click that link, this is going to take you to the American Express website and here's the affiliated ID. So if you look at this ID, this ID is how American Express is tracking how many visitors are going to apply or coming to American Express website through this website and then they get paid. So let's take a look at another website to show you the same thing. So here, okay, I'll copy the WestJetRBC World Elite MasterCard and then let's copy this link, copy link address and I'll paste it right underneath our first link. Okay, so here I can see that here's a link here, right here and that's how they know that this is an affiliated person who's signing up for this credit card is coming from this website. So that's how they keep track of how many people are signing up and that's how they make money. Now, there's no problem giving you an affiliated link and making money off you whenever you're signed up and giving you the top recommendation to have to mention the best or top travel viewers. There's no problem with that. But what is troublesome is that they're not giving you any disclosure or not telling you that these are affiliated links and they're making money. So these are not honest and unbiased opinions and also they're not telling you that by signing up for their top recommended credit cards in case something goes wrong. Let's say you purchase travel with American Express Gold or American Express Cobalt credit card and your trip got canceled and your loss is $3,000. So these websites, they're not going to pay $3,000 due to the losses based on you signed up for that credit card based on their recommendation. They're not going to stand by you and they're not going to tell you that they didn't mention that at first and you signed up thinking this is the best credit card. But if something goes wrong, then you need to cancel your trip because this card doesn't provide trip cancellation insurance, so you're not going to get back any money. And then this credit card or any other website, reverse Canada or any other affiliated website which are not giving you the full truth behind any credit cards, they're not going to pay you back that money. So that's the point what you're losing. So let's say whenever you're getting a best credit card recommendations, they should give you all the good things and the bad things. Not only the good things, hiding the bad things. So that's the point where I had problem with these top or best credit card ratings. So that's one thing I wanted to show you and then the other thing I'm going to show you. Another way you can tell that these are, this website is promoting credit card companies because they're affiliated is by looking at a press release which you should buy American Express. And let me show you that one. Okay, so here is a press release which is, let me close my note here. So I just closed it. So reverse Canada names the American Express Cobalt card as the top overall card for 2018 news provided by American Express. So this is a press release issued by American Express. You can read the whole thing in order to do that just search for just go to the website link which I'll actually give you right here. I'll put it in the note again. So if you need to look at the look at this news press release, then you can just, you know, go here. So newsware.ca and you can search for American Express or yours Canada or you can just copy or you can just copy this link. Actually cannot copy but you can just, you know, do a search for by putting news where American Express yours Canada and you're going to get a hit. So I will just close it. So what happens here? This is a press release issued by American Express. And they're making it a big deal that reverse Canada chose their credit cards for six categories, top spot in three of six categories rankings. So here they're, you know, they're giving you all the details and everything. And they're telling you on top of cobalt, American Express cars on multiple hours in the multiple in the following categories. So American Express gold second place is star root. Fourth, American Express gold second place again, another category blue sky, another category platinum card is another category. And they're even giving you that for more information on the Rewards Canada top travel, yours great card, visit yours Canada.ca. So this is just a cross promotion. So Rewards Canada is promoting American Express, American Express is promoting Rewards Canada. So that's how it goes. So it's just, you know, these are not honest and unbiased best or top rankings. These are all just promotional rankings. So you're not going to get any unbiased or honest opinion from these websites. So that's what to look for. So if you are, if you want honest, unbiased opinion, go for small websites, individual websites which are not affiliated with any anyone. For example, I can actually show you my own website right here. So let's say if you go to my own website. And if you go to a credit card review. So let's go to credit cards, Rewards here. And I'll just show you one credit card. I have lots of credit card articles, but I'll just show you one of them, which I reviewed a credit card. So let's take a look here. Okay. Okay, so, okay. So here is TD cashback visa infinite credit card review. So let's go to that article. And if you go to the article, then you will see that I have a disclosure right at the top. Although I didn't have to, but I put disclosure mentioning that this is an honest and non affiliated review right here. So these are not compensated or affiliated with any credit card companies. All credit card articles are 100% unbiased. And if you go through the whole article, you're not going to find any link anywhere. So there's no link. If I giving you link, these links are just my to my YouTube videos and to my another page on the same website. It's not any affiliate ring. So that's what to look for whenever you'd like to find any trusted opinions or reviews on any credit card. And I'd like to show you something else here. Let's say if you're going to the problem with Canadian websites like Rewards Canada and other big websites who are reviewing or giving top ratings, they never mention or disclose anything that they're affiliated. So for example, if you saw through the whole website of Rewards Canada, you're not going to see anywhere that they're mentioning that these are affiliated links and they're affiliated with the company credit card company. They are paid by the credit card company. So that's a big problem because you're not getting the truth. Even if you go to the main page, let me just take you to the main page. Let's say we're going to the main page here and same thing. You're not going to find anything anywhere that mentioned that, you know, on this site has lots of affiliated reviews or anything. On the contrary, if you go to similar sites or big websites or small websites in the US, so these websites in the America, in the US, they're always disclosing the truth.  So you know what to look for. So you are getting the truth upfront and you know what to expect. And it wouldn't be a big problem because they're giving you the truth. For example, I'll take you to the pointsguide.com website, which is a big website in the US. There are lots of articles and top recommendations for credit cards and stuff. So let's take a scroll down towards the bottom of the website and I'll show you what I mean. So if you go here, so you keep going, going, going. And let's take a look here. Okay. So right here, the credit card offers that appear on the website are from credit, are from credit card companies from which the pointsguide receives compensation. This compensation may impact how and where products appear on the site. So they're giving you information here and they're disclosing that these are all affiliated links. And then if you go to any top cards or any listing or any page which has any credit card, then they're mentioning you on that page again. So they're giving you actually, actually, sorry, this is taking you to the American Express application. That's not what I want to show you. I wanted to show you another credit card review where they're telling you, you know, these are affiliated. But you can check it by yourself later on. So the pointsguide.com is the website. So you can see here here again. So each page everywhere you see that, you know, they're disclosing everything. I'll take you to another big website, which is million miles secrets.com. So this is another website where they talk about miles and credit cards. These are all US credit cards. So this is the main page. So same here. If you scroll down towards the bottom of the website, then you're going to see that there is a disclosure and which is right here. So many of the compensation credit cards appear on this site are from credit issues from which this site receives compensation. Compensation does not impact the placement of cards. So they're again, they're giving you full disclosure and you know what to expect. And you know that, you know, there's money exchange going on behind these rankings, ratings or reviews. And here, even if you go to any individual credit card review or top listing, you'll see the same thing. But in Canada, we never see that. They're never telling you. So that's why you have to be careful based on these to follow these recommendations or to listen to these recommendations. There's no actually 100% top or best credit card. Everyone's unique situation is different. So what credit card is number one for me is not going to be number one for you. For example, if you look at the American Express Gold Rewards credit card, this credit card is very good in order to collect points, especially if you're making lots of travel purchases. It offers two points for lots of categories. However, in terms of buying full travel like plane ticket, hotel or book vacation, this is not a good card because you're losing on full travel insurance coverage. Also, this card does not give you concierge service, which is a very big thing in terms of when you are arranging your book trips or travel. And if you have problem in foreign countries, concierge can always help. Another card is the RBC Visa Infinite Avian credit card. This credit card may not give you lots of points. It's not going to give you two points for travel purchases. However, it has full insurance coverage and it gives you concierge service access, which is Visa Infinite Concierge Concierge. So no credit cards are the same and it's almost impossible to give you a recommendation for top 10 or top best credit cards. So whenever you're choosing credit cards, figure out what you need, what you're looking for. Are you looking for points? Are you looking for travel insurance? Are you looking for concierge services? And are you looking for everything then make up your mind based on reviews or ratings from sites which are non-affiliated and provide honest and unbiased opinions. So that's all for now. If you like what you see, please give it a thumbs up and consider subscribing. And there will be more quicker videos coming up. So I'll see you shortly. Thank you.


In theme 6 in GRISP, we're all about what contributions we can provide as researchers to support the economic development of the rice sector as a whole. The extension, the outreach activities, the dissemination of new technologies and knowledge to farmers. And I think in 2011 we have particularly seen very rapid developments in the whole area of concepts and tools for using what we might call modern information communication technologies to bring those kind of knowledge solutions to the farmers at a much larger scale than we were able to do before. So one of the things that we are particularly excited about is first mobile phone applications for farmers. There are various ways of doing this as a farmer being able to call a toll-free number and by answering a set of 12 or 13 questions about their field and their management, obtaining a precise recommendation for fertilizer to apply or nutrient management in general. But we're also thinking now about these types of things for smartphones, so extension workers who could have applications on their smartphones, phones that would allow making direct decisions or decisions of poor farmers in the field, helping them to solve local problems, make more precise prescriptions to their own management in the field. And there are many other applications in the pipeline, also linking this to financial services and crop insurance. So we will see more of this coming but 2011 I think has been a groundbreaking year in that sense that we released the first of these applications in the Philippines in January and have since then seen a rapid evolution in this whole new area and one could call it a whole new way of doing agricultural extension.


Tom here from Lawrence Systems, and we're going to talk about hypervisors. Specifically XCPNG as it compares to some other options. Now, if you're interested in hiring us, head over to LawrenceSystems.com. If you're interested in furthering discussion on some topics, head over to forums.LauranceSystems.com, where I pretty much daily am in there in the forums replying and having fun talks about tech topics. Now, XCPNG has been around not as long as some of the other hypervisors, but its core has. And that is the Zen server in the background. So it is based on the Zen hypervisor, and then so is Citrix. Citrix, well, still is based on Zen hypervisor, and much of Amazon still is based on the Zen hypervisor. Now, this open source project, like I said, had been around a while, and Citrix took it and put their spin on it. So they created the Citrix Zen server. Then Citrix made a decision to change the license fees pretty much substantially on what they called an update. So you updated the system and suddenly you lost features, and they're like, yeah, the features you used to have in that last version are missing out on a new version unless you have license fees. Now, I have an entire detailed history, and I won't go into it right now, but I'll leave a link to the video I did where it breaks down the history of Zen server and XCPNG and how they came to be. It started as a Kickstarter campaign, and they decided to make the entire system open source. And companies are really embracing open source. I would say it's not like I would say any one particular year is the year that open source took over. It has been a slow, methodical growth, and a lot of it is because of companies misbehaving with licensing fees and people realizing that the critical infrastructure of their business is held ransom, so to speak, by arbitrary changes in the company going, if we charge more money, we'll make more money. That seems like a good strategy. So let's just charge more with, by the way, we're also going to remove some features, and we're not going to listen to you as a customer and your infrastructure is based on this, so you don't have any choice to pay. Spin it around, you look at a lot of the open source models where you get all the software for free, but then you're going, but how is someone going to support this software? Well, that's the open source model that, to me, works really, really well. And if you don't believe me, look at Red Hat. They were just acquired not that long ago by IBM in 30 plus billion dollar acquisition, and for a company that gives away software, it's something to think about, and they just sell support. And that's how XCPNG works. So let's talk about what it is and what it's not. What it's not is a partially open source solution. It is a fully, 100% free, costume nothing and licenses solution. And I think that's really important. That way, when you're setting these up, you don't have any worries about that.  Is this, you know, if I get it, well, what happens if there's a problem? What will I do? Well, between the forums and everything, if you're a home user, it's amazing the level of support that's offered for free. But I will cover this right up front, that they sell two different things. They have a whole support agreement here for their tickets and everything else, which, look, I didn't even have to call anyone. I have the price posted right here as of January of 2020, that it's $600 per host per year, which gives you six support tickets, SSH Pro support, one business response time, or $1,200 per host per year for unlimited support tickets, SSH Pro support, one hour response time on critical issues, initial setup assistance, et cetera. And this is one of the things I like about them right away. Now, I have this pulled up as well. This is the coordination tool, Zen Orchestra, that allows you to orchestrate with XCPNG. I look at them as one and the same. They are the same developers that run both sides of this. They sell the XCPNG, or I say, not sell, sell support for it, but give away the code and do all the development. And they also have all the open source for the Zen Orchestra. And that is the orchestrating tool by which you can manage it. And I'm going to show a little bit of this real quick in a second here. But this is really turnkey. And this is an important piece of it. So you have the hardware, you have the Zen server, then you have the management platform XOA. You can manage Zen server one of two ways, well, one of three, actually. You can do completely do it from the command line and script it. You can use a Windows-based management tool. That's free to download that they have. Or you can use Zen Orchestra, which is an entire web interface. Now, some people don't like the fact that Zen server itself, like the XCPNG tool, doesn't have a web interface itself at all. But I'm actually fine with this. And the reason why is because this allows you kind of a one to many relationship. So Zen Orchestra can connect to many, many Zen servers. And then you can coordinate separate pools of Zen servers and manage them all through one web interface that lets you consolidate them. And we have clients that are doing this. We have clients with data centers and they manage each stack, each group of hosts in Zen Orchestra. And then they have several groups of hosts which are referred to as pools in the Zen system. So one Zen Orchestra can manage many, many pools with one install in one instance. And it can do this over the web interface, even has the ability to do this across a VPN, if need be. There's all kinds of different use cases you can use for it. Won't dive too much into it. I have plenty of videos on how that works. But the other thing that's really nice, is the penetration of management, backup and disaster recovery and cloud enabler. Now the backup and disaster recovery is an important thing because a lot of people say, well, how do you back up Zen server? What do you buy this third party utility? And a lot of companies that make hypervisors, they have relied not only on selling you a hypervisor, but then rely on another third party company to sell you a backup solution. They have a full backup disaster recovery solution that allows you to back these machines up, back up them to standard open virtualization formats. Actually, you can export these in and out. You can save them to external drives. You can then figure out how to drop them over on whatever cloud external provider you want without a problem at all. This is a really nice feature. When you look at it compared to some of the other systems, because this gives you turnkey between running Zen Orchestra and the XEP and GPRO. Now you're probably thinking Zen Orchestra costs money, but like I said, there is a fee if you want the full system. So if we go to Zen Orchestra and we go to pricing, free. You can play with the free version.  If you're a home user, you can try the free one or you can even grab all the source code. They have an entire guide on this that lets you grab the source code and compile it yourself and you completely will be able to do all the features. They don't hide anything. It's all there if you compile from source. I have a few videos on that. So that encourages home users to do this. So once again, you're not locked into a license fee. These are support fees and that makes it a really good choice. This is one of the reasons I like ZenServe so much. Now let's go back over here and talk about some comparisons here. So let's scroll down and XEP and G versus ZenServe are free and XEP XO versus VMware. This is a couple of the comparisons are going to be one. Why would you use this over Citrix? Well, the license fees at Citrix are ridiculous and confusing and I'm not sure why Citrix does some of the things they do but that's fine. Good news is if you are a Citrix user, you can install this right over the top and import all your stuff without a problem at all. There's 100% compatibility because they're all running in ZenServe and we've done this for a few, well, more and more clients all the time. We've installed over the top of a Citrix install and everything brought right over into XEP and G. I still recommend backing up first because things can go wrong but it does work. So here's all the features and I'm not going to read through every little piece but a couple of the people asked me commercial feature-wise, it does have SRIOV networking and it does support GPU pass-through with high-end graphics cards like some of your high-end Tesla ones and things like that. If you're a person who want to build your home lab and you want to talk about GPU pass-through, you probably are talking about more of a consumer-grade system. You can do that from the command line. It's not exactly exposed in the full system. So it does exist. It is a possibility to do it but it's not exposed to the web interface in there. So let's also talk about directly with VMware and a nice little chart they have here. I'll leave links to all this.  So when you look at the two together, and like I said, I combine these into one but they do separate out because it's delegating where the functions are. Web console for VM management, basic VM admin, live VM migration, live VDI migration. High availability. I've even done videos on each one of these topics of we even moved a phone server with active phone calls on to show the V, live VDI migration between servers is, you know, as smooth as it is on other platforms and I can not drop a phone call when I do it. So I have videos on each of these topics. You do have a console view, hyperconverged solution. Now I touched on this briefly. I'm not going to dive into it in this particular but they have Exosan and Vsan are the two different hyperconverged solutions for some of the storage. So that does exist in there. A little bit more enterprise level but it does go to that scale of enterprise level. Tool updates, automatic updates. This is kind of cool. When you have the foals and orchestra, when you compile it yourself, you're going to update yourself but if you pay for it, that's one of the things you get is the full service in terms of they do all the updating for you. That's what you're paying for is that automation level. Thin provisioning is supported on both platforms. Health checks, role-based administration, ACLs, single sign-on, self-service, scheduled tasks. You can schedule the moving, starting, stopping of VMs. There's a lot. As a matter of fact, because this is all written, it's all Linux on the back end with a really nice web interface in the front end, you can do things from the command line and integrate them into cron jobs or you can create in the interface an entire schedule to do things including not just that high availability but load balancing when a server becomes under too much load. You can then tell it under those circumstances move those VMs to a server that has less load. That's completely supported in the all free version by the way here. None of these are paid things you get. Basic backup. Now the backups are really extensive in here. This is one of the things I like to touch on. So yes, you can do a basic backup but if you want to do delta backups, that's completely a possibility. File level restore is actually a pretty cool feature where you have it configured properly. You can not just grab that old VM. You can pull a file out of one of the old VMs without having to do a full VM restore. It'll let you do that. Continuous replication and of course the load balancing like I had mentioned. So file level restore and delta backups are great because if you're going, well, I want to snapshot my VM but there's only so many megabytes available for me to upload this and how do we do that if you're saying it's an integrated backup solution? Delta backups, you're only synchronizing the changes. So once you do that initial VM upload and you backup the directory where the deltas are created, no problem. It's a very small matter of fact. They're so small and so fast we use delta backups to hourly backup all of our machines here at the office and all the virtual stacks we run and we set this up for clients as well. It's a really nice solution so you can keep a lot of iterations without going through and having an entire, you know, separate backups and separate tools and separate licensing, et cetera. It's all done through one single pane of glass. Now, the last thing I'll cover a little bit is I'm going to talk about the Web interface on here and I'm going to open up both versions just to show you they're the same. The one I compiled from open source and the one that we have for that was the paid premium. So, right here is the open paid premium, sorry, paid premium one. It's completely up to date. I can go over here. I can set up jobs, backups, all the features, blah, blah, blah. Everything's in here. I can import a VM, import a disk, create new storage, et cetera. We go over here. Matter of fact, well, something a few people have asked me for and it even has the whole hub. They've got templates in here and things like that to start even loading. You want to load PFSense in here. Click install. It'll grab and download and install it. WN10, CentOS. They've got a lot of cool integration plus you can import standard formats. You come over here to the compiled one that is 100% open source. Here's all the features and here's all them working. The only thing that's missing right now, I never compiled the hub features in because you do have to pull some of those in, but I still have the new VM storage. I can still import VMs. I can go over here to the backups. You can see stuff working. You can see the delta backups that I can do. This one failed for a reason. It's got a whole notification system in here for when it failed. This is something I did when I was moving a server. It's all in here and works perfectly fine. I do this to show that without any license fees all this will completely work. But because I like to pay support, we encourage our clients that we set these up for. They frequently want to buy support packages. We do encourage them to buy, especially when you talk about a host support package at like $600 a year. That is so inexpensive compared to the cost of a lot of other hypervisors. The last little thing I'll cover here, right here is a lab server and we even have one of our other demo servers when we set them up. You can just go in here inside of XTPNG, connect to the lab server, and now we go over here to the hosts and now I've connected more hosts. This is that idea of from one control plane I can manage many hosts. These two servers are in a host I call the puddle because it's a pool and there's only two of them. That's my silliness anyways for those asking why I call it that. This is their lab server. We do more of our testing. Once again, this is just a standalone server that I can quickly attach and spin up here and then I can now from one control plane control all the VMs and without them being in the same pool I can even migrate VMs between any connected hosts. When you see the other hosts in here, for example, these initials represent a client server we set up on our network before it was deployed. We did all the migration form and did all the testing when we built their server. I did a review on that particular box. It's a DDEL server we loaded XTPNG on. We just attached them really quickly here to do all the testing setup, maybe move certain servers and build them on there as needed and then we deliver them to the clients. This is the nice thing about this and then if I connected over a VPN I can actually reconnect and orchestrate from here but generally our process is to install the XTPNG and Zen server directly on their machine so they have their own control plane and they have an internal IT department that's going to be managing it so they wanted control plane done internally but the possibilities are there. The power of it, it's a fully, like I said, open source which means you are not beholden. If they decide to, they're not going to change something on you like Citrix said, well, yeah, your license that we used to give you which Citrix had a free license, they changed the terms and conditions on you and removed features because they said, well, that's what it says in our license, we can do that. This is all done under full GPL. All these features if you wanted to compile yourself because you want nothing to do with that company you can do an external server and we're even running into some consulting work we've done with companies that do type of military work and one of the things I thought was really interesting is they just don't like the fact that so many things call out to licensing servers all the time that's kind of a, you know, something that bothers them a little bit, they're going, you know, our infrastructure is very dependent on these third-party companies who license and those activations constantly call out, it kind of makes us feel like we just don't own it all the time and in some ways they're kind of right they don't because if they don't renew the license they will expire and when you talk about something like ESXi it's not a bad platform it's been around for a while it's a very mature, well-developed product it has massive market saturation but it they charge dearly for it and I have, you know, I've had some people love it, some people hate it, some people have hated the problems they've had with it and tit for tat feature-wise XC PNG is not only ready for the enterprise it has been deployed in enterprise we have done consulting with so many companies that I cannot, there's not a lot of people who just close names but I have seen very large companies that are all using XC PNG for their systems. Frequently, almost every time we see it is because they are already familiar with the Centric Zen server market and they wanted to switch over to XC PNG because the same thing, they were angry at the license fees, they said the support was poor, the licensees were high and it's a bad combination for things especially when Citrix raise the fees and remove features and want even more fees for features they are used to having and not want to patch and upgrade. The folks over at XC PNG, you can hit their forums up, you can see that they are very actively developed and they've listened to the crowd there's been features finally added that people said, wow, I asked Citrix for this years ago, they never did it, XC PNG has built in many, many more features and you know, keep up with their blog you'll see they are adding a lot more things all the time including a lot of private networking and systems including the way to take servers that are maybe remote to each other and create GRE tunnels between them so you can have shared private networks between them, there's a whole other project they're doing to expand that capability for building out your infrastructure which once they, once they get the time to do that I'm going to do some videos on that topic as well so they've taken it and gone further than a lot of other things do and it's all integrated these aren't third party plugins, these are all integrations they have which back to the original thing I said they're all open sourced and license free so if you're a home user looking to start out on this I have a lot of getting started videos on this I'm going to do some new ones because we're at the version 8 my last ones are based on 7 but they've only added features, everything I said in my version 7 getting started video is still relevant so this is the reason why for people always ask why I use XCPNG why I like it, we've built a lot of our stuff on there, yes I've looked at some of the other ones the runner up for this is probably going to be because someone's going to ask would be Proxmox, I think it's a good solid system I don't feel I've seen any Proxmox systems that scale this way but Proxmox is still a great option I don't have anything that I would tell people to use it, we have clients that have it in their back end, I'm not the most familiar with it but they seem to really like it it's a solid platform as well so I'm not going to downplay the fact is but my choice is and for the consulting work we've done and the integrations we've done for some and seen some large systems where we've helped migrate you know P2V migrations we just did another one so we physically moved all the servers on there XCPNG has been my go-to it's been a really solid performer and having everything from backups, turnkey all built in to one system easy control plane makes my life really simple and that's one of the reasons we like it alright, thanks and thank you for making it to the end of the video if you like this video please give it a thumbs up if you'd like to see more content from the channel hit the subscribe button and hit the bell icon if you'd like YouTube to notify you when new videos come out if you'd like to hire us head over to laurancesystems.com fill out our contact page and let us know what we can help you with and what projects you'd like us to work together on if you want to carry on the discussion head over to forums.laurancesystems.com where we can carry on the discussion about this video other videos or other tech topics in general even suggestions for new videos they're accepted right there on our forums which are free also if you'd like to help the channel in other ways head over to our affiliate page we have a lot of great tech offers for you and once again thanks for watching and see you next time


I'm Nick Brophy, the Director of DAV, Patriot Boot Camp, and I'm here with two of our boot campers, if you will, Joe and Kelly Thompson. Tell us, Joe, Kelly. First of all, how is it so far? That's fantastic. I'm completely blown away by the size and scope of this place, and everything that's been given to us so far is unbelievable. I can't think of everybody enough. Good. Kelly, anything you want to add? We're surrounded with subject matter experts, and I've already learned so much. I'm grateful to be here. And they're very attentive too. That's probably the most I've heard them say since they've been here because they've been taking copious notes. But we want to learn a little bit about you all on the DAV. I'm sure the DAV family wants to learn a little bit about you. So a couple questions here. Military service. Both of you are veterans. Can you tell us a little bit about your service, where you served, the years, what branch, and then what kind of jobs you were doing? Kelly? Sure. So I was active duty in the Navy station in San Diego. I did five years of service, got out in 2017, and I worked in the supply department. Excellent. Same thing, 2011, 2015. Four years Marine Corps, stationed in Pendleton, right by San Diego. I was a glorified gas pumper, bulk fuel, and it was an awesome four-year stint. Great. And I was four and done. Well, thank you for your service and one veteran to the next. So how did you guys get connected with us? So when she goes to bed, I usually can't sleep. So I was just typing online business ideas, like business help, and just going down a rabbit hole. And then I just wound up typing in veteran. And then I saw Patriot Boot Camp and I was like, that is exactly what I'm looking for and what the DAV provided and the registration and everything. It was just exactly what I was looking for. Yeah. We're always looking for resources, especially to help us as veterans. I feel like that's a unique niche for business owners and entrepreneurs. And this was perfectly fit the criteria. Great. And for those of you out there in social media land, it's interesting that these two actually applied with two separate businesses and have, I think, combined forces since that application process, right? And what's the name of your business now that you both are working together on? Yeah. We are Ethos Administrative Group. So we're a virtual assistant service for businesses. Awesome. All right. So why do you think veterans make good entrepreneurs? Definitely, persistency. We refuse to fail. And if we do fail, we're going to pivot.  But, you know, just like a battle mindset, you're not going to lose because, you know, you're used to losing. And if losing is death, we're not dead. So we're going to keep trying until we win. And that's pretty much what separates entrepreneurs that fail and entrepreneurs that don't. It's just keep going until it works. Spoken like a true warrior, Kelly. Yeah, you know, we're committed to excellence. So we want to provide that for our clients and, you know, internally as well. And I think that's something that we have been, you know, establishing since we started. Yeah, good. All right. Well, as a married couple, and I think our only married couple of this particular cohort, tell us about something that you may have learned about the other, whether that's with regard to business. It's always interesting to hear that we're learning about one another all the time. I would say this definitely heightened a little bit more of the mindset that we both have when it comes to business specifically. She has a whole different outlook on everything business wise than I do. So when we both take in separate notes over and over, then we combine them together. Say, like after the fact, it's just an explosion of like ideas that come together and something that I would never imagine like if we weren't here specifically. Yeah, you know, we also are parents of three, so we combine forces at home and it's a whole nother specialty when you combine forces in the workplace and really using our strengths and weaknesses, you know, with each other and it's been great. Good. Sounds like a nice complimentary group. Well, I know we still have a little bit left in the rest of the program. I don't want to take you too much removed from that, but I appreciate all of your time. Thank you all both for being here and trusting in DAV Patriot Boot Camp to lead you into the future. Thank you.


Well, today we're delighted to be at Terry Fox Secondary School to talk about how we're going to better connect trades training programs for students in British Columbia. It doesn't get any better than this. This is the culmination of three years worth of work to be able to roll out our new and expanded youth programs, putting the $7.8 million of additional funding that we got from the ministry over the next three years to expand and rename the programs at the same time to discover, explore, train, and work. I didn't know all these things that they were opening up for future students and I think that's an excellent thing to allow students to actually get a feel for the trades and to be able to see what opportunities are out there and I think it's an excellent way to get students involved in trades and show them where your life can end up. Just like the changes we've made to the BC curriculum, we want to be sure that the system of trades training programs starts early, as early as grade five, and that students have the choices they need and the information they need to make a choice to be a skilled trades person here in the province.