Okay, do you guys hear me? And I see you can see my slides. That is wonderful. The only thing I'm missing is water, but okay. I guess we'll have to do without that. So, we'll talk about MySQL sharding today. So, anybody knows what sharding is here? Okay, some of you do, right? Anybody thinks sharding is painful? Oh, okay. That's good. So, first let me mention a few words about Percona and why really we are speaking about that. So, we as a company focus on really helping our customers to succeed with MySQL and more recently in ADB. We provide a whole bunch of services running from support to managed services, and we write a lot of software for MySQL. Colin has mentioned some, right? We have Percona Server, Percona X3B cluster, as well as tools for backups, Percona X3 backup, and Percona Toolkit. There are, I think, two things which are important about us. One is all our software is open source. We are not having some enterprise version, which is closed source, so there is some other way to require you to pay if you want to use that. And we are also really vendor neutral. That is not our goal to really make you to use Percona software. That is our goal as a company to really provide you the best solution for yourself. And, frankly, that's maybe MySQL, that's maybe the Percona server, that's maybe MariaDB or Amazon RDS. If you are happy, we are happy. So, let me start with the story when I talk about Sharding. A couple of years back, a customer comes to me and says, oh my gosh, you know, I read this all wonderful stories about Facebook, their Sharding, and there is a Dropbox Sharding, and there is everybody in the world is Sharding. So, can you guys please advise me how to Shard? I say, hmm, okay, let's look at your system first. And I would ask you what exactly is your database size, right? Probably it's like, well, it's about two gigs in total. Hmm, interesting, right? Do you guys have a lot of traffic? Actually, no, it's about 100 queries a second. Well, maybe you just guys just launched and you're planning to grow approximately 1,000 times in the next three months. Well, no, actually we've been in business for about five years. We're growing about 7% a year and we're expecting that for the next year, right? Well, wonderful story, right? And, well, the case in this case is obviously with such a small system, oh, thank you, you don't really need to Shard, right? Because even if the Moore's Law is kind of, you know, kind of getting dead, right, they're still expecting performance improvements in the system much more than 7% a year, right? So the outcome for that is before you really decide how you're going to Shard, you guys really need to decide wherever you need to Shard at all, right? And I think that is a very important question because if you guys go to different conference, read a lot of famous bloggers, a lot of them work for really large and famous companies and they are often going to talk about how to operate MySQL on extreme scale, that may not be exactly your case and in your case the different principles may apply. Now, what is interesting is with modern technology you can really go quite far with our Shard, right? And I think with our semi-obsession of distributed systems in the recent years, we may forget how much you can actually get from a single simple MySQL box, right? And let me throw out some numbers. From a single MySQL server you can get more than 100k queries a second. You can get more than 100k rows updated or inserted a liter per second. If you're going to some data kind of scanning queries you can get probably more than 5 million rows in memory. MySQL can deal with 10k of concurrent connections and if you are having some good box with a good storage and networking you can actually have 10 terabytes or so on a single MySQL instance, right? And this is not the marketing data and it's not extreme, right? So if you scale the storage, that's one of the things which I hear a lot of people tell me really. Do you really have anybody on the planet having 10 terabytes on the single MySQL instance? And I'll tell you, yes, I've seen many people doing that and I have seen people up to about 50 terabytes on the single MySQL instance, right? That is kind of extreme and I wouldn't quite go that way but I think that gives you a good perspective. Now, guys, these are not some extreme marketing numbers, right? These are something that you can get from a pretty mid-range system even in the cloud, right? And for a change, I'll show you some marketing numbers, right? So I stole that from Oracle. Where is Dave here, right? Dave, he ran away anyway. So I hope nobody is watching. But anyway, this is the marketing slide which is actually also true, right? But it is on a larger box with relatively simple query but you can see we are getting well over half a million queries with MySQL 5.7 on the high end, right? So 100,000 queries is quite conservative. Now let's do some math, right? So today we have sort of a system and we are planning for maybe 3 million active users and those users have 30 interactions per day and with all our kind of fault we can kind of adjust an interactive probably for single interaction you are going to have unlikely more than 10 queries, right?  I was talking about the wrong slide, okay. Anyway, that's the math I was talking about.  Oh, I did twice.  Okay, anyway. So if you do this over math, right, what do we come out to? It is about 30,000 of queries a second, right? So really it is not a lot, even with those quite large numbers of users, interactions, and so on and so forth, right? So if you look at examples, the companies we worked with, which avoided charging, that's quite range a lot, right? We had worked with pretty large enterprise household name here in US, but the more household name it is, the more they don't like you talking about them and naming them, right? I don't know why. But anyway, so those guys are using Drupal for everyday workflow for everybody for 200,000 employees, right? Can use Drupal, no shrouding require, MySQL server loaded about, I think, 3% in average. Of course, they would use some replicas, right, for high ability purposes, but from performance alone, you don't need that. We also worked with a number of e-commerce merchants, right? And some of them are selling well over the $1 million per month, so that's a very pretty serious e-commerce businesses don't need any kind of shrouding. So, as you guys already told me at the start of this talk is charging is a pain, right? Yes, well, and I know some of us like pain. Well, maybe many of us like pain. I'm not quite sure what kind of audience is, but, well, pain is not always good, right? Let's all agree on that. When it becomes too much pain, we still stop liking it, right? So, in a lot of cases, even if the sharding is pain, and in a lot of cases, you want to either delay it or avoid it, right? So, what kind of pains do sharding bring to us? Well, a lot of that has to do with complexity. Complexity for developers, right? Now I have to, instead of querying one system, I have to figure out where my data is, and again, even if it's sharding done automatically for you, if you're a good developer, you still have to think about that because your performance profile is going to really depend on the data distribution. There is more complexity from operational standpoint. There is more kind of complexity in technology because you have more bits and pieces. You have to deal with much more complicated failure scenarios compared to just single box, essentially, which works or doesn't, and much more complicated performance profile with bottleneck network involved in much more sense and so on and so forth. Now, MySQL sharding is frankly especially painful because compared to many other systems, like MongoDB in a good example, MySQL sharding is very manual, right? MySQL came from an era when this kind of many service operating together as a single entity wasn't really a thought, right? And it was after thought, it was bolted in, bolted on, right? And in many cases, it was bolted on by the users, right? Let's say Facebook would bolt on their own sharding, right? Or the tests, which you mentioned, YouTube, their own, right? The guys like Oracle or MariaDB working on MySQL Fabric, MySQL Router, MaxScale, all those things have came much, much later, right? So if you can't avoid the sharding, you can often delay it, right? And why can't you avoid the sharding? Well, frankly, even though the modern systems became much, much more powerful and can handle a lot of beating for a single node, you can't build a Facebook on a single node, right? Or even a system which is probably 1% of a Facebook size. So, but in those cases, such systems are typically not born overnight, right? And it can take quite a while to get there, right? And you may not need to shard and waste a lot of your development and operational resources on dealing with that issues until you have much larger resources and much larger development and operational teams. So what few strategies do we have for avoiding the sharding, right? Or delaying the sharding? And they go across those several dimensions, which is using architecture, functional partitioning, replication, caching, queuing, and using some what I will call supplemental technologies, right? Let's go through them one after another. So when you look at the architecture, we'll talk about a lot of architecture questions in other top areas as well. But what I think is one of the changes in architecture recently was how people start to evolve to either microservices, each of owning kind of their own data, right? And maybe actually using different data storage technologies, right? On the back. Or some people would call it not quite microservices, but service architecture. But in any cases, you essentially do have a certain number of blocks rather than one monolith system, right? Which have different data stores, right? So the load and amount of data each of them have to store is different, right? So for example, you can say, hey, this is my interface which provides, let's say, user profile services, right? And this is maybe my billing system. If they leave in a different... If they design these different services, they probably don't have to live in the same database. Now, this also can be seen through an avalanche, which I call functional partitioning, right? And what that necessary means is what we want to keep our separate data separate, right? Think about, I don't know, even such a basic side as a per corner, right? We have a lot of different bits and pieces out there, right? We have our blog. We have our Drupal-based website. We have, you know, some other components. Really, each of them are quite independent, right? And they live on the separate databases to start with. I think it's important in this case to really maintain them to be independent and not accidentally mix them when you don't do that, because then if you need to scale and separate them on different systems later on, that's maybe hard. To give you one of the simple practical advice is if you have systems which operate in the outside on a different databases, give them different MySQL users with different permissions, which only have access to the databases they are designed to have access for, right? So that means, for example, if your main website needs an access to the content in Drupal, that wouldn't happen by accident. When you say, oh, let me just move those to, let's say, blog and Drupal website on different systems, you have things broken down because there are some joins or anything else you didn't expect. Okay, replication, right? So anybody here use MySQL replication? Yes, well, I think the MySQL replication is probably by far the most popular strategy both for MySQL high-vibability as well as for MySQL scaling, right? What replication allows us is to mainly scale reads. But what is a good thing is in the web applications, reads are typically a very large portion of a workload, right? Majority of them for many applications. What you need to be aware about MySQL replication? What MySQL replication, built-in replication that is, is asynchronous, right? And don't let words trick you. Semi-synchronous replication is also asynchronous, right? So semi-synchronous is not synchronous, that is actually asynchronous with some better recovery properties. If you're looking for other MySQL replication options, I would consider Perconex Ruby Cluster or other Galera-based solution. If you are the Amazon Cloud Fun, the Amazon Aurora also does its own unique replication approach to MySQL. The next trick which is very powerful is using some sort of caching with MySQL, which also allows us to scale reads, right? So anybody used things like memcache here or radius maybe for caching, right? Or if you guys have your application reaching in Java, you may actually cache it just in the application level, right? Not to go to a database all the time. Caching is really, really powerful, right? And I've seen applications which through use of a caching alone would able to reduce the traffic to MySQL by more than 95%, right? So that can be pretty cool. So MySQL also built in query cache. But the thing what we know about the query cache in MySQL is what it talks. I mean it was designed in MySQL 4.0, right? It did not get any attention until now. It doesn't scale very well both with many CPU cores as well as the large cache sizes, right? So if you have some really, really bad application, some really, really bad queries and low concurrency, you actually may benefit from MySQL query cache. But most high-end installations have to disable that. The interesting thing about the query cache show that is one of the things where Amazon Aurora seems to have implemented a number of changes. Amazon Aurora query cache is much, much better than MySQL query cache. And frankly, I believe a lot of benchmarks published. That is really the cause, right? Just much better query cache. Maybe this kind of a competition between the giants of Amazon and Oracle will make Oracle also interested to make some better query cache-like solutions for MySQL, too. Well, let's talk about applications, query cache, memcache and radius. Two other things I think are quite important to know. One is you often can cache MySQL and MySQL itself. Building with summary tables for some very heavy queries instead of running them all the time, right? You can call it like materialized use, if you like, which are like glorified summary tables. It can be very, very powerful optimization techniques. And now I think which I know a lot of DBAs or database developers don't think about is things as HTTP cache, right? In a lot of cases, caching on HTTP level is absolutely the most powerful, right? Because frankly, the best way to give your customer the best performance ever is if he doesn't even have to go from his, you know, system, right, to request a refresh an object. If it's cached right there in the browser memory, right, over disk space. And you can often achieve that for a lot of data by implementing proper HTTP caching policies, right? So, read on that. Queue in, that is another kind of big secret. If you look at that, any large scale system out there does use some queuing. At least for some of the things, right? Because of many things, right? Queuing helps us to scale writes for a variety of reasons, right? But it also allows us often to balance the spikes, right? Think about that. If I go ahead and something fun happens and I upload the spike of, you know, maybe 10x more YouTube videos, right, which needs to be transcoded than average, well, it would be overload the system trying to transcode them all at the same time in real time, right? By just putting them in the queue and then getting to them when you can get with your resources, right? Maybe doing some dynamic scalability on the background. That is much, much better architecture. And that also applies to a lot of the database things, right? Let's say if you build in something like Twitter and you have a lot of followers to deal with, it's much better idea not to try to put all of those notifications in real time, but put that to the queue, right? And then do whatever you have to do in the background. So there are a number of systems which we use for queuing. It's Revit, RevitMQ, ActiveMQ. Radius is very popular. And what we see a lot, getting a lot of traction those days, especially for certain use cases is Kafka. Anybody use Kafka here? Okay, well, great to see fair amount of hands. Now, the other thing that I mentioned is use different technologies, right? Well, I hope that's not a secret for you, but the MySQL is not a silver bullet, right? Nothing else. And as much as it's great for certain things, it absolutely sucks for a lot of other things, right? And frankly, it's very interesting for me being involved in MySQL for so many years to look back and say, well, let's say eight years ago, we will do something like we'll try to write apparel scripts which will go through, let's say, 10 slaves or 100 slaves, right? Run some queries in parallel and then aggregate those results of the group buys and all the other crazy stuff, right? And then what happened? Well, actually, we decided what those things are much better done on Hadoop or Spark, right? Then you need to crunch through a lot of data in parallel. Just don't do MySQL for that, right? It will be just pain and probably little gain. There are also technologies for full tech search. MySQL 5.6, 5.7 have a full tech search. Thankfully, it works in a DB now, right? So we don't have to pick between transactions or full tech search abilities, but it's not really scalable solution, right? It's what you can call like a low-duty, right? If you're implementing search for home DVD collection, right? Or something like that. If it's not overloaded, you can use MySQL full tech search, right? If you are building the search for many hundreds of gigabytes or terabytes of data, you will be much happy using elastic search, Sphinx, Solar, right? Or some other custom-made full tech search technologies which do that in parallel. And you will be also able to have much more flexible search with much better search quality. There is also selection of document stores you can use. There is also Cassandra, which we saw for some very simple but very data store, but very scalable and reliable one. It has been amazing. The next thing to consider is what you actually want to consider optimizing your system before sharding, right? I don't know. I see a lot of people who are being kind of lazy or they just like to go to a boss, ask for a lot of hardware, and then kind of play with distributed systems instead of just going and freaking tuning the indexes, right? I don't know why. But that is what we see the time and again. So what you can do, and you should do in terms of optimization, hardware, think about getting fast CPUs, right? Especially because MySQL executes single query in one core, don't buy into advertisement, oh, it will give you CPU with 14 cores, but they all will be less than two gigahertz or something like that. For MySQL, that doesn't work well, right? MySQL likes faster cores, and for more than CPUs, you essentially want to look at a turbo boost frequency, right? Not a phenomenal one because that is what your CPU is going to run at when you're just running one very, very heavy query, right? You want plenty of memory because accessing the data in memory is much, much faster than going to disk. And, guys, please understand, that happens even if you're using very fast storage. The fastest fusion IO, right? Or some NVMe storage is still going to be orders of magnitude slower than just going to the memory to fetch that, you know, 10 bytes you need from a field in the row. Fast flash storage, right? So, solid state drives. I mean, for operational databases, you should not be running anything else, frankly. Anybody else is here suffering with spinning disks? Why? What? Oh, okay. Okay. Well, so, I mean, I know in some cases we have some issues, right? I don't know. It may be politics. It may be what somebody doesn't believe what SSDs can be reliable, you know, 10 years later, right? It could be what they just spent all their storage budget for the next 10 years buying EMC for $10 million five years ago, right? Or some stuff like that.  And you also want to have a good network, of course, right? A lot of the modern application that comes to MySQL can be dominated by network through latency, right? Because I recently looked at, let's say, how much... If you look at the gigabit network, right, for example, often you will have... What's going to be around you, right, in terms of in the data center? I think it's probably about 300 microseconds, right? Now, if you look at the MySQL, it can really process the simple queries typically in less than 40 microseconds, right? So if you think about that simple selects or something, latency is going to be dominated by the network latency, at least with one gigabit network, right? Another important thing about the networks is to think about the distance and number of hops. I have seen so many people making a mistake of placing MySQL server and app server very far, right, from each other, either distance-wise or just having a lot of network equipment in between which can add up a lot of latency, right? If you go through a few routers and so on and so forth, it's all about the environment. Well, I probably would be preaching to the choir in this case, but I think Linux is a pretty good operation system to run MySQL, right? Would you agree with me, guys? Yeah. Right? Yeah. And actually, that is also the more popular ones, right? I think a lot of large-scale deployments are done on Linux these days, right? I see, I think, a little bit more, actually, three BSD deployments, right, which start popping up over the last couple of years, but Linux is still absolutely dominating. New MySQL versions generally scale better, not surprised, but scale better with multiple connections, with many CPUs, with more complicated queries because of Meriboy optimizer. But for simple queries for a single thread, MySQL has actually become slower and slower over the last releases, right? Last benchmarks I've seen, MySQL 5.7 is somewhere like 15% slower than MySQL 5.0 running single thread benchmark like SQL Bench, right? You can look up Mark Colligan has wrote about that. MySQL 5.7, you guys heard about that? That's a pretty interesting GA release. I see there's a lot of still things kind of in flux, there's some changes implemented, right? So maybe I would not just ration up great to 5.7 today, but wait, the dust settled. But if you guys are starting the new development and plan to go in production in a few months, 5.7 is a great choice. And we are also working on the Percona server, 5.7, which should be out sometime in February, and we have a release candidate available already so you guys can check it out, give us some feedback. Configuration, right? Some people may disagree, but MySQL default configuration sucks. And it is by design, right? Because really MySQL was designed something like you install new server, it sits out there in the corner, not taking a lot of resources, it can run some applications, right, and so on. So it was not designed as to go in and kind of take over the source of the box and not get anybody else to run. What that means is what you want to tune MySQL server, right? And here's a link to a webinar I did on this topic. It's not rocket science. Typically you get like maybe 10 options you tune in MySQL and you get it 95% where it needs to be compared to defaults. Storage engine, right? When it comes to storage engine, INNADB is a default storage engine and it is absolutely fantastic for a lot of workloads. In some cases, you may check out where TokuDB for high inserts and if you want to get some much high level of compressions in INNADB, you can explore that. Anybody try TokuDB here? Okay, well, I see some hands. That's good to hear. I have a feeling for sharding. So when do you want to shard? Well, here's a challenge we have, right? If you shard too early, we are likely to waste our resources because development and operations are going to be more complicated after we shard it, right? But if you shard too late, you can run into a wall, right? Because your application just unable to handle the performance required on that. So in this case, what I like to be thinking about is the term called architecture runway, right? Sharding is one of the architecture considerations you guys have and think about this way and take it as a part of your architecture runway planning with all the other things you guys consider as a part of architecture thing, cleanup, technical depth, other optimizations and so on and so forth. And what you need to understand in this case is how long it's going to take you to implement the sharding. Depending on your team pace, depending on your application complexity and so on and so forth, it may vary a lot, right? Like, guys, I can honestly tell you people have come to us, asked for sharding advice and tell me, oh, that's fine, we'll go ahead and implement over weekend.  You're not going to implement over weekend. What they actually did, right? They had the application simple enough, small, very agile team which will go and implement sharding over weekend in the application, right? On the other hand, I have seen some other legacy application which was written for, you know, 10 years, ton of queries, complexity. It may take a year or more really to implement the sharding, right? And I think that's important to make yourself an honest assessment and balance with your capacity how much you guys can run your current application, with how long it will take you to implement sharding, right? So that brings us to the capacity planning, right? So for us not to run into a wall, we need to know where that wall is, right? Makes sense? So we want to figure out how we can do that, to understand that we want to do some estimates, maybe do some benchmarks, be conservative in this case because it's much better to predict a wall 20% lower, right? Than 20% further down, right?  And one very important thing is do not ever plan for linear scalability, right? So if you're looking, let's say, at your system and saying, oh, well, currently I see only 10% of the CPU used. So that means I can handle 10x of query on this system. That is a very, very bad math, right? That's not going to happen for many reasons. Now, I have been kind of very critical of sharding, but I have any benefits of that, right? Well, and as if almost everything in this wall there are always like positives and negatives, right? If you look hard enough. So what are the positives for sharding? Well, sharding is really your road to the ultimate scalability, right? If you guys are building something on Facebook to scale, then you will need to shard sooner or later. And sharding may allow you to reduce some other complexities, right? Or eliminate them. So for example, I have seen people saying, hey, you know what? Yes, we do some caching, but you want to avoid having that extensively complicated caching level, right? With all those kind of invalidating cache properly, right? Our developers don't like that. It's complicated. Well, we just better buy the bullet ones. We shard it. We'll have most of our data in MySQL memory. And accessing data in MySQL memory is pretty fast as well, right? You guys have seen the numbers of more than half a million in a second. Asynchronous replication. That's another one, right? I have seen a number of customers saying, oh my gosh, our developers they absolutely hate dealing with asynchronous replication. Because it has this property of it works most of the time, right? I can go in most cases on my slave and get the data, which will be reasonably accurate. But sometimes I get just some silly stale data. Developers don't like it a lot because there is no built-in guarantees in the replication about the data stainless which it will be to provide. So in many cases, we see people just sharding when they say, hey, we read from masters, we write to the masters. Our slaves are used for availability purposes and for some other applications. For example, some reporting. Because reporting, I know reading a little bit stale data not a big deal. Sharing also helps us to have very security and compliance as well as possibly having the data closer to users. So for example, a lot of software as a service applications often would shard things to really separate the data between different customers completely. Different customers live in a different MySQL instances so even if developers do some mistakes, it's very hard for this data to intermix, right? And frankly, some of the customers may even require this kind of isolation as a part of that contract so that is a benefit, right? Now you guys also may hear some shard on the global scale, right? Some of you may have heard about those recent law changes and as well as like political changes in the countries. There are some countries may require you to keep the data for the users in that country. It's much easier to move to that kind of environment if you shard already, right? Makes sense. So often that will be kind of not purely technical consideration but something beyond that. Now costs may also be an important factor here. Now, if you take a look at the majority like managed hosting providers and the cloud providers, they really kind of want to take advantage of your pain, right? So what I mean by that is you would find what the margin, right, or how the cost escalates to a very, very high powerful system is phenomenal, right? You can see high-end system being charged, you know, 10x as a low-end system even if you go to, I don't know, let's say, the store to buy one, right? The difference in price will be only like 2x or so, right? So what that means in practice is what you often find if you're cloud providers or, I mean, even if the systems you buy, the systems which have an optimal price performance ratio, right? To give an example, you can get four socket systems with Intel CPUs, right? But those will be very, very expensive those days, right? They're much more expensive than the two socket systems even though they don't even give you like typically 2x of performance, right? So if we don't have to go for that the most expensive hardware, we can have a much more optimal system and Sharding allows us to pretty much pick the systems of any size because if you have implemented Sharding well, you can Shard over 10 systems or 20 systems, right? And that really should not matter. And as I mentioned, that's especially important in the cloud because cloud may have a relatively lower instance size what you can get compared to physical hardware you can buy, right? In terms of the top performance which is accessible in the cloud. So when do we Shard, right, as a summary? Well, first is Sharding can be a good idea when it's easy in your case. Some applications are easier to Shard than others in both in development and operations. Second is when scaling up is an impossible to expensive. I mentioned cloud, but enterprise can also have a silly politics, right? I remember the guys who had to Shard because there was a MySQL, DB80 and the OPS team and the OPS team believed what 16 GB ought to be enough for everybody, right? And they would not provide the virtual instance with more than 16 GB of memory, right? So they had to Shard, well. Now, you also may want to do that then your application is growing so far so Sharding is imminent anyway, right? And in this case, you may want to invest in Sharding which will give you huge performance gains compared to some small optimizations trying to pick up 5% here and 10% there. So what are a few Sharding questions we can talk about? Well, Sharding level, key, you need how we handle have ability in Sharding technology. Let's look at those in more details and that will make sense. What is the Sharding level? Well, there is two ways to think about that. One is you Shard only on the MySQL level and other you kind of think at your Sharding by deployment unit to a full stack level, right? So you can say, hey, I take my MySQL and web servers and the cache, right? And I Shard all of that completely, right? Now, you often do the second one, for example, if you want to place those things in different regions, right? Or something like that. Or to get either better security isolation and isolation at all, right? Because if you have essentially some very thin load balancer which kind of splits your customers to different pods, right? Or wherever you want to call those nodes then they're completely independent, right? And the failure affecting one of that is probably not going to impact everybody else, right? And you may have just, you know, 5% of unhappy customers which is much easier to deal with from all kind of things, right? Including providing them a good customer service. Sharding keys. When you pick a key on the database for Sharding, we can see a couple of things. One is what most accesses, right? Simple accesses should go to a single Shard, right? If most of your accesses hit all the Shards, that is bad design, right? Now, no Shard is large in terms of data size or loader generates, right? So, for example, Sharding by country may be a good idea in the first one, right? Because most accesses for many applications will go to a single country. But then there are way too large countries, right, for most applications. And they're very disbalanced, right? I mean, you have a country like China or US with a ton of population that have somebody like, I don't know, let's say Czech Republic, right? Which is, you know, much, much smaller. Another thing to consider is when you Shard, you don't have to pick one, right? In certain cases for certain data access points, you may want to pick a couple of Sharding keys and double store the data one way or another, right? So, to give you example, systems like Flixter, for example, right? We just social network around movies. So, it will have two very different, but the most important data access pattern. Even through a movie, when you have all the comments and a vibe about a movie, or about you as a user, what you have said and so on and so forth, right? So, in such case, it would make sense to double store some of the data which is very fast to access for one dimension or another, because, well, storage is cheap. Sharding unit, what do we do here? Do we Shard by physical MySQL instance schema, or do we kind of Shard it on a very logical level where you can have multiple Shard objects placed in the same set of schemas and database? Well, it depends, right? Typically, if you are dealing with something like software service where you have, I don't know, that's a small number of customers which pay you lots of money, right? Maybe 1,000, 10,000, 100,000. You can actually even set up a small physical MySQL instances, especially those as if containers, right? You can, you know, run pretty large amount of them effectively. But if you have a ton of users, something like a Facebook, then typically you end up having multiple kind of Shard keys short in the same set of tables or schemas. And I think to consider what is hybridity, right? Because the more service you have, the more there is chance for one of them would fail, right? So, pretty much, if you have rolled out the large Shard, you most likely will need to implement some hybridity as well, right? And it even becomes more important than just single server, because if you just have one single physical server, I've seen instances when people would have no downtime in many years, right? And they just rely on backups and it works for them.  Every hour even probably. So, if technology doesn't compensate for that, that would be a disaster. So, pretty much two common approaches to that. You can shard even over master-slave clusters, right over the orphaned effort, or use PXE or some other Galera-related technology. Let's look at some sharding approaches which exist those days. Roll your own, which is probably the most popular one, especially on the large scale, right? I mean, all those big guys typically have something very tied in with everything that they're doing. And in my school world, it's heavily custom. VTES. VTES is actually, in my opinion, the open-source technology is to watch, because I think it has been proven in production in YouTube. But those guys are very, very serious about making that into a good open-source project, and putting a lot of very good documentation, and so on and so forth, right? So, I'm very excited to see what that would be. There are some other technologies, like JetPants and ShardQuery, which kind of exist out there in an open-source, but they don't have nearly as supported momentum out there. Other things. This is a proprietary technology which has a built-in... MySQL Comportable has built-in Sharding, right? From technological standpoint, it's actually quite interesting, but because it's kind of closed-source in a small community, I didn't see it having a lot of traction. MySQL Cluster, that is another interesting technology, which does a certain level of Sharding automatically for you. Anybody runs MySQL Cluster here? Anybody tried Ryan MySQL Cluster here? You guys know it, right? People who tried Ryan MySQL Cluster were much more of those when they were still running it, right? This is rather complicated technology with a lot of kind of tricks. It's absolutely fantastic for certain workloads, right? But it's not as great for many other things. It's relatively complicated, and what I will also mention is there's actually had been a lot of interesting work done with MySQL Cluster over the last couple of years, or last three years, so if you gave it a look like five years ago, you may want to check it out again, right? But I would say also, when you speak about MySQL Cluster, it's kind of something that gives you a medium level of Sharding, right? Maybe if you need to Shard and get, let's say, 10 nodes with all the data fits and memory, that may be the application for MySQL Cluster. If you are doing math and saying, oh, I will have 1,000 nodes in production in a year or two, that is not nearly what MySQL Cluster can support, right? And in this case, it's not going to be good enough. A few others. MySQL Fabric, that is a Sharding effort by Oracle, which I think is interesting, and I don't know what I mean by interesting in this case. I mean, I really would be interested to see it more in production. So far, there is a lot of a developer talk from Oracle about that, but I haven't seen it really deployed at the large scale. Tizor database virtualization engine, that was also pretty nice open source project, but as they kind of moved out as a company and focused on different things, I don't think it's been well maintained anymore. Scale Arc has some rule-based Sharding engine, which is also commercial, but I actually see a lot of Scale Arc deployed, right? So that is a pretty common solution, especially by enterprise customers who don't mind using this kind of proprietary software. There was alternative to Scale Arc here, Scale Base, which I keep here as a zombie on the slide, as a reminder, what a lot of proprietary technology companies may die and kind of the technology we had can just disappear and stop being available. So a few things in summary, right? You guys can see what there are multiple technologies available for Sharding. In the space of MySQL, unlike some others, there is no just standard way to Shard, right? There is no some Sharding solutions, which 99% of the people are using out there, right? And you just can't follow. So you can see it as a downside or you can see it as an opportunity to become creative. That's up to you. So if you guys are looking for Sharding, what we, as a Berconna, can do? Well, there is a lot we can do, right? We have both helped a number of companies to implement Sharding as well as supported them with Sharding environments, youthful support, or managed services. And if you guys are very interested to learn more about MySQL, is that MySQL is a kind of big part of your life. Consider coming to Berconna Life in Santa Clara in April. We'll have a huge amount of MySQL content. And this year we'll also have content about a lot of other open source operational and analytical databases, right? MongoDB, Cassandra, and others. Well, that's it for me. And I would be happy to answer your questions. Oh, okay. Well, I mentioned that I didn't clarify the abbreviation. So when I mentioned PXC, that is Berconna X3B cluster. Yes, yes, no, but that is a good catch, right? So is that actually what happened? You entered PXC and there is no PXC out there? Okay, can you send me a screenshot of our marketing, right?  You can't even get us to the front page? Okay, any other questions? And if you want to troll me, that's fine, right? Okay. Well, yeah, two things. First, let me put in another plug, right? Actually, what I would be, one of the talks I'm very interested in is Perconna Live. Facebook is going to talk about how they're doing backs up at scale, right? And I think it's, for me, it would be very interesting to see what exactly strategy is the very important, right? Now, backups, you need for charging, right? And what I see a lot is that people have some sort of fuzzy backups, right? So, for example, say, hey, we take the shards and backups plus the backup binary logs, right? So if you ever have to need to restore the shard to a given point in time, approximately, they can do it, right? Having something like on a live starting system doing the kind of a global checkpoint that is possible but typically is quite, is a bit painful, right?  So not to impact the master when making some sort of, well, note on the binary log position which corresponds to them and taking the consistent backup, right, for everything if consistency is needed. Now, another thing about the shards is consistency between the different shards, right? It's typically a little bit loose. It's not like you have a foreign keys, right? And it's kind of absolutely going to be out there. So, because of various failure scenarios, if there is certain data consistency which has to be maintained between the shards, then it is good to have certain scripts to check it and fix it because data drift will happen, right? And you better have some tools to fix it. And that is what people can do, right? They can, well, in the worst case scenario, if something happens, I restore backup, maybe it's not kind of exactly consistent with some other shards, right? But I can, well, I know there is my source of truth, right? For the data, let's say this user lives on this shard and then I go and check the things to ensure the relationship with other shards are consistent. Does that make sense? Okay. Well, I think you always have to be looking at what exactly problem you're solving. So, because you can really get, like, 100% consistent backups, right, across, of course, but that's going to cost you, right? And in a lot of cases, people just don't pay those costs. Because, again, I'm on other things. Well, like, here's the thing, right? I think backups in general, if you think about backups in scenario where there is redundancy and there is no redundancy, there are two different things, right? Because if I, let's say, have my laptop, right, and I just lose my hard drive or something like this, right? Then in a lot of cases, I will have a full restore, right? And that is going to my use case.  If I lose hard drive on this laptop, I will just fail over, right, to the slave. I don't do backup restore in this case. When does backup restore happen? Well, if you have some developer with dirty fingers, right? Anybody seen those, right?  What happens in this case, though, is typically they trash one of the tables. Well, some developers are very gifted and will trash all your tables, right? But typically it's going to one of a few, right? So you have to restore the tables which were trashed or even the data which was trashed, right? Which is a very different backup scenario, restore scenario, right, than you may be thinking about, right? So that is why when you have a replication, often you have to optimize for how you can restore pieces of the data, right, and maybe kind of do partial binary log recovery to catch up those pieces of data you restore, right? And fun stuff like that. Make sense? Okay. Well, I think, yes. In some case we will see that as a consolidation, right? Possible. I think things have actually become a little bit better in this regard more recently with MariaDB and now in MySQL 5.7 offer multi-source replication, which allows you to find in replication if you want to get all the stuff together, you may be able to do that. We also did some work with a tungsten replicator to do that. But, well, it is possible, but that's also, I would say, not totally easy because in a lot of cases people would have created the ideas which are conflicting and so on and so forth, right? So that's going to depend a lot on application, what exactly you're going to do. Well, yes, I think that is a... And I think in a lot of cases it's also process of the during the sharding, right? Because when you speak undo the sharding, is that a consequence of a failed sharding project, what they implemented the sharding and now that application version which supports sharding just blow up, right? And we want to go back. Or it was in production for five years and then they say, hey, you know what? Application is not growing and now hardware is so powerful anymore, right? Those are two very different cases because in the first case we'll have to kind of temporary maintain some scaffolding so we can always go back to uncharted system, right? Okay, any other questions? Well, I think it depends a lot on the scale, right? Line orders I see very well used on the kind of smaller scale. If you look for large scale resource, the AWS is probably the most popular one we see. We see some more of a rack space, right? People who has been using rack space managed service started to use rack space cloud a lot, right? We like the support approach and other goodies. So it has been worrying, right? Google cloud SQL also looks interesting. But I think, you know, when it comes to us, people really choose cloud for their databases, right? They choose cloud for their applications and their databases are going to live in that cloud and that's what we have to deal with, okay? Anything else? Yes, go ahead. Talk a little bit more about what? Balance in the demand spikes. Oh, okay, okay. So when I spoke about the spikes, right, I was talking about the queuing, right? And here is an interesting thing, right? If you think about read and writes, right? Which of those are really hard to deal with? A lot of people speak about writes, right? Writes are hard. You have to actually modify the data. But actually, reads are much more hard in many cases because when you are doing reads, you have to get the user data right now, right? Until you have given that user the data, I mean, he doesn't get his value, right? When you speak about the writes, in a lot of cases, we can just put them in some sort of queue and apply later. And frankly, my SQL is already doing some effect for you, right? Think about what happens when you update the data in the debit table. It goes essentially in the queue like a log, right? And then it's going to be flushed from a buffer pool to the disk sometime later, right? If you log files at a large, it's maybe much, much later, right? And that is essentially the idea for writes. If you have some writes which are very expensive and you can afford certain delays, right? You put them in the queue and have some background working to apply that at the given pace instead of slowing all your users, right? So that, I think, is the main thing. Now, if you look at the overload management technique, which I would mention, is, well, not all the features are created equals from a user value standpoint, right? So make sure you can disable them in the load spikes, right? If you can bring up in more capacity, right? Or if you don't have any other tools in toolbox. I remember for years, for example, you guys could see Wikipedia search sometimes being down, right? They're saying, hey, guys, you know what? Wikipedia, you didn't donate enough money, so we don't have enough servers, so search is not working, right? Come back later. But if you just use Google as a search, right, it still would be working and performing very, very fast, right? And I think that is very important to prioritize what a user experience, right? And not trying to make it kind of bad performance or a bad experience for all your users for all features, but better to have, you know, certain users to have a downtime, right, and certain features not work, and others to be, like, absolutely fantastic. That makes sense in terms of load management. And what that means, however, right, is what you want to design the features so you can disable them easily, right, in production. So then the feature becomes broken, right, or it creates too much unanticipated load, you can maybe disable that. And what's that? Well, yes, I think you're speaking about another interesting thing, right, which is kind of a good development practice, right? If you're developing the large features on the scale, often gradual rollout is great, right?  Well, yeah, absolutely. So it can be back and forth, right? It doesn't have to be black and white. Yes? Well, I think that is a very good question, because if you think about that, MySQL is not really great analytical data-raise, right, in this case, I mean, it's because those days you have a lot of data, and if you want to analyze data, if you want to have features, typically, right, being able to crunch a lot of data in a parallel, right, and use all the multi-course. In this case, it's some sort of ETL process, right? And I mean, there's a lot of integration those days, right? You can use Scoop with MySQL to get the data in Hadoop. There is multiple tools right now how you can get from binary logs to Hadoop. You can use tons and replications. There is, I know some work, how Kafka is being integrated, right, with MySQL. So you get some data out, and then put it into whatever system you would like to do, right? So lots of choices those days. Yeah, and I would also mention that some people, right, we see who really love MySQL and they like pain, they can use multi-source replication to try to get it to one Frankenstein box, right, and try to run queries out there. Yes. Well, after all, he is dead, right, if you haven't noticed it, right? Unfortunately, with CPUs, we pretty much don't have a choice those days, right? I mean, in server-grade CPUs, Intel owns the market, right? We may have something to come out of your power, or like ARM is kind of picking up a head on the server, or there is a risk five, right? But it's all probably like five years ahead. For now, you typically would look at the zeons, right? And the choices you have is pretty much going to be defined with how many sockets you have, right, and how much memory you support, because low-end zeons would only support like 32 or 64 gigs of RAM, right? Not enough, right? And then that is pretty much its main direction, what you're looking for. As I mentioned with MySQL, for most workloads, you want faster cores, not necessarily so many of them, right? But that, again, varies, right? In some cases, you may want it with more cores if your workload has enough concurrency. So that's what I would mention. Any other particles you're looking for? Well, I would also mean a lot of memory, right? I mentioned for that. If all your cloud fields in memory, that's fantastic. And for solid-state drives, SATA and SAS are bad, right? Because that kind of bandwidth that SAS and SATA interface provided is just way too narrow. So you want to get some NVME, or like other PCI Express-based storage if you're looking for best performance. Yeah, NVME, right? Or other PCI Express things, right? For example, Fusion IO is not quite NVME because NVME is kind of standard, but it's also very fast because it goes through PCI Express bus, not through some, you know, funky interface which was designed for spinning disks. What's that? Well, for Flash, if you have a chance, avoid SAS and SATA drives, right? Going to limit your bandwidth compared to PCI Express. Well, I think this is kind of a little bit Apple and Oranges, right, if you look in this case. So if you speak about Galera, that is a replication technology, right? Wherever you, so, in this case, right? And Galera doesn't necessarily help. It helps you in terms of that you have a little bit more predictable latency in replication, right? But still, you can't scale your write 10x, right, with Galera. Wherever it's you're using one or another, you need to think about that, right? Also a thing is, actually I would say because of how Galera works, right, often the limit of a data size per instance you would keep if Galera is smaller than for a conventional MySQL, right? I mean, I wouldn't go into, let's say, 10 terabytes Galera cluster because if you ever need to do SSD, that's going to be very painful. Any other questions? Okay, guys, and I will hand it out today and tomorrow at the Percona booth at Expert Hall. You know, if you want to stop by, please do. We should also have some toys out there, right? If you'd like to. Thank you.


Hello everybody, it's Eva from Bohemian Crafting back again and I would like to share with you how you can make a magnetic paper clip, also shaker magnetic paper clip, and it's very easy, but with me here is just a few types, I made to show you that you can make some vintage style, shabby chic style or just simple for your planners and it's really very easy and it will hold lots of pages together, if you will using good magnets, it will hold lots of papers together for shaker effect I use those can pull rings and I will try to show you how I made it, I will try to show you probably on one magnetic clip and other stuff, you will realize yourself how to made it and if you've got only one magnet, don't be sad, you still can do your magnetic paper clip, just grab your magnet and go around in your house and I believe you will find a hundred things which are magnetic, which you can use like that opposite side of your magnetic side so as you can see all this, it's magnetic, it's all what you can use for your paper clip and I will try to show you with that shakin effect, you need to take your can pull ring and there are two different types of those can pull rings, there are magnetic and non-magnetic, here is already magnet, but non-magnetic can pull rings, they are mostly on drinks, on drink cans, they are made out of aluminium, it's like this one, it's not magnetic on food cans, like human food or pet food, they are made out of metal and they are magnetic, so you will need only one magnet with those, with those you will need one or two magnets, it depends and if you will do not use those can pull rings, I will show you how you can save your magnets and how you can do it, just win with one magnet, so we gonna try to make that shaker cut, take your can pull ring, take piece of acetate, it can be clear or stamped I've got here, I've got here one, we have very light stamped, so take your can pull ring, put your glue on that smooth side, on this nice beautiful side, put your glue on it, all over, if you are using glue, like this uhu glue, let it dry for a few seconds to make that top coat, that's how you do it, put that sticky top coat of that glue and then put it to your piece of acetate where you want it, maybe like here and let it dry, let it dry properly to make sure that it will hold together, when it's dry take your scissors and fasten it out, your can pull ring all around like this, make sure you are doing it nicely and carefully that all that piece of acetate will still holding all that can pull ring, now take some cardstock, some good quality cardstock, if you don't have good quality cardstock, if you've got just some light weight cardstock, I would recommend to use on one side some plastic acetate, I mean something like this, those they will hold, they will make that card stronger and you will be sure that those sides will not rip off, so take your piece of cardstock, fold it in size of your can pull ring here fold it, put your can pull ring on it, make sure that one off side it depends if you wanna have it closing like this or like this to make those sides together here so you will hear, do not fussy cut out here or maybe you wanna close it like this, so do not fussy cut out this folding, just put your can pull ring on it and fussy cut out all around, make sure that fold will hold together, fussy cut out your piece of card around your can pull ring you can first draw that shape of can pull ring and then fussy cut out without holding that can pull ring on your card, fussy cut out like this, so you will end up with something like this now you can distress your edges or stamp your image, I'm gonna stamp a little bit this one where is my ink, I'm gonna stamp it a little bit just slightly to make some pattern like this as I like that, it's very light now take some kind of sequins pieces or some beads, do not place there anything magnetic like I did, because it will be not shaken place your pieces inside of your can pull ring, maybe I can distress it a little bit and remember I am working with that magnetic piece, I am working now with that magnetic piece so I've got a little bit easy work, take your glue, put your glue all around, be very careful I think I've got my glue here, sorry for that put your glue all around, I will show you if you've got glue like I've got now that it's leaking so heavily to avoid that you will have there so many glue, just spread it with something use some of your tools and spread it nicely and now it's time for waiting a little bit you can put your paper right now on your can pull ring but with this glue I've got my experiences that it's good to wait a little bit not too long but a little bit and then any kind of surfaces will be glued down so fast maybe I am still a little bit eally with that glue in press it down and let it dry before you will do that other side let it dry faster, it will be probably open later when it's dry leave it closed take your magnet and drop it down and it will show you where you can put your magnet I think best way I can put my magnet is somewhere here and now you need to cover this magnet with some embellishment so here I used this picture from Tim Holtz collection I decorated it with those corners between this card I placed a piece of fabric and I glued on that fabric this one and I glue it on the top so you can decorate it and when it's dry you've got your paper clip with this, this is very easy paper clip I made this is piece with that hole it was label from some drink, I don't remember which kind of drink I'm guessing it was wine because here are those grapes so I'm guessing it was wine I saved that and I use it now like paper clip, magnetic paper clip my magnet just dropped down because I did it just a few minutes ago so it's not dry yet I have to let it dry so I placed one magnet here then I fast cut out piece of cardstock I glue it to this end I've got already one magnet here so I dropped I spoke that turn it around then I dropped my magnet, it was stuck on that magnet from the bottom I glued that magnet on that place and I covered it with this label so it's magnetic paper clip for planners because you can mark with this your notes anything you need to be seen like important with this this is not magnetic can pull ring I placed this can pull ring on cardstock I fast cut out that shape of that can pull ring then I placed then I glued that can pull ring on piece of cardstock in this hole I placed my magnet then I decorated with that ball and with this bead I turned it over on that other half of paper I didn't have to place another magnet because this filigree is magnetic it's what I told you if you will have some magnetic material for your opposite side of magnet you don't have to use two magnets if you don't have some magnetic piece some magnetic opposite side you will have to use magnet on this paper on this other half of your paper just fold that your basic paper drop your magnet it will automatically catch up that magnet from opposite side and cover that magnet with some embellishment oh my god my explanation is horrible I hope you understood and believe me if you will grab your magnet even if you've got only one magnet you will find something magnetic in your house like this one look at this maybe your mommy have a sewing machine your daddy have your garage full of things maybe in your clocks are those batteries just go with your magnet through to your house even those bottle caps they are magnetic and those will be brilliant like magnetic paper clip can you imagine that to embellish that some shabishik or some grungy make some mixed media and you will have amazing paper clip so this is my sharing for today I hope it was understandable when you will make just one small note when you will make paper clip with different types of shapes I will do it now I've got this heart and from that other side I would like to have this one so I took piece of cardstock I fold it from this side I place that card that heart where I will probably wanted to glue it down so still I've got large space for my magnet I draw around that shape of my heart now I will fascic it out I will glue it one magnet here then I will glue it this fascic it outside on my heart then I will flip it over this will be same here I will drop my magnet glue that magnet down and then I will glue it over it this card and it will make my magnetic paper clip with those two different shapes this is my sharing for today I hope it was understandable I hope thank you so much for watching a to to thank you for your interest for all amazing comments grab your magnet and go to go to find your magnetic pieces thank you so much for watching and have a lovely, lovely Sunday bye


So one of the things that medical professionals deal with all the time, whether it be students or actual trainees or attendings, is just how we best use our free time.  And if you're not studying, you feel like you should be studying. And you kind of go into this deathly cycle of never realizing how to best use your free time. One of the most important tips that I give to any kind of pre-med or medical student is make sure the things that matter most to you right now are still part of your life, even as life gets busier, because unfortunately as you get to later into medical school or later into training or as your life isn't attending, things don't get easier. As in your schedule will now be filled up with different forms of responsibility, you kind of wish that you went back to the free time of your old high school days, your old college days, and even your old medical school days. So one of the most important things you can do for yourself right now is to find the bits of free time that you do have and try to make the most of it by including the priorities that matter to you right now.


See, the forces of history and primetime television have conspired to make bearded white dudes with an affinity for denim the face of Dixie. In reality, the South is one of the most diverse regions in the United States. We're home to 60% of the country's African American population. And of the 10 states with the fastest growing Latino populations, 8 of them are Southern. Despite our less than illustrious history of race relations, diverse new groups of folks keep moving down South and improving the place. We got descendants of Vietnamese refugees adding their traditional spices to crawfish boils down in the bayou. We got a 10,000 strong Syrian Lebanese community in Mississippi.  Hell, we even got different types of white folks. Like the descendants of German immigrants in Texas who speak 100 year old dialects and still f*** with polka. To understand how the South became so diverse, we got to talk about slavery. I know there's a certain portion of the audience that don't want to hear this stuff, but it's a critical part of the origin story. It's kind of like how DC has to show Bruce Wayne's parents getting popped in every single movie to explain why he became an emo crime fighting bat. Here's some numbers that might put things in perspective. Back in 1690, there were about 250,000 people in the South and 80% of them were Native Americans. By 1790, the overall Southern population had hit 1.7 million, but by then Native Americans made up just 5% of it. So what happened? Well, if you didn't look like a cast member of Gossip Girl, it was basically a century of nightmarish tragedy. This time period went a long way toward destroying Native American populations in the United States, but with the influx of settlers and slaves, new cultures were created as well. Like the Gulligichi people who were brought to the colonies as slaves in the 18th century and still inhabit the South Coastal region today. The Gulligichi were relatively isolated in the South's low country, so over years of forced assimilation to colonial customs, they still passed on their language and traditions that tied them back to West Africa. Now these people have a unique culture that's distinctly West African and Southern. Take their language, which scholars say is one of the last true surviving mixtures of English and African dialects. They came from different societies and places in Africa and they had to learn to talk to each other, as well as to the Europeans who were around, and so they very creatively crafted this language that exists nowhere else in the world. So do say, I don't want me for coming down here now, for talking to the people and them, and I've been here, you tell me, say don't talk. Or their food, which basically helped create the bedrock of Southern cuisine by mixing African grains with Native American spices. Basically, no Gulligichi, no Gumbo. Now there are no recorded instances of the Gulligichi ramping a 69 Dodge Charger over a police car, but does that mean they have any less claim to be in the face of the South than the Dukes of Hazard?  I think if people when they thought of the South like among the first things that come to mind were like outcast instead of the Dukes of Hazard, I think that stigma wouldn't be as pronounced as it is. You guys have had pretty bad branding. Yeah, I think if Andre 3000 just put out another album again, that would solve a lot of our problems. Another group who played a crucial role in defining the South's identity is the Seminole Tribe of Florida. Thanks to the near constant efforts of the federal government to relocate or eradicate the Seminole, they've ended up embodying the Southern spirit of defiance more than any good old boy with stars and bars mudflaps on his lifted F-350.  Back in 1817, old hickory dickory d**khead kicked off a series of wars on the Seminoles because he wanted to recapture freed slaves that were living on their land, then steal their land, and then relocate them. Seminoles were outmanned, outgunned, and thus nearly obliterated in these wars. A group of Seminoles were relocated to Oklahoma, but many stayed behind in Florida and never surrendered. They remain the only Native American tribe to have never signed a peace treaty with the U.S. government. There's nothing more Southern than that telling the government to go f**k themselves, right? Hmm, fighting to the very end against a tyrannical government that wants to steal your land and freedom in a glorious lost cause? Basically, all the cool and inspiring s**t Confederates made up about themselves is actually true about the Seminole people. The reason all this matters is because, as crucial as these groups are to defining the South, their contributions are rarely acknowledged, and some of these communities may not be around too much longer. The Gulligichi have seen their historically insulated communities broken up by real estate developers. And though I would bet on the Seminole to hang on longer than anyone would expect, their lands are under attack by climate change. There's not a single magic bullet that can put an end to these threats, but we can't begin to solve them if we don't even recognize the people exist in the first place. There's a solid quote by a South Carolina historian named W.J. Cash who said, There are many Souths. And the sooner we all recognize and appreciate that truth, the longer it will remain one.


Can I log in as Jenkins? I'm not sure. I've definitely logged in as Jenkins. Anyway, we are live now. So yeah, welcome to the regular Q&A session for Google Summer Rockbook students in the Jenkins project. Apparently today is the last Q&A session for students in the Americas time zone. Let's agree that we need another one. So the application deadline is April 9. It's the next Tuesday. If somebody is interested, we can have another meeting next Monday, maybe just in the chat.  It's generally on-demand meeting. So yeah, that's it from me. No specific introductions or status updates I needed for this meeting. So we can just proceed with questions. So yeah, if any of students has questions related to the JSOC process, et cetera, let's start from there. And if we have time, we will switch to the project specific questions. OK. So does anyone have questions about the process? And deadlines, any organizational questions, whatever. Can I just share my screen? I'm going to share my screen and ask just one question about the external workspace manager. It's definitely a project-related question. OK. If there is no process questions, let's just take a look at what we have in projects. OK. So I'm going to share project-specific questions. So here in this, I want to, there is a working flow I just come up with. So first, there is a disk pool ID.  And then when I find it is predefined, I need to check whether this instance is launched on EC2 because we're not in on EC2. You cannot use EFS, right? And then after I do all these checks, then I'm going to configure it to this slave on-demand. That is, when every step were executed, I'm going to monitor on it. Is this working flow, right? What? I think what you explained is what we actually expect. So what was the reason of my comment that, yeah, there are two ways of implementing the plan. One way is trying to pre-mount EFS. And yes, it was mentioned somewhere above. We should do that. But there is another way to spin this story where you do EFS mapping on-demand. So, for example, you execute pipeline and then you need more space. And at this point, you request a new workspace to be provisioned. And Jenkins does workspace provisioning and then performs mapping of the workspace. In such case, it would be closer to what we have in the plugin, for example, for common workspaces. Okay, I think that makes sense to me. And I think that is more doable and more clear because it was, so actually, now I just found this clear goal of this project. That's good. Thanks, Alex. I'm done with my share. Wait a minute. I can stop it. Okay. Okay, done. Thank you. Any other questions? You may also hit the participant limit. We need to rotate. Any other questions? There are so many people on the call, so there should be something. Yeah, hi. It would be nice if you adjust the sound settings because you're so silent now. Hello. Oh, yeah, now it's better. Yeah, so I had a couple of questions about my proposal. Okay, I'm sorry. I cannot share my screen right now. I'm not on my laptop. If you want. Hello. Yes. Yeah, so the first question was, you'd asked how would I start Jenkins from the jar file? Yeah, your proposal. Yeah, it's here. So you're talking about just search for the, yeah, here. So yeah, you see benchmarks jar file. So the tricky thing about Jenkins, that Jenkins itself is a runtime service. So you can adjust the classes and perform unit testing. It requires to have some context. For example, what we do in Jenkins test harness, we have a Jenkins tool, which actually spins up Jenkins. So you can actually use it as an application. Yeah, I had, I did suggest that. I just wanted to ask if that is good enough. Well, we look for the network key selector. Could you please repeat your question? Yeah. I'm sorry, I'm not able to get you. Yes. So there is enough. So that's why so difficult on the center. Okay. Just second. Okay. Now you're muted. So I can just answer, I believe. Okay. So yeah, my question was about the, it's another proposal. Yeah. So here you have benchmarks jar. So when you execute this benchmarks using Java, my micro benchmark framework, you, if you want to use Jenkins rule, firstly, you need to ensure that it's actually possible because Jenkins rule is designed specifically for JUnit. So this test rule is compatible with JUnit, but I'm not sure that the JMH can actually use JUnit rules as a part of the execution. So if you want to use Jenkins rule, it's something you need to check. Yeah, I've already tested that. It does not seem to work because possibly when we run a JMH test, it folks into new Java virtual machine. So we cannot run an instance which was originally used. Yeah, right. So you would even need to somehow work around it because the Jenkins rule, there is no magic inside. If you take a look at the code. Yeah. So I wanted to ask, would it be possible during the project to use something like the, I mean, like the method used by a Jenkins test on this? Well, if you get it working, that's perfectly fine. Okay. Yeah, so Jenkins rule, there is no magic there. You can probably copy this code and make it running as a part of JMH, which may be useful to have it as a separate library so that it can be reused by others. Okay. Because I had already tested that when the setup codes for, when the setup methods for JMH run, they run in the same process. So I think we'll be able to get it to work. Okay. So if you do that, it will be definitely useful and you can probably do it as a separate library so that it can be reused by other projects which need to perform as they stand. Okay. So I'll add that to my proposal. And another question is, you had asked about moving the documentation from the Viki to GitHub. Yeah. So is there any benefit of doing so? Well, firstly, the documentation is good because when the documentation is located in GitHub, then you can patch the documentation in terms of this code. So for example, you propose a tool request with new feature and write in this tool request, you write the documentation. So you merge the documentation and the feature at once. It's one of the benefits of such approach. Okay. Yeah, whether it's built in Viki for users, it's debatable. So if you just want to pay documentation, of course you can keep it on the Viki. But yeah, in the different plugins, you must use GitHub for the documentation. Okay. Yeah, so my advice is if you want to create a tank of new documentation, then yeah, better to use GitHub. You can write the documentation in a ski dock in Markdown. So it's something standard. Unfortunately, the Confluence format, well, it's used only in Confluence, I believe, but yeah, Markdown and a ski dock are much more preferable from projects standpoint, because at some point we can generate read the docs, for example, for this plugin if we need, or do something like that. Does it make sense? For example, we have to take the problem, not the live plugin, but this plugin has some documentation hosted in GitHub. So yeah, you can just... Well, generally you can just copy it and have everything here, or you can create multiple page documentation if needed. In this case, it's your decision. So if you want to keep documentation in Viki, just do that. If you want to migrate it, yeah, it's probably a good deliverable for the project, but yeah, obviously it's not according to set. Okay. Yeah. Okay. So let's move on to other question. If there is nothing left, but yeah, we can discuss more question and if you have any other projects, questions specifically, strategy we will return to them. Alex, Jack, do you have any questions? Oh, yeah. Do you hear me? Yes? Yes. Yeah, I ask in chat, but from your experience, what's the best lens for draft, for this kind of draft, for this Google draft? Because I could add additional steps and make appendix, but I don't know, should I... Or should it be like short and then it could be developed? Okay. So there are two approaches being used by students. One approach is to have a good project proposal. I mean, keep it short so that everybody understands what you want to do and then this type of top-level things like what would be deliverables, what would be the benefits for the community, what would be the timeline. So there are sections which I actually expected in Google Summer of Code documentation. One of the approaches which is widely used. Another approach is to actually use project proposals as a kind of mini design where you document everything you want to do at the level of digitalization you can achieve during the application phase. Both approaches are fine. So whatever is preferable for you, our expectation is that you have basic information. So it can be done in three or four pages, maybe even less, so you can start from there. If you want to add details like the mini design, like some research summaries, et cetera, yeah, if you perform such research, it makes sense to put it to the proposal because at some point it will be used to do the design for your project. What we do in Jenkins, we usually use community bonding in order to do a kind of mini design together with these mentors so that your proposal has some reality mapping and that you also create a list of tasks at least for the first coding phase. So when the coding period starts, you really can start coding. This is our objective for community bonding. So yeah, if you have time now, if you want to expand your proposal and to put some design ideas, et cetera, it definitely won't hurt. But yeah, if you just create a short proposal with three or four pages, with all summaries, et cetera, yeah, it's also key. Okay, I think I focus on short at first. Okay. Maybe align some aims, some developers because starting digging deeply. Yeah, I understand there is some, it could be additional, it could take additional time. Yeah, right. So yeah, summary deliverables is what we expect to be well written regarding design, regarding project ideas, if it's a brain dump, it's perfectly fine. But yeah, well, if you focus on the first part and then if you have time to do something for the second, this approach I would recommend as well. Okay. Do we have other topics? Just a small... Yeah. Do we need to provide any more contact information because the Google's, because Google's guidelines are just to provide the address and telephone numbers. Is it necessary to provide them? It's not necessary to provide them in Google Doc. So what we really need from you, we need email, but again, email, if you join the JSOC public mailing list, we need to get emails from them. So in your application, it's not necessary to specify any contacts, contacts maybe except GitHub ID or Jenkins user name if you have it. All other contacts are up to you. So if you are not comfortable about keeping these contacts, just remove them. If you want to provide them due to whatever reason, it's also perfectly fine. Okay. I would like to clarify, Oleg, in the official application on the Google website, you need to provide everything they ask, but as far as the student proposal is concerned, it's what Oleg said. Yeah, thanks for clarification. So be sure if you want to be using your email to send spam, you won't be using your phone number unless there is an emergency. So sometimes it happens, but even if there is an emergency, we can contact the JSOC support. Because, well, it may happen, for example, if Jenkins project consults your three to Jenkins world, and for example, if something goes wrong, then yeah, this phone number may be used, but in such case, mentors will just coordinate with you before your trip. So yeah, no need to put these contacts. Okay. I have a question. What will be the schedule for, like, these Hangouts meetings when community bonding phase, so it will be the same, like once a week, and then on the coding phase. Yeah, so what do we expect to happen during the community bonding phase? Firstly, all projects will have separate meetings with mentors and with students, and it will be the base meeting for you as a student. Then we will have weekly JSOC meeting, like we have now. It will be kind of office hours for Q&A, for status discussions, etc. So it will be an optional meeting. It won't be required to join it every time, similarly to these Q&A sessions. And in addition to that, there are also special interest group meetings. So you may have seen that some projects are linked to special interest groups or to sub-projects in Jenkins. Some of these entities also have meetings. So usually it's something like one is a meeting once per two weeks or so. And yeah, these meetings is something we would really recommend you to follow, because these meetings are just a pool of stakeholders for your project. When you need a review, when you need feedback, when you want to demo something. So going to these special interest group meetings is something we would advise. Again, the main meeting for projects is a kind of sync up with mentors. We usually recommend to have at least one meeting per week. In many cases it's two meetings per week. And yeah, it will be the main meeting for communications. Okay, thank you. Yeah, we think that you will be able to discuss the time with mentors, so that all of you will be able to find a comfortable time. You don't look very happy about it. No, I'm happy. It's okay, I just I had a huge problem with this Google handout. It's not supported in my laptop, so... Good news is that if you and mentors want to switch to another communication way it's also possible. Yeah, if you decide that you do a meeting for example in Zoom or whatever please feel free to do that. And actually, there is a wider discussion about Google Hangouts on the Jenkins organization level because we have a limit of 10 participants on the call, which is not enough for many meetings. And there are discussions about moving to other platform but we wait till at least delivery foundation is fully formed to have a discussion on the CDF level. So Jenkins project now transitions with one from one umbrella organization to another one and we are hopefully we will be able to use meeting platform of Linux foundation once everything is established. Actually, I just think that I have another question about evaluation process on the face. So it will be evaluated by mentors or there will be some additional people who will be involved at this face and we will have some wide meeting with several with presentation and with outcomes deliverables and so on. So what our plan says that after coding phase, there will be public presentation performed by all students. So this public presentation would be one of the ways to showcase your project and show the progress and this presentation will be a part of the evaluation. Generally, mentors are the ones who do the final decision but there are also stakeholders. For example, I mentioned special interest groups and these groups they meet regularly there will be some stakeholders and you can use them as additional source of feedback and of course, mentors also may use these groups as additional source of feedback if they want. Okay, thanks. Yeah, JSOG program itself heavily relies on mentors to make the decision as Oracle means we have a way to make the decision if something goes wrong but hopefully you won't need it. Okay, maybe next question. As I understand for there is a JIRA which I already signed up so the work will be like inside the JIRA on the project or it will be something different what will be the workflow for the proposed features and improvements. It depends on the project so Jenkins project has JIRA and JIRA is a default standard for most of the Jenkins components but some components opted out to get half issues for example if you work on a component which prefers half issues then it might be there but usually it's around JIRA but JIRA is used mostly for tracking feature requests etc so nobody is going to do micromanagement via JIRA so we don't do time tracking we don't do all other crappy things which you can probably do in JIRA so for us it's generally a back lock some kind of tracker okay so basically I could break down my timeline and put some tasks into JIRA and work sequentially on them so usually it's what we do using the community bonding so we have meetings with students so we make decisions what we really implement sometimes mentors want to change the project a bit to make it closer to the Jenkins community needs or to adjust the timeline to make it more feasible and then after such discussions yes a number of tickets is being created in Jenkins JIRA and then you can use it as a back lock no need to create tickets right now unless you really want to request a feature even if you don't want to work on this project but during community bonding it will happen anyway okay I also I also quickly look through the previous year project and yeah it's quite impressive and the question is this don't remember the name girl still continue to work on her proposed feature or is it like finished from 2018 from 2018 yeah there is presentation on Jenkins site to remember some if you clarify what is this project about maybe I will be able to read it because yeah right now I don't understand which project you mean I just forget okay no problem I will I will send you the link later in more details if you just think that this year all former successful JSOC students participate as mentors so for example we have Alex on the call he was a JSOC student in 2016 other students Xingyu and Wutuan they also participate as mentors this year so yeah so you roast I also have some question I like the program to my fellow students and yeah maybe it's particular question to you Oleg if there will be possibility like another student to take part on on that part which I am proposed to what do you think it will be okay I mean not the copy of my proposal but like additional features with login or something else is it yeah so what is the requirement of JSOC in that yeah there may be project applications and project which have some relation to each other but yeah the main requirement that each project can be separated separately and that each project can be successful separately even if something was wrong with its counterpart so if you create proposals which can coexist with each other yeah it's perfectly fun we will still need to find mentors on our site so we can actually guarantee anything that both projects will be selected but yeah even the project applications are independent enough it may be possible okay okay I will tell my fellow students about this okay Google Summer of Code is mostly about working in open source communities so any kind of collaboration they perfectly fine okay Martina I have to drop in few minutes but yeah do we have any questions left? I'm done for now thank you I'm really excited to ask questions in GitterChat if you have something specific to discuss because yeah my name is Liss and GitterChat are the main communication channels and this is just a way to have syn cups when needed since thank you so much okay so any other questions yep I can take over Oleg I'm not logged in as Jenkins so I will not be able to terminate the the broadcast but I can log in as Jenkins if that helps if we have no questions left maybe we could just close on the broadcast okay and take the remaining questions on in the chatroom it doesn't seem to have any questions okay I meant additional okay so if there is nothing clear to discuss thanks everybody and see you in the chat bye thanks bye