My name is David Tolley, I'm with salesforce.com. If you were in this talk previously, Jim Evans is from salesforce.com as well. It's almost like we sponsored the talk or something. But anyways, I'm on something called the productivity cloud at Salesforce. It's a pretty interesting group that I work with. We don't really create features that you all as potential customers would interact with. We create features and stories that help our engineers to be more productive. So things like faster results for your commits, more reliable results, so instead of having to debug your test results and figure out if it was an infrastructure failure or a real test failure, our team helps with that. We try and make our developers life easier. Not only in that, but we experiment with new technologies. I can't really get into any specifics, but any time a new technology comes out, like Greg Wester over here, he loves new and shiny things. So anytime something new comes out, he jumps on the wagon. We experiment with it, do a proof of concept, give it to people if they like it, awesome. Some examples of things that we've done is something called strict unit test, where if anything in the JVM tries to access anything like an IO or something like that outside of the space, it'll just fill the test. Not only that, we created a new J unit for a framework at Salesforce, migrating from an older J and your three framework. That was all handled by our testing cloud.  All the data and all the metrics that I'm gonna be talking about just has to do with our platform. Salesforce has been around for over 10 years. We own a bunch of different companies like Heroku, ExactTarget, and these aren't small companies. They have hundreds of employees or thousands of employees. Salesforce.com itself has, I don't know, 13, 40,000 employees. But just a little disclaimer, I'm just talking about the platform right here. Before I get into the history of Salesforce, I just kinda wanna give a little bit of recognition to Simon and Jim and the rest of the committers here. In his keynote, he mentioned something. There's something along the lines of two or three times more jobs for Selenium than a web driver than there was for some other testing frameworks. And a lot of people give them grief. They either love web driver or they hate web driver. Some people call it an necessary evil. I'm able to provide for my family because of them, and I'm sure a lot of you here are able to do the same thing. So let's give them a round of applause and just, they help our lives. Let's give them a round of applause. So Selenium has a really interesting history at Salesforce. We're one of the early adopters, and I'll be talking about how extensive our test suite is, but around 2008, one of our soul distinguished engineer now, Doug Chasman said, it'd be great if we could test our UI. So they looked at Selenium MRC, they did a proof of concept, and it went really viral. The great thing about Selenium is it's so easy to write a test. You don't need to worry about dependency injection. You don't need to worry about your code being testable. You don't need to write your code in a way that other people can write a code. You're dealing with a UI. If you can look at a UI, if you can open up a browser, you can write a test for it. So it had really awesome, really quick developer adoption, which is okay in a certain respect. It's good, we're getting more tests. You know we're testing our application, but Selenium MRC is very, very flaky. We spend almost as much time writing new code as we do maintaining our existing test suites. So around 2012, we have about 15,000 Selenium tests, and for most people here, 15,000 is kind of a ridiculous number, but it's okay because today we have around 50,000 Selenium tests. Not only that, we have two committers on Salesforce payroll. Jim Evans and Luke Edmund Samu that Jim talked about. And unlike some companies, every single one of our Selenium test is run through our CI process. So every single commit that goes through CI every time someone checks in code, it's being tested against 50,000 Selenium tests, and all of our other tests we did as well. It's a pretty cool concept. So Salesforce engineering, just the platform, is pretty huge. We have over 1,000 engineers. We have architects that, they drive our scale. They make sure that whatever we're trying to do now is gonna be a scale in the future. They drive our overall design. Then we have the feature developers. They create the new features. They create the enhancements. They're chiefly responsible for the unit tests and some integration tests. But the really cool thing about Salesforce is our quality engineers are just as important, and even in some respects, more important than our feature developers. No, we ensure product quality through test automation. So we don't rely on a lot of manual testing. And manual testing just doesn't scale. Of course, whenever you feature new enhancements or release goes out, you need to do some type of manual testing. But for the most part, every single one of our Q engineers is a developer in their own right. But instead of feature code, they're writing test automation code. They're chiefly responsible for the integration test and almost solely for the UI functional tests. So Selenium, WebDriver, those different testing frameworks. We also have hundreds of projects. The product team side ranges from one to 10 people, 10 plus people. And it's pretty cool that I say one person. There's a really cool thing about Salesforce and being an engineer at Salesforce is we have something called PT on. So if you have a really fantastic idea or something maybe that's outside the box or something like, this isn't working out, I wanna fix it. Management will give you the time where there's a day, week, month, depending on what type of scale it's gonna be. They say, okay, you have time, go make a proof of concept, let's see how it goes. So there's many, many small one or two person teams of Salesforce where they're just prototyping these concepts, prototyping these new technologies, working on some new cool features. And that's a pretty cool thing that we have going on. But not only that, our developers and our Q engineers are really tightly integrated. The QEs are involved from the get go of new product or new enhancement, a new feature. They're working with the product owner, the project managers, whenever they're developing a new feature, they'll go on there, they'll give their feedback. Not only are the developers saying, how long they think it'll take to develop a product, the engineers and the QEs are saying, I think it's gonna take this time, right? Like not only are you gonna take time to develop it, it's gonna take this, around this amount of time to test it, to make sure that's viable. So each one of our engineers, whether they're developers or the QE type, the average is about two to three commits per day. That's pretty significant when you're talking about over 1,000 people, right? So around 1,500, 2,000, 3,000 commits per day, possibly at peak times. And again, every single one of those 1,000 plus commits goes through our entire CI process. And that's over 450,000 tests. And if each one of those commits were run by themselves, it'd be somewhere in the neighborhood of one billion tests per day. That's kind of a big number for people. A lot of people in smaller companies, smaller startups, they kind of shake their head at maybe 500 selenium tests per day. When you're talking about hundreds of thousands of selenium tests per day, it's a whole new ballgame, the whole new level of complexity that goes into it. So the CI process at Salesforce is pretty much basically the same as it is with any other company. We're really huge on agile development, but we're not so rigid that we say, each team has to be two-week sprints or each team has to be three-week sprints. We give a team the authority to kind of pick out their own time frames, right? That's not a really rigid framework. But we do have three major releases per year. Every four months, really big enhancements, really big new features will come out to salesworks.com. But along the way, there's a lot of frequent, smaller releases as well. Almost on a daily basis, whether it's fixing a bug, whether it's a product enhancement, whatever needs to happen, we have the ability to release every single day. Along with being able to test or release, test or commits, test or code changes, being a public company, we have to have an audit trail, right, to be SOX compliance and all that great stuff. So we have a really cool in-house tool called Gus. And the great thing about that is it's an internal, agile project management system that we developed. It has a really cool sprint wall. You can put your stories on there. And it provides a really great audit trail. So whenever you're writing your code and you're trying to commit it, essentially what happens at Salesforce in your commit description, you'll say what story it's linked to on Gus, what your changes is doing, what release it's going to, and then you'll submit it. And the cool thing about some of the automation that we have going on there is it'll look at that commit message, it'll go to Gus, look at the API, it'll compare the commit message to Gus, do some light verification, make sure that they match up. And then if it does match up, it'll send it to something that we call pre-checking. Pre-checking is just a really simple, fast test suite that we have. It runs our unit tests and it runs our JS unit tests. At Salesforce, one of our developers developed something called X unit testing framework where it's a JavaScript unit testing framework. It's really heavily used at Salesforce. I believe that he's open sourced it. It's a fantastic product. It really helps us along with our synium tests to verify that our UI is up snuff. So we get that result back in about 30 to 45 minutes, something along those lines. So fairly quick results back, some type of results back. If it passes that, it goes into our big integration and functional testing suite. There we have around 200,000 integration tests. So that's a lot of tests. Then we have the big huge time consuming chunk of tests of our 50,000 synium tests. And it's a lot of tests that we have to deal with. And because of the 50,000 synium tests, they take a long time to run. So that's one of the main reasons that we can't run all of our tests against every single change list. Essentially what we have to do is we have to take a group of change lists, batch them together, and run our tests against that. So whenever a change list does get run, we do some pretty cool things. We try and paralyze it as much as possible. We have different runs for different flavors of IE, different versions of Firefox, Chrome, Safari. Last year Salesforce released something called Salesforce One, which is our mobile app for Salesforce. It's a really cool tool. There's a ton of Appium tests that we have for that. Luke, one of our employees, that's a committer for synium. He's one of the people in charge of Solenoid. He does a lot of the Android tests for it. So there's a lot of cool things that we try and do to paralyze it, and we run a lot of tests. But the tipping point is we got to this point where we had a critical mass of tests, right? 50,000 tests running that take 10 hours to run. We have to batch all these tests together and run. We're not getting any really easy to act on results, right? If you're in a group of change lists that's 10 different change lists, you're getting back 100 different test failures. It takes a long time to try and figure out why the tests failed, right? It's not, these tests aren't being run on your own change list, so it's not guaranteed that your change list is the one that caused the failure. So there's a lot of logic and some automation that we do to try and figure out what's going on there. Not only that, so we run each one of these 50,000 tests, but every time there is a failure, we rerun those failures. It's something that we call flappiness at Salesforce. Some people call it flaky tests, but essentially a test will run, it'll fail. We rerun it and then it passes right. I'm sure a lot of people have experienced that. The longest run takes over 10 hours and again, we batch all of these commits together. So when failures do happen, it takes a lot of effort to debug trying to figure out what happened. And Salesforce is always trying to hire new people. We're trying to scale. We're trying to push out new products, new services as fast as possible. So we needed to not only scale our engineering team, but also our test automation suites. Like if results are continually taking 10 hours with 50,000 tests, you can imagine if we had 60 or 70 or 100,000 tests whenever we have new features, it just doesn't scale well. You know, we're releasing, you know, three major releases per year. There's a lot of testing that has to go on and so we needed to refactor our longest run test suite which happened to be a slain. So I was hired on to Salesforce last September, about a year now. Greg actually recruited me and my main, the main reason that I came on was trying to refactor our sodium test suites. It was trying to figure out why our test suite took so long, what we could do to improve it and what we could do to make it better for the future. And when I got there, everyone had their own ideas of why our test suite took so long. But no one really knew. I'm sure a lot of people here have talked about DevOps, right, or heard the term DevOps. Essentially that's developers being able to act on data, right? They have access to real-time pertinent, you know, information of what their code is doing in production. They can figure out, you know, how many times their code is throwing errors, you know, what type of CPU and IO problems they're having. You need that same kind of mentality when it comes to refactoring a test suite. You can't just say the sodium test has taken a long time. It's because of the slain MRC. Let's delete the whole test suite and move on. You can do that. That's not a very good way to do it. So I needed to get some data. One critical fact that we didn't know was how many slain MRC tests that we had versus how many web-diverted tests that we had. So we needed to create some methods around that to generate some data. So that was one of the first things that I had to do. So we had to figure out what metrics we wanted to deal with. We had to figure out which tests take the longest. I mean, the main reason that we're refactoring this test suite is because it took so long, right? We had to figure out what tests failed the most often. Not only does it take up a lot of developer time waiting for no longer in a test suite to come back, but whenever a test fails often, they have to go on there and make frequent changes. That's a big time sink for developers. Which tests are the most flappy? So whenever a test fails and reruns, that's obviously some kind of issue. Whether it's a test failure or infrastructure failure, we need to figure out which tests are the flappy, the most flappy, that happen more often than other tests. That's something that we should refactor. And along with having 50,000 tests, there's obviously gonna be some type of duplicates in there, right? There's teams that are said, okay, I'm gonna write this test for this feature two or three years later. So I was like, okay, I'm gonna do this as well. Now, there's a lot of duplicates in there. So essentially what we did is, we developed tools and developed methods and code to get those metrics for us. That was a key critical part in this whole process was developing those metrics and cataloging those metrics. So once we ran all of our tests, ran all the metrics that we needed to get, we divided each test by team. We sorted them in our backlog based on those metrics, based on which ones failed the most often, based on which ones took the longest to run. Then we started to act on those tests, right? So you can't just stop production at a company like Salesforce. You can't say, we're not gonna develop any new features now, we're just gonna refactor tests for this next spring or next month or two. We have to kind of sneak those into the backlog, right? Like our salespeople don't understand, we're gonna spend time fixing test code. Some people just don't understand that.  So we can have this backlog of tests, we have it sorted by priority. And so each spring, we ask the team to take two or three industry factory, right? Anytime we find a duplicate test, we run code coverage with it, we can say, hey, these tests are duplicates, that's to leave them. That's a great win for us, it's really easy low hanging fruit. So what's the current results that we have? It's still a work in progress. It's not done yet. I mean, 35,000 centium tests, trying to migrate those. Even though we have 100 teams and they're doing two or three per sprint, today we have about 5,000 tests to be migrated. That's not a lot, but 5,000 tests is larger than most people's like complete slay them sweet, not just refactor slay them sweet. Not only that, but since we're continuing to see you developing new features and new enhancements, we have thousands of new web driver tests that have been added to our backlog. The great thing about migrating to a web driver though is each one of our tests is able to be ran on all the different browsers. At Salesforce, a lot of our customers are enterprise customers, right? So it's a huge, huge use of internet explorer. A lot of companies don't really have that problem, maybe they only run in my Firefox, but we have to ensure that our number one customer and number one browser is definitely being tested. With SlimRC, we weren't able to do that very well, but with web driver we were able to. But not only that, now that we have the metrics and now that we have the data of what's tests, our SlimRC tests versus web driver tests, we're actually able to split up that sweet. The great thing about that is instead of taking 10 hours for the full SlimRC suite to run, we can run these web driver tests separately and they get back some kind of faster results. Again, our main job is to make people more productive. If they have results back faster, they're able to act on that faster, they're able to do a lot more with it. So there's a lot of great stuff going on. Beyond refactoring tests, Salesforce is investing heavily in new technologies. So we're going really big into dynamic testing infrastructure. We have, for each run, our testing infrastructure is thousands of dynamically generated test instances. We're going from an older technology to a newer technology. We're also embracing the SlimM grid. Dima's going to be giving a talk after lunch about SlimM grid. I encourage you to go to listen to that one. But essentially we want a huge farm of dynamically generated windows or browsers of different flavors connected to the SlimM grid that our tests can connect to. Right now we do a lot of static instances for our test which we want to get away from that. Part of trying to make developers more productive is to eliminate infrastructure problems. Whenever you're on running tests against static instances, you're running to, your test might fail for benign reasons, right? A test might fail because a previous test didn't clean up or maybe the browser before didn't close, there's a lot of problems that can happen. Not only in that, but we're moving our existing or we want to move our existing SlimM test framework from an older Salesforce hacked JNM3 framework into something that is faster and more lightweight. Right now our framework is really tightly coupled into our app code. So if you look at some of our tests, they're calling Salesforce Java classes directly instead of using APIs. We want to move away from that. We want to make it completely abstracted away from our app code. If we're able to do that, if we're able to use all of our publicly available API jars that we have, we'll be able to open source it at that point. That's cool because we have thousands upon thousands of customers that use Salesforce and a lot of them have custom workflows, the custom data that you're using on Salesforce and maybe they've created their own apps and put them on app exchange or something like that. But if we're able to open source this testing framework and let our customers test their workflows through WebDriver and this framework, it's just a win for us, right? So not only are we testing our code ourselves, they're testing their code. And if they find a problem before they're doing some type of release on their app on the app exchange, they can send us an email or a call and we're able to fix that for them. It's a pretty cool process. It's kind of interesting how Selenium started and the WebDriver transition happened at Salesforce. So around 2008, I said, we started WebDriver or started Selenium Marcy.  Let's try and get a prototype of that. And so they weren't really thinking about the scale of how it was gonna happen in the future. They were mostly saying, I wanna prototype this, I wanna get this proof of concept out as quickly as possible. So instead of architecting it in a way where it was completely separate, they put the WebDriver code in the same Selenium based test as the Selenium Marcy tests were. So we need to refactor that as well, get that into its own separate classes and do a lot of stuff around there. So going forward, there's gonna be a lot of different things that we can do. So I think one thing that people might wanna hear about is, you might be in the same situation that we are. You have a big backlog of Selenium Marcy tests. How do you get people to understand that it's worth the time and effort to refactor that test?  We tell them it takes a long time. We tell them that it takes a long time to debug our tests or refactor them or maybe we can't release as often as we want to because our test suites take so long. But whenever we try to explain that we wanna take maybe a month or so to refactor our Selenium Marcy suites, they look at that number and say, no, we're not gonna do that. We're not gonna take the time. It's not worth the money. So a lot of times people just complete their Selenium Marcy suites and start over. I'm not a big fan of the idea. There's a lot of useful data that those old test suites have. So I'm a big fan of trying to systematically refactor tests and move them over to a faster testing framework. So the first thing you have to understand is you have to be data-driven. You can't just have hunches. You have to be able to go to an executive and say, hey, our test suite takes X amount of times. And it's because of 10% of our tests that take over two minutes each or something like that. You have to understand why your tests are slow. Is it because it's Selenium Marcy suite itself? Is it because the way that you're writing your test? Are you doing really a long running test? Or are you using API to generate your test data? And then just using the web-diver to verify that that exists correctly. You have to be able to figure out what your problem areas are. What I mean by that is, if you're able to historically look at your CI build and your test failures and see a trend over time of which tests fail the most often, that's a huge data point to have. And you have to be able to figure out why those tests fail, right? So not only are you looking at which tests fail the most often, you have to understand why they fail. If it's an infrastructure problem, that's one thing that's completely outside of your test suite, right? If it's an infrastructure problem, then you can go to your ops team or your networking team and try and figure that out. So then you need to collect, once you have the metrics, or once you know what kind of metrics that you wanna talk about, you have to collect those, right? You have to put them in a meaningful way. Lucky with us at Salesforce, they gave me the leeway to take a few months to put in the test code that I needed to, to put in the classes that I needed to and the methods that I needed to to generate those metrics. And then you have to put them in, like a tool that we use, like I said, Gus, put them somewhere where people can easily see those metrics and easily act on that. One thing that you have to kind of explain is, when a test fails, one of the biggest time scenes is trying to figure out why that test failed, trying to debug why something failed.  So why does that happen? And if you can get some kind of concept or wrap your head around the idea of how many hours per release or per day or per sprint are spent troubleshooting your cinema C tests, that's probably the most important data fact that you can generate. If you're able to say you were spending 10 man days per release, fixing test code, you know, people are gonna be able to act on that. That's something that someone can grasp that maybe doesn't actually write tests, but if we're saying we're wasting 10 days per release, you know, people are gonna act on that. And some of the benefits of this factor, I mean, it's pretty obvious, but if you can release more often, we're gonna be able to get more benefits and more features out to customers. If you can release more often, whenever an issue does happen or whenever a bug is found, you can get your change list up to the CI process faster.  Okay, it passed all of our sodium tests, it passed all of our integration tests. You can get that result back in an hour or something like that. You can push out that code. You have some level of confidence that it's not gonna completely break your production. And again, find out how much money can be saved by not debugging existing tests. Executives care one thing a lot, it's money, right? They don't wanna waste money, they don't wanna spend time figuring out. They don't wanna spend time wasting here and people saying all our sodium tests, these aren't great, why? They want hard data, they want hard numbers. And it's up to us to explain to them why this needs to happen. So with that, did any of y'all have any questions? I'd love to hear your questions if you have any concerns or thoughts or if you're trying to go do this migration yourself. Right. In that code coverage, you find like which UI, I mean this elegant test is covering what type of code, how do you get that code coverage, please? Right, so code coverage is anyway, it's multiple tools that you can use. I mean there's like Emma, there's Kobutora, Jacoco, things like that. The way that we're able to generate that matrix is kind of not necessarily using our existing CI process in our order, but we have a completely separate run set up to do that. One of the colleagues on Greg and I's team, his name's Petal, he developed a system called Argus that runs each test serially. So it's kind of not the most efficient process, maybe one day we'll discover a better way to do that. But essentially we have all our Java code, we compile it, say with like Emma, code being in there, and then we run each one of our tests serially and then we can collect that data. So we know which test, which piece of the app code. So then we can collect that data and put into a database and kind of correlate that data together. A lot of times we're not gonna find the exact same test, right? I mean, they're not gonna test everything exactly, but they're gonna be similar enough that we can say, hey, we don't need two or three of these, right? So then we can go ahead and delete two of them, take that one test, put it into our backlog and refactor that later. Yeah, that's an interesting question. I wish I could tell you exactly what we're doing in the future. For right now I think I can say that we have a very large server farm of static instances that we have. So pretty much just bare metal machines that we have in a data center server, right? You know, thousands and thousands that we have hooked up to what we call auto build, it's our internal CIS system. But not only that, can I talk about VMware? Well, I said VMware, so why not? We are older dynamic infrastructures based on VMware. So what happens is whenever a change list needs to be tested, the batched change list will say, okay, I have these change lists, I need this type of infrastructure to be brought up. So it'll bring up an app and DB server. It'll bring up, say, six instances of some type of Windows VM per Selenium run, right? So again, we break up our Selenium runs by browser type. So six instances of, six serve basically Selenium slaves for IE eight, for IE nine. Then we run all of our tests against those different browsers. We're going to something newer, more dynamic. We've invested a lot of money into a new technology that's coming out, not to use on production, but just for our testing infrastructure. It's really cool and exciting, but PR said I couldn't talk about it right now for some reason, but I'm sure you can guess it's one of the really cool new open source dynamic frameworks out there. We've invested millions of dollars into it and we're going to invest millions of dollars more into it. But yeah, every single run gets an entire dynamic stack and test resourced is brought up into it. And it's all done through several data centers that we have. Right. When I mentioned WebRiver, it's solely paid job based. So we did a really great job with our implementation of WebRiver, maybe not in the classes that we put it into, but in our page object way that we wrote it. All of our WebRiver tests are done through page objects. The great thing about that is, I'm sure you all know about page objects, but it's basically abstracting your test away from the backend WebRiver code itself. So whenever a test fails, say 10 test fails for one particular bug, instead of having to edit 10 different tests, you can just edit the page object and all of a sudden your 10 different tests are working. So there's a couple things that we do whenever we refactor a test. Not only do we convert it to a Tobib driver or in the page object framework, but we also see if we can test it in a different level of test. I mentioned a JX unit framework for JS unit testing. We try and look at the test and say, you know, what are this really doing? Is it really testing widgets logic? If so, can it be an integration test? Is it just testing JavaScript? What I've seen is a lot of people just test JavaScript with WebRiver or Selenium. If it's solely testing JavaScript and clicks and know what happens after the click or animations or something like that, we can actually migrate that down to an X unit test. The great thing about the X unit test and X unit framework is one test takes microseconds to run. I think we have like 10,000 tests and it runs in under a few seconds. So it's really lightweight, really quick. So not only do we just say, hey, we're gonna migrate this over to WebRiver, we're trying to see if there's some other time in a test suite that we can run it in. Because no matter how much faster WebRiver is than Selenium RC, there's test suites that are even faster than WebRiver. We try and make that determination as well. How many times do we run our failed tests? Right, we run it at least once, do you? Greg, do you? So a test of fail, we'll rerun it, put it in, we'll flag it into something that we call Yoda. It's basically like a test failure, a test run logic, right? It collects that data over time and then we're able to determine what we need to do with that test. If it's a flapper test, we can go in and refactor it. If it's, the test just needs to be deleted, we can delete it. If there's something about the code or the infrastructure dealing with that test, we can address it there as well. But the reason that we had to develop Yoda is that, again, we batch all these change lists together and run them as a big group. Maybe there's one or two or maybe there's like 20 or 30 change lists in each run. So we developed this Yoda technology that can look at the tests, see what tests they're hitting or what files they're hitting, see what kind of change lists have historically been dealt with those tests and it's a big complicated process. We have an entire team, just three or four people that's all they do is trying to figure out why the heck our tests fail and try and get that out programmatically. So yeah, it's a big deal for us and it's a pretty cool tool that we have. I'm sorry? No, we actually do only, I mean, Salesforce is a big Java company, right? I mean, obviously we have other products that are investigating newer technologies. We've bought a lot of companies along the way. I mean, people don't know that we own Heroku, right? I mean, that's a pretty cool company, a pretty cool newer technology, newer stacks. But for the most part, our platform is written in a Java language. And again, each one of our QE engineers is pretty much a developer at heart. A lot of them used to be feature developers, but they have really a big passion for testing or something like that. So they're all pretty knowledgeable. So we don't really try and extract it away from QCumber, with QCumber or anything like that. They're actually writing real Java test classes. Again, using the paid Java framework, so it's a little bit abstracted. So you create your paid objects for a certain page, you put in all your web driver code in that paid object and then the actual test themselves are just interacting with the paid object and doing some type of verification. But no, we don't use QCumber or anything like that. Yeah, so again, like I said, this is just a platform, right, of what I was talking about. There's a, historically, a Salesforce has been one huge application, one huge code base. There's really significant effort going on right now to mavenize our entire build and to break out certain projects and certain features into their own code base. And in their own CI processes and then put them into like an extra repository. And then whenever the main app runs, pulling down those compiled jars and like that. But yeah, our testing infrastructure, I mean, thankfully, I work for a company like Salesforce where they can invest millions of dollars into our testing infrastructure. Yeah, we do have a pretty significant, I don't think I can get into specifics. We have thousands of instances, right. Yeah, so we have thousands of instances. Again, we invested in newer technology and it's going to significantly increase our existing testing capability. A lot of really cool things that we're gonna do with it. Yeah, as I mentioned with the DM over here, we did move to Page Object. One thing is web-driver tests in general are much faster than SlimRC tests. With the older SlimRC tests, it does JavaScript injection into the browser. Okay, so essentially what we did is we used the data to collect information on the particular test to put it into a backlog, a Q-type system to see what we wanted to refactor first, right. So we have a nice backlog, each test broken down by team and sorted by priority versus how long they take to run, how many times it's failed and different things like that into the backlog system, right. So that's pretty much where we collected that data and then we put it into the system I mentioned before, GUS. So we have a big, nice backlog along with all that data available to each one of the scrum teams. And again, each team pulls back from that log, each sprint, refactors a couple of tests and goes from there. To make sure that what we did actually works, we collect data on the newer test as well. So as I mentioned, we were able to separate out our SLAMRC tests from our web driver tests. So we're able to collect that same data on our web driver suite now. So we have a database of information on our tests, on our current web driver tests as well. Assertions for the web driver test themselves or for the data. Yeah, we abstract that away. We don't want our test suite to do anything more than test, right. So essentially it runs, we send all our data to something that internally we call a Cloudy Boy or Paper Boy, it's basically an API on top of a database. So each one of our test results go into a database and then we're Salesforce, so we can do a lot of cool reporting with our data. So we're able to do a lot of really cool metrics, a lot of cool spreadsheets and reports based on that information. So we have, historically we have a static server farm where they're always up and running. This is our older implementation. So if a test runs on that, those VMs are just up and running and all we do is inject like server information into the environment. It goes into our web driver test suites and it says, what server am I gonna hit? It goes and it just hits against that. With our newer implementations and the VMware and the one that's coming out, per run we dynamically spin up everything. So we don't have any stagnant instances in our new systems. Each commit gets, again per web driver build, they get six dynamic instances. So say for IEA, IE9, IE10, each one of those will just automatically spin up six new Windows images. But not only that, we spin up a brand new app and DB instances as well. The app and DB obviously runs the code that's supposed to be tested and that also acts as our test runner as well. So with the newer implementation, everything is being run on pristine, dynamically generated instances. Exactly, we're huge on mobile. Again, one of our employees is one of the committers for Solenoid, which is the Android version of web driver. I gave the caveat that we were just talking about the platform, but we have a huge infrastructure dealing with mobile testing as well. I'm sorry, it's kind of a mixture of both. One thing that I'm actually exploring right now, I don't know if you've, have you seen Jason Huggins' tapster bot? Basically it's a web driver on top of like an iPhone, it has like this cool like a Raspberry Pi based servo system to actually interact with the screens. Again that's, like I said, our team likes to explore these new kind of cool technologies, and that's one thing that I'm exploring right now. So we have physical devices, we have emulators. Sometimes we use SOS labs to do a lot of our mobile testing. There's a lot of cool things that we do. Automated, layout testing, so each sprint we collect something called a gold standard where we kind of take screen shots of our releases, but we don't really have an automated way right now to kind of correlate that information. I mean obviously one of the cool things about web drivers is you can crawl through pages, you can take screenshots and there's a lot of cool things you can do with that. And then if you have a gold standard of what a page should look like, I mean the sort of trivial to kind of compare and contrast you know between what web driver says is current versus what it should be, but currently we don't really have anything like that. It would be something that we could explore in the future. The credit set has a lot of custom objects that our employees have, right? So much so that we can't necessarily test every single custom object for every single CI run, right? Not only in that, there's customer specifics on separate pods that they don't necessarily want us to touch right there, so they don't want information. But we try, we use something called dots, like data on template or something like that. We're basically, we can take an organization, we can take a bunch of custom objects, we take a bunch of test data and kind of zip it up into one file, and then for each test, literally for each test we create a new organization, new custom objects, everything for that test. And we're, so in that respect, we're able to test against custom objects, custom workflows, custom organizations without having to do that through the UI. We don't have to set up through the UI. Everything is set up through that dot. And we're continuously improving that dot. We're taking customer data, we're taking these things, we're combining to the dot, so we're continuously testing new workflows and new data. We don't use it for like load testing and things like that. There's better tests out there for them, like JMuner test and we have some custom tools that we've built in-house. WebDriver and Selenium tests are great for testing that something comes up when it's supposed to test, right? I'm not a big fan of using it for load testing. There's just a lot of interstices and a lot of things that happen if WebDriver going through the JSON wire, talking to the browser, talking to the different instances. There's a lot of stuff on top of that that would get in the way of like a true load test. But we do have load testing that we do at Salesforce, just not through the WebDriver. Cool, I think we're done. Thank you all so much.


I'm super pumped. Yo testing out this week random note. I did yoga twice this week to heal my Yeah, my body right I've been tight my entire Woo yo today's dunk session was a huge test in my entire journey I finally started yoga. I've been dealing with this hamstring tight hips tight back all these different things And yoga was the answer, but I had these fears in my head these stories in my head.  I don't want to do it. It dehydrates me with the hot yoga Finally did it. This was the day after day two of yoga. So I did Monday Wednesday dunked Thursday So this was like, okay. I did it twice this week. I'm extra gumby. I just did it yesterday I lost all my fluids. Let's see how I jump and so you're gonna see here How that story got completely eradicated 2021 your boy is doing yoga and I'm gonna try to get flexible if I can get these hips Mobile if I can get these hamstrings healthy anybody in the freaking world I'm gonna feel like the authority on hip range because I've never met somebody as tight with the imbalances that I have and I want you guys to avoid that And I want you to take care of your body. So I have the same thoughts. I don't want to stretch I don't want to do this if you want to go hard in the gym You have to go hard in recovery and everything else health first.  Let's go so we can have fun dunking. Let's go Almost more agile than ever. I'm not I'm really light on my feet. I think it's from the yoga I'm actually feeling really great really bouncy that last one. I tried to go so hard I gotta I gotta get gradual into that. I can't go 112% yet. I still gotta go like 98. Let's go Have a great 2021 I'm more pumped than ever today's podcast the first one of 2021 was amazing Hope you guys enjoyed. Thank you for watching Thank you for tuning in so much more to go so much more content learning about myself. Look at this studio I'm feeling good. I'm painting what's going on. Have a good freaking decade And life every moment is the best now. Love you to the loop


After a year of planning and preparation, Marine Week is finally in Cleveland. Hi, I'm Sergeant Todd Hunter here in the city's public square. We're just moments ago, Mayor Frank Jackson issued a proclamation kicking off Marine Week. I encourage all Clevelanders to join me in recognizing the contribution the United States Marine Corps has made to our nation and to the city of Cleveland. General? The event is in its fourth year after successful bids in Chicago, Boston and St. Louis. And while the location is constantly changing from city to city, the concepts of community, country and core remain the same each year. It's all in an effort to bring the Marines and local community together for citizens to gain a better understanding of how the Marine Corps serves the nation. As hundreds of Marines reach out to Clevelanders through volunteer service projects, youth engagements and numerous outreach events, I believe you will see the esprit de corps, our sense of unity that defines our warrior ethos and attracts the best and the brightest. Marine Week Cleveland goes to June 17th, and the Marines encourage anyone in the area to come out and see what the Marine Corps is all about. Of course, if you're unable to make it, you can get all the latest from Marines.mil or any of the Corps's official social media outlets at Facebook, YouTube and Flickr. From Marine Week Cleveland, I'm Sergeant Todd Hunter.


Welcome friends to another session of trigonometry. In this video, we are going to discuss a theorem which says the length of the circumference of a circle bears a constant ratio to its diameter. So all of us know that circumference to diameter of a circle is a very famous constant that's called pi. Circumference of any circle divided by the diameter of that circle is always a constant and that constant we know that as pi. So now we are going to prove that circumference to diameter ratio for any circle is constant. We will not be evaluating the value of pi but we will be proving that for any circle or for all circles the ratio of circumference to its diameter is always a constant. How do we prove it? So let us draw two concentric circles which is shown here in the figure. So there are two concentric circles with the common center O. Let us also inscribe a regular polygon. So if you see I have shown few of the sides of a regular polygon whose vertices are a dash, b dash, c dash, d dash and so on and so forth. This is a n sided regular polygon. Regular polygon means all the sides are equal. Regular polygon means all the sides are equal.  So O A dash, O B dash, O C dash and O D dash these are radii of outer circle. Similarly O A dash, O B dash, O C dash and O D dash cut the inner circle at point A, B, C and D. Now if that is so then O A, O B, O C and O D are radii of inner circle.  Clearly O A by O A dash is equal to O B by O B dash. Why? They are all equal to R by capital R where R is the radius of the inner circle and capital R is the radius of the outer circle. This hence, hence by converse of, converse of Thales theorem which you would have already studied Thales theorem or which is also called basic proportionality theorem. I am writing the acronym B P T, basic proportionality theorem. We can say that, what we can say is AB is parallel to, if you see AB is parallel to A dash, B dash. This is coming from the reverse, oh sorry the converse of basic proportionality theorem. So we can say AB is parallel to A dash, B dash, A dash, B dash. Similarly, if you see, similarly BC is parallel to B dash, C dash and others like CD is parallel to C dash, B dash and so on and so forth. So all these are parallel. So AB is parallel to A dash, B dash like that. Okay now when they are parallel, so what we can say about these, let me redraw these two triangles. So this is, let us say this was our triangle and yeah, so this is O, this is A, this is B, this is C, sorry A dash and this is B dash. And AB is parallel to A dash, B dash, that means this angle is equal to this angle and this angle is equal to that angle. Now again in triangle OAB and triangle OA dash, B dash, what do we observe? We observe that angle OAB is equal to angle OA dash, B dash because AB is parallel to A dash, B dash. Similarly, angle OBA is equal to angle OB dash, A dash therefore by AA similarity criteria, angle angle similarity criteria, we can say triangle OAB is similar to triangle OA dash, B dash. Hence when two triangles are similar, what happens? The ratio of corresponding sides are equal. So hence we can say OA by OA dash is equal to AB by A dash, B dash and this is equal to small r by capital R. Why small r by capital R? Because OA is equal to small r and OA dash is equal to capital R, right? Radius of the two circles. Now let us find out the perimeter of the two polygons now. Hence I am saying perimeter of inner polygon upon perimeter of outer polygon. I will tell you why I am doing this but let us calculate this.  This upon N times A dash, B dash.  This is AB, so N times AB, N sided polygon it was. So N times AB and N times A dash, B dash. This is A dash, B dash and this is AB, right? This is what the ratio of perimeter of the two polygon. Okay, so hence which is nothing but AB upon A dash, B dash. And we have just proved above here that AB by A dash, B dash is equal to small r by capital R, okay? By similarity we just proved above. Now when N tends to infinity, right? Polygon, the regular polygon tends to circle, the shape will convert, get converted into circle. What can we say about the perimeter we can say? As N tends to infinity, circumference of inner circle upon circumference, circumference of outer circle, outer circle is equal to small r by capital R. Hence, hence, circumference, circumference of inner circle by small r is equal to circumference of outer circle by capital R, right? So if you divide both sides by two, what will you get? You will get, dividing both sides by two, you will get circumference of inner circle divided by two r is equal to circumference of outer circle by two capital R. And what is two r? Simply D diameter. So hence I can say, circumference, circumference of inner circle, inner circle, divided by the diameter of inner circle, diameter of inner circle is equal to circumference of outer circle, circumference of outer circle divided by dia of outer circle, right? Now this ratio doesn't depend on the size of the circle, that is the radius of the circle. So hence it is true for all circles. So hence we say the ratio of circumference of inner circle upon the dia of inner circle is a constant. That is what we had an objective to prove, right? Whether it is any value of r, this relation will always hold. So hence what do we learn? We learn that circumference of any circle divided by diameter of that circle is always a constant value, which we now know is pi.


Brian. This is Tom O'Brien of TFNN. We go five days a week. We go seven hours a day. We go 24 hours a day on the internet at TFNN.com. Always remember, folks, whatever you think about, you bring about whatever you focus on, Garose. Hope everyone's having a great day, safe day. Just making a great night, folks. Create agreements, new agreements based on love and respect. Take the responsibility to make new agreements with those you love. If an agreement doesn't work, change the agreement, create a new one, and use your imagination to explore all the possibilities. My good wise, let's take a look at it out here. We have the Dow Industries up 301. Nasdaq's up 26. S&Ps are up 23. Gold. Gold contract up $11.20. Trade to 2,015 an ounce. We have Silver up about $0.55, $22.94 an ounce. Look at this copper market, man. Copper, doing an ABC up to 401, 404. We're at 375. It's up five and a half pennies right now. Oil. Oil is a buck 65. $78.30 a barrel, notes and bonds. A 10-year note, up four ticks. Trade in $1104. The 30-year up nine ticks at $118.26 and $Kingdollar. $Kingdollars down $421 ticks. Trade now to $104.302. Euro at $107. Yen, $149. British pound, $125 to $1. US dollar. Offer number is<PHONE_NUMBER>. Give us a call, folks. I don't know what's going on in your world. Down in the world of the S&Ps, let's take a look at it, because I suspect what we have here is that you've got to remember something. President's Day is Monday. We'll close Monday. So we'll see what the action is tomorrow. But most times what you do have, you've got to remember people in the northeast, okay? Their children are off. There's a school vacation. So more than likely tomorrow, even though the PPI is coming out in the morning, we're probably going to have lighter volume tomorrow also. We'll see how this shakes out. But right now, what you have is this, is that you come off the high. You come off with volume. Two and a half weeks ago, we also had volume. And I think that's the target right now. I still think that's the target that's going to get down and retest. I don't see anything more than that. I can tell you that, because it's not enough sellers. But we'll see how this shakes out. So right now, we had that contraction. Well, yesterday, the volume wasn't that bad. You know, he was still coming at 68 million yesterday. Today, we're at 40. So you're going to do 50. And that's the type of contraction that you're looking for. Now with the highs, we only did 56, okay? We're already at the lows of the highs, because the lows of the highs was 500.24. So you're already into the bar. So that's saying you're going to go after the bar. Well, it's going to get intriguing here, of course, is that if the PPI comes in bad, meaning that would be good for the marketplace, you know, we'll see. The test is going to be tomorrow. It blows by and it blows by it, you know, if it basically oscillates around it, that would be saying that it will come back and you're going to have a small consolidation right here. And the X100, we take a look at the three Qs. And we're going to have A, Mac, I would have to close out here today, which, you know, we'll have some action here. So on the three Qs, you have the same contraction. Okay, yesterday what we did is 48 million, you're only at 31. Let's see, the bottom of that is the 435. Yeah, we haven't hit the bottom of that high yet. You know, I suspect we will tomorrow though. We go into the gold contract. And what do you have with the gold contract? You know, my take on this gold contract is that I think we basically have a blowout bottom, but we'll find out how this shakes out. You get 1.9 million contracts right now, which is great contract volume, by the way. Yeah, it's good contract volume. And we are back inside the range. You need to get back in 2007.40. And you're back inside the range once again. And we go over to the dollar. You know, we look at the dollar, you know, it's all about the dollar and interest rate structure still. And that's not going to change, folks. Okay, so the dollar did just over 0.618 by ticks. Okay, of the whole move down. Now the question is going to be, does it get back inside the lower range? You know, right now you're 104, 255, you are down 248 ticks. That being said, though, there's a lot of ranges here, and you're still on the higher range. We go take a look at the note and bond market. And that's getting some juice behind it also. Pull this back up. Okay, so if you take a look at this, we got to 110.17 today worked 110.04 1.7 million contracts. That's good contract volume, but it's still going to need more. But this is the building cost to repair the damage that was done on Monday, because that was some heavy damage. There's no two ways about it. Now that damage was still with the light of volume, then we went up on. So that's still saying that there's still more buyers than sellers out here. Volatility, I expect the volatility is going to be choppy. It's going to be out here. It's going to be all of the above. That being said, we're on the other side of the interest rate structure, you know. And now let's go to copper, because copper, and you can see it on the, you know, I brought up a couple of copper stocks yesterday, TGB in particular, and TGB was moving before the copper contract was moving. You can see that the last high of the copper contract, we made it to 397. When you put this on a larger one, I've been looking at this a lot lately in the last six months, because on a larger generic contract, when you bring this back, this 405 is sticking out like a sore thumb, actually. You can see, is that the 405? Yeah, here's, well, that's actually 435, so I'm not even going up to that point. The 405, the way this is on a lot longer term basis, you can see it's a nice setup, man. This level here that we came down to on a monthly basis didn't get into its lows. The lows would have been 301, we rejected three, sorry folks, so far away sometimes. Yeah, we rejected 365. So that's saying that, you know, even 435 is gain.  It actually, yeah, see, it moved. This move before the copper contract moved, TGB knows that copper is going higher, man. And every penny that copper goes higher, you know, that equity loves it. I should love it, because it got millions of ounces of copper. Stay right there, folks. Come back, we'll have Mr. Tim Ward. We're going to go over these markets. Come right back.


is the same okay time to go back home imagine that i would be actually doing the broadcast from home but technology is amazing happy new years by the way i want you to stay i just finished my radio broadcast and i'm having my dinner at 10 15 p.m made myself a really simple korean meal a very korean meal with minimal panchans or side dishes i just got my bowl of rice yes i am actually using this bowl i have my fried eggs with kimchi and i have a side of kim seaweed labyrinth my sleeping schedule is so messed up right now i think i also couldn't fall asleep because i was watching the dead sea wait was that what it's called the silent sea wait the quiet no i can't remember the name silence of the sea wait let me check the name it's the silent sea i got it confused with the silence of the lambs said silence of the seas anyways the silent sea and that had a lot of jump scares so my heart was pounding the entire time obviously it's not going to be helpful if your heart's pounding like that to fall asleep so that was my mistake but yeah i ended up finishing the series in one sitting i didn't know what i was going to be about at all i just knew that it was something to do with space but i didn't know it was going to turn out the way it did i think i just always kind of enjoy anything related to space or space travels because that's something that i always wanted to do ever since i was young like i just want to be able to go to the moon go to space so i think i'm like romanticizing it tomorrow i'm planning on exercising editing what else is there to do yeah just those two all right i'm gonna go enjoy my meal so i'll see you guys tomorrow hours of footage i thought i didn't even vlog that much i took another nap i don't know what it is i just been feeling down i'm getting ready for radio at least radio gives me a reason to like get ready and actually look put together or else i would just look like crap during my entire quarantine i'm probably because i'm not moving around a lot or doing much i don't feel hungry so i literally just had the smoothie for breakfast and haven't eaten anything since it's five p.m soon though and i'm trying to eat something before radio today and not after because i'm trying to fix this habit of eating so late man this is only day three and i'm already feeling like this i thought this would be super easy it's not so this is the tteokbokki i have i was holding out it's called it's not too spicy just the right amount that i like comes with sauce fish cakes and rice cake i'm gonna add some veggies to this too i have onions that's about it oh i'm also gonna boil some eggs so i'll do that first while this is thawing for a little bit i make this delicious tteokbokki and even added mozzarella cheese on top well if i'm gonna eat i want to make sure it's good you know hopefully this will make me feel better and make me more energized for my show i put it in this green bowl after elementary school when i go to the street cart vendors to eat tteokbokki they would kind of serve it in this bowl this is a very nostalgic dish the rice cake is so chewy i don't know what day of quarantine it is but yesterday i did not vlog at all yesterday was just not a good day i slept the entire day i it was also the first day of that time of the month so i had really bad cramps like i don't remember the last time i had cramps this bad so basically i just had tylenol and fell asleep woke up again for a little bit started getting really bad back aches again so i took another tylenol when to sleep and then this morning i woke up at seven and now i'm having my breakfast just apples with peanut butter but this peanut butter is making me feel better because it's darn delicious i got it from target when i was in new york it's from good and gather organic creamy peanut butter spread it's so creamy and soft and since i woke up early i'm going to try to have a more productive day i'm so over this though like i'm so over it i just wanted to end already so frustrating they gotta clean my closet this is the best i could do right now i just have too much crap also i have this box here that i'm planning on donating i just have to wait for the end of quarantine to send them so every day at 10 a.m and 4 p.m i am required to take my temperature and submit it into the app that i've downloaded it's called chaga 경리자 안전 보호 and it looks like this they sent me this sticker that can detect your temperature along with some masks some hand sanitizers some garbage bag that has a radioactive image on it and i'm not a lot to throw anything out until three days after my quarantine is over um so this is really cool i just take this sticker put it on my wrist like that and leave it there for about 10 minutes immediately the color has changed brown means below 36.5 green means 36.5 and yellow means fever so like 37.5 plus so as you can tell it's green which means my temperature is normal it says do you have any of these side effects today are you coughing no does your throat hurt no do you have trouble breathing no and then i submit it yes so i have to do that twice a day as soon as i put it on this it's gonna turn back to brown oh whoa whoa whoa whoa so cool and i can use the same sticker for two to three days it's already friday so i'm halfway there i bought some things to eat while in quarantine so i thought i would do a little haul of the things i bought at market curly a bagel a garlic bagel from 4b sweet chili sauce chicken breast mini punggabbang oh my god the kimchi fried rice because the reviews are good for this anyways i'm very excited about these punggabbang i've been craving punggabbang but because i can't go out to buy from the street vendors i decided to buy my own and this is sweet potato flavored so there's sweet potato cream inside i got a mini spring roll set i got myself an avocado oh my god so unripe so green i got an oat milk and i got great yogurt now i have some things to eat while i'm in quarantine breakfast is served actually no wait breakfast is really served i added some cinnamon powder to my apple phoebe got me this cup as a gift i have way too much greek yogurt leftover so hey air fried piece of the bagel i'm gonna eat this bagel with the greek yogurt this is almost like cream cheese i'm really sad i only had bagel once in new york shame yum yum my warmum some spring rolls oh that looks so yummy all these black lines come out whenever i film in the kitchen this is something to do with my kitchen lights so healthy and delicious i basically just edited the whole day today i slept a lot so my eyelids are like out of control right now so if you haven't watched part one of my new york trip yet please do so starting tomorrow i gotta edit part two of my new york trip also when i went to new york i didn't bring any of my skincare stuff because i was like i'll be fine using my moms and then towards the end of the trip i started breaking out like crazy i have like pimples here on this side and as soon as i got back i've been trying to cure it as soon as my quarantine's over i also need to go to my skincare place to treat these i got too cocky i was like my skin will be fine and then bam in the beginning i was like oh my god quarantine is never ending like what the hell and now that i only have a few days left i'm like can it's can time slow down a little bit because it's going back to a fast and i'm not ready to go back into the real world something other than pj's but still cute and comfortable this set is from cordera it's a spanish brand so i'm guessing it's pronounced cordera and it's a sustainable and ethically made brand everything is upcycled it's super stretchy super comfy perfect loungewear thank you cordera for sending me the set moon t-shirt so something i've been watching during quarantine is singles inferno or solo jiok in korean kind of like been my guilty pleasure during quarantine i'm on the last episode right now i don't know who i'm rooting for i feel like there has been a lot of dating shows recently in korea dating shows are nothing new here i mean it's been around for a long time but i feel like right now there are a lot of dating shows coming out at the same time it is fun to watch and i also think about like how i would act if i were ever on a show like that and i feel like i would just end up making friends with everybody so in the end i'd be friends with all of them but i wouldn't be able to find love unless there's somebody that i really like then i feel like i would really throw myself at him but i don't think you would know until you are actually put in that situation do you guys ever imagine yourselves being in a dating show how do you think you would be let me know because i think it'd be so fun to read the comments i wish the camera would just automatically turn off by itself turn off turn off currently it's like the coldest today i have leggings i'm gonna wear another pair of pants on top of these i'm gonna wear these sweats for jacket like should i wear a beanie i just took a shower yesterday but i know it's gonna keep me warm uh i'm torn i don't wear hats the day after i take a shower because you know it's a waste but i should dress warm but i look warm or what i'm ready for the cold all righty let's get going oh my god the fresh air feels so nice i am back home now it was nice to be able to leave my house for 40 minutes um the weather it was cold but the air was really nice there's no fine dust today so it was nice to be able to breathe some fresh air the brain poke wasn't too bad it didn't go that deep so thank goodness now i just have to wait for the results to come out tomorrow oh my god i really have to do something about this nasty hair we're back in new york last day of at home broadcast woo i'll be in the studio tomorrow evening wearing a robe perfectly embodying the at home experience and quarantine i'm really not a work from home kind of person i need to be out guess what with quarantine i was supposed to be done an hour ago my quarantine ends at 12 p.m noon sharp today but guess what i overslept to one p.m and i woke up to like six missed calls from the people that take care of the quarantine and like five text messages asking what i tested whoops yeah i didn't know i was gonna oversleep this much i was supposed to go to the gym but now i can't because it's too late i'm gonna stop by the film store to get my instant camera developed i freaking forgot to bring my film camera to new york so i had to buy two disposable ones at urban outfitters i don't have time fresh all right i'm in chocolate today and we kicked out the show with chayol by kim hyolyn


Hello, and thank you for clicking. In managerial accounting, you will learn everything you need to learn about this topic. From A to Z, you'll have multiple choice, true, false, note, and PowerPoint slides. Also, this topic is covered on the CPA BEC section. See you there.


Hello everybody. My name is Saskia Hilteman. I work at the Erasmus Medical Center in the Netherlands, and today I will talk a little bit about storage management. And in particular data libraries. Data libraries provide a convenient way to share datasets with your users. This is great for commonly used datasets such as reference data or data needed to follow GTN tutorials. Users can access this data by the top menu bar under share data and then proceed to data libraries where they can browse the different data libraries available to them and import datasets directly into their history to start using them. Another nice way that users can access the data and data libraries is directly from within the tool form. So instead of first importing to their history, they can open a tool, browse datasets directly and choose one of the data library datasets to use as input for their tool. One of the main advantages of data libraries is that it avoids duplication of data. So there's only ever one copy of the file on disk no matter how many users import this file into their history to use it. It is also nice for the users because it doesn't count towards their quota so they can use large datasets without it eating up their quota. Libraries can either be publicly available so shared with all users or be specific only to certain users or groups. So permissions can be managed on the library or the dataset level by using roles and groups. Only admins can create libraries, but then they can delegate the permit permission management to ordinary users. So they can add data sets to this themselves. There are multiple ways of getting data into a data library. So the first one is from history. You can just upload data to your history and then import it into a data library. This is very simple, especially for if you have small datasets. Another way is from a directory on the server that belongs to the user. Admins can import from any directory. And if your data is available somewhere by URL, you can import directly from the URL as well. There are a couple of configuration settings that you will have to set in the galaxy.yaml file for this. There's the user library import variable which specifies directory of files where users can upload data from. So this directory should contain subdirectories which are named with the user's email address. So very similar to the way the FTP upload directory is structured. And you can also have these points at the same location so that users can upload data by FTP. And then if they have been granted permission to manage data library, they can import this data from their FTP directory into a data library. Then there is the allowPathPaste option which will allow admin users to import data from anywhere on the server that the user that is running the galaxies process has access to into a data library. So this is the main overview and now I will just give a demo of how this looks in action. So now I will quickly demo how this looks for the users. So the users, if they have their galaxy server and they go up to share data and data libraries, they can browse different data libraries that have been made available here. And click on one of those to see its contents and then select any files they want to import. They can select the ones they are interested in like this and export directly to their history. And then they have a choice of whether to import it as individual datasets or together as a collection. They can send it either to their current history that's default or create a new history. And then they just hit import. Now you see that these files have appeared in my history and now I can start using them. The other way that users can really access this nicely is directly from the tool form. So let's say we want to do some text reformatting. We have the file not yet in our history here. This button that says browse datasets. There we can, it shows us the files in our history, but also the top here data libraries. So again, we can browse around here and say and choose which file to use as input for this tool. And you see that this file is now selected as a tool input, even though it's not uploaded to my history first, and then we can run the tool. So that is very nice for the users. And next I will show you how this looks for you as an administrator. So now I am logged in as an administrator. And if I go to this shared data and data libraries menu now, you see that here I see a list of all the different data libraries that exist on this server. But now we also have here some extra buttons. We can edit the description and the name of this data library. We can manage permissions. So here you can see that we can define which users and roles have permissions to manage the permissions to add items to this library or to modify the library. So here we can select users which we allow to manage this data library for us. And if we go to one of the data libraries you see here there are options of adding more data sets to it. So at the top here you have this plus data sets button. So you can either select files from history, or since it's enabled here for admins, we can specify a path on the server which to import. And if you specify a directory, it will keep all the directory structure within that directory. And this is a very nice resource for sharing large data sets with your users or data sets are used by a lot of your users. So in the hands on portion of this tutorial, we will show you exactly how to configure this, the setup so stick around for that. So now that we've got things set up, we will show you how to add data to your libraries. We'll show three different ways. The first one is uploading data that is in your history. Second one is uploading data from the user directory we just configured. And third from the import directory which is only available to admin users. So if you switch to your galaxy, you can access the data libraries from the top menu share data libraries. So here would be our list of data libraries we don't have any yet so we'll create one now with this plus icon. Just give it a name, example library. You can give it a description to for your users and save that. So it's created a new library for us. We can click on it to add data to it. So if we want to organize things into folders first we can create those folders by the menu here. So let's say we create one with the name of history files. Save and this directory will upload some data sets from our history. So to add data sets you click the plus data sets button. And here you see the three options from your history from a user directory or the import directory. So let's start with showing the history. Click there you see that it'll show you all the different files in your current history and you can also switch to between different histories here. And here you can just select whichever tools you would like to import here. And when you're done you say add. And you see that these files are now in the data library. So in here at the top you can browse up and down different folders. So we go back up one level we see here this is the folder we just made. So let's make another one and repeat this process except uploading data sets from the user folder. So user, userJer, save, click on that. And let's add some data sets this time not from the history but from the user directory. So here you will see all the options. So if you remember when we looked at the library folder on our server we had here this example file. So here we can select the files we want and tell them add some metadata if we want about the reference genome for example, the file type just like the regular upload button and we hit import. This is all pretty easy and we can repeat the same thing for the admin import location. So let's say we want to add a data set here from the import directory. It looks the same as from the user directory except here you see it's only the files that are accessible to the admin folder. So the admin wild type here and import. And those are the three main ways to get data into your data library which users can now use. And in the next session we will show you how you can script this, how you can automate this, if the files are available on the web somewhere by URL you can also automate this process. Okay, so next we will show you how to automatically populate data libraries from data sets available on the web. So for this you create a little YAML file that describes data sets you want to import. So we have included a example YAML file here in example repo we cloned before. So let's look at the content of that one. So if you look in that file we see that it has some metadata we can give the library a name and a description like we did before manually. And then we list the items that we want to include so here we can say the URL of where the file is located online. We can give it the file format and then the other information we might want to associate with it. And we can create directories in the same way and populate those with the several files. So here you see we define three files into directories. So to be able to use this you will need your Galaxy API key, your admin API key. If you don't already know this you can read here how to get one. I will show you if you go to your Galaxy instance under user preferences. You see here a lot of user settings including one manage API key. And it'll give you the API key and if it's not set yet you can click this button to create a new key. But copy this value because you will need it to create the data library in the next part. Okay let's do it. So this is going to use the ephemera tool. If you've already done the tutorial before you can reactivate that virtual environment. If you haven't done that yet or the virtual environment is missing this will show you how you can create it again. So let's go ahead and activate that. And then next comes our command to install or to populate the data libraries. So this is an ephemera command set up data libraries. You provided the link to your Galaxy instance, your API key and the YAML file. And we gave it some extra, couple extra parameters or training, set some defaults that it's mostly only useful for GTN materials relating to how to make descriptions and things like that. The legacy flag allows newer galaxies to still use the older API. And in this case it's a little bit nicer to use the legacy API to replace libraries in place. So if we update the YAML file and run the playbook again it will not create a second data library with the same name but it will change the existing one in place. Which is kind of nice. So let's do this command and see what happens. So remember to get your API key and to get the location of your Galaxy server. So for me that looks like this, set up data libraries, dash G the link to my VM, dash K my key, and the link to the example library YAML file. Miss type dash K should be dash A for API key. There we go. You see it's making this library name a sequencing project and I'll import the URLs next. Okay. This command is also safe to rerun. So if we do this again. It'll just detect that this library already exists and it won't do anything. So if we would like to add say a single file to this library can update the YAML file, add that file, run the playbook again and it'll only add, it'll detect the new file and add that. So let's see in our Galaxy, if we can see this now. So let's go to share data, data libraries. And we see that in addition to the one we created manually earlier, we now have this most sequencing project which we defined in the YAML file. And it contains here the one file and the directory with the two other files in it that we described in our YAML file. So this is a really nice way to automate the population of shared data libraries. So this automated population of the data libraries is very useful. It is used by Galaxy Europe, for instance. They want to support all the tutorials on the GTN. So they want to make all the data needed for those tutorials available as shared data libraries. This means that users don't have to update the or upload this data set every time. This will avoid a lot of data duplication and also is very useful for users that maybe have a slow internet connection. So they created a shared data repository that defines the data, this YAML file for all the GTN data. So that this can be easily updated when the GTN also updates. And this has also been expanding to other usegalaxy.star servers. So you will see the same libraries across multiple servers. And if you would like to also, on your instance, support this GTN trainings and offer the data sets on your server, you can join this initiative. You just have to provide them with a non-admin API key and the data library to upload data into and then they can sync it automatically for you. So I hope this has shown that data libraries are a great way to share data with your users as an admin or to allow users to share data with each other and will avoid a lot of data duplication and make it easier on the users as well. Thanks for joining and don't forget to fill in the feedback form if you have any comments about this tutorial. Bye.