Is it? Okay. Good. What for? Should I wait for the crowd to disperse? All right. We're going to get started with the next talk. So I'll get that sticker handing out. Can you go on the hallway, please? Okay. Great. So I'm Matt Trinich and I'm going to be talking about the title is Better Testing Through Statistics, but it's really about the approach, the data approach we took to analyzing CI test results in the OpenStack project. So in the OpenStack project, we've got a pretty involved CI pipeline. So whenever any developer in the project wants to push a code, push a change into any OpenStack project, it automatically spins up a lot of work. And each one of these jobs is running in a VM in a public cloud provider. So when you push the patch, it spins up all of these test jobs, which has run on donated cloud resources, mostly it's all on clouds that actually run OpenStack. So it's kind of dog-fooding. And some of these jobs are pretty simple, style rule checkers, Python unit tests. But some of these jobs, DevStack Tempest and Multinode Grenade are actually running real clouds. So it spins up a VM, deploys an OpenStack cloud inside of that VM, and then we hit that cloud with a test suite which makes API requests against the cloud, which spins up second-level guests. And it does a lot of work. Glossing over a bit of the details on how the CI works for brevity's sake. But what this means in aggregate is when you push a change, you're running between five and 25 clouds. DevStack is the tool we use for DevTest deploying an OpenStack cloud. So depending on the project you're pushing the patch to, spinning up between five and 25 clouds. As part of that, we're running about 10,000 integration tests, about a thousand and a half tests per each cloud that we spin up. As part of that test run for each cloud, we're launching 151 guests. That number has actually gone up more recently. And we're generating a gigabyte of logs uncompressed for each run. That's actually tripled since I generated these numbers. I should update them. In aggregate, we run about 12,500 jobs a day. And our failure rates are actually really low. The failure rate of an individual tempest test, which is an integration test there, is about 0.01%. So it's not entirely a fair metric because some tests are more likely to fail than others, but if you look at them all equally, that's the failure rate. And an individual run likely has a chance of failing of 0.77%, which is still less than 1%. But when you're running 12,500 jobs a day, that's something that a developer is going to notice. Also, if you push a patch and it works fine and it fails as part of that 0.77%, you'll notice because you'll be blocked by the CI system. So this is just a graph I generated to show the scale. These are the number of tests we're running in a given day from the end of 2016. And you can see we run a lot of tests in peaks and valleys based on developer, awake time, and who's on vacation. You can see there's a nice gap in December right here when everyone takes the end of the year off. That's just to show the scale of how much throughput we're having to go through the system. So we have some tools to deal with this. The first one was we save all of the logs from all of those test runs. Every test we run generates a bunch of artifacts, whether that's logs from the OpenStack services that are running, the test results, or any other data like syslog, kernellog, servicelogs, like Apache logs, all of that is stored on disk. And we store about 10 terabytes worth of data at any given point of time of logs. That ends up being about four months' worth of logs from test runs. That's as far back as we can get with the amount of free resources we have donated from corporate sponsors. The problem is dealing with all of this ad hoc doesn't give you a good view.  If there's nondeterminism in the system, which there clearly is if we have a 0.77% failure rate, looking at one result at a time is not going to help you track that down because it's impossible to find something that fails so infrequently just by looking one at a time. Also, because of the nature of how the CI system works, finding performance regressions is almost impossible because we're running on heterogeneous environments on different public cloud providers. So we have noisy neighbor problems. We have differing hardware problems, a lot of different systems that we're running all the identical tests on. So figuring out when something performed poorly just by looking at one result doesn't actually tell you anything. And also an interesting question that I often was thinking about was figuring out how often something passes or fails like an individual test is almost impossible when you're just looking at one result at a time. So we had to come up with an approach to deal with all of this data. It starts by, instead of looking at the one test result you're dealing with, look at everything. Look at things at a larger scale. Start using statistics and data mining techniques to find trends in open stack, not the tests or the CI system, but in the thing we're actually testing. So we have all of this data. We can leverage other techniques to figure out what's going on in the thing we're testing, even if the tests aren't purpose-built for performance or some other trend we're analyzing. We have to make sure all of the data we're using is open and accessible to everyone and that there are APIs for accessing it all. If the data is just a blob of 10 terabytes of log files you can download all at once. That's not helpful, but if you have a way to query it that is useful. So we started introducing systems on top of the CI system to start analyzing the data. The first one was Graphite. This was introduced long before I even thought about any of this. And we just, all of the infrastructure services that run the CI system push data into a StatsD demon that gets rendered on Graphite, which includes also job results. So you can use this to see how often an individual job run fails. So like a test job, one of those boxes on the first slide. But it can't dive in deeper than that. It's also time-based because it's StatsD and it's a time series database. So you can't actually see the individual job it goes through. You can see in a certain bucket of time how many things happened. It's counter-based, but you can't see what that came from. And linking that back is important if you're trying to find what changed, what broke, what made something worse. We also have Grafana, which uses the same demon, but it provides a little bit easier to use interface for generating graphs. A lot of projects in the open stack use this to maintain dashboards for test failure rates. Same data set, same limitations. It's just prettier graphs and easier to use. We also have an elk stack, Elasticsearch log-cabana. So all of those large artifacts that we generate get pumped into the elk stack. We have a cluster running. I think it's trying to remember how many Elasticsearch nodes it is. I think it's at least a dozen. We hit some scaling limitations with it because we're running on publicly donated public cloud resources. We can't... A lot of the scaling things for Elasticsearch is just throw more memory at it. We actually can't because we're on a public cloud with limited resources. But it gives us 10 days of results and it gives us a search engine on top of job artifacts so we can start looking for trends in logs and start searching for certain patterns, which is very useful. We have a tool called StackViz, which allows you to go out to analyze test results. It lets you look at the test result, an individual test result at a high level, but also visualize what it's going on. So in this particular case, it's an integration test suite, the tempest that I mentioned earlier, and it runs with four parallel workers making API requests, and you can see which tests are running at the same time. It also highlights which one failed and shows why it failed and where it failed and how long it took and information about that. In a different way, it helps people understand what's going on in the system under test. And this comes to the big thing that we constructed relatively recently. It's been only about a year or so called OpenStack Health, which is basically a visualization dashboard for what is going on in OpenStack that we're testing. I'm sorry the picture is so small, but it's an interactive website and I didn't want to do a live demo to get it in cleanly. But this basically just renders exactly what's going on in the test system at any given point of time and lets you dive down really deep into individual test results and you can start abstracting trends from all of the data graphically through a web interface, which is very useful for a lot of people. So that's the website if people wanted to go play with it. It doesn't really mean a lot because we don't know what's being tested, but it's cool and lots of pretty graphs and lines. It's designed for the developer community to be a single access point for all of the data from the CI system, which we call the gate. And right now it leverages two things that I haven't talked about, subunit to SQL and elastic recheck to source all of the data for generating those visualizations. The architecture is really simple. It's a REST API server written in Python with Flask, and then it just queries the multiple backends and it's designed to be modular. So as we add more backends or other things that we want to integrate the data into for visualization purposes, we can just plug it in pretty easily. And then there's the JavaScript frontend which just runs client side written in Angular with D3 to generate all of the graphs. So about the data sources for OpenStack Health, because this is also some of the places where we've started doing different things than we haven't seen elsewhere. Subunit to SQL is a project I started. It's designed to store test result data in a SQL database. Pretty simple. It stores the overall run and then the test, whether it passed or failed and how long it took, or the time stamps around it, and then a bunch of metadata. And Subunit to SQL is just a library that provides a Python API and a database schema for interacting with that database to define how to store data and access it. There are also some CLI utilities to store data into the database and retrieve data from the database as Subunit V2, which is a protocol for test results that's used in some Python test runners. The OpenStack Infra project that runs all of the CI infrastructure actually runs a public SQL server that you can log into. The credentials are documented. It's a terrible, terrible idea to run SQL on an open port on the Internet and tell people, here's how you log in. But it's something we do because we wanted open data access for everyone. I've given this presentation a few times. I'm still waiting for the denial of service attack where someone just loops, show process list and crashes my SQL. But it hasn't happened yet. And because of our resource constraints on that SQL server, we only have six months' worth of results, which is more than enough for most purposes. The architecture which we get the results into is a bit convoluted and involves an asynchronous message passing. I'm not going to go into details. I thought I actually removed the slide. I'm sorry. The other tool I was going to mention is Elastic Recheck, which is the other source of data for OpenStack Health. And it's designed to answer one question. Have you seen this recently? So part of the problem we were having with the non-deterministic failures, with those small failure rates, is how do we know when they're happening more than once? We look at one test result and we see the failure pattern and that's it. And there are some people like myself and a few others who look at these failures pretty frequently and we'd have a mental model of common failure patterns and a bug that goes with it. But we don't really scale and people also have a hard time keeping track of more than a dozen or so specific failure cases. So we wrote Elastic Recheck to leverage the Elk stack that we run to define queries that fingerprint failure conditions that are tied to a bug. And we can look at when a failure occurs and report back to the user. It says, this test failed probably because of this bug. And we report that back to Garrett and IRC. And then we also have a dashboard view independent of OpenStack Health which shows failure trends across different all the different conditions. When we started Elastic Recheck we thought between the people who were working in this space, we thought, okay, maybe we've got 10 to 20 different failures conditions that we know about in the CI system. Turns out at any given point we track about 150 to 200 race conditions that are going on in the test system. This is just a snapshot of the graphs that we generate for the dashboard view. So here you can see a bug where Libvert Block Migration stalls during the test job. It's got a bug that's been confirmed and we hit 14 failures in the past 24 hours and 50 failures in the past 10 days and you can see the frequency of it. And it's been a very useful tool. It also lets us identify when a chain if we can catch a failure condition soon enough it also lets us identify when it started so we can try to track it down to what introduced the braking of the race condition. The other thing it lets us do is you can see up there in the other corner it says uncategorized. It lets us track failure conditions in the CI system. We have no idea about what's causing them which is very useful for identifying race conditions and sorting things out. So what all of this lets us do is make decisions based on data. For a while I was the maintainer of Tempest Test Suite and a common question we had was should we skip this test? Is this test buggy? We had no data on that. It also lets us identify tests that are actually useful. We have 1,500 tests in that test suite. How do we know all of them are actually useful? Are they just hitting an API surface that only queries the database so we're essentially testing the database query? We don't really need to be testing that and we can look at the test results to see how frequently is it failing and how it's actually catching bugs. And also when things do go bad it lets us figure out trends in failures to isolate it to specific configuration or cloud regions or other environmental things about the test run. It's been very useful just to have data to point to for making a decision because otherwise you're just guessing. You have no idea what you're making is the right decision or not. The other thing which I think is pretty cool is it lets us find trends among the noise. Performance regressions are the obvious example for this. So as I said before we run in a heterogeneous environment on multiple cloud providers but we're running real things. We can't do real performance testing in the CI system because of the nature of public clouds and different clouds. But if we collect all of the data we can start seeing trends. So this is an example from I think January where someone introduced a change to the volume system in OpenStack and it caused performance regression and you can see the average moved up but still really noisy data. The scale makes it look small but that's 50 seconds. One horizontal bar is 50 seconds so that's really big distribution but looking at it high level and just graphing an average lets us see that there was a performance regression and we caught it and were able to fix it in the period of a week because we had the system in place. We can also use the same techniques for finding race conditions and analyzing all sorts of things just by looking at the data at all. It's the same standard big data approach that everyone likes talking about and just applying the test results to a real system under test you can find real value. There are some issues we have with the system as we constructed it. Right now we've got too many varied data sources I mean that's half the presentation is just going through all the different things we have. The solution was supposed to be OpenStack Health but contribution has been pretty limited so we can't we haven't grown OpenStack Health to solve that varied data source limitation and even if we integrate all of the data you still have to keep in your head all of the limitations with the different data sources. OpenStack Health uses subunit to SQL so it only uses two different types of test data and if the infrastructure falls over we don't capture that because of that graph that I skipped over if the infrastructure falls apart we don't actually collect results for the database. An elastic search we're limited to document searching which in our case is one line which is a limitation if you're constructing queries you have to know you have to know that and trying to figure out a way how to make this simpler for people to use has been a big challenge which I think contributes to the fact that people don't contribute. But moving forward some of the things we want to do is integrate all of the thing all of those data sources into OpenStack Health so we have a more complete picture of what's going on that developers can interact with to know how to interact with the API directly if they're not interested in that. We also want to use the data to optimize how we run the tests all of the tests at all of the levels run in parallel and they're scheduling involved to figure out how we run things we have all of this data figuring out a way to pipe that back in so the scheduler can make smarter decisions about how to utilize our limited resources will be very valuable and also to start playing with some of the machine learning things and look at automating the failure detection a lot of cases when we're writing elastic recheck queries to fingerprint failure condition it's looking through a log and finding the stack trace and seeing if that corresponds to the API requests that cause the test to fail that's something a computer can do really easily we rely on intuition for certain pieces of it but there's a lot we could do with trying to automate that and be smarter about how we detect failures and that's something I think is really interesting and I wish I had more time to work on so here are some links to get some more information just to get repositories for some of the projects I mentioned and all of the people who work on this are active on the OpenStack developer mailing list and on OpenStack QA on the free node IRC and with that I think that's the last slide the next slide just says questions with question mark so if people have questions about time I normally give this talk with 40 or 50 minutes so I cut things down and I hope it was clear for everyone so are there any questions great you can also play around with the dashboard if people want if I have time I don't actually know if I have time you have like 5-ish minutes okay well it's good I cut the slides in half so if people want to see the dashboard I can try to bring it up on 1024 by 768 and see if that's actually usable see one okay let's make sure I'm on the wi-fi too which I'm not and all the time goes to this now yeah it's not getting the lease we can watch a bar move that's a good way to end it if it's just a bug probably below you well then it should be fair let me make sure it's trying to connect to the right one and not the one in the previous room I don't think a live demo is going to work today but I can show people once I get on the internet assuming I can and if there aren't any other questions that's all I had I'll sit here and keep trying until he says get out I won't tell you to get out yeah I'm just going to make a great YouTube video you can try but last month it's encrypted oh yeah it's not getting the lease you said try this one the wi-fi was supposed to be fixed on Linux we're going to have to roll to the next fix okay that's fine and finally connected but okay thanks everyone


For me this is massive, coming to see these kids as you mentioned this time of year, Christmas time where everyone's in holiday mode and together. Some families are unfortunately going to spend it here in the hospital and we can do our little bit by coming and saying g'day and times like this where you appreciate what's going on in your own life. I think for them it's absolutely priceless. You can just see the way some of the kids react. I think I'm a bit of a warrior and then you see what these kids do, like the strength of these kids and what they're going through which puts things in perspective and it's nice to be able to come and make them feel good for a couple of hours. Yeah you know Christmas full stop it's all about you know being family, having fun with your mates and you know unfortunately some of them are going to be in here for Christmas and to do the little things to like enjoy the day and hang out with them and you know let them have a bit of a laugh and have a chat and we love coming to do these kind of things to see the smile on the kids faces. You know you just put everything into perspective I think and yeah it's a great initiative by our club to come out here and visit the sick and you know it's very hard to come in and see the kids going through what they are but you know they're the true fighters and you know they're the ones that put smile on our face rather than it being the other way around. So yeah look this time of the year where it's busy you know it's all about staying safe and you know it's very hard especially on the parents, working parents having to take time off work and whatever so yeah look it's me being a parent myself it's not easy when these things are just randomly occur but yeah look God bless all these kids and hopefully they all get better soon.


My name is Jamie Lunky. I'm a senior fellow in the FA Hayek Program for Advanced Study and Philosophy, Politics and Economics. I'm here with my colleague Dick Wagner, the author of a book we're going to be discussing today. James M Buchanan and Liberal Political Economy, A Rational Reconstruction. Thank you for joining me here today. Well thank you for inviting me and looking forward to our conversation. So this is a dramatic oversimplification of the structure you just built. But you have all of these superstructures, constitutional politics, self-governance, post-constitutional politics, the moral and social philosophies that come out of this analysis. And at the root of all of them is this idea that public finance should be a way of understanding relations between free democratic peoples rather than simply being a science that exclusively speaks to relationships between rulers and their subjects or a despotic kind of public choice. At the time that Buchanan is doing this early work, how revolutionary is the idea that these public activities can be founded on a democratic basis rather than more of a despotic basis within the science of public finance? Well there were certainly precursors to the work that Buchanan did. In his doctoral dissertation Buchanan cited two precursors to there. One was Knut Wichsel, a Swede who throughout his Buchanan's career, Buchanan regarded as the single most important person for his constitutional work, who thought of the same kind of question of how could you have a parliamentary system suitable for Sweden where in the underlying Swedish parliamentary activities would reasonably strongly reflect a consensus among the Swedish population and how could you go about, Knut Wichsel asked questions, how could you go about imagining a mode of doing Swedish political business that would fit that image? And then the other source that Buchanan mentioned was an Italian fellow Antonio de Vittemarco, who Buchanan later spent a year in Italy studying a variety of Italian authors. De Vittemarco only had one book published in English and Buchanan had read that. It was published and translated in 1936 and it too offered two different ways of ideal types of democratic forms, what he called a cooperative democracy and what he called a monopolistic form. And the difference between the two forms fit with Buchanan's image of self-governing republics versus republics dominated by special interest groups and those were those corresponded to De Vittemarco's two forms. And so in many ways what became known as public choice in the very late 1960s, early 1960s was already in play in Italy in the 19, actually going back to 1888 and into the 1900s because you had a variety of Italian public finance theorists who likewise start to develop a explanation of how governments actually do their business as against someone saying how a government, they think a government should do its business. Like the long tradition of public finance was you might say concern with the practice of statecraft. What would be a good tax system? Adventist myth, you know, and his wealth of nations had these four maxims for a good tax system. At the time that Buchanan was a student, the two main figures in public finance were Brits, Francis Edgeworth and Alfred Pagoo. Now Edgeworth formulated a good tax system as one that would minimize the utility losses the taxes imposed on people, however you would ever determine that. There was an Italian named Emil Carri Pugliani who wrote a theory of public finance about 1903 maybe and his concern wasn't in trying to say, to advance his thoughts on what would be a lovely tax system, but was rather in saying, well the world has a rhyme and a reason to it. And Pugliani said about saying, well there's a rhyme and a reason to all that happens under the sun. Economizing actions are useful principles for understanding that. And he said about trying to explain what he thought would be the logic by which tax systems were established, were revised over time. Pugliani has never been translated into English, but has been translated into German, it was translated in 1960. And the sponsor of that translation was a man named Gunter Schmulders. He was doing work in what was known as fiscal psychology at the time and is kind of a forerunner of behavioral economics today. And Schmulder probably chose to translate Pugliani in not Davidi because Davidi had already been translated into German in 1934. And so he presented Pugliani, but he also wrote a foreword where he said, Italian public finance has long had a political science character to it where you have an integration of economics and politics. And with these ideas often giving a very good fit with reality. Suppose you took a bunch of these Italians, see the public choice got named at a meeting in 1968. It had a couple of prior meetings in Charlottesville when I was a student there. And it got established in 1968. I posed a hypothetical question.  And suppose you took these Italian dudes like Davidi and Pugliani and so forth. And they had fallen asleep. Then they woke up and here they were or in Chicago in 1968. And they were jumping into discussions that were going on about what was then called non-market decision making. I think those characters who had been right at home, the Italians and the Americans would be speaking the same language, sure they'd have somewhat different emphases and so forth, but they're all speaking the same language. And I think what, in my estimation, what Buchanan foremost did was really serve to connect what was a short-lived Italian orientation towards collective activity and brought it forward under the rubric now of public choice.  And today was no exception. So thank you so much. Well, thank you so very much. I always love talking to you too.


Welcome to digital asset news to get top stories and cryptocurrency digital assets and bring them out of bite-sized pieces Let me tell you what's going on the crypto market in 30 seconds or less first up Elon Musk once again tweets on the top Cryptocurrency Bitcoin and allows the entire market to take a bit of a dip Bitcoin going down 5% all it's going down even more So the question has to be asked is it time to drop Elon on top of that We'll take a look at a couple of interviews for Vitalik buterin of Ethereum fame and Charles Hoskinson's response as they have nothing but good things to say about each other so potentially there is a Recourse on the horizon and lastly we'll take a look at what is going on in Miami as the major Bitcoin conference is underway Which I think could actually lead to major price action So we'll take a look at those three top stories and everything else in the link below You


Hello, everyone. We've just approached the hour, so I'd like to show you what UX research we've been working on at the moment. So in my last update, you may recall that I just discussed that the UX team had started working on improving GitLibs navigation. To recap, we tested with six users, and these were a mixture of existing users and non-users. The non-users were typically using a competitor tool such as GitHub or Bitbucket, and we gave each user the same set of tasks to work through. And these tasks were designed to purposely move the user between the sidebar, their profile and contextual navigation, which appears when you're inside of a project or a group. And on this slide, you can see how the navigation looked during our first round of testing. As you can see, it's not too dissimilar from our existing navigation. The main difference is the pinnable sidebar still existed. This design highlighted a massive 30 usability issues. Now, all usability testing sessions are recorded, so if you head over to issue three within the UX research project, you can watch users as they try to get to grips with GitLibs navigation. Since that testing took place, we've gone through two further rounds of usability testing on the navigation. I'm going to spend today explaining in brief some of the changes we've made and how users responded to them. So this slide shows how the navigation looked during our second round of testing. This was a prototype created in InVision by Pedro. It included a link to the prototype in the slide, so if you'd like to explore it, then feel free to do so. The design highlighted 29 usability issues, so we've improved ever so slightly. So let's first talk about a couple of the things that the design did improve. So in the first round of usability testing, we found that it wasn't immediately obvious to users that there were two levels of project navigation. And we found this out by asking users to navigate to the issues board. And a lot of people didn't realize that you had to click issues first in order to trigger the secondary level of navigation and get the board view. So in this round of testing, we changed the navigation so that the items at a secondary level would appear on Harbor rather than the click of primary level items. And subsequently the success rate of this task improved. We also witnessed in the first round of usability testing that users struggled to identify their personal projects from their projects list. And by adding the tab of personal projects on this page, every user in the second round of testing could confidently state what their personal projects were. And finally, in the first round of testing, users didn't realize that the Tanooki is interactive. And this may not sound like a massive deal, but it is for two reasons. And the first is that users were forced to manually adjust the URL in their address bar. And that's a really messy way to navigate. And it's subject to users making mistakes. And secondly, the UX team is planning to create a more personalized default dashboard. And we know that something card users want because they're told us in various research studies and the Tanooki could be one of those ways in which users access this dashboard. So they do need to realize it's interactive. So in this round of testing, the position of the Tanooki has moved from the center of the screen to the left hand side of the screen next to the hamburger menu. And this private subtle change made users realize that they could navigate using the Tanooki. But even though the UX team has alleviated these couple of pain points, you might still be wondering why there isn't a greater reduction in the number of usability issues. Well, that's because when you add new features into a design, such as the new icons for adding a project and assigned issues and assigned merge requests, you can create new usability issues while you're attempting to resolve the existing usability issues. So for our further round of testing, we decided to test two concepts. And again, these are clickable prototypes that users could navigate around. And they were created in-framer by Chris and Dimitri. The main differences between these two prototypes is that the breadcrumbs were displayed slightly differently. So we have a horizontal and a vertical version, as you can see, I click back and forth. Also, the content within the top section of the screen, such as the search bar, to-dos, and so on, were arranged slightly differently. So part of the testing was to find out which set of breadcrumbs would help users to identify where they were within GitLab. And the other part of the testing was to see if it would improve the way that they navigate around GitLab. So both of the prototypes had the exact same changes to the global and sci-fi navigation. So there's been one very consistent pain point through each round of usability testing we've done so far. And that was that users had no concept of the different types of navigation that GitLab has. And we witnessed this in a couple of different ways. But again, due to the limitations of time, I'm going to very quickly talk about the two main ways. So at the beginning of the previous testing, we had a task which took users directly into a project. And it exposed them to the contextual navigation for the first time. And then later on in the testing, we had a couple of more tasks which forced users to revisit the contextual navigation. However, the success rate of the later tasks was really poor. So users hadn't remembered that the project navigation existed. They never mentioned it or actually looked for it again. So this tells us that not only do they not remember it, but they have no concept of it whatsoever. And there are also two extremes of user behavior when interacting with the sidebar. So some users didn't instinctively interact with the sidebar because it isn't open by default. So for example, it took one user around nine minutes to find the sidebar. And those that did find the sidebar easily typically proceeded to treat it as their primary form of navigation. And subsequently this meant that they often failed tasks where the content couldn't be found within the sidebar. So these new prototypes in the third round of testing tried to address those issues. The contextual navigation has now moved to the sidebar. And we now have a global dropdown menu which contains some shortcuts such as groups of projects you've recently viewed. As we had two prototypes, we've doubled the amount of users we tested with. I spoke to 12 users in total and that comprised of eight GitLab users, four non-users, and they were split evenly, so six users to each prototype. So the results, 14 usability issues were discovered, which considering the amount of changes we've implemented since the previous prototype, that's actually pretty good. We've halved the number of problems. Generally the majority of users seem to understand that the sidebar was contextual after six or seven tasks, which is about halfway through the testing. And this is an improvement from the previous research which showed that users could complete a whole usability testing session without understanding that the sidebar was contextual. The breadcrumbs were inconclusive. Half of the users we tested with utilized the breadcrumbs to understand where they were located within GitLab. However, it was uneven split across both prototypes, so three out of six on each prototype used them, so there was really no clear winner there. And we had a lot of positive feedback about the new global dropdown menu and users' behavior reflected that as well. So what's next? Well, we'd like to get the number of usability issues even lower, and you can read what those issues are in issue seven of the UX research project. We've realized from the testing that there will be users who are apprehensive about moving to a contextual sidebar. So one user discussed how by referencing stuff in the center of the screen, he didn't always spot the contents of the sidebar changing as his attention was focused elsewhere. So perhaps there's still some improvements that can be made visually to help users shift their gauge to the new sidebar. And we're also asking some seasoned users of GitLab. I mean, they do tend to sometimes have a better understanding of the existing contextual navigation. And then we're kind of asking them to change a behavioral pattern that they might have passed for a couple of years, and that takes some getting used to. So we want to make sure that what we're proposing to implement works for both existing and new users to GitLab. And we're looking into ways in which we can effectively transition users to the new navigation and keep their frustrations to a minimum. So that concludes my update for today. Does anyone have any questions? Going once, going twice. Now what? Super.  Thanks. Bye-bye.


Hey guys, Ken Ross here, and I'm a business consultant that specializes in reducing costs for businesses By looking at their essential expenses and today boy. I have a doozy today I watched the Apple keynote presentation for the iPhone the Apple watch and the iPods pro AirPods Pro I hope you had a chance to check it out And if you don't go ahead and check the description down below where they talk about the brand new products from Apple I love covering events like this because it's always Really great to see something new and exciting in the world of technology And especially when it comes to telecommunications technology Apple has always done a great job with its iPhones and this time it's no exception They have quite a array of new features and they've they've Stirred up a lot of excitement for people one of those things I really want to cover because I feel like in this area of cost reduction Something I specialize in I don't want you to fall into the pitfalls of Understanding the carrier relationships when it comes to these releases So what I plan to do is to help you understand Right What is really going on when apple says you can get carrier discounts? And the iphone is essentially free or the iphone is heavily discounted. What does that really mean for you? How is the Providers of service actually setting those things up So i'm going to actually spend a little bit of time breaking that down because if you're not reading the fine print If you're not looking at these things, then you really won't understand What's going on here? So let's get into that If you're paying attention, right? And if you were really excited about the iphone 14 like I was and the iphone 14 pro and the plus and all these things Especially the new watch and the the new airpods too. I want those You did not necessarily pay as much attention to to What the pricing details were for the new iphone? You knew kind of around Hey, it's going to be the same prices as the old phones But they heavily emphasized in their key, you know Carrier discounts, right? What does that really mean? What what is going on when it comes to these carrier discounts? Let's talk a little bit about that. The first thing is I want to talk about AT&T, right? If you go to their website today and you go to Apple.com and you check out the iphone 14 released information and it talks about Hey, this is the discounts that the carrier is going to give you you have to read the fine print You have to understand what's going on. So let's give it an overview So AT&T you have to make up to the first three payments of about $28 On their monthly installment plan as if you were paying for the phone Before they start affording you the discounts and the way they're going to issue those discounts to you is via a credit on your bill Let me make sure that's really clear for you. It's going to be a credit on your bill You're going to see a charge for the installment plan for your phone And then you're going to see a credit That's how it's supposed to work and that's over 36 months I hope you see a trend here as we go along The next thing is sprint sprint is actually doing a little bit different type of deal They have anywhere from $340 rebates all the way up to $900 rebates And they segment out what parts of it are part of the The credit that you get and another part that is prepaid On a prepaid card that you get that that you can use online or in specialized apps And you only have six months to use it So the time clock is ticking when you get this rebate card You have to use the value asserted value that you got from Sprint before the time runs out or you lose the money that you were you were afforded as the discount for the phone Next up is T-Mobile T-Mobile Offers anywhere depending on the phone and the plan that you choose anywhere from $200 to a thousand dollars And it's heavily based on plan right and the phone that you trade in So when you actually go to T-Mote or when you go to the app of website You have to select which phone you're going to trade in and then Which carrier discount you want to apply to that phone? And it does all the calculations for you and it gives you a rate based on not only the phone that you choose The condition of that phone the plan that you're in all of these things are going to factor into your rebate And in the end the carrier is going to afford you that discount T-Mobile also has The same type of system that AT&T has they are going to discount your phone Based on your bill. So you're going to use for the next two years. You're going to pay on an installment plan On this iphone and they are going to give you credits. So you'll see a Charge and you'll see a credit on your bill Last up my favorite Verizon wireless. I always pick on Verizon on my channel. I love Verizon for the craftiness that they do sometimes Um, I love them so much to death that I I'm still a customer of Verizon But I love to pick on them. So what are they offering right? What is their deal? Anywhere from 400 to 800 for an eligible trading phone, right? And that is also based on plans. So if you have one of the unlimited plans The tiered unlimited plans that they're on You're going to get a lower discount than if you're on the plus plan with all the bells and whistles That's where you're afforded the most money That's also in form of a 36 month credit. So you have to use The service for 36 months in order to get the rebates now Want to make sure I make this clear to you because I I don't think most people are getting this, right? What this is essentially doing to you is it's locking you into a carrier Not necessarily a rate plan per se, but a carrier for a certain period of time And why is that important to carriers because they are actually out to keep and retain their customers, right? The iphone is such an a compelling product. They're willing to give you a discount To get the phone To keep you as a customer reason why this is so important to them is that they are in the in the 5g war, right? We're trying to acquire as many customers as we can In our territory and expand our reach of 5g and have a huge 5g customer footprint I would encourage you to heavily consider what that means, right? If you are wanting to invest in a cell phone And a cell phone plan and all the things that come with it Asking the carrier to to pony up some dough up front to pay for your phone Is is basically akin to Them giving you a loan so that they can continue to use you in their master plan And that's exactly what they're doing here. So don't be confused Don't misunderstand the message here. Apple is doing something that's great For consumers. They've produced a great product. They're also doing what's great for business And there is no reason for these carriers to want to offer these discounts if there was no other incentive for them Please know that I'm always here as a resource to help out with that I'm going to tell you exactly what I would do So when it comes to the release of any new thing, especially when apple comes out with new products They're very compelling people rush to the stores to go get them. They want to grab them up as soon as they can A lot of folks have Great deals with their carriers or even with apple themselves to upgrade their phones every other year or do those types of things I highly suggest That you remain in control of your technology that you don't allow Companies and other organizations to utilize this tactic to keep you around because honestly I really do feel like the convergence in the telecommunication space Is very much upon us and these types of deals right now where they're locking you into up to three years With a carrier to afford a discount, right? The only way you reap the benefit of this discount for these iPhones is to actually keep an iPhone On their plan on on Verizon or Sprinter team up whatever plan you're on For the time period that they give you if you don't what happens is you have to actually pay up the balance So are you going to keep your phone for three years? And in the end what that means is that when you give up your phone Verizon owns your phone. You don't own it yourself So you have to pay them To keep the phone in order to trade it in in order to get into something else And this vicious cycle of remaining under the the care of a carrier I like the pun pun intended the care of a carrier Remains and they remain In control of What it is they can do to you as a customer their rates could change they could add things They could subtract things and you have to kind of go for the ride So that is my word of caution to you. That's my my thought about this and what I do specifically is is I remain Independent I buy my phones outright I pay for apple care if I need apple care. I protect my own phones I don't buy carrier protection plans. There are many options out there to Ensure your phone to replace your phone You should look at all of them You should actually analyze and look at your bills and understand your statements because I know There's a lot of people out there and I've met a few of them In my travels Around the state here and talk to people who are paying Extra fees and extra things because they're locked to this carrier and they don't see Everything that's in their bill is something they can understand They're out to confuse you they're out to keep the information away from you and you need to remain independent You need to be able to move and say no I can no longer accept what you're doing here So I'm moving on help yourself when it comes to this do not over invest in your carrier Because of a new phone And please tell me in the comments section below what you think about Apple and the new products what what you think about the information I brought here I'm going to put links to different resources a link to the actual Keynote presentation that apple released that talks about the different products. Well, please Please Understand and really Analyze what it is you're out to do here because Ultimately, you could be you could be trapped. You could be in a situation that you did not understand before That's what I have today guys. Thank you for your time I really do appreciate you taking this journey with me on the iphone 14 and what it is the carriers are up to Please subscribe to my channel. Please let other people know i'm out here Given the best information I can based on the what I see going on in this in this crazy telecommunications space And stay tuned to this channel for more updates. I have a lot more content I'm I'm going to be delivering to this channel in the weeks to come I have a lot of things in the works Would love it if you you'd tell your friends About this if you'd like this video if you'd share it and visit my website I'm ken ross calm and until I see you next time I'll see you around


I would be really curious to see what happens when you guys get together and collide with me and we all partner together on making you guys the best in the country.  Do titles mean anything? No. No, they don't mean it. No, titles, titles are for losers, okay? You earned a title, good. Listen to me. Leaders make leaders. You know how I know what a great leader looks like? A great leader, it makes more leaders. You don't make more followers. Marcus, great leaders don't make followers. You don't need followers, bro. You need more leaders. So what we need to do is we need to set the standard, set the tone and set the pace for what it looks like to be great. Now we're in the era of the worst salesman in the history of time. You guys just got off two great years. What does that mean?  You got to say, hey Bob, if you don't buy it today, Sarah's going to come get it tomorrow. So I love you, but it's just that if it's out here, you can't buy it.  But now that inventory is coming back in, things are changing up. Dealerships are starting to not all hold their prices high. They're starting to come down. You see some of these scared dealers starting to let their profits go now? Now you can't say, hey, Bob, go anywhere you want, shop wherever you want. Nobody's giving great deals. You're going on a 747 plane on most car dealerships lots because they ain't got none. Plane demand, brother. Love you, man. Love you. If you guys believe there's another level, there's another gear, there's another area you can go to. There's the great separator. It is all in studying. Do you see this mind? You see this mind? What's the difference between a horse and a human, a dog, a pig, a mind? It's a mind. My mind is unbreakable and unstoppable.  You guys have the best company in the world. Hey guys, what's going on? It's Andy. A lot of you leave comments telling me that you need help. Do me a favor. I'm going to tell you the best way to get a hold of me. Shoot me a text message right now,<PHONE_NUMBER>,<PHONE_NUMBER>, I'll help you with whatever you need.  Let's get back to the video. You know where the first place average people go when they have a problem? Excuses and blame it on the world. You know the first place successful people go? They go look in the F and mirror.  I am obsessed. I am obsessed. I am psycho. I've lost my mind to becoming the best sales professional in the world. Notice I didn't say amateur, professional. See, I understand what the customers want and I understand what the sales pros you need to do. I know both sides. Here's what I want. I want to teach you guys to be public speakers. I want to teach you guys how to overcome any deal, any time, any place, anywhere on the phone, in person. It doesn't matter. What I really want, I want to make some legends. I want you to find your leaks. Think about it right now. Right now. What is my leak? What is my hole? What is my...  How would they roll in the Trojan horse? What's my Achilles heel? Is it your confidence? Is it your belief? Is it you don't have very good word play?  When I say, hey man, hey, what's your best price? Are you a sniper and you can bury it or are you like a machine gun sprayer and you just got to keep saying a whole bunch of words and hope that something sticks? Would you like to know what to say? So at the time when you're backs against a wall and a guy says no, you say, boom!  This is what I'm going to teach you. Now listen to me. If you don't think that you need training, I won't explain this to you. Everybody in this world needs a coach. I have a coach in the gym. I'm 42 years old. I have 6% body fat. I'm a lean machine, best shape of our life. Age don't mean... Okay? You guys want to level up in every area of your life? Kids? Family? Dealership? Cells? All of it? An individual can be beat, but a team can't be beat, can it? When the tide rises, all ships rise, here's what I'm going to ask every single one of you. Raise your hand if you believe that you can freaking take yourself to a whole another level and you think that that bank account you wrote down, you think it should be way bigger by now.  And by the way, don't look around and look at anybody else.  So my question is, if you want to be a professional and you want to be the best in the automotive space, you will be here. I would be really curious to see what happens when you guys get together and collide with me and we all partner together on making you guys the best in the country. Hey, guys. I just want to tell you the true one percenters, you made it to the end of the video. Do me a favor. Share it with the friend that wants to go to another level. Make sure you like the video, comment below so I know who you are. Set your notifications and then subscribe to the channel. We got daily sales training videos dropping. I'll see you soon.