Hey folks, today we're going to talk about something that on the surface seems almost like a small deal, but the more I get into it, it's going to be an even bigger deal and it's a big deal for the Nintendo Switch. So today we learned that Silicon Studio is bringing middleware and engines to the Nintendo Switch. As I said, it seems like a minor piece of news. There's obviously going to be various engines and middleware brought to Switch over the lifetime of the system. Same was true with Wii U, although there was far less of these middleware and engines brought over and the reason that this is a big deal is because middleware and engines is how games are essentially made. They're kind of the backbone of video games. So the more of these that are on the system, the more games that could be on the system because a lot of different studios use a lot of different engines. No, not everyone's using Unity, not everyone's using Unreal Engine 4, etc. There's a lot of different ones out there. And Silicon Studio is bringing the post-script middleware Yibis, Y-E-B-I-S. They're also bringing an all-in-one game engine called O-R-O-C-H-I, or Orochi, which has over 40 dev tools for it. They're also bringing their next-gen engine, which is Xenco, which is X-E-N-K-O, which will be out in April, so later this month. And all are going to be made compatible with Nintendo Switch, and the company has already offered middleware and graphic technology for the Switch SDK. And all of this was brought to light by Nintendo everything. Now here's where this becomes interesting. There are some games out there that have been made specifically with Yibis in terms of Yibis 3, and it is now confirmed that it is Yibis 3 support that the Switch is getting. So it's not just like an older version of Yibis. This is the current version of Yibis as confirmed by the studio who created it. So, here are games that use that, Figureheads, MotoGP15, Ride, and Dragon Ball Xenoverse. Now we know that Dragon Ball Xenoverse is coming to Switch already, however, one game I haven't mentioned yet also uses this engine and really opens the door for possibilities, and it's from a studio that has been confirmed to be doing something on Nintendo Switch, but we don't know what. And the game that supports it is Dark Souls 3. That's right. The engine that Dark Souls 3 uses is now fully supported in its latest version on Nintendo Switch. That's mind-blown. People have been wanting portable Dark Souls for a long time, and now we could possibly get Dark Souls 3, and it could possibly be the trilogy, because we since Nintendo fans have never had a Dark Souls game, you could see a release that has Dark Souls 1, 2, and 3 all packaged together for $60. Nice and clean on Nintendo Switch. That would be insane. And see, this is why this news becomes a big deal, because Nintendo Switch is selling extremely well. It is using architecture that developers are familiar with. Since it's an Nvidia, GPU, CPU combo, it is technology that developers have worked with before, whether it's through the shield, or whether it's just in general with PC gaming. Because what this GPU does, what the CPU does, makes porting to it extremely easy from PC. So when you have a game that you've already developed on a PC, it's not that hard to bring it to the Nintendo Switch, even if you have to cut back on some features. You need to cut back on draw distance, you need to cut back on the level of detail. Maybe you can't have as high of anti-aliasing as you want. But that's all irrelevant. The Switch can run it. It's just going to be, is it running it to a visual standard that you find to be acceptable as a developer. Now, obviously this doesn't mean the Switch is all powerful and is going to get every third party game under the sun. But if this Yebis 3 support leads to Dark Souls 3 coming to the system, that just opens the floodgates. Because the Nintendo Switch is doing very, very well for itself, but it's doing very, very well for itself in general thanks to Breath of the Wild and that potentially being one of the greatest games ever made. And the hype for that is still driving sales to this day. But it needs more. Right? We also got Mario Kart 8 Deluxe with the two and two arms and all that stuff. But it needs a third party games because that's what was missing from the Wii U. I mean you could argue Zelda was missing too if you don't call it remakes. That's besides the point. It needs big third party games and right now I think the biggest third party game it has is Skyrim which is a five plus year old port or you know maybe it's only a year old port if it's the remastered version but either way Skyrim is a pretty old game now and Minecraft which again it's been out forever not nearly a system mover these days. But beyond that what is it NBA 2K18? Is that the next biggest title? I mean there's Sonic Forces I suppose as well. There's not a lot of really big third party titles that have committed to the System Man. Heck we don't even have Call of Duty committed yet for this year. So this could be huge. You could see a Dark Souls 3 coming on over. You could see Bethesda deciding to bring you know some bigger games over. This is huge. I know a lot of third party developers are taking a wait and see approach with Nintendo Switch but in order to wait and see there's got to be something on the system to wait and see with. You can't just be like yeah we're going to wait and see oh Zelda's selling really well oh Mario Kart 8's selling really well oh arms is selling really well and the Nintendo Switch is selling really well but there's no third party games on it to gauge if your game is going to sell. So someone's got to take the leap and you're not going to get a whole lot learned from say NBA 2K18. That's not going to teach you a whole lot except maybe EA could convince EA to bring Madden to the platform. There's already going to get FIFA. Hopefully the full version but I'm just kind of throwing it out there that when you have an engine that supports a major well-known game even though there's another game using it that's going to be on the platform. I feel like this is a perfect opportunity a no excuse opportunity to see Dark Souls 3. It may be the entire Dark Souls series on Switch. The thing is we know the studio behind Dark Souls 3 has already gotten the game to run on Switch because they announced that they got it to run on Switch in some capacity already. So having the full engine supported is only going to allow them an easier time bringing the game to Switch if they can you know if they want to continue pursuing that. And again I think if they bring Dark Souls 3 they have to bring Dark Souls 1 and 2 with so this is a bit of a big project for them to bring all three games to a brand new platform. But it can be done. We saw it with Bayonetta and Bayonetta 2 obviously Bayonetta 2 was made for Wii U exclusively but you know it wasn't a hard time for them to bring Bayonetta from the 360 and PlayStation 3 era over. So yeah this is exciting. It doesn't mean anything quite yet in terms of seeing Dark Souls games but it's a step in the right direction. Nintendo Switch is getting things right and they are now starting to get support from graphic engines that Nintendo would have never gotten support for before. So this is awesome. I'm really excited. Maybe I'm overreacting I don't know but this is exciting news to me. And as I said it's going to fly under the radar even though we're talking about it there's not going to be a lot of places talking about it. Man. This is Nathaniel Ruffeljantz from Nintendo Prime signing out.


standard standing committee. I'm going to start off with I guess a bit of a confession. I think standards are a good thing. I guess I wouldn't be standing up here if I didn't. Hopefully you'll recognise that fairly standard piece of user interface equipment there. And that's a wonderful thing isn't it? Because of that standard it means that pretty much any competent driver can get into any modern car. You can drive it around reasonably. Okay you might not be able to work the sat-nav too well or Bluetooth sync your phone but you'll be able to drive it, you'll be able to get it to where you need it to go and you didn't need to be trained about how to drive that particular car because of the standard of that control. Here's another example.  The previous speaker in fact talked a lot about this particular application. Standards give you that interoperability RJ45 lead. Let's be honest you don't even really think about it nowadays do you? You plug it into your laptop, your server, whatever it is and it just works. And it works because of all of those physical standards, the electrical standards, all of the other standards around how those things operate. It just works. So standards give us the interoperability.  A great example car production line there. It's because of the repeatability of the standard processes. It allows us to really optimize the tooling, the process flow, the logistics, everything about that process. So we get the increased quality, we get the increased speed and we get the reduced cost, all of those good things that we want. And we're all playing at this. We're all playing at this. I've got on this chart just a selection of some of the major standards bodies including of course the open group and Fujitsu ourselves up there along with some of the other IT service providers and some of the other major tech suppliers. And my point from this chart is not ready to look at well who's a member of what but just if you look at that you can see there's a fairly broad spread of little ticks across the chart. So there's a pretty consistent pattern here of the major tech suppliers and IT service providers involved in these standards bodies. So there's clearly something here, isn't it? And actually good to see on this chart Steve there's probably maybe more ticks on the open group line than a lot of the other organizations. So hey, must be doing something good. So what's the value? I've talked about some of these things already. We've talked about the accelerator pedal or I guess I should say here the gas pedal and the brake and what that gives for us is workforce flexibility because it means that we define standards with their best practice methods inputs and outputs and that means that we can we can fairly freely distribute tasks pieces of work around a workforce because everybody's been trained and is familiar with that same approach. So great way of getting value out of the standards. Interoperability, again previous speakers spoke a lot of this. The standard interfaces means we can flexibly distribute processing and information around what services or hosts we're using and it also has an effect to commoditize the services which again is good for for cost and quality and efficiency. And then finally process optimization because again standards give us those best practice repeatable processes. It means we can optimize optimize time quality cost of the flow of components and the tooling that we use in our production processes to really get the best end result. Some examples of these things just taken from the open group portfolio workforce flexibility. You may be used toga in your enterprise architecture practice so again you can flexibly assign architects to architecture assignments because everybody's familiar with that way of working. Interoperability example of taken here again previous speaker spoke about this but OPAS is aimed at doing this sort of thing in the process industry defining those standard interfaces and in process optimization IT for IT gives us a standard model to follow which means again we can really drive the efficiencies in that process the best practice approach and get the best services at best speed best quality lowest cost. However we can do more than this. So I said I would give you three ways to get value from standards and standards bodies and that last part's important because it's not just about the standards it's the standards bodies as well. So the standards bodies of course give us access to the latest industry standards the techniques and the know-how etc but the standards bodies also give us publication and presentation platforms that we might not or we might not otherwise have access to and it gives us networking opportunities as well and these things shouldn't be ignored these things give us real value. So how do we get value from these things well fairly obviously in the first example of the access to the latest standards we can use our intellectual property in our internal processors our internal tools and methods and that's great and there's also actually a side effect here one of the other things I do within Fujitsu is we sort of research market trends and the like to see what the latest directions are and although the standards making process can take some time again we heard the previous speaker talk about standards coming out in a couple of years time a really interesting thing is to see is to see well the standards bodies are just starting to work on because that gives a real good signal about industry maturity and industry acceptance the point at which some new trends some new technology the point at which you start to see the standards bodies beginning to think about making a standard in some new some new field that's a real good signal about industry maturity and industry acceptance. So secondly publication and presentation platforms gives you as a business a certain way to demonstrate capability to the marketplace possibly influence the marketplace through applying your intellectual property into standards and then out into the marketplace. And finally the networking opportunities gives us a certain visibility and a knowledge of customers and partners that we might not otherwise have opportunities to get. So again just to clarify this thinking about some examples. So in the first case access to the latest industry standards again using togaf in your internal enterprise architecture frameworks and using certification perhaps to skill up your people to learn these tools and techniques. The publication and presentation platforms well hey here's one example presentations to industry conferences and putting forward your own intellectual property to become part of industry best practice. When it comes to networking you can join the working groups informal groups at events like this and that allows you to form these relationships and have those interactions with customers and partners that you might not otherwise get. These things have real value it's is no coincidence I guess that I think at every open group event I've ever been to then somewhere on the agenda there's going to be a network in evening so these things are there for a purpose. So some practical steps to take when it's about access to the internal standards and access to the latest industry standards then clearly you need to get your internal process owners engaged those internal people who make those internal tools internal methods and standards get them engaged with the standards bodies using the intellectual property and you've got to promote them as well to your internal technical communities it's not just through the internal standards that these things have value but also all of those cases where you've got somebody out there working on some project and maybe they're thinking gee you know what's a what's a good way to do to solve this particular problem or oh wow I could really do with a template to represent some model or something like that so you have to make sure that all of you people know of the material that's available through the standards bodies so that you can take advantage of these things. Publication presentation platforms well again pretty obviously proposing your presentations to organizations and events like this or proposing intellectual property into the standards making process and then when it comes to networking opportunities that's all about meeting the stakeholders and making sure that the engagement of your company is clearly visible so those are some ways we get value but I've worked for a number of companies through my career some some very large some very small the smallest one my employee number was just five so that tells you something about how big the company was and one thing they've all had in common is that if you wanted to get some people get some resources allocated to do something then you've got to put together some business case go through some sort of investment appraisal and you know somewhere along the line in the process of doing that there's going to be an excel spreadsheet or something like that with some some cost and some benefits in it and you're going to have to prove some return on investment so that they can get your uh you can get your resources allocated actually do some piece of work so how to go about that well here's some practical steps that I hope will help you when you're faced with that sort of problem yourself so first of all let's think about you know what are some practical things that can be counted because when it comes down to that excel spreadsheet um yes you can talk about great things like flexibility of workforce and so on but how do you actually put that into the cells on the spreadsheet so what can you practically do so using intellectual property you can you you can think of things like okay I can count how many internal methods guidelines templates etc published I can think about I can try and count how many projects have used the internal standards and I can think about and measure how many people have been trained in these approaches when it comes to demonstrating business capability you can you can count for example hey how many presentations did you people stand up and give or how many IP submissions did you make into the standards making process and then for networking how many stakeholders did you go out there and meet how many opportunities how many suspects might that have generated and then next how can we move on from there to actually calculating some sort of value well the one I'm going to focus on is the one which in my experience actually produces the most countable the most um evaluatable benefits how many projects have used some internal standard or method and you can get to that with some fairly sort of simple calculation which I'll run through you can look at something like downloads times a use rate times a saving for applications so I'll explain what I mean by that I'm going to assume that within your company that the place that you put your your internal standards some sort of web-based repository perhaps SharePoint something like that I'm going to assume that you can get some counts out of that that tell you how many people have hit the page how many people have downloaded that use rate that's how you then take a laugh at the fact that just because somebody's downloaded it doesn't necessarily mean they've used it in some for some practical purpose so you need to look at some proportion of so how many times from the download did that actually get put into some sort of productive use and then finally savings per use okay great it's been downloaded it's been used what what did that do for us as a business and the simplest way to think about that is usually some sort of time save type of thing that because you use the standard approach the standard guideline you downloaded this template whatever it is that saved you x amount of time you can put some staff time value around that and come up with some saved value and you can determine these things the last two things by surveys for example just sample surveys just simply asking people okay I can see you downloaded this did you use it yes no so you can get some proportions out of that and again you can survey people to say great I can see you use that template or that guidance whatever it was how much time did that save you a couple of days a couple of hours a couple of weeks maybe whatever it is but you can get some credible values for these things so this is an example I've taken straight actually out of a piece of work for Fujitsu so these are some absolutely genuine numbers I've changed them slightly to protect the innocent but basically this is lifted straight out of some internal work so top left there there's some stats from one of the internal internal standards publication sites that gives us a way to judge a number of number of users number of downloads so that gives us a number for that we surveyed the user base got an approximation of the use rate 50% in this model savings per use seven and a half hours basically a working day actually I think that was a fairly conservative number but when you're trying to roll this up into a business case justification it's all better if it's a fairly conservative set of assumptions then put some staff rates around that because Fujitsu is a Japanese company you can see there's some figures there in Japanese yen but translate that into British pounds because that's where that's where I'm based and that's where my budget comes from and so then that gives you a benefit side of the equation on the cost side well this is just an example of a cost model I'm sure you can build up similar things but there's just some staff time in there for the people developing standards some time to actually generate the collateral to consult with some subject matter experts and then to go out to do all the promotion and internal training type of things so then that gives you a cost for what does it take to actually develop this standard and get it pushed out into the field for use and the great news at the end of all of this there's a cost there's a benefit whoopee we've got a positive return on investment so good business case so my point here is that with some fairly simple modeling and some fairly straightforward and defensible assumptions you can show a very good return on investment and it's not just me saying this if you look out in the industry there are from some organizations some interesting bits of industry research that show some figures for how standards improve business performance so there's an ISO study the full reference is given on the slides that shows profit contribution from standards ranging from 0.15 percent to 5 percent of profit contribution and a study from the BSI British Standards Institute that shows an increase in turnover from using standards of between 1.7 percent and 5.3 percent and I can tell you from the model that I've presented on the previous slide if you then roll those overall costs as a proportion of the overall enterprise and how many projects and do a little bit more modeling actually you come out with some numbers in a similar order of magnitude so that gives good confidence that you know this is for real these are real benefits real quantifiable benefits we get from using standards so I've talked all the way through about three ways that a business gets value out of standards now I couldn't resist standing up here talking about stand talking about standards and business value without mentioning a fourth way now have we got any real network geeks in the room i.e. TFRFC 2549 anybody know that one okay here's another clue ip over avian carriers with quality of service and the date might give you even another clue first of april 1999 this is utterly fantastic piece of standards work this is a standard this is absolutely for real you can go look it up it's actually a standard for how you uh how you transmit internet traffic over carrier pigeons beautiful little picture taken straight out of the standards document the carrier pigeon with a little payload memory card or something like that tinder tied around its foot and even better than that bless them the bergen linux user group has actually implemented this there's a little bit of console output there it shows how they pinged how many was it nine packets of data that's nine carrier pigeons and unfortunately there's a packet loss right there of 55 percent don't worry nothing bad happened to the pigeons it's just they didn't like make it to the far end in the time allowed round trip time of about of about half an hour there but hey this actually works so here's another way to get value actually it's pretty rare but standards can have a sense of humor god bless them at the itf so in summary standards are a good thing they give us flexibility of workforce they give us interoperability they give us process optimization we can get to the value by using them obviously in our products and our services and in our skills development certification and things like that we can also through the standards bodies demonstrate capability the market and through contributing ip possibly influence that marketplace and networking of events like this again that has real value shouldn't be ignored that value and return and investment can be estimated to show a positive business case and finally yeah standards can be fun thank you very much we're always fun at the open group standards bodies can be fun too absolutely no thank you that's that's great it's great to see somebody put some numbers on on this and and thank you for sharing what you could of the the internal work that goes on because it's a problem we obviously believe there's ROI on this stuff but it's nice to see some of the numbers don't lie right there we are so um first question that came in um it says do you find value in working with and networking with your competitors um i guess another way of putting it how would you describe the value of that so do we find value yes of course yes of course because one of the important thing about standards it's not just that it's a standard approach but through all of that collective knowledge that we contribute and the governance processes that organizations like the open group and others put around that standard to making process it means that what comes out of it is not just a standard but it's a best practice as well and we wouldn't get to that unless everybody was involved you know you're competitive as your partner as your customers so yeah absolutely absolutely and and we hear from from customers particularly large customer organizations that one of the benefits of coming to a place like this not just here but a place like an open group event is that the kind of neutral non-sales environment that they can can participate in from a customer perspective yeah so second question how does an organization like Fujitsu go about deciding who participates in a standard's body who parts oh so who in turn i i think that really just comes down to skills and knowledge because the different standards bodies will have different subject areas that they look at and if it's a question of you know who gets involved in a particular particular working group or in some particular standards activity i would say it's just simply down to skills and knowledge because if you want people involved let's take toga as an easy example to talk about here if you want people involved in toga development then clearly you're going to want to put your experienced enterprise architects into that so i would say it is just mostly down to appropriate skills and knowledge right and something i've always personally been interested in is third organizations talk about using participation in international standards groups as a way of personal development or professional development for employees is that something you see yeah yeah very much so because it is something that's different to the normal day-to-day work and for a lot of people that can be a very a very beneficial effect you know getting exposure to the world outside the company and just broadening your experience space a little bit yes it's absolutely a good developmental thing okay question that's somewhat related to the the first one about about the competitors says is there a cut-off between protecting internal IP i guess i would say balance but it's not my question is there a cut-off between protecting internal IP for advantage and sharing IP with the standards group inevitably yes there is and it's something that i've had some personal experience of inside Fujitsu of course yes you've got to consider if you have something you consider a very unique piece of intellectual property you know be it some technique or whatever it may be then you have to make a judgment about is it best do we think it's going to give us the best advantage to to keep that internal or is it best for our business to actually contribute that out to the marketplace hopefully get some recognition as part of that process and possibly influence the marketplace with this technique or tool or whatever it might be that you've that you've contributed i don't think there's any easy way to to make that judgment but yes absolutely that that judgment has to be made in very much a case by case determination yeah yeah and it's interesting that the people it's a kind of a mindset shift that needs to go on to participate in the standards world and that you know the people some of the people that find that hardest are my former colleagues i would call them uh intellectual property lawyers it's all about protect protect don't give everything away and it's it's very difficult it's a completely different approach to standards to see the benefit of contributing something maybe not the crown jewels but contributing something for the benefit of it of it becoming a standard so it's a it's a balance that i've seen companies have to you know walk that line yes yeah um so one one last question um chris before we before we move to the next one um how do you go about sharing the learnings or how do the individuals go about sharing the learnings that they get from participating with the rest of the organization okay well um no great secret of rocket science it is very much a it is very much a process of internal presentation sharing reports internal communications bulletin boards all of these types of things so for me personally after this conference um yes i'll be i'll be writing up some key findings um and presenting about a number of internal events and i'm sure you know all of your companies must have similar types of things that go on uh internal events where you can share the the learnings from from standards bodies and from standards events i'm sure you must have those things but it is something you have to positively plan and do it just doesn't happen by itself and so it is one of the things you need to do to be able to get the value out of the standards to think about how you are going to propagate these things internally as soon as you get back the day job kicks in and it's easy for it to get it's easy to forget so you've got to actively plan that in christ this was great thank you very much


When we say art imitates life, we mean that a work of art is inspired by real life events or it mirrors real life in some way. We can say this about any kind of creative project.


My piece of advice would be to not just rely on LinkedIn, indeed those job boards, but also to look in the company websites for the jobs that you want and subscribe to their newsletter for new jobs so you are always up to date on what positions are available in the companies that you like.


It's kind of interesting given the title of your book, because as he was describing this test framework they built the set like years and years ago, it seems bizarrely similar to Ansible. They built something that looks a lot like Ansible as an integrated test framework, this is before Ansible even existed, because it's like, oh yeah, we kind of scribe the top, you know, all these hosts on the network and we can have these YAML files which say, you know, run these jobs on these hosts and I'm like, wow, this is like, just keep reinventing the wheel. Ladies and gentlemen, thank you for coming to the final session at DevCon 2018. So we have with us Micah Abbott from the Project Atomic at Red Hat and he's going to be talking about this using Ansible as an integration test framework, so welcome Micah. Thank you. We made it guys, last day of DevCon, last talk, thank you all for being here. I know everyone wants to get home most likely or go off to do other things. So yeah, my name's Micah. I work at Red Hat. I'm a senior quality engineer. I've been working on Red Hat Atomic host and now Red Hat Core OS and this is about how we were misusing Ansible to test Atomic host. This is going to be more like a retrospective on how we, on what we were doing because as we're moving to support Core OS we're going to take, we're actually taking a different tack and I'll get to that towards the end of the talk. So I'm going to start with talking about how we got to this point of using Ansible and then some of the problems we faced when we used this method and the solutions and sometimes just work arounds to the problems we encountered. So how did we get here? So back in December of 2014 I was hired as the first QE guy to work on Red Hat Atomic host. We had like little to no test process, little to no automation at the time. So I got tasked with the normal activities of you know, come up with a test plan, automation and we were really starting to invest in continuous integration and delivery. So I was also tasked with starting to help contribute in that area as well. The goals we had for testing Atomic host after we got ourselves settled was testing the integration of many parts rather than the separate parts itself.  We didn't want to be responsible for testing each individual piece. We wanted to leverage the testing that already happened. But we wanted to make sure we put it all together, it worked correctly. We also wanted to make the automation easy to develop, easy to use and easy to integrate with the continuous integration system that we were working with. And we also had to overcome the challenges of working with an immutable host like Atomic host. For those of you who aren't familiar with it, it's based off of OS tree and RPM OS tree developed by the one and only Colin Walters back there. Sorry, Colin. But it presents certain challenges that you wouldn't normally get when working with a traditional Fedora or REL system. Most of the file system is immutable, so you can't write to the places you're normally accustomed to writing. There's no concept of YAM or DNF, so you can't install packages. And we're also trying to learn the container best practices. How do we put software and applications into a container and apply that to how we test it as well? So since we didn't really have anything specifically geared towards testing Atomic host, I started asking around to the people who did, who were working with it. And I found someone who had started working on this makeshift framework called UAT framework. It was a combination of Python and Behave and it utilized the Ansible API to talk to the host. So Behave is a behavior driven development tool built on top of Python. So I was kind of encouraged that maybe we could leverage that and get more test cases contributed from people outside of the QE organization. So in Behave particular, you write a test case saying I want to upgrade Atomic host and then it's up to the person implementing the test to go write the necessary code in the back end to translate that. So I was hopeful that we would just get a bunch of test cases in natural language and we would just be implementing it, but that didn't really pan out. This whole framework had a lot of layers of abstraction. So we were really slowed down as we tried to pick apart the pieces in the model and figure out where it broke in the different pieces, in different stack, in different use cases. Additionally, the Ansible API at this point, we're using Ansible 1.9 we started. It was really simple, but also really simple in terms of how to use it. But it was difficult in making it do the things you want it to do. So it was a weird juxtaposition. So we spent a lot of time underwinding errors, like I said, going through the abstraction and figuring out where the problems existed. The API was really poorly documented. I was basically in the Python shell selling help on the different functions, trying to figure out what it did. And then when Ansible 2.0 came out, it was like a whole new API. Introduced a lot of breaking changes and all of a sudden, we were left with a decision point. Do we continue to try to push this framework forward, or do we try to go in a different direction? And so we decided we should evaluate our options. We needed something that was easier to develop and use. We realized that a lot of our tests were basically just running commands over SSH on the remote host. And we wanted to be able to run the tests across the different variants of atomic host. So we had a fedora version, well, multiple fedora versions, the red hat version and the centOS version. So if we had a test that could run across all three platforms of little modification, that'd be awesome.  So I didn't want to invest the time to build up an entire new framework, a test framework, to achieve these goals. And since we were already using Ansible under the covers, I should just do this with an Ansible playbook maybe. So we started doing that. We took the existing sanity test case that we had for or should say sanity set of tests that we had for atomic host and we turned into a playbook around as a proof of concept around February in 2016. Being completely new to Ansible as a playbook, using a playbook, we didn't have, I didn't have any roles. It was just including files full of tasks, kind of looked like this. This is actually the get check out from the first commit I made to the repo. There's a couple of Ansible file modules, a couple of shell callouts and just a bunch of includes that are just relative to the the test itself. But we started to get some success. So I convinced the powers that be to host this, the set of tests called atomic host tests, awesome name, right? Under the project atomic GitHub organization on GitHub. We added vagrant support to the framework so that you could do a vagrant up, get an atomic host and it would automatically run the playbook for you. This, this, this sanity playbook or any other playbook didn't really take off in terms of people using it. It was sort of like a nice like, oh, look what we can do with vagrant, but didn't really plan out again. And then finally, I, I've been doing a lot of this work by myself with some help from the development team and other people. But we finally got a second quality engineer to work on these tests with me. And then that's when we started to like really, we're starting to get, get better at it. Like we were starting to learn more about Ansible, how to the proper way to use it. Start to incorporate roles, things like that. We were testing across multiple versions of atomic host. We were hitting a lot of the goals. So we were feeling really good about it. But there was a lot of pain. So number one, Ansible is terrible at handling reboots. You know, Ansible is a descriptive language where you're basically telling, you're describing the state of the system. So it's hard to describe a system as reboot.  It's either doing some action. Reboot doesn't really fall into their, their model. The default output from Ansible playbooks is God awful. It's just JSON. Wait, where'd it go? It used to have, it'll come back later. I'm sorry. I forgot the order of my slides. Awesome. But anyways, it's like, it's just like JSON without any line breaks. It's really difficult to parse and debug and triage. So we were fighting with that when we've got errors. When we, when we chose Ansible, I had the idea of like, well, if we fail as early as possible, it proves it will prevent us from shipping an atomic host that has problems. What, well, the problem was that is, you know, if you fail early in your test, you're missing a lot of covers. Yes, exactly. So the other problem we had with this is that once the playbook fails, Ansible basically terminates all your SSH connections and it's hard to go back to the host and grab debug information, logs, or whatnot, you know, using just the normal Ansible model. And we had some difficulty selecting, like, which test from a playbook to run originally, because I said we were still kind of like Ansible novices and we hadn't quite solved all the problems yet. But number one, like Ansible is not a programming language. So like, we were completely misusing it as such. Like, we were trying to do, you know, conditional operations and complex operations that would be pretty easy in any other language. But in Ansible, you're trying to do it with the YAML and it's DSL. And that's when we kind of realized we made a huge mistake. But we were already too far into it, so we had to just keep pushing forward. And so we started to attack the problems that we had and try to figure out solutions to the different problems. A lot of these solutions, I gotta say, didn't come from myself. I came from the other engineer in our team. He's not here today, but his name's Mike Nguyen and he's been very helpful. And Jonathan as well over there, he helped us with this. Specifically, improving the group reboot handling.  The SSH connection can get terminated before Ansible knows it's actually been successful in issuing the command. So the common way to solve this now, and at the time when we came up with this, Ansible didn't really have a good way of telling its users how to handle reboots. It wasn't until about Ansible 2.0 where they actually put out an article saying like, well, this is how you do reboot guys, that makes perfect sense, right? So it's an asynchronous action. You add a sleep before the shutdown in a shell command that's logically combined. And that actually worked pretty well. I mean, we're actually able to reboot pretty consistently, but we still had other problems in that space. If the reboot command doesn't succeed, the host hasn't got down. So Ansible doesn't know it hasn't gone down. And when it comes back and looking for the host again in a further task, it just assumes that it's rebooted. But if you're expecting something to have changed because of the reboot, your tests are gonna fail. So what we came up with was comparing the boot ID from the proc file system. I'm not gonna show you the code because it's just taking the boot ID before you reboot and then comparing it to the next one when it comes back. So here's, this is the slide I want, this is the part I wanted to show you. This default output from Ansible is awful. So here I've run a command on the Atomicoast, Atomicoast status. It's supposed to give you a nice pretty output of, when you run it by as a user, gives you a nice pretty output of the state of the system in terms of the version you're running and the commit ID. But right now, if you try to parse that, you'd be pulling your hair out of your head because it's a mess. So Mike wrote a callback plugin to basically make it pretty. It looks a lot like this. It basically takes the result object from Ansible, breaks it into the different parts, like standard out, standard error, that the message, you get the return code and what not. It actually uses the line breaks that come in through standard out. So the same command now looks like this in our logs. So when we have an error in the test, we can go into the logs and easily pick out, the return code is not zero, for example. So there's a failure there, not in this example, but if it was a failure. And we can look at the standard out and we can even look at the standard error if it was a failure. So this is like a huge one. It made our lives so much easier once we could actually look at output and understand it. So the problem of capturing information from the host after a failure. Mike came up with this kind of hack way to use a handler to go onto the host and pull out the journal. It's a role that called a handler on failures, which then would go in and set up some names and grab the journal. I mean, you shouldn't have to do this much work with a test framework to be like, give me the logs from a failed system. But hey, we're butt into this so we had to do it and this actually works. I mean, it's not pretty, but it works. We were able to get logs and journals and what not. The failing fast problem. So how do we get the rest of the system that we missed or test the rest of the system that we missed? So Mike came up with this idea. I mean, Mike should really be giving this talk if he's not here. He's in Hawaii. Tough life, right? So he came up with this idea of a meta playbook. Once we switched to using Ansible 2.2 or 2.4, I can't remember which. We were able to use block and rescue more frequently or better suited to our use case. So if you can see this on the screen, we've got a block and rescue definition where it includes some tasks, sets a fact based on the results of the include task call. And then at the very bottom at the end of this, it rolls up all these facts into one. And then prints out a nice log of these parts of the test have failed or passed. But again, you shouldn't have to do this in a test framework. It should happen more naturally, right? So the problem of it being not a programming language, I mean, there's nothing you can really do to get around that necessarily. You can work around some of it by using inline Python. And I was told when I gave my slides to be reviewed by somebody that I should be prepared to defend this. Because I guess there are people who are better at Instable than I am who would just write a bunch of YAML to do this. But for example, you can do these kind of hacks where you're using like splits on strings and capturing those. If only another variable is set, are there else to use another variable? I mean, not fun to read by any matter of the, but these are the kind of hacks and tricks you can do with Python in your Instable paybooks if you care to do so. So selecting the subset of tests, that was a pretty easy one. We just started using tags. We used tag everything into one functional group. So this is the upgrade set of tasks. And this is the reboot set of tasks. And that made development a lot easier because then we could just say, oh, run these set of tests when we're trying to debug which one failed or whatnot. But it still wasn't really great because if you forgot to tag in a piece of the functionality, then you're wondering why it didn't work correctly. So the way our repo is set up right now is that at the top level, we have a roles directory, a callbacks, a callback plugins directory, and then a test directory. The test directory has all the playbooks, which require the roles at the top level. So you could do like a relative call to the role, but they didn't look pretty to us. So we just simlinked everything. Again, not the best solution, not the prettiest solution, but it got us around it. And it's been working so far. I mean, simlinks are simlinks. So the conclusion here. It was a lot of pain, not just some pain, a lot of pain, but it worked for our use case. And our use case was pretty well-defined. We had this immutable host, and the other thing I mentioned here is all these tests were basically single host test cases. So we were just running one playbook against a single host and just testing functionality. And that's it. We didn't get into the larger running a Kubernetes cluster across six nodes type of thing. We had a lot. It was relatively easy to develop, maintain, and execute these playbooks. It didn't require any specialized software on the host, which was big for an immutable host. All we needed was SSH, which every Linux system has, and Python, which most Linux systems have. And it was easy for us to run these from a Jenkins job. It made us, yeah, and the other big gain here was the ability to write the test once with a little bit of sugar to make it work across multiple variants. And then we end up with like this. This is our ad hoc dashboard basically on in the repo itself on GitHub. So here we're testing 11 different variants of atomic host that we run on our internal Jenkins. We then publish our results to an S3 bucket so you can get the logs. You can still get the logs. You can't see the jobs themselves publicly, but you can see the results publicly, which was important to us for the community distributions like Fedora and CentOS. And we got a nice badge just generated from badges.io. So it kind of looks like we know what we're doing. It's mostly green, so we must be doing something right. Yeah, and so this is like, this is our bar for success. Like we got, we're testing 11 some odd variants plus the rel variants that are internal. So we were doing pretty good. However, this is where, this is the swerve at the end of the talk. Don't do this. Just don't use Ansible as a test framework. It's not suited for it. It's, you saw all the problems we had. You saw the workarounds we had. Use something else. Like a bash script would probably be better at this point. You could use Ansible to like prototype some tests that fall into this model of, I need to run some commands on a host to make sure they ran correctly. That's a fine use case I think for Ansible because it has the underlying tech to do that or the line functionality to do that. Again, it's not a programming language. There's a, I think there's actually a pretty, there's a blog post out there that has this title like Ansible is not a programming language. And it instructs you not to do exactly what I just did. We had to use some hacks, workarounds and general abuse of Ansible. If we were going to start over, we would do something, we would use something like pie tests or avocado or open QA or whatever the, anything other than Ansible right now. I mean, I'm sad to say. And like I said at the beginning, now that we're working on Red Hat CoreOS, we got really lucky that the CoreOS guys had a really slick test framework called COLA that's written in Golang. It does things like provisioning in all the different clouds they support. It does things like capturing the logs and the errors automatically. You can write native Go functions and run them on the host. So it really expands like what you can do in terms of testing an immutable host. And I'm actually looking forward to using that in the future as we test CoreOS to completion. That's it. Thanks for listening. Is there any questions? Yes, sir, Adam. Oh, wait, microphone, sorry. So I'm just curious. This whole thing got written specifically for Fedora Atomic Host called Two Week Atomic and that has its own set of tests which run anything called Auto Cloud. And I didn't know until now that there was this whole other set of tests which is also being run on Fedora Atomic Host but it presumably isn't looping into the release process in any way. Just how did that happen? And are there plans to change it? Or communication. I mean, I don't have a good reason other than poor communication and we had much like a lot of the open source community just have a limited amount of time to get the things done you wanna get done. I remember having discussion with Kushal Das, right? Yeah, I had a few discussions with him and he, I think he actually got some of this working in Auto Cloud but we never got it like fully plumbed in. Yeah, this stuff came first, to be clear. Like you were first then Two Week Atomic stuff came after. I just checked the timelines. Oh, okay. Seven, 20, 14 or something, you know? Well, that's when I got hired. We didn't start doing this until like 2016. Yeah, I mean, I think there was a little bit of overlap. Yeah, so. Anything else? So go ahead, Mike. So in my workplace, we're using Ansible for patching homogenous workstations and dozens of servers running very different applications. And usually our patching verification is, oh, make sure the service is running, make sure it's listening on a port. It seems like, even if we were patching them manually, it seems like this would be a good way to do those basic system and verification. Yeah, I think that's a valid use case for Ansible, you know? Describing the state of the system, and Ansible does a very good job of like once you describe the state of the system and make sure it does everything it can to make sure it is that state. So if you're saying like, apply this patch and make sure this service is started, it will make damn sure that as best as it can that that service is started and that's like your verification check. Right, I bet like making sure actually started successfully though. Yeah, yeah, yeah. I mean, you can, I believe like the service module for Ansible has like ways to configure it, how to trigger success. Like you can look for a particular port being open like you mentioned or a particular string that gets output in the logs or whatnot. Don't quote me on that because I haven't actually used those. Right, I guess a simple example is the service is enabled but oh, it doesn't start at boot. You have to manually start at five minutes later. That's the kind of thing we want to detect. Which I feel like this would be a good way to solve it to be honest. Yeah, I mean, Ansible has its use cases for sure. I mean, like I said, as a test framework, like we didn't, I mean, Jonathan and I had this discussion and we kind of came to the conclusion of we got the mileage we needed out of it but it's time to go on to different things. Like use Ansible for configuration management like large scale deployments like that kind of thing. Don't use it to test software. Anything else? I'm sorry if I disappoint you by showing up and telling you that it doesn't work, that you were expecting to be like find this magical unicorn of software but them's the breaks, okay. Is there a plan to try and be more unified with testing isn't as cool a thing in the future? Because I mean, the whole thing about this basically is that no one's maintaining it as dumb. So in Fedora side, the auto cloud stuff is kind of dead. It just sits there and runs, but no one's maintaining it. So is there anyone planning to sort of move forward with the COLA or anything else and try and be more integrate and didn't have all the things tested using the same thing? I'll say yes, there was somebody from Fedora QE. Remember his name, Jonathan or Colin? He was commenting on the Fedora tracker like looking for collaboration in Fedora QA. Kalev or Kalev? Not Kalev Lambert. Oh, Camille, right? Camille, yes. So Camille's reached out to us. He's, and I was actually encouraged that he's getting involved as early as possible. We're not quite there yet to like, especially on the Fedora side, like Fedora Core has still like very much being defined. So we're not quite there yet to be like, hey, here's our tests and here's what you, we don't even have an image for them to test against. So we got to crawl before we can walk. But yeah, the short answer is yes, we're having those conversations. Yeah, I don't know if I've been told to sort of try and create it as well as to make sure it's all getting hooked up compared to this last thing. That's it. Have a good DevConf, everybody. Thank you very much, Micah. So that was a final talk for the DevConf. And right now at 3 p.m., we have, we are distributing a lot of fantastic prizes for a trivia game in the closing ceremony, which starts at 3 p.m. in the keynote room that's the Metcalfe Large. So see you guys there. You don't want to run into a casual start-up service after a reboot. You want to verify that, I mean it's going to move. But you know, your virtual organization cluster has to be just coming to the end of the month, so it's going to find out that it's going to be there. It's going to be canceled.


Good morning. Myself, Dr. Shabnam Parvi, post-casual trainee in the Department of Radio Diagnosis, IPG MIR, and IPG MIR Polkata. Title of my presentation is HRCT thorax findings in nematode arthritis and its correlation with spirometric indices. The nematode arthritis is a systemic inflammatory disease most commonly affect the joint and usually associated with auto antibody production. Plural pulmonary complications are responsible for significantly increased morbidity and mortality. Lung involvement ranges between 67 to 10 percent in defense studies. Pulmonary involvement accounts for 10 to 20 percent mortality. Common nematode arthritis associated IELDSAR, non-specific interstitial pneumonia, usual interstitial pneumonia, organizing pneumonia, DIP, LIP, diffuse alveolar damage and acute interstitial pneumonia. High tidal of nematode factor, anti-CCP, IR age, male gender, family history of nematode arthritis, smoking, et cetera, increases the likelihood for developing nematode arthritis associated IELD. In the aims and objective of the studies, where assessment of pulmonary affection according to duration of the disease process, pattern and frequency of findings on HRCT thorax, correlation of spirometric indices with HRCT findings. We use the studies for review of literature. This was a cross-sectional observation on the study. Sample size was 50. It had been conducted in the Department of Radiogenesis and Metrology, Pulmonary Medicine in IPG email. Inclusion criteria as a patient within 55 years of age after satisfying clinical examination, we see a similar criteria of hematode arthritis with pulmonary symptoms. Exclusion criteria as a patient with history of pulmonary TB, chest radiation, injury, lung transplantation, occupational lung disease, connective tissue disorder, and pregnant patient. Now coming into result and analysis. There is female preponderance, male is to female ratio, was two is two, two is two one, and we divided the patients according to duration of the disease early up to five years and late more than five years. In HRCT findings, the most common findings was down glass opacity followed by bronchiopterosis, interstitial thickening, rural thickening. Lung involvement in HRCT thorax was 45% in early disease group and 84% in late disease group. Correlation between disease duration and lung involvement in HRCT is statistically significant. We also found that 99.6% patients had rheumatoid arthritis associated ILD in early disease group and 52.6% patient had rheumatoid arthritis associated ILD in late disease group. Rheumatoid arthritis associated ILD and correlation between disease duration and ILD pattern in HRCT is statistically significant. 43% patients were having ILD among the patient with lung involvement and USIP pattern was maximum. This is the cross table between spirometry and lung involvement in HRCT thorax in early and late disease group. And correlation between HRCT thorax and pattern of spirometry is more significant in late disease group. 43 years old female patient with 2 years of disease duration shows air trapping in basal segment of bilateral lower lobes. 43 years old female patient with 9 years of disease duration shows small sub plural ground glass nodules in apicoposterior segment of left upper lobe and postrovisal segment in right lower lobe. HRCT thorax in 50 years old female with 1 year disease duration shows GGO and fibrosis with fractional bone care passes in both lower lobes and medial senile window shows multiple different sizes medial senile lymph nodes in pariotic and left paratracheal region. It was an NSIP with medial senile lymph adenopathy. HRCT thorax of 42 year male patient with 8 years history of rheumatoid arthritis shows honey combi throughout the lung field predominantly in the lower lobe with few patchy areas of ground glass opacity. And there is evidence of peri bronchial vascular interstitial thickening with fractional bronchial passes in both lungs. The axial medial senile window shows ratio between pulmonary artery and ascending outer is more than one. This is a case of usual interstitial pneumonia with pulmonary artery hypertension. Discussion female and male ratio is 2 to 1 in our study 62% patients were disease duration of less than 5 years and 38% patients were disease duration of more than 5 years. Most common findings in HRCT thorax is GGO followed by interstitial thickening, pleural thickening and air trapping. Correlation between disease duration and lung involvement and eyelid pattern in HRCT is statistically significant. Most common type of rheumatoid arthritis associated ILD is USIP pattern and correlation between spirometry and lung involvement in early and late disease group of rheumatoid arthritis is highly significant. So we conclude that type and manifestation of lung disease in rheumatoid arthritis varies and can easily affect any lung compartment, including lung panenchyma, pleura, airways and vasculature. In this study, UIP is most common ILD and GGO is most common findings. As the duration of illness increases, need for spinning of pulmonary involvement with PFDs and HRCT is to be emphasized along with periodic chest examination. Thank you.