Well, I guess it's about that time, so should get started. Thanks for coming. My name is Richard Marshall. I am the lead platform architect at IAC Publishing Labs. And today I'm going to talk about our container story. And it's a story that's not of building brand new things in a new, beautiful, green environment, but rather building things intertwined with the old and adapting existing applications to the new model. In other words, our brownfield story. So first I'll talk a bit about what IAC Publishing Labs or ICPL is, where we've come from. And then it's sort of the story time of the last three years that have led up to our first deployment of containers in production. A bit about where things are headed. And finally, some of the things that we've learned along the ways and the challenges that forced us to learn them. So some context. ICPL, we're an operating company for a variety of online brands. As the name suggests, we're part of the greater IAC family with sister companies like about.com, Vimeo, Home Advisor, among a number of others. Though we ourselves operate a fairly sizable amount of brands, for the context of this talk, the most important one is Ask.com. As Ask has a long history over 20 years at this point. And while it may no longer be a corporate entity of its own with ICPL taking on that role, its legacy is still alive and strong. And we might not really even be on the map when it comes to scale in comparison to the big fish in the industry. We still have a fairly substantial footprint in data centers around the world with hundreds of physical systems, thousands of VMs, all to run a fairly large collection of applications for a variety of properties. And like many other businesses that have been around as long as we have, we went through the process of transitioning from a fully physical environment into one predominantly based on VMs. And in our case, that happened about six years ago. And now we are starting down the road towards containers and starting that transition process all over again. About three years ago, in the early end of 2014, the first glimmers of interest of the container concept started to emerge within the Ask development organizations. And it mostly came in the form of a desire for an in-house platform like Heroku or Google App Engine or something along those lines. And at the time, we decided to look into the pre-docker OpenShift origin. And it seemed to fit the bill for most of what our users wanted. We spun up a pilot environment, tested things. It went very well actually. It was stable. It did everything it said it was going to do. However, because this was kind of one of those initiatives that was driven more on the interest in the technology and less of a actual business driver, we kind of ended up at a bit of an impasse where the dev org wouldn't buy into the process of working towards putting real applications on this until operations gave them a timeline for when we would be able to go to production. And ops reciprocally wouldn't do that until dev bought into it, because there's a lot of time investment in actually doing that. And so that kind of catch-22 lingered for a while. And eventually, we just let the pilot environment rot in place. And it's still there. There's still a few internal services running on it. Amazingly, they never need any babysitting. But once you stop touching things, they often stop breaking. So it was close to another year before things really started to get interesting for us when it came to containers. It was midway through 2015. The Docker buzz was literally everywhere. You go to any conference, any meetup, mailing lists, wherever. Everyone was talking about Docker. Regardless of your stance on it, you couldn't ignore it. Combined with that, what really triggered the renewal and interest from our development teams was the advent of the open-source container orchestration systems. So things like the announcement of Kubernetes and Mesos with its marathon and things like that. It really did feel at the time. Well, so now it's not exactly as if there is a consensus on that end. But it did at the time seem like there was a new orchestration system being announced every week. And so we spent maybe a good month or so evaluating the options that were in play and looking at the maturity levels of those various projects, what use cases they were targeting. And while not strictly speaking a container orchestration system, we decided to do our initial pilot using CoreOS's fleet. That mostly was due to, at the time, our perceived view of its simplicity and our existing knowledge base when it came to managing services with SystemD. However, as we tried to scale that up beyond our first development team that was making use of it, the limitations really started to show. The lack of namespacing, user tracking, and various other things really made us realize that we were forcing a use case on it that just didn't make any sense. And so in our initial evaluation of all the options, we had seen a lot of promise in Kubernetes. And we had at the time decided to give it some more time to mature before we really dove into it. The problems we ran into with our usage of fleet kind of shortened that period of letting it bake. And so we built out a Kubernetes environment. And fortunately enough, the migration or the transition period for our early guinea pigs was actually incredibly easy. And the adoption of that early POC platform ramped up pretty quickly. We got a bunch of different dev teams on there, all trying out different things, building out there some internal tooling on top of it that was not really important. But if it went down at four in the morning on a Saturday, who cares, those sorts of things that nobody actually cares about when it comes to the business criticality of it. And in parallel to that work, we started trying to use Docker in addition to the orchestrated container environment. We wanted to really start to get our feet wet with operating Docker on its own. So using it more as a deployment and packaging tool and focusing less on how containers could change everything, but rather how they could simplify and add some efficiencies to our existing CI and deployment processes. And out of that came our first production deployment of Docker in maybe not exactly. We deployed our internal asset management and commissioning system on top of it. And while it may be an important system for us and is critical to our day-to-day operations, it's not something that has any direct impact to business revenue or user experience if it went offline. So maybe not production in that sense. And just as that year was wrapping up is when the announcement of the restructuring of our business occurred. And so that was when the formation of ICPL happened. We all got brand new email addresses. And we got a little bit of a change in focus as far as our business is concerned. And that led into the early end of 2016, which did bring a lot of change in our business. But for the most part, it really didn't change the focus of this initiative that much. It was mostly kind of a rethink about what our primary objectives are as a business. And things did stall a little bit when it came to the project after the announcements in December, which wasn't really too surprising. But it didn't take that long, a month or two later. We were back up and running with our Kubernetes pilot and back on track to grow that environment. By the end of the first half of the year, pretty much every single dev team in ICPL was making use of Kubernetes in some degree. The majority of them using them for spinning up testing environments to do CI against as part of their CI flow, some building small tools, various other internal things. But the further we got with that pilot, the more it became apparent that to have any reasonable timelines for getting to production, we would need some sort of on-ramp that didn't actually include all of the complexities at once. And so to build off of the work that we had done in the previous year with deploying Just Docker with our commissioning system, we piggybacked on another initiative from our production engineering group to improve the automation around our general deployment flow. And essentially just, as I said before, we use Docker as a packaging tool just to get the artifacts onto the system and wrap up the various processes that were already on the same sets of VMs. Just wrap them up in the Docker image, deploy them. Maybe it's not the full container dream, but it's a step in the right direction. And it gave us a much shorter ramp to actually get there. Something that we could see being done with before the end of the year. And during all of that, one of our front-end teams was working on a system to build rapid prototypes for brand new sites that we needed to launch. And this coupled with the refocus of the business, that was kind of the new driver, is let's try lots of new things, spin up new sites, see how they work, how well they monetize. And this was something that needed to be deployed very quickly with very little interface with operations. And so it seemed like a perfect fit for getting started with trying this out in Docker. And it would give us in operations a real chance to operate systems with Docker with live traffic on them without any real significant risk to the business. And fortunately, that was a very successful thing. We didn't run into any real problems that blocked the project. We learned a lot along the way. We ran into some stuff I'll talk about later in the lessons learned portion of the talk. But it really lit a fire under the whole, the entire containerization effort as a whole. And because up until that point, everything had really just been an experiment. We hadn't done anything concrete that was impactful to the business and not just some toy that we were trying out. And so it was also the time to take what was a very small focused team that was building out this POC and working with a small number of dev teams and expand that to the entire technology organization. And so it was taking something that a meeting with three or four people could really iron out what we wanted to do next and figure out how to introduce six or seven different teams into that process and how to train them and how to figure out what knowledge gaps we had. And all of that to not only the knowledge issue, the learning curve issues, but also what are we missing to actually make a production environment a reality? Do we need a production grade internal registry? Do we need a CI and build system that has Docker support natively as part of it? And the list went on and on and on. And during all of that, the Kubernetes build or the Kubernetes environment continued. And it was an important part of our, became more and more of an important part of our CI flow for many teams. And we also kind of came to the conclusion that to get Kubernetes to the point where we felt we could have it in production by the end of the year, we weren't gonna be able to completely roll the entire thing ourselves because we needed to divert a lot of our time that we were working on that to the Docker on VM ramp to get into production as of last week. And that was when the decision was made to shift gears and start using the new Kubernetes based OpenShift Origin. And so that decision did kind of upend a lot of what we were doing and required some sort of rethinking of how we were gonna make that happen. But so far it hasn't been a serious blocker to any of our development users who are building out their test systems and early applications on the platform. So far we've only run into a few problems with the differences between the Kubernetes exposed by OpenShift and the bare Kubernetes that we were running before. So as of right now, last week, we launched our first front end production service on Docker serving about 10 million requests per day. As of tomorrow, we will finish deploying the rest of that service and hopefully that will jump that figure up to about 40 million requests per day, which is still not a significant portion of our overall traffic, but it's not nothing. That's a, we're still serving, well, I can't remember, the like 30 requests per second per node on our Docker based applications. And the OpenShift environment build out is ongoing. It's been a very interesting process to bring together all of the teams across operations and really just dig into doing that. Because before this point, it was kind of the convincing process of, hey, we're really trying to make this happen. Let's get on board and help make this happen, working with network engineering, working with release engineering, working with all of the different ops teams, dev teams, and with the shift with the, or not really mandate, but the business initiative behind getting OpenShift going. It really enabled us to just say, all right, we're gonna do this. And when we did that, it was quite a change in how we made forward progress. I'll talk about what led us to that change in a minute. So looking forward, we're gonna containerize all the things, well, not really. The goal is to containerize all of our front end applications by the end of the year. Some of those maybe running in production on OpenShift also by the end of the year, perhaps, we'll see. And we've come a long way, but we are by no means anywhere near the finish line. Not that there really ever is a finish line in this industry, but even on just this container journey, we're not close to done. But we did make it to that big milestone of serving live customer traffic on containers, which is pretty fun. So to get where we are now, we ran into a lot of issues. We had a bunch of speed bumps, ran into various challenges. And a lot of that came from the fact that we just didn't have a business driver behind this. There was a lot of technologists in our company that really wanted to try this stuff out. There were perceived efficiencies that we could be gaining by making this transition. But as far as a overall business incentive to deprioritize other things to be able to make the time to do this, it just wasn't there. And so with the amount of unknowns that we had, it's like how can we justify letting the other things slip? And given all of the concerns about the relative security of containers, it might seem sort of funny that our security initiative is what catalyzed actually having a business incentive to invest time into containerizing applications. Because the main thing that sort of set that fire was an incentive from our security organization or an initiative from our security or mandate from our security organization to improve the sort of latency between a required security patch for our front-end applications and a deployment of that. And part of the delay in that process has been historically within our company the time it takes to validate everything from the application all the way down into the OS. And so there was touch points across numerous operations teams every time a front-end application needed a library change that had a security patch. And that time to market of getting that deployed was considered unacceptably long. And given the sort of split in dependency management with containers, it was able to reduce the touch points between different teams. The infrastructure team that manages the OS and all of the bits inside of there didn't have to be involved in the QA and LMP process for a new deployment of a security fix for a front-end application anymore. And so that is really what gave us the ability to have that incentive from upper management to say, yes, you are going to prioritize this work. And that really enabled us to make forward progress and get to production where we are now. Would we have gotten there without that eventually? You know, there was enough interest and enough desire from developers and folks in operations to eventually just have it happen organically. But it would have been a significantly longer journey.  And honestly, the learning curve was probably the most challenging thing that we had to overcome before we, you know, in the months and year leading up to our first production deployment. There's a very big learning curve involved in all of this. You know, it's a completely new way of doing things for a lot of different teams. And we had some teams that were very self-motivated for this. You know, the front-end application teams were the most excited, the most desirous of having these sorts of platforms available to them. But not everyone had that same self-motivation and self-starting aspects to try and actually make it happen on their own. Or they just didn't have the time. And as things started to move beyond the just being a pilot project earlier this year, the importance of that training for Docker, Kubernetes, our expanded usage of CI systems with Docker support like GitLab CI, the importance of that exploded.  But we knew that something was coming and we were gonna need to get ready for production at some point. And so we really needed to try and figure out how to focus the training for different types of teams. You know, not everyone needs to know all the things about all of this stuff. You know, the dev teams and the RE groups and the performance and LNP teams. They need to be focused more on the build and image management side of things and how it might fit into the CI pipeline or the performance pipeline, testing pipeline. Our production engineering organization needed to be more focused on actual management of application runtime state and debugging running containers. While the infrastructure teams needed to be more focused on the actual Docker engine and how it interplays with the host OS and the kernel and the networking stack. And so that means that the training kind of needed to be tailored for each different team and the communication around that also somewhat needed to be handled with a lot of care because the fear of the unknown is a very human trait. And when a lot of companies are talking about how they, oh, we were able to get rid of all of our ops teams because we moved to containers, which is somewhat of a silly statement, but it's something that people have a fear around. And so when you're making these pitches to teams that have heard things about, oh, well, in the post-container world, there's no need for release engineers anymore. The communication that you use to talk to teams has to be very carefully constructed. Especially for us, it's like, we need all of these teams on board. And if any one of them is feeling concerned about their job security, that's a problem. We needed to figure out how to carefully manage those expectations of where things are going and how to ensure people that their jobs aren't going away, because that's absolutely not what we're trying to do. And so as far as training is concerned, most of the training is kind of ad hoc, trying people reading the documentation themselves. A couple teams did actually pay for online training courses from sources like Udemy and things like that. But what really paid off for us was ensuring that there was a point contact for every single team that could speak for that team in regards to how things were progressing for them, instead of us having this like giant meeting with 60 people in it and saying, oh, how's your training going? We're trying to train everyone in one go on stuff that they might not actually care that much about. Another thing that's easy to get caught up in is doing all the things right now because we really wanna do all the cool stuff. And this did bite us a little bit. We really got caught up in trying out lots of new things, both in our how we deployed the hosts, how we maybe let's move away from our VMs and run this on bare metal. Let's try out a brand new distribution that none of our operations team have any experience with. Let's ditch the ability to use our configuration management systems because our host OS doesn't actually allow you to install anything in it. And don't get me wrong. I really like CoreOS, which is what we were building a lot of our early platform on. But at the end of the day, having that level or that new aspect of things in play was just one more thing that wasn't really necessary. And so as the farther we got along with this, the more we realized we just need to try and take every single thing and really question whether or not we need it right now. And so at that point is kind of one we sort of made the transition of like, okay, let's stop with this physical migration plan because all of our physical machines are really more geared towards building hypervisors than the configuration that kind of makes a little more sense for containers. We're not gonna buy a whole bunch of new hardware that's optimized for that. And so that's kind of where we ran into some hurdles. We stumbled with that for a while, but eventually we transitioned back to our private cloud environment. We built things on top of CentOS, which we have a lot of experience with and really just tried to pare it out on complexity. And another big thing, this is all very new. Containers themselves as a concept aren't really that new, but a lot of the software we're making use of right now and how it interacts with the Linux kernel is fairly new. And this isn't a criticism on the part of projects like Docker or Kubernetes, it's just when you use new things that are interacting with things in brand new ways, you're gonna run into unexpected scenarios and we ran into a lot of them and we're still running into them. For example, in the period between Docker one, from moving from Docker one six up into one seven and all the way through one nine, we had pretty significant blocking bugs in every single release. Some of them were interactions with the version of the kernel we were running. Some of them were just various bugs that we like ran into with image pulling, network access, all sorts of other things. We got through it and we did run into a few of those situations where upgrading from one version to another version introduced something new and you get into that, well, which one is worse? Do we downgrade? Do we keep rolling forward? And a lot of this is stabilizing and becoming less and less of a problem, but you still need people that are gonna be able to dig into this and figure this stuff out. It's not something that some, not every company's gonna be able to just install it and hope then, well, it's gonna work out the box. We also ran into a lot of problems with buggy hardware causing problems for at CD. And at CD is incredibly important for things like Kubernetes and Fleet because when it starts to go haywire, everything else falls apart too. We also very recently ran into some device mapper bugs. So things aren't going, bugs aren't something that's going away.  You know, last week we ran into a bug with device mapper, a lock ordering issue causing a deadlock when Docker was trying to delete containers or destroy containers and that was fun, but fortunately, CentOS had already fixed that. We just hadn't rebooted our nodes yet. And another fun thing that we ran into was it's pretty important to read the manual for stuff. When we expanded our deployment of Docker on VMs, we ran into issues with our actual physical hypervisor configuration, breaking Docker. It might seem a little bit weird that the underlying configuration of the hypervisor would break Docker running in a VM on top of the hypervisor, but it turns out when we deployed our new 10 gig hypervisors with 10 gig interfaces, we didn't read the manual that that new driver enabled large receive offloading by default, which is a TCP optimization that if we had read the manual before putting these things into production, we'd have known that it was a enabled by default and be completely is breaking to forwarding of packets. Amazingly, we didn't actually run into any problems with the single layer of bridging with just the VMs, but once we added that second layer within Docker, all TCP traffic didn't work. And initially that was pretty weird because it was only on some VMs and it took a little while to figure out, oh, it's only on the 10 gig hypervisors. Oh, and okay, that's fun. Took a lot of debugging to figure out what the heck was going on there, but moral of that story is always read the manual. So yeah, questions. What was that second part? So yeah, the first question of why do we choose CoreOS? And what else did we look at? So when we first went that route, we looked at, looked at just running on our existing CentOS infrastructure. We also looked at the atomic project from Red Hat and RancherOS. We decided to go with CoreOS mostly because of the release cadence. With the early, at that point, Docker was a very fast paced project and we wanted to be able to keep up with it at that early stage in the process. And we also just had, we also found the fleet system to be interesting and wanted to, and sort of decided to go with that as our first platform. See any other important parts of that? Yeah, I mean, it may not have been easy to do, but it really wasn't that hard when it was small scale with a couple of people that were focused on it. When it became problematic for us, when it came to CoreOS, wasn't that there was problems with CoreOS. It was just us adding that as an additional layer of overhead when it came to learning or training up all of the teams on it. As far as what are the gains versus all of the pain that we have gone through at this point? Let's see. The big thing that's really come from the development teams that have been making the most forward progress on this have really in, they see the value as giving them the ability to have a lot more control over what goes into the environment in which their application runs because traditionally the application environment was very gated by operations. We had one or two flavors of the host environment that you could run your application in. If that wasn't ideal for your specific application use case, you would work with the release engineering team to build out chef cookbooks that installed the platform stuff that you needed to make your application work. And there were inefficiencies with that. And so from the developer's perspective, getting that level of control over being able to, oh, I want to build my, it's a bunch who makes more sense for install the dependencies of my application or Santos makes more sense or Debian makes more sense or whatever it is. That was seen as being very valuable and still holds true for most of our dev teams that are making use of this. Additionally, when it comes to the orchestration side of things like the Kubernetes environment, the ability to spin up environments, run tests against them and tear them down very quickly is something that we haven't had in our VM infrastructure. We have APIs for all of that, but they're slow. Spinning up a new VM in our VM infrastructure takes 10, 15 minutes. Spinning up a new container or the collection of containers in our Kubernetes environment is however long it takes for your Docker image to download, which is fairly quick unless you're doing something silly. And so that flexibility of being able to spin up testing environments based on Git commits, we'll actually tie that into a per commit based CI flow was something that was seen as a very valuable thing to developers. And that's what's really driving a lot of the adoption of Kubernetes within our company. And it might seem kind of silly as far as like a win, but the relative being able to do something that's kind of on the bleeding edge and doing something new is very exciting for a lot of people across our organization. And that brings sort of level of excitement and energy that is good when you have an environment that has had a status quo for a very long time and bringing in that sort of big change can kind of revitalize a lot of forward progress on making things better. Sure, so why did we transition from bear Kubernetes over to OpenShift? So part of that came from challenges that we ran into with managing this sort of the security components of Kubernetes on its own. The integrating user management, access control, security isolation was something that was, we were finding challenging. And the sort of set of checkboxes that OpenShift checked off when it came to like our security auditors looked really nice when it came to being able to go to our security folks and be like, hey, this will give us mostly the same thing we're trying to achieve with Kubernetes on its own. There's limitations to that, but for the most part, it'll get us what we want. And it also checks off all these boxes that you are really driving us to be able to achieve in a timeframe that we can actually manage. Was it all hugs and puppies making that transition? No, there was components of the upstream Kubernetes that developers were making use of that had no analogy in OpenShift yet. People were making use of the jobs system, people were making use of various other newer components within like the extensions APIs that just aren't in OpenShift yet. And for the most part, that hasn't actually been a big deal. People making use of the deployment configs within Kubernetes was effectively a search and replace operation on some manifest files. But was it easy to do that? Deploying OpenShift is not without its level of effort. Yes, there's an installer, but it's not a no-touch installer. And it was another challenge that we're just working through that has enabled us to make forward progress because of the security initiatives that we have. And that sort of business driver is what's really enabled us to make a lot of this progress. And because of the sort of checkboxes, the bits that OpenShift checks off on that is what allowed us to make that forward progress. Anything else? Cool, thank you for coming.


Commander 788 here and it's time for another vintage GI Joe toy review and as promised this review will continue Tiger Force Month. For the entire month of April and 2016, all of the main reviews on this channel will look at Tiger Force toys. And this is the review I wanted to do last week. I wanted to kick off Tiger Force Month with a bang. Well, I may not have been able to review this toy last week but it's still an exciting toy to look at for week 2 of Tiger Force Month. And once again I get to thank Arthur who sent me that really awesome care package full of vintage and modern GI Joe's. Finally, I get to dip into that box for a main review on this channel. Arthur, this one's for you. HCC 788 presents the 1988 GI Joe Tiger Force Tiger Rat and Pilot Sky Striker. This is the Tiger Force Tiger Rat and the pilot code name Sky Striker. This vehicle was introduced in 1988 and was also available in 1989 and was discontinued for the year 1990. The Tiger Rat was in the first wave of Tiger Force vehicles from 1988. The second wave from 1989 only had two vehicles, the Tiger Force versions of the Devil Fish and the VAMP. Tiger Force was a sub team within GI Joe and the idea of Tiger Force as it was expressed on the back of Tiger Force Duke's file card was to use captured Cobra vehicles and re-outfitted GI Joe weapons to battle the forces of evil. There is absolutely nothing new in Tiger Force. Tiger Force entirely consists of reissued toys. The only new character introduced was the pilot of the Tiger Rat, but even this action figure was made up of parts from older action figures. The Tiger Rat was a reissue of the 1984 Cobra Rattler and the Cobra Rattler also came with a pilot. It came with Wild Weasel. The Tiger Rat, like the Cobra Rattler, is inspired by a real-world jet, the A-10 Thunderbolt 2. But there are some modifications in the toy version and we will talk about all of them. This wild outlandish color scheme for Tiger Force vehicles may seem so out there that it could never happen in the real world. No one would really paint their jet like this, right? Tiger Force was inspired by the NATO Tiger Association, which has annual NATO Tiger Meats and the NATO Tigers were formed to promote solidarity between NATO Air Forces. For their Tiger Meats, the Tiger Association members paint their jets with wild colors, including Tiger Stripes just like this. The 2016 Tiger Meat will take place in Spain next month. The Tiger Rat did come with an action figure, the pilot, and you did hear that code name correctly. He is called Sky Striker. Nope. This is a Sky Striker, G.I. Joe's first combat jet from 1983. And if there was ever a G.I. Joe vehicle name that should not be reused for something else, this is it. Everyone associates the name Sky Striker with this jet. You can call that figure a turkey sandwich for all I care. This is my Sky Striker. G.I. Joe has reused names before. Viper, which was the name of the Cobra Glider, got reused for a Cobra action figure. Falcon, which was the name of the G.I. Joe glider, got reused for a G.I. Joe action figure. And the vehicle name Armadillo got reused for a later action figure. But I don't think that's so easy to do with the name Sky Striker. The name Sky Striker is so closely associated with G.I. Joe's first combat jet that I just don't think you can reuse it for anything, really. I mean, a Sky Striker is a Sky Striker. And whatever they want to call this guy, that's not a Sky Striker. Of course, we can't look at the Tiger Rat without looking at the toy of which it is a reissue, the 1984 Cobra Rattler. I think the Tiger Rat also takes its name from the Rattler, Rat, Rattler. We're really not stretching our creativity too far here. As far as I can tell, the Tiger Rat and the Rattler use exactly the same mold, right down to the 1984 date stamp on the Tiger Rat. The most obvious difference is the color, and I really liked the Cobra Rattler in that classic Cobra Blue. The Cobra Rattler was a very welcome and needed addition to the Cobra Air Force. The Tiger Rat's colors, however, are just wild. That bright yellow with the black stripes, it just screams, look at me, unlike the original Cobra Rattler.  The Tiger Rat has that same tilt-wing design that the Cobra Rattler had, and that was always a really nice feature on the original. The Tiger Rat also has the same missiles and bombs as the Cobra Rattler, of course, with different colors. If you choose to get either one of these toys, you might consider displaying it with the wings tilted up. It really makes a nice display, and it shows off all those missiles and bombs. So the Tiger Rat has a lot of the strengths of the Cobra Rattler, but it also has all of the Cobra Rattler's weaknesses, and we will talk about those in detail. Let's take a look at the parts and the features of the Tiger Rat. Before we get started, I would like to point out that this particular example has both United States and Canadian stickers. I'm not sure how that happened, but it is interesting. Let's start at the front, as we always do, and we have this nose cone here, which is made out of a slightly softer plastic than the main body of the toy. I wouldn't say that it's softer rubbery, but it is slightly softer than the main body plastic. And then we have this nose cannon, and this, the blueprints call a Jawbreaker CDY9B nose cannon. I really like this feature. I would use this more than I would use the missiles and the bombs. There's lots of nice detail on there, and it has a little thumb wheel here, so you can spin the Gatling gun just really cool. Very cool feature, very cool detail. This nose cannon is inspired by the A10 Thunderbolt, which has a similar nose cannon. On the A10, it is the GAU-8 Avenger 30mm auto cannon. It looks great, it looks fierce, it looks like it means business, and it is inspired by the real world. Now we have the canopy, and the canopy is in two parts. This main part, which if you squeeze it and pull it open, you can open it up and put the pilot in. So it has this front windscreen here, which if you kind of squeeze that, you can pop the tabs out and that is removable. Without the canopy, we can look inside the cockpit and see some of the cockpit detail. And there is some detail there, some instrument panels, some electronics. It's not a lot of detail, but it's more detail than the sky striker, I'll give it that. Let's look at these stickers on the sides, and I guess these stickers are supposed to be like tiger teeth and eyes. They look more like shark teeth to me. And that does harken back to another GI Joe jet with shark teeth, the X-30 Conquest. Notice the color fade from white to brown to yellow. That must not have been very easy to do, I mean, they probably would have saved money just by making this whole jet yellow, but they actually did this kind of color fade there. That is very impressive for a reissue vehicle, which really just stamped out an old vehicle. They did kind of add something there. You can fit the pilot in the cockpit like so. He can go in straight legged, you don't have to bend his legs. And it's a bit easier if you take the windscreen off in order to get the pilot in. But you can pop him in there and then pop the canopy back on and closed. It's a little bit of a tight fit, but it was a tight fit for Wild Weasel too. It's not a very big cockpit. Next we have the gun turret, kind of a midships here. This gun here can elevate, goes down pretty low, and it can elevate pretty high, and the turret can rotate all the way around 360 degrees, and the canopy on there can open. You can put a figure in the gunner's position, and I'll demonstrate using Tiger Force Troop Wire, why not? He will go in facing forward, facing the front of the jet, and you do have to kind of wedge him in a little bit, his shoulders a little bit wide, but he goes in there pretty deeply, and the canopy can close over him. There are a couple of removable panels, one on this side, which you can pop out and see some detail of the inner workings of the aircraft. The other one is around the other side, here toward the back of the fuselage, you can pop that out and see some engine detail there. The Tiger Rat came with a couple of damaged panels, just as the Cobra Rattler did, and you can use these damaged panels to replace the removable panels, so your Tiger Rat will have some battle damage, so you can just replace that one, you can see some bullet holes in the side there, not bad, swing that back around, and just pop the damaged panel in on the other side, and now you have a damaged panel on that side too, so yeah, you can have a battle damaged mode for your Tiger Rat. I don't know how you feel about these, I was never very fond of them, any part that you're going to remove and just not have on the vehicle is a part that is likely to be lost, so I'm not a big fan of these, but I guess they look okay on the jet, they do look kind of cool, those bullet holes, but yeah, that's something that's going to go missing pretty easily. We have these two things on the side here, and my understanding is that those are supposed to be lights, like flood lights or something like that. The Tiger Rat has three jet engines, and that is a bit of a departure from the real world jet that inspired it, it has one jet engine here at the tail, and that's pretty much static, you can see the turbine sculpted on there, it has an engine on each wing, and each of these engines has a removable engine cover, that allows you to see some really nice engine detail, looks really good, and this is a feature that I really liked on the Cobra Rattler, and I do indeed like it very much on the Tiger Rat. The Tiger Rat is a VTOL jet, or vertical take off and landing, which means that it can take off and land kind of up and down like a helicopter, without taking off on a long runway, and more specifically the Tiger Rat is a tilt wing aircraft, meaning that it swings tilt, pointing the jet engines up, allowing the jets to lift the aircraft into the air. This is a departure from the A-10 Thunderbolt, the A-10 was not a VTOL jet, but tilt wing design is a real thing, tilt wing planes have existed, but tilt wing aircraft are usually propeller driven aircraft, not jets. There are jet VTOLs, but they usually go about it in a different way. Now it's time to look at those great missiles and bombs, and to do that we really need to turn it upside down, and we can see that vast array of weapons, I mean it's really just bristling with weapons, and as you can see most of them are colored red to provide a contrast with the yellow and the black. Each of the bombs and missiles fits on the airplane on this peg, in a dumbbell shaped slot, and that is universal, so you can move them around and swap them out and put them in any configuration you wish. Let's start with this one, and the blueprints call this one a Loretz constant scan two stage missile, and it actually does come apart in two stages. If this is based on a real world missile I have not been able to identify it, then we have this one which the blueprints call an EO-118 electro-optical renegade bomb. By electro-optical what they're saying is this is a camera bomb, it's a bomb that's equipped with a camera that transmits a bomb's eye view back to the aircraft.  I've looked at the blueprints a couple times, and I don't think they're identified, they're just missiles. On the wingtips we have bomb racks, each with three bombs. The bomb racks are removable, and the bombs are removable from the racks. The blueprints call these PL Mark 48 radar sensitive cluster bombs. A cluster bomb is a bomb that releases multiple sub-immunitions. These are red, and they have a little shark face on them, and that is adorable. Now we have to talk about the weakest part of this toy. It was the weakest part of the Cobra Rattler, and it is still weak on the Tiger Rat, and that is the landing gear. When I say the Cobra Rattler and the Tiger Rat use the same mold, I am not kidding. These tires still say Cobra V-Tall. These landing gear are extremely fragile as they were on the Cobra Rattler, they're really just the wheel attached to a long thin piece of plastic that's holding it on, and the front and the back, and it takes a fair amount of force to push the landing gear up and pull them back down, which means they are going to break very easily. They are woefully under engineered. To make matters worse, when these back landing gear are out, they don't snap into place completely down like that. They tend to tilt inward, which is going to make your jet sit cockeyed. It can be a real chore to get the Tiger Rat to sit on its wheels rather than tilting over like that. As with the Cobra Rattler, I will demonstrate the landing gear one time, and then I'm not going to touch them again because I don't want to break them. There is no mechanism, you just push them up and pull them back down. Alright, landing gear up, back one up, back up, and then the front one carefully up.  Pull the gear down ever so gently until they sort of lock into place, and once you get them either up or down, however you'd prefer to display the Tiger Rat, just leave them. We have to look at the paint deco on the Tiger Rat because I think that's a feature. We rarely got painted vehicles in GI Joe, and when we did get them, they were usually like more traditional camouflage like the slugger, which I loved. I only wish we could have gotten more of that, but for the most part, GI Joe vehicles were unpainted. But on the Tiger Rat, you have multiple paint sprays, and they are really bold. They are not subtle at all. Now, you may love Tiger Force or you may hate it, but you have to admit, this was something special. This did not happen all the time. And I think they really needed to do that because this is a reissued toy, and they're asking kids to buy a toy that they already have. So if they're going to do that, they really have to add something special, and I think the paint does that. Maybe we would have preferred them to use different colors, but the Tiger Force vehicles were inspired by the NATO Tiger Meat, so even in their garishness, they are grounded in the real world. Let's take a look at SkyStriker. Here we go. A look at the SkyStriker. Starting with his accessories, and the accessories like the rest of the action figure are just reused and recolored parts. And these accessories come from the 1984 GI Joe slugger driver, Thunder. This helmet headset advisor reuses the mold from Thunder's helmet headset advisor. But that's not all this action figure gets from Thunder, and we'll get to that in a minute. Let's go ahead and pull this helmet off and take a closer look at it. This is a three-piece assembly, and it does look pretty cool when it's all put together, but you can take it apart by pulling the headset off, and this headset fits on the helmet with a couple pegs, and it has a couple antennae on there. The pegs of the headset also fit through the holes on this clear visor. This visor is clear, and the one that came with the 1984 Thunder was black. GI Joe did have other clear visors, like the one that came with this 1982 Hawk action figure, but you can tell if you have a SkyStriker clear visor. For one thing, the SkyStriker visor is a little bit smaller than those 1982 visors. So the 1982 visors only had pegs on the inside of the visor to connect it to the helmet, but SkyStriker's visor has holes in the side, and SkyStriker's visor has a little notch there for the nose. At the bottom of everything is this standard helmet, and I would call this a burnt amber color, and this type of helmet was issued with a lot of GI Joe action figures, including grunt. That's the same type of helmet, different color. Let's take a look at the articulation on SkyStriker. He had the standard articulation for 1984 GI Joe action figures, not 1988 figures. That means he could turn his head from left to right. He could also lift his arm up at the shoulder about so far. He could swivel his arm at the shoulder all the way around. He had a hinge at the elbow, so he could move at the elbow about 90 degrees. He had a swivel at the bicep, so he could swivel his arm all the way around. The figure was held together with a rubber O-ring that looped around the inside. That allowed him to move at the torso a bit. He could move his legs apart about so far. He could move his leg at the hip about 90 degrees, and he could bend at the knee about 90 degrees. Let's take a look at the sculpt design and color of SkyStriker, and once again this is where he borrows from thunder. That head sculpt originally was thunders, it's just a different hair color. A SkyStriker has brown hair, and thunder was a ginger. There's nothing wrong with this head sculpt, it's a really nice head sculpt, I like it. It's just a head sculpt that belongs to somebody else. It's not a terrible thing when GI Joe figures reused parts, but I have a harder time with it when they reused heads, because it makes GI Joe look like the clone army. The rest of SkyStriker comes entirely from Wild Weasel. That is, Wild Weasel's body just recolored. On his chest he has a bright orange shirt or jacket with tiger stripes, that's easier to see on the back. Now this orange color is closer to the yellow end of the spectrum, but that is definitely orange, and he has red orange pouches and a scarf. He has black straps for a parachute, but neither he nor Wild Weasel came with a parachute. His arms, again exact copies of Wild Weasel, continue the orange and black tiger stripe theme, and we have brown gloves. On his waist he has a couple brown belts, and they are pretty well detailed. They continue all the way around to the back, and it looks like this lower belt is for a holster, for his brown pistol on his right leg. On his thighs he has some maps and charts, and it looks like he has grey trousers. Those grey trousers continue down the lower legs, and we have a pouch on the right leg, and around on the left leg we have a knife, sculpted on, and then we have brown boots. He has zippers down the inside of his leg, and I think that's an excellent detail. But this detail really belongs to Wild Weasel, it's hard to give SkyStriker credit for it. Even though this is a copy of the body of Wild Weasel, I really prefer it on Wild Weasel. I have a problem with the way they recolored it for SkyStriker. For Wild Weasel this is a red pilot's jumpsuit, probably a G suit, and this is perfectly appropriate for a pilot to wear, a jumpsuit like this, but since they've colored the top half orange and the bottom half grey for SkyStriker, it looks like he's wearing a jacket and trousers. And really, I think he ought to be wearing a jumpsuit as a fighter pilot. This continues my observation that Tiger Force had two color schemes. One Tiger Force color scheme had pretty good military colors with greens and browns and blacks, and those looked really not bad at all. Then they had another Tiger Force color scheme that had this kind of really bright Tony the Tiger orange color with black Tiger stripes, and it's a lot harder to justify sending a troop into battle wearing this kind of color. Now I don't have quite as much of a problem with SkyStriker's color scheme. Now I do prefer more realistic military GI Joe figures, but I do know that you can't have all just realistic military camouflage colors in your toy line. You can't look out at the toy line and only see greens and browns and blacks. I get that. You need some kind of color interest. And the way you get that color interest is to give those brighter colors to troops that are not going to be like ground combat troops, like pilots, like sailors, divers, and technicians. So yes, SkyStriker's color scheme is loud, but if you're going to give somebody a loud color scheme, it's not quite as bad if you're going to wrap an airplane around them. I would have preferred SkyStriker and really all of Tiger Force to have a color scheme A like flint here. But in SkyStriker's case, I think the color of his jet is more problematic than the color of his uniform. Let's take a look at SkyStriker's file card. This file card was printed on the back of the box in which the Tiger Rat was packaged and it has his faction as GI Joe and it has a portrait of SkyStriker here. It has his codename as SkyStriker and it just says Tiger Rat right here. I think they mean Tiger Rat Pilot. His file name is Alexander P. Russo, his primary military specialty is Tiger Rat Pilot as it should be. Secondary military specialty is combat technician. His birthplace is Providence, Rhode Island and his grade is 05 Lieutenant Colonel and that's a surprisingly high rank for him. Larry Hama, the writer of the GI Joe comic book, wrote most of these GI Joe file cards but some of the file cards were written by Hasbro employees. Now Hasbro is based in Rhode Island and they like to make their characters New Englanders. This section says, you might say that flying is in SkyStriker's blood. He grew up on an Air Force base in Maryland where his father was in charge of flight operations. Fascinated with fighter jets, SkyStriker decided to make a career for himself in the Air Force. Upon graduation from the United States Air Force Academy, he went on to flight officers training school to become a pilot. After he received his wings, and you don't have to put that in quotes, I think we know that he didn't literally grow wings. SkyStriker flew X-30 fighters during several GI Joe attacks on Cobra Island. He shot down over 15 Cobra Mamba copters, severely crippling Cobra's island defenses. SkyStriker was tapped for duty on Tiger Force because of his fearless attitude in the face of danger. 30 is referring to the Conquest X-30 which is a GI Joe jet that was released in 1986. And this attack on Cobra Island is probably referring to the Cobra Civil War which GI Joe did participate in in the GI Joe comic books. This bottom section has a quote. It says, the man was born to fly.  It was unbelievable. When it comes to combat acrobatics, this fly boy takes the cake. Well, isn't he just super? This is another GI Joe file card that describes the character as just born to do his job. And that is not my favorite type of file card and really not my favorite type of character. I really don't think that necessarily makes a character interesting or compelling. I'm much more interested in a character that isn't born to do his job, but works hard and learns and becomes proficient. I can be very brief about SkyStriker's GI Joe media appearances. He had no appearances in the GI Joe animated series. He came out in 1988 and unfortunately that was in the gap between the end of the Sunbow animated series and the beginning of the Deke animated series. He also had no comic book appearances as far as I can tell, even though Tiger Force did have a couple appearances in the GI Joe comic books and special missions. Looking at the figure overall, the figure is made up of entirely reused parts. There's nothing original about it. So at best it's going to be about a middle tier figure. I should hate the orange with black tiger stripes on this figure, but I don't mind it as much as you might think. SkyStriker is not a ground troop and by giving these bright colors to pilots and vehicle drivers, you can brighten up the toy line while having the ground troops with more realistic colors and camouflage. Looking at the Tiger Rat overall, I think the jet looks like something special with its bright yellow color and its bold black stripes, but I'm torn. As a child, I would have hated this vehicle. The younger me would have thought this vehicle was an eyesore. I'm still here. What are you doing here?  Well I think as an adult collector that it really stands out on a shelf and it really makes a statement.  That's not fair. I think the color change distinguishes it from the Cobra Rattler, which is a jet that I love and if you're going to reissue a toy, you have to do something to make it fresh and new and I think the color change does that. Nerd. Settle down, Beavis. Now younger version of me.  So this vehicle cannot be a top tier vehicle due to the lack of originality, but I still like it and I'm going to say it's at the top end of the middle tier.  That was my review of the 1988 GI Joe Tiger Force Tiger Ratt. We have more Tiger Force toys coming all this month. You don't want to miss it. Make sure you thumbs up on YouTube, subscribe on YouTube, like me on Facebook, follow me on Twitter. That is the fuel that keeps this channel going. Don't forget to check back next week for another vintage GI Joe Tiger Force toy review and until then, never forget only GI Joe is GI Joe.