Okay, so let's get started because we don't have a lot of time Especially after the try-ons we have we have done so welcome everybody to this after the day of the open-stack Summit where myself actually Even myself Will give you some insight in our ideas this travel with with on the open-stack journey If you have more than three or four slides then it's always good to have an agenda So we start with a short introduction so that you know who is talking to you and also what is the business of our materials We are coming from because this is also important for for the later slides Then actually I will give some insights why we actually did look at open-stack Why we actually have selected open-stack as a technology to be used and then most of the presentation Actually we will spend on our journey, but what was the challenges we were facing and also later on then e-figure some technical Details on the actual setup you're using at the moment, and then we wrap up with a one-slide one-slide summary So while they're still getting prepared I start with myself So my name is Ute Seidle and originally I'm a teacher for mathematics and physics and I spent some additional years at the University and back then actually I got Infected by the Linux and the the open-source virus and luckily I never recovered from from that one and before I joined Amadeus more almost 10 years ago I was working in different areas as a consultant as a trainer and spend some additional years also at HPC environment supporting a big compute clusters for German car manufacturers like Audi or BMW and inside Amadeus I also have I had different roles and responsibilities and since our last three organization I belong to these CTO office, which we internally call the Infrastructure and architectural governance unit and with that one I hand over to Yves to introduce himself Good you may actually ask yourself why Amadeus and VMware sharing the stage at the open-stack summit and on the first part Why VMware and and actually for some reason I Get to this one later So maybe I introduce Amadeus a little bit first so normally I did ask always okay Who knows Amadeus and we had just a chat in front of the room about that one So for the people coming from the US if you have heard about world-span or saber Then Amadeus is kind of the European counterpart of it If you're coming from China or Asian or travel sky Amadeus is kind of the European counterpart and actually compared to open-stack We are quite old company Amadeus was founded by European Airlines in the late 80s and according to those founder airlines We actually have distributed our main functions So we have our headquarters in Madrid in Spain because Air Iberia was involved the main development site is in Sofia and T-police close to Nice in France because the French France was involved and The data center so the heart of the service We do provide is in Germany because the German Lufthansa was part of the game as well And this is actually where I work. So our main data center is located close to close to Munich Now we started like the same person travels skies as a classical a global distribution service provider computer reservations for travels by by air but since then we have evolved a lot and if you look at a picture you you can see that we Cover many aspects of traveling today. So it's not limited to traveling by airplane also by cars if it's a rental car booking hotels going by ship even up to the extent to Travel insurances and also getting closer than to to the airport. So for some Airports and airlines we do provide so-called departure control services Which actually makes sure that you can actually get your porting pass that you actually get the tax for for your luggage And for us actually this was a big game changer when we went into this business and we went even further Now providing as well scheduling of the runways for some pilot airports there as well So since we started in the late 80s, we have changed a lot We gained a lot of more more business and we have been quite successful You're not the first one in the market, but we are now more since quite some time We are leader in the market now as I mentioned our main data center is in central Europe Close to Munich and we do have operational sites across the world the main sites are on the east coast and the yes And in Australia to provide the kind of Follow the Sun principle to cover global operations But we also have people are ramping up in Bangalore in London somewhere else in in Germany and in other areas as well And now I get to the point which I by accident wanted to get on earlier Why VMA and Amadeus who coming from totally different businesses are actually teaming up here at the OpenStack Summit and yeah Well, the story is quite easy We have a relationship with VMA or VMA has relationship with us for for many many years So we started quite early with our compute virtualization With some GSX servers and we're having running Windows guests on them just to better utilize the capacity and Sustain as you can see on on the second part of the slide the world has changed a lot on our side So we're talking about 1000 plus your six hosts today 6000 plus guests running and both numbers are actually growing to the natural growth to the migrations from micro from mainframe to the distributed world and also also new business and because of this long Lasting and helpful relationship. We actually came to a commercial agreement Which is a kind of all you can eat a contract for us Which makes it much easy for us to as Amadeus to use VMware products So some of them are mentioned here for Software defined compute for software defined networking and for software defined storage and of course here for software defined Infrastructure so VMware integrated OpenStack, which is the distribution we do use for our Installations which we cover on later on So as I mentioned we are World leader and in the travel industry and we have been successful for many years. We are still success We are still successful. So the question is why should we change things never change a winning system? So why should we look at things like cloud type things infrastructure as a service now The truth is quite quite easy and I skip over this slide to save some time Our customers have changed as well Sometimes because they wanted to change but also sometimes because they needed to change because the customers of our customers have changed The entire T world has changed So if you really look in the crystal ball and if you look what's going on there what other companies are facing We still can continue to do the traditional business with our traditional customers But our new customers we want to attract there are new business lines We want to explore in order to do so we need to change things And then looking what other companies are doing facing the same Challenges are coming from the same background then looking at infrastructure as a service seem to be a quite a good thing for us And if you boil this down a little bit to what we need actually is so our workloads Which which traditional was served out of the data center in Central Europe? So inside our own walls behind our fences We need to make this work those workloads moveable and even more Cloneable or replicatable to a large large extent being movable within our premises So maybe from the main main data center to the disaster recovery site But also going outside our premises off-premise maybe to a hosted cloud or even to a public cloud And by doing this we didn't want to give up on the freedom Selecting actually the hardware we need to run this keep in mind We still have a large data center and we will have this for quite some time So we don't want to give up or we don't want to introduce Artificial lock-in to buy a particular hardware or to buy a particular software if you want to join the open stack as the open stack train Now Open stack. Yes does provide infrastructure as a service, but there's a big difference It's it is open source. So now looking at open source technology is not really new for Amadeus and actually even before Linux was born internally The applications we have developed the source code was available to everybody inside inside Amadeus We did have kind of this open source mindset already many many years ago And we are not early adopters But we are not light up that does either of Linux and other open source technologies like Apache or Chaper So we are quite used to how to interact with a particular community How does it work when you have our community versions and enterprise versions and the second aspect which we are looking for where? APIs and and standardized API's and such kind of things are also not new to us either because that Market we are in the industry. We are in we have to work with competition We have to work with so many different customers the standardized API's for something we were used to anyway So this is something which is not new to us, but actually it's a requirement what we want to have Now getting back to the question open stack why I would counter the question why not I think in these days There are not many options left if you want to have a cloud type or infrastructure as a service implementation On-site so in on-premise If you want to have open source a few years ago I think this question might be more Applicable, but in these days does not much left and the second point I want to make is that when you want to go this way or when you want to select a particular product, please make sure that you Stick only to the technical aspects of it and I will touch on on that one for most of the parts of my slides during the presentation there's a huge non-technical aspect or even at least a semi-technical aspect of choosing a solution as well and actually this one I have seen quite often will determine if your project if your journey is successful or not Another point which was important for us by selecting open stack as the technology for infrastructure as a service was that okay We have to think beyond if you have infrastructure as a service That's good That's cool for the infrastructure team because then it's easy for them But doesn't provide any business not good enough for us because we provide really applications or even services on top of that One so what we need is first of all easy way to integrate existing platforms existing applications as well But also a possibility maybe to explore new paths on the on the platform layer or on the software layer So kind of doing our two two things up with one set of software Like application packaging and here I think about Linux containers or going even one level up And of course integration and interfacing with existing environment because we cannot simply throw away the data center We cannot simply Build a new one some some bells and start from scratch there We still have to do a traditional business to serve our traditional customers And we have the new stuff on on top and it would be good if we can integrate this as much as possible So the overall journey I think there are four Phrases of it Before you start and this is what we what we did as well take a deep breath sit down and breathe Think through the situation. Where are you coming from? What's a current situation? But also where do you want to go and then based on that make sure that you get the right start to it because later on It's much more complicated to fix your your path or your travel when you have screwed up at the start And while you're traveling make sure that you stay focused that you all that you don't lose the momentum That you attract as much as possible To support you on on your journey and as I said already Don't lose the focus always keep in mind where do you want to go and what's the ultimate goal on that one? now for the moment we Remove two of the three main parts of infrastructure as a service or infrastructure So we forget about storage we forget about Network we just look at compute and the quotes here not so long time ago You would have heard inside Amadeus and this will give you an idea how complicated already Well, it was to introduce software defined compute a car of Virtualization in our data center. So this makes clear. Okay. There's a lot of work to be done Non-technical work to be done if you really want to be successful only on the STC part But also then on the software defined infrastructure part Part of selection for a specific workload You will see later in slides that since I've been trying to address But trying to schedule certain work builds on certain And I guess that's what you need to do enterprises to bridge the gap on which bridge from like the hardware Source to Kind of like in compromise way To say okay, you will go around on this hardware that has the characteristics that you Yeah Thanks So some of the things listed here we did on purpose because we know we had to do this some of them We did more or less by accident, but later on with hindsight it turned out this was something something good So we did actually sit down and looked around. Okay. Where are we coming from? What is the existing environment also? What is the existing mindset of the people going back to the quotes on on the previous slide? So became quite clear and I had just a chat with somebody else who's facing similar challenges Do why why are you doing this? This is important. It's not good enough This is just something for the infrastructure unit or for a particular department or division in your organization Actually, it goes to the extent that you have to show it will add business value It will help your company to be successful even more successful in the future and this will also help lay down to get the the power or the The support from the from the senior management or even from from the sea level management Which you will need especially for for the latest point which says secure the resources if you can and this is not easy especially if you are coming from a IT landscape, which is large and which is already running in production Try to fence out bits and pieces that you can start as small as possible and this helps a lot It may not look cool at the beginning if you start with just two or three notes Not with five hundred or five thousands, but it makes it much more easier to identify Okay, what are the dependencies who are the key people which I do need who are which are the processes which have to change Which have to be adapted but also work around blocking stones Which you which you may face and you can extend later on so you have separated your scalability Problems or challenges from the let's say mindset or non technical problems As mentioned already identified the key people and it's not only inside the infrastructure unit And so this is the same on our side They are the core, but you need also the people in the lion organization They are your champions to the spread the word to support you and as I said at the end you have to provide a service You have to provide a business and unless infrastructure is your business You need the layers on top as well or need support by those people on top as well Ideally you great win-win situations and this should be a no-pray now if you cannot easily come up with those with such situations I think there's something wrong anyway on that one. Yeah, I did List some quotes related to compute Virtualization and actually this is the angle where we are coming from so we have acquired a huge a compute virtualization environment We have some highly skilled people there We also got started educating the people preaching them now It's not bad and hypermises are secure as well and things like that So those this was the angle where we are coming from so we started from from that part of the Of the infrastructure not from the network part and not from the not from the storage part Yeah, and then it happened a customer came along a new customer with some let's say interesting Requirements and we could have sat down saying okay Let's see how we can build this the source provide this service with existing modules It was clear we had to change bits and pieces, but actually we took the decision. Okay, this is good This is exactly our pilot where we start from scratch where we build a new stack based on open stack Which will be our future stack and we started really green field with with that one We did select the right people which we needed there based on the experience based on the mindset and based on other soft skills And it was not that we said okay this particular team or this particular team So we didn't care about the line organization actually picked the people we need we picked the experts and we pulled them together Not really not to virtual teams to to real teams So there were rooms provided for them and they were sitting there and then working there And of course in order to do so you need resources and luckily on our side This was initiative which was fully supported by the by the highest management level you can get inside Amadeus So they did Take care that we actually get the rooms that they get it get the equipment that they get pulled out of their Current assignments that they actually can concentrate on that one And then this is something Amadeus specific in the past. We are used to the habit do it your own Nobody else can do it better than us But actually here we changed this also lots we teamed up with quite a few companies PM They're being one of them because we were quite new in the open stack business So it is really wise to look okay. Who has much more experience either as a provider or as a customer Or some something else. So we did team up also with other companies Where we had relationships also healthy relationships like like redhead or or mirantes So based on that there was quite a bit of known stuff We knew the VMware part and we knew okay This is the way we want to go but there was a lot of new stuff and a lot of unknown stuff as well and here My advice is don't be afraid if things change out of the sudden if you didn't plan it that way That's that's okay. That's that's part of the of the adventure So one of the new things we needed was new mindset And this goes a little bit back to the quotes you have seen on on the previous slide So we needed people really think differently who don't think okay I have done my work now for 20 years to accept it the same way and it was successful It is still successful. Why should I change it? You need people really with a different mindset Also different minds of regarding how we approach failures that I might do is because our business is so critical We are quite afraid of failure something fails that it can impact our customers However with open-stake or going this way you have to change this a little bit You have actually to embrace failures and and think okay your failures will always happen And you have to be confident by experience or by the skill set of the people that you can handle that one And then you can decide if you want to handle Availability on the infrastructure level on your platform level on the software level on both or mixture of all of them And also change the way how you work So at our side we are used to maintenance windows of a few minutes within three months or six months So our approach for change is always was to pull them up to stack them up have a big change And then we then we throw it in and we can manage that one But it doesn't work with with open-stake if you have a release cycle of six months for open-stake release You cannot pull up the changes for six months to do so and of course and for the sake of the password I meant I will mention deaf ops as I said earlier So the main development side was and still is in France and the data center where corporations is working is in Germany So even geographical we were quite quite separated But it doesn't work that way and to end responsibility means that also the developer understands if I do this particular change This could be the impact to the customer or just the impact to my colleague who actually has to run the code Or maybe I experience this much better if I run the code my run the code myself Being in the business for so many years and we still have mainframe to give you an insight here We have a lot of platforms so-called platform one or platform two applications If you can avoid don't try to migrate them to put them on on the new stack You may get it working, but it's much more challengeable is much more complicated and it doesn't make the things the things easier So we took the decision. Okay in order to do so we start from scratch We rewrite the application. We don't want to use the existing application and make it kind of working No, we really do some development work and and start from scratch Of course, it's not that black and white you cannot out of the sudden do the same business With a total new application so at some areas and we actually had to step back and to say okay This one we would like to have to have it this way But actually we cannot be fully stateless We have to cover state to a certain extent and we have to make some some commitments there Sometimes to cut corners sometimes to do workarounds to do things which you would not like to do But we have to do it in order to to progress But that's also something good with that one because you have noticed 2.5 or 2.8 Platform applications and they're actually a good bridge between the traditional business and the new business So we didn't go from zero to 101 step It needs a little bit more steps, but that's okay because it gives us also more time More time to learn and last but not least and at least here you need support from the upper management level so our old organization We had a lot of silos so there was application management and actually there was several application management units and even Inside infrastructure and inside the infrastructure era We had a storage department and there was a backup team and then there was a network division and things like that now or Our executive vice president he took the old organization actually turned it by by 90 degrees And he created what we know called so-called a service management units and they are responsible end-to-end Starting from the operating system up to the application providing the service that this one is running and providing the service to the Customer that he or she actually does expect and as part of the of the same re-organization a so-called Infrastructure unit was created. So this was already separated out to a certain extent and Their task is not only to provide infrastructure based on the hardware we have in-house But also to provide services may be coming from hosting providers or from public cloud So we already kind of created this this the same layout you have on open stacks You have an infrastructure part and all they provide is kind of API type things to the services on top Then this is a material specific if you are if you are in an environment Which is this for a while and was quite diverse and with a lot of walls and silos So we have a huge tools landscape and of course we have at least two Ways to approach a particular thing to automate it Which is okay, but it really doesn't make your your open-stack journey easier So we pull tools and automation together in one department one unit to make sure those things get harmonized and consolidated Because automation is something you do you do expect as part of open stack it makes the life much more complicated if you want to also Introduce automation while you introduce open stack and this is actually what what happened so as part of the of our open-stack journey We did some new things and and tried to change things which are done in particular way in the past To do it in a newer better smarter way And last but not least of course we did create particular dev ops teams One at the moment which actually will then run the environment for the for the customer so now some more details or facts which actually will then Never hand over to to Eve to provide even more technical details So at the moment we have we ever in the great open-stack version one which is ice-hole space So we are one of the customers who contribute to the large ice-hole space installation out there But we are looking forward to upgrade to 2.0 which will in kilo base the plan is to do this in in November So this year so ideally before we go into the Christmas freeze Everything is done. We are quite settled on the software defined compute side with these here six We have also NSX Used in in this particular environment six dot one But we are we are also looking forward to upgrade to six to two because it will provide some additional features helping for this project But also for interfacing project with is related to improve how we provide disasters recovery services These are we kind of got started, but we are not as advanced as we could be but that's okay Despite the fact we had the support from upper management We had we have only limited resources and we still have to serve also the traditional Business with the same people. So there was no Urgent needs to tackle this one straight from the beginning. So we kind of have some I think for the control for the control brain. I think we use visa on And and if it touch on that one later later on we will expand it and it can wait That's that's okay We have three installations one in Central Europe in our main data center and actually we have two other installations in the US East Coast and West Coast The one in Central Europe actually was the first one and lessons learned from that installation We actually have then used also to avoid the same mistakes when we built the US locations the Central Europe one is also used for us internally Testing things before we hand them over also getting used to things Spreading the world helping the people educate the people and also bringing on new projects Services which are running on different infrastructures the service implementations at the moment to pull them back to run on an open stack Yeah, and last but not least and parallel to those technical things. We are also working with teams like process management Security offers to get them on board especially to avoid statements like the hyper virus was not secure and If I stick to the security topic for us, it's also something told you in the past all the data were behind Defenses inside our walls, but now we have things outside our walls even outside the continent Which is something totally new for Amadeus we just acquire changes Change approaches different setups different technologies Which also have to work together with our existing technologies because also the environments in the U.S. They are part of our PCI DSS audit security audits because we do deal with with credit card data Training I cannot cannot stop mentioning that one train the people spread the word They need to need to have a foundation to start because only then they can Work with on and of course we try to expand physically expand the installation base We have especially in our main data center Yes, and with that one actually I'm now shutting down and tending over to Eve to give you some more technical insights Before I start What we do is we deploy multiple lower compute Notes are basically on VMS on management VMS and each of the normal compute is actually Interfacing to the v-sphere API or the v-center API is talking to a cluster object. Yeah So no one compute for us is not one single hypervisor is multiple hypervisor species is fixed up to 64 So it's a large pool of capacity we're interfacing with Glantz is also talking to the recent API to upload images onto a v-sphere data store Cinder creates VNB case on v-sphere data store to map them to instances So it's it's very like you you always have the center in between the open-state drivers and And the hydrographs and the same for for neutron at neutron side We have a blogging that speaks to the m-sx and builds those virtual networks that spend as over the networks between Hypervisors, so we have really a complete virtual networking technology Okay, so you heard about the requirements to have specific hardware for specific needs and that Rover requirement together with an h.a. requirement to have To have specific hardware pools within each rack So there are multiple racks and in each of those racks We have hypervisor pools for compute and we have hypervisors for database clusters Actually two different database clusters and also one for data analytics now since the unit is a rack You can't like put multiple of those big boxes in there Supposed to do database stuff and have a lot of local SSD capacity So it's it's it's a single hypervisor and with the logic that we have at the moment They are single No, we're compute notes and is that we we map them to a single availability zone Which is not ideal because what we are want to be able to do is to move workloads around And so for for the next round that will happen November will we use it that and likely will use Combination of host and DRS groups in these here There are DRS groups, which is basically just a collection of hypervisors within a cluster And we can address them on through either image metadata or through failures In that case it would be much metadata. So we take like a database instance We'll attack the image and say run only on those specific host groups and then we can we can Take that down to a single availability zone Similar cluster So those are like the challenges that we had to address to get to this point where we can select specific hardware for specific workers specific workload types and Along with that is is a storage consideration. So there was a tendency to use a lot a lot of local disk capacity instead of share storage and We decided then to go for shared storage more than local capacity because of multiple Issues and those issues are not v-sphere specific. You will encounter them also in a In a KDN based environment and the first one is spawning times So if you have shared storage, you have shared image cache and so your spawning times will just be much faster You only spawn once and not every time not with every hyperlibrator Also, there is a lot of network traffic that is caused by this spawning process if you don't have shared storage And now like in the v-sphere world, you want to be able to move workloads around DRS does a good job in Automatically rescheduling based on load and moving things from one hypervisor to the other and without shared storage It's simply not possible and also for administration purposes if you want to evacuate a hypervisor Shared storage is just helpful. I mean there is there is what VMware calls advanced v-motion You can also move data around between storage, but that's just again causing network traffic and it's lower So shared storage is a good idea Today, it's it's NFS based But likely it's going to be visa and so what you could compare to you are to set distributed storage system That is using the local disk capacity Okay networking we do is neutron and NSX so the deaf environments the production environments etc etc they all take the same Network definitions so no matter if you look in test that on production You'll find the same subjects the same routing tiers etc etc so it's a completely Virtualized networking biome and the network environment is completely configured Within the heat templates that also hold all the information of the application logic and the instances for the application On a physical perspective, it's a layer 3 network. It's an ECMP layer 3 fabric And since we're using overlays, we can build their two networks across those racks Without any problem, and there is an infrastructure utility rack that holds the gateways that go to the outside world the management connections that go back to the audience data center and the internet access and As you said we have a management cluster where the actual open-stack control pane runs So in the IO Open-stack is is based on VMs that run on this management cluster and that management cluster is also across racks and is using visa Okay, so couple of other tools that I used as I said all the application environments are defined in heat those heat templates are all held in an internal gate and All the images and packages are held in the architecture So with that combination, you basically have your whole definition of what the application the networks the whole environment looks like in a centralized version controlled Repository and and then you you pull from it and you deploy your application using using heat and The images also Now the application comes up or the environment comes up at the first thing that comes up is a puppet master And that puppet master is actually then like the focal point of configuration management that deploys the the application decoys things like the couch base cluster And the base application it also configures a local DNS server in inside of the environment And the other instances that come up Auto-registering that DNS records into also this public master What else yeah, there is a little interesting thing which is puppet runs then ansible to deploy a little shift and actually They try to go up the whole application runs in a passing line And that is pulled on open shift and open shift then uses containers To run a lot of the code. So most of the code is ran in containers and sometimes the database and other Monolithic functions are running on the end base that's basically it for the other tools and Yeah, we are at the end. So I take over again to to sum up Yeah, on high level what was what made it successful on our side or what helped us a lot is that we kept the size small and that we State focus that we did select the right application. Actually, we didn't pick any existing application We did create new applications to To bring them on the open stack at journey and selecting that the right right key people And if you look just at the infrastructure part, but we have built in our main data center So we are I think in Clovis. We are more than 900 people But only four or five people actually did work on on this particular project from infrastructure point of view So if you have the right people the subject matter experts the gurus the champions this this helps a lot and of course there was a Good selling of divine win win situations. So actually this particular project. There was Interest from both sides from global operations and from development to go that way. So there was an A natural joint interest to to go so to go that way and to create this to create this new kind of culture Like like DevOps mentioned before The point I wanted to make is with the second item. Yes, it is technology opens open stack is technology But it's really people first It's up to the people who make the thing successful or make it fail and people on all levels Technical managers technicians are people managers or even sea level sea level people If you're not starting your own business if you cannot really start full-time Greenfield It's okay to fence it out and ignore the prone field at the beginning But don't forget it at a certain point you have to look back.  It's okay to leave it beside at the beginning But later on you have to you have to look at that one and this is one of the challenges We have in the midterm future once you have successfully Gain a lot of experience with this set up that how to expand and actually how to migrate existing services We have them and last but not least. Yeah, don't do it yourself a team up as mentioned So we have teamed up with the MMR to a lot of the infrastructure part But we have also teamed up for instance with the threat head We are co-developing the threat head on the open shift side for the orchestration Because this is the community way of doing this of influencing things to make sure that in open shift version 3 is the functionality We do need and we have teamed up with other companies as well for the for the same reason Yes, and with that one and looking at my my watch. I think we have still two and a half minutes left I think it's a lot for your for your attention Also for staying awake and staying in the room as much as possible I think we have time for two or three questions right now and if there are more questions We are still around the entire day today at the party and tomorrow and we also I think easy to find on the internet If you're Google for us Because of OpenStack So and it's kind of more concurrent if you to use But honestly like the complexity of the stack is so big That you lose time at so many places that that might not even be the biggest challenge Especially as we're using lean clones also on the B-Sphere side, which speeds up some things compared to other technologies Yes The practical challenges we had on different layers Okay, you got me Yeah, yes and no So did the person running the project? I didn't have a chance to get briefed before before I got here Actually, there has to be a schedule. I don't know if Arthur discussed it with you in the detail already Oh And they're actually copies of each other they are So Right and of course we start with the one in our main data center first because this is only for internal test purposes So we can we can break it here I think this goes mainly back to really the support from from DVM where people as I said only a few people Did work on that one inside inside Amadeus, but we got huge support out of the M Yeah, I have to say kind of expected because we are used to getting this kind of support But was really a lot of support of dedicated people like if for instance you really made his hand dirty To to make this happen. I think the cluster was running up and running in in June already Would you like something like that really within a couple of months? We made it running and we couldn't have done this on our own We wouldn't have had the skill set and and even if are all the Spreading the word to getting the bind of the people no chance Yeah It's a self written application are providing some services for for hotel customer This is Amadeus specific so the majority of our application or this our core services is provided by application We do develop on our own you you can't buy them You will find them somewhere else which sometimes makes it easier because we can easily change the code or adapt the application But you're also on your own on on that level. Yeah Oh Yeah, long-term of course absolutely we have to so especially in the e-commerce environment you're thinking about To actually that's a pilot just just started But we are there's so many things to to be considered We didn't only change the way how we're going to provide the infrastructure We also changed the way how to want to provide the platform the platform there as well And and the project will focus on on that one first and then the other one should should be easier long-term Yes, everybody should run on on on that layer midterm We started with something from the e-commerce area, which is different to the core business to certain extent But we just got started here I guess actually we are over time but nobody else claiming it. Yeah, okay. Thanks a lot


And we like to think that in most markets that we service, we can keep people busy day in and day out with work. Welcome to the Smarter Building Materials Marketing Podcast, helping you find better ways to grow leads, sales, and outperform your competition. All right, everybody. Welcome to Smarter Building Materials Marketing, where we believe your online presence should be your best. Salesperson, I am Zach Williams, the longest on my coast, and Beth, I think love. And we've got an awesome show lined up for you today. We talk a lot about labor on the show, and today we are bringing someone on who is, I'm going to say on the forefront. You can correct me later, but we're excited to welcome Andrew Spurvey. He is the Chief Marketing Officer for QuickContractors.com. Andrew, welcome to the show. Thanks so much for having me. So, Andrew, before we dive in with our million questions about labor, why don't you give us a 30,000-foot view of what QuickContractors.com is, kind of what you guys are all about, and then what your role is within the organization. Sure, and again, thanks for the invite. Really happy to be here. QuickContractors.com is a Canadian-based company. I believe we are the largest labor force that represents big box retailers in Canada for services. So if you were to walk into a big box retailer and purchase a product on a shelf for special order, a strong likelihood that it would be our company that would execute the installation of that product in the consumer's home. So the point of sale is traditionally with the retailer, a PO would be issued, and then we would dispatch the technical expertise to go and fulfill that job via our proprietary SaaS software as a service platform. And we are also expanding into the US as well. I believe we're in 16 or 17 states now. And we have approximately 5,000 field technicians in the field that are supported by approximately 100 or 110 employees now. Well, I think the question everybody wants to know, I want you to answer, Andrew, is how in the world are you getting labor right now? If that is your business, like how are you finding people and like how are you coercing them to work for you? Very good question. So we have full-time recruiters that are just spending their days in and out being as creative as possible to attract tenants. So, you know, you got your traditional mediums like running ads through various, you know, publications and websites, like Indeed and ZipRecruiter and that, but also reaching out to the trade schools and building those relationships with the teachers for certain trades to let them know that we are hiring basically at all times. And, you know, your question about winning them over, well, it's about volume. We like to think that in most markets that we service, we can keep people busy day in and day out with work. So, you know, if you're looking for, you know, as much work as you could probably handle is in most markets, we're able to have our technicians keep turning the wrench as much as they like.  You don't have to worry about work. We take care of that part. Well, it's a mixed bag. There's some technicians that are working with us 100% of their efforts in not doing anything else. And then there's some of those that would use us as supplemental work where perhaps maybe their days aren't being filled. And so we operate on an app that we created out there where those technicians can, you know, populate their capacity, draw a geo-fence of what, you know, of where they would like to service and what service categories. And you know, you just kind of input your pricing and then jobs would pop into their calendars based on that. Now, one of the attractions for many of them is when, you know, they go into a home and, you know, they do a job or a number of jobs throughout the day. Traditionally, you get home at the end of the day or at the end of the week and you need to prepare invoices and you know, get your counting in order. Where we perhaps oversimplify and it's very exciting to many of them the moment that the customer signs off that, you know, it's a certificate of completion, the job is done, our system will automatically invoice us on their behalf. Therefore they don't need to prepare any invoices and their accountants really like that because we have a dashboard and their accounts can, you know, log in and track all that making, you know, really easy for them at tax time for all the remittances. So we're talking about, you know, calendar, capacity, invoicing, it's all wrapped into one platform. So that's one of the things that I think it's very attractive to many of them. Yeah, absolutely. So it's very similar, and I like hate it when people use these analogies. It's like Uber for this or Uber for that. I was thinking the same thing, but it is. It's like, you're an Uber driver as a contractor. That's huge.  I can tell by Andrew's face, we're not the first person to say it and you dislike it more than us. If for our listeners, you missed a pretty massive eye roll that was awesome. You know what, at the same time, it is, I guess, an honor or a compliment to be compared to Uber in that sense. But yes, definitely hear the word daily from clients, friends, you know, customers, so pretty much everyone. So yes, definitely in a similar sense, you could call it Uber for contractors, yes. So our audience is building product manufacturers. They are finding it difficult, a lot of them to find the people to do the work, right? And then they find people to do the work. The other issue for them is how do I convince the contractor or the pro to use my product to change their mind? Because this is an audience, primarily contractors and installers, they find a brand that they like, changing their mind can sometimes feel like a monumental Herculean effort, right? Right. I'm curious to get your perspective, Andrew, do you agree with that? Like is it hard to change their mind? Or manufacturers bringing the wrong type of argument or ammunition to get somebody to change their mind about trying a product? So the first thing you said was to find them. There's a lot of trial and error when it comes to finding good quality technicians, contractors.  There are a number of boxes that need to be checked. For example, criminal background checks. Unfortunately, there are a lot of folks that we would really like to hire in the trades that don't pass that stage of the vetting process. And it's a very unfortunate, but you need to protect the brands that you represent. And it's important for the reputation to mitigate any potential in PR nightmares, right? And then of course there's the insurance checks, there's workers' compensation insurance, liability type of insurance. And of course you need to be able to validate the skill set, whether it's through referrals or doing checks on their work, a lot of good quality professionals will have a portfolio of pictures to show you the work, that sort of thing, seeing their work in person. And that's just getting into all of this legwork. And so it's all encompassing just for one technician, contractor for one particular trade. All of what I described there is definitely a monumental task for just the one. So there's this ongoing, I guess you would say it's some coaching involved when you're representing large brands, such as big national retailers. And I think the next point that you brought up there, Zach, was building material manufacturers, distributors to position their products to those contractors and to try to swing their business from one particular product to yours. It's definitely tricky in a sense when a contractor is already comfortable with a particular product line, with a particular brand, and they've been through the trial and errors and they're accustomed to the nuances of how to install them. So introducing something new to them, when I say it's tricky, it's definitely possible, but it requires like marketing, of course, a lot of relationship building for sure. And big box retail and also LBM yards is definitely a place for those manufacturer reps to meet those type of contractors, to introduce the products and do demos and that sort of thing. So big box is definitely an important part of that whole chain when it comes to building materials, because I think every contractor in the field will use big box building material stores for at the very least convenience, like a convenience store when they need something on the job, that's where they go. I think the big question for us is just, how do I, as a manufacturer, this is like a general sweeping statement because it depends upon what specialization the contractor installers in, but what they're really after is how do I convince a contractor to try or choose my product? In your experience, like what is the messaging or tactic that you see works the best to get a contractor to go, you know, I use XYZ product, but I'm willing to give yours a shot. Like what have you seen work to make that happen? What I've seen work again is the retailer big box environment where it's kind of like merchandising some specific message into design or video is definitely becoming more important in a sense of delivering the message and perhaps there's a simplicity to the product installation and so on and so forth, but believe it or not, Instagram now has this massive impact on building materials and the contractors. I've been meeting contractors lately and the words that are being used by them is that Instagram is invaluable to their business now in terms of growing it and promoting themselves and TikTok of course too, it's the most widely viewed medium, but at this point I'm seeing more of the building materials side on Instagram today. So it would be a matter of understanding who your specific audience is because many manufacturers and it's not just building materials, it's a lot of industries when you ask them who their target audience is, they're going to say, well, it's everyone, right? Is there a particular age demographic? No, no, our customers are everyone. It's the husband and wife, it's the newlyweds that just bought a home, it's the senior citizens and that sort of thing. So the type of advice that I would give is no, you need to actually do your research to figure out who will be your largest buying segment for your particular product and sometimes you're going after the end user, but you still need to find a contractor to do the fulfillment and it's just really spending the quality time and making the decision on who is it that you're targeting and then just going and executing on that target plan. So make your plan, figure it out. There's a lot of guerrilla tactics out there and that sort of thing, but the organic growth of Instagram for those contractors that I'm seeing out there is actually incredible. And if it is true that your true target segment is everyone, for example, we install a lot of toilets, right? Everybody needs a toilet and that's true, right? Or we install a lot of dishwashers, if it breaks down, you need a new one. The demographics really it's very, very wide, but I would still suggest that you need to hone in and if you're looking for more than one segment then you need to still define them and target them separately through separate campaigns, yeah, absolutely. So it's just massaging your message, really figuring out and of course you can A-B test it, which is nice these days, you can just look at the data of the engagement, but targeting the messaging to the target audience that you're looking for. And I know a lot of people talk this kind of language out there, but actually doing it and really spending that quality time, like I say, whether it's an in-house expert or someone that knows the field, salespeople that's selling the product in the field typically are a really good resource to figure what does the consumer really want. Many years ago, the American Marketing Association, I don't have it handy to tell you when, but once upon a time they published the statistic and I think it was a survey and it was a very, very low percentage of professional salespeople on commission. So that was the kind of key tag there. The salespeople on commission who would actually use the marketing materials that were provided to them by either the marketing department or the manufacturer or the distributor, or even their particular employer for the messaging in the field for sales. The successful ones, again, they're on commission where you need what you kill, as they say. They kind of took, I guess, the materials that were being provided them and kind of created their own to suit their needs. So I'm sure that's still relevant today and professional salespeople will kind of create their own messaging in a sense on how they're delivering it, trying to tap into the actual sales message. From a branding perspective, that's a nightmare. I just had a heart attack. I don't know about, I don't know how, but you're feeling sad, but like, I'm sweating. I think you're right, Andrew. I've heard of similar things and who can blame them if it's hurting your sales. But if I'm a manufacturer, I'm like, hopefully not texting and driving, but I'm definitely sending a quick text to my VP of sales being like, we better not be doing this. I'm gonna need some information. Right, yeah, I totally hear you. The contractor world is, if I find a less politically correct bunch. I think you're right. Let me ask you, Andrew, I mean, you guys are doing a lot to obviously recruit existing people to the trade. Are you doing anything from a training standpoint or are you seeing anybody that's doing a good job at getting people to actually come into the industry versus already recruiting people that are already in it? Yeah, so from a training standpoint, we definitely used to host a lot of big large events across the country where we would have people come in and kind of pandemic, the times have changed where we've moved significantly more towards a digital platform. We are still thinking about going back and doing some personal ones now. We actually in fact are doing an event late next week, getting people back together again for the first time in a while. And so I think it's time to get back out there in that sense. We use a platform called Teachable for training, teachable.com, I don't know if you've heard of it, but it's really nice in a sense that you get to use your materials, there's questions and answers, usually like a multiple choice that you get to set up, engage the knowledge base of the audience that you're trying to teach and then make sure that they've got that knowledge base to at least a minimum standard before you go to endorse them to move on. I'm just curious how you're finding ways to, are you training people to get into the trade? And in addition to that, I'm curious, are there other companies that you see or organizations, whether that's manufacturers or someone else that's doing a good job at finding ways to get people into the trade? There is, for lack of better terms, companies everywhere, including ourselves, are blowing our brains out of recruiting budgets and trying to try anything at everything. And some of those recruiting budgets, speaking from experience, can get out of control sometimes when in your pipeline, you see there's a possibility that you might not meet the demand. So across the board, I mean, we try to recruit within a sense, offer incentives to our employees, family, friends and that sort of thing to get the word out there that we're always looking. We do a lot of in-store assembly, merchandise, for example, where it's bicycles and barbecues for retail stores and it's kind of nice if a student, someone, and even one day a week for as many days as they would like to be available, we get to customize a roadmap to match their schedule to work with our app, right? So with our company, we've got individuals that recognize, hey, I can make a business within a business. So I'll refer to a gentleman named Mo that works for us, he's a technician. He was the one-man crew, new immigrant to Canada, who was out there installing dishwashers. And he came up with a question one day, hey, would it be okay if I train someone? I'll find someone, I'll train them myself and I could actually start a business here and we said, absolutely, that's great. So Mo went on in pretty short order, went from one vehicle to, last I heard 12 vans earning something in excess of a million dollars a year. Oh, wow. It's come up before and I'm pretty sure it'll come up again where maybe it was 20, 30 years ago, the trades were really well respected and really well paid. I remember 30 years ago, even in the big box retailer or lumberyard, you might have a carpenter working on the weekend or a plumber in the plumbing department, that was somebody getting near retirement, but they were getting a really good wage back then and then it just seemed that there was a cycle or a behavioral shift in the mindset of just the general public, where it seemed that the trades started to perhaps not be respected as much. And I don't know why that happened, perhaps even the dollars started to decline a bit than what they were earning. And now today I'm finding that there's a shift. If you were to ask me three, four years ago, I found that there was a stigma still with the younger folks heading into trades. Oh, I definitely think you're right. I think you're seeing, you're definitely seeing other people, especially younger generations, want to use their hands and see the value. And especially when you look at the cost of, gosh, college education in reference to what you're gonna make out of it. I think that you're absolutely right. Now, is that shift gonna happen overnight where labor problems are gonna be solved? No, but I do think you're seeing a lot of organizations begin to educate more around that in order to get more people into the trade. Because I mean, you can make great money doing it too, right? Andrew, question for you, if I'm a manufacturer and listen to this, I'm going, okay, clearly you guys have got a really dialed in system for how you find work, find talent, train, recruit, et cetera. And a lot of manufacturers are trying to find ways to either A, partner with organizations that are doing that well or B are even considering doing that themselves, right? I'm curious if you can, in closing, give a little bit of insight and recommendations based on what you know in your organization. What advice would you give them if they're trying to do something similar in the recruiting and training space with pros? Yeah, there's definitely a lot of manufacturers out there. There is a shift in the marketplace where the manufacturers are thinking about test piloting direct consumer programs and offering up labor at the point of the sale, like in the e-commerce checkout platform.  And just to go through the trial and error and the pain points of who is manageable, who isn't and who didn't show up and upset the customer and who did and so on and so forth, it definitely can be done, but I would say find somebody that has the definite expertise and perhaps a profound expertise with a real good track record of doing specifically just that and making sure that there's some runway there and that you're in it for the long haul. Alternatively, you could find a third party that has a really good track record and does the volumes and that has the capacity or perhaps go around the country perhaps acquiring the smaller players that are in that space that could fast track you to what your objective is. One thing that we get asked or we'll get called, opportunities are facing us all the time and we'll get a call from a large builder and it's a large subdivision and they no longer have technicians for this, this and this task and do you have people readily available and there's companies out there will say, yeah, we're ready to go, but truth is there's no idle capacity sitting anywhere. You're gonna need to develop that capacity and grow it and that's the business we're in and we're constantly looking at opportunities from all kinds of angles to grow that, but why we're successful and I'll share this, it's because honestly, nobody wants to be in the business that we're in and we don't have any competition. That's hysterical. That is some honest talk from you right there, Andrew. Yeah, and then with retailers, they want national pricing. You've got one little town in the middle of nowhere that has a big box store and they want the same price with the flyers in the website and all of that. They don't wanna chop up prices or regionalized pricing for different markets. So we as a company in our risk model in certain markets, we actually lose money on every job we install and that's just those losses kind of averaged out where the gains are in the market. So it's a tough pill to swallow for many or also a business analyst. It's just part of our model. Well, I think you hit it on the head. So Andrew's advice is if you want a successful business, do the things that nobody else wants to do. Absolutely, Bill said that once upon a time. Absolutely. That's great. Andrew, this has been awesome for our listeners. What's the best way for them to connect with you if they wanna reach out? I can be reached at Andrew at quickcontractors.com and that's contractors with an S. I can also easily be found on LinkedIn if you just search hashtag QCrocks or you could search quickcontractors on LinkedIn. You'll find me for sure. But I'm always open to have discussions with anyone. I'm a very open-minded, open door kind of person and I just love creating more relationships out there because you just never know. Wonderful. Again, thank you so much for coming to the show. This has been awesome. And for our listeners, if you enjoyed this content, make sure you go to Venvio.com slash podcast to subscribe and get more. Until next time, I'm Zach Williams alongside Beth and Bobbie Glove. Thanks, everybody.