don't get the time to document things And you have a change Which has been undocumented And that spells disaster going ahead Once you start these things Start piling up it starts Leading to disasters which We did end up on We used to Actually deploy applications by Running a get pull on the Server and Again we Like our deploy used to happen Like a month or two months maybe Sometimes even two months apart so It was just piling We were in no way we could Have done frequent enough deployments So it used to be like we are putting All together we put The code together and in a Traditional way we used to just Release them and fix everything that comes up At one time Some of the problems document is not updated All the time if we Had to Reach a scenario where Our server crashed or some it Was compromised and we need to Start over from scratch we would have Probably taken good 10 15 probably One 20 days to rebuild Things and this was something that was A very scary situation once we Went live into production Code deploys We probably at times we Actually there was a release where We took three days To debug all the problems that Came because of all the configuration Errors while deploying the code Tracking Of all the changes For the config like we have The code changes being pushed to get But the configuration changes were not Really tracked properly anywhere except For the wiki where it was not really reliable Nobody was Handling the ops task as a Dedicated thing we Were doing the development work But then just pushing things through the Server and let's forget it nobody Else is looking after things at all Whenever the problem arises somebody Will go usually that's me because I was the one who's more Associated with the ops task I Understood the servers better so The what happened was we had one Of the Places just before the production Release we Went to a place where We had a major major failure We our configurations were Not right our code we were Having get conflicts and We probably ended up having A like a week's worth Of work where we were doing overtime And we were running into major issues And management was really Pissed off at us what is this This should not be happening what are You guys doing so That point we really realize that We need something We need to figure out what is there So after the release We set up together And said let's analyze Our failures and look into what is The next step we can do So Along with that we started Having the increase So we realize that one of the things Is that we have a single environment Everything is going there so We decided to have Two different environments one Was a development where everything Would be pushed more regularly And So then we made another environment Where it would be much more Stable and that is what users would Use so there wouldn't be a problem With doing a deploy And if the deploy Is broken then It wouldn't affect what is being used We started going from Monolithic application there was One PHP big PHP application At that point we started breaking It up into a few Microservices which were there And that gave us a lot more reliability Because some of the things had to be Constantly processed in the background And we realized it can be handled Much better with Node.js We had a JavaScript developer with us By that then Still We were Doing a lot of Configuration what I tried is To have a homegrown script Kind of a thing where it's Repo and I'm trying to organize Configuration files within that But With a little bit Work I realized it's just Still becoming pretty complex With various things trying To update together So what would happen is at times They were making changes At one place but The same change needs to go At another place like there is a service Which is a dependency of another service Both needs to have a same value And this was not being reflected correctly When I was making these changes So while doing that One of the first things That I came across Was here at rootconf 3 years ago Somebody introduced me to This tool called Ansible When I visited the website at first I was like I don't know it didn't really Make too much sense to me and I was A bit skeptical about that tool But I still started using it And it's one of the things That I love and I still use it To date So with Ansible The biggest change Happened is now we have a very good Tracking of what are the configuration Changes which are happening to our machines They are much well Reflected there It's in the code, it's there in the Git repo and all the Regulation changes are in one place And we have the dependency between Them mapped out correctly This has been The biggest change that we Could have and That made our system The reliability aspect that we are facing Was way way More simplified just by Introducing Ansible at that point There were things like we also added Some deployment code so we were Doing Git pull but the Git pull would Happen and would happen via Ansible So what that did was Earlier I was doing things Manually here now anybody can Just use the Ansible script And do it by themselves So it didn't really depend Only on me I can Hand it out to the others and Where people can Work with the same Code That allowed us to Do some delegation of the Task and also There was a lot more consistency With doing these things But of course this was the first Step and it wasn't Solving our problems we still had Some really major problems as we were Moving with things We are doing deployments via Just doing a Git pull, Git push And we couldn't Really track what is going into Dev and what is going into The production machine And when was it being done It was not too like a Git push may be done today But the deployment happened two days later When we see it more appropriate So there was no real way to Get the data we didn't have Anything back then So Again the deployments We use we are We have a complete JS stack We have no JS in the back end We use Angular in the front end So one of the things was we used to do A NPM install while doing the deploy And at times we used to Observe that Maybe some network issue Maybe some issue with the NPM At times our builds were not reliable All our dependencies were not there correctly So the same build Which was there in working Completely fine in Dev Would not work in production Just because the NPM install failed And we realized one of the Things that we need to do here Is to have a reusable build How I'm sure pretty much everybody here would be familiar With the CI systems And once you build in your CI system Every the whole Artifacts are available to you And if you can test it on one system At least you are sure that Your build artifacts are working correctly If there's a failure then it's somewhere else Which was not the case At this point We managed to sort out the whole Configuration issue That itself allowed us to build Our systems back If we had to do that Would be in a couple of days Which was a big big difference And we would be relieved Because the business pressure would be high If we were to take some 10 days And not have something up and running That would really be a huge Huge business loss But besides that since we went To microservices architecture We Started initially with just a monolithic application It was just one place Where things used to happen Now with Now with the various microservices The code integration Wouldn't really always happen correctly So this was A sort of a bigger issue that We needed to sort out That took us to Next step One of the big changes that happened In this was My role shifted from a developer To a full time ops guy So now I could actually look at the system Then manage the system properly And we needed to do that because We started having a higher Volume of Usage in our systems We had more users We actually had more systems There So We needed somebody who would do The ops work full time And we did hire a couple Of other developers to manage things So this brought In Sure Besides that we started experimenting With Things like AWS cloud because Even though Rackspace is pretty solid The costs were already escalating Even with we had about 8 to 10 Machines and with that the costs Of machines were really really high As compared to something like AWS And we were Looking at things like having Infrastructure which can be Built up whenever required and But you just have it running while You still want like a dev environment Doesn't need to be always running When you are doing certain parts During this phase We started looking at What are the things that we can Actually Start doing Sorry It's moved around So here is where we started Being Having a little more complex We started getting a little more complexity In our environments So Also the frequency of the releases now Increased now since we had More developers and we did Have things working and Moving and Also with new frequent Requirements coming in we were Actually Doing a few releases Every week and Or maybe even 2-3 days We were still nowhere close to having Something like a configuration or Being able to quickly move So much bigger difference from Phase 1 where we were just Pushing Things together and having a big Release at the end of 1-2 months It was much better place where we Were maybe 2-3 days To a week Then we Started working on having project management Tools to tell us What is happening, what is release What is to be released, the tracking Of the features, tracking of when Releases happen, we introduced Jira, Confluence, Bamboo, Slack chat And Slack Which was for These things We did try to introduce CICD pipeline at this point But miserably failed With bamboo We didn't really have anybody With expertise in that And we didn't know really Where to go with things We tried to just connect Tencible with bamboo to do all the stuff And it didn't work at that point So Yeah, one of the Even though the builds were there They were not getting consistent Builds across the environment With the multi, like now that We had about 10-12 machines There was something in production Something in beta Something in dev, the consistency Tracking of the deployments Across the environment is what CIE tools should have sorted out By doing the CD part correctly Which we didn't get right at this point So that was the biggest thing that we had And we were still manually Managing the infrastructure Just the configuration is happening But the base system was still manually If you had to install a new package Or update the package Probably updates would happen with Ansible But setting up the base system Was still manual So with the next release The focus was on improving the code quality And we hired a person To do the Q&A And the same with the Q&A We also Realized the person Who was handling the Q&A Is also doing good with Jenkins So we moved to Jenkins During this release And That was one of the big things That came into a picture With Jenkins being there And some kind of Q&A tests being there He wrote some integration tests At this point But there was still a big issue That now with frequent releases And A lot of work happening And more developers We were running majorly into a lot of Git conflicts So I did also Have a complete workshop With all my developers And we went through the whole Understanding of the Git And sorting out the Git conflicts Locally But we were still facing certain kind of Issues while dealing with the Deployed on the server environment So started working on Ansible for doing Dynamic And destroying the infrastructure dynamically We were looking At AWS at that point And AWS has Very good support within Ansible Where you can bring up the infrastructure Configure it and have it up and running So I started playing around with That So since we replaced bamboo With Jenkins during that period That was one major change And we started focusing on Getting that part of the setup right So we were doing the Business test having all the builds Being done in bamboo And building the deployment pipeline And the same Reuse those builds which have been done In each In the dev stage We can do that and reuse it For our beta And if that works Then push it to the production Around the same time In mid 2016 Where we started working on New As I mentioned we had Three products now And we started the work on the second product By then so we started Applying this from day one on the new project And that did bring us A lot more consistency And we realized where we Were going wrong with the first product And also since In the first product we started Applying this things much later into the thing Building the things on top of What is already existing Which is a pretty humongous infrastructure And also has a lot of things Which we don't know right Even pretty recently We didn't know some of the things What has gone in so there were inconsistencies And unless we were in a place Where we could do everything from the start This things would still remain So we Ensured that we have Full automation and full control Of the systems Of the builds Of the deployments Everything right from day one In the new project So with these More projects coming in And more developers being there And bringing in more automation Also brought in Increase in the cost now Because we used to have more and more machines And with these more machines Costs started just escalating big time So management Had a push that You guys really need to Start working on how do you reduce The cost for this So Also Building time for the infrastructure was We brought it down from Days to like we could probably By then start doing it In a day probably 8 to 10 hours Which was a major achievement For us to get from somewhere With 2 to 3 weeks to down to 8 to 10 days 8 to 10 hours sorry But again one of the things Was I was trying to do too much with Ansible I tried to even Start new machines Bring them start the Bring the load balancer set everything Up together started writing the code To manage the infrastructure within Ansible And I realized that Somewhere I am not doing This right I am creating Something way more complex than Is manageable and I am taking Too much time And Maybe luckily or some Unluckily I don't know but the biggest hurdle Happened when we decided we are going With Google cloud and not AWS And Ansible uses Something called lip cloud To Talk to Google cloud They are working on With the native apis api calls But that was not there I I am not sure if it is still done or not But when we were working on it It was not there and we couldn't have L7 load balancer we wanted L7 load balancer ahead of Our machines to load balance All the STDP All Traffic that was coming into our machines And we couldn't do that with Ansible So I started looking Out for solutions I reached out to some My sister had been friends And One of the That is where we Moved to our next level of Automation that came in So The scenario is we Have three projects by now We were almost close to Having 20 to 30 Machines it should be 20 not 10 But we were 20 to 30 machines There and We didn't need all these machines running all the time So We started working on how to Cut down the cost and how to manage the infrastructure Such a bigger infrastructure Compared to the earlier things How do we manage this So I was Told by some of the friends To bring in Packer and Terraform With Packer I Start doing my builds and Having image Ready with all my base Things built into it And with Terraform It will just take the Packer image And have virtual And your whole infrastructure will It will start a machine and also With Terraform I could Actually define my Infrastructure where the load balancer was there My networking was there My machines were there My database was there In a much much simpler way I didn't have to write a logic on What to start first, what to start later Terraform Just understands how do you want to What are the kind of dependencies that come in When It's also more Declarative code Which is much simple I realized that it's much much simpler to manage And I was I didn't have to write a logic on How to do things To define what do I want And Terraform would give that to me Then We still keep Ansible We just reduced the use of Ansible to do the We are still doing deploys Using Ansible and we are doing Configuration using Ansible And it's working brilliantly And what we have done is we have Tied all the three things together And created some Task within Jenkins So now I don't need to do Like they don't require My involvement, everything has been Fed into Jenkins Only there are certain kind of Parameters that My release managers understand And they will just define these parameters Whenever they need to do a release Even our dev can do that They just need to specify what is the release number What is the environment And the release can be done right from Jenkins environment For every release, only when things fail Then I need to look into it That has allowed us To reach a stage where We can actually do multiple Code deploys a day We are doing that in the dev environment And in the basically In the non-production environment We are not yet at a phase Where we are so confident Of how the builds we are trying to Work on that but hopefully In next couple of releases We will be there where we are doing Multiple push to production in a day With This whole thing We can actually start off a new If I want to build a new Infrastructure, I can actually do it In some 15 to 20 minutes Now I have the Code written in Packer It can It creates a machine image And stores it already in the Google cloud images And Terraform Can just take that image And start building the machine Probably it will be done in another 5 to 10 minutes And Ansible just kicks in And configures all the things Once the infrastructure is ready This has allowed us to have the complete Devop cycle that you really need Where the dev Creates the code It goes to our CI system Does the build The build is pushed into The system Which can be used Then brought up in minutes And there are also Once the new infrastructure is brought up And a deploy has been Done there The tests run on that And give us the immediate feedback And We can have this Completely tracked into a Couple of central places like Jenkins Has a very good amount of information Which is already there We are pushing things to Slack And the next step is to push it into a monitoring system Which gives us a complete dashboard Of what has happened At what point Of course this is today But even today we do have a few more problems That we will be sorting out In near future One of the things is that We have Code and data on the same machine That certain places That means that we cannot just bring down The machines at this point And have a new machine brought up Unless we handle the data correctly So we are working on Bringing in that separation And so The data machines can be separate And they will be having backups And All the things handled On the data separately And the code machines would be much much It will become very trivial to scale up And down Just destroy and bring in a new machine Whenever you want Whenever we are doing a new release We can just bring up a completely new machine And whenever we are ready with the new machine Bring down the old machine We want to get into that kind of a phase So one of the factors Which is stopping us from doing that Is this part So Another thing is I have written a lot of code But it was written in a place Where we were still figuring things out So one of the things I would Really love to do is Rewrite my code such that I can If I have to start a new project at any point I can just reuse the existing code And be able to Do this with Very less work So that is still remaining In bringing into our automation So idea is to Be at a stage where If any If any eventuality occurs We are ready to be prepared for that Like having Being there in multiple regions In today the cloud providers Provide you multiple regions We are not using that We are still in one region And if that region has a failure My code is not ready to Immediately be able to switch to a new region Our data is there That data needs to be backed up That can bring it out from a new region And use it there So some of these things still remain And stopping us from having A very reliable infrastructure And having a fully automated system Which can really work without Me having to look after it all the time Yeah So that pretty much concludes What I have In my presentation Anybody would like to Ask any questions here Yeah Yeah So my question is Currently the scenario inside My company We have migrated into monolithic Into microservice And we are doing deployment By Jenkins But the problem is Currently on a daily basis We are having 5 millions of ad requests Which gets double On weekends Right So now we are on a stage We are thinking to have auto scaling But in Jenkins We need to configure All the servers Manually So is it any plugin or a way Where something like auto discovery Can happen in Jenkins When we scale up We can deploy the code on the scaled up VMs So this is something that I am The next release is The deploys don't happen to the systems The deploys will happen To my packer build So my packer will build and keep Everything The base ready So when Jenkins has a new build That build will get baked into my Image which is there So using that image I can start the new machine So auto scaling I have not used auto scaling yet But with the autos I am guessing that you can specify Which is the version of the image that you want to use So say if you want to go from one Machines to two machines You will start one machine with the new image Once you are sure it is working Then you can basically Say that take the old machine out And bring Start one more machine with the new code So you can So the whole thing is baked Into your You don't need the Jenkins To know which are the machines Jenkins deploys the code to your Image instead of the new machines And what I do is after that There is of course configuration required So my configuration which are the Machines to be picked up is I use dynamic inventory in Ansible To handle that So Jenkins doesn't need to know again It's Ansible which knows which are the Machines which are relevant to this Basically you need a Management tool so Ansible is one Probably if you are in AWS Then you can use their own tools Or if you are outside and Depending on the size of your company And your requirements you can figure out What are the tools relevant to you Ansible is what we use You can use any configuration management tool That is relevant to you If you like to speak or Ansible I am always happy to do that Probably there are some red head guys around I am not sure if they have anyone from the Ansible team Okay the next question is from here Hey Hi I have a question On the tool whatever you talked about So consider that you have a Cluster already created using Your system So tomorrow I want to upgrade The same cluster so how Is it going to be With zero downtime Sorry what is this Cluster of what The application You spin up instances So we have Node.js Let's say we create a cluster of Node.js Machines which we have not really Experimented with it but what I can think of Is that We would go is Instead of Sending all the requests We will create one machine With the new version of the code And only push Have our Load balancer only push a certain amount Of data sorry certain amount Of traffic to that it can probably Be a certain user that We can Specify or it can be certain of Rules and once we are sure of it Will bring up a completely new cluster Which is Which is all the new The new release let's say we want A cluster of we We want release will have one machine Of v2 release Once we are sure that it's working correctly We'll start sending more like We'll build a whole cluster of v2 release And send all the Data to that now and get the v1 release now So we'll take down the whole machine We are planning to basically get to what is Known as a immutable infrastructure So there you don't once you have The machine up and running you Don't modify anything on that you Just use that machine as it is And whenever you want to do something New you bring up a new machine and Once it's working take down the whole machine Sorry can you just wait for the mic Yeah how do you handle Capacity in this case It is something So you have a constraint on capacity So how do you Capacity would still not be affected Right because you're Like say you were just Sending your to like Essentially you will only send a little bit of traffic Let's say 2-5% of traffic to the New uses so if there's a problem With it it won't everything is Still going through the old version You need to ensure that there is some kind of Compatibility or We are looking at things like Feature toggles or The things to build that kind of Compatibility the new release where You can have both the version of the code Living side by side on different Machines of course or it can be different Containers if you prefer that But basically we have Two versions of the code running side by side So still majority Of the traffic say 95 to 98% Is still going through the old version Till the time the new release is up and Running completely so you like say If you had a cluster of five machines On the old release You started one machine on the new release And once you are sure Of everything is working on the new release You build that cluster of five machines On the new release and take down the old cluster Because you are sure That everything is working so you Start directing the old traffic to the new cluster And once you are sure of that You can take down the old cluster That's one of the reasons to be in the Cloud and building the dynamic infrastructure Yes for a short period you do Have a much higher Users you are using 10 machines Instead of five machines for a short Period but that's a tradeoff That we are willing to do For having a better uptime And better reliability Wouldn't it be better if we Do this on a strategy If you have a strategy for this Instead of spinning Completely a replica of the existing Cluster and just wire it off And wire it on to the new Version of the cluster Will it make more sense Sorry can you Instead of so you said You will test on one box on the New version right So instead of you know to continue On the same logic So why don't you just go and Do it on a batch basis Instead of doing it at once If you have a thousand nodes you are going to create another New version Depends on the size I am probably thinking in terms of A much smaller size where we are right now But if you are much higher Like you said maybe in thousands Maybe in hundreds You will probably start with Like if you are 100 on v1 You start a cluster of Phi on v2 And then start Make it into a cluster of 10 And release 5 of the old machines So slowly start You can do it in batches Where you start Once you are sure of ok It is killing better with 10 So let's take down 10 of v1 And bring up 10 of v2 So you have 20 of v2 And 80 machines of v1 So you still have 100 machines But 20 percent of them Are on v2 and 80 percent of them Are on v1 So you can do that kind of a strategy Of course your strategy would be determined On basis of your use case And the constraints that are there What you have so I have not covered everything probably There will still be something that is more relevant to you And you are in a much better place To determine what is the correct thing for you Ok everybody We have plenty of time for questions We are running a little early But a few announcements first If you want to ask questions stay back But there is an OTR session Of the record session on disaster stories happening It is hosted by Trouble and Todd Highly recommended And it started at 3.45 It goes on till 4.45 We will reconvene here at 4.40 So that lines up nicely And those who want to ask questions Can stay back Question I have a question here Over here Just wait for the mic or else we can speak Hello I am speaking So you said at some point In time you are running two versions Of the application at the same time And I assume that this application Is connecting to the same database Because you need to show the data Change the database schema for the new version Of the application So yes this is something that we are factored in So basically what we are working towards Is trying to At least so far we have not run into any issue And we are not Completely doing this But what we are trying to work on Is that where we run into issues Are like when the column We are using MySQL So in MySQL if the column is deleted Then the new version is trying to find that column So instead of that we are Refactoring our schema In such a way that We don't delete the column anymore We mark the column as unavailable In the new release So we have something called a data model Which is the code which interfaces with your database So in the data model we mark That this thing is unavailable It is not to be used anymore So yes In a way we are kind of building a bigger database Than we should Migrate it completely to the new version That column will remain And later we will take a snapshot of this And then phase out the column Can you explain The usage of Packer and Terraform You are using Packer to So Packer is a tool Which kind of Defines the rule or it will take The basic image like we have We use Debian so it takes the Top level Debian image What has been released And we specify what we will go into That image to create our own custom Image Do we have The next speaker is Jim Perrin He is the Centaurs Arm Lead He manages the Centaurs CI team I think When KB is interfering And He is going to talk about Creating IoT devices with Centaurs Since before the Y2K So if you are young enough Old enough to remember that I will let him start off You are a terrible person Thank you So most of the talk That seems really loud to me Is that loud to you guys? Yeah don't do that So most of the talks we have heard today Have been talking about why Containers are great How we are using containers for new technology Or improving things for the server Side and OpenShift makes it all fantastic And Vashik will absolutely Stand up here and say OpenShift Will make it all fantastic See But we can also abuse this sort of technology We can take it and twist it In ways where it wasn't quite intended To go in that fashion But we are going to do it anyhow And it actually works out pretty well So how do I know this Because I Have been assist admin since around 1999 and thank you Nigel For making me feel old on stage Fantastic And I've been a member of the Sentos project since 2004 I decided that I wanted to dabble In a little bit of evil in my career And so I started out I moved from Education consulting into oil Consulting A little bit of Other things in there as well And about two years ago I started Getting involved in the 64-bit ARM effort On the server side Development for ARM workstations Or ARM workstations ARM server John Masters actually Was the person who got me involved In that And I've been doing that now For the last two years Give or take And there have been a lot of lessons The ARM folks to stop thinking Embedded and start thinking server And the more we played Around with that idea the more we realized It actually goes the other way as well You can take a lot of the server mentality And transfer that into the embedded space And it does make a lot of sense So the first thing we have to kind of look at Here is the idealistic Approach for IOT And for how Linux Distributions are actually done And one of the ways that we do that Is look at What drives IOT Most of the folks here are either Ops or engineers and that's great That means that we want to play with this technology We want to tinker around with it We want to build and develop it But we have to come up with a way that's convincing And pitch this to management and say Hey here's why you should let me do this In my spare time, here's why it's a good thing And really it comes down to money Because a lot of the money that's listed In IOT over the next 5 or 10 years Gets pretty ridiculous When you start talking about the amounts The low estimate For 2020 Was given at $470 billion That's a pretty Significant chunk of change Now that does encompass everything IOT related so that You're talking about the low end RFID tags all the way through gateways Through Amazon with Alexa Through Google with whatever the hell it is Google's plugging in with Nest A lot of various things like that In addition to all of the individual devices You've also got All of the data that comes out of that Somebody has to process that data Somebody has to look at collecting it Figuring out if you're going to track The actual data real time If you're going to look at trending data If the numbers matter or if all you care about Are we going up, are we going down Is it cyclical, that sort of stuff But these are basically The drivers for why people Want to get into the IOT market The other side of this Is in addition to Abusing the Data analytics side of it You actually get a chance to look and see How quickly you can respond To something with your business And if you are Using triggers based on the Data then you don't have To make a lot of the low end decisions Anymore you can sit there and say When I hit this amount of Widgets in my company Order more, you don't have to have A person who sits there and tracks that Inventory and says okay we're getting close I'll go ahead and put that order in You can just program this automatically And essentially start doing CI on your Business as well as On your code Everybody stands up and says IOT is horribly secure For the most part they're right And this is one of the places Where I think that A Linux distribution Be it fedora, be it rel, be it sentos Actually makes a lot of sense In the IOT landscape Because right now you've got companies like Samsung doing all of their Well I won't just single out Samsung But all of the IOT companies Are doing their own stack Their own Linux deployment, their own software Deployment and these are primarily hardware companies Lots of the hardware vendors Don't necessarily know the right way To deal with software Who does? Ostensibly software developers So I mean IOT is Kind Hey it doesn't like that slide Okay there should be Yeah alright It won't do that That's interesting We don't need Typical IOT type stuff It's skipping a thing So Basically what happens When you have companies When you have IOT companies Who are developing their software in a silo They'll oftentimes Especially for hardware companies Skip putting out updates Because once you've bought their product You're a customer Everything after that you become a cost center And this is one of the places Where having a distribution Model actually helps To worry about each of the individual CVEs Or each of the individual Security problems that come With building a Linux distribution Up from scratch You can farm a lot of that work out Onto a stable Base platform like Fedora Like Sentos like RHEL And then just focus on what you want That piece of hardware to do So if you're making a camera You can just focus on The user interface for the camera And then let the Sentos team Or let the RHEL team Or let the Fedora team worry about That pesky open SSL update That just came out or dealing with bash Or any of the other things And everybody looks at IOT And says they're talking about small scale devices We're talking about chips The size of your fingernail Well, that's not We'll see if it comes up with a slide That I want it to We're talking about the servers The ones on the right The one on the bottom is a legitimate product You actually can go get that They make bluetooth heated Insules for your shoes Why? I don't know But people buy it The equipment on the right is An HP Enterprise IOT Gateway It is no different than Your typical x8664 server The fundamental difference For this piece of hardware As an IOT platform versus anything else Is the way in which The software is delivered These gateways are done as a Firmware style operating system So there isn't necessarily The login That you would expect You're not going to sit there and run YUM update Or DNF update or anything like that It is a one shot software piece It's taking a generic CPU style system And making it task specific And this is pretty much When I say I want to put CentOS on these systems I'm talking about the industrial gateways I'm talking about the cash Registers that handle your point of sale transactions When you go into a store These are all The IOT style Servers that I want to see CentOS running on I want to see God help me Fedora running on One of the Last week at Red Hat Summit One of the use cases that we had brought to our attention Was an MRI machine And that one caught me off guard I wasn't expecting to see medical devices As something talked about for IOT And the reason that they were brought up As an IOT style device Is these are One shot installed systems That ostensibly move around Throughout a lot of countries We don't necessarily have the ability To stop in any one place And get updates The people who brought this up to us Were part of the Doctors Without Borders team And they were moving an MRI machine Around Afghanistan For some of the mobile hospital type work They're not going to be able to plug in a system And do a YUM update These are high end medical devices That have to be certified At each step of the way You're doing An IOT style update Or the RPM OS tree Who's familiar with RPM OS tree Let me ask that question before I start About half of you RPM OS tree Is a very good way To let you generate The tree of updates That's going to be pushed out To every single system One time So you don't have to log into each individual server And you do the batch transaction Once and you push that out Primarily we've been using that As a container platform Because it makes a really nice way To drop containers on top of that It also works in the IOT space For exactly this Because I can have a golden image That gets pushed out And most people should be familiar With the concept of golden image So You generate the image And you get that image certified For a medical device And when it's time to push an update For that, you go through that same process Again, it's a read-only install That doesn't let you move From one to the other ostensibly And so for MRIs It actually made a lot of sense And the guy who brought this to our attention Really wanted to see this done So this is something That I think we could actually push Along with automotive Management systems Most of you aren't going to connect your car On your home wireless network Tying it in through a bluetooth Cell phone or 3G connection 4G connection via cellulers Pretty much the way that's going to happen And the last option is basically the home gateways The Amazon Alexa style devices That act as Brokers between what you're doing And the cloud as a whole If you have a bunch of light bulbs In a house that are IoT connected You don't necessarily want to have to talk to a server In China to tell your lights to turn on In your house. If your network goes out You can't turn on the lights, that's kind of a problem So a home gateway Acting as a broker for that makes a lot of sense And a distribution That has a Method for tracking And updating CVEs Or other security vulnerabilities almost immediately Helps a lot in that plan And I answered my question Before I got to that slide So The reason that we want to use The enterprise style operating Systems or the Long lived operating system is basically Nobody's going to replace the refrigerator Every two years Simply because somebody stopped Producing a software update For the in-dash panel on the refrigerator Or The dryer has an application So that you can be notified When your laundry is done When they stop pushing the updates to that You lose some of the functionality that The hardware manufacturer Is using as a selling point for it Having a distribution That can continue that 10-year Life cycle makes it easier For these appliances To continue to be useful And productive despite what The manufacturers think At that point you now know that It's a forcing Function of the business rather than We don't have the time to put into first software On this So it means that it's easier for the user To upgrade to do things right And a lot of it comes down To it's the exact same tool chain That people are already used to using A lot of businesses are already certified On rel or on sentos Or on fedora They already have that workflow down They already understand how that operates And this allows us to continue that Into the lower end for the The lower end chips as well And into the IOT space It's the exact same process Things behave the same way It gives everybody that consistent platform That they understand already Does anybody have any questions so far? Does anybody think these ideas are Completely and totally crackpot yet? I have no problem heckling The audience feel free to give it back So if you have a question stop me and ask I have no problem with that Basically this is what we get out of it The vendors can focus on making the hardware Do what they want the hardware to do They don't have to care about the operating system So we don't need to have 18 vendors Coming up with 19 different ways To produce an operating system We can give them the platform And they can focus on making the device Do what they want Within the sentos project We've got a few vendors Who are already trying to talk to us To get us to do things like this I single out Samsung They're fresh in my head for IoT They actually have a few devices That are using I want to say it was built on Fedora 21 That was their embedded model For this And they came to talk to us At Red Hat Summit Asking for ways in which they could Get updates to the system for this So they didn't have to constantly refresh All the time And the RPMOS They actually seemed to work For what they were doing Because this was basically a DIY TV chip It was an embedded chip That they were using in a lot of their different technology And they wanted the customer base To be able to play around with it This gives everyone a way to do that So if it's running a standard distribution You get to play around with it You get to do what you want with it You can try and break it if you want to And contribute features back And again, the atomic updates Are basically similar to The typical firmware style So it makes things very easy To just push the update See if it breaks If it does, it rolls back automatically You get stuck in some of that Who here is in ops Or has been in ops In an RPM distribution in the last 10 years A few of you Anybody ever had a System lock up midway through a YUM update And have to roll back through all the transaction tables And everything I know you've had to do that a few times The RPMOS tree style Prevents a lot of that headache So you don't have to go through this And I keep harping on that point But it's one of the biggest selling points of it And it actually That aspect seems to make the most Sense to me So how does this actually work One of the Current IOT Workflow All pull comes from the automotive industry A large chunk of the automotive Industry is either standardized on QNX or Yachto Linux And I Don't want to take away from the Yachto folks too much It's a great project But it's really cumbersome to work with And we've had a few folks come up to us To complain about Their workflow process with Yachto And what they describe looks something like this They do a get clone of the Yachto code They do a base build of Yachto They start to add in The changes that they need They put their application code Inside that same directory They cycle their builds And their configuration files They copy it off, they test it And they iterate through that And every single time they do They get a fresh build of Yachto Which means they can't go back Because they're already into the next tree By the time somebody files a bug They don't necessarily Have a sane ability To roll back to a previous Get snapshot, see what was there See what was built, test against An issue that somebody has filed And then produce a patch They have to send that person an entirely brand Or send that vehicle an entirely brand new build Of Yachto And of the vehicle software To see if that actually works So That workflow is problematic For a number of reasons If you're already well through Your software development cycle You have to roll back a number of things Based on one particular issue If you're using Fedora Or Rel or Sentos Reproducing a build And taking a known working Set of RPMs Makes things very reliable And you can take a snapshot Of somebody who has filed a bug Or filed an issue And go back, duplicate That exact environment, see what it is And then figure out what needs to happen From there, it's a lot easier to do That in Fedora or Sentos Than it is to do within Yachto So On the Sentos side It's basically package Your application As an RPM Or a docker Containers, do we have to call it Moby Containers If we're talking about community work Okay, so as long as I use the lower case D Then we're good Okay, I'll include the trademark Then later in the slide Or change it to a lower case D You expose the package To a repository Either a container registry Or a yum repository You install it And you deploy That's pretty much it The only thing that you need to do in between Is validate that it works the way you want it to That's it So you add a CI pipeline into this Or Gotta help you tie it into OpenShift And everything is fantastic The other thing that you can do Just deploy it Or just publish it as an RPM Expose it to a yum repo And build a layer on top Of RPMOS tree You can actually If you're not familiar with RPMOS tree You can unlock The Tree Install an additional application Close the tree up And then push that layer as Just the layer You can add it in as a second tree And push that That workflow is still a little problematic But it does actually work And we're talking with a developer To kind of Move the workflow The reason that we push that Is if you do it as an individual container You have one container running on One device, one application That seems like kind of overkill To involve a container at that time If you're running multiple Applications If it's an in-dash card player You're probably going to want A thing to play music A thing to answer phone calls A thing to pull contacts in If you want Containers in that space make sense But if you're talking about a camera Your camera is going to take pictures Or video and that's pretty much All it's going to do There isn't necessarily a need to have Multiple things layered on top of it The other thing that this gets you In this particular workflow Is the ability to add in Anything else that's already packaged up So if you want to start Looking at sharing out storage In the home gateway and you've got Six different devices You want to be able to share things between them You can put something like Gluster In persistent container storage And share that out so you can actually Have seven or eight things that all Accessing the same point And I'll single out some of the Gluster folks In the room just because Nigel was nice Enough to throw out that introduction I'll see about making this part of his Task later on to do CI for this I told you My revenge would be swift The big thing out of this And the thing that I keep coming back to Is making sure that everything is consistent The ability to have A workflow Where you can reproduce An environment where somebody files A trouble ticket Is kind of key The ability to Test against things and Reliably roll back and then Create new Tweaks is really important But this also leaves out the smaller Devices, the ones that we saw on Those, the tiny ones We've talked about the big stuff We've seen the The rebranded servers That now count as IOT But the tiny stuff We've still left on the side So we need a way to deal with these Otherwise we're abandoning An entire chunk of IOT So how do we deal with the small ones The Zephyr operating system Is actually really nice On the One of the IOT working groups Red Hat has joined And is part of the Light, trying to remember what light stands for It's the Leonardo IOT Technology Gateway or something like that I can't remember what it is But Red Hat is involved in this And one of the demo platforms That the light working group For IOT is using Is the Zephyr operating system Zephyr is a real-time operating system So it's much It's more tailored to Real-time collecting of information A lot of the lower sensors need to have Where the smaller chips are Sensor designed So that you can pull in and collect information For temperature things The railroad industry is actually looking At a large portion of Because real-time actually matters for trains Who would have thought Finding out If the signals Are working right, if the gate comes down on time As the train is passing through Real-time matters in that instance So The Zephyr operating system Is Tiny enough that it can be built For a lot of these Low-end devices that will never get fedora on That will never get sentos on That just has no hope of running on But Zephyr is kind of purpose built So You can work through The Zephyr builds And duplicate this in QEMU So the QEMU environment Allows you to do the build on sentos Natively On fedora as well Run it Through the virtual environment to test And make sure that you've got all the functionality That you want and then deploy the build So you essentially add it into Your current CI chain You can do the build and test Right in the same operating system Right in the same pipeline that you're already Used to using and then crank out The small chip set If you don't want to do the native Compilation, fedora actually provides All the cross-compile tools So you can build on your x8664 laptop For arm64 For a lot of the other chips That they provide the tooling for So essentially you never Have to leave your current environment Even to deal with the stuff that you can't Necessarily run sentos or fedora on The point of all this Comes back to keeping it simple You want to have the It was said earlier today Developers are lazy And I know this because I am one I will say that again And I will steal from that guy Because I'm a developer I'm lazy, I'll use his words And it's kind of the same concept It's basically use the core that you already know Sentos is already there Fedora is already there Rel is already there You just need to apply it slightly differently It gives the developers a consistent base So they don't have to learn about Debugging Alpine and debugging Sentos and debugging Fedora and debugging Debian it's all the same Platform it's all the same libraries So everything works the same way That they're already used to And that gets us almost to the point A little bit ahead of time I was told 15 minutes for Q&A We're at 17 So questions? Anybody, anything? Yes, one sec He's going to bring you the mic And if we can get somebody on the opposite side Of the room to ask a question next It'll be great to see him run back and forth Have you done Linux from scratch There's a project out there where you just Take the source code and try to build it from scratch So What I had to do was Try to do something like Sentos from scratch Because if you are building Something like to cross compile The whole Sentos for Another target It's kind of Difficult Because there's no standard instructions Of how to build a Sentos Like build one Not just install Sentos and run it But build a Sentos So if you Truly want to do this Sentos on IoT Some instructions Along those lines will really help the Adoption And second thing Is there a definition Of what is minimally A Sentos Like should it just be the kernel DNF and SLNX Or should it be something else So Is there a definition for that That definition will really help Like You bring up two very good things The first is directions For building A version of Sentos And that changes So it's more difficult to provide Than you would think Initially when you look at what it takes To build Sentos You assume it's just a bunch of RPMs I'll cycle this through And that's not necessarily the case For each series of builds Things get kind of tricky You have to walk The build platform back and forth Even sometimes for little things Like libnss The nss library Has several checksums in it That break They're basically code time bombs MariaDB and MySQL do the same thing Because they validate SSL certificates And one of the nss Tests actually goes out To PayPal Grabs the PayPal certificate Pulls it in and then tries to validate it And if PayPal has updated their SSL certificate Like they do fairly frequently Because they do things right That test fails So It becomes Problematic for Okay, follow these directions To build a distribution But know that you're going to have time Bombs in the code when you try to build These 12 packages so you have to do This step, this step, this step Because every time that package gets updated Something new is in there For Golang Golang up to 1415 You could build that against GCC No problem there Now, Golang requires Golang To build So if we push those instructions And we say, okay, here's how to do it The problem becomes maintaining those Instructions every single time And we spend more of our time Focusing on building the instructions Rather than building the distribution It's not that the tooling isn't out there Or that it's not discussed We're not the only ones out there So scientific Linux is out there There's another one from Stanford University I want to say it was called Springdale I think they've renamed it to something different Oracle does it There are a bunch of different ways The other side of this, when you come down To what's A minimal distribution Or minimal version of CentOS That gets tricky as well You're bringing up, do you need a kernel For example For a bare metal install Absolutely, yes And we won't support it if it's not our kernel If you're talking a container You don't need a kernel in a container Because you're using the host kernel So Which one is CentOS, which one isn't They both meet the definition We ship them both So that gets tricky as well I would say Being self-serving And standing up on stage talking about this I don't necessarily Want you building your own version Of it and putting your own tweaks into it What I want is you contributing to the stuff That we're already doing I would want to expand the community out And say, hey, we've already got CentOS here Come help us do what we're doing And tailor it to fit your needs Or add an additional Package into an additional repository Instead of building it again from scratch Building it again from scratch Is pretty much what got us in the position Where IoT is right now Because Samsung has done it one way The folks from Dell have done it a different way With the Edge X Application There are a number of different examples For IoT products Where they've done it their own individual way That they felt met their needs at the time And it's gone horribly wrong for them The idea is this So generally If you look at other operating systems I mean the only In one of Your slides earlier you said Developers are lazy and the job of An operating system or a distribution Is to give some sort of Consistency to a developer So when he comes and Sees, hey What are you running on this piece of hardware And If I say CentOS Is running on this piece of hardware I know exactly What to expect from there So I agree with Whatever points you have mentioned But any definition Which you can give like minimally Like for example what First thing that comes to my mind Is CentOS has a silenax Like while Debian based Or Ubuntu they will not have that So I can expect So if I am going to Distribute Some application which is going to run On that specific piece of firmware And it's not necessary that you Are always going to be supporting that Piece of firmware Like there's some random chip out there Which I have got for Low cost and I have This decided to Use it so Can I build some firmware which is Which kind of Ascribes to the definition of CentOS Even if it's not like certified By CentOS or Whatever is the authority So how do we define Like This is like CentOS Like the most minimal basic requirement Not like list of packages These packages should be there in CentOS But more like hey it has the Linux kernel it has a silenax Like some definition would Really go a long way I mean it could be a starting point The container Argument is the reason that we Haven't put out a A firm definition of that yet Because if we require A kernel then even The official CentOS docker container Doesn't meet that definition So if we're just calling it a kernel Can't do it it won't work If we say it has to Have silenax we'll open Susie has silenax as well So that doesn't work if we say The way that the Distribution as a whole has said That we will deal with this this is to say If you are running code That we have signed with the distribution Key that is code that we will support If you have if you're running Your own kernel that's great You're completely welcome to do that As soon as we hit something where The kernel is Singled out as causing a problem You're on your own at that point As far as trademark goes If you want to do something that ascribes The CentOS the trademark guidelines For the distribution allow you to do that So you can sit there and say hey this is based On CentOS or this is Powered by CentOS but I have added X on top of it So there are guidelines on the website For doing that you absolutely can Do that as long as You're following the trademark guidelines And the tooling around it As far as the For the rest of it I want people contributing To the project rather than Trying to copy the project And duplicating it It doesn't seem like there's a lot of effort In building a Linux distribution It's not necessarily the Compilation that requires The work Although there's a ton in it It's the long term maintenance of it It's the ability to continue supporting it For that amount of time A lot of the value is actually there And that's one of the things that gets Pushed in the IOT landscape Because the updates for IOT just They're not there in most cases They're hardware vendors They're selling the hardware A lot of the Developer boards and a lot of the Even the android cell phones That you look at today Most of them are running a 310 or 318 Kernel Even though there are known problems An android itself has moved on Within the google aspect To a 4x based kernel They're just not being updated So that's one of the things that gets triggered It's that long tail of maintenance That's actually the Biggest part of the value But saying that Yes, it's maintenance that makes this Thing function Maintenance isn't new It's not hot, it's not the sexy thing That pulls you into a product So it's, you know, people get sold On the features, they get sold on the The upfront UI, the interface The flashy, the support Comes afterwards about a year into it When they start saying, hey, I need an update for this Because there's a problem Any other questions? Yes Hang on, the microphone So I'm quite interested Interested with the GPIO stuff Which I didn't find with Fedora So Dirt Center as some What's the GPIO On the 32 bit Side we have some limited GPIO support It's not Fully baked in the way that You would likely want And we don't necessarily have the hot plug For certain devices for Adding and removing While the device is powered up There are a couple of kernel tweaks in that That need to be done that aren't there On the 64 bit side We don't have that at all currently Because most of the servers That we've been aiming for don't have GPIO support So there hasn't been a need to do that In the future We probably will turn that on For some of the smaller scale devices I know that On the Fedora side they were working to enable Some of it, but Because Fedora builds Specific functions into the kernel A single kernel built for all the devices Just per architecture Getting some of those features Turned on causes issues For other architectures And that causes the whole Build chain to fail So there are some reasons why Fedora Doesn't have it just blanket enabled That are a little different You're not making him walk that far Sorry I could just walk over there So if you are finished With the answer So my question is CentOS is building Atomic images Based on OS3 But it's container oriented It has Docker inside by default Do you build or do you plan to build Other Atomic Not Atomic but OS3 based Images Or trees That would be as minimal as possible With devices without Docker Because maybe people don't want Docker there They want to just use OS3 install or something like that Eventually We have to have the workflow Around that Before we can start producing those images We have to have The developer environment and the workflow For people to consume that So we have to get that built first And then say this is the tree The easy way Is to say Here's how you build your own tree From this list of signed CentOS packages From this repo Here's how you can do it yourself That seems to be the first step to me Whether or not we say Here's a base tree This is what you need to use As your initial platform and work up We'll see where it goes from there I think we're good Thank you everybody You can pick up your feedback Forms and drop it at the bin Outside Not the dustbin but there's a bin next to the IFF counter So I'd like to thank A lot of the people who have done the work To make today successful Especially Hasgeek For hosting us along with Rootconf For the people who are running the logistics here So the hall monitors The people who have been passing around the mic The people at the cameras Who haven't been able to probably Move around a lot The people who are handling the bikes And Beavis has been stuck at The Video thing And that's going to end up on YouTube And it's a very thankless job for now And eventually we'll be grateful to him Thank you for doing it And a big thank you For all the speakers today And all of you who showed up Today and yesterday for the workshops And I think this is the end of DevCon


One of the important conditions that we have in marriage from the Islamic point of view Islam emphasizes that when you marry someone You know one of the things that you had to see whether they are kufu of one another or not and kufu means You had to see whether they are compatible to one another or not and one of the most important dimensions of that Compatibility is the issue of iman and faith You know If you look around there are issues when you have people two people from two different backgrounds As far as faith or even mazhab is concerned They are issues Initially things are very good Or we love one another we can handle whatever comes But life is not like that way Especially when your children come in then they are disputes Do you go to for Shabaya Shab-e-Qadir here or do you go for tarabi over there? You know many many issues come up and this is where the issue of iman comes in for us one of the best example is the example of Fatima Tuzahara and Ali bin Abi Talib, alaihi salatu wa salam I think the the true love and relationship is Tested at the time of difficulties When things are okay, everything is okay But that's not the actual time when you see the worth of a person You know, this is a very interesting story. You might have heard it before about Bibi Fatima Tuzahara once was not feeling well And Amir al-Mu'minin asked her That what does your heart desire from the sweets of this dunya? What do you want now?  We are not talking about medicals here And this is where you know Sayyid al-Fatima says that you know my heart desires Ar-Rumman Qalbi ashtahi rummanan, you know my heart, you know our desires from granite and When Amir al-Mu'minin heard that he said okay, I'll try First of all, he didn't have enough money at that time. So he had to go and first borrow Then he goes to the bazaar finds The rumman and then he bought it while coming back to home. He's so a And a poor person sitting by the side of the road Who was saying that I am Marie's also You know, nobody's helping me So Amir al-Mu'minin sat down with him And he said what do you want? He says, you know, many people have been passing by for more than one day here Nobody turned or sat down with me to ask me, you know, how am I doing and whether they can help me or not You are the first one. He says, what do you want?  We don't know because he is on that level and He says, you know, I have this desire my wishes to get a pomegranate He had only one And he thinks about it. I have a wife at home who wanted this and I promised I'll get it She's not feeling well This man on the street is asking for the same thing Then he says, but I remember this ayat of Quran Allah subhanahu wa ta'ala says when a beggar comes to you do not refuse in a rude manner So Ali decided to sacrifice there and Gave that Rumaan to that, you know poor old man who was not feeling well He added actually he brought a rocket for him and fed him And he felt very comfortable and kind of better at the time And this is where Amil-e-Mu'minin now has a bigger challenge goes home Kind of embarrassed. What does what do when what does he say to his wife? And this is the time of test and trial This is where kufu comes in whether they are compatible to one another on the level of values and Eman or not You know, sometimes you will have similar situations where your respect and love will be questioned You know, you have nothing in my heart for me. You give to somebody outsider But now look at the example that we have from Sayyid-e-Fatima salamatullahi alayha When he realized even she realized she says Amma innaka magmumu While he you feel sad for what happened Fawa izzatullahi ta'ala He says I swear by the majesty of Allah subhanahu wa ta'ala Inna ka lma at-ta'amta dhalika sheikh al-rummane When you fed that promigranate to that old man Zala an qalbi ishtiha al-rumman The desire that I hate in my heart for the promigranate Just went away You're not You know questioning him not being upset with him actually saying Ali. Don't worry And and this is where you know Allah subhanahu wa ta'ala and now we come to the medical spot of the story Somebody knocked at the door And I'm a little more meaning said who is it? And he says this is Salman Ali opened the door and Salman was there, you know holding a tray which was covered And Ali said mimman ahadaya Salman Salman, this is from whom? He says minallahi ilal rasul wa minal rasul ilayka This is a gift from Allah subhanahu wa ta'ala to the prophet and from prophet it has come to you Amir al-Mu'mineen, you know took the tray picked up the cover and He counted nine promigranates Look says Salman says ya Salman لو كان هذا إليya لكان عشران لقوله تعالى من جاء بالحسنة فله عشرام ثالية Salman if this is from Allah for me There should have been ten promigranates here not nine Because Allah promises in the Quran if you do one good deed, I'll reward you Ten times and this is where Salman took it out from his bag They were close to one another don't worry about you know, and this is where Salman He says ya Ali wallah he cannot actually run Yes, they were ten but I just want to test you You know, and this is this is where remember this promise of Allah subhanahu wa ta'ala and this is Surah An-Fal Surah An-A'am Surah number six at hundred and sixty Where he says man jaa bil-hassana falahu ashraam thaliha If you do one good deed Allah subhanahu wa ta'ala will reward you ten times


Good evening and welcome to episode 395 of the Private Property Podcast on my host, Usamandamwa Kumalo. It's the Tuesday edition. I said Wednesday, I feel as though this month certainly, you know, fled by, speed and pie. It is the Tuesday edition of the Private Property Podcast. And if you join us for the first time, welcome to the family. You tune into the only talk show that deals with property on a daily basis right here in South Africa. And to all our regular viewers on Facebook, on YouTube, on Instagram, welcome back. You know how we do. Every single weekday, you and I have an appointment at 7 p.m., where I'm always in conversation with a property expert who helps us make better property decisions. It doesn't matter where you are on your property journey, we're certainly here to make sure that you are well catered for and get the right insight and info to help you along the way. And talking about the right insight and info, we also have a whole host of other great shows every weekday at 8 p.m. that you can tune into. It is a Tuesday, so we'll be bringing you the farming podcast right here at 8 p.m. And you can catch the same show every Thursday at the same time. And on Wednesdays, Estie Carlson brings you the first time home buyer show and Chad takes us through the home shopper's show every Monday and Friday, where he takes us through incredible properties that you can find on www.privateproperty.co.za. Well, those are the great shows every weekday at 8 p.m. Another great thing that I love talking about is of course the great competition that we're running on our Facebook page, where we want to find out from you some of the great property advice inside that you've picked up on while watching the show. And all you have to do is to share your comment of what have been some of the nuggets that you've learned while watching this show. And of course, if we pick your name in the lucky draw, then you get to walk away with 500 grand in cash. All you have to do to claim that prize money is to be watching us live, drop us a text and of course do so before the end of the show. And if we don't get a lucky winner before the end of the show, the money rolls over into the money bag. Yesterday's winner was indeed watching. I think we had what was 2000, 1,500 grand in the money bag here to make use of was watching. So she got to walk away with that, meaning that our money bag is back at 500 grand. So we're certainly looking forward to who the lucky winner is going to be. Well, this evening, I've got our guest, Shanae Stratiwi, who is the CEO at Rent Master. We're going to be looking at top insights on how to navigate the shifting rental markets. But we're going to put a really different spin to it. There's a great article that Shanae wrote that we really want to reflect on a little bit because the reality of being a landlord is very important to understand the numbers side of things. And we can never downplay how important that is, understanding the numbers, being able to use different data sets and different data points in order for you to make better decisions. But there was also a human component when it comes to managing property. And of course, we have different people living in your property. So that's some of what we're going to be reflecting on. And I want to find out from the landlords this evening. It's the tail end of the year. December is literally in a couple of hours time.  We've seen so many lows. I think some people have seen some highs, but there's certainly been quite a number of lows. What has that landlord journey been like for you this year? Do you share with us down here below? Absolutely looking forward to hearing some of your insights. Shanae, good evening and thank you so much for joining us on the show. Thank you so much for having me again, Zama. I really enjoyed joining you and your audience and sharing some of our insights. So yeah, let's get to it. As you see, there's a whole human element to what we deal with. And today I'm super keen to share some of that because usually I'm giving you data point after data point. So there's some fascinating data points so we can share some of those. But I'm keen to hear from your landlords too. And I think let's first start with what and I was asking the landlords how this year has been for them as landlords, certainly keen from you what the data has been telling us around how landlords have been doing or how their portfolios have been performing in this specific year because I think it's been quite an intense year. It's been quite a tumultuous year in different contexts and as we really slowly start reflecting on the year that was and of course gearing up for the upcoming year. What have been some of the trends that you've spotted when it comes to the rental market and the different regions that we've got? So I certainly can't ignore where we started from 2020 because 2020 with COVID and hard lockdown, we have to have a look at that and see what it did to our economy, what it did to our rental collections and we had an all-time low and we must understand that that's not a quick recovery in terms of the economics of things. So when we have a look at double income homes going to single income homes, we're looking at how are people recovering in terms of their affordability? How are they recovering in terms of being able to pay their rental or catch up on their date? And we certainly see a decline in consumers' credit profile. How are institutions such as banks and rental companies like for instance, in-master with their orders, how are we helping these consumers who fell onto hard times rehabilitate themselves? So that is the trend that we have to bring into 2021. What are we doing as landlords, as property owners, as agents, as institutions in order to get consumers back on the bandwagon? Because they didn't have a choice about it. Let's face it, many people didn't have a choice about the financial quandary that they found themselves in. So that is what we look at in 2021. We see a recovery in the payment profile behavior, meaning how many people are paying on time. That has been a certain recovery, sitting at about 83% on a national average. But what was more interesting to me is that with the lowering of the interest rates, you had first time homeowners. Meaning those tenants that were good tenants now are in a position to buy their first homes because of the drop in the interest rates. Now I'm looking forward to 2020 to see what does monetary policy and what do our interest rates say for these individuals that just got to own their first properties and are they going to re-enter the rental market? But landlords were left with high vacancies. So in this last quarter, I've just read from PP and Creative Bureau, their vacancy survey and you know what amazing data they have. And what they are telling us is that there is a slight decrease in this vacancy rate, showing a marginal improvement. And what we're seeing is we're seeing very low or little or zero escalations in rental rates. So yeah, that's pretty much the trend from this year. And you know what Shana, as when I think about some of the trends that we've seen this year, one of your other unfortunate trends, and it speaks to the article that you had also written, was that you also saw a lot of separation. Divorce rates were significantly higher than in other months, and not just for the South African context, but we also saw it in other parts of the world where you know, married couples are divorcing or certainly getting separated. And we know that something like that has an effect for both the landlord and the tenant. And I think it's one of those things that we don't usually think about even when we enter into for instance a lease agreement with our partners. We don't really think what would happen if we break up.   And that's what I want you to look at. I think first let's start on the owner side, because that's you know, that's the more admin heavy part because you would be married to somebody. You know, what are the implications, especially of course as it relates to property when owners who are landlords of a particular property, they are separating and now need to deal with the admin of the property or properties that they essentially manage. Yes, so you're very right. Now the sweet spot we're lying and during this last year we've encountered and have to have a lot of difficult conversations. Meaning the last thing you want to be dealing with in a you know relationship breakup and in today's world we have a lot of individuals that are cohabiting. Now if we look at a landlord perspective, you've got individuals that are going into marriages or cohabiting, they co-own properties. If you're married, clearly whether you're married in community of property or out of community of property would have influenced how you contracted with your tenant. So when you are out of community of property and your property so happens to be part of your assets, that wouldn't necessarily be affected when you are changing your marital or your relationship status. As long as the contract that you've got in place is with the two correct contractual parties and I think that's what we ultimately want to get to. What we ultimately, the principle is that you want to enforce a contract and at the end of the day you start off with good intention. You need to understand who are the contracting parties and you need to plan for the fact that those contracts have to reflect who the parties are accurately should you need to take your action. So at any point in time if there's a change in those contracting parties you need to change the contract to reflect them. I'll give you a very, very sad situation. What happens if during COVID times the property owner passes? It's a reality that's happened. What happens if that property falls into a later state? Who is the executor? Who is giving authority on whether the property can be rented out and to which bank account in the estate that has to be declared is the money going to because that needs to be given to any agent and we need to make sure that the tenant is aware of that changing circumstances. I think those are such important things that we need to always consider and as you said I think one of the unfortunate things during COVID is that people have also passed away. Never mind a couple separating by landlords we've seen landlords pass away and so many things have unfortunately gone haywire because of that. There have been instances that we've also seen on social media with tenants saying my landlord passed away and their partner suddenly wants a place or somebody said I need to move out within a month and yet you're only four months into a year long lease and people just wanting some kind of help to best manage that. I want us to go for a quick break, see who the lucky winner of the competition that we're running is and when we come back we'll be taking your questions and comments this evening as we look at the insights on navigating shifting rental markets and of course also understanding the human component of being a landlord and of course being a tenant and how you can best manage those situations because if anything COVID has taught us is that there's so many other things that happen that we hadn't quite thought of but there are ways that you can best manage that. But in the meantime let's have a look at who the lucky winner of the 500 Reins in cash is going to be. And that lucky winner this evening is Fatima Sadian 500 Reins up for grabs for you this evening. Do make sure that you drop us a text before the end of the show in order to claim your price and if you want to be just like Fatima all you have to do is to enter our competition on the Facebook page it's the pinned post on the Facebook page we have over 6,000 comments the more you enter the more you comment the more you stand a chance of walking away with that cash price and of course you have to be watching us live in order to claim that money. Going back to our conversation looking at top insights and how to navigate the shifting rental markets I'm joined by Shanice Trethewey who's a CEO at Rentmaster and taking your questions and comments on our Facebook page we've got Christine Dachawa asking good evening please advise if there's a rental gorge within an area or a landlord can increase according to their bond then it differs to the very next house or flat and I love this question Shanice essentially it's how does a landlord set the rental price are you setting it based on you know your bond installment and if you know it ends up having the price point being significantly higher than the neighbors how and we find this particular challenge with new landlords who don't quite know you know how the market typically works and think that the way that you set the rental amount is by looking at what your either your bond is going to be or what the overall expense is going to be and thinking well that's how I'm going to set my rental how do we go about setting that rental price point sure I mean I've had this question so many times and I find it fascinating because I really do want to give landlords a formula but the way I see setting your rental right is by understanding all of the different components so let's start in a simple way and say all right you've got a bond in property so you need to understand what numbers and what payments do you need to make in order to not get yourself into trouble that is a starting point that is what I call in my world a some cost whether you get that in return or not that's going to be going out your bank account your bond payment needs to be made your property agent needs to be paid your managing agent needs to be paid your lights and water so you need to understand what are all of those subsequent to that you then going to go and like for instance at Grandmaster what we do for our landlords if we we get a property report and tpn credit bureau provides a property report and based on rental data in your area as well as surfing the net seeing what other people are advertising what their properties look like you can start seeing what the rental index is the rental index is this is the average rental and this is the amount of vacancies and how much competition you've got now you've got these two sides and so the negotiation starts you present it and put it out there you see what response you get and then based on that you start adjusting I just have to say to landlords that you need to be aware of three things one you need to be aware of interest rates and where they go in because your bond payment is going to be affected by that and secondly inflation you need to understand what's happening because costs if they rise are going to affect what that profit is so you can't uprise yourself but please make sure you've got a margin that you can work with a buffer and I think that's such an important one because I think when we look at understanding the numbers understanding what you can afford but also I think you know sometimes it actually goes to the first starting point about what the intention of that property is are you buying it for capital appreciation are you buying it for cash flow do you understand the difference of the two do you understand how the numbers have to make sense you know if you choose either one of those particular you know strategies if you're looking for instance to buy and hold and to rent out because that also then informs certain things I know some landlords who with certain properties when they're buying for you know capital appreciation would argue that look I'm okay with topping up certain expenses of a property in the beginning stages because I'm taking a long-term view of this property and because I know that in you know five seven ten years time the value would have appreciated and that's what I want to unlock you know in the event that I sell it or perhaps refinance it in order to access that equity or if you're then looking at you know buying a property for cash flow you're very clear that you're not interested in in paying any kind of amount you know the property needs to be able to more than pay itself you want to be able to make x amount of money when you of course have that particular property so understanding those dynamics you know certainly helps I know that some people buy a property to say they want to eventually give it to their children or this is where their child perhaps will live when they're in velocity so for now it just needs to pay for itself it doesn't have to make any money but it's just that thing that they want to be able to give to their kids in the future so I think it's very important and that's why I love that question because it has it has you thinking through as a new time you know landlord or certainly an aspiring landlord some of the the fundamentals when you have that particular when you're thinking of a property why you want to actually buy it and and and what needs to be triggered in the event where you're buying it for you know capital appreciation or to potentially give to your children and that's something that's so important so thank you very much there Christine for that question now so now we spoke about what happens when you know a couple who were who are landlords of a particular property in the event that they separate and and what would typically happen but then what happens on the flip side when you now have tenants who co-habitated or perhaps who even married living together and they're breaking up and this could have been you know at least agreement that they both co-signed as opposed to just in the one person's name so they're both co-signing and they now want to separate what needs to happen what does the tenant need to do in terms of notifying the landlord what does the landlord need to do in terms of either changing the nature of the lease agreement so it does become somewhat complicated in the scenes of when you're contracting with two individuals and they are going on to the contract from a landlord perspective firstly you need to understand who is on that country at the end of the day the principle still stands there should be an agreement in place that you should be able to enforce at the end of the day we're dealing with humans a conversation is the place that we start instead of being obstructive or not answering landlords are overwhelmed just as tenants are because in the COVID environment everyone's facing pressures so look we must just be human about it so let's start at a place that we can all reference and that is the contract at the end of the day we want to know that that is reflective of what is happening on the property so when we have a couple that's co-habiting who signed it what we've seen in the red master environment is that people bring us a couple and one of the partners has signed whereas they're both intending to share the rental because they're doing each doing their part in actual fact both of them are liable both of them are over 18 and it would be right for both of them to sign that lease agreement and that's what I say to all of our landlords make sure anyone that is in that environment that is over 18 and a contracting party should be signing that lease agreement now in the event that that changes we need to understand two things we need to have the notice from the leaving party and we need to evaluate the financial impact what happens if the if the tenant that's left over reasonably cannot afford your rent that's an awkward question but it's one that needs to take place it's one that you need to ask and it's one that green master asks of the tenants when these changes happen we need to rerun the credit check based on the changing circumstances you know we go to doctors and at that point he's not going to look at the blood test from six months ago he needs to make a call on what's happening right now and and we need to do the same thing so when you are specialist in it you're going to look at what's going to affect it can that person afford it and are you going to be prepared to re-contract with them based on the fact that they have that profile and they are are they going to honor their obligation a tenant needs to give that notice a landlord an agent if you've got a property management agent needs to change the legal documentation and tenants must be aware that if your partner moves out you don't change the lease agreement the partner that's left could be on a lease document and who are people going to go after unfortunately landlord you'll have to take action against the person who left the relationship in the property and and that's that is an unfortunate legal situation so let's avoid it and I think you know it's such an important one to note because uh the the the what are the realities that you know is that when you know a breakup is is happening uh regardless of the nature of the breakup uh and and of course you know lease agreement is is affected you sometimes don't stop to alert your landlord especially in the event where you can afford the place in the meantime right so let's assume in the moment you can actually afford the place so you're not under financial strain I think it would be a significantly different thing when you know you cannot afford it on your own your landlord is going to be one of the people that you know you have to think about perhaps notify if the lease agreement has to be terminated and and I want us to talk a little bit about that you know it's in the context where you your partner and you are separating and certainly no longer going to be living under the same roof and one party the one whoever is staying can unfortunately not afford to stay in that particular apartment because of you know the rental and them paying that amount of loan and they now want to terminate the lease agreement and it would be terminating it prior to when the lease agreement is meant to actually end uh what what happens in those situations is we tend to find a lot of times you know tenants just struggling to we'll call it get out of the lease because that's the you know that those are the words that they use that they can't get out of the lease or they need to find somebody else to take over their lease uh not knowing that you can give notice there may be penalties and sometimes you may rather you know have a conversation with your landlord about that uh perhaps take us through the kinds of options that the tenant may have uh that they can you know have that they effectively cannot afford your place and they're letting you know as soon as possible about it yeah so that is a very sensitive topic and you know what we alluded to earlier is that owning property is understanding data understanding the money but the magical eyes and being able to bring that middle ground between the human element and understanding the data and finding a way to negotiate and reduce it to writing for being fairly representative of both parties so and this is why lease agreements are so important right this is why for instance we've got things like the tp and d spec which red master bases the lease on and that says the landlord has rights they have a right to cancel the lease early and that is based on the fact that they are going to occupy it or sell it um they've got to have those reasons and a tenant can also cancel the lease and the terms of that cancellation is captured in the lease agreement they need to give so many days notice if you've got a fixed term you need to give you 30 days notice if you've got a month to month lease you need to give seven days or however whatever is stipulated and you've agreed on now in these situations we first evaluate how far from the end of the lease is that because that's going to it's going to determine how penalized or how how the landlord's impacted if you're two months into a 12-month lease he had anticipated having someone there for another 10 months so the likelihood is that he's going to say well i've got a i've got a levier one month penalty and i can say that i'm going to go ahead for advertising and you need to to take that cost on because i just finished going through all of that now in those cases what happens is that's when tenants go well if i find you someone then i can avoid that penalty and i can avoid the advertising cost okay that is part of the fairness of the contract if you are a month away from the end of a 12-month contract it becomes a more difficult negotiation because what happens is that you still have to give you 30 days but it does mean that you the court see it as how fair are you in putting such a vast or such a big or heavy penalty on the tenant when they would have exited a month from there okay and you would have had to advertise so these are the sensitive and subtleties that you need to go into understanding it however the tenant can cancel they will be liable for penalties and that is a point of negotiation and then they exit the property and unfortunately tenants need to understand that lowered affordability means they need to make and come to terms with their change in lifestyle and denying it is only going to get them into financial trouble and that is a big reality that a lot of consumers have had to face in the last two years changing their lifestyles in order to afford rent which is an obligatory payment we've got a last question that i'm going to squeeze in before we wrap up the show coming through from ulu ni sibani asking do you have to have a good credit record to enter into a lease agreement so that is a credit agreement a credit profile is showing your risk profile it means it's showing how well you pay so as i said the reasons why you may have a good or poor credit record is based on how you honored your payments good intention is one thing but what i say to all tenants or consumers you need to be very careful about how you spend your money and honor your obligations because when people look at you on a piece of paper they're going to assess you based on that you can't be denied for instance um you know you can't be denied saying i can't apply for property because i have a poor credit record but the landlord has a right to understand whether they are okay with your payment profile behavior to believe that you will pay it so from that perspective you need to be very aware that it contributes to an individual's view of your risk profile and they have a right to decline your application to a property and you need to show them that you can rehabilitate yourself and so for instance something like doing your rental via debit order contributes to your credit profile and what it means is over time you can show that you rehabilitating yourself or you are honoring it and then you can get to a point of a better profile and therefore seek better properties unfortunately it is a risk a question of risk now as we as we wrap up any final you know comments to both landlords and tenants when it comes to uh you know better tenant landlord relations especially the context of any significant changes to either one of them that could of course affect how they are a landlord or their ability to continue being a tenant in a particular property at this point in time what i've experienced is that no one's about being emotional we've all faced so much stress in the past two years and we would have responded in a very rational way we are acting out of desperation out of needing recovery of need out of financial pressure and we need to be very measured in the way we interact if you feel that you cannot interact you know you've got agents you've got people that have an impartiality and you need to understand that at the end of the day talking about it actually reducing it to documents and making sure that you come to a fair solution is what is going to be in the best interest of you as a landlord because you also are going to have to face consequences if you do unfair practice similarly with a tenant you've got a responsibility to your landlord to speak and respect what you contracted to do so the point is let's understand stick back be measured in our approach it is tough everywhere and we just need to understand the humanity and be fair and that's where we're going to leave it at this evening Shanice thank you so much for joining us on the show thank you so much for having me and that is Shanice Tredaway who's a CEO at rent master wrapping up the Tuesday edition of the private property podcast and unfortunately Fatima Sadian has not raised her hand to claim that 500 rounds so we're going to have a roll over tomorrow a thousand rounds is up for grabs I do have that if you have entered uh you're certainly going to be watching if you haven't already do go and enter and of course you've already entered also you know enter more in order for you to increase your odds at winning well that's it for myself as I'm under one tomorrow this evening on the private property podcast and of course I will be hand over the reins to Umbali Norco who's coming to your screens at 8 p.m on the thumbing podcast and of course second all things agriculture until then hope you stay home and stay safe