So I think consumer loyalty is a difficult topic. It's not an easy topic given the human nature, right? We want to try different things It's it's a difficult emotion to hold basically But at the same time You know when there is a deep loyalty towards a brand we really enjoy it and we really love it, right? so I think I'll start with a question where So for me some of the Tata brands. I'm very loyal, right? So whenever I travel I like to travel on Vistara or I like to stay in Taj I mean given similar hotels, you know, I would prefer a Taj Basically, and we are in Taj. I love the food here. So would love to know One or two brands which you are very loyal to and why So for me from a consumer durable point of view if we ask that since we are born and been using Olegate is a very favorite brand since my birth and if you take about financial sector I as you said that Tata is a good brand But similarly my personal is a public sector brand any public sector brand is a very because the trust element is there So I prefer a public sector brand Yeah, I think more a new age customer now So I think I have a lot of loyalty towards so matter every time I want to order food I always have an option of so matter versus we on the other friend whenever I'm booking a cab I always have an option of Ola versus over and I've seen I've spent a lot of extra money on Uber at times But it's just it's just I don't know in for some reason. It's just yeah I travel a lot Uber is definitely a favorite But the brand I do really like and I find myself keep going back to them is a decade long While, you know, I'm not a big Outdoor or a sports enthusiast or anything of that sort, but whenever I do Need something in that sort of category so to speak For some reason I always keep thinking of decade long, right? I think you know kudos to them for building a brand around that category that mind You know brand recall is there the experience in the stores, of course as if you've been to those stores The associates are very knowledgeable, right? You can try the product out. So lots of great things around it. So I really like them for me One of the brand that I really look forward as a loyalty customer is punished. It's a jewelry brand. I Think it has the true essence of word loyalty has been taken into actual implementation across all touch points of And as a customer Whether it is to do with service transparency communication Whether it is to look from a perspective of it's a status symbol the kind of experience that you get when you are in Versus So One of the brands that I'm extremely loyal to but not for the reasons that probably would I ask the questions that the brand is shopper stop And the reason I'm very loyal to it. The first reason is it was the first company I started working for so it starts from there And therefore it's very close to the heart. So that's how I continue to be a very loyal customer of shopper stop But if I have to answer your question in true spirit is the brand that I keep on going back to again and again is one plus In spite of a lot of things being said about Android versus Apple and this thing so for me It is not Android. It is for me. It's one plus So over a period of time consistently, I've been moving on to the newer models and then moving on to The watch the earbuds and whatever one plus keeps on coming including a one plus TV. So that's Great So What are the key characteristics of consumer loyalty 2.0, right? How you are using technology In your business or in other some other business you would have seen would be great to understand the insight So maybe you can start with you Abhishek on this question So For us and I'll talk about our industry life insurance. I will divide loyalty in two parts First part is loyalty because when a customer takes a product from us the customer has contracted to Be with us for next 15 years 15 years 20 years 10 years depending upon the policy term If the customer decides not to pay us for those period of time the transaction overall is not very profitable to me So the first part of loyalty is when the customer is already have acquired a customer just to ensure that the customer keeps on paying the contracted What was the contract for so that is one part of the loyalty because Customers keep on at writing in this parameter. That's one part The second part of loyalty is when you have acquired a customer but a customer and Research shows that whenever a customer buys a life insurance policy you end up buying six seven life insurance policies throughout your life and These six seven policies are normally spread over three or four life insurers So the second part of loyalty comes in that out of the six seven policies how much what is the maximum share I can get So these are the two parts of loyalty for us third part of loyalty which and I that's a very unfortunate senior nobody talks about is is the distributor loyalty Because insurance is sold through distributors like customers have a choice of choosing one of the 25 life insurance companies So as the distributor to sell any of these companies It's very important that I as an organization Retain my distributors because that they are the bread and butter. So these are the three dimensions of loyalty for us Technology plays a huge role It's plays a huge role because data analytics and ML models actually tell me which customers are likely to at right Which customers are likely to abandon us even after taking a paying the first premium? So once we know about these code of customers we take in extra efforts to retain them similarly Normally insurance is bought whenever there's a change in the life stage So depending on the data we know when the life stage of the consumer is changing That's the most right time to pitch the next product to the consumer So that's where you go to the customer third part is even when the first sale is happening You leave customer with something So what we do is we tell customers that you have wealth protection needs you you have wealth enhancement needs you have protection needs Today that product you are taking with us will only take care of this need This need will still remain unfulfilled. Just leave that thought with the customer or even the protection need that you have taken It's not 100% fulfilled. It's probably 60% fulfilled that 40% gap is still there Leave that thought with the customer so that next time when you are initiating a conversation with the customer You don't start from scratch. You start from the where you left off earlier. So these are the things we use Day in and day out to get loyalties from our customers and distributors Bobby can your business? so I'll give you a little different perspective from loyalty from a segment industry, which is The biggest financial and emotional decision that a human being can buy which is his home. Yeah, and You may go for an FMCG product Which may you you will be using every day versus you can look at luxury like cars Which you can change after every three four years or you could have something which is ultra luxury or watch Which is like a legacy but there is always a way or a choice a customer takes to say, okay I would like to move on or I have not like my decision was wrong. I can go back But when you buy a home You don't take a decision for yourself, but the entire family Right and in our business how we look at it is from a perspective of one customer who buys with us at Ashwin Shade we are analytics says that we have some five to seven percent Population of our customers who three generations have bought with us. It shows three things one Prosperity for that customer loyalty third more trust and faith in our brand So using technology coming back to is why it is so important in in the segment that we are operating which is real estate is We use multiple cues to understand how the person is moving up in his life for example Let's take an example Shreedhar is a friend and he has moved from an X organization has become the CEO of a Y organization Right his income has changed his status has changed his lifestyle has changed his head office now is in BKC The first thing that analytic our software will do is identify and first congratulate Shreedhar and say Congratulation Shreedhar for the next big leap in your life Right time for you to look at a different lifestyle Your office is in BKC a great product of ours is in Bandar West Why would you like to look at it now when Shreedhar has a Emotional connect because the brand recognizes him He is also connected because he now knows that yes I'm anyways I should look for contemplating because his initial office was in Andheri So he was staying somewhere for travel is the key and time third He is getting through his Facebook. We are aware that his son is getting married Then it's a right time for him to upgrade a bigger house So we use technology at every single spear of the person's touch point once he is a customer of ours Or has come visited our product whether online or offline We then track him and identify the cohorts that we could create out of it and based on that Intelligent list propose a product for example is not looking for himself But as is an investor his roots are from Bangalore and we are coming up in Bangalore as a farmhouse project You would he would be the first one to know That you know we are coming up Would you be interested being part of the inner circle you would like to do a personalized presentation to you So our technology is evolving with the customer the way he is today touch base We realized initially that 60% of our consumers were basically the ones who are using laptop So we created our design or conversation related to those but in the recent last year when we did our whole analysis 93% people are using mobile handset. So our entire communication or touch and feel our experience The touch points to him shifted from an X to completely to mobile So those are the technological stuff that we are using with Salesforce some of the AI analytics to predict a customer that based on His last five years pet and when can he buy or we should prod him for the next buy is also something that we work around Thanks, so first of all Bobby guy. Hope all your predictions come true that I become a CEO soon No, thanks for that And you know, I think that's a great segue and I work for a technology company sales force We obviously, you know, we are a B to B company, right? So loyalty again, you know from our perspective when we are dealing with our own customers and you know Ashwin shade group is a is a fantastic customer of ours. He just you know talked about us a little bit One thing is of course, you know, we have to continuously innovate on the product Right. In fact, just when we were sitting in the lounge. He was talking about what are we doing in the genitive AI space, right? And and that's a often asked question, you know So what I'm trying to say here is that we have to continuously innovate so that we make the life of our customers You know easier better so that they're able to reach to their, you know consumers In in you know, his case, you know, the home buyers and so forth, right? So that is one way of ensuring that at least, you know, the product that they've invested in is something that You know continues to deliver value, you know or time, right? It's not something that is gonna just sit on the shelf and you know, nobody's using it Right. So that's a very basic level of thing that is a table stakes for us The other aspects are of course, you know, ensuring that he also realizes, you know value from it, right? And value realization is something that has become more and more important to ensuring, you know loyalty because you know, everybody is operating in a very Tough economic, you know environment I would say and you really have to stretch that, you know, rupee to the last mile and even beyond So everybody is is getting asked those questions, right? Where is my investment going? What is the ROI on this and how am I, you know gaining benefit out of this and then they're looking for actually black-and-white numbers So it might not seem very Intuitive that house is related to loyalty But I can tell you that, you know, nine out of ten conversations that I have with my customers If they if we are able to help them, you know with the business case and help them understand The ROI on the on that investment that they made They're very likely to renew with Salesforce, you know in the following year and so on and so forth, right? So to me that again is a sign of loyalty. I would probably just stop with those two There are a lot of other reasons But also at the same time, you know from a B2C perspective It is also constantly keeping in touch with the trends and the the pulse of what consumers are doing Again, you know, it varies by industry whether it's retail or consumer goods or, you know banking and so on and so forth So it really requires you to have a pulse on the ground Understand market trends. Of course, we look at what competitors are doing and you know Ensure that the latest in technology that has to offer is something that we are able to infuse in the product That ultimately helps you to deliver a better experience personalize that You know ensure that, you know, you're getting a lot more value out of the data insights that you're gathering I think that's what ultimately is the secret sauce to ensuring You know loyal customers. I don't think there's one Formula that works for everybody But our job is to simply to make sure that the technology is something that is benefiting our customers. Thanks Okay, so I'll put the answer in two parts, right? The first part is more to do with B2C of what we do as to like what we help with our customers And what we've seen across, right? I think the biggest part of Consumer loyalty in B2C companies is probably to do with two factors. The first one is building on aspiration I mean, I'll just take an example of something like a Nike I mean people are loyal back and back again to Nike because once you have worn a Nike People don't want to be seen again in a shoes lower than that basically. So that's the first part The second part is to build it on a On on the front of convenience in that case now when I say convenience It's it's as simple as you have a paytm versus a what's the phone pay versus a Google pay for that matter People have found their own convenience in different apps like today for example for me Even if I close my eyes, I open a paytm I can scan a QR code and pay to someone without even seeing the screen for that matter today So either it's on the brand front or it's on the convenience front majorly how customer royalty in B2C companies are built up At least for all the new age customer that are coming up, honestly Earlier what used to happen is the loyalty used to be built on the group brand like we all are saying here that we are great Loyalists of Tata as a company now any product that Tata will bring in our loyalty from day zero will start from hundred percent And only if the brand screws up really bad I will go down from there But I think the new age way of like the new age trend of how this is built is day zero that is zero Like like on day one that loyalty is zero actually and now brands have to build it upwards Basically, and I think that's been a big shift because the amount of consumerism that's happening across the country today It's a Western trend that's coming out in India that that will always be there because now there's an option There's a choice of many that's coming in right when it comes to B2B companies Particularly especially in how we look at customs you my loyalty and I think it's very close to what she there also mentioned for Salesforce Right. See honestly, we provide an entire content stack to our customers, right? No customer would ever want to change their content stack unless we screw up really really bad Because there's so much amount of context so much amount of data that we already have with the customer I think the same is for Salesforce as well I mean, it's difficult to move to a different stack than to remove us In that case and and over a period of time you ought to build that but I'll tell you what there's something That's also called as customer delight that you need to offer to the customer in order to make them more loyalist for you That is to do more with how do you consult the customer? Like we've seen in the most of the cases where B2B product companies have come up, right? They are very very transactional I mean when you have bought it after that a customer success manager is just there for you when you need them And it's more support than success and there's a vast difference in our B2B companies I'm pretty sure people in the room have had used one or two B2B products particularly that support versus success is the entire key of How loyalists are built through we've realized for the longest time We never called our customers unless they had a problem as soon as we started calling them up sell cross sell up So we've seen that loyalist because the demand is infinite at the customer's end The only the only ways is that person considering me versus someone else? I think one big factor we have changed in the way we run our business in B2B companies is meeting people face to face Do they think of pepper when they are not thinking of content? Do they respect in us as individuals versus respecting us as a company particularly? I think these are all smaller factors which are very personal relationships We have realized either it's a B2B company, but we are actually selling it to an individual It's it's it's probably the CMO or the CDO or the chief content officer's Consideration that the work is either coming to me or going to someone else. I think that's how I put in customer royalty in B2B and B2C So customer loyalty and retention is fundamental for any brand I'll just go back During my maybe 30 years back how a customer loyalty used to be just an example I remember during my maybe 10 year old I was 10 year old I used to go with my mother to a Kirna shop where every month we used to buy the Grocery that those times there was not multiple grocery shop and only to one grocery shop We used to go and every time we buy the grocery for the month He used to give some sugar candy something small four four or five So it so happened that I really remember that kind of a gratification The Kirna store was giving because that every there was a motivation for us to go to that shop again and again Because he's just giving that and he's I'm I'm that kind I was an influencer kind of a person in in Motivating my mother to why not we go to the shop to buy grocery. So that was customer loyalty then So why because there was only one point of sale During that time but right now the point of sale and the customer touch points have increased Multi-fold and with digital and communication being extensively Distributed it's not one place where the brands need to contact or connect with the customer There are multiple channels in which the Touch base is there and accordingly the Brands are positioning their customer retention or loyalty or whatever second is the segmentation Segmentation if you see from a bank's perspective, we are a public sector bank We are on 15 crore customers of which I can say 30 percent or less than 28 years old 30 percent And we also have nearly 15 percent over 60 years and above. So once you fits all Loyalty program weren't work here in our case So we need to really understand what segment of customers we are dealing with and what kind of a loyalty or Retention we are going to bring in that for that segment and that's where the technology has really played a very strong role in Identifying the different segments and positioning the suitable products Like for example the IIT guy will be more interested in getting his 45 subscription But my senior citizen will not know what is part if I need to really explain I can't send them a sort of a description that you increase your balance and we'll send you a subscription So certainly segmentation is another very significant element, which is playing a very a critical role in This loyalty program in the second phase now third thing is Previously in banking if you want to say some consumer Characteristic it's only the balance which used to determine. He's having 10 10 lag balance He's having 50 lag balance. He's having 100 1000 rupees balance the balance used to be the characteristic But we did with digital going extensively and cash literally out We all have in from the finance sector. We have a lot of transactional behavior of the customer So today I can see that which customer is not only having balance But where he is actually spending water is brand affinity water is product affinity so that I can really offer the right kind of loyalty or reach Reach reach to the right customer with the right product and technology is really playing a strong role here again third thing Sorry, I think fourth point. I wanted to say is that earlier it was used to be gratification like customer loyalty and Retention was always used to be gratification buy one get one free use your Shop with us and get some offer or something now. It's more not more more. It's going to be gratification It's kind of expectation from the customer. It's not that what you give I want It's what I want you I expect the brand to give so there's a lot of AI the technology the consumer behavior understanding a lot of things going to play and Literally determining that what exactly the consumer want from that particular brand and which type of customer what you want from the brand so these Co-ords or these kind of consumer behavior is very strongly supported by the latest Technology or digital and also the consumer awareness on this is playing a lot of Role in this so I think that these were the fundamentals of customer loyalty Which brand should look at going forward particularly in financial sector great, I think sugar candy is a great technology as well to you know Obviously You know I have sweet memories of going to a particular shop and so I really resonated with that example and You know Santil you talked about segmentation so You know probably going deeper into that personalization obviously can help right and we are now in an era where there is data There is technology. There's a lot of data analytics as well. So In your business You know what role does data analytics play you have talked about it a bit But if you can elaborate it in more detail, that will be fantastic Yeah data analytics play a very critical role as I said see I'll tell you for example the number of Customers is a very high and transactions what particularly financial transact because digital has really Increase the number of transactions data analytics play a very critical role in identifying the consumer behavior particularly and also the what we call the After segmentation from the from segmentation then we come to transactional beer then consumer behavior and then They're positioning of the products to them. So in in our our field data analytics play a very very strong role great Christian in your business or any other example? Yeah, so I think I'll tell you about one this one of these companies called as notion how many of you've heard of notion basically Yeah, it's one of my favorite products basically the way I use it It's almost I use it almost like I open a notion would almost four to five times a day or you're writing on notion board. Yeah, so I Someone at notion told me one thing right and I think that's always stuck with me The process of consumer loyalty or a post sales behavior of a customer doesn't start after it's a customer It start when it was a prospect Basically, and I think a lot of times when we do these segmentation data analytical holds We always put it as sales and post sales and that's because the nature of the organizations the way we are structured all of us Is that one person owns it from demand generation to lead the other person owns it from lead to customer the third person Owns it from customer to a loyal customer the fourth owns it from a loyal customer to an advocate Basically because the nature of these four things are different after a point in time in larger organization They all start operating in silos basically the biggest one and the right use of data and I see hyper personalization is something We all know of it. I mean, there's no one in the room I don't think would not know of hyper personalization But I think hyper personalization is beyond the fact by using their first name and their account number and all of these things I think that's that's done in dusted. It's it's almost it's a 2020 thing. That's their right Today, I'll tell you hyper personalization actually should start with a customer when the demand is generated Because since then that customer loyalty process starts right from there itself. I'll just give you a small example I was just seeing the Bajaj Fincer website recently the way the website is seen to me is very different from what it would be seen to you or you in that case and It's it's probably because whatever amount of data they have today the way they have segmented their cohorts If not the exact individual but the way I have segmented the cohorts My needs are put in extremely differently than yours and yours in fact our navigation bars are very different altogether I mean, that's a that's a good level of hyper personalization I've seen at the first level basically so what I what I mean by this is use data analytics cohort segmented I think there are a ton of tools from your clever taps of the world to more engages of the world to have Engages of the world that there are tons of tools. Honestly, there's sales force. There's Adobe There's there's there's a kind of tools already that exists But the way you're using it to hyper personalize information is critical the second part I would say is hyper personalizations on e-mailers is also old a lot of players when they say we have Personalized information and given that's the only channel. We have as a medium I've seen that that brands are moving ahead of that probably more shorter formats of communication hyper personalization With some use of generative AI of course in that case I'm pretty sure you all have seen some of the campaigns of a celebrity coming in taking your first names and probably a Shah Rukh Khan Mondele's campaign that came up, right? I mean those are all fine But in fact beyond first name and last name even if you are giving the customer the information that they want and that's what I call Is unassisted sales basically like you're selling to the customer without taking their first name But what they're exactly wanting would be the ideal case So the key takeaway from what I wanted to say was just hyper personalization using data analytics on personal information of a customer is not enough Even if that's not there that's okay There are brands who are doing much greater or not doing this job But actually giving sales to a customer or actually telling them what they are looking for is more critical and important So the courtization has to happen as per needs and not as per information or the Psychography and demography only of a consumer. So that's that's the key take I wanted to mention Yeah, some great points being made here and you know again from my perspective Maybe you know I'm already a convert but I believe that you know every company is going to be a technology company Right and then some way or the other it doesn't matter whether you're making cookies or biscuits or making cement Or selling software at the end of the day You are going to be you know so dependent on technology to run your you know business on a day-to-day basis, right? But I think at the heart of it is again What has been touched upon already is data and you know We've all heard the statements about data as a new oil Let's et cetera et cetera right and I think you know We are really seeing that you know being played out in the market every single day Right and especially with the hype around genitive AI and again, of course, you know I say it high because I think we are still in a very nascent stage There's a long way to go but obviously the initial results are very promising and Companies are investing in it because they they feel that this can deliver result real results But I think what they're also finding out very quickly is that you know unless the data foundation is there Your genitive AI can only go so far, right? So one of the fundamental things that we are doing for our own purpose within Salesforce is to really emphasize the collection of data and Collecting it at a very granular level and making sense out of it is something that again, you know We have a booth outside Please stop by you know We'll be able to tell you more about it is this you know product called data cloud right and at the end of the day What it really means is a is a way for you to store customer data from across the entire Enterprise all your interactions whether it's on the web or store point of sale et cetera et cetera social media Everything comes into that. We are able to harmonize it You know create a a very rich profile or a 360 degree view of the customer as we like to say and Once that foundation is created you can do a lot of things with it, right including segmentation which hopefully leads to hyper personalization Use it for customer service and and simply understand what our customers looking for and that can feed back into your product Innovation and so on and so forth, right? So there is enormous value in it But at the end of the day if you don't get that basic stuff, right? Then a lot of the you know upstream stuff will not make a lot of sense And I think your investments will not go, you know very far So again, you know just emphasizing that point. Yeah, thanks so for Importantly for us when we look at data and we look at tech It's like for us. It's like in a dark tunnel a ray of light Which is basically for us and it gives us a direction I'll give you the things that we are using data and how it is important is We are trying to now look at version 2 from the loyalty because they basic personalization Invites and all happen. So we are trying to now understand a little more psycho Graphic personality understanding of a consumer for example why a customer is loyal to a brand What is brand is basically not just product. That's what Second we try to look this is the look alike that we have created through tech. We start understanding. What is his aspiration? What is he looking in the society at larger? So we link his LinkedIn. We link his Facebook meta We link his Instagram. We also look at other forums for example again. She either right a guy is extremely professional in his corporate life, but he's in a he's very nature-loving for a person He likes to be associated with brands, which are environmental friendly Some he is associated with those brands which give back to society So am I that brand which will allow him to constantly not only be loyal but become an advocate for me? Second we also look at stages of people's aspiration. I'll give you an example We were talking to one customer is bought a penthouse with us a 30 crore product And we're trying to understand his aspiration is thought-based is Bhavi. I Want to be seen with certain kind of people? I don't want to be seen with certain kind of people who are not my type Which means brand is also in that segment by taking our brand is his status going up Is he seen that he's with the who's who of the society or he's seen with someone who is just going to arrive So those kind of tech data is extremely important for us to take the loyalty version to next level Where we are able to understand the reasons behind why a customer buys a product and Frankly when I was talking to one of the guy other customers It's not just because of features and advantages because lot of products have I'll give you an example apple The camera quality is in another brand which is half the cost is better than that But still a customer wants to pay premium and want to be with apple is not just because of the product But he wants to get associated in a certain segment where the people say oh this guy has an apple Okay, let me have a perception if I see a guy coming out of may back I will create a perception imagery about that customer in my mind if he's the sports He comes out from a sports car So those are the things that we take from tech and data and then we implement back in our business to come back on the loyalty front So I think enough has been said about tech data analytics and this was so let me give you a very basic view Ultimately we all are humans Selling our products to humans That's the base unfortunately, what has happened Unfortunately that because of the so much of data overlaid these models analytics somehow I feel we keep on forgetting that basic thing that we're humans selling to humans and We have actually forgotten what these interactions between two humans can create wonders Tech can always be an enabler Take can always be the backbone, but tech cannot be everything and that we have to take in mind We're gonna let me just give a very simple example I was in a town called Kholapur day before yesterday and Since I was visiting Kholapur we have a branch over there I also tried to look at some data with us about Kholapur customers and this thing and from the iterator I just realized that on that day We had approved a claim for a customer That means customer had passed away and we had approved a claim for the customer's family That claim was going to be credited into account on the day. I was visiting Kholapur that happened by chance So I thought I should pay a curtsy visit to this customer Just a curtsy visit thanking be with the family at difficult time and just thanking them and just ask that Did they face any problems when they were the family was going through the claim process and I think that was my intention I Went along with our the local branch leadership team. I went and visited the customer's family What we realized over there that customer was staying in one of the most upmarket location of Kholapur The customer was a well-renowned CA who had passed away his Son was now running the CA firm He used to work along with his father and they had very influential very good kind clientele This being a CA and when I spoke to him he told me that when my father was taking this policy I was telling him Kya Zoratya kyu falto mein paise dal raheo. It's not required But then when I realized that for past three years My father had just paid close to two lakh rupees and we are giving and he has got 60 lakh rupees credited into his account He said And I have now become a proponent of life that's customers words. I have now become the proponent of life insurance. I Will offer life insurance to all my clients and I would like to associate with you So that I often your product to my customers because I have seen gone through that entire process The point I'm trying to make over here is you know all being said done data technology and this thing Ultimately, we should never forget is human selling to humans humans dealing with humans So we should endeavor to build the data layers the technology layers the analytics layers around this part not try to Actually replace one with another that should not be the intention Unfortunately, we seem to be going in the direction where tech is everything in personal interactions are everything and The human touch is going out. Yeah, completely agree. I think you know with all the development in AI Humans are developing. Yeah, right. So, you know human factor is the most critical factor So, I think, you know, we have we are probably last 10 minutes. So I will combine two questions in one. So one is, you know Someone talked about the environmental consciousness as well, which I think a very relevant point So, you know, how are you looking at that as a factor and that that also relates to you know, what we as a You know species are doing for sustainability, right and taking care of what we have created here And with that, you know, how can brands move beyond transactional to more emotional connections? Which is where you know again that people to people interactions play a very very important role as well So two points environmental consciousness and emotional connections. I think that Rightly said that more than the product or service or experience now leadership brands are more focusing on Common cause are for the global good. So in it for example in even in our bank bank over at our Recently, we have introduced a product called green deposit the deposit money raised from the deposit will be Used for financing Green initiative projects, which are be as mentioned what all the categories? So this this this is nothing to do with the money is the money the interest rate is the interest rate The product is totally different, but for what cause it's going to be Utilized so that makes the difference and certainly the Affinity to the brand will make a strong impact by we are we are strongly hopeful that this this product will certainly take Bring a lot of affinity to our brand and similarly sometime back also We were doing the initiative like when you take a car loan or a home loan. We planted three for each car and home loan and there are a lot of movement or awareness happening in this direction and Most of the top brands now are moving into not just product sale or not just giving the customer the best experience But moving forward taking a conscious call on how we are going to really Support the future in terms of bringing such kind of a green initiatives and other things. So that's from us Yeah, so I have a very different take on it and I'm wanting to be very honest in this case, right? See this entire Moving towards brands for a purpose kind of a thing is a is a three four year old phenomena that has come up, right? I think the recent Viacom report that it come in people in the age bracket of 18 to 30 Basically would prefer brands Which have more purpose in that case like for example? I would prefer getting a holy color from a brand who is actually recycling the flowers from hotels altogether But I want to be very brutally honest in this case. This is only happening in the opposite 10 15 percent phenomena. That's it India as a market. We are almost almost 120 crore people now I don't think apart from the top eight to 10 percent people the bottom layer of people still Focus on purpose more. They focus on price the more and as a market. We are still educating over there See, this is a very very western culture phenomena that has come in. I've spent a lot of time in the west I can tell you for sure people over there are very concerned about what's the fabric of my clothes I am gonna wear a Birkin stock because oh, that's gonna be the the core of the bark of the tree that I'm using India may that's a bombay may South Bombay chaleja probably what I'm gonna I'm which is mass market. There's a very long tail in India Like it's the longest tail in the world of people that exist and I don't think that purpose thing has come over there Unless the brand is themselves telling ketchup. Do my account call. Okay, so I make jar like it. That is fine Bo brand who's a car right, but the consumer is choosing that Giving an extra premium because of that. I don't think that has happened and I want to be brutally honest in this case With all the fanciness around it This is the truth today that's happening But yes that upper segment of brands that exist absolutely I'll give an example recently. There's a company called as fool who's come up with this entire recycling of flowers I was speaking to the people at Hotel Lila Lila group They have initiated like six different or big initiatives overall But they are even using the bags being made by some Jodhpur Manila group in that case They're taking these initiatives, but I have personally just seen that the top one person I think the brands are doing a phenomenal job by not asking the customers for a premium But still kind of giving back to the society with what brands like Bank of Baroda Every lot of financial services company is doing anything that is very phenomenal because brands have a leverage to do that And they're ideally doing it Nietzsche, I would say it's more convenience So that's that I agree with that point But the other side of it is that you said that still we've got a long way to go I agree, but if you take the anger generation segment if you take India is like less than 30 50% of population is less than 30 years the anger population is more wanting to Attach themselves to not just the product and more into a social purpose or cause maybe we are far away in India But the population if you see in India is that this this generation is looking forward for that kind of Attachment to brands. That's a very scientific study also has been done on that Yeah, and I'll just make a quick comment I think consumers in India are very very smart, right? If a brand is trying to associate and that's a you know, it's not authentic then they don't like it But otherwise, I think you know Indians are in general I would say very environmental conscious all across many and in all it's not just you know I've I've been in villages and will villagers are very Sentimental about their trees and stuff like that, right? So I think As a whole we love our environment and we want to protect our environment, but we want to feel authentic. I Think authentic is a great word. I think you know in this context and again from from my perspective on from my employer's perspective I think you know, it's also important to Establish that emotional connection even though we are a software company I think it's important because at the end of the day the buyers are still you know human beings, right? Not a no-name corporation that buys software automatically or robotically So we do have to appeal to that human being That human being ultimately they have you know their own set of beliefs values, etc And we have ours and I think it's ultimately about aligning our values to you know That set of values that exist out there and I think you have to be authentic And I think you have to do it, you know in a very consistent manner over and over again, right? Like he said I think a lot of companies have picked up on this fad or trend But hopefully it stays and we definitely believe that you know the real business of any business is to bring about change In the society and the communities that we live and serve And I think you know hopefully I can say I've been with you know sales force for the last seven years I see that they're very you know religious about it in that sense We we have things like you know giving back to the community we have 56 hours of paid leave to actually go to volunteer work You know I can go to an orphanage or a blind school to spend eight hours and their eight hours You know I don't have to take time off, right? The company actually pays me for that So there are lots of things I think companies can do but you have to figure out what is right for you, right? I Agree see a Fundamental believe at Ashwin shit as a brand is that you know you can't cater to everyone Okay, brands are basically magnets which attract certain kind of people who have a certain kind of a value and the Ecosystem that they would like to be associated People like to be associated with people who have a viewpoint So is if as a brand you have a viewpoint that you would like to give back to society That's a viewpoint in the certain kind of set will like to know more about if you're a transactional brand There are a lot of people in the market who only look at price and they'll say okay I'll join hands with you Emotionally what I see is that if you think that this is the way you want to go be true to it throughout the journey and don't have a Cut short corner where you are showing the reflecting your other side of it, which is a fad and which is not a reality Emotionally when you are having a connect with the consumer on this parameter, which is just sustainability So when you're seeing Implement in the projects like for example in our projects We have been committed by the chairman that for in the next five years two thousand corrodes of worth investment We are going to do only in eco-friendly projects are if we have redesigned our entire design team Who is now specializing with green all our projects has to be lead certified all our project should have solar panel All our project should have electronic vehicle charging stations Yeah, it's a cost but we have identified that when what rightly said by him that if you ask a customer a premium Then you are not doing actually that so what we have done We have re-engineered our model of construction where we have absorbed that cost and not charging the customer Hi, then it becomes a purpose like in today our organization. Everything is paperless We have realized that yes every single day for every reports There is the print and the team member keeps coming back to us So we have done that another thing that I think is very important from a perspective of larger industry As a brand when you are touching to that consumer on an emotional perspective Don't be true for a certain time but keep on graduating and increasing in that level rightly said that today The average age of India is 29 as for Mackenzie Which means this is the population who will become 16 the next 30 40 years If you don't have a viewpoint they don't connect and that's the reason I think Emotionally having a point of view and then standing true to it will make the difference in the loyalty of the brand Okay, I'll quickly wrap up but Whatever I'm going to say there is at least one person in the room who knows what I'm going to talk about We actually as an organization since we are into life insurance we have a purpose purpose is protect dreams and aspirations That's what we as a company do but we were thinking that What how can our customers help? protect dreams and aspiration of other people So we realize there's a huge problem in India and that problem is Lack of organ donation Five years back Gaurav. It was five years back, right? Yeah, the reason I'm saying person who is Gaurav although he's part of Group now me and he worked together when he was part of Edelweiss We started organ donation campaign five years back. We have done a lot of activities to promote the case of organ donation I'll not talk too much about it Three months back three months back one of our customers family is not customers One of the families contacted us and informed us that Their 25 year old son Had got in touch had come in contact with one of the awareness campaigns that we were running for organ donation And he had signed up to become an organ donor through our program Unfortunately this person met with an accident But because he had signed up for an organ donor program and he had actually told his family That he wanted his organs to be donated when he was not there His organs were harvested and it was donated and five people five different people got a new lease of life Only because that person had signed up for that campaign and decided to go through it so You know when you hear these kind of things It's very sad for that particular family But I actually admire and salute the courage of that family to go ahead with that decision It's when you see these kind of incidents happening You realize that you know Kailog as a bull thing key all these CSR things and good for society These are all lip service, but when these things happen You really realize that all the investments that you did I'm not talking about money. I'm talking about Thinking I'm talking about time. I'm talking about efforts All that you do has been really worth it and extremely satisfying to yourself So that is what I wanted to add to your question This was really amazing. Thank you. Thank you so much


Yeah, thank you very much. So a few words to myself. I work for Genua. We are a company building BSD based open BSD based firewalls And I'm a open BSD committer since 10 years now so I can combine my profession with my hobby So what we'll talk about first of all why do we need something else? It has been said already is this log is a very old program and just works and that's what we expect from it Where are we now starting position? What did I do to make? The local logging better on on the machine. What did I change with remote logging sending this log messages to other? Machines and I have a short summary of what I did Okay, why do we need logging and why do we need reliable logging? So we do system analysis We want to know what on the machine is going on if you debug a problem And if some log messages are missing and we don't see that then it makes our task harder Second thing is we have attackers in the internet And they try to avoid that we see their traces They try to avoid that we see log messages So if they can bring the system to state where it doesn't lock anymore They can make their tech and we don't see that it did something evil And then the third thing is we certify Yeah, we are on time And We do a certification for our product with common criteria and there someone wrote into the documents Yeah, we do a reliable syslogger and we don't lose log messages. So We had a bunch of dirty hacks in our product to make that and I thought Let's move all the intelligence into the system into the syslog The it's all that the community and community can have an advantage for it And we have cleaner code and don't require our hacks So what can go wrong? First of all syslog is doing if it does remote logging it with UDP forever and UDP is not known for being reliable It's basically unreliable. You can just lose it if you lose UDP packet. You don't see the log message Then we have UNIX datagram Sockets for local delivery they you need file descriptors then you have to need need access to the file You can be a run change routed then you don't see it anymore. So it's def lock That's the the classical thing and there are things that can go wrong So and then we have time stamps and time zones and you run change routed the classical syslog takes the time zone from The local file system and the client was logging it So when you have when you forget to to put your ETC local time in your change route environments you get syslog messages that are different time zones some of them are UTC some of them are local time and Timestamps is also a problem when you do remote logging when you have a global environment and every syslog is Logging in local time and then you have a common log file with all things and every machine has a different time Stems that's really hard to debug. Okay Where did we start from so that's how? syslog works when you you just go through So that's how a syslog works when you you are a C programmer So there's a libc function. That's called syslog. You can say okay. I have several Severities here. We have an error message You can do some printf style Formatting Then everything goes to libc there the priority is processed. So this is converted to a number You have timestamps added there the print is done and then it's sent to devlog that's a UNIX files UNIX domain socket and The syslog D is listening on that So then the syslog D is receiving the thing writes it to a log file or sends it via UDP So this this has been there for forever more or less So now we look into details how that works So we have some priorities facilities levels severities and options that we can pass There the first confusion starts the man page speaks of level the RFC calls it severity The header files calls it priority while the RFC Means the combined thing between severity and facility is the priority so what? So what do you do? We have an open log That's this Lipsy function that a program calls when it starts running and there it can say it's okay My name is FTP demon. So tag this name to all the log messages So the user knows where it comes from it can say, okay and I want that my pit file my pit number is locked in all messages and If logging goes wrong in the message so if cannot reach this log the messages should show up at the console then you can say log cons and there are some other options there and Then we have the facility facility means that every program that there are some classes of programs For example FTP programs then we have you use the news the syslog the mail So it's basically what a server in the beginning of the 80s could have and then we have local Facilities local zero to local eight. That's everything else What's local 16? I don't know So those are our numbers and the the facility and the Severity they are all together and then they're called priority and The Lipsy generates such a string So it this one is in a decimal notation those two are together And you have time stamp. That's the classical BSD timestamp Then the program name the pit and the message so what can we change here? So we we write that to dev log that means we need a file descriptor so when do we acquire the file descriptor? We can either do it with open log or we can do it when we do the syslog call and there We have a special switch. That's N delay when you say no delay to open log. It opens the file descriptor immediately So that's it has an advantage. So we have opened it at once so before you change route And also when you go into to a change route environment, then logging still works although you have no socket there But then you got to stick up to the syslog D that means he closes all The the the listening sockets and you have to reconnect Lipsy will do that unless you're in change route and unless you have specified to syslog D to To add an additional lock socket into the change route So and then the other art added to pledge to Open BSD where you can specify that only special classes of System calls are allowed, but you want to allow every Program to write its log messages because they are the errors. They are the traces for the attacker and that's common so all this concept makes it quite complicated because You need this file descriptor you can run out of file descriptors because you have to log and delay you have to reopen it during operation You have to allow somehow syslog Socket is somehow the eunuchs domain socket. So you have to pledge eunuchs everything So that's not good. And when you say lock cons And the lip sees makes a sent to sees. Oh syslog D is not running Now I have to open the console file descriptor dev console So you need another device in your change route. Don't forget that and you need a file descriptor so if you run out of file descriptor and want to lock that doesn't work and So that's that's the implementation nightmare, especially in in situations where something went wrong So what's the solution you have a new system called? So if you invented it, it's called sense syslog and there we move tasks from Lip see to the kernel So how does it work? It's has an integer return word value like most of our system calls then you also see some Some in some error if something didn't work for example a no buffs if you run out of m buffs and You put in the message. It's not null terminated or in most cases it is But basically you give a lot as a length of the message and you pass some flags So there's only one flag supported at the moment and that's lock cons where you have to open Want to log on the console if the send to the syslog fails And the advantages that you don't need a file descriptor Because the colonel has opened the console anyway, and it knows what to do So it's not to ask of the program anymore to grab the console the colonel does it So here's an example what I call I put in the the string the length and the flag No, no does not come Okay, the question was whether I need the pit here or not and I need it all the other Processing is still in lip see colonel is only responsible for sending it. So lip see will generate the The priority the timestamp the name that you gave with open lock Otherwise, you have to store all that in the colonel it adds the pit So the pit is not reliable. Everyone can send whatever pit he wants and all the print F dispatch Processing it's still done in lip see So how do we get the message out of the colonel again before it's just like the just opened and listen socket on the unix domain socket And here's just like the creates a socket pair The socket pair is a system call and you get it's something like a pipe It's a socket with two ends and you can write in one end and get it from the other way and the other way around and Then syslog details the colonel with an yachtle it opens the fklok and does this I octal with the file descriptor of one end of the socket pair and says oh, this is my logging file descriptor and Then the colonel Registered it and it knows one logging file descriptor and every time you call send syslog it uses this and Syslog D listens on the other end and colonel writes into this socket pair because it knows which one it is because of this I octal the colonel also knows when the sent to doesn't work then it writes to the console It has it open anyway You can ktrace everything that's going through there. We have added some ktrace Oops, just have to ktrace the program and you see what it's logging Normally you write a file descriptor here minus one is written because the program doesn't have a file descriptor for that and What it's new I can count the arrows the colonel can count the arrows. So what does reliable logging mean? So either when we see oh, we cannot lock anymore. We hold the system until it works but we don't want that and So my solution for reliable logging is counting the errors and lock the errors when it works again So you see there was a problem the system works and works and continues and then we say and be careful We lost five messages and Here the colonel can count the error We have one colonel one central place and every time the sending to the syslog d fails it increments the counter So we have it on this slide The problem is with error logging at error handling That the syslog function that is implemented in libc and more or less standard is void So the program never knows or never knows whether it failed And even if it would know it cannot do anything because it cannot lock So doing it in the program is the wrong place and we do it in the colonel we counted there and then when We continue logging and the syslog the colonel tries to write to syslog again and again and once it recognized Oh, it works again, then it writes out a message To tell the user it didn't work So it says send syslog drop two messages and it writes the air know of the first failed Senses lock So I think that's a new buff for not connected something like that. I think that's not connected So if you lock something before the syslog d is running you get a not connected So next thing the timestamps so as I said before we take the local time in In every change route because it's done by libc when the log messages generate generated That's bad and when you look at the format you have no year. We have no time zone We have no daylight daylight saving time indication. So One time in a year. You see for one time every log message There are double lot messages and then you have one hour gap half a year later We have only second precision no sub second precision and Colonel when the when the colonel is logging something like the the dmask buffer the boot up messages They don't have a timestamp anyway So syslog has to do something for colonel messages So what is always doing for colonel messages is adding the timestamps looks the messages. Oh, there's no timestamp let's add one and I extended that so that the timestamp is always optional Syslog tries to parse it and if there's no timestamps. Yeah, then add one that also corresponds to the way What is written in this syslog rfc? Yeah, we also have devices that won't write timestamps. So syslog has to be capable to add one If the device that we get the remote message from doesn't have a clock So and what what I changed is that lip see doesn't generate a timestamp anymore So we always take the syslog timestamp and we have one central instance that is responsible for generating timestamps New people coming to the talk You'll just go in find a place and be quick So syslog d generates the The timestamp it's one central instance. You don't have going timestamps going back It is a difference now before you locked when the message The you did the timestamp in the program when the message happened now you do it in syslog. So you have a delay there and depending on when syslog gets scheduled it may be Bigger or not bigger So We can use higher precision as a new So the solutions for all this is you can turn syslog d you can give it the option minus z and then It generates in the new easel format In utc that is the solution for we have no daylight saving time no time zone Whatever and there we have advantages. We can also use millisecond precision So this format is in you are syslog rfcs And you see here you have the year the month the date the hour minute Second and you have a precision and this z means it's zulu time. So it's more or less utc gmt Whatever you call it Here we are allow up to six tickets precision or fractions of seconds and you should What's Henning saying there? That's correct And what does our syslog d do? I I have to quickly repeat it. Sorry Also, so Henning says that utc is without leap seconds and gt gmt is with And we log in our systems without leap seconds Thank you So You have the precision here You are allowed to specify by rfc up to six tickets I only do three because I did a measurement how how long is the The delay between logging in the program and reading it in syslog d and I don't get a better precision And that's roughly correct and the rfc says by specifying the number of tickets. You can say how precise you are It's like physics so We have other advantages by using send syslog directly So there are one some situations where you don't want to call the libc functions for you as Um application programmers just use libc syslog does everything for you But we can do other things we can call send syslog from a system signal handler. It's signal handler safe Um, we do have it In our libc when we see that a mem copy does overlap. We have mem move That's when you have memory regions that do overlap and you have mem copy That is not allowed to use if the regions do overlap because then it could do bad overwritings And if a programmer does it wrong then open bsd libc will generate a log message But we don't want to call other libc functions from there. So we just did it for a raw census log Then we have the stack protector handler that sees that a stack cookie has been overwritten by buffer overflow and in that cases you don't want to call other libc functions to To make it worse. Yeah, and it calls printf and does some formatting.  We also use that But don't do it It's never necessary except for those cases So there are other places where we can lose log messages We have the the demask where the kernel prints the boot up messages and prints some debug messages if you enable device driver debugging And basically it's a it's a ring buffer. So kernel writes and syslog reads. There's a it doesn't read on def k log and it can overflow And if syslog is reading slower than your device debugging messages appear Then it will override and you lose some messages And it's very hard. I once I tried to debug the usb stack and I felt no, you have this message and have that message And uh, how can that happen and I read the code and and it couldn't happen. Yeah, I was just an override And those messages had nothing to do with each other There were a lot of other messages in between So the kernel is counting the bytes it is overwriting It can count the bytes and when um syslog doesn't read and I see the counter is not zero And I've lost some bytes and the first thing the kernel writes to the syslog is this message And then you see ah, there was a gap now. Um, I know what's going on when I'm debugging this device driver So what do we do when we log remotely? So syslog d can do a lot of things So we have a sender Some some log host that is sending us messages We can get messages from some processes who do logging the kernel generates its messages himself And then we can write it to a file We can spawn a process and pipe it into it We have a memory buffer where we can store um syslog messages Temporarily that's interesting for diskless machines. And then we have a special syslog c uh command that communicates with over eunuch's domain socket with syslog and then it reads out the messages Um, yeah, it has been added to open bsd It was there when I looked into it, but it's an open bsd feature Then you Really? No, no, no Yeah, but I think I saw it in other bsd so So Pipe is there and free bsd so whoever copied it from Okay, so Henning says so Henning says everybody copies from open bsd. There's no other direction But I'm doubt that that's right and I would also say it's not good if it's if it would be right because There are also good other things in other bsd that we should share So Henning so I'm I'm supposed to repeat everything that Henning says and now Henning did some jokes about himself Okay So we can send the the messages to other remote logging facilities We can write something to the console We can write something to every users terminal with write all we can write something to special users You can put users in your syslog files or root gets messages or Henning or whatever Um, and you can write to special tty's So let's look at the at the details. That's more or less what I already explained When you have the process it uses either the new Method send syslog or devlog we kept the old method to be backwards compatible not everyone is using libc There's a pro module Syslog where you can say I use unix domain socket then uses that perhaps you have some programs around that still use that And then you can add additional unix sockets. That's basically for putting your devlog in all your change root environments You can specify multiple of them on the command line same minus a minus a bar And then we have devk log. That's for the messages generated generated by the kernel himself So let's look at the remote logging We have ipv4 and ipv6 when I started looking at syslog. There was only ipv4 And we can do udp tcp tls on both sides when I started we only had udp So how does udp logging look like We have a single udp packet and that contains a log message and that's it Then we have some rfcs talking about the size we have old rfcs that say Ah, it should not more than thousand and then we have others who say yeah, perhaps we should process 2000 And then we have one rfc if you don't make it longer than 1180 the the payload Then you can assume that it doesn't get fragmented And if it doesn't get fragmented then the chances are higher that it will reach the destination um So what I did is I took this value from one of the rfcs And truncate every message that I sent to udp to this length So how does it look like We have the priority we've seen before We have the timestamp depending on how you set your syslog With minus z or not you get the the the bsd timestamp or the easel timestamp So then the rfc says here comes the uh the hostname Um the if Rfc is called bsd syslog format But our syslog doesn't have at the hostname. So I don't know which bsd syslog they refer to We have an option Marcus, I think I did it where we can say minus h to add this Hostname or ip address to to every message um So and then there that uh there comes that what we receive from libc it doesn't change Yes, um the question was whether I added ascending or receiving and the answer is we added when we send it so we So what what what what I do when we when we receive a message we start parsing it We are looking at the priority that we filter it with our config file Um, we we we look at the timestamp. We add one if there is none I've always so added the the minus z option if you find the old format I replace it with a new format because here you don't see anything if you have global logging and and then It also adds this when it does in udp sent But that's also an open bsd extension another question It's the question is what happens if the message is longer and the answer is it gets truncated. I just cut it off So how does it look in tcp?  We have the old syslog format doesn't know anything about tcp. Then we have newer rfcs um They have one rfc of about syslog in general the next one is about syslog over udp Next one is about syslog over tls and there's no tcp But then you have another rfc with a bigger larger number and there um Somebody has written down what he's found What syslog implementation who has found that use tcp? So it's a collection of things that are out there and usually it just works So what we do in the tcp stream? We just put a new line to split the messages And then there are other implementations that said oh, we want new lines in our syslog messages But that doesn't work if you write in the log file because then you can't because there of course the new line is the delimiter And I replace every new new line that I get somewhere with the underscore by the way um, so somebody also use null delimiter to split tcp messages and And then there are so I split when I read it. I think with new line and null And then there's an octet counting format That's the new sheets. That's already that's also in the in the tls rfc That says are you write the decimal number with a number of bytes and you make a space and then here are 60 bytes till here And then the next message starts So what I do when I send a message add a new line here, so I'm compatible with everything except when somebody explicit explicitly is expect No delimiter, but I don't think That that is the case. So everybody splits splits that new line. It's just written in that rfc And another advantage is when you put a new line here It's much easier to to parse those messages if you just print them out You get a new line every time um, you see a new message and Just the the the octet counting in front So tls format there we have an rfc and it says Octet counting is the only supported format Then you must support 2,000 bytes And you should support 8,100 And what I did is um I added a global constant That says syslog messages have that maximum size and everything else is truncated So it's truncated in libc. It's truncated in kernel and it's truncated in syslog It can even truncated multiple times because we are adding all those timestamps and changing things And so it can get longer and then gets truncated again So truncation is the the right thing what to do with syslog messages If they don't fit because most of the time people can get out figure out What's going on at the beginning and it's much better than dropping it. Don't drop them So what I did also is converting syslog to lib event before you had a poll loop And doing all this tcp with multiple senders and multiple listeners and doing tls Um, it's quite hard with poll. So thio Uh recommended to change to lib tls that the lib event that was the first thing I did And then I linked to lib tls to do the tls stuff and I wrote a little wrapper um that does encryption of lib event buffers by sending them through lib tls instead of doing a plain read and writes And this is one c file located in the syslog directory and it has already been copied to our ldup d Demon that also needs that feature And newer the things that in open bsd We have a rather old lip tls because we think the new one lip tls too because this is Lip event. Uh, sorry. Sorry. Sorry. Lip event. Uh, the lip event is is feature bloat. So we have a lip event one 14 Something yeah Yeah Yeah, we we are still api uh api Compatible so things that when you develop something with the open bsd lip event and Copy it to linux and take the old lip event there and it still works But I've done a lot of cleanup there and internally it looks quite different That's also the reason why I didn't add this feature to our lip event Because then we wouldn't be api compatible anymore anymore. I've looked in lip event two. They have something like that So if you write portable programs, just use lip event two and if you do open bsd programming Just add it here Locally So what you also have to do when you do tls you have to care about Identifying your communication partner and doing some Certificate checks and as usual certificates are complicated You have this public key infrastructure and nobody Can set it up and understands how it should work and what it really does So I've implemented everything what you what you may need And now I've put it on four slides for every use case So um what you always have to do if you want to accept encrypted messages You have to provide a server certificate. So our syslog d uses a server certificate here and You put it in etc ssl Together with the host name you have you can also put an ip address here So the idea is to enable receiving Tls it's not enabled by default you put a t here A minus capital t and you say the host where you That's an ip address where you want to listen on you can also say 000 to listen on on a star socket You put a port if you leave it out the tls port is 6,514 We also have a 514 for udp and there's no well known port for tcp Because the 514 tcp A number it has been used by some remote shell whatever old unix tool um So but people use it anyway just uses nobody it does remote shell And so what I did here is a real ad does the same so you can put certificates for your Your machine and keys and private keys you just put them there and then syslog d uses it You can also add the port here then you can use specific certificates for syslog and relady Um So why do we do that? Because when we have a certificate here a server certificate here the sender knows with whom it it's talking So the sender can identify The drain of all the messages and that means the attacker cannot redirect the messages to himself And he cannot see our messages So another thing Funny thing about tls is you can have client certificates so then The client provides a certificate and we want to figure out whether it's the correct one first We have to enable it enable using tcp at all tls at all and we specify a certificate authority file and um Every time a sender tries to connect to us We ask him to show me your client certificate and we Check whether it's signed with this ca and if it is we um We take the message otherwise. We don't take it that means the syslog d can identify the sender who wants to talk to us And the attacker cannot inject messages. So nobody else is allowed to send something to us So that's the other side when we want to send something out So first the first thing I had to do is Thinking about the format how to specify that I want to use tls You have to give it in the in the config file of syslog syslog.con And what was there classically is that you use an ad sign And then the ip or host name and then you send you the udp packet there And what I wanted to do was able to specify or take tcp take ipv4 take ipv6 so you can say Here tls tcp or udp And then you can also optionally add in 4 or in 6 and then it will you will be forced to that protocol to that ip version And yeah, I thought about how to specify it and then I took just Something that looks like in your uh the prefix of your url because i'm sure that that will not collide with the hostname And I can parse it reliably if somebody has a better idea And those things that doesn't look like I'm I'm open for suggestions so Then If we want to send something out the receiver syslog Host the receiver lock host has a server certificate as every tls server and we want to validate it So you can either specify a ca file if it's your private pk pk Or if you want to send something to well-known syslog these at the nsa or at google Or wherever you want to have your messages Then just take something that is provided by upstream. That's the default certificate store in open bc so syslog must know the ca and The the hostname must also be in the server certificate so that Nobody else that belongs to the same cryptographic infrastructure Like komodo can can spoof it and um, so the hostname you specify here must be in the common name of the certificate Or in the subject out name lip tls handles that handles that for us And so we we can identify to whom we are sending messages And so we the attacker cannot redirect it um And the the messages are confidential And if everything this pka and this thing is too complicated turned on by default you can just disable it So for test environments that might be useful or if you don't care who's reading your messages And just want to get it off get them off use minus me So that's the final certificate slide So here we um use client certificates ourselves you can specify them on the command line And then the receiver can check them. And so he's sure that he's getting the The Messages from us and the tucker does not inject anything. So how does the error handling work? um We wanted to say that we we log all incoming connections um So what I first implemented is that we Write a message every time we accept something and somebody closes the connection to us And then somebody complained. Oh, it fills up our all our lock files. So I converted that to debug messages So syslog is in a special facility So syslog can and you can identify all the messages that are generated by syslog itself It's a facility and I changed the priority to or the severity to debug And so it's not locked by default, but you can turn it on and see Who is connecting to me and when you set up your syslog environment? It's maybe useful and you can leave it on if you don't have too many messages, but if you have Thousand clients rebooting every hour, then of course you don't want to see all that We lock connection errors so when When something goes wrong with a connection to a client we log an error and you see oh something is wrong And I lost some messages or may lose some messages Um, we count the dropped messages.  We throw them away. We get a sick term We throw them away then we count the messages that are in the buffer and lock that so you can see um How many messages were not sent to a tcp destination It's not completely reliable because the kernel also locks some messages and the kernel throws The buffers away if there's an error on the tcp connection And but when you see oh, we have a connection error Then you know or there may also some log messages be lost because the kernel has thrown away the kernel buffers And more than that are locked here that are the messages that are Dropped in user land in syslog to user land. So it looks like this syslog d because syslog generated the messages Dropped to messages and it also locks the name of the host where it wanted to send them and what are also implemented when you have multiple messages um then Syslog d writes message repeated. It doesn't write the same stuff all the time It's important for local logging because otherwise you can fill up your log files easily But it's bad for remote logging because when you have multiple syslogs then you're not sure to which message the The last message adopt belongs then you can turn it off either for remote logging or even globally with a minus r option So now let's go to the conclusion so i've um The new message flow i've here and i've made it italic everything that has changed so for the program nothing has changed the syslog call looks the same And here in libc we have no timestamp anymore And we do a syscall the special sense syslog syscall and not a send The kernel has the send syslog and it knows about logging errors and counts the errors And syslog d still receives it. It's no different whether we receive from a socket pair or from or from a unix socket It adds the timestamp. So we have one single instance responsible for timestamps Um, it writes it to a log file that didn't change and it can also send to tls or tcp So that's new So what also has to be done to be reliable logging to have a reliable log system on your machine First of all syslog may not die. Normally when you write programs, it's a good idea to die and fix the buck If anything goes wrong die, but if if syslog dies You're blind. You don't see anything what's going on. Perhaps the attacker wanted to kill syslog by some funky thing And you don't know what's going on. So syslog tries to run It logs all the errors it gets at startup or and whatever and maybe you lose some log systems You can't bind to the udp socket and you it doesn't work or tcp socket is wrong. It doesn't work Gets a log message, but still running other messages appear So we count the dropped messages internally We make tcp and tls support because they are more reliable and more confidential than udp We changed everything to lip event. That means we get safe signal handlers. We don't have to to Do all those dances with sick prog mask to to avoid some races here and there All the signal handlers are not real signal handlers. They run the main program flow lip event handles that for us We cope with file descriptor exhaustion When you write a server that accepts a lot of things like incoming tls syslog messages You we run against your process limit and then And then you Critical file descriptors you cannot open anymore. For example, we run with pref sap. That means we have file descriptor passing between a low privileged process that does all the The network handling because now we do tls. We have to be careful here And we have a privileged program that opens the log file We have to do it at every c cup for example and we have to open consoles and tty's and Read user databases to send messages to special users so And then we do file descriptor passing. So we have to we need a reserve of file descriptors And syslog handles that because incoming tcp and tls connections will always leave about 10 file descriptors Not allocated. They are for internal use Then already said we have pref sap What's important to to avoid blind rob attacks There's there's an attack when you have when you do fork you have the same address Random address layout in two processes And if you can probe the one you can gather information about the other And to fix that you have to make an exit call Normally we accept the child here We accept the parent because it was much easier to to implement in syslog d because syslog d does a lot of startup things with for and then it does the fork and So if I accept the child all the startup is is lost and the child needs the startup the startup In this initialization And the the parent only serves some some requests like doing do dns open some lock files And there it's better to lose all the states So I do the the exec and I have just a minimal program just doing the iMessage requests Easy things and we run pledge in both processes the child and the parent So I've already written some tests usually before I change programs. I I write some regression tests Um, and then I fix the bugs and then I add the new features That's the the the evil thing about writing tests. You always find bugs and then you have to fix them So don't do that um So the tests work in the way that they generate some some config start the syslog d Then they have a client in the server process the client process Generate some lock files, then it's stop everything and then I check And what can I check I can check the standard error output of syslog d syslog d runs in debug mode where every Debug message and all other lock messages are printed there. So I can see what's going on Um, the client writes a log, but it's sending to syslog. It can use native send syslog methods or udp or tls or tcp or whatever um The server also has a log what you've received from syslog d We have a log file. We can look into that. I spawn some pipes there and I look into that I make some console redirection to see that everything goes to the console that is specified there uh, I create some fake users and that I enter in the via user temp File and then syslog looks into that and writes it there on the console and redirected the console and look into the user's console Or the user's terminal. Sorry. It's the terminal redirection and I look into the terminal that from my fake user Um, syslog d creates a ktrace. I can grab in the ktrace if it does the the correct system calls And I can look in the fstat if it's has opened the right sockets Yes a question Oh, um, that's not a regression framework itself. It's the Other question repeating the question was how I test That something has been written to the console with our regression framework and the answer is um It's not the framework itself. It's something implemented in this syslog test. It's some pearl modules that do things And there's the thing that you have in x. You have the x console It's a program and it tells the kernel redirect everything that's written to the console into this window And I use that mechanism to lock everything that is written to the console into A process and the process writes into the file and at the end of the test I grab in the file So and of course software is never finished. I always say only that projects generate stable software um So what else can we do here? So the thing is when you prepare such a talk you look into to the code and think oh, that's all buggy that could be done better So the first thing I thought is um All those initialization errors that are written to standard error or the console when syslog goes up Cannot be written into a log file because they are not there And I could count the errors. I have a diff for that. It's on tech And that's meant as a hint for some co-developers to do the review Um then I went to the our Quality assurance guy at genua and said oh now we have reliable logging and we don't lose anything and he tested it and says Oh and the file system is full You stop logging. Yeah, and it's true if syslog gets twice an error when writes your file descriptor and file system exhausted Then it stops logging there even if the file system gets empty again So if a diff for that, but it is it's on top on the other diff So I will send it out when the first one has been reviewed and committed Also have a test for that. I fill a I create a file system with def vnd fill it up write a log file to that get the error and then Um check that and I empty it again and check that it's rocked again and Read all the debug messages that syslog de creates for that So then I have read the old rfc with the hostname and there's a new rfc how to Write remote syslog messages and they have version numbers there and other features and we are somewhere in between but not quite And perhaps we should get more rfc compliant That's also when when I was preparing the talk and then Yeah, you you go through this conferences talk to other developers.  No, didn't fix it, but I promised him to put it on the slide And then yesterday at the social event I was talking with imbra from the dragonfly project and he said Oh, yeah, you know when you when you when you do this startup Um, then you get the dmesh things and when syslog reads the dmesh buffer, it's a timestamp there And wouldn't it be better when the kernel adds the timestamp? It could use the the uptime and syslog can create it to to regular timestamps Then we would know in watch order those messages come and so the problem is in our company. We have a central configuration thing and they put every log message in the database and then they sort by timestamp And all those messages coming from a kernel at the same timestamp because when they were read And then they log it showed randomly to the user That's a bad thing to do And the idea was um, if the kernel would log it we would see When when the the dmesh messages Appear we can see where were the delays. How long did it take? That's one thing and then I thought about it when we would do that also for the syslog call We would get the the the time When the program does the log and that when syslog you get scheduled and reads the log and we could get more precision So I'll think about that if I want to influence it Yeah, and that's it So turns out you have already talked about the my First of my notes about been uptime in the kernel and kernel timestamps, but I have another one Uh, you said you're opening a socket pair and you're marking a socket as Used by syslog by this special ioc tool, right? Yes so there is a race condition if you do that and the Then some program tries to represent itself as a syslog demon and then it waits before until the syslog d gets sick up When it closes everything and tries to reopen again um You do the ioc tool on the def klog file descriptor So you have to open def klog and depending on your fire permissions in slash def only root can do that Okay Yeah, so you have to check for root and you have to check to be not true that in this case, right? um Yeah, because there's no device. Yeah, otherwise Okay, so you need to have access to def klog to to be a syslog A reader for sensors lock.  Yeah Very good more questions For your new syscall, why are you using avoid pointer? Is there a valid use case why it's not a character pointer actually? Oh, that's theos idea Okay, so I have to ask him the question because it's if it's only because it's like right and read that stupid There's no valid use case for So there's not to be a character you this is not system d after all Yeah, so so there are two possibilities either theo has a good reason for it Or it was just to stupid to copy it correctly from the main page. Okay I'm curious, uh, what's the motivation for uh using the additional socket pair instead of uh reading and writing directly from the klog for the Senses log messages. So in klog you have the real kernel messages. They have a different format They are different from from user messages. So you want to to have a different uh a difference the klog buffer is more or less The thing that is in dmask So so if you call the log function in the kernel, it gets added to the dmask And it can be read by syslog d from def klog. That's very that's very distinct from userland logging So we added the new syslog um the syscall send syslog and their new mechanism was Needed and theo did it with a socket pair and I think it's a good idea So but I don't know exactly why this approach was chosen Anymore questions Then let's thank Alexander for his great presentation Thank you