This is Think Tech Hawaii. Community matters here. Yes, sir. We're back at 4 p.m. It's Think Tech Asia. And of course, it's with Russell Liu, an American lawyer practicing and teaching in China in Beijing, who joins us here today to talk about a very important subject, money. And money is a very important day-to-day subject today because we lost 1,100 points in the U.S. stock market, something to do with the emperor's new clothes. Okay, Russell, welcome to the show, Russell. I'm glad to back, Jay.  It's really a big holiday period coming up in China. Next week, we're going to see floods of Chinese tourists going around the world. Money, okay. And one of the things we say is shenanigans. And as you know, out on the streets in Honolulu, we're getting more global. Bank of Hawaii has a big banner. Let's take a look at what it looks like. Let's see. Oh, yeah. What does it say? Xinyin Kuala Lumpur? A few minutes ago, our cameraman was out there, got this gray shot. It says, Xinyin Kuala, happy new year. So they're getting ready. They're getting ready because it's a time of holiday Chinese. It's always red. The colors, good luck. Because they pass these red envelopes called Hongbaos with money in it. But we're going to talk about this today, Jay, how it's a kind of cashless society. Everybody gets their red packets or Hongbao. So interesting. You know that the US, as a leader, starting, what, in the 50s with American Express, got into this whole notion of credit cards and then credit cards proliferated all over the world, ubiquitous, actually. But now, with technology, the Chinese and other countries in Asia are leapfrogging over all of that. And they're finding another way for you to make purchases without using a credit card, which is much more efficient. And you've been following this. I know how much you admire this system, Russell. So tell us how it works. Sure. You know, as you just said, Jay, Americans, we pioneered the credit card system. We pioneered a system that relies on gathering credit information, checking out to see, and it's really using money, foreign money. It's using against future earnings. But this way, the Chinese don't do it that way. It's a cashless society. We don't carry cash. Carl Maldon used to come on the television at advertisements with American Express. He used to say, don't leave American Express at home. Don't be without it. Carl Maldon, boy, that's a name out of the past. Street of San Francisco. That tells her age. Sure. But again, the Chinese say, now, don't leave your smartphone at home because it contains the apps. Wait for that. Don't lose it. And now, we pay. Don't lose it. But again, in China, we don't carry cash anymore. The typical person will carry maybe 100 renbi for the whole month in your wallet, and you carry a smartphone. You don't use it unless it's a backup. It's interesting because in China, 84% of the population uses no cash, a cashless society. Yeah. Well, OK. I mean, cashless society, it's interesting because you don't have to have a credit card. You can get along without any credit cards, at least in China right now. You have to. It's kind of like a debit card, this whole thing. And things is more than one of them, I think. And you can spend really any amount you want as long as you have it in your bank. And it's instantly transmitted, instantly. You get an instant credit for it. You buy anything you want using a QR code from your phone. So you shine the QR code at, say, a newspaper stand, am I right? That's right. And then you get a newspaper and the whole transaction is immediate. But it goes from there way beyond. Can I buy a car with a QR code? I'm sure you can. I'm not a rich Chinese. I can't take my car. I just don't have money to bank. You still have to have cash to bank. It may not be rich, but you're famous for us. Yes. But you know what's interesting? It's the Chinese who started the cash. 2,800 years ago, China invented the paper money. And now, full circle, now they're inventing a new form of money, a new platform, a new payment platform. It's a cashless society. So they've come full circle where they're innovative, but they're using technology. So now, within the past years, five years, the movement has been towards a cashless society. And this is very interesting because this cashless society means that everything that the Chinese will do, whether it's going to the supermarket, the fruit stand, you just flash a QR code, the vendor will scan it. Let's take a look and see what a QR code looked like. This is my QR code. We've all seen QR codes. There it is. There it is. This is a typical QR code on the WeChat system. And if you notice the QR code, Mr. P, that's my Identify Handler. That QR code also has information in my bank, okay, contact information. Now, it's interesting because that's your picture. That's my picture. So your picture is embedded in the QR code. Yes, it's embedded. And it's very interesting because in 2011, when Tencent, the big tech John in China, Aleksandran China, started this WeChat. They didn't envision it. Don't leave that on too long. Somebody's going to copy it and steal your identity. It'll be protected because that's what these Tencent started, WeChat. We would just flash your QR code. Somebody had just met us. We wouldn't pass cards around anymore. We don't pass out our mobile phone number. They would scan that QR code. And that QR code would go online into their system and then I could communicate with them. But fast forward. After 2011, they came up with the idea if we could use it something more meaningful. And that's because Alibaba came out with Alipay and both system work on the same idea. That you take your QR code, the vendor will scan it or the vendor will have their own QR code. And with their QR code, also on our smartphones through WeChat, we use a scanner. It'll scan that QR code and it will ask how much you want to pay. And we transfer the money to Tencent to the vendor. So that goes both ways. It goes both ways. It could be your QR code or his QR code. And it's very interesting because people ask, well, where did this technology come from? You know, it's interesting because this technology actually came out in the 90s by the Japanese automobile makers. What they would do was instead of using the traditional bar code, they developed this QR code system. Q and R, what does it mean? Quick response. And so that's how they keep track of the cards that slapped that QR code. And you know, that's the genesis of the QR code. Is this like the octopus card in Hong Kong? Yeah, well, the octopus part is from the subway. But in China, you go to Shanghai, you can use some, I believe now they're starting to use it, or the QR code, you can get on the subway. With the QR code. QR code. So it's just another thing you can buy with the QR code. And so what's really interesting is that this QR code can do many things. You can buy things, buy groceries, buy clothes. You can buy your train tickets. You can buy your movie tickets. You can use it to pay the taxi driver. Many, many things. Or you can use it to invest money. But you know, getting back to the new years for the Chinese. And this is an important occasion because we send money to our friends and family. Use your family to the old days. The red envelopes. I didn't bring it to that. I had one in my pocket. You can imagine what it looks like. You know, the reason why somebody gave me one and why we're still behind in the US, we have to give them out of the red packet. So in China, we would use our WeChat and we can actually send the money and it has a red envelope. So the red envelope is on the screen. It's on the screen. It's not a real red envelope. It's a red envelope and it'll have the Hongbao and we would put in the amount and we would just send it to our WeChat. So the other guy gets a red envelope on his screen. That's right. And he gets a payment into his account using WeChat. That's right, exactly. And so, you know, you could be in one part of China. You could be in your hometown or maybe some village. As long as you have a little smartphone, you know, and your son is out in Beijing. But this is so different than a credit card, really. Because in a credit card, you wind up, as you mentioned, it's a credit card. It's with credit and it catches up later. But with this kind of QR code and WeChat, the cashless society goes both ways. So you can put money into your account, take money out of your account and so forth. And theoretically, if you can balance it, you never actually have to put money in, cash money. There is no cash going in. It all happens inside that account. Well, what's a fascinating thing about it is, as Americans, you know, we look and say, well, the Chinese are very smart, because they realize that every time that money transfers from one account to the other, it's cash flow. It's moving the society. It's a robust economy. Sure, that's right, too. That's correct. And just every time, economics point of view, it's a multiply effect. And it becomes a very strong society. Yeah. Now, in a credit card, you have to pay credit card fees. I mean, America expressly charges you an annual fee, others too, sometimes. And of course, if you're late by just a few days, you wind up paying less than late. But if you don't pay immediately, you wind up paying interest at some really exorbitant rate. Even now, even when interest rates at the Fed are really low, you still pay a lot of interest on any credit card. But on the WeChat thing, on the QR code, on this cashless society, you already have the money in your account. That's right. So there is essentially, am I right? No charge in doing this. No charge. There is no charge. And it's fascinating because it's changing the world because a lot of developing countries where there's no credit rating system, it's too expensive to develop that, takes too long. And people won't have good access to credit anyway. They're starting to adapt what the Chinese are doing in China. Because it comes out of a bank account. So you spend what you have. And it's really interesting, Jay, because no Chapter 7 bankruptcies, no Chapter 13, you pay what you have. You don't pay against borrowed money, borrowed time. So it works as a functional practice of things because you know how much money you've got in the bank. You are it. And you are it. So what about I work for an employer, right? And I'm going to get some RMB at the end of the pay period. Do they pay me in my account? That same cashless account? Well, you know, I don't work for a Chinese employer, so I don't know. But I know that Chinese New Year, they send the red packets with money in there to employees. That's your bonus. Yeah. And it goes instantaneously to your account. And you know what strikes me is that- But what about compensation? I can do a contract, right? Yes. And he owes me a thousand RMB a week, okay? Is he going to put that RMB in my account or is he going to pay me by some other method? I'm sure that's been done through WeChat. I am positively short sending WeChat. And you know, because what happens is the employer will take the deductions out for the taxes every month, the income tax. And what happens is that the accountant will just simply use WeChat from the account and just pay the employee. Yeah, I mean, that would be the most sufficient thing of all. And it's great because, you know, the employee doesn't have to worry about going to the bank. It frees up the bank from long lines. And it also limits the possibility that people are going to steal your paycheck or you lose your paycheck. And it takes care of multiple problems. Remember, the Chinese are very practical. They're very pragmatic. So this is a great way where you just become efficient. I miss money. So if I have one of these accounts, these QR code or what do you call it, WeChat accounts, do I need to have a bank account also or is this for the average person, his only account? Well, let me say yes and no. Let me talk about the second half of program about how the impact of WeChat and foreigners. But for the Chinese, everybody has a bank account in China. So it's easy to link. But it's that account that you use the WeChat for. That's right. It's just one account. It's one account. And so I look, for example, I have a Chinese bank account and I simply link it to the WeChat account. Also, I link my passport, my photo, and it gets approved. And there's some tight security on it. So when I make the payment, they'll verify it because the vendor is not going to work because the money comes straight to them. And it's all there. Like you saw on the QR code, it has my picture also. But the interesting thing about it is that, how big is the society? Well, the other fact is not only transmitting money, but the fact is we would talk about cash flow. Cash and economy is turning over constantly. How big of it? Well, in 2015, talking about spending, e-commerce. E-commerce is the big thing in China. Amazon, we're starting to do that now. E-commerce, where Amazon actually sells more clothes than Macy's through the web page. So Alibaba is the Amazon of China. Can I order something on Alibaba with my QR code? Well, actually, for Alibaba, you simply, you can order it online. And I haven't done it in the Chinese, but you can actually pay through your phone. And that's how it's done. Is it a password or is it a QR code? Well, I believe most people that have an account with Alibaba, this is an account. They hit the password and they can make payments. But it's interesting because now Jeff Bezos, Amazon has set up a mechanism where you can actually, in China, I can order something from Amazon, the U.S., it comes on there. And you'll see in the articles is we ship to China. So they're shipping to China. And the foreign exchange curses the issues because I believe that Amazon has teamed up with either Alibaba or Tencent. They do a company that will do the foreign exchange simultaneously, the transactions. So it's interesting that that's being done. And I'm going to talk a little later in the show about what's going to happen, the impact of Chinese travelers going abroad. I'll tell you what WeChat's doing. Yeah, I want to ask you one question before we take a break. And that is, can you tell me the difference between Tencent and Alibaba and WeChat Pay and Alipay? Tencent is the founder of WeChat. And WeChat has WeChat Pay, which is part of WeChat. WeChat is a communication. It's sort of like the Facebook hyper on steroids, okay? And Alibaba runs Alipay. And there are two different types of payment systems. They're competing. They're competing. And it's interesting, but I'll tell you how big it is. Let's first look at the numbers. 2015, we had $1.5 trillion done. American dollars. American dollars through these payment systems. 2016, that more than Kodrupa, five times $5 trillion. 2017 from $5 trillion and went up to $15 trillion in the economy just in one year. Just in the... Now imagine what I'm talking about. I'm talking about cash flow and a robust economy. And this is only the everyday Chinese person. War money passes hands, that's how the economy is. And they predict that by 2021, they're going to hit the $45.5 trillion mark. Fantastic. It's fantastic to know that. You can have more than one account. Do people have more than one account? Do they have the WeChat Pay and the Alipay votes? They do. I have both. Do you? I do have both. Why? Well, I'll tell you why. It's interesting. The demographics are very different. They find out that Alipay are used more by the high wealth people. So I suppose that the products, maybe you can buy a car out of Alipay, are these people through Alipay. I think that Alipay has a greater percent, maybe 50% of the market. I believe WeChat has 40%. So close, they're close. Apple Pay is 1%. Yeah, how about Amazon Pay? I don't know. Jeff Bezos is making a move and he's going to team up with probably one, either Alipay or Chenson. That's going to be much more effective. You can see the size of this. It's 84% of the society is using its cash. We don't carry cash. After this break, I want to cover two things with you that are really worth discussing. Number one is how the Chinese government is expanding this whole notion to outside of China. And what is the purpose of that? What is the effect of that? And what is the future of that? The second thing I want to discuss as we come back is the dark side. The dark side is a dark side to WeChat Pay and Alipay. And if you want to know, you'll have to wait. We'll be right back. Don't do it in your team's house. Know your limits and plan ahead so that everyone can have a good time.  She said, what are you doing? Research says reading from birth accelerates our baby's brain development. Push! Read aloud 15 minutes. Every child, every parent, every day.  Okay. And the second thing I was, you know, the second cliffhanger was the thing about the dark side. And Russell, let me be clear about it. What I meant by that is something that Richard Hornick talked about at the China seminar a few weeks ago. And he'll be on the program, talk about it again. And that is the way the Chinese government is collecting data on people and trying to do mind control, his term. And so my understanding is that everything you do in the economy, every time you exercise your QR code, every time you buy or sell something in this cashless society, it goes, it's available to the government.  And if you're a good guy, according to the government's standard, whatever that may be, then you get benefits. And if you're not such a good guy, you don't get benefits and you get barred from things. And so everybody has a little profile. And part of the profile is this account. So correct me, Russell, this is your chance. Well, here's where we look at it. You're looking through the sunglasses of an American talking about the political issues. I'm looking through the sunglasses of Chinese, which is clear. And I'll be honest, does Richard live in China full-time anymore? No. And I live in China full-time 15 years. I really don't care, because you know what? As long as I'm not doing anything illegal, why should I care? It helps me in the conveniences of life. And you know, through a different culture where you have 1.3 billion people, how do you keep order? And it's important because with this kind of system, people do things differently. For example, I go to the best spot. I leave my sunglasses next to a counter. I walk back. I forgot. Oops, run back. It's happened to me. You know, three, four minutes later, sunglasses are gone. Lost and found. Nobody turned it in. People don't turn in. In China, I've lost my wallet three times, but with the cameras watching around, people in society now, they don't take it. They've actually turned in three times, twice in the rail station. It's funny you should say that. I don't know if I ever told you. But my second trip to China, I left my wallet at the TSA security line coming into the country. And I left it there. My God, it's incredible. You arrive in a foreign country, you leave your wallet somewhere. Okay, this is a real horrendous thing. And I'm walking away. I'm hundreds of yards away down this big airport hallway. And there's a guy chasing me. One of the security guys is chasing me to give me my wallet back. Oh, my God. I was so relieved to get my wallet back. But I was also very impressed that they would chase me. And he said, be careful next time. Don't leave your wallet around. That really touched me, I tell you. Well, that's happened to me.  And I found that it just created a lot of order in society. And we have to realize that China is, I'm against the country, and they will have to do what they have to do. But I've lived there 15 years. And I talk about it like this. China is like a 50 lane freeway. There's a lot of room you can go as long as you don't cross the edges and get into political things. Most people are in that situation. So who cares? You know, they have a national ID system, but maybe a lot of things work. You know, they have cameras on the street to watch cars causing traffic infringement, speeding. But we have cameras too now. We start to have cameras here. Again, all of these things, again, are different cultural and different foundational concepts. But, you know, I find that, again, maybe Richard Hornick, you know, I live there full time and it's easy to criticize on the outside. It really is, you know. But imagine now that you have WeChat. Sure, the laws are different. The cybersecurity laws, you know, the government has these laws where they want to watch for terrorism and so forth. They have a back door to the server. Yes, if you're a multinational doing business in China, you're worried about your IP being protected. But again, that's an issue that I'm not. OK, I appreciate that. And, you know, I want your view of it. And I agree with you that one thing China has established is the stability of this society. It is compared to other countries, quite stable and successful economically. But I wanted to go to one other point with you, and that is the spread of this whole cashless society thing. Whether and to what extent China has exported this kind of technology, this kind of, you know, transactional arrangement to other countries and why. Let's look at two formulas versus what impact does it have? Does it have their lessons that Americans can learn from?  Amazon Go, a few weeks ago, opened their first store in Seattle, which is unmanned.  That clicks to your account. And anything you take off the shelf clicks and charges your account. You pay for it that way. Artificial intelligence. That's artificial intelligence. And where did they get it from? Well, the Chinese have been using it, the scanner, the QR code. So that impact is making its way even into the U.S. We're going to see more of that. I predict more of that. The cost of doing business in the U.S., the brick and mortar, the labor cost, we're going to be more and more into a service economy. You won't have a lot of Macy's who sells people. Macy's has their artificial intelligence or some sort of model they're working on. And Walmart has something like that. So it's in the works. It's in the pipeline. So there is an effect on it, okay? Second of all, so is this model, this paid model, is it making an impact around the world? Yes, it is. Through Southeast Asia, Singapore is considering. India has, the government has just adopted this. They're moving towards a cash society. They're following the Chinese. Well, they're following the Chinese. It's not the Chinese system. It's not the Chinese system.  I forget the title of it, the Amazon, the Amazon store, that the Amazon would try to export their technology to other places and sell it to a retailer in Yon. Is that what's happening in India? Is it the Chinese selling their technology in India? Or is it India emulating what the Chinese is doing? India is emulating it. However, let's look at MasterCard's thing. MasterCard has got agreement with China Unipay, and they're trying to create a universal system that is very similar to this. So they're going into Africa, again with the Kiorco. So that's a new industry standard being set. It's set up by a company called EMVCO. Is that a Chinese company? That's a group that's put together by MasterCard Visa and China Unipay. China Unipay. So it's going on board. It's a venture between the American MasterCard and global MasterCard companies and China. Right. So they have an Africa, the Kiorco system, and they were in Nigeria where they had some success. So we're seeing this making its way around the world. And why is it successful? It's because it's where most of the world is not like the US. Most of the world is developing. It seems more like the case every day, actually. But it's interesting because Visa and MasterCard, they can't operate in these worlds. So they've got to change their product. So they're adapting and they've got to work with the Chinese to come out with a product because if not, they're going to be extinct dinosaurs. Are you saying that although India right now is emulating China in this regard, because India is fully capable of developing its own technology. I mean, they've got a lot of technology in India, a lot of smart technology people. Are you saying though that over time, not only will people emulate China, but they will want to use and be networked on the Chinese system so that and China would like to see that happen? I know where you're getting at, Jay. Yeah. I know where you're getting at. Go ahead.  Go ahead. So where you're getting at is now are these system merging? Are the China technologies being used out of China? Yes. WeChat, Tencent has recently has been working on a plan. And what that plan is it provides that you have to, we talked earlier in the first part of show, you've got WeChat on your phone, don't you? I do. Now what they're doing, they believe they're teaming up with a big MasterCard or one of them. And they're allowing the linkage in WeChat with your MasterCard or Visa so that you can use it and the transaction will automatically go to your card. Right. And the likelihood there is that it's going to go in the direction of the QR arrangement, the cashless society arrangement, the debit approach, rather than the credit card approach where you have all these other charges and where there are security issues that are ultimately charged to the consumer somehow. Yes. So what I suggest from what you said is that over time, we will have a global system that the Chinese will have a part of it, that the MasterCard Visa, you know, stands at American credit cards, may migrate in that direction. In fact, MasterCard has MasterPass which is a debit card. Yeah. So we're going to see that link with, for example, WeChat. Yeah, there won't be a card anymore. Well, that means it's going to change to retailers. They're going to, people aren't going to have a card. Right. It's going to be linked to that account. In the Amazon store, it's just a QR code when they walk in. They took it from the Chinese. I mean, that's the same thing. But now you can download WeChat and the Apple Store of Samsung, and now they are going to allow it. So it links to like your debit card MasterPass something. Okay. What about security? I mean, you know, we worry about credit card, you know, terrorism and credit card theft of identity and all that. So if we have the cashless society and a debit card basis, how safe are we? Are we safer than with credit cards? Well, that's because that hinges on government regulations of that industry, government regulations of WeChat and Apple. Can somebody knock off my QR code? Can somebody copy it and then take money out of my account, buy things out of my account? But I still believe that a lot of places now, you have a QR code, but now a lot of vendors will show you only their QR code and you scan their QR code. And you scan their QR code so your QR code doesn't go out. You see? Some vendors will scan your QR code. But in China, the standard more is, I see more is that more vendors will have their QR code on a sign and you just simply scan them. And so far, I haven't heard of any major theft from these retailers.  Okay, so maybe with that kind of technology, the possibilities are limitless, really. We can minimize the security issues that we have now. We have plenty of security issues now. That gets another thing, Jay. You know, it's fascinating because, again, what WeChat is doing is they're letting foreigners to get on WeChat, link your account. You can buy things wherever you got the QR code. Second of all, that's the first step. The second step is they're working on agreements now where WeChat, the Chinese upon travelers, which is a huge industry, will be able to link their account to retailers in the U.S., where they travel, use the QR code. The retail industry is going to change and I'm certain it's going to change just by that. And it's going back to the question, where are we in Hawaii? Are we up to this? Do we know about it? You know, and it'll be a great boon because imagine going across borders instead of carrying 10,000, it carries smartphone. Well, clearly this is all leading to a global system. Yes. A global way of spending money, a global way of consuming and consumerism. It's a global way of doing business, really, on so many levels. And I guess the question, which we'll have to leave it at this question because we're out of time, the question is if you have a global system, a global payment system like this, whether it be emanating from China or elsewhere, whether it be the purely WeChat thing or some kind of hybrid, when you make a system like this global, do you increase the risk of some kind of global collapse, some kind of global meltdown, or is it safe? Is it safe, Russell? You don't have to answer that. Well, so far it's been safe. Well, I'm in China. There's a lot of government regulations in place in China. I feel very confident. And from what I've seen is that there are problems, yes, that they have to work out. Again, it's really going to be the government that's going to have to back it to make sure that you won't have a lot of these issues. Great, great discussion, Russell. Great, great important topic. Let's do this again in a couple of weeks. Great, great. Russell, I'll be back in China. Okay, we'll do it in China. We'll catch you by Zoom in China, yeah? Great. Xin yan kuai lai. Yes, Xin yan kuai lai.


And here we go. Hello and welcome. My name is Shannon Kemp and I'm the Chief Digital Manager of DataVersity. We'd like to thank you for joining the DataVersity webinar today, why you need end-to-end data quality to build trust in Kafka sponsored today by InfoJix. Just a couple of points to get us started. Due to the large number of people that attend these sessions, you will be muted during the webinar. If you'd like to chat with us or with each other, we certainly encourage you to do so. Just click the chat icon in the bottom middle of your screen for that feature. And for questions, we will be collecting them via the Q&A in the bottom right-hand corner of your screen. Or if you'd like to tweet, we encourage you to share highlights or questions via Twitter using hashtag DataVersity. And if you'd like to continue the conversation after the webinar, you can go to community.dataversity.net. And as always, we will send a follow-up email within two business days containing links to the slides and the recording of this session and any additional information requested throughout the webinar. Now let me introduce to you our speaker for today, Jeff Brown. Jeff is a director in InfoJX's product management group responsible for delivering customer-driven solutions across InfoJX's data 360 platform. He holds an MBA from DePaul University and a Bachelor of Science in Engineering from Michigan State University. And with that, I will give the floor to Jeff to get the webinar started. Jeff, hello and welcome. Hi, everyone. Thanks, Shannon, for the introduction. And let me just get started with my slides. Like Shannon mentioned, my name is Jeff Brown. I'm the director of data quality and analytics here at InfoJX. And today we're going to talk a little bit about Kafka and its impact on current organizations and what we're seeing in the market and how we help different customers throughout different industries as well. So I want to start off with just a quick stat. So recently Forbes had mentioned that 90% of the world's data was created within the last two years. So as you could imagine, this is quite a bit of data being generated, and a lot of that is being generated from streaming sources like IoT, different types of devices that are streaming data. And even in the corporate world, there's a lot more data that's being moved around that's being transferred from system to system. And inside of each of your organizations, I'm sure you're seeing this, that the generation of data is just starting to overwhelm the business. So some of that data is also coming from a new platform or a recently new type of messaging platform initially created by Apache Kafka. And if we just take a look at Kafka at a quick glance, and some of these numbers are brought to us by Confluent, which does a lot of these different types of studies and offers software as well. But 60% of the Fortune 100 companies are using Apache Kafka. So that's quite a bit just in its most recent adoption as well. And if you look at over 100,000 organizations globally are using Apache Kafka worldwide. And I imagine it's a fairly steep slope in terms of adoption most recently, and some of these numbers might have even gone up in the recent months. If we look at a 2018 Apache Kafka report from Confluent, we see that 90% of Kafka projects are considered mission critical within the organization. If we then look at what are the adoption and why is why organizations going to Apache Kafka, 62% of the deployments are actually replacing existing technology. And we're seeing that with our large customers as well, where they're moving from some of the old file-based systems into this new event-driven architecture to get more data, higher volume, and higher throughput as well. So why are organizations moving to a streaming-based architecture? So even beyond Kafka, just a streaming-based architecture. Today we are going to focus about Kafka, but in general we do have a lot of customers and we see a lot of research being done that streaming data is becoming more and more popular even in enterprises today. So if we look at what is Apache Kafka, Apache Kafka is really a real-time streaming message system that is built around the concept of a publish and subscribe. So similar messaging systems do exist where you're either it's a Q-based type of system, so if you think of the MQs of the world. But in the publish and subscribe system, you have publishers for Kafka, publishers, which are really the creators of the data, the messages, that are feeding the messages through different topics and consumers on the other end are then subscribing to these different topics and then receive the messages from that different topic. So it's a little different from MQ where you could have a Q being sent messages, you could have the concept of a subscriber taking messages off of that Q, whereas in this new Apache Kafka world, you have messages that are being created and are just submitted to one or many different consumers that can then take the messages off of the different topic and they don't disappear, they are retained for a particular amount of time. And messages are stored in topics across Kafka in many partitions, really helps out a lot of the different redundancies that enterprises are looking for to keep multiple versions of their data so that if anything were to happen in a particular partition that it would have a failover. So if you look at it in a more visual concept for Kafka and just a general data pipeline, if you look at the flow of data, you've got the producers on the left-hand side and as I mentioned, this could be devices, it could be log data for machines, it could be application logs, it could be Internet of Things or it could be general applications or third-party vendors and even files to a certain extent could be generating messages that are then put into a topic. They are basically brokered by the Kafka platform, they're published and brokered by the Kafka platform and the messages are then consumed by the consumers on the right-hand side through a subscription-based type of model. So if you just look at the flow of this chart, you can see that it goes from left to right and this is just a very generalized pipeline flow but the concepts are the same. When we talk about producers, that's anyone who is creating messages, posting it to a topic and then the consumers are really subscribing to those messages, to those topics and receiving those messages in real-time. Some of the advantages that we hear customers talk about are listed here. For the majority of the customers, they are really looking for that real-time data availability. So with Apache Kafka, it enables organizations and enterprises through their event-driven architecture to reduce the lag time within the data that's being posted and even move from one producer to a consumer so that data is now available in real-time or even near real-time as soon as it happens, which allows the consumers to have faster responses to critical data-driven decisions. There's also the concept of Apache Kafka acting as a centralized access to data. So if you think of that concept in the middle of that chart that we had seen, the Kafka platform gives that central access to the data in one location. So instead of building out multiple integration points, if you look at the point, one of the points on the screen on the right-hand side, it reduces the amount of integration points as well. So that IT, who is basically handling a lot of the Kafka management in terms of the technical platform side, it gives consumers a centralized access to the data and from an IT perspective, it is a reduction in integration points. So if you have thousands of applications and thousands of downstream systems looking for that data, you don't have to build out an integration point for each one of those. Kafka acts as that centralized data platform to a certain extent. It is also built to scale to massive amounts of data. So high volumes of data as well as high throughput. To use Kafka, it is built to handle a very large amount of data where sometimes some of the other systems and even message-based systems would potentially drown in the amount of volume that is being handled. It also acts as a type of data storage layer. So one of the advantages of Kafka also is that it can provide a certain amount of data storage so that if you wanted to retain data, messages, let's say, for a week or a certain amount of messages even in terms of counts and amounts, you are allowed to do that so that if you are subscribing to a topic, you can potentially even go back in time prior to the time that you're subscribing to it to retrieve data and to retrieve messages. And again, if we're looking at the last bottom right, there is a certain fault tolerance that I mentioned that is built in into the system. So it provides that additional security type of data potentially being removed or deleted or not available within the entire platform. Some of the key drivers to move to Kafka also include the ability to make faster business decisions. So the faster the data is put into the decision-maker's hands, the faster the decision can be made. So this is also an advantage when we talk to customers, we ask them, why are you moving? What's so technical? Some of it is we can no longer live in this batch-based world, we need access to more real-time data. It also creates a unified data hub so that business can consume the data from a centralized point so that helps the different applications, the downstream applications go to a single location to receive the data and to even receive some of the data that might be critical to their decision-making and critical to their downstream systems. It also gives better data access to data scientists and analytic teams. So this is also in conjunction with some of that real-time data. So depending on what type of data your data scientists are looking to consume, they now have access to this real-time data within their organizations because they can have it as soon as it's generated, as soon as it's posted to a topic they can then consume it and then make decisions, build it into their models so that they can then either train on it or learn from it within their systems. And that same goes for the different analytic teams. If they're looking to make analytic decisions, you can have data that's faster and at a higher volume available to you. And lastly a lot of the companies that we speak to, Kafka's part of their digital transformation strategy, and what I mean by the digital transformation strategy is that they're trying to move a lot of their on-premises type of systems to a cloud-based system. They're moving to a more event-driven based type of architecture and these are large companies. These are Fortune 500 companies that perhaps in the past may have not moved as fast in terms of adoption of this sort of newer technologies but we're seeing that more and more throughout our customer base that as they move to Apache Kafka or they have it in their sites within the next 18 months that they are, that that is a key part of their digital transformation strategy and a lot of that has to do with the real-time availability of data and even some of that redundancy that I mentioned before. However, it's not always rainbows and sunshine for these organizations that are adapting Kafka. As we talk to them and as we talk to them and understand some of their challenges we've collected some of them here to talk about. So what are they saying? A lot of times since it's such a new architecture and it's a new jump into a piece of their business flow that audit is not just going to let them adopt this blindly so audit will not let us move forward with our Kafka platform without being able to validate the data. So sometimes it's audit driven so that they're saying everything looks great in terms of the architecture, we love the concept of having data in real-time but there's no way that audit is going to let us move forward with transferring this large amount of data from one system to another in terms of producers to consumers without any type of insurance around it so that audit needs to know that what was created is being sent etc. So that they need validation on the data that is in motion. So if we think of Kafka's data in motion that the data is from producers to consumers really needs validation before it's being pushed out or posted to some of their downstream systems and auditability of the data and process is still a very big key focus for the different companies that we're speaking to. We also have several companies talking to us, several large companies across industry saying we're moving all system to system communication from a file based to Kafka messages. So what we're hearing is that they are no longer generating or extracting any type of data or files in a batch based type of manner, they're moving strictly to Kafka based system and what this means is that it's a new means of digital communication between the systems and there's a true recognition of real time data access. So this may be driven by business saying we need the data faster, it might be delivered by IT but all in all they're saying no longer are we creating any type of files and storing it or moving data in that manner we're moving away from any files and moving strictly to event driven architecture or moving to Kafka as our centralized type of data hub so that we can have access to this data in real time. And lastly we're also hearing some hesitation as well from customers from this is from a large bank. We don't trust the stability of our Kafka platform to expand its usage. So again, so there's reasons why they're moving and why they're pushing to go towards it but there's also some hesitation saying we've got our Kafka platform in place we're not in production, we're getting we're ramping up to production but we're really not moving forward because we just don't trust yet, we don't have the validation on the data, we don't have trust in the overall Kafka platform not because they don't trust Kafka itself it's just because it's so new that they don't have their arms completely wrapped around it and they can't get transparency and clarity of the data inside of the different topics and streams that are being generated and pushed to these critical business systems. So really they do require insights into their operations. What we're also seeing from customers and from just companies that we're working with is that it might be a new technology but it's going to be the same challenges that they've seen before. So what we're seeing is that they're running into the same data quality issues even with their Kafka platform. So if you take the old adage, garbage in, garbage out just because data is being pushed through faster at a higher volume does not mean that the data quality is any more improved than it was before. So I think that's a key concept that resonates with a lot of the different enterprises that we work with is that it is even more dangerous now or potentially even more risky endeavor that they're taking on because it's the same data that was from the producers that are producing the data except it's now being pushed to the decision makers hands even faster so that the quality is still the same they're not doing anything to the quality it's being put in the hands even faster. So it's almost like turning up the conveyor belt even faster data quality is going to be the same as what it was coming in and it's going to potentially have an even a faster response time in terms of the decisions that are made. So we've got a list of questions that we know what the organizations are saying what should they be asking to sort of squash some of these challenges they should be asking do we know if all the transactions that were supposed to be sent were sent how do we know that all the transactions that were supposed to be sent were sent do we even know when or if duplicate data is being sent through the different topics through the different streams how do we know if duplicate data has been sent do we know if the data across transactions and even let's say the messages whether it's policies or claims or different trades in terms of financial was aggregated and transformed correctly how do we know? So if audit saying prove to me that this data is being aggregated in the way it's supposed to be they're having trouble answering that in an honest manner in an honest fashion because they don't have clarity into that into the data that's being pushed from system to system again how do we know that all the transactions supposed to be sent had arrived how do we know that they even are being sent in a timely manner or even in the correct order so as we're talking to the customers they're saying we actually don't know that everything is being pushed a lot of times since Kafka is fairly a new project or a new platform that they're taking on it's mostly about setting up the infrastructure making sure that it can handle the value making sure that all the topics are being subscribed to appropriately making sure that it can handle the size of the messages we're talking to customers and they're talking about messages from three kilobytes to even three megabytes so it's a broad range of different sizes that these messages are being sent so some of it is just operational can we get the messages up can we get Kafka up and running but really when you take a step back once the business starts to adopt the real-time data that's being sent over through this new platform these questions will arrive so I need to know that you're not sending me multiple claims in the same message or even multiple claims in the same topic or across topics I need to know that I'm not doubling up the amount of transactions that are in my daily trades and even how do I know and how am I notified if the right amount of messages came over in the same day so there's a sort of threshold a threshold tolerance that's being built how do I know that the number of messages that were supposed to be sent weren't either above an upper limit or a lower limit that are within this sort of tolerance or threshold within the organization so there's the whole concept of the monitoring of the Kafka streams and then the context of the messages themselves is different so there's almost an operational view and then a context or business view of the questions that these customers should really be asking themselves and so what is the impact to some of these challenges so if you take the negative from all those questions a duplicate message was sent a message was sent late or it was incomplete or it was transformed and aggregated incorrectly from an IT and operations perspective they're unable to monitor the different data volumes for the different anomalies like I mentioned what if all of a sudden messages let's say there's 10,000 messages that were supposed to be sent in a topic in a day and that drops to 100 messages and the customer or the consumer was really expecting to have 1,000 messages well they need to be able to respond to those anomalies appropriately so that they can find out and do a root cause analysis because maybe something is wrong technically maybe the data is not arriving on time or maybe their source isn't being posted so there's a couple of different challenges and impact to the IT and operations so if we look at how does IT also then predict any of the different data volume that's needed for retention so if they're unable to monitor the streams and do an accurate prediction of the amount of data that will be stored and retained inside of their Kafka platform this could be an issue this could run into some different type of processing errors a different type of downstream systems that are being impacted because data may not be delivered in a timely manner and then again just having all these different types of monitoring allows IT to potentially identify underlying infrastructure issues so it could be a hardware type of issue from the business side those that would be consuming the data you could have incorrect data that's being consumed that you're making the wrong business decisions on and again like I mentioned you could be making decisions faster now that data is being delivered faster you could also ultimately lead to customer loss you could have a harm to reputation you can have revenue loss regulatory fines etc depending on the type of data that is falling through the cracks or the data that is not being delivered on time or the data that potentially being delivered incorrectly so all of this could lead to a sour customer could lead to real money so real dollar in terms of regulatory compliance fines and then ultimately if you take a step back what this means is that there's a reduced overall trust factor that comes into play so that the business doesn't trust the data whether that's to put in the regulatory report decisions on or to do any other type of reporting on so that it leads to reduced trust reduced adoption moving forward do you build data trust within your organization and especially how do you build the trust in Kafka and provide that end-to-end data quality when I think of the data pipeline I kind of revert back to some of my manufacturing experience throughout my career I take a look at any of the data pipeline in any business across any industry if you look at the source data that's coming through when you do different types of quality checks on a car that's being built you don't put quality checks at the very end right so as the parts and as the raw sources are being put together you're doing the different types of validation and monitoring and quality assurance on those parts same goals for processing of the parts so if you think of that in the manner of well now these parts are being put together in terms of processing they're not ready to be packaged up and sent out but once they are you've got that QA and the quality assurance that's being put on that as well if you think of this similar to a data pipeline you've got raw data from the left you've got the source data coming in you've got third party data this all needs to be monitored and validated you can't wait until the last minute so that at the data warehouse we're actually doing the validation on the data so if you think of potentially Kafka being in the middle here or even on the book ends of the process you really need validation and data quality throughout the process you can't say I'm doing data quality on my data warehouse because I'm doing cleansing or I'm doing different types of validations on it at the very end by that time it's already too late you don't know who's consuming that data you can't track down the source system by the time it gets there potentially is aggregated and transformed in a manner that is even unrecognizable from what it looked like at the source so from an infigix perspective we talk with an enterprise or we talk to data strategy teams we are really looking at the validation and data quality from an end to end perspective all the way from raw data raw source to that finished product whether that's in a data warehouse or extracted to a data mart data lake etc to your MDM system we're looking at it from end to end and that's the same way that customers enterprises need to look at it because you can't just rely on it to say you know what I'm just going to add the cars rolling up the line I'm going to do some QA on it well that doesn't make any sense because you're going to end up with problems that you can't then do anything about because what if it's if you look at it from a car analogy what if it's something to do with with the engine well you're going to have to take it out and it's a lot easier to fix it up front than it is to fix it at the end so when we talk about end to end this is really what we are we are referring to from an infigix perspective if we take that producer to consumer type of approach when we talk about how we provide that validation that data quality that reconciliation even and balancing we're looking at it holistically so not only do we look at it from what you need to be able to get in line to the message before it's posted to the consumer and you need to provide that validation but you're also looking at data that could be coming from source systems that are not Kafka messages or are not streaming messages so you need to be able to provide that data ingestion from all those different types of sources along with the Kafka message so that you can read the data that's ingestion from Kafka so you can read the messages you can even potentially read in data that's not a Kafka message aggregate it, transform it and then spit out a Kafka message as well for those downstream systems to consume so we look at it as we insert ourselves into the key process flow so that you can provide that insurance that the message, the context of the messages is being validated through different business logic and rules engine but also from a threshold and monitoring IT perspective that you're able to provide that visualization and monitoring of those amounts and counts of the different topics as well so it's all interrelated in terms of data quality and as you look at how do we create a new data quality project well you can no longer look at I'm pointing to a data lake I'm pointing to a data warehouse and that's where my data quality efforts are being focused on again by that time it's too late the new cars rolling off the line so by providing validation up front and as early and even as often as possible whether it's producing systems, whether it's sour systems beyond this you need to be able to provide that level of validation more so around Kafka you need now you have higher volumes higher throughput a new technology business consumers that are looking to make critical business decisions faster you want to give them the assurance that you really are providing them with the highest quality data that's not only from Kafka but even before that so if you look at it from before and after Kafka and even within Kafka that's what we're looking to provide and that's what has really resonated with customers and has been successful we look at it from a multifaceted approach in order to build trust from producer to consumer going back it's not only data quality so you need to be able to have the conformance checks the different uniqueness checks completeness checks all of the key dimensions that you need to put into play but you need to have reconciliation so if we look at Kafka as a data in motion data movement mechanism you need to be able to say that this number of transactions and messages were created from the producing system and this number of messages were received from the consuming systems and they need to be able to reconcile the counts and amounts and even at the transactional more detailed type of level that it needs to be reconciled and it needs to balance throughout so that you can build that trust within the business so that they can go on and expand usage and trust their decisions and lastly even the integrity of the messages and data that's being sent from left to right if we go back and use that diagram as an example the integrity did any messages fall off was everything that was supposed to be delivered in a timely manner was delivered again getting back to the integrity of messages to making sure that I know that whatever my data that is leaving a producer or leaving some of those producing systems is ending up in my consuming systems and even downstream further in the highest integrity highest data quality as well so how do you provide a 360 degree standard to data and trust again if you take a step back you look at it from producer to consumer so you're setting up the data flow the data pipeline of the data that's moving from producer to consumer but you have to look at it from a top to bottom approach as well so not only do you need to identify the data that's bad you need to have a rules engine that's flexible enough to have business users be able to create rules to be able to transfer their business logic into real data quality rules but you need to be able to have different mechanisms to monitor that data as well so monitor the stream so you're monitoring continuously it's not a one and done you don't point it at a data source say give me the data quality no you're really trying to put in that reoccurring monitoring of the data and messages that are coming through and flowing through your data pipeline flowing through your Kafka platform so you have I've got the rules I'm able to build the rules I've migrated my business logic into some usable data quality rules I've got my monitoring how often I'm doing these checks and what needs to be done in terms of this check first that check first okay if that fails what do I do next so that's more of a monitoring who do I alert which key stakeholders do I alert but once I have that data or that bad message that's being identified what do you do with it do you have a mechanism to then remediate that data to quarantine that data to then notify those key stakeholders to say here's a topic that could be then producing a bad data stream and then split it into a good data stream and a bad data stream but bad a good topic and a bad topic do you have the mechanism to keep the key stakeholders and the providers the producers of that data accountable so that you can go back and say you are the highest offender of the quality that's being monitored in our Kafka platform today and here's the proof here's some of the data we've got it in our system it's been assigned to an internal queue to be worked through a workflow you can assign it to users and groups to then look at or even send off downstream but not just having that raw data and having the bad data and having a mechanism to remediate it and track those remediations you need to be able to communicate it appropriately through visualizations through dashboards through KPI through different metrics and they need to have bar charts and dashboards and line charts and all the different type of graphics that you can imagine within a single system so that all the different roles are being addressed so anywhere anyone from the data quality analyst role or data analyst role to a mid tier let's say assign manager all the way up to executive so that you're able to roll up those KPIs and metrics and all the different bad data type of metrics and even monitoring or good data metrics so that you can communicate it appropriately throughout your system and so that's really our 360 degree standard to providing that data trust that's how we view it and that's what we provide Infigix provides it through the data 360 platform so we've got a three pronged approach to how we tackle some of these items we tackle it from a data governance perspective so not only do you need to know what data you have but you need to know who owns it you need to know where it's at you need to know when it's changing so you need to be able to obtain that metadata and catalog it appropriately assign owners and do workflow of changes etc so that's from data governance perspective from a data quality perspective too from a metadata perspective on data governance more so you're knowing your data from data quality you need to be able to trust your data so you need to provide that data quality baseline you need to get at that transactional level you need to be monitoring the data that's flowing through and that's what we're talking more so about here today around your data quality being able to build that data quality and trust and then finally getting your hands on the data and being able to harness it be able to transform it even more so within the system and that's more for our data analytics approach so if we look at data quality a little deeper for streaming data what does that mean for the companies and customers that we're working for we talk about the end to end pipeline what are some of the key functionality features and characteristics that are beneficial to provide data quality on streaming data so you need to be able to provide data quality on real time and batch so validation on real time and batch data so that's streaming data that's file databases some of the more traditional types of data in a required time window so if we're talking Kafka we're talking fast high volume so you need to be able to say I need to captured every single message in real time or if I'm monitoring a topic I need to say how many messages came through in a certain time window and even more so I may not need a decision or validation on my data so I can do it more in a micro batch type of mechanism minutes or seconds or an hour every hour you need to be able to provide that balancing and reconciliation like I touched on before need to provide the reconciliation from source all the way to target and Kafka sits in the middle so if you've got data sources that you're converting to Kafka messages maybe you want to move beyond the Kafka message and get before it turns into a Kafka message to those source systems so there's a lot of tools out there that are transmitting files and database data tables into Kafka messages so what if you want to get at that data to say I know that was whatever was in my table ends up at the consumer and I've got validation logic all the way from every step throughout that process again I need to be able to provide that transformation and aggregation logic so you need to be able to have that rules engine to be able to form and aggregate aggregate the data on streaming and non-streaming data as well I'm looking to enrich streaming data also so if you've got different topics that are generating let's say claim information or policy information or a financial trade I'd like to enrich that message so that it could be enriched from a non-streaming source or it could be enriched from another streaming source another Kafka message I need to be able to do that so it's all these different type of operational checks and validations on streaming data and even non-streaming data and again being able to visualize I need to be able to identify and manage exceptions I need to provide that upper control limit lower control limit I need to know if messages are degrading inappropriately so that I can put in some of those key statistical controls to say if over the last four data points of the last seven data points Fiverr are on a negative slope identify this key stakeholder like some of the western electric type of statistical controls that are often implemented I also need to have machine learning on some of this data as well so there's a large amount of volume that's being touched that's being monitored that's being validated within Kafka stream so I need to be able to have machine learning around that to identify anomalies to identify outliers to learn from to build my models in and I need to have that in a tool that's also doing the validation on that message so it's redundant to have multiple systems doing machine learning as well as some of this data quality validation if you're if you have that that message and that data in your grasp you might as well do machine learning on it as well to build out better insights in your in your enterprise from a data 360 streaming functionality if we dig more into the infagix product we're seeing a lot of basic functionality that is that is very critical so it can can be basic and it can be used in a very much more complex manner we'll touch on that later but you need to be able to ingest streaming data stores you need to be able to output to data streams as well you need to be able to convert that streaming data to batch data I need to be able to do some sequel on some of that streaming data as well some sequel statements because I may have certain sequel statements that I want to transfer over from some of my existing checks and I need to put them on some of my Kafka messages I need to be able to join different types of streaming messages with both streaming messages and batch messages I need to be able to provide that deduplication of messages that we touched on early one of the key questions that companies are asking how do I know if data has been sent multiple times or if I'm receiving duplicate data you need to be able to provide that deduplication type of mechanism if we take a step back in terms of use cases like I touched on we're looking we're hearing from many customers that they have both streaming data inline type of validations and data checks that may happen in batch so when you look at the end-to-end data quality on a Kafka data pipeline we have different customers and different use cases for both if you just do sort of a high view at this you've got the ability to be able to read messages you need to be able to provide some type of a sequel on that streaming message and then I need to say I'm going to push it to the good data so I'm going to push out these are the messages that are validated and then I need to identify some of those that are bad data so that I can then potentially do something with those failed messages I can send it to a workflow I can remediate it as well I can route it or internally so internal remediation or you know what I'm taking those bad messages don't post them to the to the consuming system I want them to go to a topic that only exists to basically post the bad messages or the bad data because I don't want that diluting mainstream tools that are then being used to make business decisions on so this is more of that inline type of check if we're looking at how do we do traditional types of data quality rules or even some of that more the more reconciliation rules that you may have a static reference data set or you may have some pre-built data quality rules inside of the tool and you need to be able to read multiple messages and sort of aggregate them and batch them together to provide an even profile of a potential cue you can do that as well so this is an example of I'm able to read small batches convert them to a batch provide some type of data quality on it again push out the past push out the failed or I can push them out to a single topic with that contain both past and failed with additional attribute on the message that says here is the status and it could say past it could be failed and then the downstream systems can then do their validation on it itself so you need that flexibility to be built into the tool so that you can create these new types of topics because you're going to be faced with these types of use cases whether it's today or tomorrow it's not going to be a fully real time based type of validation on your systems you will have some of this batch type of checks and then again if we're looking at some of this a more complex type of ruling so that you're saying I've got streaming data that's coming in I need to validate both streaming and non streaming data sources and I need to output it in a sort of a single swoop in a streamlined approach I may need to join non streaming data sources with streaming data sources so I'm taking non streaming data such as like this database and I need to join it with some streaming data sources and then output it to an external system or to a a Kafka message so that you've got your options on whether I'm capturing Kafka data, Kafka messages from a topic or I'm capturing files and databases in some of the more traditional types of data I need that all within a single tool so that I can have all of these options at my fingertips because you could have a large Kafka project that's going on today and that is the goal is to remove all of the files from any type of data transfer but in all reality if it's let's say a third party vendor that's providing you a file or you just have a CRM that doesn't provide any type of Kafka messages you will be able to be faced with some of these use cases that push in I need this reference data it's not streaming or I need this streaming data to then be enriched with something that is from a reference dataset so you will be faced with these type of use cases within the overall scope of how you adopt Kafka moving forward and again just to put in a plug Infigix provides this as a single solution to build out the data so if we're looking at data quality rules so when you're building out these rules we have over a hundred different type of predefined rules that go against a single type of attribute I mean we have everything from conformity, completeness, uniqueness, etc. we've got the major data quality dimensions covered so that you can easily implement them we have the built in type of validation in terms of the rules we also have the visualization inside of the system too as a single type of a mechanism to build out those dashboards and then above on top here we have workflow and routing in order to remediate and resolve those issues so that you can then do something with that failed data and again don't trust us we're hearing this from our customers today so it's not just Infigix speaking it's more of a representation of what we're hearing and responding to in the market right so we're working with an international bank with over a hundred billion in assets that they're looking at doing the reconciliation on their streaming architecture so that they're looking at streaming messages, they're looking at static data across their use case we're also working with a large financial institution that's delivering our Kafka capabilities as part of their general data integrity and data quality controls group so that's more of a traditional type of checks and balances and then one of our largest health insurers in the world and a 30-year Infigix customer we're working with them now to plan for their Kafka adoption in preparation for their event-driven architecture that's going through this new implementation within the next 12 to 18 months because they are getting beat on by audit to make sure that they have this validation in check and that they can prove out that their Kafka platform is fully secured and fully insured as well so some of my key takeaways before we open it up for questions is really organizations are moving towards adapting Kafka this is something that we've seen across all industries just as they move from mainframe to more of the distributed database driven world we are now seeing this in a communication adoption that they are adapting Kafka in a pretty rigorous type of adoption phase across all industries again the same operation of data quality issues as before garbage in garbage out so you need to make sure that the Kafka messages are being validated faster delivery higher data volumes lead to data quality increased quality issues if they're not managed properly so it could get out of control and then lastly our core message the entire data pipeline from to end must be validated monitored and approved for the data quality type of basics and steps and metrics that meet your companies needs from and in order to build out that trust and to optimize some of your streaming data investments as well so we're talking from the adoption of Kafka in terms of your organization but there's a lot of money that's being invested into those different types of projects you want to make sure that it is the highest success and having validation from end to end is really one way to have business buy-in and to get clarity and insights into your projects moving forward okay so you can find out more on InfraJigs.com we've got several infographics e-books, data sheets and blogs that you can use as a reference to find out more about InfraJigs some of our positioning on end to end how we deal with it as well you can also contact me my email is below you can contact feel free to contact me directly and we'd love to chat with you and find out more about some of your challenges in terms of adopting Kafka in your organization also or what you're seeing in the market okay I'm going to open it up for questions yeah thank you this has been a great presentation if you have questions for Jeff feel free to submit them in the bottom right hand corner of your screen in the Q&A section and just a reminder to answer the most commonly asked questions I will send a follow-up email for this webinar by End of Day Monday with links to the slides and links to the recording of this session so diving in here Jeff any lessons learned for streaming large structured databases for on-prem to cloud oh I'm sorry I was on mute because I was taking a sip of water so we're actually seeing a lot of that attention from the customers that we're talking to as part of their digital transformation strategy so they're moving a lot of their on-prem data warehouse some of their core business systems that is on-prem and they're adopting you know and we're seeing Azure really taking a large piece of that market share more recently than not in terms of what the big big companies are adopting so we're seeing a lot of that we've got a data lake on-prem we're now moving to Azure to move that data and they're building in these types of checks and balances to make sure that anything that is on-prem makes it to the cloud and again it's sort of that same left to right you're you then go into the producers are on-prem and the consuming downstream systems are on the right so we're seeing the same attention being drawn to what does it look like in the database then what does it look like when it's posted to the cloud whether that I just want to know that you know you kind of start macro and then move into the micro in terms of when I'm moving data I look at my database on-prem and I've got 10,000 transactions today it's moving to the cloud at the end of the day or just doing a check you know every hour and then I need to know what's actually been posted to the cloud that's at a more macro type of perspective and then you get into okay now that I do know that all the transactions were sent in terms of constant amounts let's dig into the context and apply business logic to make sure that nothing was inappropriately transformed or inaccurately aggregated so we're seeing those types of checks that are being implemented from our customers that are doing that got a large on-prem type of Hadoop data lake and then I'm moving everything to my private cloud and Azure in our system so we're seeing a lot of that I don't know if that answered your question but again if it doesn't feel free to reach out to me directly yeah definitely and that's great and Jeff if third party is providing their raw data as files what's the best way to ingest the data are there drawbacks to to lay on the data into a Kafka topic for cleaning and put the clean data into another topic so that's a good question so I mean there's quite a few mechanisms and companies out there that provide if you're just looking to move a file to a Kafka message that do that but they don't have a lot of that business logic type of rules engine in place so that when you are taking files from even if it's on-prem or from a third party it just depends on the criticality of the data that's being posted to that consumer I always like to say marketing campaign data is a lot less critical than financial statements in terms of regulatory reporting so you need to get every single piece of data and every single tidbit of data for that financial report prior to being posted rather than if it's marketing campaign we can sort of maybe be a little more lax in terms of what's being posted to the classroom system so send all of it or there's certain school of thought where you say send me everything that's bad just let me know what's bad and I can then work on it or work with the third party and again it's holding that third party accountable a lot of customers that we talk to say we buy data whether it's from a financial provider or some type of trading platform and we buy it and it's crap so we don't have any way of tracking it but we know we're paying a lot of money for it and we just need to hold them accountable so before they ingest that they say don't stop it don't do that inline validation where we're sort of playing bouncer where the validation is playing bouncer to say only let the good stuff pass let it all pass through but push some of it to the side and maybe redundant that storage at that time but let me know so then I can have that data at the ready to then feed it back to that third party provider and say the last file you sent me was only 80% complete and I have proof and hear the values I want you to go back and fix it that's really that ongoing type of improvement if you think back to some of the manufacturing continuous improvement initiatives you really want to stop bad data or bad occurrences that happens and then that way if it does happen again you know that it's fixed immediately and it's not going to be an impact downstream 11 I think we have time oh okay I was just trying to get myself on mute and I think we have time for one more question here any insight on handling privacy and security concerns in moving data to the cloud yeah absolutely just in that's just a general privacy type of concern whether it's GDPR some of the California regulations that are coming through that will only spread I mean security and privacy of data is here to stay it's not going away it's only going to get stricter so as you're moving data to the cloud you really want to be able to identify some what is PII what is being used for some of our PCI private or credit type of initiatives what is our HIPAA data so you need mechanisms and solutions and a plug here for infigix but you need a tool to be able to identify what is PII what is sensitive data you need to be able to know where it's at who's using it who owns it can the people that are using it use it in what manner can they be approved to use it is there a mechanism to allow approval to be used so there's all that type of rigor to make sure that yes data that's going to the cloud five years ago I would say the majority of customers we talked to said you know we're just dipping our dipping our toe and sort of navigating the cloud space now as a company as a vendor we do we do not see a single RFP that does not have what is your cloud option so with the structure around security and in light of keeping your company's name out of the news there is a there's a lot of security that needs to come into place when you're moving data from out from into the cloud but there's lots of tools to then to keep that you know security of transmission security of data at rest as well but yes there's lots of precautions to say the least well Jeff thank you so much for this great presentation and thanks to InfoJix for sponsoring today but I'm afraid that is all the time we have heard today just again a reminder I will send a follow-up email by End of Day Monday for this presentation with links to the recording and links to the slide. Jeff thank you so much and thanks to all of our attendees thank you