Welcome to Resiliency Radio with me, Dr. Jill, your go-to podcast for the most cutting-edge insights in functional and integrative medicine. I'm your host. In each episode, we dive deep into the heart of healing and personal transformation. Join us as we connect with renowned experts, thought leaders, and innovators who are at the forefront of medical research and practice. We want to empower you with knowledge and inspiration, aiding you in your journey to optimal health. Today, I have a very special guest, Dr. Leonhard Weinstock, on the topic of mass selectivation disorders and the gut specifically. It's becoming an epidemic and I can't wait to dive into this topic. But first, let me introduce our guest, Dr. Weinstock. He is a board-certified gastroenterologist and internal medicine. He is an associate professor of clinical medicine and surgery at Washington University School of Medicine. He received his medical degree from the University of Rochester School of Medicine and completed his postgraduate training and was chief resident in internal medicine at Rochester General Hospital. His gastroenterology fellowship was performed at Washington University School of Medicine. He is an active lecturer and has published over 140 articles, abstracts, editorials, and book chapters. He's given lectures throughout the world and he's currently researching our topic mass cell activation syndrome, also small intestinal bacterial overgrowth, restless leg syndrome, mold toxicity, and rosacea. And Dr. Weinstock, you and I might be the only ones who know how all those things are connected, right? Yes, they're connected for sure. So welcome to the show. It is absolutely an honor and delight to have you here. Jill, thank you so much for inviting me. I'm honored. Oh, thank you. Well, we got to talking because you guys are producing a documentary in the future at the end. Stay tuned because this topic is so important that Dr. Weinstock and his team are really trying to raise funds to bring awareness and we will be sure and give you all the information if you want to donate or support the cause. I want to be the first to say, I think this is so critical. But before we go there, let's talk about you. How did you get into medicine? What drew you to medicine and then what drew you into gastroenterology? Tell about your journey. Well, a lot of the people in my family just saw me as a caring person, an individual, not a businessman, and always encouraged me to be a professional. My dad was a dentist, uncle, a pediatrician, and so I had models. And so I always kind of knew I was going to do that. I was a little bit of a renegade. I did pre-vetinary medicine for a little bit in Vermont. But then after I worked on some dairy farms and worked with a family where one of them had cancer, I just saw a different side and I thought, you know, I could get into this. And I liked the people to people more than the people to cow contact. And so that's the way I went. And, you know, it did work out. And the specialty of GI was something that I saw during my residency, mystery cases and mystery symptoms and people going years and years with abdominal pain, where nobody ever figured out what was wrong. And I thought, oh, this is kind of interesting. Maybe I could get into that and learn more and get into the depth of it and then come up with answers. And then also you could look with endoscopes and colonoscopes and do a surgical side of things. So GI had a lot to offer. Wow. And what I see there is something I always see in some of the best physicians that I know. And that's the curiosity, right? Like the idea that you could actually solve problems. I think whenever we go into medicine and think we stop learning when we graduate, then we lose that curiosity and clearly with what you're doing and even the openness to mass selectivation and how does this affect the gut, our topic today. It really does take a curious kind of person to keep searching and looking because as you well know, the complexity of the gut and the body and our environment are really exponentially increasing. So let's dive into, first of all, just what is many of my patients and listeners have heard of mass selectivation syndrome. But for those who haven't, let's do an overview of what is this issue and why are we seeing more of it than we used to 20 years ago? Oh, yeah. Well, okay. So the mass cell is one of our white blood cells that lives mainly in the bone marrow. But when there's inflammation, a burn or injury, they come out of the bone marrow and go to the site and they orchestrate how much inflammation should be going on, how much new blood vessels should be created to help heal. But if one of these guys has a mutation on the gene that controls them, then it allows for these chemicals to come out of the cells and create a lot of damage and symptoms, both near and far. And so the fact is, it's incredibly common. It's been estimated that 15 to 20% of the country has some degree of mass cell activation syndrome. Sometimes people say, well, it's more of mass cell activation with symptoms and systemic syndromes. But those in the know know that you have to keep your mind open. And not only does it come alone, but it comes in with L.S. Danlos syndrome and POTS, postural orthostatic tachycardia syndrome, thus forming the evil triad. This has been a particular interest because at least in my clinical practice, it is just rising, especially since the pandemic. Let's talk about triggers though, because that may pull it to us, right?  Oh, yeah. I mean, first of all, there are temporary triggers, and then there are permanent triggers. So as often is the case, kids are sick with a variety of conditions at birth or in childhood, and they have, you know, headaches, asthma, eczema, food allergies, GI problems. And then in teenage years, women, young women have severe periods. Men, young men can have psychological problems, panic attacks, ADHD, and so forth. But they get worse along the years. And if they get infectious mononucleosis, that can set off a permanent change and more damage to the genes, making them more uncontrolled. And then during adulthood, things also occur. And unfortunately, during the pandemic, COVID occurred. And COVID also put patients with a genetic aberration and activation of the mast cell on a regular basis. So, you know, whether we can get out of this virus that lives in our body or not, and therefore it's affecting the mast cells. But that's what you're talking about in terms of what's happened since 2020. Yeah, it's weakened immune system, and also more, it's almost like two ends of the spectrum, right? We have more autoimmune, more activation of mast cells, more immune, like irrit, I always think of it as like poking the sleeping bear, right? There's these cells that are supposed to protect us, but there's all these things in our environment that are kind of irritating our systems more than ever before. Do you see environmental toxicity? Like, obviously, I deal a lot with mold. Do you see things in the environment affecting this as well? Oh, yes, we're getting ready to submit a paper on mold. This individual that is in the case report, he'd go to different houses. He was a home inspector and worked for a company that dealt with selling homes and knocking down bad homes and so forth. Every time he'd go into a moldy home, he'd have to run out of the house and he'd vomit, he'd have severe pain and having diarrhea and just imagine that happening on a regular basis. But some of these times, it was so severe that he went to the hospital and his gut was paralyzed. Well, it turns out that he did in fact have mast cell activation syndrome from years and years back and then treating that and keeping him out of those homes led towards a significant improvement in his status. So, you know, that the mold is incredible. It can activate good mast cells as well and you can get the chemicals that we see in mast cell activation syndrome, but it can certainly activate mast cell activation syndrome and patients can suffer until they get out of the environment and they do healthy things to their body that everybody who listens to your show knows about. Yeah. Oh, I love that you talk about that too because what happens, I feel like, is just the load of our bodies, whether it's infections or toxins or inflammation is getting worse and I think that's why we're seeing more. I want to go back to something you said that I think was really important and there's something that we were taught in medical school called mastocytosis and this is a proliferative disorder, right? You want to just describe because that was kind of the old thing that we were told to look for and to me, the mast cell activation, even though it's actually more prevalent, I think you can correct me if I'm wrong, been mastocytosis, but again, from the old school medical education, we weren't taught a lot about mast cell activation, but it's becoming a bigger issue. Do you want to differentiate that just a little bit? Oh, absolutely. Well, first of all, it's something that people could diagnose relatively easy if they thought about it, but it was like 0.3 per 100,000 people per year get it, so that's an extremely rare disease. It seems like in med school, they love the rare things, you know, like gigantism with growth hormone exes and mastocytosis, but really the fact is that mast cell activation syndrome was not described in literature until 2006. Oh, no wonder then. Right. Exactly. After I graduated and they don't teach it now, and that's part of our problem. That's what we're trying to do with this documentary and increasing awareness amongst physicians and people in general, so they could get help for decades worth of problems, and it is decades worth, but the thing about mastocytosis, it is a malignant disease. It can be indolent, very slow, and then you get all the symptoms of mast cell activation, diarrhea, hives, itching, rash, abdominal pain, brain fog, et cetera, et cetera. They're very similar in terms of symptoms, but if you do a bone marrow, you'll see many mast cells because it starts there as some malignant disease, whereas with mast cell activation syndrome, you see some in the periphery, in the, let's say, gut or lungs or bladder, but you don't see any in the bone marrow. It's very, very different. And we're seeing much more of the MCAS than the mast. Oh, okay. So, right. So if we're thinking mast cell activation disease, that's really three things. That's MCAS, that's up to 17% of the population, that's systemic mastocytosis, which is like one out of 100, less than one out of 100,000, and then mast cell leukemia, which is one out of 500,000. So that's the mast cell disease, activation disease, which are very different, but also similar with respect to symptoms. Okay. So MCAD is the umbrella over MCAS and mastocytosis and the mastocytosis leukemia, that makes perfect sense. And again, it's just for those, because we do have doctors who listen, so I want to make sure we're educating everyone because it's so important. So you work with the gut, but this is a disease that absolutely affects the gut, but it affects so many other systems. Do you know anything about percentages or how many, because you can do a biopsy in the gut to actually show, let's talk just a little bit about diagnosis. What would you do to actually diagnose mast cell activation syndrome? Okay. It's absolutely doing the main seven chemical tests, four blood tests, and three urine tests. The problem with the biopsies is that it's very common to define 20 or more per high power field. And there's some argument about that by the pathologists and others, especially the allergies. So when you're diagnosing mast cell activation syndrome, you have to have two or more systems involved with typical symptoms, whether they be hives or gut symptoms or itchy eyes, ringing of the ears and other problems. Two or more systems involved. Fatigue is a big one.  I think I read that top symptom was the brain. Maybe I'm wrong, but one of the biggest ones is generalized fatigue in that brain fog, right? Oh, absolutely. And then you add the third most common symptom being muscle aching. And that's the same thing as fibromyalgia. So you got to say, how many fibro patients are really MCAS? Right. And chronic fatigue syndrome patients, how many of them are really MCAS and so forth.  Now as a gastroenterologist, I see them being very, very common. But Dr. Affron saw patients as a hematologist with just a wide array of symptoms. And 50% or more had GI symptoms being predominant and significant. So the gut is affected in part because the gut is one of the main interfaces where you've got the mast cells living underneath the lining of the gut and you eat some gluten or dairy for many patients. And that activates the mast cells setting off the chemicals. So it's the chemicals that we measure. And that includes prostaglandin, histamine, chromagranin. And we do measure triptase because if it's high, you want to look for mesocytosis. But in the, there are two camps. One camp feels like triptase is very specific to mast cells, which is true. But in fact, it's not very commonly elevated in mast cell activation syndrome. And so the problem is when a person with multi systemic disease sees an allergist with hives and asthma and they say, I think I have MCAS, they get a blood test and it's triptase is normal. And the allergist says, no, you don't have it. And that's a real problem in terms of management. And the allergist also thinks it's very common to have anaphylaxis, but our group doesn't think so. And then there are three urine tests that are done, prostaglandin, histamine, and leukotriene E4. And so with those seven tests, my, about 70% of my patients who I think have MCAS are positive for one or more. And then the other group of, let's say the 30% that have negative markers, if they respond to basic mast cell therapies like antihistamines, vitamins, flavonoids, and they get better than they get have, they're allowed to have the label of mast cell activation syndrome. Hey everybody, I just stopped by to let you know that my new book, Unexpected, Finding Resilience Through Functional Medicine, Science and Faith is now available for order wherever you purchase books. In this book, I share my own journey of overcoming life threatening illness and the tools and tips and tricks and hope and resilience I found along the way. This book includes practical advice for things like cancer and Crohn's disease and other autoimmune conditions, infections like Lyme or Epstein bar and mold and biotoxin related illness. What I really hope is that as you read this book, you find transformational wisdom for health and healing. If you want to get your own copy, stop by readunexpected.com. There you can also collect your free bonuses. So grab your copy today and begin your own transformational journey through functional medicine in finding resilience. And we'll put that paper because I know the consensus statement that you have been an author on is one of the things that at least for me as a physician has been a game changer because you list all the criteria in that paper and kind of say let's shift from just triptase as the only thing that we look at. Is it true that histamine and triptase are going to kind of go up and down based on their acute exposures so that if you caught it during a acute flare, you might get it, but because it can go up and down. And then they're also very volatile in the blood and urine. Is that correct? So, so correct. Yeah. So for to address that, ideally when I do the blood tests and urine tests, I say, well, are you feeling poorly today? And many people are feeling poor poorly. And so that actually is most of my statistics of the 70% positive are because they're at baseline. And many people simply have fatigue or flushing when they see me. And so I'll do it.  But don't overdo it. So you're set back for a week. And then come in for the test.  So two of our blood tests have to be spun cold. And that can be done with little jackets that keep the blood cold in the centrifuge or going to the hospital. So yeah, so histamine and prostaglandin are the plasma. And then the urine has to be collected cold and kept cold and then frozen when you bring it in. So that can be mailed to the research lab that reference lab that looks at those levels. So and that's in your article, I know the consensus statement, some of the details. But that's real important if you're a physician or even a patient listening, if these things aren't kept cold from the collection, especially the urine, it's very likely to make a false negative result, correct? Yeah, absolutely. Okay. So the other thing you mentioned early on was genetics. How many what percentage of people have like, I'll just tell myself, for example, I grew up in a farm severe, severe eczema, severe allergies. And then, you know, Crohn's disease and gut issues and all this, I am sure I have all kinds of mast cell issues as far as genetically. Do you see the majority of patients have genetics that are positive? Do you even test those? Okay, so the testing for the kind of kid genes that are positive require a lot of blood and research labs. So there is one gene that's commonly tested for patients who have mastocytosis that's available. But that's really never positive for MCAS. So Dr. Muldering has described many different mutations of the genes. And you ask, why are we seeing it more? Well, is it because we're more aware and a little bit smarter? Or is it the epigenetics and all the toxins in our environment that are changing our genes after we are born?  And, you know, then we have both weird, we have a hyper immune state, but we also have a state where the mast cells don't know what they're doing in terms of controlling infections. And so people tend to heal poorly on their skin. They tend to go from simple viral bronchitis to bacterial bronchitis, and same thing for the nose and sinus problems. So it's kind of a yin and yang there. Yes. Yes, I've seen that because again, there's like autoimmune and this native immune activation, all the cytokines, which again, we saw on COVID, we see in mold, very similar. But then on the other hand, they're actually very susceptible to not being able to fight out intercellular bacteria or infections or things. So it's this very bizarre immune system dysfunction that kind of the worst of both worlds. So let's talk about someone who does have this. What would you do as far as starting treatment and do they require with you as a gastroenterologist? It sounds like you're doing tests and doing a great history and then doing interventions, but they don't all necessarily require an endoscopy or biopsy. Is that correct? Correct. Yeah, I've been dissuaded from doing that.  They want to be pretreated with intravenous, benadryl, pepsid, and some cases versed and some severe cases solumedrol. So you don't want to, you know, from the propofol get into a situation where you activate, get activated. I've had some patients just get into severe hives and it's been hard to get them out of that because they're very sensitive to chemicals. Yes. Yeah. Well, let's talk about that. We wanted to talk about treatment, but before we do triggers, you mentioned what are some of the common, like top seven or 10 things that you see as triggers and maybe those. Okay. Oh yeah. Well, I just put a table together and presented an article because I had a patient who for his 53-year-old man, 30 years of attacks of down pain, diarrhea, vomiting, and it turns out he was a pain salesman and whenever he had exposed long exposures to pain, well sales or conventions, all those volatile chemicals, organic chemicals made him flare. So that's a table that's going to be in hopefully an article that will be published soon. But basically it's environmental, it's environmental, infectious, chemical, implants, and temperatures. So a lot of the common ones are heat and cold. Dietary was another big one and most common ones are gluten, dairy, yeast, and high histamine foods. If you go and you've got some implant, that can be a problem.  Patients of mine, they go into Macy's and they encounter the terrible odor from perfume and bang, they're on the ground, passed out. Some people, like my first patient ever, she said that the big box stores were her kryptonite. She had both POTS and MCAS and she would faint going there and part of it was the fluorescent bulbs. Wow, amazing. It's almost like an ending list of things that could trigger. It's very interesting because back when I was diagnosed with Crohn's at 26 and of course looking back, I absolutely had mast cell activation probably from birth, but the Crohn's one of the biggest things that changed for me and I didn't even know the commonality was a low histamine diet. I just knew these certain foods were triggers for me. So more than any other thing that I did as a dietary intervention, the low histamine foods. And again, I just was like, oh, this thing fermented this and these other things and like aged meats and cheeses and bone broth and all those things, right? And avocado and spinach. And I had no idea back then. Now, of course, it's like, oh, it was histamine foods and it made a huge difference in the activity of the Crohn's, which again, probably was connected to severity and with the mast cell being activated. So very interesting. Let's talk about, let's talk about interventions first, like what do you do as far as natural and medications kind of go through the list because there's a long list of things. Absolutely three week diet trial.  Gluten, dairy, and yeast and low histamine is kind of what I say. And no, it's a little bit flippant to just say, okay, three weeks, you know, you got to do this, but you got to do it because you don't know how the medicines are going to work. But there's basically step one treatment for me are all but one are over the counter. So what I do is each one blocker, each two blockers of this in English, that's for Modedine. We used to have Zantac, but now we have pepsid from Modedine and there's two others that are harder to get a little bit. But sometimes one person does better and one versus another. That's the same thing that could be said for the non sedating anihistamines, claritins or tags, isole, Allegra, et cetera, the trade names. So those two are like the cornerstone, but then the vitamins C and D are important. So you want to get your vitamin D levels up to the ideal levels, 70 or so. And usually 1000 or sometimes 500 of vitamin C works better than the higher dose. And then a flavonoid. And if somebody really has terrible brain fog, then I'm going to go more with Luteolin. If there's just more body problems than body systemic problems, then Corsitin. And then the only prescription medication at this point, pretty much for all my step one patients is low dose naltrexone. And you do that with ketotaphone because that's a favorite. So ketotaphone is definitely the next step after that. Chromalin. So part of the step one does include ketotaphone, chromalin and cingular, but the problems are A, cost, B, you have to do chromalin very, very slowly. Otherwise it can react and activate the mast cells. And then you've got Montelukas cingular, which I'm not quite sure, but probably 5 to 10%, especially in mast cell women, get psychological disturbances. I would agree. I feel like that one's a wild card. Like it doesn't always work as well as it should, right? According to the mechanism. Right. Well, especially if LTE4 is elevated. And then ketotaphone. So ketotaphone is very good, especially if there's insomnia, can be helpful. Yeah. And then of course, there's some big players, right? Like immune modulator steroids. If you really, because some of these, in my book that I wrote last year, I have the preface is about a young woman who passed away and she had mast cell activation disorder and she was so sick. And I'm sure it was the triad, right? And I saw her way late in the game, but I wanted to kind of bring to the public the severity of these kinds of things. And I'm sure you've seen that too. So when you have someone who's maybe bedbound, in a moment, we'll talk about the triad, because that maybe fits in more with the triad and the disorder. I don't know about these really severe cases. How do you stabilize them? What are the bigger guns if needed that could be used? Well, you mentioned steroids. We do want to keep them off. If I have somebody with severe diarrhea, though, budesonide is a nice one because it doesn't get absorbed from the gut. And then steroids, if they have severe pain, they're in the hospital, then I'll give them my IV, the IV protocol actually created by Andrew Andy in California. Anyway, he came up with an idea of IV benadryl, IV pepsid, IV versed, or adivan, and IV ketaryl, which I'm not thrilled with because it can produce ulcers, and then a fluid challenge. And he gives those to his patients who have pots, but also MCAS. And it can be very helpful. So I have actually some patients who are getting intravenous fluids and the three meds intravenously to keep them out of the hospital. Or if they're in the hospital, I'll give that. So the steroids can be given with that complex as well. But again, the more and more steroids, the more risk. Yes, I couldn't agree more. And I wait those at the very end, but for those really severe cases, it can be kind of life-saving. And you mentioned versed and lorazepam and adivan. Benzodazben can actually have an ant, even though we don't love those, they're very habit-forming orally. They do have a mass cell, like an effect on the mass cell activation. Is that correct? They stabilize it. There's an animal study that proves that, and there's empiric data just from managing patients that it's a very good drug. I mean, yeah, it's step two on the step one, step two, step three would be Zolaire. So if you've got hives itching, hives, asthma that's refractory to therapy, seeing an allergist to get those, that drug, it's a shot once a month, anti-IGE can be very helpful. Excellent. And I've seen in my experience, and I don't usually prescribe that, like you said, I usually refer out to an immunologist or allergist. But in the Zolaire, I feel like it's just in my clinical, my small little experience, maybe 30% of people react to it too. Do you have people? Yeah, for sure. So it's immunogenic. So that's problem. But there's one good article about it that looked at multiple things, GI symptoms, and there was a 30, 40% improvement in GI symptoms, even improved neuropsychiatric things. And I wanted to talk about that if we have a moment. Yes, let's do. Let's go into that because it's huge, right? People don't. So let's just frame this because you wouldn't necessarily think mast cell in the brain, right? But obviously, huge, huge impact on these things. So dive into that. Tell us how that could present with mast cell. Well, one of the questions I have always asked is from, you know, once I knew about some of the manifestations is, did you have panic attacks as a kid? And so many people did. And then depression, I started taking a history really delving deep about depression, bipolar, ADHD. And there and Dr. Affron did a whole article on different reports. And there were many neuropsychiatric disorders associated with mast cell activation. And so we just published a paper looking, it was a case series of eight people who had a variety of psychological problems that were made worse by psychiatric meds. They didn't tolerate it or made worse, and or they were suicidal, some of them all their lives. And just diagnosing MCAS and treating them for MCAS got all their psychological problems better. So we're now going to do a more of a epidemiological study to see how often it is and, you know, how people have responded once they're diagnosed with MCAS. So for me, this is like one of the most exciting things that we've done because so many people tell you they react terribly to SSRIs and other things. Yeah. And to know that there's something else out there. And that makes so much sense. I mean, years and years ago when I was really studying just histamine, which is only one little thing out of the many hundreds of chemicals that mast cell produced. And there's a definite correlation with focus and lack of focus. And even IQ, there's a couple studies on IQ and histamine, which is crazy, but it definitely interacts with the brain on a profound level. You mentioned the triad. To me, this is one of the most fascinating things because as I've understood the triad, and this is what your documentary is going to talk about, it really pulls together some of the most complex cases that are mysterious as far as what's going on. Do you want to just frame what is that you mentioned before, but let's talk again about what is it and how do all these things connect and our common presentation? Absolutely. Okay. So the evil triad is MCAS, Ehlers-Danlos Syndrome and POTS. And either they run together just because they're common or more likely that there's etiology involved. And if you talk to any POTS expert who knows something about mast cell and knows a lot, they'll realize that about a third of the POTS patients are caused by mast cell activation because the mast cells live in the nerve bundles of the parasympathetic and sympathetic chain.  So if we say, okay, a third of POTS patients are due to MCAS, then we want to worry about hypermobile EDS. And it was Dr. Afrin's idea that the MCAS patients not only have inflammatory and allergic mediators, but they also have growth mediators. And that's clear from many of the patients can get subcutaneous nodules, fibromas, and also increased rates of cancer. So his idea was that the patients with mast cell who were secreting growth mediators would make the tendons and ligaments grow, which would therefore make somebody hypermobile. Interesting. And so that's one way to connect it. And so in studies that I've done for, you know, I studied restless legs in 180 patients of MCAS, how many percent had it, 40 percent, but 20% had EDS, 20% had POTS. So it gives you an idea just looking at a patients who had GI disorders came to see me, but I looked at whether they did or didn't have restless legs. The concordance rates was 20% POTS and EDS. So they're very common together. Wow. Now I have a question. And you might be the author on the papers I'm referencing, but I've read some papers on rosacea and SIBO and also restless leg and SIBO. And for me, it's been a big aha because so many of my patients, when we treat the SIBO, those things will actually get through. Again, you've probably been author on some of those papers. Where does SIBO fit in? Because in my again, just very humble clinical opinion, I feel like some of the microorganisms in our gut can actually produce excess histamine and trigger mast cells. Is there a connection between bacterial overgrowth or fungal overgrowth in the small bowel and manifestation of mast selectivation? Yes, there is. So I looked at 130 patients with amcasse, did breath tests on everybody. And most of them had bloating of abdominal pain and bowel habit changes. 30% had positive hydrogen and 10% had positive methane curves. And the inflammation that we get on the gut lining is something that's important because that is one of the triggers and that's on the trigger table as is SIBO that can make the mast cells and dwelling in the gut lining worse. So that's really important. Restless leg syndrome, likewise, I think it's a systemic inflammatory problem. We've been studying it for quite sometimes in dorfins or low inflammation is high in many of the secondary restless leg syndrome. And actually amcasse throughout that 40%, that's the highest positive concordance rate of secondary restless leg syndrome of 40 different diseases. Wow, it's very really big. And so a lot of my patients that I treat with LDN for mast selectivation, their restless leg syndrome gets better. That's reducing the inflammation. And that just doesn't happen. So there is the SIBO connection and maybe it's decreased parasympathetics. So you have decreased vagal tone and you're not flushing out the bacteria. And that's why you get SIBO in amcasse. So maybe it's the migrating motor complex and that small bowel motility that's actually impaired as the thing that causes. I think you're 100% right. That would make no sense. Because again, I see those infections often actually causing like the people who have the chronic viral load or a tick borne infection often have some issues with the small bowel motility and the vagal nerve and it seems like it's all connected. This is so fascinating. I'm just so grateful that you're out there doing the research publishing. And whether you know it or not, I've been quoting your work for years. You've been putting it out there and helping docs like me who to really understand what's going on. And I'm so grateful that you have been curious and keep looking for those answers. Obviously, we want to talk about the movie, the documentary where people can, you know, donate or help support that efforts. But before we do, is there anything else on the horizon that you've been studying, looking at maybe seeing some correlation? What's kind of the next steps in this area? Well, the next step is definitely the psychological issues in amcasse patients. You know, a number of us have large numbers of patients and we'll hopefully send out a bulk blind email. If you do or don't have psych problems, please just answer the questionnaire. And then we'll come up with an idea of frequency of amcasse in patients who have refractory or difficult to manage psychiatric problems. Some of the people on our website discussion group feel that most of their patients who have had bipolar disease have mast cell, which is really interesting. Yeah.  And I've often postulated, I wonder if maybe 95% of our mental illness is actually organic, whether it's inflammation, immune dysfunction, mast selectivation, other infections, you name it. I think there's so many times where it's actually an underlying cause versus just something that you were born with. And I'm sure you're seeing that too. So where can people find your work, your research, and then we'll talk about where they can donate to the film. Okay. Well, under gidoctor.net resources, there's a number of little slide shows, PowerPoints and articles that I've written. And then what we're doing with the documentary, we're trying to make it a learning experience for everybody, whether they be medical professionals, something to open the eyes on patients who have had to deal with symptoms for decades and are now finding answers. And then we explain what's going on in the movie to about mast selectivation syndrome. And then we are doing something unique. We're tying it to a library, an online library with papers and PowerPoint presentations. And that way we help patients and potentially doctors who are interested. We have to get the doctors interested. We have to get this into the medical schools. It's just absurd that it's not. And as far as where to go, it's mcastfun.org. So it's m-c-a-s f-u-n-d.org. No matter how little or much you choose to donate, that'd be great. It's a not-for-profit movie. And so we're not making a dime and we're putting a lot of efforts in. We've filmed half of it, but now things have slowed down and fundraising. So anybody anywhere can even a brick laying for $10 would be fantastic. Well, I am one of your biggest fans. And I think this is so critical. Unlike you, I think that we need to educate our fellow physicians because it is an epidemic. And the more docs we have that can help us treat these patients, the better. So I am just, I love that you're doing this. I'm your biggest supporter. I'll try to help you get the word out. And I'll be right there with the donations as well. If you heard this, if you're driving your car wherever you listen to your podcast on the show notes, we'll have these links. So don't worry if you've missed it, you can find that on my website or anywhere you've listened to this podcast. Dr. Weinstein, thank you for your heart, your curiosity, your work in the world and your ongoing efforts. I know this isn't easy because you also have a full clinical practice, but we are so grateful. And thanks for taking your time today to talk to us. Well, that's a wrap. Thank you guys for joining me with this wonderful interview with Dr. Weinstock. I hope you will check out mcasfund.org and support this documentary. I think it's just a critical effort that we need to educate the public and other physicians on the prevalence of mass selectivation disorders. And I know many of you who suffer from that will appreciate it. But we hope you've enjoyed this show and I hope you'll stay tuned for more empowering episodes with new episodes I release every week. You can find them all on iTunes, Spotify, or wherever you listen to or watch podcast. And you can find the complete transcripts and more information and links on my page, JillKarnehan.com or on YouTube. And if you want to have links to any of the products that were mentioned in this show or others, just go to drjilhealth.com. Thanks so much and I'll see you next week.


Hi there guys, welcome Thank you for coming to my talk today. If you guys just listened to Burton speak maybe a couple of you In the room next door. He's a my colleague at chart beat I'm a platform engineer on to a chart beat. He works on the data science team but we handle building the systems that chart beat runs on and Chart beat is a pretty cool company. We're scrappy 75 in Union Square And we build a web analytics product that is used by tons of big publishers Across the world anytime you load up your favorite publisher probably we're on there unless you use ghostry and We are tracking you we Sort of pioneered this idea of engaged time where we think that people should care about the amount of time that People are spending on their articles and reading their content as opposed to just paying attention to page views Which of course is what apps metrics and so forth are based on Now at chart beat we handle really big data 300,000 requests per second about 50 billion page views a month We like to think that we have the most data per engineer since we're only about 25 or 30 engineers But we have really big data problems and picking databases for us is a problem because they could be either expensive or not performant Sometimes that's not the case sometimes when we find a happy medium, but other times we don't so we have one Particular database out of many many databases that we host Out of our 800 EC2 machines One of those databases alone cost 15 grand a month to operate and we do so with Mongo running these three I to 8x L's which I think is the largest instance class we can get on the old AWS EC2 system So pretty expensive very beefy machines Mongo likes to have big machines to work correctly. I should preface that this is Mongo 2 Mongo 3 has some performance benefits. We could probably scale this down a little bit But in any case we can see here that with our database that we've built We're able to get the same sort of performance guarantees and handle and process the same data for about two grand a month as opposed to 15 grand a month. It's pretty cool So what is Wade this thing that we're talking about today? It's a distributed fault tolerant horizontally scalable database So lots of words there And it's a database framework and I'll talk a little bit in a minute about exactly what that means But it provides strong consistency and high throughput by using a replication Strategy known as chain replication. This was done out of research done about 10 years ago in Cornell And there's been a couple other implementations that use chain replication But we'll talk more about what that means in a bit out of the box Wade supports something called command replication So basically when replication happens, we don't actually send the objects across the network We send sort of little pieces of information that tell the next node in the in the chain what to do with that object We can solve what's called the read write update loop Which is a problem that we face when building databases oftentimes not the first database to solve this But we do make it pretty easy to do so Ultimately clients and users of the database write query and update commands written in Python that live on the database Wade is built in Python, which for better or worse. It is a database. You hope it's really performant So we have decided so forth for now since it's more like an academic thing to keep it in Python So it's easy to understand by people who aren't used to writing databases You know other folks you use scala and earlaying and all kinds of fancy languages to make their stuff We're just saying let's keep it simple so To frame what Wade is and what it's not it's not an all-in-one solution. That's why we called it a framework So to give you an idea about What that means let's talk about a database we might be more familiar with my sequel So what are the parts of my sequel? We've got our drivers and clients the things that we interact client side to talk to the database There's a part to my sequel that understands things like replication Making backups partitioning the data the sort of administrative layer that also handles any user permissions and so forth There's also the query parser in the planner The sequel language and grammar on top of which my sequel is built a superset of sequel We also have things like stored procedures and last but not least the layer of sequel that understands the data on disk as it's written And in the case of my sequel you have multiple options for choosing your storage layer Now where does Wade fit in with this? This is a kind of a loose analogy. So it doesn't really map out but basically With Wade you have to implement a couple features that normally would be given to you namely we have to define the methods the in Python how to mutate our data on the servers and We have we are responsible for writing the data to disk if we want to we could keep it in memory Or we could use something else now laying out the data on disk You might be saying oh, I have to keep track of file pointers and locations and binary data Maybe but there's a lot of great utilities out there that help you do this called like level DB rocks DB These things that you hear about when you're choosing back ends for databases you can use those tools With Wade to help you define design your system So without further ado, I do want to do a quick example of how this thing works in practice So it is a distributed database framework and as it's a framework. We've created an implementation of this framework Which is an in-memory key value store. So this is an implementation of Wade again Wade You can build lots of different things. This is just the one that's easy to demo So in our example here on the right-hand side I have two nodes that are sitting here doing nothing at the moment and I'm going to restart those nodes right now So it's distributed. So we've got two replication factor two We have two copies of our data and then I configure those nodes and tell them to talk to each other so they know about each other and on the left-hand side here I have my ipython terminal ready to go to instantiate a client to talk to the database and The interface is pretty simple So I've written some code that we haven't seen yet that has that lives on the server and we're interacting with that code right now So we're going to send a couple commands to the server One is to get some data and the other ones to set some data now as I said before everything in Wade is either query or update Method and here at the beginning of this client. Can y'all see this finally? Yeah So We are querying the data so we have client query Our method name is gets this is something I created. I named it the get method That zero number represents the partition ID Wade relies on partitioning But we'll talk more about that in a moment But right now we're just talking to the zero partition And then we have our arguments for our get function, which is just a JSON dictionary in this case We just have one key or the one thing that we're trying to do which is get a value So we're just giving it the key which is Adrian and then last but not least we have a little bit of metadata Which is sort of a tag to identify the application to the database right now We're not doing anything with this but in the future This might be useful to say have metrics break down by application at the database layer We often have this problem where when we're trying to debug performance issues Knowing who the colors of our database is pretty useful and looking at things like network traces doesn't really give you much information So we're gonna try to get our value and there's nothing there I just restarted my database. There's no data because it's all in memory now Let's actually set some data. So the syntax is very similar except now it's an update method The name is set. We have our zero partition We have our arguments to our set method, which is a key and now a value of 12 for whatever reason and then a debug tag Again included we set that value and now we get it and it's 12 and on this on the right hand side here We can see our two nodes. I put some strategic print statements there To kind of help us along as we can see both nodes in our cluster got that update and that was replicated Any questions so far? Okay, so this sounds like kind of magic So let's actually look very briefly at what the actual key value store looks like so we're implementing an interface the store Interface that we give to Wade we register this thing with Wade It runs on the database and so when we execute that update command called set It executes this function here given these arguments for now Let's ignore the object ID and the object seek stuff really we just care about the arcs and as we can see in this in the line here We're doing the self dot data equals or dot object ID K equals V. That's our key value store Remember, this is our example is just in memory. It's a bad example We probably wouldn't keep this stuff in memory. You wouldn't want to replicate it in memory key value store Maybe you would But you could easily Initialize this with a level DB database and then talk to the level DB database instead and create a wrapper around it So that's our key value store. It's pretty simple I hope that gives you some insight as to what Wade does so behind the scenes Wade is handling things like replication Message passing between the nodes. It's handling partitioning the data just so long as we tell it what partition to write our data to and Aside from that and that nice easy to use interface Wade is built on top of something called chain replication Which I mentioned at the beginning So chain replication this nice handy excerpt from the paper really kind of summarizes what it is is that it's an Approach intended for supporting large-scale storage services a database is that exhibit high throughput and availability without sacrificing strong consistency guarantees most databases that we're aware of they Sad they either give us one or the other they give us high throughput or they give a strong consistency things like sequel You can tune it to be very performant, but it's best for consistency It gives you strong consistency always and that's kind of its main feature that we blow in love Things like for example, RIOC if you have ever heard of that database Extremely high throughput massively horizontally scalable, but eventually consistent. We lose out on the strong consistency We can ask RIOC for strong consistency, but with a major performance benefit penalty So before we talk about chain replication something that we're more more familiar with Which is known as a primary backup strategy to replication Drivers clients and so forth interact with what's called the primary node solely Riots go to the primary node and then in parallel the primary node sends those rights to backups Almost always in parallel. This is how mongo works for example mongo calls these primaries and secondaries This is pretty common in databases in general if you've ever used a database you've probably heard of this So and as you can see here, there's like a little back-end which represents the actual storage engine that you might be using if it's sloppable and One thing that to note too is that the reads they're not replicated obviously But they go only to the primary and I have this dotted line here on the right-hand side that says reads from backup There's a big caveat there is that if we read from the backups we lose strong consistency guarantees You could imagine a situation where a rights went to the primary and a read Simultaneous right after that went to the backup, but before the backup has acknowledged the right You'll be reading stale data certain scenarios. That's perfectly okay. You know your data model You know if that's acceptable, but you do lose out on strong consistency Chain replication on the other hand we always get a strong consistency and we get high throughput now with chain replication It's a little bit different our replicas are instead of being the backups being sort of like siblings of each other We have a strict ordering of nodes and The driver sends these update commands these rights to the head of what is called a chain So this these three nodes here these colored nodes they represent what's called a chain and then rights are sent through the chain and Each node subsequently forwards it on to the next in order So the head always receives the object of the message first and the tail is the last one to receive it And the tail is actually sort of the arbiter of truth It's the thing that's actually committing the data and it's the thing from which we read the data So the tails kind of like the primary here It's the place where we decide whether a right is successful or not because that's where we read from as well The benefit here is that we get sort of like we can have each of the nodes in the chain sort of be Processing something while more messages are coming through from the top So we get really good throughput in this example, but we don't get we kind of sacrifice latency a little bit So in our primary backup model, we're writing to our replicas in parallel So the maximum amount of time it takes for the request to be successful is the max latency It takes for that one of those replicas to respond in the case of chain replication It's the sum of the latencies between the head body and the tail Now the replication factor is configurable both with Wade and with chain replication the case here We have replication factor three. We have three replicas three copies of our data We could have two in which case we would have no body just ahead in the tail And at least Wade you can even run it with one node The node can simultaneously act as the head and tail for the command, which is pretty cool I think that kind of sums it up As I mentioned before Wade does partition your data. It's pretty common in databases. You have to tell the database how to partition your data Databases like Kafka Redshift just name a database a lot of them have this idea of partitioning in Wade They're called objects, which I'm kind of maybe thinking we should change that Because makes hope makes you think of key value stores, which is not what this is but they're named objects because basically it's a replicated state machine and That's a chain as a replicated state machine, which are called objects. That's what we talk about So an object is sort of like a chain. So the ordering of nodes is is a chain You can have multiple objects live on the same chain. That makes sense and Lordy skipped ahead so But we have to tell it how many partitions we want also kind of like with Kafka if you ever worked at Kafka We're kind of making up the number of partitions. You should you should have right now Right now one of our production systems running few thousand partitions like three three thousand partitions or so And that seems to work just fine for us The number of those partitions or objects is limited for the lifetime of the system. So we say put a lot in if you need them So let's dig a little bit deeper into chain replication and talk about what happens when a message actually goes in So if that wasn't clear in those arrows before hopefully now It's clear as we follow this cycle that the message goes into the tail and then comes back out Through the head, but we have to make this full trip through the system for the right to succeed now Chain replication kind of has this two-phase commit thing going on with it that when the messages come in to the system When they reach the head they are not actually written immediately to the disk Such as with primary backup rather they're added to a pending set in memory We're sort of recording the intent to write the message But we don't actually do so until finally the message reaches this tail and again the tail sort of the arbiter of truth It's it represents the known state of the system And so once the tail says this right was successful I've committed it it responds successfully to the body the body's like cool you were successful I'm gonna commit my data remove the command from the pending set and then return to the head and so forth And eventually it goes through so on the left-hand side as the message to coming into the system We are pre-committing and then once the message is on the way out of the system We are actually committing it presuming. It was a successful right Sure, what's up So if something goes wrong I'm gonna say that I'm gonna I'm gonna tell you what in a little bit what goes on But just know that if something goes wrong This is fail-safe and there is an algorithm to read to know what's wrong and like say like okay It'll make more sense in a little bit. You had another question over here somewhere. Oh Yeah, yeah, we're gonna talk about that a little bit because it is a little bit like how does this work exactly? So this is where we're talking about it. So how do we actually like Order the messages and send them through the chains and know like okay Is it is it appropriate to remove the command for the pending set and store it? Is it okay to commit? Well, basically we're we have to introduce a timeline So that as I kind of mentioned briefly, this is a replicated state machine So if we have some state that we're operating on which is like the object or values or whatever our you know Key value source doing or you know not key value sort rather what weight is doing what we've asked is to do We need to make sure that the commands that we've created that live on the database are Executed in the same order on every node if they execute in the same order on the on every node given the same inputs We will get the same output value on every node in the system and everything is is synced up together So when a message comes into the system It reaches the head and the head is responsible for assigning an incrementing number to what is like called the sequence number of that Command so it's an incrementing number per object that represents the nth update of The state of the command so for example when the object first starts out its sequence is zero It hasn't been updated ever and then if we make 10 updates to it We say we called our set command 10 times the sequence number is now 10 and The head is responsible for receiving a message and saying oh, yeah, I got a message from a client I'm gonna assign it a sequence number and then once that sequence number is assigned every node Downstream in the chain the body and the tail they check what they know The current sequence number is and they make sure that the new command has a sequence number That's exactly one more than the one that knows about so it says that hi. I am the body I just received an update from the head the update has a sequence of 10. I Currently have sequence 9 10 is good 10 is the next logical step But if I were the body and I got a command for 11, it'll be like what happens at 10 I missed command number 10 something's wrong with the system and It's possible at this point that the head and the body have diverged in which case we need to execute some Algorithms to restore the state of the system and basically use the tail Which again remember is the is the source of truth we ask the tail Hey, what is the latest sequence you know about and in which case the tail overwrites the state of the other nodes? So this is sort of the more complicated diagram of this. I don't want to spend too much time on this but basically Wade is that the nodes themselves are generic They ask a series of questions to know the state of the object and as we see here We have three nodes the top little green box as a client or upstream node could be either it's generic And then after the body of the know, there's a downstream node potentially so as we see the message enters the system We have some command in the top here. It's a set command. It's for the object zero, which is the partition We've got our arguments Adrian Actually, this is bad arguments. It would be a key and value and all the other stuff We talked about before and then we our object sequence We don't know it could be assigned or not depending on whether it came from the client or if it's coming from an upstream node So it enters a system. We check if it's valid We ask if the object sequence has been assigned if it has that mean it's came from an upstream node If not, we're the head. So then we ask are we the head of the chain? Yes or no if we are the head of the chain It's now our responsibility to assign a sequence number. So we pair up the store and the pending set We ask both of them. Hey, what's the current sequence number? You know about for this object and then we add one to it and that becomes our new sequence number Now our object has a sequence number attached to it We add it to the pending set the node then asks itself if it's the tail for this chain If it is the tail then we're done. We know that we can commit and that's why we go straight into the commit step If we're not the tail that means that we're not the end of the chain We need to forward the message onwards and that's why we potentially have a downstream node And then if the downstream node responds successfully We also commit the message and if not we receive return or rejection upstream the system either to a client or another upstream node any questions there I Should point out to that the sequence number is something that while I'm talking about it right now It's not necessary to understand how Wade works. Wade just asks that you keep track of it So Revisiting the key value store now we can look at our get and set methods again if we look at our set method We notice that we have this object ID that's object sequence That's the partition and the object sequence is again that number that's always Incrementing and so all Wade asks you to do is store it just store that object sequence for us and be able to retrieve it And so in this case we have a very simple this seek map Business here, which is storing the current sequence by object ID. So that's pretty simple I've omitted some methods here that you would need to implement to say give Wade the object sequence when it asks for it But that's pretty simple So enough about chain replication any questions about chain replication before I move on Yes So in the case of let's suppose that a Network of partition occurs somewhere in the chain such that the tail is still accessible from the client, right? But the head may not be in this case if the if the message if there's a network partition that occurs then If the data makes it to the tail, that's good The data is correct. If the data doesn't make it to the tail then it doesn't commit So it's so what would have happened is that there would have been pending commits In the head and the body, but they never would have been removed from the pending set They would still be pending so the state of the system is still consistent Well rather so Well, let's let's be clear here. So in this diagram of chain replication The the writer the client writing a message into the system It that respect that request to write does not return until Every node in the cluster has committed So it's not like I send a message and then the heads like Immediately responds and says great. I'm gonna handle everything asynchronously. It's all synchronous So the availability thing is something that's a little bit questionable. I haven't talked about it too much But chain replication does provide availability if you implement it as per the paper I'm kind of missed a big step here Which is that all of the messages are actually being proxy in the original chain replication design? All the messages are actually proxied by a master node and that master node can then detect whether a chain is up or down So I said it goes to the head in reality in chain replication At least as the paper describes it the messages first go to a master node and then farms it to the head so what means we maintain high availability because the master node immediately knows upon writing Whether or not the chain is is up and then can execute a series of commands to bring the chain back into it and Bring it back up. That's kind of That's really it's interesting stuff, but it's not really kind of part of the conversation here. You had a question Um, I honestly haven't we're we're still in mongo to we tried to upgrade to mongo 3 But we had a lot of issues doing that It's probably just unique to our particular case in our data model But you know, I don't want to say that our thing is better than mongo because mongo is a great database Rather, I want to say it's comparable to mongo. So for us we were able to get really good performance and We were able to beat mongo, but that's not to say that it's the best option Mongo is a great data general purpose database. Wade is a little bit heavier Lifting you have to write specific code that lives on the database. There's a lot more code. You have to write You use Wade when you need this sort of like I have very specific query pattern that I need to optimize for Does that answer your question? So We talked about that so enough about chain replication Let's talk about another feature of Wade that you can build in yourself This doesn't necessarily come with it, but it's easy to do with Wade is To eliminate what's called the read write update loop So oftentimes especially if we're talking about mongo if we need to make a change to an object That involves two round trips to the database We need to first read the object onto the client side serialize it stream it over the network Mutate it somehow and then write the object back to the database. So this can be kind of expensive We are putting potentially network strain because we're having to travel this huge object potentially huge objects across the network And then, you know, we have this sort of decoupling of where the storage is and where the code is and so forth But with Wade what we can do instead is that since the storage the update commands themselves live on the database The database can actually keep the value on itself and we just send the command that says Execute my function mutate. So this is very comparable to say stored procedures in SQL Which you might be familiar with although more powerful in another way because instead of SQL we write Python Now This is this has some benefits because the object doesn't have to travel the network we don't have to saturate the network anymore and Another sort of subtle thing here that we can solve is an issue of consistency If two clients happen to read the same object at the same time and then write it back and Overwrite the right of the other clients that was concurrently occurring. Does that make sense? so if the mutation method happens on the database and it happens in a very specific atomic fashion then We don't have to worry about these, you know multiple readers act to save the same objects I'm not having we don't have to do any locking basically. So we have a lock-free way to update arbitrary data So that was the whole read write update stuff something kind of similar to that although very distinct is this thing called command versus value forwarding so With in the case we were talking about before We're we don't have we're avoiding the case where we have to stream the object between the database and the client Here we're talking about avoiding having to stream the object between the nodes During replication. So when a node when a command enters the database and it's being replicated We have two options Again, the mutate method that we talked about it's living it lives on the database So we have a couple options here one we can Mutate the object on every node in the database and just forward the command on Between the different nodes just say here's my set command. Just do it and the next node in the database says okay I'll just execute the set command again That's what's happening on the left-hand side every node is executing this mutate method now on the right-hand side We have something called value forwarding instead of sending the command over the network We send the value which is probably what we're more familiar with and typically, you know How regular databases work that don't even have you know database sides stored procedure kind of stuff We just set the value on Now the benefit of both of these is sort of subtle on the left-hand side Because we're sending the commands over the network They might be really small compared to the values or the objects on which we're operating in which case We can avoid all kinds of network traffic by having to read the object and you know send it across the network The downside to this is that every node has to do the same computation So if your command or update is very expensive to compute maybe something like that, you know NLP stuff that Burton was talking about You may not want to have to do that on every node you might want to just have it do it once And then you know the other knows just have to say okay. Here's my value. I'm just gonna store it So there's a trade-off here on which one is better in what scenarios right now weight supports command forwarding We assume that that's probably more useful in most scenarios There might be situations where value forwarding is useful But the most useful scenario is that is where the computing the value is very expensive But on the left-hand side again just to clarify we save having to send the value across the network We just get to send this nice tight little command Okay, any questions there about command versus value forwarding What about sharding Yeah, it is sharded. Yes, so weight is called So this is so the command and value forwarding stuff is built into a way You don't have to think about it. It's just given to you Does that make sense? So you don't you're not in charge of it. You just The only thing you have to tell weight is what partition of a specific object belongs to so you have to tell it Hey, my update command is going to the zero partition or the third partition or the 10th That's all you have to do is just choose the partition Okay, I think I might be this understanding of question Maybe Well, I'm gonna say that for now the migration to another language is probably Not gonna happen. It might happen if this thing becomes a thing, right? And that's a big if For now Python is great one thing that even though we said, you know, Python may not have performance Well, it can have performance if we see libraries and stuff like that We can gain an immense performance using things like level DB to back your back Wade is an excellent choice All the level DB or at least most of the clients are written in C So yeah, you get to write Python and talk to C Wade is also built on top of a framework called Pi UV, which is based on lip UV Which is in a very efficient event processing framework for that We're using extensively code is a little messier because we use it, but we gain immense performance Any other questions about command versus value forwarding? Okay so Last but not least I kind of mentioned it before Chain replication has this idea of a master or proxy node Wade our design is a little bit different. I don't know if it's right or wrong But instead of having a master or proxy node the client directly talks to the head of the chain avoiding that extra step and What we've done is say it is built an asynchronous process called the overlord which talks to the chains itself and detects failures So Basically when a chain when the commands in a chain might be out of order because based on the object sequence You know the node receives an object sequence and it's like sorry. That's not the one I'm gonna accept it's not one more than the current one. I have I'm gonna stop accepting updates. Well That's great. It means that the database doesn't go into an inconsistent state But that's kind of bad because it means it just stops working. So the rights just happening So we have this other process as overlord that's periodically checking all the chains and trying to write to them And if any of the chains that can't write to them for whatever reason it starts to execute a series of commands to handle Reconfiguring the chain. So for example, if like in a chain the body You know something happened to the body in a chain and the head and the tail are still available It'll just remove the body and then make the head and the tail be the new chain But you know, you might want to like add a new node in there and replace the body and stuff So you have a lot of choices there. This is not part This is part that hasn't received a lot of love yet So we definitely need to think more about how it works, but currently it works It's just not the most performant thing. It takes a while to bring this chain back up From being unavailable and in order for us to be available like chain replication talks about we're gonna need to solve that problem Any questions there on the overlord? I'm gonna skip this but that's basically like the the gist of what the nodes think they are That's like here are the nodes listed This is my little local example and here are my chains or my objects or partitions We have two zero and one and then they're composed of node zero and one and one and zero and the ordering of that is the head and the tail So the first node is the head, the second node is the tail This is I'm gonna really skip past this briefly But this is basically like how do we decide where our objects live and our partitions live There's a lot of good research that was done on how to help you determine where your partitions should be There's some work here you can look at And that's pretty much it I'm not really Python Our client is if you've heard of G event It's G event compatible under the hood. It is multi-threaded So if you don't have G event, it will use Python's thread and look threaded library implementation Only threading is scary in Python, but it's okay if you're just waiting on IO, which with a socket you are So I'm not sure I haven't used twisted before But we've used it in a variety of different like applications For example, we've used it on web servers before and it works You know it writes in parallel and you can use the same client object and share it with all your you know You could use you can instantiate one client and use it on your web server. That makes sense How is the latency so I can't really say I Don't have a lot of comparison numbers with me But just based on the way that it works like the theory behind it Wade is gonna have a higher latency So what reads will be fast because reads go to the tail That's just one node look up there, you know really simple Like all pretty much all other databases When you do the riots if you recall from our chain here, let me pull back a little bit We have to wait for each of those red lines Before we get a success of the driver whereas with the other database Same thing except those red lines are in parallel So it's whichever one respond first so with this model we get slightly better lower latency The difference though is that you trade off latency for throughput So basically the solution to getting more rights per second is to have more writers So you spin up more writing processes So instead of making you know one client writing 200 I'm making this up one client writing 200 requests per second You'll have two clients writing a hundred requests per second each, but overall you still get 200 requests per second Okay, what's up? So that's something I really didn't talk about here Just know that that object sequence number that I talked about gives us a very good state and that the tail always represents The true state of the system so suppose in this example that the body Something bad happened to the body in this case. We would just remove the body and The head and the tail we would just tell the head and the tail. Hey by the way instead of There being a body you're just talking to each other now. That's really simple now suppose that the Suppose you wanted to add the body back in What we do is say because the tail is the state of truth. We would just tell the tail send all your data to the body Okay Right now there is there are ways to minimize the downtime But there is some and the downtime is by partition So that's why we feel like you should choose a lot of partition IDs So that way Presuming if you have a lot of partitions each one is smaller and if each one is smaller the system can more expeditiously bring it back online So, you know, I said it having two objects and it takes forever for one of them to come up You have a thousand and each one can kind of come up slowly, you know, you can get better availability It's something we're still tackling because it does feel like it should be better than it currently is Yeah Yeah, but I mean it's pretty good. It's still alright It's it may be not be as good as some other databases, but again, we still get high throughput and Still get strong consistency, which are great, you know Any other questions for the crowd? Hi correct So basically the reason why at Charby we can get away with this is because the clients themselves don't actually write data to us That happens all asynchronously so, you know Reading the data is the hard part for us. We have Okay, if you probably saw our dashboard at the beginning I kind of gloss over this Which I think this is actually kind of cool. You see this line here. This like graph thing So this is by the way, this is Gawker. They're partners strategic partner of ours. This is their dashboard And I don't remember when I took this screenshot, but we've got like some of their top articles that are here Pokemon Go is a government surveillance Psyop conspiracy And they had at the time they had 12 about 11,000 people on their website this graph here that you see These are the number of people on their site over the current day So I took this around 11 a.m. So they haven't reached the peak of their traffic yet But as you see this graph here, this is the database the database that powers the graph is the thing I was talking about the thing we're replacing and so We have several thousand customers With several thousand webpages that we keep track of so it's really it's a lot of data for us to keep track of and Mongo Just wasn't doing it for us. So That's nothing against Mongo. Just our particular data model just needed something that was more hand-tuned and customized Correct Some Yes, yes, exactly Correct correct so for us at chart beat The size of our data is bounded to some extent because it's a time series and so we roll data off so Yeah, this isn't Unfortunately right now. This is not resizable. Just bad. I know But you know for now it's yeah, it is what it is Yeah Yeah, yeah, yeah, no, no, no, no, it's more of my own insecurity So there is a there is a couple considerations under the hood the database itself is storing Python dictionaries of Mapping data from partition ID to you know something so if you chose more than max ant partitions It would not fit in memory. So there are some practical limitations Correct so all the objects are Basically the code is written such that if the object doesn't exist yet Then it uses a default value of zero and then if it does it actually writes the data So this is kind of my configuration. This is static. You choose this thing you create this file And then this is the this is the truth henceforth now in the case of reconfiguration You know like the overlord might change some stuff. This will change But this is like sort of the canonical truth of the system. That was a good question Any other questions? No Yeah, yeah, it's really not an issue whatsoever For better or worse our current production system for whatever reason we decided to do replication factor two Which I know if you have like sensitive data, you'd be like, oh my god, you need three But with two, we don't even have to worry about that latency because it doesn't exist for us Any others? Thank you guys wasn't my box