's like, sometimes we feel a bit sick of the fact that we're having to wait for things to happen.  But it's not happening quick enough. We want things to root. We want things to take, take shape for things to happen that we have, you know, once kind of put out there. Okay. So there is going to be a lot of words that you need to say back. Okay. So it's like voicing back. Okay. It looks a very scary card. But if you imagine swords are thoughts, all these thoughts coming at you, everybody trying to tell you what to do. Everyone trying to give you that advice. We've got that theme going on the beginning of the week. And this is you saying, I'm ready. I'm ready to give it a go. I'm ready for whatever the universe brings to me. And if it means that I've got to get going with something, you know, that I've not got a hold back, then that's exactly what I need to do. There is going to be a decision about money. Okay. You've got two sevens here. Sevens are usually the crossroads times. You've got the seven of ones, and you've got the seven of pentacles. Okay. The ten of swords. So these two cards really are saying that you have got so much that you can do, you know, there is so much. And all you have to do is to pick the direction that you want to go and focus and have it, have it, you know, and it's like you're very determined. So maybe you've waited long enough, okay, for you to have the motivation to start that new project, to start that new enterprise, something that you, you've said, I'm gonna, I'm gonna, I'm gonna, I'm good. And then it's like, I've done it. I've done it. I've started it. I've done it. Okay.  I'm doing this for me. So by the end of the week, you've got the temptation card, which is kind of that naughty energy, mischievous energy being tempted to do something or not do something, you know, that card just says, you know, that maybe you've got to focus on what it is that you want and don't get, maybe don't get distracted. Okay. So you've got the hanged man, you've got the eight of cups and the sun. Okay. So the hanged man means that everything is a little upside down. And that goes along with the waiting card. Okay. That goes along with that one. Okay. So this is coming out at the end of the week as well. Okay. So hanging around, you know, maybe you're hanging around, waiting, waiting, waiting, maybe in this time of hanging around, you might have learned a new skill though. Maybe you've met the right person who is going to follow you down a pathway that you're meant to go. Maybe the waiting has allowed you time to balance yourself and get you ready. And so the waiting could be linked with before you met someone else. It could be before you started that new job or project. It could be, you know, so whatever has been waiting, there's been a reason for it. But now it's time to let go. Now it's time to say, I'm done. I've had enough. Can't do this anymore. Okay. So you are either letting go of something or someone or a situation or anger. I'm letting go alongside the sun, which means that's a very good thing. Because the sun means a boost of confidence. The sun means you're doing it. Okay. So so yeah, alongside the temptation, sometimes there's a temptation to carry on waiting and to be patient. And other times it's like saying, no, I've done enough waiting. I'm on it. I'm sorting it. So yeah, that's what I see. So I'm going to bring up the cancerian barrier now, the banner. And let's have a look and see what Vera's got to say about cancerians. I needed some coffee. For those that don't know, it's 2.25 am. So they're really feisty. That's again, I think the same group of cards that jumps out. For cancer, from the 16th to the 22nd of May, 2022. Let's see what the shameless dreams and they keep going at it sticking to and again, again, again. Okay. So for cancerians, it's going to kick off the week. And it says feast of plenty, choices and the consequences. And I'm a first of all, and really drawn to that beautiful big clone, shiny. So a lot of opportunities. And yes, there are two sorts. And there's a crystal also. Like, it feels like a crystal. And feast of plenty. Maybe you've been had been looking for something, maybe a job, a house, and maybe even a car, you couldn't find the car that needed. You just couldn't find it.  And you gotta, okay, this is what I want, or this is what I want. And yes, not every choice will be the right one for you. But so think of it carefully, but feast of plenty. Just look at that, all the opportunities, put them together, sort it through, and then make, start making choices. Okay, this is way out of my league. This is getting better. Sort them out, and then sort again. Then make it smaller and smaller until, but sort from the heart. So you know that when it keeps popping up, like I had in a previous reading, about a week, and my eye kept on seeing, excited, excited, excited. That had a meaning three times in a row. So some opportunities, you're going to put aside again, but just, there may be so many, you've never experienced that before. Maybe you had more one or two choices, would I do it, would or not. Only you got a lot of them. And yeah, sorting out, middle of the week, two cards came out. I'm first going to show you this one. The dust devil, the dust devil, moving out of stagnation. Now that can be, well, what I said in the beginning of the week, when you've been waiting for so long to even see any opportunities, job wise, money wise, a new car, maybe even friends.  But keep in mind, while you're going through it, because yeah, if you had never had the opportunity, like, yeah, now I've got all those opportunities. Oh my God, you start sorting. And slowly, you're moving away. All of a sudden, you start to get excited. And when you go through it, the universe is your partner. And yeah, this is this beautiful light. And when I move it a certain way, sometimes I just see a slight glimmer of a heart. That's the universe is talking to you. So, together with what you're already doing, moving out of the stagnant energy, that's also, you felt, well, if you don't have any opportunities for a job or whatever, it's like, you can feel like the root girl, the disowned self. But everything you've gotten on your path has had a reason.  You had, some of them were not good, others were better. But you're going to move out of that stagnation.  Because when you all of a sudden get those opportunities, and I say, again, that can be on any level, on any ground, money-wise, job-wise, or just hobby-wise, your heart's compassion, all of a sudden the opportunity comes, and then you need some time, the divine feminine, just to be with yourself in the cosmic wound. Like I said in the beginning of the week, the universe is your partner. And that's what I got for Cancerians. Lovely. Thanks, Vera. You're welcome. So, we've got a lot of messages here. The truest, realest one is coming. Ego aside, you still have your identity. It's true. It's true. Yeah. Thank you, Vera, great reading. Totally resonated with me. Interesting times to come with our gifts. Yeah. Oh, and the question who sent you to something for PayPal, that was for the exchange for reading. Yeah. Yeah. So, yeah, we're doing some readings later. Well, I am. I don't know if it's going to be very late for Vera because I've got to have a little break in between this. I really have to, so I won't be doing it on this video, okay? So, I'll come back and do readings later. So, I'll leave it up to Vera to decide. So, Countess Blessings, you bless beloved beauties. Oh, that's lovely, isn't it? Thank you for your empowering time. We love it. There's, well, Leo, Sarge and Scorpio, all new beginnings. There was, wasn't there? New beginnings for all of those. I don't even remember the cards. I've, oh, no. Beautiful reading, Ellie, powering ahead. So true. Moving on. Culling and cleansing. We're uniting, yes, with people. Rest and rejuvenation for many of us. Thank you. Yeah, we do. We do have to. There you go. Thanks, Vera. Good one. Crab here. Mary Chain. Okay. All right. So, I'll let you have a little rest while I, you're not going to last out for readings, I don't think. But anyway. I never know. So, my first mark of coffee and I only had a few sips from it. Okay. All right. So, Cancerians. We start off the week. That's going to be obstacles and challenges. Okay. Is it an obstacle or is it a challenge? Okay. It all depends on the way you look at it, really. You know, we can see things as negative or we can see things as positive. And I think this is going to be a time to do that. Okay. So, it's going to be, we're doing it. Okay. I'm just going to treat this as a challenge. Okay. Something's come up and I'm going to get through it. Okay. Now, this may mean that you have to let go of something. Someone, something, this card has come up many times in these readings. Okay. So, just letting go walking away card. The eight of cups. It's like, I'm done. I've done the best I can. It's obviously not good enough though. So, I'm done. Okay. It's time to be selfish. Okay. My princess of wands is I want it and I want it now. You know, that's kind of like justice. I want justice. I want to be able to have a good outcome. Okay. And then you've got that I'm not talking card. The two of swords. Sometimes we have to keep things to ourselves. Sometimes we have to kind of admit that maybe you know, life isn't as easy as it seems, you know, and sometimes these obstacle and challenges have to come up. Maybe you don't want to talk to a certain person about these obstacles or challenge. You just want to get on with it and do it yourself. That's going to be how it feels at the beginning of the week. So, hello, Megan. This is my eldest who's canceran. Okay. And Stephanie out here. This is canceran as well. Spiritual union in the middle of the week. So, connecting with people of like minds. Now for some of you that may be linked with love life for others that could be meeting of a friendship, connecting with somebody who you would normally connect with, you know, but not very often. And that is one of those connections. There is going to be issues about secrets. Okay. The high precess can mean secrets, secretive energy. Okay. I am dealing with it. I'm dealing with it. I'm, you know, the most sarcastic card in the deck in any deck of cards is the three of swords. It means, oh, sorry, I'm going for it. I'm getting to that. The beginning of the problem and I'm going to blast that problem. Okay. You then have this, and that is a meeting or it's somebody who you connect with and that seems to be a meeting or it seems to be something that you need to get sorted out. Maybe you need to make something happen. Okay. Maybe that meeting is with that spiritual person. Maybe it's got to be secret. Maybe you've got to get to the bottom of what is actually going on between the two of you. Maybe the two of you have got something that you want to talk about that is secret. Maybe it's about things that maybe aren't meant to be known by anyone else. Oh, Phil. Thank you, lovely. Phil, thank you. Oh, that's very kind. Okay. So this is now the end of, okay, the week. Okay. So the end of the week for Cancerians, you've got the truth energy. Okay. Feeling as if I've got to know the truth. Remember, you had that kind of secrecy card, the middle of the week. And then at the end of the week, you have the truth card. So maybe all of that secrecy is coming out. Maybe you then know what is going on. Okay.  I need to meet this person who is going to be for my future or this person has got information about my future that I need to know about. We've got the two of cups, which means meeting somebody of great significance. The same as that card is. Okay. Now that can mean romantic, but it can mean meeting of the minds. It can mean somebody who knows you, you know them. It can be a past life connection. It can be somebody who you just, you don't know how you know them, but you just do. Okay. There's going to be a lot of nagging. A lot of people trying to tell you what they think you should do. Imagine the sorts of thoughts, all these thoughts coming at you. You're going, I'm not listening. I'm not listening. But sometimes it's like those, those kind of comments do get to you. Just don't mean pretend we don't hear them, but they do. Okay. You've then got that, I'm dissecting things. I'm dissecting what I've got to dissect. I've got to find out the truth about this new person that I'm linked to or I'm getting close to. You know, is it a past life thing? Maybe you want to know, why am I so drawn to you? Is it a past life thing? What is it about you that I need to know more about? So all around one person close to you. Okay. Cancerians. Very interesting. Did you have a past life with that person? Did that past life do something? Just don't mean, which means that you've got to come back here and maybe change things in some way. Ooh. Intriguing. Okay. Intriguing. So where are we now? We are, we're going on. So we've done the water size. We're now doing the air signs now, aren't we? Okay. So we're doing Aquarius. Okay. Over to Viola. So let's see what the Shane streams have in store for Aquarius for the week from the 16th to the 22nd of May 2022. Aquarius. Let's see what I have. Ooh. Again, two for the middle of the week. Soak for Aquarius. Well, for the beginning of the week, this car came out and I'm just drawn to the beautiful bubbles that come up. Beautiful light. Deep diver. And diving into a task in which tells me that there's something you're very compassionate about.  Like you're going to go full for it. And which is needed to get it completed, to get where you want to go with it. And that can be that you need to do some research or just starting a new job, maybe home, but you have an opportunity and it's gonna, you're going to dive really deep. But don't be surprised that you're going to run into some obstacles like a wall, which you're going to have to break down to reach where you want to go. And this one came together. The last compass, getting back into integrity. Maybe that what you've long been willing to do for a long time. You're finally able to dive really deep into it. Diving into that task. And yes, that needs some time together with the obstacles.  You wanted to go for it. You have to get back into your own integrity to be true to yourself, to actually be you in the things you need to do. And that's why they throw up some obstacles. You need to clear them. But I don't see them big obstacles. It's just a little glitch. But that feels like they want to know, find out if you are actually really serious about what you want to do. So they need to put up a little obstacle. And that tells you to just getting back into your integrity. And so it's going to be a pretty week, a little bit all over the place. Because by the end of the week, the hollow bone, teachability. And with that part of diving into that task, with the obstacles to force you to go back. And I feel that that word force you to go back into your integrity is there to teach you. There are things to be learned from it. And no matter what age you are, it's all about teachability. The right timing. And I'm more drawn to the bottom of it than to the top. Because, yeah, it's, I think it's more about a week where a bundle of really big lessons can be learned. And some things will be repeated.  And that's what I got for Aquarius. You ticker it, you were just ready. I was just doing that ticker, but I couldn't do it whilst you were reading because it would have put you off. I forgot to put me ticker at the bottom. Don't worry about that, Ali. Oh, it's done now. Okay, so let's do for Aquarius. All right. So Aquarius. Oh, thank you so much for the super chat. Vera, can you say thank you to this lovely person? What's the name? Svetlana. Thank you so much. So much for the super chat. You might nearly have enough money to at least half of it, I think by the time, you know, we've finished here. So this is all going towards Vera's flute. Okay. She's nearly got enough money for her real flute. All right. So she's got a flute.  She's getting on really, really well. And so, yeah, good. So thank you so much for the super chats. We really appreciate it. Okay. She's very moj. All right. So Aquarius. Okay, let's have a look at Aquarius and see what's going on. Just concentrate there. Right. Okay. So let's have a look. So we've got, I've used the wrong ticker as well. Tick or take. We've got enough to change that in a minute. Anyway, we start off the beginning of the week with the psychic tarot cards. You've got this one, which is the base chakra. That looks orange, but it's actually red. Okay. So this is the base chakra. So the base chakra or chakra, whichever way you want to look at it, is all to do with money, security. It's used to kind of that, you know, anything to do with reproduction, harvest, just sort of mean spreading the seed. And I know this sounds rude, but it is linked with the male organs. Okay. Spreading the seed, you know, getting what it is that you want, kind of the animal instinct of wanting to grow things. Okay. That's what that energy is. And that's what energy they're showing you at the beginning of the week. Okay, Aquarius. So think about things that you can do now that are going to take root now and going to bear fruit later. Okay. So it could be that you're now, maybe the stuff that you've done in the past is now ready to bear fruit now. It could be that that's what they mean, you know, here. Okay. So things have been upside down. Things have been kind of like on hold with the hangman. Everything is on hold. But this says that, you know, I'm bored of waiting. I don't want to wait anymore. The Leo character says, where's the action? Where is it? I want more. And so this is saying it's getting you down. The weight is getting you down. You've had enough of waiting. And so with the base chakra card, that is definitely you are fired up and wanting to get on. Okay. You're not going to be putting up with everything being on hold for much longer. Okay. You're going to take charge. Okay. The beginning of the week, Aquarius. By the middle of the week, we've also got another chakra card. There's obviously seven in this particular deck. Okay. So this is the heart chakra. So this is to do with your love. It's to do with emotions.  It's all going to be part and parcel of that. So you've got an idea. There's emotional overflowing. Okay. Of emotions.  And you thought, I've done with this. I've done with all of that energy. Why is it coming up again? What is that about? Why, why, why? You then have the moon card, which is your psychic ability. It's that psychic ability. It's like, I just know. I know that I need to cleanse. I know that I need to get these emotions out of the way, because partly I know that somebody new is coming into my life, or I know that I'm going to be going on an emotional adventure. Okay. And that's going to be important to me. Okay. So lots of that focus of, you know, that emotional energy there. There's going to be a meeting. So that meeting can be doctor, solicitor, bank manager, estate agent. There's some important meeting coming up. It could be that special date. It could be meeting that special person. It could be meeting a person that is going to be giving you that information that is going to be vital for your future. Okay. So there's going to be important meetings linked with your heart and your emotions. And by the end of the week, we have the new beginnings. Okay. So new beginnings energy is where you're just about to start, start a new adventure. So in the tarot cards, this is the full card, which is I'm doing it. I'm going for it. I'm not going to hold back anymore, because my gut feeling is telling me it's time. Okay. So the whole of this reading for Aquarius, it's all about no more procrastination, no more dithering, no more saying, I'll do it tomorrow. This is it. This is your future. And you are literally delving into it. Okay. So you have had obstacles. There's no doubt in that you've had obstacles that have frustrated you. The nine of ones indicates that there has been obstacles. But the rejections maybe have been the obstacles, being rejected, being treated badly. Okay. It could be that one person has always put more energy in than the other in relationships. Okay. With the six of cups. But what's happening is that you are in charge of your own destiny, that you don't need the approval of someone else. If someone else isn't giving you emotional stability back, sit on with it. You don't need it.  You are the person that is going to make yourself happy. And then if someone comes along who can make you feel special and give you all of that love, that's great.  To be happy before you meet that special person is really important to find that balance. Okay. And I feel that this, whether this is a relationship, whether this is to do with finding your life path, you know, it's like, yes, you might have been rejected in the past. It might not have worked out in the past. But for now, that's it. You're ready to go. Okay. So, yeah, that's what I see for Aquarius. Let's get the banner up for Gemini. Gemini, Gemini. There we are. There. Okay. Over to Vera for Gemini. I had to put my coffee aside. I was doing something. Okay. For Gemini's, for the shaman's dreams. And I gotta say, we are all sacred dreamers, dreaming the world into being. What will you dream? So this is about something that you have been dreaming about. Deep down somewhere for some hidden, and it's wondered out for Gemini's upcoming week. And that one to stay out. So for Gemini, for the beginning of the week, and it says, preparations come to fruition, which it says the netcaster, you know, those fishermen, when they have made a big net, they just throw it out at the right spot. So they, when that net is sinking, and they pull it in, they can have, when they have prepared everything properly, and have the right spot where a lot of fish, where they throw out the net, they will haul in a lot of food, a lot of abundance. So this tells me that Gemini's, a lot of, I heard a lot of Gemini's have been preparing. And all that preparation is coming to fruition. It's going to be very, very, very beneficial. And gonna be, I got a feeling, abundance is on your path. And that it's also, yeah, like you're going to finally be able to breathe. Because in the middle of the week, the first breath, the beginner's mind. And which tells me that everything you have been working on, everything you've been studying for, information you got it. It's been a net net, you pulled it in. And you finally be able to breathe. It's like a new beginning. Everything is opening up for you. So that's good thing. And this could be because of the last card I pulled about your spirituality, your intuition has been getting better. Because the last card that came out, the watch is transpersonal. Yeah, I'm drawn to this, like there's a big magic going on. The connection with the universe has gotten stronger. So yeah, there's a lot of change, positive change. Like, and I'm literally like, I can breathe. I'm like, finally, that wasn't such a word in between, but that's okay. So yeah, I think a lot of good things for, for, for, for Gemini's because yeah, all the preparation, you can finally start breathing while you're picking the fruits of your abundance, because all the preparations you've done. And it's kind of, yeah, it's going to open up. It's, it's, it's, this feels really about spirituality. So I got a good feeling, a lot of Gemini's that have been waiting, oh, it's ever going to come.  So that's what I got for Gemini. Lovely Vera, thank you. Love that Mad Max. Next time I'll have to put your details up on the banner as well, Vera. So yeah, but in the chat, she can put up her calendar. So if anybody would like, I mean, Vera does this nearly every week, she does this. At the goodness of her heart, she does this.  But when Greg doesn't steal my spot. Oh, that's funny. I like, I like that he's doing it, Ellie. I do like it. I'm wanting to read with him for such a long time. He's going, no, no, I can't do that. It's like, you read differently to I read. He's getting more confident. He is, he is, definitely. I mean, that's what he used to do. And I love in our global predictions. So if you wanted to follow us with our global predictions, that's what we're doing, ask us a question about what's going on out there. And we're doing that about an hour, an hour and a half before we do our date night every Tuesday. So you're welcome to come see us for that. Okay, so let's have a look and see what's going on with Gemini's. Okay, so Gemini's when I see the 16th through to the 22nd of May, you've got the intuition card with the psychic tarot. Okay, the intuition card. It's like something's telling you that the door wasn't open to you. But now it is. There is, it's like your psychic ability has now kicked in so far that you can now see what was once not open to you. Okay, so it's like your abilities will all of a sudden you'll wake up and you'll go, wow, they're just talking to me now. They're talking to me every day, every day. I'm asking a question in my head and they're answering it. You know, everything is coming about now. Whereas before it, it wasn't okay, it was, it was like you'd have to sit there and meditate and you'd have to do this and this and this. But it's like now it's just they're talking to you. So really concentrate really on, you know, what are your thoughts and what are their thoughts because they do feel different. Okay, so and I feel that's what you're going to be doing. Okay, so we have the 10 of cups, which is feeling brilliant, feeling centered and happy with yourself. And what I was saying in last reading, okay, and that was for aquariums. Sometimes we have to be, I don't know if it was Aquarius actually, I'm not sure. We have to be ourself. We have to be happy within ourself. Because then if there's another person that comes along in our life, they are a bonus. They are not there to come along and make you happy. Yes, they make you happy, but it's not their sole job. And sometimes we just have to say, do you know what? I'm happy within myself. I'm centered and happy within myself. This is just an added bonus to me. This completes me. That's what it's like. It completes you. But if you're waiting to be rescued, if you're waiting to say I'm unhappy, I'm unhappy, I'm unhappy until that person comes along into your life. You know, that's going to be a lot of pressure for that new person coming into your life. It's going to be, and they might feel that pressure, that maybe why you don't get that second date.  Maybe somebody who, you know, isn't happy themselves and desperate for that, for that relationship that can scare any person off as quick as blink. It can scare people off. Some people have got enough going on in their own life. They don't want to baby another person. And unfortunately, I know that sounds quite harsh, but my guides want to say it. Okay, so linked around Gemini energy, but also this is a message for everyone. So then you can start seeing clearly, once you found that connectedness, that bold connectedness that is, this is who I am, like it or not, I can't change. This is who I am. So once you get to that stage, you then have this positive energy flowing into your future. It's like, I know where I'm going. I know what I'm doing now, just to remain and know what I want. Because sometimes it's like, we don't know what we want. And we're kind of going into something a bit half-hearted and, you know, and wonder why it doesn't work, but it might not work because you haven't fully understood what it is that you're doing, what you're doing for. Okay, so this to me says, I am now standing in my ground and I'm not going to back down because I know what I'm doing. I know what I want and what I desire. And, you know, if somebody challenges me, then I will stand up for myself. So those three are very positive cards. Okay, from feeling as if I was a bit lost to what's going to be happening to you, Gemini's next week is going to be from black to white, white to black, whatever way you want to look at it. Okay, it's like completely opened up. The intuition is like spot on. The door is open to you. Okay, so by the middle of the week, it's going to affect every area of your life because we've got movement, choices and decisions. Okay, so once one door opens, once one energy clears and it reveals everything else, it's like, that's it. Okay, go, go, go. So you may have connections with the past, understanding more about past relationships, maybe finishing up old existing energy, okay, that need to be changed. You know, perhaps that energy was, this is stagnant energy, but we need to kind of put a lid on it and say, no, this is finished. I know we've kept in touch, but this is done. I don't want to keep you on the back burner, just in case I want to go back, I'm going to say we're now done. So there could be kind of like finishing things, finishing old relationships, kind of like putting those, you know, well and truly in the past. Putting up that boundary around yourself is going to be very important as well. Now all of this, if it's not about love life, remember, Gemini, this could be about how you feel about yourself and how you are treated by other people.  Who do you want to kind of blend with?  Do you want to blend with somebody who is fun and there's lots of laughter and, you know, they've got a good energy there?  Okay, so this is to do with your wisdom. You're being very wise this week. Your wisdom is beyond any expectations. It's like something has clicked inside your head, which is just really making you very happy sincerely. And then comes the world card, okay, which is the universe in this deck. But this is where things have turned out better than you thought. It's like, well, didn't expect that, but brilliant, excellent. Okay, then comes money, security. Okay, security comes from feeling secure inside. It's the mirror image. We bring that to each other. The wheel of fortune. Okay, everything turns and moves on for a reason, you know. And yeah, it's been a battle. Yeah, it's not been easy. But look, look, you've got through it. And, you know, there's lots of love and light energy coming your way. So, yeah, keep up the good vibes. Good vibes equal good energy coming to you. Bad vibes, scratchy vibes, anger, guilt, resentment, fear, that's all going to grow and get more of that. You get more of what you give out. You're creating your future with every single thought that you have. Okay, so it's, it's difficult, but you can do it. You can do it. And you've done it. So, okay, so this is our last one now, which is Libra. Okay. And over to Vera, the Libra. I'm finding my joy. They all said one day I will find my joy. And I felt if I'm finding it back. So, last one of the whole bunch. So, let's see what the shaman's dreams have in store for Libra. 16. Oh, that one went out to the 22nd of May 2022. Let's see. I need two more cards. Thank you. So, Libra, that one. Okay, let's see. Oh, it's going to start really good. And it feels like you've been waiting for the opportunity. And it's going to be there at the beginning of the week, the father gate. The gate has opened. And this is going to be about you to take that bold step forward. Maybe, and I feel like that was a little bit too early. Maybe this is about your spirituality. That you're finally saying, well, everyone is telling me I'm ready because that gate is already open. But I'm not stepping through that gate. Well, I got a feeling for those Librians that that week is going to start a kick off with that bold step forward through that gate to show yourself the way you are. And by doing so, you also got a dust devil card moving out of stagnation. Because when you've stayed longer than everyone thought you needed, well, you didn't feel ready. So the energy was a little bit stagnant, stayed in a safe place. But this week, I strongly feel that for a lot of Librians that father gate, that is finally, that you're finally ready to take that step forward and doing so. It's like, you see that dust cloud? That's the dust that's been blown out of that stagnant energy.  I literally hear while I'm talking about it like, that's gone. And so be aware that when all of a sudden your feet don't feel like there are, you have to pull them out of that mud that is really sticky that you don't fall over when you take the next step. Because when you put, because there's no, you don't have to pull, you can walk, just relax because that stagnation is gone. And a little have to say that stagnation will be gone. And that leads you to the end of the week. They say feast of plenty, choices and the consequences. Well, one of the choices is you took that step, both step forward through the Vardigate. That dust, well, all the dust has been blown off. You're moving out of that stagnation. Those are choices because when you choose to step forward, you'll find the consequences. Well, yeah, there are consequences. Well, they can be negative. But I don't feel that this is negative.  And yes, these are taking in one is fully lighted lit up. And that's the energy that you're going to feel. And it will take some time. Slowly, you feel comfortable, you get more energy to do more things. Instead of that you have to drag yourself forward or push yourself to do it. It's going to come a lot easier. And that's what I got for Libran. Thank you. Oh, and the night, Mary Jane. Night night. Thanks for hanging out with us. Appreciate you. Mary Jane. See you soon. And our clear skies, Sweden. Yeah. And cheeky. Mr Prescott. Cheeky cheeky. Silently listening. Yeah. Okay. I got to the end when I thought, oh, wait a minute. Here's Libran. So we're going to be doing one to one readings. Well, I am definitely doing one to one readings. I've got my dinner on at the moment. So I just need to have something to eat. And then I'll come back on. So about 15 minutes after this one, we'll come back. So if anybody wants a reading, please let us know. And it's by donation. Okay, tonight, just donation tonight. But I'll do the link. It will pop up on your phones. If you've had the notification for this one, then you'll have the notification again. So it will be on all of these. Okay. Notifications are getting better. They are. They are. But obviously, nowhere near what it should be. If Greg's got millions of followers, and we only get 30 people, 50 people tops, just I mean, coming in, then yes, we are definitely, definitely a shadow band. But hey, we have to make the most of what we have. So, okay. So this is Libra. Okay. So Libra Energy, we've got Material Harvest card. Okay. Here. All right. So this card is, you know, everything is there. Everything is ready. Okay. So sometimes it's like you've planted the right seeds at the right time. That could be those seeds could be anything, really, could be a job. You know, you've trained and learned the right things at the right time, ready for this specific job that's come up for you. You know, something is happening and making things click into place for you at the beginning of the week. Libra, let's have a look and see what the tarot have to say. Okay. So, right. It's time to break free. It's time to break free from whatever has been restricting you or holding you back. Okay. No more of that holding you back. You know, this is how you've been feeling. It's like, well, which way do I get, how do I get out? How do I get out of this situation that I'm stuck in with the 10 of ones? And the just judgment card says, we're breaking free now. Okay. That's it. And stop listening to gossip. Stop listening to any doubts that come into your mind that maybe you can't do this or maybe there's this reason why you haven't done this or that going on. It's like, yeah, it's like, wow, what's going on? What is happening? So that is now all change, all change. Okay. So now you have this, the material harvest is coming in, especially the beginning of the week for you guys. So we have had this card where you've had to sacrifice things in order for the greater good. And like I said in about the shadow banning, yeah, we sometimes you've had to, you know, it's like, for the greater good, speaking your truth, even though you are penalized for speaking your truth, what would you want at the end of the day to stick your neck out and to awaken more people to what the truth is? You know, and this is why it's like, I for one will always whether, whether he's, he's my husband or not, I'm going to stick by Mr Greg Prescott, because he has stuck his neck out. And that has cost him dearly cost him dearly, but he's done it anyway, and he'll carry on doing it. Obviously, he has to be careful with this channel, but he that's what he's done. He's sacrificed. And so maybe this is you, maybe you've sacrificed something in order for the greater good. Maybe you're being asked to do that around the middle of next week. And you've got to kind of decide what you're going to do. Okay. So the four of swords is stand your ground stand up for yourself. Stand in your truth. Okay. And the moon card says your psychic ability is is very strong that you're able to you're able to to know what's best. Does that mean it's listen to your higher self, listen to that. There is people around you. And I feel this might be to do with maybe children. The princess card can also be young people, young energy. And so maybe there's standing your truth going on.  I don't know where that comes in, but look out for that. Okay. Around the middle of the week. And the end of the week, patience and planning. Okay. So there are ways to get around certain issues. So find your way, you know, be patient and plan ahead. Plan what it is that you need to do and plan how you're going to get there. That's my goal.  Everybody's had this car. Well, most people have had this card. The reconnecting with old. Okay. So it's reconnecting with old souls. Family that you've had, galactic family that you've had. We're all reconnecting now. Maybe for this last this last point, this last connection, you know, to that bad energy, you know, we've all come together to battle through it, to get past it, you know, we do it all together. The nine of cups. That's I think that's what it's meant all the way along with these readings. There may have to read to me read meanings. And that could well be, you know, you know, people from your past, literally, okay, people from your past past lives, or it could mean galactic energy, getting people together. There is a character around you around the end of the week, who is being a little bit tough. Okay, there's a little bit of kind of forcefulness about them. You might be feeling as if, you know, they're being a little bit too kind of heavy. You know, you don't particularly want to, you know, connect with them. Maybe they're just too bossy. Maybe it means about money. Maybe there's a lack of money. Maybe that person owns you money. Maybe there's an issue about that. You don't particularly want to reconnect with them. But at the end of the day, fair is fair. Okay, so you've been patient. You've been planning. Okay. But now's the time. It's time. Okay, so that's what I see for Libra. We did it. Hi, guys. So thank you, everybody, for hanging out with us. We do appreciate you. Thank you for all the super chats. That is for Vera's new flute. So, yeah, it's a gift. So thank you, everybody, for doing that. I hope that's all right, Greg. So, yeah. When he says no, I know he wouldn't. He wouldn't. He loves you too. He loves you too. I love you, baby. I love you. Okay, we're going to go away. Vera, how are you feeling? At this point, tired. Okay. Well, I'll send you the link and you'll either come in or you won't. It'd be up to you. Okay. I get it. I understand. This is my kind of like early evening kind of time for me because my body clocks on a different level altogether, really. That's okay. I made it a little calculation. And if it does go the way it exchange rate can all change, of course. I'm over half, way over half. Yay. Oh, that's lovely. You deserve it, Vera. You deserve it.  And I really want your business to flourish. So if you want a reading with Vera, she's going to put it in the chat. So, you know, she's very reasonable. And otherwise, if you cannot find it, when you do it in search bar, Raven, Wolf Star readings, my draft booking site shows up. Oh, that's good. So I checked that. So it's Rising Wolf. No, Raven, Raven, Raven, Wolf Star, Raven, Wolf Star, Raven, Wolf, as in the Wolf Star readings. But when you the first three words already, yeah, your guides didn't let you know, didn't let you remember the rising part. No, that's why you're Raven, Wolf Star. And I thought, wow, that's a good clue to it. Raven, Wolf Star. And then I heard a scream. Readings. That flows. All right, lovelies. I'm going. Bye for now and see you back later. If anybody wants a reading, I'll see you later. Okay. Bye. Bye, Vera. Thank you so much. You, well, you thank you too for the opportunity. It's okay. It's okay. Bye. Bye. I'm trying to end the broadcast and it won't let me.


my name is Jane Leonard so I'm part of the volunteers of the gardening department and we're based down in the gardening department section okay so I've been learning all about gardening I was never a gardener before and so I've come completely fresh never done anything so I have been learning how to bed up the parterre here how to help the gardeners we've done the summer bedding we're just changing over now to the winter bedding so that's bedding up all the plants going around the estate I'm a very good weeder and helping out with the with the general team gardens


So first of all, thank you for coming and I've been seeing a lot of old faces as well as new faces, okay. So today I will be telling you several stories, yeah. So no demo, so yeah, stay comfortable, okay. So, sorry, shit, okay, yeah, that was me eight years ago, hopefully I still look the same. Okay, I work for PocketMesh, we are one of the biggest mobile DSP, yeah. So in terms of what we do, we are doing programmatic mobile advertising. So just a little bit of background of what is actually programmatic mobile advertising or in a more industrial standard term, it's called open RTB, real-time bidding. So I believe all of you has, in some way, used any applications which has this like advertisement banner, right. So basically that's the area we work for. So start of the, whenever you start this application, okay, the application basically will send a request to what we call that exchange, which you can just realize it as an auction house. So then the exchange will propagate the request to several different DSPs. So now we are actually one of the DSP, Demonsite platform, participating in this auction. So we have to answer one single question, do we want that opportunity or not to show advertisement here? And also we need to give how much money we want to pay for this. And of course the pricing itself is not fixed, so it really depends on a lot of factors. So we need to do some computation. After that, the price as well as the image itself together will send back to exchange. So now we are the one participating in this auction.  And then eventually, exchange will help us to kind of deliver the image eventually into the application itself. So basically that's how more or less RTB is doing. What is the lag of this whole thing? Yes, yes. So feel free to ask me any question, okay, I'll try to answer as far as I know, okay. So a good question, but basically we cover here. So the challenge, okay, first of all, this thing looks pretty straightforward.  But whenever you talk about the numbers, now this thing become a little bit tricky. So from the exchange to our servers, and back to the round trip, we have to respect 100 milliseconds latency. So beyond that, the exchange will basically cut off the connection and treat us as we are not going to buy for that. And in terms of throughput, every day we'll be bombarded like 60 billion requests a day. If you do a little bit of translation, that translates like 700,000 requests a second. So by doing that, and also we have thousands of customers and thousands of these images. So we have to make a very quick decision, and the throughput is also huge. So that kind of poses a unique challenge to our server infrastructure. So starting off the, I mean, that's the legacy setup. Back in the old days, we were using ELB, by the way, all our servers on AWS. So we don't really host our own dedicated servers. So all the exchange traffic, first of all, hit on the elastic load balancer, which will be load balancer across different bidders, where we do the bidding itself. And so load balancer is on the transposition layer. So it's not application layer.  And we have an upper sheet host on the same machine with JBoss. The reason picking upper sheet is more like we do need to have a SSL termination. So doing the SSL termination in JBoss is less robust and efficient. So that's the reason for us to pick up this setup. From upper sheet into JBoss, it will be like a AJP protocol. So pretty standard setup. And along the way, a lot of things start to border us. So obviously, if you look at this graph, since these two components are hosted in a single EC2 instance, we basically need to pass the request twice, first time at this layer. And once the request is passed, it will be propagated to JBoss. I mean, basically, we need to process the request one more time. So this makes the processing less efficient. We are burning more CPU cycles, of course. And basically, upper sheet itself and the JBoss, they are also competing with the CPU cycles. And also, the other one is like, it's still at reasonable overhead on the latency itself. So remember, we have like 100 millisecond, 200 millisecond deadline. We have to respect. That's including the network round trip. So really, what we have, the time, what we have for processing each of the request is not that much. So that gives us some room to think. I think it's all started with the Java 7 end of the line. So we say, oh, since anyway, by the way, JBoss was on Java 7. So we say, huh, since we are going to migrate to Java anyway, why not we just do some experiment and let's say pick up a lighter weight server. So that's the reason for us to pick up Tomcat. So by doing this alone, suddenly, our 99% latency goes like, whoa. So basically, that's like a pure win. Right now, we have one less component to care about, the Apache. And in terms of latency, this is like a pure win for us. By the way, I'm only talking about 99% latency. So on average, it was still very, very fast. It's way faster than this. But again, always aim for the slowest. And the setup for Tomcat is actually very straightforward. We pick up the APR, Apache Portable Runtime, as our means to terminate SSL. And again, since the server actually taking a lot of TCP connections at any particular given time. So we also set the very relatively, very high max connections. And yes. And that's the pretty standard setup for SSL connector. So basically, the rest of them, we just take it as it is. We don't do too much optimization, too much tuning. And just Tomcat there. And then we have the latency improvement. So it's like, oh, everybody's like, yeah, we go party. Everything looks fine. So this taught us a story. Basically, Apache itself, if you are using it as a low balance, then probably it starts to make sense. But if you just use the Apache as front end for your Tomcat, then probably you don't really need that. If we can process like 65 billion requests a day just using Tomcat alone, I think you can do that also. So continue with the story, with the latency story. So the easiest way for us to implement a timeout mechanism to respect the 100 milliseconds latency is just to throw the request into the executor service and wait for the executor service to finish. So for those of you who are not so familiar with the executor service, I'll just go to a very brief introduction of these things. So first of all, we have the set of a selfless stretch, which are processing the real HTTP requests. Then we'll be basically encapsulating the request itself into the task, where basically just the pipelines and send into the executor pool, like the queue. So then the executor itself will have a set of worker stress, which will be pick up the request from the queue and carry out the real actual computation itself. So this gives us a nice side effect also, like here we can also specify how many stress will be alive. That means we will have a built-in way to throttle how much load we'll be posing onto the backend, let's say, database, backend, whatever the instruction we have to support for these activities. So again, so this thing will be propagating to here. And therefore, it carries out the action itself. Occasionally, occasionally, one of the requests will be running relatively longish, not in seconds, but probably 200 milliseconds, 300 milliseconds. By that definition, we don't want that anymore, because from the action point of view, this auction is gone. So we just reply as no bid. We are not going to buy that. So that's the executor part. Now we're talking about the task itself. The task, OK, sorry for that. Yeah, I skipped too fast, sorry, sorry. So the task itself is just an abstraction encapsulation of the computation itself. There are several stages of the pipeline itself. So each stage, there are several stages which are in memory only. So those operations are very, very fast, basically like the blue stage. Basically bump into the RAM and you check some parameters and oh, it doesn't match. OK, I say no bid. If this stage pass on to the next stage and pass on to the next stage, as you go deeper and deeper into this pipeline, it's getting more and more heavier. Eventually, we do have some external lookups. So we want to optimize such a way that if, let's say, you can already detect. At this stage, we have already gone through the timeout deadline. There's no need to carry on the operation anymore. We just need to reply. So this has a built-in kind of way to say let's cut off the connection earlier, let's cut off the request earlier. So what we decided, actually what we learned from the Apache story is like, because we have removed one process from the operating system, obviously there are much less context switch because the Apache doesn't need to wait for the Tomcat to respond what is the HTTP request before Apache kind of send a reply back to the wild internet. So we learned that, OK, by cutting off the number of context switches, we have a win in terms of latency. So from here, naturally, this executor itself is just another queue built into the GVM itself. And every time we call this, we have to pay multiple context switches because the calling strap, which is normally the self-led strap, has to wait after you submit the task. And then the executor itself will wake up another worker's track. Remember the slides I showed here? Another worker's track has to be wake up. Then once the worker's track finished, it kind of send back the signal to the original self-led request, a self-led strap. So that's probably several times of context switch, which again we learned from Apache story. This is in our way of reducing the latency. So what we really do is why not we just remove the executor itself and call the task itself directly. Since we already have the pipeline built-in mechanism to terminate the request earlier, there's no big race for us to carry on the request all the way to the external resources if, let's say, we have already passed the timeline somewhere up here. So then, yes, you have already seen this slide. So our 99% now goes even faster going to within 10 milliseconds. Now it's like one digit millisecond latency, which is like a pure win for us. Of course, we can't really do too much on the network stack because that's what AWS is providing to us. We take as granted. But what we can really do is to optimize the internal latencies. Yeah, again, so pure win. There's no additional overhead. Actually, less code also. So far, any question? Or do I go too fast or do slow? Questions? Yeah. Why don't you terminate the s itself at the e of e? If you terminate SSL at ELB, you have to go for application layer load balancing. So ELB itself, it doesn't maintain a steady back end connection, TCP connection. It has to, every time, set up new connection for each and every application, sorry, each and every request ELB receive. That means also we will do nothing but to kind of set up TCP connection and tear it down. That's another TCP connection and tear it down. So we have already done some benchmarking that doesn't come for us. So the only solution left to us is transport layer load balancing. Any question? No? Good. Your guys are very smart. So let's carry on to the next story. So along the way, we have identified a very, very strange pattern you can probably see from here. This is the server CPU. We have many, many servers. Occasionally, one of the servers will go haywire, 100% CPU shooting off the roof. Then what we actually, we really observe is during this period of time, if you happen to look into the server lock, everything looks fine. There's no excessive exceptions. There's basically no nothing. It's only the server is hitting 100% CPU. And it kind of auto heals itself. Probably that's the, we can phrase that, the best part of all the developer ones, where the bug will heal itself automatically. The only thing for us is like, what is happening over here? What is happening over here? So we have checked all these things. No clue. And it actually auto heals itself. And it's like, oh, it doesn't make sense. So until one day, until one day, we run the command line, perf top. For those of you who don't know, the perf is just a profiling tool for Linux. So it doesn't, it's not tied to GVM itself. You can also provide native applications. So I mean, we were in a state like we have no clue. So might as well just run the command C. So what we really see from the log of perf top is like there is a very distinct line, item pointing to garbage collector of the JDK native code. OK, that kind of start to give us some clue. OK, something is going on for the GC for that machine. Then one of our developers, James, who is the handsome boy here. He said, oh man, what if we can attach a profiler into that machine and see what is the GVM it's actually doing. But the attaching profiler itself doesn't really solve the problem, but we have a good side effect. By attaching the profiler, and full GC was kicked out. And this full GC was actually the clue to solve the problem.  So this is like, oh yeah, now we still have more information. So what turns out to be is like eventually we find out for the server set up, we were running before. We only set the maximum heap size. So we don't really specify the distribution between the young generation and the old generation. So for those who are not so familiar with the GVM memory layout, most of your allocation actually is done in the young generation. And when the GC kicks in, there are different stages of GC. There are young generation GC, minor GC, which will be only doing something here. We will not be touching that. The reason is more like there is a paper showing that 80% of your allocations actually died out very, very young in a short period of time. They are eligible for garbage collection. But there are a lot of data that lives quite long time. So they just push the data into the old generation and forget about that and then postpone the garbage collection as long as possible. What really happens is like, since we do not specify the distribution of these two partitions of your heap, GVM is free to resize. Resize the young generation and the old generation. So if you happen to read any of these performance tuning guys, they always tell you, do not specify the distribution between the young generation and the old generation. GVM itself has a lot of built-in heuristics to kind of adjust itself. So adjust the distribution between the young generation and the old generation so that it can go to an optimal setup. However, for our use case, it's very, very unique for us, I would say. What really happens is like, the old generation keeps on growing and the young generation keeps on shrinking. So then while the consequence is there, if the young generation is very, very small and if you are doing a lot of allocations, then we have no choice but to run a lot of minor GC, young generation collection. So this is actually the reason kind of shooting the server into 100% CPU usage. And now we kind of, OK, no other problem. The solution will be, again, very straightforward. First of all, we said explicitly give a flag for the GVM. Telling the GVM, do not specify, do not auto-resize the young generation and the old generation, which, again, going against all the guys, but we probably have no other choice to go when the server is on fire and the production is on fire and the project manager is besides you, man, can you fix the problem? Yes, you have to fix the problem. So another side effect is that we discover, OK, since the reason we, if we can send in a full GC command to the GVM, such that it will clear out all the old generation and it will solve the server fever, yeah. By the way, the server fever is more like, it's a cute name for us to name that even when the server goes like 100% CPU usage, where the server is not really dying. It's just like, well, it's not really that healthy also.  So, turns out that's a very neat command, GCMD, which you can send to any GVM. And you can run additional operations. For those who you want to see and probably can run a manual command for this GCMD to see all the options, one of the options called gc.run, this will trigger the full GC. So the other opportunity for us to run this is because our traffic, OK, our main traffic is coming from US. So that kind of gives us an opportunity where the traffic day-to-day goes like a sine wave. So there will be a peak, there will be a low peak. So this gives us the perfect opportunity for us to kick in the panadom, the SH during the low peak. Well, we still get hit by the GC a little bit.  We have to time out a little bit. But at least it's off the server fever for us. So and this is the graph after we apply a panadom here. That's the point we apply a panadom. This is a server start to having fever, like what, 38 degrees, but not that high, like 40 degrees. So we're just kicking in the panadom and all these things. And you can also see a trend along the day. This is more like a daily graph. Along the day, the server start to converge the CPU usage. Why is more like the build-up effects of these old generation getting larger and larger, young generation getting smaller? But once we apply the panadom, things like getting much better, they kind of start to converge again, which is like, OK, now we kind of solve the problem. For this time, while we have already solved the problem, we explicitly go against all the conventional wisdom, where they tell us, do not specify a young generation.  Well, yeah, I kind of get a drop down, but yeah. So how's the time? Good. So any question? So when you do the full GC, when you trigger that, is that it stops the world and then you lose lots of requests? Yes, yes. Yes, that's the trade-off we have to make. Because otherwise, when the server goes like a fever, we'll be losing more bits and we'll be losing more money. So, yeah, might as well just lose some small money than the big money.  Even if you put more of your heat up everything? Yes, yes, because the new generation, the young generation can keep on shrinking, keep on shrinking. And it becomes to a point where every request or every total request comes in, we have to kick in minor GC, which is like the main reason driving the CPU usage going up very, very high. Can I show the GC from a zoom? Right now, we are more like lean towards the open source so we don't really want to pay the commercial service as for now. Probably that would be a good solution for us. Yeah. Yeah, yeah, we should still wait. The minor GC is going to take longer, or a lot longer. Because we are using G1 GC, so the minor GC doesn't really collect all the young generation. It's like, depends on how much time it has, you collect as much as possible, but it doesn't really have to collect the whole region. So that gives us a little bit of flexibility. Is there any memory link in the application that you needed? I would say no. I would say no, because after we applied the Pandora SH and it go back to the similar level compared to previous day. So in terms of memory leak, that could be a possibility but we have more or less ruled out that possibility. Also in the server chat, only one server seems to go high. Neither the servers are better, not better. So is that it is missing in there? Sorry? In the server, CPU is at high. Only one of the servers are high. Okay, during that server fever, right, the server actually, the throughput for that particular server is comparable, the same compared to the rest of the servers. So basically that server is not like a processing less or processing more data, it's processed more or less the same. Only using more CPU usage. It's not always the same server, right? Yeah, it's random. So it can't really have any high, strong correlation between which server goes to fever when, yeah. Okay? Okay, let's move on. So our traffic starts to, as we approach to the end of the year, our traffic starts to grow. So early last year, we start with $30 billion. Then this time it grows to $40 billion. So as you know, anytime you have something start to grow, some new problem will arise. So, okay, this is a problem. This graph comes from the exchange. So that's the external view. How is the performance of Pocamas? Okay, so the blue line shows how much traffic the exchange sent to us. The green line on the other hand is saying what is the percentage, this is a 50%, what is the percentage of throttling? Okay, let me give you a very brief explanation on what is the throttling. So the exchange itself mostly debuts that infrastructure very smart. So obviously there's no way for them to know how is Pocamas doing? What is the CPU usage, right? All we are exposed to them is just like black box. Just send me that big request via HTTP. So from then they are building something like heuristically. First of all, they will keep sending as much requests as many requests as possible to DSP, so to Pocamas. And if let's say for some reason, they observe some kind of slowness in terms of time loss. So in this case, 100 requests sent to us, 50 response goes back in time. So that kind of give them some kind of idea, say, huh, these guys probably are having problem. Let's not send that fast, let's not send that many. Let's send a little bit less. So the next round they will be sending less. So this thing, this is basically throttling. And for us, this is really a bad situation because basically we are losing like from this graph, 80 requests where we can make a purchase, make some money. So this is like the missing opportunities. So obviously, throttling is bad, okay? So we want the green line always stay as close as zero. So the other observation we have seen is like, since we are using ELB, ELB has this built-in matrix also. So this is something I support from ELB into New Relic. As you can see the ELB, this is the unique TCP connections ELB has seen during that period of time. So it's kind of going up and also the spillover. Spillover refer to where ELB has already received a request from the outside world, but haven't be able to make internal connection to our bidders such that it kind of like waiting for too long and they say, okay, I'll just drop the TCP request, TCP connection. So as you can see, there's a steady trend going up. So again, this kind of start to bother us. Yeah? So as more and more TCP connections are established with this ELB and as well as the backend server and we have seen more spillovers. So one of the observation is like because we have already set ELB to be keep alive as long as possible. We rule out the possibility where ELB was early terminating all the TCP connections from exchange. So the only possibility left to us is like Tomcat itself is doing something wrong. So it's kind of cutting off the connection way too early. It has to be, it's supposed to keep alive as long as possible. So then saying, let's just say the, revise the keep alive timeout. Remember I told you earlier before, most of the parameters we just take as using the default. Okay, we don't change anything. So then this is another parameter we change. So we make it like five minutes for Tomcat. So we want that connection to be stay alive for five minutes. So that it doesn't terminate or destroy that early. But this makes no difference. So again, it's like, okay, what, okay. Then the conventional wisdom kicks in again. Okay, when it comes out of the dock. Okay, so okay, let's go back to the Tomcat documentation again and see what else we can do. Turns out, turns out there is not only keep alive timeout, but also there is a keep alive request, a counter. So that's what really makes the difference is like keep alive will be keeping alive as long as five minutes or this counter itself was default to 100. 100 over here. So it's like either five minutes or 100 requests. So after 100 requests has been served over that TCP connection, gone for you. So that's the catch for us. So how do we solve the problem? Easy, just make sure there's no counter, there's no upper limit. So this connection will stay as alive as five minutes. Okay, so again, so let's quickly, so the blue line in here representing the quick deployment. Let's just deploy everything together and solve the problem. So of course, during this deployment, the throttle rate goes like 100% because no cell is responding, they were nappy all the time. Then after that, what really matters is like, okay, everything goes to zero. So a single line of configuration, change all of them, make all the difference, yeah? So time to party and all, and the reason is as we are pushing Christmas, so the traffic keeps on growing, keeps on growing. Nothing kind of can stop the traffic from sending to us. So basically the throttling back again. So this time, we know definitely it's not going to be the max keep alive request account because we have already solved the problem. And yeah, you have already seen this before. And the other observation we have already seen, I can't really start to notice. It's like, for any given time, the server, okay, this is during the low peak of our traffic. So you can only see like 6,000 established TCP connection per server. So this number can go up to 10K to 12K. So there is always this pattern, is this stable for a while? Then suddenly it kind of like take a dive, goes from 6,000 to like 2,000. For each line, it's like five seconds away from each other. So it's like within a minute, we have lost like 60% of our traffic, of our connections. So this kind of started bothering us. And after that, this guy auto heals itself again. He's like, oh, another bug of this auto heal, auto healing. So we know we have already solved more or less the garbage collector problem, all right? We know we have already solved maximum keep alive request. We go and look at the log, everything looks normal. We go look at the new relic monitoring system, everything looks normal except this thing. Okay, so really what we hear is we were in this state for a while, literally, until one day, until one day. When we were doing garbage collector tuning, accidentally actually this flag was added, for no good reason, because we are told to. So we just add a flag. So normally the flag will enable the logging like this line. Most of the time, most of the time, the time stop is like in a range of milliseconds, five milliseconds, 10 milliseconds, maximum like 50 to 100 milliseconds, that's like when a GC kicks in, minor GC kicks in. But what really happens is like we start to see some very strange auto liars. It's like in several orders of magnitude larger than a normal one. So these start to kind of not so usual. And the other observation we have observed is like, whenever we see this four second delay, four second stop, we are going to do the dive. We are going to do the deep diving here. So that kind of start to give us some kind of clue. Okay, there's something related. There's very high correlation between these two events. Whenever you see the four, there will be a dive. So that kind of give us something to start with. So again, the conventional wisdom say when you have a problem, go and ask style overflow. Okay, so okay, let's just do some Google searching and Google search lead us to one of these questions, which doesn't have many answers. There's only two answers. So the second answer, the second answer shows us you probably want to enable a safe point statistic flag. Then the thing is like, what is really a safe point? Okay, so again, go ask Google and when in doubt, read the doc, okay? The doc says, whenever GVM want to do something tricky, as in if let's say they want to do a garbage collection, obviously you have to stop all the stress. Otherwise, what happens like, still one of the pointer is still on your stack and the memory pointing to has been already reclaimed by the garbage collector. So what happened when you access that pointer again? Stack fault. So obviously that shouldn't happen, right? So GVM will be, for the safe point, it basically says the GVM will coordinate all the stress to stop at the same time so that it will carry on some tricky stuff. So one of the things like garbage collection, there are many, many possibilities for this thing, like code reloading and let's say compilation when just done a git compile your interpreted code into binary code, you have to replace the original metacode, dispatched to the native routine instead of the interpreter mode. So all these things requires the GVM to go into the safe point where the hotspot implementation where all of you are using the Oracle implementation, it requires all the stress to be agreed on the same safe point. So basically that kind of acting like a global lock. So if you, again, if you happen to use Azure, ZIN, then their implementation of safe point is more advanced. They are more like several tiers, they can be post threat level. So because we are not using ZIN, so this is something we have to live with. So when can the GVM kind of stop a threat and go into the safe point? So this is a lot of potential opportunities, right?  So let's just enable the flag. So what we really see is like, okay, so there are a lot of possibilities for this reason, for the safe point reason. There's no, and what we really observe is like there's no correlation between which type of the event happened. So again, we are getting closer to the problem, but that's not really a clue what is really happening. What really catches our eyes is like, every time we see this lock happen into our sub-lock, there will be a distinct lock which linked to one of our component. For internally, internally the beta is designed such that we will have a central report for all the bidding instructions and the bidding instruction will pass down to the compiler which is built into the beta itself. The compiler will do some pre-computation such that to make it fast and adopt the result into the matching engine and therefore that matching engine can take the request from outside world. So what really happened is like, one of the steps of this compilation is gone haywire. It was designed, it was working pretty okay. Until the point we have more customers and the outcome for that step is not linear, it's more like exponential. So it grow pretty crazy. In terms of, we had an array of 200 million long elements. So that kind of like consume about one giga to two gigabytes of data and we are doing a sorting on top of that array. So kind of that's like when the GVM see this kind of thing to happen in a very tight loop, GVM tend to optimize the code to forgone all the safe point checks. So basically you'll just send a tray into that loop, very tight loop and just do the computation itself and never share the safe point. So when that happens, when that happens and all the stress has to wait for that single try to finish because we have to agree on the same safe point for the GVM to stop. So this actually is the real problem. So let's say, okay, then we say, okay, let's just do a quick experiment. Let's stop the compiler from doing any compilation and this is what we really see. For a very long period of time, the TCP connection stays very, very stable. Okay, so now we know the problem. Let's just deploy the fix. After we deploy the fix, you can see on the graph is clearly something is really effective. And the other side effect, a good side effect, is remember the Panadode or S-H because now we start to kind of link back, connect the dots, right? So we start to realize, okay, because of this compilation stage where I take in too much RAM, then for G1GC, whenever you have a huge array, huge object allocation, you always, by default, go to old generation. It doesn't even stay in the young generation because that's the one of the limitation for G1GC. So they call that humongous object. So that object goes directly into the old generation. So this kind of, we now know the culprit of these old generation growing, young generation shrinking. And after we apply the fix for this problem, it actually solved two problems. So the other problem, the other problem solved for us is like now we do not need to apply Panadode daily. So the distribution, basically the old generation stayed more or less very stable across the day. And yeah, this time one solution solved two problems. We definitely need to go to a party. And just before I end the talk, these are some numbers where I want to kind of show off. So starting off the day, we are talking about 30 billion requests. Now we are handling 65, sometimes even close to 70. So that's like two times the amount of traffic. In terms of number of servers, we started with like a 60, around 60, not really 60. 50, 50, 55, 58, 59, something like that. Now we are talking about 47. That was day before yesterday. Yesterday, this number goes to 45. Today, today, this number goes to 41, okay? So if I go in to deliver this presentation one more time, you will see a much smaller number over here. And in terms of 99 percentile latency, and you can see this like 10 times better. So down here, this is one of our exchange, our, the send us daily report. This, I think that this report sent us, sent to us like the day before yesterday. You can see the total number of requests sent to us, two billion, and unfortunately, we still have a little bit of time loss, like a 6,000. Yeah, and unfortunately, we still have a little bit of transfer error, like a 5,000, but if you do the math, then this is like a negligible, right? And another graph, really, really want to show you. And I know when you talk about latency, people tell you, do not talk about average, do not talk about anything. Only talk about 99 and the max, okay? So I just want to show you anyway. What is our average time? So if you happen to see the average, it's like a 600 microsecond. It's not even a millisecond. Medium is even smaller, so not even funny anymore. So 99% stays comfortably, below 10. So that's today. And yep. Any question? Any question? Do you have some solution in mind where you will separate the requests, which suppose you have some... This is something we like to have, we don't have now. The reason it's more like a simulation of these big requests is quite difficult because you need to simulate a pattern and simulate all the different parameters we have to compute with. So yeah, definitely that could be something that can be something to be done, but we haven't done something yet. We haven't done anything yet on top of that. Really like to have that. And by the way, PokerMessage is also hiring. Yeah. Yeah. Like PayPal. Okay, if you like ping me, yeah. Okay, so probably I will stay around for a while and if you guys have any other questions, you can ask me personally. Yeah. Yeah. Thank you.


In this video, I want to talk a little bit about rearranging equations. And this is an algebra issue, but it's one that sometimes causes students some issues. And as an example, I'm just going to go ahead and take a look at a couple of the equations that we use in one of the early chapters in my physics class. So here's one of the equations we'll work on rearranging. And then my next equation, and we'll work on rearranging that one. Now when we work with doing algebra on equations, we have to keep in mind the order of operations. And different teachers teach it different ways, but you have to do parentheses, exponents, multiplication and division go together, and then addition and subtraction. Some people call this PEMDAs or other such things. So what that means is that this quantity in parentheses over here, this Vi plus Vf, are closely bound together, more closely bound together than the multiplication in front of and behind it. And over here in this equation, because there aren't parentheses, this multiplication is more tightly bound together than this addition. So keeping those rules of operation in mind is really going to help you as you're working through. So let's say I was going to rearrange this equation and I wanted to solve for the Vi. Well it's right now bound to the plus Vf inside the parentheses, but more loosely to the multiplication and division, because multiplying by one half is like dividing by two. So I could go ahead and multiply both sides by two, and that would leave me with Vi plus Vft, but now the two has moved over to the other side. But it's multiplied since it was originally divided. And then I can divide both sides by the time, and when I do that it cancels out on the right hand side. Now I didn't show the actual multiplication and division here, although I could have. I could have said here that you divide both sides by time, time then cancels, and that's how I get my equation below. Once I've done that, now I'm down to something which is just in parentheses. So if I was going to now work with this, because there's nothing else that can come outside of the parentheses, and I can now subtract the Vf on both sides, which would leave me with the two delta x over t minus Vf equals Vi. And I've got my full equation if I'm trying to solve for Vi. I could do the same sort of thing over here on this equation. And again, let's say I'm going to solve for Vi. Well, in this particular case, because I have the At, which are bound together, but they're added, what I can do is subtract off that At on both sides. And I'm left with Vf minus At equals Vi. I never had to separate those two because those weren't the things that I'm solving for. So let's look at these same two equations now. Look at a different example. Let's say in both of these equations, I wanted to solve for the time. Now I'm going to go ahead and recopy them just so we can keep track a little easier. Here I've got my delta x is 1 half Vi plus Vft. And here I had my Vf equals Vi plus At. And now I want to rearrange these equations as if I was solving for the time. So again, we want to go through and do some operations. Now in this case, again, I've got a divide by 2 so I can multiply both sides by 2, leaving me with 2 delta x equals Vi plus Vft. And then because these are bound in parentheses, I can actually go through and divide by the whole thing still held in parentheses. And that's going to leave me with 2 delta x over Vi plus Vf is equal to t. Just scrolling up so you can see better. Over here on this side, well, again, the At are bound together, but the Vi is more loosely connected. So I can go ahead and subtract that off first, which would leave me with Vf minus Vi equals At. Now we can deal with the division because they were originally multiplied. The inverse operation would be the divide. And it divides the entire quantity there. So I'm left with Vf minus Vi. And you might want to put some parentheses around that now just to clarify. All divided by A equals t. So as you're working through doing algebra rearrangements, remember that the order of operations matters. And we're kind of working backwards through things and figuring out which things are more closely connected and which ones are more loosely connected. So that we can do the inverse operations to be able to rearrange things. If you have any more questions on rearranging equations and you're one of my students, go ahead and contact me and I'll help you walk through some of the steps.