Good afternoon. Hi, everyone. Good afternoon. Sorry, it's getting started just a few minutes late here. Just a few things at the top, and then I'd be happy to dive in and take your questions. So earlier today, the Secretary arrived in Jakarta, Indonesia, to attend the 10th ASEAN Defense Minister's Meeting Plus. While in Jakarta, he will conduct an ASEAN United States informal meeting to discuss opportunities to expand the ASEAN United States Security Cooperation one year after the elevation of the ASEAN-US relationship to the Comprehensive Strategic Partnership. He will also meet with regional counterparts to express the United States' deep commitment to working alongside allies and partners in support of a free and open Indo-Pacific. Also earlier today, Secretary Austin approved a request from the DC Homeland Security and Emergency Management Agency to provide 30 DC National Guard personnel to support the DC Police Department during the March for Israel on the National Mall. These 30 guardsmen will be unarmed and will provide traffic and crowd management support to DC police from 5 a.m. to 7 p.m. today, and will remain under the Operation and Administrative Command and Control of DC National Guard. The Secretary has additionally authorized the activation of approximately six guardsmen from the DC National Guard's Civil Support Team. These guardsmen will similarly be unarmed and will remain under the command and control of the DC National Guard while providing support to DC Fire and Emergency Medical Services. For more information, we would refer you to the DC Homeland Security and Emergency Management Agency. And finally, turning now to our forces in Syria and Iraq. Since October 17th through today, we are tracking that there have been 55 attacks on U.S. forces. There have been 27 attacks against U.S. forces in Iraq and 28 attacks in Syria. And an update for you from last week on injuries. We are also tracking 59 U.S. personnel who have received injuries in the TBI or non-serious categories since these attacks began. Approximately 27 personnel have been evaluated for TBI and 32 for other non-serious injuries. As of today, all 59 have returned to duty. And with that, I'd be happy to take your questions. Tara, you want to start us off? Yes, thanks. To just start with Syria, can you give us any additional information on the air strike that happened over the weekend? What sort of battle damage assessment was there? And did that strike result in casualties on the ground? So something that we are continuing to do is battle damage assessment, but I don't have anything to read out in terms of casualties. As we mentioned over the weekend, we are aware that there were IRGC-affiliated members in the proximity of the facilities that were struck by our aircraft. But I don't have more on casualty numbers or anything else to read out. So you don't have a sense of how many people were in the area? And then just one more on the summit that's taking place. Much has been said about maybe the potential takeaways from this will really be mill-to-mill and maybe improved mill-to-mill relations. Sorry, I'm checking out the president. Okay. Yes. What has the secretary communicated as far as what would be basically a good deliverable to help restart this relationship? Well, as you mentioned, this is the president is leading the summit, which is starting tomorrow. And so I don't want to get ahead of anything that the president might announce or any outcomes from that summit. We've stressed from the very beginning that we believe mill-to-mill cooperation, coordination, and communication is important. Right now the PRC does not have a national defense or a minister of defense. And so at the 80 a.m. plus that the secretary is at starting tomorrow, we don't have any indication that there will be a meeting happening between the secretary and his non-designated PRC counterpart yet. So there are no plans for the secretary to sit down with anyone at this point. But again, I'd let the president's meeting that's happening tomorrow. I don't want to get ahead of that. Just one last. Can you characterize the state of the relationship right now? Are there any conversations going on? Any sort of deconfliction line talks? Are they picking up at all? Are we even trying? I mean, we continue to be both publicly and privately continue to engage with the PRC. We are certainly committed to having an open line of communication and mill-to-mill communications with the PRC, but we just haven't had that resuscitated from their end. Jen. Sabrina, about a week ago, Iranian-backed Houthis downed an MQ-9 Reaper drone, a U.S. drone, $32 million drone. So far there's been no response by the U.S. and the Houthis are saying they will threaten Israeli ships in the Red Sea. Isn't the lack of response by the U.S. military inviting more aggressive actions by the Houthis? Yeah. Thanks, Jen, for the question. I wouldn't say that it's inviting more aggressive for the response from the Houthis. We've seen the Houthis do this before. We've seen one of our MQ-9s shot down before in the past. I'm not saying that we're not going to respond. We always reserve the right to respond at a time and place of our choosing, but I just don't have anything to forecast for you right now. I mean, as you know, our main goal is to contain and to make sure this conflict is contained within Gaza. Right now, that's where we see it. We see the conflict remaining within Israel and Gaza and between Israel and Hamas. So as of today, I'm not going to broadcast any response or get ahead of any response that the Secretary or the President might want to take, but I think we've been pretty clear in our message when it comes to deterrence, and we haven't seen this conflict wide enough. We're not deterring the Houthis. The Houthis are continuing to fire missiles toward Elat. They're threatening ships now. I don't understand how a deterrent message is being received by the Houthis if there's no response to downing an MQ-9. Well, you have to remember that one of our ships did take action when the Houthis did fire missiles that were headed towards Israel. We did engage in that. We did bring those down. Again, I'm not going to get ahead of any action that the Secretary or the President may or may not take. We have a very robust presence in the region right now that is sending, I think, a very strong message of deterrence. And while I certainly hear your question, we always reserve the right to respond at a time and place of our choosing. Yeah. Thank you, Marina. A question and then a follow-up on an earlier question. There's some reports in Arabic language media today, I went to accounts from Gaza, where witnesses are saying that they've seen troops speaking English to each other and American flags on their uniform. Can you give us an update generally on what U.S. forces are doing currently in Israel and specifically can you respond to the claim that there's potentially U.S. troops on the ground in Gaza? Well, I can't respond to that specific claim. I haven't seen it, but what I can tell you is that there are no U.S. forces on the ground in Gaza. Thank you. Carla. Oh, I'm sorry. Thank you. And then to follow up on that earlier point with the Chinese and the U.S. military communications, you said that there hasn't been any reciprocation. Now, does that date back to the LOC visit in 2022 or when was the last time there was any reciprocant communication? I would have to take that question in terms of when the last time there was a mill to mill communication. I mean, the secretary has not engaged with his counterpart. And as you know, that position is vacant around the speaker's visit, I'm not aware that there were communications ongoing with the PRC, but it's something that I can take that question and get back to you. Carla. Thanks. Back to the attacks on U.S. forces, sure, Iraq and Syria, I have three questions. First, will the Pentagon acknowledge that deterrence is not working with Iran? Second, why is the U.S. not hitting targets in Iraq when it has been proven that there are weapons facilities and weapons being used against U.S. forces in Iraq? And then third, does this building still an obligation to strike any and all Iranian and Iranian-backed weapons storage facilities to protect forces in Iraq and Syria? So many questions. Okay, so starting at the beginning, your first question was, do we think deterrence has failed? Was that... I'm not going to acknowledge that deterrence is not working. When it comes to Iran. So again, Carla, this war we want to see contained to Israel and within Gaza. We do not want to see a wider regional conflict. Because of today, that has not happened. We have not seen this war spillover into other neighboring countries and into the region. In terms of attacks on our troops, separate issue, we are in Syria and Iraq for our mission to defeat ISIS. We have seen Iran and Iran proxy groups continue to attack U.S. forces because we know Iran doesn't want us. We know Iran does not want to see U.S. forces on the ground and in the region, frankly. And so we saw this back in March. We've seen this before where they attack our forces. On the issue of deterrence, just taking a step back, is deterrence working? We feel that it is. We have not seen this war spread into a wider regional conflict. We have responded with the—we conducted three different strikes. We responded most recently this weekend. And again, we will always reserve the right to respond at a time and place of our choosing in the future. Does that understand? The strikes that happened, that we've done three rounds now, they're not to prevent attacks on U.S. forces in Iraq and Syria, they're to prevent escalation of the conflict. I think again— Is there anything to prevent the strikes against U.S. forces in Iraq and Syria? I guess I'm trying to understand. So the strikes that we are taking is to signal and to message very strongly to Iran and their affiliated groups to stop. That is the purpose of those strikes. And we are seeing— Because they keep striking. I understand that. But again, I think we are being very deliberate on how and when we conduct our strikes against these groups. And I think that Iran is certainly seeing that message. And I'm sorry. I don't remember. You had a third question. Sure. Sure. So why is the U.S. not hitting targets in Iraq? And does the building feel an obligation to strike weapons facilities if they know that they're there to prevent attacks on U.S. service members in Syria and Iraq? So in terms of why we aren't hitting where we're hitting, we're hitting places where we feel we can make the most impact. And just this weekend, you saw us hit a training facility and a safe house that also operates as a headquarters node. So we feel very confident in targeting and precision of these strikes. And just because they're in Syria does not mean that they're disrupting access to weapons, access to capabilities that these groups have interrupt. And I'm sorry. You're a third part. You kind of explained it. If you know of a facility that has weapons, do you feel, does this building feel that they have the obligation to strike it, to prevent enemies from using those weapons to target U.S. forces as they have done on a daily basis? Well, I think as you've seen, we've taken action. We've hit weapons storage facilities. And it's important to remember, the facilities that we hit are now destroyed. These groups cannot go back to these facilities and take weapons out because they're gone. So we feel that we are being very precise with our targeting and ensuring that we can inflict the most damage to these groups and their capability. So General Writers said exactly two weeks ago that the mechanism by which the U.S. is providing military intensive Israel is different from a legal and budgetary standpoint in comparison to the military assistance to Ukraine. How and why and also what figure we're talking about in military assistance to Israel, like millions, billions, tens of billions, what is it? Yeah, no, so thank you so much for the question. So in terms of what General Writers said, I don't have much more of an update to provide you in terms of the dollar amount right now. What I can tell you is we are providing assistance to Israel through very different mechanisms than how we provide assistance to Ukraine. So Ukraine, you're seeing the PDAs or the USAI drawdown authorities that we are using. Israel is mainly through FMS, FMS, and then direct contracting that Israel does with private commercial companies. So again, I don't have that amount in front of me right now, but the mechanisms of which we are providing it to Israel is different from Ukraine. Can you get into why that mechanism can flow over the other mechanisms that's been provided to Ukraine as a specific reason? Different countries, different mechanisms, different relationships that we have with each country. Again, Israel had, before October 7th, a very capable professional military. Ukraine is building up to being a very capable fighting force in their counter-offensive. They were attacked in February of almost two years ago, but Ukraine needed the support to build up their military that Israel already had. So again, the funding mechanisms and how we're supplying aid. It's two different countries, two different wars, and again, two different areas of responsibility. So it's a little different. I haven't heard any figure. Do you think we're going to be able to find out how much need to have been provided to Israel? Yeah, something that we are, of course, I know General Reuters spoke to this, it's something that we're working towards. I just don't have that right here at this moment. Yes, Joseph. In the record on the MQ9, I hope you said that it was about Yemen territory. Yeah, the MQ9 was operating in an international airspace when I was shot. I believe it just wasn't going to be salvageable at that point. We know that the Houthis did make an attempt to try and salvage some of the remnants and they also were not successful. So it just wasn't salvageable at that point. One other one, on military assistance to Israel, we've seen targeting civilians and journalists in Gaza. Well, that's quite a ringtone. That was, yeah. Oh, my gosh. It's also expanded beyond those borders. Yesterday, there was a group of Lebanese journalists targeted in southern part of the country. The reporters said that they had coordinated that trip with the Lebanese armed forces and the UNP's keeping force in Lebanon. So this came about a month after a Reuters journalist was killed. They also say that was done by, as a result of the Israeli strike. So what's the final doing to ensure or push the Israeli military on this aid and provide to make sure that it's not targeting civilians and specifically journalists? Well, we certainly, I mean, I think you asked me this last time and I'll reiterate that we believe that freedom of the press and protection of innocent civilians is incredibly important during this time. Incredibly important when people are on the ground in Gaza, reporting on what's happening on the ground. And then, as you mentioned, I think it was you referenced Lebanon with some of these journals, journalists were stationed. Again, you know, we are going to continue to urge for Israel to continue to uphold the laws of armed conflict, humanitarian laws. And again, we know that this is an incredible. I don't burden is not the right word, but told that it takes for journalists to cover these what's happening on the ground. And it's incredibly important that their safety is protect. Yeah, in the back. Yeah, so two part question. The president said yesterday that hospitals in Gaza must be protected. Is the US telling Israel not to attack hospitals or to shift away? It is operating at these sites. And then second, do you have any conclusive intelligence that these hospitals are legitimate military targets? So we've been very clear with. I think you've seen in the readouts that the Secretary has had with Minister Galant that we believe that the protection of innocent civilians is not only a priority, but it must be upheld. And so you've seen in our readouts time and time again, that is something that the Secretary continues to emphasize with Minister Galant in all of his calls. And you've seen that from from the not just this department, but across this administration. I think what Jake Sullivan said yesterday is exactly what we keep reiterating to our Israeli counterparts is that we don't want to see a firefight. In a hospital, we absolutely believe that hospitals should be places where people seeking medical treatment and tension need to be. And those places should be protected. But you have to remember that. And, you know, we do have information that Hamas and Palestinian Islamic Jihad uses some hospitals in the Gaza Strip, including the Al-Shifa Hospital, as a way to conceal and support their military operations and hold hostages. They have tunnels underneath these hospitals. And so Hamas and PIJ members operate a command and control node from Al-Shifa Hospital in Gaza City. They have weapons stored there and are prepared to respond to an Israeli military operation against the facility. So this is I'm just telling you what we, as the intelligence community assesses, is happening in Gaza City. How Hamas is using these hospitals to operate. But absolutely, we do not want to see a firefight in a hospital where there are innocent civilians. Fadi. So you thought of having forces on the ground, right? So I take this intelligence assessment is gathered based on information provided by Israel. Sorry, what I just read out, that's from our intelligence community. So do you have any assets inside Gaza? We have no assets or boots on the ground in Gaza. So there's no gathering on the ground. How about this? That's right. So you have no one entering the hospital to verify this information? We have no boots on the ground in Gaza. How was that assessment, you know, how did you conclude that assessment based on information received from your Israeli counterparts? Well, I'm not going to get into more specifics on how our intelligence is collected. What I can tell you is there are multiple agencies, as you know here within the department and across the administration, multiple agencies that collect and assess and analyze intelligence. But I'm not going to go into any further details on how that intelligence is gathered. Are you confirming that you have no assets on the ground or assets entering these hospitals to basically confirm information you're sharing? I will say it again. We have no boots on the ground in Gaza, Louis. I'm sorry, Jared, and then I'll come back. What you just read. Yeah. That just sounds different to me. It's more expansive than what Mr. Sullivan said yesterday. So am I correct in saying this is new information that you are providing to us? Yeah, that is. New language that being assessed by the US intelligence community. This is newly downgraded information that we felt was important to get out today because there have been a lot of questions about the hospital and how Hamas operates.  Clarify and invite any questions here. So this is an independent US assessment separate from the Israeli, their own assessment or this information that they provided to you within you to analyze. Yeah, I'm just not going to get into more specifics on the intelligence. I can tell you that this is information that was downgraded and collected by the intelligence community. Yeah. What do you mean? Do you mean it's classified? Do you mean it's classified? Yeah. That's what we're going to have. I'm not going to get into how or where this intelligence is coming from. This is done by our intelligence community. It was put together and downgraded so you can have a better idea of what's happening on the ground. Can I just wait? I think he wasn't done. Sorry, this is on the subject. OK. All right, go ahead. Just the level of confidence, usually you, when intelligence shares some information or declassify, you talk. Yeah, I'm not going to go into the assessment of how we feel. I'm just telling you what the intelligence community has been able to declassify today. And the fact that I'm reading it out to you, we feel very confident in our sourcing and what the intelligence community has gathered on this topic, but I'm just not going to go into more specifics on the intel itself. So you had a high level of confidence because you said we are very confident. I'm saying I'm confident reading it to you from this podium today in our assessment. Did you have one more question? Then I'm going to go to the phone. Just the follow up button. I can use questions on multiple levels, deterrents in the region. And from the Department of Science, did Iraq's attempting to rein these groups in from sort of localized attacks in Iraq and Syria and US bases? How do you speak to intelligence that we gather? I'm going to, yeah?  Is there a concern that the IRGC may or may not have full operational control of these groups? Is that something that might be factoring into what's going on right now? Well, we certainly know, and we've been pretty public about the fact that the IRGC supports arms, quips, funds, trains these groups. So they do maintain some level of accountability over these groups. I think some of these militias, of course, carry out their own attacks, maybe not necessarily running it up all through their command and control nodes. But we know that Iran certainly has a role to play with these groups. I'm going to get head to the phones here. Dries Reuters? Nothing heard. Chris Gordon, Aaron Space. Thanks, Sabrina. On the US Air strikes, you and other US officials have said broadly the message to Iran and these militias have stopped. So does that mean no attacks on US troops in Iraq and Syria by these militias? Are these airstrikes intended to stop attacks completely, to limit them, to degrade them? What's the end here? What is the practical, achievable goal of these US airstrikes? Yeah, Chris, thanks for the question. We do want to see a stop to these attacks on US forces. Next question, Jeff Shogle, task and purpose. Thank you. Has the Defense Department or the Biden administration as a whole made a decision that any airstrikes or other form of retaliation against attacks on US forces will be in Syria instead of Iraq? And I also just wanted to follow up on my colleague's question, what is the end game here? There's a cycle of attacks and a reprisal and more attacks. How does this end? Thanks, Jeff. In terms of why, I think your question is sort of what Carla had asked earlier on why we are selecting targets in Syria. Look, we feel confident about the targets that we select, the ability to degrade further capabilities of these IRGC-backed groups. I'm not going to get into why we pick certain places over another. The attacks right now on our forces are basically split pretty evenly between Iraq and Syria. And so we feel confident in our targeting of the places we were able to hit within eastern Syria. And on your, how does this end? I mean, again, we want to see these attacks stop. So how these end is really up to how these IRGC-filiated back groups choose to conduct themselves. Again, we want to see these attacks completely stop on our forces in Iraq and Syria. So we will always and continue to protect our troops in the region. We will make the decision to decide when we choose to respond. But how this ends is when these attacks stop. And last question on the phones, and I'll come back in the room, Heather, USNI. Hi, thank you so much. Nothing sounds flipping, but you keep saying that the war hasn't expanded past Israel and Hamas. But I was wondering what the DOD's definition for expansion is since we're seeing these attacks in Syria and Iraq against US troops that only started after the Hamas attack in Israel. Yeah, thanks, Heather. So again, the attack, the brutal terrorist attack that happened in Israel was on October 7. These attacks started happening on October 17. Again, we have not seen the conflict that is happening within Israel and within Gaza expand out to the region. And you're seeing, or that's why we have said and continue to emphasize from this podium here, that we have surged assets to the region. We have, when we announced that we have put A-10s, F-15s, F-16s into the region, we have also put the forward carrier strike group. The AIC is also in the SENCOM AOR. That is all to send a clear message of deterrence to any actor, state or non-state actor, to if they seek to take advantage of this conflict, to really think about that again. And again, I'll emphasize right now, we are seeing the conflict contained to Gaza. And so that is what we want to see, and we don't want to see this expand beyond that. All right. Coming back in the room. Yes, in the back. I have a question. Sure. Have you seen any indication of progress by Iran in enriching uranium? I don't have any updates for you on that specifically. Sure. Did I see anyone else? Are we? All right. Thanks, everyone. Have a good afternoon.


Okay. Hola. Es un placer restaurante, voy a hablar en español. No, it was only a joke, but it's really nice to be in Madrid, and I really like Spain and Spanish culture, so it's nice to be here. So hi. Today I'm going to talk about how we can get data from Kafka to Parquet. So first, to give you a little bit of context, I work at bold.com, and bold.com is the biggest e-commerce platform in the Netherlands and Belgium. So we're a huge web shop that sells everything from toys, books, electronics, and now lately we even started selling alcoholic beverages, which is kind of awesome. So and then a little bit about me. I'm Gabor. I work as a data engineer at bold.com. And previously, I've worked in a research institute in Hungary, and I was mostly involved with contributing to open source projects like Apache Flink streaming, and I also implemented some machine learning algorithms on top of these distributed systems. So now I've been working at a measuring team and recommendations teams at bold.com. So today I'm going to talk about a really simple problem, or at least a seemingly simple problem, to have a data stream and turn this stream into files, just like files based on time. And although this seems easy, we struggled with it a lot, and we tried several solutions, and I hope we learned a lot from this struggle, and I hope you're also going to learn a little bit. So I'm just going to go through our solutions and then conclude. So what is this problem exactly? So first, we have click data, so basically events about users viewing some products, clicking on recommendations, all sorts of things. And we store this data in Kafka, and I think you probably know Kafka. How many of you know Kafka? Okay, yeah, yeah, sure, I'm at the right place. So it's a distributed message queue, simply. And we store that data there, because we can get that data in real time. And of course, we need a distributed message queue, because we have a huge amount of data. Currently, we have around 10,000 events per second in this specific topic, and it's going to increase, because our company is going to grow, and we're going to measure more and more. So then the challenge is putting that data into files on HDFS, on how to distribute it by system specifically. And why do we need this? We need it because, like, yeah, everybody is going streaming nowadays, but we still have analytics teams that are using tools that cannot read Kafka, or if you just want to gain insights from really historical data one year ago, plain old files can work easier for you. So basically our task is, how are we going to do that? How are we going to get from Kafka to HDFS files? So before I go into that, I want to touch one topic, which is the difference between batch and stream processing. So I guess probably you mostly have an idea about it. Batch processing is when you have, like, a finite amount of data, and you have a program that gets this data output to result, and that's it. The program is finished. Maybe you run it daily, but it's just plain simple finished. In stream processing, you have a continuous data stream, an infinite data stream. You process the data 24-7. Your job is running 24-7, and you're continuously creating output. But to give you an analogy, I like this analogy. Let's say you have a problem. You want to fill a pool with water. So batch processing is a bit like having buckets and carrying buckets of water around and filling it into the pool. So it seems like a terrible job. And stream processing is like when you have a water hose, and you just hold a water hose, and you can fill the pool. So actually in this analogy, the water is your data. And you can see that batch processing, batches are the buckets, basically. But what does it get to in our problem? So in our case, we have a stream of data in Kafka. We have that water hose. We have that stream. And we want to fill up buckets, basically. We want to create some batch files so that people can use it in batch processing as well. So in expectation, it should be easy. In reality, we can go crazy with that water hose like that dog. And we actually went a little bit crazy with this. So but why? Why is it difficult? It's just dumping data, right? So it's difficult because we have some requirements. First, we want a scalable solution. And it's not surprising. We are at a big data conference. Like I told you, we have a thousand events per second. It's growing. We want something scalable. Second, we also want to dump the data exactly once. And what do I mean by that? So we have one message in Kafka, and we should have exactly those messages in the files. So I don't want to lose any messages in the files. And I don't want any duplicates. Why do I care about duplicates? Because let's say there's a data analyst in our company, Katie, for example. And she will need to read this data several times. She doesn't want to start every single job by saying distinct. Katie doesn't like distinct. Katie wants the duplicated data. So we are thinking about our data consumers and not ourselves. So exactly once is one requirement. Another requirement is event time. Having these files in event time. And what do I mean by that? So let's say we are reading event streams. One event comes in at 9 a.m., a user clicked on something. But actually the user clicked on that item at 8.55. So the time when the event happened and when we got to process the event is a little bit different. So the event is a little bit late. So where should we put that event then? Should we put it into the batch file of, let's say we have batches of one hour, batch files in one hour. Should we put it for 9 a.m. or 8 a.m.? We want to put it at 8 a.m., why? Because also from a data consumer perspective, Katie, our data analyst, she doesn't really want to care about like, yeah, if I need only the events from between 8 and 9, which files should I look at? She doesn't want to look at the neighboring times and files. You just want like, I want the events that happened between 8 and 9. So we need files in event time. And last but not least, we want the columnar format because it's more optimized for reading. Again, Katie doesn't like loading files that take a long time. So that's what we want. To give you a little bit idea why this columnar format is better, let's say you have a data set like customers viewing products. So you have a customer ID, the product ID viewed, and you have a product category. And there are basically two ways to store this data on disk. One is the row-oriented format. So in this case, you store this data record by record. So first, you're going to store all the fields for the first record. So the customer ID, the visitor ID, the visited product, and the category in one record, and then for the second record, and then for the third record, and so on. Whereas in a column-oriented format, yeah, it's pretty trivial. You store it by column. So first, you store all the customer IDs, then you store all the visited products, then you store all the categories. Yeah, that seems easy. But why is this row-oriented format is not so good for analytics for reading the data? Let's say we have another data analyst, Peter, over there, and he doesn't like slow loading. And Peter is not interested in the categories. Peter is only interested in two columns, just the customer ID and the product visited. So in this case, if we store it in row-oriented format, the system that Peter uses needs to read all the data anyway, because he's reading it row by row, every row, so he needs to read those books. And if you're talking about big data, this can be an issue. This can make it a lot slower. So in contrast with the column-oriented format, because we store it in columns and we're not interested in, or at least Peter is not interested in the category, we can just decide not to read that column. And it makes it basically faster. So both our data analysts, Katie and Peter, they do love, they do love fast loading. So we want a column-oriented format. And the go-to column-oriented format in this Apache Hadoop landscape and Fire Landscape is parquet, so we want it parquet. So let's get back to our requirements. So I told you about the requirements, but what tool are we going to use to solve this problem? So the first thing that popped into our mind was Apache Flink. And that's also the distributed stream processing system. How many of you have heard about Apache Flink? OK, cool. OK, so this is not going to be really new. But Apache Flink seems suitable for these requirements. Why? Because it's distributed, it's scalable. It gives you exactly one guarantee. Even if failure happens, it has obstructions for handling this event time. And it has parquet output connectors, input connectors. So that seems cool and easy. But so yeah, actually in our problem, the question mark becomes, yeah, we are going to use Flink for that. But we'll see how it goes. So first, let's go to our first solution. We first wanted to use Flink windowing. And Flink windowing is basically windows are just chunks of events based on time. So you can have a window from 8 to 9 or 9 to 10 and so on. And what happens is that Flink will kind of store the data for one window. And then when the window finishes, it will do something, an aggregation. So in this example, I'm collecting events for two hours. So let's say again, let's say we want to write hourly batch files. So batch files for every single hour. First, you can see that in the Kafka queue, I noted with brown and blue squares. So the blue squares are for events between 19 and 20 o'clock. And the browns are events for 18 and 90. So you can see that the events are out of order. And that's also what happens in reality. You don't see your events come in order because you all have distributed systems. I also did a simplification here because I'm just showing one ordered queue. In reality, you have a distributed queue in Kafka. You have multiple partitions per topic. But you can easily generalize my examples to multiple partitions. So yeah, so we start reading the data. And we collect this data per Windows in Flink. So we have a window for events between 18 and 19 hour and so on. And we just read this data. And as soon as we know that one window has finished, we've seen all the events before 19 o'clock, we're going to write that file. And Flink just gives this data in memory. So that looks pretty straightforward and easy. But how do we handle failures? Oh, we're lucky in this case. Yeah, we can celebrate. It's out of the box. We don't really even need to care about fault tolerance. We don't need to understand it because Flink will handle it for us if we are using these windowing abstractions. But there's always a but. But we might use too much memory. Like I said, this is also a bit of simplification. But Flink keeps this state, keeps these windows in memory. And in our case, if we just assume that somebody decides we don't want batches for one hour, but we want it for two hours and we're using the same amount of memory, we need to keep two hours of data in memory. So maybe we're like collecting, collecting it in memory and boom, at one point it's gonna run out of memory. So we could just increase the memory in the machines, add more processing units and so on. But it feels a little bit bad because we want to do a simple thing. We just want to read some data and dump it into files. Why should I keep these files in memory? And so we actually experienced a lot of, like when the loads changed, we experienced these out of memory exceptions and we decided like it's better to come up with a solution that doesn't require that much memory. So we went on to the next solution which is called the bucketing sync. And in think, think the, basically the output connectors are called sync. We are syncing the data so you can have a sync for Cassandra, a sync for, for I don't know Kafka as well or any other database. And the bucketing sync is basically something that writes into files called buckets and buckets are also files based on time. So this seems exactly what we need. We want buckets of time for every single hour. And yeah, that should work well. So in this image as you see, now we are still reading the data from Kafka and we are dumping it into files, but we don't need to store it in memory. As soon as we read the record for some buckets, for like an event between 19 and 20 o'clock, we will just dump it into the file. So basically nothing is stored or many more things are stored in memory. So that sounds good. But what about handling failures? And here comes a little bit more tricky part and we need to understand Flink fault tolerance mechanism a little bit. So Flink handles failures by making checkpoints and how does it make checkpoints? It actually marks one point in the data source when a checkpoint happens. And I noted that with that green line as you can see. And when the data processing reaches that checkpoint, Flink will mark the part of the files, the point in the position in the files that has been written until the checkpoint. So I also marked it with small green lines at the files. So the checkpoint goes and we keep on processing. We're writing more data to the files and boom, suddenly a failure happens. Yeah, and failures can happen. Like for example, in our case, we had a commodity Hadoop cluster and people were like sometimes maintaining node and if somebody picks the node your job was running on, then your job's gonna fail. So it's that easy. So you need to account for failures. So in a failure scenario, what Flink does, it draws back the reading to the checkpoint and because Flink marked the position of the file when the checkpoint happened, it can just simply cut off the end of the file. It can just truncate the file and keep on writing the data. So this is the big idea that ensures exactly one's guarantees. If we kept the data at the end of the files, we would have duplicates. But by knowing exactly the position of the files at the checkpoint, we can just truncate them. So that sounds good and it should work well, but there's always a but. In our case, HDFS doesn't support truncating. It doesn't support cutting the end of the files. So we just cannot do that and that's kind of sad but when we were doing it, it was a problem. The good thing is that the Flink community has already fixed it and it can actually support systems that don't allow truncating, that doesn't support truncating by just copying the relevant parts of the files. So we know until which position it's still correct data and it can copy that data. But there's another but. Parking doesn't allow flushing. So that means at the checkpoint, we cannot know the positions what we were writing. We cannot know that simply. So that leads to a problem. We just cannot know what was the position when our checkpoint ended. So at a failure, we don't know what parts to cut in our files. So that led us to another idea. That means closing files at checkpoints and what do I mean by that? So we don't just write one file per every hour. We keep files in a directory per every hour and we're gonna write multiple part files. And first, we're gonna write one part file but when we get to the checkpoint, we just close the file that we've been writing and start a new file. And as the writing goes on, we're only going to write to the new file and when a failure happens, we can simply remove that old file and roll back to the checkpoint. So actually that helps us because we don't need truncating, we don't need to know the position. We just close the files at every checkpoint. So that kind of solves our problem but there's always a but. We need, in order to do this, we need it to change plain bucketing sync code which is like kind of feels tricky. And I think that's one really good point in open source projects because when you're doing something and something doesn't work but you could kind of use the code and change the code, it's pretty useful and in Apache projects the code quality is normally really good. So what we did, we just looked at the bucketing sync code and just changed it to our needs to close these files. And then even better, Flink is already supporting this closing files at checkpoints with something called streaming file sync. So the community actually solved it for us as well, but we've been experimenting with this before the release of Flink 1.6 which got this feature. So it's a really huge could us to Flink community because yeah, that's awesome, we won't need to worry about this anymore. But we had another problem. We, like I said, we've been writing data but we also cared about late events. So we actually had like, we said like we want to keep late events if they are within 12 hours late. So what happens in that case is that we are going to write the data and we're writing those files and at first there will be like large files and the small amount of files. But as we're moving to late events we will have like, let's say we're writing a file for nine o'clock by seven o'clock, nine o'clock in the morning by seven o'clock in the evening it's still within 12 hours but we're gonna have a lot of, we're not gonna have a lot of events for nine a.m. So we're not, maybe some files will only contain one record, one single record. And we will end up with a lot of small files and that's bad for HDFS because HDFS is not really good in handling small files but not just to this specific HDFS technology. Handling a lot of small files is not really good for processing system. For instance, in this case some people wanted to process in the company or process this data with Apache PIG and PIG first before doing any processing reads the data and reads the metadata about the data sets to estimate the size of the data and so on to create optimizations and it just couldn't handle too many files because so we got something like too many connections or something like that. And that's not really good because we cannot really support Katie and Peter our data analysts easily. So and by this time, like we've tried a lot of things. So we were like, why? Why can't this simple thing be done easily? Like we, why is this, why is this streaming for tolerance and all these systems? Why are those so complicated? Why do we need to understand them? So actually one teammate of mine Karst came up with an idea like, why don't we just run a daily job that or an hourly job that treats all the data in Kafka and dumps out the relevant things in an hour? That's pretty simple, right? So we did that and that's basically, we just, let's say for the events between 18 and 19 hour, we just read the whole Kafka queue and only dump the events relevant to that hour and then later we will run another job and we only dump the events for, for the events between 19 and 20 o'clock. So that's simple, of course, but yeah. So I kind of feel like sometimes sticking to batch processing is like carrying those buckets can still be, still be, still be cool and we can still be happy about it because we avoid all the complexities. Of course, this is not perfect solution because first we need to reprocess the data. Every time we read the data we need to process maybe the whole Kafka queue or we need to process an additional data anyway. So that's not really, really perfect and it also doesn't support the so-called semi-real-time case and I call it semi-real-time because there are some analytics people who want to use data like not everyday batches but they want to get what happened in the last half hour we were getting into the Christmas season with Black Friday coming next week so gaining insights. What happened in the last half hour or last 10 minutes is really important and maybe they cannot use Kafka for these cases. They still want to use files and if we're gonna run a daily batch job and run a batch job every day then kind of our latency will be one day and that's not acceptable for some use cases. So it's not for working for that. But of course like for the reprocessing problem we can use some optimizations for this batch processing. For instance Kafka now supports indexing based on event time which means you can kind of, Kafka keeps track of your, of the position of every hour in the data or you can set up which time range. So you don't need to read the whole Kafka queue to get the time frame that you need. You can rely on this indexing and you only read the relevant parts. So that's it basically, that's what we did but of course I'm not satisfied with it totally so what would be a proper solution for the future? And we haven't done this but actually one idea is like we were aiming for plain simple files and those are practically working well for a lot of scenarios but what if we instead of storing the batch data in files we just use the database and actually the team is working on something like that because we're using Google Cloud for some stuff and they are working on getting this data into BigQuery which is Google's analytics solution instead of just plain files. You could, yeah so that's one other solution. You could use different tools like and of course that database like BigQuery will take care of partitioning your database on time and you don't need to worry about it. And you can use another tool, not Apache Flink but maybe Kafka Streams or I'm sure that there are hundreds of solutions out there that might solve this problem better. We went with Flink because we've already used Flink and it seems like a good idea. Another idea is because our last problem was like a lot of small files that Apache Pig and other systems HDFS cannot really handle. So you could just write the small files and merge them in the end. That's another solution. Or you can just, like I said, we had small files because we were keeping up late events for 12 hours. You could just say that I don't care about those small number of late events, let's just drop them and let's maybe only keep late events that are five minutes late but not 12 hours late. So these are all options. But to get back to a problem, like I said, one interesting problem is how do we support semi-real time? And there are different solutions for that and everybody's talking about Kafka architecture and Lambda architecture nowadays. So by supporting real, so Kafka architecture basically means you do everything in streaming. That sounds cool because you have one code base, you have one system that can handle both streaming and batch workloads and all these things. Whereas Lambda architecture means you have a batch system combined with a streaming system and these are two separate systems. You have maintained two code bases and you specialize it to these two use cases. So in our example, how could we support real time? Maybe we could support real time by doing the same batch processing by reading the data and writing the data in daily batches and then create another job that simply drops the late events. So that sounds good because first, we got to keep all these late events in the historical data. So we got to keep exactly the same amount of events but we can still support our real time use cases and we can still satisfy our data analysts who wants events in the last few minutes, the last half an hour. So actually, this was just one example of this kind of Kafka architecture, Lambda architecture thing. And I think a lot of people are excited about Kafka architecture and me included. But in some cases, Lambda architecture, like separating these two domains can keep it simple because these two things have specialized solutions. So the main takeaway I want to give you is that I think streaming is not trivial. So we can think and other people can say that Kafka architecture is the way to go but I think in a lot of cases, there's a lot of complexity. In this case, the fault around and we needed to understand the fault around system which is in a batch system, it would just be like, we just restart a job if it's a bad job. In streaming, it's completely different and it's more complex. So I would say streaming is not trivial but I should be fair and out streaming is not trivial yet. And what is awesome that I've seen talks yesterday from data artisans, Ayusha was talking about these and Confluent is also here at the conference. I've heard a talk about Apache Puzar and these guys are all working on something called the Kafka architecture which is they are working on making it easy for us to just use Kafka architecture, use one code base and use it for streaming and batch as well. And I think in the coming years, it's gonna be more usable but I would say for some use cases, it's not there yet and it's just difficult and it's a struggle. So I would say just keep it simple and if you have a use case and you find like, how streaming would be awesome but doing a batch processing is still feasible then go for batch processing. Or if, but if you're adventurous and you see that like, yeah, let's try streaming, go for streaming but then understand the system because understanding fling fault arounds and understanding how these things works were really important for us to deal with the problems and understand what can go wrong. So in that case, try to understand your system better. So that's it basically. I also wrote a blog post about this topic like not everything is unclear did what I was talking here but you can check that out and you can reach me on Twitter and I'm happy to answer questions. So I saw some hands raised over there. Really nice talk. I wanted to ask you a couple of questions. I recently, you need a project where we were doing this kind of movement, moving data to to park it and the point that at some point and it is not the first time I had this discussion we tried to decide whether going to park it whether going to hive we'd optimize a row column in our format. I don't know if you guys have done any performance analysis regarding that. What are your concerns? And regarding the small files problem you've been facing I just wanted to know if you have considered to apply somehow of dynamic partition in grouping the packets of time with less utilization to improve the performance. That's all. Yeah, so I'm not sure what you mean by the dynamic partitioning but the answer is your first part of the question. Yeah, so park a format is just one choice and it's like every choice is the trade-off and I talked about our requirements but you can kind of loosen those requirements up. So for instance you could just use Apache Avro or any other format you want if you can still get out the performance for your analytics teams. So maybe using a simpler format and not like a row-oriented format could help you because then you wouldn't have these problems with the flashing and all these things. So that's something. But yeah, in generally I don't know it really depends on the use case what you can do. Oh, here's one. Yes, so that's one other thing. We do transformations, especially in the age of GDPR we need to analyze some parts of our data sets and that's one of the things why we need a like also distributed processing system and not just a simple solution like dumping the data really, really directly because then we can also do arbitrary transformations on these records. Yeah, okay. I don't see any more hands but I will be staying here for a while so you can catch me for questions if you want afterwards. Thanks. Thanks everybody. Thank you.


Hello, hello, hello. Hello, everybody. Option traders, stock traders, investors, speculators. This is Lee Lowell from SmartOptionSeller.com. Today is Saturday, June 12, 2021. Welcome to another edition of our Saturday YouTube videos. We talk about options trading strategies and things about those options trading strategies, specifically what's on your screen right here, the time decay of option prices. This is what we do. We talk about option strategies and things that affect option prices. And then a little later on the video, we do our Saturday synopsis, which I take a look at the stock charts, see what's happening, see what kind of patterns are developing, try to get a gauge for what's going to be coming down the pike. And if we can see any good setups happening and what strategies, options strategies we would put forth in those trades. So let's jump right in and talk about time decay of option prices. I get questions every week about all aspects of options trading. And I've got a couple this week specifically talking about time decay and theta and what is time decay and how does it affect option prices? And is it real? Does it really happen? And the answer to that is yes, time decay is real. So let's step back for a second. So we're all on the same page here. And let me explain to you what time decay is as it affects option prices. Now, anyone who's ever bought or sold an option, and maybe if you haven't done a lot of it, but let's just say you've bought or sold an option, you pretty much know the option price on that day that you execute the trade, that's what you either paid for it or you receive for it. If you were the option seller, you receive from the option buyer. So you have that option price. Now, that option price doesn't remain static. While the trade moves towards expiration, as we all know, all options have an expiration date. And as you move towards that expiration date, the value of that option will change, obviously, and it will change according to how the stock moves. So the option price is going to change based on where the stock moves, how much time is left to expiration, and the volatility in the market. You have all these different things that affect, excuse me, that affect the options value or the options price or the options premium. Those are all the words that describe the options value. So what we're talking about today is time decay. And what that means is that every day that an option is in existence, it's going to lose a little bit of its value just due to the sheer passage of time. As each day ticks off the calendar, the option value is going to give a little bit away. The option value is going to get stripped away a little bit each day as it moves towards expiration. And the thing about it losing some of its value is that it doesn't move in a straight line. It doesn't lose like, you know, five cents per contract per day. It has, it's on a curve. And I'm going to show you that time decay curve. So what I want everyone to understand is that while the option, while you're holding that option position, whether you're an option buyer or an option seller, the option value slips away each day. So if you're an option buyer, you're not so happy about that because you're just giving money away day after day, just because time is ticking off the clock. Now, if you're an option seller, which we are, we love time decay because we like making money just because time is going by. So most option buyers don't understand that. They don't think that, well, you know, if the stock doesn't move in my favor, you know, I'm going to lose money, but it's more than that. Yes, if the stock doesn't move in your favor, you can't make any money, but you're losing money to the passage of time as well. And I always use this example when I'm trying to teach time decay to people, think of an ice cube sitting on the counter, right? At first, you don't really see a lot of that melting away of the ice cube. But over time, you can see it starts to melt away faster and faster and faster until it's just a pool of water. It's the same thing with option prices. If you're holding a very long-term option, like, you know, a six-month option, you're not going to see a lot of daily time decay of that option. You won't see a lot of its value get stripped away until you get closer to expiration. And that's what's called the time decay curve and how you measure that the time decay in an options price, like how much that option will lose per day is calculated through theta. Theta is one of the Greek outputs of the option pricing model. And here at the smart option seller, we talk about delta a lot. Delta is one of the most important Greek outputs of the option pricing model. We talk about delta a lot, but theta on the other hand is very important as well because it can tell you, it can alert you how much money you're either going to make or lose per day as you hold your option position. So let's just take a look at my little cheat sheet here and talk about time decay of option prices. Number one, all options are affected by time decay. Every option will lose a little bit of its value each day. But time decay is not an equal opportunity employer. Time decay is greater for certain options and it's greater for options within the expiration cycle. At the money options have the greatest or the largest time decay per day, no matter where it falls on the expiration scale. What are at the money options? At the money options are options with a strike price that is closest to the current price of the stock. So if the stock is at 100, the 100 puts and calls are called the at the money strike prices. So those have the largest amount of time decay, the largest theta per day.  Now the longer you give or the longer you have until the expiration day, the slower the time decay will occur. So if you're an option buyer, it seems intuitive that maybe you want to buy a longer term option because if the stock doesn't move in your favor right away, you won't lose a lot of money to time decay. So that's why I always tell people if you're going to buy options, stick to longer term options because you're not going to lose money just because time is ticking away. So the longer the expiration, the slower the time decay. Now as I said, time decay is an option seller's best friend and an option buyers worst enemy because at each day passes, you're going to lose a little bit of value just due to the sheer passage of time. So as an option seller, you want that, you want to see the option decay. So option sellers think that well, if I'm going to get all that time decay, well maybe I should just sell very, very short term options like one day options or one week options. And yes, the time to get time decay is very high in those short term options, but you have to play the at the money options. And when you play at the money options on very short dated expiration scale, you don't have a lot of error. You don't have a lot of room for error. The stock could jump or fall on you very hard. And the time decay will be offset by what you'll lose because the stock moved against you. So you have to be very careful. So right here at the bottom I'm selling near dated options isn't always the smartest move because and I'll show you on in the option chain and we'll look at if you sell at the money very short dated options, you don't get a lot of wiggle room. The stock could jump on you and you'll give up all that time decay profit because the stock has moved against your position. So be careful if you're selling short dated options because you want all that time decay, you have to be pretty sure where the stock is going to move to, you know, in your favor. And here theta shows the actual dollar amount per day that is given up due to time decay. So let's take a look at the option chain. Actually first, let's take a look at the the option time decay chart. This may hate this may help some of you visualize how time decay works in an option. So this is your standard time decay chart for options trading. Now what you'll see is it's broken up into 30 day chunks starting at 120 days going all the way down to zero days. Now the curve shows you how much the option loses to the daily time decay. And on the on the left hand side here it shows you how much the premium is left in the option how much how much price it still has. So as you can see here in the first 30 days from 120 days down to 90 this 30 day block you can see the curve barely barely goes down the option is not losing a lot of value to time decay. It goes from 100 percent to roughly 95 percent still that option still has it's you know 95 percent of its value. So it hasn't lost a lot to time decay. Loses about 5 percent of its value. Now in the next 30 day chunk from 90 days down to 60 days until expiration still doesn't lose that much. Only a little bit loses about 10 percent of its value from this curve. So you can see the little triangle here. You see the triangle small here here this triangle this right 90 degree triangle right here. So from 90 days to 60 days it really doesn't lose a lot of value either. The next 30 day chunk from 60 days to 30 days it loses a little bit more of its value from about 85 percent to 70 percent. So about 15 percent of its value. So you can see as the closer we move to expiration this 90 degree triangle starts to get bigger as we get closer to expiration. And then the last 30 day chunk from 30 days down to zero days you can see how much it loses it loses this whole chunk loses can lose 70 percent of its value down to that last day. So these 30 days right here is huge. Okay. So a lot of a lot of option sellers will try to concentrate on selling options in those last 30 days of life. But once again if the stock moves really large in those last 30 days that stock movement could overtake the money you'll make on the time decay itself. So you need to be careful when you're trying to sell very short dated options. Okay. There's always there's always a trade off. You know there's no free lunch anywhere. But just know that the time decay if you're an option buyer it's much harder for an option buyer to succeed in the long run over and over again if you're buying these short dated options because if the stock doesn't move in your favor you're just going to give up all this money to time decay. All right. So let's take a look of let's take a look at how that looks on an actual option chain the actual option prices and we'll see what happens. So here's the option chain for Microsoft options. Okay. Microsoft last closed at $257.65. Okay. So we got calls on the left puts on the right. So what you want to look at is the two columns with the theta column. All right. So if you have your option chain with your broker and you want to know how much you can lose or make per day you look you have to have the theta column and every broker every platform in their option chain should have a theta column. All right. So as I said the at the money options has the largest theta. That's the amount of money you will lose per day per contract as each day passes. Now we're looking here at Microsoft $257.65. We're looking at the June 18th 2021 expires in six days from now next Friday next Friday June 18th and when you look down the strike column you go to the the at the money options which would be the $257.50 strike. Now it's the same theta is the same for puts and calls. Same amount of money will be lost whether it's a put or call doesn't matter. So the $257.50 puts $257.50 calls are over here you're going to lose 16 and a half cents per day starting at six days left 16 and a half cents these are the calls here scroll over here so roughly 16 and a half cents as well for the put options these at the money options 16 and a half cents when you go down in the strikes or you go up in the strikes you'll see how the theta numbers get smaller as you move down as you move up in the strikes or as you move down in the strikes the theta gets smaller the highest data as I said is at the at the money strikes so between now and tomorrow let's just say the market was open today and tomorrow these $257.50 puts and calls are going to lose 16 and a half cents per contract just because we flip the calendar over to the next day so if you're a holder if you're a buyer of of either of the puts or calls your your option value your option position is going to lose 16 and a half cents a contract that's 16 and a half dollars 16 and a half dollars you have to multiply this by 100 because each option contract is worth 100 shares of stock so 16 and a half dollars you will lose between today and tomorrow just because we turn the calendar over to the next day if you're a buyer or a holder of that option you're like wow that sucks I'm gonna lose 16 and a half bucks just because we move to the next day so as an option buyer if you're playing these short-term options you better make sure that stock moves in your favor very quickly otherwise you're holding on to something that's just slipping away from you just because time's ticking away on the other hand if you're an option seller you're like yeah I'm gonna make 16 and a half dollars in my sleep I'm gonna go to bed tonight I'm gonna wake up tomorrow and I'm gonna be 16 and a half dollars richer per contract as long as the stock doesn't you know really fluctuate wildly against you okay this is assuming that this is assume well this this is gonna happen regardless but if you're a holder of that option or you're selling that option if the stock moves big enough in the direction you will either you can either overtake this money that you'll lose just because the stock moved in your favor or against you depending on if you're an option buyer seller okay so let's take a look at what happens let's go out to a longer dated option on Microsoft here let's move out to the November options has 160 days until expiration so the at the money strikes are either the 255 or the 260 we don't have the 257 and a half strikes listed so here you can see the theta is about five cents per day so the longer you have until expiration the smaller the amount of money you'll lose per day as time starts to take away okay so between November and October let's look at this 30-day chunk right here a little over 30 days in October now remember the theta is about five cents a day in October the theta is still a little over five cents a day so you haven't lost that much money in that 30-day chunk between November and October about two cents you know $2 per contract now if we go to the September's you'll see that the theta is now up to you know 5.7 5.8 cents so it's gotten a little bit bigger not a lot lost about five cents or five dollars per contract per theta per theta now if we go to the August options now we're at 6.8 cents so now you're looking at you know a dollar per day that you're losing I'm sorry 10 cents it went from 5.7 cents to 6.8 cents so now it's you're losing about 10 dollars that that between August and September the theta has gotten larger okay went from 5.7 cents to 6.8 cents or roughly 6.7 cents so you're starting to see the theta get larger now if we go down to July the July options the theta is you know 7.8 to 8 cents so it's getting a little bit bigger and then as we move into June as we are looking here the theta gets quite large the 257 outputs 16 and a half cents so as you get closer to expiration you're going to lose more money you're going to lose money faster or the theta if you're an option buyer if you're an option seller you can gain that money faster okay depends if you're an option buyer option seller now we can look at the an option calculator let's go to ivolatility.com you can use their their option calculator and you can see I have the same uh Microsoft I'll put the the same price 257.65 so we're kind of looking at the same numbers six days until expiration June 18th uh options you can see the theta here is now they have it about 17 and a half cents so that all depends on you know not every calculator is the same not all the numbers will match up but but they'll be somewhat close so ivolatility has 17 and a half cents theta per day where the option chain at Interactive Brokers has about 16 and a half cents so let's move this out to 150 days or so and we'll see that the theta now is five you know roughly five cents five dollars um five cents per contract is what you'll lose overnight five dollars in actual dollars is what you'll lose overnight for these longer dated options okay so if we go down 120 days which is a 30 day block theta is now 5.4 cents 5.3 cents you're only losing a small amount so if we go back to let's look at this put in 31 days until expiration the theta is 8.2 cents roughly 8.2 cents and if we go down to one day before expiration we'll see that theta is now 42 for almost 43 cents per day so between you know one day left until day of expiration you're going to lose at least 43 cents if you're an option buyer if you're an option seller you'll gain 43 cents just due to the passage of time okay so that's how time decay works time decay is a constant it's always moving and it gets bigger and bigger as you get closer to expiration so if you're an option buyer and you think you know where the stock is going to go and you say i'm just going to buy a one-week option because i know the stock is going to move in my favor well if the stock doesn't move in your favor you're going to give up a lot of money to time decay now if you're an option seller as we talked about you'll make that money due to time decay but let's take a look here so the 257.5 puts with six days left 16.5 cents of time decay well you if you're going to sell these options you need to be pretty sure that this you hope that the stock stays within that range okay if you're let's just say you sold the put and the call you sold a you know a straddle an at the money straddle and all of a sudden the stock you know blast higher on you you're going to lose money and the time decay won't help you as much so be sure if you're playing short-term options be sure you know where the stock may be going okay so i'll bring up the the cheat sheet one more time time decay it's there it's always there it's part of an options value and just be aware that it it does exist and theta can tell you how much money you can lose or make per day depending on if you're an option buyer or seller all right so there's your your little lesson in time decay let's move on to our saturday synopsis where we take a look at the charts we look at the indexes we look at individual stocks to see what's been happening see what i see moving forward looking for certain patterns support and resistance points and get an idea of you know what what we could be thinking about doing so let's open up the chart we look at the spy which is the exchange traded fund for the sp 500 index as i've been talking for many many weeks many many months now we have been bullish i've been bullish the market is going up the market is going up and that's what the market does in the long run it goes up because the stock market is made up of companies that produce products that people want that people pay for that if they sell enough for these products they make money their earnings go up and their stock price goes up that's how the stock market works and the s and p 500 and the dow Jones 30 and the nasdaq 100 are all made up of stocks that are picked by the the the makers of those indexes and they keep stocks in there that are worthy that are quality stocks they deem quality stocks and so those companies are going to produce products that people buy and their earnings are going to go up now not all stocks are successful obviously but the index itself will continue to go up over time because people keep buying the products and their earnings go up it's just this repeating cycle so as of now with the pandemic at least here in the us we have almost half of the us i think it's half now that or more than half have had at least one dose of a vaccine and i think you know low 40 percent of all adults here in the us have a full vaccination so the us is completely opening back up people are going out they're spending money that means they're buying more products that means stock prices should go up in the long run in the short term you know day to day week to week prices are very erratic but in the long run prices will go up stock prices will go up so i drew this i have this little triangle here this is called an ascending triangle and this is the s and p 500 now between this week and last week the last few weeks we really have not had a lot of movement have not had a lot of movement and that's what happens during the summer you get the summer dull drums people go away take vacations they you know they just step away so trading gets subdued and you get get these these you know sideways patterns for a while and that you can see that right here we haven't had a big move in the s and p 500 for a little bit but this ascending triangle is a bullish setup pattern okay you got the uptrending price action and you have this resistance area and it and and the and the stock keeps you know bumping up against the resistance just waiting to break through and once it breaks through then it moves on to the next leg of the bull run which is exactly what's happening here in the s and p 500 and you can see the last two days this is thursday and friday this week the sp 500 moved above or the sp y moved above the resistance line which is an excellent sign that the next bull leg is going to happen when you have enough people watching the same patterns it becomes this self-fulfilling prophecy that okay we see it breaking out it's time to get on board so you get this rush of orders these buy orders that will continue to push it up until the next pattern develops whatever that may be so i can see here this is a good sign s and p 500 moving above resistance this to me is the signal that the next bull leg is going to happen okay but once again it's the it's the pattern the ascending triangle pattern that i like that it is making me bullish so if you've been thinking about getting on board you know this could be your signal now i talk about a self-fulfilling prophecy if enough people see the same technical pattern they get on board you know sometimes that's regardless of the the fundamentals of the company okay people keep saying stock prices are stretched you know their fundamentals don't don't reflect where the stock price should be well that's how it's always been i mean if you look back at the history of time okay here's the monthly chart of the s and p 500 2008 and you know that we had the meltdown and it just gone straight up from there all along the way people are saying the fundamentals don't match up with the stock price the fundamentals don't match up with the stock price the fundamentals don't match up with the stock price but yet stock prices keep going up that's just how the stock market works yes we'll have pullbacks over time which is good we need those pullbacks to kind of reset things and shake out the the weak hands but it goes up over time because that's what the stock market is supposed to do as long as the the stock market is composed of companies that keep creating products that people want to buy then the stock prices have to go up right who's to say that a fundamental of a company has to coincide with a certain stock price it doesn't make sense yes you can look at long-term parameters of of price to earn it p e ratios and whatever else whatever other fundamentals people concentrate on of course stock prices are going to look stretched at some point but yet they keep going up so you have to reset those those fundamental values i'm a technical guy i look at the charts more than anything else and because i assume everything with that's known about that company gets reflected in the charts so i look at the charts and so do millions and millions of other people when they see this thing breaking out above this ascending triangle they're going to start buying it as well that's just how it works okay so that's the smb 500 let's take a look at the dow let's take a look at the dow the dow has been very flat you can see right here very flat over the last couple weeks not really going anywhere has not hit an all-time high recently here's the all-time high back on may 10th about a month ago and you know what can we draw what can we see well you know the dow just like the sp 500 is going up you have the you know you can draw these these um you know when you draw trend lines it's a little bit more art than science okay it depends where do you want to you connect the lows you know do you want to connect it here and then you can also have you know the resistance line here you draw it over so it's sort of making a little triangle pattern as well it's in the eye of a holder depending on how you see it you know so the dow could keep working its way sideways for a little bit but you can't deny that it's going up and we have the the 20 day simple moving average the the 50 day simple moving average and the 200 day simple moving average all of them are sloping upwards i'll take this off so we can see better they're all sloping upwards which is a good thing with stocks are when stocks are in an uptrend you want to see the moving averages in an uptrend as well okay my rsi indicator down here which is an overbought oversold indicator is right in the middle of the range you know 50 something it's telling me we're not overbought we're not oversold it's just got a nice slow move higher which is good which is what we want to see if you're bullish okay people always try to pick tops people always try to pick tops saying this is it here comes the next bear market it may go down for a little bit but then it keeps going up so i don't i don't get on the short side because the market just goes up you have to be very nimble and have to be very quick if you want to make money on the on the down swing so let's take a look at the nasdaq which was has been the weakest of the three but has really started to catch up now i have these i leave these patterns on here over over the week so we can see how things have been shaking out over time now the nasdaq has been the weaker of the three but when it starts when people start to buy it catches up very quickly okay now we can also sort of see we have the the w pattern okay we talked about the w pattern you can kind of see the w pattern here and once it gets above the resistance line it'll probably start to move pretty good uh let let me just let me just blow this up a little bit so we can see it better so you can see the triangle pattern okay move i mean the the w pattern blow this up a little bit a little bit of manual labor as i like to say so you can see a w pattern is pretty bullish okay apply we got one more do this one right here okay so it's all about looking for patterns it's all about looking at trend lines support resistance so now we have the w pattern which is is a bullish pattern okay all it has to do is get above this resistance line doesn't always happen but it's there for you to take a look at once it gets above this line the nasdaq should start to move up on all time new highs as well so we keep an eye on things see what's happening otherwise you know it's been in this sideways channel for a bit um you know it could get knocked back down and it could start to move back down towards maybe the the 200-day moving average if if there's a lot of sellers that come in so keep an eye on the nasdaq you want to see it get above this resistance level if you do your own charting as well all right let's take a look at some individual stock charts but for the most part i the indexes look strong to me looks like they want to go up especially in that that that sp 500 so let's take a look at some individual charts i want to show you the the ascending triangle pattern on some other some other charts to show you how strong that pattern is so let's take a quick look at cisco cisco systems cisco has been making or had been making the you can see the ascending triangle pattern it's got the upslope and the flat top and just this past week the last couple days it broke through it okay it broke through it it's got the momentum building up it's trying to blast through it it's got all this pressure and once it goes through it goes through pretty strong and it should have some follow through you want to see a couple days worth of the breakthrough before maybe deciding to say okay this is a strong indicator then i want to get on board so wait for a couple days to see if it blasts through cisco looks pretty good so uh oracle as well had made the same pattern okay not that long ago oracle ascending triangle this is this is a classic pattern oracle once it finally got through look how hard it blasts through it's pulled back a little bit and it may be gearing up for the next next leg higher again sometimes when it blasts through it likes to pull back a little bit uh as long as it doesn't fall back through the resistance line or the triangle it should probably it's probably gearing up for its next leg of the move higher okay so that's oracle cisco what else i think was um i think ibm may have made a pattern yep ibm as well here's ibm international business scenes also ascending triangle pattern blasted through pretty good this week okay so you want to look for these patterns on the chart how do you get a little bit more education on these well let me take you to another website which i've shown before chart patterns dot com chart patterns dot com here's the ascending triangle pattern it gives a quick little synopsis of what these patterns are and what you can look for okay you scroll down the page you can click on any of these links it'll show you examples you click on the ascending triangle box right here and it'll give you more of a definition of what it is okay you can see these are these are the exact patterns that i've just shown you on the charts and once it gets through the resistance it's off to the races okay so these are the things you want to look for on the charts it's all about pattern recognition support and resistance and watching the trend the trend is which way is the stock moving okay and we know our physics an object in motion tends to stay in that same motion until something moves it in a in a different direction it's the same thing with charts it's the same thing with stocks here's the s and p 500 it's going to keep moving in that same direction until something comes along and knocks it back down well what what would that be well we had the pandemic last year of february march of 2020 something like that otherwise if this if the company is still making good products and people are still buying it's going to keep moving up until something happens maybe a disastrous earnings report or the ceo of the company has been fudging the numbers for the last five years something like that will will turn the tide but otherwise if the news is decent and earnings are decent why would the stock start to go down it wouldn't so you watch the trend lines you watch the momentum and and you play it until it shows you otherwise okay so let's take a look at some individual stocks we always do here let's take a look at apple we always look at apple now apple is really hasn't gone much anywhere since no last august last september it hit a high last august meandered around hit a new high back early in this year and it's just been kind of consolidating it really hasn't gone anywhere in you know nine or ten months just kind of hovering between you know 110 140 dollars a share and you can see look how tight this range is right here it's a very small range right here it's going it's gearing up when it when it when a stock trades in a very tight range like this it's building up this energy it's going to explode one direction or another and it's been hugging on the 200 day moving average does not doesn't seem to want to drop below it we can see here one two three it has you know had three attempts to go down below the 200 day moving average not a lot of people want to see apple drop okay so they're defending this 200 day moving average and it bounces off and it goes back up but it's been really tight been really tight eventually apple is going to move hard in one direction or another it just needs a catalyst to do that what that catalyst is i don't know but eventually something will happen obviously i wanted to go higher i'm long apple and i've just been kind of waiting around for something to happen and it's just not it's not following through in one direction or another or another so you got to have some patience and this is a thing where if you're an option buyer and you've been buying apple call options hoping that'll go higher this is where that time decay has been killing you right it just apple's not going anywhere so you're losing money day after day if you've been an apple buyer option buyer if you're an option seller this has been great for you because the time decay has just been eating away and and giving you a profitable position all right so that's apple we looked at microsoft uh in that in the example for time decay and microsoft has been in this nice little uptrending channel okay you can draw the channel lines you connect the tops you connect some of the bottoms how far back you go i don't know you know three to six months you you draw some tops draw some bottoms and you can see it's been in this pattern now and it bounced three times off the bottom here so if you're looking to potentially get long microsoft you know follow the patterns if you see that it's bouncing off the trend line maybe it's time to buy so here was your last buy so now microsoft may start to go up towards this top line we can draw we can remove this and and oops let me get rid of this and we can draw a new one you know we can extend the line out you can go from here just draw it a little bit longer so it'll give you an idea of where it may move to if it continues to move up okay so that's the channel um let's take a look at some other charts tesla tesla's had a pretty ugly chart for a while but just like apple it's been it's defending this 200 day moving average this is the the granddaddy of moving averages you this is the last line in the sand and you can see it's dropped through the 200 day moved above it dropped below so it's really fighting this 200 day moving average right here the longs people that are bullish do not want to see it really go far below the 200 day moving edge because that could really spell some trouble on a technical basis so you really need tesla to get going again you can also maybe draw another little triangle pattern um there's a congestion pattern eventually it's going to blast out higher or lower in a strong way you can see all these other triangle patterns that we drew so tesla has a habit of of making these triangle patterns and then and then going in one way but then falling back again so tesla's this is a very ugly chart i really can't decide what what i think about it you can also draw some support here down around the 550 level that seems to be a floor for now that tesla does not want to drop below so you got a floor here maybe um you know if you want to get long at some point on tesla however that may be buying some shares of stock or selling some put options or put option spreads or buying call options whatever you know you want to make sure that it doesn't drop below 550 so maybe maybe you wait to see some more confirmation that it's ready to start moving up but tesla it's a very ugly looking chart i'm i'm not making really an assessment either way in our newsletter we have sold put spreads put option spreads is more of a neutral to bullishly directional trade but we have a lot of cushion okay we have a lot of cushion on this trade so hope we don't want tesla to fall too far but we do have some cushion there let's take a look what else uh walmart is another big one that we like walmart um you know the biggest of the biggest physical retailers uh and so also kind of just trading in this narrow range all the the moving averages the 200 day 20 day 50 day they're all kind of converging they're getting very close to each other when that happens eventually the stock will explode one way or the other um you know let me get rid of this get rid of this line see uh if i can see any kind of pattern here you know not really not so much it's just kind of meandering around um nothing to really gain from this chart just it's got a little bit of an uptrend but overall it really hasn't moved much since last august too just like the apple trade just kind of in that same range nothing really going on with with uh walmart what else do we have some of the the pharmaceutical companies got a boost this week elie lily ran up 20 the other day i think that's the biggest move in one day moving the history of that stock uh i was very surprised so there was some news out uh so this is elie lily murk all the stock whoops i don't know what's going on with murk here's bad tick there's a bad tick in the charts let's not look at murk um bristol mires bristol mires you know had pretty good move this week so some of the healthcare stocks had had a good move higher this week but really hasn't gone much bristol mires let's take a look at um what other uh we have amazon we always look at amazon and netflix these are the big ones these are the stocks that people are interested in amazon still in this wide range really hasn't gone much either even though it's a very expensive stock it really hasn't gone much um since last summer just kind of staying in this range you know this would be great for option sellers uh if you're if you're willing to take that risk whether you're selling strangles straddles uh iron condors whatever this would be a good this is a good stock to do it on okay but not really going anywhere um netflix still kind of hanging around the lows within its channel netflix also has a wide channel here we can we can redraw this we can stretch it out a little bit more okay stretch it out a little bit still in this range but starting to move down towards the lows here's the last move a little bit of a down move so at the moment it's on a downswing where will it bottom don't know yet don't know yet but but if you're long net netflix you really don't want to see it fall below this this long term line not much happening with netflix let's look at some of the meme stocks these have been actively game stop game stop they had that they were up big and then they dropped big so this is game stop um you know once again i'm not playing these but it's fun to watch because the the volumes and the trading ranges are very large it had uh you know we had a triangle here had some sideways action then it finally blasted higher but it's come down a good bit amc also got in on the action just the last couple weeks look at this rally but it's dropped a bit as well so if you're playing these stocks please be careful um there is a there is a website that um let me see i think i have it on my phone here it that shows you the short interest the the way these stocks are why these stocks are moving is because they have a lot of high short interest in them what that means is that there's big institutions and hedge funds are selling short these stocks hoping the price goes down and the uh the redditors the wall street bets group on reddit uh try to buy these stocks up and try to force the these institutions to buy back their shares in a short squeeze so let me see if i could find what's that website that i use here we go so it is the i think it's high short interest dot com let me type this in let me type this in here and see if i get it high short interest okay yeah so here's the website right here high short interest dot com it'll show you all the the highest stocks with the the greatest short interest meaning there's a lot of short sellers now you can see here some of these stocks work course is is the the biggest uh the high highest shortest stock right now 40 of the flow 40 of the available shares available for that stock as being shorted right now okay clover was another big mover this week um geogroup link so you have these stocks that are that are highly shorted so you got to be careful these let's look at um i know clover moved pretty good this week look at that so the stock went from you know you know eight or nine dollars all the way up to over 28 29 bucks but it's come back this week so it it rallied and then it lost half of its half of its move up so you got to be careful with these stocks um those are though there's some of the exciting stocks happening right now but you can you can lose a lot of money so please be careful what other stocks we look at paypal paypal is starting to move out of this little triangle pattern and it's it started to blast higher and it's above the moving averages which is good so paypal could be on its next leg higher if you want to look at possible bullish activity or bullish positions on paypal i'm trying to see what i've seen during the week um procter and gamble okay so procter and gamble let's take a look at this procter and gamble had been making the ascending triangle pattern it was bumping up against the resistance pretty good here thought it might go through but it got knocked back down fell down through the triangle right here so this has sort of been negated you know not everyone works out sort of been negated here so um we we take a look at well i'm going to remove this now so now we we look for what other patterns is there well you can you can maybe draw a little you got your resistance again up here but now you have maybe another little could be in this channel right here now you know it's all in the eye of the beholder that's procter and gamble not much happening now waiting for it maybe it'll try to move back up again uh let's see let's see let's see oh amd we always look at amd still still kind of just hanging around very very tight range this week not much happening not much happening but it hasn't fallen through below this this low line here around 74 dollars or so and up here you know that's just not much happening had a couple w patterns that didn't follow through so amd is just kind of kicking around not really doing much sideways action there let me see what else we got let's see what else we have disney disney is sort of okay so i had drawn this down trending channel before this we've had this on for a few weeks now probably so disney's in this little downtrend channel okay downtrending it's right now it's bumping against the upper line of the downtrend channel so if you're thinking oh maybe maybe i'll buy some put options ourselves short some shares of disney you might want to wait until you see that it starts to move down again because if it pops above the the upper resistance line and gets above this 50-day moving average that could mean disney's on the next leg higher finally but if it if it if it doesn't get through it and starts to tick back down that could be your signal that disney's still in a downtrend okay there's another one chlorox i wanted to show you that which is in a downtrend chlorox has been in a long downtrend here look at this thing it was in the uptrend and then something came along to knock it down and now it's in this this pretty wide this this downtrend channel right here okay so this is chlorox now you don't want to get long a stock in a downtrend until it shows you otherwise here you you wanted to stay long until it showed you otherwise now it's in a downtrend so chlorox has gone from one extreme to the other okay so if you're if you're thinking okay well maybe i want to get short or or bearish position on chlorox you have you have the momentum on your side so i just want to show you these patterns all right so we're getting close to 50 minutes here i think that's all for now lastly let me show you the vix the vix is the volatility indicator getting back down to lows 15 under 16 percent getting back down to those those values right before the pandemic hit back here so we're right in that so volatility's come down that means option prices have come down people are more complacent and that just means the stock market has had this nice slow move higher and that's what happens volatility and market action moving in opposite directions all right so that's it for your saturday synopsis of stock market stocks still look strong keep an eye on this s and p 500 keep an eye on the 425 level roughly if it gets through that a couple days it's probably going to keep moving higher all right so i hope you all have found this today's video helpful give me a thumbs up leave me a comment don't forget to subscribe hit that red button in the bottom right hand corner of the video send me an email we'll go to our website smartoptionseller.com here's our put selling basics free guide if you want to learn about selling put options which is what we do put your name and email address here we'll send your free copy also our services services tab we have our two newsletters and our one-on-one coaching and last week i know i mentioned our deep dive webinar into put option selling that's sort of um still in in process here so if you have interest in a deep dive seminar you know keep keep checking back here or send me an email saying you're interested we'll make sure you're on the list okay that's all for me today i hope everyone has a great weekend and a great trading week ahead this is lee lowell signing off