Hey, my name is Shelly Reed. I am the manager of the Legal Services National Technology Assistance Project, and we're excited today to have Ellie Mattern here with us to talk to us about BYOD, bring your own device policies, and all the things that go along with it. So I'm going to turn the program over to Ellie and sit back and let's listen. Thank you. Hi, everyone.  So no problem on that. I do like to take questions as I go. So if you have any questions, please feel free to come on off mute and ask your question out loud. That would be helpful for the recording. And if not, if you're shy, that's totally OK, too. Please feel free to drop it in the chat and I will read it out loud, you know, to the best of my ability. Shelly, you can still see my screen, right? That's correct. OK. So as Shelly mentioned, I'm Ellie Matter and I am the I don't know if I introduce myself from the Director of Technology for Community Legal Services in mid Florida. We're an LLC funded program, as I assume many of you are. And if not, I welcome, welcome. I'm so excited to be here. So a little bit of a few definitions at first. So what does BYOD need to bring your own device policy? Some of your organizations may have them or may be considering them. Part of the reason Shelly and I decided to put this presentation together is that Community Legal Services was considering putting in place a BYOD policy. And I knew that I wanted to do some research on it anyway this year. And so it was a great opportunity for me to share with the community the things that I had learned and bring them back. So a BYOD policy defines the circumstances under which employees can use their personal devices for work purposes. Devices can include laptop, smartphones, tablets and other devices. I include those other devices because while most of the time people are talking about phones and computers, I also have recently imagined the idea of a VR headset that was someone's personal device, but they're meeting other employees in VR land to talk about compute employee related issues. So it really could be any device that an employee owns. I before we get too far into it, I do want to just, you know, take a little pause on this road and talk about to policy or not to policy. So I think that it is incumbent on anyone who is a director or an organization member to think about, does our organization need a policy for this? There can be a an operational default to say, like, yes, let's put a policy in place. But that's not always the best option. And so our detour, because this is the hill I will die on, is do we need a policy for this? So let's talk about reasons to policy and a reason not to policy. So reasons to policy can be compliance issues. Some of you are probably LSE funded. LSE has a lot of compliance requirements. We have legal requirements as law firms. Funders have various requirements where we need to collect certain data. There those are all great reasons to have policies around things. They're great reasons, largely because of the next two points, because you can instill managerial consistency. That means when a case is closed, it is always closed with one of these few place case closure codes. It's, thank you, Shelley. Yeah, LSE legal services corporation. If you have multiple units within your organization, so if you've got a housing unit and a family law unit to make sure that they are closing cases the same way to make sure they are doing the same things with their cases. So management consistency across the organization is one reason to have a policy. And then it's also can be helpful for your employees. It's a clear expectation. If you have a policy around answering the phone that says, Hi, welcome to Community Legal Services of mid Florida. We're happy to assist you. That means that every time somebody answers the phone, they're going to answer it the same way. That makes it very normative for an employee to know what is expected of them and then how do they go above and beyond that to be like an extra stellar office employee or how do they fall below that in a way that may impact their performance review? So those are all great reasons to put a policy in place for a variety of things. Reasons not to policy. So I employee morale. I have heard from many employees that too many policies can stifle their forward momentum on a project if they feel like they always have to go and look something up. If they just want to get out there and do the work and hit the ground running, but then they feel like they have to wager through a myriad of policies. That can be really damaging to employee morale. And so you want to be mindful of not putting someone in a place where they feel like you're putting barriers to them doing the work that they want to do, to doing the work of the firm, the organization is made for. Management of policies. So as an admin and as one of the people who has to manage these policies, if you have 7000 policies, that means you have to keep track of 7000 policies. And keeping track means has ever been trained on the policy. When was the last time it was updated? And then constantly recycling through, do we need to update this policy? Is it antiquated? Is it useful? So management of policies is really its own door that an organization should be aware of. And then last but completely not least is bias. And a lot of times when we talk about bias, we think about gender bias or racial bias. And while those can play a role in this bias, what I'm talking about is bias between one employee and another employee. So let's just throw out a completely fake policy for right now. And let's say everyone in your organization adopted a policy that said anyone who wants to go on vacation has to give at least seven days notice. OK, seven days no problem. But, you know, the friend of the manager comes to the manager and says, I forgot I have a wedding for my brother in three days. I just didn't put in for it. And the manager says, no problem, go to the wedding, you know, have fun, toast them for me. Well, that's a problem because in doing so, they have created a difference between employee A, who's following the policy, and employee B, who has been given special permission to avoid the policy. And now, if there was an employment law issue at the organizational level, that employee A has a reason to say, well, that employee B had preferential treatment. They didn't follow the policies and I did. Or the policy didn't really mean anything because nobody followed the policy. And so before you put any policy into place at your organization, you want to be consistent. And this is such an important point. I'm going to bring it up again later. No policies unless you plan to see it through. Okay, now I'll get off my soapbox and we'll actually be here to talk about what you all are here to learn about. For those of you who came in late, I do want to reiterate, I love to take questions in the middle. If you have questions as you go, please drop them in the chat or come off mute. I will see you and just ask your question. Okay, so we've gotten off the hill that I'm willing to die on and we've all decided that we do have a way to do it. We've all decided we do want a policy for BYOD. What are the reasons we would want specifically of bringing your own device policy? So here we're going to be balancing security and privacy. So what are the risks to an employer that the employer is solving for? Here, we're looking at not exposing client confidential information or any organizational confidential information. Accidental exposure can come in a lot of different forms. It is so obvious and ubiquitous for people to think about hackers or somebody, a stranger looking on somebody else's device, but accidental exposure can happen in the home. If you have the photos of a domestic violence victim on your phone because you're getting ready for a trial tomorrow or an injunction case tomorrow and your phone or your child, friend, spouse, partner is flipping through your phone looking for photos of your last vacation, you could have just exposed client confidential information to somebody who has no business seeing those photos. And so accidental exposure of confidential data is something that is really, that people should really think about when it comes to BYOD. The next one to think about is hacking and malware attacks. People treat their personal devices differently than they treat their devices that are handed to them by their employer. They just do. And so being aware that you're not gonna have an IT support person or anyone else who's keeping track of what's going on with that device or second guessing, whether there's not any of the traffic coming from that device is safe, is something the employer needs to be aware of if you allow the incorporation of BYOD or employees to use their own devices. And then also you have to worry about stolen devices in public wifi and those two both go down to whether or not someone nefarious is accessing information from that device because it was left in a car, because it was left open and unlocked at a coffee shop, any number of opportunities at that point. So let's take a different approach and think about what is privacy or what is the problem with it for employees? So here we're looking at privacy infringement. A small detour where I say that some of these apply whether you're giving out employee or employer devices or BYOD devices, but we'll keep track of that as we go. So tracking user location. I am right now on my CLS-issued device, my CLS-issued laptop.  The same thing goes for a BYOD device. If you are adding things to the devices to make it easier for the IT infrastructure to track them, which means location sharing, it means browser history, it means app usage, or really if you're looking at anything that is on the employer or employee's personal device, that needs to be included in your BYOD policy, because everyone should be fully aware of what they're committing to when they agree to this policy. Another privacy infringement is, let's say an employee leaves the firm. They're retiring, they're quitting. How do you get with some of their device back into the organization? How do you know what's in their device with many of these? Now, many of you probably work at law firms, and by law firms, I mean law firms or nonprofit law firms. And so I can hear you, you say, Kelly, but the people who leave are lawyers and they're duty bound by their professional responsibility to not disclose client confidential information. Cool, cool, you're right. Except that there are a lot of people in your organizations who don't have the same professional responsibility obligations. You can usually hang a lawyer's bar card over their head and say, thou shalt not do the wrong thing, because you could lose your license.  And so being mindful of who has access to that kind of information and what data they're putting on their phone really does matter when a client or when an employee exits the firm. So we talked about employees quitting or retiring. We're gonna assume that when employees quit or retire, they are largely leaving on good terms. What happens when an employee is leaving and it's not good terms? This employee's been fired and they've been fired for cause. They've been fired for any number of reasons. How do you assure that what they have on their phone gets deleted or destroyed or got backed by IT if required? How do you ensure that that at least even if it doesn't come back to the firm, how do you ensure that it gets uploaded into the case file? Those are all of the challenges that you're going to face if you agree to implement a BYOD policy. Or that you're trying to solve for with a BYOD policy. So we've gotten this far. You're still interested. You're like, look, my employees are using their devices whether we tell them to or not. I get it. In that case, here are some terms you may wanna put in a BYOD policy that are best practices for the employees requiring strong passwords, requiring encryption. This is something for the employer where they provide training on different cyber threats. So at CLS, we have a monthly training on cyber threats every month. It can be on phishing, it can be on cyber attacks. And I truly believe that if you're training people in one area of their life, so if you're training them at work, this does percolate through to their private life because they may just take that extra moment to think, should I click that link? Is this text message real? Those kinds of things, which is what you want them to be doing all the time because it makes this all safer to begin with. You also wanna consider whether you should limit the type of data or the amount of data an employee is allowed to have on their phone. Are they allowed to have photos? Are they allowed to have case files? What about pleadings? These kinds of things you wanna talk about. So, here's a fun thing. I want you all to think about your organization for a second. Okay, let's pretend you're all nonprofit law firms, which many of you probably are. And if you have a communications department and if your communications department uses social media, so you guys post on Twitter, TikTok, Facebook, any of those, does that person have a company-issued phone? If they do not, there's a very high probability that they're taking their photos of outreach events, of pro bono clinics, of anything that they attend for the employees on their personal device and uploading them to social media. Many social media companies require cell phones and personal accounts to be attached to commercial accounts. And so, even if they're not, even if you've issued them a company phone, they may have their personal account attached to that account, and so you need to know what information's going from one account to the other. If they don't have, if they haven't been issued company devices, then the images that they're taking on their phone are things that you wanna be aware of. Again, I can hear you thinking, Ellie, that's not client confidential information. We don't care about that. Great. If you don't care about it, you don't care about it. However, if you like to promote your organization and do reports at the end of the year for your funders or board members, they look at all these beautiful events that we attended, look at all these awesome outreach things we did where people came and received legal information. That content is stuff that you will need. Those photos taken at those events are the proprietary or are owned by the organization and therefore you need to be aware of them if they're living on private devices. And then again, talk about if there's any consent for monitoring. I'm gonna pause here and give anybody a chance, any questions so far, okay? Hearing none, we keep going. How do we limit client or company liability when it comes to BYOD policies? I promise you that we are gonna talk about it again and here we are. If you implement a BYOD policy, you are going to implement it and then you are going to commit to it. So whatever that means. I will tell you that I do not think there's a single organization I've ever worked for where employees were not also using their personal devices for work. And so while this conversation about BYOD policies does paint it as optional and it is, the reality is most people are using their devices anyway. And so to not at least think through what that looks like for your organization and what the best practices should be for your organization is the tantamount to putting your head in the sand. A lot of times with BYOD policies, companies limit their liability by requiring employees to indemnify them. If company data is exposed through their device. That may seem draconian, but I think it depends very much on what the company risk tolerances and whether or not they're willing to assume the responsibility if something goes awry with that individual's devices. So in preparing for this presentation, I thought, okay, so what are the benefits of BYODs? Everyone's pretty much using their cell phones anyway. Why put a policy like this in place anyway? Why should we encourage that kind of behavior? And so these are some of the ones that I came up with, or that the internet came up with. It increases customer satisfaction. And I can personally test anecdotally that having teams on my phone does make it easier for me to interact with people if I'm moving around the office. I am no longer constrained to the notifications I get on my phone or emails that come to me. If I am in flight, I can respond to something in flight. And one of my personal rules is like don't keep people from working. And so if somebody is blocked because they've sent me an email and they're waiting for me to reply, if I can take five seconds to respond to your email just saying like, go for it, then that is a benefit to the organization because there's one person who can continue doing what they were doing without having to context switch to another task. It does reduce costs. So a lot of people say, how does it reduce costs? Well, you could alternatively buy a cell phone for everyone in your organization and ask them to carry it around with them and use only that company issued cell phone which would increase your overhead costs in terms of hardware. It would increase your IT infrastructure costs in terms of management and service. Of course, then you have to have data plans for each of these phones. So there are a lot of costs to doing this all in house by allowing the employee to assume that cost and that risk or that is something that the employer is saving money on. And then you do have the ability to decrease duplicate work. If someone is sitting on their couch at night watching law and order and they have just like a brilliant idea rather than having to write it down on a scrap paper and then try to remember the next day they can immediately go into their notes for that file or put it somewhere that they would remember when they was in work in the morning. So there are really two types of BY, well, three types if you wanna count not having planned it all but there are two types of BYOD plans required and often.  And those can be really good for certain kinds of policies because they apply to everyone equally. Everyone has to know about them. And, but I don't feel like they're the best practice of BYODs and this is my personal opinion and intelligent minds can disagree. For me, I think that often policies are the best option for BYOD policies and I'll tell you why. So the benefits of an often policy is that it gives you the opportunity to put the policy in front of an employee and say rather than this just applying to every single person at the organization I need you to read this and agree to it if you want to be able to use your personal device. And so in doing so, it's not just one policy in the booklet of policies that you hand them during onboarding. It's something that the organization has to or the employee has to opt into. It also gives a sense of autonomy because the employee is saying affirmatively I want this, I wanna be a part of this. It reduces IP oversight. So rather than just the next two are kind of in tandem. So it clearly identifies the number of employees that it applies to. So right now our organization has like 130, 120 people in it. If somebody leaves the firm, there is no way for us to know whether or not they are using their personal devices. We just have to assume they are or assume they're not. By having everyone sign an opt-in policy, which is the plan at CLS, which I plan on rolling out. So CBD, my next update can be on what that will allow process look like. But the, now we will have a way for the IT department to actually know, okay, 89 people that are a part of this organization have opted into the BYOD policy and they have agreed to these terms to be bound by. And so when someone is off-boarding, now we can say, hey, do they have an opt-in BYOD policy? Yes, they do. Therefore, let's add these four steps to their off-boarding, which means reminding them that they owe a duty to clients to not share confidential information, that if they have any CLS data that they need to either delete it from their phone or upload it to the appropriate places on our network. And it allows you to build the infrastructure around how to keep that data mindful. And so in my opinion, I do recommend opt-in policies. So I have written, which I didn't need to click, I'm going to. I have written a BYOD opt-in policy, which I'm going to share with everyone in the chat. Well, I'm not because I can't figure out how to click it, but I will share it with you guys shortly because it's actually made record time through my presentation. This opt-in policy is still in draft form. It has not been approved by CLS. So all caveats and disclaimers apply. That said, I really, really welcome feedback from this group if there's anyone who feels that I have missed a clause or that there are things to be included in the BYOD policy that we haven't already talked about. The other thing is it may be a good jumping off point for your organization. If you're thinking about adding a BYOD policy to consider using an opt-in policy and you're welcome to use mine as a template. So as I mentioned, my name is Ellie Mattern. Please find me on LinkedIn. And I'm going to stop sharing my screen and drop that opt-in policy in the chat. And then I will open it up for questions and maybe we can just have conversation. I'd love to hear if any of your organizations have implemented a BYOD policy and what that looks like. I'm going to throw a poll in if I can get it to work and just to see how many have policies. Maybe I'm not. That's okay. I was going to say, see if everybody fell in asleep. Maybe totally possible. All right. So in the chat, I am about to drop the opt-in policy. Let's try it again. For anyone who is available to do so if you're not on your phone could somebody just click that link and confirm that it is open. I tried to share it with anyone who has the link but I just want to make sure that it works. It did open for me. Perfect. All right. So not getting the poll to work. So I would love to hear how many people have policies that their organizations or don't have policies or considering a policy. And that's why you're here today. So just put, yes, we have a policy. No, we don't have a policy or maybe thinking or thinking about it. While we're waiting for those, I have a question, Ellie. So one of the things that I was concerned about, I had a little bit of experience in e-discovery and all that kind of stuff. So in your research, is there much concern in the community about personal, information on personal phones and then that makes the personal phone discoverable? I haven't done any research on that. I, especially because we're law firms, a lot of the information, a lot of the information that attorneys or paraprofessionals are going to have is going to be either client confidential or attorney work product. And so they can likely skirt the edge of that. And because of that, I wasn't looking into that. However, with public companies, I would be, if it wasn't a law firm, I would be really concerned about whether or not something's discoverable. And also not just discoverable, but I would think about chain of custody. So if you have someone who is working after hours on their personal device and they find something or screenshot something and then the attorney wants to get that in at court, then what does that look like? And I think depending on the way in which that is stored and depending on the way in which they can authenticate it may make a difference. And it could be different in each state, but that is something I've been thinking about is evidence that's collected on a personal device where that live in the discourse of BYOD policies. I see a couple of chats. Doug says they don't have one yet, but they're working on it. It looks like a couple of people don't have them. Steve said we have a mobile device use policy. However, it's not specifically written for BYOD and math has a BYOD policy that's written into their technology policy. Since we're talking about kind of since Amanda brought up technology policies in general, we have quite a few other technology policies as well that are around cyber risk or about traveling with company devices. And right now we have them all separate and I will be instituting the opt-in BYOD policy. There is a question from my perspective as to whether or not it would make more sense to consolidate them all into one technology policy. But I think I haven't made that executive decision yet, but I can see Amanda why you may want to wrap up BYOD into a larger technology conversation and policy. I am, so Jason asks, is the BYOD policy the only opt-in policy we have? I am not sure. I don't, I can't think of any other opt-in policies. I, but I seem to recall one in the back of my mind. It's like scratching at the back of my mind. But I will, I'm not sure. I can't think of any off the top of my head. Amanda said, we have had luck with using Intune to have staff and a little personal devices so they can at least ensure their devices are meeting some basic security policies. That's awesome. I'm going to say that. One of the things that we do plan on using is we use an inventory tracker for our IT issued devices. And so I plan on adding the opt-in policy to that device tracker so that we can see under each individual employee, you know, what laptop they have, what monitors they have, and then also whether they've opted into the policy so that we're tracking it somewhere in our inventory. And I am looking for our technology policy that I have cleaned up a little bit and I'll throw that in the chat here shortly. But I'd love to answer any, get your questions answered. I mean, you came to the webinar today so I'm guessing that you perhaps had questions. This is your chance to engage with Ellie and engage with other people in the community to get some answers. So please take advantage of this opportunity. Jason, I think you raised your hand. I think you can come off mute on your own, if so, go for it. Jason, we can't hear you. So I don't know if your mic's not working. While you figure that out, Mary, I see your question about the percentage of staff using our BYOD. So I've just put the BYOD policy recently together by done all the research on it. And so I'll be taking it to the executive team and rolling it out for the rest of the year. Can you hear me? Yes, I can. Go for it. Oh, okay. Sorry. So, you know, we even set out our, you know, we have a pretty good draft of our BYOD policy. You know, the one thing I guess I'm kind of struggling with at the time is, yeah, cell phones. We, you know, we have roughly 160 employees and that's a lot of cell phones to issue out for, you know, to keep track of and stuff like that.  Yes.   But, you know, just curious to know your comments on, you know, what you've kind of came across and what you kind of resolved. You know, I do like that opt-in policy, you know, that I think that could be very useful in something like this.  I didn't think about that as far as, you know, updating them to, you know, you know, Facebook account and now that's something that we would have to have records of. So just curious to know how you tackle that and what things you thought about as far as, you know, coming up with a policy. So I've actually thought of a couple of things. So there are, on the realm of the spectrum, there's like everybody uses their own device, everybody uses an internal device. And when you think about internal devices, I think, well, that's super expensive. What does that look like from an organizational cost perspective and then also resource management perspective on the IT infrastructure of managing. So for me, what I have thought through is, okay, so who really needs a cell phone that's company issued and what does that look like? And so at CLS, we are going to roll out the, but opt-in, the YOD policy people use their personal devices. We have two, yeah, two company issued cell phones for members of the executive team that needed them for work reasons. But, and this is what I impart to you, Jason. I had an idea recently was that not all cell phones need to get called. And so what we did for our communications department was I asked the IT department and I asked the comm department, hey, does anybody have an old phone that you're not using anyway? And it turned out that somebody did. And so we wiped the phone, it's not connected from any kind of phone service. And I told the, and we delegated it to the communications department and said, this is the phone you will take photos on for CLS. And it will be, it's a company issued phone now, but it doesn't have cell phone service. And so it didn't cost us as much because A, it was donated to the organization for free. It also, we don't have to pay maintenance in terms of every month cell phone issuing. However, now that device is owned by the organization. If they were to leave the organization, we say we need that phone back. And so that was something I had thought maybe a way to resolve people who want to have a company issued phone, but maybe we don't have enough money to issue everyone a phone. Now it could be an IT nightmare to have a bunch of different phones. But since they're not, as long as they're not too old and as long as you're making sure that people are updating them periodically as a part of the IT infrastructure anyway, you could do a phone drive through your organization and say, hey, if you've got a phone sitting in your drawer that's not working, that may be the phone you had before the cell phone you're using now, donate it to our organization and our IT department's going to use those as company issued devices. That's an idea I came up with. I don't know if it helps at all, but that's one thing to consider. Yeah, I think that that is a good idea. We do have office cell phones that people can use and that's before I think really people's, before they really became remote and mobile, but now you have, just everyone's just so mobile. So, and people are calling on clients, they need to call clients. I guess it's, there's so many factors you got to consider I guess is, yeah, who, what are they doing with their phones? What types of services are they texting clients? I hope not. You know, it's that type of thing you really need to think about. You know, we're looking at upgrading our phone system and as they, you know, get in a mobile app and put those on people's devices that they'd be calling and so we'll still have a record of that within the cloud. But, you know, and then the texting would be handled through the, you know, through the telephone provider as well and that wouldn't be directly on the phone would be within the, or in the cloud. So those are some things that we've been thinking about to try to figure out, you know, not, or to alleviate the, you know, issuing work cell phones to, you know, the attorneys. So, you know, those are, you know, some things I'm thinking about and but it, like I said, it comes sometimes down to, you know, who really needs them and if, but also keeping within security policy of that, of what the organization develops. But it is a lot of moving parts. Definitely. And there were a couple of points in there that I, you know, that client communication, you know, you have to have a policy on client communication and any communication documenting in your case management system. You know, so all of these things tie together. It's really important to make sure that your employees understand that there has to be that documentation. So you may allow them to use their own devices, but things need to get put into the case management.  Did your question get answered? In a roundabout way, yes. I was really just curious as to why there might be a need to do a allowing staff to bring their own device. I mean, is it a financial sharing of the burden because I know providing equipment to a nonprofit organization can be expensive. So are you trying to share that cost with the staff or is it more of a convenience factor that that's their personal preference? I'm coming at it from both sides. I'm looking at it from the business side, from a financial cost, looking at it from the IT side, from a management stock's cost, and also the eDiscovery background as well. I spent six years doing eDiscovery projects as well and having to go to a partner's house to do an eDiscovery document collection because he had forwarded emails to his home computer. So it's messy and I'm just trying to figure out what's the, I think if I better understood the motivation or the desire for what is driving this request for bringing your own device might help better formulate our response. And I was just curious just what other people, what's the purpose that you're trying to accomplish by allowing it? I totally welcome this question to the group but I will answer from my perspective. So I joined Community Legal Services a year and a half ago and there are attorneys at the firm who are using their personal computers and always have been. And everyone else is using a, and those attorneys don't want to move off of their personal device. And almost every person in the firm is using their cell phone in some capacity. So at our organization, it is already pervasive that people are using their devices. So it's about how do we mitigate risk of something we know the employees already plan on doing, they are doing and they want to do. And how do we secure that as much as possible? I think if you were to come into an organization that had no employees, you were starting the organization up and you had the IT infrastructure to say, hey, we're gonna issue everyone their own devices, everyone's gonna use it from the beginning and everyone's really excited to keep their work life with work and their home like with home and they're never gonna mix the two, then there's no reason to have a BYOD policy except to say that at our organization, we don't use our own devices. And that is a totally acceptable BYOD policy. We only use company-issued devices. So I think some of it is about knowing where your organization is and how their employees want to work. Yeah, and that's, I understand it entirely. I mean, it's kind of the genies out of the bottle, you can't put it back in there. For several people have already grown accustomed and they've got their own personal lives, which in some cases may be of a higher quality or a higher capability than the ones that I'm gonna provide them. We try to provide people with new quality equipment, but we don't do cell phones. That's just a decision that was made years ago. So everybody's using their personal cell phone and yeah, we've given them a voiceover IP app that they can install. So now you've extended the organization's needs to a personal device. And then so at that point, I mean, like I said, the genies out of the bottle, I was just trying to better understand that'll help formulate what's the right spot. I don't think we're gonna shut it down and start over though. So I gotta figure out a way to make it work. I did see Adrienne or Adrienne ask a question. She or he, I don't know. If you have a corporate dev and it gets confiscated or corporate device and it gets confiscated due to being used as evidence in a legal or regulatory matter, that is not a big deal. What are the implications of the BUIOD? I assume that was a big deal. If you're gonna get the device back, it's not a big deal. I guess there's a question there. I don't ever wanna see a device of mine out. I don't wanna see the device of mine to someone else and allow them to do whatever they want with it because that to me is a breach of client confidentiality. I have, probably two or three years ago when there were all of those stories about TSA asking to take someone's phone or laptop into a private room and do, we don't know what with, I had often wondered what would happen if they came up against a lawyer who refused to give over their device based on client confidential information or even somebody who has trade secrets on their device. I think a lot of that hasn't been litigated but I think you'd see a lot of people go to bat for the, or get the bars involved to go to bat for why that device should not be seated to that regulatory organization. And if I've misunderstood your question, please, I see you came off mute though. No, I have never encountered it before. I know that it is a possibility. And so, that's a huge reason for me not to use a personal device for corporate use because now you're without your phone and everything on it. Plus, like you said, there could be potentially other private data that shouldn't be viewed. So, I was just wondering if there were situations that you were aware of or something like that happened. I haven't seen it and I've been watching for it. I just have it. I don't think it's been tested in court really yet or if it has, I welcome any of you to share court cases with me. I'm really fascinated by what the implications of both of that is, but I haven't seen it yet. Thank you. Also a great question. So, going back to Paul's question a little bit, I think there's also when attorneys work with, say, rural clients who work in agriculture or things like that, many of those clients use WhatsApp. So, having WhatsApp on company computers sometimes can be problematic due to policies. So, it can, or if they're going out in the field, it's very convenient for the attorney to use that app on their phone because they're out in the field, may not have access to, say, their laptop. So, there can be many reasons why use of a personal device for work. I think convenience, I think for all of us, it's very nice to be able to roll over, look at my calendar and know whether I can be in workout clothes all day or if I need to, if I need to be a little more professional that day. So, I think that's where we, a lot of people just use for the convenience of it. Yeah, I think it does come down to convenience. I think one other thing I'll say, Adrienne, and I'm not sure that I'm saying your name right, please feel free to correct me, is while we're talking about BYOD policies today, the first policy that I implemented completely at CLS was a travel and technology policy. And we have basically set as an organization, you can go anywhere in the US that you want, any in the US is fine, and you can have your devices with you. You may not travel out of the country with a CLS-issued device and you cannot travel with any of your CLS-issued, like connected apps while abroad either. Right. And that's, you know, we don't wanna be the one to test that PSA case. Well, no, and not only that, there's a lot of different regulations in different countries, which then you may not be properly following. That, and I also, so the conversation we've had at our organization about can people work remotely from these other countries, and especially I understand that I'm impetuous, especially when you see so many people with digital no-mads on your scrolling, you're like, doesn't that seem lovely? The problem is that I said, you know who doesn't want the responsibility of deciding which nation states are safe to take a laptop to and which ones aren't? This gal. So since it's not gonna be my job to decide, yes, you can go to this country, but no, you can't go to this country because that one's not cyber safe. And also keeping that up to date because it could change moment to moment or at least week to week. And I could have approved someone going with a device two weeks ago, and now all of a sudden there's a really big problem and they can't take their device to that state. And so because of that, you're right. We also don't want any of the regulatory risk about, and so Singapore doesn't actually allow Airbnb's. It's illegal, even though there are some that you can rent. Well, somebody rented an Airbnb in Singapore and then that turned out to be someplace that, or at least they couldn't last time I checked when I was actually visiting. And if somebody rented an Airbnb and then regulatory bodies broke into that place and said, hey, you're not allowed to be here and confiscated the device, what recourse do we have? And just because we don't wanna deal with any of that risk, I told everyone, you wanna go to Puerto Rico, you wanna go to Guam, you're great, anywhere else outside of the US or like anywhere that's not a US territory or a US state can't go there with our stuff. Right. All right, so that's two policies for a price of one. Well, if we don't have any questions, I certainly don't want to take up more of your afternoon, but this is your chance. We still have a few minutes on the schedule. So if you have a question, please let us know. If anybody else has implemented a BYOD policy, I would love for you to, I'll drop my email in the chat. If you don't mind sharing, I would love to see yours so that I can compare it to mine and see what provisions I may have forgotten or should include or maybe three words that would be stupendous. Shelly, if it doesn't look like we've got too many questions, maybe we can stop recording and then just hang out a little bit afterward in case anybody wants to ask another question. Perfect. So I'm gonna go ahead and end the recording by doing my sum up and then we'll turn off the recording and if people have questions, sounds like a good plan. So thank you so much for joining us today. I appreciate everyone stopping by and I hope that this session has been helpful. We always are looking for ideas on what kind of sessions would be helpful for you. So please give us feedback and let us know. And thank you so much.


Lace up your sneakers, it's time for the 19th Annual Rafters College Town Classic 5 Mile Race or 5K Walk. There are age group awards, water stops, and mile markers. Of course, what's a road race without the post-race barbecue? Come and support the Jimmy Fund and the Amherst Police Relief Association on May 20, 2012 at 10.30 am. For more information and to register, go to RaftersRoadRace.com.


One of the reasons why people have at-home diagnostics is because of the rapid growth both in technology which is in a large part due to the boom in the electronics industry but also because of the growth in genetics and genomics information and data. So right now in healthcare there are kind of two paths which are being taken. So there's the at-home diagnostics where the consumer is really doing all of their tests in healthcare but then there's also the approach to healthcare where you have rapid diagnostics which the physician can perform and the key kind of I think transition point that needs to be discussed is if you have someone at home who's doing their own diagnostics which is wonderful and then they get a result how do they interpret that result right is one thing if you find out you're at home you're sick you have strep through it right that's pretty easy to interpret right but if you're at home and you're doing your own cancer test right and you find out your breast cancer what do you do with that information that is something you shouldn't find out by yourself that is something you should find out with a physician sitting next to you and obviously those two scenarios are pretty easy to tell apart right but there's there's gray areas in there so this new area where science and clinicians and lawmakers need to come together to work collaboratively in order to advance the field


I guess we will start right? Yeah, okay. Hey guys, my name is Pawan and that's Janmejain. Both of us work as developers in a company called ThoughtWorks. We work on the studio's division which is basically the product division of ThoughtWorks. So this talk, we call it performance division in the one. It's, yeah, like we called it that but then what this talk has like evolved into over a period of time is that, so this is how you think, like what are all the things that you need to keep in mind? What are all the things you need to keep in mind if you want to scale up your app rather than scale out? And what are all the things to keep in mind if you want to build Hydropotaps is basically what this talk is all about. So this is a gist of all the things that will be covered in this light deck. This is more for people who look at it online like we have shared it on LightShare but we don't need to go through it right now. So the context of what we are doing, the app in reference. So where most of the things that we have done is this presentation is basically a result of the last two and a half years worth of work that JJ and I have done on this app. This app is called Go. Go is basically a release management tool. It runs your builds, it runs your deployments. You can think of it as Jenkins and all of them being orchestrated by one single tool. So which makes Go massively distributed, like scheduling and a process-funding system, which makes it complicated. So the predominantly Go is built with Java as a backend and we use Rails for the frontend for the web app and the UI part of things. It's a managed environment which means that it runs on a JVM, which means that we don't have any control over how JVM does its memory management and things like that. We don't have any control over any of that, it's managed for us, which makes it all the more tedious. Contrary to what people think that it's a good idea to use a managed environment, we actually faced a lot of problems because of that. And like JJ already said, most of whatever we have done and our toolings have worked only, like mainly on Linux, we can probably find an equivalent of that in other Oasis. And over a period of time what we have learned when we do this presentation is from feedback we have got is the approach and learning of what we are going to talk about is independent of text stack. It doesn't need to be like, even though we have Java as our reference app, it doesn't need to be Java specific. What we are going to talk about is independent of text stack. Performance performance, do it practically, good chance that you'll get some, but amount of output which is, performance will likely get out of it is going to be, the ratio is going to be very hard. How many of you have, which has had performance issues when it was required to product? So, which means that most managers are like, oh, last week's had a performance issue, this week should not have any performance issue. I'll assign you like, I'll give you devs. Devs, go and look for, you know, that's an anti-pattern according to us. So yeah, rather good emphasis that don't be out there with the gun looking for trouble. Let the problem come to the right set of tools when it comes to figure out exactly what the problem is and figure out exactly how to fix it, like minimal effort. So that's the key of performance during the workplace. So, the dashboard is low, probably you have like a page which everyone comes with, like a whole page of the app that is low. Or the JavaScript is low, that's the performance issue. Builds are taking too long. In part it's to propose a build server. So it translates. The builds are taking too long after a VCS check-in. That's low if the workers which are running on different other machines and they're talking to the server, they're taking too long to get work when they start executing the build. So that is one kind of premise. We check-in to STM, but code doesn't find out about it in like a one minute, sort of a little time limit. That's the problem. The process of working too much resources, you've given it a 16-core, 18-box. That is required for a basic setup in like one build and one agent. That's not good, right? So basically what that means is in the context of your app, most people think of UI not responding as low. But UI not responding is like a very, like one thing of the app. It's something of the whole. Like on a, probably on a simple website, that might be the only thing that matters. But on a real web application, let's say it has like background posters, passing cards, things happening. They're measuring the throughput at what is low, itself will become a very hard thing to define. So even if your UI may be sloppy, the application throughput itself may be very slow. It might not be sending enough mails, it might not be sending enough messages. It might not be modifying enough files that are supposed to modify the background. It can be anything. It can be, right? So just because your end users are not saying, oh my UI is slow, doesn't mean that your app is actually slow or it has a performance issue. That's what it really means. Like even in our case, that's what it meant. Like all the background things that are supposed to happen. If any of that is low, it means the app has a performance issue. It's still a slow, inspired by users seeing it as really fast. So in terms of acceptance, right? Like most features or most environments that you work on need to have some end point. So typically in your performance fixing work, let's say somebody has come back to you and said, okay, do we have a performance issue on our production app? What do we do? Right? So first thing you need to do is define how slow is slow. Right? Because some people may just be very aggressive. Like they may not have the patience. Like they may have a really huge and complicated setup. Like typically this is the case with products. Like if your product is being in different kind of environments, like if it's not hosted and you know if you're selling it to different kinds of people, like people and setups. And if they have some huge or weird case and they don't realize that, but they're just being unreasonable for that, just to tackle that case. The first thing you actually need to define is how slow is slow. For different kinds of setups, what is an acceptable response time or acceptable throughput is something that you need to first define. Right? Okay, let's say we define that. How fast is fast? You don't want to spend the next one year fixing a problem. Right? You need to figure out, like typically performance fixes, like for geeks, it's actually a big time there, but in a ring. So they typically say, oh my God, this is a really nice problem. Let me just put all my effort in fixing this. But then they may get carried away and they may work on it forever. So for you to figure out when to end it, you need to also have an upper limit of how fast is fast. So you need to know a lower limit of how slow is slow. You need to have an upper limit of how fast is fast. That kind of figures out what is the boundary of your work. And these things have different definitions, right? It might be that I as a manager want my app to have a throughput of 5 meters per second. That's a different definition compared to I as a user want my page to respond within like 20 milliseconds or half a second. Right? These requirements may be mutually conflicting. They may be mutually independent or they may be somehow related to each other. So identifying what is your benchmark is kind of important. Yeah. So benchmark using realistic setup. Like it works on my dev box, doesn't it? I think we all know this. It doesn't matter if it works on your dev box. It needs to be on a realistic setup. And it needs to be like if it's a product, you better make sure like you have like profiles of different usage patterns and things like that. So make sure that it's realistic. I don't even need to get into that. One thing, like going back to the thing that we said, don't go on a goose hunt of where is the problem. The reason we say that is because it's hard. Like it's finding a little in a haystack, right? Instead of what you would rather do is reduce the time to fix issues. Like issues will keep coming up, right? Like bugs will keep coming up. So what you need to do according to us is have a repeatable and automated way to set up an environment where you can easily get to the problem that a customer is facing, right? So that is what is more important than actually, I would rather spend my effort. If somebody feels that okay, we need to fix performance issues, I would rather spend my effort in trying to automate things, but not actually fix anything because what you would rather do is wait until you hit the problem, immediately reproduce it, fix it, and then get on. So that is where, like in our experience over the past two and a half years, that's what we have found to be most valuable. Where we have automated scripts to like, within 45 minutes, we can get to any customer setup with just one execution of, one line execution of the command. That's the kind of automation we have. And the important thing is repeatable is essentially even more important because once you find a problem and once you fix the problem, you want to run it on the exact same load profile in the exact same environment and your profiler should suddenly show that the problem has vanished, right? The profiler will show you this method is taking like 20-90 seconds. Every time the next time it shows like this method is taking like 1 millisecond. Now that's good, but that has to be on a repeatable and automated setup so that you can't make human errors. You can't just go from like 3 agents on go to like 1 agent on go. And then you say, oh, now my point is shutting fast enough. That's not good. Yeah, so like you said, don't go overboard in fixing performance issues. Like contrary to what, like you don't give any hype to it. It's like any other problem on your system. Like the reason we wanted to start our talk with this was because there used to be a time in our product where any performance issues would give jitters to our product manager would use the website to our development team. Everybody would be scared because like they're scared because of the unknown involved in it, right? But the problem is it's like any other thing, like it's software at the end of the day, right? All we did, like it's an expertise that we didn't have and it's an expertise that is not very well documented, but then it's easy to gain. I mean, it just needs some, you know, like our whole approach now is we don't, like if it's a performance issue, we treat it like any other problem. We say, okay, fine, it's fine. Like what's, like it's a bug. It's a part of the bug tracker. It's a part of the bug tracker. We treat it the way we treat any other piece of work, right? So that's the thing that we wanted to start off with.  You know, don't go overboard. You can easily define acceptance criteria for it. The moment you can figure out what is the work involved and what does it mean for me. And if I have automated scripts for it, it doesn't need to get the importance of the risk factor that typically is given. That's the thing that we wanted to talk about. So a typical cycle that we figured out after a lot of fixing and finding issues is this. It's essentially a fish. So you start at this point. You measure what you have. With this measurement, somewhere here, you realize that you have a performance issue. Then you fix the performance issue. You again come back and measure. And you realize oops, my fix hasn't worked.  And you treat all it as an attachment as you need to. Eventually, because what we just said was find your end goal, right? Decide what your end goal is. Once the end goal is reached, you are bought out of this. And you come back and stop. So what is the problem with me starting at fix? The problem starting here is that I have not yet found the problem, right? Like you get a part from the code part. You say, oh, you found a problem. You fix it. That should not be how you follow it at all, right? Saying it's low, all right. That is a performance problem. But I haven't yet figured out what exactly is low. Is it taking out of one second to the page is taking the load? Is it taking 50 seconds? It takes like 15 milliseconds. Or it takes like 500 milliseconds. Or it takes like 750 milliseconds. That's important, right? I want to take the 750 milliseconds first. And you have no idea how many times where we didn't have to do anything here about the wrong memory configuration of that. You know, like, you have no, like, that's why it's very important for you to measure, because if you measure, it may be working fine for you. Which means that then you say, hey, user, do you have enough heat associated with it? Is your process swapping? There are so many things that you could easily find out that you don't even need to get into this place. Because this is expensive. It's developer's time, right? Let's say you're billing at $40 an hour. That's how much money you will spend in order to do this. But if you have had an automated setup already, that would be free. It doesn't require any of those things. And the automated setup will allow you to do this as many times as you want, and you just decide that you want to start any from, that's it. Like, this is the most expensive part of the process. So don't do this. Avoid doing this, right? So that's the message. What if I stop at fix? Like I said, if I stop at fix, I haven't yet measured and informed myself that the fix actually works. And I stop at fix, I deploy it to the customers and the customer says, oops, it still stopped. Guess what, I fixed nothing. So you have to go to fix and come back to measure. You have to ensure that measurement shows you very, very clearly that the problem is gone and that you should make the best of it. So, yeah, that's... Why do I need to iterate? Because one fix that is tried may not work. You may have to try another fix and a third fix and a fourth fix. And also, one fix might affect it by a certain percent and that's not good enough. According to your benchmarks, you've decided that it has to load in under 500 milliseconds. And if it's 550 milliseconds, it's just not good enough. And then you've got to add another 50 milliseconds here and then iterate and figure out, oops, now it loads in for 90 milliseconds on. And in our experience, if your constraints plays a massive role here, like if you are familiar with theory of constraints, most of the times, only the first constraint shows up in measurement. If you remove that, guess what? There are lots of other constraints that have not even been uncovered in your measurement. So the first fix, only remove this constraint. But then there are so many other constraints for you to remove by the time you hit your acceptance criteria, right? So that's why we have to, it's very important for you to iterate. Just by fixing the problem that you uncovered does not make your app fast, typically. And sometimes it's so important that you might have fixed something and actually fixed something. That is a real fix. It's not a fake one that is done without measurement. And you may come back and realize by fixing that and made something else worse, right? Because that's the nature of contention issues. Locked contention, if you fix one, it moves to another place. If that another place is more contented for than the original place, then you've made it worse. Right, and guess what? That other place may be contented for by a more important part of the app than the web page itself or than the backend itself, right? And then you have made not only this thing worse, you have also made other piece that was using this contented portion worse. So it's very important to go into this side. How long should I iterate, obviously? Your benchmarks will tell you. When you have achieved your benchmarks and then you have not made anything else worse, over here you will realize that your benchmarks have been met. So, inclusion, unlike functional bugs, where intuition works marvellously, intuition works really, really bad when you're looking at performance issues. So, please do not go by your intuition. Even the intuition tells you that it's a GC issue, please take a profile and figure out this lock and menu if you're not a GC issue. And when you're absolutely doubly sure, only then go and apply a fix. Always go by proof, be mathematical. The easy and obvious one is page response time. You have no idea how many times we get asked about, oh, do you guys use jmeter? Do you guys use apache benchmark? Do you guys use a flip-up? And most people, when they are thinking of performance testing, they are really thinking of load testing and they are really thinking of loading only the user pages. I mean, that's the team. That's the easy part. So for us, the easier and obvious one is that the not so easy one is how do you define app throughput? How do you measure your app throughput? How do you measure your message queue performance? How do you measure your CPU or your chart? How do you measure your database performance? Lock contention. If I have like really long, one thread holds a lock for a long time. There are like 100 other threads which are waiting for a lock. And if it's in a managed environment, how do I even get that information? Because everything on your vision is fine. Your IO is fine. Your CPU is fine. Your CPU is fine. But my app is actually slow, right? So probably it's a lock contention issue. How do you even measure that? How do you measure the environment-specific things? Like memory usage and GC channel. How do you even figure out environmental issues? My hard disk, for some reason, my hard disk is slow. Like Ram was our ops person for a long time. He knows of a few instances where our own production server had a problematic IO driver. And we spent a lot of time trying to figure out why the hell is my DB taking so slow? Like, oh, my 20s like really non-performance. But it turned out that our performance had dropped about 16 times because of a bad disk. How do you even measure such things? Those are the things that we think are more important when it comes to performance testing. Like using Jmeter and AP is easy. Using these things and understanding and interpreting the output. And trust me, this is a lot of data. Like JJ actually plotted a 24-dimension queue in order to plot all this data. It's a good intellectual thought experiment. We'll keep it aside because there's no correlation that can be established. So that's something that we quickly realized. So you need to have people look at this output and figure out what the hell is going on. So that's why we think that what to measure is a super important thing. And don't just go with Jmeter. And don't just go with AP. Think of all the things that you need to do. And remember, understand your environment. Like we said, we know how to do these things on Linux. But we have no idea how to do these things on, let's say, Solaris or on Windows or on Mac. We don't know. Probably for you guys, if you have deployment from those environments, you need to understand it really well. For instance, something for us to... Like we have very dp heavy. Our dp is always under cvr load. If I don't understand that my dp is under cvr load and it's bringing a lot of IO. And if I don't understand that my server is running on SSD and it is indeed running on SSD, say. In that case, if I use anticipatory IO scheduler on Linux kernel, I'm sorted. Anticipatory IO scheduler is just not made for that. It's made for magnetic lists. You are using the wrong tool in the wrong place. It's like round pegs and square holes. It doesn't make any sense. So, yeah, those kind of things is what we need to understand. And the reason you need to understand all this is because we are talking about scale up. Because it's very easy for you to scale out. But for us, the really interesting engineering problem is scaling up. So that's what we are trying to address here. Because most of the times, your CPU is really powerful. You have really fast RAM and you have really fast IO, right? And your network is fast. We are saying, okay, fine. We'll not leverage any of it. We'll just scale out. But for us, that's like a pop-out. What we are trying to do here is let us understand the computer science behind it. Let us scale up. So that's what this is all about. So the tools we use was, yes, for user load, we use something called Apache Benchmark. It's a very nice tool. We use the ht-perf. That's also good. AB and ht-perf are native. JMeter is actually on Java. So we actually found that JMeter itself has performance issues because of which you can't really load up to what you want. So for us, AB and ht-perf is a much nicer option than JMeter. So the app throughput, so we typically, for us, like, let us do, so Java has this thing called jstat and jmap and things like that. So you can do ht-scan. So there are Java JVM tools and there are equivalent tools for your VM. So you should probably start like, exploring those tools where it gives you these kind of counts. Then you have, we use something called ActiveMQ for our queuing. ActiveMQ exposes really nice statistics over management console. So we use that for measuring this. It might be rapid and queuing. So most queuing systems do have this in mind when they are queuing. So any serious queuing system will expose this data. Then in order to make a CPU or your churn, the most obvious one is the load average. Then we have iostat, we use iNutrify. For DB performance, we use a profiler called Yo-Kit. If you are working on a JM, you are great. It's like, don't bother looking at any other thing. It's not that expensive. It is comprehensive. It's one of the best JM profiling tool out there. I don't know where they would be without Yo-Kit. Yo-Kit is like, when you perform an open heart surgery, all the tools that our surgeon would need, those are the things that Yo-Kit provides. Yo-Kit is available for Java and for .NET. So people are working on data environment, it's there for you as well. Yo-Kit basically shows out every single thing that can ever be anything similar for Ruby. For Ruby? For Ruby, you can generate a snapshot that you can look at. For lock contention, again, Yo-Kit? If you are dealing with other environments like non-java and non-.NET environments, you can probably if you don't find anything that monitors lock contention, you can write your own extensions. Locks are normally implemented either as spin locks or semaphores. And you can just write rapidly around that and you can create a specific lock. And you can just grab and move on that map to figure out. Yeah, so again, manage environment specific gc locks. So you can just look at gc locks or equivalent of that anything. Environment specific things. So views like net stack, ps, dmessage. Like yeah, I mean dmessage. So basically anything that will tell us the machine itself is so This file is incredibly useful. We are trying to reason anything in the user space. It's like the most compact source of information from the bundle site for you. So yeah, this file is amazing. Go back and look at it. This is obviously the next bundle. I mean, going back to what you said roundpicks for round holes. So figure out what is the right tool for you and use only those tools. And in order to be able to use what tools to use, you need to understand what next should be like that. You know, that's the chain of reasoning. So tools comes at the end. The understanding comes first. So try to understand your environment your problems and then figure out what tools to use. With least possible overhead here. Like I just explained, right? If you don't have a lock condition profiler you want to write like a simple enhancement to your runtime. But remember if you start if you start flashing to this your lock messages for your lock condition profiling every single time then you're probably going to be locked on IO all the time and you're not going to get like right kind of data from that lock condition profiler. So ensure that you ensure you expand your buffer size appropriately. A professional profiler like your kit is like really really good at doing that. No, your kit is not open source. Yeah, that's a sad thing about your kit. Be sure that your buffers are like directly configured to the right size. If you're at the right time, there's a dedicated set of lock when it's trying to flash. So make it all them. So what did we fix eventually after going through all that exercise? We are going to get into the nature of fixes in some way. So it's important to understand your library. It's really the fine print in documentation. If you can afford to read code, it's often that problems that you see is not caused by your happens. It's caused by a library and fixing that issue is really as simple as just taking this library and stopping with another one and then patching it and applying the patch code production to it. So yeah, don't miss out on that one. Don't assume that all libraries are like perfect. Be mechanical. Start with measuring code to fix, come back to measuring code to fix, come back to measuring and then stop. So don't try to skip steps because you've done it like 10 times. Oh, now I kind of know it. You want to go wrong. Which is where we say execution works. So the last 10 times you have to go wrong. The last 10 times you have to go wrong. So execution works very, very cool. So you should be okay with using an exaggerated code to magnify the problem.  It's okay to load that page really heavily so that you see how to fix, right? You see a difference from 5 seconds to 1 second versus 5 milliseconds to 1 millisecond, right? So that magnification sometimes very clearly tells you that you are doing the right thing. So feel free to use the exaggerated code and then add concurrency, higher number of requests to get the right data. You can use the same you have to use the same code profile to test your fix that you use to reproduce the problem, right? So that in the same situation your code is better now compared to rigorous add up the numbers. If you have like five calls in a method and each call takes x, y and z amount of time, you have to add that x, y and z and then decide the method's time. Over that x, y and z and figure out what the delta is. Is this method taking all the time or are the calls taking all the time? We are interested in all this reasoning, don't just be superficial. So that's important. You should think about varying on complicated fix like you to see all the corner cases where you are trying to fix logs, right? So you don't introduce a race condition, you don't introduce some kind of data corruption, not foregoing a log statement that helps you reason out in case of bugs. Typically in logs it's safer to be more granular than less granular. Whenever we reduce the granularity of logs we are generally a risky operation and the risk is data corruption. There is no performance but it's risky, right? So that's what we attack. Fix problems, don't attack three problems and then think okay alright I'll fix these three problems and then I'll profile again and I'll measure and verify that everything is fixed. No, don't do that. Attack one problem at a time so that you know exactly what is the impact of the fix on other areas, right? If I'm fixing my page load time, a scheduling should go for it also. Because then I made a more important area worse and that's not good. I mean, I know that we know we can take on four problems at the same time and the same patch fix all of them. It's okay, don't do it. It's fine. The overhead of So having said all that, correctness always comes first. It's okay to have a slow binding application. So these are just what we just covered. So let's say somebody told us what the problem is. We understand how to go about doing it. Now what are typically the nature of the problems that you fix in terms of lock contention. By synchronization we talk about lock contention, right? So typically what happens is if you look at one class calls another class calls another class calls a DV, right? So typically you have a lock here, you have a lock here, you have a lock here, you have a lock here. So let's say you made this granular that doesn't matter if this is very like it's very wide. If you fix this, this could still be wide. If you fix it, everybody accesses the database, so database lock you know. So typically when you fix one lock, it has the nature of granularity of locks once you fix one place it keeps moving on. So it's very important for you to keep that thing in mind and fix it. So more than one shared, if you have one, two data structures which get accessed across different methods, it requires synchronization. You just cannot do lockless concurrency there. I mean it's just theoretically impossible, right? So don't even bother trying. If you have more than one shared memory and things that get accessed, it's just you have to synchronize. There is no way you can synchronize it. It must be particularly proved. Transactional memory is a repair. But even transactional memory, if you have two different transactional things you have to put them in one transaction otherwise you can't have two different transactions. So that's the equivalent of this. So you have to put them in a single one. There are globally unique objects so interned objects are pretty useful. You'll see how we leverage interned objects. John has a notion of Permanent Pool. It's got permanent generation pool. So Permanent Pool usually keeps things like classes, it keeps things like unique copy of a string. A string pool is equal to a string pool anywhere else in the system, but it might not be the same reference. Interning means that it is the same reference essentially. The concept comes from common lists but it's used widely in Java.  It's xc and pxc. Comparance of it. Fixers, jumps, places unlike functional bugs. Functional bugs are typically I did not take care of a corner case. I fixed it. It's localized in one place. Locked intentions are typically they move all over the place. So you need to take measures properly. So look beyond local impartments. You haven't made anything worse. Okay, so nature of fixes with respect to GC, GC's garbage collection. When you hear from a user that unresponsiveness is intermittent as in once in like 40 seconds I see it being very slow. Otherwise it's very fast. It might be a GC issue. It is likely not easy to fix because it may begin either in your code base if you are lucky or in somebody else's code base and it's a library that we are using if you are unlucky. If you are really, really unlucky, it's code. So yeah, because maybe you are a library, like I said the use for GC is Java's GC logs. I'm sure other VMs expose GC logs as well. If they don't, it's really simple. You have to go to the garbage collection and start logging there. Again, take care of offering if you are doing it yourself. Provide you to see if majority of allocations are just from a handful of calls that means if there's a method called foo, bar and bars and these are the three most important methods in my app very frequently called.  If there are 200 such calls that are generating 1% of the garbage or 0.5% of the garbage then it's very hard to fix that. Each one putting in is extremely hard to fix. Then try to play with the GC arguments and try to figure out what is the right kind of...  Unless you have one full year to only fix that you will probably not deliver it. So you would rather fix GC algorithms of your VM than use the right algorithm for you. So throughput collector is probably not the right thing to do. You're not even the content. So with your maximum omitted allocations now only if the VM says alright because I'm a VM, I can let you so you've got to know the maximum limits like that. Don't configure them far apart because if you configure them far apart it might start crashing. It's a modest cost for one but when it crashes your request will be admitted please go. So malloc and free are system calls that are expensive so try to not use them. Going to skip over DBDB has got nothing to do with managed environments. It's nice insights but you can read it up only. Come back to thrashing. So thrashing is about number of processes and your number of VM cores. So if you have like two few cores and you have... what's going to happen, context which is going to happen. Every time context which happens you'll see few caches get quite cold. If you are not on x86 your tlp cannot be populated without what is it called? page fault. Page faults are expensive and blah blah blah so you don't want that ratio to be extremely odd. A little bit of odd ratio is fine because you are normally blocked on ION and not on cpu so that makes the world happy. If you have too many logs that will cost too many sleeves. Logs are normally permitted as semaphores so the semaphore keeps so semaphore registers the locking thread on a listener list of the locker and when locker comes out it says oops the first looks like the thread is waiting for me so I just wake up the first thread and I will let go. And the first thread gets locked, the rest of them keep sleeping. So sleep based logs cost context switching very important. So like I said, context switching gets your cpu dashes l9 to l3 so yeah being it's in the game. Two frequent ION obviously ION weights are locking. When you try to read something it will wait you all just cannot return unlike mine. So if you go to the disk it will come back from there and return your data. If you are lucky it will be cached but most of us I guess are no conscious of them in the way.  It's like auto of magnitude slower so you never trust your disk. Overlogging can be hurtful neither your OS nor your NVM cache will be able to save you if you are overlogging. Like your login will start to waste every one millisecond.  Concept of buffered ION exists for a reason as in even if you may as well load a huge chunk in one shot than loading like one line and keep loading it. Keep a buffer over there you can use memory mapped ION If you are using memory mapped ION please don't close your file descriptor because then memory mapping also goes away and then it comes back and then it goes away. Assigning very low max memory limit can cause 2 pin GCs. The way GCs work in a native thread system like Java is all threads wait, I'll go and garbage collect I'll create some memory I'll remove the start and then I'll create some memory. So basically crashing is about all the, crashing is basically about all the common pitfalls for us when this is pitfalls that we have faced, there may be other pitfalls that we might have or will face so the first important problem we had was prejudice. This prejudice comes from intuition so we had this bad notion called SSL is bad but once we made it go through our measurement exercise it actually turned out that a lot of our problems was not because of SSL. In fact it was not even close to what DB was causing us in terms of problems. So using a synchronicity to avoid fixing root cause. So I have a lock, what do I do a user clicks on something it takes forever to come because it's locked on the back end so what do I do? I return the user thread really fast requested returns with that makes the click really fast but the actual through portable system is still slow. So just by introducing a message was there does not make your app fast. In fact it makes your app slow because it's asynchronous the real work that will happen will now happen at its own sweet time which means instead of synchronously finishing it in 5 seconds it might now take 15 seconds or 20 seconds so you need to be really, you need to understand exactly what it means in case of java it again means object serialization. I will read that the entire object graph gets serialized sent across to the other thread that's what it says ok and now I am ready to read it it reads the entire object graph and then it starts processing it. So many extra synthesizers wasted there. So for us the prejudice was oh use a message to there you will be fast. And then make the mistake by what point so yeah. So that comes from prejudice lack of testing we can't have more than 150 workers but the that myth got carried over for like we are a 5 year old app that myth existed for the first 3 years and about 2 years ago we said ok fine what is this let's just go and test it and then we could easily go up to 1000 agents and 1000 workers, workers lives and nothing happened and we were like what's that right so it's all about in any project you have you have prejudices it's very important for you to keep all of those aside be mechanical be rigorous. So that's why lack of setup understanding we have a misconception that 64 bit JVM is slow but guess what for all our like we asked Ram to move his current setup to a 64 bit JVM because we just realized that oh that has got nothing to do with the slowness all the time like it does have some overhead when you have smaller heaps sizes but for larger heaps sizes it's fine it doesn't really make you that slow. And it says that 64 is so much faster that you will not believe it is MMU can address what 16 times more more references with with one page than 32 bit it has a dedicated system call instruction there are two ILT 0x 0x 80 anymore so your register is up bigger there are a lot of these things that are it's a sick tool obviously faster and typically 32 bit JVM or 64 bit JVM has to go through an emulation right which means that emulation adds overhead. Yeah so there is this native emulation layer that allow you to address it. Yeah basically like this was clearly a lack of understanding so these are common pitfalls that you run into a long running project yeah like so we spoke about this already they are uncharted waters in all projects so you need to be careful about them you can read this up online if you want we are running short of time so yeah so know what you are getting in the like the most of the time when you jump into the deep end of the river it's not only the problem is not that it's deep it might have crocodiles in them like these crocodiles are unknown basically you don't know what you are getting into so make sure that you know that there are crocodiles waiting there for you and these crocodiles are what makes everybody scared of performance issues but if you know how to get around them and how to get around them instead of being here you have to be here exactly that's the so again you can watch stories typically like this is a tour presentation that we do these watch stories are all the things that we experience in our project and how we fix them but we can't share our code in a public forum so what we have done is we have gotten the same kind of problems in a sample code and we will show you how it was before and how you can probably fix it so the what we are doing here is basically we want to fetch an XML document from a remote service the fetcher, the client has some configurations which are persisted in a database the configurations name is passed inside is passed so you say okay use this client give me the data use this client let the client fetch the document from a remote service and give me that so that's what this method does so you go to the repo use the name and find that client if you don't have it you create a client and you put it back into the repository or persist it then what you are doing here is you want to eagerly fetch all the documents that are available in the remote service and cache it locally so that you don't want to keep making remote service calls this client exposes a good page count number of documents and you are trying to make RBC calls to each one of them and you are putting them into a cache so you took all the elements that the remote service gave you you want to cache it, you want to just log it and you return it and please notice that it has service level check level synchronization over there but then somebody who wrote this code knew that it was synchronized because you are caching it and cache data structure is being accessed if multiple people access the same cache data at the same time you need to synchronize it so it's obvious and the other things also if you are creating a new client you want to implement the ID of that and somebody outside wants to know how many clans your system has so you have this little better method over there that is also synchronized this is the implementation somebody has done this deployed beautifully works on their local box they even took care of caching they deployed it this in production this particular method call got 1000 concurrent requests at the same time what do you think happens yes lot of caching happened so the answer was lot of caching happened and yes that is true it is over caching what else and that locks over same object same time there are subtler problems here if you see you are doing a log dot debug here right and this string gets evaluated this anyone has any idea how big this thing is an analyst of all the DOM that you have loaded which means it's going to spew out pages and pages of stimulus or example probably right and it's locked at debug most people don't run their production apps with a logger level set at debug which means that you are computing this string and logger is ignoring it right and in java because strings are immutable one string two strings three strings multiple strings here which means that every time in my example of 1000 concurrent requests so there are about 4000 strings that were constructed which got completely ignored so this is done where the moment this stack frame goes out of stack frame pops the reference to this is gone so it garbage connects and guess what in production you are logging in for you are not even using the string right exactly it's insane amount of garbage being generated for no apparent reason right so that's a subtler bug then synchronization is here which means that this axis and this axis is blocked and then there is one more subtle bug you are caching an array list right most caches if you have configured a cache cache has this notion called number of objects in memory in this case there is only one object in memory and that object has this many items inside it and each one of them is huge so if there were 1000 accesses to different clients let's say there were 20 clients cache tells you for issue 20 objects but the cache size is 3gb right that's nasty that's really nasty and probably cache explorer management you say oh my god my heap has gone massively high and cache obviously has a odd reference because you don't want it to gc so you have a 3gb hard reference data if you look at cache you have 120 of it what the hell is happening right so you immediately discount cache cache is not problem right right so that is where like these are all the like I said these are not very obvious problems but these are all really massive problems in a highly concurrent app with high throughput requirements so now we are going so we had a fixed thing in the middle which would show you how it looks with fixed and very quickly show it to you it's very ugly so we are going to refactored fixes so to quickly cover because we are running out of time we have wrapped this with these log dbox we are not causing unnecessary gc churn unnecessary gc churn the synchronization keyword has gone from there it has gone over here instead of instead of caching all the elements and caching an array list and caching each element separately so my cache actually reflects how many objects it has how many we have objects right and I don't have transactional anymore so I am not trying to cache in transactions because the transaction gets rolled back by caching under if you guys want just a quick dumb call what happens if you cache inside a transaction which is offline it's a crazy crazy problem you don't want to cache inside transactions even if you have transactional caches it is a crazy problem yeah so in a concurrent app so we can talk about it offline it's a really insane issue the call get client here so you can the key over here is for the element so this entry is being cached with this key synchronization and the get client call so this is called a double check clock so it's called a double check clock extremely useful construct and synchronization yeah so like what we are trying to do here is basically we want to show you all the things that could go wrong and we have fixed a lot of them and this is only 20% of the actual slide actually go back and read the slide it already makes sense here again double check locking here this one and this one and I can try to get it from the cache you can't find it take a lot of credit from the cache you can't find it to an actual file get it if it's still not then create it the create method itself is transaction but the whole method is not so I can legally put things into my cache without going wrong right and the synchronization keyword is gone away it is only over here so we basically reduce the granularity of the cache we reduce the decision we are caching the right things in our cache cache will tell us now the right number of objects in the cache and then there is something and how do we create the synchronization that's really important this is an intern string string being interned over here so it's a globally unique reference across the entire game so I can literally lock on that the create line method is transactional but it does very little so that's nice I'm not accessing any cache over there the synchronization here is also gone away if you look at it it's an autonomous operation that I'm using increment and get it doesn't use comparison swap it will increment it by one try to do comparison swap if it fails it will increment by one try to again it's a loop right so this is called a spin lock in the kernel terms it's called a spin lock this prevents process preemption in the kernel space but over here it's not even semaphore so it's not making you flip non-schedulable then flip to schedulable piping your cache is clean so this is a really nice constant go back and I'll look at all the atomic operation the new game supports so with that really yeah so if you are interested this is probably about 20% of all the things that we have covered in that questions will be off by any current question back and forth thanks guys


Hello, everyone, and welcome. My name is Melissa Armo and I own a company called the Stock Swoosh. And today I thought I would review the tracking results for the month of October. If you'd like more information, you can email me at Melissa at thestockswoosh.com or call me at 929-3200-GAT. If you'd like more information, you can pick up a phone and call me or email me. And you can feel free to watch me on Fox Business Network, where I will be discussing stocks and the market. Main trading is about chunking it out. You're looking for a move.  I like to focus on shorts, occasionally I won't go long. I focus on gaps. I focus on stocks that are gapping. It's a system I created where I look at 26 points every day and I rate the gap. And based on the rating system, that's how I determine what I will train that day, whether long or short. As I said, however, I prefer it to short. So the results for October, 10-2 was a TB day off, 10-3 was TTS, 1500 profit, 10-4 Tiva. Tiva's had a lot of nice downward moves, 3,000 profit, 10-5, no trades, 10-6 was cost, 1,100 weekly total of first week of October, 5,600 average per day. That first week was 1867, 10-9 was cost, 3,400, 10-10, Dow, 625, 10-11, HDSN was a loss. 10-12 was a profit, that was a good one, 2,800, 10-12, JPM was a loss, Tiva was a winner again, 1,800, 10-13 TB day off, weekly total, 6,525, average that week, average that week per day, 1,631, 10-16 was a loss, Apple was a winner, 2,800, 10-17 was hog, loser, 960, NWI, 2440, winner, 10-18 spy was a loser, IBM was a really nice gap, 3,400 profit, 10-19 was Apple, that was a nice one, 3,285, 10-20G was a loser, and CLG was a winner, 2,800, it was a big week, 11,204, 10-16 week, average per day was 2,241, then 10-23, Matt was a winner, 1250, 10-24G was a loser, 1,200, low G was a loser, 1,000 and Matt was break even, 10-25 Juniper was a loser, 1,200, AMD was a big winner, 6,300 was a big day, 10-26 CLG was a loser, and CLG was a winner, so there were two trades in the CELG, missed the first drop in the exit on CLG, we took it, had a really nice move in the afternoon, late morning, 10-27 TB day off, weekly total 9,350, average per day that week, 23-37, 10-31 was MRK, which was a winner, 930, 1031, UAA was a 1,000 loser, QCOM, two winners in that two trades, that had a big move, 1,000 plus 1,500, so October total was 35,080 bucks, average per day, 1,949, this is with an advanced risk of $1,000 or more per trade, and when you look at the whole, like I said, when you're looking at the whole month, you have to be very specific for what you're doing, because you can't trade days when there isn't anything that meets your criteria, there were days in there was on television, so I didn't trade, however there will be days where you don't get any good gaps, and there will be some trades that are losers, most of all though, it's very important to get out when you're up, and to focus on the good ones, and to not over-trade, as you go back and look and see those results from October, you know, average trade per day is one or two, max, and I really think that's very important, I really don't like to be in trades long, and I don't trade all day, even though the market's open until four, as a day trader I'm focusing on the morning, because I focus on gaps, and if you came and learned and did my gap course, that's what you'd learn from me, how to trade in the morning, how to pick which stock is going to have the momentum move in the morning, so the October trading results average per day was $19.49 a day, and that's if you did an advanced risk. Either way was a profitable month, and if you want to be a day trader, it's important to have a specific strategy and a structure, it's really a structure that you're following every morning, instead of going all over the place, and I think a lot of traders miss that, they don't have a structure. The nice thing about being in the trading room with me is, you can follow me and do the trades, I call the trades live in the room, so the class is called the Golden Gap course that I teach, it's a fall two-day course on how to strategically find pick-and-play stocks at our professional bearish gaps. The class is online and you can be anywhere in the world and take it. Only have two more classes before 2018, if you're interested, you can do the course November 18th and 19th from 9 a.m. to 5 p.m. Eastern time, cost of the class is $4,999. If you want to sign up, you must email me for sign up registration papers at melissa at the stockswish.com, and again like I said, all those trades are called in the live trading room, the entry, the stop, the exit, and honestly, if you learn my system, you don't need me to give the calls, but I think that it's important to have the support of being in the room with me right after the class. It helps you make money faster, right away, and also get the system, understand it, and apply it for yourself to do for the rest of your life. Really, gaps work for swing trades, options trades, long-term investments, or day trades. I just happen to like the quick day trades. So empower yourself to trade, to trade the market, you can do it. If you'd like a free trial for this week, you could email me. I give a one week free trial at melissa at the stockswish.com or info at the stockswish.com. Don't forget the upcoming class dates. 2018 is almost here. October is a good month. We'll see if we can replicate and have a very positive November and December going into close out a very strong year. Thanks everybody and have a great day.