So this session is about benchmarking and performance testing. We had a session yesterday where we talked about how we're doing some of the automation at hike. And that was more on the functional automation side of things. This one is more specific on benchmarking and performance. So KP, Prathish, and I, I'm going to kind of co-present this one. I'm going to give you a little bit of background, some context setting, and then Prathish is going to jump in and do a demo of some of the things that we have. Unlike yesterday's talk, this will probably reveal more information for some obvious reasons. Here, our goal is to actually walk you through kind of our journey of how we started in 2014. And up till this point, how we've done some performance testing. How many people are aware of hike? Wow, awesome. The numbers just went up, right? So we have, as you guys know, about 100 million users using this out of that, about 40 billion messages are exchanged every month. So from a volume's point of view, I think we are one of the top apps right now in India being used, which obviously means that performance testing, benchmarking, becomes extremely important for an app like this. But I want to hear from you, right? Why do you think we should care as an industry? We should care about performance and benchmarking. What are your thoughts before we get started? Why should we care about these things? We as an injurer, not just hike, right? Any other company. Why do they care about performance and benchmarking? Want to make this interactive, right?  User experience, OK? That's an important one. What do we mean by user experience? Want to make sure the experience on the app is seamless, right? Everything scrolls smoothly, loads fast. There is no lag. People get to what they want to do very seamlessly, all of that stuff, right? So that's a very important aspect from that point of view. Anything else you guys can think of why this is important? Time, but that's the same point, right? I mean, we basically want to give a seamless experience to the user. That's only part of the problem, in my opinion. So a non-performance app can actually impact the behavior or the functionality of your application itself, right? So it's not just UX part of it, but it could also lead to people not being able to do what they want to do. So if I send a message to you, and it's an instant message, and it took three hours to get to you, right? Even functionally, you got the message, but it's kind of broken, right? I mean, it's not instant anymore. I might as well send a pigeon over, and you'd get it faster. Anyway, here we're going to talk. So we're talking about these, and there are other factors why benchmarking and performance is important for sure. Let's actually jump into and look at from a user's point of view, how they would see an app that is not performant, right? This would be a user's experience, and we certainly don't want people destroying their phones. Only if Hike was in the business of making phones, we would love that. So let's talk about how this all started back in 2014. We were about to ship out a release of the latest, greatest version of Hike. We were shipping every two weeks back in that point, and we were about to ship a release, and at this point, we didn't really have, we were just starting our whole automation effort inside Hike, and someone was basically manually testing the app, and they said, you know what? When I actually force-killed the app and started again, it feels like it's taking a little bit longer. It doesn't feel like it's very crisp, and it's very great, right? We didn't have really data to say what that means. So we said, okay, let's actually try and pull some data, and we actually kind of ran some data, looked into this, and what we found was there was a shoot of 130 milliseconds on the app launch. 130 milliseconds, who cares? Yeah? But 130 milliseconds, cumulatively, over a period of time, soon hits 300 milliseconds, which is human-eye-perceivable, right? So you can actually notice a 300 millisecond lag, and that's noticeable, so we wanted to basically kind of catch that and stop, and actually if you see release on release, the blue one, the red one, and the yellow one, we're actually seeing that every release somewhere we are kind of shooting up slightly. So we said, you know, soon this is gonna hit a point where this is gonna cross the level where when a user launches, they actually see the app still kind of launching, and that's a bad experience. So we wanted to address that issue. So we decided to actually pause the release. We decided that the release won't go out. We need to actually find out what's the problem before we ship out this release. And you know, obviously when you take a call like this, many people will jump on it, right? Because people want the releases to go out on time. So we had a lot of stakeholders jumping in, and at this point, a lot of people were saying, here on my phone, this is working perfectly fine. I don't see any lag. Other people were like, no, this looks like some lag. And it doesn't seem like a scientific way about going about an app that is used by 100 million people, right? So what we decided is we decided that we need a little bit more scientific way around how we do this. And so we decided to actually look at analytics to make certain decisions on when something will ship, when something will not ship, and put a basic framework in place that will help us make more informed decisions, right? So the first part of this talk, we're gonna talk about how we went about creating this basic framework which helps us benchmark things. And then we'll talk about diving deeper into the performance side of things. But when we talk about benchmarking, right? What do you guys do typically in your company? Do you pick random 10 devices? Do you pick random 10 use cases and try and do performance testing? Or there's a little bit more structure around this, right? So you look at, in the market, what are your top devices being used by the users, right? So, sorry, correct. So top devices that your app is installed on, you wanna basically look and do performance testing on those app because that's gonna give you the biggest buck for the bank, a bank for the buck. What else would you do? That's only one part of the story. So you wanna look at specific parts or use cases of your application where you think things might not be most performant and you might want to do that. So one is looking at the devices, one is looking at the use cases. There's more to it, right? Crashes, I wouldn't put that into this segment of the bucket that would go into the functionality side of things. So usage, data usage on the customer side is important. So basically what we're talking is when you do profiling, you need to do profiling on the user side of things, right? And you need to do profiling on the device side of things. What does this mean? We'll dive into a little bit more detail. So on the user side of things, we wanna basically look at height specific data. What is height specific data? We wanna see how many chat threads do you have open? Like how many active chats do you have open, right? If someone has five active chats open and an app launch would not take as much time, if you had 500 chat threads open, it might take longer time. How many stickers do you have downloaded in your phone? That would have an impact. How many groups you have created and so forth. So there'll be a whole bunch of user specific data that'll impact and there will be device specific data that will impact. This is things like how many contacts you have in your address book. How many photos you have on your gallery, things like that. So there are all of this, which is user specific data. And then we look at devices, right? So like you pointed out, like Pooja pointed out, what are the top devices being used by height users? But that's not sufficient because we might be missing a completely big market that we are not into. And one of the reasons could be because your performance is really bad and people are not using your app on those devices. So you also want to look in the market, what are the top apps right now? Sorry, what are the top devices right now? So you take all of this and you mishmash and you create basically a kind of metric that we came up with which is basically we segmented our users into different buckets. We call them 50, 80, 85, and 99th percentile. And then we look at some of these things in terms of the number of messages, group chats that are active, one-on-one chats, sticker packs, status updates, address books, so forth, right, like the long list of things. These numbers are obviously cooked up numbers. We can't reveal the actual numbers, but this should give you an idea of how we try to segment our users into different buckets. Once we've segmented our users into different buckets, now the next thing is to actually look at, this is the user-specific data. The second thing is to look at the device-specific data. So in India, what are the top phones on which people are generally using messaging apps? And inside that, which are the top phones on which or devices on which Hike is being used? So we looked at both of this information. Obviously, this is, again, for various reasons, blurred out. But again, the point is that you do this kind of an analysis, you pick up this list, and then you need to come up with how, now you're gonna benchmark based on this. So I think I've spoken enough, but given you a broader framework of what went through, now we'll jump into a demo and actually look at step-by-step, what do we analyze, yeah? So KP, over to you. So before I jump into Hike or Market, there are companies that do that for you. Nareesh talked about profiling of users and profiling of devices. So now regarding all these parameters, we have our own benchmark application simulating the real market scenario. We have created the environment, we have devices which have pre-loaded data and the top-most devices that we'll be using. But that's not enough, right? I mean, the major question that comes here is, what do you benchmark your application on? What are the actions, what are the activities for which you want to check whether your app is performing well or not? Can I get some ideas? What would you like to check in your application? If it is working, I'm good to shift. Or at least this is the basic functionality which should be working. That's a very good point. App launch, anything else? I'm sorry? Yes? That is also a very good point. Of course, if you're making constant calls in the app, some data is getting loaded dynamically. Yes, so that's like putting your app in. App loading and then specific screen loading times you want to measure. Putting your app in the back. So rendering of data once you bring your app back to foreground, all right? Background, foreground, activities. Exactly. So for us, it's a messaging application. You'd say the chart trade messaging, right? That is one parameter that I would like to shift to performance side, because user does not actually perceive that. But still, we will be covering that on a later stage. In our case, we work both offline and online. So it's not important for us. Definitely, if you have some kind of, yes, sure. All right, so we've got a bunch of ideas. And I think that's a very rock solid idea. So let's jump into the demo. Let me show you the application that we have developed. It's hosted on our local machine, of course. So this is the UI that looks like basically these are the actions currently that we are benchmarking our application on. Of course, like we discussed, this is not enough. There's a long way to go from here. Open this dropdown. So like we talked about app launch time, right? So there are two major aspects, force kill and force stop. So we are measuring app launch time after both of these. Then the next thing that we talked about, the core of the application itself for us, it's the chat thread opening. That comes next. Then there was seamless dynamic data loading. So that is chat thread logging, chat thread scrolling. You don't want scrolling to lag or any kind of animations to lag. Then we talked about external library. So when you start, try to start a new chat. All the contacts in your devices are imported and listed in the application itself. So that is one screen which should open seamlessly and instantly. So let me just select any particular action.  We do it periodically. So I have a list of APKs. Let me just... One thing I forgot to tell initially is a lot of what we are talking is Android specific. We're not going to discuss iOS specific stuff here, but this should give you a pretty good idea of how we are doing on Android. So a lot of what we are going to talk is Android specific. So of course, when we run this application, the benchmarking sooth, we get all the logs. Now, as a customer insights team or as an analytics member, it's not very user friendly going through the logs, right? So they need to have it in a very formatted manner. It makes their life easier. We dump all the data to a graph. It's just a basic comparison. So as you can see here, there are two points that I like to mention. So first is usually you have a set of actions. You would like to perform it, take the reading, parse it and get the data, but that's not enough. Sometimes now we are simulating the real users device. Sometimes there might be multiple applications in the background. And because of that, your phone might be lagging and the effect of that impact of that, you might see on your foreground application, which is hike messenger. So one iteration of the reading is not enough. You might end up with outliers sometimes, which might cause a panic. So it's always better to take five to 10 iterations, take an average of those. Despite of that, you might still end up with outliers, multiple outliers at the same time. So the only way to overcome that is run the suit again, get a good data. So here you can see three particular application versions are there. There's a jump and so these jumps, I selected this specifically because this data is with the outlier. Actually we were having a uniform data and then this is where we caught that, okay, we should have multiple iterations because we are having one outlier, very high, which is screwing up the overall average. So this is pretty much the application. Any questions around this? Sure. So for benchmarking, we are considering the wifi speed itself. Right now I'm only concerned with the activities loading. I'm not concerned whether they're interacting properly with the network or not. That test gets pushed down a little bit lower. We'll come to that part as well. When we start playing around with networking, network and other aspects of performance testing. So here I'm more concerned with my application as a normal application is working fine or not without considering other environments. So generating the load, we talked about the user profiling and device profiling, right? So the numbers that we showed, let's say we talked about 99, 85, 80, 50. So let's take any one bracket, 80 percent right. I have the numbers. We create backup files of those and then when we sign up, we have the functionality of restoring from backup. If you reset your application and then when you try to log in again, sign in again, we restore the backup for you so that your chat messages and all your data in the application basically does not go away. So we do that. We perform a fresh sign up with the backup file restored. So I have the user data and the contacts are there and the device data as we discussed, that is simulated. So I have the environment setup and then I run my test suite on that. So for example, it starts at the opening. I need to take 10 iterations. My UI test will perform 10 of those actions because that's how the user works and in the backend, my thread will be working which will keep thinking for the log lines. It'll collect those log lines. Then there's my parser which will pass through the log lines, get the values, push them to DVR, wherever you want to store them and then send it to the UI for the formatted manner. It doesn't- Now just the visualization part. Everything else we're gonna show how you can do through. Now this is just rendering the data that's already collected. Yeah. This is a CI system which actually kicks off these. This is about rendering those and visualizing those. Here you're not kicking off the build. This is just for a visualization. There's other ways, other part from where it actually gets kicked off. Any other questions? All right. This was the demo that we actually went through. All right. So we talked about benchmarking. Now, but as you saw, a lot of questions came around network as well. This was completely from apps point of view. There are other things in the device itself, especially Android being an OS which facilitates you to play around. There are a lot of things, a lot of parameters that you want to take care of. Right? Example, network. One of those. Any other examples that you can think of that we should consider should be optimized memory, CPU usage, memory usage. Right. What we like to call it hike is the four pillars of benchmarking. Memory, battery, CPU and network. If I do not look at these four aspects, I might end up somewhere where my app is actually working fine in the UI, in the benchmarking side, but in the performance side, it's not that optimized. Let me share an incident that happened with us. Quite a few releases back. We saw that our app was getting killed in the background. It was an internal release, of course. So we were still in the development phase. Now we didn't know what was causing the app getting killed in the background. So we dig deeper. We found that we had integrated a new feature then, regional keyboards. We were supporting, we were starting to support 10 regional languages. Now when we go in the chat thread and you open the keyboard, and if you have already selected that, the custom keyboard comes. So we found that whenever that was getting loaded, particularly the app was getting killed. Not a good sign. We cannot ship at that moment. So we decided to dig deeper. We found that the external library that we were using for our custom keyboard was internally using a native Android library, which was shooting up the SO memory map values. I'll talk about all these terms later. So this was one particular aspect of memory which was shooting up, and that was causing my app to be killed in the background. So why does this happen? Android has made it very clear that if the app exceeds a particular threshold value of memory consumption, it will kill the application. So they do not assure you that if you remain below the threshold value, they will not kill the app. They might still kill the app, but at least you have to be optimized in that way, because if you cross the threshold value, your app is definitely getting killed. So we dig deeper into that, and that is where we realize that memory is probably the most important aspect among these four. So let me get a little bit deeper into memory. So on a layman term, if we talk about very basic, so there are two major parts of memory which Android plays around with. There's a private memory that is allocated to your application which is private, dirty, or USS, that is your unique set size. That part of memory will only be shared by your application and it's released only if you force kill or force stop your application. It might be released if you push it to the background also depends on how your application is making use of the memory. The next part that comes is shared dirty or PSS. This is proportional set size. Now this is a chunk of memory which lies there and it's shared by a lot of processes. One of those processes is your application and let's say a couple of them are using it. Now if we add this private dirty and shared dirty and say this is the memory that is being consumed by your application, that's not right because a lot of processes are sharing this PSS value. So what we do is what Android does is they divide this PSS equally among the number of processes that are sharing this, then that chunk and the private dirty that is allocated to you, that gives you the total memory. Apart from that, Android does not really, it does not have at all the concept of swapping memory, plays around with paging and memory mapping. Memory mapping in turn itself has a lot of different, different branches. So I'll just name them a few. There's SO, JAR, DEX, there's ART, OART. And there's a lot of aspects. I mean, if you start looking at each one of those, it's a day long conversation. But yeah, most important ones, like I talked about the SO native map. So what does it contain as a native code, like native elements, like your code itself, then JAR is any external libraries that you're using. DEX is basically your Dalvik executable. So these things. Now you as a developer cannot control each and every aspect. You can bring them down as a whole. What you can control really is your private dirty and your heap memory. So how do you calculate these values? I mean, a lot of companies are out there who are providing benchmarking and performance. I'm pretty sure the basic, the base which is being formed for all these readings is ADB. So Android Debug Bridge, they have provided us with a set of operations which you can perform and get the readings directly from the kernel level. So we have one for battery, Thumbsys battery. We have CPU info. We have mem info along with the package name of your application. And then there is this particular file, proc.net, xt, qta, guid stats, which digs into the networking files and tells you how many packets you received and sent. And it gives you a very, very clear bifurcation. Let me just quickly shift to the terminal and show you what kind of readings these give. So that will probably give you a better picture. Okay, we could quick time check for three minutes. All right, so probably we'll not dig into the... You can show one or a couple of them. Sure, sure. All right. You got a phone connected right now. So it's a physical device. So one quick point that I'd like to add here is, if the phone is connected, the question asked, raised is, how do we do battery testing? There are two ways of going about it. You can directly go into the kernel files and fetch the values from there, how much current is being flowed and... But I'll not go into the details and it's very complicated to do that. The other way is, you can connect your device wirelessly through ADB and then trigger your test suite on that. So for battery, the command is ADB shell, it dumps its battery itself, then two devices connected. All right, so as you can see here, I have only one device connected. It's showing two. The other one is actual device, which is being showed when you actually connect your device, right, with the device ID and everything. However, to run the test cases over battery, wirelessly for battery, actually, we've connected it through the TCP IP protocol with the IP address of the device. So what happens is, I disconnect the device, I run the ADB devices command again, and that's the only device that's connected. I have the device connected without the cable. Now if I run the battery command, all right, this is why we pray to the demo gods. All right, I'll do one thing. Since we have a time crunch, I'll just directly connect the device. So this is something that you get. What we are concerned with is the level. Now, when you scroll down and you look at the battery value, this is exactly what you see as a user. So this is all that I'm concerned with. Then quickly jumping to CPU info. So you see a list of services, all the services that are using CPU right now, right now, no operation is being performed in Ike. So the package is not there. Then quickly jumping into mem info, because that is a very important aspect, providing the package name. So a lot of data is thrown when you do that. So most important of them that we talked about, the native heap, Dalvik heap, and the PSS total and the private dirty, you can see there. That is what we should be concerned about. And when you come down, there is SO map, JAR, APK, TTF, DEX, there's a lot of values. You need to delve deeper into the concept of each one of those and see how they impact your application as a whole. All right, so these are the basically commands and how we are running our performance route. So optimization, when we started with our performance route, now there are four aspects that we have to check across so many templates. So we ended up with a time of somewhere around 13 hours. When this was massive, I have to run it on an urgent basis, it gets a problem. So what we were doing then was first run the CPU, then battery, then memory, and then finally network, get all the data and then send out a report.  We observed taking five, 10, or 20 iterations, I'm almost getting a level value. Why not reduce my number of iterations? We came down to somewhere around 10 hours from there, still a big number. So we thought, all right, I can do one thing. I have to connect wire for CPU memory, a CPU memory and network, wirelessly for battery. Why not run these three parallely? I have the same set of templates from the analytics team, like we talked about the user profiling. I know on a daily basis, at least a user sends out 50 text messages. So that's a test case for me. But I have to check CPU, battery, memory, and network for all those. Let's perform it once, get the values for all three at once. That brings us down to somewhere around seven hours. From there, we said, why not run everything wirelessly? Why do connect wire at all? Run all four at one go, the templates remain same. They remain constant. Run everything at once, bringing us down to somewhere around 3.5 to four hours. That was a big dip that we have seen. And this is actually a lot more realistic because people won't connect their phones and do messaging. They typically disconnect. So this is a more realistic test. So while it improved the performance of the test suite itself, it also gave us a lot more realistic read of how the users would actually see the performance. So of course, this is not enough. There's a lot, a lot more to do. So talking about future enhancements. We could quickly pause before we jump into that. Any questions so far? I think we rushed through a bunch of things. Yeah. So I did not show you the application actually. What we do is we have been running it on a bunch of applications, versions, right? APK versions. So let's say I'm on version one today. I ran it. I see I'm pretty good to go. I've optimized. It's using very less CPU, very less memory. That becomes my benchmark value, my threshold value. So I'm going to go back to the baseline. I build version two tomorrow. I get all the data and see what's the jump. If there is a dip, good enough. I should do that in every activity of my application. There is a jump. Why is it? Then we try to delve deeper into the code. What exact activity caused it? Because we are benchmarking activity-wise and we are performing template-wise. So one template we talked about, 50 text messages being sent out. So I know that is causing a shooter. We can always run trace views on that through DDMS. Get the exact spike and then bifurcate in our functional basis everything from the code itself. So that of course requires a bit of debugging because of course you need to verify where exactly is your memory going high. And finally we run these commands to get the values and see what is shooting up. So to your question, there's no golden standard that you should meet this number. It's basically a relative scale and which is where benchmarking becomes important. So you keep benchmarking and any time there's a shoot up, you know something's going wrong, right? It should not exceed certain thresholds. So there are certain thresholds that are set. It should not exceed certain thresholds. If the thresholds are exceeded, then it basically fails the test. Someone had a question over there, yeah. You made certain assumptions. You're saying that if I have 100 million user base, I need to test against X number of devices. If we don't follow the industry, whatever you said. I thought this is the first time I've actually heard something like that. It didn't really affect us. I mean, if tomorrow we had 500 million users does not mean I will increase. That's, so what we are talking is mostly client side performance, right? There's obviously server side performance that needs to be done. And server side is where the number of users will impact you, right? On the client side, number of users I have 100 million or 500 million users. Actually on the client side, it doesn't impact you. Provided your server side, your handling concurrency and all of that stuff on the server side. So decouple the two, right? On the client side, you're mostly focused on, there are, again, we did profiling, we figured out what is the average number of chats you would have, right, typical user. Bucket them into different percentile because we want to make sure across the different percentiles people have seamless experience. So we might optimize only for the highest performance bucket and then screw the others, then that's not the right thing to do. You want to make sure across all the buckets there is a reasonable performance. Of course, if someone's sending like 500 million messages every single day to lots of people, their performance will be slightly more than someone else who's sending in the 80th percentile. So those are the things that we are measuring on the client side. It doesn't really matter, in my opinion, how many overall user base you have. Talking about the number of devices that we are actually using, at this point, we are essentially testing it on three devices. This is something we'll talk about in future enhancements, is we want to increase that number of devices to more. But for now, we are actually benchmarking on three devices, which is our top three used devices. Okay. Did I answer your question? We are lucky we are only in India. We talked, right? We picked the top three devices that are used on Hike. You need to start somewhere, right? Again, there's a point we'll talk in the future enhancements where we want to improve this thing. But what we are seeing is in the top three, we are able to catch, we are able to get 80 percent feedback, right? So 80 percent feedback is good enough. There is obviously room to improve on getting the rest 20 percent feedback. But again, starting somewhere, I think even like we started with one device. Now, we've gone to three devices. Gradually, we'll grow to more devices. We did a talk yesterday where we talked about APM and Dexter and stuff like that we're using. Yeah. Yeah. It's APM. Yeah.  That's what I think KP was explaining about using. Go ahead. So you know that this particular activity is creating a higher value in your memory consumption. You go to your code, you start a trace view on those activity in it methods, and then you go function by function. So it gives a very detailed analysis of each function, how much memory they are consuming, how much time are they taking to initiate? So you can benchmark on those values and then you know, take an analysis, what exact function is there, which is causing. In fact, initially we talked about one example where we saw a shoot up of 130 milliseconds, right? There where we actually dug in. We realized like deep down in the chain, call chain somewhere a developer had added a new call which accidentally goes and does an IO operation. And when you try and do an IO operation on a UI thread, it's obviously going to introduce some kind of a lag. Now that lag might vary significantly depending on the device type, right?  Which is where the benchmarking is important. So we have built our own tool. There's a time crunch, probably once we finish, we can sync up offline. But essentially building on top of what is out there, we've not like built, we just built our own stuff. You will keep hearing this not invented here syndrome, very big in hike. We basically try to do a lot of stuff on our own. Not a great thing but also on another side it's a nice thing, you know, from actually kind of pushing the boundaries. Okay, then let's go through the future enhancements, hold on to the question, we are running short of time. So let's quickly jump into future enhancements. So the first one is we talked about right now we have CI as part of our, we basically trigger this weekly right now. And we want to actually make this part of our, with every check-in we want to basically trigger this. So that's one enhancement that we are basically working on right now, where it'll become part of this thing. We talked about device coverage improvement. Right now we are basically on three devices. We want to improve the device coverage. Yes, in yesterday's talk we talked about how we are set up Dexter which can support up to 80 devices. So the idea is to essentially have, you know, this suit run on those 80 devices that we have set up. So that's a future direction that we want to do. As of now we only do APK size benchmarking on the, you know, like just before making a release and it's also kind of manual check someone runs an APK size check and verifies whether, you know, but we want to actually make this also part of, so when we talked about the four pillars, we are planning to add APK size as part of that thing that we'll get regularly checked because we are also very sensitive of not increasing the APK size given our users are very data sensitive. So we want to make sure that that becomes a part of our whole benchmark. Along with these, we run through four commands. That's not enough. Android is providing you to, they give you the freedom to read into, dig into the kernel files themselves. So there's another command proc stats along with Meminfo which gives a different kind of data. As you compare those, you might find a difference of data for the same aspect. Now we're not very sure what the difference is, but I mean, you can look into any of those, you know, you have to take it with a grain of salt. So probably these are the future enhancements. And so yeah, to finally summarize what we talked about, we talked about analytics driven benchmarking, simulating the real market scenario in our own test lab for performance and benchmarking. Then we talked about the four major pillars, CPU, memory, battery and network. And we focus mostly on memory because probably that is the most important aspect. Then we, yeah, this is deep analysis into memory. And then we talked about the parallelization of test suits going completely wireless, running everything and getting the real data as the user would actually end up using your application. So that's kind of, yeah, a quick summary. So we have three more minutes for questions. Yeah, awesome. Not true in our case. So I'll just repeat his question for the benefit of everyone. So he's asking, what about server side performance? You know, can you throw some more light on for server side performance? Because at the end of the day, if you only measured client side, it would massively get impacted by the server side performance. In our case, that's not 100% true because we also have peer to peer messaging, offline messaging which doesn't hit our server at all. So for a lot of our use cases, like I can transfer a one GB file to you, our server will never even come to know about it. That's just gonna be directly through a Wi-Fi direct to you. So we are doing, on the server side, obviously there's a really impressive team at Hike which does a lot of performance testing on the server side. Here we are mostly focusing on the client side of things and how we do performance testing on the client side of things. Because for us, without the server also, there are a lot of use cases that exist. So it's not true in our case. Maybe in other cases, it might be true where server is an integral part of every use case. In our case, that's not true. Okay, one last question, let's talk offline. All right, yeah, so actually that's a good point. We should, so just repeating what he said, like on the performance side of things, you know, 2G network, 3G network, Wi-Fi network, 4G network now, there are a lot of different kinds of network and the performance can, you know, get impacted by different kinds of network. So how are you doing performance testing on it? KP talked about mostly right now, we are doing all our testing on Wi-Fi. So that's again, I would say, is an area of improvement that we need to add. We do do some, go ahead, sorry. So adding to that, we are actually testing on these three networks as well, 2G, 3G, and 4G as well. So normally if you see, if you're sending any data over the server somewhere, the amount and the chunks of data remain same despite of the network. But we have optimized our file transfer and message transfer and everything. Such an extent that you change the network and our chunks that are going, go in that manner. I will not delve into the details of the same, but talking on a very abstract manner, we have, we always have log lines, we switch the network, we run the test case and we see that whatever chunks they are supposed to go in with respect to the network that we have selected, whether they are going or not. If I hit that particular threshold value, it assures that my time limitation, whatever I have a threshold for that, I'm meeting that. If that chunks are not met, it means I'm increasing that and I need to dig deeper into my code and see where exactly is it failing. So why I said this is an area of future improvement is because there's a lot of interesting work that is done on the way we work with different network types. And those kind of tests actually right now, we're trying to push at a lower layers of our tests rather than keeping it at this level. So right now we have few tests that we run across multiple networks, but ideally we want to push those at lower layers because that's where a lot of ALGO and other interesting things are built to basically determine how you do network-specific use cases, which right now, we are actually not fully automated. So that's an area of improvement in my opinion. All right, thank you everyone. I hope you had something interesting to take away from this session. Thanks guys.


It's The Cube. Here is your host, Jeff Crick. Hi, Jeff Crick here with The Cube. We are in downtown San Francisco, very special segment of our On the Ground series, really highlighting our Women in Tech feature that we've had as an ongoing segment. And we're really honored to be joined here by Michelle Lee, who is the Under Secretary of Commerce for Intellectual Property at the U.S. Patent and Trademark Office. It's a mouthful. Congratulations. Welcome. Thank you. Michelle Lee recently put in the job by Obama, and you did your acceptance at South by Southwest. That's pretty innovative. Yes. I had the pleasure of being sworn in on South by Southwest on the plenary stage of the festival, and it was in a group of technology innovators and entrepreneurs, and most people in that community never see a Washington, D.C. style swearing in. So we decided, you know what? These are our customers. These are who are here to help support and promote and support their efforts. So we decided to have the swearing in on the stage of South by Southwest. They were very excited, and it was a special privilege and honor to be sworn in as the first woman head of the United States Patent and Trademark Office in our country's 225 plus year history. Yeah, that's amazing. Now, we're going to get you into the secretary job next, so I don't know if Obama will still be around when you want. If I do that, I'll be next in November next year. You've got the candidate. So that's terrific. It's a big job. I didn't realize, $3 billion budget, almost 13,000 people. So you're busy. It's a big group of people to keep moving. A little busy, yes. I would say so. So let's, before we get into that, let's get into your history a little bit because I'm always curious to know, kind of, what was your journey? How did you get here? You got a really interesting story. You grew up in the Bay Area, got phenomenal education at MIT with a bachelor's in electrical engineering, a master's in electrical engineering, a master's in computer science, and then you took a little bit of a turn and went and got your law degree. So I wonder if you could tell us a little bit about your kind of academic journey, those choices, and how you ended up going from hardcore tech at MIT to a Stanford law degree. Yeah, no, I'd be glad to. And I was, as you said, born and raised in the Silicon Valley. And as a young girl, I loved math and science. And I can remember building a handheld radio with my dad in our living room. And I was always passionate about science and invention and creation and making. And all the dads on the street that I grew up on were engineers. Many of them started companies based upon a clever invention they patented that they then obtained venture capital funding for to build a company that would bring their invention to the marketplace. So that was a lesson that I had very early on growing up in the Silicon Valley. And it's what inspired me to study electrical engineering and computer science at MIT. I was one of a small number of women at MIT studying electrical engineering and computer science at the time. And the numbers got even smaller when I went into grad school. But I had a wonderful technical education. I thoroughly enjoyed it. And I was about to get my PhD in computer science. You were going to go PhD? I was. That was the route that I was on. And my thesis advisor was terribly disappointed when I did not go on with my PhD in computer science. Now that usually goes to whole different tracks. I remember when I got my bachelor, I talked to the professors about PhD. And they're like, no, no, no, not unless you really want to be a professor. You want to make the investment on a PhD track. It's a long and hard thing. So you not only made it. So was that really kind of for academia or you just love the topic so much? I grew up in the Silicon Valley where science and being an engineer was the most honorable thing you could do. And so that's what I grew up with. And that's the path that I was on. But then when I was at MIT getting my master's degree, I realized that there are a lot of things you can do with a technical degree. And at the time, there was the Apple versus Microsoft case going on in the courtrooms and it had to do with how do you protect the graphical units or interface of the icons, the scroll bar, the trash can. I thought, wow, that is interesting. We're applying really old case law to really cutting edge technology. And I thought, you know what? That's what I want to do. That's how I'm going to make my contribution. And that's when you made your swap. So I decided, I worked at HP Labs Research Laboratories on a computer programmer and I worked at the MIT Artificial Intelligence Lab. And I was having a wonderful time and then I was working on my graduate degree. And I dawned upon me that there are lots of things you can do with a really good technical background. And that's what I wanted to do. I wanted to work on cutting edge issues in law. I wanted to help innovative companies protect their inventions and help them commercialize it in the marketplace. So I went to Stanford Law School, graduated there. And ended up working in the Silicon Valley for about two decades. Representing small startup companies, big fortune tech 100 companies. On a whole range of intellectual property and other legal issues helping them grow and expand their business. And around 2013, or 2003, excuse me, Google asked if I would join their patent team and head up their patent efforts. Right, so just to back up a little bit. So it's high tech issues, but it's kind of ironic because it was mice, trash cans, and desktops, right? That originally got me to move off of you. They were talking about this. But then you went into the legal world. You went into Fenwick and West, obviously a really top end firm. Deals with a lot of startups, a lot of big companies.  So then, again, why the shift? Why did you decide to go from a place where you get to touch a lot of different cases into Google? Where now you're just working for one very specific company. What is a big change? I mean, the change from being a legal advisor to a whole range of clients, different technology areas, different industries to a client. But, and I thoroughly enjoyed my time at Fenwick. Great clients, great startups, all different varieties. But at the time when Google asked me to join, it was at the time where I had used the search engine. And I thought, you know what, the search engine's pretty good. And I wasn't really looking to change. I wasn't looking to go in-house. I was very happy with my practice and so forth. But they said, why don't you come on over, meet the folks. And you had a really bright, impressive group of individuals. And I've had the privilege of working with many bright individuals. And there was something exciting about the individuals at the company and their mission. And the search engine, of course, was better than any other search engine. And there were a lot of search engines. I don't know if you remember. Well, you're smarter than me. I thought the search engine wars were over. I was like, who's going to give Google any money? Web crawler is dead. Altavista is dead. Excited home is dead. And even if they come up with a cool search engine, you're only one click away from going to another one as soon as the next best algorithm comes in. You're smart. All those competitive search engines were still on the market competing for. It was unclear which one would prevail. And I thought, you know what, I like the company. I like the product. I like the team. I'll join it for a little while, see how it goes. And one year led to two years led to eight going on nine years. And I have to say, it was an incredible journey. It was one job, but it was really like many, many different jobs because of the changes that the company underwent. So where was Google size-wise from the time you joined to the time you left approximately? It was a relatively young company. I can't remember the precise number of employees, but for the most part, it was US ReserviNU only. We had a small number of products and services, search ads, and a couple of others. By the time I left, we were in every country of the globe. We were in a wide range of areas, including mobile phones, driverless cars, Google Glass. I mean, just all these areas that you never even imagined the company would be in. And when I joined the company, we had a small handful of issue patents. And by the time I left, the team and I helped build that patent portfolio to over 10,500 patent assets in eight short years. And it was an exciting journey. Wow, busy time. So then did you get the call or did you get the itch when you decided to make the move to the public sector? Well, I think what happens is, given one's career, if you've had a certain number of experiences in the past and I have, I've had a very rich set of experiences. I represent all kinds of startup companies. I've written patents. I've asserted patents in patent infringement cases. I've defended against patent infringement cases on behalf of clients. I've bought, sold, and licensed patents. Sometimes for very large sums of money. And you develop this experience and perspective from being on the tech side, on the innovation side, to helping startup companies navigate these issues from small to very large. And you get to the point where you have an ability and a perspective to contribute something of value that your country needs. And when you're asked to serve in those roles, I think it's a privilege to consider it. And if you're going to all swing it, I think I firmly believe in each of us giving back to the country, to our country, which is a wonderful country. It's given my family and myself tremendous opportunities and really giving back in ways that you can. And so it was the desire to both give back and to have an impact on some issues that are very important to this country at this time, to the president at this time. And really, to American businesses, there are a lot of changes that are going on that, you know, growing up in the Silicon Valley, we had the incentives that nurture and promote the innovation that we see. And you can't take it for granted that that will continue to occur in the future. You have to make sure that you actively manage it and that our policies are right. Right, right. And so the ability to contribute to that, to help play a part in ensuring that we have an innovative society where intellectual property continues to incentivize innovation for future generations is pretty important work. Yeah, yeah, although I'm sure the taxicab unions might have issue with the latest innovation that we're seeing with our friends at Uber, that's a whole nother topic that we won't get into that. But so let's jump into your current role, your current job. So first off, kind of, you're out here, what are you doing in the Bay Area? I know you got some, their satellite office now, you got a directive to really get outside of Washington. I know that's where you got your start, really. So talk a little bit about kind of what's going on with the USBTO and what's going on with the satellite offices and kind of up to date on what you're up to. So the trip here to California is driven in part to, again, reach out to our stakeholders. So I met with the UCLA Technology Transfer Office and students and entrepreneurs in the Los Angeles area. Earlier today, I met UC Berkeley campus with Skydeck and Accelerator and Incubator. And really, and the USBTO offers so much to support startup and entrepreneurship activity. And we've got the opening, as you had mentioned, of our regional office that will be located in the Silicon Valley in downtown San Jose, opening on October 15th of this year. We're very excited about it. And as far as I'm concerned, there is nothing but upside to these satellite offices, these regional offices, because the USBTO is there and has so many resources to offer inventors and entrepreneurs how to file for a patent. What are the considerations that you need to think about in terms of various forms of intellectual property? What do I protect with a patent, a trademark, a copyright, a trade secret? What are those things? How can I get access to an attorney if I don't quite have the full legal resources? How can I accelerate the filing of my patent application if I want to get venture capital funding? And so a whole range of services. It's really the USBTO bringing its full range of services out to the local innovation community. I mean, businesses knew this long ago. If you've got clients in a region, you're out in the region. You've got to be with the customer. So how many people will be at the offices in Silicon Valley and where are the other regional offices? So the regional offices are in Dallas, Detroit, Denver, and the Silicon Valley. And let me tell you, there were a lot of cities all over the country who wanted to have a United States Patent and Trademark Office in their home city. How'd you pick Detroit? That surprises me. Well, I mean, there's a lot of innovation, oftentimes a mechanical innovation that occurs. It's a city that is rich with history in terms of inventive activity. I mean, the automobile industry. But also, there's a resurgence now of young companies in that area. Someone out of Ann Arbor, or a second of the driver? Correct. So the university's there. In fact, I was there at Tech Week earlier this year. And there's a lot of activity there. And we're working with the incubators there. And we're looking to support their activities. So we're very excited about it. But we have offices in every time zone in the continental United States. So they were equally distributed throughout the country. And the idea is, well, we can't be in every city across the country. We're trying to spread ourselves out. And each satellite office serves a region, the Silicon Valley office, the West Coast region, the Detroit office, the Rust Belt region, and the Dallas office, the Southwestern region. And of course, we've got our home office in Alexandria, Virginia, which serves the East Coast. So we're very excited about that. And yeah, that's opening October 15. Well, good for Detroit. I hope they get it going out there. It's fascinating. I mean, clearly it's a huge history of innovation. And the universities are such big drivers of this kind of activity. And to have University of Michigan right there, it's actually a pretty popular school for kids out here to help move that forward. It's terrific. So what are some of your kind of current priorities in your role given the current environment that we're sitting in? Yeah. So there are a lot of priorities that we're working on. Give me your top two or three. It's like picking your favorite child. You can't do that.  So I mentioned the opening of the satellite offices. That's clearly a priority. I think that's all good for our country. And I think, well, another priority is ensuring that we achieve balanced and meaningful legislative pattern reform. There's been a lot of discussion lately amongst companies in the press, in our business communities about the cost of abusive patent litigation. Some people point to what's called Troll patent litigation. I don't like to use the word because I have yet to come across a definition of a Troll that everybody can agree upon. But nonetheless, there are costs to society for entities that engage in abusive litigation tactics. And the legislation pending before the House and the Senate now are designed to address and curtail some of those abuses so that we can curtail the abuses but also enable those who need to enforce their patent rights to enforce it. And it's important to get that balance. So that's what the legislation is dealing with in terms of in Congress. And that's a priority for the administration and also just making sure we issue the very best quality patents in a timely manner. I mean, our businesses rely upon it. They need it to build their companies. They need it to obtain the venture capital or financings that they need to develop the technology. And they need to be able to recoup their investment after they've worked so hard to bring it to the marketplace. Talk about the challenge, though, of just the speed of innovation. And you're at Google, which is a really valuable experience to understand how fast things move. And as you said, you guys did 10,000 patents in eight years or whatever the number was, and don't misquote you. For older institutions, the government specifically, it doesn't move by the same speed as Silicon Valley does, which moves at a different speed than the rest of the country. Now, we're even in a really crazy spot with the cloud and mobile and social and enabling people to innovate even faster. So I would imagine that the legal precedents kind of lag as they usually do, because usually it's case law that kind of sets new trends. How are you trying to keep up with that? Are you able to? Are you changing priorities, procedures? It's got to be a really challenging time. And then, of course, you always get into who's got the patent. And then it's actually funny, the garbage can, the desktop, and the mouse. Because at what point, how far out does something like that go? You look at like Facebook started as a dating app for a couple of lonely guys at Harvard. And now it becomes a single sign-in vehicle for a ton of other applications. So I would imagine trying to keep in front of that, as well as kind of the defining law that dictates where things go. It's got to be getting more and more challenging. So let me say it's a great problem to have. We are lucky to be living in a country that is as innovative as it is, and that is moving as quickly as it is in creating new inventions and ideas. I mean, so many countries across the globe want to have a system just like ours, where we have this problem to deal with. I love the fact that we get all different kinds of technologies happening, evolving so quickly, and that our examiners have to deal with it. That is a good problem to have. So to answer your question, how does the agency adjust to that, adapt to it, stay on top of it, our examiners are really good. But we don't do it single-handedly. We partner with the public. So one of the reasons why I'm out here is to say, hey, look, we're out here. These are all the great things that we're doing to support the startup and entrepreneurship communities. But when I was at UC Berkeley and UCLA, I said, look, you guys have some of the world renowned experts in various technology areas. We have to examine patent applications in these areas. And the technology is changing constantly rapidly. Come to the PTO. Help us train our examiners. They're super bright. They're super hardworking. But you are on the cutting edge. Come and explain it to us. As a result, you'll get better quality patents issued faster with better customer service. Everyone's excited. Our examiners love it. And the experts in the field love coming to the PTO or coming to our local satellite office, being able to give an hour presentation on their area of expertise, it makes complete sense. So that's one way of staying on top of the constantly changing technology. But it's a great problem to have. I mean, I'm glad we have this problem instead of not enough innovation, not enough change, same old technology. I think our economy will be worse off if we're in that situation. Yeah, no, and we're jamming. And it's just interesting, too, because not only do you have all these enablers that push the pace of innovation and the breadth of innovation down and wider within a company, but, too, now we've got this API economy where a lot of these things in Uber is just a great example, where it's basically, there's an integrating a map and a payment system and a GPS for the guy to drive around. I mean, it's more complicated than that. But really, it's this aggregation of existing technologies in new and innovative ways. And I imagine that the opportunities and challenges are pretty significant. And we're also taking advantage. I mean, given my Silicon Valley background, of course, I'm a huge fan of big data. And I believe that big data can help businesses make better informed business decisions. And so we are bringing some of the big data techniques to the USPTO. We measure a lot of things in our process. So we're analyzing that data, seeing where we can refine, seeing where we can improve the training to our examiners based upon measurements of results. And we'll fold that back into training so it's a feedback loop. And we're also looking to share a lot of our data with the public and make it accessible. So we have a lot of data about who's inventing what, in what region of the country, and how active. Because if you think about if somebody invent something, one of the first things they're going to do is file a patent application. And you're going to know, after a certain period of time, what area they're doing the work in. So it's a very good leading indicator of where the innovation is occurring in our country and to what level of activity. So we're looking to share publicly available and publicly releaseable data to the public so that the businesses can make better informed business decision about the competitive landscape. Do you go out to shows? Do you go out to Spark Summit and Hadoop Summit and all those things? I went to South by Southwest.  I'll give you a tip. Spark's the next crazy big thing coming. It was a tech week. There you go. OK. So now let's shift gears one more time. We've been going. We could go for a long, long time. And let's talk about why we're here, which is really increasing the amount of women that stay in technology. I think there's a big issue where women drop out. Both we read about as grade school girls like 4th or 5th grade, there seems to be some evidence. And then also, there's evidence that women drop out of tech careers as well relatively early in their career.  You've really bridged technology as well as law, as well as public service, which is phenomenal. You've got a daughter, which you've talked about in other interviews, which is terrific. So it's personal. I do. What are just kind of your priorities around this area and really helping women specifically get into tech, stay into tech, be successful in tech? Well, thanks for the question. I very much appreciate it. And it's an issue that I care about personally, but it's also within the mission of the United States Patent and Trademark Office. Because if you step back, the mission of the US Patent and Trademark Office at the highest level is to promote American innovation. That's across the country geographically and across all demographics. So men, women, boys, girls, every ethnicity, and so forth. So that is the goal of the USPTO is to promote American innovation. And if you think about it, there's no age requirement on a restriction on filing a patent application. You have an age restriction for when you get a driver's license when you can vote. But there's no age restriction. Others done it all. There's none whatsoever. And in fact, we have some minors who apply for patent applications. I would love to see more American children, boys and girls across the country being inventive, creating, making, and filing for a patent application. So on the issue of girls and women in tech and STEM, you're right. I mean, they are lagging in numbers. They tend not to enter into the STEM field. They drop out at higher rates. In fact, the Harvard Business Review came out with a recent study that said, well, first of all, Girl Scouts of America came out with a study that said 57% of all girls surveyed said that they are not considering a career in STEM. So 50% of girls surveyed are not considering a career in STEM. Girls got age girls. So that would be. I don't know what the ages are, but there were girls surveyed. Where were they surveyed, OK? And then what happens is when they do enter the STEM field, according to the Harvard Business Review study, 10 years into the profession or the field, 41% of the women leave the profession compared to 17% of men. So you have fewer women entering in the STEM fields and they're dropping out at a higher rates. And so as the first woman head of the United States Patent and Trademark Office in our country's history, and the Patent Office goes back to the Constitution, so it's been around since the founding of our country. Patent office and the Marines, right? Those are the two founding institutions. We're in the Constitution. So from day one of our country, right? We had a Patent Office. So it took a little while, but they got a woman head. And I give credit to President Obama and so forth for putting me in this position. And I'm very honored to have it. But I also looked at the numbers of the USPTO. And what we've found is about 16% to 18% of the Patent applications filed have at least one woman as an inventor. 15% to 16%, that's still pretty low. So that's an issue that we are looking at. We are engaging in particularly targeted outreach to encourage starting from the beginning at elementary school level. We've got to get more girls interested in STEM. And so to that end, we sponsor a program with Invent now called Camp Invention. And you're going to like this for kids across the country, really. We have 100,000 kids who enter into a one week summer enrichment program where they come to Camp Invention. And basically it's a very carefully designed program where they're given a design project, build a pinball machine. They go into a room, they get wire, metal, wheels, batteries, balsa wood. And then they have to design a pinball machine or whatever the objective is. They have to sketch out a design. They have to prototype it. They have to build it. They have to tweak it. They have to revise it. If it doesn't work, they have to iterate again. And then they sketch out the basics of a patent and they start thinking about a business model. So it's hands on learning about the application of science and STEM technologies. It's not textbook reading of science and technology. You should have seen. I visited Camp Invention in Alexandria, Virginia. There was not a board kid in that room. They were all engaged. They were collaborating. Fantastic. That's the kind of programming that's going to get all of our children across America interested in kind of invention, creation, making. That's one example. Another example is that we have established a Girl Scout patch on Intellectual Property. And you know what? I was a Girl Scout. And in order to earn this Girl Scout patch on Intellectual Property, you have to learn a little bit about patents, copyrights, trademarks, trade secrets. And you have to actually build something and create something. And if so, at the end of the process, you earn an Intellectual Property patch. And that's great, because like I said, I was a Girl Scout in the patches I remember earning. We're on sewing and first aid. Right, right. And I think we can do a little better than that. And we should be doing a little better that in this day and age, don't you? 15%.  But it's interesting because what you're doing is really taking kind of another order of degree of really perception of what they're working on. It's one thing to kind of design and build an invention that's cool and put all those things together. It's a whole other thing to think about the concept of, how do you patent it? How do you trademark it? The intellectual property behind the invention. So that is super innovative. And do they get kind of separation between those two concepts? Yeah, I built it. OK, now your task is to define it in such a way that you basically own the concept. They do. They do. There's a difference between creating it, inventing it, prototyping it, and then kind of sketching out the basics of what is novel here that might be new, that nobody else has done this way before. And then the next step, which we would like to encourage them to go in, is what can I do as a business matter to capitalize upon this invention that I've created or this thing I've created? And what kind of company might I establish and what need in society might it meet? Right, right. So we're very excited about programs like that. And we also have a collegiate inventors program where we have a competition for college students across the country. And they have to submit their inventions. We have a committee that evaluates them from industry and academia. And the winners share prize in the amount of $100,000 total. And many of them, there's no restrictions on the prize money. They can use it for anything they want. And many of them put it into a company that they're starting. And they oftentimes file a patent on their invention.  And if you think about it, immunizations are usually given by shots drawn from vials that need to be refrigerated. In this case, if you're going into third world countries where you don't actually have as easy access to refrigeration and the maintenance of the proper temperature for the vaccine, a patch that slowly releases the vaccine into your body or the immunization agent is actually a much preferable way. So it was an incredible invention. I think she got a patent on it. Where did she go to school? Do you remember? I think it was SUNY, one of SUNY's schools. Yeah, Stony Brook. I can't remember, but it's from SUNY, yeah. I was going to ask if you're doing stuff in college. Because we've interviewed Maria Clavet, who's a president at Harvey Mudd, and also on the board of Microsoft. And I asked her this similar question. And her read on it is really that the young girl's stuff is important, but it's less important as really that early years in college. She said, you know, that's where you can really get them with a really solid CS class, EE class, and really lock it. Because they're kind of mature enough to understand and to get it and to fall in love. And that's where she spends a lot of her time, obviously. She's running Harvey Mudd, so she's got a vested interest there. But really going after the college thing. I think every step along the way is needed. I mean, you can't start too early. You can't get our kids excited about invention, creation, and science, technology, engineering, and math too early. The earlier you spark it, the more you maintain it. And every stage along the way, the better. That avoids the attrition that we were talking about. So we're very excited about that. And at the USPTO, we're taking a number of steps as well to make sure that we have that next generation of inventors ready to file patent applications and to start companies and to make our society and our economy strong. Because it will only stay strong as long as we continue to innovate. Absolutely. So, Michelle, I really appreciate your time. I know you're busy. You're flying all up and down the coast. You're visiting people. So I'll give you the last word before we sign off. What are you excited about the next six months? It's kind of on the top of your agenda. Let's see. Where's the list too long? The list is very long. In the next six months, I guess I'm excited to see where we end up on the patent reform legislation. I'm very excited to see the improvements that we'll continue to make to these proceedings that are before the Patent and Trademark Office called Post-Grant Review, which allows the public to bring a patent back to the office to test the validity of the patent in a manner that's cheaper than district court litigation. So it's playing a critical role in our overall ecosystem in making sure that we have the very best quality patents in the system. And that patents that should not have issued can be removed. And patents that issue or should have issued stay in the system. Awesome. Well, thank you very much. Thank you. Awesome. So I'm Jeff Frick with Michelle Lee. We've been talking about the US Patent and Trademark Office. You're watching theCUBE. See you next time.


Once logged into your RPW account, select Capital Works Claim. You will then be asked to carefully read the following key messages before progressing with your Capital Works Claim. Once you've read the messages, please click Next. You will then need to select the contracts for which you want to claim Capital Works items. As an example, this account currently has two claims. Once completed, click Next. The next section is the activities available to claim. You need to place a tick in the Claim this activity column for each individual activity that you wish to claim at this time. For each activity claimed, the approved quantity will be copied to the claimed quantity. You can type a lesser amount into the claimed quantity if you wish, but you cannot claim a larger quantity. An example of this would be if a gate had been installed which covered a space of 4 meters. You must also input a completion date for each activity in the date activity completed. Once you have selected all the available claims and double-checked the quantity, click Next. You are now at the point of submitting your Capital Works Claim. You must ensure that you have selected the appropriate activities that are to be included and that all details are complete and accurate. Please double-check before proceeding. You are now ready to submit your claim, but please make sure that you have read and understood all of the relevant guidance notes on Glastier Capital Works contained within the Glastier contract before clicking Submit. You then need to select the two options at the bottom of the form in relation to submitting your geotagged photographs and agreeing to the declarations and undertakings. When uploading your geotagged photographs, you need to provide a before and after photograph as shown. For this example, we will upload them via the document upload menu on the home page. Thank you for listening. If you want to get involved click Next. If you want to make sure that you have signed the contract for the coming date event, in fact, the date should be prepared by one from now on. If you are in a hurry click Next. The date to submit the geotagged photographs has been ready by one. For you to have taken photos of the geotagged photographs, please take a look at the geotagged photos from the geotagged members of the English geotagged group. For the geotagged photos, please click Next in the links to the gheadoi form in the description below of the photo. The depth of the cave is the depth of the water, but we don't know the depth of the cave. The entrance is about 15 metres. There are a lot of people in the cave that come from the cave. The whole cave is located in the cave, and the cave is the size of the cave. The new product of the camera is the Kavlunna Khauliat Kwaik Kavalaf. This is a brand that has been a very popular product in the country. And it is very popular now with the Kovalhaingawir brand, I am looking forward to it! The new product of the Kavlunna Khauliat is a brand that has been popular in the past for the last few years. It is the Kavalaf Glass Steer and the Contrad Glass Steer Kavlunna Khais. Here, the best option for the world of photography is the Kavlunna Photography and Geo Tag and it's written in both English and English. There's a lot of photography and beauty in Llan Lwitho and I'd like to share with you what it's all about. In the end, Llan Lwitho is one of the most popular Llan Lwithos in the world. Thank you very much.


Software is essential to our lives. Imagine living without the internet, Google, Facebook, or your smartphone. We expect our software to work correctly. A bug in a program can make it really frustrating to use, or it could even correct your computer. But it could also be a lot worse than that. What if the software controlling a nuclear power plant fails? Or the software controlling an airplane? Unfortunately, software engineering techniques can help us make software that works correctly. Making good software is difficult, because often software is huge and complex. Some large systems have tens of millions of lines of code. If you printed them out on paper and stacked them like a book, you'd get a stack that's around 100 meters high. That's as high as a 25-story building. If you wanted to read through a program that's size to try to understand how it works, it's likely to take you about 50 years. If some software takes almost a lifetime just to read through, imagine how long it would take to write it. Of course no single person can do this alone. We need teams of developers working on software, sometimes three developers, sometimes ten developers, sometimes a hundred, and sometimes more. This is where software engineering comes in. How do we get a team of developers to write huge software where all the pieces fit together, it works reliably, and it does what the user wants? Software engineering is about so much more than just programming. In fact, the actual programming part is usually only about 20% of software projects. Even today, we are not good at getting software development right. A survey in 2009 found that only around a third of software projects succeeded, while a quarter of projects failed outright, or were cancelled before the software could be delivered. Fortunately, we now know a lot about what works and what doesn't. In this chapter, we'll look at some approaches that add doom to failure, but are still used surprisingly often, and some approaches that seem to work a lot better and can help us create software that does what it's meant to.


Mr. President, any instructions to our negotiators returning to Geneva on the medium-range missiles? I don't think there's something we'll talk about right now. Do the prospects look good for the agreement? I don't think there's something we'll talk about right now. Do you expect anything from us this year? We're never going to have anything. Mr. President, I want to commend Bobby. House leaders shared such a good showing of the Republican unity during action two weeks ago on the Democrats' budget resolution. The budget you fought against, I think, is a catalog of all the wrong priorities. First, big defense concept, critical time in our discussions with the Soviets. It's a lousy way to go into negotiation with them, doing what they were doing to us. Second, tax increases on the heels of tax reform. Some of those who fought tax reform last year just won't give up this campaign to raise taxes. And third, dimension-expending increases that had to be for the deficit. And now in the Senate, Bob, I understand that the Budget Committee Democrats refused to eat cake. He commented these warnings that they have been recommended a plan to the Senate to similar to that adopted in the House. I think we're in a strong position to stick together and oppose any of these plans to raise taxes on gut defense. And I hope that you and the Senate will turn aside the pleas of those who say that's made a deal. Even if there's no deal, I think it can be made on these scores. We do need to fix the process. It's broke now and the country's paying a steep price. And once we achieve permanent and fundamental reforms, then perhaps we can have the tools to get on with the deficit reduction in the right way by cutting spending and not raising taxes. And the floor is open. I really don't want to comment on that. I think it's a key issue before us. And I think that we've got a lot of educated public to do on what the problem is. Can I just say since the House finished the Budget today, we may get it up this week, I guess. Mr. Avid. Oh, there. We'll see you again. Ambassador. Yes. Good to see you. How do you do? Good to see you. Welcome. Come in. Over there. You've been in here a number of times. Before we get out of any mundane business here having to do with trade and so forth. I thought I would mention the fact I want to make sure I get your name right. Your son's ironible in Shenzhen. Are you soon going to be married? Right away. Yeah. Very soon. The President, like the boys that get dinner for you. I'm going to say thank you very much. Well, it is more than great honor for them, I think, that you have mentioned it. Thank you very much, Mr. President. Well, then in a very short time. This is our teacher. I mean, it's just great to hear that. The physics being a rock. Thank you very much. Good to see you. Did you ever hear of a poem called Teacher Ritzy by Mark? Well, let me read one of this here first, please. Mary Fiori, would you like to see me? Nice to see you. How do you do? Very close. Good to see you. Nice to see you. I have to be surprised that more people are in the molders of their dreams. The gods who build or crush their young beliefs of right or wrong. You are the spark that sets a flame in the poet's hand or lights the flame of some great singer's song. You are the god of the young, the very young. You are the guardians of the million dreams. Your every smile or frown can heal or pierce a heart.  Yours the pride of loving them, the sorrow too. Your patient work, your touch, makes you the god of hope that fills their souls with dreams to make those dreams come true. And let me say to you that because I teach, I can dream and thank you very much. I accept this award as a symbol and recognition of all of America's finest teachers. And thank you for recognizing me. Well, I'm very pleased with honor to be able to do that. And on behalf of the state of New York, I would like to present you as a servant of my appreciation for what we do for educators and teachers. Well, thank you very much. I am very pleased to add this to what is something of a little collection. You're meant to be in the number. Well, thank you. You're welcome. I'm very pleased. President, we're going to do one big group photo here. So we're going to have you center on the door and we'll have some of them together on the right side here. Let's get the baby. Just squeeze in as tight as you can. That's great. One more. Thank you, sir. Here, as you can see out there, as we've been away from the head and the rose garden, his roses is true themselves. I've never seen a fight so boom and where it's always been. Congratulations. Thank you. And so what you mean. Thank you, sir. Oh, sure. Donald. Donald. The congressman. Oh, oh, oh. The president. Yeah. Yeah. It's so good. Just you can sit and do nothing. Oh, sir. Thank you. Thank you. Thank you. Good morning. Oh, thank you. Mr. President, since I'm in our congress room, if you would, if we three would be together here, what would put it? Better put, Donald. Better put your eye. Right. That's great. Thank you. Right here. Thank you. Thank you. So, thank you. Thank you. Thank you. Thank you. I always thought you had to take off your glasses when the barbecue was working on me and I won't spend the time reading you, so I turned my hands away. You can read that way. Well, we're not commenting on things other than just that we hope that this is going to proceed to something meaningful for all of us. Mr. President, Mr. Geider and Mr. Lange seem to be coming back empty-handed from Tokyo. How big a problem is this? They seem to have been stonewalled. Well, I'll just wait and hear from them. Did the Special Envoy Abe convince you, sir, to rescind your sanctions? Did Shintaro Abe, when you spoke with him yesterday, was he persuasive in encouraging you to rescind the sanctions and let it round? No, we've met several times. We had a president meeting here and both of us hope that everything can be worked out. Is Nakasone going to be welcomed? Always. We're good friends. Should I turn my hand off? I'll take a mile tonight. Seven days, one a year. Even while you turn them off at night instead of a week? The agenda here isn't anything planned or formal. It has to do with it. We've been involved there. We had a report for some of us here the other day, and did you like it? You should have just taken a few minutes to chat with us. Well, listen, we used to do it. Come in. Thank you very much. Hi, Ken. Senator, how are you? Great. How are you? Good. See you as always. The chair is still warm. Well, no, I'm very pleased to hear this incident. Let me say something to you. If it had been brought to your attention, I understand that the Wall Street Journal contains a story that I have made the decision.  Well, that story did come to my attention, Mr. President. I have not made a decision as yet, and I'm waiting to hear it. Well, I thank you very much. I did call Will about it this morning, and I was not too...


The Padres just keep making people mad. This one's brought to you by DraftKings. Trent Grisham up at the plate. Love his little rhythm to get in timing. Love his whole presence in the box right here. Very casual, very cool, very calm, very collected. A little hip action, a little get in the rhythm. Rizzo does that. Nice pitch by Kershaw there to get the count. Even loses them there. Two-one pitch. It is hit hard. Go! Let's go! Screams at his teammates in the dugout. Then hits them with the little smirk. Now, this is very crucial for me. I think there's nothing wrong with what he did after the home run. He looked at his teammates. He didn't look at Kershaw and show them up. He just looked at his guys in the dugout because he tied the game in the sixth inning and let out a huge let's go. Nothing wrong with that, in my opinion. The smirk, though. Did the Dodgers talk first and then he hit him with the smirk? Or does he throw the smirk, the smug face on his own? Who cares, really, anyway? He holds up three because that's his ninth home run is what I read. I don't really get that. Kershaw's thinking, damn it, I just tied the game in the sixth inning. Then he holds up nine fingers there and he hits them with another smirk. Just dudes full of the smug smirk. So that's fine, I guess. And then the Dodgers are yelling at him. You can hear it. So he yells back. I'm not sure what he says here. It kind of looks like y'all soft, mother fuckers, soft. But I don't think it's soft.     As he casually jumps on home plate like a little kid who just hit his first little like home run.  Is excited with his team again. So it's a little bit of both. Dave Roberts didn't like any of it. This was his reaction. That's not a picture. That's his actual reaction to all of this just went still on us. And then here was his quotes in the press conference afterwards.   But I just felt that, you know, to over kind of stand home plate, certainly against a guy like Clayton who's got the respect of everyone in the big leagues and what he's done in this game. I just took exception to that because I think that there's a certain respect that you give a guy that, you know, you know, if you're homer against him. Roberts is a little guy but he knew as soon as he hit it. So there you have it. You can celebrate home runs if you hit them off shitty pitchers. But if you hit a home run off one of the best pitchers in the game, how dare you celebrate that? That's not even an accomplishment. Didn't even tie the game in the sixth inning or anything between two teams in a divisional race. Baseball's got to get out of its own way but this is all good because the Padres are just getting more bulletin board material. This one's brought to you by DraftKings. Go pick up a Padre. They're 21 and four in their last 25 games. They've got six players above league average and Jerks and Profar is about to join that squad and become the seventh Padre with an OPS plus above 100. So go get DraftKings, use promo code Johnboy and just find the cheapest Padre, put them on your squad and he'll do good for you. It's basically about as simple as it gets. They're the best team in the history of baseball. Astrix not really, don't take me too seriously.