Hello, and good morning, good afternoon, and good evening, everyone. Thanks for joining the fifth installment of the A&A Avatar XPRIZE Meet the Teams interview series. Today we're here with Team Northeastern and several of their team members. This is going to be a really interesting session, so we are glad you took the time to join us. My name is Colin Pairtree. I'm the team coordinator for the A&A Avatar XPRIZE. I'm also joined by my colleague and technical consultant, Jackie Maury. And we are here once again with Team Northeastern. There are six members of this team joining us today, and they have a lot of great interesting information and content to share with us today. And so we are very excited to move forward. Before we begin, just a few notes about this webinar. You will all be muted during the call, but you're welcome to use the chat function at the bottom of your screen to get in touch with us or to say hello to the others on the call. We also invite you throughout the session to submit questions that you have for Team Northeastern into the Q&A function. We'll take some time at the end of the call to add those to our discussion, so we invite you to present any questions to us. We'd be happy to take a look at them. A quick overview of our agenda today, we're going to say hello and welcome to you and also to Team Northeastern. I'll be passing it over to the director of the Institute of Experiential Robotics in Northeastern in just a moment. And as I mentioned, we'll take some time at the end of the call to do some questions and discussions with this team. So be sure to submit once again those questions. So once again, we're joined today by several members of a university-based team in Boston, Massachusetts from Team Northeastern University. The members of this team on the call are decidedly multidisciplinary in their work and studies, and that leads to a highly dynamic approach to their avatar system that you're about to hear more. We're going to be viewing some presentations from these team members, so be sure to once again submit your questions. We'd love to have those questions from you as discussion points and we'll make time throughout the session to address those. To begin, I'm really excited to introduce Tashkin Pideer, who is the director of the Institute for Experiential Robotics at Northeastern University, and he is the leader of the avatar XPRIZE team of Northeastern. Tashkin, welcome. It's a pleasure to have you here and the rest of your team, please take it away. Sounds good. Sounds good Colin. Thank you so much for the introduction and, you know, hello everybody and welcome to this session. So I will share my slides and then I'll try to give you an idea about what we're trying to do and why. So let me start with the presentation. So what keeps you going? You know, why avatar? Why did we decide to participate in this? So sometimes it helps us to reflect back in time and see what the technology achieved, right? So these are the early attempts to fly, a technology that we take it granted these days, but it all started with failures. It all started with exploration. It all started with trying to push the limit, trying to push the envelope towards something that hasn't been done before. So with this in mind, you know, those videos are long videos, but you know, I'll just keep scrolling. I want to take you to five years ago when we worked on the DARPA Robotics Challenge. So these are robots embracing the risks, right? So trying to achieve things that haven't been done before and, you know, every single team on the field experienced one failure or another. This is one of the famous videos of the DRC. Again, you can find it on YouTube. Now, we take flight granted and you saw some of the grandfathers of the Atlas robots from Boston Dynamics on the previous video. And now this is where the technology is, right? So for those of you who think that, well, robots will never get there. Or for those of you who think that technology will not get there, I think we are making as a community great, great progress. So now what is in our inventory in terms of projects that we worked on and why are we doing the Avatar XPRIZE challenge? So this is a quick time travel. This is exactly the team that I worked on in 2015. And this was an avatar. This was the DARPA Robotics Challenge to address the disaster response scenarios. Actually, what you see is a robot doing a number of tasks. You know, it just drove this vehicle. It's going to open the door, go through the door. Again, a disaster response inspired by Fukushima. But what you don't see is a 20, 30 people team that's 1000 feet away sitting in the garage, not seeing what the robot does, but seeing through the eyes of the robot and trying to complete these tasks. So these robots were not fully autonomous. They were called supervised by human operators. But then we started pushing the limits of task completion when it comes to robots that are complex in shape and form and enabling the operators to operate these systems to complete hard tasks, such as picking up a drill, turning it on and running it. You know, I was one of the team leads for the Team WPICMU and we did fairly well in that competition. But aside from that, you know, here are some of the visuals, right? So a complex robot, a team of people who look at the robot on the field, but then another team that is really controlling the robot behind the scenes and interpreting the data and understanding what happens. Now, how do we change that paradigm, you know, through the avatar XPRIZE and how can we achieve that, you know, many to one to perhaps one to one, right? So I will talk about some of the lessons learned from the DRC, even though it is five years ago, it is still pushing us to in our research, in our study. We truly believe in maximizing the utility of the human and the robot team, human and the machine team and achieving these complex tasks, making robots of all sorts and forms an avatar for the human capabilities. Oh boy, did this become important in the past six, seven months, right? So how many times did you think, I wish I had a robot avatar that can do my job somewhere far away from where I'm located? So here are some of the visuals from the challenge again, just to give you an idea of where we were and because I want to talk about some of the things, what we are, you know, there's no way that an untrained operator can operate a human or robot by looking at these screens. You need to train your operators, definitely. So what is the secret sauce? What did we do? We looked at the tasks very carefully during the challenge. They were defined, but then there were a lot of variability as well, right? Uncertainty is variability. You cannot plan everything and say that, OK, so the humans will be able to control these systems from end to end. For a human or robot like this, you get the first plan out, OK, it's just simply opening the door task. It is very intuitive for a human, but how will you do it with your avatar?  And then this factorization is critical because it enables human robot teaming, right? So where do you want your operator input to come into picture and what does the robot do autonomously, right? So one of our key contributions here or your key success stories here was the behaviors of nudging, right? So when the robot thought, OK, here's where I'm going to put my hand to engage the door handle, we allowed the operator to be able to nudge that decision because there's always that deeper human intuition where we think, oh, wait a minute, you won't be able to grab that handle because you're way too off from the center and things like that, right? So this is where we were building and that enabled us to formulate complex models of computation for task completion. Again, perception will be a key aspect because we will split at the end of the day, we will split the autonomy of the robot with the supervision of the human operator to achieve the true promise of the avatar technology. And again, that's a step-by-step process. Manipulation and motion planning, motion and task planning has been a huge focus of our teams. And what we can do is we can generate multiple tasks, multiple solutions to perform the same task using a complex system. Now, very briefly, what did we learn, right? So I mean, I think the capabilities make sense. But what was really critical as an outcome of the DARPA Robotics Challenge? Reliable hardware is critical. You cannot just put everything on the software side and hope that the hardware will operate. You have to have a strong hardware design and creation. Reliable software is critical. You know, those videos, those pictures have stories behind them, right? So for example, here, we dropped the drill during the day one tasks because a one-off software bug triggered and the robot just opened its hand when it was holding it. So reliable software is critical. This is, I think, one of the most important aspects from my perspective for the challenge, for the Avatar XPRIZE Challenge. We need to validate robot or the avatar behaviors. And then at the end of the day, we are aiming to maximize the utility of the human robot team. This is essential. A human understanding the capabilities and limitations of the robot avatar is important. And this is where we are aiming towards. And this is how we think that we'll be able to complete the tasks or the scenarios that are outlined in the challenge. And a rigorous validation strategy is critical.  And then a capability from the avatar side on understanding the errors based on uncertainties. It was based on the limitations of the technology to understanding the source of errors is critical. So this was a quick wrap on what we have done for the DRC five years ago, but then what have we been doing since then? You know, we've been working with NASA's humanoid robot technology, Valkyrie, and basically moving more and more autonomous behaviors to the robot side to achieve more complex tasks. So this is still a couple years old results where we were able to autonomously, you know, the human just says, go and pick up the box and then deliver, right? So this is a very intuitive way of, you know, that that supervision, human supervision, and then we have a whole line of autonomous behaviors where the robot can navigate through tight corners, understand, you know, what to pick up and what to do with the object and so on. Again, all these videos are available, will be available later on. And then, you know, more complex manipulation tasks as well. So how do you turn a valve to be able to adjust, you know, antennas and things like that? Valkyrie is a complex system, so it is not really intuitive for a human to control it, you know, joint by joint. So it needs to be at the supervisory level. But this is much more recent work where we simplified, if you just remember, although I very briefly, very quickly went through it, if you remember our DRC interface, it was much more complex. So now we can put a lot of things behind the scenes and then, you know, we can even add the autonomy at the interfaces level. So we have a much simpler user interface, if it's just to the, you know, keyboard mouse type of an interface. Oh, by the way, you know, with the emergence of the virtual reality interfaces, now we can do even better. We can allow the operator to pick up, you know, okay, so you want this robot to walk, right? So robot can plan out its steps and then all what remains to the operator or the human far away is to validate that plan and say, go and execute or, you know, reaching, right? So if you want to, you know, it's interesting when you start controlling a human robot, you know, things are really complex, right? So do you reach with the right arm or do you reach with the left arm, right? So we can leave these decisions selectively to the human operator and these are just some of the results that we recently obtained. And then here's another idea, although this is a little bit far away from the concept of avatar, but this shows us how we use the virtual reality tools to achieve this transparency in the human robot collaboration. And I think it will be another key capability for the avatar challenge. How does the robot understand what the human is going to do versus how does the human understand what the robot actions are, right? So it's easier said than done when I said that we put more autonomy on the robot, but then whenever you have human in the loop, if the human doesn't have an idea about what the robot is doing, that doesn't become a true team arrangement in my opinion, in our opinion. So we are trying to push the limits on that front. This wraps up my quick overview of, you know, specifically what I do in my lab and, you know, some of the visions that we have for our team's participation in the avatar XPRIZE. Next, I'm going to pass on the button to Professor Peter Whitney and he will talk about what he works on and some of the hardware technologies that we are developing. Peter. Thank you, Tashkin. I'm sharing my screen now. So Tashkin took us through a big picture of all the pieces that we need to be successful in the avatar challenge and my focus and one of the critical things that Tashkin pointed to is reliable hardware. So I'm going to talk about some of the research I've done in the past and how we're going to bring some of those elements in, in particular for hand design and tactile interaction. And after my presentation, I'll pass it off. We have some student groups that are working more on the human machine interface side. So exoskeleton design and bringing some of these elements all together. All right. Just a second. So actually, before I came to Northeastern, I actually worked for Disney and this is a project some of you might be familiar with. This is actually an avatar system named Jimmy. And we developed the, we developed a system to create physically interactive experiences via robots for guests in Disney theme parks. So that's actually me behind the scenes behind the wall, tele-operating Jimmy. And I'll speak in a second about some of the technology behind the scenes. We looked at a couple of different approaches. This is some older work using pneumatic transmissions to generate very smooth, lightweight robot arm motion. And in the rest of the video, Jimmy is driven through a water based hydraulic transmission. So highlighting some of the features and capabilities of this kind of a system. So this is tele-operated needle threading. Where are my cheeks? And of course, if you're going to build human safe robots, I feel like you have to test it on your own children. If you want to show people what it's really capable of by Jimmy. So in this clip, actually, there was a brief second where it shows behind the scenes what's going on here. There's actually two identical robots, but they're not electrically coupled through motors. They're actually physically coupled through a hydrostatic transmission. And the concept is not unlike what some of you might have seen. This is a kind of a popular educational exercise for students. You take some syringes and connect them and fill them with water, and you create a sort of manual remote transmission. And so this concept took this concept in and improved it in several areas. So one thing, you have a lot of friction in the seals of this sort of a hydrostatic transmission. So we use special nearly zero static friction rolling diaphragms. It's sort of a soft fiber reinforced element. And it gives us a nearly frictionless ability to transmit force in motion. And one thing that's not obvious in that Disney video is the operator can actually feel what's going on. So picking up that egg and moving it, you don't stumble and drop it because you actually can feel contact. So it's kind of hard to demonstrate to you the sense of touch in a video, but we'll come back to that in a later slide. So what we've just seen is basically the sort of passive teleoperation. And of course, if you've got hoses connected, you can only go a few tens of meters before the friction in the fluid lines becomes too much. And so obviously, we can't stretch hydraulic hoses across the country or over long distances. So instead what we can do is we can use this similar kind of transmission to build a motorized arm. And so here for an avatar, you'd actually have two copies. So we have a motorized avatar, and then we would have a motorized haptic interface. So Towshkin talked a lot about shared autonomy, and what I'm presenting about is direct haptic telemanipulation or teleoperation. So again, for the avatar system, we would have two of these systems, and they would be coupled electrically over arbitrary distances. The video from Disney, we used rotary actuators for all of the joints, but in recent work here at Northeastern, we've developed these ultra low friction linear motion actuators. And maybe I skipped over it, but you might be wondering, well, why do we need these hydrostatic actuators that are driven by motors? Why not just use motors directly? And the main reason is that if we want many actuated degrees of freedom, if we have to put motors at all of these joints, it makes the arms incredibly heavy. And so heavy arms can be dangerous. It sort of filters out and blunts your sense of touch. So by using this transmission, we're able to take every single motor out of the hands, out of the arms, and mount all of those motors remotely. So here, this actuator can actually push with up to 100 pounds of force, but it only weighs 70 grams itself. So we can take a couple of these and build a two degree of freedom gripper. So we have a sort of wrist flexion extension motion and then a grasping motion. And so to operate this, we have two motors you can see at the top. These are direct drive brushless motors, and they're remotely operating this gripper here. And so you can see tele-operating. We can get very, very fast, very, very smooth motion. And we can also, the system is very easily back drivable. This is showing actually measuring the pressure in the hydraulic lines. So the sense of touch is incredibly sensitive. This is a five degree of freedom system doing some simple tele-operation. This is kind of interesting. This is showing squeezing different bits of foam. The traces above, you can see in orange, that's actually the force that we're measuring. So that memory foam has sort of a relaxation property. We actually had a project to do seafood handling. So being able to pick up seafood and actually sense the mechanical properties of the filet there. This is the students having some fun with high speed seafood processing. This is a slide from a related project that we have. Actually on doing underwater avatar type tele-manipulation for explosive ordinance disposal tasks. So this is kind of highlighting the fully haptic tele-manipulation strategy. So you'd have two duplicate arms. And here we're showing just with sort of a glove gripper interface. So there's a duplicate gripper that's actually allowing force coupling between the avatar and the operator. So they're getting a fine sense of touch at the gripper. And later Carl is going to talk about some efforts to actually build a full exoskeleton system. Before I hand it off, I want to talk a little bit about sort of a connection that I have to avatar that's sort of interesting is you know when you think about avatar and everything you'd want it to do and all the sort of tasks that you'd want to do. I've had sort of similar ideas about you know what would be the ultimate task, what would be the hardest task that we could ask a robot to do. So this is Baxter, some of you might be familiar. It's a series elastic actuated robot that was built by Rethink Robotics. It's not incredibly accurate, but it does have more compliance and force sensitivity. But my question to you is would you be willing to take a Baxter give him a straight razor and then have Baxter shave you with a straight razor? Maybe in the time of COVID this is not so extraordinary a concept, but if you think actually about the task of shaving you know if we take a straight razor and we think about well not only do we need to be able to control the position of the blade as you're trying to shave someone's face, but you need to be able to control the contact forces. And then you also for example need maybe a reflex motion. So if you notice that there is some kind of a slicing motion you can quickly pull out of the way. And of course this kind of task can't be executed if there isn't a sense of touch. So we got the idea not with a fully autonomous system yet, but could we actually take on straight razor shaving with this hydrostatic transmission technology? And so we actually recently did a test here. So this is a little more. So this is actually a three degree of freedom Jimmy style manipulator. That angle is great. Now just keep trying to keep your head in that same angle. Okay. All right. So when we're gonna put the razor on. Yeah. Straight your skin nice and tight from here. Yeah. Okay. And then I'll come straight down on that side. Okay. So just a point to note out here is this is a friend of mine who is a professional barber that's actually going to shave me with the straight razor here. So there's no motors involved in this case. This is just again the passive direct physical manipulation. Lean your head over to your left a little more. The idea here is that is that he has a very fine sense of touch.  Feels pretty good. Does it feel like it's slippery enough? Yes. It's not too sticky. So yeah, this was kind of a pre avatar challenge for myself thinking about you know an ultimate an ultimate test that that really it's getting more and more natural. Yeah. So we can do it. It really captures the challenges of robotics and really kind of brings right to your mind you know what are the weaknesses and what are the limitations. And if you think about that task and you think about handing it over to a robot I think it helps identify areas that need improvement. So yeah, I just briefly want to mention some sponsors that have funded this work. Some of this is co-work with Toshkin and also wanted to mention some graduate students and collaborators that have participated in this work. So thank you. I'm going to hand it over to Carl Swanson. Carl is one of many undergraduate students that are working on several different projects to kind of piece together different elements of technology with a primary focus on human machine interface. So thank you and I'll hand it over to Carl. Thank you, Professor Whitney. All right. I'm going to start sharing my screen then. All right. So now as a team we're going to give a brief overview of the system that we're developing now. So my name is Carl Swanson. I am a senior at Northeastern University currently pursuing a combined degree in computer engineering, computer science. Outside of classes I've spent a lot of time within Northeastern's aerospace organization working on anything from drones to avionics bays to NASA student launch which is a competition involving rovers and drones as well. So in a project lead role I've worked within Northeastern's NUAV group which is Northeastern unmanned aerial vehicle working on companion computing between drones where multiple drones work together to accomplish various tasks. On co-op I've worked as a software engineer or co-op at iRobot working on their new Tara Lawnmower. Then I worked at Square Robot working on an AUV, a Thomas underwater vehicle that went into oil tanks to inspect and for damage using various sensors and then I'm currently working at Amazon Robotics on other things. Growing up reading Asimov my interests in robotics rely in removing humans from dangerous situations and additionally giving them time back into their lives. And so the avatar project really presents like the perfect mixture of that. I mean so this project the avatar system can remove a human from a dangerous situation for like Fukushima's like that power plant. If someone has to work in outer space like an astronaut inside a space station needs to very carefully manipulate something to fix a satellite. Same thing there can work there but it can also allow us to get more time back into her lives like Professor Padir mentioned earlier removing a human from like a workplace when they don't necessarily have to be there. And now Anya will give her a I'll lead over to her. Yes I am Anya Derek. I'm a fourth year Electrical Engineering student at Northeastern University. Outside of class I've done two co-ops my first one being at Teradine in the semiconductor industry and right now I'm at Lutron Electronics doing design and development. My interests are mainly in PCB design so that's why I joined this project. And then outside of class I typically volunteer for STEM mentoring and teaching as well as some side research projects related to corporate response to social movements. And the rest of the EE team consists of Shay and Peter. So Shay is okay. Shay is a fifth year Electrical and Computer Engineering student. Shay is currently on co-op at Draper Labs but she has also done co-ops at Boston Engineering and Parkstone in Singapore and Shayne's interests are mainly in electronics as well as wireless capabilities which is why she decided to join this project. And Peter downward Peter is currently so he couldn't join us. Peter is a fifth year Electrical Engineering student. He's done two co-ops as well. His interests are mostly in the mix between electrical and mechanical systems so he's done a co-op at Electrical GT as well as Professor Padir's robotics lab where he worked on a five degree of freedom robotic arm and he is also very much interested in sustainability and the technology sector and kind of integrating that with mechanical and electrical design. And then additionally on the software side we have Peter Albanese and Spencer Sotian. Peter has a bunch of experience in better robotics both at the university and on co-op. On co-op he's worked at Kinetic North America and MCAS Instruments and he's currently working on some research with machine learning and artificial intelligence at the Northeastern Burlington campus that calls this research institute. He's very excited to work on this system that's going to include haptics and really work on providing like biomimetic motion to the human user and he's excited to create new software solution for a project that is really cutting edge and hasn't been seen before. Spencer additionally has a lot of experience with better firmware development and electrical engineering. On co-op he worked at analog devices, MCAS Instruments and then currently Amazon Robotics and he has a specialized interest in real time operating systems which will be extremely useful for us when we move into that on the system itself and is interested in low-level communication protocols like SPI and things like that. And now we'll lead off into the MECI team. Hi, my name is Mike Polkarri. I'm just going to introduce myself and the four other members that are of the mechanical end of things. So like I said, my name is Mike Polkarri. I'm a candidate for a Bachelor's of Science in Mechanical Engineering and it's my senior year at Northeastern. I've been lucky to have two vastly different co-op experiences at Northeastern. My first was with A-123 Systems where I got to work in process engineering. We mainly work in streamlining the manufacturing process for electrodes in electric car batteries. Right now I'm working at Viking Detection in a mechanical design role aiding with the design of an under vehicle x-ray system that's designed for law enforcement. For me personally I love being an engineer because engineering bridges the gap between science and creativity and now it has the potential to bridge the gap between technology and the human experience. And so being part of Northeastern's avatar team is a once-in-a-lifetime chance to do that. I'm going to hand it over to my friend Al to introduce himself real quick. Hey, I'm Albert Saunders. I'm a fifth year mechanical engineering major at Northeastern University with minors in material science and mathematics. I've had three co-op experiences so far. My first two in the biomedical device sector where I worked at diffuse synthese on the shoulder reconstruction team, a similar role at spine frontier on the spine implant team and right now at Amazon Robotics. I also work part-time on the information technology team for the Boston Red Sox at Fenway Park. And I'm hoping to use the great knowledge that I've gathered in classes and on co-op in robotics shoulder biomechanics and material science to inform our team decisions on mechanical design and biomimicry. I got into engineering for a very simple reason. I just want to help people using science technology and I feel like at this particular point in human history, the avatar competition presents itself at a great time. You know, remote telepresence is becoming more and more impactful and necessary and I feel like getting on this project at this point in time is very filling to me and I hope to help as much as I can. I'm really excited to work on this. So to introduce the rest of the mechanical engineering team that couldn't be here today, we have Albert Demers who is a senior mechanical engineering major with a minor in robotics. He enjoys volunteering through Northeastern's outreach program to teach robotics to elementary school children. He has held electromagnetic co-op positions at Stone Ridge in Stron and Raytheon and he hopes to utilize his knowledge of robotics to achieve effective telepresence. We also have Sepair Kansalar who is a combined BS MS in mechanical engineering major. He is a former product design development engineering co-op at Proctor & Gamble. We're very lucky to have his experience in design for manufacturability, injection molding, mold design and the like. Not only that, he is also professionally certified in SOLIDWORKS and he enjoys studying a variety of robotics topics and I'll hand it back over to Mike to introduce John Corey. Thanks Al. Last but not least we have John Corey. John is a senior mechanical engineering major here at Northeastern with a minor in mathematics. He's also the co-founder and lead project manager at Northeastern's VR club. Currently he's working at FLIR unmanned ground systems in an R&D position. In addition to all this, he's also a self-made entrepreneur with a number of patents pending on various products and he's excited to apply his robotics and VR skill sets towards making a motive and expressive telepresence impossible. So the current avatar system was designed with a few high level design requirements in mind. The team desired a wearable exoskeleton driving robotic arm with five degrees of freedom, a three kilogram maximum load at the end effector and a .5 meter per second maximum end effector speed. The team wanted to focus on biomimicry first to really effectively communicate the human presence from one spot to a remote location while maximizing the positional accuracy of the end effector. Kind of like what Professor Whitney was talking about earlier to improve safety and allow it to be used around humans. So with those requirements in mind the current exoskeleton design has four degrees of freedom, three at the shoulder and one at the elbow. The components are 3D printed to minimize the weight to about seven pounds. All of the joints are directly driven to eliminate backlash and the shoulder joint has three axes of rotation that intersect at one point and that back plate is fixed behind the user for usability and comfortability. Straps comfortably fix the exoskeleton to the operator's arm for effective motion. And coming over to the robotic arm design, it currently has five degrees of freedom, three degrees at the shoulder, one degree at the elbow, and one degree at the wrist for pronation and supination. This has brought the total weight down to about 15 pounds total actually weight due in part by the spanners that are single 30 millimeter carbon fiber tube which offer us high stiffness at a low weight and we utilize 3D printed bonding coupling ends for ideal torque transfer. I'm going to pass it back over to Mike to talk about differentials and get into more of an integrity of our design. Okay, so with specific regard to the robotic arm it is comprised of three main sub-assemblies. They are two identical differentials and one shoulder rocker. The differential itself was designed with modularity and ease of iteration. Differential to mimic either the shoulder or the elbow and to be swapped freely if necessary. Each differential is comprised of four sub-assemblies that you can see here. They are the motor housing, two identical timing belt transmissions, and a bevel gear drive. This design grants that differential two degrees of freedom. They're mainly made of 3D printed plastic but critical components are made of steel such as the gears and the overall weight is approximately four and a half pounds. Similarly, the shoulder rocker is an additional motor sub-assembly that is only present at the shoulder of the robotic arm. This rocker grants the shoulder one additional degree of freedom to fully mimic the functionality of the human shoulder and has identical belt transmission to that of the differential and is made of a single piece of 3D printed plastic. Additionally, this is mounted at a 30 degree angle to biomedically match the human shoulder and make that transition more intuitive. Another thing we've been working on is a virtual reality operator interface and essentially this would allow the user to take in visual data through the use of a VR headset letting them look around the room. It mirrors the roll pitch and yaw of the head and allows the user to convey subtle body language. I will let the video speak for itself. It's presented by our teammate John Corey. The avatar camera system is able to copy the roll pitch and yaw of the operator by converting the gyroscope data from the VR headset into motor commands. This not only allows for real-time visual feedback but also helps give the avatar personality by depicting the subtle head movements of the operator. This feature, in addition to others incorporated throughout the system, brings us closer to achieving the desired simulation of the user's persona at the point of the avatar or son's icon. Thanks guys. All right, now we're going to go into a little bit more detail on how our software stack is going to work. So on the software side we're going to be employing ROS as a robotics middleware. ROS is going to handle much of the heavy lifting, including the physical simulation, motion planning, and importantly the message passing between applications on our arm. Following preliminary research we're going to be using ROS control which is a common ROS package to tie together our simulation and hardware. This is going to allow us to run hardware and loop testing, permitting us to visualize how our arm is moving not only in the real world but also in software. So as I mentioned before we're leveraging ROS control to provide this interface between the simulated real life and simulated in real life environment. It also gives us the ability to tune easily the PID loops for our system and gives us the ability to send simple effort commands to our arm's joints which then are seamlessly translated into electrical signals sent to our drive boards. ROS is going to assist in creating a software structure where each one of our nodes can easily access information from all the other nodes on our system. In such a complex system simulation is more important than ever. By simulating our robotic arm bugs that may cause our arm to physically break in the real world we're instead just simply fail in the simulation. It's a lot easier to fix problems that are like a simple line of code rather than buying more parts spending money and reordering broken materials. To simulate the motion of our arm we're going to be using gazebo here. It's a real-time physics simulation that hooks into ROS and by using an accurate model of our arm with mass distributed correctly across the model we will additionally be able to simulate PID tuning in software. Using gazebo has additional benefits as it's going to allow us to visualize the requested and actual position of the robotic's arm as it is commanded by the exoskeleton. All right at this point I'd like to go through and run a demo so I'm going to switch over to my phone. One moment that will take me just five seconds here. All right can everyone see me there? Good? Yeah. So as you can see our system here we have the Apollo system in the strap to the user to go in. It's currently powered by a power supply that means I salvaged off the previous projects. Unfortunately due to COVID we were forced to split up some of the project parts between different members so I could work on it. So currently we only have the exoskeleton here while the robotic arm itself is in another location. Currently for this demonstration I'll be showing driving a single motor and then reading the encoder positions off of two of these motors right there. Set that up real quick. So it's fairly easy for the user to strap in such just mount this right above the elbow. Connect it down there and then if you watch my arm here I am mirroring the movement off of this single joint onto that joint over there onto that single o-drive powering that motor. It mirrors it quite well so I didn't like make my new change like one degree. It's extremely responsive and this will allow the user to perform very delicate tasks and in general love to act like a human would in normal situation. Now while I'm working with this Anya is going to give a couple just details on how we're choosing code for our system and the power systems in general. So as you can probably see hanging off the side there we are currently at the moment using o-drives to control our motors. O-drives are open source and affordable so they gave us the flexibility we needed in times of COVID to quickly and rapidly just integrate everything. We did create our own encoder PCBs that are mounted. There are slots inside the exoskeleton where the encoder PCBs are mounted and we specifically picked on-axis magnetic position sensors to do our encoding specifically the AS5047B because it's high precision high speed it has 14-bit core resolution so it really gave us what we needed in terms of having precise motion and being able to mirror it. At the moment we're using SPI as our communication protocol for the encoders but we did pick an encoder that does have options for both absolute and incremental positioning so we can in the future potentially add that to our system and then we do also plan on developing custom controllers in addition to the o-drives just to give us more flexibility in terms of picking our communication protocol although all drives that recently start supporting CAN communication which is what we do plan on using but having our own board will enable us to kind of integrate the controller part of our system with the encoders and kind of have it all in one single board with the communication protocol that we want and it's sort of future-proofing us in a way because if we end up adding any functionalities to our boards in the future it will be easy to just add that on instead of starting from scratch pretty much but yeah we are using or planning on using STM32 microcontrollers for those boards which are going to be modeled after the o-drive but we're going to add our own player to it to show another demo of how we're able to accurately read the additional values from here this one this is from the one of the shoulder joints and then the arm joint itself that one is discreetly set there and it shows that we can track individually each one of these joints as it moves to a relatively high degree of precision that's it for the demo i'm going to hop back on my normal computer and we can do any questions yeah well Carl hops back on this is definitely just the beginning stages of our project we're definitely looking forward to integrating all of the work that professor for the year and professor Whitney have been doing in their own labs and kind of putting it all together eventually but COVID has definitely presented itself as a challenge in terms of getting our team together and kind of starting the piece together our system wow thank you very much for that demo and the walkthrough of your entire system is all your team as well whether they were able to be here or if you were just showing their their head showing screen it's clear to me that as i said before that there really is a multidisciplinary approach to this team tons of knowledge base that's coming through for each of your team members so it's really impressive to see that hear about it and also really cool to just have despite the the challenges that were presented by current state of affairs in this world just having that in literally your backyard your living room to to work on that pretty interesting car so thanks for showing that demo anyone who's listening in in the audience you're welcome to submit questions as we go through we're just going to be doing a discussion for the next 10 15 minutes or so so we invite you to write into us and post questions to this team i do want to hop back to maybe go through the beginning of the presentation peter the way that you were using that system for not only jimmy but also the shaving experience that's physical connection and it presents a really impressive level of dexterity and also compatibility mapping the motions of your of yourself what kind of how similar can you get that sort of level of i guess maybe fidelity is a good word but quality of motion with a system that's teleoperated there's a lot of fine motions happening in that one what's what's the state of the art as far as what we can do with something that's farther away yeah that's a great question so i think the in the video on the gripper i showed the there was a teleoperation and then there was also teleoperation and scooping up scooping up the fish so all of that teleoperation is just a positional command so there the operator did not have a duplicate gripper to serve as like a haptic glove of and so that's actually work that that's that's ongoing right now so we're we're currently trying to design a system to allow someone to wear a duplicate copy of the gripper so that that so that we can have a basically a force reflection coupling between the gripper and the the avatar hand so in terms of the the performance that could be achieved actually in the in the electrically coupled version you actually can get better performance than the the direct physical teleoperation and the reason is that for example when Jesse the barber was shaving me he's manipulating one end and that's going through the transmission and then through the hydraulic line and then through another transmission actuator so there's there's there's two actuators with the fluid in between and so any amount of imperfection or friction in the actuators you go through that twice but in an autonomous system if you can measure the pressure in the hydraulic line that that basically allows a measurement of the of the contact forces where you can skip one of the actuators so you actually the the measured pressure is better than the sense of touch that an operator would have in the passive system so that's yeah that's we have an area of research on how do we make best use of measured forces and measured haptic information in the in the overall control loop but yeah really interesting I wouldn't almost wouldn't have expected that but it makes sense that there's a lot of different hardware the things that it's moving through actually transmitting that signal as it were in more physical sense versus just a singular more more simplistic although seemingly complex channel yeah if you if you remember the example with the the two syringes like the the the school experiment you're pushing on a syringe and you feel the friction of that syringe seal and then the pressure in the high in the water line and then it's pushing the second syringe so yeah you feel it twice in that case have you guys decided what the actual form of the avatar robot will be yet you've got all the parts going there I just wondered what you're going to do for the final I'll take this question on how about that as as as you mentioned I we think that we have the foundation of blocks to put it together and we've been waiting for the final scenarios to be announced so that we can analyze them and and you know decide where to go and now that they are out as of last month you know to answer your question in short no we did not settle on okay here it is yeah however we have I think in our inventory we have quite a lot of platforms that will serve well for the for the announced scenarios for the semi-finals but then eventually you know we are converging as as the team mentioned on integrating all these different little pieces together to have the whole system set up yeah I think it's great that you're you're trying to put that human experience into the the whole robotic system and I I love the little the demo you showed with the two eyes and and trying to get the persona of the operator in there even with just two camera eyes I think that you know you can do a lot and it doesn't have to be you know fully formed humanoid robot it can do as long as it can connect to two people together that that's what we're going for that's why we are known as the institute for experiential robotics to enrich to enrich human robot experiences thanks tashkin I just had a quick question coming through the q&a from anonymous attendee and it's quick why not full hydrostatic I don't know who exactly to direct that question to but anyone can feel free to dive in yeah I can take that so that's that's a great question I mean we we do have projects where we're envisioning building a fully hydrostatically actuated system sort of a trade-off between complexity and performance and another way to think about it is that if you think about from your fingertip to your hand to your wrist to your elbow to your shoulder the the closer you get up towards the body up towards the shoulder you know the more you need your joints to be stronger and and more precise and the closer you get out towards the fingertips the more you need the system to be able to behave compliantly and to be able to measure forces so out at the fingertips out in the hand the very low friction hydrostatic driven remotely driven system I think is definitely the best option I mean if you if you tried to build a direct drive motorized gripper that's going to be incredibly heavy to have those those big motors out there where the mass penalty is is the worst you know the farther away you are from from the shoulder so I yeah there's a there's a range of solutions which could involve you know maybe having hydrostatic hand and wrist and then having a more traditional motorized upper upper body and another thing is actually the exoskeleton design that we're working on it's actually fully motorized but but each joint is direct drive and it's and it's actually designed not to exert the full torque that the avatar is capable of it it has a force scaling because what we found is for the human operator they don't nest the the the interaction forces you don't necessarily need to be able to feel the full magnitude of it but it's it's about changes in forces you know if you're touching something or if you're sliding along and you're feeling friction it's those it's those force vibrations that are really giving you that that tactile and proprioceptive feedback so yeah so again in in terms of making things simpler you know not having motors and motor control and a hydrostatic transmission and all of the the complexity of that i'm just having a simple lightweight electric system that's not capable of full strength but it's capable of full haptic quality is the direction that we're we're going towards and so that system doesn't have a gripper at the end and so the that is is slated to be hydrostatically actuated got it yeah carl or anybody anything to add about the the exoskeleton system itself and where things are going next as far as how you're developing it next stages of testing other things to that are going to be integrated with that yeah so i think professor ready to explain quite well there the plan's been forward i mean this it's a back to travel system so the user can't experience like haptics on it by himself that is one of our next steps um we've taken kind of like a complete 180 from our our previous direction which was a custom developed pcb using a ti chip and now we're moving on to using a more open source on the solution called oDrive which is one of one of our sponsors now as well and it's an amazing uh so it's a single board that allows it performs like all the pd tuning um for your robotic system and allows you to very easily create a system that can be back driven so we're currently using you might have seen uh the taped board onto the arm there i was working with that's our solution now it's going to change very soon so we have a mechanical mounting option but what we're trying to now is move that onto a footprint that can be put onto the back of the um the current motors we have so currently we have these pcbs that on your designs that only read off the encoder value and then report that back to the oDrive once we can pull the chips off the oDrive and make our own pcb we can get them to much smaller form factor and this is going to let us move forward to actually like back drive these motors um and give the user an idea of like a true haptic feedback and make them feel like they're really in that situation so that's where we're going on this year i think end of this year our goal is back drive the um dexel skeleton to give the user that true like haptic feedback i just have to say that the fish demo and haptic feedback together just boggles my mind i can't imagine really feeling fish not that you're doing that but uh it was something i'd never thought about um how something could handle an object that has such a strange texture well i also really appreciated the quote-unquote high-speed seafood handling as a so i think that maybe there's there's more of a future there's fishing is a big industry and you know northeast um what so you mentioned you talked about shaving as an as an ultimate task for the use case of this avatar i'm curious um maybe albert or mike or ania if there are other use cases that are sort of inspired um your development or the way that you've applied your knowledge to this challenge maybe i'll go to ania if there's any thoughts that you have on now what's a use case that that that really has driven the work that you've done um i don't know if anyone else has anything specific in mind i can jump in if you want yeah go ahead michael well i think one well in terms of very specifics i think the obvious one that jumps to a lot of people's mind is surgery um there's only so many high level surgeons in this world and obviously you know travel being a thing is um limits the access of care for a lot of people so if you can get a telepresencing robot to the accuracy and responsiveness needed for some kind of complicated surgery that could save a lot of people's lives and additionally there's also like search and rescue situations anything where the human life would normally be in danger and you require a human level of interactivity those things need to be done through telepresence thing in the future and so i don't know if that's specific enough but that's sort of the things that drive us and why we're doing this and um i think just to add to that mike um for me at least the public school system has been instrumental in getting me to where i am and and proceeding to where i'm going and i think that in the current state of affairs a lot of school is being shut down and things like that people are looking for a solution as to how teachers are going to interact and successfully teach their students and how coach is going to interact and successfully teach and coach their athletes and i think that this avatar system um if worked on and done correctly could really be instrumental in and being the solution there imagine students um being in a classroom or like separated in different classrooms and instead of having much teachers having to come in you could have robotic coaches maybe in every single home um just the the opportunities are endless so yeah and uh i had one last thing to add um following on what what albert said and and it reminded me of something jackie said earlier about the emotional connection you know even if all you have is sort of the um you know two eyes and a and a pen and a tilt in a yaw um you're sort of able able to get kind of an emotional connection um i have a very short video um if there's time here i'm going to try to share my screen this is actually a a clip this is from back in disney days but this is this is actually trying to focus on that emotional connection as sort of the the the primary focus in a robotic system so i think it actually doesn't take a it doesn't take a a really human mimetic system to necessarily be able to convey a lot of emotion and so one of the things that um you know interested me at disney and that and that interests um me about avatar is is the ability to have an enhanced emotional connection with someone you know in in ways that just with a sort of a flat screen and no physical interaction is is missing and i think that you know you don't need uh sort of future level technology to actually get there i think that um there's really interesting things that can be done with with today's technology yeah it's just the eyes and the and the mouth and the tilt of the head and the movements it's we don't really care what your skin looks like there's something subtle too about even the the color of the the nubs at the end of the ineffectors that kind of gives it some brightness as well and then even just a little bit of that fluid motion of waving beckoning and saying hello is is definitely a way to convey some emotional connection even though it would it's really just a a mechanical being right in front of you yeah i appreciate you showing that it's a really great tie-in to connection especially to what albert mentioned about having a having or conveying presence in a way that doesn't um doesn't limit the experience that that maybe students would have teachers would have and really important roles that require connections we are just coming over the time um if you was wanted to learn more about the work that you're doing uh whether it's related to the avatar enterprise or just at the institute for experiential robotics where could they go is there the best website any more videos that anything they might be able to see about your work um yeah i would say um searching the news archives on northeastern's website um there's several stories about uh past projects that are related to this work and uh yeah i think that's that's a good a good place to go yeah the the institute itself is actually just being stood up so i don't know that we have a um a fancy website that has a an archive of all of the projects related to affiliated faculty um but we're working on it great yeah well look forward to when that's uh standing tall with more work about the avatar x prize and more to come from from your team as you continue to develop your work so i want to thank you all we are out of time for today's session i appreciate everybody taking the time to join the session whether you were a panelist or an attendee or viewing the recording later on thanks for jumping in so on behalf of the avatar x prize we really appreciate um having tashkin peter carl albert ania and michael as well as thanking all the other members of team northeastern for your contributions to the team and your hard work on the avatar x prize um it is really a bit of pleasure to speak with you today hearing more about your approach to the competition and learning how all of your studies and work are coming together to integrate into this challenge so everyone this has been the fifth in a series of meet the teams webinar interviews you can view the previous meet the team sessions and other innovation driven content by visiting x prizes youtube channel if you have questions about the avatar x prize or you may want to have questions more for team northeastern you can email us at avatar at x prize dot org and you can visit avatar dot x prize dot org for more information about our competition and to see the full list of qualified teams so we are wishing you well from los angeles and hope you're staying safe and healthy and i hope you enjoy the rest of your mornings afternoons and your evenings take care


This is a talk about performance and user experience. My name is Davey Shaffer. I'm the release manager for 1871, which is kind of cool. It's also a three-year commitment, which I did not realize when I signed up. I have a release that's got to go out today, so I'm going to work on that at least. I'm at Davey Shaffer on Twitter, so I'd like to put it first. If you have questions, please reach out, and I have my phone here, maybe I'll see you. Which probably grew up as the world's largest CDN. We also do some really cool security and other products as well. Create APIs to check us out at developer.acmi.com. And this may be a little bit too much, but in every single topic, I always went to the problem, which I co-organized with PGA. So Justine did not give me URL, which is image.com.org. This is really important, so please do check it out. All right, so let's get started. Woo! All right, so I'll try to do that slide, because you might get them to come back. So first of all, why performance matters? So essentially, there have been a number of studies about this. One of the most important ones probably is Amazon. So Amazon loses a 1% of their annual revenue per 100 milliseconds in their cycle slot. So what that means is for 2015, they would have lost a billion dollars. And to put that into perspective, that's 15 cents for every single person on the planet, or $3.32 per US citizen. And that's per 100 milliseconds. Google, some of you have probably heard of, they found that if they were half a second slower, they'd got 20% less searches. Bing, when they were two seconds slower, they got 4.3% less revenue per user. GQ, popular magazine, they found that when they caught load time by 80%, traffic went up by 80%, and time spent on the site went up by 32%. Some other case studies really quickly. Walmart got one second improvement, like 2% increase in conversions at sea. They found that by adding 160 kilobytes additional imagery, they had a 12% increase in bounce rate, so that people coming in and immediately leaving. And statements, 1% improvement led to a 10% increase in conversions. So what we can conclude from this is that users could receive tiny changes in performance. And that's actually as little as 14 milliseconds, which is infinitely small. And slower sites make customers less happy. Less happy is a poor user experience. Pretty easy question. Now, this slide will not make much sense, but if my slides had been coming up, you would have noticed that every single slide was 100 milliseconds slower to render than the one before. And it's a really interesting thing when you kind of click onto that, like sort of 400, 500 milliseconds, then you're like, wait a second, something's not right. So there's an old sort of maxim, which is 10 seconds. That's what you have. 10 seconds to render everything, and if you don't get it, your users are going to go away. That's no longer true. It's very far from true. It's actually more like 4 seconds these days. And just yesterday, somebody's slide is totally up to date, Google released a study that showed that you have about 3 seconds, but more than 53% of your users on mobile will just go 3 seconds. And imagine numbers that turn out to be. So who is responsible for performance? The answer is everybody. Everybody is responsible for performance. It does not matter if you are a designer, sysadmins, devops, back-end developers, front-end developers, browser vendors, everybody has a role to play when it comes to web performance. So hoping this is where it's likely to be really necessary. So modern web pages are getting bulkier and bulkier. And what we're finding these days is I have a beautiful bar down here, like it's just gorgeous. 63% of web pages today are images. 17% is JavaScript. We have 3% is HTML and CSS. Also 3% fonts are 5% and then other, whatever that is, 90%. But 63% is images. So anything you do to make your images more better is great. Page weight also is going on quite dramatically. So while 60% of sites are under 2 metrics which is still quite large, we have like 8%, 3 to 4 megabytes in size. Did anyone see there was an article like, like in the New York Times or something, it was a one-word article with that headline. And it was something like, when you receive a CC to the entire company, should you respond and be asked to be removed out of the CC? And the article, the whole article was no. And it was a 6 megabyte page. Additionally, the number of requests per page is going on hugely. Actually, one more thing that will happen. The average page size now is larger than new, the original new. So it's over one and a half minutes. So the number of requests per page is also unknown. So this is a request for sub-resources, images, JavaScript, CSS, etc. Where you have 18% of sites now loading over 15 resources. And even on the very high standard, more than 150, you have 15% of them. It's huge. So let's talk about how we can fix some of this. So as I said, everybody has a role to play in this. So are there any designers here? I don't know, at least one, just to speak on the road there. Number one thing, compress your images. That sounds silly, but if you don't, we're going to do it for you. And we don't know necessarily what you were going for in terms of quality. So be careful with that. Pressure if it is or we will do it. Is this Agents, Dead Opsi folks? Anybody? Every few? Okay, so number one thing for you, please enable HTTP2. It's potentially an incredible performance when it's completely transparent. Just go turn it on. Amazon now supports it. Acquire obviously supports it. Just go for it. But along with that now, all browsers require SSL or TLIs actually. So check out Let's Encrypt.org. This is one of the slides that did not convert revenue. Let's Encrypt.org is a fantastic certificate authority. It's completely free. It's transparent. It's open and it's automated. So you can get these fantastic 90 days certificates that work in every single operating system. Go back to HTTP, the older IDs, every single browser. They're completely free. As I said, they're automated. You can track and rotate them every 90 days. Something else that you can do is you can use the cloud to scale based on the perceived performance. So typically when we scale, we're looking at metrics like load, memory, CPUs, network utilization. One metric that we don't use, and I think we can use to further inform our scaling decisions is perceived performance. So there's something called a speed index port, which is from the WebHTest site. And there's a node module called marsildurant slash webHTest API. You can use this to go ahead and grab the perceived performance, the speed index of your site at the current time. So everyone can link that. Try and follow through this. So basically it's really just simple JavaScript node stuff. So you pull in the module, WebHTest, great to go away. You're then going to go ahead and instantiate that, and you're going to ask to run the test. And so here I'm running it again, stateshap.com, my site. I tell them I want the page speed, and then you have this callback. And what that callback will receive is the URL at which the results will end up at when they're done. So this is not a quick test. You could spin up your own WebHTest instance, and then you're not sharing resources, so that's an option for this. But essentially you get this URL back. And then I have this check response call that's going to go ahead and pull that URL till I get an actual response. Let's see if I can make something else a shot. So basically I have a check response callback. It does a git request on that URL that was pulled, where my results are going to be. It will continue to retrieve the data until it has it also. The way the JavaScript works is there's a data event. So as data is streaming back, you will get that. You just sign it somewhere and then when it's complete, you can then run more. So I have this on hand, which is the end of it. This is when we've got the entire response. And we very simply here just say parse the JSON, this is all JSON APIs. If we get 200, yes, then we finally finish our, the whole response is now there. They've actually finished the test. And we can actually go ahead and get both the first view, which is an uncached view of our website, speed index, and also a repeat view. So it actually adds the ability to catch the resources that are accountable and give you a speed index with that in mind. So you can take that information and make decisions to scale on. So what you're seeing here is basically the test story in every five seconds of people until it gets a response. In this case, I have the two speed index shown. All right, so back-end developers who would consider themselves a back-end developer. Okay, and I know all of you full-stacked people like you can just do all of this, right? So this might seem obvious, but cache aggressive. There's many levels of which we can catch. So native-based caching, which is something like Memcache or Cassandra. Full-page caching using something like Memcache or Cassandra again, but also varnished squid. There's also something really cool about the next slide, including which I had not run into before, and if you don't know it, they're also known as ESIs. They were created by Acline and some other companies I don't care about. And they are partially supported in varnished squid and other open-source products. So I'm not trying to sell you things. I'm trying to make you clear this is available through open-source stuff. And what these are are special HTML tags that you can put in your output that you can then have different caching requirements around. So you can say this fragment has a different TTL. This fragment does not get cached at all. You can do it per user session.  You know some things like geographic area and stuff like that, which is really cool. So the way that these work is you have two tags. You have the ESI include tag, which is at the top here. So they're XML tags, really, I guess. ESI include source and then whatever you want it to actually include. So this is what your varnished or squid is actually going to pull into the software. But this is the cache resource with whatever settings that we have on it. And then you also have the option to have an ESI removed. And what this is, is a more static version that is less memory-intensive, potentially generic. And that will be removed if the include is successful. So here what I have is I'm including a greeting and it's going to be personalized.  So nice follow back there. And that will work with any language that's all up on the caching area. Alright, so let's talk about images. As I mentioned, images are a primary source of paintwork. Anything you can do to improve your images is going to have the most drastic effect on the performance of your site. So the first thing is formatting your changes. So, typically, you will look at an image and go, alright, it's kind of a photo. So, copy it to the J&M, or it's a small number of colors, it's blocked, maybe we'll look at if, or jiff, and paint, or pfg. There are no new formats. So there's WebPaint, which is supported by Chrome. And there's JPEG XR, which is supported by the latest IE and Edge. And these are closer to JPEG and that they are full-colored formats. But they've got much better compression, potentially much better compression. The same thing where you test, like as a PNG24 versus the JPEG, you can see which one's smaller. You can do the similar things here. So what you can do is you can make changes on the server side, compare the results, and decide, okay, this smaller is WebP, so for Chrome, I will serve them a WebP image instead of the original JPEG. So to follow along with that is just compression. So, who's familiar with the JPEG level? Zero, 1200. Who knows what they mean? They don't need anything. So they don't need anything. It's actually what, so it's completely down to the compression, as to what they mean. So, if you have different tools, 60 is not 60 everywhere. So we ran into an issue just this week where LibGD, which is one of the image fibers that PHP uses and WebPageTest also uses, their 60 is different to our media that we use on the Apple App. You have the same image quality, the same size, but 60 is not even right. So JPEG levels are kind of useful. You just say, yeah, we just have generic 60, 80, whatever, we'll just pass everything through that, we're okay with that, but you get decent images that are okay size. But there is a better way. So let's talk about perceived quality. So as I mentioned earlier, as a designer, if you want high fidelity in your images, you should be responsible for compressing. But I'm going to cheat a little bit and say as developers, we actually have a great way to compress in a way that this idea of perceived quality. So perceived quality is quality that humans can see. It's not how computers can derive what humans can see. So nobody can see that, but it's an amazing image of a lady standing in her own starting. And the trunk here is like a terrible renderer. So I had an image that was originally 16.8x, and it was just huge. To make it look terrible, I had to resize it down to 50 pixels, because I literally did the JPEG slider a little bit underneath. So I'm like, okay, resize it down to 50 pixels, and then resize it back up to 5,000, which is completely bracket. But using those JPEG numbers, the original is 16.8x, 90 immediately drops to 5.8x, 84.2x, and I go all the way down to 0 here, which is 393 kilobytes. And what you can't see but I can, is that there is very little difference between the two images. Very closely, especially on a display like this, as opposed to like that, you can see a little JPEG harder acting in some of the more detailed parts of the image, and maybe as a designer, you wouldn't be happy with that. But maybe not. So what can we do to actually compress in a way that is doing it in such a way that it is about perceived quality and not about just size? So how can we programmatically what number we can get to as far as that compression level before we start to get degradation in that image? So there's a really cool algorithm called SSIN. It is actually an Emmy award-winning algorithm. I don't much like to go out on the stage to accept a statue, but this is the truth. It won an Emmy, and it was originally developed for like video compression, so individual frames. So what the SSIN algorithm does is it is not only is not only determining three different actions, brightness, contrast, and structuring, which is a pre-visual part of an image that we assume is perceived quality to be around. And most importantly, SSIN is fast at doing it. But what we can do is we can use DSSIN, which is a way to measure the distance between two SSIN scores and determine how much degradation is happening in reception.  So with the SSIN tool, it is possible to render these awesome eight maps that you can actually see visually. You know like the FLIR, X, Y, R and the red and the digital, like yellow is hotter, etc. You can actually see that kind of information where the degradation is happening. That's useful for humans. But what we can get out of it is these scores, so these DSSIN scores. So where we went from 16.something next down to 5.something next for that 90% or 90 compression ratio, we get a DSSIN score of 0.09. So that's how far it is from the original source. What we can do is we can set a perceptive quality threshold. We can say that we are okay with that image becoming within a certain distance of the original. So for example, we decide that our threshold is going to be 0.2 or 0.0 to a sorry, that we can compress it all to different levels. Until we get a compression level that is within that threshold but no more. So at the original working point, I'm sorry, and then at 30 is where we get 0.017. As soon as we get on to 20, it's 0.2, what was it? 0.20, so we're over the threshold. So we can make this decision. And this is actually a discussion that you can have with your designer. You can say, look, let me know when you start seeing degradation that you're unhappy with and we will set the threshold. So there is an open source tool called SSIN, there's many implications of it. Some of them suck, some of them set fault. But they are out there, go and check them out. But long and short of it is in this particular example, we save over 90% of the size without losing any perceptively visual inquiry. Say what? I cannot believe this part of the image but it should be if it's not. Because it's kind of awesome. If you didn't, I mean, I should gather like really anyone here in the building. I'm glad that there's a Wikipedia page, that's where that came from because I know what it means. So moving on from the back end, I have a few people here. I have a few. Everyone has a multiply. So for front-end people we have what's known as predictive browsing. More commonly called pre-browsing. And pre-browsing is a really cool technique which is really new. And what it allows you to do is to preemptively perform some of the necessary parts of the connection for subsequent navigation. So there's four main parts of pre-browsing. There is DNS pre-fetching which means, you know, let's go ahead and fetch the DNS entry group and give it your up. Let's reconnect to a server. And I just sent a request which is to open up a TCP height. There's pre-fetching which is getting the data and then finally there's pre-rendering which is actually rendering the requested data off-screen as if it were visible in the browser and it navigates that. So if you've seen any sites that do like full-page transitions and like immediately the next page loads that's how that's happening. So let's see, I have pretty diagrams of all this work. So if you have a Chrome browser or any browser the first one is DNS pre-fetching. This is the first part of making your request. So when you go out to the DNS servers you go ahead and resolve the DNS. We then open up TCP connection to reconnect to the server pre-loading, pull that data down we just hold it in memory, we don't do anything with it and then finally there is that pre-rendering where we run it off-screen so that's kind of a whole new request. And these things are all really actually simple to do. So they're all using link text in your head. So here we have link, rel, DNS, pre-fetch pre-fetch, href, equals and then there's no problem just slash, slash and then you know what I mean there's no problem with the DNS so just look up the example where we get the DNS entry for it The next one then is pre-rendering it's pre-connect So here we now specify the protocol because now we're actually out of making an HTTPS so we need to know where we're going to open up a TCP connection so very similar except that it's well equals pre-fetch pre-fetch well equals pre-fetch this time you can use a relative URL because it's in their own page and then finally pre-render which is basically the same word now I want to caution you, awesome image if you use pre-render you remember earlier those pretty bar charts and stuff that have to happen to render a page those still have to happen and if you're still doing stuff then you may be running into resource intentions trying to render something else so that's something to bear in mind you do like JavaScript to add the pre-render after your existing page to render for example to kind of solve some of that just be cautious especially on mobile because you could be loading a lot of tables On top of pre-routing there is a new standard that is being work on which is locally known as rel equals pre-load and it's similar to pre-fetch so the difference between pre-fetch and pre-load is all the pre-routing stuff is to do with subsequent navigation so you're saying you're coming to the home page for a site that 90% of the people are going to click on excellent next so we're going to do all pre-routing for that the pre-load however is to do with the current navigation so one of the things about it is subsequent navigation and optional pre-load is meant to be a management thing and it's a high priority and it's for the current navigation so you can say I want to pre-load my CSS or my fonts and typically it sent this ahead rather than the link because of that link slash resource 70 home and rel equals pre-load if you request a site typically the browser has to pull down the HTML parse through it, find the CSS add by the JavaScript add, etc then the CSS has some resources to respond to the CSS and has to part through the CSS with link pre-load what you can do is just say based off the thing you're going to need and you can start fetching those during those parsing pages and by the time it eats them you're going to have your own cache and that's really cool when you have really high bytes so with all of these things we start to look at new architectures and essentially if you take nothing away from this talk if the basics we want sucks and the reason it sucks is we don't have to come up with a number of tools that are quite frankly genius but also they're hacks so minifying and concatenating JavaScript and CSS to minimize the number of requests that are being made we also do inlining small projects from CSS called critical path so we will take just the amount of CSS and JavaScript necessary to render what's above the fold what's visible on the page and we'll put that in the HTML so there's no suffering last year there's a problem with that there's an outside of that which is that it's in every page which are paid the main page and 80 URIs which are also paid we also do things like domain chart so all browsers have a limit of the number of concurrent connections they can make to the same domain most browsers is 6, i.e. is 8 but that's not many we're now loading 150 resources so we'll chart, we'll do s1, s2, just write the load across the main even if it's the same servers in the background so as i mentioned all of these things are hacks they're really clever hacks but they aren't so with htt2 we have something called multiplexing which is this really cool idea of using one connection for many arrow over buttons and along with that we also get server push which is very much like link preloading except that the server is just saying here's the error you can also do things like set dependencies between resources so you can say if you are now loading this i also want you to now load this but not until it's done so if you're now loading the css i also want you to now load the fonts but get the css first and we can also assign links which is a great way to actually say the priority of the three things you're sending and then between the browser and the server you'll negotiate and the common resources you get to use things so with htt2 we get this fantastic priority tree which is still being worked on for the browser but we have this idea that like all of these different things that come together to determine how pages deliver across the wires that let's just send everything out at once and see what happens so when we look at delivering a page now yes what we can do now is things like this so we have a browser and a server we send an http request and we get a response we can now also instead of inlining our critical path we can do server push-ins with separate files which are individually casual so we know whether they need it the same way as we do with any request they can just say I've already got this don't send it to me again we've got the last mod up we also send like our loader that's really important during the rendering we can send them and force those down we can also say well in sending my critical path but we also want the fonts because we have headers that are shown at the top of the page so let's make that a dependency when you come downloading the CSS grab the fonts also grab the rest of the JavaScript because they're probably going to scroll at some point we really hope they're going to scroll and then finally we follow along with the last CSS like the majority of our CSS the minor part so this can get really really interesting now I want to point out that this is all new this is an idea this is not something that is I'm not saying that this is the best practice like I wanted to get there everybody is discovering all of this stuff right now so you have a role to play in this you're interested in performance you should be looking at these types and these different architectures and checking it out so in summary performance matters time is money we know that it's user satisfaction so we can say performance is your experience and all of that is money so I'm at Shaffi on Twitter I am bshaffi.com the slides will be on the big shaffi.com slash slides and I would love to have questions thank you so the question was how do you fight the battle between shipping new features and adjusting for performance performance is money so while features are also money like we know that losing money is the opposite of making money and bad performance is losing money so it's juggling out every single thing has as much priority as shipping features and you just make the trade-off as you do between different features like I don't think it's a step for concern it's in the same place alright thank you very much


Good I'm 840 here. So I just watched this 2019 documentary from from Hulu. It's called jawline and It's about this 16 year old kid trying to make it in the world of live streaming So his name is Austin tester and he has about 20,000 followers and He's trying to make it big time and he's also trying to escape from poverty at home dysfunction at home from being bullied at school And he projects this Like sweetness naivete He's good-looking. He's got all this energy and he presents this real positive message telling people to follow their dreams And he's got this enormous following among teenage girls Not enormous. I mean you'll have four or five six people watching him in live And when he does a live meetup there about five girls who show up so He's struggling to digitize his digital footprint and he's making about $60 a day and The thing that struck me was not so much Austin Who was was struggling to make it as a live streamer? What struck me was the the pain among his followers and And the remark was it's not normal happy people aren't watching live streams It's people who are troubled people who are miserable people who are being bullied people who feel disconnected People whose reality is awful. And so they want to distract themselves by a live stream and Here's a comment from the director of this documentary I think everybody in that world of live broadcasting is escaping some sort of situation and so it's not just the The personalities in live streaming who are trying to escape a situation. It seems to be overwhelmingly the audience Trying to escape misery that we've developed like these two personalities According to the documentary director, there's the for the teenagers There's the personality they display at school and then there's the more idealized fantasize self that they display online and She thinks this is no but we've always displayed different personalities because we there's no There's no like true Luke. There's no true self because who we are Changes depending on the situation. I'm gonna speak and think differently in Sydney, Australia than in Los Angeles, California I'm gonna think and speak differently when I'm talking to a fellow orthodox Jew as opposed to talking to say a Sakela Jew as opposed to talking to a religious Christian as opposed to talking to someone who's not at all religious I'm going to speak differently to a Hindu from India then I am to Say an indigenous Mexican American so We're always changing depending on the context in which we're speaking acting operating and thinking So that's why going to a new place.  It's called jawline. It's on who Lou came out in 2019 or so check out the rescue Great documentary from National Geographic about the rescue of those Indonesian soccer players who got trapped trapped in caves when when the water came flooding in


Hello, you're watching The Daily Roundup by People's Dispatch, where we bring you some of the top stories from around the world. Let's take a look at today's headlines. Colombian Army leaders confessed to civilian killings. Horn of Africa faces severe drought and hunger. U.S. court halts execution of Melissa Lucio and four soldiers killed in Israeli attack on Syria. In our first story, 10 former members of the Colombian military have publicly admitted their role in the extrajudicial killings of civilians. Between 2007 and 2008, 120 people were murdered in and around the town of Ocana in the Catatumbo region. These were later branded as combat killings of left-wing guerrilla fighters. A historic public hearing on the incident was organized by the Special Jurisdiction for Peace or JEP on April 26. Around 50 of the victims' family members attended the meeting in the Norte de Santander department. A former military general, one civilian, four colonels and five army members admitted their participation in abducting and killing the civilians. The JEP was set up under the 2016 peace accord between the now demobilized revolutionary armed forces of Colombia or the FARC and the government. It found that the military carried out over 6,400 extrajudicial killings between 2002 and 2008. This coincided with an incentives program launched under President Alvaro Uribe. Soldiers with higher combat kills were given rewards including vacations and promotions. The JEP is examining atrocities committed over 50 years of armed conflict in Colombia. It stated on April 22 that over 5,700 people had been killed or disappeared between 1984 and 2016. These were linked to military-backed paramilitaries in a campaign against the FARC-linked Petriotic Union or UP. The tribunal noted that 4,616 victims were murdered whilst 1,117 were forcibly disappeared. Out of this, 538 people had no links to the UP whatsoever. Moving on, over 15 million people are facing acute food insecurity amid a severe drought in the Horn of Africa. Areas in Kenya, Somalia and Ethiopia are facing their driest conditions in more than 40 years. The region is at risk of facing a fourth consecutive season of failed or poor reign. The United Nations has warned that this could lead to one of the worst climate-induced emergencies in its history. Some areas in Somalia are now at risk of famine. Nearly 6 million people are acutely food insecure including 81,000 people in the IPC's catastrophic level. Between 5.5 to 6.5 million people in Ethiopia and 3.5 million in Kenya are also facing drought-related acute food insecurity. Across all three countries, around 5.7 million children are acutely malnourished. Over 1.7 million out of this are suffering already from severe acute malnutrition. Factors including low harvest and global rise in prices has made access to food more difficult. The price of a food basket has increased by 66% in Ethiopia and 36% in Somalia. In some of Somalia's worst affected areas, the price of water has surged by a shocking 72%. Water shortages have also increased the risk of disease and infection. Factors including lack of food, water and pasture has triggered mass displacement. In Somalia, over 759,000 people have been forced to leave their homes since January. This figure is projected to reach 1.4 million in the next 6 months. In our next story, a U.S. court has issued a stay in the execution of 53-year-old Melissa Lucio, the Hispanic mother of 14, was convicted for the murder of her 2-year-old daughter in 2007. Her case drew widespread condemnation given the gross violation of established procedures and rights during her interrogation and trial. Lucio and her family maintained that her daughter had fallen down a flight of stairs while they were moving houses. Her daughter Maria suffered from a physical disability which made her walk unsteadily. She died 2 days later. Immediately after the incident, Lucio was taken into police custody. She was interrogated for over 5 hours during which she asserted her innocence over 100 times. According to reports, police officers deployed coercive methods including intimidation. After undergoing this manipulation, Lucio made statements that were used to incriminate her. She was charged and sentenced for fatally beating her daughter. Lucio's lawyers argued that her supposed confession had been coerced. They also highlighted her vulnerability given her history of being subjected to domestic violence and sexual assault. Lucio was also not allowed to present evidence which challenged the validity of her confession. They added that false and unscientific evidence had misled the jury to believe that Maria's injuries could have only been caused by abuse and not the fault. Information related to her disability was reportedly also omitted. The Texas Court of Criminal Appeals issued a stay just 2 days before Lucio was to be executed on April 27. The case will head back to a lower court which has been ordered to review 4 claims raised by her lawyers. And for our final story today, at least 4 Syrian soldiers were killed in an Israeli missile attack near Damascus on April 27. The Syrian Defense Ministry confirmed that 3 others had also been injured. The Syrian Arab News Agency or SANA reported that the missiles had been launched from the Israeli city of Tiberias. It added that most of the projectiles had been intercepted by Syrian air defenses. Venice Day's attacks also caused some material damage. Meanwhile, the Syrian Observatory for Human Rights has reported 9 fatalities. It also stated that an ammunition depot was among the reported targets. The missile attack took place a day after an Israeli military drone crashed in Syrian territory. The military stated that it was investigating the incident. However, it has not commented on Venice Day's attack. Israel reportedly also struck several targets near Damascus on April 14. The missiles were allegedly fired from the occupied Syrian Golan Heights. Israel has carried out hundreds of such attacks on Syria since 2011. It claims to be targeting positions of Iranian-backed militias including Lebanon's Hezbollah. Israel has also stated that Iran's position near the northern border is a quote-unquote red line. It often launches these unprovoked attacks by making illegal use of Lebanon's airspace.