a pan udych chi yn y debyg o abusell. Felly mae efallai. Beth bod yn oes bywch maith? Efallai, iawn, mae'r e rumannau dyna fydd yn ôl y dyma. Mae'r ddechrau? Mae'n ddefnyddio eu zeithloeth oher Danien. Mae efallai maith am unig? Mae'r ddechrau i ddyn nhw? Mae'r ddechrau i ddim yn dda, allan. Felly, mae'n gwneud hynny bod yn dweud am y maesol. Dwi'n credu ymddangos. Dwi'n credu ymddangos o bodi'n credu. Felly, mae'r methu sy'n ddweud, i'n ffodol? Mae'r methu yn ddweud. Dwi'n credu, mae'n ddweud. Dwi'n credu. Dwi'n credu. Mae'n credu eisiau yn ddweud. Dwi'n credu. Mae'n credu eisiau yn ddweud. Rwy'n credu'n credu. I don't think we'll get that out anyway so... That's it. I'm proud for you to keep me in the roof. Keep me in the roof too. Keep me in the down. Cool. Yes, I forgot the roof slips. Oh dear, I've just killed. Let's see if it comes back. No, it's back. Yeah, is it still recording? Okay, super. Haven't you recorded the camera yet? What's that? Haven't you recorded the camera yet? So I did and then I looked for playback and... So now it's on recording. A few minutes to go, I suppose, five minutes? I'm trying to get out of the way. I'm filming. Oh yeah, that's all. Sorry? Yes, yes. I'm John. John, nice to meet you. Nice to meet you. I love you. I love you too. You belong to... Yeah, go for it. So it's just starting soon. It's an example. Yeah, yeah. I don't know how far it is. Do you know how far it is? I don't know how far it is. Okay, so it's all recording. I'm below. So we have 60 minutes to go. I want to use the spoon. I'll sit on the other side. Can you have a look at those? Yes. Goli, can you sit in the centre? Hi, I'm David. I'm learning. David, you want to put it over there? I'm on your back. He's going to have the door to the back. There is line to breath. Listen to me. He's getting too nervous. I can't turn around. Mi'n hyffordd yr wrth gwrs. Mae'r twfawr yn gwybod, mae'r gwirionedd yn Llyfrgell Llyfrgell, rydw i'n ddim yn ddwybod i chi, mae'n gofynio'n ddim yn gweithio'r gweithio. Rydw i'n gweld iddo yn gweld i'r mwy o'r Llyfrgell. Mae'r cyhoeddych yn ysgolig. Mae'r llei'r Ysgolig yw'r ysgolig Ysgolig yw'n gweld i'r ysgolig, With Patrick, I've done a couple of ten-minute shows last year and this is the first full-length play. This play was developed at the Huntington Theatre in Boston. This is another chance for Patrick to look at the play and see how he's going to develop it further. It's important to know what people actually think. If you don't tell me or tell Patrick what the feedback is, we'll never know. Thank you so much for coming and I hope you enjoy the play. Sitting here has the time to kill. I see. You are travelling. Oh, business. Back to Houston. What is your business? I work for NASA. I'm an astronaut. With the space ships? Exactly. You go up into space? Just once. I used to be on the shuttle. Payload specialists before they shut the program down. And the stars do they look different from up there? You've never imagined a sky so dark and so full of light. There are so many stars and they're so bright. It feels like you could just reach out and touch them. But you can't because there's so much glass between you and them. On the other side, it's cold and empty. There's not a soul for all of eternity. It sounds very beautiful and very lonely. Sometimes it is. I would like to see the stars like that one day, inshallah. Something. Would you like a coffee? No, thank you. I'm sorry. My name is Malik. This is not to disturb you. I'm just waiting for my flight. I am bringing you a coffee. No, no, please, please, please. I bring you a coffee just the way you like it. Two sugars, two creams. This is the way you like it, yes? How do you know that? Tori, a Dunkin Donuts. She knew. That is where you get your coffee, yes? This is also where I get my coffee. I do not like the Starbucks. Lots of people buy coffee at Dunkin Donuts. You must be confusing me with someone else. Tonight, you like to get coffee sometimes. You do not sleep very much, I think. Two sugars, two creams. For me, this is just the start of the day. I don't know you. A cup of coffee. You don't know me? No, in friendship. I'm just passing through. I'm getting training here at MIT. That's why you see me. I'm a frequent flyer, waiting for a flyer. An astronaut, the stars. You are not going to him. I have my bag. I'm here at the airport. I'm ready to go.  You are here more.  But you never go. I have a flight. Maybe you use an old boarding pass, or perhaps you print it out on a computer. You are hiding, perhaps, in a place where you can be seen but not seen. I know what it is like to be invisible, but you have a choice. I think it is my uniform, my skin, my English. The travelers here, they don't see you. The TSA, they don't see you. But Toria, Duncan, Donuts. Carlos at the newsstand, they see you. I see you. You remind me of someone. And I wonder to myself, what is she doing here? Is she okay? Are you okay? I'm fine. What is your name? Sarah. It's nice to meet you, Sarah. No one is supposed to know me yet. No one knows me here. It's okay. I'm only bringing you a book. Airports are the safest places on earth. No, no, no, no. It is still the same. It's not easy to find a place where you don't have to live. If I end up talking to someone, I can be anyone. I don't have to be me. Last week I was a sculptor. You can be anyone you want. I do not mean to disturb you too. Quiet. Here. It's a place that's no place. That's what people think. That there is no here, here, but they are all. I love this place. People are on their best behaviour. No one shouts or screams or hits. You are still safe. I love that airports have chapels. I go there sometimes. I go there too. Once I rode the bus between the terminals all day blending in. It was like I vanished from the face of the earth. It was perfect. Someone like you should not disappear. I was fine. I was fine. What am I going to do? No, no, no, no. I will go. I do not mean to ruin anything for you. I know what it is like to want to be away to one's silence. I have made a grave mistake. You must stay. I will go. I will see you. I hope you will forgive me. Is your flight? No, I am leaving from another gate. I just had some time to kill. I see you are travelling on business, yes? Back to fitness. I am a heart surgeon. That is very impressive. Thanks. I look young, but I have had a lot of training. I am sure you are very skillful with the knife. If you ever had a heart problem you could come and see me. It is a very serious business to have a man's heart in your hand. I can't let myself think about it. Even for a second when I am actually in surgery. If I look down and see this heart, this limp and useless muscle in my hands and think if I don't get this working in the next ten minutes he is going to die. Then I freeze. In the operating room there is no time for second thoughts. No, no. When someone is gone, they are gone forever. Yes. It's nothing. Next time, but leave by the coffee. No, no. This I do. Dori at Duncan don't know she gives me two for the price of one. It's a very American thing. It's two for the price of one, but I like it. It's okay, right? Can a heart surgeon have double chocolate coffee? I've noticed you are staying here very late sometimes. Even more than before. It's not always safe to travel home alone late at night. I don't know how to say this to you. You are laughing, but your eyes, they are very tired. I don't know what to do, Millie. My neighbors fight so loudly and viciously. They have this baby and the walls are so thin. Stella, the baby, she cries and shrieks while they are going at it. I can't be the only one who hears them, but no one does anything. They know that this man likes to hurt people. They are afraid. To them fight like animals. Terry and Fran. When we passed in the hall the way he looks at me, I can tell. He knows I can hear them. A man like this can be very dangerous. Fran and I, we see each other in the lobby. We don't chat. We barely even say hello. I'm careful not to look too closely, but I can see the bruises. If you get too close, then you will be in danger. But the baby, I should do something. I should call someone, but I don't know who the police are. What would you do? To protect a child. Mother, yes, I would want to do something, but Sarah, what can you do? You must not get too close. I'm empowered. I should call. No. When I hear Stella crying, I can't help you thinking. You have a good heart, Sarah. You want to do the right thing. I don't know if I can take one more night of it. Maybe you don't have to. There is a place here, a closet, small, but it is big enough. There is a cot inside. No one goes down there, down past the doors. There is a place you could sleep. No, I couldn't. It is quiet. Away from everything, it is safe. You would be rested to not answer now. But think about it, because sleep is important. You must rest. If you have rest, then you will know what to do. It's tempting. To get to the room, to get to that part of the airport, there is a code, a secret code, a number. I will give it to you, but you must promise you cannot tell anyone when you got it. It is a very secure secret. I don't want to get you into trouble, really. No, no, don't trouble. Don't worry. You should not worry. Don't think about it. What are you making? I made a hat. Very skillful. It's pretty horrible. This is for a boyfriend, yes? No, no. Sorry. I don't need that complication in my life right now. That is good. Simple is very good. Do you have a big head? You have to start over. At least I know it fits. This is for me. You've been so kind. It's cold out. You need a hat. But this one is awful. I'm not sure where I went wrong. But you made it for me. I'll make another, a better one. You like this one? It's horrible. Please, please. Give it to me. I'll tell you make a new one. I will have this one. No one has ever made me a hat before. The next one will be better. You will think about what I said about the coat and the coat. Yes, you need peace and sleep mentally. I will. I followed you from your job. What are you doing here? I'm going somewhere. I have a flight to catch. Oh, you're flying? Please. Do you want to see family? There's no way they let you out already. I have to play football with the DA. Please, don't touch me. Do you think I want to hurt you? Have I ever even said so much as boo to you? Last night, Frenny and me, things just got out of hand. What do you want from me? Social services took Stella. I always said no kid in mind would ever end up with those assholes.  Stella is my little girl. And now she's gone. Because of you. I can't help you. You're the only one who can. Talk to them. Frenny won't press charges, but they won't listen to us. All part of a pattern they keep saying, but that's not the way it is. Frenny and me, we're just human. That's all. She understands I've never hurt Stella. Everybody knows I'm a good father. I've seen Frenny's bruises. Well, she gives as good as she gets. No, wait, please. I'll beg you. Talk to them. Tell them. Help us get our daughter back.   You don't have to like me. I see the way you look at us. You think you're better than me? I don't care. I'm not. A little girl needs her parents. Sometimes life is hard and your parents aren't there for you when you need them. Well, just tell them you made a mistake. Tell them you know you love our daughter and that we never do anything for her to. You never saw her bruise on Stella, did you? I've seen Frenny. I hear through the walls. Frenny's back home now, and I'm looking after her. She'll never forgive you without Stella. Maybe you shouldn't be forgiven.  Frenny won't stop crying. It's like a river of tears that's sprouted from her eyes and just keeps flowing and flowing. She's a mother.  I was trying to help. Well, you make things worse. But you can fix it. Can't you see? I can't. I can't fix anything. Not for you, not for me. You have to, Earl. Can you get back for your trip? You listen again to those walls and you listen to the sound of a mother's heart breaking. Then you tell her. Then you tell her why you won't help her get her daughter back. I'm not responsible for your life. You stuck your nose in where it don't belong. Now you're part of it. I'm not.  When you get back, I'll be waiting for you right here. And you will help me. Here in Phoenix, another shot to be on that fly. You look familiar. Free from fly. I have a lot. Wish we got free from fly mouth for this job.  What's that? New gadget for the coves on these boarding passes. Oh, I printed mine at home. You can get a lot of that lately. It says this one's no good. I have a flight to catch. Please, I just need to get through. You got the wrong code. You're going to solve all that problem. Please, I'll miss my flight. I'll go back to the ticket counter. I'll mail straight it out. I can't go back out there. Sorry. It's been a hard day. Last night, my neighbors and the people I worked for there were so horrible. The world was so... I just need to get through to where I belong. I just need to go. Miss Sarah, just calm down now. No crime. That does not work on me. I can't stop thinking about her. Baby is so fragile. Oh, I know it's hard to leave her. Impossible.  Yeah. He's grown. They never stop dropping you crazy, believe me. Did you, daughter? I understand, but... Please, please just let me... I don't think this ticket is fulfilling, I think. I've got close to a photographic memory.  I'm sure you're very intelligent. That's why they made me a super boss. I remember things, the tech. I missed my other flight. My boss called, and now he wants me to meet a whole other different client. I still end up missing your daughter. What? Oh, yeah. Well, there's no way around that. I'm a mother. Naughty. You have a picture. No. Not with me. I don't carry them on. I left them on code. Right. Yeah, I didn't think so. Pete, come over here and take over from me. Yo, Pete! Pete, code 829. 829. If you could just come this way, please, Sarah. This is a real ticket. I just bought it. Come with me, please. Where are we going? Sarah, come this way, please. But I really need to just... Don't make me ask again. Once it please goes away, it gets locked up. I'm on the table. I'm scared. Have you ever traveled outside of the country? No. What are these? They're knitting me. What's in here? I didn't. Oh, it's half the dinner when you fly. Food is very expensive. Why is your ticket a one-way ticket? I don't know when I'm coming back. Are you in a hurry to get out of town? I have meetings. A trip.  New clients all that. What kind of company is this? A mattress company. I do graphic design. Well, no, I meet with hotel chains and advise them on mattresses and hotels buy a lot of mattresses. My company... The agent something they should not be. How do you mean?  Your company, the things they do, the people you work for. Is it drugs? What? Raise your arms. Do they sell drugs? No, just mattresses. Do they put drugs in mattresses? No, there's no drugs. They're rude, cruel. They say things that people shouldn't say. Do they make you do things you shouldn't do? I'm just trying to do my job. I'm going to quit. But you're still flying for... I need to go to my gate. I really need to go. Why? Have I done something wrong? Have you? No. No. Did you check any load? No. I don't have much. I benefit. You lie to me, sir. I need to get my gate. Lying to a federal security officer is a crime. I bought a ticket. Punishable with a fine. Ask at the counter. I was just there. It was all the money I had. There's another room where you will have to take off all of your clothes. If you ever lie to me again, you will go to that room. And it will be a long time before you come out. Why are you doing this to me? Aren't you a mother? You asked me just like that. And the tears. Let me ask you. Are you a mother, Sarah? Are you? That's what I thought. Don't lie to me.  Am I clear? I'm sorry. Do we understand each other? Yes. I understand. Okay then. We're done. Have not lost. Not the security line. You heard? In this place I hear everything. I see everything. You met Madam Moon. She can be very funny, but she is also very hard. Fears. Maybe that is the word. Fears.  Something has happened. I made the call. Last night I had to do something I had to. All night Terry and Fran were screaming at each other. The banging and the crashing over and over and over again. I picked up the phone. Should I call? What will Terry do to me if I call? He'll know it was me. All night dishes breaking, doors slamming. Are there fingers in the jam? The phone was in my hand and the baby was crying. Screaming. So I called. What's your emergency then? How do they ask? What do I know for sure? I don't see. I just hear but it's enough. Why did I wait so long? That's what they should have asked me. You should not go back. The cops let him go. Just like that. He found me at the entrance to the terminal. I knew if I could just get through security. Frannie's already gone back to him. He knows it was me. I'm safe here but I'll never get through security again.  He's out there right now waiting for me. He knows where I live, where I work. Don't go back. Not to the job, not to the apartment. I went to work after being up all night and they were on my case.  I tried to explain but they mocked me. Useful what happened is a joke poking at me all day long. So I quit. Yet another failure and a string of failures. They were hurting you. How can I go back to any of it? Sometimes you cannot go back. Sometimes the door closes and you are not meant to open it again. I just wanted to... The baby kept crying. I just wanted to protect her to find what's best for her. I know, I know. Where will I go? We will find you somewhere safe. I need a gate where people will be stranded all night where I can blend in. There are blizzards in the Midwest. What's the gate from Minneapolis? I know some place better. Remember that room I told you about? The one that got inside where it is still there? I don't know. No, no. Do you think you are safe in your apartment? No. You will be safe here. No one will bother you. It is forgotten. You really think it's okay? As long as you are on this side of security, there is nowhere you can be more safe. Sarah, you are in danger. And now it is my turn to help you. We have a safe place for you. You will come with me. I will give you the code. It is very peaceful there. You will be safe. So far from the heart of the building. In the terminal and airport seems like a quiet place. It's an illusion. You will soon get used to the sound. What's this? Something from a long time ago. It is the hand of Fatima. Maybe it will watch over you. It is okay? Sure, it's not. This isn't... Am I taking your bag? No, not now. There was a time where I used to sleep here. But now I have my own apartment. It is not much bigger than this. Are you sure this is alright? Yes, yes. You should not worry too much about other things. This is okay. When you need a shower, you can go to the admirals club. I worked there that night. I have the code. No one will see you. I don't want to get you into trouble. I make a great many things around smoothly in this airport. You would not think so. People look at me. They say, oh, there is Malik with his broom. He is nothing. It's not what I think. There are people in this world who would like to be seen. I am not one of those people. I have been here a long time. And there are some things that I know on this room is one of those things. I changed this door when I was sleeping here. There is one lock on the outside. I have the only key. There is a copy. And there is this lock on the inside. When you turn it, no one can come in from the outside. No one will disturb you. There is something I need from my apartment. It's small, but it's... Sometimes the small things are the most important. But if you leave, this man is waiting for you. And Marlene, she knows you now. You will never be able to return. We will find a way of getting you things. Maybe I can go and get them for you. It's too much to ask. You are not asking. I am offering.  And I will get your things in the morning. It will all be okay. I don't know what to say. It's been a long time since... This is not just for me. Not just for you. It is for me. It will make me feel better to know that you are safe. Marlene, people like you. People know you. You're friends. You do a good job. These days that doesn't describe fair men's pain. But that's you. Good only. My English is not so good. But I work hard, yes. Do you know why I got promoted this week, Marlene? No, but congratulations. I got promoted because I do a good job too. My job is to find things that are not right. To catch lives. And I can be friendly about it. Like you. You clean things. You're friendly about it. My job is just like yours. My job is to keep the fill off the plane. To keep gun baggage from sneaking on guns and bombs and box covers. My son, Jimmy, he's in Afghanistan right now because someone like me wasn't doing her job on September 11. Right here in Miss Air Force. You got kids, Lly? For 10 years, I am an American. I have my papers. That's okay. I only see those right now. Maybe later. But before you were an American, what were you? I lived in Algeria, and this was many years ago. You lived there with your wife, your children, maybe a daughter. That's what I think. A daughter. She'd be wrong now. When's the last time you saw her, a wife? I know what it's like, Jimmy's been gone a year. And that's hard, but for you maybe it's been a lot longer. A daughter of a family with care. A young woman, not too flashy. I'll bet you spray.  Lly? Pussy. I found it. It was a lost bank. I was just returning it to the baggage claim. Good Samaritan. You were doing someone a favor. Some of the guys here, they said, oh, he's stealing because they're suspicious.  But they don't trust anybody. That's what they get paid for. But it weighs effort too. For me, it's trying to figure out who to trust. Why do I want to trust you, Malay? I am not a thief. That's the truth, isn't it? I checked all the records. Do you know how many bags are lost in our airport every day? Not as many as you think. Doesn't fit most, but they're all reported. Because people value their things. So I checked all the reports. All of them. And this is not a missing bag. This is not a lost bag. It is not a stolen bag. Yes, this is what I tell them, and I am not a thief. No, you are not. Yes, thank you. Can I go now? But if you're not a thief, what are you? I'm a janitor. I clean. It's just clothes and shoes and a hairbrush and toothpaste. Yes. Why? Where are you going? Where are you taking? I do not understand.  Your English is almost as good as mine. Where is she? Your daughter. My daughter. That's who it is, right? Now she's wild, the style's too young. Now she's hiding out, sneaking into the country to see her long lost papa.  Is she here? Who? On her way somewhere else. You're smuggling her into the country. Where's your daughter, Malay? I do not know. Where is your daughter? I cannot see. You tell me. I will make you tell me. You will regret. See the light of day again if you don't tell me. I swear to God. Where is your daughter? Stop. Where is she? You want to help her. You can't help her. You can't hide her. You tell me where she is and maybe it won't be so bad. But you gotta tell me where is your daughter. Dad was seven years old. I was here, they were there. I did not want to be a soldier anymore but I could not do the things they asked me to do. So I left. I was trying to find a way to bring them over but they were in a war and their fathers are not there to bury them. I wish you were alive. So do I. Look. You don't make a mistake. I am not a thief. I didn't think you were. I never thought. I think you should let me go now. I have work to do. There are places that need to be cleaned. Yeah, okay. You go back to work. I'll keep this to remind me. You're telling me the truth but you're not telling me what I want to know and I don't like that leaf. I don't like that at all. I'm sorry. I'll be watching you. Are you all right? Yes. I am thinking of people and places. People I lost a long time ago. And it still hurts? Yes. When I look at you, it helps. You are the one helping me. I don't. I'm glad. You are bright. Like a candle. Fragil. Help me. I'm tougher than you think. I've been thrown up. Maybe we're not so different. Maybe not. Did you have any luck at my apartment? I have failed you. What? Marlene, she has taken the suitcase with your face. She took it? She brought me to our office. She had many questions. What did you tell her? I told her nothing about you. You are safe but I have lost your face. They're gone? Yes, she has the suitcase with everything. She has the envelope? No, no, I could not find it. It should have been on the nightstand on the milk crate by the bed. It should have been right there. I looked hard. A white envelope. There were some photos in it. You're sure it wasn't there? I did not see it. It had to be there. It was there. Marlene does not have it. Just to close. How could not be there? How? It has to be. It is hard not to have your things, but Marlene does not know she does not know that you are here. I'll have to go and look for it. No, no, no. I have seen this man, this Harry at your apartment building. I have seen him also here in the airport. You saw him here? This morning. Yes. Outside the terminal. Pacing like a jaguar. He must be some way on he get past him. What did his wife look like after he had finished meeting her?  And he loves his wife. He does not love you. He thinks you are taking his daughter to lose a daughter. She's the end of the world. And he blames you. Such a man will stop at nothing. I did the right thing. He did. And now you must let me help you a little longer. Do not rush away, not now. I will try again. I will look harder. This is my mistake. For some mistakes, there is never a remedy. But this I can do. You will let me fix this mistake. Yes, I will get you some proof. And then tomorrow as soon as I can, I will go back to your apartment. I will find your envelope. I will look harder. Something that I can't afford to lose. I understand. But I must be careful that he does not see me. And you must be patient. Can you be patient, please? I'll try. I will. You won't be fine. See that plane? All the runway? Taking off? I see. Yes. Watch it. There it goes. Keep watching. All the way up. All the way they go so fast. Stay with it. Focus on the light. Stay with it. It's still there. And now it's gone. Like a star that disappears. Even after I can't see it? I know it's still there. It's not visible. Sorry. I remember you, Sarah.  That was a mistake. I'm sorry. You don't belong here. You don't have tape. I did. You saw it. So you should have left. You belong days ago. But here you are. I'm just on my way back. Connecting. I'm leaving. If you lie to yourself, it's easy to leave. You lie to everybody else. Don't worry. You don't need to run. It's not today. I'm not on duty. I'm not bothering anyone. I promise. I just need to stay here a little longer. Until what? Until the world outside disappears. That was your hope before. Wait. Oh, I'm sure. Does it work? What? I'm here. Honestly. Maybe I don't see it because the world seems completely covered in bloodshed and darkness right now. But if it works, I need to know. It used to? I mean, hell, I can read. I can knit. Drink coffee. Watch the world go by. I can do that. Couldn't I just do that? You could try it.  I don't know. Don't lie to me. No, she doesn't. Call her. Whatever story it takes, but tell her something because not knowing is hard. I mean, parents understand our kids will grow up and eventually grow up and live their lives in faraway places. I'm glad I raised Jim with me and man with sense of responsibility. I'm glad he signed up. People say the word denial is a bad thing. You deny the idea that they might never come back because if you don't, what kind of life is that? My mother doesn't care where I am. Shit. Of course she cares.  Do you know what that means? That's right.  You don't know anything about me.   I will be spitty next. I will break things.   Take some time to do what. Listen to the echoes of my empty house. No one can actually get any better when you're there in the ground. That's where they stick.  I've had three days. I can't take one more. Sorry. Tell me, screw you, Marlene. I'm staying right here. Go ahead. But don't you dare tell me. I'm so sorry for you. I'm so sorry for you. I'm so sorry for you.  I'm staying right here. That's much better. My mother kicked me out when I was 19.    There are plenty of ways to mess up our kids. Whatever happened to Jimmy, it's not your fault. Don't try to be not to make it. Take it, please. You're going to miss your flight. You're testing me. Maybe that's it. This isn't your airport, Sarah. It's mine.  How about your son? Two pretzels and an orange juice. Regular pretzels or low salt? Regular. That'll be $6.79. We have mustard and honey, powdered sugar, ketchup. Who puts ketchup on a pretzel? Some people put it on everything, I guess. No, you. Maybe you're thinking of someone else. Sarah, the myth. That was not good. It was nothing. Don't think you changed anything. Okay. How are you? You might think you know me, but you don't. I know you though. Not even a little. What are you doing here? I work here. Since when? Not long. I don't like this. You don't have to like it, but I work here. You shouldn't be here. And you should? I work here. Didn't they tell you to take some time off? To collect yourself, to adjust? Where else am I done go? Don't. I can rip that down right out your throat. This is my place. You can't help. Well, it's a lot better if you're getting out there, isn't it? Where's your manager? He ran. He's here like once a day. We just serve pretzels and drinks. It's not exactly complicated. You should not be unsupervised. You're clearly a risk to passenger safety. Passenger safety? I need to talk to your manager. He's not here. He shouldn't just perish. People shouldn't just perish. He's just not here. He'll be back tomorrow. It's okay. Take a breath.  Not from you. I'll have to make a report. I'll talk to your manager and his boss and his boss. I'm not bothering anyone. Why? I'm harmless.  Please. Whatever you're looking for, you can't find him here. Who's your supervisor? I am a supervisor. Who's your boss? He should know you're not ready to come back to work. I am ready. He won't think so. You know what he'll say. Take a little more time, Marlene. He'll feel sorry for you just like the rest of them. I'm ready. You'll see it in his eyes. I'll find out his name and I'm going to give him a call. He wouldn't dare. He's probably one of our customers. I'll tell him you were down here harassing the pretzel girl. He won't believe you. Yes, he will. Because they don't trust you anymore. They're watching for cracks. They know you've changed, but they just aren't sure how. I can't stay in that house one more minute. I need to be here. Then we have something in common. From your apartment. I can see that. It is all right. It's more than all right. I want to make it...  It is still the same, but maybe it's still a bit different. It does. It is not. It's not. Thank you. I'm glad you like it. Did you find the envelope? It's not there. I don't understand. It's everywhere. No one could have looked harder than I looked for this envelope. I would have found it. I don't... It is gone. Gone. Even without them, you will remember. No matter how much you remember, you cannot change what happened. Even though your life has made you so sad. Sometimes I think I'd make a million different choices. That I'd go back to search out for the route at one moment, one decision, and yang it out. But no one can go that far back without ceasing to exist. That's not a trade I'd make. Seren, you are very wise for someone so good. Wise. Me. Look at me, pretzel girl. There is no one who looks at me and says, Oh, Derek Rosara, she's so wise. And there is no one who really sees you. But maybe tonight they will. I have something for you. What's this? I would like you to have it. I hope it's fake. Really? Will you wear it? Put it on. I will buy you some dinner. You will go to the bonfire. You will eat a real meal. Even though it is in the airport, it is a real restaurant. No more pizza and pretzels for just one night. I don't mind pizza. You are safe here. And this is good. You must stay here.  You will eat a little food. You will smell beautiful flowers. You will wear a beautiful dress. You will see that way you are. And who you are right now, right here. This is okay. Will I? And once you see this, the whole world, that will be open up for you. You make it very hard to say no. Is it working? Yes. Yes, you win. Let me try it on. It will fit. You will see. Shoot. I will wait for you. Marlene! Marlene! Marlene! Marlene! What? Are we old friends now? I'm surprised to see you down here. Not many people are down here. It is a long way from the... Yeah, not my world. Why are you here? I can go anywhere I want. Of course, yes. This is my airport. Yes, but it is late and you have... What's in there? Supplies. For my cards, I keep it locked. So no one can steal anything. No one can put dangerous things in there. It's always locked, secured. Very safe, very secure. I wasn't looking for you. I would not think so. There is no reason. Exactly. I'm a little lost. Oh, well, I can show you. I'm not lost. I know where I am. I am done filling my card, so I am leaving to clean. I work hard. I clean things. I fix things. Oh, no. I remember you. I remember you. I remember you. Why are you here? I was looking for you. You have found me. Now we can work together, yes? You know what? Down here. Deep down, there's a hum. Everything cut out except for the heartbeat. It is not always quiet, but it is very peaceful. Except the heart. You don't see it, but it keeps pumping. Hearts aren't meant to keep pumping. I should be. I expected it to be lonely down here, but I don't know. I think I like it. Some people it is okay. Do you get lonely when that happens? Is it better to just give in too? Sometimes. But looking into that pit, into the darkness, I'm afraid if I do, I'll never crawl out. Do you miss her? You're done. Of course you do.  I just...  And then... And then... And then it was gone. I shouldn't be here. I'm sorry. A long time. Maybe only now do I start to feel my heartbeat again. My son. I know what happened to your son. And is she your daughter? Still there in you. Still saying. She's there, yes, but not the same. I am not the same. For a long time, I've not let the world see how black I am. Inside, I stay good old in my league, but I am empty. And then one day a light shines, and I start to be closer on the inside to what people see on the outside. Every day this light in me includes. I saw a coffin going around, but they wouldn't let me see his body. They wouldn't let me see him up there, and I'll feed him down on the kitchen table. Been there a week now, but I can't bear to touch me trying to build a red white blue. That's my son. Your son is Sean. And you are broken, and nothing can put you back together again. Aren't you supposed to try and make me feel better? No. This is not why you come and look for me. I know not to try. To the world. The empty sound. I was never here. No, you never talk to me. I'm fine. Yes, of course, you are strong. I should go. Yes, you have much work to do, and so do I. A little turn around. It is very confusing down here. I will take you. I took her away. I couldn't get out. It was important that you did not come out by accident. She must not find you. I locked the outside lock. You locked me in. I needed to take her away, so she would not find you. And then what? You forgot about me? Of course not. I could never forget about you. Why didn't you come back? I was coming back, but then my boss he sees me. I must do a job for him and then another job and then another. I'll hurry as fast as I can. I worry that they are watching me. I cannot be locked in.  Yes. But I had to be careful. I came as fast as I could. You are safe. Don't lock me in here. Never. I will.  I understand. It was a mistake. Are we friends? Of course. Because I thought you were my friend. I thought all sorts of things. I am. Friends don't lock each other in tiny little closets. I know. And then they'll come back. Please. Come.  Please. Let that play. Let that play. This is crazy.  Look out here. Don't leave, please. Just do a game for now.  And don't follow me. Stop trying to help me to protect me. Just stop.    You have to take it. Neither do you. You can't get past security. I've got a cousin who works in the bag. Just leave me alone. If you want your daughter back, I don't have anything to do with that. Is that what you tell yourself? Granny was screaming. You were having a fight. I could hear. You don't know what you're saying. You've got a big imagination. The ambulance came. But she exaggerates. She was back on her feet in an hour. I've seen her bruise it. Ask yourself. Did you ever even talk to her about this? You think you did her a favour? You think she'll thank you? You put her daughter into foster care. No, you did. You did more harm to Franny than I ever could. I was helping. Didn't help Stella. I'll tell you that. And I never laid a hand on her. Tell that to her case worker. They won't listen to me. Franny tells them they won't listen to her either. Well, maybe she's better off. Or I'll reach my hand in there myself and yank it out. I'll scream.  We're both the outside. I can drive you to the office. We'll be front of the line in the morning. I'm not going anywhere with you. Franny left. She went back to live with her mother. We're still not around. She can't even better look at me. Like I'm some horrible, deformed creature. It's the tears. They make some sort of fun house lens. She can't see me straight. It's all distorted. I can't help you. You won't. I can't. I think you might. I think you know something about losing things. Important things. A landlord in me? You may know what your apartment is. Just to make sure nothing bad had happened to you. Those belong to me. Oh, I know. Look at him. Almost worn thin like you looked at him a million times. Please. So fragile. What was the name? Give them to me. Just a few hours old, yeah? In you. Even more like a child than you do now.  Yeah, I definitely see that. Darker skin though. The daddy black. A little jungle fever for me, Sarah. I don't mean anything to you. What did grandma and grandpa say to that, eh? I'm sorry. I'm sorry, Michael. I'm sorry. What happened to her? I made a choice.   You know how it feels. I do. And what am I supposed to do now? Suffer. Just like the rest of us. Or get your act together and win her back. At least you still have a chance. You think so? Maybe. Lyon. You think I got what I deserved? You think I'm scum? Maybe I am. Think I'll hold on to this. Don't do this. You come at me and let off this tomorrow. You walk in there and you say, I'm sorry I called. It was a huge mistake. Terry and Freddie are wonderful parents. You help me get my Stella back and I'll give you back what you owe us. No. I should have called months ago that I was a coward. I knew what would happen and you.   I'll finish you. I'll finish you right here. Then we'll find out how much you really love Stella because if you touch me, you will never see her again. For most of us, when we lose someone, they're gone for good. It doesn't have to be that way for you. Don't blow it. Drag behind me. Inside now. What about you? Time to finish our little talk, Sarah. You, the girl for the errand boy. I see you at the apartment. Go home.  What you want is not here. What I want is right behind that door. Go away.   I want your daughter. Oh, Sarah's got a big mouth. This will not bring her back. She bragged about what she did?  I'll tell you one more time. Get out of my way. No. Sarah! You can't have it. Give it to me now.  You will never open that door. I said, give me the...    I know what you have inside of you. I will finish you. I will finish both of you. You will not take it away from me. No, this time, please. You must stay just for a little while. Please, be calm. Are you okay? I'm okay. Where is Terry? He will not touch you. I can help you. I'm getting pleased to let me out. Soon, soon but not now. I am sorry you must wait. I will be back soon. With the leak? He could move a camp. Posed to. I saw people who were keeping secrets. Works here. Has been sneaking around. Someone gave out a security cover. Trying to blow something up. Or maybe smuggling something in. They smug a person. It's not your daughter, is she? No. Because your daughter is dead. Don't want to choose that time. But then I'll find Sarah. Are you pervert? Is that it? Never laid a hand on me. Not yet. Let's see. So happy.  Tell me what happened. Don't keep lying to me. That's not who you want to be. You want to be more than just Sarah in the locker. We all make mistakes.  You can't make me wonder. What is it about her? Why do I keep seeing her? You disappear. You're almost it. It doesn't have to make any sense to you. You don't have to lie anymore. I'm sorry if it offends you. That's it. You offend me. And you. You definitely offend me.  The world is a dangerous place. You notice. We try to keep them safe, and then we fail. If you had your son back, what would you do to protect him? Everything? Give me your keys and your badge. You don't work here anymore. You go quietly. No HR, not a word to anyone. Just go. Understand? I understand. I never see you in this airport again. I don't see you ported back to Algeria. I'm a medic. Good for you. Then I'll have you sent to an American jail. You will never see a son again. Come right now. He told you. So am I. If he hurt you, you can tell me. I can still find him for a little while. Whatever he did, we can. He tried to protect me. That's all. That's the way you want it. It's the truth. I don't think you know very much about the truth. What's that he gave you? Just some photos. Let me see. I can't give them to me now. This isn't mine. She's never really mine. Well, maybe she always will be. Something in her, something in you. Did she have any? I followed them after the final meeting. After I gave her away. I could barely walk, but I followed them. The airport, I couldn't let her go. I couldn't, and then they were all on their flight home. Home. All I could do was watch them go. I'm going to do now. Something else. Do you have some place to go? Not really, no. Because there's room in my house. In your house? It would be temporary, just until. It's very kind of you. No, I don't need to be here anymore. Well, that's fine. I don't see you again. And if I hear you whisper, something with a tweet, Facebook post, email or something, I'm just calling the newspaper. I know you won't. Get out. What is it that you do? Malik. Maybe you are a sculptor or a surgeon. You know what it means to hold a man's heart in his hand. I used to be a tempo, doing graphic design for a mattress company. I'm not sure what's next. There will be an astronaut. You have to see the stars. I brought you a coffee. Two sugars, two creams. That's the way you like it. You shouldn't be here. No, but this place is not so easy to leave. I needed to see you. Your hands are shaking. They are mine. And what they have done belongs to me. I'm not sorry that I called, but I am so, so sorry for what all of this. I am the one who needs to be forgiven by so many. Maybe someday you will give this to me. I already have. I should not have. I'll let you. But I wouldn't let it happen again. Not from anyone. Yes. Was this for your daughter? She'll be watching over you now. You should not worry about me. You are going to fly. I bought a real ticket. I am glad for you. I'm ready. You are ready for a thousand flights. Before it in the office, in Marlene's office, there are many hard things in life. Goodbye is the hardest. Yes. Sarah, have a good life. Stay safe. Blow old. All of these things I wish for you. For you I wish peace. Goodbye. It's called The Prince William Henry. But if you'd like to go, just go through and have a drink. We just need five minutes to clear up. Thank you very much.


So it's a great pleasure to have Julian here with me all the way from London. He happened to be the author of the first test automation book on mobile. I believe it was 2007-2008, Julian. Yep, roughly then. And Julian also happens to be the first automation engineer or the quality engineer, sorry, for Google outside US in Europe. He was the first person to set that practice up and then went on to do lots of amazing things. Recently, I've had the fortune to work with him at Geo where we were helping a bunch of teams with similar topic. I believe he's at the fag end of his PhD thesis on using analytics to improve quality of mobile apps. So, you know, it's again a great pleasure to have him directly talk on this topic. I think you guys will find this really useful. And so without much delay, I want to hand it over to Julian. Thank you very much, Nuresh. I look forward to helping people learn more about using analytics to improve the quality of their software. So let me start then by looking at the fact that we use analytics today, whether we realize it or not. So we use it in terms of test execution and test automation. Here are some screenshots from various tools working with Appium, for instance. And we can see graphs and percentages and colors and bars. All these are generated by some form of analytics. As users, we use analytics today. When we go to the App Store, we look at the ratings, the reviews, the scores. And we use this information to help us decide whether we're going to install the app on the left, the middle, or the right. And I suspect that not many of you will be installing the one in the middle, for instance, with an overall score of 1.8. So the next question then is, are analytics useful or can we test in isolation? So the first thing to look at is how do we measure our testing? Are we using internal metrics such as the number of tests we have, the tests we ran, the results of the tests we ran, and the bugs we reported? All that's well and good. But let's think also about the external metrics. So the discussions with the developers. How many of the bugs are accepted and actually fixed? Discussions with the customers or the representatives, such as the customer service team, and even discussions with sample customers. An old saying, which I think is still opposite to us, is those who cannot learn from history are doomed to repeat it. And this is attributed to George Santiana. And if we look from a testing perspective and we move up a level to the sort of systems thinking, then probably most of our focus is within the development team. And by that I mean the team including the testing, the project managers is essentially internal to ourselves. And yet we probably claim to do DevOps. But our views of production and the end users may be obscured from us. And the challenge then is to look at other ways that we can increase our vision and insight on the usage and behavior of our mobile apps. The Romans had a God called Janus and Janus looked both forward and back. And for many of us, we tend to look forward. We complete a sprint or a project or release and we move on to the next task, the next sprint, the next set of items. But yet we have to deal with bugs and issues that come from the past from our previous releases. And it's important to think how do we look both ways so we can learn from the past from the work we did and also learn from what's happening today so we can improve our work in future. I'm going to give you some examples now of applying mobile analytics to improve the qualities of mobile apps. The first one is the Wikipedia app known as Qix. I've been involved in this project for about seven years now. It's an application source and it's available through Google Play and it's also available for people to download in places where they can't use Google Play or it's impractical. For instance, in some countries where Wikipedia is banned, where Google is banned or where people get in trouble if they're using Wikipedia. So we reckon we've got millions of users out there, but we know we've got at least four or 500,000 active users of the applications. And two years ago they had a real problem though. The crash rate was as high as about 9% in some conditions for the main application. So we decided to see what we could do with analytics to deal with these problems. And we were able to reduce the crash rate by 38% within the first release. It took us 25 lines of code which included the comments to make those changes. And we were able to reduce the crash rate down to about 0.4% after six releases. And the development team continued to use these practices for well over 18 months afterwards. And all these are good indicators that we did something useful here. So here's an overall presentation showing the reduction in the crash rates. The purple line represents the application we picked to do the experiment on. And we deliberately left the rest of the applications unchanged until the third release. We wanted to see if the improvements were random chance or connected to something else in life apart from our work. What we did find is our work was effective. And then with the fourth release, we'd applied the improvements from the first experimental app to the rest of the code base. And we saw ongoing improvements. This was presented in something known as Android vitals, which is part of the Google Play Console. And Google Play Console is the developer's interface into Google Play. So as end users and our end users go into Google Play and they can download our application. So this is the internal sort of backdoor view of the system. And we can see the crash rate for this period, which was around summer of 2019. We can see that it started around 4.59% three months ago. It had dropped to 3.12. And as I mentioned, we got it down to under 0.4. Let's look at the complexity of the code. I'm hoping that many of you have seen code before, particularly given you're working with Appian. So we can see essentially that the red code is code that was deleted, the green code was code that's added. And this was all we really did. We looked through, we found that we'd missed a couple of conditions where nulls could be thrown. And we dealt with the null pointer exceptions. And these were two of the three top crashes that were causing the high crash rate in the application. Moving on to another application then. Here's an example of a much larger user base in the order of millions. And we can see here a number of things in the reports from Android vitals. The first thing is that we can see an orange line. And this is called a bad behavior threshold. And Google sets this site-wide. So for all the millions of applications, they all have the same bad behavior threshold. And for crashes, that's just over 1%, so 1.09%. For these things called A&Rs, which is when the application is not responding, or when the application freezes, it's around 0.47. And it looks pretty good. It looks like the application is doing quite well. It's well within the boundaries. Android vitals is great, and it provides platform-level analytics. And by that, it means it collects the data at the platform level, so at the Android level, across all the applications on end-users devices, if they're using Google Play, which several billion people do. But many of the projects also use some other form of analytics. One of the most popular is known as Crashlytics. And here's an example of another application where we're using Crashlytics to monitor the crash rates. We get much prettier graphs, by the way, than we do from Android vitals. And what we can see here is we did some similar improvements in two stages. We had a hackathon in early November, and the crash rate at that time was floating around a couple of percent. We were able to do an interim release on the 27th of November, which was released 66. And we made a second release on the 6th of January 2020, and that was released 67. And the top line shows very clearly how we're able to smooth and improve the reliability of the application. Academics have studied this topic before. This was a three-year-long study of two applications with millions of users. And they built in analytics, custom analytics, into these two applications and collected the data of billions of sessions to try and understand what was going wrong. As many of us know with our applications, we've got little idea just how many hundreds or thousands of device models are out there. So it's impractical for us to test on everything. In this case, they noticed that the session duration for users on Kindle Fire was about a third less than the average, by the average I'm talking about the median of all the users. And they're trying to work out what the heck it was, and they finally realized that the battery drain varied by three-fold across similar hardware specifications. And the developers ended up adding custom code for the Kindle Fire to reduce the brightness while the application was being used. It was a fairly small reduction in brightness, and it resulted in 40% less battery drain. And this meant then they noticed that the user sessions increased again, which is a good feedback loop. Here's an example which is published in an article on Medium and is from a fairly large company known as Bumble Tech now, and it owns a dating site called Badoo, which I've worked with in the past. And they actually looked at some open source code and usage analytics to work out why their users were complaining and having problems. I'll go through this fairly briefly, but I just want to walk you through some of the aspects of what they ended up doing. So they essentially wanted to understand where the problem was before it affected all their users. They had 500 million users globally, and they were using the WebRTC code for communications. They countered the number of tests, the lines of code in the test for the media codec, and these are across the Android versions. And one of the things we can notice here is that before version 4.3 of Android, they really didn't have any tests. And in versions 5 and 7 we can see the numbers grew considerably. And for this reason it's very safe to say that in Android 4.3 Google wasn't testing the compliance of devices with their specification for video encoding or decoding, and they only really started doing this with version 5 of Android. So what they noticed next was that it depended on the devices and the versions. So here's an example of some code that we might add to our application that gets information about the device details, the model, the market name, which is the one that customers see, rather the internal model details, and collect that. And this sort of information can be reported back using mobile analytics libraries such as Firebase. Firebase is the most popular library in the world. It's in roughly three quarters of all the Android apps. So for those of you using Android phones, it's running in the majority of the apps on your device, if not all of them. So this is an example where it can be used to report this information back. Now, Firebase actually collects most of this information automatically when the library initializes, so as developers that we don't need to do this, but nonetheless this is an example of the code. The next thing is they looked at the encoders on Android Lollipop, so in essence they're getting the codec details, and as we can see, they have a distribution, and this is drawn on a logarithmic scale, which means that we get very large variations across the scale of the graph, so from 1 to 10 to 100 to 1,000 to 10,000 examples. So we can see that the top one is by far the most popular the Q-com video encoder. So using all this information, then they worked out that many of the devices had a subset of the chipset, so the spread from the MediaTek, Huawei and Marvel chipsets, and they were able to make changes in their code to enable hardware encoding of the videos, and that gave a much better improvement for the end users. Let me give you a quick overview of mobile analytics. First of all, let's think from a developer's perspective of where do they get feedback from? So we have the developer in the middle of this picture, and they commit code to the code repository and other developers typically review that code, as pull requests, for instance, and they write comments. What did you intend when you were writing this bit of code? What about error handling? Those could to me, et cetera, and they provide feedback. They've probably got some sort of automated build and continuous integration, an automated test running with Appium, et cetera, and developers get that feedback. There may be some form of pre-launch testing, and if they're developing Android apps, they're supported in Google Player Standard, so it's fairly easy to do, and it has this concept of alpha testers and beta testers, and again, those can provide feedback. And finally, we release the apps to users globally, and they can provide feedback through the app store or through social media, or through getting in touch with us directly if they have that ability. Oops, sorry about that. And all these different sources of feedback are possible. So one of the challenges, then, is how do we get feedback based on usage, and here the gray or the silver dotted arrows indicate that we're getting digital feedback, and we also get it back from the users in the field. In terms of the feedback cycles, then, we've got a mobile application, and we've probably got test results, and this may be from interactive testing by humans, our automated tests using unit tests, tests using Appium, and whatever our favorite flavor of the month is for test automation. You may have Apple Tools, et cetera, et cetera. And all this is a fairly tight feedback cycle that goes round and round quickly, and we get the feedback within seconds, minutes, or hours, depending on the length of those tests and the complexity of the number of devices we're doing the testing on. We then get some sort of in app store feedback. For those of you with mobile apps, you'll realize that the amount of feedback varies tremendously. For some applications, we maybe get 5% of the users providing some sort of rating. For other ones, it's 1% or 0.1%. So perhaps as few as 1 in 1,000 users actually provide feedback, and they're the ones who've got something special to complain about or to praise us for. They tend not to be the run-of-the-mill users. So we get that sort of feedback, and we can apply it on our application. Again, we can have social media, Twitter, et cetera. In-app analytics provides us feedback from the population of usage, so from virtually all the users, with a few exceptions. And we can use this, again, as an additional source of feedback on our applications. In terms of our application, this is a sort of conceptual view of our application. We have the GUI, and whatever the user interface layers, which may be audio, et cetera. And these are the ones that the humans who use our application interact with. We have some sort of application logic, what's special to our application, and we have the operating system. And all of these have a different perspective on the application. Visual testing and heat maps, for example, work well on the GUI interactions. Mobile analytics, the sweet spot is really with the application logic, and the operating system can also provide us information about how the application is doing. And that's how Google collects the data that feeds into Google Play and Android vitals. The mobile apps need to first of all collect the data and send it. Without it, there is no mobile analytics. It goes to some sort of data collector. This is normally provided by a third party, such as Google or Flurry or Facebook or whoever is providing the analytics service. But we can host it in-house if we wish, and we're using appropriate tools. That then goes into a database. It's analyzed, that's where the analytics comes from, and typically presented in a user interface in a dashboard, such as Google Play Console or Firebase Analytics, et cetera. It's important to know that the steps may be delayed. For instance, the user may be in flight mode, and the data gets sent either when the application resarts or the network becomes available. I'll move on to some quick examples of analytics. We'll start at a very high level. This work was done by OpenSignal. It's from a few years ago, and what they were trying to do is understand the distribution of Android phone models that were using their application. Their application tested the network's strict signal strength, and with mission, they were able to collect this information from the users each time they used the application, along with some other details. And we can see in the left-hand chart that the most popular device at the time using the application was a Galaxy S3, and then it was a Galaxy S5 and then a Galaxy S4, et cetera, et cetera. And they had several thousand devices building up the matrix. They reordered the matrix by manufacturer, and now we can see on the right-hand side that the Samsung devices were by far the lion's share, around 40% of all the devices. And this is sort of information that may help us through analytics of usage to improve our testing. So for instance, we'd want to test on similar devices to the most popular ones. Google used to provide a dashboard that was updated regularly every week or so about the overall market stats for Android, including the screen sizes, the Android versions, et cetera. The dashboard still exists, and Google host it, although they took it offline for a few years, it's now back. But their strong recommendation for teams is rather than trying to seek global statistics, look on your per-app statistics. So it now redirects the development teams to Android vitals for this sort of data. So if you're not aware, then it's well worth asking your development team for access to Android vitals, and look at this sort of analytics, which helps us understand the Android releases, the mobile device models, the crash rates, et cetera, per device for our application. For those of you working with iOS, then there are similar analytics of the different capabilities of the devices, which helps us to understand our testing. Many of you will be familiar with Google Analytics. It's, again, running your web browser right now. Whatever software you're using is collecting information on your usage. This is from an Android application, and what we can see are the flows from the Android users coming in and the different web application pages that they're going to. And we can see the navigation, for instance, from home to join a meeting to participant panel, et cetera, et cetera. So a bit like what we're doing now, in fact, with this online conference. And by understanding these flows and the distribution of the flows and the red lines show the drop-off points where the users go away, we can again use this sort of information to help us improve our testing by modeling these flow paths. And furthermore, where we can see the large drop-offs, it may be worth us investigating why. Is there a problem in the application with the usability or reliability or performance that's causing the drop-off? If we use Android Vitals as part of our development and testing practices, a key stage is the release of the software. For small apps, we may just release the app straight out in the app store to all the user base. But for many of us, this is not practical or sensible. Particularly, we've got millions of users. So we release gradually, and Android Vitals automatically generates additional reports for us in the first seven days of release. And here's an example of that sort of report. And what we can see is the crashes per thousand devices on average, and then specifically for this release. And in this particular case, it looks like our new release isn't doing quite as well as the average across our user population. So it may be worth investigating what's going wrong with this release as it's going out, rather than just pushing ahead and releasing to the entire user base. It also provides similar information for the application not responding. And again, in this case, we can see that the crash rate is doing slightly better for the new release. But the A&R rate, the freezers, is much, much worse. So it's telling us something's gone wrong with this release, and perhaps it's an area in our testing that we didn't even consider. So it may be worth us looking back on our testing to work out ways we can improve the testing practices. And here's an example of some details of a particular application and looking at the crashes per day, per hour. So we can see this fairly typical chart of hourly usage. And we can see, unsurprisingly, that there are lulls perhaps in the middle of the night, for instance, when very few people are using this software, and then peaks perhaps in the office hours if it's a business application that helps us understand the worst cases. It gives us statistics on how often it's happened in the last 30 days, how often this crash has ever happened, the number of impacted users, etc. And when the crash last occurred. It also gives us a breakdown by app version, by Android version, and even by device. And we can drill down into any of these areas. And then it gives us the stack trace. And from the stack trace, the development team can work backwards and try and make sense of what the heck went wrong. Here's an example where Android Vital doesn't give us quite enough information to understand the problem. And this is typically where development teams will use additional in-app analytics, such as Firebase to collect the extra information and perhaps to collect breadcrumbs, which are little markers that are written that lead up to the crash and made available when the crash occurs. From experience, Microsoft App Center is superb at this sort of reporting. So well worth looking at these tools and not just taking the market leader necessarily. The next consideration then is heat maps. And here's just a quick example using the Wikipedia app for us. And we can see that there's examples of people sliding in the screen and zooming in and zooming out. And these colors indicate where the touch points are. We can look individually on an individual session and see how users interacted. We can also look in aggregate to understand patterns in the usage. So there are lots of useful piece of information we can learn from this, such as our users trying to click on areas that are not live, so dead spots on the application. And if so, it indicates a usability area that may want to choose and improve in the UX. And this is exactly the sort of area that is hard to test with standard test automation tools where we want to look at the visual side of it and also what the end users are doing. So we can look at lots of data, but the other challenge is to understand the causation or is it just correlation that's happening. And what these two words mean, correlation means that when one thing happens, another thing happens, but they may be caused by a third thing, or is one caused by another thing. So here's a quick example from some research which looks at the network and the power usage as a user's typing. And the typing may be doing things in automatic search, so it's sort of sending the search results back each time the user presses the key press, in which case this is appropriate and it is causation and a good behavior. Or it may be there's a bug in the software and it shouldn't be sending each individual key press to the server. So until we understand the use of the application in the context, we can't necessarily decide whether the behavior is correct or not. Let's move out a little bit from the details of the tools and try and understand what are we trying to achieve here. The first thing is we establish our measurements. This is the biggest circle and we understand what we need to measure. The metrics tell us how much. So things like crash rates, the number of tests that pass and fail, how long they take to run, our user base, whether it's growing or decreasing. We then need to understand why. Why are these things happening? Why is the user base increasing or decreasing? Why is the crash rate changing? Why are user journeys changing? And ultimately we're trying to get to the goals. We're trying to improve the software. At work of a couple of people from Microsoft Research, so Booth and Zimmerman from, believe it or not, 11 years ago were looking at how we can use software analytics broadly across our development practices. And they realize that we could look at the past, the present, what's happening now, and the future. And we use the information that we collect and we try to obtain insights. Many of us focus on what happened. We look back on the previous release. We may even have retrospectives for this purpose. But we don't necessarily work out what can we do better in future. So it's useful to consider the past to understand how well we did and what caused the problem. So if there was a crash in production when we made a new release, how and why did it happen? What are the patterns that led to that? What can we improve in our practice? When the application is actually being used we may realize that something gone wrong. Maybe one of our servers has gone down. I've got a problem and returning 400 errors or 500 errors. So we need to work out what to do next. So these are analytical questions that help us understand what to do right now. And then we look in future and we do modeling for instance to plan for the busy seasons. If we're in e-commerce, we know Black Friday is coming up. We know Christmas time is coming up. So we can use analytics to understand these things better. In terms of time scales then we can look back on trends across multiple releases. What we may not realize is that some of our users continue to use old releases not just for months but for years. With a Wikipedia application we have something like 30 different releases that are in use right now. Some of them we wrote probably four or five years ago. But Google Play Console collects the data and tells us this. This also helps us understand that when we fix problems and make a new release, that only ever reaches a subset of our user population. It doesn't ever reach 100%. Even if we try to force upgrades it doesn't always happen and sometimes all we do is upset our users with the force upgrades. Anyway, so we can think about the current release and how it's doing. We can work out what's happening in terms of production, so the DevOps side of it. And then we can move forward to what can we do today? What's the best thing we can spend our time on? And then what can we do in this sprint? And then looking further ahead, what can we do in future sprints? This may be where we're looking at different testing tools, different approaches and so on. So let's consider then on using analytics to improve our development and testing practices. I suspect that many of you struggle with deciding what to test and how much testing is enough. And perhaps the bigger question for us all is the best thing we can do right now to improve the process and or the product. So I argue that much of our test automation work is wasted effort. We do lots of testing, we do lots of manual testing, but is all of it useful to the project team? From experience, and this includes working at some of the top companies in the world, roughly 8% of all the reported bugs weren't addressed. This sounds kind of incredible, doesn't it? Every five bugs that we file, only one was addressed and the other four are eventually abandoned. So they cannot reproduce, will not fix or they put down at priority 4, 5 or 99, which basically means no one's ever going to fix them. And eventually someone cleans up the bug database and says, right, let's get rid of these old bugs and we start with a clean sheet again and the same thing happens in future. So how can we target our testing so there's less wasted effort and it's more fruitful? A couple of different areas. One is based on information and the other is based on insights. So by information, things like the popularity across devices. So for instance, if your app is used on Huawei devices, perhaps we want to be testing on Huawei devices to make sure we keep that user population happy with a new release. Is it based on locales? So are they using the app in Hindi, in Gujarati, in French, in German, etc., in Chinese, in Japanese? We can look at the user flows and all this helps inform our testing. And so for crashes, perhaps we can help reproduce the crashes by writing automated tests so that when we make changes to the code we can see if the crash fixes actually worked. And our goal really is fast reproduction of issues that are happening and firsthand learning in a controlled environment. And by that I mean that when something's going wrong in the field it's out of control. We don't control the usage the users do. So we want to be able to reproduce things locally so that we can understand them and address them. Testing based on insights then is looking at rates of change. So perhaps we've lost a whole bunch of users recently with a new release and we want to understand why we've lost them. It's when we cross thresholds so we're moving from one data center being enough to needing several data centers across a region is looking at anomalies that are cropping up and patterns. So we're trying to learn and be agile and respond quickly. One of the old questions that probably still affects many of your projects today is how many devices are enough for our testing. There's some academic papers looking at games and they realize that basically their games success was affected by the ratings and the reviews. So they decided well let's try and test across enough devices to cover the ones that users use when they're writing reviews about our application. So if the user is using a pixel 4a then we're going to test on that one because those are the ones who bothered to respond. So an interesting piece of work and what they realize is that the number of devices were different between free and paid apps. This was for Android applications and that something like eight devices was enough to give them 80% coverage. So good use of analytics for this. One of the benefits then of using analytics data is we're no longer limited to what the team believes we're actually getting real data from production across just about all the population. And there may be a couple of cases where the data isn't being sent either because users have opted out or because they're behind a network proxy or in flight mode etc. But other than that we're getting data from the population. We can reduce the cost of operations and use the actual usage from production to help driving our testing analysis and make our tests more real. Given this is the APM conference I think it's important to think what can we do with APM and APM 2 supports this concept of plugins. So potentially we can write a plugin that helps to drive our testing at scale. For instance to model the user journeys using analytics data. Interestingly I was looking at the documentation on GitHub today and it says go and check out the extension CLI docs and then it says to do. So I encourage the development team to address this to do and then we can look at how we can actually integrate data from Firebase, from Android vitals etc. into our testing. For all of us it's important to pick just the right amount of work that we can do to improve things. So something that's not too hard not too easy but just right to affect a meaningful improvement. So for some teams it's enough to fight the battle internally to get access to the analytics. The analytics is not available by default. You have to find the person responsible for the App Store account and say please can I have access to it and look at the crashes etc and look at the usage data and once you've got that information you can then start to use it. Similarly for Firebase and for the other tools that we're using for analytics. It may be that you already have access and you can get the data out so how can you drive a simple test based on the Android version. So write your first plugin based on that and thankfully the Appian project has a couple of sample plugins available for that purpose. I won't go through this quote in full. It's from a good friend of mine Joseph Reeve on changing behavior and one of the things he observes is by adopting a new tool such as mobile analytics the big improvement is the change in practice rather than a new tool. Because the tool gives us new insights into what is happening in production for instance and therefore we can change our practices without necessarily getting to a blame game in what was going wrong in the past. So the slides will be available and you're welcome to read the full details. So with that I'm about out of time so I'll thank you very much. You're very welcome to get in touch with me. I'm Julian Hardy at gmail.com in all sorts of places. So with that I'll hand over to the organizers and the Q&A session. Awesome, thanks Julian that was you surprised me, you finished much faster than I thought you would. How you were asleep were you? Couldn't afford to sleep while your talk is going on. Anyway, there are a few questions so we'll utilize the remaining time to answer those questions. I'd actually like to start with the question myself Julian if that's okay. Of course. So you talked about using analytics to figure out what are the most used devices and then prioritize testing against those devices. Now one of the classic challenges that I've faced myself is that when you do that you of course test the most commonly used ones but there could be potentially ones which could have been most widely used but there was an issue in that and hence that whole class of devices are not just getting used. So how do you address that part of the open ended stuff where you could have basically fixed an issue on some device and that could have been a top device now. So you've raised a really important point which is that silence is sometimes as informative as the signal. There's a very nice book called Silence on the Wire from Nostarge Press which discusses listening to the silencers and how they help us. So the silencers in terms of device analytics is where we look at the usage and we realize that a device isn't in the list. So let's say it's the Pixel 4a which is my new phone I'm a little behind the time so I've just bought one and it's a popular device so I can look at the analytics for the applications and say where's the Pixel 4a? Now I'd hope it'd be in the first 100 most popular devices. If it isn't then it indicates that perhaps something's wrong with our app on this device. So then we'd understand by taking the particular device and doing some testing is there a cause for this. If we looked at the example from BumbleTech they were finding problems with the different media codex so again they'd be losing users who use those models. So in essence we need a comparison because the analytics we get for our application isn't the picture of the world it's just the data that we receive. So we can get this information from various sources if we work with companies got lots of money we may be able to go to paid for services like AppAnnie who provide global analytics on usage. We can find some information from sitestatistica.com that provides some reanalytics on devices and usage and we're essentially doing a gap analysis. This is our data this is their data they say it's 5% of the population ours says 2. What's wrong between 5 and 2? Let's go and investigate. Similarly with Android versions Google used to publish a nice distribution dashboard that's no longer available unfortunately because that was a great reference point to say we'd see that a new version of Android would come out Android 11 etc and typically with an Android release it takes about two years to achieve the majority to be the most popular one so that helps us gives us some clues. So there's lots of information around and available to us and we need to compare against that. Now the other challenge is that even if we know it's a Pixel 4a do we even have one? So then we go to the companies who have device farms we ask friends etc who have those devices if they're willing to help and if we're really sophisticated we may have facilities to talk to our end users and say hey you've got a device that's interesting to us would you be willing to help us out and to do some testing and although that may be hard for some of us to do is a consideration that we can use. Alright thanks Julian can I request you to just stop your screen sharing so people can see what we're doing. I will do that. We have three questions here that I'm going to quickly go through sorry I have to pause and stop the share. Thank you. Alright thanks there's a question here which says do we have any automated way to test analytics test analytics are triggered from the app for all expected use cases and expected properties they want to make sure that the analytics itself is getting fired correctly. Brilliant question and what I meant to cover in my talk so thank you very much Vinith for asking that question. So I'll give you a couple of answers the first is there's a start up called iteratively it's now been acquired by a company called Amplitude because the market moves very quickly and they actually focused on ensuring that the analytics was called correctly so you could do the equivalent of unit testing as part of your CI to make sure that the analytics was being called correctly because perhaps someone's made a mistake in implementing the code and doesn't actually call the analytics that they intended to in the application that's the first choice the second is that if we're using tools such as Firebase App Center most of the in-app analytics they actually go through a getting started and typically at the end of the getting started go and test this, go and test to crash go and test this, go and test that so we do have ways to do interactive testing and the third way and it's not complete but it's open source and it's on GitHub is that I've written some code to help test analytics along with Joseph Reeve the chap I gave the quote from at the end of my talk so those are the three different sources what that doesn't really seem to be is much to test the analytics at scale and by that I mean that if the analytics says there's a thousand users using your app right now is it really a thousand or is it only 900 or is it 10 million and that's an area that I'd love to do some more work in and hoping to do that ideally once I finish my PhD right, thanks Julian we'll go to the next question that is from Sham Maler does Google publish crash rates of app as free to get info for app which I'm testing another good question and this the short answer is it is free if your app is in Google Play they don't charge anymore for this so anyone who has access to the data can use it free of charge the challenge typically is finding the person within your organization your development team who will give you access to Google Play console I'll give you a couple of tips Google enables them to give granular access so on a per application basis and they can also give limited access so they can give you read-only access which is plenty good enough for this sort of information and by read-only I mean it means that you wouldn't be able to make a new release the app and push it to the app store but you would be able to get the real-time analytics the historical analytics and also potentially to download historical information so in short talk to your development team and get them to add you as an authorized user in Google Play console perfect, the next one is interesting it's from Ninkos how will the new data privacy policy from Apple affect the gathering of analytics data on iOS devices short answer is I don't know I think the industry is having to work this out if we look at the industry though it involves some billion dollar trillion dollar companies such as Google so what is for sure is that someone will find an answer to this so the industry will find an answer one of my recommendations would be though is to start considering making it easy for users to opt in to providing analytics in the application the iOS ecosystem is different from Android anyway because Apple each time you update the operating system so iOS on your device typically ask do you want to share information with developers etc that's how they handle it and in terms of the new privacy it's going to have an effect and similarly with Google's changes which are currently mothballs in getting rid of third party tracking cookies again it's going to change the way the analytics works so don't know we'll follow it and there will be answers but I don't know the details yet all right thanks Julian there's a last question and then we'll wrap up there's a question from Anil which feels we must track when we are tracking analytics how do we give priority to different fields which we are tracking in analytics another sort of big conundrum there isn't an easy answer to this so some teams say well basically let's collect everything so we'll collect details of the user what they did, what was in their shopping cart etc etc so that we have everything we need to diagnose problems when they occur to understand the usage patterns however other teams say well that's potentially a breach of privacy the human privacy independently of what Apple allows for instance just this is not a good thing to do to track humans in this level of detail so we need to work out what we're tracking and broadly I'd separate and distinguish it from the human side how much we're tracking about the individual the process side what is the user doing independent of who the user is then finally the technical side the technical is things like the crashes, the ANRs the start-up times cold start warm start etc so the technical side is something that's typically collected by these tools anyway so we just need to use it in terms of the application what I recommend is we focus more on testing the flows so for instance the screen names are more prevalent through the application than who's doing what and we're very careful about tracking the human specific aspects of it so Jane age 23 coming from Mumbai is probably information we shouldn't be capturing by default in the analytics and ultimately we need to work out what of this data we're using and why so if we're looking at plugging this into Appium it's probably going to enable us to select a device to run the tests on it may enable us to configure the journey through the application so all those seem to make sense do we really need it to provide what goes into the shopping cart and the user account well hopefully not hopefully we can sort those details out so broadly my recommendation is we look at the technological side first and then we track the application logic next and we try to avoid tracking the details of the humans and what they're doing so I think we are out of time thanks a lot again Julian for this very insightful session your entire PhD thesis kind of compressed into a 45 minute talk Thank you Nuresh for hosting me