Well, I don't know how I'm going to follow a green parrot on a genie, so I'm going to try it. I said, well, okay, my name is John Kelly, I'm here from Atlantic Language, and we're going to talk about our experience with the Brazilian CAPIS program, and also the topic course that we had to put together for these. Before I go any further, I'd like to say that this was a... Well, it wasn't just myself to put this talk together. There was Gigi Rousseau, who's our academic manager, who's here today. Okay, she was involved in this as well. And also, Rogine Walsh, who's our director of studies, who is dying to be here today. And unfortunately, we have the... Well, fortunately, we have the Cambridge exams today, so she's not here. But anyway, so, just to give you an idea, my presentation is basically... Well, organizers follow us. First of all, we're going to have a look at the background of what is the CAPIS program from Brazil, but a very quick look at that. We're going to have a look at an overview of the TOEFL course that we had to put together for these students. Then we're going to have a look at the challenges and solutions. What was the outcomes for this, for Atlantic Language and Galway? And it has been a significant project for us over the last year. And then we're going to have a look at, again, some of the reflections and conclusions from this. First of all, the CAPIS program was set up at the end of the 1950s, early 1960s in Brazil. It's basically... Well, I'd say a training betterment program for professionals from Brazil. The part we're looking at is the Science Without Borders program, which I'm sure some of you have heard about. It's basically a mobility program for people from within the sciences community in Brazil to go overseas, have experience and go back home and be able to implement new ideas and maybe fresh forms of thinking back home in Brazil. So, let's have a look at the profile of a typical CAPIS student. At the moment, there are about 1,400 CAPIS program students, or Science Without Borders students, here in Ireland. I have one of them. You've actually got one here. We weren't expecting that actually. So we actually have a live one here. So let's have a look now. Let's hope this is good. So the profile of a typical CAPIS student. Now, this again is from our interpretation from this. Well, let's have a look. So we have a science engineering background, also medicine and mathematics in the results of medicines, also includes dentistry. Twenty to thirty years old, usually. At least the ones we've had in Atlantic language anyway. Middle-class, usually. Mostly from large cities and quite large towns in comparison to Ireland. Again, from all over Brazil. One of the first exercises we do with the students is just to call up a map of Brazil on the interactive whiteboards and we asked them to show us where they're from. Absolutely. Just flabbergasting sometimes the distances that are between students. We've had 3,000 kilometers between one of our courses in January. It was 3,000 kilometers between two of our students. From right up in the north eastern corner to right down to the border with our students. It was incredible. Now, the English level in A2 to B1, initially English level when they come in, actually quite high for students. In comparison, we found with the European students, they hadn't really studied for three or four years while at university and when they come in, it is actually incredible how fast they improve. It's one of the things that really surprised us when the students come in. Now, react positively to challenges, extremely positively actually. One of the few things that you will never hear as any complaints and we have loaded these students with work as part of the course. Again, conscientious, extremely conscientious in how the work is done. We've actually learned quite a lot as regards sometimes how underachieving sometimes some students are in comparison to these students. Also, balanced, amenable and appreciative. Our end-of-course parties or end-of-course classes are probably some of the most memorable that I've ever had from the students. Be perfectly honest with you. You are genuinely sorry when these courses finish. The good news is that there's another group waiting to come in, so it's actually great. Then, sense of humour is incredibly important in many classes and in these classes especially so. Now, the Irish connection to all of this was started about three years ago with trade missions to Brazil. The agreements were signed to bring students over here. Then there were partnerships were set up with institutions here in Ireland. We've got the University of Limerick here. We have INUI Galway, DIT RSI and UCC and amongst others. To basically cater for these students in the sense of they come in for three, six months, nine months or a year of actual study in the University. They're mostly under-graduates although there have been Masters and Doctors people have come over as well but they've gone directly into their programs and they haven't come through any of the language courses. Our connection with this is through INUI. We were approached by INUI Galway too. They felt that some of their students needed language support so they came to us at Atlantic Language Galway to see what could we implement a tuffle course. That's what they had specified and we said yes and then the students started to come to us.  If somebody said to me this time, exactly this date last year that I'd be standing here in front of you talking about tuffle I would say there's no way because we had never done tuffle at Atlantic Language before. Never. And suddenly we had four weeks, literally four weeks warning or notice to get this up and running we were expecting 20 students to come in and we had to get a brochure. We had to get literally a curriculum and materials to get the books and everything together and we knew that we had 20 students coming on the Monday and we had to get moving on this really, really quickly. So it really was one of the most intensive projects that we worked on. That of course right in the middle of our normal summer curriculum as I'm sure you all know yourselves this was right in the middle of July and this was the middle of June.  So let's have a look then. I'm talking about the tuffle exam. The tuffle exam was specified by in UIG because through the CAHPS program that's how the students are selected to come over here as regards their English level. So they wanted a program that involved tuffle in some way so that they could basically, for equivalency scores through IELTS so that they could know exactly what level the students had. So that's what we ended up then using basically what was the IBT, the tuffle IBT which was an internet based exam but without the component of actually doing the exam online for lots of different logistical and technical reasons as well. So let's have a look at the tuffle exam for maybe for people who are not familiar with what the tuffle exam is. It's basically an American exam. First of all it's based around American English and American vocabulary which is a little bit unusual for us then here because we're so used to the Cambridge exams and then suddenly we have to make this switch over. But it didn't prove to be such a challenge really. When you think about it there are certain vocabulary. It was more the layout of the exam itself that was challenging for us. So basically it's the idea of the exam. It's actually a very practical exam in the sense of it tests the effect of communication of ideas. Focuses on use of language as opposed to knowledge. Not really interested in the exam. It's not really interested in how much you know. It's basically what you know and how you're able to apply that. It simulates academic and student life communication. All the elements of the exam are based on campus. You have interactions with lectures. You have interactions between students and then you have tutorials, things like that. Now, what were our objectives then? Our objectives were obviously to increase the learner's general language skills. This was what EUI had requested. And our focus on this was yes. EUIG had said, OK, right, we do the talk like that. But when we sat down to talk about this and we're trying to get this together, we said, well look, that's not really the main objective here at all. It is in the sense that the students have the taste for EUIG. But we were thinking, hold on for a second here. What's going to happen to these guys in September when they are suddenly, they have like two months with us and suddenly they're into classes with Irish people, Irish lecturers. So when we put the course together, we looked at this, not just getting them to pass the tough exam, but to see how helpful could we be, what tools could we give them to really, you know, this wasn't abstract. It wasn't like a Cambridge exam, so maybe somebody's going to use this in the future. We knew that these students were going to be sitting down at lectures in EUIG in September. So that made it a hell of a lot more real to us. And so we incorporated that into the program. We had to prepare the learners for the Tuffle IBT, OK, which was a modified version of it, as I said in the sense that they had to do an exam. We had to put the exam together ourselves also. And basing it on the Tuffle IBT exam, which we said was a computer-based exam, but modified in the sense that it was going to be on paper. In short, learners obtain a minimum of 80 out of 120 Tuffle IBT scores. That's what EUIG was basically asked to do. Now, we have been very successful at that. All of our students have passed at what EUIG, they have been incredibly happy with that. One or two barbed lines, but most of them actually with excellent results. Surprisingly good results in some cases. For the Tuffle IBT exam. It is quite a challenging exam. Provided, there was also the cultural exchange element in the sense of here we have, literally, people getting off an airplane on a Saturday and coming in their first contact with Ireland is probably through the teacher. And this was actually, we sort of realized this very quickly, that we had a hell of a responsibility here that they were going to be three, they literally arrive on the Saturday, you know, go to their host families on the Sunday, and this is their first real contact with Ireland, any English-speaking country in lots of cases, with us. So there was that cultural exchange element and we sort of realized that very quickly that what we were doing was incredibly important to the students themselves. Now, so let's have a look then. This was our July 2013 trial run. These were, I think, 19 students in total, I've got Gigi here with and myself. These were mostly doctors, dentists, we had three physicists, really good students, and very clever and very challenging. Probably one of the most challenging groups of people that I've ever had to be in there with or even Gigi as well, in the sense of these guys didn't let you get away with anything. I remember one day making a mistake of just saying something as an offhand comment, okay, out came the phones, and next minute I was being quoted a Danish research paper on something. Right guys, thanks. That's how challenging it was and they were really good people and very formative experience for us in the sense that we realized, oh my god, these are the people we're dealing with now. One of the things that we realized very early was that we really had to up our game, really had to do it because we were dealing with very, very special students here. Now, okay, so this was the first influx of over 100 students into NUIG in September. Okay, we'll be coming back to this a little bit later. Now, so challenges for learners. Now, so in the sense that it's okay talking about us, but what about the students? What was their challenges were they facing? Well, obviously enhanced speaking skills was incredibly important. A lot of the students did not have contact with English as spoken English. Reading, yes, but spoken English, no. Maybe for two or three or maybe four years before they came to Ireland. So, they were good, they had a really good level, but they were missing some of the flexibility sometimes with higher level structures and things like that. Now, next one, we obviously increased academic vocabulary, a huge academic loading of vocabulary on this. As part of the TOEFL itself, just the nature of the exam, the very American based college type exam, but also just for general we had to make sure that we were filling in all of the spaces for the students. As I said, our approach to this was the TOEFL on one side, but the students first on this. We had to make sure that they were completely covered as regards to their vocab, and a very intensive part of our program was the focus on the vocab. Now, improved writing skills, again writing skills, pretty obvious, these guys were going to be not just for note-taking but they were going to have to produce papers in the new IG, or even in the future as well. Essays, you name it so they needed quite a lot of work on that. Fine-tune and listening comprehension, again they needed those reflexes for listening obviously for lectures, can you just have the lectures and then note-taking connected in with that as well. Now, how about the challenges for the tutors? Let's go back, what did we face? This was the interesting bit as well. Well, first one, we had to adapt the TOEFL IBD exam to in-UIG's requirements. That was the first thing we had to do. We had to make sure that the students passed the in-UIG requirements. All the time we were expecting an inquiry. Well, one of the things was we assumed we were going to be inspected in some way. So we were expecting somebody to arrive in at the school someday from, the Brazilian, let's say the Constance and Dublin or the embassy, whatever, and say, okay, I'm here from CACA, so we're here to see what you're doing. So that was one of the first things we decided, okay, what happens if we're going to be inspected by somebody from Brazil or even from in-UIG? So we always said, in the sense of, okay, let's make sure we're doing a really good job on this. Developed TOEFL specific work practices. Now, I'll come back to that a little bit later. One of the things we discovered was this was not the CAA and this was not the FC. This was not take a book and walk into the classroom and have a nice cushy 12 weeks. All right? That didn't happen. We had no material. We had to put the course together ourselves and this was one of the most intensive parts of this. We had to invent, put this whole course together ourselves. There was nothing there for us in a very short time. Now, as I said, we had been working on this over a year and now we have a really solid course on this but it did take a hell of a lot of work. Deal with single nationality class. This was something unusual for us as well. We have quite a diverse student that's a population at Atlantic Language Galway. But suddenly you went from having two or three Saudis, Spanish, Italians, whatever in the class and then suddenly you have 12 Brazilians.  This was changed. That was quite unusual when we saw that first. Now, not unusual in the bad way but it was in the sense that you didn't have this sort of diversity sometimes inside of the class. Now, continuously challenged and motivated. We were talking about really, really good people here. Quite brilliant in some cases. We had three of our physicists, okay, mathematicians who kept us on our toes on the first one. We had, for example, Jefferson, for example, was one of the, from Salvador who was probably one of the most brilliant people that I've ever come across in the class. And, I mean, he mentioned the mathematics or astronomy or something, okay. You had to be absolutely sure you were, you know, because he was able to hold a conversation with almost anybody on this. Now, keep it interesting. This was another thing as well. We had these guys for three hours every morning and we had to make sure that they were in there, that they were working, they were challenged all the time and also in a very interesting way. Also, we had the personal crisis. We were talking about groups of young people a long way from home and as the course progressed, we began to see, you know, the little problems began to arise. Long-term relationships. Some people had left, were trying to continue relationships that they'd left to back home. Boyfriends, girlfriends, were left at home. There were breakups. There were medical emergencies. There were financial emergencies. There were, there are. What am I saying? They're silly. There were all sorts of these things. One of the most interesting cases, again, is when a woman rode in Galway one night, gets sideswiped by a taxi, gets thrown about, literally, 15 metres through the air and lands up in the forecourt of the G Hotel. On his head. Ends up in hospital. Two days later, he's back on crutches in class. Black and blue on one side of his body. Not seriously hurt. Lost his bike, lost his bag, lost his phone, of course. But he's back in class two days later. So just to show you the sort of people and these were just some of the things that we've come across. Nothing but nothing major. Now, let's see, what sorts of solutions? This is probably something that might be useful to you guys. The rest of the story, okay, but let's see if you can take some of this away with you. First of all is an extensive in-depth grammar revision, which I'm sure you could all have more or less predicted in the sense that they needed higher level structures that they didn't really have. We had to work quite a lot on those. Mapping of persistence. The challenges we had. We discovered in the first one or two courses we ran, we had trouble trying to basically correct errors. They didn't seem, there was a very simple little errors that were repeated again and again and again. We had to basically stop and look at this and say, what can we do about this? We had to make out specific correction sheets for each exercise. We had to note down their errors. We had to photocopy these, get them back to the students, get them in class every single day of reading. It was probably the most intensive corrections that I've ever had to do. I've never had to, even for Ed Cambridge Advanced, I've never had to do this in the sense we had to go through all of these to try to improve their writing levels. Intensive real-world reading from a variety of sources. We had to use real material. Now the advantage of this was that they were actually reading real material back home and textbooks in some cases in English. This example from Wired Magazine, new scientist. Full articles from this. These guys were going home, reading these over the weekends, producing summaries, vocabulary reports, and back in on Monday morning. On the rare occasion, this wasn't done. Now I'm talking about seven or eight page articles from Wired Magazine which would be challenging even for any of us to do the same amount of work. There was no complaining. They were back in on the Monday morning and every single thing you asked these people to do. The first course, we were sort of surprised people were doing this. The next course, then we began to realize, hold on, we can actually start loading these guys with more and more. It was always dark. It was incredible. Some of the work I would have sort of said to myself, oh my God, if I had to do this for the weekend, they were able to do it. Extensive use of mind mapping note taking. We had to go back through our experience to the EAP. We had experience with mind mapping. This proved to be incredibly useful again, to tell them how to do this and go back to an extremely organized note taking for summaries. Every time they read something, they had to produce a summary on this. Again, why? One of the writing questions is what's called an integrated question in the TOEFL, which involves listening, okay, reading, and then you have to summarize all the points and synthesize the points together. Now, focus on summarizing effective linking. We'll go back again, in linking incredibly important. Extended group discussions based on course takes. This is something that we hadn't really used before. I hadn't really used this before as intensively, but if there's something about a large class, when you get 12 students from the same country together, they react differently. There's nobody there pussing, saying, oh, I'm not interested in this. They were. And we divided them up into six and six groups. This worked incredibly well. You take your article and then we would say to them, okay, why is this important? Why is this important today? How will this be important in the future? Also, for example, as regards Brazil, how could this be adapted? How could this be brought back to Brazil? Where could you see this industry in Brazil in 10, 15 years time? And they worked away with that. And it was incredible. They could spend 20, half an hour, 45 minutes on this. We just let them run with it. And it was incredible. The enthusiasm that they were ever put into this was really incredible. Now, so bridging techniques connecting course topics to Brazil, Brazil's future. This was something we found incredibly useful. We had experience doing this before with the Chinese about seven, eight years ago. Now, we used extensive bridging techniques. And again, we were able to apply that again, and it worked. Also, then there was the CPD, the Continuous Professional Development Element because it was okay for, let's say, Gigi and myself coming in first. But we knew we were going to have more and more groups coming in. So we had to start training other teachers. Now, I was lucky because I came from a science background. I'm not from an arts background or a languages background. Okay, I'm actually a marine biologist. So I was able to, okay, the first time around it was easier for me. And that's when I was like, okay, I knew where these guys were coming from. But we knew, okay, that we were going to have to try and extend this into other, let's say, teachers who are not from the science field. So one of the major things that we had to do, well, basically, I had to do it from January to March, was to set up a course, a day-by-day course, where it was all laid down, all the materials in the sense that, and depending on the situation, that it would not be challenging for the new tutors coming in. It would be acceptable to the students in the sense that teachers, coming in on the course, that they would be able to just transition from a general language class into a tougher class on a Monday morning and there would be no tears and everything would work. Now, it takes about a two-week transition, three-week transition, but it actually worked. Now, for reflections on this then, what did we learn from this? Now, it's taken from, again, from Atlantic language and what did we learn from this? Well, first of all, expect the unexpected. That's one of the things. I'm sure it's all, but definitely for us now. We've begun to notice that we're getting a lot of different courses in and with very short lead times on the courses. Be flexible and innovative. We have to be very flexible as guys who's coming in. We have to be very innovative as to how we're actually using new techniques and it's definitely this title has shown us that really we have to work extremely hard and come up with very innovative solutions with new people being and new students coming in. But I'm sure that's the same across the board for everybody. Practical course design.  As I said before, not like a Cambridge exam where you hope students will be using this in the future. We know that they're going to be using this in a very short period of time. Market requirements are placing different demands of language that we've sort of realized that we're going to be getting a lot more of these courses. Courses with very short lead times. Intensive courses. One of the courses that's growing as well are the Intensive SCs, Intensive CAs for us as well. They put in a program of that as well. The word seems to be Intensive. We have an Intensive Isles course coming on again in a week or two. So everything seems to be Intensive these days and short term and then new courses coming in. You don't even know that they're coming in. Now, so that's basically the Atlantic TOEFL program. Now, it's okay talking about all these students, but what we're going to do apart from Vera, I'm going to give you just a very short video clip of some of the faces here. Some of these people that I've been talking about. These were students from our first intake here. It's only short, it's only about a minute long now from our first intake and this was shot around the end of August early September last year. First to come to Ireland because it's a country really different from Brazil. The weather, the people and I am feeling really welcome here. I like the pubs that are really different from what we have in Brazil and the university, the buildings are really different too. It's really old and beautiful and it's a smaller city in contrast with the big city in which I live. So I love to walk to everywhere and people are really cool and they help us a lot in the city's rich and I really love this country.


The Equitable Life Assurance Society presents, This Is Your FBI. This Is Your FBI, the official broadcast from the files of the Federal Bureau of Investigation. Presented transcribed as a public service by the Equitable Life Assurance Society of the United States and the Equitable Society's representative in your community. Perhaps you received one of the postcards your Equitable Society representatives sent out this week. He sent these cards to people in your community who are evidently headed for successful business and professional careers. The kind of people who should be interested in the Equitable Society's plan for men and women on the way up. In about 14 minutes, I'll give you the interesting details on this plan for every man who believes in his ability to get ahead. It is offered by the Equitable Life Assurance Society of the United States. Tonight's FBI file, The Hollywood Frame Up. The announcement a few months ago that Dr. Albert Einstein had successfully completed a series of four equations which he hoped would eventually explain the fundamental laws of matter and energy was a front-page newspaper story in almost every city in the land. Very few people in the world with the exception of Dr. Einstein understood those four arrangements of symbols and it is safe to say that even now very, very few people fully understand them. However, at that same time as the announcement was being made, other people were working on other equations or formulas that anyone can understand. Formulas like one scheme plus manpower equals one crime. Each of us can understand that equation, can accept it because it deals in units our mind can picture. One crime happening someplace is a reality to us. But a million and a half major crimes, that unfortunately does not have the impact it should. Yet the fact that more than that number of major crimes were committed last year that the total will be surpassed this year is important to each of us. For the victims of those crimes will not be names gathered at random and written on police blotters the nation over. The victims will be you. Tonight's file opens in a saloon located on an alley that runs through the slum section of a large eastern city. It is late at night and only two customers remain leaning against the bar. One of them is talking to the tired bartender. And anyway, who says it's better this way? Not the Italians make with the realism. All the time documentaries, documentaries, documentaries. Nobody in Hollywood just makes for plain picture now. They might as well turn the studios into parking lots. Your glass is empty, pal. Yeah, fill it up. Give him a refill, too. I got enough, Doug. What's the matter? You want to live forever? Trick up. Okay, great. Give me a new one, too. Too right coming up. Hey, you see that picture, city edition? No. Again, they made with the documentary. I hear they got good actors. Actors. They got amateurs. Sure, they look good on the screen. They're real newspaper guys. What do they know about acting?  Are you scared? Sure. Gable is scared. Jimmy Stewart is scared. Why shouldn't I be? Here you are, pal. Here, then. Yeah. How would you like to have a job where you can't go against the juice? I don't see nobody stopping you from drinking. No. But I ain't got a job, either. You just said you'd come in from Hollywood to be in a picture. They make him here. That's right. But the studio threw me a curve. I got a director from New York. A stage director. Hello, Ben. Hi, Artie. Hey, let's have a beer. He's one of them realism nuts. I come in this morning. I got a hangover and poached eyes. He starts beefing. I tell him to shoot me in long shots and nobody'll see my eyes. He says it would violate the spirit of the drama. So I belted him. Down he went. That's where this works. And he comes up to be spanned as my nuisance. I give you the sign. He'll say that John is closing. Just do what I say. We're making a flicker, I said. A 30-cent flicker. Get wise, dear self, I told him. Wait till you're in this business more than 20 minutes before you start making up rules. All you guys come out from New York with that shred of various way of acting where a guy makes believe he's a grapefruit. I told him we ought to go back to Hollywood and let a movie guy make the picture. They call you a director, I said. You couldn't direct traffic. That's when he fired me. Finish your drinks, man. In a closing time yet, it is here. What's your name? Bolstead? Come on, pal. I got a car outside. I'll give you a lift back to your hotel. There's nothing planned at the hotel? Let's go someplace where we can get a drink. Okay, let's go. Leon Earl, did you get it? This way, pal. The car's out here. What's the name of this crib we're going to? Joe Asis. Go ahead, pal. Okay. Well, what happens now? Show him already. Sure. Let's take him down the alley. Good morning at the local FBI field office. Police detective Tommy Adams is approaching Special Agent Jim Taylor's desk. Hello, Tommy. Hi, Jim. Got a minute? I need some help. Sure, name it. Ask your lab to find out what they can about this bullet, will you? Okay. Where's it from? The back wall of a jewelry store named Courtney's. Two men robbed the place this morning. Any descriptions? No, both of them are masked. Isn't this Courtney's got a burglar line? Yeah, but Courtney didn't use it because he was expecting the holdup. Come again? When the store opened this morning, a man came in. He introduced himself as a representative of the Crenshaw movie studio. Yeah. He told Courtney that the studio's making a picture called Eye Witness here in town. Well, that part's true, isn't it? Yeah. Then he said that the holdup of the jewelry store was part of the script. He asked Courtney if they could use his store for the scene. No, I get it. What did they take? $26,000 worth of diamonds. Ouch. Any description on this, man? Nothing good enough. We got something to work on, though. One of the holdup men dropped his hat in his wallet. It belonged to an actor who was working in the picture. His name is George Black. He ought to be easy enough to locate? Yeah, the chief is checking on him. Okay, Tommy. I'll send this bullet along to the lab. As soon as I hear anything, I'll call you. Thought you said you'd call me last night. We said I'd call you if I had trouble. How'd you make up? With the actor? Yeah. I took him to his hotel. Anybody rumble? I told everybody he was loaded. Did he come to? Once in the freight elevator. Did he put up a beef? He didn't have time. I tapped him out again. He must have quite a collection of lumps. Where's the stuff? Over there. Did you look it over? Mm-hmm. What do you think we'll score for? Well, there's two ways of doing it. What do you mean? We can pedal these rocks now and take a couple yards a piece. We let the stuff cool for six months. We can get 10 Gs for it. It's a pretty big margin. You want to let it cool? Cool? Yeah. Let's put it in a deep, deep freeze. They've been made now of the police. It ain't against the law to sleep in this town, is it? You George Black? That's my billing. I'm from police. Okay, I'll take three tickets to the ball. Now let me get back to the case. Wait. I want to question you about the Courtney holdup. Read that again and get a left this time. A jewelry store was held up this morning and when the bandits ran, your hat and your wallet were filed. Look, that's a funny routine, but I think you'd be better off working with a girl. I'm serious. Suppose I show you my wallet. It's here in my back... Hey. It's gone. I know. It's a police headquarters.  I got loaded last night and I've been sleeping it off all day. How could I stick up a store? We can talk about that downtown. Hey, wait a minute. I just remembered something about last night. What? I got loaded in a crummy joint. I was with a guy. He must have clipped my wallet. Look, let me go back to the joint. I can find that guy. I'm afraid I can't. Oh, now look, you ain't giving me a chance. You got me marked guilty now. I've got to have your book done the evidence. Put a rep like this. Look, after getting canned in the picture, I'm dead. I'll wind up in television wrestling. All right, let's get going, Black. So you won't give me a chance to square myself? Sorry. All right. Then I got a ticket. Shake hands with a sucker, Jim. Tommy, where did you get that eye? I forgot the duck. Well, if this happened on the jewelry case, you better fill me in. We're in on that one, too. You are? Yeah. That bullet you dug out of the wall at Courtney's matches one in our unidentified ammunition file. Oh? A jewelry store in Los Angeles was robbed two years ago. A jeweler was killed with the same gun that fired the bullet you brought in here. George Black lives in Los Angeles? That's right. The last time I saw you approximately 10 hours ago, you were going to try and find this George Black. I found him, all right. Oh? He, uh, gave you this over there? Mm-hmm. I saw him at his hotel. He denied everything. Just plain denied, or did he have a story? He had a story. He said he was drunk last night, and somebody must have picked his pocket. Well, that would explain the wallet. How about the hat? We never got that far, Jim. Huh. He said he remembered being in the saloon with some men. And if he could find them, he could prove they took his hat and wallet. And tell me, how did you feel about his story? Ah, I hate to say this after what he did to me, but I kind of believed him. Despite of all the evidence? Yeah. Well, in either case, we'd better get to work and find him. What do you have? Hello, Pete. What do you want to drink, mister? I wouldn't drink in here if you threw in a picture of the chef in tights. Nobody sent for you, doc. I got up the day after drinking here last night, and I had three heads. Well, one of this joint is always empty. You weren't in here last night, pal. I wasn't in Scranton. I got a good memory for faces, and I'd never seen yours before. Look, I was standing right here with a guy named Ben. You know him, don't you? No. You got a memory like a blue-sert suit. A lot of these joints around here look the same, pal. You must have been in one of them other chimneys. What's the matter? You get rolled? Look, I know I was in this joint talking to this guy Ben. You know him? You kept calling him Ben. Hello, Pete. Hey. He was here, too. He's the guy that turned out the lights on me. Marty, you ever seen this guy before? No.  Look, you're coming with me. Like where? Down to the cops, and you can tell them what you guys did with my wallet and my hat. That's what you think.  In just a moment, to this exciting file which shows how your FBI helps protect the security of America. Now a quick look at the future of America as it stands at the threshold of the second half of the 20th century. Leading economists say we are safe in predicting that America's industrial productivity will reach new highs during the 50s and that the resulting prosperity will be more widely shared than ever before. By 1960, it is expected that the average family's income will have risen to over $4,000 a year. Remember, that's just the average man's income. If you figure yourself as above average, then your salary should be due to climb to far higher figures than that in the next few years. If you really have faith in your future, then you'll be interested in a special type of life insurance designed to order for your type of person. It's the equitable life insurance society's plan for men and women on the way up. To people who are confident they're going to get ahead, this plan offers three important advantages. First, as your salary goes up, your insurance can keep pace with it. When you get that better job or that big promotion comes your way, you can adjust your insurance to measure up to your increased income. Second, while you're waiting, your wife and children have the life insurance protection they need. This means that you have the peace of mind, the freedom from worry about your family. That's essential to a man who wants to concentrate on getting ahead. Third advantage, the equitable plan is flexible at all times. It can expand or contract as you see fit and offers you many desirable options, which your equitable society representative will be glad to explain to you. So why not get in touch with him right away? Phone him and ask for full details on the equitable plan for people on the way up. Or send the postcard care of this station to the Equitable Society. That's E-Q-U-I-T-A-B-L-E. The Equitable Life Assurance Society of the United States. Now back to the FBI file, the Hollywood Frame Up. In connection with tonight's program from the files of the Federal Bureau of Investigation, we bring you a message from Mr. J. Edgar Hoover. Mr. Hoover's message is, and I quote, Each time this official broadcast dramatizes one of our files, it is done with a specific purpose in mind. Frankly, our purpose this evening is to create apprehension. If someone could teach the American citizen to look about him constantly, to be less bland and more suspicious, our crime taxes might start downward instead of constantly increasing. Our greatest fault is complacence. Second in seriousness is the American habit of viewing everything unpleasant as something detached, something that doesn't concern us. Each of us seems gifted or cursed with the belief that while misfortunes of law and fraction may happen to the other fellow, we ourselves will be spared. Nothing of course could be further from the truth. The victim in approximately 99 out of 100 cases which pass through our field offices is an innocent person. While it is true that thieves fall out and that there are such things as gang wars in which criminals fight each other, it is well constantly to remember that like any military organization, America's army of criminals has one basic target, to profit from another's labors. That could mean you. Tonight's file continues at the local FBI field office the following morning. Morning, Jim. Oh, morning, Tommy. How's the eye? They feel better if we could locate George Black. Oh, I checked with the Crenshaw studios. He was fired from the picture they're making here. I know. I wired Washington last night to see if they had anything on him. Heard anything yet? Yes, and if he's got a record, it's not under the name of George Black. I got a couple of pictures of Black from the studio publicity man here in town. I sent them along to Washington too. Pardon me, Tommy. Oh, sure, Jim. Especially, Jim Taylor speaking. Yes, that's right. You have where? When? Thanks very much. That was Memorial Hospital, Tommy. They just brought in George Black. A friend again last night. Which friend? The actor. Where? At Pete's joint. Nice shot. He was around there looking for us. Oh? Yeah, Pete was given only I never saw you before as when I walked in. So, what happened? He spotted me. Wanted me to go to cops with him. Remembered you that good? Yeah, so I belted him out again and dumped him in an alley. No, you shouldn't have done that already. You think I should have gone to cops with him?  I couldn't. Look, Artie, this guy's a lush bucket. We see him again, we don't know him. You'll think maybe he drew a blank. They could have worked. But don't, we'll tap him out again. This time for good. Mr. Black? Look, if you've got more pills, give them to the guy next door. I'm a special agent of the FBI. You're my credentials. Oh, I suppose now I stole the plans to Sandy Hook. I'm afraid there isn't much to be funny about, Mr. Black. I ain't trying to get laughs. Mind if I ask you some questions? Go ahead. The gun that was used in the hold up here yesterday morning was also used two years ago in Los Angeles to kill a jeweler. It's possible that you were in both of these holders. Look, mister, I did comedy and vaudeville, not a shooting act. This killing took place just about two years ago. Are you in Los Angeles then? Two years ago. That's right. Yeah, yeah, I was laying off at Santa Anita.  Look, I'm no crook. I don't even know any crooks except my agents. Then why weren't you willing to go to police headquarters yesterday with Detective Adams? Because I know a frame when I see one. And this one is so big, you could have stuck Whistler's mother in it. Oh, who do you think is framing you? There are two guys who belted me out night before last and took my hat and my wallet. And all the names are? Yeah, one guy's name is Ben, the other guy's Artie something. I don't know their whole tags. Where'd you meet them? In a saloon on Water Avenue near 15th Street. I don't know the name of the joint. The bartender's name is Pete. Can you give me a description of them? Not a good one, but I'd know them if I saw them again. Tell me, did you go down to this saloon last night after slugging Detective Adams? Yeah, the bartender said he never saw me before. Then this Artie character walked in. I told him to come on down to the cops with me. And the next thing I know, I'm laying here inhaling pills. What made you think you could solve this whole thing by yourself? I played a detective in a picture last year. Well, that's the movies, Mr. Black. It's a little more difficult in real life. Well, I found that out. Well, I'm not saying that I believe your story, but I'm going to go out and check it. Okay. And until I get back here, you're technically under arrest. You get to see George Black this afternoon, Jim? Yeah. What does the story sound like? Well, he told me he got into this whole jam because he got drunk at a bar with somebody named Ben. You know something, Tommy? You might be right. You mean he convinced you to? Partially. This Ben could be a petty larceny thug named Ben Hamilton. How do you know him? I dug up records of everybody named Ben. I sent the pictures down to the hospital and Black identified this Hamilton. Does he think Hamilton's trying to frame him? That's it. He went back to the saloon after he slugged you and the bartender denied seeing him before or knowing Hamilton. All right. Where do we go from here, Jim? Well, I've got a call in for the saloon where Black claims he was hit. You think the bartender will talk on the phone? He may. Hamilton was arrested once with a hoodlum named Rocky Thompson. Well, he's in jail now, out west. He may be able to get someplace using that name. Oh, pardon me, Tommy. Sure, Jim. Hello. Oh, fine. Hello. Pete, the bartender there. This Pete? Pete, this is a friend of Ben Hamilton's. No, no, you don't know me. Rocky Thompson. Yeah, that's right. Yeah, we were away together. Yeah. Look, where do you think Ben will be in? No, no, just on my club. Okay, thanks, Pete. Well, that part of George Black's story checks they know Ben Hamilton around the place. Now, if we get a... Hey, wait a minute. Huh? Tommy has, uh, Courtney's jewelry store got descriptions of those unset diamonds that were stolen. Sure, Jim. Why? Well, George Black can leave the hospital tonight. Let's give him a chance to prove his own innocence. Yeah, Ben. Give Artie me another pair, huh? Okay. You want something, mister? No, don't tell me you still don't remember me. Hiya, Ben. Artie? I think maybe we better go someplace else. That's okay with me. Hey, don't go, boys. I got something in my kick you'll want to see. Pete, this guy leaves. We'll come back. Uh, look, Ben. The diamonds from Courtney's. Where'd you get them? I went up to your apartment and took them. I figured as long as I was taking the rap, I might as well have the loot. Ben, how'd he get into your joint?  Well, that kind of spills it, don't it? Maybe. But it ain't gonna do you much good. Have I got to build them again? Yeah. This time, go for big casino. Ben, not in here, huh? Artie, take him out in the alley. Oh, now wait, I'm dead. You heard him, doc. This gun says start moving. Okay. How you doing, man? Drop it.  Good to see you, Jim. Hey, what is this? I think you can guess. Well, thanks for the cooperation, Mr. Black. You did some very convincing acting. Mr. Taylor, if acting in saloons counted, I'd be the yearly winner of the Academy Award. John Hamilton and Archie Melrose were returned to the West Coast to be tried on the charge of murder. They were both convicted and executed. Their accomplice, Pete Morgan, was sentenced to 20 years as an accessory to their crime. One call to Pete the bartender made by Special Agent Taylor is what convinced him that George Black was probably telling the truth about having been framed. However, there was still the problem of proving that Ben Hamilton and his friend had committed the crime of robbing Courtney's jewelry store. Taylor, therefore, went to the store and had a set of glass duplicates of the stolen diamonds made up and wrapped as the real diamonds had been wrapped in the safe. George Black was then called upon. It was his job to convince Hamilton and his accomplice that the pieces of glass were really the diamonds and that he had gone to Hamilton's apartment to get them. Special Agent Taylor and Detective Tommy Adams were outside the alley door listening to the conversation and were prepared to enter if George Black needed help in sight. The gun which Archie Melrose was pointing at Black as they came through the alley door turned out to be a luger, the luger with which the jeweler in Los Angeles had been killed. And so another case from the files of the Federal Bureau of Investigation was closed. And closed as 97% of all FBI cases are with the conviction of the arrested criminals. In just a moment, we will tell you about next week's exciting FBI file. Now one last word on the Equitable Society's plan for men and women on the way up. It's a plan for the man who knows that some Valentine's Day, he'll come home with this kind of present for his wife. Honey, you get that fur coat after all. The boss just made me district supervisor. If you're that kind of man, then the sooner you get in touch with an Equitable Society representative, the better. Ask him for full information on the Equitable Society's life insurance plan for men and women on the way up. Or send a postcard care of this station to the Equitable Life Insurance Society. Next week we will dramatize another case from the files of the Federal Bureau of Investigation. A fact will recount of the activities of a ring of international Jewel thieves. It's subject, smugly. It's titled, Package of Death. The incidents used in tonight's Equitable Life Insurance Society's broadcast are adapted from the files of the Federal Bureau of Investigation. However, all names used are fictitious. And any similarity thereof to the names of persons living or dead is accidental. Tonight the music was composed and conducted by Frederick Steiner. The author was Jerry D. Lewis. Your narrator was William Woodson and special agent Taylor was played by Stacey Harris. Others in the cast were Herb Ellis, J. C. Flippen, Ed Gargan, Edmund MacDonald and Carlton Young. This is your FBI is a Jerry Devine production. This is Larry Keating speaking for the Equitable Life Insurance Society of the United States and the Equitable Society's representative in your community. And inviting you to tune in again next week at this same time when the Equitable Life Insurance Society will bring you another thrilling transcribed story from the files of the Federal Bureau of Investigation. Package of Death on this is your FBI. This is ABC, the American Broadcasting Company.


Hi everyone, Joe for Jasper's casebreaks.com with 2019-2020 Panini mosaic basketball fast break edition fast break edition edition Pack break number 12 random team break number 12. So we'll do the break itself first, but at the end of the break We'll re-randomize your names and the top three will win spots in that number block right there, which looks like it's sold out. Oh Maybe they're in there. I've got to do one more and then I think That'll eventually sell out that so we're pretty close ladies and gentlemen. We're pretty close to knocking on another one of those uh noir national treasures Mixers two box mixer big thanks to these folks right here as always we're gonna do the break itself first So let's roll it. Let's randomize it seven times One and seven in final time after seven. We've got Matthew down to Teddy All right seven times for the teams One and a seventh and final time. Oh nice Brandon. Thank you No need to shout though Gris down to Lakers take a deep breath R e l a x as Aaron Rodgers once said All right, so there's the first half of the list right there and There's the second half of the list right here. No trade windows and stuff like this. Let's print Let's real. Well, maybe it's the stage winners that win them when spots and Oh nice, we've got a Lucas Samanich fast break autograph out of a fast break pack. That's not too shabby spurs That is for Ryan s and the spurs and there's NBA debut hachimura. All right, nice autograph out of that One lonely pack. It's pretty good. Let's flip back to this screen right here. So Matthew down to Nancy and everybody in between Let's roll it. Let's randomize. It's gonna be top three after 11 six and a five 11 times one two ten and 11th and final time after 11 times big thanks to all of these folks from four down to 30 appreciate you giving this a shot There's one more opportunity to get into that and the top three Matthew Matthew and Shawn Wow Matthew with two of the three spots.  There you go I'm Joe. I'll see you next time for the next one. Bye. Bye


Let me introduce myself. I'm Diego, I love testing and thanks to that I started to contribute to Selenium some years ago. I maintain two projects that are related to infrastructure, open source ones. And right now I'm a software engineer in Sauce Labs. If you want to ask questions after the talk or if you want to get in touch afterwards, that's my Twitter handle. And if you want to check the projects I work in the open source community and you want to help me perhaps into those projects, you can check my GitHub profile. Before continuing, the reason of the coffee is that I am Colombian and that's the country that has the best coffee in the world. And right now I'm in the country that has the best tea in the world. So that's like a good match, I would say. Every time I have the opportunity to talk in public, I take the opportunity to mention that if you want to contribute to Selenium, if you want to learn more about Selenium, there are these different channels that you can join, the IRC channel, the Slack channel. And there is a Google group where you can send questions, but not only ask questions, if you have the expertise about Selenium, join the channels, join the Google group and help us to answer questions from other people who are getting started. So you can also find bugs in the website, contribute to the documentation and so on. So the message here is we are an open community, so come to us and help us build it in a stronger way. Okay, so what I'm going to do in this presentation is I'm going to talk, give an overview of what I have done in the last years while building test infrastructure, what I have been through, what my challenges have been and how I have tackled them. And I hope that you take something valuable from this, maybe you can take one of the tools that I will show, all open source, all publicly available and you can try them in your office, in your work. So the first thing I will talk about is about browsers and this will perhaps like the biggest part of the presentation because this is where I took all my base concepts that I applied into mobile testing regarding building infrastructure. And afterwards I will go and take those concepts and put them into emulators and simulators and try to do the same thing into real devices. Afterwards I will talk about all the maintenance required to these type of setups. And having said all those things I will mention how important it is to scope your efforts before starting to do anything and I will just round up things by giving my thoughts about what it means to build test infrastructure. So before I will just tell a short story about how I got into testing and test infrastructure. So I graduated from university, I got a computer science degree, I got a job in my hometown in Colombia. I was a developer and mostly people were testing manually. We were giving the software to the end users and they were testing it manually. And I had almost no knowledge about testing. Afterwards some years after I moved to Germany I started as an automation engineer and they told me just write all the test cases you can for API testing. So I just wrote lots of tests and afterwards I did the same for UI testing and I ended up going to a point where my test suites were getting slower and slower and I came to the conclusion that I had too many tests so things were getting slower. My first approach was to tackle that in a way that I have to improve the infrastructure and that's when I started working in infrastructure. So no one was paying attention to the local grid we had, I started paying attention to it and therefore I became the infrastructure guy. So I wasn't doing tests anymore, somehow I became the person who maintains infrastructure. I fixed a few things, improved things, our test suite got better, got from 1 hour 15 to like 20 minutes and our response to that improvement was to write more tests. And then I got into the cycle of improving infrastructure, writing more tests and I got to a point where is it where I am right now that I think infrastructure is not everything. First you need to have an important test strategy and then you can think about what you need as an infrastructure. But let's get back into the topic about infrastructure. Let's start with desktop browsers and my first infrastructure around early 2015. We had a Selenium Grid with six AWS machines, they were decent with good capacity. We were testing in these four browsers and they were something that I like to call handcrafted VMs. So this means that someone went by hand into the VM and installed Java, installed Selenium, configured things by hand and so on. And there was a point where I was seeing that some tests were failing and we had very old browser versions. So my question to the team was, do you know when we, what was the last time when we updated these machines? And everyone was like, no clue, like I just run the tests and I have no clue what is happening over there. Then I took the initiative to update infrastructure. So I went and I started writing some bash clips. I was not so good into bash clips. I think I have improved but I still have a long way to go. So I started downloading like the browser drivers, everything trying to update things slowly. And the first thing I got after upgrading things and improving things was people telling me, now my tests are failing. Like, what did you change? And this was because we were using old Selenium libraries and I was updating to a new Selenium server and things were just breaking apart. And then in my head was like, but I'm trying to make things better and now people are telling me that nothing is working. So my head was like, but this doesn't make sense. So then I got into the first point where you have to test the test environment before actually using it into your normal pipeline of testing. So I requested more VMs and happily they gave me those VMs. So I had to install the operating system, all the dependencies, Java, X-Virtual Frame Buffer, all the browser, make all the configuration. And I was doing that by hand every single time when I figured out there has to be a better way, which is the first important concept about infrastructure. It could be any type of infrastructure but the first important concept is infrastructure as code. And that's why there are different tools out there, free to use, open to everyone, that are Puppet, Ansible and Chef and other ones that you can use to configure your machines to have them in the state you want. If you want to have Java installed, if you want to have Ruby installed, if you want to have Selenium downloaded from a specific URL and put in a given path and make it work, you can use these files by providing scripts and executing them in a simple way. Now the tests were still failing because people were telling me that somehow every now and then tests were failing. And I started to monitor the machines and seeing that in the morning nobody was writing me, we were using Skype as a communication tool. So nobody was writing me during the morning and during the afternoon, around noon, like, hey, my tests are failing, what's going on? And this test was passing in the morning. So I started to monitor and see that some people were not doing driver.quick, for example. So the browser was still there, it was released, it was using memory, it was using resources. And I just came to the point where I saw, okay, maybe I'll try rebooting. And that was like the quick workaround at some point that around lunch, before going to lunch, I was rebooting the machines and then coming back and everybody was happy. But the main conclusion here was that I found out that when processes depend on each other or when you are running many browsers in the same machine, things are not isolated. So that's the second big concept about infrastructure that you need to have things in an isolated way. That's why there are different technologies that we can use for that. The most popular one right now is Docker, where you can use containers to isolate processes and run tests. The other one that we know since many years ago is VirtualBox. And one that is very common to be used in the Linux world is QM, which is an emulator that helps you spin up operating systems from scratch in a Linux environment. The third thing that happened was that people were complaining still, because they are never happy. And they said, my tests are slow now. They are not breaking. Well, sometimes they're breaking, but when they run, I see that they are slow. And what was happening is that we were serving around six teams. And one team wanted to run 10 tests in parallel, the other team wanted to run five and so on. And when I checked and was doing the sum, we were trying to run around 20 to 30 tests in parallel, and we didn't have the same amount of CPUs and RAM. So we were trying to do a lot of magic with just a few VMs, and that doesn't make sense. That's the first point where we can see that if we want to test in a big scale, we need all the resources for that. Then I came to the third point that is very important when we are building infrastructure and its performance monitoring. So we have to check how the machine is behaving, and we have to measure how much CPU and RAM each single test will need. This is very common in the open source world, where in the Docker Selenium repository, for example, people say, this is not working. I try to run 20 tests in parallel, and it's not working. And I ask them, what machine are you using? Yeah, I have an EC2 instance, like X2 something. I have two CPUs and a gigabytes of RAM, but it doesn't make sense. You have to check what resources you need before you build infrastructure. And the tool I really like about this performance monitoring is called Prometheus. Prometheus is a tool that works in different environments. You can use it to monitor the processes locally, a bare metal machine. You can do that. You can do it for containers, you can do it for Kubernetes, and et cetera, et cetera. So it's a tool that also allows you to export the values to Grafana or Kivana or whatever you want. And you can see over time how the machine is behaving, and then you can realize I am using a lot of CPU, I'm not using so much CPU, I need more machines, and so on. Then let's talk about the setup details of each type of browser. For example, Chrome and Firefox are quite simple because they run on Linux, so we can use them in Docker containers. And there are two projects that I recommend to use that you can start your grid locally and use Chrome and Firefox. And normally you need around one CPU and one gigabyte of RAM per browser and running, so per container. It could be less, it could be more. For example, if you are testing a website that is maybe just some text and images, it would need less RAM than testing Netflix, for example, that has a lot of images and a lot of animations and videos. So that's what you need to start monitoring so we understand what type of infrastructure we need. Then let's talk about Internet Explorer and Edge. Normally the way we could do that, we could tackle that is with virtual box or using QML. And there is a link from Microsoft that gives you ready to use VMs for different environments, virtual box or parallel or VMware, but you can use them right away to start machines with Internet Explorer and Edge. And the other one, the friends from IOCube, they also documented very well how you could actually put a Windows operating system inside a QML emulator and then put that inside Docker. This information has been available always around the Internet, you just have to Google it, but finally someone took the time to document it properly and put it into GitHub. So the normal rule here is that you need to build the machines, you need to build the images yourself, and if you want to do it properly, you need to buy the Windows licenses to have everything legally. Otherwise, if an auditing comes to the company, maybe we would run into legal issues there. And nested virtualization is needed if we're using the QML approach and I will talk about that in a few minutes. Now let's talk about Safari. You could also use virtual box or QML for that and these are links of GitHub repositories that you can use and they show you step by step and sometimes they provide scripts that you can just execute right away and you will have an OSX operating system running inside virtual box right away and you can do your testing there if you want. But there is also the third link that someone took the time to run OSX the operating system inside QML and for that you need the Linux operating system. Normally the thing that you need to take into account here is that the complete operating system is started so you need, in the same way as Windows, you need around two or three CPUs and you need around three gigabytes of RAM per virtual machine running. And the tricky part here is that different to Windows, the operating systems from Apple, the OSX, they are free to use. You don't have to pay anything for them but the underlying hardware has to be Apple. So if you don't want to run into legal issues, you need to buy a few Mac Minis, a few Macbooks to run that in your company. Having said that, we could have different layouts for the stack. We could have a combination of Linux as an underlying operating system with nested virtualization and nested virtualization is basically the feature that Linux has where you can run an operating system inside, sorry, a hypervisor inside a current hypervisor. And a hypervisor is just software that is able to manage operating systems. It allows to allocate RAM, CPU, etc., etc. And that's the tricky part when we are doing these environments. We need to have proper nested virtualization. So the stack looks like this. You have the host, either Apple or Linux. You can have your tools to manage the infrastructure like Puppet, Ansible or Chef and then you choose what virtualization container you want, what type of emulation you want. If it's Docker, QEMO or VirtualBox and then you put the browser on top. But you could also have different combinations. You could say I just want to use QEMO or Docker or you could say I just want to use QEMO. So the advantage here is that when you choose less virtualization emulation platforms the code that you need is less because if you are going to use Docker, VirtualBox and QEMO then you need to manage three different platforms when you are going to boot your machines. So that's an architecture decision that I leave it up to you. So normally what I have done is to either choose QEMO or choose VirtualBox because I have just one single thing to choose. The next part that is quite important is to build the images to be used for testing. As we build Docker images, for example, we also need to build either VirtualBox images or QEMO images and so on. So the first step is that we need a base image, either Ubuntu, Windows, etc., etc. We need to install the operating system requirements in each of these base images. So Java, if it's Python, if it's Selenium, etc., etc. We need the networking utilities and so on. And we need to start then the binaries that we're going to need to start running those tests. So Chrome Driver, the Chrome Browser. If we're going to start recording video, FFNPEG. If we're going to do testing in different languages, we need to start the language packages because by default only the western languages come. So if you're going to load a page in Chinese, it won't work. So you need to start all those things that are very important. Then we need to run some custom splits on the operating system because normally you would like to disable Windows Update. You don't want it running while you're running your test. You need to disable perhaps Siri in OS X because that will take more resources and more RAM. And you have to set some permissions so actually things work out out of the box. And don't forget about testing because we need to test the images before actually using them for testing. So we need to check that the elements are placed in the correct folder, etc., etc. We need to tag the image saying this is Windows, this is OS X, and so on. And we need to store them either in Docker, or in a registry locally, or in an Artifactory, and so on. So the problem here is that if we have these 10 base images, so Windows 7, Windows 8, the different versions of OS X and Ubuntu, if we use the 10 latest browser versions, let's say Chrome and Firefox 10 latest, then Safari and Edge 5 and 6, and then we can have the latest three of Internet Explorer. And let's say they're going to support the latest five-celling versions. So all these combinations give us around 1,700 images to maintain. So that's a bit too much and there has to be a better approach. So what normally is done when we want to build a test infrastructure is to build a base image that is in the upper part, and this doesn't change. You will always have OS X, Mojave, for example. And when we want to run tests, we just mount these binaries that are used to run the tests. For example, if we want to run a test with Chrome 75 and Selenium 3.141C59, we just mount them to the base image. And for example, in Docker, this is called Volumes. In QM, this is called Side Discs. In Virtual Box, I think it was called Mounted Drives or something like that. So every single technology has the same thing. So this can be used and it's all documented in each tool. Okay, so this was the largest part of the presentation that doesn't relate at all to Appian, but now we're going to take those concepts and go into simulators and emulators and afterwards into real devices. So emulators, we can just talk about Android and the needed steps here are we need to have Appian properly working, we need to set up all the Android SDK tools and we could use either real hardware or QM. My recommendation, as that was the thing we were doing with browsers, is that we use containers if possible, but for that we need nested visualization. There is a project that a friend of mine that we did, I helped him a bit when we were working in a previous company and it's called Docker Android. It's a Docker container that can run an Android emulator. The downside of that project is that you always need nested utilization. So we need always a Linux operating system or we need a tool like VMware to run it in Mac or Windows. We were using this Prometheus thing to measure the performance and normally around when the emulator was booting it was using around three CPUs and three gigabytes of RAM, but it could change depending on the app you're using. Also, there are some default images, but if we need to change things we could do some slight customization to it and I will mention that in the next slides. The challenge is about emulators is that basically there are too many combinations. All the Chrome versions, Chrome driver, if we want to use different Android versions like 7, 8, 8.1, 9, and if we want to use the 10 beta. Also, if we need Google Play services, for example, because it could be that the developer builds an Android application with a recent version of Google Play services, but the one that is in the emulator is older. So then the test will fail by the way. So we have to be in sync with the developer and understand what they are building currently. And the downside is that when we start an emulator, normally the Chrome version there is quite old, so that's something that we have to upgrade ourselves as well. And finally, this will be pure Android. So if our interest is to check how the application behaves in an LG device and in Samsung, those operating systems in those devices will slightly change, but testing on pure Android should be good enough to check that our basic features are working properly. There is also a way that we can overcome some of these challenges and thanks to this project there is already, well you cannot see it properly perhaps, but there are already pre-made Android emulators in containers from version Android 5.0 up to 9. And what Booty does in the repository, he just mentions that please be aware that the Chrome version there is quite old. So what should we do? If we want to upgrade Google Play services or the Chrome APK, there are two open source projects, someone took the time to understand how the Google Play API works and built a wrapper around that. So this person opensource at that and you can use these two projects, you can read the documentation and you could actually download newer APK versions for given Android and device configurations. So you can download them.  If we want to take a different approach, well we just extend the Docker image, we start the emulator, we install it, the APK, we shut it down and with that we can have a new Docker image that will be potentially bigger in size but it will be already set up for our tests. Simulators, there is a similar challenge in the way that we have to do all the setup in the same way we did it for Safari. So if you are able to set up a Safari running in OSX in a containerized environment to go to emulators is not that complicated. The only thing we have to do, the only thing is to configure Xcode which is already a big task but if we use Ansible or Puppet or Chef that shouldn't be that hard. The downside is always that we need hardware from Apple otherwise we could run up into legal issues. The challenges here are that we should always figure out the mix and match between the versions of Xcode, Appium, iOS and so on. But normally this is not so hard because since Apple is the only producer of iOS and the devices, normally this works in a much faster way and a much straightforward way. And the main challenge I have seen here is that it's really hard to tell like our manager or the higher upper management that we need to spend 10,000 euros or 100,000 rupees or whatever it is to buy a few Macs or a few Mac Minis to use for testing. So they always expect to give these machines to the developers like the cool stuff but it's very rarely seen that it's okay for them to buy these machines just for testing. So let's jump into the real devices. Normally setting up real devices is not that hard. There are some projects like the one offered by the Appium project where we can set up through USB connections a real device and a real device. We have to always be aware that they need to be charged the whole time and sometimes we also, depending on the device, we need to tweak the configuration of the device. For example, if you have seen an LG device, the LG, when you plug the charger, there is a pop-up saying, Torbo Connect charger up to like, will be ready in 20 minutes. So this pop-up, if it happens during a test, that will break the test because it's an element on the screen that somehow Appium could bump into. So we need to check all these little features that each device has and it could be like a feature of the operating system but that could break the test. Also, we need to figure out ways of avoiding the screen going to sleep. And for iOS, it's slightly easier. As I said, there is less device fragmentation so the plug-in to the machine should be more simple but as well as simulators, we have to need, we need to have Xcode configured, Appium properly configured with all the dependencies installed for real devices. And this is the hard part. The first part, the initial setup is quite simple but this is the hard part because we run one test and we run two tests and we run five tests and eventually there will be some leftover files in the devices that could affect future tests. If we're creating, for example, images or downloading stuff, then when we are going to do a check on that, that could be a conflict. We need always to find a way to clean the device after the test runs. We need to potentially remove the application. Normally, Appium can do it if we pass some capability. We can also clean the temporal files with some scripts depending on the operating system but one important thing is that the devices will have queues. So when we add a new device into the grid it could be the latest iPhone. Everybody would like to test in that one so if we have bigger teams, if we have different 5 or 6 teams there will be a queue for that resource. And going back to Android, one of the main problems is that there is a huge device fragmentation. For example, in Germany it's okay to test in Samsung and maybe in some LG but if you go to a different country if you have customers, for example, I don't know, I think in China the popular phone is not what we have in Germany and so on. So those are the issues that we need to be aware and that's what I've seen when I had the opportunity to visit the data center in SOS Labs. Sometimes we have to buy devices that you cannot find in Germany but some other people want to test. So that's the challenge that we have to do when we want to build our own grid. And the scaling up, normally it could happen that when you have an Android device or an iOS it could happen that after many times of using it you will have the device that looks like this in this vertical way but the more you use it the battery will explode and will end up like this, like bloating up. And this is a normal thing because phones are not made to be tested over and over again. So the solution for this is like recycle them, throw them away and take one from the drawer. That's the only solution you have. We always need to have extra devices ready to be used when the other one dies. And in the end, to be honest, what I have found after building infrastructure for a few years is that it's mostly a test strategy question. So when we face the challenge of saying, okay, we need to build mobile infrastructure, then let's ask a few questions and say, do we really need it or do we just want to do responsive design testing? Like see how the application looks in mobile devices. Let's see that we just have a web application and we just need to see how it looks in mobile devices. Then we could use the Chrome emulation for that. But if we really need to do compatibility testing, so see how our application works in a Samsung device, in an LG, in an HTC and so on, then probably we need the devices. Sometimes we need to test how the application works in several devices. So some devices have more memory in RAM, some devices have more CPU and so on. So that's a valid case to use real devices. And sometimes, for example, the emulators we find that can be used for testing in a simple way, they are not able to test the native ARM libraries. So we need to have the real devices that provide those features. And in the end, if we need to test how it works for a Samsung device with their plain operating system or other devices, then it's a real use case to use real devices. So let's go into maintenance. And this is perhaps one of the shortest parts of the presentation, but in real life, this is where we spend most of the time. The first thing is that we need to keep an eye always on the releases. And that was the first thing that I found out when I started working in infrastructure. When I had my first Selenium Grid back in 2015, we were never checking when the new browser was coming out. We were never checking when the new Chrome driver was coming out. So we just forgot. And then tests were failing. So we have to keep an eye on the repositories. We have to keep an eye on when is the next app being released, when is the new phone coming up. If we have a compromise about time to market, if we need really to test in the latest iPhone, we need to find a way to buy the latest iPhone even before it's released to the public. So it depends on what we can do and get those devices ready. So what I have found is that if we have time to market pressure, we need to have all these updated things really fast, like Android, iOS, Chrome, et cetera, et cetera. But sometimes we can wait a bit for Appium in the sense that if there is no feature that we need for this new release, then we could stay like maybe hold it for the next release and test it properly. Then what is really important is that we need to test the test infrastructure before offering it to our users. And that's the first thing that I learned many years ago that we need to test our new versions in a separate environment. And that's why even if we are now working in test infrastructure, we need to write tests also to test the new devices, the new versions. If there is a new feature and so on, we need also to test the features we have in our infrastructure. For example, if you're recording videos, we need to run a test to actually verify that the video is being captured. And also, I recommend like every two or three months run some performance testing for the emulators because many times it happens that Android releases something new and the performance is way better than the previous one, so we need less resources. But it could go the other way around. It could be something that is using many more resources and it could just use all our machines. We end up using or having less capacity for parallelization. And sometimes it just doesn't work. There could be an app in release and there is a bug and we cannot do anything. We just need to read the release notes. We have to understand. We have to report the bug. We should report a way to reproduce that bug. And also, we need to learn how to build from source code the infrastructure. If we are using Selenium Grid, we should learn how to build it from source code. Same thing for Appium, understand how the structure of the project is because in that knowledge lays down the good service of the infrastructure. And one important thing is that since we're keeping an eye on releases, let's try to test the betas. That's where we will have what could break or what could work in our infrastructure. Let's talk about the scoping. So we have spoken about all the things that we can do to build an infrastructure. All the tools that are out there. So there are no secrets. Everyone can find the tools that they need and you can build your own infrastructure. But the important part is that before even downloading tools and building stuff, we need to scope things so we actually have clear expectations of what is needed. So the first thing is the coverage. Always ask, why do we need mobile? So question the people who are asking for mobile because if you ask anyone, so what browser versions do you want? What devices do you want? So they will tell us I need everything because I need to test and be sure that it works in all combinations. And that is not a good answer because then we should ask them like do you need to do responsive testing? Why do you need real devices? Do you want them for just automation or do you want to do exploratory testing? And exploratory testing is a really good use case for real devices because people would like to get the device and check the application if it looks fine. Some things are not easy to catch by an automated test or sometimes people just would like to have a device and jump into the bus and see how it works in a real use case. So if the connection is good or not. And for that we don't need to build an infrastructure. We just need to buy a few phones and that's it. What about emulators? Normally everything is shifting left. Therefore if we want to do that emulators are a good option to test early in the pipeline because we can spin them as much as many resources as we have. And one important question is that if we are serious about starting testing infrastructure if it is a long term solution. If it's a long term solution I think it's wise to invest time to research and build it properly. But if it's something that we're going to use like a proof of concept in the next six months I would say that you could just build something small but not think about scalability and future features and things that could maybe take time from you that will not allow you to have something working right away. Also we need to ask how many people we want running the tests how concurrent they need to be. So for example if we have a team of five people one single team including the CI. So we could assume that each person will use one slot for the concurrency and maybe the CI should use four or five. So with that we can dimension how many phones we need how many servers we need and so on. So I consider this setup something that is low or middle effort that you could manage without focusing entirely on it but something above that I would recommend that either you talk to your manager and say then if we're going to do this this should be my task only I cannot do anything else or to say you know this is really hard I don't like this maybe we should send someone to do that and allocate the proper resources for that. The next thing is about the quality gate. So the quality gate is if we want to put our infrastructure in the middle of the pipeline. So when we run tests when we are doing a pull request if they all pass we merge the feature and then we release to the public. So if the pipeline is in the middle it's critical because if the infrastructure is not working then we cannot release. So if we think about that how important the infrastructure is in the pipeline then we can also decide how much we need to invest or not.  So are you in this company to be a tester or to be a DevOps or what's your role? If you would like to try it I would say go for it because it's really fun to build infrastructure to try it out and to see the challenges by yourself but if not if you really love testing I would advise to push back into this because this will take your focus and your time from it. That's what I did, I mean I couldn't really do more testing I just had to focus on infrastructure but in the end I loved it and this is what I like to do. And I will not talk about price because many variables are different but I would just say that we need to take this into account we need to take into account a person on the devices all the time we need to buy machines for iOS so like Mac mini's or so on we need to have machines for Android as well we need to have something like a small data center the phones cannot be like where the heat is they need to have some air conditioning and good features and good protection they need to be like in a lock or something like that and we need to buy the devices as well. So let's go to the final part. So always take this into account please that there is a lot of work to do it's like so much work that it could be for an army and that was kind of one of the reasons why I joined SOSLO because I really love doing infrastructure but I don't want to do it by myself the whole time I would like to actually take a vacation I would like to actually do I don't know research hackathon something else and that's because it's a lot of work this is super important whatever you do infrastructure as code is a key component and one important thing is that doing test infrastructure is harder than normal infrastructure because there is nothing else built that has to be booting a real operating system to be used for testing and then being thrown away so it's harder than that and it's a continuous task so we should not think that we just do the initial setup and that's it the maintenance goes on and on because I see this as like establishing a long term relationship when you want to get married when you want to start a new job or so this is the amount of hours I have spent working since March 2015 in infrastructure and this is around 1000 days up to the day coding, training people, fixing bugs and features and contributing back to open source and this is around two years, nine months where I have been working on this including Saturday sometimes but normally Monday to Friday and with that I just want to tell you that if you want to try it you'll have my blessing but be aware that this is a lot of work but it's also a lot of fun so just give it a try and if not then we're back in the open source community to give you advice and if you need any help just let us know and that's it for me I just wanted to thank you all for coming and if you have any questions and we'll transfer them thank you Thank you Diego we have around a minute for questions and after that I think we can catch Diego on the hallway track any questions for now? There is a question Have you tried Intel's HAXM configuration with QEMU? Quick emulator Can you repeat the question please? Intel's HAXM Yeah Have you tried Hypervisor? No, I haven't tried Hypervisor because my selection was to use QEMU since it could work since I could spin all the environments I need on Linux so that was my choice back then it was not so mature as it is now Yeah, that's the reason why Alright Okay I want to ask that I saw that you have a library that able to install the APK of the Google Play service for the new version Is it by default you check the version or by default you install all the new Google Play service? What I was doing was to wait until someone reported that it was not working because it was not a common use case but when it happened it was very annoying So I was waiting until someone said my test is not working because the Google Play service version is higher than the one so in the app it's higher than the one in the emulator so we just have to build everything again Okay And do you have an average time how long it takes when you launch an image and booting up until it's ready for emulator and simulator? For emulators it normally takes around one minute sometimes 40 seconds those are like the times I have seen lately with us in Solstice but they are ready to use that's also one thing that we pre-boot things and they are ready to use so when you want to run your test it's already done but normally it takes about 40 seconds and the simulator since we are booting up the whole OSX operating system that takes about one minute or a bit more I think we have time for one last question So we are using one Selenium grid of five VMs basically they are the window VMs and each VM has eight gigs of RAM and we trigger our test suite with four instance of browser one browser that's it we are not targeting everything at one time so four instance on each VM and every time the execution time of the whole test is somewhere around eight to nine hours so they execute continuously so every time it is very consistent that 10% of the test case fails just because of the environment issue and they are passing locally every time so it happens like at least for 10% of the test cases how we can fix that? The approach I have always taken is to run one browser per container or per virtual environment so if you are running four browsers per machine what I did was to run one browser per machine so that VM I would break it in two so four CPUs or three CPUs per VM and run one test per VM I know that this reduces the amount of concurrency that you can have It will take now 36 hours but the tests are more stable but that's the question we need to ask ourselves we need magic with just a couple of VMs we need to go to the management if you want faster releases we need more VMs Then what's the advantage of having Selenium Grid? Like that's the only purpose you don't want to execute sequentially you want to run in parallel Exactly, but it's a matter of resources and I was lucky I had good management in the sense that they gave me more resources but we cannot do magic with just a couple of little boxes Thank you