Greetings from the National Archives Flex, a building in Washington, D.C., which sits on the ancestral lands of the Ndukatshtang peoples. I'm David Ferriero, Archivist of the United States, and it's my pleasure to welcome you to today's conversation with Megan Kate Nelson about her new book, Saving Yellowstone, which looks at the fascinating and complex historical context behind the establishment of this highly popular national park. Before we begin, I'd like to tell you about two programs coming up soon on our YouTube channel. On Tuesday, March 8th, at 1 p.m., Mary Sarah Builder will speak on her recent book, Female Genius. Builder introduces us to Eliza Harriet Barron's O'Connor, a pathbreaking female educator in the 1780s. She argued that women had equal capacity and deserved and equal education and political representation, and her University of Pennsylvania lecture was attended by George Washington. And on Thursday, March 17th, at 1 p.m., historian Laura F. Edwards will discuss her new book, Only the Clothes on Her Back. By studying the clothing of ordinary people, Edwards tells us we can uncover the hidden history of power in the 19th century United States. This month marks the 150th anniversary of the establishment of Yellowstone National Park, the first federally protected national park. On March 1, 1872, President Ulysses S. Grant signed the Yellowstone National Park Protection Act into law. This legislation described 3,472 square miles of wilderness in the Montana and Wyoming territories that would be dedicated and set apart as a public park or a pleasuring ground for the benefit and enjoyment of the people. Among the many records related to Yellowstone in our holdings are some of the earliest photographs taken by photographer and explorer William Henry Jackson. Seventy years later in the early 1940s, another celebrated photographer, Ansel Adams, captured the grandeur of Yellowstone in images. These pictures and Jackson's have been digitized and are available in the National Archives online catalog. As today's guests will explain, the establishment of Yellowstone National Park occurred at a crucial time in American history amid the nationwide turmoil and racial violence of the Reconstruction Era. Joining Megan Kate Nelson in conversation will be Andrew R. Graybill, director of the William P. Clements Center for Southwest Studies at Southern Methodist University. Megan Kate Nelson is a writer and historian living in Lincoln, Massachusetts. She has written about the Civil War, U.S. Western history and American culture for the New York Times, The Washington Post, Smithsonian Magazine, and Civil War Monitor. In addition to saving Yellowstone, Nelson is the author of The Three-Cornered War, Ruin Nation and Trembling Earth. Andrew R. Graybill is a historian of the North American West with a particular interest in continental expansion, borders, race, violence, and the environment. He is the author or editor of four books, Policing the Great Plains, Bridging National Borders in North America, The Red and the White, and Civil War Wests. Now let's hear from Megan Kate Nelson and Andrew Graybill. Thank you for joining us today. Hi there, Megan. Hey, Andy. How are you? I'm good. It is so fun to be your interlocutor today, so thank you for pulling me into this program and thanks also to our friends at the National Archives for putting on this program. So we're going to begin with a two-part question. First, can you tell me a little bit about your career path to this point? And then second, how this project developed all these interesting prior interests of yours? Absolutely. Yes, and I would like to echo my thanks to the National Archives, not only for hosting events like this, which is amazing for historians to be able to come on and talk to people across the nation hosted by the archives, but also for everything that they do for historians and researchers and keeping all of our national documents safe and available to the people. So I went to graduate school at the University of Iowa in American Studies. And what followed was about a 12-year career in academia. And as many, if there are folks out there who are in the academy or they know people who are, they know that you're often making decisions about your employment alongside decisions about family, about where you want to be geographically, whether you want to be near family, near landscapes that you love, et cetera, et cetera. I don't think we're alone in that in academia, but it really shaped a lot of my decision making and I had a series of tenure track jobs and then decided to leap off of the tenure track, Four Family Reasons, in 2009 and put together some adjuncting and continue to work on my second book, Ruination. And then in 2014, it just became clear to me that I wasn't really going to get another tenure track job in the field that I wanted and a situation that I wanted. And I had this idea for a book about the Civil War in the desert southwest and I thought that it had trade book potential. And I think for a lot of academics, and I don't know if this was true for you, Andy, but you think about all of the amazing things you get to do as a professor, right, as someone involved in this work. And you get to write and research and teach and advise and administrate and do all of these different kinds of things, which is why it's such a compelling profession. And I really decided at that moment, just around 2014, that I really liked writing, most of all, and researching. And so I, well, how could I possibly do that? And decided to give about two years to the project that became the Three Cornered War, to do some research, an initial research trip through the southwest, archives and libraries and historic sites and parks. And, you know, trying to then figure out if I could write a book proposal that I could sell to an agent and actually get paid to write a book, which, you know, we know in academia, you don't make much money off your books in academia usually. So if I was going to be, if I was going to be a writer professionally and sort of take myself seriously and describe myself as a working writer, you know, that was really important to me. So I did that and I did. I managed to get an agent and to sell the Three Cornered War to Scribner. And that was kind of how I began kind of living my life as a writer and a historian, kind of out here in the world. I might add the Pulitzer Prize finalist, Three Cornered War, for that matter. So how does saving Yellowstone maybe come out of the work you did for that book or does it? Oh, it absolutely does. So I was finishing up writing Three Cornered War, and I had been writing the final chapters for John Clark, who is a protagonist in that book, who is the surveyor general of New Mexico Territory. And so I had done all this research into the history of surveys and surveying in America. And I ran across, you know, the Fernand Haydn's 1871 survey and realized that it had led to the passage of the Yellowstone Act and that the 150th anniversaries of both of those events were coming up. And this was around 2018 that I was starting to think about this. And, you know, for historians, and I think for Americans in general, anniversaries are really interesting moments. They're important moments for us to really consider why something happened in the past in event or the significance of a landscape in that moment and then why it's important today, right? And to kind of have a reckoning about this. And so I thought, well, that's a good reason to write a book about Yellowstone and its exploration and preservation. And then I had that revelation, which, you know, probably shouldn't have come so late. But the revelation that this was all taking place and right in the middle of reconstruction in 1871, 72, this really pivotal moment. And, you know, there are so many great books written about Yellowstone, about that situated in the context of conservation, in racial violence, in, you know, history of the National Park Service. And I thought, well, what kind of story could I tell that would be new? And it's like, well, what would happen if I looked at reconstruction from a really unusual place like Yellowstone? I mean, I had just written a book, The Three Cornered War, about looking at the Civil War from an unexpected place, the Desert Southwest. And so I thought, well, what if I did that here? Like, would it tell us something different about reconstruction? And would it tell us something different about Yellowstone? I want to come back to that reconstruction era context. There's a great story that you tell in the acknowledgments about how your research at the National Archives branch in College Park, Maryland, was interrupted all the moment in March of 2020. So can you describe that experience for the audience? Maybe then explain the implications of the COVID-19 shutdown for your work in a most unusual context on the Yellowstone? Yes, it was actually, you know, right around two years ago this week that I arrived in D.C. I was doing research at the Maryland College Park campus of National Archives, NARA II, and was looking at some really great documents there related to the Hayden Survey.  But yeah, I had on Thursday, I think, you know, we were hearing news about the pandemic, things were starting to happen. I think the real I got a real sense that things were going to change when the NBA kind of shut down their schedule. I was like, oh, OK, this is for real. And I was in the microfilm room because I had to look at the Hayden records on microfilm because they're too fragile to look at and have been digitized. And an archivist just kind of stopped by my table and said, get whatever you need done, done because we're closing tomorrow. Tomorrow is our last day. And I had already booked. I was supposed to be there for another week.  I was like, and I like I took all the photos that I could of all of the the microfilm material that I had left. And then I was just if you've ever been in NARA II, you know, I was just running to the to the elevator bank and going up and down floors, submitting slips in the artwork section and in map section. And I was running around trying to get a look at as many documents as I could before I had to leave. So I mean, I think most researchers know those last couple of days in the archive are always a little frantic, but this was especially so in that context. But I did get to look at so many interesting documents, why I was there. And so I really got a lot of bang for my buck out of that that one week. And and if you to allow me, I would like to show everyone some of these documents because they really are great. And and one of the benefits of doing a project where you're working with federal records is that so many of them are saved and available. And I had done some great research at the National Archives in the D.C. branch for three cornered war actually in the John Clark for the John Clark section, because he had sent so many letters back to the General Land Office. And so there was just this giant carton of all of his letters, which were in pristine condition, which was amazing. But, you know, this. So when you're when you're dealing with federal officials, so many of their records are saved and are available, which is wonderful. So so these were the major three record groups that I got a hold of while I was there, all having to do with the Hayden surveys, all in record group 57 and all available for me to look at the records of the USGS were, of course, in microfilm. And then the maps and the artwork I actually got to see in person, although you'll see that they had protective coverings on them. But, you know, it's just a delight whenever you're doing historical work in the archive to see like I took this picture of just Hayden's signature, right? Just to know, well, you know, how he was signing his name, the little flourish at the end. And that it's so Hayden when it, you know, if you haven't read the book yet, when you start to get to know him, he's such an interesting character, really ambitious, really kind of aggressively social. He's a very good lobbyist for that reason and always about kind of emphasizing himself and his work. So he was a great person to research anyway, but it was just nice to see his handwriting and to see the letters going back and forth. So a lot of what I was looking at was from the section of these records that were the letters received by Hayden during this period, when he was thinking about going to Yellowstone and when he was lobbying for the act. And so I found this great letter from Henry Dawes to Hayden on December 8th of 1871 on the House of Representative Committee of Ways and Means Letterhead, which I absolutely love. Dawes was one of the most powerful people in the House at this moment, a Republican from Massachusetts. And Hayden had taken his son, Chester, on the Yellowstone expedition with him as part of this group that I call in the book, the political boys. They were all this kind of group of outdoorsy young gentlemen who were the sons of important movers and shakers in Washington, including, you know, Chester as the son of Henry Dawes. And Dawes had always been a supporter of Hayden's, and he had voted to give him $40,000 to fund the Yellowstone expedition. And what's important about this piece, and it's basically just Dawes saying, sorry, I knew that I was supposed to be in my office, but I wasn't today. You know, it's the beginning of the the season in D.C. and I'm really busy, but I will be there tomorrow. It's sort of indicating to Hayden that he can drop by and see him. And this is in the week before the Yellowstone Act was presented to Congress and introduced as a bill. And so here is, to me, proof that Hayden is lobbying congressmen in order to try and convince them that they should save Yellowstone as a national park. Dawes was very much amenable to this, very receptive. And so I used this exchange to kind of build a scene in the book where Hayden goes to talk to Dawes and to really use his influence to spread the word about the Yellowstone Act and to lobby him directly. So that was a great piece of evidence to see. And then there's also after the act was passed, just almost a week after there's this, you know, kind of short letter from a Kansas senator, Alexander Caldwell, who wrote to Hayden and asked him if he could have some of the views of Yellowstone, which are as we heard in the introduction, the National Archives holds a lot of William Henry Jackson's photographs. And so this is when he's talking about the views of the Yellowstone. That's what he's talking about. And Hayden had used many of Jackson's photographs and then a couple of Thomas Moran sketches in a little exhibit he put together in the rotunda. Including minerals and some fossils and other things. Also as part of his lobbying project. And so Caldwell would have seen the images there. He also probably saw illustrations taken from those images in the report that Hayden submitted to Congress in February of 1872, right before the House voted on the on the act.  And so this is after they've already passed the act. But there's continued interest in Yellowstone and in seeing images of Yellowstone. So the relationship between Hayden and all of these politicians continues on. And he's actively lobbying them in this moment, too, to return to Yellowstone to have another survey that goes out with an even larger team. And they, in fact, did vote to give him seventy five thousand dollars, which is just an incredible amount of money. So he was successful in that, too. So so this was a nice document for me to see around that whole continued discussion of Yellowstone and the importance of those visual images, because I think, I mean, you know, Andy, like in Western history, whenever you're talking about 19th century, the 19th century West, so much of what Americans understand about the West is coming through visual culture, is coming through paintings and illustrations and photographs. And this is how people and I think this is how we still, if you haven't been to Yellowstone, you probably know what certain features of it already look like because you've seen images. So so that was a really great kind of moment for me in the archive. And speaking of images, I knew that there would be in this collection because I'd already read about them a series of letters from Thomas Moran, the painter to Ferdinand Hayden. Moran had been sent to the survey by Jay Cook, who was an investment banker in Philadelphia. Moran was also from Philadelphia, so they knew each other from that scene. And Cook wanted Moran to produce a lot of visual images of the Yellowstone that he could use to promote the Northern Pacific Railroad. And so so Moran joined in July and immediately began really kind of traveling and working with William Henry Jackson. They became pretty good friends and mess mates.  So I knew that there were these letters and in this moment, and this is also after the act was passed, Moran was finishing up what he was calling his big picture, which is the epic, huge painting, the eight by 12th painting, the Grand Canyon of the Yellowstone, which is on the cover of the book, which I'm delighted about and readers of Saving Yellowstone will actually kind of get into the nitty gritty with Moran and the painting of this picture and its exhibit. And he's writing to hate in this entire time, kind of asking for some money for a frame. He's asking for all kinds of things. And in this particular letter, he writes to him on March 11th. He asks him for a photograph. He asked Hayden for a photograph of himself because he wants to insert a couple of people into the foreground of the painting, and he wants one of them to be Hayden. So he's like, if you have a photo, you know, I would thank you very much for it. And, you know, I've read about this and and people have quoted this letter quite a lot in discussions of Moran and Hayden and the Yellowstone. But when you see it in person, what you notice is about halfway down the page on this right side. Moran has actually said, I would like something about this size, right?  It's just and it's and it is. It's like it's tiny in the letter. And and Hayden's figure, which is actually turned, his back is turned so that the front view is of no use to him in the end. But they are tiny. Those figures are very tiny in comparison to the gigantic landscape of the Grand Canyon of the Yellowstone. But this to me and Andy, I'm sure you've run across this in your own work, too. When you look at these documents, you can see all sorts of things like people doodle in the margins. They write in a specific way, they double underline or they do really interesting things in their letter writing that often don't get translated in a transcription. So this is to me one of the benefits of seeing documents kind of right there in front of you, because you really get that sense of their materiality and that tactality. So I knew when I saw this document, I knew that I had to include that. And in fact, it is a detail that made it into the final copy of the book. Moran is is drawing this little thing to tell him what he means.   So once I left the microfilm room and was running around like a crazy person in that in those final days in the National Archives, I was going to the map room because the National Archives has a series of the maps that were created from this survey. And so it was really great to see these. And I was very happy to find this particular map of the Upper Geyser region. And the map is actually created by another topographer, but it was taken from notes that were made by Anton Shamborn, who was the topographer along with the survey, one of Hayden's favorite topographers. He thought he was a genius. He thought he was a remarkable artist and scientist. And he had been working for the Army for a while and based in Omaha. And Hayden had taken him taken him along on a couple of trips already. And he he made it. Actually, Shamborn was on his list of one of the top couple of people that he wanted with him because of his talent at creating maps. And we also know how significant maps are in shaping the way that we think about places and shaping the way we think about definitely landscapes and sort of unknown places are previously unknown places in the 19th century. And Yellowstone was in 1871, one of the last unmapped places in the continental United States. No one really knew what was there. So it was really important to Hayden that he produced maps. And in fact, his charge from Congress included map making as part of it. So I knew I wanted to look at those maps as not only evidence of that, but also evidence of Shamborn's contributions. And I won't spoil it for you. But if if you have not read the book, but Shamborn has a sort of a different path that he takes. And so I was interested to see if his work would actually show up in the records of the Hayden Survey. And they did, which was really fantastic. Another artist who is along on the expedition, who doesn't get a lot of attention was Henry Elliott, who was a sketch artist. And he had also been a guy who Hayden had known for a little while, had been on a couple of surveys with him before. And he was in charge of doing the kind of large frame sketches, the big kind of not nothing that would turn into a painting, but that would be useful for Hayden in writing his reports, you know, very long, very detailed views from the summits of mountains sort of orient the viewer. And a lot of his illustrations and sketches ended up in Hayden's report. And the reason I was delighted to find this and this is in the Hayden artwork collection is that this is a view from immigrant peak, which is in the Paradise Valley, which is north of Yellowstone from August 28th, 1871. So this is after the team has actually left Yellowstone Basin. It was very important to Hayden that he leave Yellowstone before the snows came, which often came in the early part of September. And he knew that was going to be the case and he did not want to get trapped there. So they were actually out of the basin already. By the end of August. And this this moment doesn't actually make it into the book. And, you know, as historians, we have a lot of these things. We have a lot of evidence that doesn't make it. But I loved seeing this because I had and this is his sketch. And it's actually a very long piece. And you probably can't see on my hands on the zoom, but it's a very long piece of paper. And I had descriptions from other records where Elliot would just take these big roles of parchment and then he would just lie down and just start sketching across the whole thing. And so there is a scene where he's doing that on Mount Washburn in the book. But he does it again on Emigrant Peak. But the only person there with him is Albert Peele, who is a graduate of Penn. He was a student of Ferdinand Hayden's and a quite talented mineralogist. And he kept the most amazing diary of this of this expedition. And so he has this this moment. So this is his diary entry from that date, from August 28th, detailing how he and Elliot got up and they decided they were going to climb this mountain. And people said that they couldn't do it, right? And and Albert Peele has this moment later where he writes about how hard it is to be on this survey, that it is actually physically taxing to go do this work because you are walking through the landscape, you're collecting, you're stooping over, you're climbing mountains. You know, when they get to the Grand Canyon of the Yellowstone, they're they're actually lowering themselves into the canyon and then hiking back out. And if you've ever been there, you know how deep that canyon is. It is very deep. So no one could believe that they wanted to go and climb Emigrant Peak after coming out and when they were supposed to be resting and they were supposed to be eating. And, you know, basically the work of the expedition is done. But Peele and Elliot went up there together. And I just love this description because, you know, he's saying, Elliot reached the top long before I did. So he was clearly a good Mountaineer. And then after Elliot had finished his sketching, we commenced the descent. So there's the reference. And when I found the actual sketch, I was just delighted. It's just one of those moments as a historian that you really think is really cool. And just to give you a sense of how high this mountain actually is. This is Emigrant Peak. Yellowstone is sort of off in the distance there on the far right side. And the Yellowstone River is kind of going through this valley. So they were encamped at Butler's Ranch, which is over kind of to the right.  And you can see this is from mid September 2021 when I finally made it to Yellowstone after being trapped in my house for two years writing this book. And and there's already snow up there. And so it's an Emigrant Peak is known as a challenging peak to climb, you know, even today. So here are these these surveyors going and doing this. And it was just a nice moment in the archive to kind of bring these sources together. So I'll stop there with the share. But yeah, I mean, it was just such a it was a very rich time in the National Archives, but then I had to go home and. Do basically write the bulk of the book from my living room using digitized archives. So thank goodness also for all the people who have worked so hard to digitize, you know, the Congressional Globe and all of the documents that go into the Congressional Serial Set newspaper, the people for newspapers.com, you know, all of these institutions and businesses that have done this really made it possible for me to do that research. Hmm. Fascinating. Yeah, a book written entirely from home. Our profession is pretty solitary, but that sounds like taking it to a different level entirely. So you talked about the fact that there are a lot of books about Yellowstone. There are. Yours is a propulsive narrative. It's a real joy to read. I highly recommend it. But let me put it to you this way and maybe in the scholarly parlance. What's your what's your argument? I know that that is kind of in some way second as perhaps it should be to the narrative flow of the book. But what what do you see as the main argument of through line of your book? So the book really argues that when we look at reconstruction from Yellowstone, we understand that it was a national project. You know, most of the time we think about reconstruction and we we teach about reconstruction as as a solely Southern experience, right? That it's the military occupation of the South bringing the South back into the Union politically, which only happened fully by in 1870. And then also the rise of the KKK and the federal government's attempt to to really stamp them out. And that, you know, that history is incredibly important. And it and it should be taught and it should be taught more, actually, than it is now. And we should know more about it than we do currently as kind of a general American public. But I think what tends to happen there is that that chapter on reconstruction is is the chapter on reconstruction. And what comes after is the chapter on Westford expansion. And so the stories are not ever really told together. And when you do tell those stories together, you see that that there are federal projects going on to extend federal power on the part of the Republican Party into both the South and the West, and that it creates a lot of interesting tensions, right? That they're actively trying to protect the 14th and 15th Amendment rights of Black Southerners in this moment in 1871, 72. But at the same time, they're completely denying the citizenship rights or even the citizenship potential of native people in the West. And that Republicans could. Completely embrace both of those projects together. I think we think now that that might be contradictory. But in that moment, they didn't believe that that was contradictory at all. That was part of a larger Republican project of creating this this empire of liberty and an empire of of white and then also some limited black settlement across the country. So you build your book around three protagonists. You talked a little bit about the geologist, Ferdinand Hayden. I'd love to hear a little bit more about him if you care to share the railroad magnet, Jay Cook, and then, of course, Papa Lakota. Holy man, sitting bull. Can you take a moment and describe each man and then talk a little bit about how you found the three of them to be sort of the the ideal protagonist for your story in some ways? I think that in describing them, you'll answer that second question. But take it away. Sure. Sure. So Ferdinand Hayden was an obvious choice to be a protagonist in the book. He was the leader of the survey. He was the the main lobbyist for the Yellowstone Act. And he's also just a really fascinating character born into poverty in a family. You know, his parents got divorced when he was quite young. They did realize, his parents, though, that he was quite smart and they managed to send him to Oberlin for his education. And that's where he really discovered a love for science. It took him a little while to find his way, but he really liked he discovered being out in the field and being a geologist was one of the ways you could do that. That was one of the kind of field sciences where you could go out and be in the landscape and do this work of collecting. And it turned out that he had a really good eye for fossils. Like he could he could look at a piece of rock and he could not only find the fossils within it, but he could determine how important they were. And so from the very early 1850s, he was already going out on collecting trips and had determined that he was going to make a living and a reputation for himself as a scientist. And he started working with both. Well, they were at that time military surveys sent out by the U.S. Army into the West. And then he lobbied to get control of his own survey for the state of Nebraska in 1867. And so by the time 1871 rolled around, he had been appointed U.S. geologist. He was fairly well known. This was in the time before the U.S. GS had been created. So if you were a surveyor like Hayden or like John Wesley Powell or Clarence King, you had to lobby every single year to get funding. So these guys had to go out during the exploration season, do all of their work, write it up and then lobby to get money and then turn right back around and get their team and go go back out into the West. So that whole that whole business of exploration and the study of science in this moment really intrigued me. And and Hayden was a little bit of an outlier. Like he was a little brash, you know, he came from nothing. A lot of the scientists during this period had money. They were they were elite men who were working with elite institutions, and he was just this kind of a man on fire, right? Like he just wanted to establish himself and he was a little obnoxious and very ambitious. He was a great leader of men. He ran a really great survey team. Everybody loved him. And he ended up being actually a pretty interesting writer of popular science, which very much helped him establish his reputation and to preserve Yellowstone. And he drew the attention of Jay Cook because Jay Cook also appreciated someone who could write well and who knew the power of PR and marketing and lobbying. Jay Cook had made his fortune in the Civil War, selling U.S. war bonds to fund the war effort. And really during reconstruction, he was searching around for a project that would give him that same sense of patriotic fervor and purpose, but that would also make him a lot of money. Right. After all was the goal. He ran his own investment bank called Jay Cook and Company. And he in 1870 took on the the job of financing the Northern Pacific Railroad, which was meant to be the second transcontinental. And it was meant to it was called the centennial line. It was meant to be finished by 1876 in celebration of the nation's centennial. But he was having problems already trying to get that line built. It barely reached the Missouri River from the Great Lakes, from Minnesota by the summer of 1871. And so he had a great interest in what Hayden was going to discover in Yellowstone. That's why he sent Thomas Moran to go join the survey because he wanted visual images that he could use to promote the railroad. And he really wanted to, you know, the track was going to need to go through Montana Territory, you know, and across Idaho and then and then into either Washington or Oregon. And he had planned that those tracks would go through what is now Livingston, which is about 50 miles north of Yellowstone Basin. So he thought he could bring tourists there. He thought if people were introduced to this area, then he could bring actual settlers there to create businesses and that it would help his his railroad flourish. What he did not anticipate was the resistance from Citing Bull. And when Jay Cook sent out survey teams in the fall of 71 and then also again in 72 and 73, Citing Bull and his people pushed back. And as William to come to Sherman put it later, they fought every inch of the line, right? They were not about to let the railroad go right through their homelands, which extended from the Missouri River to the Yellowstone Basin. And they knew as well as anyone by this period in American history, they'd seen the transcontinental line go in and they knew that the railroad was just going to bring more white settlers, right? This was the railroad was a mover of settler colonialism. It was going to bring all of these people. And they were going to be forced to either fight them or give up their lands. And Citing Bull really became in this moment a voice for resistance among the Lakota people, who's not the only voice. There were other chiefs and other leaders who were more amenable to making peace agreements. Red Cloud was one of them. But Citing Bull really emerges in this moment as the voice of resistance. And you can see when you read the records of territorial officials and Indian agents, all the people kind of dealing with this region, sent back to the to D.C. Citing Bull's name just kind of populates those records in this moment. He kind of emerges into the American imagination right now. And so one of the arguments of the book is that the road to Little Bighorn really starts here that it's his resistance against Jay Cook's project that kind of hones his message and brings him more allies who are determined to keep white settlers and developers and business people on Railroad magnets out of their traditional homelands. So I really felt like those three men, yeah, they're all around the same age. Cook is the oldest, Citing Bull's the youngest. They're all staking their claims in the Yellowstone at this moment, and they all believe that this place can kind of do something for them and for their communities. And I think those three vantage points, the vantage points of science and then capitalism and indigeneity are three of the main when we talk about the 19th century West, we are almost always talking about those like three components, along with the federal government and Ulysses S. Grant kind of helps situate us in that moment too and in that context in the book. Those are the major sort of forces that we're talking about in this region during this period. So it's interesting because you describe these three men as all having some specific and in some cases overlapping interests in the area. But my understanding is that ironically had the Yellowstone area held any prospects for mineral wealth or any sort of any other sort of economic development. It almost certainly wouldn't have been set aside by the U.S. government as a national park. So can you discuss that? Maybe we'll also reflecting on the role of the federal government more broadly in the post Civil War West, which the historian Richard White has memorably described it as, quote, the kindergarten of the American state. How do we think about all of that? Yes. And in fact, this was one of the arguments that Hayden made and that also Henry Dawes made when he got up to speak for the act in the House in February of 1872, they were really arguing that the land was useless. And this was very typical. If you wanted to preserve land in America, you had to demonstrate that it couldn't be used for any other purpose because people would want to come in. Right. And so and one of the things that he was supposed to do, one of his charges from Congress was to determine its agricultural potential, its mineral potential, its hydrological features. They wanted to know what waters had headwaters so that if, you know, if the federal government had those, they had riparian rights. So that was significant. And in fact, Hayden did discover the headwaters of the Yellowstone and the Green River and the Snake, which all feed into different waterways. And this is why he called it the heart of the continent Yellowstone. And so, you know, they wanted to know what the federal government wanted to know if the land could be developed. And so one of the arguments that Hayden was making was that, you know, the geothermal features in the basin and the kind of rough landscape of the eastern side of the park with the lake and then the river and the canyon really precluded any kind of development, any kind of ranching, any kind of farming, most certainly. And he knew because, you know, when they were in the firehole basin, they kind of ran out of food and they couldn't find there wasn't a lot of vegetation there. And that meant that there was not a lot of game animal action there. And so they were in a little bit of dire straits and they had to head back to Yellowstone Lake and send the wagon back to Bozeman for more food to survive the last part of the survey. What's interesting about that argument, there was a Republican from California, Cornelius Cole, who said, well, if it can't, if no one's going to do anything with it, then why don't we just let it be? Why would we have to preserve it if no one is actually going to develop it? Like he had a sort of laissez-faire attitude toward preservation. Just like let it let it go. If it can't be used, then no one will come in and use it. No one will destroy the geothermal features because they won't need to. But clearly, most of his Republican colleagues believe that people would come in. And in fact, the Hayden goes into a part of the Yellowstone Basin that he believes has never been seen before by white men and promptly runs into a bunch of miners kind of taking the waters and who are trying to, you know, heal themselves from rheumatism or, you know, from consumption. And this was a very common thing going to hot springs in order to to heal yourself. And so there were already people there who were going to try and develop it. So that was the other arm of the argument. A, it's useless. B, we have to keep these kind of small entrepreneurs out of Yellowstone or Yellowstone will turn into Niagara Falls, which had been so overcommercialized by 1870 that people just really there had been a lot written about how horrible Niagara Falls was because there were just hawker stands and everything was denuded and it wasn't beautiful anymore. It was this terrible commercial landscape. And so they didn't want that either. But yeah, if they had discovered gold or if they had found a way to use immediately the geothermal energy from Yellowstone, we might have a different picture of it now. But in that moment, Hayden was able to make the argument that not only was Yellowstone the most unique landscape. Probably in America and was unique in all the world, given just the thousands and thousands of geothermal features. He was he was able to argue that, which really appealed to Americans in the wake of the Civil War, right? They're looking for something to believe in about their country and to be optimistic about. You know, so so he's able to kind of make that argument. And then he's also able to make an argument that preservation will keep this area out of the hands of kind of grasping local entrepreneurs and we'll save it also for science so that we can kind of understand our continent better. And that was convincing to congressmen at that moment. It wouldn't always be. But it was at that moment. Yeah. So that's an interesting story that the idea that Yellowstone basically developed as a large part, or at least in part because of Jay Cook's sort of profit driven motives, although at the same time, sort of in a contradictory fashion, want to prevent it from being excessively developed. How do you get just how do you strike just the right balance? But in terms of thinking about the uniqueness of the landscape and its importance and representing America, whatever that might mean, I think it may still be true. Didn't Moran's grand canon of the Yellowstone hang in the US Capitol? Does it still as kind of a symbol of the American patrimonial? Can you say anything about that? Oh, yeah. Congress purchased that painting in June for June, 1872 for $10,000, which was the largest sum of money that had been paid for a single painting since Frederick Edwin Church's Niagara. And Moran was delighted by this, right? Like this was he lobbied. He lobbied. It was actually the committee on the library. So the Library of Congress was actually the body that purchased the painting. And, yes, they initially hung it in the Capitol, the US Capitol, where it was near to the Great Mural that had gone up during the Civil War, rest for the course of empire makes its way. And if you've ever seen this image, this is the very famous painting of just a surge of white settlers with some Calistoga wagons and oxen and horses going up and over this mountain range. And they can see the glory of the sunset over the West kind of beyond them, right? So so those two paintings are really working together to shape this idea of American conquest in the West, that that these are landscapes that prove that America is nature's nation, that, you know, we may not have all the great ruins of Europe that tell us that we have a storied past, but we do have Yellowstone and we do have Niagara and we do have these amazing natural features that no one else has in the world. And they prove that we are exceptional and special. So, yeah, so Moran, he painted the painting and he first exhibited it in New York. And then it went on a kind of tour of the East, including the Smithsonian, where it hung beside a series of George Catlin's Indian paintings, Indian portraits, and then moved to the capital. Right now, as far as I know, if it has not been moved, it is now in the foyer of the Department of the Interior. And I had a conversation. It was really actually a great opportunity. I had known about the Hayden survey because I studied it not in a history class, but in an art history class. Because of all the amazing visual culture coming out of those surveys, I studied it in grad school with Joni Kinsey, who was a great historian, especially American landscape painting and a historian of Moran. And so I got to have a Zoom chat with her about the Moran painting. And she had seen it in the Department of the Interior and was kind of outraged that it was there because it's in a kind of small space and that painting is so huge that it really needs to be in a gallery where you can see it from a distance and then walk up to it. And you can see it with a couple of other Moran paintings. That would be ideal. If you could see it with the painting that is the cover of the Three Cornered War of the Grand Canyon and then also maybe the Mountain of the Holy Cross. And they've been displayed together in that kind of triple image before and it's stunning to see them like that. But yes, that painting still belongs to the U.S. government and is one of its greatest treasures. Glad you mentioned sort of the importance of some coursework and getting you to think about some of these questions. Because I want to go to a loop back to your acknowledgments, which I'll confess, I always read first. Oh, yeah. The same way that one should approach the eating of dessert. You explained that you dedicated the book to a pair of your high school teachers, Ann Moore and Marlis Farrell. I hope we're getting their names right. Given the current scrutiny of the teaching of history across a range of academic settings, whether secondary, collegiate level, so on, could you say a bit about how their instruction inspired you? Absolutely. Yes. So Ann Moore was my history teacher and Marlis Farrell was my English teacher and I was lucky enough to have them twice. So for American history and American literature and then for AP US and AP American lit and just extraordinary instructors, the both of them from. And I still I don't know if you do this, too, but they always tell me they're like, call me and call me Marlis and I can't. It's Mrs. Moore, Mrs. Farrell.  And it was it was pretty traditional history.  But but she really helped me hone kind of my sensibility as a historian and in looking at all of the evidence and really analyzing it and then also honing my writing as a historian. And Mrs. Farrell assigned us projects that allowed me to play with form like we wrote our own poetry. We wrote about poetry and plays and novels.  There's a switch that kind of goes on in your brain, right? And so, you know, I read a lot of books with both of them and got exposed to a lot of different kinds of history and literature, which was really wonderful. But I think most importantly, they were teaching me about how to express myself in writing and how to argue effectively and to write really good tight sentences. So and I'm eternally grateful to them for that. And they also, you know, I mean, teachers are so important and we have learned this through the pandemic that teachers are our most valuable resource and they are also the resource that we devalue the most. And that's why I think as I was writing this, I was thinking a lot about them and I was thinking a lot about, you know, the kind of responsibility that teachers are given and that so many of them are so excellent and wonderful and they need more resources, they need more respect. They need us to really kind of cherish the work that they do and to respect it and to recognize it as something really important in our culture. So that's really why I wanted to dedicate the book to them and to all K-12 teachers who have just been doing heroes work over the past two years. Right. As the son of an English teacher, high school English teacher, I appreciate that especially, long retired. So I love your concluding thoughts at the very end of the book and this is what you write at the end of the epilogue, quote, the United States is both beautiful and terrible. It is both fragile and powerful and what lies beneath the surface of this nation is always threatening to explode. So I'm wondering how a book about one of the most beautiful spots in the country lead you to such an arresting conclusion, I guess I would say. Yeah, well, I really thought, have you been to Yellowstone? I have, but it's been a long time. Yeah, yeah, so it is, and I know we were talking before we went on air about how dangerous this place is, right? And we hear about this every year, people fall in, they go off the paths, they're not supposed to do, and they can fall into boiling springs, they can fall into geysers. Hayden himself broke through the surface into a mud pot and had the boots burned off his legs during this. And I was so struck by the descriptions that all of the explorers were putting down in their diaries and letters, talking about how when they were on horseback and then when they were on foot, they would hear the echo of their feet or the hoof prints because the ground they were moving across was hollow. There was nothing beneath it. And I was like, that is terrifying. And so I've always thought that Yellowstone is such, I mean, it's such a beautiful, amazing place, but it's also a really weird place, right? And it was Phil Sheridan who called it, you know, this strange country. Like it's really strange and part of that strangeness is that combination of being beautiful and terrible. And so I was really thinking of Yellowstone as I was writing as a place, but also as a metaphor for the nation, right? That perhaps, you know, Congress wanted to save Yellowstone because it was so unique and so special, but also maybe because they had that sort of sense that it really did represent the country, that it had these beautiful, amazing components to it, but it was sitting on top of a caldera. It was sitting on top of an active, what we now call a super volcano. It was very close to the surface and that meant that you could kind of see those forces that had created the earth coming out of the earth and creating these features. And that that interplay of what we can see on the surface and what lies beneath is really the real stuff of history and of American culture. So the one time that I was there was in the late nineties, maybe early 2000s, that I remember seeing, I guess, evidence of the fires from 1988. I also remember sort of being, I did get off, well, not off the paths, but I did get out of my car and do a little exploring with a friend who took me. But I'm wondering if you could say a little bit about the current state of the park today. When I was there, it was bumper to bumper on the main roads. And there's a worry I know among some that it's been loved to death. What's your sense of how things are at Yellowstone on its 150th anniversary? Well, I mean, they have had a huge uptick in, in visitorship, like I think by 30% during the pandemic, because, you know, people can't travel internationally. So they're going to these places in the United States that they've always wanted to go to. And they've moved up a slot even in the most visited parks list because of that. And they have, you know, more than 4 million visitors a year and they're all almost all visiting in a very short window between mid-May and about early September. And there really are only these two big loop roads and four entrances. And those roads are two-lane roads. And so it really can get jam-packed, especially if people are pulling off to the side of the road to look at all of the amazing charismatic megafauna, the bison, elk, the wolves, bears, you know, that are all out for us to, to view and hopefully not approach. But, and I know that there was a climate study done and published in July of 2021, talking about Yellowstone Basin and the greater Yellowstone ecosystem of which it is a part, which is one of the largest, mostly intact temperate zone ecosystems in the world. And so it's a really good place for scientists to study something like climate change. And what they've determined, I think wouldn't surprise you, Andy, as you live in the West, the summers are getting hotter and drier. There's less snowpack, many, many more raging wildfires.  It was terrible. And really only kind of pushed out in those early weeks in September, but what they've determined is that what we're gonna see then is a water table that's getting lower. There's not gonna be as much water in the rivers. We'll see changes in the geothermal features because those are all fed by groundwater. So the water kind of goes down into these subterranean systems, gets super heated and then shoots back out as steam and water and geysers, right? And if you have less water, then those features, old faithful may not be going off every 90 minutes. It may be more like two hours or two and a half hours or something like that. So we are seeing real change in this place. And it's a good place to monitor it, but it does mean that Yellowstone is still a really wonderful scientific laboratory for scientists studying all kinds of things. And I do know that for the 150th year, the park is making a much more concerted effort to bring in indigenous nations, 28 of them that have historic relationships with Yellowstone Basin, because for thousands of years it was used as a thoroughfare, a camping site, a hunting ground. There was indigenous presence and the roads that you follow through Yellowstone now are following indigenous pathways, right? So even though no one nation claimed it and it was not negotiated out by treaty, it was an indigenous space and Yellowstone National Park, it has several initiatives. So there's going to be a Native History Center established at Old Faithful, which is the most visited place in the park. And they are also doing an installation of a crow village at the Northern entrance. And they're initiating a lot of other projects, I think, when consulting with indigenous groups to really, I think, create a path forward that will engage with the park's history in a much more, in a kind of full way, right? That engages within acknowledges that complexity and the fact that this was contested land. I think we're close to the end of our time, but I have to ask you an obligatory question, but one that I'm very interested in all the same and that many of your readers will want to know the answer to as well, which is, what's next? Well, what is next is I'm staying in the West, Andy. You'll be happy to know, I came over from the Southeast for a three-cornered war and now I've stayed in the West. Yeah, so the next project is called The Westerners and it is in part a reply to what I think is still a dominant myth, despite our best efforts in the Western history community, a still dominant frontier myth that really drives a lot of our popular understanding of history and also popular culture, Westerns in particular. And it's going to have a similar kind of approach as the three-cornered war and saving Yellowstone in that it will be following probably seven or eight people, but the time span will be much longer. So it's going to start in the early 1800s and end in the 1880s. So a much broader sort of synthesis of US Western history but focused on the people who I'm calling the real pioneers, right, the true pioneers, not just the white families in the Calistoga wagons moving from East to West, but men and women from all different communities who are moving in all different directions during the 19th century. Oh, I can't wait, sounds great. Nice long stretch. Since I know some of us, myself included, have tended to go a bit more micro. I love something that opens the way that you describe and is a lot more macro. I don't think we have any questions from the audience and now that it has passed to a plot East Coast time, I think we're supposed to wrap up, but thank you for a most stimulating conversation. I really enjoyed it. Yeah, thank you for joining me in this. It's so good to see you. I hope I see you in real life. Well, I am gonna see you in real life. I'm gonna see you in a couple of weeks here in Boston. On your territory. For those who've not yet read the book, I will make a pitch for it and say, absolutely worth your while to take a look at Saving Alistone. It's a really elegantly written and really compulsive narrative.  I wish we could all say the same about all of us who endeavor to write American history. Oh, thank you. Thank you. And if I can just pitch here at the very end, if you want to know more about the book or about the work that I do, you can go to my website, which is www.megankatemelson.com. And there you can also click on buttons to buy the book in any format. It's in hardcover. It is in ebook and it is an audio book if that's how you prefer to do your reading. Wonderful. Thanks so much. All right, thanks Andy.


We are representing the data science group in United Airlines. Today, we will be walking through one of the use case that we worked in video analytics space. It is actually calculating deep learning times using deep learning. Just to give a quick introduction about the speakers, my name is Vikas Krover. I am senior manager in data science team at United Airlines. I have a colleague with me, Jayit, who also works in the same team as analyst data scientist. Just a quick disclaimer, this is not an official position of United Airlines, so don't quote it anywhere. Really quickly talking about the agenda, we will be starting with a quick introduction about our group. Then we will define the problem statement, giving the complete context why this problem is important in the context of United Airlines. Then we will talk about computer vision and deep learning concepts which we used for solving our problem. Then at the end, we will do a deep dive into how we approach the solution and there is a quick demonstration of the video itself. At the end, we want to leave 10 minutes for Q and A session. I am hoping to get a lot of questions. So at United, we are in the business of uniting the world and we are continuously transforming United by growing our fleet and our network. Even today, we are the world's most comprehensive network catering to 354 destinations and 48 countries. We are continuously investing in our product and in our people and our focus areas is to provide the best experience to our customers and also run most efficient operations. So talking about the data science group, so we deal with big data since United Airlines is one of the biggest carriers in the world. There is a lot of data both in commercial space as well as in operation space. So we deal with cloud and edge computing software as well. Talking about the use cases. So data science group is involved in both commercial and operations use cases. So we work on optimizing the digital journey of a customer. Then we also have specific use cases in the operation world where we do predictive maintainers and make use of natural language processing and social media data. Today we will be talking about one specific use case in computer vision world. So let us define the problem statement. As apparent from the title of the presentation itself, we want to calculate deep planning time but why is it a problem in first place? So let us look at the flip side of deep planning time which is enplanement or in simple words onboarding time. So it is very easy to calculate enplanement time, why is that? Every customer has got a boarding pass and at the time of boarding, gate agent scans the boarding pass. So the total time that it takes to board the first passenger until the last passenger is called enplanement time. It is very easy. Once you scan the boarding pass, it is in the system. From system you will know the exact view of enplanement time. But what are our options for deep planning time? One option is we can do a manual time keeping. So we can have an agent who will start the timer as soon as the first passenger is out of the aircraft and will stop the timer when the last passenger is out of the plane. But as you can imagine it is not going to be efficient, it is not going to be accurate all the time. The other option is to scan the boarding pass once more similar to what we did for calculating the boarding time was not a great customer experience. And since we are a customer facing industry, we do not want to do anything which affects the customer experience negatively. So that is why we as a data science group came up with this elegant solution where accurate measurement is possible. And so this is the problem statement. We are using the video feed at the gate to accurately measure deep planning time. So even now we have the cameras installed right at the gate of all the aircraft. So we can use those video feeds to calculate deep planning time. We have defined the problem statement, but so far I have not talked about why this is so important in United Airlines. To understand this, let us understand concept which is specific to airlines industry called aircraft turnaround. Some of you might already know this, but for benefit of the people who are not so aware about the airlines industry, I will explain this briefly. So United Airlines actually operate using hub and spoke model. So this is different than the point to point model that usually Indian airline carriers operate in. But since the scale is so large in United States, we cannot operate in point to point to point model. And you can see this with the help of this diagram. So on the left side we have hub and spoke model. So for connecting nine points, a hub and spoke model only requires eight routes. Whereas to connect the same nine points, we require 28 routes in point to point model. In India, point to point model is fine since we do not have too many destinations to serve. But since United Airlines is the business of connecting so many remote destinations and we want to give the opportunity to our customers to connect from point A to point B, does not matter where those points are in United States geography. So hub and spoke model is the most efficient way to plan these routes. So now we have established that United Airlines work in a hub and spoke model. And in a hub, you can imagine that there will be lot of aircrafts going into a hub and lot of aircrafts going out of hub. And a lot of times it will happen. The same aircraft which is coming to the hub will be required to be ready for the next departure flight. Since we have lot more flights than the number of fleet. So the entire time that it takes for an aircraft to be ready for the next departure flight from the time it arrives at a gate on a hub, it is called turnaround. And all the events that constitute this turnaround time are the events that have to be performed. So deep planning time is one of those initial events. If we do this quickly, then we get a heads up for rest of the events. The later events can only start when deep planning event is finished. What happens if this deep planning time is slow or we are taking a lot of time for deep planning? As now we understand it will affect the turnaround time of the equipment. So it can happen that your next departure flight might be delayed and we do not want that to happen. From a customer perspective it will be a negative impact. So if deep planning time is large, then you will be pissed off as a customer. Even if the rest of the flight is great, you would not care. And it can also impact the next departure flight. So in urban spoke model, people have to change their aircraft to reach to their final destination. So if the connection time is really tight, so let us say you are arriving at a hub at 9 am and your next flight is at 9.45. If you are not able to reach gate at time, then your next flight is missed. And as United Airlines will have to make alternate arrangements for the customer. So it is a increased cost for us. So I think now you can appreciate the fact that doing deep planning in the most optimized fashion is very important to United Airlines. But we are talking about the first step in this presentation. How do we get the accurate view of deep planning time? Once we have an accurate view, we can go to the next step where we will look to optimize this deep planning time. And we can figure out the opportunities where we have to optimize and there can be numeric use cases and it can have a very large impact. So in this presentation, we will be talking about how we are planning to approach getting this accurate measurement for deep planning time. And we will be making use of computer vision concepts and deep deep learning concepts. So Jit will be walking us through this journey over to you Jit. Thanks Sikas for building up the context. So let us talk about computer vision. Computer vision is all about making sense out of the visual data. That is giving computer enough ability to visualize, perceive and gain a high level understanding of the videos and images that you have seen. So before moving on, let us quickly talk about how the history for the computer vision have been for the last couple of years. So as humans, we have always tried to solve two fundamental problems in computer vision. First one being making, creating out the good mechanical vision that is camera. So historically, if we go forward, go backward, then we can see the first, the first computer mechanical vision was inspired out from something called a spin-hole camera. And pass over to where we are right now. We have got the cameras that can easily fit it into our pockets and we can easily record out the videos. Right? Second fundamental problem is, right, we have got the data. How we can make sense out of the visual data? So what can be the possible applications for it? So historically, if we go backward, then we can figure out the applications can range from as simple as doing hand-written digit recognition for the zip code that was suggested by Professor Yan Likun in 1996. And pass over to where we are right now, where there are multitude of applications in computer vision that we would be probably seeing in the next future slide. So there was something called as ImageNet movement in the computer vision industry. So this ImageNet competition was hosted by Stanford. So the high-level idea was, Stanford came up with a very big data set that had around 1.4 million images and around 22,000 different categories. So it was a de facto movement in the computer vision world. So a lot of different, different research community came up with different algorithms. So ImageNet hosted out a competition called ILS3R, which is Image Scale Large-scale Visual Recognition Challenge. So if you can see, if we have a look at it, it was until 2012, where this dropped in the, what do you say, the error percentage was significant until 2012. And this solution was inspired from what we call as the deep neural architecture. It's the first deep architecture that gave genesis in 2012. And it was until 2015, when we first surpassed the human level accuracy, that of 5.1%. So in 2015, we have already surpassed the human level accuracies in the image, different, different image analytics tasks might be. Image classifications, image captions and all those things. So why are we even talking about computer vision? So there are two factors for it. It's possible because of the data that we have got and the powerful computing that we have got over there. So most of us present here are canning a smartphone and it would be a good, that has got a one, two or even three cameras in it. So it would be a good approximate to say that there are more number of cameras present in this conference room as compared to the total number of people, right? Apart from it, by the time I just counter to five, there is already 50 GB of data that's already generated on YouTube. So we will think of the enormity of the data if we include all other data sources as well. It might be YouTube or it might be Netflix and all those things. And apart from it, we have got the data. We need powerful computing. Image data in itself is very, image and video data in itself is very time consuming and it's very complex. We need a lot of compute power to actually make sense out of this. So thanks for the hardware community as well. We have got a multitude of different solutions for it. Probably we can use a CPU for a less influencing devices to the GPU for a, where the influencing cost is more to FPGA and TPU for, let's say, edge influencing and those sort of things. So now let's quickly look at some of the very key concepts in the deep learning for video analytics. So before proceeding further, what is a video? So video is nothing but a continuously moving image. Anything that you can apply to a single, what do you say, image, you can easily extrapolate that particular idea to a video, right? You can probably dump out the frame sort of the video, run out your influencing pipeline on that particular image, and you can easily extrapolate it to the video. So anything that can be done to the image can easily extrapolate it to the video. So there is one very fundamental, what do you say, visual recognition problem for image classification. So let's have a very high level idea of what it is. So let's say we have given an input image, input labeled images of cats. We have some input labeled images of dogs. So there is some human that have marked that this particular image belongs to a cat class, this particular image belongs to the dog class. And we are passing on this images to this fancy looking neural network, what's called deep neural network, you can probably think of. So this image is getting flown through this particular deep neural network and the neural network is learning out that, yeah, okay, I've got it, it's actually a cat. The weights are going on in and the network is learning it out. So it got it, it's a cat. Okay, let's go out for the second iteration. We give the second image and we figure it out. We give it to the neural network, image came, come in, converted into the normal clones versus deep neural network architecture. And it says that it's actually a dog. But my neural network says it's actually a cat. But it isn't it, it's a dog, right? So what's happening is something called as back propagation. So the neural network is refining itself to learning all these representations and encapsulating out that this particular image belongs to the dog class. So all these different, different, there are some high level, what do you say? Computations that would be happening and the back propagation would be happening, weight would be propagating out through this neural network and it would be updating depending on the context. So it's learned out, okay? So the high level idea is if at first you don't succeed, try another billion times. This is the, this is what image classifications are now using to work on. So, right, so there are some other visual recognition problems. We're gonna try to encapsulate it in the airline context, what we can probably do. So there's something called as object detection. Object detection is something where we have bought our input image and we have got a multiple object in that particular image. We are actually localizing on that particular object. So let's say there's a person standing on the ground of the aircraft. So there's a person over there, there's an aircraft, there's a baggage loader. So it's uniquely identified that this particular morning box belongs to this particular class. There's a person class, there's an airplane class, there's a, what do you say? Again, second is instant segmentation. So in instant segmentation, we are classifying each and every pixel in the input image to what do you say, the predefined class. So we have got three classes over here. One is the person class, one is, let's say, a background class. Third is, let's say, an airplane, an aircraft class. So we are pixelating and kind of just mapping each and every pixel to that particular class. So we are saying that these particular pixels belong to a person class. These particular pixels belong to the aircraft class. Now let's look at action classification. So in action classification, what we are doing is something related to object detection, but we are kind of digesting out what actually the particular object is performing in that particular image. As you can see here, the gentleman sitting out over there is probably walking out. So the network is saying that there's a person and there's a person who's walking out and doing something. So action classification. There are other problem is image capturing. So in image capturing, we are given an input image and the image is digesting out, understanding the context and giving you a story about that particular image. So there is a caption that the model is interpreted out from this particular image. Last one is semantic segmentation. So it's somewhat related to instance segmentation. Only difference is for the similar kind of object, it's uniquely identifying them. So there are three person class. There are three persons belonging to one class, that's a person class. So it's uniquely identifying them. There's a person on the left, that's person one. In the middle, it's person two and in the right, it's person three. And there's a class, it's uniquely identifying them as well. So in our model, we are using something called as object detection. We are localizing all the objects that are presented in that particular. So moving on to the deep learning framework and the considerations that we did have. So as you know, world is always full of choices. So we did also have this scenario where we had a lot of, what do you say, alternatives to figure it out, what convolution you're in at what we can probably go forward with. So there are a lot of considerations that you need to make up, right? So you need to worry about how, you need to take care and take into consideration how the trade-off between speed to detect and how's the accuracy considerations as well. So you can probably come up with a very, what do you say, complex architecture and can just throw it out for the influencing. But it would consume resources, right? You need to worry, you need to take into consideration those factors as well. The considerations is the resource ability. You need to take care of how's the time and money, because when you're going out for a very complex model, you would need to pay associated costs as well. There's something called as influencing costs, right? So you need to take into consideration those factors as well. Other considerations are, there are a lot of pre-trained models available right now. So we have got Cafe2, we have got Pytos, we have MxNet, N-Reroids and others. There are a lot of different architectures that are available out in the market. You need to drill down to select out the one thing. So we move forward with selecting Cafe2. And apart from it, you need to worry about the scalability issues as well, right? So for a company like United Airlines that have got so much rights for presence, we need to take into consideration. We can come up with a simple solution, but you need to take into consideration how would scale up for different halves and different airports and all those things. You need to take into consideration that aspect as well. So some of the possible options that we drill down to was using RCNN, pasta RCNN, pasta RCNN and YOLO NFS. So there's a graph, there's a nice graph that talks about how the relation between GPO time and the overall map for multiple algorithms that we have got. So we want to drill down to something where we have got a model that has got a less amount of G2 time, and at the same time, it has got a considerable higher overall map. Map is a mean average position that we are talking about. So we drill down and settling up to somewhere in SST. So here's something, SST that has got a less amount of G2 time. And at the same time, got an overall considerable amount of map. I wouldn't say too much higher, but it's considerable. We can just think of those. So let's look at how the pipeline and what are the evaluation metrics that we are using. So again, to reiterate on what Vikas said, so the deep learning time is the time difference between when the first person was detected in the frame to the last person detected in the frame, the time difference between them, that would be called as a deep learning time. So we need to take into consideration what possible gate you would be having. We have got thousands of gates. We need to drill down to one particular gate. We can probably run out the solutions and figure out how is it doing. So we need to take into consideration the homogenous fleet aspect as well. There might be multiple fleets. There might be multiple aircraft that's coming into that particular gate that has got might be that 180 passengers, capacity might be 250 capacities and different, different aircraft that might be coming in. So we selected out a gate that serves to a homogenous fleet. So there's an apple to apple comparison that's happening also. I promise we need to take into consideration the available infrastructure as well.  You need to figure out something where you can use your available infrastructure perfectly. So we drill down to selecting out that gate that has got the most high-resolution camera and at the same time, the position of the camera was perfectly right. So there isn't any issue of just going through and manually setting up all those things. So the application uses a video source to graph frames and DNN is being used to process of this data. OpenCV is being used to stretch out different DNNs into a single pipeline that we have built out. So right now our model is using phase detection and head-force estimation and it's taking care of the duplicate phones when a person is coming out of the plane. So there's an aircraft gate and the person is coming out of an aircraft. So our solution is detecting out the phase that there is a phase over there and it's dumping into the database. But when the person is going into the aircraft, there isn't any influencing. Since the phase isn't visible, so there are some security persons of gate agents and all those, just going into the aircraft for cleanliness and all those things. We need not to worry about those things. We need to take into consideration passengers as a fact. So since they are going into the aircraft, no person would be, no phase would be detected out. So they would be kept aside from what do you say influencing out. So the duplicate count issue wouldn't be there. So this is a very high-level overview of what our pipeline looks like. So step one is frame extraction. So as I said, video is continuously moving images. So we are extracting out the frames from the video and we are using OpenCV library for extracting out the frames. Once we have the frames, we are passing on to the second module that's the influencing request module. So it's running deep learning algorithm to just infer out the different different, what do you say, the rivers. For this we are using FAT1, the deep learning framework we are using, again based on assistive, we are using something called a skew's net to take into consideration the model weight, weight, files and all those things. And apart from it for head for estimations, we are using a CNN architecture. So once we have these results, once we have the influencing request and we have the results, in step three, we are storing these results into the database. So we are dumping all these results in the real time into the database and we are using a visualization tool for making out the dashboard that you would be seeing. So for the database, we are using something called as Grapna. For the database, we are using in plus DB and for visualization purpose for building up the dashboard, we are using something called as Grapna. So both of them are open source, you can probably try it out for your applications as well. Very nice solutions to work on. So now let's quickly look at how is the evaluation matrix. So it's something different here. So there is something called as IOU. IOU is Intersection over Union. That is area of overlap upon area of union. So area of overlap is let's say we have a green box that was actually a particular object that was present in that particular image. Red box is what you detected it out. So the union between those, the common area between those would come in the numerator and the intersection of entire area would come in the denominator. So as you can see, the poor approximation would be when we are not kind of overlapping perfectly the actual boxes. It's a poor IOU, right? The good IOU would be when you are just trying to encapsulate. You know, not perfectly, but it's somewhere in the between. So not so good, not so bad. Excellent IOU would be when you are perfectly encapsulating all that particular object in that particular image. So it's an excellent IOU. Depending on IOU, we can define TPSP and FN. So TPSP is now here we need to take into consideration the threshold aspect. So threshold is something that you can probably play out with. It again depends on what application you are using, what application you have got. And we can change it accordingly, according to our application. So once we have the TPSP and FN, we can easily calculate out the precision and the recall, right? So the solution that we have built out, we tried it out on a publicly available data set for wider. So it's giving around 83% on the wider data set and it has got a multitude of amount of faces over there. But in the real time, when we tested it out, it's fairly larger than this. So it's the only faces that are visible on the video frame, videos that you are working on. So now let's look quickly, look at the demonstration. So before proceeding in, let me just give you what you would be probably seeing out over there. So these are the frames that are extracted out from the video. So at frame one, there was total number of people detecting the frame was zero. It goes up to till, I'm just hypothetically making it up. Don't take it too much seriously. So let's say frame 110, we got an event at that particular time stamp. There was a 1% detected in the frame. It would be marked as a deep learning start event, right? Once it's keep on happening, keeps on happening until that point that, let's say at frame 55,000. At that particular time stamp, we got a single iteration. So this is keep on going. Let's say after this, let's say at frame 60,000, we got a zero person being detected in that particular frame. But we cannot mark it as a deep learning end event, right? So there might be a time differences where the persons are coming out. So we would wait out for some threshold amount of time before marking it as a deep learning end event. So this threshold amount of time is defined on a lot of factors. There might be the free type you're probably serving in and apart from the business knowledge that we have got. So this is the deep learning end event and at this time, the bridge is completely empty. So this is a video that we would be showing you in of our solutions and actions. So there's an aircraft door where the camera is installed out. So the persons are coming in. As you can see, the faces are masked out for the obvious reasons, since it's of AI information. So there's a warning box detected on each and every face. And there's some text written out over there called packed captors. So the passengers are perfectly captured and stored in the database and it would mark as that the passengers are correctly being captured out. So at the same time, and the influencing is going on, the results are being dumped into the database. You can use the database for different visualizations of the things, right? So we have got the deep learning event. We have got the results now let's look at how the dashboard look like. So this is what we built out for our internal consumption. So it's a visualization that you probably see. So on the top left, there is a total number of unique passengers detected. So it's currently 72. We have got, we can encapsulate what the current gate is. So there might be multiple gates that this solution might be working on. So currently saying that this particular gate the solution is running on. The next is what's the current passengers detected in that particular time sense. So it's a real time, real time, what do you say, the statistics. So there is one person detected at that particular time sense. This is a deep learning status. So as you can see the deep learning event is still going on. This green spike tells you about that. And apart from it, there's a data where we can even figure it out if there is, let's say, crowd on the bridge. If there is a crowd on the bridge, there would be a peak that would be happening here. And we can easily figure it out. There is some crowd and we can take accordingly, we can just do, necessarily get trapped over there. This is the real time, what we are dumping into the database. So according to the time sense, we are dumping these number of people were detected and we are dumping into the database. And this is the kind of a historical deep learning time. For this particular gate, what was the, for that particular flight at that particular gate, what was the deep learning time that is stored? So these are all the results. So this is for our internal consumption that we are using it out. So what do you have with us? So thanks for coming for the session. Before we open the floor for questions, you can reach out to us on LinkedIn. Both of us have put our LinkedIn contact here. We are actively looking out for people to join our group. So let's start the question and answer session. Yeah, so that's again manual. Yeah, so the question is good that we have started using video analytics and deep learning in airlines industry. But for this particular use case, isn't it an overkill to utilize video feed. Can't we just use the airline agent's entry when the gate is open, when the gate is closed? Yes, you are right. So that's how it is being done currently. But again, it's a manual process and it is susceptible to errors. And we have seen that we can't rely on that information if you have to go about optimizing our turnaround time events. So that's a major idea. Since we already have the video feeds, why not make use of it? So it's not like we have to come up with the new infrastructure altogether to come up with the solution. And another suggestion is that we can use on the feeling when they have to apply it. Yeah, we can do that. The only problem with that is your customers should give a buy-in. So since here, one thing to point out here is we are not use storing this customer's information anywhere. We are just utilizing the frame information for calculating the deep learning time. When the first person is directed and when the last person is directed. Other people have also questions. Thanks. Yeah, good question. So the question is did we create value out of this solution or not? So this is the first step. We have reached out to stakeholders, to hub managers, and they have liked the solution. Now the next step will be to implement it on various hubs and see it in action. So this is the first step where we have gotten a really good approach to calculate the deep learning time. Now we'll go about where we can optimize it. And essentially first we have to figure out where the opportunity is. So did you look at tracking people from frame to frame to further reduce your double counting? Yeah, we are using some sort of a threshold. So we are in, let's say there are five frames and I'm just hypothetically making it up. In a one second there is a five frame. So we are not probably capturing it out each and every frame. So it's after every threshold amount of time we are capturing all the frames. So were you assigning a unique identifier to those people as they go? No. So the genesis is we are worried about what do you say, the flow of the passengers, right? So even if you're not accurately trying to measure it out, what's the total number of persons detected or not? And there would be some duplicate count issues as well. But at the last of the day we are only worried about the flow, right? At t to t plus one second if we got the flow, that this is the flow. And there is a significant drop in the flow at this particular time. Then we can easily encapsulate that this is a deep learning and event, right? So this is how we are approaching it right now. Do you install some specific cameras for, to do this task or do you use TTTB to tell them something about it? Yeah, so this, so this solution is actually pretty robust. It doesn't require installation of new cameras. So whatever cameras we have, CCTVs and there are hubs where we have sophisticated cameras as well, it can utilize it. So since we are not, so objective is to not calculate the number of people who are coming out. We have to calculate what's the deep learning time. So if you think about it a bit more deeply, you can say we can live with some error rate. So that's why we don't have to install sophisticated cameras. Apart from it, there is a concentration as well, right? When the video is being captured out, there is a, let's say it's very low FPS when they are saving it all, right? If you are saving the video at a very high FPS, then there would be a significant amount of space is there, storage concentration is there. Yeah. This is the beauty of the solution. This is the beauty. We are using that particular. So you are saying it's ethical to use it. Yeah, so I think you have got a good question. So as I said, this is a POC. It's not implemented. I'm sure if there are considerations like that, we'll get into it. But I don't think that should be a problem since we are masking everything. So and we are only utilizing it to make our operations efficient. Yeah, one question here. Are you differentiating between passengers and the crew? So there could be a time lag between the passengers coming out and then this crew slowly coming out a little later. Yeah, so. So you take care of that. Yeah, right. So all our queues actually have a vest. So we can easily differentiate between queues and the passengers. So here we haven't done it, but that's very easy to do. So we can, yes. So if we have enough videos and we will, we have got enough videos actually here. So we'll be differentiating between passengers and the crew and that's very easy problem. Yeah, I can continuation of the earlier question. So there could be various category of passengers like we share bound and women with kids and all of that. So your basic assumption is that all passengers will take the same amount of time to come out or how is your distribution? No, so that's not the assumption at all. So if you see in the dashboard here, although there is no crowd here right now, but we want to track this as well, whether there is a crowd on bridge. And because it can happen that due to some considerations where the people are not moving out faster. So we want to take care of that also. So the idea here is to calculate the entire deep time time irrespective of the distribution, whether people are moving at the same pace or they are the pace is really different. So it will be fine. We just want to detect the first person and keep detecting until there is no person in the frame. Then we'll say the deep learning time is ended. Thank you.


And I'm an undergraduate Indian student studying in National Institute of Technology, Durgapur. So today I'm going to speak about my GSOC 2016 project that I did with Public Lab and some methods that we designed for making software outreach. So a little about Public Lab. Public Lab is a community and a nonprofit democratizing science to address environmental issues. We make some low-cost DIY techniques to analyze environmental data and make it accessible to common people to use it. Here's some tools and techniques we have developed like the time-lapse kit, a spectrometry and using balloon and kite mapping and dust we know, and multispector imaging. We basically take very common materials that are available like some CD or wooden equipments and just make it out so that it can be easily accessible to common people. And here's some Public Lab software that we use for analyzing those data. Like Plots2 is basically the main content management system, Public Lab.org. It is basically a research-based content management system so that we can write down research content. Spectreworkbench is a project that is used to analyze spectrum data that we get from any spectrometer. And MapKinator is used for designing local maps on a community. And Webjack is a project that defines, that is, used to directly transfer audio data from the, some micro-handler like Arduino to directly a web browser. And there is Infragram. So let me start with my GSOC project that I worked during this summer. That is the Plots2 CMS. The Plots2 is actually the main content management system of Public Lab.org. It is a research-based content management system that is designed in Rails. So, giving a brief point, it uses a software stack. The backend is on Ruby on Rails. And the front-end, that is, the design is a Booster-based design. And we use a separate module of Node.js for making the editor content. So here are the key features of Plots2. What you see is what you get on Magdown-based editor. A wiki-editing and revision-tracking system, a tagging and tag-based organization system, a question-and-answer system, and activated and upgraded grids. So the last point, activated and upgraded grids is a very important feature of Plots2. That is, using some small parts of code like, suppose you want to post an activity of a spectrometry. So you just write activity spectrometry, and it will link to all those research nodes that are related to spectrometry in that content management system. So we are now trying to modularize the code a bit. So we have separate projects of the editor like, where you write your research content. We have shifted it to a separate project on GitHub, in other GitHub projects like the public lab editor and the inline Magdown editor. So what's unique about this project is that it separates it from other content management system. So it is the power tags. So power tags is the most powerful feature of the CMS. So basically power tags are very common tags that are used anywhere in the content management systems. But they use a special format, that is a key value format. The key explains the feature you are explaining. Suppose you want to write a question and have some extra features. So we have specially designed a power tag for question. Whenever you put a tag such as question colon some name, so it differentiates it from a normal research node and it labels it as a question. And there are other features like creating events. Suppose you want to add an event or you want to add a date to a research node when you are posted and it shows out a calendar. So just add an event colon the date tag and it shows out a calendar beside the event. And suppose we have a map-based content on that too. Suppose you are posting a research node on some location, some location-specific content. So you just give the latitude colon the latitude coordinates and longitude colon longitude coordinates and it just displays a map on the research node. And there are activity and updates that I have told earlier. You just put a tag like something activity colon the name just in your editor. This is a special kind of power tag that is used. These are known as inline power tags. So whenever you are writing a content suppose you want to post or show a table of other contents that are related to that. You can just put an inline power tag like activity name and it just lists out all the content related to it in tabular format in that. So now I am going to speak about some software outreach for the techniques that were developed during my Dishog days. So when I started contributing documents in my personal experience I found it hard to find communities that are really welcoming. So how to make outreach so that you can welcome newcomers and expand your community. So here are some good points. Like we started with the first-timers-only project. So we labeled each issues with the first-timers-only label. It is basically a special type of issue that very newcomers like who don't have even contributed to open source can take up. And here is a sample first-timers-only issue. So I just want to say that this first-timers-only tag is a special tag and a Twitter bot is there which tweets about all the issues met with this first-timers-only. And we followed a break-me-up technique that is breaking out larger issues into smaller ones so that newcomers can approach each part of the issue. And this is the most important part. Collaborate with like-minded projects. Like there is a website called first-timers-only and offer grabs that helps first-timers to find new projects which is welcoming to them. And say thanks. Like whenever you try to merge a poll request say thanks and take some time even you can tweet to them and participate.  Do participate in them and just expand your projects. Here are some stats like how you have reached so many contributors within a few span of months within like one year. Thank you. Any questions? Okay.